emqx k8s Cluster fail to add pods

[and-dzh3]

Environment

• EMQX K8S Cluster
• emqx:v4.1.0

Description

After few Kubernetes nodes have gone and newly added, (due to running on spots instances) emqx pods go to restart loops .
Many of these logs
([email protected])1> 2020-09-28 16:04:01.021 [error] Mnesia('[email protected]'): ** ERROR ** Mnesia on '[email protected]' could not connect to node(s) ['[email protected]']

Only after recreating manually, 1 or 2 affected pods issue resolved

[terry-xiaoyu]

@and-dzh3 It shows that it cannot connect to [email protected], may because the k8s network is not ready, or the node names have been changed.

[and-dzh3]

@terry-xiaoyu network ready and everything looks good about Kubernetes.
I added to config EMQX_CLUSTER__AUTOHEAL: "on" looks better, but ... why auto-heal is not on by default? and why the error messages generated are not informative?