Embed etcd cluster mayhem

[Tochemey]

Hello,

I really don't know whether this is a bug or not. This is the scenario to what I am experiencing. I start an embed etcd cluster of 5 pods on k8. I scale down the deployment to 3 and I started observing the following log message:

023-05-18T12:31:28.921Z | {"level":"debug","ts":"2023-05-18T12:31:28.921291Z","caller":"rafthttp/stream.go:570","msg":"dial stream reader","from":"75efc854671362d7","to":"ea075d516e9254b4","address":"http://10.244.0.146:32380/raft/stream/msgapp/75efc854671362d7"}
2023-05-18T12:31:29.240Z | {"level":"debug","ts":"2023-05-18T12:31:29.2401Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"c5444e9ce8ab8cde","error":"failed to dial c5444e9ce8ab8cde on stream Message (dial tcp 10.244.0.144:32380: connect: no route to host)"}
2023-05-18T12:31:29.240Z | {"level":"debug","ts":"2023-05-18T12:31:29.240182Z","caller":"rafthttp/stream.go:570","msg":"dial stream reader","from":"75efc854671362d7","to":"c5444e9ce8ab8cde","address":"http://10.244.0.144:32380/raft/stream/message/75efc854671362d7"}
2023-05-18T12:31:29.240Z | {"level":"debug","ts":"2023-05-18T12:31:29.240222Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"c5444e9ce8ab8cde","error":"failed to dial c5444e9ce8ab8cde on stream MsgApp v2 (dial tcp 10.244.0.144:32380: connect: no route to host)"}

All embed etcd server started with the same initial-cluster and token with cluster state as new. My expectation was that the cluster should auto-adjust.

I am sure I am missing something here or doing something wrong.

[jmhbnz]

Hey @Tochemey - Thanks for raising this question. When you say:

I scale down the deployment to 3

Was this purely by reducing the number of replicas for example on the k8s side, or did you also update the etcd cluster configuration to remove the peers that are no longer desired?

Additionally can you please confirm what etcd version you are running?

[Tochemey]

@jmhbnz I only scaled down the deployment from 5 to 3. I am using the etcd v3.5.9.

[jmhbnz]

Hey @Tochemey - As far as I know when you scale down the number of pods in your etcd deployment from 5 to 3, the etcd cluster itself will not automatically adjust. Scaling down the number of pods only affects the number of running etcd instances, but it doesn't update the etcd cluster membership.

To properly scale down the etcd cluster, you'll need to manually update the etcd cluster membership to reflect the new desired size of 3 members. We have an example here that includes removing a member: https://etcd.io/docs/v3.5/tutorials/how-to-deal-with-membership

[Tochemey]

I am programmatically starting the cluster using the embed etcd. Do you mind guiding me on how to go about that?

[jmhbnz]

Ahh the embedded etcd side of things is definitely not my area of expertise sorry! One of the other @etcd-io/members might have some ideas here.

[Tochemey]

I would like you to point me some folks in there who can help me figure out this. Thank you so much for jumping to help. I really appreciate it.

[ahrtr]

Just scaling in/out etcd instances (deployment replica in your example) isn't enough. The deleted etcd instances are still in the member list, so you should also remove the member from the member list using the etcdctl member remove <ID> command or etcd client SDK interface MemberRemove

Similarly, you need to execute etcdctl member add [flags] or use MemberAdd when adding new member.

Please also read https://etcd.io/docs/v3.5/op-guide/runtime-configuration/

[Tochemey]

Hello @jmhbnz and @ahrtr I would like to know how to check a cluster status via the client SDK.

[ahrtr]

Please refer to https://github.com/etcd-io/etcd/blob/main/tests/integration/clientv3/examples/example_cluster_test.go

[Tochemey]

@ahrtr I have checked those examples and I could not find anything in that regard

[ahrtr]

• Member list: example_cluster_test.go#L29-L47, equivalent to etcdct member list;
• Get endpoint status: example_maintenance_test.go#L26-L43 , equivalent to etcdctl endpoint status.

Please read more doc and examples in https://pkg.go.dev/go.etcd.io/etcd/clientv3#pkg-examples

[ahrtr]

@Tochemey Can we close this discussion?

[Tochemey]

Sure we can thanks