new cluster stuck in CREATE_IN_PROGRESS using auto_scaling_enabled=true

[Lappihuan]

creating a new cluster, in my case with v1.30.4 is stuck in CREATE_IN_PROGRESS using auto_scaling_enabled=true without auto_scaling_enabled it goes into CREATE_COMPLETE and HEALTHY.

there is no autoscaler pods in the magnum-system namespace, nor do i see a release in the helm-releases.
i don't know how to confirm this but i suspect this chart is way out of date, so it never goes past this line.

for v1.30.x the supported version of cluster-autoscaler would be 9.37.0+
https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#releases

[MaximMonin]

My cluster goes to HEALTHY with v.1.30.7 and auto_scaling_enabled=true, but autoscaling pods not started due to scheduling issue. To fix it all control plane cluster nodes should be labeled with:
kubectl label node controlnodename openstack-control-plane=enabled

[nguyenhuukhoi]

I have this problem too but kubectl label node controlnodename openstack-control-plane=enabled not working for me,

[nguyenhuukhoi]

I can do it if I set openstack-control-plane=enabled label on worker nodes.

[MaximMonin]

I can do it if I set openstack-control-plane=enabled label on worker nodes.

It seems to me that you do not quite understand the architecture of the solution - autoscaler is created on the management cluster.

root@os-stage-ctl-b:/home/ubuntu# kubectl get pods -A
NAMESPACE                           NAME                                                             READY   STATUS      RESTARTS     AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-647c4d77dc-cgc2p       1/1     Running     0            9d
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-67fc9db87c-p7d8t   1/1     Running     0            9d
capi-system                         capi-controller-manager-685f8c946f-6hhjr                         1/1     Running     0            9d
capo-system                         capo-controller-manager-6bdf5576d4-smqpx                         1/1     Running     1 (9d ago)   9d
cert-manager                        cert-manager-5c887c889d-7fh4r                                    1/1     Running     0            9d
cert-manager                        cert-manager-cainjector-58f6855565-mmznl                         1/1     Running     0            9d
cert-manager                        cert-manager-webhook-6647d6545d-bqc7g                            1/1     Running     0            9d
kube-system                         coredns-ccb96694c-dfcxr                                          1/1     Running     1 (9d ago)   9d
kube-system                         local-path-provisioner-5cf85fd84d-nws4s                          1/1     Running     1 (9d ago)   9d
magnum-system                       kube-rgjp9-autoscaler-697dcb57c8-hhkns                           1/1     Running     0            2d
root@os-stage-ctl-b:/home/ubuntu# kubectl get nodes
NAME             STATUS   ROLES                       AGE   VERSION
os-stage-ctl-a   Ready    control-plane,etcd,master   9d    v1.31.4+k3s1
os-stage-ctl-b   Ready    control-plane,etcd,master   9d    v1.31.4+k3s1
os-stage-ctl-c   Ready    control-plane,etcd,master   9d    v1.31.4+k3s1

[nguyenhuukhoi]

What you mean? I see that cause by autoscale cannot schedule on controlplane node. I mean management cluster.

…

On Wed, Jan 8, 2025, 5:21 PM Maxim Monin ***@***.***> wrote: I can do it if I set openstack-control-plane=enabled label on worker nodes. It seems to me that you do not quite understand the architecture of the solution - autoscaler is created on the management cluster. ***@***.***:/home/ubuntu# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-647c4d77dc-cgc2p 1/1 Running 0 9d capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-67fc9db87c-p7d8t 1/1 Running 0 9d capi-system capi-controller-manager-685f8c946f-6hhjr 1/1 Running 0 9d capo-system capo-controller-manager-6bdf5576d4-smqpx 1/1 Running 1 (9d ago) 9d cert-manager cert-manager-5c887c889d-7fh4r 1/1 Running 0 9d cert-manager cert-manager-cainjector-58f6855565-mmznl 1/1 Running 0 9d cert-manager cert-manager-webhook-6647d6545d-bqc7g 1/1 Running 0 9d kube-system coredns-ccb96694c-dfcxr 1/1 Running 1 (9d ago) 9d kube-system local-path-provisioner-5cf85fd84d-nws4s 1/1 Running 1 (9d ago) 9d magnum-system kube-rgjp9-autoscaler-697dcb57c8-hhkns 1/1 Running 0 2d ***@***.***:/home/ubuntu# kubectl get nodes NAME STATUS ROLES AGE VERSION os-stage-ctl-a Ready control-plane,etcd,master 9d v1.31.4+k3s1 os-stage-ctl-b Ready control-plane,etcd,master 9d v1.31.4+k3s1 os-stage-ctl-c Ready control-plane,etcd,master 9d v1.31.4+k3s1 — Reply to this email directly, view it on GitHub <#429 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRLKQMNS4CI5BEC46FGD3L2JT3ZXAVCNFSM6AAAAABOJ6TOMSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZXGMYTGMJSHA> . You are receiving this because you commented.Message ID: ***@***.***>

[MaximMonin]

What you mean? I see that cause by autoscale cannot schedule on controlplane node. I mean management cluster.

Ok, it seems it depends on management cluster installation. k3s ha cluster in my case enable to schedule it to os-stage-ctl-* control-plane (k3s uses it for worker nodes too)

[nguyenhuukhoi]

Hello, from https://github.com/openstack/magnum/blob/1c3d7d070b60a36ccfb7c753b26f12609d818cec/magnum/drivers/common/templates/kubernetes/fragments/enable-auto-scaling.sh#L124C3-L135C29

It allowed us to schedule on controlplane node with control-plane=enabled label.

But I see that

https://github.com/vexxhost/magnum-cluster-api/blob/main/magnum_cluster_api/resources.py#L94

does have tolerations allow us schedule on controlplane.