Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken Kubeadm Config Patches for kubeadmConfigPatches #3849

Open
rooty0 opened this issue Jan 18, 2025 · 4 comments
Open

Broken Kubeadm Config Patches for kubeadmConfigPatches #3849

rooty0 opened this issue Jan 18, 2025 · 4 comments
Assignees
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@rooty0
Copy link

rooty0 commented Jan 18, 2025

What happened:

Start the KIND cluster like this

kind create cluster --config kind.yaml

My kind.yaml is

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: test
nodes:
  - role: control-plane
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: KubeletConfiguration
        registerWithTaints:
          - effect: NoSchedule
            key: mri-agent
            value: presence

What you expected to happen:
The worker node should have the taint assigned

How to reproduce it (as minimally and precisely as possible):
Take the config and run it, It is easy to reproduce

Anything else we need to know?:
When I run an alternative following configuration everything works ok:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: test
nodes:
  - role: control-plane
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            register-with-taints: "mri-agent=presence:NoSchedule"

Thought I'm getting the following: Jan 18 00:28:49 test-worker2 kubelet[188]: Flag --register-with-taints has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

Environment:

  • kind version: kind v0.26.0 go1.23.4 darwin/arm64
@rooty0 rooty0 added the kind/bug Categorizes issue or PR as related to a bug. label Jan 18, 2025
@BenTheElder
Copy link
Member

So kubeadm only respects kubelet configuration for the first node.

Note that these are kubeadm config patches
/remove-kind bug
/kind support

There's an issue somewhere discussing having v1alpha5 have kubernetes config patches and splitting out the patches and applying them to kubelet config after kubeadm generates it, but that's not implemented

kubeadm config patches patch the config going to kubeadm, and in kubeadm the kubelet config is cluster-wide and shared by all nodes after the first kubeadm init node unfortunately.

It would be good to add a warning about this to the kind config docs.

@k8s-ci-robot k8s-ci-robot added kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jan 21, 2025
@BenTheElder BenTheElder self-assigned this Jan 21, 2025
@BenTheElder
Copy link
Member

BenTheElder commented Jan 21, 2025

You will have to use the argument version for now, and ignore the warning for now. Kubelet currently marks all* flags with config equivalents as deprecated, but that stance has been walked back for some flags already.

@rooty0
Copy link
Author

rooty0 commented Jan 22, 2025

Thanks for the clarification @BenTheElder , will do

@BenTheElder
Copy link
Member

This situation is super confusing. It will be complicated to implement a new config format that does the patching ourselves and we've been a bit wrapped up in other fixes, I think we should document the limitation at least for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Development

No branches or pull requests

3 participants