Deploy snowdrop team k8s cluster on RHOS

[jacobdotcosta]

No description provided.

[jacobdotcosta]

Error deploying kubernetes 1.24

failed to pull image "k8s.gcr.io/kube-apiserver:v1.24.3": output: time="2023-08-09T07:10:31Z" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

[jacobdotcosta]

Seems to be related to this problem.

containerd/containerd#8706

fixing the /etc/containerd/config.toml file which had disabled_plugins = ["cri"].

[jacobdotcosta]

Although that step is fixed a new error is ocurring when applying Calico.

fatal: [snowdrop-k8s]: FAILED! => {"changed": true, "cmd": "kubectl apply -f /tmp/calico.yaml", "delta": "0:00:00.040075", "end": "2023-08-09 07:38:23.804883", "msg": "non-zero return code", "rc": 1, "start": "2023-08-09 07:38:23.764808", "stderr": "The connection to the server x.x.x.x:6443 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server x.x.x.x:6443 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}

[jacobdotcosta]

https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#installation

Prior to Kubernetes 1.24, the CNI plugins could also be managed by the kubelet using the cni-bin-dir and network-plugin command-line parameters. 
These command-line parameters were removed in Kubernetes 1.24, with management of the CNI no longer in scope for kubelet.

[jacobdotcosta]

Deployed Calico but still container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

[jacobdotcosta]

After installing Calico, and downloaded the container images by hand, the Node became ready.

$ kubectl get nodes
NAME           STATUS   ROLES           AGE     VERSION
snowdrop-k8s   Ready    control-plane   6h26m   v1.24.3

[jacobdotcosta]

Error creating the temporary PV

fatal: [snowdrop-k8s]: FAILED! => 

{
  "changed": true
  , "cmd": "mkdir -p /tmp/pv001\nchmod -R 777 /tmp/pv001\nchmod -R a+w /tmp/pv001\nchcon -Rt svirt_sandbox_file_t /tmp/pv001\n"
  , "delta": "0:00:00.007847", "end": "2023-08-09 14:09:35.777125"
  , "msg": "non-zero return code", "rc": 1, "start": "2023-08-09 14:09:35.769278"
  , "stderr": "chcon: can't apply partial context to unlabeled file '/tmp/pv001'"
  , "stderr_lines": ["chcon: can't apply partial context to unlabeled file '/tmp/pv001'"], "stdout": "", "stdout_lines": []
}

[jacobdotcosta]

Another Calico related error:

Aug 09 14:14:08 snowdrop-k8s containerd[71572]: time="2023-08-09T14:14:08.335246941Z" level=error msg="StopPodSandbox for \"ff95a64aaf7571c06d775eaf5324b19cb2416912de1ccd59bdd8a194d64c7139\" failed" error="failed to destroy network for sandbox \"ff95a64aaf7571c06d775eaf5324b19cb2416912de1ccd59bdd8a194d64c7139\": plugin type=\"calico\" failed (delete): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"

[jacobdotcosta]

snowdrop-k8s deployed with k8s version 1.24.3.

Resuts in snowdrop/k8s-infra#337