Skip to content

Latest commit

 

History

History
188 lines (158 loc) · 7.05 KB

debugging.md

File metadata and controls

188 lines (158 loc) · 7.05 KB

Debugging

kubevirtci/cluster-up/kubectl.sh version

will try to connect to the apiserver.

Retrieving Logs

To investigate the logs of a pod, you can view the logs via kubevirtci/cluster-up/kubectl.sh logs. To view the logs of virt-api, type

kubevirtci/cluster-up/kubectl.sh logs virt-api -f

Sometimes a container in a pod is crashlooping because of an application error inside it. In this case, you normally can't see any logs, because the container is already gone, and so are the logs. To get the logs from the last run attempt, the --previous flag can be used. To view the logs of the container virt-api in the pod virt-api from the previous run, type

kubevirtci/cluster-up/kubectl.sh logs virt-api -f -c virt-api -p

Note that you always have to select a container inside a pod for fetching old logs with the --previous flag.

Watching Events

Both, Kubernetes and KubeVirt are creating events, which can be viewed via

kubevirtci/cluster-up/kubectl.sh get events --all-namespaces --watch

This way it is pretty easy to detect if a Pod or a VMI got started.

Entering Containers

It can be very valuable to enter a container and do some investigations there, to see what is going wrong. In this case the kubectl exec command can be used. To enter virt-api with an interactive shell, type

kubevirtci/cluster-up/kubectl.sh exec virt-api -c virt-api -i -t -- sh

Kubelet Logs

After all you might not see errors in the logs provided by Kubernetes. In that case you can take a look at the logs of the kubelet on the host where the issue is appearing. Depending on the error it is getting logged to either the system logs or to the kubelet logs, you can use the following commands to view them:

journalctl
# or
journalctl -u kubelet

References

Using a Debugger (delve)

This shows the basic principle on how remote debugging can be done.

  • Add delve to the container
  • Start delve on a specific port ( dlv attach <pid> --headless=true --api-version=2 --listen=:1234)
  • Use kube-proxy to forward the port to your machine

A go debugger can be attached to kubevirt processes in the following manners:

Local execution of the kubevirt process

Not all processses can easily be executed locally, since some of the processes require specific dependencies that exist in the runtime pod and node For processes that are easily executed locally, such as virt-controler, the following program arguments can be passed to virt-controller --kubeconfig /path/to/kubeconfig --leader-elect-lease-duration 99h in order to successfully debug. Since virt-contoller and other control plane components rely on leader election to achive consensus among multiple replicas, and we only want our local replica to be the leader, even in the absence of other replicas, we can temporarily grant control to our local replica by specifying the --leader-elect-lease-duration argument, as shown above, with an example value of 99h, granting the lease for 99 hours.

Note /path/to/kubeconfig must point to a running kubrenetes cluster with kubevirt installed

Note It is recommended to scale down both virt-controller and also virt-operator deployments to 0 replicas, because otherwise it will override the virt-controller deployment manifest.

Remote debugging running pods

Remote debugging may be more complex to setup, but has the advantage of executing the procesess in its real runtime environment, where all dependencies ae satisfied, therefore it can work for any go process

Step 1 - recompile go code, build and publish container image

Compile the required go cmd with -gcflags='all=-N -l' to produce a debuggable binary.
Validate that it is in fact debuggable by running file <binary file> and expect to see with debug_info, not stripped in the output. For Bazel based build, add --@io_bazel_rules_go//go/config:gc_goopts=-N,-l --strip=never to the bazel build command. Build a container image, with the compiled binary and publish to a registry that is accessible to the cluster consider adding dlv executable to the image for step 2

Step 2 - modify workload manifest to consume debug container image and execute delve

Having scaled down virt-operator, patch or edit the respetive k8s workload manifest (e.g. virt-controller Deployment, virt-handler Daemonset ) to:

  • Consume the debug image
  • Execute dlv, either in a separate container, and attach to the debugged process as shown above. kubectl debug can be used, to easily add another containr to the pod. Note that shareProcessNamespace: true is necessary, for dlv to be able to debug a process running in another container. The other alternative is to modify the app container cmd to have dlv exec the process in the same container: dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec <binary>
  • Turn off probes, tweak securityContext and anything else that could interfere with slow execution

Here's a working example of a virt-controller deployment patch

cluster-up/kubectl.sh --namespace kubevirt patch deployment virt-controller --type='json' -p '[
   {
      "op":"add",
      "path":"/spec/template/spec/containers/-",
      "value":{
         "name":"dlv-debugger",
         "image":"golang:1.23-alpine",
         "command":[
            "sh",
            "-c",
            "apk add --no-cache git bash && go install github.com/go-delve/delve/cmd/dlv@latest && \\
dlv attach $(pgrep virt-controller) --headless --accept-multiclient --api-version 2 --listen=:2345"
         ],
         "securityContext":{
            "seccompProfile":{
               "type":"Unconfined"
            },
            "runAsUser":0
         }
      }
   },
   {
      "op":"add",
      "path":"/spec/template/spec/shareProcessNamespace",
      "value":true
   },
   {
      "op":"replace",
      "path":"/spec/template/spec/containers/0/image",
      "value":"registry:5000/kubevirt/virt-controller:debug"
   },
   {
      "op":"replace",
      "path":"/spec/replicas",
      "value":1
   },
   {
      "op":"replace",
      "path":"/spec/template/spec/containers/0/imagePullPolicy",
      "value":"Always"
   },
   {
      "op":"replace",
      "path":"/spec/template/spec/containers/0/securityContext",
      "value":{
         "runAsUser":0
      }
   },
   {
      "op":"replace",
      "path":"/spec/template/spec/securityContext",
      "value":{
         
      }
   },
   {
      "op":"remove",
      "path":"/spec/template/spec/containers/0/readinessProbe"
   },
   {
      "op":"remove",
      "path":"/spec/template/spec/containers/0/livenessProbe"
   }
]'

Step 3 - port-forward a local port to the target pod desgnated debug port

kubectl port-forward <pod> 2345

Step 4 - connect to localhost:2345 with your debugger