Skip to content

Commit

Permalink
Improve CDI alert runbooks
Browse files Browse the repository at this point in the history
Use kubectl jsonpath instead of jq

Signed-off-by: Arnon Gilboa <[email protected]>
  • Loading branch information
arnongilboa committed Jun 25, 2024
1 parent a6bc998 commit 9dcd7bc
Show file tree
Hide file tree
Showing 4 changed files with 20 additions and 12 deletions.
10 changes: 7 additions & 3 deletions docs/runbooks/CDIDataImportCronOutdated.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ For golden images, _latest_ refers to the latest operating system of the
distribution. For other disk images, _latest_ refers to the latest hash of the
image that is available.

In case there is no default (Kubernetes or KubeVirt) storage class, and the
DataImportCron import PVC is pending for one, the alert is suppressed as the
root cause is already alerted by CDINoDefaultStorageClass.

## Impact

VMs might be created from outdated disk images.
Expand All @@ -37,10 +41,10 @@ VMs might fail to start because no boot source is available for cloning.
event is displayed: `DataVolume.storage spec is missing accessMode and no
storageClass to choose profile`.

2. Obtain the `DataImportCron` namespace and name:
2. Obtain the `DataImportCrons` which are not up-to-date:

```bash
$ kubectl get dataimportcron -A -o json | jq -r '.items[] | select(.status.conditions[] | select(.type == "UpToDate" and .status == "False")) | .metadata.namespace + "/" + .metadata.name'
$ kubectl get dataimportcron -A -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="UpToDate")].status}{"\t"}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' | grep False
```

3. If a default storage class is not defined on the cluster, check the
Expand Down Expand Up @@ -77,7 +81,7 @@ object:
6. Set the `CDI_NAMESPACE` environment variable:

```bash
$ export CDI_NAMESPACE="$(kubectl get deployment -A | grep cdi-operator | awk '{print $1}')"
$ export CDI_NAMESPACE="$(kubectl get deployment -A -o jsonpath='{.items[?(.metadata.name=="cdi-operator")].metadata.namespace}')"
```

7. Check the `cdi-deployment` log for error messages:
Expand Down
16 changes: 10 additions & 6 deletions docs/runbooks/CDIDefaultStorageClassDegraded.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,16 @@

## Meaning

This alert fires when there is no default storage class that supports smart
cloning (CSI or snapshot-based) or the ReadWriteMany access mode.
This alert fires when there is/are default (Kubernetes and/or virtualization)
storage class(es), but none of them supports both smart cloning (CSI or snapshot
based) and ReadWriteMany access mode.

A default virtualization storage class has precedence over a default Kubernetes
storage class for creating a VirtualMachine disk image.

In case of single-node OpenShift, the alert is suppressed if there is a default
storage class that supports smart cloning, but not ReadWriteMany.

## Impact

If the default storage class does not support smart cloning, the default cloning
Expand All @@ -21,28 +25,28 @@ If the default storage class does not support ReadWriteMany, virtual machines
1. Get the default KubeVirt storage class by running the following command:

```bash
$ export CDI_DEFAULT_VIRT_SC="$(kubectl get sc -o json | jq -r '.items[].metadata|select(.annotations."storageclass.kubevirt.io/is-default-virt-class"=="true")|.name')"
$ export CDI_DEFAULT_VIRT_SC="$(kubectl get sc -o jsonpath='{.items[?(.metadata.annotations.storageclass\.kubevirt\.io\/is-default-virt-class=="true")].metadata.name}')"
```

2. If a default KubeVirt storage class exists, check that it supports
ReadWriteMany by running the following command:

```bash
$ kubectl get storageprofile $CDI_DEFAULT_VIRT_SC -o json | jq '.status.claimPropertySets'| grep ReadWriteMany
$ kubectl get storageprofile $CDI_DEFAULT_VIRT_SC -o jsonpath='{.status.claimPropertySets}' | grep ReadWriteMany
```

3. If there is no default KubeVirt storage class, get the default Kubernetes
storage class by running the following command:

```bash
$ export CDI_DEFAULT_K8S_SC="$(kubectl get sc -o json | jq -r '.items[].metadata|select(.annotations."storageclass.kubernetes.io/is-default-class"=="true")|.name')"
$ export CDI_DEFAULT_K8S_SC="$(kubectl get sc -o jsonpath='{.items[?(.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].metadata.name}')"
```

4. If a default Kubernetes storage class exists, check that it supports
ReadWriteMany by running the following command:

```bash
$ kubectl get storageprofile $CDI_DEFAULT_K8S_SC -o json | jq '.status.claimPropertySets'| grep ReadWriteMany
$ kubectl get storageprofile $CDI_DEFAULT_VIRT_SC -o jsonpath='{.status.claimPropertySets}' | grep ReadWriteMany
```

<!--USstart-->
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/CDIMultipleDefaultVirtStorageClasses.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Obtain a list of default virtualization storage classes by running the following
command:

```bash
$ kubectl get sc -o json | jq '.items[].metadata|select(.annotations."storageclass.kubevirt.io/is-default-virt-class"=="true")|.name'
$ kubectl get sc -o jsonpath='{.items[?(.metadata.annotations.storageclass\.kubevirt\.io\/is-default-virt-class=="true")].metadata.name}'
```

## Mitigation
Expand Down
4 changes: 2 additions & 2 deletions docs/runbooks/CDINoDefaultStorageClass.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ does not have a specified storage class remains in a "pending" state.
command:

```bash
$ kubectl get sc -o json | jq '.items[].metadata|select(.annotations."storageclass.kubernetes.io/is-default-class"=="true")|.name'
$ kubectl get sc -o jsonpath='{.items[?(.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].metadata.name}'
```

2. Check for a default KubeVirt storage class by running the following command:

```bash
$ kubectl get sc -o json | jq '.items[].metadata|select(.annotations."storageclass.kubevirt.io/is-default-virt-class"=="true")|.name'
$ kubectl get sc -o jsonpath='{.items[?(.metadata.annotations.storageclass\.kubevirt\.io\/is-default-virt-class=="true")].metadata.name}'
```

## Mitigation
Expand Down

0 comments on commit 9dcd7bc

Please sign in to comment.