Add crds deletion as in kueue update steps

[pawloch00]

Fixes / Features

• This is an fix to upgrading a cluster with xpk also upgrades kueue but does so improperly #296 . It follows steps listed in https://github.com/kubernetes-sigs/kueue/releases/tag/v0.9.0 . Multikueues are not enabled in xpk, so only step 3 is executed: 3. Delete old CRDs
• Tests:
• Created cluster with kueue v0.8.1, switched kueue version manually to v0.9.1 in xpk code, ran cluster create again. Cluster was sucessfully updated to kueue v0.9.1 with all pods in kueue deployment working properly.

Testing / Documentation

Testing details.

[XPK] Enabling Kueue on the cluster
[XPK] Task: `Get kueue version on server` is implemented by `kubectl kueue version`, hiding output unless there is an error.
[XPK] ['Client Version: v0.10.0-devel-33-g00a05fcf', 'Kueue Controller Manager Image: registry.k8s.io/kueue/kueue:v0.8.1']
[XPK] v0.8.1 0
[XPK] 0.8.1 0.9.1
[XPK] Try 1: Delete multikueueclusters crds
[XPK] Task: `Delete multikueueclusters crds` is implemented by `kubectl delete crd multikueueclusters.kueue.x-k8s.io`, streaming output live.
[XPK] Waiting for `Delete multikueueclusters crds`, for 0 seconds
customresourcedefinition.apiextensions.k8s.io "multikueueclusters.kueue.x-k8s.io" deleted
[XPK] Task: `Delete multikueueclusters crds` terminated with code `0`
[XPK] Try 1: Delete multikueueconfigs crds
[XPK] Task: `Delete multikueueconfigs crds` is implemented by `kubectl delete crd multikueueconfigs.kueue.x-k8s.io`, streaming output live.
[XPK] Waiting for `Delete multikueueconfigs crds`, for 0 seconds
customresourcedefinition.apiextensions.k8s.io "multikueueconfigs.kueue.x-k8s.io" deleted
[XPK] Task: `Delete multikueueconfigs crds` terminated with code `0`
[XPK] Try 1: Set Kueue On Cluster
[XPK] Task: `Set Kueue On Cluster` is implemented by `kubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.9.1/manifests.yaml`, streaming output live.

• [ y/n ] Tests pass
• [ y/n ] Appropriate changes to documentation are included in the PR

[GallagherCommaJack]

tested this on an existing cluster & it doesn't seem to have resolved the issue
still seeing it fail to install jobset
error below

[XPK] Try 3: Install Jobset on [REDACTED]
[XPK] Task: `Install Jobset on [REDACTED]` is implemented by `kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.7.2/manifests.yaml`, streaming output live.
[XPK] Waiting for `Install Jobset on [REDACTED]`, for 0 seconds
namespace/jobset-system serverside-applied
[XPK] Waiting for `Install Jobset on [REDACTED]`, for 1 seconds
customresourcedefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io serverside-applied
serviceaccount/jobset-controller-manager serverside-applied
role.rbac.authorization.k8s.io/jobset-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-metrics-reader serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-proxy-role serverside-applied
rolebinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding serverside-applied
secret/jobset-webhook-server-cert serverside-applied
service/jobset-controller-manager-metrics-service serverside-applied
service/jobset-webhook-service serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration serverside-applied
Error from server (InternalError): Internal error occurred: failed calling webhook "vdeployment.kb.io": failed to call webhook: the server could not find the requested resource
[XPK] Task: `Install Jobset on [REDACTED]` terminated with code `1`

[logicalhan]

Honestly, I would do three things.

1. yes delete the CRDs. All the kueue ones. Just in case.
2. delete the kueue-system namespace. purge all the objects from the system
3. then install kueue from a fresh slate.

[pawloch00]

Honestly, I would do three things.

1. yes delete the CRDs. All the kueue ones. Just in case.
2. delete the kueue-system namespace. purge all the objects from the system
3. then install kueue from a fresh slate.

This may break existing installations, for now we are going to follow steps listed in kueue update instructions.

[pawloch00]

@44past4 please review

[pawloch00]

@logicalhan If this solves your issue, please close it.