Add KaaS robustness feature tests #714

cah-patrickthiem · 2024-08-28T15:29:55Z

This PR will add tests for the K8s cluster robustness features defined in the scs standard: scs-0215-v1-robustness-features
Here is a detailed listing of what is tested:

SCS-0215-v1 Robustness Features Test Coverage
1. API Server Rate Limiting
Test_scs_0215_requestLimits

Verifies basic request limit configurations
Checks API server configuration for required settings

Test_scs_0215_minRequestTimeout

Validates min-request-timeout setting
Checks configuration in API server args

Test_scs_0215_eventRateLimit

Confirms EventRateLimit admission controller configuration
Verifies plugin is enabled in API server

Test_scs_0215_apiPriorityAndFairness

Checks APF feature gate enablement
Validates API server configuration for priority and fairness

Test_scs_0215_rateLimitValues

Verifies specific rate limit values
Checks recommended settings:
- QPS: 5000
- Burst: 20000

2. etcd Management
Test_scs_0215_etcdCompaction

Validates compaction configuration:
- Mode: periodic
- Retention: 8h

Test_scs_0215_etcdBackup

Verifies backup CronJobs setup
Checks backup configuration:
- Hourly backups
- Daily backups
- Proper paths and schedules

3. Certificate Management
Test_scs_0215_certificateRotation

Check_Certificate_Rotation_Configuration:
- Verifies kubelet certificate rotation settings
- Validates serverTLSBootstrap and rotateCertificates

Check_Certificate_Controller:

Confirms cert-manager deployment
Validates certificate controller functionality

There is a positive and a negative test case.

…axMutatingRequestInflight" and "minRequestTimeout".

cah-patrickthiem · 2024-11-15T11:46:59Z

For reference, here the successful test logs of sonobuoy:

cat results/plugins/scs-kaas-conformance/sonobuoy_results.yaml | yq
name: scs-kaas-conformance
status: passed
meta:
type: summary
items:

name: out.json
status: passed
meta:
file: results/global/out.json
type: file
items:
- name: Test_scs_0200_smoke
  status: passed
- name: Test_scs_0215_requestLimits/Check_Request_Limit_Configuration
  status: passed
- name: Test_scs_0215_requestLimits
  status: passed
- name: Test_scs_0215_minRequestTimeout/Check_minRequestTimeout_Configuration
  status: passed
- name: Test_scs_0215_minRequestTimeout
  status: passed
- name: Test_scs_0215_eventRateLimit/Check_EventRateLimit_Configuration
  status: passed
- name: Test_scs_0215_eventRateLimit
  status: passed
- name: Test_scs_0215_apiPriorityAndFairness/Check_APF_Configuration
  status: passed
- name: Test_scs_0215_apiPriorityAndFairness
  status: passed
- name: Test_scs_0215_rateLimitValues/Check_Rate_Limit_Values
  status: passed
- name: Test_scs_0215_rateLimitValues
  status: passed
- name: Test_scs_0215_etcdCompaction/Check_Etcd_Compaction_Settings
  status: passed
- name: Test_scs_0215_etcdCompaction
  status: passed
- name: Test_scs_0215_etcdBackup/Check_Etcd_Backup_Configuration
  status: passed
- name: Test_scs_0215_etcdBackup
  status: passed
- name: Test_scs_0215_certificateRotation/Check_Certificate_Controller
  status: passed
- name: Test_scs_0215_certificateRotation
  status: passed

[Displaying results...]
sonobuoy results *.tar.gz
Plugin: scs-kaas-conformance
Status: passed
Total: 17
Passed: 17
Failed: 0
Skipped: 0

cah-patrickthiem · 2024-11-15T11:52:14Z

In order to make the tests pass on your K8s cluster, you would need to apply the following configurations:

API Server Configuration
Location: /etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    # Admission Control
    - --enable-admission-plugins=NodeRestriction,EventRateLimit
    - --admission-control-config-file=/etc/kubernetes/admission-config.yaml
    # API Priority
    - --feature-gates=APIPriorityAndFairness=true
    - --enable-priority-and-fairness=true

Admission Configuration
Location: /etc/kubernetes/admission-config.yaml

# event-ratelimit-config.yaml
kind: Configuration
apiVersion: eventratelimit.admission.k8s.io/v1alpha1
limits:
- burst: 20000
  qps: 5000
  type: Server

etcd Configuration
Location: /etc/kubernetes/manifests/etcd.yaml

apiVersion: v1
kind: Pod
metadata:
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --auto-compaction-mode=periodic
    - --auto-compaction-retention=8h

Kubelet Configuration
Location: /var/lib/kubelet/config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
rotateCertificates: true

Certificate Management
Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml
etcd Backup CronJobs
etcd-cronjobs.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
 name: etcd-backup-hourly
spec:
 schedule: "0 * * * *"
 jobTemplate:
   spec:
     template:
       spec:
         containers:
         - name: etcd-backup
           image: k8s.gcr.io/etcd:3.5.9-0
           command:
           - /bin/sh
           - -c
           - |
             ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
               --cacert=/etc/kubernetes/pki/etcd/ca.crt \
               --cert=/etc/kubernetes/pki/etcd/server.crt \
               --key=/etc/kubernetes/pki/etcd/server.key \
               snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db
           volumeMounts:
           - name: etcd-certs
             mountPath: /etc/kubernetes/pki/etcd
             readOnly: true
           - name: backup
             mountPath: /backup
         volumes:
         - name: etcd-certs
           hostPath:
             path: /etc/kubernetes/pki/etcd
             type: Directory
         - name: backup
           hostPath:
             path: /var/lib/etcd/backup/hourly
             type: DirectoryOrCreate
         restartPolicy: OnFailure
---
apiVersion: batch/v1
kind: CronJob
metadata:
 name: etcd-backup-daily
spec:
 schedule: "0 0 * * *"
 jobTemplate:
   spec:
     template:
       spec:
         containers:
         - name: etcd-backup
           image: k8s.gcr.io/etcd:3.5.9-0
           command:
           - /bin/sh
           - -c
           - |
             ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
               --cacert=/etc/kubernetes/pki/etcd/ca.crt \
               --cert=/etc/kubernetes/pki/etcd/server.crt \
               --key=/etc/kubernetes/pki/etcd/server.key \
               snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db
           volumeMounts:
           - name: etcd-certs
             mountPath: /etc/kubernetes/pki/etcd
             readOnly: true
           - name: backup
             mountPath: /backup
         volumes:
         - name: etcd-certs
           hostPath:
             path: /etc/kubernetes/pki/etcd
             type: Directory
         - name: backup
           hostPath:
             path: /var/lib/etcd/backup/daily
             type: DirectoryOrCreate
         restartPolicy: OnFailure
---
apiVersion: batch/v1
kind: CronJob
metadata:
 name: etcd-compaction
spec:
 schedule: "0 */8 * * *"
 jobTemplate:
   spec:
     template:
       spec:
         containers:
         - name: etcd-compaction
           image: k8s.gcr.io/etcd:3.5.9-0
           command:
           - /bin/sh
           - -c
           - |
             ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
               --cacert=/etc/kubernetes/pki/etcd/ca.crt \
               --cert=/etc/kubernetes/pki/etcd/server.crt \
               --key=/etc/kubernetes/pki/etcd/server.key \
               compact $(etcdctl endpoint status --write-out="json" | awk -F'"' '{print $4}')
           volumeMounts:
           - name: etcd-certs
             mountPath: /etc/kubernetes/pki/etcd
             readOnly: true
         volumes:
         - name: etcd-certs
           hostPath:
             path: /etc/kubernetes/pki/etcd
             type: Directory
         restartPolicy: OnFailure

Location: Apply via kubectl
kubectl apply -f etcd-cronjobs.yaml

cah-patrickthiem · 2024-11-15T11:59:28Z

For reference, I used a self configured KubeAdm cluster to develop those tests.

mbuechse · 2024-11-15T11:59:35Z

Impressive! I'm not sure I am competent to review it, but I will give it a shot. About these preconditions, wouldn't it be good to put them into a 'Testing and implementation notes' supplement? This can happen within this same PR.

mbuechse · 2024-11-15T12:01:15Z

For reference, I used a self configured KubeAdm cluster to develop those tests.

Impressive again! Just for increased safety, could you please also test on moin once we have the necessary permissions?

cah-patrickthiem · 2024-11-15T12:03:47Z

Impressive! I'm not sure I am competent to review it, but I will give it a shot. About these preconditions, wouldn't it be good to put them into a 'Testing and implementation notes' supplement? This can happen within this same PR.

I talked about including the configurations with @tonifinger. We came to the same conclusion. Also, my guess is that there will be more configuration snippets from the other tested features in other PRs.

cah-patrickthiem · 2024-11-15T12:06:13Z

For reference, I used a self configured KubeAdm cluster to develop those tests.

Impressive again! Just for increased safety, could you please also test on moin once we have the necessary permissions?

Sure, I can do that.

tonifinger · 2024-11-19T15:03:40Z

...s/kaas/kaas-sonobuoy-tests/scs_k8s_conformance_tests/scs_0215_v1_robustness_features_test.go

+               t.Errorf("Required setting %s not found in API server configuration", setting)
+           }
+       }
+       if !foundSettings["EventRateLimit"] {


Can this check also be carried out in the conditional statement in line 67? I think the error can already be logged there.

Yes, you are right, I changed that.

tonifinger · 2024-11-19T16:09:20Z

...s/kaas/kaas-sonobuoy-tests/scs_k8s_conformance_tests/scs_0215_v1_robustness_features_test.go

+           config, err := clientset.CoreV1().ConfigMaps(loc.namespace).Get(context.Background(), loc.name, metav1.GetOptions{})
+           if err == nil {
+               if data, ok := config.Data[loc.key]; ok {
+                   if strings.Contains(data, "eventratelimit.admission.k8s.io") {


How is this different from the test in Test_scs_0215_requestLimits()
Isn't this check already handled on this line L67
On second though, if this does the same check, I think it would be better to only handle it here in Test_scs_0215_minRequestTimeout(), as this is the testfunction related to "EventRateLimit"

No, these are different tests. The first test checks if the EventRateLimit is enabled in the API server command line flags, while the second test specifically looks for EventRateLimit configuration in ConfigMaps. The second test is more thorough as it searches multiple locations for the actual configuration details.
The first test only verifies the admission plugin is enabled, while the second test verifies the configuration exists and is properly set up.

tonifinger · 2024-11-19T16:17:40Z

...s/kaas/kaas-sonobuoy-tests/scs_k8s_conformance_tests/scs_0215_v1_robustness_features_test.go

+   }
+
+   if isKindCluster(clientset) {
+       t.Skip("Running on kind cluster - skipping APF test")


This must raise an error as well. Otherwise this will be unnoticed by the scs-test-runner.py in case someone does run this the testsuite against a kind cluster.

I thought about the "skip tests if kind cluster" topic again. I tend to exclude those skipping statements. The tests should fail if the cluster cannot support the test features, that is the whole purpose of the tests.

What do you think?

I agree. we should not allow to skip any tests.

will change that

tonifinger · 2024-11-19T16:21:14Z

...s/kaas/kaas-sonobuoy-tests/scs_k8s_conformance_tests/scs_0215_v1_robustness_features_test.go

+   }
+
+   if isKindCluster(clientset) {
+       t.Skip("Running on kind cluster - skipping rate limit values test")


Same as above and to all other t.Skip related conditionals.

tonifinger · 2024-11-19T16:31:14Z

...s/kaas/kaas-sonobuoy-tests/scs_k8s_conformance_tests/scs_0215_v1_robustness_features_test.go

+               for k, v := range expectedValues {
+                   if !strings.Contains(config, fmt.Sprintf("%s: %s", k, v)) {
+                       allFound = false
+                       break
+                   }
+               }
+               if allFound {
+                   return
+               }
+           }
+       }
+
+       t.Error("Recommended rate limit values (qps: 5000, burst: 20000) not found")


In the standards, these values are described as RECOMMENDED and furthermore “SHOULD be adapted to the needs of the environment and the expected load”.
We should therefore not regard the values described in the standard as fixed values. Rather, we should check whether we meet them as minimum requirements.

See: ../scs-0215-v1-robustness-features.md#kube-api-rate-limiting-1

yes you are right here, I have overseen that it is just recommended. However, for that reason I exclude the test for now. Maybe some test for this could be added in the future if needed. The overall check for the presence of event rate limits is there.

tonifinger · 2024-11-19T16:50:34Z

...s/kaas/kaas-sonobuoy-tests/scs_k8s_conformance_tests/scs_0215_v1_robustness_features_test.go

+   }
+
+   if isKindCluster(clientset) {
+       t.Skip("Running on kind cluster - skipping etcd backup test")


Same as above.

tonifinger · 2024-11-19T16:59:41Z

...s/kaas/kaas-sonobuoy-tests/scs_k8s_conformance_tests/scs_0215_v1_robustness_features_test.go

+           LabelSelector: "component=etcd",
+       })
+       if err != nil || len(pods.Items) == 0 {
+           t.Skip("No etcd pods found")


Same as above this must throw an error as well. We currently don't consider someone using something else as etcd for k8s.
If there is the need to use something else then etcd the standard itself needs to be updated first.

cah-patrickthiem self-assigned this Aug 28, 2024

cah-patrickthiem force-pushed the 549-testing-kaas-robustness-features branch from 4e8fc4d to 6d98860 Compare October 16, 2024 13:43

mbuechse linked an issue Nov 4, 2024 that may be closed by this pull request

[Testing] KaaS Robustness features SovereignCloudStack/issues#549

Open

3 tasks

cah-patrickthiem force-pushed the 549-testing-kaas-robustness-features branch from 5c2f787 to d0c4d95 Compare November 15, 2024 11:43

cah-patrickthiem added 5 commits November 15, 2024 12:44

Added the first robustness feature test called maxRequestInflight.

3c2aaba

There is a positive and a negative test case.

Added two more tests regarding the robustness K8s standard, namely "m…

4d11602

…axMutatingRequestInflight" and "minRequestTimeout".

add results path to .gitignore

7013a9d

Added remaining robustness tests.

9f4ec85

added/adjusted the remaining tests

cbcca65

cah-patrickthiem force-pushed the 549-testing-kaas-robustness-features branch from d0c4d95 to cbcca65 Compare November 15, 2024 11:44

cah-patrickthiem marked this pull request as ready for review November 15, 2024 11:56

cah-patrickthiem requested review from tonifinger and mbuechse November 15, 2024 11:56

tonifinger reviewed Nov 19, 2024

View reviewed changes

Addressed review commands.

cb8e7a3

cah-patrickthiem removed the request for review from mbuechse December 5, 2024 14:15

Merge branch 'main' into 549-testing-kaas-robustness-features

363d62a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add KaaS robustness feature tests #714

Add KaaS robustness feature tests #714

cah-patrickthiem commented Aug 28, 2024 •

edited

Loading

cah-patrickthiem commented Nov 15, 2024

cah-patrickthiem commented Nov 15, 2024 •

edited

Loading

cah-patrickthiem commented Nov 15, 2024

mbuechse commented Nov 15, 2024

mbuechse commented Nov 15, 2024

cah-patrickthiem commented Nov 15, 2024

cah-patrickthiem commented Nov 15, 2024

tonifinger Nov 19, 2024

cah-patrickthiem Dec 5, 2024

tonifinger Nov 19, 2024

cah-patrickthiem Dec 5, 2024

tonifinger Nov 19, 2024

cah-patrickthiem Dec 2, 2024

tonifinger Dec 4, 2024

cah-patrickthiem Dec 5, 2024

tonifinger Nov 19, 2024

cah-patrickthiem Dec 5, 2024

tonifinger Nov 19, 2024

cah-patrickthiem Dec 5, 2024

tonifinger Nov 19, 2024

cah-patrickthiem Dec 5, 2024

tonifinger Nov 19, 2024

cah-patrickthiem Dec 5, 2024

Add KaaS robustness feature tests #714

Are you sure you want to change the base?

Add KaaS robustness feature tests #714

Conversation

cah-patrickthiem commented Aug 28, 2024 • edited Loading

cah-patrickthiem commented Nov 15, 2024

cah-patrickthiem commented Nov 15, 2024 • edited Loading

cah-patrickthiem commented Nov 15, 2024

mbuechse commented Nov 15, 2024

mbuechse commented Nov 15, 2024

cah-patrickthiem commented Nov 15, 2024

cah-patrickthiem commented Nov 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cah-patrickthiem commented Aug 28, 2024 •

edited

Loading

cah-patrickthiem commented Nov 15, 2024 •

edited

Loading