From d61bb454e218b2f13eec4a1e288ea3f63ad91fd0 Mon Sep 17 00:00:00 2001 From: Mariam Fahmy Date: Sun, 12 Nov 2023 18:46:05 +0200 Subject: [PATCH 1/3] Proposal for extending forEach to cover generate rules Signed-off-by: Mariam Fahmy --- extend_foreach_to_cover_generate_rules.md | 182 ++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 extend_foreach_to_cover_generate_rules.md diff --git a/extend_foreach_to_cover_generate_rules.md b/extend_foreach_to_cover_generate_rules.md new file mode 100644 index 0000000..68cc034 --- /dev/null +++ b/extend_foreach_to_cover_generate_rules.md @@ -0,0 +1,182 @@ +# Support `forEach` in Generate Rules + +- **Authors**: [Mariam Fahmy](https://github.com/MariamFahmy98) +- **Created**: Nov 11th, 2023 +- **Abstract**: Supporting `forEach` in Generate Rules + +# Table of Contents +- [Overview](#overview) +- [Definitions](#definitions) +- [Motivation](#motivation) +- [Proposal](#proposal) +- [Implementation](#implementation) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Prior Art](#prior-art) +- [Unresolved Questions](#unresolved-questions) + +# Overview +[overview]: #overview +Generate rules are in charge of creating new resources. The rule can either define the new resource using the `data` object or use an existing resource in the cluster with the `clone` object. Currently, only one resource can be generated. + +This proposal is to extend the `forEach` to cover generate rules. + +# Definitions +[definitions]: #definitions + +1. Trigger: It refers to the resource responsible for triggering the generate rule as defined in a combination of `match` and `exclude` blocks. + +2. Downstream: It refers to the generated resource(s). + +3. Source: It refers to the clone source when generating new resources from existing ones in the cluster. + +# Motivation +[motivation]: #motivation + +There are some use cases where a single "trigger" resource should result in the creation of multiple downstream resources. + +Examples: +1. Generating a list of new network policies based on a comma separated string in an annotation using a `forEach` and `Split()`. +2. Migrating from ingress to Kubernetes Gateway API requires generating at least two resources: `Gateway` and `HTTPRoute`. + +# Proposal +In this proposal, Kyverno generate rule can be extended to allowing the generation of multiple resources using `forEach`. +This could be done by using the `list` attribute which is a JMESPath expression that defines the sub-elements it processes. + +Examples: +1. Generate multiple network policies when a new namespace is created: +```yaml +spec: + rules: + - name: generate-network-policies + match: + any: + - resources: + kinds: + - Namespace + generate: + synchronize: true + foreach: + - list: request.object.metadata.labels.networkpolicies | split(@, ',') + resources: + - apiVersion: v1 + kind: NetworkPolicy + name: '{{ element }}' + namespace: "{{request.object.metadata.name}}" + data: + spec: + podSelector: {} + policyTypes: + - Ingress + - Egress +``` + +2. Generate ConfigMap and Secret for each container: +```yaml +spec: + rules: + - name: generate-configmaps-and-secrets + match: + any: + - resources: + kinds: + - Pod + generate: + synchronize: true + foreach: + - list: request.object.spec.containers + resources: + - apiVersion: v1 + kind: ConfigMap + name: cm + namespace: "{{request.object.metadata.namespace}}" + data: + foo: bar + - apiVersion: v1 + kind: Secret + name: secret + namespace: "{{request.object.metadata.namespace}}" + data: + foo: bar +``` + +3. Generate Gateway and HTTPRoute from Ingress: +```yaml +spec: + rules: + - name: generate-gatewayapi + match: + any: + - resources: + kinds: + - Ingress + generate: + synchronize: false + foreach: + - list: request.object.spec.rules + resources: + - apiVersion: gateway.networking.k8s.io/v1beta1 + kind: HTTPRoute + # the name must be unique + name: httproute + namespace: "{{request.object.metadata.namespace}}" + data: + spec: + hostnames: + - '{{ element.host }}' + - list: request.object.spec.ingressClassName + resources: + - apiVersion: gateway.networking.k8s.io/v1beta1 + kind: Gateway + name: gateway + data: + spec: + gatewayClassName: '{{ element }}' +``` + +# Implementation + +We will extend the generate rule as follows: +```go +type Generation struct { + ForEachGeneration []ForEachGeneration `json:"foreach,omitempty" yaml:"foreach,omitempty"` +} +``` + +```go +type ForEachGeneration struct { + List string `json:"list,omitempty" yaml:"list,omitempty"` + + Resource []Resource `json:"resources,omitempty" yaml:"resources,omitempty"` +} +``` + +```go +type Resource struct { + ResourceSpec `json:",omitempty" yaml:",omitempty"` + + RawData *apiextv1.JSON `json:"data,omitempty" yaml:"data,omitempty"` +} +``` + +- `foreach` will only support generating Kubernetes resources that are defined as a part of the rule. + +- Nested `foreach` can be supported but there is no real use case for it. + +- At least one of the following: `data`, `clone`, or `foreach` must be declared. + +# Drawbacks + +N/A + +# Alternatives + +N/A + +# Prior Art + +N/A + +# Unresolved Questions + +N/A From 578db5e72aef56464513275f15c34f0cc7bd5fd5 Mon Sep 17 00:00:00 2001 From: ShutingZhao Date: Fri, 19 Jul 2024 19:03:39 +0800 Subject: [PATCH 2/3] update design Signed-off-by: ShutingZhao --- .../extend_foreach_to_cover_generate_rules.md | 99 +++++++++++-------- 1 file changed, 56 insertions(+), 43 deletions(-) rename extend_foreach_to_cover_generate_rules.md => proposals/extend_foreach_to_cover_generate_rules.md (55%) diff --git a/extend_foreach_to_cover_generate_rules.md b/proposals/extend_foreach_to_cover_generate_rules.md similarity index 55% rename from extend_foreach_to_cover_generate_rules.md rename to proposals/extend_foreach_to_cover_generate_rules.md index 68cc034..54695c8 100644 --- a/extend_foreach_to_cover_generate_rules.md +++ b/proposals/extend_foreach_to_cover_generate_rules.md @@ -1,7 +1,8 @@ # Support `forEach` in Generate Rules -- **Authors**: [Mariam Fahmy](https://github.com/MariamFahmy98) +- **Authors**: [Mariam Fahmy](https://github.com/MariamFahmy98), [Shuting Zhao](https://github.com/realshuting) - **Created**: Nov 11th, 2023 +- **Updated**: Jul 18th, 2024 - **Abstract**: Supporting `forEach` in Generate Rules # Table of Contents @@ -17,18 +18,18 @@ # Overview [overview]: #overview -Generate rules are in charge of creating new resources. The rule can either define the new resource using the `data` object or use an existing resource in the cluster with the `clone` object. Currently, only one resource can be generated. +Generate rules are in charge of creating new resources. The rule can either generate the new resource using the `data` object or clone from an existing resource with the `clone` object. Currently, both generate patterns create single target resource. -This proposal is to extend the `forEach` to cover generate rules. +The `forEach` declaration is supported in validate and mutate rules to simplify rule applications of sub-elements in resource declarations. This proposal is to extend `forEach` support to offer flexible options for resource generation. # Definitions [definitions]: #definitions 1. Trigger: It refers to the resource responsible for triggering the generate rule as defined in a combination of `match` and `exclude` blocks. -2. Downstream: It refers to the generated resource(s). +2. Downstream/target: It refers to the generated resource(s). -3. Source: It refers to the clone source when generating new resources from existing ones in the cluster. +3. Source: It refers to the clone source when cloning new resources from existing ones. # Motivation [motivation]: #motivation @@ -36,15 +37,18 @@ This proposal is to extend the `forEach` to cover generate rules. There are some use cases where a single "trigger" resource should result in the creation of multiple downstream resources. Examples: -1. Generating a list of new network policies based on a comma separated string in an annotation using a `forEach` and `Split()`. -2. Migrating from ingress to Kubernetes Gateway API requires generating at least two resources: `Gateway` and `HTTPRoute`. +1. Generate a list of new `NetWorkPolicy` based on a comma separated string in an annotation using a `forEach` and `Split()`. +2. Migrate from ingress to Kubernetes Gateway API requires generating at least two resources: `Gateway` and `HTTPRoute`. +3. Create several `ProjectIamMember` Crossplane GCP CRs based on a comma-separated label of a namespace. +4. Clone the `Secret` to a list of `Namespace` looked up by the source `Secret`'s label. # Proposal -In this proposal, Kyverno generate rule can be extended to allowing the generation of multiple resources using `forEach`. +In this proposal, the generate rule can be expanded to support the creation of multiple resources or the cloning of a source to multiple targets using the `forEach` declaration. + This could be done by using the `list` attribute which is a JMESPath expression that defines the sub-elements it processes. Examples: -1. Generate multiple network policies when a new namespace is created: +1. Generate multiple `NetWorkPolicy` when a new `Namespace` is created: ```yaml spec: rules: @@ -58,7 +62,7 @@ spec: synchronize: true foreach: - list: request.object.metadata.labels.networkpolicies | split(@, ',') - resources: + dataList: - apiVersion: v1 kind: NetworkPolicy name: '{{ element }}' @@ -71,7 +75,7 @@ spec: - Egress ``` -2. Generate ConfigMap and Secret for each container: +2. Generate `ConfigMap` and `Secret` for each container: ```yaml spec: rules: @@ -85,7 +89,7 @@ spec: synchronize: true foreach: - list: request.object.spec.containers - resources: + dataList: - apiVersion: v1 kind: ConfigMap name: cm @@ -100,7 +104,7 @@ spec: foo: bar ``` -3. Generate Gateway and HTTPRoute from Ingress: +3. Generate `Gateway` and `HTTPRoute` from Ingress: ```yaml spec: rules: @@ -114,7 +118,7 @@ spec: synchronize: false foreach: - list: request.object.spec.rules - resources: + dataList: - apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute # the name must be unique @@ -125,7 +129,7 @@ spec: hostnames: - '{{ element.host }}' - list: request.object.spec.ingressClassName - resources: + dataList: - apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway name: gateway @@ -134,36 +138,45 @@ spec: gatewayClassName: '{{ element }}' ``` -# Implementation - -We will extend the generate rule as follows: -```go -type Generation struct { - ForEachGeneration []ForEachGeneration `json:"foreach,omitempty" yaml:"foreach,omitempty"` -} -``` - -```go -type ForEachGeneration struct { - List string `json:"list,omitempty" yaml:"list,omitempty"` - - Resource []Resource `json:"resources,omitempty" yaml:"resources,omitempty"` -} -``` - -```go -type Resource struct { - ResourceSpec `json:",omitempty" yaml:",omitempty"` - - RawData *apiextv1.JSON `json:"data,omitempty" yaml:"data,omitempty"` -} +4. Clone the `Secret` to a list of `Namespace` looked up by the source `Secret`'s label: +```yaml +spec: + rules: + - name: clone-seret-to-multiple-namespace + match: + any: + - resources: + kinds: + - Secret + namespaces: + - default + context: + - variable: + name: selector + jmesPath: request.object.metadata.labels.sync + default: a=b + - name: namespaces + apiCall: + urlPath: /api/v1/namespaces?labelSelector={{selector}}?limit=5 + generate: + synchronize: false + foreach: + - list: "{{namespaces}}" + cloneList: + - source: + namespace: default + name: regcred + kind: v1 + apiVersion: Secret + target: + namespace: "{{element}}" + name: regcred ``` -- `foreach` will only support generating Kubernetes resources that are defined as a part of the rule. - -- Nested `foreach` can be supported but there is no real use case for it. +Note a generate rule allows only one of the following `data`, `clone`, `cloneList` and `foreach` to be declared. -- At least one of the following: `data`, `clone`, or `foreach` must be declared. +# Implementation +tbd # Drawbacks @@ -179,4 +192,4 @@ N/A # Unresolved Questions -N/A +- Support nested `foreach`? From eb3a306344b8f91398e23ce58ad1a72a6ecbb67e Mon Sep 17 00:00:00 2001 From: ShutingZhao Date: Thu, 25 Jul 2024 19:09:06 +0800 Subject: [PATCH 3/3] update proposal Signed-off-by: ShutingZhao --- .../extend_foreach_to_cover_generate_rules.md | 209 +++++++++++++----- 1 file changed, 158 insertions(+), 51 deletions(-) diff --git a/proposals/extend_foreach_to_cover_generate_rules.md b/proposals/extend_foreach_to_cover_generate_rules.md index 54695c8..ea3063a 100644 --- a/proposals/extend_foreach_to_cover_generate_rules.md +++ b/proposals/extend_foreach_to_cover_generate_rules.md @@ -1,9 +1,9 @@ -# Support `forEach` in Generate Rules +# Support `foreach` in Generate Rules - **Authors**: [Mariam Fahmy](https://github.com/MariamFahmy98), [Shuting Zhao](https://github.com/realshuting) - **Created**: Nov 11th, 2023 - **Updated**: Jul 18th, 2024 -- **Abstract**: Supporting `forEach` in Generate Rules +- **Abstract**: Supporting `foreach` in Generate Rules # Table of Contents - [Overview](#overview) @@ -20,7 +20,7 @@ [overview]: #overview Generate rules are in charge of creating new resources. The rule can either generate the new resource using the `data` object or clone from an existing resource with the `clone` object. Currently, both generate patterns create single target resource. -The `forEach` declaration is supported in validate and mutate rules to simplify rule applications of sub-elements in resource declarations. This proposal is to extend `forEach` support to offer flexible options for resource generation. +The `foreach` declaration is supported in validate and mutate rules to simplify rule applications of sub-elements in resource declarations. This proposal is to extend `foreach` support to offer flexible options for resource generation. # Definitions [definitions]: #definitions @@ -36,19 +36,94 @@ The `forEach` declaration is supported in validate and mutate rules to simplify There are some use cases where a single "trigger" resource should result in the creation of multiple downstream resources. -Examples: -1. Generate a list of new `NetWorkPolicy` based on a comma separated string in an annotation using a `forEach` and `Split()`. +Use cases: +1. Generate a list of new `NetWorkPolicy` based on a comma separated string in an annotation using a `foreach` and `Split()`. 2. Migrate from ingress to Kubernetes Gateway API requires generating at least two resources: `Gateway` and `HTTPRoute`. 3. Create several `ProjectIamMember` Crossplane GCP CRs based on a comma-separated label of a namespace. 4. Clone the `Secret` to a list of `Namespace` looked up by the source `Secret`'s label. # Proposal -In this proposal, the generate rule can be expanded to support the creation of multiple resources or the cloning of a source to multiple targets using the `forEach` declaration. -This could be done by using the `list` attribute which is a JMESPath expression that defines the sub-elements it processes. +Based on the use cases mentioned above, we can introduce `foreach` for generate rules in two phases: +- Phase 1: support the generation or cloning of a list of resources for each generation rule by adding `dataList` or `cloneList` to the generate pattern; +- Phase 2: support iteration through the given list using the `foreach` declaration. + +Let's break down each phase to following: + +## Phase 1 - support `dataList` and `cloneList` + +To allow generation of multiple downstream resources for a single trigger, `dataList` and `cloneList` can be added to `generate` pattern. + +The following rule snippet showcases generation of a `NetworkPolicy` and a `ConfigMap` using `dataList`: +```yaml +spec: + rules: + - name: generate-multiple-targets + match: + any: + - resources: + kinds: + - Namespace + generate: + dataList: + - apiVersion: v1 + kind: NetworkPolicy + name: default-networkpolicy + namespace: "{{ request.object.metadata.name }}" + data: + spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + - apiVersion: v1 + kind: ConfigMap + name: default-configmap + namespace: "{{ request.object.metadata.name }}" + data: + foo: bar +``` + +The following rule clones a source secret `regcred` and a configmap `default-configmap` from `default` namespace to the new created namespace using `cloneList`. Note that the [cloneList](https://github.com/kyverno/kyverno/blob/main/api/kyverno/v1/common_types.go#L736) is already supported which allows cloning sources by label selectors. To make it backward compatible, this proposal introduces new `list` under `cloneList` pattern to provide flexibility when selecting sources. + +```yaml +spec: + rules: + - name: clone-multiple-sources + match: + any: + - resources: + kinds: + - Namespace + generate: + cloneList: + list: + - source: + namespace: default + name: default-regcred + kind: v1 + apiVersion: Secret + target: + namespace: "{{ request.object.metadata.name }}" + name: regcred + - source: + namespace: default + name: default-configmap + kind: v1 + apiVersion: ConfigMap + target: + namespace: "{{ request.object.metadata.name }}" + name: cloned-configmap +``` + +## Phase 2 - support `foreach` declaration + +In the second phase, the generate rule can be expanded to support `foreach` declaration in order to iterate through given list. This could be done by adding the `foreach` key under generate pattern, and a `list` attribute which is a JMESPath expression that defines the sub-elements it processes. For every `foreach` pattern, either `dataList` or `cloneList` will be supported. + +The following example defines a generate rule which takes a list of namespaces, which is derived from a comma-separated label value, and creates networkpolicies into new namespaces. + +Since the resource names for a kind needs to be unique within the same namespace, the user needs to define a dynamic name for the downstream resource, i.e., add `{{elementIndex}}` to the name. -Examples: -1. Generate multiple `NetWorkPolicy` when a new `Namespace` is created: ```yaml spec: rules: @@ -59,14 +134,13 @@ spec: kinds: - Namespace generate: - synchronize: true foreach: - list: request.object.metadata.labels.networkpolicies | split(@, ',') dataList: - apiVersion: v1 kind: NetworkPolicy - name: '{{ element }}' - namespace: "{{request.object.metadata.name}}" + name: my-networkpolicy-{{ elementIndex }} + namespace: "{{ request.object.metadata.name }}" data: spec: podSelector: {} @@ -75,7 +149,42 @@ spec: - Egress ``` -2. Generate `ConfigMap` and `Secret` for each container: +This example takes a list of namespaces, which is looked up by the `apiCall`, and clones the `Secret` to corresponding namespaces: +```yaml +spec: + rules: + - name: clone-secret-to-multiple-namespace + match: + any: + - resources: + kinds: + - Secret + namespaces: + - default + context: + - variable: + name: selector + jmesPath: request.object.metadata.labels.sync + default: a=b + - name: namespaces + apiCall: + urlPath: /api/v1/namespaces?labelSelector={{selector}}?limit=5 + generate: + foreach: + - list: "{{ namespaces }}" + cloneList: + - source: + namespace: default + name: regcred + kind: v1 + apiVersion: Secret + target: + namespace: "{{ element }}" + name: regcred +``` + +More examples using `foreach`: +1. Generate `ConfigMap` and `Secret` for each container: ```yaml spec: rules: @@ -86,25 +195,24 @@ spec: kinds: - Pod generate: - synchronize: true foreach: - list: request.object.spec.containers dataList: - apiVersion: v1 kind: ConfigMap - name: cm - namespace: "{{request.object.metadata.namespace}}" + name: my-configmap-{{ elementIndex }} + namespace: "{{ request.object.metadata.namespace }}" data: foo: bar - apiVersion: v1 kind: Secret - name: secret - namespace: "{{request.object.metadata.namespace}}" + name: my-secret-{{ elementIndex }} + namespace: "{{ request.object.metadata.namespace }}" data: foo: bar ``` -3. Generate `Gateway` and `HTTPRoute` from Ingress: +2. Generate `Gateway` and `HTTPRoute` from Ingress: ```yaml spec: rules: @@ -115,15 +223,14 @@ spec: kinds: - Ingress generate: - synchronize: false foreach: - list: request.object.spec.rules dataList: - apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute # the name must be unique - name: httproute - namespace: "{{request.object.metadata.namespace}}" + name: httproute-{{ elementIndex }} + namespace: "{{ request.object.metadata.namespace }}" data: spec: hostnames: @@ -132,49 +239,50 @@ spec: dataList: - apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway - name: gateway + name: gateway-{{ elementIndex }} data: spec: gatewayClassName: '{{ element }}' ``` -4. Clone the `Secret` to a list of `Namespace` looked up by the source `Secret`'s label: +Note a generate rule allows only one of the following `data`, `clone`, `dataList` and `cloneList` to be declared. + +## Support `context` and `preconditions` for each sub-element + +Similar to mutate and validate `foreach`, the `context` and the `preconditions` attributes are supported for generate rules for finer conditional checks. + +The following `foreach` rule looks up the label value from the incoming request, and creates a configmap for each container if the label matches static value `foo=bar`: + ```yaml spec: rules: - - name: clone-seret-to-multiple-namespace + - name: generate-gatewayapi match: any: - resources: kinds: - - Secret - namespaces: - - default - context: - - variable: - name: selector - jmesPath: request.object.metadata.labels.sync - default: a=b - - name: namespaces - apiCall: - urlPath: /api/v1/namespaces?labelSelector={{selector}}?limit=5 + - Pod generate: - synchronize: false foreach: - - list: "{{namespaces}}" - cloneList: - - source: - namespace: default - name: regcred - kind: v1 - apiVersion: Secret - target: - namespace: "{{element}}" - name: regcred + - list: request.object.spec.containers + context: + - variable: + name: selector + jmesPath: request.object.metadata.labels.sync + default: a=b + preconditions: + all: + - key: "{{selector}}" + operator: Equals + value: foo=bar + dataList: + - apiVersion: v1 + kind: ConfigMap + name: my-configmap-{{ elementIndex }} + namespace: "{{ request.object.metadata.namespace }}" + data: + foo: bar ``` - -Note a generate rule allows only one of the following `data`, `clone`, `cloneList` and `foreach` to be declared. - # Implementation tbd @@ -192,4 +300,3 @@ N/A # Unresolved Questions -- Support nested `foreach`?