Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added documentation for DeletionPropagationPolicy for cleanupPolicy a… #1426

Merged
76 changes: 76 additions & 0 deletions content/en/docs/writing-policies/cleanup.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,50 @@ rules:
- delete
```

### Deletion Propagation Policy
ShivamJha2436 marked this conversation as resolved.
Show resolved Hide resolved

The `deletionPropagationPolicy` field is an optional setting in `CleanupPolicy` and `ClusterCleanupPolicy` that specifies how Kubernetes should handle the deletion of dependent resources. The available options are:

- **Foreground**: Deletes the resource and waits until all of its dependent resources are also deleted.
- **Background**: Deletes the resource immediately, and its dependents are deleted asynchronously.
- **Orphan**: Deletes the resource without deleting its dependents, leaving them orphaned.

> **Note**: If `deletionPropagationPolicy` is not set, Kyverno defaults to the API server’s behavior. This default allows Kyverno to delete the primary resource asynchronously, giving the API server the flexibility to manage the deletion of dependents as per cluster settings.

An example `ClusterCleanupPolicy` with `deletionPropagationPolicy` is shown below. This cleanup policy removes Deployments with the label `canremove: "true"` if they have fewer than two replicas, on a schedule of every 5 minutes, and deletes dependents in the **Foreground** mode.

```yaml
# ClusterCleanupPolicy with deletionPropagationPolicy
apiVersion: kyverno.io/v2
kind: ClusterCleanupPolicy
metadata:
name: cleandeploy
spec:
match:
any:
- resources:
kinds:
- Deployment
selector:
matchLabels:
canremove: "true"
conditions:
any:
- key: "{{ target.spec.replicas }}"
operator: LessThan
value: 2
schedule: "*/5 * * * *"
deletionPropagationPolicy: "Foreground"
```

{{% alert title="Note" color="info" %}} Since cleanup policies always operate against existing resources in a cluster, policies created with subjects, Roles, or ClusterRoles in the match/exclude block are not allowed since this information is only known at admission time. Additionally, operations[], while permitted, are ignored as the only trigger is schedule based. {{% /alert %}}

Values from resources to be evaluated during a policy may be referenced with target.* similar to mutate existing rules.

Because Kyverno follows the principle of least privilege, depending on the resources you wish to remove it may be necessary to grant additional permissions to the cleanup controller. Kyverno will assist in informing you if additional permissions are required by validating them at the time a new cleanup policy is installed. See the Customizing Permissions section for more details.
ShivamJha2436 marked this conversation as resolved.
Show resolved Hide resolved

An example ClusterRole which allows Kyverno to cleanup Pods is shown below. This may need to be customized based on the values used to deploy Kyverno.

## Cleanup Label

In addition to policies which can declaratively define what resources to remove and when to remove them, the second option for cleanup involves assignment of a reserved label called `cleanup.kyverno.io/ttl` to the exact resource(s) which should be removed. The value of this label can be one of two supported formats. Any unrecognized formats will trigger a warning.
Expand Down Expand Up @@ -96,3 +140,35 @@ spec:
Although labeled resources are watched by Kyverno, the cleanup interval (the time resolution at which any cleanup can be performed) is controlled by a flag passed to the cleanup controller called `ttlReconciliationInterval`. This value is set to `1m` by default and can be changed if a longer resolution is required.

Because this is a label, there is opportunity to chain other Kyverno functionality around it. For example, it is possible to use a Kyverno mutate rule to assign this label to matching resources. A validate rule could be written prohibiting, for example, users from the `infra-ops` group from assigning the label to resources in certain Namespaces. Or, Kyverno could generate a new resource with this label as part of the resource definition.

### Deletion Propagation Policy in TTL-based Cleanup

The deletionPropagationPolicy can also be specified for resources with a TTL-based cleanup label. This field provides flexibility in handling dependent resources when a resource reaches its expiration. For example:

- **Foreground**: Ensures that all dependent resources are deleted before the resource itself is removed.
- **Background**: Deletes the resource first, while dependents are removed asynchronously.
- **Orphan**: Deletes the resource but leaves its dependents in place.

For example, consider a Pod with the TTL label cleanup.kyverno.io/ttl: 2m. After two minutes, the Pod will be deleted, but the dependents will be handled according to the specified deletionPropagationPolicy. If the policy is set to Orphan, the Pod will be deleted, but its dependent resources will remain in the cluster.
ShivamJha2436 marked this conversation as resolved.
Show resolved Hide resolved

```yaml
# TTL-based cleanup with deletionPropagationPolicy
apiVersion: v1
kind: Pod
metadata:
labels:
cleanup.kyverno.io/ttl: 2m
name: foo
spec:
containers:
- args:
- sleep
- 1d
image: busybox:1.35
name: foo
deletionPropagationPolicy: "Orphan"
ShivamJha2436 marked this conversation as resolved.
Show resolved Hide resolved
```
ShivamJha2436 marked this conversation as resolved.
Show resolved Hide resolved

In this example, the TTL is set to 2m, which removes the Pod after two minutes but leaves any dependent resources because of the Orphan policy.

Although labeled resources are watched by Kyverno, the cleanup interval (the time resolution at which any cleanup can be performed) is controlled by a flag passed to the cleanup controller called ttlReconciliationInterval. This value is set to 1m by default and can be changed if a longer resolution is required.