Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fresh objects for Synchronization executions #261

Merged
merged 1 commit into from
Mar 18, 2021
Merged

fix: fresh objects for Synchronization executions #261

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,6 @@ jobs:
run: |
go build ./cmd/shell-operator


# MacOS build works fine because jq package already has static libraries.
# Windows build requires jq compilation, this should be done in libjq-go.
# TODO Does cross-compile can help here?
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,9 +279,8 @@ jobs:
strategy:
fail-fast: true
matrix:
os: [ubuntu-latest]
k8s_version: [1.16, 1.17]
runs-on: ${{ matrix.os }}
k8s_version: [1.16, 1.19, 1.20]
runs-on: ubuntu-latest
env:
ACTIONS_ALLOW_UNSECURE_COMMANDS: true
steps:
Expand Down
203 changes: 87 additions & 116 deletions HOOKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,18 +346,18 @@ Temporary files have unique names to prevent collisions between queues and are d
Binging context is a JSON-array of structures with the following fields:

- `binding` — a string from the `name` or `group` parameters. If these parameters has not been set in the binding configuration, then strings "schedule" or "kubernetes" are used. For a hook executed at startup, this value is always "onStartup".
- `type` — "Schedule" for `schedule` bindings. "Synchronization" or "Event" for `kubernetes` bindings. "Synchronization" or "Group" if `group` is defined.
- `type` — "Schedule" for `schedule` bindings. "Synchronization" or "Event" for `kubernetes` bindings. "Group" if `group` is defined.

The hook receives "Event"-type binding context on Kubernetes event and it contains more fields:
- `watchEvent` — the possible value is one of the values you can use with `executeHookOnEvent` parameter: "Added", "Modified" or "Deleted".
- `object` — a JSON dump of the full object related to the event. It contains an exact copy of the corresponding field in [WatchEvent](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#watchevent-v1-meta) response, so it's the object state **at the moment of the event** (not at the moment of the hook execution).
- `filterResult` — the result of `jq` execution with specified `jqFilter` on the above mentioned object. If `jqFilter` is not specified, then `filterResult` is omitted.

The hook receives existed objects on startup for each binding with "Synchronization"-type binding context:
- `objects` — a list of existing objects that match selectors in binding configuration. Each item of this list contains `object` and `filterResult` fields. If the list is empty, the value of `objects` is an empty array.
- `objects` — a list of existing objects that match selectors in binding configuration. Each item of this list contains `object` and `filterResult` fields. The state of items is actual **for the moment of the hook execution**. If the list is empty, the value of `objects` is an empty array.

If `group` or `includeSnapshotsFrom` are defined, the hook receives binding context with additional field:
- `snapshots` — a map that contains a list of objects for each binding name from `includeSnapshotsFrom` or for each `kubernetes` binding in a group. If `includeSnapshotsFrom` list is empty, the field is omitted.
- `snapshots` — a map that contains an up-to-date lists of objects for each binding name from `includeSnapshotsFrom` or for each `kubernetes` binding with a similar `group`. If `includeSnapshotsFrom` list is empty, the field is omitted.

### `onStartup` binding context example

Expand Down Expand Up @@ -438,6 +438,8 @@ During startup, the hook receives all existing objects with "Synchronization"-ty
]
```

> Note: hook execution at startup with "Synchronization" binding context can be turned off with `executeHookOnSynchronization: false`

#### "Event" binding context

If pod `pod-321d12` is then added into namespace 'default', then the hook will be executed with the "Event"-type binding context:
Expand Down Expand Up @@ -467,13 +469,17 @@ If pod `pod-321d12` is then added into namespace 'default', then the hook will b

### Snapshots

Shell-operator caches a list of resources for each `kubernetes` binding. Another bindings can access this list via `includeSnapshotsFrom` parameter. Also, there is a `group` parameter to automatically get all snapshots from multiple bindings and deduplicate executions.
"Event"-type binding context contains an object state at the moment of the event. Actual objects' state for the moment of the execution can be received in a form of _Snapshots_.

Shell-operator maintains an up-to-date list of objects for each `kubernetes` binding. `schedule` and `kubernetes` bindings can be configured to receive these lists via `includeSnapshotsFrom` parameter. Also, there is a `group` parameter to automatically receive all snapshots from multiple bindings and to deduplicate executions.

Snapshot is a list of cached kubernetes objects and corresponding jqFilter results. To access the snapshot from particular binding, there is a map `snapshots` in the binding context where the key is a binding name and the value is the snapshot.
Snapshot is a JSON array of Kubernetes objects and corresponding jqFilter results. To access the snapshot during the hook execution, there is a map `snapshots` in the binding context. The key of this map is a binding name, and the value is the snapshot.

`snapshots` format:
`snapshots` example:

```yaml
[
{ "binding": ...,
"snapshots": {
"binding-name-1": [
{
Expand All @@ -489,10 +495,10 @@ Snapshot is a list of cached kubernetes objects and corresponding jqFilter resul
},
...
]
}
}}]
```

- `object` — it is a JSON dump of Kubernetes object.
- `object` — a JSON dump of Kubernetes object.
- `filterResult` — a JSON result of applying `jqFilter` to the Kubernetes object.

Keeping dumps for `object` fields can take a lot of memory. There is a parameter `keepFullObjectsInMemory: false` to disable full dumps.
Expand All @@ -508,21 +514,31 @@ kubernetes:
keepFullObjectsInMemory: false
```

To illustrate `includeSnapshotsFrom` parameter, consider the hook that monitors changes of labels of all Pods and do something interesting on schedule:
### Snapshots example

To illustrate the `includeSnapshotsFrom` parameter, consider the hook that reacts to changes of labels of all Pods and requires the content of the ConfigMap named "settings-for-my-hook". There is also a schedule to do periodic checks:

```yaml
configVersion: v1
schedule:
- name: incremental
crontab: "0 2 */3 * * *"
includeSnapshotsFrom: ["monitor-pods"]
- name: periodic-checking
crontab: "0 */3 * * *"
includeSnapshotsFrom: ["monitor-pods", "cm"]
kubernetes:
- name: configmap-content
kind: ConfigMap
nameSelector:
matchNames: ["settings-for-my-hook"]
executeHookOnSynchronization: false
executeHookOnEvent: []
- name: monitor-pods
kind: Pod
jqFilter: '.metadata.labels'
includeSnapshotsFrom: ["monitor-pods"]
includeSnapshotsFrom: ["cm"]
```

This hook will not be executed for events related to the binding "configmap-content". `executeHookOnSynchronization: false` accompanied by `executeHookOnEvent: []` defines a "snapshot-only" binding. This is one of the techniques to reduce the number of `kubectl` invocations.

#### "Synchronization" binding context with snapshots

During startup, the hook will be executed with the "Synchronization" binding context with `snapshots` and `objects`:
Expand Down Expand Up @@ -565,19 +581,14 @@ During startup, the hook will be executed with the "Synchronization" binding con
...
],
"snapshots": {
"monitor-pods": [
"configmap-content": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
Expand Down Expand Up @@ -609,19 +620,14 @@ If pod `pod-321d12` is then added into the "default" namespace, then the hook wi
},
"filterResult": { ... },
"snapshots": {
"monitor-pods": [
"configmap-content": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
Expand All @@ -630,12 +636,12 @@ If pod `pod-321d12` is then added into the "default" namespace, then the hook wi

#### "Schedule" binding context with snapshots

at 12:02, the hook will be executed with the following binding context:
Every 3 hours, the hook will be executed with the binding context that include 2 snapshots ("monitor-pods" and "configmap-content"):

```yaml
[
{
"binding": "incremental",
"binding": "periodic-checking",
"type": "Schedule",
"snapshots": {
"monitor-pods": [
Expand All @@ -651,6 +657,15 @@ at 12:02, the hook will be executed with the following binding context:
"filterResult": { ... },
},
...
],
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
Expand All @@ -659,49 +674,67 @@ at 12:02, the hook will be executed with the following binding context:

### Binding context of grouped bindings

`group` parameter defines a named group of bindings. Group is used when the source of event is not important and data in snapshots is enough for the hook. When binding with `group` is triggered with the event, the hook receives snapshots from all bindings with equal `group` name. Also, adjacent tasks with equal `group` in the same queue are "compacted" and hook is executed only once. So it is wise to use the same queue for all hooks in a group.
`group` parameter defines a named group of bindings. Group is used when the source of the event is not important, and data in snapshots is enough for the hook. When binding with `group` is triggered with the event, the hook receives snapshots from all `kubernetes` bindings with the same `group` name.

`executeHookOnSynchronization`, `executeHookOnEvent` and `keepFullObjectsInMemory` can be used with `group`.
Adjacent tasks for `kubernetes` and `schedule` bindings with the same `group` and `queue` are "compacted", and the hook is executed only once. So it is wise to use the same `queue` for all hooks in a group. This "compaction" mechanism is not available for `kubernetesValidating` and `kubernetesCustomResourceConversion` bindings as they're not queued.

`executeHookOnSynchronization`, `executeHookOnEvent` and `keepFullObjectsInMemory` can be used with `group`. Their effects are as described above for non-grouped bindings.

`group` parameter is compatible with `includeSnapshotsFrom` parameter. `includeSnapshotsFrom` can be used to include additional snapshots into binding context.

Binding context for group contains:
- `binding` field with group name.
- `type` field with "Synchronization" or "Group" string.
- `snapshots` field if there is at least one `kubernetes` binding in the group and in `includeSnapshotsFrom`.
- `binding` field with the group name.
- `type` field with the value "Group".
- `snapshots` field if there is at least one `kubernetes` binding in the group or `includeSnapshotsFrom` is not empty.

### Group binding context example

Consider the hook that is executed on changes of labels of all Pods, changes in ConfigMap and also on schedule:
Consider the hook that is executed on changes of labels of all Pods, changes in ConfigMap's data and also on schedule:

```yaml
configVersion: v1
schedule:
- name: incremental
crontab: "* * * * *"
- name: periodic-checking
crontab: "0 */3 * * *"
group: "pods"
kubernetes:
- name: monitor_pods
- name: monitor-pods
apiVersion: v1
kind: Pod
jqFilter: '.metadata.labels'
group: "pods"
- name: monitor_configmap
- name: configmap-content
apiVersion: v1
kind: ConfigMap
nameSelector:
matchNames: ["settings-for-my-hook"]
jqFilter: '.data'
group: "pods"
group: "pods"

```

#### "Synchronization" binding context for group
#### binding context for grouped bindings

During startup, the hook will be executed with the "Synchronization" binding context with `snapshots` JSON object:
Grouped bindings is used when only the occurrence of an event is important. So, the hook receives actual state of Pods and the ConfigMap on every of these events:

- During startup.
- A new Pod is added.
- The Pod is deleted.
- Labels of the Pod are changed.
- ConfigMap/settings-for-my-hook is deleted.
- ConfigMap/settings-for-my-hook is added.
- Data field is changed in ConfigMap/settings-for-my-hook.
- Every 3 hours.

Binding context for these events will be the same:

```yaml
[
{
"binding": "pods",
"type": "Synchronization",
"type": "Group",
"snapshots": {
"monitor_pods": [
"monitor-pods": [
{
"object": {
"kind": "Pod",
Expand All @@ -715,7 +748,7 @@ During startup, the hook will be executed with the "Synchronization" binding con
},
...
],
"monitor_configmap": [
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
Expand All @@ -734,68 +767,6 @@ During startup, the hook will be executed with the "Synchronization" binding con
]
```

#### "Group" binding context

If pod `pod-dfbd12` is then added into the "default" namespace, then the hook will be executed with the "Group" binding context:

```yaml
[
{
"binding": "pods",
"type": "Group",
"snapshots": {
"monitor_pods": [
{
"object": {
"kind": "Pod",
"metadata":{
"name":"etcd-...",
"namespace":"kube-system",
...
},
},
"filterResult": { ... },
},
...
],
"monitor_configmap": [
{
"object": {
"kind": "ConfigMap",
"metadata":{
"name":"etcd-...",
"namespace":"kube-system",
...
},
},
"filterResult": { ... },
},
...
]
}
}
]
```

Every minute it will be executed with the same binding context with fresh snapshots:

```yaml
[
{
"binding": "pods",
"type": "Group",
"snapshots": {
"monitor_pods": [
...
],
"monitor_configmaps": [
...
]
}
}
]
```

### settings

An optional block with hook-level settings.
Expand Down Expand Up @@ -834,4 +805,4 @@ settings:
executionBurst: 1
```

Hook with these settings will be executed once in 3 seconds.
If the Shell-operator will receive a lot of events for the "all-pods-in-ns" binding, the hook will be executed no more than once in 3 seconds.
Loading