Skip to content

Commit

Permalink
++ changes from review
Browse files Browse the repository at this point in the history
  • Loading branch information
@diafour
diafour committed Mar 17, 2021
1 parent ab553a7 commit a8278c8
Show file tree
Hide file tree
Showing 5 changed files with 118 additions and 138 deletions.
201 changes: 86 additions & 115 deletions HOOKS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -346,18 +346,18 @@ Temporary files have unique names to prevent collisions between queues and are d
Binging context is a JSON-array of structures with the following fields:

- `binding` — a string from the `name` or `group` parameters. If these parameters has not been set in the binding configuration, then strings "schedule" or "kubernetes" are used. For a hook executed at startup, this value is always "onStartup".
- `type` — "Schedule" for `schedule` bindings. "Synchronization" or "Event" for `kubernetes` bindings. "Synchronization" or "Group" if `group` is defined.
- `type` — "Schedule" for `schedule` bindings. "Synchronization" or "Event" for `kubernetes` bindings. "Group" if `group` is defined.

The hook receives "Event"-type binding context on Kubernetes event and it contains more fields:
- `watchEvent` — the possible value is one of the values you can use with `executeHookOnEvent` parameter: "Added", "Modified" or "Deleted".
- `object` — a JSON dump of the full object related to the event. It contains an exact copy of the corresponding field in [WatchEvent](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#watchevent-v1-meta) response, so it's the object state **at the moment of the event** (not at the moment of the hook execution).
- `filterResult` — the result of `jq` execution with specified `jqFilter` on the above mentioned object. If `jqFilter` is not specified, then `filterResult` is omitted.

The hook receives existed objects on startup for each binding with "Synchronization"-type binding context:
- `objects` — a list of existing objects that match selectors in binding configuration. Each item of this list contains `object` and `filterResult` fields. If the list is empty, the value of `objects` is an empty array.
- `objects` — a list of existing objects that match selectors in binding configuration. Each item of this list contains `object` and `filterResult` fields. The state of items is actual **for the moment of the hook execution**. If the list is empty, the value of `objects` is an empty array.

If `group` or `includeSnapshotsFrom` are defined, the hook receives binding context with additional field:
- `snapshots` — a map that contains a list of objects for each binding name from `includeSnapshotsFrom` or for each `kubernetes` binding in a group. If `includeSnapshotsFrom` list is empty, the field is omitted.
- `snapshots` — a map that contains an up-to-date lists of objects for each binding name from `includeSnapshotsFrom` or for each `kubernetes` binding with a similar `group`. If `includeSnapshotsFrom` list is empty, the field is omitted.

### `onStartup` binding context example

Expand Down Expand Up @@ -438,6 +438,8 @@ During startup, the hook receives all existing objects with "Synchronization"-ty
]
```

> Note: hook execution at startup with "Synchronization" binding context can be turned off with `executeHookOnSynchronization: false`

#### "Event" binding context

If pod `pod-321d12` is then added into namespace 'default', then the hook will be executed with the "Event"-type binding context:
Expand Down Expand Up @@ -467,13 +469,17 @@ If pod `pod-321d12` is then added into namespace 'default', then the hook will b

### Snapshots

Shell-operator caches a list of resources for each `kubernetes` binding. Another bindings can access this list via `includeSnapshotsFrom` parameter. Also, there is a `group` parameter to automatically get all snapshots from multiple bindings and deduplicate executions.
"Event"-type binding context contains an object state at the moment of the event. Actual objects' state for the moment of the execution can be received in a form of _Snapshots_.

Shell-operator maintains an up-to-date list of objects for each `kubernetes` binding. `schedule` and `kubernetes` bindings can be configured to receive these lists via `includeSnapshotsFrom` parameter. Also, there is a `group` parameter to automatically receive all snapshots from multiple bindings and to deduplicate executions.

Snapshot is a list of cached kubernetes objects and corresponding jqFilter results. To access the snapshot from particular binding, there is a map `snapshots` in the binding context where the key is a binding name and the value is the snapshot.
Snapshot is a JSON array of with Kubernetes objects and corresponding jqFilter results. To access the snapshot during the hook execution, there is a map `snapshots` in the binding context. The key of this map is a binding name and the value is the snapshot.

`snapshots` format:
`snapshots` example:

```yaml
[
{ "binding": ...,
"snapshots": {
"binding-name-1": [
{
Expand All @@ -489,10 +495,10 @@ Snapshot is a list of cached kubernetes objects and corresponding jqFilter resul
},
...
]
}
}}]
```

- `object` — it is a JSON dump of Kubernetes object.
- `object` — a JSON dump of Kubernetes object.
- `filterResult` — a JSON result of applying `jqFilter` to the Kubernetes object.

Keeping dumps for `object` fields can take a lot of memory. There is a parameter `keepFullObjectsInMemory: false` to disable full dumps.
Expand All @@ -508,21 +514,31 @@ kubernetes:
keepFullObjectsInMemory: false
```

To illustrate `includeSnapshotsFrom` parameter, consider the hook that monitors changes of labels of all Pods and do something interesting on schedule:
### Snapshots example

To illustrate the `includeSnapshotsFrom` parameter, consider the hook that reacts to changes of labels of all Pods and requires the content of the ConfigMap named "settings-for-my-hook". There is also a schedule to do periodic checks:

```yaml
configVersion: v1
schedule:
- name: incremental
crontab: "0 2 */3 * * *"
includeSnapshotsFrom: ["monitor-pods"]
- name: periodic-checking
crontab: "0 */3 * * *"
includeSnapshotsFrom: ["monitor-pods", "cm"]
kubernetes:
- name: configmap-content
kind: ConfigMap
nameSelector:
matchNames: ["settings-for-my-hook"]
executeHookOnSynchronization: false
executeHookOnEvent: []
- name: monitor-pods
kind: Pod
jqFilter: '.metadata.labels'
includeSnapshotsFrom: ["monitor-pods"]
includeSnapshotsFrom: ["cm"]
```

This hook will not be executed for events related to the binding "configmap-content". `executeHookOnSynchronization: false` accompanied by `executeHookOnEvent: []` defines a "snapshot-only" binding. This is one of the techniques to reduce the number of `kubectl` invocations.

#### "Synchronization" binding context with snapshots

During startup, the hook will be executed with the "Synchronization" binding context with `snapshots` and `objects`:
Expand Down Expand Up @@ -565,19 +581,14 @@ During startup, the hook will be executed with the "Synchronization" binding con
...
],
"snapshots": {
"monitor-pods": [
"configmap-content": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
Expand Down Expand Up @@ -609,19 +620,14 @@ If pod `pod-321d12` is then added into the "default" namespace, then the hook wi
},
"filterResult": { ... },
"snapshots": {
"monitor-pods": [
"configmap-content": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
Expand All @@ -630,12 +636,12 @@ If pod `pod-321d12` is then added into the "default" namespace, then the hook wi

#### "Schedule" binding context with snapshots

at 12:02, the hook will be executed with the following binding context:
Every 3 hours, the hook will be executed with the binding context that include 2 snapshots ("monitor-pods" and "configmap-content"):

```yaml
[
{
"binding": "incremental",
"binding": "periodic-checking",
"type": "Schedule",
"snapshots": {
"monitor-pods": [
Expand All @@ -651,6 +657,15 @@ at 12:02, the hook will be executed with the following binding context:
"filterResult": { ... },
},
...
],
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
Expand All @@ -659,49 +674,67 @@ at 12:02, the hook will be executed with the following binding context:

### Binding context of grouped bindings

`group` parameter defines a named group of bindings. Group is used when the source of event is not important and data in snapshots is enough for the hook. When binding with `group` is triggered with the event, the hook receives snapshots from all bindings with equal `group` name. Also, adjacent tasks with equal `group` in the same queue are "compacted" and hook is executed only once. So it is wise to use the same queue for all hooks in a group.
`group` parameter defines a named group of bindings. Group is used when the source of event is not important and data in snapshots is enough for the hook. When binding with `group` is triggered with the event, the hook receives snapshots from all `kubernetes` bindings with the same `group` name.

Adjacent tasks for `kubernetes` and `schedule` bindings with the same `group` and `queue` are "compacted" and hook is executed only once. So it is wise to use the same `queue` for all hooks in a group. This "compaction" mechanism is not available for `kubernetesValidating` and `kubernetesCustomResourceConversion` bindings as they're not queued.

`executeHookOnSynchronization`, `executeHookOnEvent` and `keepFullObjectsInMemory` can be used with `group`.
`executeHookOnSynchronization`, `executeHookOnEvent` and `keepFullObjectsInMemory` can be used with `group`. Their effects are as described above for non-grouped bindings.

`group` parameter is compatible with `includeSnapshotsFrom` parameter. `includeSnapshotsFrom` can be used to include additional snapshots into binding context.

Binding context for group contains:
- `binding` field with group name.
- `type` field with "Synchronization" or "Group" string.
- `snapshots` field if there is at least one `kubernetes` binding in the group and in `includeSnapshotsFrom`.
- `binding` field with the group name.
- `type` field with the value "Group".
- `snapshots` field if there is at least one `kubernetes` binding in the group or `includeSnapshotsFrom` is not empty.

Consider the hook that is executed on changes of labels of all Pods, changes in ConfigMap and also on schedule:
### Group binding context example

Consider the hook that is executed on changes of labels of all Pods, changes in ConfigMap's data and also on schedule:

```yaml
configVersion: v1
schedule:
- name: incremental
crontab: "* * * * *"
- name: periodic-checking
crontab: "0 */3 * * *"
group: "pods"
kubernetes:
- name: monitor_pods
- name: monitor-pods
apiVersion: v1
kind: Pod
jqFilter: '.metadata.labels'
group: "pods"
- name: monitor_configmap
- name: configmap-content
apiVersion: v1
kind: ConfigMap
nameSelector:
matchNames: ["settings-for-my-hook"]
jqFilter: '.data'
group: "pods"
group: "pods"
```

#### "Synchronization" binding context for group
#### binding context for grouped bindings

Grouped bindings is used when only the occurrence of an event is important. So, the hook receives actual state of Pods and the ConfigMap on every of these events:

- During startup.
- A new Pod is added.
- The Pod is deleted.
- Labels of the Pod are changed.
- ConfigMap/settings-for-my-hook is deleted.
- ConfigMap/settings-for-my-hook is added.
- Data field is changed in ConfigMap/settings-for-my-hook.
- Every 3 hours.

During startup, the hook will be executed with the "Group" binding context with `snapshots` JSON object that contains all objects from bindings in the group. Note there is no `objects` field as for "normal" binding:
Binding context for these events will be the same:

```yaml
[
{
"binding": "pods",
"type": "Group",
"snapshots": {
"monitor_pods": [
"monitor-pods": [
{
"object": {
"kind": "Pod",
Expand All @@ -715,7 +748,7 @@ During startup, the hook will be executed with the "Group" binding context with
},
...
],
"monitor_configmap": [
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
Expand All @@ -734,68 +767,6 @@ During startup, the hook will be executed with the "Group" binding context with
]
```

#### "Event" binding context for group

Hook is executed on event from every binding in group. So, if pod `pod-dfbd12` is added into the "default" namespace, then the hook will be executed with the "Group" binding context with `snapshots` JSON object that contains all objects from bindings in the group:

```yaml
[
{
"binding": "pods",
"type": "Group",
"snapshots": {
"monitor_pods": [
{
"object": {
"kind": "Pod",
"metadata":{
"name":"etcd-...",
"namespace":"kube-system",
...
},
},
"filterResult": { ... },
},
...
],
"monitor_configmap": [
{
"object": {
"kind": "ConfigMap",
"metadata":{
"name":"etcd-...",
"namespace":"kube-system",
...
},
},
"filterResult": { ... },
},
...
]
}
}
]
```

Every minute it will be executed with the same binding context with fresh snapshots:

```yaml
[
{
"binding": "pods",
"type": "Group",
"snapshots": {
"monitor_pods": [
...
],
"monitor_configmaps": [
...
]
}
}
]
```

### settings

An optional block with hook-level settings.
Expand Down Expand Up @@ -834,4 +805,4 @@ settings:
executionBurst: 1
```

Hook with these settings will be executed once in 3 seconds.
If the Shell-operator will receive a lot of events for the "all-pods-in-ns" binding, the hook will be executed no more than once in 3 seconds.
4 changes: 0 additions & 4 deletions pkg/kube_events_manager/kube_events_manager_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -160,10 +160,6 @@ func Test_MainKubeEventsManager_HandleEvents(t *testing.T) {
t.FailNow()
}

//mgr.Start()

fmt.Printf("mgr Started\n")

// First event — Synchronization, second and third are Event
eventCounter := 0
done := false
Expand Down
4 changes: 2 additions & 2 deletions pkg/kube_events_manager/monitor.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -212,8 +212,8 @@ func (m *monitor) Snapshot() []ObjectAndFilterResult {
return objects
}

// EnableKubeEventCb turns on event callback for each informer. This will call eventCb for
// buffered events in each informer.
// EnableKubeEventCb turns on event callback for each informer. This also calls
// eventCb for events saved during "Synchronization" hook execution.
func (m *monitor) EnableKubeEventCb() {
for _, informer := range m.ResourceInformers {
informer.EnableKubeEventCb()
Expand Down
Loading

0 comments on commit a8278c8

Please sign in to comment.