Update options and add more examples for add_entries processor (#7412)

* Update options and add more examples Signed-off-by: Hai Yan <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * A few small edits * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
opensearch-project · Jul 2, 2024 · 509d8f0 · 509d8f0
1 parent 0f889fe
commit 509d8f0
Showing 1 changed file with 181 additions and 21 deletions.
diff --git a/_data-prepper/pipelines/configuration/processors/add-entries.md b/_data-prepper/pipelines/configuration/processors/add-entries.md
@@ -10,55 +10,215 @@ nav_order: 40
 
 The `add_entries` processor adds entries to an event.
 
-### Configuration
+## Configuration
 
 You can configure the `add_entries` processor with the following options.
 
 | Option | Required | Description |
 | :--- | :--- | :--- |
 | `entries` | Yes | A list of entries to add to an event. |
-| `key` | Yes | The key of the new entry to be added. Some examples of keys include `my_key`, `myKey`, and `object/sub_Key`. |
-| `metadata_key` | Yes | The key for the new metadata attribute. The argument must be a literal string key and not a JSON Pointer. Either one string key or `metadata_key` is required. |
+| `key` | No | The key of the new entry to be added. Some examples of keys include `my_key`, `myKey`, and `object/sub_Key`. The key can also be a format expression, for example, `${/key1}` to use the value of field `key1` as the key. |
+| `metadata_key` | No | The key for the new metadata attribute. The argument must be a literal string key and not a JSON Pointer. Either one string key or `metadata_key` is required. |
+| `value` | No | The value of the new entry to be added, which can be used with any of the following data types: strings, Booleans, numbers, null, nested objects, and arrays. |
 | `format` | No | A format string to use as the value of the new entry, for example, `${key1}-${key2}`, where `key1` and `key2` are existing keys in the event. Required if neither `value` nor `value_expression` is specified. |
 | `value_expression` | No | An expression string to use as the value of the new entry. For example, `/key` is an existing key in the event with a type of either a number, a string, or a Boolean. Expressions can also contain functions returning number/string/integer. For example, `length(/key)` will return the length of the key in the event when the key is a string. For more information about keys, see [Expression syntax](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/).  |
 | `add_when` | No | A [conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. |
-| `value` | Yes | The value of the new entry to be added. You can use the following data types: strings, Booleans, numbers, null, nested objects, and arrays. |
 | `overwrite_if_key_exists` | No | When set to `true`, the existing value is overwritten if `key` already exists in the event. The default value is `false`. |
+| `append_if_key_exists` | No | When set to `true`, the existing value will be appended if a `key` already exists in the event. An array will be created if the existing value is not an array. Default is `false`. |
 
-### Usage
 
-To get started, create the following `pipeline.yaml` file:
+## Usage
+
+The following examples show how the `add_entries` processor can be used in different cases.
+
+### Example: Add entries with simple values
+
+The following example shows you how to configure the processor to add entries with simple values:
 
 ```yaml
-pipeline:
-  source:
-    ...
-  ....  
+... 
   processor:
     - add_entries:
         entries:
-        - key: "newMessage"
-          value: 3
-          overwrite_if_key_exists: true
-        - metadata_key: myMetadataKey
-          value_expression: 'length("newMessage")'
-          add_when: '/some_key == "test"'
-  sink:
+          - key: "name"
+            value: "John"
+          - key: "age"
+            value: 20
+...
 ```
 {% include copy.html %}
 
+When the input event contains the following data:
+
+```json
+{"message": "hello"}
+```
 
-For example, when your source contains the following event record:
+The processed event will contain the following data:
+
+```json
+{"message": "hello", "name": "John", "age": 20}
+```
+
+### Example: Add entries using format strings
+
+The following example shows you how to configure the processor to add entries with values from other fields:
+
+```yaml
+... 
+  processor:
+    - add_entries:
+        entries:
+          - key: "date"
+            format: "${month}-${day}"
+...
+```
+{% include copy.html %}
+
+When the input event contains the following data:
+
+```json
+{"month": "Dec", "day": 1}
+```
+
+The processed event will contain the following data:
+
+```json
+{"month": "Dec", "day": 1, "date": "Dec-1"}
+```
+
+### Example: Add entries using value expressions
+
+The following example shows you how to configure the processor to use the `value_expression` option:
+
+```yaml
+... 
+  processor:
+    - add_entries:
+        entries:
+          - key: "length"
+            value_expression: "length(/message)"
+...
+```
+{% include copy.html %}
+
+When the input event contains the following data:
 
 ```json
 {"message": "hello"}
 ```
 
-And then you run the `add_entries` processor using the example pipeline, it adds a new entry, `{"newMessage": 3}`, to the existing event, `{"message": "hello"}`, so that the new event contains two entries in the final output:
+The processed event will contain the following data:
+
+```json
+{"message": "hello", "length": 5}
+```
+
+### Example: Add metadata
+
+The following example shows you how to configure the processor to add metadata to events:
+
+```yaml
+... 
+  processor:
+    - add_entries:
+        entries:
+          - metadata_key: "length"
+            value_expression: "length(/message)"
+...
+```
+{% include copy.html %}
+
+When the input event contains the following data:
 
 ```json
-{"message": "hello", "newMessage": 3}
+{"message": "hello"}
 ```
 
-If `newMessage` already exists, its existing value is overwritten with a value of `3`.
+The processed event will have the same data, with the metadata, `{"length": 5}`, attached. You can subsequently use expressions like `getMetadata("length")` in the pipeline. For more information, see the [`getMetadata` function](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/#getmetadata) documentation.
+
+
+### Example: Add a dynamic key
 
+The following example shows you how to configure the processor to add metadata to events using a dynamic key:
+
+```yaml
+... 
+  processor:
+    - add_entries:
+        entries:
+          - key: "${/param_name}"
+            value_expression: "/param_value"
+...
+```
+{% include copy.html %}
+
+When the input event contains the following data:
+
+```json
+{"param_name": "cpu", "param_value": 50}
+```
+
+The processed event will contain the following data:
+
+```json
+{"param_name": "cpu", "param_value": 50, "cpu": 50}
+```
+
+### Example: Overwrite existing entries
+
+The following example shows you how to configure the processor to overwrite existing entries:
+
+```yaml
+... 
+  processor:
+    - add_entries:
+        entries:
+          - key: "message"
+            value: "bye"
+            overwrite_if_key_exists: true
+...
+```
+{% include copy.html %}
+
+When the input event contains the following data:
+
+```json
+{"message": "hello"}
+```
+
+The processed event will contain the following data:
+
+```json
+{"message": "bye"}
+```
+
+If `overwrite_if_key_exists` is not set to `true`, then the input event will not be changed after processing.
+
+### Example: Append values to existing entries
+
+The following example shows you how to configure the processor to append values to existing entries:
+
+```yaml
+... 
+  processor:
+    - add_entries:
+        entries:
+          - key: "message"
+            value: "world"
+            append_if_key_exists: true
+...
+```
+{% include copy.html %}
+
+When the input event contains the following data:
+
+```json
+{"message": "hello"}
+```
+
+The processed event will contain the following data:
+
+```json
+{"message": ["hello", "world"]}
+```