-
Notifications
You must be signed in to change notification settings - Fork 515
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOC] Add set processor documentation #5989
Merged
Merged
Changes from 6 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
d453547
Add set processor documentation
vagimeli d9c195e
Merge branch 'main' into set-processor
vagimeli bb45a5a
add text and examples
vagimeli d6b2ea9
add text and examples
vagimeli 520aed5
Update set.md
vagimeli 2609707
Merge branch 'main' into set-processor
vagimeli 2db46be
Update _ingest-pipelines/processors/set.md
vagimeli 8e7c8ef
Update _ingest-pipelines/processors/set.md
vagimeli 596cc0b
Update _ingest-pipelines/processors/set.md
vagimeli fe64844
Update _ingest-pipelines/processors/set.md
vagimeli a8fe146
Update _ingest-pipelines/processors/set.md
vagimeli c85b8ca
Update _ingest-pipelines/processors/set.md
vagimeli 181e0a7
Merge branch 'main' into set-processor
vagimeli File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,155 @@ | ||
--- | ||
layout: default | ||
title: Set | ||
parent: Ingest processors | ||
nav_order: 240 | ||
--- | ||
|
||
# Set processor | ||
|
||
The `set` processor adds or updates fields in a document. It sets one field and associates it with the specified value. If the field already exists, its value is replaced with the provided one. | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
The following is the syntax for the `set` processor: | ||
|
||
```json | ||
{ | ||
"description": "...", | ||
"processors": [ | ||
{ | ||
"set": { | ||
"field": "new_field", | ||
"value": "some_value" | ||
} | ||
} | ||
] | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
## Configuration parameters | ||
|
||
The following table lists the required and optional parameters for the `set` processor. | ||
|
||
Parameter | Required/Optional | Description | | ||
|-----------|-----------|-----------| | ||
`field` | Required | The name of the field to be set or updated. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | ||
`value` | Required | The value to be assigned to the field. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
`overrideEnabled` | Optional | A Boolean flag that determines whether the processor should override the exisitng value of the field. | ||
Check failure on line 37 in _ingest-pipelines/processors/set.md
|
||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
`ignoreEmptyValue` | Optional | A Boolean flag that determines whether the processor should ignore empty values. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
`description` | Optional | A description of the processor's purpose or configuration. | ||
`if` | Optional | Specifies to conditionally execute the processor. | ||
`ignore_failure` | Optional | Specifies to ignore failures for the processor. See [Handling pipeline failures]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/). | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
`on_failure` | Optional | Specifies a list of processors to run if the processor fails during execution. These processors are executed in the order they are specified. | ||
`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. | ||
|
||
## Using the processor | ||
|
||
Follow these steps to use the processor in a pipeline. | ||
|
||
### Step 1: Create a pipeline | ||
|
||
The following query creates a pipeline named `set-pipeline` that uses the `set` processor to add a new field `new_field` with the value `some_value` to the document: | ||
|
||
```json | ||
PUT _ingest/pipeline/set-pipeline | ||
{ | ||
"description": "Adds a new field 'new_field' with the value 'some_value'", | ||
"processors": [ | ||
{ | ||
"set": { | ||
"field": "new_field", | ||
"value": "some_value" | ||
} | ||
} | ||
] | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
### Step 2 (Optional): Test the pipeline | ||
|
||
It is recommended that you test your pipeline before you ingest documents. | ||
{: .tip} | ||
|
||
To test the pipeline, run the following query: | ||
|
||
```json | ||
POST _ingest/pipeline/set-pipeline/_simulate | ||
{ | ||
"docs": [ | ||
{ | ||
"_source": { | ||
"existing_field": "value" | ||
} | ||
} | ||
] | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
#### Response | ||
|
||
The following example response confirms that the pipeline is working as expected: | ||
|
||
```json | ||
{ | ||
"docs": [ | ||
{ | ||
"doc": { | ||
"_index": "_index", | ||
"_id": "_id", | ||
"_source": { | ||
"existing_field": "value", | ||
"new_field": "some_value" | ||
}, | ||
"_ingest": { | ||
"timestamp": "2024-05-30T21:56:15.066180712Z" | ||
} | ||
} | ||
} | ||
] | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
### Step 3: Ingest a document | ||
|
||
The following query ingests a document into an index named `testindex1`: | ||
|
||
```json | ||
POST testindex1/_doc | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
{ | ||
"existing_field": "value" | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
#### Response | ||
|
||
The request indexes the document into the index `testindex1` and then indexes all documents with the `new_field` set to `some_value`, as shown in the following response: | ||
|
||
```json | ||
{ | ||
"_index": "testindex1", | ||
"_id": "1", | ||
"_version": 1, | ||
"result": "created", | ||
"_shards": { | ||
"total": 2, | ||
"successful": 1, | ||
"failed": 0 | ||
}, | ||
"_seq_no": 0, | ||
"_primary_term": 1 | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
### Step 4 (Optional): Retrieve the document | ||
|
||
To retrieve the document, run the following query: | ||
|
||
```json | ||
GET testindex1/_doc/1 | ||
``` | ||
{% include copy-curl.html %} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not always true, because if the parameter
override
is set to false and the specified field exists, the value of the field will not change.