From 04ba3d829889e026bd3c030e1cd590b8588a2c14 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 3 Nov 2023 17:09:28 -0600 Subject: [PATCH 01/53] Develop new content for dissect processor Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 174 ++++++++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 _ingest-pipelines/processors/dissect.md diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md new file mode 100644 index 0000000000..7391f0921c --- /dev/null +++ b/_ingest-pipelines/processors/dissect.md @@ -0,0 +1,174 @@ +--- +layout: default +title: Dissect +parent: Ingest processors +nav_order: 55 +--- + +# Dissect + +The `dissect` processor extracts values from an event and maps them to individual fields based on user-defined dissect patterns. The processor is well-suited for field extractions from log messages with a known structure. + +## Example +The following is the syntax for the `dissect` processor: + +```json +{ + "dissect": { + "field": "source_field", + "pattern": "%{dissect_pattern}" + } +} +``` +{% include copy-curl.html %} + + +## Configuration parameters + +The following table lists the required and optional parameters for the `dissect` processor. + +Parameter | Required/Optional | Description | +|-----------|-----------|-----------| +`field` | Required | The name of the field to which the data should be dissected. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`dissect_pattern` | Required | The dissect pattern used to extract data from the field specified. | +`append_separator` | Optional | The separator character or string between two or more values. Default is `""` (empty string). +`description` | Optional | A brief description of the processor. | +`if` | Optional | A condition for running this processor. | +`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | +`ignore_missing` | Optional | If set to `true`, the processor does not modify the document if the field does not exist or is `null`. Default is `false`. | +`on_failure` | Optional | A list of processors to run if the processor fails. | +`tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | + +## Using the processor + +Follow these steps to use the processor in a pipeline. + +**Step 1: Create a pipeline.** + +The following query creates a pipeline, named `dissect-text`, that uses the `dissect` processor to parse the log line: + +```json +PUT /_ingest/pipeline/dissect-test +{ + "description": "Pipeline that dissects web server logs", + "processors": [ + { + "dissect": { + "field": "message", + "pattern": "%{client_ip} - - [%{timestamp}] \"%{http_method} %{url} %{http_version}\" %{response_code} %{response_size}" + } + } + ] +} +``` +{% include copy-curl.html %} + +**Step 2 (Optional): Test the pipeline.** + +It is recommended that you test your pipeline before you ingest documents. +{: .tip} + +To test the pipeline, run the following query: + +```json +POST _ingest/pipeline/dissect-test/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "message": "192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] \"POST /login HTTP/1.1\" 200 3456" + } + } + ] +} +``` +{% include copy-curl.html %} + +#### Response + +The following example response confirms that the pipeline is working as expected: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "response_code": "200", + "http_method": "POST", + "http_version": "HTTP/1.1", + "client_ip": "192.168.1.10", + "message": """192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] "POST /login HTTP/1.1" 200 3456""", + "url": "/login", + "response_size": "3456", + "timestamp": "03/Nov/2023:15:20:45 +0000" + }, + "_ingest": { + "timestamp": "2023-11-03T22:28:32.830244044Z" + } + } + } + ] +} +``` + +**Step 3: Ingest a document.** + +The following query ingests a document into an index named `testindex1`: + +```json +PUT testindex1/_doc/1?pipeline=dissect-test +{ + "message": "192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] \"POST /login HTTP/1.1\" 200 3456" +} +``` +{% include copy-curl.html %} + +**Step 4 (Optional): Retrieve the document.** + +To retrieve the document, run the following query: + +```json +GET testindex1/_doc/1 +``` +{% include copy-curl.html %} + +## Dissect patterns + +A dissect pattern is a way to tell `dissect` how to parse a string into a structured format. The pattern is defined by the parts of the string that you want to discard. For example, the following dissect pattern would parse a string like `"192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] \"POST /login HTTP/1.1\" 200 3456"` into the following fields: + +```json +client_ip: "192.168.1.1" +@timestamp: "03/Nov/2023:16:09:05 MDT" +``` + +The dissect pattern works by matching the string against a set of rules. For example, the first rule is to match a single space. Dissect will find this space and then assign the value of `client_ip` to everything up to that space. The next rule is to match the `[` and `]` characters and then assign the value of `@timestamp` to everything in between. + +### Buidling successful dissect patterns + +When building dissect pattern, it is important to pay attention to the parts of the string that you want to discard. If you discard too much of the string, then `dissect` may not be able to successfully parse the remaining data. Conversely, if you do not discard enough of the string, then `dissect` may create unnecessary fields. + +If any of the `%{keyname}` defined in the pattern do not have a value, then an exception is thrown. You can handle this exception by using the `on_failure` parameter. + +### Empty and named skip keys + +An empty key `%{}` or a named skip key can be used to match values, but exclude the value from the final document. This can be useful if you want to parse a string, but you do not need to store all of the data. + +### Matched values as string data types + +By default, all matched values are represented as string data types. If you need to convert a value to a different data type, you can use the [`convert` processor]({{site.url}}{{site.baseurl}}/ingest-pipelines/processors/convert/). + +### Key modifiers + +The `dissect` processor support key modifiers that can change dissect's default behavior. For example, you can instruct `dissect` to ignore certain fields or append fields. + +The following table lists the key modifiers for the `dissect` processor. + +Modifier | Name | Position | Example | Description | +|-----------|-----------|-----------| +`->` | Skip right padding | (far) right | `%{keyname->}` | Tells `dissect` to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell `dissect` to skip over any padding characters, such as two spaces or any varying character padding, that follow `timestamp`. | + \ No newline at end of file From 9cea9c9845c560bb8eedfec48c045ab11c44688a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 6 Nov 2023 13:48:41 -0700 Subject: [PATCH 02/53] Add new dissect processor documentation Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 113 +++++++++++++++++++++++- 1 file changed, 110 insertions(+), 3 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 7391f0921c..d4eec2306a 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -148,7 +148,7 @@ client_ip: "192.168.1.1" The dissect pattern works by matching the string against a set of rules. For example, the first rule is to match a single space. Dissect will find this space and then assign the value of `client_ip` to everything up to that space. The next rule is to match the `[` and `]` characters and then assign the value of `@timestamp` to everything in between. -### Buidling successful dissect patterns +### Building successful dissect patterns When building dissect pattern, it is important to pay attention to the parts of the string that you want to discard. If you discard too much of the string, then `dissect` may not be able to successfully parse the remaining data. Conversely, if you do not discard enough of the string, then `dissect` may create unnecessary fields. @@ -164,11 +164,118 @@ By default, all matched values are represented as string data types. If you need ### Key modifiers -The `dissect` processor support key modifiers that can change dissect's default behavior. For example, you can instruct `dissect` to ignore certain fields or append fields. +The `dissect` processor supports key modifiers that can change the dissection's default behavior. These modifiers are always placed to the left or right of the `%{keyname}` and are always enclosed within the `%{}`. For example, the `%{+keyname->}` modifier includes the append and right padding modifiers. Key modifiers are useful for cases such as combining multiple fields into a single line of output, creating formatted lists of data items, or aggregating values from multiple sources. The following table lists the key modifiers for the `dissect` processor. Modifier | Name | Position | Example | Description | |-----------|-----------|-----------| `->` | Skip right padding | (far) right | `%{keyname->}` | Tells `dissect` to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell `dissect` to skip over any padding characters, such as two spaces or any varying character padding, that follow `timestamp`. | - \ No newline at end of file +`+` | Append | left | `%{keyname} %{+keyname}` | Appends two or more fields together. | +`+` with `/n` | Append with order | left and right | `%{+keyname}/2 %{+keyname/1}` | Appends two or more fields together in the order specified. | +`?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | +`*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as value of `*` and output value of `&`. | + +Detailed descriptions of each key modifier are in the following sections. + +### Right padding modifier (`->`) + +The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For instance, the pattern `%{helloworldkey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because pattern only has one space while the source string has two. + +The right padding modifier can be used to address this issue. By adding the right padding modifier to the pattern `%{helloworldkey->} %{worldkey}`, it will no match `Hello world` (one space), `Hello world` (two spaces), and even `Hello world` (ten spaces). + +The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}`, `%{}`. + +#### Example + +The following is an example of a right padding modifier and how it is used: + +`%{name->} %{city}, %{state} %{zip}` + +In this pattern, the right padding modifier `->` is applied to the `%{name}` key. This means that the `%{name}` key will match an sequence of characters, including spaces. This is useful for handling names that may contain spaces, such as "First Last". + +The following is an example of how the right padding would be used to extract information from the following address entries: + +```bash +New York, NY 10017 +New York City, NY 10017 +``` + +Both addresses contain the same information, but the second entry has an extra word, `City`, in the city field. The right padding modifier allows the pattern to match both of these address entries, even though they have slightly different formats. + +### Append modifier (`+`) + +The append modifier combines the values of two or more keys into a single output value. The values are appended from left to right. You can also specify an optional separator to be inserted between the values. + +#### Example + +The following pattern extracts the values of `key1` and `key2` fields and appends them together, with a space as the separator: + +`%{key1} %{key2}` + +The output is: + +`value1 value2` + +You can also specify a custom separator using the `append_separator` parameter. For example, the following pattern uses a comma as the separator: + +`%{key1} %{key2}, append_separator => ","` + +The output is: + +`value1, value2` + +### Append with order modifier (`+` and `/n`) + +The append with order modifier combines the values of two or more keys into a single output value, adhering to the a specific order defines by a newline character `/n`. You have the flexibility to customize the separator that separates the appended values. the append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. + +#### Example + +The following pattern extracts the values of `key1` and `key2` fields and appends them together, with a newline character as the separator: + +`%{key1} %key2} /n` + +The output is: + +```bash +value1 +value2 +``` + +You can also specify an alternative separator using the `append_separator` parameter. For example, the following pattern uses a comma as the separator: + +`%{key1} %key2}, append_separator => "," /n` + +The outout is: + +```bash +value1, value2 +``` + +### Named skip key (`?`) + +The named skip key modifier excludes specific matches from the final output by using an empty key, `{%}`, within the pattern. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output, focusing on specific information, or streamlining the output for further processing ot analysis. + +#### Example + +The following pattern excludes a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{ignore}`, to clarify that the corresponding value should be excluded from the final result. + +`%firstName} %{lastName} %{ignore}` + +### Reference keys (`*` and `&`) + +Reference keys use parsed values as key/value pairings for structured content. This can use useful when handling systems that partially log data in key/value pairs. by using reference keys, you can preserve the key/value relationship and maintain the integrity of the extracted information. + +#### Example + +The following pattern extracts data into a structured format, with `%{value}` represented the parsed value and `%{reference_key}` acting as placeholder for the actual key: + +`%{value} %{reference_key}` + +The output is: + +```bash +value1 value1 +value2 value2 +value3 value3 +``` From 8a33fa7079db10fcbd3e9551119948e75522a870 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 17 Nov 2023 10:16:12 -0700 Subject: [PATCH 03/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index d4eec2306a..4d44741b29 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -2,7 +2,7 @@ layout: default title: Dissect parent: Ingest processors -nav_order: 55 +nav_order: 60 --- # Dissect From 46b495af5bf481dc1c5e74d1a6fa423f565399cc Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 8 Dec 2023 15:49:09 -0700 Subject: [PATCH 04/53] Address tech review feedback Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 49 ++++++++++++++----------- 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 4d44741b29..6eaffb1996 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -225,42 +225,49 @@ The output is: `value1, value2` -### Append with order modifier (`+` and `/n`) +### Append with order modifier (`+` and `/`) -The append with order modifier combines the values of two or more keys into a single output value, adhering to the a specific order defines by a newline character `/n`. You have the flexibility to customize the separator that separates the appended values. the append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. +The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. #### Example -The following pattern extracts the values of `key1` and `key2` fields and appends them together, with a newline character as the separator: +The `append_separator` parameter must be defined in the processor configuration, outside of the pattern. It is only relevant with the `+` modifier. See the following example pipeline: -`%{key1} %key2} /n` - -The output is: - -```bash -value1 -value2 +```json +PUT /_ingest/pipeline/dissect-test +{ + "description": "Pipeline that dissects web server logs", + "processors": [ + { + "dissect": { + "field": "message", + "pattern": "%{a} %{+a} %{+a}", + "append_separator": "," + } + } + ] +} ``` +{% include copy-curl.html %} -You can also specify an alternative separator using the `append_separator` parameter. For example, the following pattern uses a comma as the separator: - -`%{key1} %key2}, append_separator => "," /n` - -The outout is: +If you ingest the following example document, you'll get the response `"a":"apple,banana,coconut"`: -```bash -value1, value2 +```json +{ + "message": "apple banana coconut" +} ``` +{% include copy-curl.html %} -### Named skip key (`?`) +### Named skip key -The named skip key modifier excludes specific matches from the final output by using an empty key, `{%}`, within the pattern. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output, focusing on specific information, or streamlining the output for further processing ot analysis. +The named skip key modifier excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output, focusing on specific information, or streamlining the output for further processing or analysis. #### Example -The following pattern excludes a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{ignore}`, to clarify that the corresponding value should be excluded from the final result. +The following pattern excludes a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{?ignore}`, to clarify that the corresponding value should be excluded from the final result. -`%firstName} %{lastName} %{ignore}` +`%{firstName} %{lastName} %{?ignore}` ### Reference keys (`*` and `&`) From 4eaefa19379abfc1603787416dfdb24a3515cf11 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 12 Dec 2023 16:17:50 -0700 Subject: [PATCH 05/53] Copy edits Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 43 +++++++++++++------------ 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 6eaffb1996..1e721e3577 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -9,7 +9,8 @@ nav_order: 60 The `dissect` processor extracts values from an event and maps them to individual fields based on user-defined dissect patterns. The processor is well-suited for field extractions from log messages with a known structure. -## Example +## Syntax + The following is the syntax for the `dissect` processor: ```json @@ -29,13 +30,13 @@ The following table lists the required and optional parameters for the `dissect` Parameter | Required/Optional | Description | |-----------|-----------|-----------| -`field` | Required | The name of the field to which the data should be dissected. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | -`dissect_pattern` | Required | The dissect pattern used to extract data from the field specified. | +`field` | Required | The name of the field containing the data to be dissected. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`dissect_pattern` | Required | The dissect pattern used to extract data from the specified field. | `append_separator` | Optional | The separator character or string between two or more values. Default is `""` (empty string). `description` | Optional | A brief description of the processor. | -`if` | Optional | A condition for running this processor. | -`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | -`ignore_missing` | Optional | If set to `true`, the processor does not modify the document if the field does not exist or is `null`. Default is `false`. | +`if` | Optional | A condition for running the processor. | +`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters errors. If set to `true`, failures are ignored. Default is `false`. | +`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified field. If set to `true`, the processor does not modify the document if the field does not exist or is `null`. Default is `false`. | `on_failure` | Optional | A list of processors to run if the processor fails. | `tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | @@ -43,7 +44,7 @@ Parameter | Required/Optional | Description | Follow these steps to use the processor in a pipeline. -**Step 1: Create a pipeline.** +**Step 1: Create a pipeline** The following query creates a pipeline, named `dissect-text`, that uses the `dissect` processor to parse the log line: @@ -63,7 +64,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -**Step 2 (Optional): Test the pipeline.** +**Step 2 (Optional): Test the pipeline** It is recommended that you test your pipeline before you ingest documents. {: .tip} @@ -86,7 +87,7 @@ POST _ingest/pipeline/dissect-test/_simulate ``` {% include copy-curl.html %} -#### Response +**Response** The following example response confirms that the pipeline is working as expected: @@ -116,7 +117,7 @@ The following example response confirms that the pipeline is working as expected } ``` -**Step 3: Ingest a document.** +**Step 3: Ingest a document** The following query ingests a document into an index named `testindex1`: @@ -128,7 +129,7 @@ PUT testindex1/_doc/1?pipeline=dissect-test ``` {% include copy-curl.html %} -**Step 4 (Optional): Retrieve the document.** +**Step 4 (Optional): Retrieve the document** To retrieve the document, run the following query: @@ -186,9 +187,9 @@ The right padding modifier can be used to address this issue. By adding the righ The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}`, `%{}`. -#### Example +#### Example of usage -The following is an example of a right padding modifier and how it is used: +The following is an example of how to use a right padding modifier: `%{name->} %{city}, %{state} %{zip}` @@ -207,9 +208,9 @@ Both addresses contain the same information, but the second entry has an extra w The append modifier combines the values of two or more keys into a single output value. The values are appended from left to right. You can also specify an optional separator to be inserted between the values. -#### Example +#### Example of usage -The following pattern extracts the values of `key1` and `key2` fields and appends them together, with a space as the separator: +The following is an example of how to use the append modifier. In this example, the pattern extracts the values of `key1` and `key2` fields and appends them together, with a space as the separator: `%{key1} %{key2}` @@ -229,9 +230,9 @@ The output is: The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. -#### Example +#### Example of usage -The `append_separator` parameter must be defined in the processor configuration, outside of the pattern. It is only relevant with the `+` modifier. See the following example pipeline: +The following example pipeline uses the append with order modifier. Note that the `append_separator` parameter must be defined in the processor configuration, outside of the `pattern`. It is only relevant with the `+` modifier. See the following example pipeline: ```json PUT /_ingest/pipeline/dissect-test @@ -263,9 +264,9 @@ If you ingest the following example document, you'll get the response `"a":"appl The named skip key modifier excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output, focusing on specific information, or streamlining the output for further processing or analysis. -#### Example +#### Example of usage -The following pattern excludes a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{?ignore}`, to clarify that the corresponding value should be excluded from the final result. +The following pattern uses a named skip key to exclude a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{?ignore}`, to clarify that the corresponding value should be excluded from the final result: `%{firstName} %{lastName} %{?ignore}` @@ -273,9 +274,9 @@ The following pattern excludes a field (in this case, `ignore`) from the output. Reference keys use parsed values as key/value pairings for structured content. This can use useful when handling systems that partially log data in key/value pairs. by using reference keys, you can preserve the key/value relationship and maintain the integrity of the extracted information. -#### Example +#### Example of usage -The following pattern extracts data into a structured format, with `%{value}` represented the parsed value and `%{reference_key}` acting as placeholder for the actual key: +The following pattern uses a reference key to extract data into a structured format. In this example, `%{value}` represents the parsed value and `%{reference_key}` acts as the placeholder for the actual key: `%{value} %{reference_key}` From 8b46f7aeec218684c12ee4eef82188947cbafaf9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 4 Jan 2024 15:00:38 -0700 Subject: [PATCH 06/53] Copy edits Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 1e721e3577..5722104b07 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -272,7 +272,7 @@ The following pattern uses a named skip key to exclude a field (in this case, `i ### Reference keys (`*` and `&`) -Reference keys use parsed values as key/value pairings for structured content. This can use useful when handling systems that partially log data in key/value pairs. by using reference keys, you can preserve the key/value relationship and maintain the integrity of the extracted information. +Reference keys use parsed values as key/value pairings for structured content. This can use useful when handling systems that partially log data in key/value pairs. By using reference keys, you can preserve the key/value relationship and maintain the integrity of the extracted information. #### Example of usage From eb0702070d63daff078b7202acaa144fd11c80f5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 4 Jan 2024 15:02:36 -0700 Subject: [PATCH 07/53] Copy edits Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 5722104b07..eaf6dddd9d 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -46,7 +46,7 @@ Follow these steps to use the processor in a pipeline. **Step 1: Create a pipeline** -The following query creates a pipeline, named `dissect-text`, that uses the `dissect` processor to parse the log line: +The following query creates a pipeline, named `dissect-test`, that uses the `dissect` processor to parse the log line: ```json PUT /_ingest/pipeline/dissect-test From a1c4a3a65b5f61ec2a96ecb5d44ab4e14a844d65 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 10:46:38 -0700 Subject: [PATCH 08/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index eaf6dddd9d..1743516f0f 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -7,7 +7,7 @@ nav_order: 60 # Dissect -The `dissect` processor extracts values from an event and maps them to individual fields based on user-defined dissect patterns. The processor is well-suited for field extractions from log messages with a known structure. +The `dissect` processor extracts values from a document text field and maps them to individual fields based on dissect patterns. The processor is well-suited for field extractions from log messages with a known structure. Unlinke the `grok` processor, `dissect` does not use regular expressions and has a simpler syntax. ## Syntax From 179f20daaee05002c46f942ece8639a490bd972b Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 23 Jan 2024 15:15:46 -0700 Subject: [PATCH 09/53] Address Fanit doc review feedback Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 327 ++++++++++++++++++++---- 1 file changed, 282 insertions(+), 45 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 1743516f0f..94ef5b6d33 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -7,7 +7,7 @@ nav_order: 60 # Dissect -The `dissect` processor extracts values from a document text field and maps them to individual fields based on dissect patterns. The processor is well-suited for field extractions from log messages with a known structure. Unlinke the `grok` processor, `dissect` does not use regular expressions and has a simpler syntax. +The `dissect` processor extracts values from a document text field and maps them to individual fields based on dissect patterns. The processor is well-suited for field extractions from log messages with a known structure. Unlike the `grok` processor, `dissect` does not use regular expressions and has a simpler syntax. ## Syntax @@ -30,9 +30,9 @@ The following table lists the required and optional parameters for the `dissect` Parameter | Required/Optional | Description | |-----------|-----------|-----------| -`field` | Required | The name of the field containing the data to be dissected. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | -`dissect_pattern` | Required | The dissect pattern used to extract data from the specified field. | -`append_separator` | Optional | The separator character or string between two or more values. Default is `""` (empty string). +`field` | Required | The name of the field containing the data to be dissected. | +`pattern` | Required | The dissect pattern used to extract data from the specified field. | +`append_separator` | Optional | The separator character or string that separates appended fields. Default is `""` (empty string). `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running the processor. | `ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters errors. If set to `true`, failures are ignored. Default is `false`. | @@ -140,34 +140,34 @@ GET testindex1/_doc/1 ## Dissect patterns -A dissect pattern is a way to tell `dissect` how to parse a string into a structured format. The pattern is defined by the parts of the string that you want to discard. For example, the following dissect pattern would parse a string like `"192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] \"POST /login HTTP/1.1\" 200 3456"` into the following fields: +A dissect pattern is a way to tell `dissect` how to parse a string into a structured format. The pattern is defined by the parts of the string that you want to discard. For example, the `%{client_ip} - - [%{timestamp}]` dissect pattern parses the string `"192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] \"POST /login HTTP/1.1\" 200 3456"` into the following fields: ```json client_ip: "192.168.1.1" -@timestamp: "03/Nov/2023:16:09:05 MDT" +@timestamp: "03/Nov/2023:15:20:45 +0000" ``` -The dissect pattern works by matching the string against a set of rules. For example, the first rule is to match a single space. Dissect will find this space and then assign the value of `client_ip` to everything up to that space. The next rule is to match the `[` and `]` characters and then assign the value of `@timestamp` to everything in between. +The dissect pattern works by matching the string against a set of rules. For example, the first rule is to discard a single space. Dissect will find this space and then assign the value of `client_ip` to everything up to that space. The next rule is to match the `[` and `]` characters and then assign the value of `@timestamp` to everything in between. ### Building successful dissect patterns When building dissect pattern, it is important to pay attention to the parts of the string that you want to discard. If you discard too much of the string, then `dissect` may not be able to successfully parse the remaining data. Conversely, if you do not discard enough of the string, then `dissect` may create unnecessary fields. -If any of the `%{keyname}` defined in the pattern do not have a value, then an exception is thrown. You can handle this exception by using the `on_failure` parameter. +If any `%{keyname}` defined in the pattern do not have a value, then an exception is thrown. You can handle this exception by providing error handling steps in the `on_failure` parameter. ### Empty and named skip keys -An empty key `%{}` or a named skip key can be used to match values, but exclude the value from the final document. This can be useful if you want to parse a string, but you do not need to store all of the data. +An empty key `%{}` or a [named skip key](#named-skip-key) can be used to match values, but exclude the value from the final document. This can be useful if you want to parse a string, but you do not need to store all its parts. -### Matched values as string data types +### Converting matched values to a non-string data type By default, all matched values are represented as string data types. If you need to convert a value to a different data type, you can use the [`convert` processor]({{site.url}}{{site.baseurl}}/ingest-pipelines/processors/convert/). ### Key modifiers -The `dissect` processor supports key modifiers that can change the dissection's default behavior. These modifiers are always placed to the left or right of the `%{keyname}` and are always enclosed within the `%{}`. For example, the `%{+keyname->}` modifier includes the append and right padding modifiers. Key modifiers are useful for cases such as combining multiple fields into a single line of output, creating formatted lists of data items, or aggregating values from multiple sources. +The `dissect` processor supports key modifiers that can change the default processor behavior. These modifiers are always placed to the left or right of `%{keyname}` and are always enclosed within `%{}`. For example, the `%{+keyname->}` modifier includes the append and right padding modifiers. Key modifiers are useful for cases such as combining multiple fields into a single line of output, creating formatted lists of data items, or aggregating values from multiple sources. -The following table lists the key modifiers for the `dissect` processor. +The following table lists the primary modifiers for the `dissect` processor. Modifier | Name | Position | Example | Description | |-----------|-----------|-----------| @@ -181,50 +181,150 @@ Detailed descriptions of each key modifier are in the following sections. ### Right padding modifier (`->`) -The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For instance, the pattern `%{helloworldkey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because pattern only has one space while the source string has two. +The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For instance, the pattern `%{hellokey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because the pattern only has one space while the source string has two. -The right padding modifier can be used to address this issue. By adding the right padding modifier to the pattern `%{helloworldkey->} %{worldkey}`, it will no match `Hello world` (one space), `Hello world` (two spaces), and even `Hello world` (ten spaces). +The right padding modifier can be used to address this issue. By adding the right padding modifier to the pattern `%{helloworldkey->} %{worldkey}`, it will match `Hello world` (one space), `Hello world` (two spaces), and even `Hello world` (ten spaces). -The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}`, `%{}`. +The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}` or `%{}`. #### Example of usage The following is an example of how to use a right padding modifier: -`%{name->} %{city}, %{state} %{zip}` +`%{city->}, %{state} %{zip}` -In this pattern, the right padding modifier `->` is applied to the `%{name}` key. This means that the `%{name}` key will match an sequence of characters, including spaces. This is useful for handling names that may contain spaces, such as "First Last". - -The following is an example of how the right padding would be used to extract information from the following address entries: +In this pattern, the right padding modifier `->` is applied to the `%{city}` key. Both addresses contain the same information, but the second entry has an extra word, `City`, in the city field. The right padding modifier allows the pattern to match both of these address entries, even though they have slightly different formats: ```bash New York, NY 10017 New York City, NY 10017 ``` -Both addresses contain the same information, but the second entry has an extra word, `City`, in the city field. The right padding modifier allows the pattern to match both of these address entries, even though they have slightly different formats. +The following example pipeline uses the right-padding modifier with an empty key `%{->}`: -### Append modifier (`+`) +```json +PUT /_ingest/pipeline/dissect-test +{ + "description": "Pipeline that dissects web server logs", + "processors": [ + { + "dissect": { + "field": "message", + "pattern": "[%{client_ip}]%{->}[%{timestamp}]" + } + } + ] +} +``` +{% include copy-curl.html %} -The append modifier combines the values of two or more keys into a single output value. The values are appended from left to right. You can also specify an optional separator to be inserted between the values. +You can test the pipeline using the following example: -#### Example of usage +```json +POST _ingest/pipeline/dissect-test/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "message": "[192.168.1.10] [03/Nov/2023:15:20:45 +0000]" + } + } + ] +} +``` +{% include copy-curl.html %} -The following is an example of how to use the append modifier. In this example, the pattern extracts the values of `key1` and `key2` fields and appends them together, with a space as the separator: +Your response should be similar to the following: -`%{key1} %{key2}` +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "client_ip": "192.168.1.10", + "message": "[192.168.1.10] [03/Nov/2023:15:20:45 +0000]", + "timestamp": "03/Nov/2023:15:20:45 +0000" + }, + "_ingest": { + "timestamp": "2024-01-22T22:55:42.090569297Z" + } + } + } + ] +} +``` +{% include copy-curl.html %} -The output is: +### Append modifier (`+`) -`value1 value2` +The append modifier combines the values of two or more values into a single output value. The values are appended from left to right. You can also specify an optional separator to be inserted between the values. -You can also specify a custom separator using the `append_separator` parameter. For example, the following pattern uses a comma as the separator: +#### Example of usage -`%{key1} %{key2}, append_separator => ","` +The following is an example pipeline with an append modifier: -The output is: +```json +PUT /_ingest/pipeline/dissect-test +{ + "description": "Pipeline that dissects web server logs", + "processors": [ + { + "dissect": { + "field": "message", + "pattern": "%{+address}, %{+address} %{+address}", + "append_separator": "|" + } + } + ] +} +``` +{% include copy-curl.html %} + +You can test the pipeline using the following example: + +```json +POST _ingest/pipeline/dissect-test/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "message": "New York, NY 10017" + } + } + ] +} +``` +{% include copy-curl.html %} -`value1, value2` +The substrings are appended to the `address` field, as shown in the following response: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "address": "New York|NY|10017", + "message": "New York, NY 10017" + }, + "_ingest": { + "timestamp": "2024-01-22T22:30:54.516284637Z" + } + } + } + ] +} +``` +{% include copy-curl.html %} ### Append with order modifier (`+` and `/`) @@ -232,7 +332,7 @@ The append with order modifier combines the values of two or more keys into a si #### Example of usage -The following example pipeline uses the append with order modifier. Note that the `append_separator` parameter must be defined in the processor configuration, outside of the `pattern`. It is only relevant with the `+` modifier. See the following example pipeline: +The following example pipeline uses the append with order modifier to reverse the pattern order defined in the preceding pipeline. The `append_separator` parameter must be defined in the processor configuration, outside of the `pattern`. It is only relevant with the `+` modifier. ```json PUT /_ingest/pipeline/dissect-test @@ -242,8 +342,26 @@ PUT /_ingest/pipeline/dissect-test { "dissect": { "field": "message", - "pattern": "%{a} %{+a} %{+a}", - "append_separator": "," + "pattern": "%{+address/3}, %{+address/2} %{+address/1}", + "append_separator": "|" + } + } + ] +} +``` +{% include copy-curl.html %} + +You can test the pipeline using the following example: + +```json +POST _ingest/pipeline/dissect-test/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "message": "New York, NY 10017" } } ] @@ -251,24 +369,94 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -If you ingest the following example document, you'll get the response `"a":"apple,banana,coconut"`: +The substrings are appended into the `address` field in reverse order, as show in the following response: ```json { - "message": "apple banana coconut" + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "address": "10017|NY|New York", + "message": "New York, NY 10017" + }, + "_ingest": { + "timestamp": "2024-01-22T22:38:24.305974178Z" + } + } + } + ] } ``` {% include copy-curl.html %} ### Named skip key -The named skip key modifier excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output, focusing on specific information, or streamlining the output for further processing or analysis. +The named skip key modifier excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output. #### Example of usage The following pattern uses a named skip key to exclude a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{?ignore}`, to clarify that the corresponding value should be excluded from the final result: -`%{firstName} %{lastName} %{?ignore}` +```json +PUT /_ingest/pipeline/dissect-test +{ + "description": "Pipeline that dissects web server logs", + "processors": [ + { + "dissect": { + "field": "message", + "pattern": "%{firstName} %{lastName} %{?ignore}" + } + } + ] +} +``` +{% include copy-curl.html %} + +You can test the pipeline using the following example: + +```json +POST _ingest/pipeline/dissect-test/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "message": "John Doe M.D." + } + } + ] +} +``` +{% include copy-curl.html %} + +You should get a response similar to the following example: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "firstName": "John", + "lastName": "Doe", + "message": "John Doe M.D." + }, + "_ingest": { + "timestamp": "2024-01-22T22:41:58.161475555Z" + } + } + } + ] +} +``` +{% include copy-curl.html %} ### Reference keys (`*` and `&`) @@ -276,14 +464,63 @@ Reference keys use parsed values as key/value pairings for structured content. T #### Example of usage -The following pattern uses a reference key to extract data into a structured format. In this example, `%{value}` represents the parsed value and `%{reference_key}` acts as the placeholder for the actual key: +The following pattern uses a reference key to extract data into a structured format. In this example, `client_ip` and two key/value pairs are extracted for the next values: -`%{value} %{reference_key}` +```json +PUT /_ingest/pipeline/dissect-test +{ + "description": "Pipeline that dissects web server logs", + "processors": [ + { + "dissect": { + "field": "message", + "pattern": "%{client_ip} %{*a}:%{&a} %{*b}:%{&b}" + } + } + ] +} +``` +{% include copy-curl.html %} -The output is: +You can test the pipeline using the following example: -```bash -value1 value1 -value2 value2 -value3 value3 +```json +POST _ingest/pipeline/dissect-test/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "message": "192.168.1.10 response_code:200 response_size:3456" + } + } + ] +} +``` +{% include copy-curl.html %} + +The two key/value pairs were extracted into fields, as shown in the following response: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "client_ip": "192.168.1.10", + "response_code": "200", + "message": "192.168.1.10 response_code:200 response_size:3456", + "response_size": "3456" + }, + "_ingest": { + "timestamp": "2024-01-22T22:48:51.475535635Z" + } + } + } + ] +} ``` +{% include copy-curl.html %} From 740af707e5eff0a048bf546001826ba9d600349d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 23 Jan 2024 16:21:49 -0700 Subject: [PATCH 10/53] Copy edits Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 94ef5b6d33..e349495b16 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -157,7 +157,7 @@ If any `%{keyname}` defined in the pattern do not have a value, then an exceptio ### Empty and named skip keys -An empty key `%{}` or a [named skip key](#named-skip-key) can be used to match values, but exclude the value from the final document. This can be useful if you want to parse a string, but you do not need to store all its parts. +An empty key `%{}` or a [named skip key](#named-skip-key) can be used to match values but exclude the value from the final document. This can be useful if you want to parse a string but do not need to store all its parts. ### Converting matched values to a non-string data type @@ -165,7 +165,7 @@ By default, all matched values are represented as string data types. If you need ### Key modifiers -The `dissect` processor supports key modifiers that can change the default processor behavior. These modifiers are always placed to the left or right of `%{keyname}` and are always enclosed within `%{}`. For example, the `%{+keyname->}` modifier includes the append and right padding modifiers. Key modifiers are useful for cases such as combining multiple fields into a single line of output, creating formatted lists of data items, or aggregating values from multiple sources. +The `dissect` processor supports key modifiers that can change the default processor behavior. These modifiers are always placed to the left or right of the `%{keyname}` and are always enclosed within `%{}`. For example, the `%{+keyname->}` modifier includes the append and right padding modifiers. Key modifiers are useful for cases such as combining multiple fields into a single line of output, creating formatted lists of data items, or aggregating values from multiple sources. The following table lists the primary modifiers for the `dissect` processor. @@ -177,13 +177,13 @@ Modifier | Name | Position | Example | Description | `?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | `*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as value of `*` and output value of `&`. | -Detailed descriptions of each key modifier are in the following sections. +Detailed descriptions of each key modifier, along with usage examples, are in the following sections. ### Right padding modifier (`->`) The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For instance, the pattern `%{hellokey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because the pattern only has one space while the source string has two. -The right padding modifier can be used to address this issue. By adding the right padding modifier to the pattern `%{helloworldkey->} %{worldkey}`, it will match `Hello world` (one space), `Hello world` (two spaces), and even `Hello world` (ten spaces). +The right padding modifier can be used to address this issue. By adding the right padding modifier to the pattern `%{helloworldkey->} %{worldkey}`, it will match Hello world (one space), Hello  world (two spaces), and even Hello          world (ten spaces). The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}` or `%{}`. @@ -200,7 +200,7 @@ New York, NY 10017 New York City, NY 10017 ``` -The following example pipeline uses the right-padding modifier with an empty key `%{->}`: +The following example pipeline uses the right padding modifier with an empty key `%{->}`: ```json PUT /_ingest/pipeline/dissect-test @@ -328,7 +328,7 @@ The substrings are appended to the `address` field, as shown in the following re ### Append with order modifier (`+` and `/`) -The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. +The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/` modifier. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. #### Example of usage @@ -369,7 +369,7 @@ POST _ingest/pipeline/dissect-test/_simulate ``` {% include copy-curl.html %} -The substrings are appended into the `address` field in reverse order, as show in the following response: +The substrings are appended into the `address` field in reverse order, as shown in the following response: ```json { @@ -434,7 +434,7 @@ POST _ingest/pipeline/dissect-test/_simulate ``` {% include copy-curl.html %} -You should get a response similar to the following example: +Your response should be similar to the following: ```json { From da6875440a2a28a3ce7a6c5eee929c7ac9f2c4cb Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 26 Jan 2024 16:24:01 -0700 Subject: [PATCH 11/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index e349495b16..7a5ace4044 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -332,7 +332,7 @@ The append with order modifier combines the values of two or more keys into a si #### Example of usage -The following example pipeline uses the append with order modifier to reverse the pattern order defined in the preceding pipeline. The `append_separator` parameter must be defined in the processor configuration, outside of the `pattern`. It is only relevant with the `+` modifier. +The following example pipeline uses the append with order modifier to reverse the pattern order defined in the preceding pipeline. This pipeline specifies a separator to insert between the appended fields. If you don't specify a separator, all values will be appended together without a separator. ```json PUT /_ingest/pipeline/dissect-test From f82b3b8c62ccc3426d66cbfe81f851167372ca44 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 26 Jan 2024 16:24:10 -0700 Subject: [PATCH 12/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 7a5ace4044..324ba55fe2 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -328,7 +328,7 @@ The substrings are appended to the `address` field, as shown in the following re ### Append with order modifier (`+` and `/`) -The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/` modifier. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. +The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. #### Example of usage From 0ff6606eda10cab02ea819095a05445a1090b472 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 26 Jan 2024 16:24:21 -0700 Subject: [PATCH 13/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 324ba55fe2..315131f82e 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -326,7 +326,7 @@ The substrings are appended to the `address` field, as shown in the following re ``` {% include copy-curl.html %} -### Append with order modifier (`+` and `/`) +### Append with order modifier (`+` and `/n`) The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. From b44f48bbae37ed3d870d321c045cdcc29b8e5705 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:35:49 -0700 Subject: [PATCH 14/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 315131f82e..60740d70d6 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -7,7 +7,7 @@ nav_order: 60 # Dissect -The `dissect` processor extracts values from a document text field and maps them to individual fields based on dissect patterns. The processor is well-suited for field extractions from log messages with a known structure. Unlike the `grok` processor, `dissect` does not use regular expressions and has a simpler syntax. +The `dissect` processor extracts values from a document text field and maps them to individual fields based on dissect patterns. The processor is well suited for field extractions from log messages with a known structure. Unlike the `grok` processor, `dissect` does not use regular expressions and has a simpler syntax. ## Syntax From da19be40cfc4f5c7ee1be8d86ab1f58a31f1d5c3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:36:33 -0700 Subject: [PATCH 15/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 60740d70d6..d91d14d751 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -35,7 +35,7 @@ Parameter | Required/Optional | Description | `append_separator` | Optional | The separator character or string that separates appended fields. Default is `""` (empty string). `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running the processor. | -`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters errors. If set to `true`, failures are ignored. Default is `false`. | +`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, failures are ignored. Default is `false`. | `ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified field. If set to `true`, the processor does not modify the document if the field does not exist or is `null`. Default is `false`. | `on_failure` | Optional | A list of processors to run if the processor fails. | `tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | From 98b790b4b621dc4d4527ab3d8e5b5352fdfce0f1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:38:38 -0700 Subject: [PATCH 16/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index d91d14d751..63805c4b01 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -35,7 +35,7 @@ Parameter | Required/Optional | Description | `append_separator` | Optional | The separator character or string that separates appended fields. Default is `""` (empty string). `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running the processor. | -`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, failures are ignored. Default is `false`. | +`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, the processor failure is ignored. Default is `false`. | `ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified field. If set to `true`, the processor does not modify the document if the field does not exist or is `null`. Default is `false`. | `on_failure` | Optional | A list of processors to run if the processor fails. | `tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | From 5cdd221e38d56ae3fbbea6f56a1f5a9937b62a4f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:39:03 -0700 Subject: [PATCH 17/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 63805c4b01..60ec3131e5 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -140,7 +140,7 @@ GET testindex1/_doc/1 ## Dissect patterns -A dissect pattern is a way to tell `dissect` how to parse a string into a structured format. The pattern is defined by the parts of the string that you want to discard. For example, the `%{client_ip} - - [%{timestamp}]` dissect pattern parses the string `"192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] \"POST /login HTTP/1.1\" 200 3456"` into the following fields: +A dissect pattern is a method of telling the `dissect` processor how to parse a string into a structured format. The pattern is defined by the parts of the string that you want to discard. For example, the `%{client_ip} - - [%{timestamp}]` dissect pattern parses the string `"192.168.1.10 - - [03/Nov/2023:15:20:45 +0000] \"POST /login HTTP/1.1\" 200 3456"` into the following fields: ```json client_ip: "192.168.1.1" From 34a47fd4864ce2fee21cda2ca0659ddaf30338e4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:39:41 -0700 Subject: [PATCH 18/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 60ec3131e5..38e6d2004a 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -147,7 +147,7 @@ client_ip: "192.168.1.1" @timestamp: "03/Nov/2023:15:20:45 +0000" ``` -The dissect pattern works by matching the string against a set of rules. For example, the first rule is to discard a single space. Dissect will find this space and then assign the value of `client_ip` to everything up to that space. The next rule is to match the `[` and `]` characters and then assign the value of `@timestamp` to everything in between. +A dissect pattern works by matching a string against a set of rules. For example, the first rule discards a single space. Dissect will find this space and then assign the value of `client_ip` to everything up to that space. The next rule matches the `[` and `]` characters and then assigns the value of `@timestamp` to everything in between. ### Building successful dissect patterns From b379bc3b6ce426c75f0857759fc0f16d289ac9df Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:40:06 -0700 Subject: [PATCH 19/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 38e6d2004a..a74cfc03e5 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -151,7 +151,7 @@ A dissect pattern works by matching a string against a set of rules. For example ### Building successful dissect patterns -When building dissect pattern, it is important to pay attention to the parts of the string that you want to discard. If you discard too much of the string, then `dissect` may not be able to successfully parse the remaining data. Conversely, if you do not discard enough of the string, then `dissect` may create unnecessary fields. +When building a dissect pattern, it is important to pay attention to the parts of the string that you want to discard. If you discard too much of the string, then the `dissect` processor may not be able to successfully parse the remaining data. Conversely, if you do not discard enough of the string, then the processor may create unnecessary fields. If any `%{keyname}` defined in the pattern do not have a value, then an exception is thrown. You can handle this exception by providing error handling steps in the `on_failure` parameter. From 9ea4d6f88c8fc542afe229b178ead2efefc1e0c1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:40:26 -0700 Subject: [PATCH 20/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index a74cfc03e5..1765bad675 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -153,7 +153,7 @@ A dissect pattern works by matching a string against a set of rules. For example When building a dissect pattern, it is important to pay attention to the parts of the string that you want to discard. If you discard too much of the string, then the `dissect` processor may not be able to successfully parse the remaining data. Conversely, if you do not discard enough of the string, then the processor may create unnecessary fields. -If any `%{keyname}` defined in the pattern do not have a value, then an exception is thrown. You can handle this exception by providing error handling steps in the `on_failure` parameter. +If any `%{keyname}` defined in the pattern does not have a value, then an exception is thrown. You can handle this exception by providing error handling steps in the `on_failure` parameter. ### Empty and named skip keys From ce9ff071b0be87ccb312cab1350a74c94a851f37 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:40:43 -0700 Subject: [PATCH 21/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 1765bad675..81cbf87098 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -157,7 +157,7 @@ If any `%{keyname}` defined in the pattern does not have a value, then an except ### Empty and named skip keys -An empty key `%{}` or a [named skip key](#named-skip-key) can be used to match values but exclude the value from the final document. This can be useful if you want to parse a string but do not need to store all its parts. +An empty key `%{}` or a [named skip key](#named-skip-key) can be used to match values but exclude the value from the final document. This can be useful if you want to parse a string but do not need to store all of its parts. ### Converting matched values to a non-string data type From 04af08675d23fcb3fc4ae3f99578b2578790b12f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:41:04 -0700 Subject: [PATCH 22/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 81cbf87098..d5b3c3ba8c 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -165,7 +165,7 @@ By default, all matched values are represented as string data types. If you need ### Key modifiers -The `dissect` processor supports key modifiers that can change the default processor behavior. These modifiers are always placed to the left or right of the `%{keyname}` and are always enclosed within `%{}`. For example, the `%{+keyname->}` modifier includes the append and right padding modifiers. Key modifiers are useful for cases such as combining multiple fields into a single line of output, creating formatted lists of data items, or aggregating values from multiple sources. +The `dissect` processor supports key modifiers that can change the default processor behavior. These modifiers are always placed to the left or right of the `%{keyname}` and are always enclosed within `%{}`. For example, the `%{+keyname->}` modifier includes the append and right padding modifiers. Key modifiers are useful for combining multiple fields into a single line of output, creating formatted lists of data items, or aggregating values from multiple sources. The following table lists the primary modifiers for the `dissect` processor. From a3e7a8d7ef0f434931907a7886703d84272a0cbd Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:41:24 -0700 Subject: [PATCH 23/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index d5b3c3ba8c..71b8833373 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -171,7 +171,7 @@ The following table lists the primary modifiers for the `dissect` processor. Modifier | Name | Position | Example | Description | |-----------|-----------|-----------| -`->` | Skip right padding | (far) right | `%{keyname->}` | Tells `dissect` to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell `dissect` to skip over any padding characters, such as two spaces or any varying character padding, that follow `timestamp`. | +`->` | Skip right padding | (far) right | `%{keyname->}` | Tells the `dissect` processor to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell the processor to skip any padding characters, such as two spaces or any varying character padding, that follow `timestamp`. | `+` | Append | left | `%{keyname} %{+keyname}` | Appends two or more fields together. | `+` with `/n` | Append with order | left and right | `%{+keyname}/2 %{+keyname/1}` | Appends two or more fields together in the order specified. | `?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | From 7b44a063fdc2f5e1c830dce61c4630efcc1add54 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:41:37 -0700 Subject: [PATCH 24/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 71b8833373..7d713ebfbe 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -172,7 +172,7 @@ The following table lists the primary modifiers for the `dissect` processor. Modifier | Name | Position | Example | Description | |-----------|-----------|-----------| `->` | Skip right padding | (far) right | `%{keyname->}` | Tells the `dissect` processor to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell the processor to skip any padding characters, such as two spaces or any varying character padding, that follow `timestamp`. | -`+` | Append | left | `%{keyname} %{+keyname}` | Appends two or more fields together. | +`+` | Append | left | `%{keyname} %{+keyname}` | Appends two or more fields. | `+` with `/n` | Append with order | left and right | `%{+keyname}/2 %{+keyname/1}` | Appends two or more fields together in the order specified. | `?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | `*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as value of `*` and output value of `&`. | From de25752795fdd63b6b3609d54d61b11c8cf67cd0 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:41:52 -0700 Subject: [PATCH 25/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 7d713ebfbe..d9edc541e7 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -173,7 +173,7 @@ Modifier | Name | Position | Example | Description | |-----------|-----------|-----------| `->` | Skip right padding | (far) right | `%{keyname->}` | Tells the `dissect` processor to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell the processor to skip any padding characters, such as two spaces or any varying character padding, that follow `timestamp`. | `+` | Append | left | `%{keyname} %{+keyname}` | Appends two or more fields. | -`+` with `/n` | Append with order | left and right | `%{+keyname}/2 %{+keyname/1}` | Appends two or more fields together in the order specified. | +`+` with `/n` | Append with order | left and right | `%{+keyname}/2 %{+keyname/1}` | Appends two or more fields in the specified order. | `?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | `*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as value of `*` and output value of `&`. | From 47195d09cda471d3093e1db58835a7e83ff8d204 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:42:06 -0700 Subject: [PATCH 26/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index d9edc541e7..4c518d6705 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -175,7 +175,7 @@ Modifier | Name | Position | Example | Description | `+` | Append | left | `%{keyname} %{+keyname}` | Appends two or more fields. | `+` with `/n` | Append with order | left and right | `%{+keyname}/2 %{+keyname/1}` | Appends two or more fields in the specified order. | `?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | -`*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as value of `*` and output value of `&`. | +`*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as the value of `*` and the output value of `&`. | Detailed descriptions of each key modifier, along with usage examples, are in the following sections. From 7d755f66c4c977862bb51a257fb4edb56003afe8 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:42:20 -0700 Subject: [PATCH 27/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 4c518d6705..6ba156c8e3 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -177,7 +177,7 @@ Modifier | Name | Position | Example | Description | `?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | `*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as the value of `*` and the output value of `&`. | -Detailed descriptions of each key modifier, along with usage examples, are in the following sections. +Detailed descriptions of each key modifier, along with usage examples, are provided in the following sections. ### Right padding modifier (`->`) From 7b03a4bd07f0e054c4fd9db1a47b75411482667f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:42:34 -0700 Subject: [PATCH 28/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 6ba156c8e3..3b3c0bf4d3 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -181,7 +181,7 @@ Detailed descriptions of each key modifier, along with usage examples, are provi ### Right padding modifier (`->`) -The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For instance, the pattern `%{hellokey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because the pattern only has one space while the source string has two. +The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For example, the pattern `%{hellokey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because the pattern only has one space while the source string has two. The right padding modifier can be used to address this issue. By adding the right padding modifier to the pattern `%{helloworldkey->} %{worldkey}`, it will match Hello world (one space), Hello  world (two spaces), and even Hello          world (ten spaces). From 01206c5375c3861d09667f3f91e5dae0f1765d2e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:42:53 -0700 Subject: [PATCH 29/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 3b3c0bf4d3..f3607b1a6d 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -183,7 +183,7 @@ Detailed descriptions of each key modifier, along with usage examples, are provi The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For example, the pattern `%{hellokey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because the pattern only has one space while the source string has two. -The right padding modifier can be used to address this issue. By adding the right padding modifier to the pattern `%{helloworldkey->} %{worldkey}`, it will match Hello world (one space), Hello  world (two spaces), and even Hello          world (ten spaces). +The right padding modifier can be used to address this issue. When added to the pattern `%{helloworldkey->} %{worldkey}`, the right padding modifier will match Hello world (1 space), Hello  world (2 spaces), and even Hello          world (10 spaces). The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}` or `%{}`. From 599afa17ff62d135abec11386e90848bd87fd241 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:43:07 -0700 Subject: [PATCH 30/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index f3607b1a6d..a67a413ee8 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -187,7 +187,7 @@ The right padding modifier can be used to address this issue. When added to the The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}` or `%{}`. -#### Example of usage +#### Example usage The following is an example of how to use a right padding modifier: From 3be93a26de6ebb7426e8643a9bb9995bca2c2f8c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:43:22 -0700 Subject: [PATCH 31/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index a67a413ee8..f6b3e80306 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -193,7 +193,7 @@ The following is an example of how to use a right padding modifier: `%{city->}, %{state} %{zip}` -In this pattern, the right padding modifier `->` is applied to the `%{city}` key. Both addresses contain the same information, but the second entry has an extra word, `City`, in the city field. The right padding modifier allows the pattern to match both of these address entries, even though they have slightly different formats: +In this pattern, the right padding modifier `->` is applied to the `%{city}` key. Both addresses contain the same information, but the second entry has an extra word, `City`, in the city field. The right padding modifier allows the pattern to match both of these address entries even though they have slightly different formats: ```bash New York, NY 10017 From c7d1e04e04e6c0329b3533a70896a4840cb8cce0 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:43:45 -0700 Subject: [PATCH 32/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index f6b3e80306..cbc1cd8573 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -218,7 +218,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -You can test the pipeline using the following example: +You can test the pipeline by using the following example: ```json POST _ingest/pipeline/dissect-test/_simulate From 5b13a02a422d59b9e7ec3b35e5039a80058e07e3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:44:13 -0700 Subject: [PATCH 33/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index cbc1cd8573..07ef328c9e 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -236,7 +236,7 @@ POST _ingest/pipeline/dissect-test/_simulate ``` {% include copy-curl.html %} -Your response should be similar to the following: +Your response should appear similar to the following: ```json { From a6d8d42007c65568b981eca71f030193d86c07aa Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:52:44 -0700 Subject: [PATCH 34/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 07ef328c9e..e0f1541a56 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -264,7 +264,7 @@ Your response should appear similar to the following: The append modifier combines the values of two or more values into a single output value. The values are appended from left to right. You can also specify an optional separator to be inserted between the values. -#### Example of usage +#### Example usage The following is an example pipeline with an append modifier: From 1c95dfe853d2247fd644b503702c09764dd80bf0 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:53:02 -0700 Subject: [PATCH 35/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index e0f1541a56..2b3da2ea85 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -285,7 +285,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -You can test the pipeline using the following example: +You can test the pipeline by using the following example: ```json POST _ingest/pipeline/dissect-test/_simulate From 736ca78e4c0411f2f77d7a06bf90efd572804690 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:55:54 -0700 Subject: [PATCH 36/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 2b3da2ea85..3914946c0a 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -218,7 +218,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -You can test the pipeline by using the following example: +You can test the pipeline by using the following example pipeline: ```json POST _ingest/pipeline/dissect-test/_simulate @@ -285,7 +285,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -You can test the pipeline by using the following example: +You can test the pipeline by using the following example pipeline: ```json POST _ingest/pipeline/dissect-test/_simulate @@ -328,7 +328,7 @@ The substrings are appended to the `address` field, as shown in the following re ### Append with order modifier (`+` and `/n`) -The append with order modifier combines the values of two or more keys into a single output value based on the order specified after the `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. +The append with order modifier combines the values of two or more keys into a single output value based on the order specified after `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. #### Example of usage @@ -351,7 +351,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -You can test the pipeline using the following example: +You can test the pipeline using the following example pipeline: ```json POST _ingest/pipeline/dissect-test/_simulate @@ -416,7 +416,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -You can test the pipeline using the following example: +You can test the pipeline using the following example pipeline: ```json POST _ingest/pipeline/dissect-test/_simulate @@ -482,7 +482,7 @@ PUT /_ingest/pipeline/dissect-test ``` {% include copy-curl.html %} -You can test the pipeline using the following example: +You can test the pipeline using the following example pipeline: ```json POST _ingest/pipeline/dissect-test/_simulate From 472d152726867f859a825ccc5c108b46e032fd11 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:56:09 -0700 Subject: [PATCH 37/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 3914946c0a..a9b0bca4c4 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -330,7 +330,7 @@ The substrings are appended to the `address` field, as shown in the following re The append with order modifier combines the values of two or more keys into a single output value based on the order specified after `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. -#### Example of usage +#### Example usage The following example pipeline uses the append with order modifier to reverse the pattern order defined in the preceding pipeline. This pipeline specifies a separator to insert between the appended fields. If you don't specify a separator, all values will be appended together without a separator. From fe543786bfb154c2b6f3c784f01151df50f7ed27 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:58:03 -0700 Subject: [PATCH 38/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index a9b0bca4c4..1e5a15dc7d 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -183,7 +183,7 @@ Detailed descriptions of each key modifier, along with usage examples, are provi The dissection algorithm is precise and requires that every character in the pattern exactly match the source string. For example, the pattern `%{hellokey} %{worldkey}` (one space) will match the string "Hello world" (one space) but not the string "Hello world" (two spaces) because the pattern only has one space while the source string has two. -The right padding modifier can be used to address this issue. When added to the pattern `%{helloworldkey->} %{worldkey}`, the right padding modifier will match Hello world (1 space), Hello  world (2 spaces), and even Hello          world (10 spaces). +The _right padding modifier_ can be used to address this issue. When added to the pattern `%{helloworldkey->} %{worldkey}`, the right padding modifier will match Hello world (1 space), Hello  world (2 spaces), and even Hello          world (10 spaces). The right padding modifier is used to allow for the repetition of characters following a `%{keyname->}`. The right padding modifier can be applied to any key along with any other modifiers. It should always be the rightmost modifier, for example, `%{+keyname/1->}` or `%{}`. @@ -262,7 +262,7 @@ Your response should appear similar to the following: ### Append modifier (`+`) -The append modifier combines the values of two or more values into a single output value. The values are appended from left to right. You can also specify an optional separator to be inserted between the values. +The _append modifier_ combines the values of two or more values into a single output value. The values are appended from left to right. You can also specify an optional separator to be inserted between the values. #### Example usage @@ -328,7 +328,7 @@ The substrings are appended to the `address` field, as shown in the following re ### Append with order modifier (`+` and `/n`) -The append with order modifier combines the values of two or more keys into a single output value based on the order specified after `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. +The _append with order modifier_ combines the values of two or more keys into a single output value based on the order specified after `/`. You have the flexibility to customize the separator that separates the appended values. The append modifier is useful for compiling multiple fields into a single formatted output line, constructing structured lists of data items, and consolidating values from various sources. #### Example usage @@ -394,7 +394,7 @@ The substrings are appended into the `address` field in reverse order, as shown ### Named skip key -The named skip key modifier excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output. +The _named skip key modifier_ excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output. #### Example of usage From 91885c4e5b167e8b36dacedf79f861a2fc6fe84a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:59:12 -0700 Subject: [PATCH 39/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 1e5a15dc7d..7de4dff419 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -332,7 +332,7 @@ The _append with order modifier_ combines the values of two or more keys into a #### Example usage -The following example pipeline uses the append with order modifier to reverse the pattern order defined in the preceding pipeline. This pipeline specifies a separator to insert between the appended fields. If you don't specify a separator, all values will be appended together without a separator. +The following example pipeline uses the append with order modifier to reverse the pattern order defined in the preceding pipeline. This pipeline specifies a separator to be inserted between the appended fields. If you don't specify a separator, all values will be appended without a separator. ```json PUT /_ingest/pipeline/dissect-test From 4818868afa26444e6f7fc999c3379f062c1f8918 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:59:25 -0700 Subject: [PATCH 40/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 7de4dff419..cb00cc8ae5 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -369,7 +369,7 @@ POST _ingest/pipeline/dissect-test/_simulate ``` {% include copy-curl.html %} -The substrings are appended into the `address` field in reverse order, as shown in the following response: +The substrings are appended to the `address` field in reverse order, as shown in the following response: ```json { From 19bb8265e904c977f3082f321b61c8212a8db5dd Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:59:45 -0700 Subject: [PATCH 41/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index cb00cc8ae5..34d334f031 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -398,7 +398,7 @@ The _named skip key modifier_ excludes specific matches from the final output by #### Example of usage -The following pattern uses a named skip key to exclude a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{?ignore}`, to clarify that the corresponding value should be excluded from the final result: +The following pattern uses a named skip key to exclude a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{?ignore}`, to clarify that the corresponding value should be excluded from the final output: ```json PUT /_ingest/pipeline/dissect-test From 240f7094ebb3e4fe7aa0ec4bd87741e85bcc8a21 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 09:59:57 -0700 Subject: [PATCH 42/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 34d334f031..1caedd5f1a 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -396,7 +396,7 @@ The substrings are appended to the `address` field in reverse order, as shown in The _named skip key modifier_ excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output. -#### Example of usage +#### Example usage The following pattern uses a named skip key to exclude a field (in this case, `ignore`) from the output. You can assign a descriptive name to the empty key, for example, `%{?ignore}`, to clarify that the corresponding value should be excluded from the final output: From f31da78d7d364a4a27d3dbaff5d2a63e0e764961 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:00:15 -0700 Subject: [PATCH 43/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 1caedd5f1a..8a55de6a99 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -434,7 +434,7 @@ POST _ingest/pipeline/dissect-test/_simulate ``` {% include copy-curl.html %} -Your response should be similar to the following: +Your response should appear similar to the following: ```json { From fa9986988e4fb002f8e8803cbce4efb7166c1f83 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:01:17 -0700 Subject: [PATCH 44/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 8a55de6a99..6abcc9d9f9 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -392,7 +392,7 @@ The substrings are appended to the `address` field in reverse order, as shown in ``` {% include copy-curl.html %} -### Named skip key +### Named skip key modifier The _named skip key modifier_ excludes specific matches from the final output by using an empty key `{}` or `?` modifier within the pattern. For example, the following patterns are equivalent: `%{firstName} %{lastName} %{?ignore}` and `%{firstName} %{lastName} %{}`. The named skip key modifier is useful for excluding irrelevant or unnecessary fields from the output. From 4e0418f43b45b821acca1c314c6185bff1e7f09e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:01:35 -0700 Subject: [PATCH 45/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 6abcc9d9f9..e22c8e7e7d 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -460,7 +460,7 @@ Your response should appear similar to the following: ### Reference keys (`*` and `&`) -Reference keys use parsed values as key/value pairings for structured content. This can use useful when handling systems that partially log data in key/value pairs. By using reference keys, you can preserve the key/value relationship and maintain the integrity of the extracted information. +Reference keys use parsed values as key-value pairings for structured content. This can use useful when handling systems that partially log data in key-value pairs. By using reference keys, you can preserve the key-value relationship and maintain the integrity of the extracted information. #### Example of usage From 0022732425ce6d4243466e7f5bdb4fa73b97c3d1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:01:48 -0700 Subject: [PATCH 46/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index e22c8e7e7d..506bbd0b58 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -462,7 +462,7 @@ Your response should appear similar to the following: Reference keys use parsed values as key-value pairings for structured content. This can use useful when handling systems that partially log data in key-value pairs. By using reference keys, you can preserve the key-value relationship and maintain the integrity of the extracted information. -#### Example of usage +#### Example usage The following pattern uses a reference key to extract data into a structured format. In this example, `client_ip` and two key/value pairs are extracted for the next values: From b0bf080bd3890dc7160ef35764823769d5684af4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:02:04 -0700 Subject: [PATCH 47/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 506bbd0b58..f2e36a0cbd 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -464,7 +464,7 @@ Reference keys use parsed values as key-value pairings for structured content. T #### Example usage -The following pattern uses a reference key to extract data into a structured format. In this example, `client_ip` and two key/value pairs are extracted for the next values: +The following pattern uses a reference key to extract data into a structured format. In this example, `client_ip` and two key-value pairs are extracted for the next values: ```json PUT /_ingest/pipeline/dissect-test From 3eb53307fd9a9c0bcfc52e34983fc1366733cad3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:02:22 -0700 Subject: [PATCH 48/53] Update _ingest-pipelines/processors/dissect.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index f2e36a0cbd..2c3d30ec96 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -500,7 +500,7 @@ POST _ingest/pipeline/dissect-test/_simulate ``` {% include copy-curl.html %} -The two key/value pairs were extracted into fields, as shown in the following response: +The two key-value pairs were extracted into fields, as shown in the following response: ```json { From 5d6a747081bbd76e90d912ca02b75f1b0a50efe1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:08:17 -0700 Subject: [PATCH 49/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 2c3d30ec96..80532b7318 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -147,7 +147,7 @@ client_ip: "192.168.1.1" @timestamp: "03/Nov/2023:15:20:45 +0000" ``` -A dissect pattern works by matching a string against a set of rules. For example, the first rule discards a single space. Dissect will find this space and then assign the value of `client_ip` to everything up to that space. The next rule matches the `[` and `]` characters and then assigns the value of `@timestamp` to everything in between. +A dissect pattern works by matching a string against a set of rules. For example, the first rule discards a single space. The `dissect` processor will find this space and then assign the value of `client_ip` to all the data characters before that space. The next rule matches the `[` and `]` characters and then assigns the value of `@timestamp` to everything in between. ### Building successful dissect patterns From 5a4c9bbc0527b196f6767ff656b189c8fc388937 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:13:02 -0700 Subject: [PATCH 50/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 80532b7318..84015dc1f1 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -171,7 +171,7 @@ The following table lists the primary modifiers for the `dissect` processor. Modifier | Name | Position | Example | Description | |-----------|-----------|-----------| -`->` | Skip right padding | (far) right | `%{keyname->}` | Tells the `dissect` processor to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell the processor to skip any padding characters, such as two spaces or any varying character padding, that follow `timestamp`. | +`->` | Skip right padding | (far) right | `%{keyname->}` | Tells the `dissect` processor to skip over any repeated characters to the right. For example, `%{timestamp->}` could be used to tell the processor to skip any padding characters, such as two consecutive spaces or any varying character padding, that follow `timestamp`. | `+` | Append | left | `%{keyname} %{+keyname}` | Appends two or more fields. | `+` with `/n` | Append with order | left and right | `%{+keyname}/2 %{+keyname/1}` | Appends two or more fields in the specified order. | `?` | Named skip key | left | `%{?skipme}` | Skips the matched value in the output. Same behavior as `%{}`. | From 59f4e8259766ba9d0b504ed71de4a67ac0e2bae6 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:23:05 -0700 Subject: [PATCH 51/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 84015dc1f1..ff4ef3ee8c 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -193,7 +193,7 @@ The following is an example of how to use a right padding modifier: `%{city->}, %{state} %{zip}` -In this pattern, the right padding modifier `->` is applied to the `%{city}` key. Both addresses contain the same information, but the second entry has an extra word, `City`, in the city field. The right padding modifier allows the pattern to match both of these address entries even though they have slightly different formats: +In this pattern, the right padding modifier `->` is applied to the `%{city}` key. Both addresses contain the same information, but the second entry has an extra word, `City`, in the `city` field. The right padding modifier allows the pattern to match both of these address entries even though they have slightly different formats: ```bash New York, NY 10017 From 84ff48320939c237b67294f1d9aa8511dabb0374 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:30:08 -0700 Subject: [PATCH 52/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index ff4ef3ee8c..70ea10ae38 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -464,7 +464,7 @@ Reference keys use parsed values as key-value pairings for structured content. T #### Example usage -The following pattern uses a reference key to extract data into a structured format. In this example, `client_ip` and two key-value pairs are extracted for the next values: +The following pattern uses a reference key to find data and organize it into a structured format. In this example, `client_ip` and two key-value pairs are extracted for the next values: ```json PUT /_ingest/pipeline/dissect-test From f17c8d8e432b27cc099e3a71a496d06be408d328 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Jan 2024 10:44:10 -0700 Subject: [PATCH 53/53] Update dissect.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dissect.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_ingest-pipelines/processors/dissect.md b/_ingest-pipelines/processors/dissect.md index 70ea10ae38..8368dbbf63 100644 --- a/_ingest-pipelines/processors/dissect.md +++ b/_ingest-pipelines/processors/dissect.md @@ -147,7 +147,7 @@ client_ip: "192.168.1.1" @timestamp: "03/Nov/2023:15:20:45 +0000" ``` -A dissect pattern works by matching a string against a set of rules. For example, the first rule discards a single space. The `dissect` processor will find this space and then assign the value of `client_ip` to all the data characters before that space. The next rule matches the `[` and `]` characters and then assigns the value of `@timestamp` to everything in between. +A dissect pattern works by matching a string against a set of rules. For example, the first rule discards a single space. The `dissect` processor will find this space and then assign the value of `client_ip` to all the characters before that space. The next rule matches the `[` and `]` characters and then assigns the value of `@timestamp` to everything in between. ### Building successful dissect patterns @@ -464,7 +464,7 @@ Reference keys use parsed values as key-value pairings for structured content. T #### Example usage -The following pattern uses a reference key to find data and organize it into a structured format. In this example, `client_ip` and two key-value pairs are extracted for the next values: +The following pattern uses a reference key to extract data into a structured format. In this example, `client_ip` and two key-value pairs are extracted for the next values: ```json PUT /_ingest/pipeline/dissect-test