From 2fa619fc752dbfa7077b81f5d2041c3389711795 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Tue, 13 Aug 2024 18:01:13 +0100 Subject: [PATCH 1/8] Adding first draft of delimited-payload token filter WIP Signed-off-by: Anton Rubin --- _analyzers/token-filters/delimited-payload.md | 102 ++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 _analyzers/token-filters/delimited-payload.md diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md new file mode 100644 index 0000000000..881be70a05 --- /dev/null +++ b/_analyzers/token-filters/delimited-payload.md @@ -0,0 +1,102 @@ +--- +layout: default +title: Delimited payload +parent: Token filters +nav_order: 90 +--- + +# Decimal digit token filter + +The `delimited_payload` token filter in OpenSearch is used to parse and attach payloads (extra metadata) to tokens during the analysis process. This is particularly useful when you want to associate additional data (like weights, scores, or other numeric values) with tokens for use in scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. + +When analyzing text, `delimited_payload` token filter parses each token, extracts the payload, and attaches it to the token. This payload can later be used in queries to influence scoring, boosting, or other custom behaviors. + +## Parameters + +The `delimited_payload` token filter in OpenSearch has two _optional_ parameters: + +1. `encoding`: specifies the data type of the payload attached to the tokens. This determines how the payload data is stored and interpreted during analysis and querying. There are three valid values: + + - `identity`: The payload is treated as a sequence of characters. For example: "user|admin" where "admin" is stored as a string. + - `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format. For example: "car|2.5" would store 2.5 as a floating-point number. + - `int`: The payload is interpreted as a 32-bit integer. For example: "priority|1" would store 1 as an integer. + +2. `delimiter`: specifies character used to separate the token from its payload in the input text. By default, this is set to the pipe character (|). + + +## Example + +The following example request creates a new index named `my_index` and configures an analyzer with `decimal_digit` filter: + +```json +PUT /my_index +{ + "settings": { + "analysis": { + "filter": { + "my_decimal_digit_filter": { + "type": "decimal_digit" + } + }, + "analyzer": { + "my_analyzer": { + "type": "custom", + "tokenizer": "standard", + "filter": ["my_decimal_digit_filter"] + } + } + } + } +} + +``` +{% include copy-curl.html %} + +## Generated tokens + +Use the following request to examine the tokens generated using the created analyzer: + +```json +POST /my_index/_analyze +{ + "analyzer": "my_analyzer", + "text": "123 ١٢٣ १२३" +} +``` +{% include copy-curl.html %} + +`text` breakdown: + + - "123" (ASCII digits) + - "١٢٣" (Arabic-Indic digits) + - "१२३" (Devanagari digits) + +The response contains the generated tokens: + +```json +{ + "tokens": [ + { + "token": "123", + "start_offset": 0, + "end_offset": 3, + "type": "", + "position": 0 + }, + { + "token": "123", + "start_offset": 4, + "end_offset": 7, + "type": "", + "position": 1 + }, + { + "token": "123", + "start_offset": 8, + "end_offset": 11, + "type": "", + "position": 2 + } + ] +} +``` \ No newline at end of file From c9bfb83b60ed8385287dd6e474c7581bb43ff870 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Wed, 14 Aug 2024 15:08:15 +0100 Subject: [PATCH 2/8] adding the rest of the details to delimited payload token filter Signed-off-by: Anton Rubin --- _analyzers/token-filters/delimited-payload.md | 176 +++++++++++++++--- _analyzers/token-filters/index.md | 2 +- 2 files changed, 146 insertions(+), 32 deletions(-) diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md index 881be70a05..b651550e58 100644 --- a/_analyzers/token-filters/delimited-payload.md +++ b/_analyzers/token-filters/delimited-payload.md @@ -5,28 +5,31 @@ parent: Token filters nav_order: 90 --- -# Decimal digit token filter +# Delimited payload token filter The `delimited_payload` token filter in OpenSearch is used to parse and attach payloads (extra metadata) to tokens during the analysis process. This is particularly useful when you want to associate additional data (like weights, scores, or other numeric values) with tokens for use in scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. When analyzing text, `delimited_payload` token filter parses each token, extracts the payload, and attaches it to the token. This payload can later be used in queries to influence scoring, boosting, or other custom behaviors. +The payload is not returned in the query response by default, additional configuration is needed in order to be able to view the payload, see [Example with stored payload]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/#example-with-stored-payload) + ## Parameters The `delimited_payload` token filter in OpenSearch has two _optional_ parameters: 1. `encoding`: specifies the data type of the payload attached to the tokens. This determines how the payload data is stored and interpreted during analysis and querying. There are three valid values: - - `identity`: The payload is treated as a sequence of characters. For example: "user|admin" where "admin" is stored as a string. - - `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format. For example: "car|2.5" would store 2.5 as a floating-point number. - - `int`: The payload is interpreted as a 32-bit integer. For example: "priority|1" would store 1 as an integer. + - `identity`: The payload is treated as a sequence of characters. For example: `"user|admin"` where "admin" is stored as a string. + + - `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format. For example: `"car|2.5"` would store 2.5 as a floating-point number. -2. `delimiter`: specifies character used to separate the token from its payload in the input text. By default, this is set to the pipe character (|). + - `int`: The payload is interpreted as a 32-bit integer. For example: `"priority|1"` would store 1 as an integer. +2. `delimiter`: specifies character used to separate the token from its payload in the input text. By default, this is set to the pipe character (`|`). -## Example +## Example without stored payload -The following example request creates a new index named `my_index` and configures an analyzer with `decimal_digit` filter: +The following example request creates a new index named `my_index` and configures an analyzer with `delimited_payload` filter: ```json PUT /my_index @@ -34,21 +37,22 @@ PUT /my_index "settings": { "analysis": { "filter": { - "my_decimal_digit_filter": { - "type": "decimal_digit" + "my_payload_filter": { + "type": "delimited_payload", + "delimiter": "|", + "encoding": "float" } }, "analyzer": { "my_analyzer": { "type": "custom", - "tokenizer": "standard", - "filter": ["my_decimal_digit_filter"] + "tokenizer": "whitespace", + "filter": ["my_payload_filter"] } } } } } - ``` {% include copy-curl.html %} @@ -60,43 +64,153 @@ Use the following request to examine the tokens generated using the created anal POST /my_index/_analyze { "analyzer": "my_analyzer", - "text": "123 ١٢٣ १२३" + "text": "red|1.5 fast|2.0 car|1.0" } ``` {% include copy-curl.html %} -`text` breakdown: - - - "123" (ASCII digits) - - "١٢٣" (Arabic-Indic digits) - - "१२३" (Devanagari digits) - The response contains the generated tokens: ```json { "tokens": [ { - "token": "123", + "token": "red", "start_offset": 0, - "end_offset": 3, - "type": "", + "end_offset": 7, + "type": "word", "position": 0 }, { - "token": "123", - "start_offset": 4, - "end_offset": 7, - "type": "", + "token": "fast", + "start_offset": 8, + "end_offset": 16, + "type": "word", "position": 1 }, { - "token": "123", - "start_offset": 8, - "end_offset": 11, - "type": "", + "token": "car", + "start_offset": 17, + "end_offset": 24, + "type": "word", "position": 2 } ] } -``` \ No newline at end of file +``` + +## Example with stored payload + +You can configure the payload to be returned in the response by creating an index that stores vectors, using `term_vector` set to `with_positions_payloads` or `with_positions_offsets_payloads` in mappings of the index, see following example: + +```json +PUT /visible_payloads +{ + "mappings": { + "properties": { + "text": { + "type": "text", + "term_vector": "with_positions_payloads", + "analyzer": "custom_analyzer" + } + } + }, + "settings": { + "analysis": { + "filter": { + "my_payload_filter": { + "type": "delimited_payload", + "delimiter": "|", + "encoding": "float" + } + }, + "analyzer": { + "custom_analyzer": { + "tokenizer": "whitespace", + "filter": [ "my_payload_filter" ] + } + } + } + } +} +``` +{% include copy-curl.html %} + +You can index a document into this index using following request: + +```json +PUT /visible_payloads/_doc/1 +{ + "text": "red|1.5 fast|2.0 car|1.0" +} +``` +{% include copy-curl.html %} + +## Generated tokens + +Use the following request to examine the tokens generated using the created analyzer: + +```json +GET /visible_payloads/_termvectors/1 +{ + "fields": ["text"] +} +``` +{% include copy-curl.html %} + +The response contains the generated tokens which include payload: + +```json +{ + "_index": "visible_payloads", + "_id": "1", + "_version": 1, + "found": true, + "took": 3, + "term_vectors": { + "text": { + "field_statistics": { + "sum_doc_freq": 3, + "doc_count": 1, + "sum_ttf": 3 + }, + "terms": { + "brown": { + "term_freq": 1, + "tokens": [ + { + "position": 1, + "start_offset": 10, + "end_offset": 19, + "payload": "QEAAAA==" + } + ] + }, + "fox": { + "term_freq": 1, + "tokens": [ + { + "position": 2, + "start_offset": 20, + "end_offset": 27, + "payload": "P8AAAA==" + } + ] + }, + "quick": { + "term_freq": 1, + "tokens": [ + { + "position": 0, + "start_offset": 0, + "end_offset": 9, + "payload": "QCAAAA==" + } + ] + } + } + } + } +} +``` +{% include copy-curl.html %} diff --git a/_analyzers/token-filters/index.md b/_analyzers/token-filters/index.md index f4e9c434e7..aee4330dfb 100644 --- a/_analyzers/token-filters/index.md +++ b/_analyzers/token-filters/index.md @@ -21,7 +21,7 @@ Token filter | Underlying Lucene token filter| Description `common_grams` | [CommonGramsFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/commongrams/CommonGramsFilter.html) | Generates bigrams for a list of frequently occurring terms. The output contains both single terms and bigrams. `conditional` | [ConditionalTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/miscellaneous/ConditionalTokenFilter.html) | Applies an ordered list of token filters to tokens that match the conditions provided in a script. `decimal_digit` | [DecimalDigitFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/core/DecimalDigitFilter.html) | Converts all digits in the Unicode decimal number general category to basic Latin digits (0--9). -`delimited_payload` | [DelimitedPayloadTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html) | Separates a token stream into tokens with corresponding payloads, based on a provided delimiter. A token consists of all characters before the delimiter, and a payload consists of all characters after the delimiter. For example, if the delimiter is `|`, then for the string `foo|bar`, `foo` is the token and `bar` is the payload. +[`delimited_payload`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/) | [DelimitedPayloadTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html) | Separates a token stream into tokens with corresponding payloads, based on a provided delimiter. A token consists of all characters before the delimiter, and a payload consists of all characters after the delimiter. For example, if the delimiter is `|`, then for the string `foo|bar`, `foo` is the token and `bar` is the payload. [`delimited_term_freq`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-term-frequency/) | [DelimitedTermFrequencyTokenFilter](https://lucene.apache.org/core/9_7_0/analysis/common/org/apache/lucene/analysis/miscellaneous/DelimitedTermFrequencyTokenFilter.html) | Separates a token stream into tokens with corresponding term frequencies, based on a provided delimiter. A token consists of all characters before the delimiter, and a term frequency is the integer after the delimiter. For example, if the delimiter is `|`, then for the string `foo|5`, `foo` is the token and `5` is the term frequency. `dictionary_decompounder` | [DictionaryCompoundWordTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/compound/DictionaryCompoundWordTokenFilter.html) | Decomposes compound words found in many Germanic languages. `edge_ngram` | [EdgeNGramTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/ngram/EdgeNGramTokenFilter.html) | Tokenizes the given token into edge n-grams (n-grams that start at the beginning of the token) of lengths between `min_gram` and `max_gram`. Optionally, keeps the original token. From a5e8e24e53dc1703c37dd79e802578392db3a06d Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Thu, 12 Sep 2024 11:07:35 +0100 Subject: [PATCH 3/8] Update delimited-payload.md Signed-off-by: AntonEliatra --- _analyzers/token-filters/delimited-payload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md index b651550e58..00099d3788 100644 --- a/_analyzers/token-filters/delimited-payload.md +++ b/_analyzers/token-filters/delimited-payload.md @@ -58,7 +58,7 @@ PUT /my_index ## Generated tokens -Use the following request to examine the tokens generated using the created analyzer: +Use the following request to examine the tokens generated using the analyzer: ```json POST /my_index/_analyze From 5ab053124aea5c2b1e4313d8f26a45513132ec88 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Wed, 16 Oct 2024 19:36:02 +0100 Subject: [PATCH 4/8] updating parameter table Signed-off-by: Anton Rubin --- _analyzers/token-filters/delimited-payload.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md index 00099d3788..59aaf45969 100644 --- a/_analyzers/token-filters/delimited-payload.md +++ b/_analyzers/token-filters/delimited-payload.md @@ -7,7 +7,7 @@ nav_order: 90 # Delimited payload token filter -The `delimited_payload` token filter in OpenSearch is used to parse and attach payloads (extra metadata) to tokens during the analysis process. This is particularly useful when you want to associate additional data (like weights, scores, or other numeric values) with tokens for use in scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. +The `delimited_payload` token filter is used to parse and attach payloads (extra metadata) to tokens during the analysis process. This is particularly useful when you want to associate additional data (like weights, scores, or other numeric values) with tokens for use in scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. When analyzing text, `delimited_payload` token filter parses each token, extracts the payload, and attaches it to the token. This payload can later be used in queries to influence scoring, boosting, or other custom behaviors. @@ -15,17 +15,12 @@ The payload is not returned in the query response by default, additional configu ## Parameters -The `delimited_payload` token filter in OpenSearch has two _optional_ parameters: +The `delimited_payload` token filter has two parameters. -1. `encoding`: specifies the data type of the payload attached to the tokens. This determines how the payload data is stored and interpreted during analysis and querying. There are three valid values: - - - `identity`: The payload is treated as a sequence of characters. For example: `"user|admin"` where "admin" is stored as a string. - - - `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format. For example: `"car|2.5"` would store 2.5 as a floating-point number. - - - `int`: The payload is interpreted as a 32-bit integer. For example: `"priority|1"` would store 1 as an integer. - -2. `delimiter`: specifies character used to separate the token from its payload in the input text. By default, this is set to the pipe character (`|`). +Parameter | Required/Optional | Data type | Description +:--- | :--- | :--- | :--- +`encoding` | Optional | String | Specifies the data type of the payload attached to the tokens. This determines how the payload data is stored and interpreted during analysis and querying.
There are three valid values:

- `identity`: The payload is treated as a sequence of characters. For example: `"user|admin"` where "admin" is stored as a string.
- `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format. For example: `"car|2.5"` would store 2.5 as a floating-point number.
- `int`: The payload is interpreted as a 32-bit integer. For example: `"priority|1"` would store 1 as an integer. +`delimiter` | Optional | String | Specifies character used to separate the token from its payload in the input text. By default, this is set to the pipe character (`|`). ## Example without stored payload From 3dbcd301a809ae8e91d2c3fc76d335997911a967 Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Fri, 1 Nov 2024 11:02:16 +0000 Subject: [PATCH 5/8] Apply suggestions from code review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: AntonEliatra --- _analyzers/token-filters/delimited-payload.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md index 59aaf45969..8e62d2934c 100644 --- a/_analyzers/token-filters/delimited-payload.md +++ b/_analyzers/token-filters/delimited-payload.md @@ -7,11 +7,11 @@ nav_order: 90 # Delimited payload token filter -The `delimited_payload` token filter is used to parse and attach payloads (extra metadata) to tokens during the analysis process. This is particularly useful when you want to associate additional data (like weights, scores, or other numeric values) with tokens for use in scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. +The `delimited_payload` token filter is used to parse tokens containing payloads during the analysis process. For example, the string `red|1.5 fast|2.0 car|1.0` is parsed into tokens `red` (with a payload of `1.5`), `fast` (with a payload of `2.0), and `car` (with a payload of `1.0`). This is particularly useful when your tokens have additional associated data (like weights, scores, or other numeric values) that you can use for scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. and attach payloads (extra metadata) to tokens -When analyzing text, `delimited_payload` token filter parses each token, extracts the payload, and attaches it to the token. This payload can later be used in queries to influence scoring, boosting, or other custom behaviors. +When analyzing text, the `delimited_payload` token filter parses each token, extracts the payload, and attaches it to the token. This payload can later be used in queries to influence scoring, boosting, or other custom behaviors. -The payload is not returned in the query response by default, additional configuration is needed in order to be able to view the payload, see [Example with stored payload]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/#example-with-stored-payload) +Payloads are stored as Base64-encoded strings. By default, payloads are not returned in the query response along with the tokens. To return the payloads, you must configure additional parameters. For more information, see [Example with stored payload]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/#example-with-stored-payload). ## Parameters @@ -19,12 +19,12 @@ The `delimited_payload` token filter has two parameters. Parameter | Required/Optional | Data type | Description :--- | :--- | :--- | :--- -`encoding` | Optional | String | Specifies the data type of the payload attached to the tokens. This determines how the payload data is stored and interpreted during analysis and querying.
There are three valid values:

- `identity`: The payload is treated as a sequence of characters. For example: `"user|admin"` where "admin" is stored as a string.
- `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format. For example: `"car|2.5"` would store 2.5 as a floating-point number.
- `int`: The payload is interpreted as a 32-bit integer. For example: `"priority|1"` would store 1 as an integer. -`delimiter` | Optional | String | Specifies character used to separate the token from its payload in the input text. By default, this is set to the pipe character (`|`). +`encoding` | Optional | String | Specifies the data type of the payload attached to the tokens. This determines how the payload data is interpreted during analysis and querying.
Valid values are:

- `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format (for example, `2.5` in `car|2.5`).
- `identity`: The payload is interpreted as a sequence of characters (for example, in `user|admin`, `admin` is interpreted as a string).
- `int`: The payload is interpreted as a 32-bit integer (for example, `1` in `priority|1`).
Default is `float`. +`delimiter` | Optional | String | Specifies the character that separates the token from its payload in the input text. Default is the pipe character (`|`). ## Example without stored payload -The following example request creates a new index named `my_index` and configures an analyzer with `delimited_payload` filter: +The following example request creates a new index named `my_index` and configures an analyzer with a `delimited_payload` filter: ```json PUT /my_index @@ -96,7 +96,7 @@ The response contains the generated tokens: ## Example with stored payload -You can configure the payload to be returned in the response by creating an index that stores vectors, using `term_vector` set to `with_positions_payloads` or `with_positions_offsets_payloads` in mappings of the index, see following example: +To configure the payload to be returned in the response, by create an index that stores term vectors by setting `term_vector` to `with_positions_payloads` or `with_positions_offsets_payloads` in the index mappings. For example, the following index is configured to store term vectors: ```json PUT /visible_payloads @@ -131,7 +131,7 @@ PUT /visible_payloads ``` {% include copy-curl.html %} -You can index a document into this index using following request: +You can index a document into this index using the following request: ```json PUT /visible_payloads/_doc/1 @@ -153,7 +153,7 @@ GET /visible_payloads/_termvectors/1 ``` {% include copy-curl.html %} -The response contains the generated tokens which include payload: +The response contains the generated tokens, which include payloads: ```json { From 174e13995a479ea08631bbd13b0e9cccb4f2eca5 Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Fri, 1 Nov 2024 11:03:02 +0000 Subject: [PATCH 6/8] Update delimited-payload.md Signed-off-by: AntonEliatra --- _analyzers/token-filters/delimited-payload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md index 8e62d2934c..cd2c135fd2 100644 --- a/_analyzers/token-filters/delimited-payload.md +++ b/_analyzers/token-filters/delimited-payload.md @@ -96,7 +96,7 @@ The response contains the generated tokens: ## Example with stored payload -To configure the payload to be returned in the response, by create an index that stores term vectors by setting `term_vector` to `with_positions_payloads` or `with_positions_offsets_payloads` in the index mappings. For example, the following index is configured to store term vectors: +To configure the payload to be returned in the response, create an index that stores term vectors and set `term_vector` to `with_positions_payloads` or `with_positions_offsets_payloads` in the index mappings. For example, the following index is configured to store term vectors: ```json PUT /visible_payloads From 72f8acb8a9e42a60d3a3dd3b6626f3f6a6f2012f Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Tue, 5 Nov 2024 11:05:56 +0000 Subject: [PATCH 7/8] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: AntonEliatra --- _analyzers/token-filters/delimited-payload.md | 10 +++++----- _analyzers/token-filters/index.md | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md index cd2c135fd2..d325521c71 100644 --- a/_analyzers/token-filters/delimited-payload.md +++ b/_analyzers/token-filters/delimited-payload.md @@ -7,11 +7,11 @@ nav_order: 90 # Delimited payload token filter -The `delimited_payload` token filter is used to parse tokens containing payloads during the analysis process. For example, the string `red|1.5 fast|2.0 car|1.0` is parsed into tokens `red` (with a payload of `1.5`), `fast` (with a payload of `2.0), and `car` (with a payload of `1.0`). This is particularly useful when your tokens have additional associated data (like weights, scores, or other numeric values) that you can use for scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. and attach payloads (extra metadata) to tokens +The `delimited_payload` token filter is used to parse tokens containing payloads during the analysis process. For example, the string `red|1.5 fast|2.0 car|1.0` is parsed into the tokens `red` (with a payload of `1.5`), `fast` (with a payload of `2.0`), and `car` (with a payload of `1.0`). This is particularly useful when your tokens include additional associated data (like weights, scores, or other numeric values) that you can use for scoring or custom query logic. The filter can handle different types of payloads, including integers, floats, and strings, and attach payloads (extra metadata) to tokens. When analyzing text, the `delimited_payload` token filter parses each token, extracts the payload, and attaches it to the token. This payload can later be used in queries to influence scoring, boosting, or other custom behaviors. -Payloads are stored as Base64-encoded strings. By default, payloads are not returned in the query response along with the tokens. To return the payloads, you must configure additional parameters. For more information, see [Example with stored payload]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/#example-with-stored-payload). +Payloads are stored as Base64-encoded strings. By default, payloads are not returned in the query response along with the tokens. To return the payloads, you must configure additional parameters. For more information, see [Example with a stored payload]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/#example-with-stored-payload). ## Parameters @@ -22,7 +22,7 @@ Parameter | Required/Optional | Data type | Description `encoding` | Optional | String | Specifies the data type of the payload attached to the tokens. This determines how the payload data is interpreted during analysis and querying.
Valid values are:

- `float`: The payload is interpreted as a 32-bit floating-point number using IEEE 754 format (for example, `2.5` in `car|2.5`).
- `identity`: The payload is interpreted as a sequence of characters (for example, in `user|admin`, `admin` is interpreted as a string).
- `int`: The payload is interpreted as a 32-bit integer (for example, `1` in `priority|1`).
Default is `float`. `delimiter` | Optional | String | Specifies the character that separates the token from its payload in the input text. Default is the pipe character (`|`). -## Example without stored payload +## Example without a stored payload The following example request creates a new index named `my_index` and configures an analyzer with a `delimited_payload` filter: @@ -94,7 +94,7 @@ The response contains the generated tokens: } ``` -## Example with stored payload +## Example with a stored payload To configure the payload to be returned in the response, create an index that stores term vectors and set `term_vector` to `with_positions_payloads` or `with_positions_offsets_payloads` in the index mappings. For example, the following index is configured to store term vectors: @@ -143,7 +143,7 @@ PUT /visible_payloads/_doc/1 ## Generated tokens -Use the following request to examine the tokens generated using the created analyzer: +Use the following request to examine the tokens generated using the analyzer: ```json GET /visible_payloads/_termvectors/1 diff --git a/_analyzers/token-filters/index.md b/_analyzers/token-filters/index.md index a8a6bd3403..1451f1233d 100644 --- a/_analyzers/token-filters/index.md +++ b/_analyzers/token-filters/index.md @@ -23,7 +23,7 @@ Token filter | Underlying Lucene token filter| Description [`common_grams`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/common_gram/) | [CommonGramsFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/commongrams/CommonGramsFilter.html) | Generates bigrams for a list of frequently occurring terms. The output contains both single terms and bigrams. `conditional` | [ConditionalTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/miscellaneous/ConditionalTokenFilter.html) | Applies an ordered list of token filters to tokens that match the conditions provided in a script. `decimal_digit` | [DecimalDigitFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/core/DecimalDigitFilter.html) | Converts all digits in the Unicode decimal number general category to basic Latin digits (0--9). -[`delimited_payload`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/) | [DelimitedPayloadTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html) | Separates a token stream into tokens with corresponding payloads, based on a provided delimiter. A token consists of all characters before the delimiter, and a payload consists of all characters after the delimiter. For example, if the delimiter is `|`, then for the string `foo|bar`, `foo` is the token and `bar` is the payload. +[`delimited_payload`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/) | [DelimitedPayloadTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html) | Separates a token stream into tokens with corresponding payloads, based on a provided delimiter. A token consists of all characters preceding the delimiter, and a payload consists of all characters following the delimiter. For example, if the delimiter is `|`, then for the string `foo|bar`, `foo` is the token and `bar` is the payload. [`delimited_term_freq`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-term-frequency/) | [DelimitedTermFrequencyTokenFilter](https://lucene.apache.org/core/9_7_0/analysis/common/org/apache/lucene/analysis/miscellaneous/DelimitedTermFrequencyTokenFilter.html) | Separates a token stream into tokens with corresponding term frequencies, based on a provided delimiter. A token consists of all characters before the delimiter, and a term frequency is the integer after the delimiter. For example, if the delimiter is `|`, then for the string `foo|5`, `foo` is the token and `5` is the term frequency. `dictionary_decompounder` | [DictionaryCompoundWordTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/compound/DictionaryCompoundWordTokenFilter.html) | Decomposes compound words found in many Germanic languages. `edge_ngram` | [EdgeNGramTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/ngram/EdgeNGramTokenFilter.html) | Tokenizes the given token into edge n-grams (n-grams that start at the beginning of the token) of lengths between `min_gram` and `max_gram`. Optionally, keeps the original token. From 705f346296a549e309c0830f4f4a4115be99f729 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Tue, 5 Nov 2024 17:45:26 +0000 Subject: [PATCH 8/8] fixing broken link Signed-off-by: Anton Rubin --- _analyzers/token-filters/delimited-payload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_analyzers/token-filters/delimited-payload.md b/_analyzers/token-filters/delimited-payload.md index d325521c71..f17bb1b1ce 100644 --- a/_analyzers/token-filters/delimited-payload.md +++ b/_analyzers/token-filters/delimited-payload.md @@ -11,7 +11,7 @@ The `delimited_payload` token filter is used to parse tokens containing payloads When analyzing text, the `delimited_payload` token filter parses each token, extracts the payload, and attaches it to the token. This payload can later be used in queries to influence scoring, boosting, or other custom behaviors. -Payloads are stored as Base64-encoded strings. By default, payloads are not returned in the query response along with the tokens. To return the payloads, you must configure additional parameters. For more information, see [Example with a stored payload]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/#example-with-stored-payload). +Payloads are stored as Base64-encoded strings. By default, payloads are not returned in the query response along with the tokens. To return the payloads, you must configure additional parameters. For more information, see [Example with a stored payload]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-payload/#example-without-a-stored-payload). ## Parameters