diff --git a/_dashboards/visualize/vega.md b/_dashboards/visualize/vega.md
index 7764d583a6..3a9f6aad4f 100644
--- a/_dashboards/visualize/vega.md
+++ b/_dashboards/visualize/vega.md
@@ -1,192 +1,137 @@
---
layout: default
-title: Using Vega
+title: Vega
parent: Building data visualizations
-nav_order: 45
+nav_order: 50
---
-# Using Vega
+# Vega
-[Vega](https://vega.github.io/vega/) and [Vega-Lite](https://vega.github.io/vega-lite/) are open-source, declarative language visualization tools that you can use to create custom data visualizations with your OpenSearch data and [Vega Data](https://vega.github.io/vega/docs/data/). These tools are ideal for advanced users comfortable with writing OpenSearch queries directly. Enable the `vis_type_vega` plugin in your `opensearch_dashboards.yml` file to write your [Vega specifications](https://vega.github.io/vega/docs/specification/) in either JSON or [HJSON](https://hjson.github.io/) format or to specify one or more OpenSearch queries within your Vega specification. By default, the plugin is set to `true`. The configuration is shown in the following example. For configuration details, refer to the `vis_type_vega` [README](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/vis_type_vega/README.md).
+[Vega](https://vega.github.io/vega/) and [Vega-Lite](https://vega.github.io/vega-lite/) are open-source, declarative language visualization tools that you can use to create custom data visualizations with your OpenSearch data and [Vega data](https://vega.github.io/vega/docs/data/). These tools are ideal for advanced users comfortable with writing OpenSearch queries directly. Enable the `vis_type_vega` plugin in your `opensearch_dashboards.yml` file to write your [Vega specifications](https://vega.github.io/vega/docs/specification/) in either JSON or [HJSON](https://hjson.github.io/) format or to specify one or more OpenSearch queries in your Vega specification. By default, the plugin is set to `true`.
+
+## Creating Vega visualizations from multiple data sources
+Introduced 2.13
+{: .label .label-purple }
+
+Before proceeding, ensure that the following configuration settings are enabled in the `config/opensearch_dasboards.yaml` file. For configuration details, refer to the `vis_type_vega` [README](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/vis_type_vega/README.md).
```
+data_source.enabled: true
vis_type_vega.enabled: true
```
-The following image shows a custom Vega map created in OpenSearch.
+After you have configured [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/) in OpenSearch Dashboards, you can use Vega to query those data sources. The following GIF shows the process of creating Vega visualizations in OpenSearch Dashboards.
-
+
-## Querying from multiple data sources
+### Step 1: Set up and connect data sources
-If you have configured [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/) in OpenSearch Dashboards, you can use Vega to query those data sources. Within your Vega specification, add the `data_source_name` field under the `url` property to target a specific data source by name. By default, queries use data from the local cluster. You can assign individual `data_source_name` values to each OpenSearch query within your Vega specification. This allows you to query multiple indexes across different data sources in a single visualization.
+Open OpenSearch Dashboards and follow these steps:
-The following is an example Vega specification with `Demo US Cluster` as the specified `data_source_name`:
+1. Select **Dashboards Management** from the menu on the left.
+2. Select **Data sources** and then select the **Create data source** button.
+3. On the **Create data source** page, enter the connection details and endpoint URL, as shown in the following GIF.
+4. On the **Home page**, select **Add sample data**. Under **Data source**, select your newly created data source, and then select the **Add data button** for the **Sample web logs** dataset.
-```
+The following GIF shows the steps required for setting up and connecting a data source.
+
+
+
+### Step 2: Create the visualization
+
+1. From the menu on the left, select **Visualize**.
+2. On the **Visualizations** page, select **Create Visualization** and then select **Vega** in the pop-up window.
+
+### Step 3: Add the Vega specification
+
+By default, queries use data from the local cluster. You can assign individual `data_source_name` values to each OpenSearch query in your Vega specification. This allows you to query multiple indexes across different data sources in a single visualization.
+
+1. Verify that the data source you created is specified under `data_source_name`. Alternatively, in your Vega specification, add the `data_source_name` field under the `url` property to target a specific data source by name.
+2. Copy the following Vega specification and then select the **Update** button in the lower-right corner. The visualization should appear.
+
+```json
{
- $schema: https://vega.github.io/schema/vega/v5.json
- config: {
- kibana: {type: "map", latitude: 25, longitude: -70, zoom: 3}
- }
- data: [
- {
- name: table
- url: {
- index: opensearch_dashboards_sample_data_flights
- // This OpenSearchQuery will query from the Demo US Cluster datasource
- data_source_name: Demo US Cluster
- %context%: true
- // Uncomment to enable time filtering
- // %timefield%: timestamp
- body: {
- size: 0
- aggs: {
- origins: {
- terms: {field: "OriginAirportID", size: 10000}
- aggs: {
- originLocation: {
- top_hits: {
- size: 1
- _source: {
- includes: ["OriginLocation", "Origin"]
- }
- }
- }
- distinations: {
- terms: {field: "DestAirportID", size: 10000}
- aggs: {
- destLocation: {
- top_hits: {
- size: 1
- _source: {
- includes: ["DestLocation"]
- }
- }
- }
- }
+ $schema: https://vega.github.io/schema/vega-lite/v5.json
+ data: {
+ url: {
+ %context%: true
+ %timefield%: @timestamp
+ index: opensearch_dashboards_sample_data_logs
+ data_source_name: YOUR_DATA_SOURCE_TITLE
+ body: {
+ aggs: {
+ 1: {
+ date_histogram: {
+ field: @timestamp
+ fixed_interval: 3h
+ time_zone: America/Los_Angeles
+ min_doc_count: 1
+ }
+ aggs: {
+ 2: {
+ avg: {
+ field: bytes
}
}
}
}
}
+ size: 0
}
- format: {property: "aggregations.origins.buckets"}
- transform: [
- {
- type: geopoint
- projection: projection
- fields: [
- originLocation.hits.hits[0]._source.OriginLocation.lon
- originLocation.hits.hits[0]._source.OriginLocation.lat
- ]
- }
- ]
}
- {
- name: selectedDatum
- on: [
- {trigger: "!selected", remove: true}
- {trigger: "selected", insert: "selected"}
- ]
+ format: {
+ property: aggregations.1.buckets
}
- ]
- signals: [
+ }
+ transform: [
{
- name: selected
- value: null
- on: [
- {events: "@airport:mouseover", update: "datum"}
- {events: "@airport:mouseout", update: "null"}
- ]
+ calculate: datum.key
+ as: timestamp
}
- ]
- scales: [
{
- name: airportSize
- type: linear
- domain: {data: "table", field: "doc_count"}
- range: [
- {signal: "zoom*zoom*0.2+1"}
- {signal: "zoom*zoom*10+1"}
- ]
+ calculate: datum[2].value
+ as: bytes
}
]
- marks: [
+ layer: [
{
- type: group
- from: {
- facet: {
- name: facetedDatum
- data: selectedDatum
- field: distinations.buckets
- }
+ mark: {
+ type: line
}
- data: [
- {
- name: facetDatumElems
- source: facetedDatum
- transform: [
- {
- type: geopoint
- projection: projection
- fields: [
- destLocation.hits.hits[0]._source.DestLocation.lon
- destLocation.hits.hits[0]._source.DestLocation.lat
- ]
- }
- {type: "formula", expr: "{x:parent.x, y:parent.y}", as: "source"}
- {type: "formula", expr: "{x:datum.x, y:datum.y}", as: "target"}
- {type: "linkpath", shape: "diagonal"}
- ]
- }
- ]
- scales: [
- {
- name: lineThickness
- type: log
- clamp: true
- range: [1, 8]
- }
- {
- name: lineOpacity
- type: log
- clamp: true
- range: [0.2, 0.8]
- }
- ]
- marks: [
- {
- from: {data: "facetDatumElems"}
- type: path
- interactive: false
- encode: {
- update: {
- path: {field: "path"}
- stroke: {value: "black"}
- strokeWidth: {scale: "lineThickness", field: "doc_count"}
- strokeOpacity: {scale: "lineOpacity", field: "doc_count"}
- }
- }
- }
- ]
}
{
- name: airport
- type: symbol
- from: {data: "table"}
- encode: {
- update: {
- size: {scale: "airportSize", field: "doc_count"}
- xc: {signal: "datum.x"}
- yc: {signal: "datum.y"}
- tooltip: {
- signal: "{title: datum.originLocation.hits.hits[0]._source.Origin + ' (' + datum.key + ')', connnections: length(datum.distinations.buckets), flights: datum.doc_count}"
- }
- }
+ mark: {
+ type: circle
+ tooltip: true
}
}
]
+ encoding: {
+ x: {
+ field: timestamp
+ type: temporal
+ axis: {
+ title: @timestamp
+ }
+ }
+ y: {
+ field: bytes
+ type: quantitative
+ axis: {
+ title: Average bytes
+ }
+ }
+ color: {
+ datum: Average bytes
+ type: nominal
+ }
+ }
}
```
{% include copy-curl.html %}
+
+## Additional resources
+
+The following resources provide additional information about Vega visualizations in OpenSearch Dashboards:
+
+- [Improving ease of use in OpenSearch Dashboards with Vega visualizations](https://opensearch.org/blog/Improving-Dashboards-usability-with-Vega/)
diff --git a/_dashboards/visualize/visbuilder.md b/_dashboards/visualize/visbuilder.md
index de4dfb1666..2b6818a00e 100644
--- a/_dashboards/visualize/visbuilder.md
+++ b/_dashboards/visualize/visbuilder.md
@@ -1,13 +1,13 @@
---
layout: default
-title: Using VisBuilder
+title: VisBuilder
parent: Building data visualizations
nav_order: 100
redirect_from:
- /dashboards/drag-drop-wizard/
---
-# Using VisBuilder
+# VisBuilder
You can use the VisBuilder visualization type in OpenSearch Dashboards to create data visualizations by using a drag-and-drop gesture. With VisBuilder you have:
@@ -19,7 +19,7 @@ You can use the VisBuilder visualization type in OpenSearch Dashboards to create
## Try VisBuilder in the OpenSearch Dashboards playground
-If you'd like to try out VisBuilder without installing OpenSearch locally, you can do so in the [Dashboards playground](https://playground.opensearch.org/app/vis-builder#/).
+You can try VisBuilder without installing OpenSearch locally by using [OpenSearch Dashboards Playground](https://playground.opensearch.org/app/vis-builder#/). VisBuilder is enabled by default.
## Try VisBuilder locally
@@ -27,7 +27,7 @@ Follow these steps to create a new visualization using VisBuilder in your enviro
1. Open Dashboards:
- If you're not running the Security plugin, go to http://localhost:5601.
- - If you're running the Security plugin, go to https://localhost:5601 and log in with your username and password (default is admin/admin).
+ - If you're running the Security plugin, go to https://localhost:5601 and log in with your username and password (default is `admin/admin`).
1. From the top menu, select **Visualize > Create visualization > VisBuilder**.
@@ -37,4 +37,4 @@ Follow these steps to create a new visualization using VisBuilder in your enviro
Here’s an example visualization. Your visualization will look different depending on your data and the fields you select.
-
\ No newline at end of file
+
diff --git a/_data-prepper/common-use-cases/metrics-logs.md b/_data-prepper/common-use-cases/metrics-logs.md
new file mode 100644
index 0000000000..3fda8597c7
--- /dev/null
+++ b/_data-prepper/common-use-cases/metrics-logs.md
@@ -0,0 +1,70 @@
+---
+layout: default
+title: Deriving metrics from logs
+parent: Common use cases
+nav_order: 15
+---
+
+# Deriving metrics from logs
+
+You can use Data Prepper to derive metrics from logs.
+
+The following example pipeline receives incoming logs using the [`http` source plugin]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/http-source) and the [`grok` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/grok/). It then uses the [`aggregate` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/aggregate/) to extract the metric bytes aggregated during a 30-second window and derives histograms from the results.
+
+This pipeline writes data to two different OpenSearch indexes:
+
+- `logs`: This index stores the original, un-aggregated log events after being processed by the `grok` processor.
+- `histogram_metrics`: This index stores the derived histogram metrics extracted from the log events using the `aggregate` processor.
+
+The pipeline contains two sub-pipelines:
+
+- `apache-log-pipeline-with-metrics`: Receives logs through an HTTP client like FluentBit, using `grok` to extract important values from the logs by matching the value in the log key against the [Apache Common Log Format](https://httpd.apache.org/docs/2.4/logs.html#accesslog). It then forwards the grokked logs to two destinations:
+
+ - An OpenSearch index named `logs` to store the original log events.
+ - The `log-to-metrics-pipeline` for further aggregation and metric derivation.
+
+- `log-to-metrics-pipeline`: Receives the grokked logs from the `apache-log-pipeline-with-metrics` pipeline, aggregates the logs, and derives histogram metrics of bytes based on the values in the `clientip` and `request` keys. Finally, it sends the derived histogram metrics to an OpenSearch index named `histogram_metrics`.
+
+#### Example pipeline
+
+```json
+apache-log-pipeline-with-metrics:
+ source:
+ http:
+ # Provide the path for ingestion. ${pipelineName} will be replaced with pipeline name configured for this pipeline.
+ # In this case it would be "/apache-log-pipeline-with-metrics/logs". This will be the FluentBit output URI value.
+ path: "/${pipelineName}/logs"
+ processor:
+ - grok:
+ match:
+ log: [ "%{COMMONAPACHELOG_DATATYPED}" ]
+ sink:
+ - opensearch:
+ ...
+ index: "logs"
+ - pipeline:
+ name: "log-to-metrics-pipeline"
+
+log-to-metrics-pipeline:
+ source:
+ pipeline:
+ name: "apache-log-pipeline-with-metrics"
+ processor:
+ - aggregate:
+ # Specify the required identification keys
+ identification_keys: ["clientip", "request"]
+ action:
+ histogram:
+ # Specify the appropriate values for each of the following fields
+ key: "bytes"
+ record_minmax: true
+ units: "bytes"
+ buckets: [0, 25000000, 50000000, 75000000, 100000000]
+ # Pick the required aggregation period
+ group_duration: "30s"
+ sink:
+ - opensearch:
+ ...
+ index: "histogram_metrics"
+```
+{% include copy-curl.html %}
diff --git a/_data-prepper/common-use-cases/s3-logs.md b/_data-prepper/common-use-cases/s3-logs.md
index 7986a7eef8..8d5a9ce967 100644
--- a/_data-prepper/common-use-cases/s3-logs.md
+++ b/_data-prepper/common-use-cases/s3-logs.md
@@ -9,7 +9,6 @@ nav_order: 40
Data Prepper allows you to load logs from [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3), including traditional logs, JSON documents, and CSV logs.
-
## Architecture
Data Prepper can read objects from S3 buckets using an [Amazon Simple Queue Service (SQS)](https://aws.amazon.com/sqs/) (Amazon SQS) queue and [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html).
@@ -20,7 +19,7 @@ The following diagram shows the overall architecture of the components involved.
{: .img-fluid}
-The flow of data is as follows.
+The component data flow is as follows:
1. A system produces logs into the S3 bucket.
2. S3 creates an S3 event notification in the SQS queue.
@@ -28,7 +27,6 @@ The flow of data is as follows.
4. Data Prepper downloads the content from the S3 object.
5. Data Prepper sends a document to OpenSearch for the content in the S3 object.
-
## Pipeline overview
Data Prepper supports reading data from S3 using the [`s3` source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/).
@@ -44,7 +42,6 @@ Before Data Prepper can read log data from S3, you need the following prerequisi
- An S3 bucket.
- A log producer that writes logs to S3. The exact log producer will vary depending on your specific use case, but could include writing logs to S3 or a service such as Amazon CloudWatch.
-
## Getting started
Use the following steps to begin loading logs from S3 with Data Prepper.
@@ -57,8 +54,7 @@ Use the following steps to begin loading logs from S3 with Data Prepper.
### Setting permissions for Data Prepper
-To view S3 logs, Data Prepper needs access to Amazon SQS and S3.
-Use the following example to set up permissions:
+To view S3 logs, Data Prepper needs access to Amazon SQS and S3. Use the following example to set up permissions:
```json
{
@@ -88,12 +84,13 @@ Use the following example to set up permissions:
]
}
```
+{% include copy-curl.html %}
If your S3 objects or SQS queues do not use KMS, you can remove the `kms:Decrypt` permission.
### SQS dead-letter queue
-The are two options for how to handle errors resulting from processing S3 objects.
+The following two options can be used to handle S3 object processing errors:
- Use an SQS dead-letter queue (DLQ) to track the failure. This is the recommended approach.
- Delete the message from SQS. You must manually find the S3 object and correct the error.
@@ -104,8 +101,8 @@ The following diagram shows the system architecture when using SQS with DLQ.
To use an SQS dead-letter queue, perform the following steps:
-1. Create a new SQS standard queue to act as your DLQ.
-2. Configure your SQS's redrive policy [to use your DLQ](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html). Consider using a low value such as 2 or 3 for the "Maximum Receives" setting.
+1. Create a new SQS standard queue to act as the DLQ.
+2. Configure your SQS re-drive policy [to use DLQ](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html). Consider using a low value such as 2 or 3 for the **Maximum Receives** setting.
3. Configure the Data Prepper `s3` source to use `retain_messages` for `on_error`. This is the default behavior.
## Pipeline design
@@ -125,6 +122,7 @@ s3-log-pipeline:
queue_url: "arn:aws:sqs:If `default_values` is `{"key1": "abc"}`, `key1=value1` will parse into `{"key1": "value1"}`.
If `include_keys` is `["key1"]` and `default_values` is `{"key2": "value2"}`, `key1=value1&key2=abc` will parse into `{"key1": "value1", "key2": "value2"}`. | -| transform_key | When to lowercase, uppercase, or capitalize keys. | If `transform_key` is `lowercase`, `{"Key1=value1"}` will parse into `{"key1": "value1"}`.
If `transform_key` is `uppercase`, `{"key1=value1"}` will parse into `{"KEY1": "value1"}`.
If `transform_key` is `capitalize`, `{"key1=value1"}` will parse into `{"Key1": "value1"}`. | -| whitespace | Specifies whether to be lenient or strict with the acceptance of unnecessary white space surrounding the configured value-split sequence. Default is `lenient`. | If `whitespace` is `"lenient"`, `{"key1 = value1"}` will parse into `{"key1 ": " value1"}`. If `whitespace` is `"strict"`, `{"key1 = value1"}` will parse into `{"key1": "value1"}`. | -| skip_duplicate_values | A Boolean option for removing duplicate key-value pairs. When set to `true`, only one unique key-value pair will be preserved. Default is `false`. | If `skip_duplicate_values` is `false`, `{"key1=value1&key1=value1"}` will parse into `{"key1": ["value1", "value1"]}`. If `skip_duplicate_values` is `true`, `{"key1=value1&key1=value1"}` will parse into `{"key1": "value1"}`. | -| remove_brackets | Specifies whether to treat square brackets, angle brackets, and parentheses as value "wrappers" that should be removed from the value. Default is `false`. | If `remove_brackets` is `true`, `{"key1=(value1)"}` will parse into `{"key1": value1}`. If `remove_brackets` is `false`, `{"key1=(value1)"}` will parse into `{"key1": "(value1)"}`. | -| recursive | Specifies whether to recursively obtain additional key-value pairs from values. The extra key-value pairs will be stored as sub-keys of the root key. Default is `false`. The levels of recursive parsing must be defined by different brackets for each level: `[]`, `()`, and `<>`, in this order. Any other configurations specified will only be applied to the outmost keys.
When `recursive` is `true`:
`remove_brackets` cannot also be `true`;
`skip_duplicate_values` will always be `true`;
`whitespace` will always be `"strict"`. | If `recursive` is true, `{"item1=[item1-subitem1=item1-subitem1-value&item1-subitem2=(item1-subitem2-subitem2A=item1-subitem2-subitem2A-value&item1-subitem2-subitem2B=item1-subitem2-subitem2B-value)]&item2=item2-value"}` will parse into `{"item1": {"item1-subitem1": "item1-subitem1-value", "item1-subitem2" {"item1-subitem2-subitem2A": "item1-subitem2-subitem2A-value", "item1-subitem2-subitem2B": "item1-subitem2-subitem2B-value"}}}`. | -| overwrite_if_destination_exists | Specifies whether to overwrite existing fields if there are key conflicts when writing parsed fields to the event. Default is `true`. | If `overwrite_if_destination_exists` is `true` and destination is `null`, `{"key1": "old_value", "message": "key1=new_value"}` will parse into `{"key1": "new_value", "message": "key1=new_value"}`. | -| tags_on_failure | When a `kv` operation causes a runtime exception within the processor, the operation is safely stopped without crashing the processor, and the event is tagged with the provided tags. | If `tags_on_failure` is set to `["keyvalueprocessor_failure"]`, `{"tags": ["keyvalueprocessor_failure"]}` will be added to the event's metadata in the event of a runtime exception. | -| value_grouping | Specifies whether to group values using predefined value grouping delimiters: `{...}`, `[...]', `<...>`, `(...)`, `"..."`, `'...'`, `http://... (space)`, and `https:// (space)`. If this flag is enabled, then the content between the delimiters is considered to be one entity and is not parsed for key-value pairs. Default is `false`. If `value_grouping` is `true`, then `{"key1=[a=b,c=d]&key2=value2"}` parses to `{"key1": "[a=b,c=d]", "key2": "value2"}`. | -| drop_keys_with_no_value | Specifies whether keys should be dropped if they have a null value. Default is `false`. If `drop_keys_with_no_value` is set to `true`, then `{"key1=value1&key2"}` parses to `{"key1": "value1"}`. | -| strict_grouping | Specifies whether strict grouping should be enabled when the `value_grouping` or `string_literal_character` options are used. Default is `false`. | When enabled, groups with unmatched end characters yield errors. The event is ignored after the errors are logged. | -| string_literal_character | Can be set to either a single quotation mark (`'`) or a double quotation mark (`"`). Default is `null`. | When this option is used, any text contained within the specified quotation mark character will be ignored and excluded from key-value parsing. For example, `text1 "key1=value1" text2 key2=value2` would parse to `{"key2": "value2"}`. | -| key_value_when | Allows you to specify a [conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"`, that will be evaluated to determine whether the processor should be applied to the event. | +Option | Description | Example +:--- | :--- | :--- +`source` | The message field to be parsed. Optional. Default value is `message`. | If `source` is `"message1"`, `{"message1": {"key1=value1"}, "message2": {"key2=value2"}}` parses into `{"message1": {"key1=value1"}, "message2": {"key2=value2"}, "parsed_message": {"key1": "value1"}}`. +destination | The destination field for the parsed source. The parsed source overwrites the preexisting data for that key. Optional. If `destination` is set to `null`, the parsed fields will be written to the root of the event. Default value is `parsed_message`. | If `destination` is `"parsed_data"`, `{"message": {"key1=value1"}}` parses into `{"message": {"key1=value1"}, "parsed_data": {"key1": "value1"}}`. +`field_delimiter_regex` | A regular expression specifying the delimiter that separates key-value pairs. Special regular expression characters such as `[` and `]` must be escaped with `\\`. Cannot be defined at the same time as `field_split_characters`. Optional. If this option is not defined, `field_split_characters` is used. | If `field_delimiter_regex` is `"&\\{2\\}"`, `{"key1=value1&&key2=value2"}` parses into `{"key1": "value1", "key2": "value2"}`. +`field_split_characters` | A string of characters specifying the delimeter that separates key-value pairs. Special regular expression characters such as `[` and `]` must be escaped with `\\`. Cannot be defined at the same time as `field_delimiter_regex`. Optional. Default value is `&`. | If `field_split_characters` is `"&&"`, `{"key1=value1&&key2=value2"}` parses into `{"key1": "value1", "key2": "value2"}`. +`key_value_delimiter_regex` | A regular expression specifying the delimiter that separates the key and value within a key-value pair. Special regular expression characters such as `[` and `]` must be escaped with `\\`. This option cannot be defined at the same time as `value_split_characters`. Optional. If this option is not defined, `value_split_characters` is used. | If `key_value_delimiter_regex` is `"=\\{2\\}"`, `{"key1==value1"}` parses into `{"key1": "value1"}`. +`value_split_characters` | A string of characters specifying the delimiter that separates the key and value within a key-value pair. Special regular expression characters such as `[` and `]` must be escaped with `\\`. Cannot be defined at the same time as `key_value_delimiter_regex`. Optional. Default value is `=`. | If `value_split_characters` is `"=="`, `{"key1==value1"}` parses into `{"key1": "value1"}`. +`non_match_value` | When a key-value pair cannot be successfully split, the key-value pair is placed in the `key` field, and the specified value is placed in the `value` field. Optional. Default value is `null`. | `key1value1&key2=value2` parses into `{"key1value1": null, "key2": "value2"}`. | +`prefix` | A prefix to append before all keys. Optional. Default value is an empty string. | If `prefix` is `"custom"`, `{"key1=value1"}` parses into `{"customkey1": "value1"}`. +`delete_key_regex` | A regular expression specifying the characters to delete from the key. Special regular expression characters such as `[` and `]` must be escaped with `\\`. Cannot be an empty string. Optional. No default value. | If `delete_key_regex` is `"\s"`, `{"key1 =value1"}` parses into `{"key1": "value1"}`. +`delete_value_regex` | A regular expression specifying the characters to delete from the value. Special regular expression characters such as `[` and `]` must be escaped with `\\`. Cannot be an empty string. Optional. No default value. | If `delete_value_regex` is `"\s"`, `{"key1=value1 "}` parses into `{"key1": "value1"}`. +`include_keys` | An array specifying the keys that should be added for parsing. By default, all keys will be added. | If `include_keys` is `["key2"]`,`key1=value1&key2=value2` will parse into `{"key2": "value2"}`. +`exclude_keys` | An array specifying the parsed keys that should not be added to the event. By default, no keys will be excluded. | If `exclude_keys` is `["key2"]`, `key1=value1&key2=value2` will parse into `{"key1": "value1"}`. +`default_values` | A map specifying the default keys and their values that should be added to the event in case these keys do not exist in the source field being parsed. If the default key already exists in the message, the value is not changed. The `include_keys` filter will be applied to the message before `default_values`. | If `default_values` is `{"defaultkey": "defaultvalue"}`, `key1=value1` will parse into `{"key1": "value1", "defaultkey": "defaultvalue"}`.
If `default_values` is `{"key1": "abc"}`, `key1=value1` will parse into `{"key1": "value1"}`.
If `include_keys` is `["key1"]` and `default_values` is `{"key2": "value2"}`, `key1=value1&key2=abc` will parse into `{"key1": "value1", "key2": "value2"}`. +`transform_key` | When to lowercase, uppercase, or capitalize keys. | If `transform_key` is `lowercase`, `{"Key1=value1"}` will parse into `{"key1": "value1"}`.
If `transform_key` is `uppercase`, `{"key1=value1"}` will parse into `{"KEY1": "value1"}`.
If `transform_key` is `capitalize`, `{"key1=value1"}` will parse into `{"Key1": "value1"}`. +`whitespace` | Specifies whether to be lenient or strict with the acceptance of unnecessary white space surrounding the configured value-split sequence. Default is `lenient`. | If `whitespace` is `"lenient"`, `{"key1 = value1"}` will parse into `{"key1 ": " value1"}`. If `whitespace` is `"strict"`, `{"key1 = value1"}` will parse into `{"key1": "value1"}`. +`skip_duplicate_values` | A Boolean option for removing duplicate key-value pairs. When set to `true`, only one unique key-value pair will be preserved. Default is `false`. | If `skip_duplicate_values` is `false`, `{"key1=value1&key1=value1"}` will parse into `{"key1": ["value1", "value1"]}`. If `skip_duplicate_values` is `true`, `{"key1=value1&key1=value1"}` will parse into `{"key1": "value1"}`. +`remove_brackets` | Specifies whether to treat square brackets, angle brackets, and parentheses as value "wrappers" that should be removed from the value. Default is `false`. | If `remove_brackets` is `true`, `{"key1=(value1)"}` will parse into `{"key1": value1}`. If `remove_brackets` is `false`, `{"key1=(value1)"}` will parse into `{"key1": "(value1)"}`. +`recursive` | Specifies whether to recursively obtain additional key-value pairs from values. The extra key-value pairs will be stored as sub-keys of the root key. Default is `false`. The levels of recursive parsing must be defined by different brackets for each level: `[]`, `()`, and `<>`, in this order. Any other configurations specified will only be applied to the outmost keys.
When `recursive` is `true`:
`remove_brackets` cannot also be `true`;
`skip_duplicate_values` will always be `true`;
`whitespace` will always be `"strict"`. | If `recursive` is true, `{"item1=[item1-subitem1=item1-subitem1-value&item1-subitem2=(item1-subitem2-subitem2A=item1-subitem2-subitem2A-value&item1-subitem2-subitem2B=item1-subitem2-subitem2B-value)]&item2=item2-value"}` will parse into `{"item1": {"item1-subitem1": "item1-subitem1-value", "item1-subitem2" {"item1-subitem2-subitem2A": "item1-subitem2-subitem2A-value", "item1-subitem2-subitem2B": "item1-subitem2-subitem2B-value"}}}`. +`overwrite_if_destination_exists` | Specifies whether to overwrite existing fields if there are key conflicts when writing parsed fields to the event. Default is `true`. | If `overwrite_if_destination_exists` is `true` and destination is `null`, `{"key1": "old_value", "message": "key1=new_value"}` will parse into `{"key1": "new_value", "message": "key1=new_value"}`. +`tags_on_failure` | When a `kv` operation causes a runtime exception within the processor, the operation is safely stopped without crashing the processor, and the event is tagged with the provided tags. | If `tags_on_failure` is set to `["keyvalueprocessor_failure"]`, `{"tags": ["keyvalueprocessor_failure"]}` will be added to the event's metadata in the event of a runtime exception. +`value_grouping` | Specifies whether to group values using predefined value grouping delimiters: `{...}`, `[...]', `<...>`, `(...)`, `"..."`, `'...'`, `http://... (space)`, and `https:// (space)`. If this flag is enabled, then the content between the delimiters is considered to be one entity and is not parsed for key-value pairs. Default is `false`. If `value_grouping` is `true`, then `{"key1=[a=b,c=d]&key2=value2"}` parses to `{"key1": "[a=b,c=d]", "key2": "value2"}`. +`drop_keys_with_no_value` | Specifies whether keys should be dropped if they have a null value. Default is `false`. If `drop_keys_with_no_value` is set to `true`, then `{"key1=value1&key2"}` parses to `{"key1": "value1"}`. +`strict_grouping` | Specifies whether strict grouping should be enabled when the `value_grouping` or `string_literal_character` options are used. Default is `false`. | When enabled, groups with unmatched end characters yield errors. The event is ignored after the errors are logged. +`string_literal_character` | Can be set to either a single quotation mark (`'`) or a double quotation mark (`"`). Default is `null`. | When this option is used, any text contained within the specified quotation mark character will be ignored and excluded from key-value parsing. For example, `text1 "key1=value1" text2 key2=value2` would parse to `{"key2": "value2"}`. +`key_value_when` | Allows you to specify a [conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"`, that will be evaluated to determine whether the processor should be applied to the event. - - diff --git a/_data-prepper/pipelines/configuration/processors/write_json.md b/_data-prepper/pipelines/configuration/processors/write_json.md index 9e94176010..8f1e6851da 100644 --- a/_data-prepper/pipelines/configuration/processors/write_json.md +++ b/_data-prepper/pipelines/configuration/processors/write_json.md @@ -11,8 +11,8 @@ nav_order: 56 The `write_json` processor converts an object in an event into a JSON string. You can customize the processor to choose the source and target field names. -| Option | Description | Example | -| :--- | :--- | :--- | -| source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is set to `"message"` and the input is `{"message": {"key1":"value1", "key2":{"key3":"value3"}}`, then the `write_json` processor generates `{"message": "{\"key1\":\"value`\", \"key2\":"{\"key3\":\"value3\"}"}"`. -| target | An optional field that specifies the name of the field in which the resulting JSON string should be stored. If `target` is not specified, then the `source` field is used. +Option | Description | Example +:--- | :--- | :--- +source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is set to `"message"` and the input is `{"message": {"key1":"value1", "key2":{"key3":"value3"}}}`, then the `write_json` processor outputs the event as `"{\"key1\":\"value1\",\"key2\":{\"key3\":\"value3\"}}"`. +target | An optional field that specifies the name of the field in which the resulting JSON string should be stored. If `target` is not specified, then the `source` field is used. | `key1` diff --git a/_field-types/supported-field-types/index.md b/_field-types/supported-field-types/index.md index dbff7c30f2..7c7b7375f9 100644 --- a/_field-types/supported-field-types/index.md +++ b/_field-types/supported-field-types/index.md @@ -23,7 +23,7 @@ Boolean | [`boolean`]({{site.url}}{{site.baseurl}}/field-types/supported-field-t IP | [`ip`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/ip/): An IP address in IPv4 or IPv6 format. [Range]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/range/) | A range of values (`integer_range`, `long_range`, `double_range`, `float_range`, `date_range`, `ip_range`). [Object]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/object-fields/)| [`object`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/object/): A JSON object.
[`nested`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/nested/): Used when objects in an array need to be indexed independently as separate documents.
[`flat_object`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/flat-object/): A JSON object treated as a string.
[`join`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/join/): Establishes a parent-child relationship between documents in the same index. -[String]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/string/)|[`keyword`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/keyword/): Contains a string that is not analyzed.
[`text`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/text/): Contains a string that is analyzed.
[`match_only_text`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/match-only-text/): A space-optimized version of a `text` field.
[`token_count`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/token-count/): Stores the number of analyzed tokens in a string.
[`wildcard`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/token-count/): A variation of `keyword` with efficient substring and regular expression matching. +[String]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/string/)|[`keyword`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/keyword/): Contains a string that is not analyzed.
[`text`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/text/): Contains a string that is analyzed.
[`match_only_text`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/match-only-text/): A space-optimized version of a `text` field.
[`token_count`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/token-count/): Stores the number of analyzed tokens in a string.
[`wildcard`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/wildcard/): A variation of `keyword` with efficient substring and regular expression matching. [Autocomplete]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/autocomplete/) |[`completion`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/completion/): Provides autocomplete functionality through a completion suggester.
[`search_as_you_type`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/search-as-you-type/): Provides search-as-you-type functionality using both prefix and infix completion. [Geographic]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/geographic/)| [`geo_point`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/geo-point/): A geographic point.
[`geo_shape`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/geo-shape/): A geographic shape. [Rank]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/rank/) | Boosts or decreases the relevance score of documents (`rank_feature`, `rank_features`). diff --git a/_getting-started/intro.md b/_getting-started/intro.md index 272d8d6981..edd178a23f 100644 --- a/_getting-started/intro.md +++ b/_getting-started/intro.md @@ -106,7 +106,6 @@ in | 1 the | 1, 2 eye | 1 of | 1 -the | 1 beholder | 1 and | 2 beast | 2 @@ -158,4 +157,4 @@ In OpenSearch, a shard is a Lucene index, which consists of _segments_ (or segme ## Next steps -- Learn how to install OpenSearch within minutes in [Installation quickstart]({{site.url}}{{site.baseurl}}/getting-started/quickstart/). \ No newline at end of file +- Learn how to install OpenSearch within minutes in [Installation quickstart]({{site.url}}{{site.baseurl}}/getting-started/quickstart/). diff --git a/_im-plugin/index-rollups/rollup-api.md b/_im-plugin/index-rollups/rollup-api.md index 61bfdf76d4..5064d2ac49 100644 --- a/_im-plugin/index-rollups/rollup-api.md +++ b/_im-plugin/index-rollups/rollup-api.md @@ -105,8 +105,8 @@ Options | Description | Type | Required `schedule.interval.cron.expression` | Specify a Unix cron expression. | String | Yes `schedule.interval.cron.timezone` | Specify timezones as defined by the IANA Time Zone Database. Defaults to UTC. | String | No `description` | Optionally, describe the rollup job. | String | No -`enabled` | When true, the index rollup job is scheduled. Default is true. | Boolean | Yes -`continuous` | Specify whether or not the index rollup job continuously rolls up data forever or just executes over the current data set once and stops. Default is false. | Boolean | Yes +`enabled` | When true, the index rollup job is scheduled. Default is `true`. | Boolean | Yes +`continuous` | Specify whether or not the index rollup job continuously rolls up data forever or executes over the current dataset once and stops. Default is `false`. | Boolean | Yes `error_notification` | Set up a Mustache message template for error notifications. For example, if an index rollup job fails, the system sends a message to a Slack channel. | Object | No `page_size` | Specify the number of buckets to paginate at a time during rollup. | Number | Yes `delay` | The number of milliseconds to delay execution of the index rollup job. | Long | No diff --git a/_im-plugin/index-transforms/transforms-apis.md b/_im-plugin/index-transforms/transforms-apis.md index df9ff19f8f..37d2c035b5 100644 --- a/_im-plugin/index-transforms/transforms-apis.md +++ b/_im-plugin/index-transforms/transforms-apis.md @@ -39,7 +39,7 @@ You can specify the following options in the HTTP request body: Option | Data Type | Description | Required :--- | :--- | :--- | :--- enabled | Boolean | If true, the transform job is enabled at creation. | No -continuous | Boolean | Specifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the `schedule` field and run based off of newly transformed buckets as well as any new data added to source indexes. Non-continuous jobs execute only once. Default is false. | No +continuous | Boolean | Specifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the `schedule` field and run based off of newly transformed buckets as well as any new data added to source indexes. Non-continuous jobs execute only once. Default is `false`. | No schedule | Object | The schedule for the transform job. | Yes start_time | Integer | The Unix epoch time of the transform job's start time. | Yes description | String | Describes the transform job. | No @@ -447,7 +447,7 @@ from | The starting transform to return. Default is 0. | No size | Specifies the number of transforms to return. Default is 10. | No search |The search term to use to filter results. | No sortField | The field to sort results with. | No -sortDirection | Specifies the direction to sort results in. Can be `ASC` or `DESC`. Default is ASC. | No +sortDirection | Specifies the direction to sort results in. Can be `ASC` or `DESC`. Default is `ASC`. | No #### Sample Request diff --git a/_ingest-pipelines/processors/fingerprint.md b/_ingest-pipelines/processors/fingerprint.md new file mode 100644 index 0000000000..4775da98b6 --- /dev/null +++ b/_ingest-pipelines/processors/fingerprint.md @@ -0,0 +1,158 @@ +--- +layout: default +title: Fingerprint +parent: Ingest processors +nav_order: 105 +--- + +# Fingerprint processor +Introduced 2.16 +{: .label .label-purple } + +The `fingerprint` processor is used to generate a hash value for either certain specified fields or all fields in a document. The hash value can be used to deduplicate documents within an index and collapse search results. + +For each field, the field name, the length of the field value, and the field value itself are concatenated and separated by the pipe character `|`. For example, if the field name is `field1` and the value is `value1`, then the concatenated string would be `|field1|3:value1|field2|10:value2|`. For object fields, the field name is flattened by joining the nested field names with a period `.`. For instance, if the object field is `root_field` with a sub-field `sub_field1` having the value `value1` and another sub-field `sub_field2` with the value `value2`, then the concatenated string would be `|root_field.sub_field1|1:value1|root_field.sub_field2|100:value2|`. + +The following is the syntax for the `fingerprint` processor: + +```json +{ + "community_id": { + "fields": ["foo", "bar"], + "target_field": "fingerprint", + "hash_method": "SHA-1@2.16.0" + } +} +``` +{% include copy-curl.html %} + +## Configuration parameters + +The following table lists the required and optional parameters for the `fingerprint` processor. + +Parameter | Required/Optional | Description | +|-----------|-----------|-----------| +`fields` | Optional | A list of fields used to generate a hash value. | +`exclude_fields` | Optional | Specifies the fields to be excluded from hash value generation. It is mutually exclusive with the `fields` parameter; if both `exclude_fields` and `fields` are empty or null, then all fields are included in the hash value calculation. | +`hash_method` | Optional | Specifies the hashing algorithm to be used, with options being `MD5@2.16.0`, `SHA-1@2.16.0`, `SHA-256@2.16.0`, or `SHA3-256@2.16.0`. Default is `SHA-1@2.16.0`. The version number is appended to ensure consistent hashing across OpenSearch versions, and new versions will support new hash methods. | +`target_field` | Optional | Specifies the name of the field in which the generated hash value will be stored. If not provided, then the hash value is stored in the `fingerprint` field by default. | +`ignore_missing` | Optional | Specifies whether the processor should exit quietly if one of the required fields is missing. Default is `false`. | +`description` | Optional | A brief description of the processor. | +`if` | Optional | A condition for running the processor. | +`ignore_failure` | Optional | If set to `true`, then failures are ignored. Default is `false`. | +`on_failure` | Optional | A list of processors to run if the processor fails. | +`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. | + +## Using the processor + +Follow these steps to use the processor in a pipeline. + +**Step 1: Create a pipeline** + +The following query creates a pipeline named `fingerprint_pipeline` that uses the `fingerprint` processor to generate a hash value for specified fields in the document: + +```json +PUT /_ingest/pipeline/fingerprint_pipeline +{ + "description": "generate hash value for some specified fields the document", + "processors": [ + { + "fingerprint": { + "fields": ["foo", "bar"] + } + } + ] +} +``` +{% include copy-curl.html %} + +**Step 2 (Optional): Test the pipeline** + +It is recommended that you test your pipeline before ingesting documents. +{: .tip} + +To test the pipeline, run the following query: + +```json +POST _ingest/pipeline/fingerprint_pipeline/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "foo": "foo", + "bar": "bar" + } + } + ] +} +``` +{% include copy-curl.html %} + +#### Response + +The following example response confirms that the pipeline is working as expected: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "foo": "foo", + "bar": "bar", + "fingerprint": "SHA-1@2.16.0:fYeen7hTJ2zs9lpmUnk6nvH54sM=" + }, + "_ingest": { + "timestamp": "2024-03-11T02:17:22.329823Z" + } + } + } + ] +} +``` + +**Step 3: Ingest a document** + +The following query ingests a document into an index named `testindex1`: + +```json +PUT testindex1/_doc/1?pipeline=fingerprint_pipeline +{ + "foo": "foo", + "bar": "bar" +} +``` +{% include copy-curl.html %} + +#### Response + +The request indexes the document into the `testindex1` index: + +```json +{ + "_index": "testindex1", + "_id": "1", + "_version": 1, + "result": "created", + "_shards": { + "total": 2, + "successful": 1, + "failed": 0 + }, + "_seq_no": 0, + "_primary_term": 1 +} +``` + +**Step 4 (Optional): Retrieve the document** + +To retrieve the document, run the following query: + +```json +GET testindex1/_doc/1 +``` +{% include copy-curl.html %} diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md index 79f30524d6..0e1ee1e114 100644 --- a/_ingest-pipelines/processors/index-processors.md +++ b/_ingest-pipelines/processors/index-processors.md @@ -40,6 +40,7 @@ Processor type | Description `dot_expander` | Expands a field with dots into an object field. `drop` |Drops a document without indexing it or raising any errors. `fail` | Raises an exception and stops the execution of a pipeline. +`fingerprint` | Generates a hash value for either certain specified fields or all fields in a document. `foreach` | Allows for another processor to be applied to each element of an array or an object field in a document. `geoip` | Adds information about the geographical location of an IP address. `geojson-feature` | Indexes GeoJSON data into a geospatial field. @@ -71,3 +72,7 @@ Processor type | Description ## Batch-enabled processors Some processors support batch ingestion---they can process multiple documents at the same time as a batch. These batch-enabled processors usually provide better performance when using batch processing. For batch processing, use the [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) and provide a `batch_size` parameter. All batch-enabled processors have a batch mode and a single-document mode. When you ingest documents using the `PUT` method, the processor functions in single-document mode and processes documents in series. Currently, only the `text_embedding` and `sparse_encoding` processors are batch enabled. All other processors process documents one at a time. + +## Selectively enabling processors + +Processors defined by the [ingest-common module](https://github.com/opensearch-project/OpenSearch/blob/2.x/modules/ingest-common/src/main/java/org/opensearch/ingest/common/IngestCommonPlugin.java) can be selectively enabled by providing the `ingest-common.processors.allowed` cluster setting. If not provided, then all processors are enabled by default. Specifying an empty list disables all processors. If the setting is changed to remove previously enabled processors, then any pipeline using a disabled processor will fail after node restart when the new setting takes effect. diff --git a/_ingest-pipelines/processors/split.md b/_ingest-pipelines/processors/split.md index 2052c3def1..c424ef671c 100644 --- a/_ingest-pipelines/processors/split.md +++ b/_ingest-pipelines/processors/split.md @@ -26,19 +26,18 @@ The following is the syntax for the `split` processor: The following table lists the required and optional parameters for the `split` processor. -Parameter | Required/Optional | Description | -|-----------|-----------|-----------| -`field` | Required | The field containing the string to be split. -`separator` | Required | The delimiter used to split the string. This can be a regular expression pattern. -`preserve_field` | Optional | If set to `true`, preserves empty trailing fields (for example, `''`) in the resulting array. If set to `false`, empty trailing fields are removed from the resulting array. Default is `false`. -`target_field` | Optional | The field where the array of substrings is stored. If not specified, then the field is updated in-place. -`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified -field. If set to `true`, then the processor ignores missing values in the field and leaves the `target_field` unchanged. Default is `false`. -`description` | Optional | A brief description of the processor. -`if` | Optional | A condition for running the processor. -`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, then failures are ignored. Default is `false`. -`on_failure` | Optional | A list of processors to run if the processor fails. -`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. +Parameter | Required/Optional | Description +:--- | :--- | :--- +`field` | Required | The field containing the string to be split. +`separator` | Required | The delimiter used to split the string. This can be a regular expression pattern. +`preserve_field` | Optional | If set to `true`, preserves empty trailing fields (for example, `''`) in the resulting array. If set to `false`, empty trailing fields are removed from the resulting array. Default is `false`. +`target_field` | Optional | The field where the array of substrings is stored. If not specified, then the field is updated in-place. +`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified field. If set to `true`, then the processor ignores missing values in the field and leaves the `target_field` unchanged. Default is `false`. +`description` | Optional | A brief description of the processor. +`if` | Optional | A condition for running the processor. +`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, then failures are ignored. Default is `false`. +`on_failure` | Optional | A list of processors to run if the processor fails. +`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. ## Using the processor diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index 34b1829b78..a1894a0d2c 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -54,6 +54,8 @@ OpenSearch supports the following dynamic cluster-level index settings: - `indices.fielddata.cache.size` (String): The maximum size of the field data cache. May be specified as an absolute value (for example, `8GB`) or a percentage of the node heap (for example, `50%`). This value is static so you must specify it in the `opensearch.yml` file. If you don't specify this setting, the maximum size is unlimited. This value should be smaller than the `indices.breaker.fielddata.limit`. For more information, see [Field data circuit breaker]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/circuit-breaker/#field-data-circuit-breaker-settings). +- `indices.query.bool.max_clause_count` (Integer): Defines the maximum product of fields and terms that are queryable simultaneously. Before OpenSearch 2.16, a cluster restart was required in order to apply this static setting. Now dynamic, existing search thread pools may use the old static value initially, causing `TooManyClauses` exceptions. New thread pools use the updated value. Default is `1024`. + - `cluster.remote_store.index.path.type` (String): The path strategy for the data stored in the remote store. This setting is effective only for remote-store-enabled clusters. This setting supports the following values: - `fixed`: Stores the data in path structure `
To learn about multi-cluster support for data sources, see [Enable OpenSearch Dashboards to support multiple OpenSearch clusters](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1388).
diff --git a/_query-dsl/compound/function-score.md b/_query-dsl/compound/function-score.md
index 8180058ae6..98568e0965 100644
--- a/_query-dsl/compound/function-score.md
+++ b/_query-dsl/compound/function-score.md
@@ -826,4 +826,197 @@ The results contain the three matching blog posts:
}
}
```
-
\ No newline at end of file
+
+
+## Named functions
+
+When defining a function, you can specify its name using the `_name` parameter at the top level. This name is useful for debugging and understanding the scoring process. Once specified, the function name is included in the score calculation explanation whenever possible (this applies to functions, filters, and queries). You can identify the function by its `_name` in the response.
+
+### Example
+
+The following request sets `explain` to `true` for debugging purposes in order to obtain a scoring explanation in the response. Each function contains a `_name` parameter so that you can identify the function unambiguously:
+
+```json
+GET blogs/_search
+{
+ "explain": true,
+ "size": 1,
+ "query": {
+ "function_score": {
+ "functions": [
+ {
+ "_name": "likes_function",
+ "script_score": {
+ "script": {
+ "lang": "painless",
+ "source": "return doc['likes'].value * 2;"
+ }
+ },
+ "weight": 0.6
+ },
+ {
+ "_name": "views_function",
+ "field_value_factor": {
+ "field": "views",
+ "factor": 1.5,
+ "modifier": "log1p",
+ "missing": 1
+ },
+ "weight": 0.3
+ },
+ {
+ "_name": "comments_function",
+ "gauss": {
+ "comments": {
+ "origin": 1000,
+ "scale": 800
+ }
+ },
+ "weight": 0.1
+ }
+ ]
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response explains the scoring process. For each function, the explanation contains the function `_name` in its `description`:
+
++ Response +
+ {: .text-delta} + +```json +{ + "took": 14, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 3, + "relation": "eq" + }, + "max_score": 6.1600614, + "hits": [ + { + "_shard": "[blogs][0]", + "_node": "_yndTaZHQWimcDgAfOfRtQ", + "_index": "blogs", + "_id": "1", + "_score": 6.1600614, + "_source": { + "name": "Semantic search in OpenSearch", + "views": 1200, + "likes": 150, + "comments": 16, + "date_posted": "2022-04-17" + }, + "_explanation": { + "value": 6.1600614, + "description": "function score, product of:", + "details": [ + { + "value": 1, + "description": "*:*", + "details": [] + }, + { + "value": 6.1600614, + "description": "min of:", + "details": [ + { + "value": 6.1600614, + "description": "function score, score mode [multiply]", + "details": [ + { + "value": 180, + "description": "product of:", + "details": [ + { + "value": 300, + "description": "script score function(_name: likes_function), computed with script:\"Script{type=inline, lang='painless', idOrCode='return doc['likes'].value * 2;', options={}, params={}}\"", + "details": [ + { + "value": 1, + "description": "_score: ", + "details": [ + { + "value": 1, + "description": "*:*", + "details": [] + } + ] + } + ] + }, + { + "value": 0.6, + "description": "weight", + "details": [] + } + ] + }, + { + "value": 0.9766541, + "description": "product of:", + "details": [ + { + "value": 3.2555137, + "description": "field value function(_name: views_function): log1p(doc['views'].value?:1.0 * factor=1.5)", + "details": [] + }, + { + "value": 0.3, + "description": "weight", + "details": [] + } + ] + }, + { + "value": 0.035040613, + "description": "product of:", + "details": [ + { + "value": 0.35040614, + "description": "Function for field comments:", + "details": [ + { + "value": 0.35040614, + "description": "exp(-0.5*pow(MIN[Math.max(Math.abs(16.0(=doc value) - 1000.0(=origin))) - 0.0(=offset), 0)],2.0)/461662.4130844683, _name: comments_function)", + "details": [] + } + ] + }, + { + "value": 0.1, + "description": "weight", + "details": [] + } + ] + } + ] + }, + { + "value": 3.4028235e+38, + "description": "maxBoost", + "details": [] + } + ] + } + ] + } + } + ] + } +} +``` +-Before using hybrid search, you must set up a text embedding model. For more information, see [Choosing a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/integrating-ml-models/#choosing-a-model). +To follow this example, you must set up a text embedding model. For more information, see [Choosing a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/integrating-ml-models/#choosing-a-model). If you have already generated text embeddings, ingest the embeddings into an index and skip to [Step 4](#step-4-configure-a-search-pipeline). {: .note} ## Using hybrid search diff --git a/_search-plugins/index.md b/_search-plugins/index.md index fca30667ee..79e0e715d0 100644 --- a/_search-plugins/index.md +++ b/_search-plugins/index.md @@ -70,6 +70,8 @@ OpenSearch provides the following search relevance features: - [Querqy]({{site.url}}{{site.baseurl}}/search-plugins/querqy/): Offers query rewriting capability. +- [User Behavior Insights]({{site.url}}{{site.baseurl}}/search-plugins/ubi/): Links user behavior to user queries to improve search quality. + ## Search results OpenSearch supports the following commonly used operations on search results: diff --git a/_search-plugins/knn/settings.md b/_search-plugins/knn/settings.md index f4ef057cfb..4d84cc80bb 100644 --- a/_search-plugins/knn/settings.md +++ b/_search-plugins/knn/settings.md @@ -12,17 +12,28 @@ The k-NN plugin adds several new cluster settings. To learn more about static an ## Cluster settings +The following table lists all available cluster-level k-NN settings. For more information about cluster settings, see [Configuring OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#updating-cluster-settings-using-the-api) and [Updating cluster settings using the API]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#updating-cluster-settings-using-the-api). + +Setting | Static/Dynamic | Default | Description +:--- | :--- | :--- | :--- +`knn.plugin.enabled`| Dynamic | `true` | Enables or disables the k-NN plugin. +`knn.algo_param.index_thread_qty` | Dynamic | `1` | The number of threads used for native library index creation. Keeping this value low reduces the CPU impact of the k-NN plugin but also reduces indexing performance. +`knn.cache.item.expiry.enabled` | Dynamic | `false` | Whether to remove native library indexes that have not been accessed for a certain duration from memory. +`knn.cache.item.expiry.minutes` | Dynamic | `3h` | If enabled, the amount of idle time before a native library index is removed from memory. +`knn.circuit_breaker.unset.percentage` | Dynamic | `75` | The native memory usage threshold for the circuit breaker. Memory usage must be lower than this percentage of `knn.memory.circuit_breaker.limit` in order for `knn.circuit_breaker.triggered` to remain `false`. +`knn.circuit_breaker.triggered` | Dynamic | `false` | True when memory usage exceeds the `knn.circuit_breaker.unset.percentage` value. +`knn.memory.circuit_breaker.limit` | Dynamic | `50%` | The native memory limit for native library indexes. At the default value, if a machine has 100 GB of memory and the JVM uses 32 GB, then the k-NN plugin uses 50% of the remaining 68 GB (34 GB). If memory usage exceeds this value, then the plugin removes the native library indexes used least recently. +`knn.memory.circuit_breaker.enabled` | Dynamic | `true` | Whether to enable the k-NN memory circuit breaker. +`knn.model.index.number_of_shards`| Dynamic | `1` | The number of shards to use for the model system index, which is the OpenSearch index that stores the models used for approximate nearest neighbor (ANN) search. +`knn.model.index.number_of_replicas`| Dynamic | `1` | The number of replica shards to use for the model system index. Generally, in a multi-node cluster, this value should be at least 1 in order to increase stability. +`knn.model.cache.size.limit` | Dynamic | `10%` | The model cache limit cannot exceed 25% of the JVM heap. +`knn.faiss.avx2.disabled` | Static | `false` | A static setting that specifies whether to disable the SIMD-based `libopensearchknn_faiss_avx2.so` library and load the non-optimized `libopensearchknn_faiss.so` library for the Faiss engine on machines with x64 architecture. For more information, see [SIMD optimization for the Faiss engine]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/#simd-optimization-for-the-faiss-engine). + +## Index settings + +The following table lists all available index-level k-NN settings. All settings are static. For information about updating static index-level settings, see [Updating a static index setting]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/#updating-a-static-index-setting). + Setting | Default | Description -:--- | :--- | :--- -`knn.algo_param.index_thread_qty` | 1 | The number of threads used for native library index creation. Keeping this value low reduces the CPU impact of the k-NN plugin, but also reduces indexing performance. -`knn.cache.item.expiry.enabled` | false | Whether to remove native library indexes that have not been accessed for a certain duration from memory. -`knn.cache.item.expiry.minutes` | 3h | If enabled, the idle time before removing a native library index from memory. -`knn.circuit_breaker.unset.percentage` | 75% | The native memory usage threshold for the circuit breaker. Memory usage must be below this percentage of `knn.memory.circuit_breaker.limit` for `knn.circuit_breaker.triggered` to remain false. -`knn.circuit_breaker.triggered` | false | True when memory usage exceeds the `knn.circuit_breaker.unset.percentage` value. -`knn.memory.circuit_breaker.limit` | 50% | The native memory limit for native library indexes. At the default value, if a machine has 100 GB of memory and the JVM uses 32 GB, the k-NN plugin uses 50% of the remaining 68 GB (34 GB). If memory usage exceeds this value, k-NN removes the least recently used native library indexes. -`knn.memory.circuit_breaker.enabled` | true | Whether to enable the k-NN memory circuit breaker. -`knn.plugin.enabled`| true | Enables or disables the k-NN plugin. -`knn.model.index.number_of_shards`| 1 | The number of shards to use for the model system index, the OpenSearch index that stores the models used for Approximate Nearest Neighbor (ANN) search. -`knn.model.index.number_of_replicas`| 1 | The number of replica shards to use for the model system index. Generally, in a multi-node cluster, this should be at least 1 to increase stability. -`knn.advanced.filtered_exact_search_threshold`| null | The threshold value for the filtered IDs that is used to switch to exact search during filtered ANN search. If the number of filtered IDs in a segment is less than this setting's value, exact search will be performed on the filtered IDs. -`knn.faiss.avx2.disabled` | False | A static setting that specifies whether to disable the SIMD-based `libopensearchknn_faiss_avx2.so` library and load the non-optimized `libopensearchknn_faiss.so` library for the Faiss engine on machines with x64 architecture. For more information, see [SIMD optimization for the Faiss engine]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/#simd-optimization-for-the-faiss-engine). +:--- | :--- | :--- +`index.knn.advanced.filtered_exact_search_threshold`| `null` | The filtered ID threshold value used to switch to exact search during filtered ANN search. If the number of filtered IDs in a segment is lower than this setting's value, then exact search will be performed on the filtered IDs. +`index.knn.algo_param.ef_search` | `100` | `ef` (or `efSearch`) represents the size of the dynamic list for the nearest neighbors used during a search. Higher `ef` values lead to a more accurate but slower search. `ef` cannot be set to a value lower than the number of queried nearest neighbors, `k`. `ef` can take any value between `k` and the size of the dataset. \ No newline at end of file diff --git a/_search-plugins/neural-sparse-search.md b/_search-plugins/neural-sparse-search.md index b2b4fc33d6..8aa2ff7dbf 100644 --- a/_search-plugins/neural-sparse-search.md +++ b/_search-plugins/neural-sparse-search.md @@ -16,8 +16,8 @@ Introduced 2.11 When selecting a model, choose one of the following options: -- Use a sparse encoding model at both ingestion time and search time (high performance, relatively high latency). -- Use a sparse encoding model at ingestion time and a tokenizer at search time for relatively low performance and low latency. The tokenism doesn't conduct model inference, so you can deploy and invoke a tokenizer using the ML Commons Model API for a more consistent experience. +- Use a sparse encoding model at both ingestion time and search time for better search relevance at the expense of relatively high latency. +- Use a sparse encoding model at ingestion time and a tokenizer at search time for lower search latency at the expense of relatively lower search relevance. Tokenization doesn't involve model inference, so you can deploy and invoke a tokenizer using the ML Commons Model API for a more streamlined experience. **PREREQUISITE**
Before using neural sparse search, make sure to set up a [pretrained sparse embedding model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sparse-encoding-models) or your own sparse embedding model. For more information, see [Choosing a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/integrating-ml-models/#choosing-a-model). diff --git a/_search-plugins/search-pipelines/search-processors.md b/_search-plugins/search-pipelines/search-processors.md index 5e53cf5615..4630ab950c 100644 --- a/_search-plugins/search-pipelines/search-processors.md +++ b/_search-plugins/search-pipelines/search-processors.md @@ -121,3 +121,7 @@ The response contains the `search_pipelines` object that lists the available req In addition to the processors provided by OpenSearch, additional processors may be provided by plugins. {: .note} + +## Selectively enabling processors + +Processors defined by the [search-pipeline-common module](https://github.com/opensearch-project/OpenSearch/blob/2.x/modules/search-pipeline-common/src/main/java/org/opensearch/search/pipeline/common/SearchPipelineCommonModulePlugin.java) are selectively enabled through the following cluster settings: `search.pipeline.common.request.processors.allowed`, `search.pipeline.common.response.processors.allowed`, or `search.pipeline.common.search.phase.results.processors.allowed`. If unspecified, then all processors are enabled. An empty list disables all processors. Removing enabled processors causes pipelines using them to fail after a node restart. \ No newline at end of file diff --git a/_search-plugins/search-relevance/reranking-search-results.md b/_search-plugins/search-relevance/reranking-search-results.md index 14c418020d..4b4deaeb92 100644 --- a/_search-plugins/search-relevance/reranking-search-results.md +++ b/_search-plugins/search-relevance/reranking-search-results.md @@ -115,4 +115,19 @@ POST /my-index/_search ``` {% include copy-curl.html %} -Alternatively, you can provide the full path to the field containing the context. For more information, see [Rerank processor example]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rerank-processor/#example). \ No newline at end of file +Alternatively, you can provide the full path to the field containing the context. For more information, see [Rerank processor example]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rerank-processor/#example). + +## Using rerank and normalization processors together + +When you use a rerank processor in conjunction with a [normalization processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/normalization-processor/) and a hybrid query, the rerank processor alters the final document scores. This is because the rerank processor operates after the normalization processor in the search pipeline. +{: .note} + +The processing order is as follows: + +- Normalization processor: This processor normalizes the document scores based on the configured normalization method. For more information, see [Normalization processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/normalization-processor/). +- Rerank processor: Following normalization, the rerank processor further adjusts the document scores. This adjustment can significantly impact the final ordering of search results. + +This processing order has the following implications: + +- Score modification: The rerank processor modifies the scores that were initially adjusted by the normalization processor, potentially leading to different ranking results than initially expected. +- Hybrid queries: In the context of hybrid queries, where multiple types of queries and scoring mechanisms are combined, this behavior is particularly noteworthy. The combined scores from the initial query are normalized first and then reranked, resulting in a two-stage scoring modification. \ No newline at end of file diff --git a/_search-plugins/sql/functions.md b/_search-plugins/sql/functions.md index de3b578e1a..9706148d76 100644 --- a/_search-plugins/sql/functions.md +++ b/_search-plugins/sql/functions.md @@ -32,10 +32,10 @@ The SQL plugin supports the following common functions shared across the SQL and | `expm1` | `expm1(number T) -> double` | `SELECT expm1(0.5)` | | `floor` | `floor(number T) -> long` | `SELECT floor(0.5)` | | `ln` | `ln(number T) -> double` | `SELECT ln(10)` | -| `log` | `log(number T) -> double` or `log(number T, number T) -> double` | `SELECT log(10)`, `SELECT log(2, 16)` | +| `log` | `log(number T) -> double` or `log(number T, number T) -> double` | `SELECT log(10) -> 2.3`, `SELECT log(2, 16) -> 4`| | `log2` | `log2(number T) -> double` | `SELECT log2(10)` | -| `log10` | `log10(number T) -> double` | `SELECT log10(10)` | -| `mod` | `mod(number T, number T) -> T` | `SELECT mod(2, 3)` | +| `log10` | `log10(number T) -> double` | `SELECT log10(100)` | +| `mod` | `mod(number T, number T) -> T` | `SELECT mod(10,4) -> 2 ` | | `modulus` | `modulus(number T, number T) -> T` | `SELECT modulus(2, 3)` | | `multiply` | `multiply(number T, number T) -> T` | `SELECT multiply(2, 3)` | | `pi` | `pi() -> double` | `SELECT pi()` | @@ -162,7 +162,7 @@ Functions marked with * are only available in SQL. | `replace` | `replace(string, string, string) -> string` | `SELECT replace('hello', 'l', 'x')` | | `right` | `right(string, integer) -> string` | `SELECT right('hello', 2)` | | `rtrim` | `rtrim(string) -> string` | `SELECT rtrim('hello ')` | -| `substring` | `substring(string, integer, integer) -> string` | `SELECT substring('hello', 2, 4)` | +| `substring` | `substring(string, integer, integer) -> string` | `SELECT substring('hello', 2, 2) -> 'el'` | | `trim` | `trim(string) -> string` | `SELECT trim(' hello')` | | `upper` | `upper(string) -> string` | `SELECT upper('hello world')` | diff --git a/_search-plugins/sql/ppl/functions.md b/_search-plugins/sql/ppl/functions.md index 275030f723..d192799f2e 100644 --- a/_search-plugins/sql/ppl/functions.md +++ b/_search-plugins/sql/ppl/functions.md @@ -11,7 +11,7 @@ redirect_from: # Commands -`PPL` supports all [`SQL` common]({{site.url}}{{site.baseurl}}/search-plugins/sql/functions/) functions, including [relevance search]({{site.url}}{{site.baseurl}}/search-plugins/sql/full-text/), but also introduces few more functions (called `commands`) which are available in `PPL` only. +`PPL` supports most [`SQL` common]({{site.url}}{{site.baseurl}}/search-plugins/sql/functions/) functions, including [relevance search]({{site.url}}{{site.baseurl}}/search-plugins/sql/full-text/), but also introduces few more functions (called `commands`) which are available in `PPL` only. ## dedup diff --git a/_search-plugins/sql/ppl/index.md b/_search-plugins/sql/ppl/index.md index 850a540bc4..602255d126 100644 --- a/_search-plugins/sql/ppl/index.md +++ b/_search-plugins/sql/ppl/index.md @@ -37,7 +37,15 @@ PPL filters, transforms, and aggregates data using a series of commands. See [Co ## Using PPL within OpenSearch -To use PPL, you must have installed OpenSearch Dashboards. PPL is available within the [Query Workbench tool](https://playground.opensearch.org/app/opensearch-query-workbench#/). See the [Query Workbench]({{site.url}}{{site.baseurl}}/dashboards/query-workbench/) documentation for a tutorial on using PPL within OpenSearch. +The SQL plugin is required to run PPL queries in OpenSearch. If you're running a minimal distribution of OpenSearch, you might have to [install the SQL plugin]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) before using PPL. +{: .note} + +You can run PPL queries interactively in OpenSearch Dashboards or programmatically using the ``_ppl`` endpoint. + +In OpenSearch Dashboards, the [Query Workbench tool](https://playground.opensearch.org/app/opensearch-query-workbench#/) provides an interactive testing environment, documented in [Query Workbench documentation]({{site.url}}{{site.baseurl}}/dashboards/query-workbench/). + +To run a PPL query using the API, see [SQL and PPL API]({{site.url}}{{site.baseurl}}/search-plugins/sql/sql-ppl-api/). + ## Developer documentation diff --git a/_search-plugins/sql/ppl/syntax.md b/_search-plugins/sql/ppl/syntax.md index 45eeb3aed2..22d6beaf26 100644 --- a/_search-plugins/sql/ppl/syntax.md +++ b/_search-plugins/sql/ppl/syntax.md @@ -8,10 +8,12 @@ nav_order: 1 # PPL syntax -Every PPL query starts with the `search` command. It specifies the index to search and retrieve documents from. Subsequent commands can follow in any order. +Every PPL query starts with the `search` command. It specifies the index to search and retrieve documents from. + +`PPL` supports exactly one `search` command per PPL query, and it is always the first command. The word `search` can be omitted. + +Subsequent commands can follow in any order. -Currently, `PPL` supports only one `search` command, which can be omitted to simplify the query. -{ : .note} ## Syntax @@ -22,8 +24,7 @@ source=
+
+
+{% comment %}
+The mermaid source below is converted into a PNG under
+.../images/ubi/ubi-schema-interactions.png
+
+
+```mermaid
+graph LR
+style L fill:none,stroke-dasharray: 5 5
+subgraph L["`*Legend*`"]
+ style ss height:150px
+ subgraph ss["Standard Search"]
+ direction LR
+
+ style ln1a fill:blue
+ ln1a[ ]--->ln1b[ ];
+ end
+ subgraph ubi-leg["UBI data flow"]
+ direction LR
+
+ ln2a[ ].->|"`**UBI interaction**`"|ln2b[ ];
+ style ln1c fill:red
+ ln1c[ ]-->|query_id flow|ln1d[ ];
+ end
+end
+linkStyle 0 stroke-width:2px,stroke:#0A1CCF
+linkStyle 2 stroke-width:2px,stroke:red
+```
+```mermaid
+%%{init: {
+ "flowchart": {"htmlLabels": false},
+
+ }
+}%%
+graph TB
+
+User--1) raw search string-->Search;
+Search--2) search string-->Docs
+style OS stroke-width:2px, stroke:#0A1CCF, fill:#62affb, opacity:.5
+subgraph OS[OpenSearch Cluster fa:fa-database]
+ style E stroke-width:1px,stroke:red
+ E[( UBI Events )]
+ style Docs stroke-width:1px,stroke:#0A1CCF
+ style Q stroke-width:1px,stroke:red
+ Docs[(Document Index)] -."3) {DSL...} & [object_id's,...]".-> Q[( UBI Queries )];
+ Q -.4) query_id.-> Docs ;
+end
+
+Docs -- "5) return both query_id & [objects,...]" --->Search ;
+Search-.6) query_id.->U;
+Search --7) [results, ...]--> User
+
+style *client-side* stroke-width:1px, stroke:#D35400
+subgraph "`*client-side*`"
+ style User stroke-width:4px, stroke:#EC636
+ User["`**User**`" fa:fa-user]
+ App
+ Search
+ U
+ style App fill:#D35400,opacity:.35, stroke:#0A1CCF, stroke-width:2px
+ subgraph App[ UserApp fa:fa-store]
+ style Search stroke-width:2px, stroke:#0A1CCF
+ Search( Search Client )
+ style U stroke-width:1px,stroke:red
+ U( UBI Client )
+ end
+end
+
+User -.8) selects object_id:123.->U;
+U-."9) index event:{query_id, onClick, object_id:123}".->E;
+
+linkStyle 1,2,0,6 stroke-width:2px,fill:none,stroke:#0A1CCF
+linkStyle 3,4,5,8 stroke-width:2px,fill:none,stroke:red
+```
+{% endcomment %}
+Here are some key points regarding the roles:
+- The **Search client** is in charge of searching and then receiving *objects* from an OpenSearch document index (1, 2, **5**, and 7 in the preceding diagram).
+
+Step **5** is in bold because it denotes UBI-specific additions, like `query_id`, to standard OpenSearch interactions.
+ {: .note}
+- If activated in the `ext.ubi` stanza of the search request, the **User Behavior Insights** plugin manages the **UBI queries** store in the background, indexing each query, ensuring a unique `query_id` for all returned `object_id` values, and then passing the `query_id` back to the **Search client** so that events can be linked to the query (3, 4, and **5** in preceding diagram).
+- **Objects** represent the items that the user searches for using the queries. Activating UBI involves mapping your real-world objects (using their identifiers, such as an `isbn` or `sku`) to the `object_id` fields in the index that is searched.
+- The **Search client**, if separate from the **UBI client**, forwards the indexed `query_id` to the **UBI client**.
+ Even though the *search* and *UBI event indexing* roles are separate in this diagram, many implementations can use the same OpenSearch client instance for both roles (6 in the preceding diagram).
+- The **UBI client** then indexes all user events with the specified `query_id` until a new search is performed. At this time, a new `query_id` is generated by the **User Behavior Insights** plugin and passed back to the **UBI client**.
+- If the **UBI client** interacts with a result **object**, such as during an **add to cart** event, then the `object_id`, `add_to_cart` `action_name`, and `query_id` are all indexed together, signaling the causal link between the *search* and the *object* (8 and 9 in the preceding diagram).
+
+
+
+## UBI stores
+
+There are two separate stores involved in supporting UBI data collection:
+* UBI queries
+* UBI events
+
+### UBI queries index
+
+All underlying query information and results (`object_ids`) are stored in the `ubi_queries` index and remain largely invisible in the background.
+
+
+The `ubi_queries` index [schema](https://github.com/OpenSearch-project/user-behavior-insights/tree/main/src/main/resources/queries-mapping.json) includes the following fields:
+
+- `timestamp` (events and queries): A UNIX timestamp indicating when the query was received.
+
+- `query_id` (events and queries): The unique ID of the query provided by the client or generated automatically. Different queries with the same text generate different `query_id` values.
+
+- `client_id` (events and queries): A user/client ID provided by the client application.
+
+- `query_response_objects_ids` (queries): An array of object IDs. An ID can have the same value as the `_id`, but it is meant to be the externally valid ID of a document, item, or product.
+
+Because UBI manages the `ubi_queries` index, you should never have to write directly to this index (except when importing data).
+
+### UBI events index
+
+The client side directly indexes events to the `ubi_events` index, linking the event [`action_name`](#action_name), objects (each with an `object_id`), and queries (each with a `query_id`), along with any other important event information.
+Because this schema is dynamic, you can add any new fields or structures (such as user information or geolocation information) that are not in the current **UBI events** [schema](https://github.com/opensearch-project/user-behavior-insights/tree/main/src/main/resources/events-mapping.json) at index time.
+
+Developers may define new fields under [`event_attributes`](#event_attributes).
+{: .note}
+
+The following are the predefined, minimal fields in the `ubi_events` index:
+
+ + +- `application` (size 100): The name of the application that tracks UBI events (for example, `amazon-shop` or `ABC-microservice`). + +
+ +- `action_name` (size 100): The name of the action that triggered the event. The UBI specification defines some common action names, but you can use any name. + +
+ +- `query_id` (size 100): The unique identifier of a query, which is typically a UUID but can be any string. + The `query_id` is either provided by the client or generated at index time by the UBI plugin. The `query_id` values in both the **UBI queries** and **UBI events** indexes must be consistent. + +
+ +- `client_id`: The client that issues the query. This is typically a web browser used by a unique user. + The `client_id` in both the **UBI queries** and **UBI events** indexes must be consistent. + +- `timestamp`: When the event occurred, either in UNIX format or formatted as `2018-11-13T20:20:39+00:00`. + +- `message_type` (size 100): A logical bin for grouping actions (each with an `action_name`). For example, `QUERY` or `CONVERSION`. + +- `message` (size 1,024): An optional text message for the log entry. For example, for a `message_type` `QUERY`, the `message` can contain the text related to a user's search. + +
+ +- `event_attributes`: An extensible structure that describes important context about the event. This structure consists of two primary structures: `position` and `object`. The structure is extensible, so you can add custom information about the event, such as the event's timing, user, or session. + + Because the `ubi_events` index is configured to perform dynamic mapping, the index can become bloated with many new fields. + {: .warning} + + - `event_attributes.position`: A structure that contains information about the location of the event origin, such as screen x, y coordinates, or the object's position in the list of results: + + - `event_attributes.position.ordinal`: Tracks the list position that a user can select (for example, selecting the third element can be described as `event{onClick, results[4]}`). + + - `event_attributes.position.{x,y}`: Tracks x and y values defined by the client. + + - `event_attributes.position.page_depth`: Tracks the page depth of the results. + + - `event_attributes.position.scroll_depth`: Tracks the scroll depth of the page results. + + - `event_attributes.position.trail`: A text field that tracks the path/trail that a user took to get to this location. + + - `event_attributes.object`: Contains identifying information about the object returned by the query (for example, a book, product, or post). + The `object` structure can refer to the object by internal ID or object ID. The `object_id` is the ID that links prior queries to this object. This field comprises the following subfields: + + - `event_attributes.object.internal_id`: A unique ID that OpenSearch can use to internally index the object, for example, the `_id` field in the indexes. + +
+ + - `event_attributes.object.object_id`: An ID by which a user can find the object instance in the **document corpus**. Examples include `ssn`, `isbn`, or `ean`. Variants need to be incorporated in the `object_id`, so a red T-shirt's `object_id` should be its SKU. + Initializing UBI requires mapping the document index's primary key to this `object_id`. +
+ +
+
+ - `event_attributes.object.object_id_field`: Indicates the type/class of the object and the name of the search index field that contains the `object_id`.
+
+ - `event_attributes.object.description`: An optional description of the object.
+
+ - `event_attributes.object.object_detail`: Optional additional information about the object.
+
+ - *extensible fields*: Be aware that any new indexed fields in the `object` will dynamically expand this schema.
diff --git a/_search-plugins/ubi/sql-queries.md b/_search-plugins/ubi/sql-queries.md
new file mode 100644
index 0000000000..517c81e1d6
--- /dev/null
+++ b/_search-plugins/ubi/sql-queries.md
@@ -0,0 +1,388 @@
+---
+layout: default
+title: Sample UBI SQL queries
+parent: User Behavior Insights
+has_children: false
+nav_order: 20
+---
+
+# Sample UBI SQL queries
+
+You can run sample User Behavior Insights (UBI) SQL queries in the OpenSearch Dashboards [Query Workbench]({{site.url}}{{site.baseurl}}/dashboards/query-workbench/).
+
+To query a demo workbench with synthetic data, see
+[http://chorus-opensearch-edition.dev.o19s.com:5601/app/opensearch-query-workbench](http://chorus-opensearch-edition.dev.o19s.com:5601/app/opensearch-query-workbench).
+
+## Queries with zero results
+
+Queries can be executed on events on either the server (`ubi_queries`) or client (`ubi_events`) side.
+
+### Server-side queries
+
+The UBI-activated search server logs the queries and their results, so in order to find all queries with *no* results, search for empty `query_response_hit_ids`:
+
+```sql
+select
+ count(*)
+from ubi_queries
+where query_response_hit_ids is null
+
+```
+
+### Client-side events
+
+Although it's relatively straightforward to find queries with no results on the server side, you can also get the same result by querying the *event attributes* that were logged on the client side.
+Both client- and server-side queries return the same results. Use the following query to search for queries with no results:
+
+```sql
+select
+ count(0)
+from ubi_events
+where event_attributes.result_count > 0
+```
+
+
+
+
+## Trending queries
+
+Trending queries can be found by using either of the following queries.
+
+### Server-side
+
+```sql
+select
+ user_query, count(0) Total
+from ubi_queries
+group by user_query
+order by Total desc
+```
+
+### Client-side
+
+```sql
+select
+ message, count(0) Total
+from ubi_events
+where
+ action_name='on_search'
+group by message
+order by Total desc
+```
+
+Both queries return the distribution of search strings, as shown in the following table.
+
+Message|Total
+|---|---|
+User Behavior Insights|127
+best Laptop|78
+camera|21
+backpack|17
+briefcase|14
+camcorder|11
+cabinet|9
+bed|9
+box|8
+bottle|8
+calculator|8
+armchair|7
+bench|7
+blackberry|6
+bathroom|6
+User Behavior Insights Mac|5
+best Laptop Dell|5
+User Behavior Insights VTech|5
+ayoolaolafenwa|5
+User Behavior Insights Dell|4
+best Laptop Vaddio|4
+agrega modelos intuitivas|4
+bеуоnd|4
+abraza metodologías B2C|3
+
+
+
+## Event type distribution counts
+
+To create a pie chart widget visualizing the most common events, run the following query:
+
+```sql
+select
+ action_name, count(0) Total
+from ubi_events
+group by action_name
+order by Total desc
+```
+
+The results include a distribution across actions, as shown in the following table.
+
+action_name|Total
+|---|---|
+on_search|5425
+brand_filter|3634
+global_click|3571
+view_search_results|3565
+product_sort|3558
+type_filter|3505
+product_hover|820
+item_click|708
+purchase|407
+declined_product|402
+add_to_cart|373
+page_exit|142
+user_feedback|123
+404_redirect|123
+
+The following query shows the distribution of margins across user actions:
+
+
+```sql
+select
+ action_name,
+ count(0) total,
+ AVG( event_attributes.object.object_detail.cost ) average_cost,
+ AVG( event_attributes.object.object_detail.margin ) average_margin
+from ubi_events
+group by action_name
+order by average_cost desc
+```
+
+The results include actions and the distribution across average costs and margins, as shown in the following table.
+
+action_name|total|average_cost|average_margin
+---|---|---|---
+declined_product|395|8457.12|6190.96
+item_click|690|7789.40|5862.70
+add_to_cart|374|6470.22|4617.09
+purchase|358|5933.83|5110.69
+global_click|3555||
+product_sort|3711||
+product_hover|779||
+page_exit|107||
+on_search|5438||
+brand_filter|3722||
+user_feedback|120||
+404_redirect|110||
+view_search_results|3639||
+type_filter|3691||
+
+## Sample search journey
+
+To find a search in the query log, run the following query:
+
+```sql
+select
+ client_id, query_id, user_query, query_response_hit_ids, query_response_id, timestamp
+from ubi_queries where query_id = '7ae52966-4fd4-4ab1-8152-0fd0b52bdadf'
+```
+
+The following table shows the results of the preceding query.
+
+client_id|query_id|user_query|query_response_hit_ids|query_response_id|timestamp
+---|---|---|---|---|---
+a15f1ef3-6bc6-4959-9b83-6699a4d29845|7ae52966-4fd4-4ab1-8152-0fd0b52bdadf|notebook|0882780391659|6e92c90c-1eee-4dd6-b820-c522fd4126f3|2024-06-04 19:02:45.728
+
+The `query` field in `query_id` has the following nested structure:
+
+```json
+{
+ "query": {
+ "size": 25,
+ "query": {
+ "query_string": {
+ "query": "(title:\"notebook\" OR attr_t_device_type:\"notebook\" OR name:\"notebook\")",
+ "fields": [],
+ "type": "best_fields",
+ "default_operator": "or",
+ "max_determinized_states": 10000,
+ "enable_position_increments": true,
+ "fuzziness": "AUTO",
+ "fuzzy_prefix_length": 0,
+ "fuzzy_max_expansions": 50,
+ "phrase_slop": 0,
+ "analyze_wildcard": false,
+ "escape": false,
+ "auto_generate_synonyms_phrase_query": true,
+ "fuzzy_transpositions": true,
+ "boost": 1.0
+ }
+ },
+ "ext": {
+ "query_id": "7ae52966-4fd4-4ab1-8152-0fd0b52bdadf",
+ "user_query": "notebook",
+ "client_id": "a15f1ef3-6bc6-4959-9b83-6699a4d29845",
+ "object_id_field": "primary_ean",
+ "query_attributes": {
+ "application": "ubi-demo"
+ }
+ }
+ }
+}
+```
+
+In the event log, `ubi_events`, search for the events that correspond to the preceding query (whose query ID is `7ae52966-4fd4-4ab1-8152-0fd0b52bdadf`):
+
+```sql
+select
+ application, query_id, action_name, message_type, message, client_id, timestamp
+from ubi_events
+where query_id = '7ae52966-4fd4-4ab1-8152-0fd0b52bdadf'
+order by timestamp
+```
+
+
+
+The results include all events associated with the user's query, as shown in the following table.
+
+application|query_id|action_name|message_type|message|client_id|timestamp
+---|---|---|---|---|---|---
+ubi-demo|7ae52966-4fd4-4ab1-8152-0fd0b52bdadf|on_search|QUERY|notebook|a15f1ef3-6bc6-4959-9b83-6699a4d29845|2024-06-04 19:02:45.777
+ubi-demo|7ae52966-4fd4-4ab1-8152-0fd0b52bdadf|product_hover|INFO|orquesta soluciones uno-a-uno|a15f1ef3-6bc6-4959-9b83-6699a4d29845|2024-06-04 19:02:45.816
+ubi-demo|7ae52966-4fd4-4ab1-8152-0fd0b52bdadf|item_click|INFO|innova relaciones centrado al usuario|a15f1ef3-6bc6-4959-9b83-6699a4d29845|2024-06-04 19:02:45.86
+ubi-demo|7ae52966-4fd4-4ab1-8152-0fd0b52bdadf|add_to_cart|CONVERSION|engineer B2B platforms|a15f1ef3-6bc6-4959-9b83-6699a4d29845|2024-06-04 19:02:45.905
+ubi-demo|7ae52966-4fd4-4ab1-8152-0fd0b52bdadf|purchase|CONVERSION|Purchase item 0884420136132|a15f1ef3-6bc6-4959-9b83-6699a4d29845|2024-06-04 19:02:45.913
+
+
+
+
+## User sessions
+
+To find more of the same user's sessions (with the client ID `a15f1ef3-6bc6-4959-9b83-6699a4d29845`), run the following query:
+
+```sql
+select
+ application, event_attributes.session_id, query_id,
+ action_name, message_type, event_attributes.dwell_time,
+ event_attributes.object.object_id,
+ event_attributes.object.description,
+ timestamp
+from ubi_events
+where client_id = 'a15f1ef3-6bc6-4959-9b83-6699a4d29845'
+order by query_id, timestamp
+```
+
+The results are truncated to show a sample of sessions, as shown in the following table.
+
+
+application|event_attributes.session_id|query_id|action_name|message_type|event_attributes.dwell_time|event_attributes.object.object_id|event_attributes.object.description|timestamp
+---|---|---|---|---|---|---|---|---
+ubi-demo|00731779-e290-4709-8af7-d495ae42bf48|0254a9b7-1d83-4083-aa46-e12dff86ec98|on_search|QUERY|46.6398|||2024-06-04 19:06:36.239
+ubi-demo|00731779-e290-4709-8af7-d495ae42bf48|0254a9b7-1d83-4083-aa46-e12dff86ec98|product_hover|INFO|53.681877|0065030834155|USB 2.0 S-Video and Composite Video Capture Cable|2024-06-04 19:06:36.284
+ubi-demo|00731779-e290-4709-8af7-d495ae42bf48|0254a9b7-1d83-4083-aa46-e12dff86ec98|item_click|INFO|40.699997|0065030834155|USB 2.0 S-Video and Composite Video Capture Cable|2024-06-04 19:06:36.334
+ubi-demo|00731779-e290-4709-8af7-d495ae42bf48|0254a9b7-1d83-4083-aa46-e12dff86ec98|declined_product|REJECT|5.0539055|0065030834155|USB 2.0 S-Video and Composite Video Capture Cable|2024-06-04 19:06:36.373
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|on_search|QUERY|26.422775|||2024-06-04 19:04:40.832
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|on_search|QUERY|17.1094|||2024-06-04 19:04:40.837
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|brand_filter|FILTER|40.090374|OBJECT-6c91da98-387b-45cb-8275-e90d1ea8bc54|supplier_name|2024-06-04 19:04:40.852
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|type_filter|INFO|37.658962|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:04:40.856
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|product_sort|SORT|3.6380951|||2024-06-04 19:04:40.923
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|view_search_results|INFO|46.436115|||2024-06-04 19:04:40.942
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|view_search_results|INFO|46.436115|||2024-06-04 19:04:40.959
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|type_filter|INFO|37.658962|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:04:40.972
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|brand_filter|FILTER|40.090374|OBJECT-6c91da98-387b-45cb-8275-e90d1ea8bc54|supplier_name|2024-06-04 19:04:40.997
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|type_filter|INFO|37.658962|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:04:41.006
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|product_sort|SORT|3.6380951|||2024-06-04 19:04:41.031
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|product_sort|SORT|3.6380951|||2024-06-04 19:04:41.091
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|type_filter|INFO|37.658962|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:04:41.164
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|brand_filter|FILTER|40.090374|OBJECT-6c91da98-387b-45cb-8275-e90d1ea8bc54|supplier_name|2024-06-04 19:04:41.171
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|view_search_results|INFO|46.436115|||2024-06-04 19:04:41.179
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|global_click|INFO|42.45651|OBJECT-d350cc2d-b979-4aca-bd73-71709832940f|(96, 127)|2024-06-04 19:04:41.224
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|view_search_results|INFO|46.436115|||2024-06-04 19:04:41.24
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|view_search_results|INFO|46.436115|||2024-06-04 19:04:41.285
+ubi-demo|844ca4b5-b6f8-4f7b-a5ec-7f6d95788e0b|0cf185be-91a8-49cf-9401-92ad079ce43b|global_click|INFO|42.45651|OBJECT-d350cc2d-b979-4aca-bd73-71709832940f|(96, 127)|2024-06-04 19:04:41.328
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|on_search|QUERY|52.721157|||2024-06-04 19:03:50.8
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|view_search_results|INFO|26.600422|||2024-06-04 19:03:50.802
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|product_sort|SORT|14.839713|||2024-06-04 19:03:50.875
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|brand_filter|FILTER|20.876852|OBJECT-6c91da98-387b-45cb-8275-e90d1ea8bc54|supplier_name|2024-06-04 19:03:50.927
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|type_filter|INFO|15.212905|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:03:50.997
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|view_search_results|INFO|26.600422|||2024-06-04 19:03:51.033
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|global_click|INFO|11.710514|OBJECT-d350cc2d-b979-4aca-bd73-71709832940f|(96, 127)|2024-06-04 19:03:51.108
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|product_sort|SORT|14.839713|||2024-06-04 19:03:51.144
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|global_click|INFO|11.710514|OBJECT-d350cc2d-b979-4aca-bd73-71709832940f|(96, 127)|2024-06-04 19:03:51.17
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|brand_filter|FILTER|20.876852|OBJECT-6c91da98-387b-45cb-8275-e90d1ea8bc54|supplier_name|2024-06-04 19:03:51.205
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|type_filter|INFO|15.212905|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:03:51.228
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|product_sort|SORT|14.839713|||2024-06-04 19:03:51.232
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|type_filter|INFO|15.212905|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:03:51.292
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|2071e273-513f-46be-b835-89f452095053|type_filter|INFO|15.212905|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:03:51.301
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|23f0149a-13ae-4977-8dc9-ef61c449c140|on_search|QUERY|16.93674|||2024-06-04 19:03:50.62
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|23f0149a-13ae-4977-8dc9-ef61c449c140|global_click|INFO|25.897957|OBJECT-d350cc2d-b979-4aca-bd73-71709832940f|(96, 127)|2024-06-04 19:03:50.624
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|23f0149a-13ae-4977-8dc9-ef61c449c140|product_sort|SORT|44.345097|||2024-06-04 19:03:50.688
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|23f0149a-13ae-4977-8dc9-ef61c449c140|brand_filter|FILTER|19.54417|OBJECT-6c91da98-387b-45cb-8275-e90d1ea8bc54|supplier_name|2024-06-04 19:03:50.696
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|23f0149a-13ae-4977-8dc9-ef61c449c140|type_filter|INFO|48.79312|OBJECT-32d9bb39-b17d-4611-82c1-5aaa14368060|filter_product_type|2024-06-04 19:03:50.74
+ubi-demo|33bd0ee2-60b7-4c25-b62c-1aa1580da73c|23f0149a-13ae-4977-8dc9-ef61c449c140|brand_filter|FILTER|19.54417|OBJECT-6c91da98-387b-45cb-8275-e90d1ea8bc54|supplier_name|2024-06-04 19:03:50.802
+
+
+## List user sessions for users who logged out without submitting any queries
+
+The following query searches for users who don't have an associated `query_id`. Note that this may happen if the client side does not pass the returned query to other events.
+
+```sql
+select
+ client_id, session_id, count(0) EventTotal
+from ubi_events
+where action_name='logout' and query_id is null
+group by client_id, session_id
+order by EventTotal desc
+```
+
+
+
+The following table shows the client ID, session ID, and that there was 1 event,`logout`.
+
+client_id|session_id|EventTotal
+---|---|---
+100_15c182f2-05db-4f4f-814f-46dc0de6b9ea|1c36712c-44b8-4fdd-8f0d-fdfeab5bd794_1290|1
+175_e5f262f1-0db3-4948-b349-c5b95ff31259|816f94d6-8966-4a8b-8984-a2641d5865b2_2251|1
+175_e5f262f1-0db3-4948-b349-c5b95ff31259|314dc1ff-ef38-4da4-b4b1-061f62dddcbb_2248|1
+175_e5f262f1-0db3-4948-b349-c5b95ff31259|1ce5dc30-31bb-4759-9451-5a99b28ba91b_2255|1
+175_e5f262f1-0db3-4948-b349-c5b95ff31259|10ac0fc0-409e-4ba0-98e9-edb323556b1a_2249|1
+174_ab59e589-1ae4-40be-8b29-8efd9fc15380|dfa8b38a-c451-4190-a391-2e1ec3c8f196_2228|1
+174_ab59e589-1ae4-40be-8b29-8efd9fc15380|68666e11-087a-4978-9ca7-cbac6862273e_2233|1
+174_ab59e589-1ae4-40be-8b29-8efd9fc15380|5ca7a0df-f750-4656-b9a5-5eef1466ba09_2234|1
+174_ab59e589-1ae4-40be-8b29-8efd9fc15380|228c1135-b921-45f4-b087-b3422e7ed437_2236|1
+173_39d4cbfd-0666-4e77-84a9-965ed785db49|f9795e2e-ad92-4f15-8cdd-706aa1a3a17b_2206|1
+173_39d4cbfd-0666-4e77-84a9-965ed785db49|f3c18b61-2c8a-41b3-a023-11eb2dd6c93c_2207|1
+173_39d4cbfd-0666-4e77-84a9-965ed785db49|e12f700c-ffa3-4681-90d9-146022e26a18_2210|1
+173_39d4cbfd-0666-4e77-84a9-965ed785db49|da1ff1f6-26f1-49d4-bd0d-d32d199e270e_2208|1
+173_39d4cbfd-0666-4e77-84a9-965ed785db49|a1674e9d-d2dd-4da9-a4d1-dd12a401e8e7_2216|1
+172_875f04d6-2c35-45f4-a8ac-bc5b675425f6|cc8e6174-5c1a-48c5-8ee8-1226621fe9f7_2203|1
+171_7d810730-d6e9-4079-ab1c-db7f98776985|927fcfed-61d2-4334-91e9-77442b077764_2189|1
+16_581fe410-338e-457b-a790-85af2a642356|83a68f57-0fbb-4414-852b-4c4601bf6cf2_156|1
+16_581fe410-338e-457b-a790-85af2a642356|7881141b-511b-4df9-80e6-5450415af42c_162|1
+16_581fe410-338e-457b-a790-85af2a642356|1d64478e-c3a6-4148-9a64-b6f4a73fc684_158|1
+
+
+
+You may want to identify users who logged out multiple times without submitting a query. The following query lets you see which users do this the most:
+
+```sql
+select
+ client_id, count(0) EventTotal
+from ubi_events
+where action_name='logout' and query_id is null
+group by client_id
+order by EventTotal desc
+```
+
+The following table shows user client IDs and the number of logouts without any queries.
+
+client_id|EventTotal
+---|---
+87_5a6e1f8c-4936-4184-a24d-beddd05c9274|8
+127_829a4246-930a-4b24-8165-caa07ee3fa47|7
+49_5da537a3-8d94-48d1-a0a4-dcad21c12615|6
+56_6c7c2525-9ca5-4d5d-8ac0-acb43769ac0b|6
+140_61192c8e-c532-4164-ad1b-1afc58c265b7|6
+149_3443895e-6f81-4706-8141-1ebb0c2470ca|6
+196_4359f588-10be-4b2c-9e7f-ee846a75a3f6|6
+173_39d4cbfd-0666-4e77-84a9-965ed785db49|5
+52_778ac7f3-8e60-444e-ad40-d24516bf4ce2|5
+51_6335e0c3-7bea-4698-9f83-25c9fb984e12|5
+175_e5f262f1-0db3-4948-b349-c5b95ff31259|5
+61_feb3a495-c1fb-40ea-8331-81cee53a5eb9|5
+181_f227264f-cabd-4468-bfcc-4801baeebd39|5
+185_435d1c63-4829-45f3-abff-352ef6458f0e|5
+100_15c182f2-05db-4f4f-814f-46dc0de6b9ea|5
+113_df32ed6e-d74a-4956-ac8e-6d43d8d60317|5
+151_0808111d-07ce-4c84-a0fd-7125e4e33020|5
+204_b75e374c-4813-49c4-b111-4bf4fdab6f26|5
+29_ec2133e5-4d9b-4222-aa7c-2a9ae0880ddd|5
+41_f64abc69-56ea-4dd3-a991-7d1fd292a530|5
diff --git a/_search-plugins/ubi/ubi-dashboard-tutorial.md b/_search-plugins/ubi/ubi-dashboard-tutorial.md
new file mode 100644
index 0000000000..ac6cd72186
--- /dev/null
+++ b/_search-plugins/ubi/ubi-dashboard-tutorial.md
@@ -0,0 +1,94 @@
+---
+layout: default
+title: UBI dashboard tutorial
+parent: User Behavior Insights
+has_children: false
+nav_order: 25
+---
+
+
+# UBI dashboard tutorial
+
+Whether you've been collecting user events and queries for a while or [you've uploaded some sample events](https://github.com/o19s/chorus-OpenSearch-edition/blob/main/katas/003_import_preexisting_event_data.md), you're now ready to visualize the data collected through User Behavior Insights (UBI) in a dashboard in OpenSearch.
+
+To quickly view a dashboard without completing the full tutorial, do the following:
+1. Download and save the [sample UBI dashboard]({{site.url}}{{site.baseurl}}/assets/examples/ubi-dashboard.ndjson).
+1. On the top menu, go to **Management > Dashboard Management**.
+1. In the **Dashboards** panel, choose **Saved objects**.
+1. In the upper-right corner, select **Import**.
+1. In the **Select file** panel, choose **Import**.
+1. Select the UBI dashboard file that you downloaded and select the **Import** button.
+
+## 1. Start OpenSearch Dashboards
+
+Start OpenSearch Dashboards. For example, go to `http://{server}:5601/app/home#/`. For more information, see [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/). The following image shows the home page.
+
+
+## 2. Create an index pattern
+
+In OpenSearch Management, navigate to **Dashboards Management > Index patterns** or navigate using a URL, such as `http://{server}:5601/app/management/OpenSearch-dashboards/indexPatterns`.
+
+OpenSearch Dashboards accesses your indexes using index patterns. To visualize your users' online search behavior, you must create an index pattern in order to access the indexes that UBI creates. For more information, see [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/).
+
+After you select **Create index pattern**, a list of indexes in your OpenSearch instance is displayed. The UBI stores may be hidden by default, so make sure to select **Include system and hidden indexes**, as shown in the following image.
+
+
+You can group indexes into the same data source for your dashboard using wildcards. For this tutorial you'll combine the query and event stores into the `ubi_*` pattern.
+
+OpenSearch Dashboards prompts you to filter on any `date` field in your schema so that you can look at things like trending queries over the last 15 minutes. However, for your first dashboard, select **I don't want to use the time filter**, as shown in the following image.
+
+
+
+After selecting **Create index pattern**, you're ready to start building a dashboard that displays the UBI store data.
+
+## 3. Create a new dashboard
+
+To create a new dashboard, on the top menu, select **OpenSearch Dashboards > Dashboards** and then **Create > Dashboard** > **Create new**.
+If you haven't previously created a dashboard, you are presented with the option to create a new dashboard. Otherwise, previously created dashboards are displayed.
+
+
+In the **New Visualization** window, select **Pie** to create a new pie chart. Then select the index pattern you created in step 2.
+
+Most visualizations require some sort of aggregate function on a bucket/facet/aggregatable field (numeric or keyword). You'll add a `Terms` aggregation to the `action_name` field so that you can view the distribution of event names. Change the **Size** to the number of slices you want to display, as shown in the following image.
+
+
+Save the visualization so that it's added to your new dashboard. Now that you have a visualization displayed on your dashboard, you can save the dashboard.
+
+## 4. Add a tag cloud visualization
+
+Now you'll add a word cloud for trending searches by creating a new visualization, similarly to the previous step.
+
+In the **New Visualization** window, select **Tag Cloud**, and then select the index pattern you created in step 2. Choose the tag cloud visualization of the terms in the `message` field where the JavaScript client logs the raw search text. Note: The true query, as processed by OpenSearch with filters, boosting, and so on, resides in the `ubi_queries` index. However, you'll view the `message` field of the `ubi_events` index, where the JavaScript client captures the text that the user actually typed.
+
+The following image shows the tag cloud visualization on the `message` field.
+
+
+The underlying queries can be found at [SQL trending queries]({{site.url}}{{site.baseurl}}/search-plugins/ubi/sql-queries/#trending-queries).
+{: .note}
+
+
+The resulting visualization may contain different information than you're looking for. The `message` field is updated with every event, and as a result, it can contain error messages, debug messages, click information, and other unwanted data.
+To view only search terms for query events, you need to add a filter to your visualization. Because during setup you provided a `message_type` of `QUERY` for each search event, you can filter by that message type to isolate the specific users' searches. To do this, select **Add filter** and then select **QUERY** in the **Edit filter** panel, as shown in the following image.
+
+
+There should now be two visualizations (the pie chart and the tag cloud) displayed on your dashboard, as shown in the following image.
+
+
+## 5. Add a histogram of item clicks
+
+Now you'll add a histogram visualization to your dashboard, similarly to the previous step. In the **New Visualization** window, select **Vertical Bar**. Then select the index pattern you created in step 2.
+
+Examine the `event_attributes.position.ordinal` data field. This field contains the position of the item in a list selected by the user. For the histogram visualization, the x-axis represents the ordinal number of the selected item (n). The y-axis represents the number of times that the nth item was clicked, as shown in the following image.
+
+
+
+## 6) Filter the displayed data
+
+Now you can further filter the displayed data. For example, you can see how the click position changes when a purchase occurs. Select **Add filter** and then select the `action_name:product_purchase` field, as shown in the following image.
+
+
+
+You can filter event messages containing the word `*laptop*` by adding wildcards, as shown in the following image.
+.
+
+
diff --git a/_search-plugins/vector-search.md b/_search-plugins/vector-search.md
new file mode 100644
index 0000000000..862b26b375
--- /dev/null
+++ b/_search-plugins/vector-search.md
@@ -0,0 +1,283 @@
+---
+layout: default
+title: Vector search
+nav_order: 22
+has_children: false
+has_toc: false
+---
+
+# Vector search
+
+OpenSearch is a comprehensive search platform that supports a variety of data types, including vectors. OpenSearch vector database functionality is seamlessly integrated with its generic database function.
+
+In OpenSearch, you can generate vector embeddings, store those embeddings in an index, and use them for vector search. Choose one of the following options:
+
+- Generate embeddings using a library of your choice before ingesting them into OpenSearch. Once you ingest vectors into an index, you can perform a vector similarity search on the vector space. For more information, see [Working with embeddings generated outside of OpenSearch](#working-with-embeddings-generated-outside-of-opensearch).
+- Automatically generate embeddings within OpenSearch. To use embeddings for semantic search, the ingested text (the corpus) and the query need to be embedded using the same model. [Neural search]({{site.url}}{{site.baseurl}}/search-plugins/neural-search/) packages this functionality, eliminating the need to manage the internal details. For more information, see [Generating vector embeddings within OpenSearch](#generating-vector-embeddings-in-opensearch).
+
+## Working with embeddings generated outside of OpenSearch
+
+After you generate vector embeddings, upload them to an OpenSearch index and search the index using vector search. For a complete example, see [Example](#example).
+
+### k-NN index
+
+To build a vector database and use vector search, you must specify your index as a [k-NN index]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/) when creating it by setting `index.knn` to `true`:
+
+```json
+PUT test-index
+{
+ "settings": {
+ "index": {
+ "knn": true,
+ "knn.algo_param.ef_search": 100
+ }
+ },
+ "mappings": {
+ "properties": {
+ "my_vector1": {
+ "type": "knn_vector",
+ "dimension": 1024,
+ "method": {
+ "name": "hnsw",
+ "space_type": "l2",
+ "engine": "nmslib",
+ "parameters": {
+ "ef_construction": 128,
+ "m": 24
+ }
+ }
+ }
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+### k-NN vector
+
+You must designate the field that will store vectors as a [`knn_vector`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-vector/) field type. OpenSearch supports vectors of up to 16,000 dimensions, each of which is represented as a 32-bit or 16-bit float.
+
+To save storage space, you can use `byte` vectors. For more information, see [Lucene byte vector]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-vector#lucene-byte-vector).
+
+### k-NN vector search
+
+Vector search finds the vectors in your database that are most similar to the query vector. OpenSearch supports the following search methods:
+
+- [Approximate search](#approximate-search) (approximate k-NN, or ANN): Returns approximate nearest neighbors to the query vector. Usually, approximate search algorithms sacrifice indexing speed and search accuracy in exchange for performance benefits such as lower latency, smaller memory footprints, and more scalable search. For most use cases, approximate search is the best option.
+
+- Exact search (exact k-NN): A brute-force, exact k-NN search of vector fields. OpenSearch supports the following types of exact search:
+ - [Exact k-NN with scoring script]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-score-script/): Using the k-NN scoring script, you can apply a filter to an index before executing the nearest neighbor search.
+ - [Painless extensions]({{site.url}}{{site.baseurl}}/search-plugins/knn/painless-functions/): Adds the distance functions as Painless extensions that you can use in more complex combinations. You can use this method to perform a brute-force, exact k-NN search of an index, which also supports pre-filtering.
+
+### Approximate search
+
+OpenSearch supports several algorithms for approximate vector search, each with its own advantages. For complete documentation, see [Approximate search]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/). For more information about the search methods and engines, see [Method definitions]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/#method-definitions). For method recommendations, see [Choosing the right method]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/#choosing-the-right-method).
+
+To use approximate vector search, specify one of the following search methods (algorithms) in the `method` parameter:
+
+- Hierarchical Navigable Small World (HNSW)
+- Inverted File System (IVF)
+
+Additionally, specify the engine (library) that implements this method in the `engine` parameter:
+
+- [Non-Metric Space Library (NMSLIB)](https://github.com/nmslib/nmslib)
+- [Facebook AI Similarity Search (Faiss)](https://github.com/facebookresearch/faiss)
+- Lucene
+
+The following table lists the combinations of search methods and libraries supported by the k-NN engine for approximate vector search.
+
+Method | Engine
+:--- | :---
+HNSW | NMSLIB, Faiss, Lucene
+IVF | Faiss
+
+### Engine recommendations
+
+In general, select NMSLIB or Faiss for large-scale use cases. Lucene is a good option for smaller deployments and offers benefits like smart filtering, where the optimal filtering strategy—pre-filtering, post-filtering, or exact k-NN—is automatically applied depending on the situation. The following table summarizes the differences between each option.
+
+| | NMSLIB/HNSW | Faiss/HNSW | Faiss/IVF | Lucene/HNSW |
+|:---|:---|:---|:---|:---|
+| Max dimensions | 16,000 | 16,000 | 16,000 | 1,024 |
+| Filter | Post-filter | Post-filter | Post-filter | Filter during search |
+| Training required | No | No | Yes | No |
+| Similarity metrics | `l2`, `innerproduct`, `cosinesimil`, `l1`, `linf` | `l2`, `innerproduct` | `l2`, `innerproduct` | `l2`, `cosinesimil` |
+| Number of vectors | Tens of billions | Tens of billions | Tens of billions | Less than 10 million |
+| Indexing latency | Low | Low | Lowest | Low |
+| Query latency and quality | Low latency and high quality | Low latency and high quality | Low latency and low quality | High latency and high quality |
+| Vector compression | Flat | Flat
Product quantization | Flat
Product quantization | Flat |
+| Memory consumption | High | High
Low with PQ | Medium
Low with PQ | High |
+
+### Example
+
+In this example, you'll create a k-NN index, add data to the index, and search the data.
+
+#### Step 1: Create a k-NN index
+
+First, create an index that will store sample hotel data. Set `index.knn` to `true` and specify the `location` field as a `knn_vector`:
+
+```json
+PUT /hotels-index
+{
+ "settings": {
+ "index": {
+ "knn": true,
+ "knn.algo_param.ef_search": 100,
+ "number_of_shards": 1,
+ "number_of_replicas": 0
+ }
+ },
+ "mappings": {
+ "properties": {
+ "location": {
+ "type": "knn_vector",
+ "dimension": 2,
+ "method": {
+ "name": "hnsw",
+ "space_type": "l2",
+ "engine": "lucene",
+ "parameters": {
+ "ef_construction": 100,
+ "m": 16
+ }
+ }
+ }
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+#### Step 2: Add data to your index
+
+Next, add data to your index. Each document represents a hotel. The `location` field in each document contains a vector specifying the hotel's location:
+
+```json
+POST /_bulk
+{ "index": { "_index": "hotels-index", "_id": "1" } }
+{ "location": [5.2, 4.4] }
+{ "index": { "_index": "hotels-index", "_id": "2" } }
+{ "location": [5.2, 3.9] }
+{ "index": { "_index": "hotels-index", "_id": "3" } }
+{ "location": [4.9, 3.4] }
+{ "index": { "_index": "hotels-index", "_id": "4" } }
+{ "location": [4.2, 4.6] }
+{ "index": { "_index": "hotels-index", "_id": "5" } }
+{ "location": [3.3, 4.5] }
+```
+{% include copy-curl.html %}
+
+#### Step 3: Search your data
+
+Now search for hotels closest to the pin location `[5, 4]`. This location is labeled `Pin` in the following image. Each hotel is labeled with its document number.
+
+
+
+To search for the top three closest hotels, set `k` to `3`:
+
+```json
+POST /hotels-index/_search
+{
+ "size": 3,
+ "query": {
+ "knn": {
+ "location": {
+ "vector": [
+ 5,
+ 4
+ ],
+ "k": 3
+ }
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the hotels closest to the specified pin location:
+
+```json
+{
+ "took": 1093,
+ "timed_out": false,
+ "_shards": {
+ "total": 1,
+ "successful": 1,
+ "skipped": 0,
+ "failed": 0
+ },
+ "hits": {
+ "total": {
+ "value": 3,
+ "relation": "eq"
+ },
+ "max_score": 0.952381,
+ "hits": [
+ {
+ "_index": "hotels-index",
+ "_id": "2",
+ "_score": 0.952381,
+ "_source": {
+ "location": [
+ 5.2,
+ 3.9
+ ]
+ }
+ },
+ {
+ "_index": "hotels-index",
+ "_id": "1",
+ "_score": 0.8333333,
+ "_source": {
+ "location": [
+ 5.2,
+ 4.4
+ ]
+ }
+ },
+ {
+ "_index": "hotels-index",
+ "_id": "3",
+ "_score": 0.72992706,
+ "_source": {
+ "location": [
+ 4.9,
+ 3.4
+ ]
+ }
+ }
+ ]
+ }
+}
+```
+
+### Vector search with filtering
+
+For information about vector search with filtering, see [k-NN search with filters]({{site.url}}{{site.baseurl}}/search-plugins/knn/filter-search-knn/).
+
+## Generating vector embeddings in OpenSearch
+
+[Neural search]({{site.url}}{{site.baseurl}}/search-plugins/neural-search/) encapsulates the infrastructure needed to perform semantic vector searches. After you integrate an inference (embedding) service, neural search functions like lexical search, accepting a textual query and returning relevant documents.
+
+When you index your data, neural search transforms text into vector embeddings and indexes both the text and its vector embeddings in a vector index. When you use a neural query during search, neural search converts the query text into vector embeddings and uses vector search to return the results.
+
+### Choosing a model
+
+The first step in setting up neural search is choosing a model. You can upload a model to your OpenSearch cluster, use one of the pretrained models provided by OpenSearch, or connect to an externally hosted model. For more information, see [Integrating ML models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/integrating-ml-models/).
+
+### Neural search tutorial
+
+For a step-by-step tutorial, see [Neural search tutorial]({{site.url}}{{site.baseurl}}/search-plugins/neural-search-tutorial/).
+
+### Search methods
+
+Choose one of the following search methods to use your model for neural search:
+
+- [Semantic search]({{site.url}}{{site.baseurl}}/search-plugins/semantic-search/): Uses dense retrieval based on text embedding models to search text data.
+
+- [Hybrid search]({{site.url}}{{site.baseurl}}/search-plugins/hybrid-search/): Combines lexical and neural search to improve search relevance.
+
+- [Multimodal search]({{site.url}}{{site.baseurl}}/search-plugins/multimodal-search/): Uses neural search with multimodal embedding models to search text and image data.
+
+- [Neural sparse search]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/): Uses neural search with sparse retrieval based on sparse embedding models to search text data.
+
+- [Conversational search]({{site.url}}{{site.baseurl}}/search-plugins/conversational-search/): With conversational search, you can ask questions in natural language, receive a text response, and ask additional clarifying questions.
diff --git a/_security/access-control/document-level-security.md b/_security/access-control/document-level-security.md
index 08de85bbf7..352fe06a61 100644
--- a/_security/access-control/document-level-security.md
+++ b/_security/access-control/document-level-security.md
@@ -191,6 +191,10 @@ Adaptive | `adaptive-level` | The default setting that allows OpenSearch to auto
OpenSearch combines all DLS queries with the logical `OR` operator. However, when a role that uses DLS is combined with another security role that doesn't use DLS, the query results are filtered to display only documents matching the DLS from the first role. This filter rule also applies to roles that do not grant read documents.
+### DLS and write permissions
+
+Make sure that a user that has DLS-configured roles does not have write permissions. If write permissions are added, the user will be able to index documents which they will not be able to retrieve due to DLS filtering.
+
### When to enable `plugins.security.dfm_empty_overrides_all`
When to enable the `plugins.security.dfm_empty_overrides_all` setting depends on whether you want to restrict user access to documents without DLS.
diff --git a/_security/audit-logs/storage-types.md b/_security/audit-logs/storage-types.md
index c0707ff424..719287ad7f 100644
--- a/_security/audit-logs/storage-types.md
+++ b/_security/audit-logs/storage-types.md
@@ -53,8 +53,8 @@ If you use `external_opensearch` and the remote cluster also uses the Security p
Name | Data type | Description
:--- | :--- | :---
-`plugins.security.audit.config.enable_ssl` | Boolean | If you enabled SSL/TLS on the receiving cluster, set to true. The default is false.
-`plugins.security.audit.config.verify_hostnames` | Boolean | Whether to verify the hostname of the SSL/TLS certificate of the receiving cluster. Default is true.
+`plugins.security.audit.config.enable_ssl` | Boolean | If you enabled SSL/TLS on the receiving cluster, set to true. The Default is `false`.
+`plugins.security.audit.config.verify_hostnames` | Boolean | Whether to verify the hostname of the SSL/TLS certificate of the receiving cluster. Default is `true`.
`plugins.security.audit.config.pemtrustedcas_filepath` | String | The trusted root certificate of the external OpenSearch cluster, relative to the `config` directory.
`plugins.security.audit.config.pemtrustedcas_content` | String | Instead of specifying the path (`plugins.security.audit.config.pemtrustedcas_filepath`), you can configure the Base64-encoded certificate content directly.
`plugins.security.audit.config.enable_ssl_client_auth` | Boolean | Whether to enable SSL/TLS client authentication. If you set this to true, the audit log module sends the node's certificate along with the request. The receiving cluster can use this certificate to verify the identity of the caller.
diff --git a/_security/authentication-backends/jwt.md b/_security/authentication-backends/jwt.md
index b6b08388b5..3f28dfecfd 100644
--- a/_security/authentication-backends/jwt.md
+++ b/_security/authentication-backends/jwt.md
@@ -122,7 +122,7 @@ Name | Description
`jwt_url_parameter` | If the token is not transmitted in the HTTP header but rather as an URL parameter, define the name of the parameter here.
`subject_key` | The key in the JSON payload that stores the username. If not set, the [subject](https://tools.ietf.org/html/rfc7519#section-4.1.2) registered claim is used.
`roles_key` | The key in the JSON payload that stores the user's roles. The value of this key must be a comma-separated list of roles.
-`required_audience` | The name of the audience which the JWT must specify. This corresponds [`aud` claim of the JWT](https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.3).
+`required_audience` | The name of the audience that the JWT must specify. You can set a single value (for example, `project1`) or multiple comma-separated values (for example, `project1,admin`). If you set multiple values, the JWT must have at least one required audience. This parameter corresponds to the [`aud` claim of the JWT](https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.3).
`required_issuer` | The target issuer of JWT stored in the JSON payload. This corresponds to the [`iss` claim of the JWT](https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.1).
`jwt_clock_skew_tolerance_seconds` | Sets a window of time, in seconds, to compensate for any disparity between the JWT authentication server and OpenSearch node clock times, thereby preventing authentication failures due to the misalignment. Security sets 30 seconds as the default. Use this setting to apply a custom value.
diff --git a/_security/authentication-backends/ldap.md b/_security/authentication-backends/ldap.md
index 49b01e332b..9f98f7f5b0 100755
--- a/_security/authentication-backends/ldap.md
+++ b/_security/authentication-backends/ldap.md
@@ -61,8 +61,21 @@ We provide a fully functional example that can help you understand how to use an
To enable LDAP authentication and authorization, add the following lines to `config/opensearch-security/config.yml`:
+The internal user database authentication should also be enabled because OpenSearch Dashboards connects to OpenSearch using the `kibanaserver` internal user.
+{: .note}
+
```yml
authc:
+ internal_auth:
+ order: 0
+ description: "HTTP basic authentication using the internal user database"
+ http_enabled: true
+ transport_enabled: true
+ http_authenticator:
+ type: basic
+ challenge: false
+ authentication_backend:
+ type: internal
ldap:
http_enabled: true
transport_enabled: true
diff --git a/_security/authentication-backends/openid-connect.md b/_security/authentication-backends/openid-connect.md
index 8efb66fbb6..8e785a9e65 100755
--- a/_security/authentication-backends/openid-connect.md
+++ b/_security/authentication-backends/openid-connect.md
@@ -181,8 +181,8 @@ config:
Name | Description
:--- | :---
-`enable_ssl` | Whether to use TLS. Default is false.
-`verify_hostnames` | Whether to verify the hostnames of the IdP's TLS certificate. Default is true.
+`enable_ssl` | Whether to use TLS. Default is `false`.
+`verify_hostnames` | Whether to verify the hostnames of the IdP's TLS certificate. Default is `true`.
### Certificate validation
@@ -252,7 +252,7 @@ config:
Name | Description
:--- | :---
-`enable_ssl_client_auth` | Whether to send the client certificate to the IdP server. Default is false.
+`enable_ssl_client_auth` | Whether to send the client certificate to the IdP server. Default is `false`.
`pemcert_filepath` | Absolute path to the client certificate.
`pemcert_content` | The content of the client certificate. Cannot be used when `pemcert_filepath` is set.
`pemkey_filepath` | Absolute path to the file containing the private key of the client certificate.
diff --git a/_security/authentication-backends/proxy.md b/_security/authentication-backends/proxy.md
index bb7d1f0151..7716b1d6d2 100644
--- a/_security/authentication-backends/proxy.md
+++ b/_security/authentication-backends/proxy.md
@@ -40,7 +40,7 @@ You can configure the following settings:
Name | Description
:--- | :---
-`enabled` | Enables or disables proxy support. Default is false.
+`enabled` | Enables or disables proxy support. Default is `false`.
`internalProxies` | A regular expression containing the IP addresses of all trusted proxies. The pattern `.*` trusts all internal proxies.
`remoteIpHeader` | Name of the HTTP header field that has the hostname chain. Default is `x-forwarded-for`.
diff --git a/_security/authentication-backends/saml.md b/_security/authentication-backends/saml.md
index a4511a5325..652345ccdc 100755
--- a/_security/authentication-backends/saml.md
+++ b/_security/authentication-backends/saml.md
@@ -244,7 +244,7 @@ If you are loading the IdP metadata from a URL, we recommend that you use SSL/TL
Name | Description
:--- | :---
-`idp.enable_ssl` | Whether to enable the custom TLS configuration. Default is false (JDK settings are used).
+`idp.enable_ssl` | Whether to enable the custom TLS configuration. Default is `false` (JDK settings are used).
`idp.verify_hostnames` | Whether to verify the hostnames of the server's TLS certificate.
Example:
@@ -302,7 +302,7 @@ The Security plugin can use TLS client authentication when fetching the IdP meta
Name | Description
:--- | :---
-`idp.enable_ssl_client_auth` | Whether to send a client certificate to the IdP server. Default is false.
+`idp.enable_ssl_client_auth` | Whether to send a client certificate to the IdP server. Default is `false`.
`idp.pemcert_filepath` | Path to the PEM file containing the client certificate. The file must be placed under the OpenSearch `config` directory, and the path must be specified relative to the `config` directory.
`idp.pemcert_content` | The content of the client certificate. Cannot be used when `pemcert_filepath` is set.
`idp.pemkey_filepath` | Path to the private key of the client certificate. The file must be placed under the OpenSearch `config` directory, and the path must be specified relative to the `config` directory.
diff --git a/_security/configuration/security-admin.md b/_security/configuration/security-admin.md
index ed293b7e91..77d3711385 100755
--- a/_security/configuration/security-admin.md
+++ b/_security/configuration/security-admin.md
@@ -201,7 +201,7 @@ Name | Description
`-cn` | Cluster name. Default is `opensearch`.
`-icl` | Ignore cluster name.
`-sniff` | Sniff cluster nodes. Sniffing detects available nodes using the OpenSearch `_cluster/state` API.
-`-arc,--accept-red-cluster` | Execute `securityadmin.sh` even if the cluster state is red. Default is false, which means the script will not execute on a red cluster.
+`-arc,--accept-red-cluster` | Execute `securityadmin.sh` even if the cluster state is red. Default is `false`, which means the script will not execute on a red cluster.
### Certificate validation settings
@@ -210,7 +210,7 @@ Use the following options to control certificate validation.
Name | Description
:--- | :---
-`-nhnv` | Do not validate hostname. Default is false.
+`-nhnv` | Do not validate hostname. Default is `false`.
`-nrhn` | Do not resolve hostname. Only relevant if `-nhnv` is not set.
diff --git a/_security/configuration/tls.md b/_security/configuration/tls.md
index d06b16a47e..a4115b8c25 100755
--- a/_security/configuration/tls.md
+++ b/_security/configuration/tls.md
@@ -52,11 +52,11 @@ The following settings configure the location and password of your keystore and
Name | Description
:--- | :---
-`plugins.security.ssl.transport.keystore_type` | The type of the keystore file, JKS or PKCS12/PFX. Optional. Default is JKS.
+`plugins.security.ssl.transport.keystore_type` | The type of the keystore file, `JKS` or `PKCS12/PFX`. Optional. Default is `JKS`.
`plugins.security.ssl.transport.keystore_filepath` | Path to the keystore file, which must be under the `config` directory, specified using a relative path. Required.
`plugins.security.ssl.transport.keystore_alias` | The alias name of the keystore. Optional. Default is the first alias.
`plugins.security.ssl.transport.keystore_password` | Keystore password. Default is `changeit`.
-`plugins.security.ssl.transport.truststore_type` | The type of the truststore file, JKS or PKCS12/PFX. Default is JKS.
+`plugins.security.ssl.transport.truststore_type` | The type of the truststore file, `JKS` or `PKCS12/PFX`. Default is `JKS`.
`plugins.security.ssl.transport.truststore_filepath` | Path to the truststore file, which must be under the `config` directory, specified using a relative path. Required.
`plugins.security.ssl.transport.truststore_alias` | The alias name of the truststore. Optional. Default is all certificates.
`plugins.security.ssl.transport.truststore_password` | Truststore password. Default is `changeit`.
@@ -65,7 +65,7 @@ Name | Description
Name | Description
:--- | :---
-`plugins.security.ssl.http.enabled` | Whether to enable TLS on the REST layer. If enabled, only HTTPS is allowed. Optional. Default is false.
+`plugins.security.ssl.http.enabled` | Whether to enable TLS on the REST layer. If enabled, only HTTPS is allowed. Optional. Default is `false`.
`plugins.security.ssl.http.keystore_type` | The type of the keystore file, JKS or PKCS12/PFX. Optional. Default is JKS.
`plugins.security.ssl.http.keystore_filepath` | Path to the keystore file, which must be under the `config` directory, specified using a relative path. Required.
`plugins.security.ssl.http.keystore_alias` | The alias name of the keystore. Optional. Default is the first alias.
@@ -137,7 +137,7 @@ plugins.security.authcz.admin_dn:
For security reasons, you cannot use wildcards or regular expressions as values for the `admin_dn` setting.
-For more information about admin and super admin user roles, see [Admin and super admin roles](https://opensearch.org/docs/latest/security/access-control/users-roles/#admin-and-super-admin-roles) and [Configuring super admin certificates](https://opensearch.org/docs/latest/security/configuration/tls/#configuring-admin-certificates).
+For more information about admin and super admin user roles, see [Admin and super admin roles](https://opensearch.org/docs/latest/security/access-control/users-roles/#admin-and-super-admin-roles).
## (Advanced) OpenSSL
@@ -150,8 +150,8 @@ If OpenSSL is enabled, but for one reason or another the installation does not w
Name | Description
:--- | :---
-`plugins.security.ssl.transport.enable_openssl_if_available` | Enable OpenSSL on the transport layer if available. Optional. Default is true.
-`plugins.security.ssl.http.enable_openssl_if_available` | Enable OpenSSL on the REST layer if available. Optional. Default is true.
+`plugins.security.ssl.transport.enable_openssl_if_available` | Enable OpenSSL on the transport layer if available. Optional. Default is `true`.
+`plugins.security.ssl.http.enable_openssl_if_available` | Enable OpenSSL on the REST layer if available. Optional. Default is `true`.
{% comment %}
1. Install [OpenSSL 1.1.0](https://www.openssl.org/community/binaries.html) on every node.
@@ -179,8 +179,8 @@ In addition, when `resolve_hostname` is enabled, the Security plugin resolves th
Name | Description
:--- | :---
-`plugins.security.ssl.transport.enforce_hostname_verification` | Whether to verify hostnames on the transport layer. Optional. Default is true.
-`plugins.security.ssl.transport.resolve_hostname` | Whether to resolve hostnames against DNS on the transport layer. Optional. Default is true. Only works if hostname verification is also enabled.
+`plugins.security.ssl.transport.enforce_hostname_verification` | Whether to verify hostnames on the transport layer. Optional. Default is `true`.
+`plugins.security.ssl.transport.resolve_hostname` | Whether to resolve hostnames against DNS on the transport layer. Optional. Default is `true`. Only works if hostname verification is also enabled.
## (Advanced) Client authentication
diff --git a/_tuning-your-cluster/availability-and-recovery/snapshots/sm-api.md b/_tuning-your-cluster/availability-and-recovery/snapshots/sm-api.md
index cd3a238f9c..5d89a3747b 100644
--- a/_tuning-your-cluster/availability-and-recovery/snapshots/sm-api.md
+++ b/_tuning-your-cluster/availability-and-recovery/snapshots/sm-api.md
@@ -181,7 +181,7 @@ Parameter | Type | Description
`enabled` | Boolean | Should this SM policy be enabled at creation? Optional.
`snapshot_config` | Object | The configuration options for snapshot creation. Required.
`snapshot_config.date_format` | String | Snapshot names have the format `