+
{:/} icon.
-An example data source connection screen is shown in the following image.
+A data source connection interface is shown in the following image.
@@ -93,13 +96,15 @@ To select a data source through the Dev Tools console, follow these steps:
5. From the **Data source** dropdown menu, select a data source and then query the source.
6. Repeat the preceding steps for each data source you want to select.
-### Upload saved objects to a dashboard from connected data sources
+---
+
+## Uploading saved objects to a dashboard from connected data sources
To upload saved objects from connected data sources to a dashboard with multiple data sources, export them as an NDJSON file from the data source's **Saved object management** page. Then upload the file to the dashboard's **Saved object management** page. This method can simplify the transfer of saved objects between dashboards. The following 20-second video shows this feature in action.
{: .img-fluid}
-#### Import saved objects from a connected data source
+### Importing saved objects from a connected data source
Follow these steps to import saved objects from a connected data source:
@@ -109,11 +114,13 @@ Follow these steps to import saved objects from a connected data source:
4. Select **Import** > **Select file** and upload the file acquired from the connected data source.
5. Choose the appropriate **Data source** from the dropdown menu, set your **Conflict management** option, and then select the **Import** button.
-### Show or hide authentication methods for multiple data sources
+---
+
+## Showing or hiding authentication methods
Introduced 2.13
{: .label .label-purple }
-A feature flag in your `opensearch_dashboards.yml` file allows you to show or hide authentication methods within the `data_source` plugin. The following example setting, shown in a 10-second demo, hides the authentication method for `AWSSigV4`.
+A feature flag in your `opensearch_dashboards.yml` file allows you to show or hide authentication methods within the `data_source` plugin. The following setting hides the authentication method for `AWSSigV4`.
````
# Set enabled to false to hide the authentication method from multiple data source in OpenSearch Dashboards.
@@ -128,89 +135,212 @@ data_source.authTypes:
enabled: false
````
+The following demo shows this process.
+
{: .img-fluid}
-### Hide the local cluster option for multiple data sources
+## Showing or hiding the local cluster
Introduced 2.13
{: .label .label-purple }
-A feature flag in your `opensearch_dashboards.yml` file allows you to hide the local cluster option within the `data_source` plugin. This option hides the local cluster from the data source dropdown menu and index creation page, which is ideal for environments with or without a local OpenSearch cluster. The following example setting, shown in a 20-second demo, hides the local cluster.
+A feature flag in your `opensearch_dashboards.yml` file allows you to hide the local cluster option within the `data_source` plugin. This option hides the local cluster from the data source dropdown menu and index creation page, which is ideal for environments with or without a local OpenSearch cluster. The following example setting, shown in a 20-second demo, hides the local cluster:
````
-# hide local cluster in the data source dropdown and index pattern creation page.
+# hide local cluster in the data source dropdown and index pattern creation page.
data_source.hideLocalCluster: true
````
+The following demo shows this process.
+
{: .img-fluid}
+---
+
## Using multiple data sources with external dashboards plugins
Introduced 2.14
-{: .label .label-purple }
+{: .label .label-purple}
-The following plugins now support multiple data sources
+The following plugins now support multiple data sources.
### Index management
-When the data source feature is enabled, you can navigate to **Index Management** under the **Management** menu. Using indexes as an example, you can view all connected data sources and select a specific one from the navigation bar on the upper right. By default, the indexes from the designated default data source are displayed. However, you can select any connected data source to view its corresponding indexes. The following GIF illustrates these steps.
+When you set `data_source.enabled:true`, you can view and select data sources and their associated indexes directly from the interface:
+
+1. Navigate to **Management** > **Index Management** under the main menu.
+2. Select **Indexes** from the sidebar menu and then select the {::nomarkdown}
{:/} icon on the upper-right menu bar.
+3. Choose the appropriate data source from the dropdown menu and then choose the appropriate index from the list. By default, the indexes from your default data source are displayed. You can choose any connected data source to view its corresponding indexes.
+
+The following GIF illustrates these steps.
-To perform operations on a specific index within a data source, select the individual index from the list. To create a new index, select the **Create Index** button, which opens a form. Fill in the required information and select the **Create** button. The index is created within the selected data source. The following GIF illustrates these steps.
+To perform operations on a specific index within a data source, select the individual index from the list. To create a new index, select the **Create Index** button, which opens a form. Enter the required information and select the **Create** button. The index is created within the selected data source. The following GIF illustrates these steps.
### Anomaly detection
-When the data source feature is enabled, you can navigate to **Anomaly Detection** under the **OpenSearch Plugins** menu. On the navigation bar on the upper right, you can view all connected data sources and select a specific data source to view the dashboard from that source if it has detectors. If the selected data source does not have any detectors, the page prompts you to **Create detector**. The following GIF illustrates these steps.
+When you set `data_source.enabled:true`, you can create or view detectors associated with a data source:
+
+1. Navigate to **OpenSearch Plugins** > **Anomaly Detection** under the main menu.
+2. Select the database icon on the upper-right menu bar to view a list of connected data sources.
+3. Select a data source to view a list of associated detectors. If the selected data source does not have detectors, then the **Create detector** button appears under the upper-right menu bar. See [Creating anomaly detectors]({{site.url}}{{site.baseurl}}/observing-your-data/ad/dashboards-anomaly-detection/#creating-anomaly-detectors) for instructions on creating detectors through the interface.
+
+The following GIF illustrates these steps.
-When you select **Detectors** from the side bar, the page displays the detectors currently configured for the selected data source. You can view and configure individual detectors by selecting them from the list. The following GIF illustrates these steps.
+You can edit the data source's associated detectors on the **Detectors** tab under the left side bar.
+
+1. Select **Detectors** and then select the {::nomarkdown}
### Security
-When the data source feature is enabled, you can navigate to **Security** under the **Management** menu. Using role management as an example, you can view all connected data sources in the navigation bar on the upper right and select a specific data source to view its existing roles. To create a new role, select the **Create role** button, which takes you to a new page. Enter the required information and select **Create** to add the new role to the selected data source. The following GIF illustrates these steps.
+When you set `data_source.enabled:true`, you can view and manage roles for each connected data source:
+
+1. Navigate to **Management** > **Security** under the main menu.
+2. Select **Roles** from the left sidebar menu and then select the {::nomarkdown}
### Maps
-When the data source feature is enabled, you can navigate to **Maps** under the **OpenSearch Plugins** menu. To edit an existing map, select it from the maps list page, which opens the edit page. On the edit page, you can view all available data sources and the ones currently used in the map. To add a new layer, select **Add layer**, and then select **Documents** from the prompt, which opens a flyout. In the flyout, select the index pattern and geospatial field. Note that the data source name is prefixed to the index pattern name. After selecting **Update**, the new layer is added. Select the {::nomarkdown}
### Machine learning
-When the data source feature is enabled, you can navigate to **Machine Learning** under the **OpenSearch Plugins** menu. Initially, the models within the default data source are displayed. To view models from a different data source, switch to that data source from the navigation bar. To inspect the details of a specific model, select the {::nomarkdown}
{:/} icon to the right of the model entry. The following GIF illustrates these steps.
+When you set `data_source.enabled:true`, you can view and manage machine learning models from different connected data sources:
+
+1. Navigate to **OpenSearch Plugins** > **Machine Learning** under the main menu.
+2. Select the {::nomarkdown}
### Notifications
-When the data source feature is enabled, you can navigate to **Notifications** under the **Management** menu. The page displays the notification channels configured for the currently selected data source. To view channels from a different data source, select the desired data source from the menu. To view or edit the details of an existing channel, select it from the list, which opens the channel details page. The following GIF illustrates these steps.
+When you set `data_source.enabled:true`, you can view and manage notification channels for different data sources:
+
+1. Navigate to **Management** > **Notifications** under the main menu.
+2. Select the {::nomarkdown}
### Search relevance
-When the data source feature is enabled, you can navigate to **Search Relevance** under the **OpenSearch Plugins** menu. On the navigation bar on the upper right, you can view all available data sources. To compare search results between indexes from different data sources, first select a data source and an index for **Query 1**, and then select a data source and an index for **Query 2**. Select **Search** to run the queries. The following GIF illustrates these steps.
+When you set `data_source.enabled:true`, you can compare search results across indexes from different data sources:
+
+1. Navigate to **OpenSearch Plugins** > **Search Relevance** under the main menu.
+2. Select the {::nomarkdown}
-## Next steps
+### Security analytics
+Introduced 2.15
+{: .label .label-purple}
+
+When you set `data_source.enabled:true`, you can view and manage security analytics resources, such as detection rules, across multiple connected data sources:
+
+1. Navigate to **OpenSearch Plugins** > **Security analytics** under the main menu.
+2. Select the {::nomarkdown}
+
+1. Navigate to **OpenSearch Plugins** > **Security analytics** under the main menu.
+2. Select the {::nomarkdown}
+
+### Alerting
+Introduced 2.15
+{: .label .label-purple }
+
+When you set `data_source.enabled:true`, you can you can view and manage alerting monitors across multiple connected data sources:
+
+1. Navigate to **OpenSearch Plugins** > **Alerting** under the main menu.
+2. Select the {::nomarkdown}
+
+To create a new monitor, select **Create monitor**. Fill out the form and select **Create**. The monitor is created within the selected data source.
+
+#### Managing alerting monitors from within the Dashboards application
+
+To manage data source monitors from within **Dashboards**:
+
+1. Navigate to the **Dashboards** application under the main menu and then select a dashboard from the list.
+2. From the dashboard, select the {::nomarkdown}
{:/} icon to open the **Options** dropdown menu and then choose **Alerting**.
+4. From the **Alerting** dropdown menu, choose **Associated monitors** to open the configuration window.
+5. Select a monitor from the list to view or edit its details.
+
+The following GIF illustrates these steps.
+
+
+
+To associate a monitor with a data source:
+
+1. Navigate to the **Dashboards** application under the main menu and then select a dashboard from the list.
+2. From the dashboard, select the {::nomarkdown}
+
+---
-## Limitations
+## Next steps
-The following features are not supported when using multiple data sources:
+After configuring multiple data sources, you can analyze the data from each source. See the following resources for more information:
-* Timeline visualization types
-* Some external plugins, such as the `gantt-chart` plugin
+- [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/)
+- [Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/)
+- [Connecting OpenSearch and Amazon S3 through OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/)
+- [OpenSearch Integrations]({{site.url}}{{site.baseurl}}/integrations/index/)
diff --git a/_dashboards/quickstart.md b/_dashboards/quickstart.md
index 358efe4003..82a2b446a5 100644
--- a/_dashboards/quickstart.md
+++ b/_dashboards/quickstart.md
@@ -4,7 +4,8 @@ title: OpenSearch Dashboards quickstart guide
nav_order: 2
has_children: false
redirect_from:
- - /dashboards/quickstart-dashboards/
+ - /dashboards/get-started/quickstart-dashboards/
+ - /dashboards/quickstart-dashboards/
---
# OpenSearch Dashboards quickstart guide
diff --git a/_dashboards/visualize/maps.md b/_dashboards/visualize/maps.md
index 8a4196e483..23e14d41c3 100644
--- a/_dashboards/visualize/maps.md
+++ b/_dashboards/visualize/maps.md
@@ -5,6 +5,8 @@ grand_parent: Building data visualizations
parent: Using coordinate and region maps
nav_order: 10
redirect_from:
+ - /dashboards/maps-plugin/
+ - /dashboards/visualize/maps/
- /dashboards/maps/
---
diff --git a/_dashboards/visualize/visbuilder.md b/_dashboards/visualize/visbuilder.md
index 7b32e818f5..de4dfb1666 100644
--- a/_dashboards/visualize/visbuilder.md
+++ b/_dashboards/visualize/visbuilder.md
@@ -9,9 +9,6 @@ redirect_from:
# Using VisBuilder
-VisBuilder is an experimental feature and shouldn't be used in a production environment. For updates on its progress, or if you want to leave feedback that helps improve the feature, see the [GitHub issue](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2280).
-{: .warning}
-
You can use the VisBuilder visualization type in OpenSearch Dashboards to create data visualizations by using a drag-and-drop gesture. With VisBuilder you have:
* An immediate view of your data without the need to preselect the visualization output.
@@ -22,36 +19,22 @@ You can use the VisBuilder visualization type in OpenSearch Dashboards to create
## Try VisBuilder in the OpenSearch Dashboards playground
-If you'd like to try out VisBuilder without installing OpenSearch locally, you can do so in the [Dashboards playground](https://playground.opensearch.org/app/vis-builder#/). VisBuilder is enabled by default.
+If you'd like to try out VisBuilder without installing OpenSearch locally, you can do so in the [Dashboards playground](https://playground.opensearch.org/app/vis-builder#/).
## Try VisBuilder locally
-VisBuilder is enabled by default. If you want to disable it, set the feature `flag vis_builder.enabled:` to `false` in the `opensearch_dashboards.yml` file as follows:
-
-```
-# Set the value of this setting to false to disable VisBuilder
-# functionality in Visualization.
-vis_builder.enabled: false
-```
-
Follow these steps to create a new visualization using VisBuilder in your environment:
1. Open Dashboards:
- If you're not running the Security plugin, go to http://localhost:5601.
- If you're running the Security plugin, go to https://localhost:5601 and log in with your username and password (default is admin/admin).
-2. Confirm that the **Enable experimental visualizations** option is turned on.
- - From the top menu, select **Management** > **Dashboards Management** > **Advanced Settings**.
- - Select **Visualization** and verify that the option is turned on.
-
- 
-
-3. From the top menu, select **Visualize** **>** **Create visualization** **>** **VisBuilder**.
+1. From the top menu, select **Visualize > Create visualization > VisBuilder**.
-4. Drag and drop field names from the left column into the **Configuration** panel to generate a visualization.
+1. Drag and drop field names from the left column into the **Configuration** panel to generate a visualization.
Here’s an example visualization. Your visualization will look different depending on your data and the fields you select.
-
+
\ No newline at end of file
diff --git a/_data-prepper/common-use-cases/codec-processor-combinations.md b/_data-prepper/common-use-cases/codec-processor-combinations.md
index 57185f2ce9..525bc704be 100644
--- a/_data-prepper/common-use-cases/codec-processor-combinations.md
+++ b/_data-prepper/common-use-cases/codec-processor-combinations.md
@@ -45,3 +45,6 @@ The [`newline` codec]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/config
[Apache Avro] helps streamline streaming data pipelines. It is most efficient when used with the [`avro` codec]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sinks/s3#avro-codec) inside an `s3` sink.
+## `event_json`
+
+The `event_json` output codec converts event data and metadata into JSON format to send to a sink, such as an S3 sink. The `event_json` input codec reads the event and its metadata to create an event in Data Prepper.
diff --git a/_data-prepper/migrate-open-distro.md b/_data-prepper/migrate-open-distro.md
index e7fdacbd8f..8b3e7a7198 100644
--- a/_data-prepper/migrate-open-distro.md
+++ b/_data-prepper/migrate-open-distro.md
@@ -2,6 +2,8 @@
layout: default
title: Migrating from Open Distro
nav_order: 30
+redirect_from:
+ - /clients/data-prepper/migrate-open-distro/
---
# Migrating from Open Distro
diff --git a/_data-prepper/migrating-from-logstash-data-prepper.md b/_data-prepper/migrating-from-logstash-data-prepper.md
index f87ca8d6be..3d87f29517 100644
--- a/_data-prepper/migrating-from-logstash-data-prepper.md
+++ b/_data-prepper/migrating-from-logstash-data-prepper.md
@@ -3,6 +3,7 @@ layout: default
title: Migrating from Logstash
nav_order: 25
redirect_from:
+ - /clients/data-prepper/configure-logstash-data-prepper/
- /data-prepper/configure-logstash-data-prepper/
---
diff --git a/_data-prepper/pipelines/configuration/processors/aggregate.md b/_data-prepper/pipelines/configuration/processors/aggregate.md
index 699d25025f..38b138a996 100644
--- a/_data-prepper/pipelines/configuration/processors/aggregate.md
+++ b/_data-prepper/pipelines/configuration/processors/aggregate.md
@@ -20,6 +20,7 @@ Option | Required | Type | Description
identification_keys | Yes | List | An unordered list by which to group events. Events with the same values as these keys are put into the same group. If an event does not contain one of the `identification_keys`, then the value of that key is considered to be equal to `null`. At least one identification_key is required (for example, `["sourceIp", "destinationIp", "port"]`).
action | Yes | AggregateAction | The action to be performed on each group. One of the [available aggregate actions](#available-aggregate-actions) must be provided, or you can create custom aggregate actions. `remove_duplicates` and `put_all` are the available actions. For more information, see [Creating New Aggregate Actions](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#creating-new-aggregate-actions).
group_duration | No | String | The amount of time that a group should exist before it is concluded automatically. Supports ISO_8601 notation strings ("PT20.345S", "PT15M", etc.) as well as simple notation for seconds (`"60s"`) and milliseconds (`"1500ms"`). Default value is `180s`.
+local_mode | No | Boolean | When `local_mode` is set to `true`, the aggregation is performed locally on each Data Prepper node instead of forwarding events to a specific node based on the `identification_keys` using a hash function. Default is `false`.
## Available aggregate actions
@@ -176,4 +177,4 @@ The `aggregate` processor includes the following custom metrics.
**Gauge**
-* `currentAggregateGroups`: The current number of groups. This gauge decreases when a group concludes and increases when an event initiates the creation of a new group.
\ No newline at end of file
+* `currentAggregateGroups`: This gauge represents the current number of active aggregate groups. It decreases when an aggregate group is completed and its results are emitted and increases when a new event initiates the creation of a new aggregate group.
diff --git a/_data-prepper/pipelines/configuration/processors/key-value.md b/_data-prepper/pipelines/configuration/processors/key-value.md
index 884ae97549..aedc1f8822 100644
--- a/_data-prepper/pipelines/configuration/processors/key-value.md
+++ b/_data-prepper/pipelines/configuration/processors/key-value.md
@@ -33,6 +33,11 @@ You can use the `key_value` processor to parse the specified field into key-valu
| recursive | Specifies whether to recursively obtain additional key-value pairs from values. The extra key-value pairs will be stored as sub-keys of the root key. Default is `false`. The levels of recursive parsing must be defined by different brackets for each level: `[]`, `()`, and `<>`, in this order. Any other configurations specified will only be applied to the outmost keys. This is an earlier version of the OpenSearch documentation. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
+You're viewing version {{site.opensearch_major_minor_version}} of the OpenSearch documentation. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
{% elsif site.doc_version == "unsupported" %} -This version of the OpenSearch documentation is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
+You're viewing version {{site.opensearch_major_minor_version}} of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
{% endif %} {% endif %} {% if site.heading_anchors != false %} diff --git a/_ml-commons-plugin/agents-tools/index.md b/_ml-commons-plugin/agents-tools/index.md index ba88edef2f..009906d4cf 100644 --- a/_ml-commons-plugin/agents-tools/index.md +++ b/_ml-commons-plugin/agents-tools/index.md @@ -18,7 +18,7 @@ An _agent_ is a coordinator that uses a large language model (LLM) to solve a pr - [_Flow agent_](#flow-agents): Runs tools sequentially, in the order specified in its configuration. The workflow of a flow agent is fixed. Useful for retrieval-augmented generation (RAG). - [_Conversational flow agent_](#conversational-flow-agents): Runs tools sequentially, in the order specified in its configuration. The workflow of a conversational flow agent is fixed. Stores conversation history so that users can ask follow-up questions. Useful for creating a chatbot. -- [_Conversational agent_](#conversational-agents): Reasons in order to provide a response based on the available knowledge, including the LLM knowledge base and a set of tools provided to the LLM. Stores conversation history so that users can ask follow-up questions. The workflow of a conversational agent is variable, based on follow-up questions. For specific questions, uses the Chain-of-Thought (CoT) process to select the best tool from the configured tools for providing a response to the question. Useful for creating a chatbot that employs RAG. +- [_Conversational agent_](#conversational-agents): Reasons in order to provide a response based on the available knowledge, including the LLM knowledge base and a set of tools provided to the LLM. The LLM reasons iteratively to decide what action to take until it obtains the final answer or reaches the iteration limit. Stores conversation history so that users can ask follow-up questions. The workflow of a conversational agent is variable, based on follow-up questions. For specific questions, uses the Chain-of-Thought (CoT) process to select the best tool from the configured tools for providing a response to the question. Useful for creating a chatbot that employs RAG. ### Flow agents diff --git a/_ml-commons-plugin/agents-tools/tools/connector-tool.md b/_ml-commons-plugin/agents-tools/tools/connector-tool.md new file mode 100644 index 0000000000..dfe4770932 --- /dev/null +++ b/_ml-commons-plugin/agents-tools/tools/connector-tool.md @@ -0,0 +1,168 @@ +--- +layout: default +title: Connector tool +has_children: false +has_toc: false +nav_order: 20 +parent: Tools +grand_parent: Agents and tools +--- + + +# Connector tool +**Introduced 2.15** +{: .label .label-purple } + + +The `ConnectorTool` uses a [connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/) to call any REST API function. For example, you can use a `ConnectorTool` to call a Lambda function through its REST API interface. + +## Step 1: Register a connector with an execute action + +The `ConnectorTool` can only run an `execute` action within a connector. Before you can create a `ConnectorTool`, you need to configure a connector and provide an `execute` action in the `actions` array. The `execute` action is used to invoke a function at a REST API endpoint. It is similar to the `predict` action, which is used to invoke a machine learning (ML) model. + +For this example, you'll create a connector for a simple AWS Lambda function that accepts two integers and returns their sum. This function is hosted on a dedicated endpoint with a specific URL, which you'll provide in the `url` parameter. For more information, see [Lambda function URLs](https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html). + +To create a connector, send the following request: + +```json +POST _plugins/_ml/connectors/_create +{ + "name": "Lambda connector of simple calculator", + "description": "Demo connector of lambda function", + "version": 1, + "protocol": "aws_sigv4", + "parameters": { + "region": "YOUR AWS REGION", + "service_name": "lambda" + }, + "credential": { + "access_key": "YOUR ACCESS KEY", + "secret_key": "YOUR SECRET KEY", + "session_token": "YOUR SESSION TOKEN" + }, + "actions": [ + { + "action_type": "execute", + "method": "POST", + "url": "YOUR LAMBDA FUNCTION URL", + "headers": { + "content-type": "application/json" + }, + "request_body": "{ \"number1\":\"${parameters.number1}\", \"number2\":\"${parameters.number2}\" }" + } + ] +} +``` +{% include copy-curl.html %} + +OpenSearch responds with a connector ID: + +```json +{ + "connector_id": "Zz1XEJABXWrLmr4mewEF" +} +``` + +## Step 2: Register a flow agent that will run the ConnectorTool + +For this example, the Lambda function adds the two input numbers and returns their sum in the `result` field: + +```json +{ + "result": 5 +} +``` + +By default, the `ConnectorTool` expects the response from the Lambda function to contain a field named `response`. However, in this example the Lambda function response doesn't include a `response` field. To retrieve the result from the `result` field instead, you need to provide a `response_filter`, specifying the [JSON path](https://github.com/json-path/JsonPath) to the `result` field (`$.result`). Using the `response_filter`, the `ConnectorTool` will retrieve the result with the specified JSON path and return it in the `response` field. + +To configure the Lambda function workflow, create a flow agent. A flow agent runs a sequence of tools in order and returns the last tool's output. To create a flow agent, send the following register agent request, providing the connector ID from the previous step and a `response_filter`: + +```json +POST /_plugins/_ml/agents/_register +{ + "name": "Demo agent of Lambda connector", + "type": "flow", + "description": "This is a demo agent", + "app_type": "demo", + "tools": [ + { + "type": "ConnectorTool", + "name": "lambda_function", + "parameters": { + "connector_id": "YOUR CONNECTOR ID", + "response_filter": "$.result" + } + } + ] +} +``` +{% include copy-curl.html %} + +For parameter descriptions, see [Register parameters](#register-parameters). + +OpenSearch responds with an agent ID: + +```json +{ + "agent_id": "az1XEJABXWrLmr4miAFj" +} +``` + +## Step 3: Run the agent + +Then, run the agent by sending the following request: + +```json +POST /_plugins/_ml/agents/9X7xWI0Bpc3sThaJdY9i/_execute +{ + "parameters": { + "number1": 2, + "number2": 3 + } +} +``` +{% include copy-curl.html %} + +OpenSearch returns the output of the Lambda function execution. In the output, the field name is `response`, and the `result` field contains the Lambda function result: + +```json +{ + "inference_results": [ + { + "output": [ + { + "name": "response", + "result": 5 + } + ] + } + ] +} +``` + +## Register parameters + +The following table lists all tool parameters that are available when registering an agent. + +Parameter | Type | Required/Optional | Description +:--- | :--- | :--- | :--- +`connector_id` | String | Required | A connector ID of a connector configured with an `execute` action that invokes an API. +`response_filter` | String | Optional | A [JSON path](https://github.com/json-path/JsonPath) to the response field that contains the result of invoking the API. If a `response_filter` is not specified, then the `ConnectorTool` expects the API response to be in a field named `response`. + +## Execute parameters + +When running the agent, you can define any parameter needed for the API call in the `request_body` of your connector's `execute` action. In this example, the parameters are `number1` and `number2`: + +```json +"actions": [ + { + "action_type": "execute", + "method": "POST", + "url": "YOUR LAMBDA FUNCTION URL", + "headers": { + "content-type": "application/json" + }, + "request_body": "{ \"number1\":\"${parameters.number1}\", \"number2\":\"${parameters.number2}\" }" + } + ] +``` \ No newline at end of file diff --git a/_ml-commons-plugin/agents-tools/tools/index.md b/_ml-commons-plugin/agents-tools/tools/index.md index 8db522006e..fba47b63da 100644 --- a/_ml-commons-plugin/agents-tools/tools/index.md +++ b/_ml-commons-plugin/agents-tools/tools/index.md @@ -33,6 +33,7 @@ Each tool takes a list of parameters specific to that tool. In the preceding exa |:--- |:--- | |[`AgentTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/agent-tool/) |Runs any agent. | |[`CatIndexTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/cat-index-tool/) |Retrieves index information for the OpenSearch cluster. | +|[`ConnectorTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/connector-tool/) | Uses a [connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/) to call any REST API function. | |[`IndexMappingTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/index-mapping-tool/) |Retrieves index mapping and setting information for an index. | |[`MLModelTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/ml-model-tool/) |Runs machine learning models. | |[`NeuralSparseSearchTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/neural-sparse-tool/) | Performs sparse vector retrieval. | diff --git a/_ml-commons-plugin/api/model-apis/register-model.md b/_ml-commons-plugin/api/model-apis/register-model.md index 30cc93be00..61d821419e 100644 --- a/_ml-commons-plugin/api/model-apis/register-model.md +++ b/_ml-commons-plugin/api/model-apis/register-model.md @@ -260,15 +260,24 @@ To register an externally hosted model with guardrails, provide the `guardrails` Field | Data type | Description :--- | :--- | :--- -`type` | String | The guardrail type. Currently, only `local_regex` is supported. -`input_guardrail`| Object | The guardrail for the model input. | -`output_guardrail`| Object | The guardrail for the model output. | -`stop_words`| Object | The list of indexes containing stopwords used for the model input/output validation. If the model prompt/response contains a stopword contained in any of the indexes, the predict request on this model is rejected. | -`index_name`| Object | The name of the index storing the stopwords. | -`source_fields`| Object | The name of the field storing the stopwords. | -`regex`| Object | A regular expression used for input/output validation. If the model prompt/response matches the regular expression, the predict request on this model is rejected. | +`type` | String | The guardrail type. Valid values are [`local_regex`](#example-request-regex-and-stopword-validation) and [`model`](#example-request-guardrail-model-validation). Using `local_regex`, you can specify a regular expression or stop words. Using `model`, you can specify a guardrail model. For more information, see [Guardrails]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/guardrails/). +`input_guardrail`| Object | The guardrail for the model input. +`output_guardrail`| Object | The guardrail for the model output. +`stop_words`| Object | The list of indexes containing stopwords used for model input/output validation. If the model prompt/response contains a stopword contained in any of the indexes, then the predict request on the model is rejected. +`index_name`| Object | The name of the index storing the stopwords. +`source_fields`| Object | The name of the field storing the stopwords. +`regex`| Object | A regular expression used for input/output validation. If the model prompt/response matches the regular expression, then the predict request on the model is rejected. +`model_id`| String | The guardrail model used to validate user input and LLM output. +`response_filter`| String | The dot path of the field containing the guardrail model response. +`response_validation_regex`| String | The regular expression used to validate the guardrail model response. -#### Example request: Externally hosted model with guardrails +## Examples + +The following examples configure an externally hosted model with guardrails. + +#### Example request: Regex and stopword validation + +The following example uses a regular expression and a set of stopwords to validate the LLM response: ```json POST /_plugins/_ml/models/_register @@ -303,7 +312,36 @@ POST /_plugins/_ml/models/_register ``` {% include copy-curl.html %} -For a complete example, see [Guardrails]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/guardrails/). +For a complete example, see [Validating input/output using stopwords and regex]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/guardrails/#validating-inputoutput-using-stopwords-and-regex). + +#### Example request: Guardrail model validation + +The following example uses a guardrail model to validate the LLM response: + +```json +POST /_plugins/_ml/models/_register?deploy=true +{ + "name": "Bedrock Claude V2 model with guardrails model", + "function_name": "remote", + "model_group_id": "ppSmpo8Bi-GZ0tf1i7cD", + "description": "Bedrock Claude V2 model with guardrails model", + "connector_id": "xnJjDZABNFJeYR3IPvTO", + "guardrails": { + "input_guardrail": { + "model_id": "o3JaDZABNFJeYR3I2fRV", + "response_validation_regex": "^\\s*\"[Aa]ccept\"\\s*$" + }, + "output_guardrail": { + "model_id": "o3JaDZABNFJeYR3I2fRV", + "response_validation_regex": "^\\s*\"[Aa]ccept\"\\s*$" + }, + "type": "model" + } +} +``` +{% include copy-curl.html %} + +For a complete example, see [Validating input/output using a guardrail model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/guardrails/#validating-inputoutput-using-a-guardrail-model). #### Example response diff --git a/_ml-commons-plugin/api/model-apis/update-model.md b/_ml-commons-plugin/api/model-apis/update-model.md index e952f9f2bb..048b5b76e5 100644 --- a/_ml-commons-plugin/api/model-apis/update-model.md +++ b/_ml-commons-plugin/api/model-apis/update-model.md @@ -64,12 +64,13 @@ PUT /_plugins/_ml/models/T_S-cY0BKCJ3ot9qr0aP ``` {% include copy-curl.html %} -#### Example request: Updating the guardrails +#### Example requests: Updating the guardrails ```json PUT /_plugins/_ml/models/MzcIJX8BA7mbufL6DOwl { "guardrails": { + "type": "local_regex", "input_guardrail": { "stop_words": [ { @@ -93,6 +94,24 @@ PUT /_plugins/_ml/models/MzcIJX8BA7mbufL6DOwl ``` {% include copy-curl.html %} +```json +PUT /_plugins/_ml/models/9uGdCJABjaMXYrp14YRj +{ + "guardrails": { + "type": "model", + "input_guardrail": { + "model_id": "V-G1CJABjaMXYrp1QoUC", + "response_validation_regex": "^\\s*[Aa]ccept\\s*$" + }, + "output_guardrail": { + "model_id": "V-G1CJABjaMXYrp1QoUC", + "response_validation_regex": "^\\s*[Aa]ccept\\s*$" + } + } +} +``` +{% include copy-curl.html %} + #### Example response ```json diff --git a/_ml-commons-plugin/cluster-settings.md b/_ml-commons-plugin/cluster-settings.md index c473af81a1..37c7cab9f7 100644 --- a/_ml-commons-plugin/cluster-settings.md +++ b/_ml-commons-plugin/cluster-settings.md @@ -108,7 +108,7 @@ plugins.ml_commons.sync_up_job_interval_in_seconds: 3 - Default value: `3` - Value range: [0, 86,400] -## Predict monitoring requests +## Monitoring predict requests Controls how many predict requests are monitored on one node. If set to `0`, OpenSearch clears all monitoring predict requests in cache and does not monitor for new predict requests. @@ -303,12 +303,12 @@ This setting automatically redeploys deployed or partially deployed models upon ### Setting ``` -plugins.ml_commons.model_auto_redeploy.enable: false +plugins.ml_commons.model_auto_redeploy.enable: true ``` ### Values -- Default value: false +- Default value: true - Valid values: `false`, `true` ## Set retires for auto redeploy diff --git a/_ml-commons-plugin/index.md b/_ml-commons-plugin/index.md index c936f36161..f0355b6be3 100644 --- a/_ml-commons-plugin/index.md +++ b/_ml-commons-plugin/index.md @@ -30,4 +30,8 @@ ML Commons supports various algorithms to help train ML models and make predicti ## ML Commons API -ML Commons provides its own set of REST APIs. For more information, see [ML Commons API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/index/). \ No newline at end of file +ML Commons provides its own set of REST APIs. For more information, see [ML Commons API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/index/). + +## Tutorials + +Using the OpenSearch ML framework, you can build various applications, from implementing conversational search to building your own chatbot. For more information, see [Tutorials]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/index/). \ No newline at end of file diff --git a/_ml-commons-plugin/remote-models/batch-ingestion.md b/_ml-commons-plugin/remote-models/batch-ingestion.md new file mode 100644 index 0000000000..64f434e652 --- /dev/null +++ b/_ml-commons-plugin/remote-models/batch-ingestion.md @@ -0,0 +1,250 @@ +--- +layout: default +title: Batch ingestion +has_children: false +nav_order: 80 +parent: Connecting to externally hosted models +grand_parent: Integrating ML models +--- + +# Using externally hosted ML models for batch ingestion + +**Introduced 2.15** +{: .label .label-purple } + +If you are ingesting multiple documents and generating embeddings by invoking an externally hosted model, you can use batch ingestion to improve performance. + +The [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) accepts a `batch_size` parameter that specifies to process documents in batches of a specified size. Processors that support batch ingestion will send each batch of documents to an externally hosted model in a single request. + +The [`text_embedding`]({{site.url}}{{site.baseurl}}/ingest-pipelines/processors/text-embedding/) and [`sparse_encoding`]({{site.url}}{{site.baseurl}}/ingest-pipelines/processors/sparse-encoding/) processors currently support batch ingestion. + +## Step 1: Register a model group + +You can register a model in two ways: + +* You can use `model_group_id` to register a model version to an existing model group. +* If you do not use `model_group_id`, then ML Commons creates a model with a new model group. + +To register a model group, send the following request: + +```json +POST /_plugins/_ml/model_groups/_register +{ + "name": "remote_model_group", + "description": "A model group for external models" +} +``` +{% include copy-curl.html %} + +The response contains the model group ID that you'll use to register a model to this model group: + +```json +{ + "model_group_id": "wlcnb4kBJ1eYAeTMHlV6", + "status": "CREATED" +} +``` + +To learn more about model groups, see [Model access control]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control/). + +## Step 2: Create a connector + +You can create a standalone connector that can be reused for multiple models. Alternatively, you can specify a connector when creating a model so that it can be used only for that model. For more information and example connectors, see [Connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/). + +The Connectors Create API, `/_plugins/_ml/connectors/_create`, creates connectors that facilitate registering and deploying external models in OpenSearch. Using the `endpoint` parameter, you can connect ML Commons to any supported ML tool by using its specific API endpoint. For example, you can connect to a ChatGPT model by using the `api.openai.com` endpoint: + +```json +POST /_plugins/_ml/connectors/_create +{ + "name": "OpenAI Chat Connector", + "description": "The connector to public OpenAI model service for GPT 3.5", + "version": 1, + "protocol": "http", + "parameters": { + "endpoint": "api.openai.com", + "model": "gpt-3.5-turbo", + "input_docs_processed_step_size": 100 + }, + "credential": { + "openAI_key": "..." + }, + "actions": [ + { + "action_type": "predict", + "method": "POST", + "url": "https://${parameters.endpoint}/v1/chat/completions", + "headers": { + "Authorization": "Bearer ${credential.openAI_key}" + }, + "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }" + } + ] +} +``` +{% include copy-curl.html %} + +The `parameters.input_docs_processed_step_size` parameter is used to set the maximum batch size for documents sent to a remote server. You can set this parameter to the maximum batch size supported by the remote server or to a smaller number for optimal performance. + +The response contains the connector ID for the newly created connector: + +```json +{ + "connector_id": "a1eMb4kBJ1eYAeTMAljY" +} +``` + +## Step 3: Register an externally hosted model + +To register an externally hosted model to the model group created in step 1, provide the model group ID from step 1 and the connector ID from step 2 in the following request. You must specify the `function_name` as `remote`: + +```json +POST /_plugins/_ml/models/_register +{ + "name": "openAI-gpt-3.5-turbo", + "function_name": "remote", + "model_group_id": "wlcnb4kBJ1eYAeTMHlV6", + "description": "test model", + "connector_id": "a1eMb4kBJ1eYAeTMAljY" +} +``` +{% include copy-curl.html %} + +OpenSearch returns the task ID of the register operation: + +```json +{ + "task_id": "cVeMb4kBJ1eYAeTMFFgj", + "status": "CREATED" +} +``` + +To check the status of the operation, provide the task ID to the [Tasks API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/tasks-apis/get-task/): + +```json +GET /_plugins/_ml/tasks/cVeMb4kBJ1eYAeTMFFgj +``` +{% include copy-curl.html %} + +When the operation is complete, the state changes to `COMPLETED`: + +```json +{ + "model_id": "cleMb4kBJ1eYAeTMFFg4", + "task_type": "REGISTER_MODEL", + "function_name": "REMOTE", + "state": "COMPLETED", + "worker_node": [ + "XPcXLV7RQoi5m8NI_jEOVQ" + ], + "create_time": 1689793598499, + "last_update_time": 1689793598530, + "is_async": false +} +``` + +Take note of the returned `model_id` because you’ll need it to deploy the model. + +## Step 4: Deploy the model + +Starting in OpenSearch version 2.13, externally hosted models are deployed automatically when you send a Predict API request for the first time. To disable automatic deployment for an externally hosted model, set `plugins.ml_commons.model_auto_deploy.enable` to `false`: + +```json +PUT _cluster/settings +{ + "persistent": { + "plugins.ml_commons.model_auto_deploy.enable" : "false" + } +} +``` +{% include copy-curl.html %} + +To undeploy the model, use the [Undeploy API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/undeploy-model/): + +```json +POST /_plugins/_ml/models/cleMb4kBJ1eYAeTMFFg4/_deploy +``` +{% include copy-curl.html %} + +The response contains the task ID, which you can use to check the status of the deploy operation: + +```json +{ + "task_id": "vVePb4kBJ1eYAeTM7ljG", + "status": "CREATED" +} +``` + +As in the previous step, check the status of the operation by calling the Tasks API: + +```json +GET /_plugins/_ml/tasks/vVePb4kBJ1eYAeTM7ljG +``` +{% include copy-curl.html %} + +When the operation is complete, the state changes to `COMPLETED`: + +```json +{ + "model_id": "cleMb4kBJ1eYAeTMFFg4", + "task_type": "DEPLOY_MODEL", + "function_name": "REMOTE", + "state": "COMPLETED", + "worker_node": [ + "n-72khvBTBi3bnIIR8FTTw" + ], + "create_time": 1689793851077, + "last_update_time": 1689793851101, + "is_async": true +} +``` + +## Step 5: Create an ingest pipeline + +The following example request creates an ingest pipeline with a `text_embedding` processor. The processor converts the text in the `passage_text` field into text embeddings and stores the embeddings in `passage_embedding`: + +```json +PUT /_ingest/pipeline/nlp-ingest-pipeline +{ + "description": "A text embedding pipeline", + "processors": [ + { + "text_embedding": { + "model_id": "cleMb4kBJ1eYAeTMFFg4", + "field_map": { + "passage_text": "passage_embedding" + } + } + } + ] +} +``` +{% include copy-curl.html %} + +## Step 6: Perform bulk indexing + +To ingest documents in bulk, call the Bulk API and provide the `batch_size` and `pipeline` parameters. If you don't provide a `pipeline` parameter, the default ingest pipeline for the index will be used for ingestion: + +```json +POST _bulk?batch_size=5&pipeline=nlp-ingest-pipeline +{ "create": { "_index": "testindex1", "_id": "2" } } +{ "passage_text": "hello world" } +{ "create": { "_index": "testindex1", "_id": "3" } } +{ "passage_text": "big apple" } +{ "create": { "_index": "testindex1", "_id": "4" } } +{ "passage_text": "golden gate bridge" } +{ "create": { "_index": "testindex1", "_id": "5" } } +{ "passage_text": "fine tune" } +{ "create": { "_index": "testindex1", "_id": "6" } } +{ "passage_text": "random test" } +{ "create": { "_index": "testindex1", "_id": "7" } } +{ "passage_text": "sun and moon" } +{ "create": { "_index": "testindex1", "_id": "8" } } +{ "passage_text": "windy" } +{ "create": { "_index": "testindex1", "_id": "9" } } +{ "passage_text": "new york" } +{ "create": { "_index": "testindex1", "_id": "10" } } +{ "passage_text": "fantastic" } + +``` +{% include copy-curl.html %} + diff --git a/_ml-commons-plugin/remote-models/blueprints.md b/_ml-commons-plugin/remote-models/blueprints.md index 0f65aee9da..254a21b068 100644 --- a/_ml-commons-plugin/remote-models/blueprints.md +++ b/_ml-commons-plugin/remote-models/blueprints.md @@ -55,41 +55,45 @@ As an ML developer, you can build connector blueprints for other platforms. Usin ## Configuration parameters -| Field | Data type | Is required | Description | -|:------------------------|:------------|:------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `name` | String | Yes | The name of the connector. | -| `description` | String | Yes | A description of the connector. | -| `version` | Integer | Yes | The version of the connector. | -| `protocol` | String | Yes | The protocol for the connection. For AWS services such as Amazon SageMaker and Amazon Bedrock, use `aws_sigv4`. For all other services, use `http`. | -| `parameters` | JSON object | Yes | The default connector parameters, including `endpoint` and `model`. Any parameters indicated in this field can be overridden by parameters specified in a predict request. | -| `credential` | JSON object | Yes | Defines any credential variables required to connect to your chosen endpoint. ML Commons uses **AES/GCM/NoPadding** symmetric encryption to encrypt your credentials. When the connection to the cluster first starts, OpenSearch creates a random 32-byte encryption key that persists in OpenSearch's system index. Therefore, you do not need to manually set the encryption key. | -| `actions` | JSON array | Yes | Defines what actions can run within the connector. If you're an administrator creating a connection, add the [blueprint]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/) for your desired connection. | -| `backend_roles` | JSON array | Yes | A list of OpenSearch backend roles. For more information about setting up backend roles, see [Assigning backend roles to users]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control#assigning-backend-roles-to-users). | -| `access_mode` | String | Yes | Sets the access mode for the model, either `public`, `restricted`, or `private`. Default is `private`. For more information about `access_mode`, see [Model groups]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control#model-groups). | -| `add_all_backend_roles` | Boolean | Yes | When set to `true`, adds all `backend_roles` to the access list, which only a user with admin permissions can adjust. When set to `false`, non-admins can add `backend_roles`. | -| `client_config` | JSON object | No | The client configuration object, which provides settings that control the behavior of the client connections used by the connector. These settings allow you to manage connection limits and timeouts, ensuring efficient and reliable communication. | +| Field | Data type | Is required | Description | +|:---|:---|:---|:---| +| `name` | String | Yes | The name of the connector. | +| `description` | String | Yes | A description of the connector. | +| `version` | Integer | Yes | The version of the connector. | +| `protocol` | String | Yes | The protocol for the connection. For AWS services such as Amazon SageMaker and Amazon Bedrock, use `aws_sigv4`. For all other services, use `http`. | +| `parameters` | JSON object | Yes | The default connector parameters, including `endpoint` and `model`. Any parameters indicated in this field can be overridden by parameters specified in a predict request. | +| `credential` | JSON object | Yes | Defines any credential variables required to connect to your chosen endpoint. ML Commons uses **AES/GCM/NoPadding** symmetric encryption to encrypt your credentials. When the connection to the cluster first starts, OpenSearch creates a random 32-byte encryption key that persists in OpenSearch's system index. Therefore, you do not need to manually set the encryption key. | +| `actions` | JSON array | Yes | Defines what actions can run within the connector. If you're an administrator creating a connection, add the [blueprint]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/) for your desired connection. | +| `backend_roles` | JSON array | Yes | A list of OpenSearch backend roles. For more information about setting up backend roles, see [Assigning backend roles to users]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control#assigning-backend-roles-to-users). | +| `access_mode` | String | Yes | Sets the access mode for the model, either `public`, `restricted`, or `private`. Default is `private`. For more information about `access_mode`, see [Model groups]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control#model-groups). | +| `add_all_backend_roles` | Boolean | Yes | When set to `true`, adds all `backend_roles` to the access list, which only a user with admin permissions can adjust. When set to `false`, non-admins can add `backend_roles`. | +| `client_config` | JSON object | No | The client configuration object, which provides settings that control the behavior of the client connections used by the connector. These settings allow you to manage connection limits and timeouts, ensuring efficient and reliable communication. | The `actions` parameter supports the following options. -| Field | Data type | Description | -|:------------------------|:------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `action_type` | String | Required. Sets the ML Commons API operation to use upon connection. As of OpenSearch 2.9, only `predict` is supported. | -| `method` | String | Required. Defines the HTTP method for the API call. Supports `POST` and `GET`. | -| `url` | String | Required. Sets the connection endpoint at which the action occurs. This must match the regex expression for the connection used when [adding trusted endpoints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/index#adding-trusted-endpoints). | -| `headers` | JSON object | Sets the headers used inside the request or response body. Default is `ContentType: application/json`. If your third-party ML tool requires access control, define the required `credential` parameters in the `headers` parameter. | -| `request_body` | String | Required. Sets the parameters contained in the request body of the action. The parameters must include `\"inputText\`, which specifies how users of the connector should construct the request payload for the `action_type`. | -| `pre_process_function` | String | Optional. A built-in or custom Painless script used to preprocess the input data. OpenSearch provides the following built-in preprocess functions that you can call directly:- `connector.pre_process.cohere.embedding` for [Cohere](https://cohere.com/) embedding models
- `connector.pre_process.openai.embedding` for [OpenAI](https://platform.openai.com/docs/guides/embeddings) embedding models
- `connector.pre_process.default.embedding`, which you can use to preprocess documents in neural search requests so that they are in the format that ML Commons can process with the default preprocessor (OpenSearch 2.11 or later). For more information, see [Built-in functions](#built-in-pre--and-post-processing-functions). | -| `post_process_function` | String | Optional. A built-in or custom Painless script used to post-process the model output data. OpenSearch provides the following built-in post-process functions that you can call directly:
- `connector.pre_process.cohere.embedding` for [Cohere text embedding models](https://docs.cohere.com/reference/embed)
- `connector.pre_process.openai.embedding` for [OpenAI text embedding models](https://platform.openai.com/docs/api-reference/embeddings)
- `connector.post_process.default.embedding`, which you can use to post-process documents in the model response so that they are in the format that neural search expects (OpenSearch 2.11 or later). For more information, see [Built-in functions](#built-in-pre--and-post-processing-functions). | +| Field | Data type | Description | +|:---|:---|:---| +| `action_type` | String | Required. Sets the ML Commons API operation to use upon connection. As of OpenSearch 2.9, only `predict` is supported. | +| `method` | String | Required. Defines the HTTP method for the API call. Supports `POST` and `GET`. | +| `url` | String | Required. Sets the connection endpoint at which the action occurs. This must match the regex expression for the connection used when [adding trusted endpoints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/index#adding-trusted-endpoints). | +| `headers` | JSON object | Sets the headers used inside the request or response body. Default is `ContentType: application/json`. If your third-party ML tool requires access control, define the required `credential` parameters in the `headers` parameter. | +| `request_body` | String | Required. Sets the parameters contained in the request body of the action. The parameters must include `\"inputText\`, which specifies how users of the connector should construct the request payload for the `action_type`. | +| `pre_process_function` | String | Optional. A built-in or custom Painless script used to preprocess the input data. OpenSearch provides the following built-in preprocess functions that you can call directly:
- `connector.pre_process.cohere.embedding` for [Cohere](https://cohere.com/) embedding models
- `connector.pre_process.openai.embedding` for [OpenAI](https://platform.openai.com/docs/guides/embeddings) embedding models
- `connector.pre_process.default.embedding`, which you can use to preprocess documents in neural search requests so that they are in the format that ML Commons can process with the default preprocessor (OpenSearch 2.11 or later). For more information, see [Built-in functions](#built-in-pre--and-post-processing-functions). | +| `post_process_function` | String | Optional. A built-in or custom Painless script used to post-process the model output data. OpenSearch provides the following built-in post-process functions that you can call directly:
- `connector.pre_process.cohere.embedding` for [Cohere text embedding models](https://docs.cohere.com/reference/embed)
- `connector.pre_process.openai.embedding` for [OpenAI text embedding models](https://platform.openai.com/docs/api-reference/embeddings)
- `connector.post_process.default.embedding`, which you can use to post-process documents in the model response so that they are in the format that neural search expects (OpenSearch 2.11 or later). For more information, see [Built-in functions](#built-in-pre--and-post-processing-functions). | The `client_config` parameter supports the following options. -| Field | Data type | Description | -|:---------------------|:----------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `max_connection` | Integer | The maximum number of concurrent connections that the client can establish with the server. | -| `connection_timeout` | Integer | The maximum amount of time (in seconds) that the client will wait while trying to establish a connection to the server. A timeout prevents the client from waiting indefinitely and allows it to recover from unreachable network endpoints. | -| `read_timeout` | Integer | The maximum amount of time (in seconds) that the client will wait for a response from the server after sending a request. Useful when the server is slow to respond or encounters issues while processing a request. | +| Field | Data type | Description | +|:---|:---|:---| +| `max_connection` | Integer | The maximum number of concurrent connections that the client can establish to the server. Some remote services, like SageMaker, constrain the maximum number of concurrent connections and throw a throttling exception if the number of concurrent connections exceeds the threshold. The maximum number of concurrent OpenSearch connections is `max_connection`*`node_number_for_connector`. To mitigate this issue, try to decrease the value of this parameter and modify the retry settings in `client_config`. Default is `30`. | +| `connection_timeout` | Integer | The maximum amount of time (in seconds) that the client will wait while trying to establish a connection to the server. A timeout prevents the client from waiting indefinitely and allows the client to recover when it encounters unreachable network endpoints. | +| `read_timeout` | Integer | The maximum amount of time (in seconds) that the client will wait for a response from the server after sending a request. This is useful when the server is slow to respond or encounters an issue while processing a request. | +| `retry_backoff_policy` | String | The backoff policy for retries to the remote connector. This is useful when there is spike in traffic causing throttling exceptions. Supported policies are `constant`, `exponential_equal_jitter`, and `exponential_full_jitter`. Default is `constant`. | +| `max_retry_times` | Integer | The maximum number of times that a single remote inference request will be retried. This is useful when there is a spike in traffic causing throttling exceptions. When set to `0`, retrying is disabled. When set to `-1`, OpenSearch does not limit the number of `retry_times`. Setting this to a positive integer specifies the maximum number of retry attempts. Default is `0`. | +| `retry_backoff_millis` | Integer | The base backoff time in milliseconds for retry policy. The suspend time during two retries is determined by this parameter and `retry_backoff_policy`. Default is `200`. | +| `retry_timeout_seconds` | Integer | The timeout value, in seconds, for the retry. If the retry can not succeed within the specified amount of time, the connector will stop retrying and throw an exception. Default is `30`. | ## Built-in pre- and post-processing functions diff --git a/_ml-commons-plugin/remote-models/guardrails.md b/_ml-commons-plugin/remote-models/guardrails.md index ca34eb335c..5330454c8b 100644 --- a/_ml-commons-plugin/remote-models/guardrails.md +++ b/_ml-commons-plugin/remote-models/guardrails.md @@ -1,7 +1,7 @@ --- layout: default title: Guardrails -has_children: false +has_children: true has_toc: false nav_order: 70 parent: Connecting to externally hosted models @@ -14,13 +14,21 @@ grand_parent: Integrating ML models Guardrails can guide a large language model (LLM) toward desired behavior. They act as a filter, preventing the LLM from generating output that is harmful or violates ethical principles and facilitating safer use of AI. Guardrails also cause the LLM to produce more focused and relevant output. -To configure guardrails for your LLM, you can provide a list of words to be prohibited in the input or output of the model. Alternatively, you can provide a regular expression against which the model input or output will be matched. +You can configure guardrails for your LLM using the following methods: +- Provide a list of words to be prohibited in the input or output of the model. Alternatively, you can provide a regular expression against which the model input or output will be matched. For more information, see [Validating input/output using stopwords and regex](#validating-inputoutput-using-stopwords-and-regex). +- Configure a separate LLM whose purpose is to validate the user input and the LLM output. ## Prerequisites Before you start, make sure you have fulfilled the [prerequisites]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/index/#prerequisites) for connecting to an externally hosted model. -## Step 1: Create a guardrail index +## Validating input/output using stopwords and regex +**Introduced 2.13** +{: .label .label-purple } + +A simple way to validate the user input and LLM output is to provide a set of prohibited words (stopwords) or a regular expression for validation. + +### Step 1: Create a guardrail index To start, create an index that will store the excluded words (_stopwords_). In the index settings, specify a `title` field, which will contain excluded words, and a `query` field of the [percolator]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/percolator/) type. The percolator query will be used to match the LLM input or output: @@ -41,7 +49,7 @@ PUT /words0 ``` {% include copy-curl.html %} -## Step 2: Index excluded words or phrases +### Step 2: Index excluded words or phrases Next, index a query string query that will be used to match excluded words in the model input or output: @@ -71,7 +79,7 @@ PUT /words0/_doc/2?refresh For more query string options, see [Query string query]({{site.url}}{{site.baseurl}}/query-dsl/full-text/query-string/). -## Step 3: Register a model group +### Step 3: Register a model group To register a model group, send the following request: @@ -95,7 +103,7 @@ The response contains the model group ID that you'll use to register a model to To learn more about model groups, see [Model access control]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control/). -## Step 4: Create a connector +### Step 4: Create a connector Now you can create a connector for the model. In this example, you'll create a connector to the Anthropic Claude model hosted on Amazon Bedrock: @@ -145,7 +153,7 @@ The response contains the connector ID for the newly created connector: } ``` -## Step 5: Register and deploy the model with guardrails +### Step 5: Register and deploy the model with guardrails To register an externally hosted model, provide the model group ID from step 3 and the connector ID from step 4 in the following request. To configure guardrails, include the `guardrails` object: @@ -227,7 +235,7 @@ When the operation is complete, the state changes to `COMPLETED`: } ``` -## Step 6 (Optional): Test the model +### Step 6 (Optional): Test the model To demonstrate how guardrails are applied, first run the predict operation that does not contain any excluded words: @@ -241,7 +249,7 @@ POST /_plugins/_ml/models/p94dYo4BrXGpZpgPp98E/_predict ``` {% include copy-curl.html %} -The response contains inference results: +The response contains the LLM answer: ```json { @@ -293,6 +301,339 @@ The response contains an error message because guardrails were triggered: Guardrails are also triggered when a prompt matches the supplied regular expression. +## Validating input/output using a guardrail model +**Introduced 2.15** +{: .label .label-purple } + +For more advanced validation, you can configure a guardrail model---a separate LLM whose purpose is to validate the user input and the LLM output. In this example, you'll configure two models: +- Chat model: An Anthropic Claude model hosted on Amazon Bedrock whose purpose is to hold a conversation with a user. +- Guardrail model: An OpenAI model whose purpose is to validate the user input and Anthropic Claude output. + +### Step 1: Create a connector for the guardrail model + +First, create a connector to the OpenAI guardrail model. Note that the OpenAI prompt instructs the model to respond only with the words `accept` or `reject`, depending on whether the input/output is acceptable. Additionally, the request contains the `response_filter` parameter, which specifies the field in which the guardrail model will provide the validation result: + +```json +POST /_plugins/_ml/connectors/_create +{ + "name": "openai", + "description": "openai", + "version": "1", + "protocol": "http", + "parameters": { + "endpoint": "api.openai.com", + "max_tokens": 7, + "temperature": 0, + "model": "gpt-3.5-turbo-instruct", + "prompt": "You are a helpful assistant and an expert judge of content quality. Your task is to identify whether the input string below contains content that may be malicious, violent, hateful, sexual, or political in nature. Your answer should consist of a single word, either reject or accept. If the input belongs to any of these categories, please write reject. Otherwise, write accept. \\n\\nHere is the input: ${parameters.question}. \\n\\nYour answer: ", + "response_filter": "$.choices[0].text" + }, + "credential": { + "openAI_key": "
+
+You can now chat in OpenSearch Dashboards, as shown in the following image.
+
+
\ No newline at end of file
diff --git a/_ml-commons-plugin/tutorials/conversational-search-cohere.md b/_ml-commons-plugin/tutorials/conversational-search-cohere.md
new file mode 100644
index 0000000000..e02f576b7c
--- /dev/null
+++ b/_ml-commons-plugin/tutorials/conversational-search-cohere.md
@@ -0,0 +1,228 @@
+---
+layout: default
+title: Conversational search with Cohere Command
+parent: Tutorials
+nav_order: 20
+---
+
+# Conversational search using the Cohere Command model
+
+This tutorial illustrates how to configure conversational search using the Cohere Command model. For more information, see [Conversational search]({{site.url}}{{site.baseurl}}/search-plugins/conversational-search/).
+
+Replace the placeholders beginning with the prefix `your_` with your own values.
+{: .note}
+
+Alternatively, you can build a RAG/conversational search using agents and tools. For more information, see [Retrieval-augmented generation chatbot]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/rag-conversational-agent/).
+
+## Prerequisite
+
+Ingest test data:
+
+```json
+POST _bulk
+{"index": {"_index": "qa_demo", "_id": "1"}}
+{"text": "Chart and table of population level and growth rate for the Ogden-Layton metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\nThe current metro area population of Ogden-Layton in 2023 is 750,000, a 1.63% increase from 2022.\nThe metro area population of Ogden-Layton in 2022 was 738,000, a 1.79% increase from 2021.\nThe metro area population of Ogden-Layton in 2021 was 725,000, a 1.97% increase from 2020.\nThe metro area population of Ogden-Layton in 2020 was 711,000, a 2.16% increase from 2019."}
+{"index": {"_index": "qa_demo", "_id": "2"}}
+{"text": "Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."}
+{"index": {"_index": "qa_demo", "_id": "3"}}
+{"text": "Chart and table of population level and growth rate for the Chicago metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Chicago in 2023 is 8,937,000, a 0.4% increase from 2022.\\nThe metro area population of Chicago in 2022 was 8,901,000, a 0.27% increase from 2021.\\nThe metro area population of Chicago in 2021 was 8,877,000, a 0.14% increase from 2020.\\nThe metro area population of Chicago in 2020 was 8,865,000, a 0.03% increase from 2019."}
+{"index": {"_index": "qa_demo", "_id": "4"}}
+{"text": "Chart and table of population level and growth rate for the Miami metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Miami in 2023 is 6,265,000, a 0.8% increase from 2022.\\nThe metro area population of Miami in 2022 was 6,215,000, a 0.78% increase from 2021.\\nThe metro area population of Miami in 2021 was 6,167,000, a 0.74% increase from 2020.\\nThe metro area population of Miami in 2020 was 6,122,000, a 0.71% increase from 2019."}
+{"index": {"_index": "qa_demo", "_id": "5"}}
+{"text": "Chart and table of population level and growth rate for the Austin metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Austin in 2023 is 2,228,000, a 2.39% increase from 2022.\\nThe metro area population of Austin in 2022 was 2,176,000, a 2.79% increase from 2021.\\nThe metro area population of Austin in 2021 was 2,117,000, a 3.12% increase from 2020.\\nThe metro area population of Austin in 2020 was 2,053,000, a 3.43% increase from 2019."}
+{"index": {"_index": "qa_demo", "_id": "6"}}
+{"text": "Chart and table of population level and growth rate for the Seattle metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Seattle in 2023 is 3,519,000, a 0.86% increase from 2022.\\nThe metro area population of Seattle in 2022 was 3,489,000, a 0.81% increase from 2021.\\nThe metro area population of Seattle in 2021 was 3,461,000, a 0.82% increase from 2020.\\nThe metro area population of Seattle in 2020 was 3,433,000, a 0.79% increase from 2019."}
+```
+{% include copy-curl.html %}
+
+## Step 1: Create a connector and register a model
+
+Conversational search only supports the [OpenAI](https://github.com/opensearch-project/ml-commons/blob/2.x/docs/remote_inference_blueprints/open_ai_connector_chat_blueprint.md)
+and [Amazon Bedrock Claude](https://github.com/opensearch-project/ml-commons/blob/2.x/docs/remote_inference_blueprints/bedrock_connector_anthropic_claude_blueprint.md) input/output styles.
+{: .important}
+
+This tutorial follows the Amazon Bedrock Claude model input/output style by:
+- Mapping the Cohere Command `message` input parameter to the `inputs` parameter in order to match the Cohere Claude model input style.
+- Using a post-processing function to convert the Cohere Command model output to the Claude model output style.
+
+Create a connector for the Cohere Command model:
+
+```json
+POST _plugins/_ml/connectors/_create
+{
+ "name": "Cohere Chat Model",
+ "description": "The connector to Cohere's public chat API",
+ "version": "1",
+ "protocol": "http",
+ "credential": {
+ "cohere_key": "your_cohere_api_key"
+ },
+ "parameters": {
+ "model": "command"
+ },
+ "actions": [
+ {
+ "action_type": "predict",
+ "method": "POST",
+ "url": "https://api.cohere.ai/v1/chat",
+ "headers": {
+ "Authorization": "Bearer ${credential.cohere_key}",
+ "Request-Source": "unspecified:opensearch"
+ },
+ "request_body": "{ \"message\": \"${parameters.inputs}\", \"model\": \"${parameters.model}\" }",
+ "post_process_function": "\n String escape(def input) { \n if (input.contains(\"\\\\\")) {\n input = input.replace(\"\\\\\", \"\\\\\\\\\");\n }\n if (input.contains(\"\\\"\")) {\n input = input.replace(\"\\\"\", \"\\\\\\\"\");\n }\n if (input.contains('\r')) {\n input = input = input.replace('\r', '\\\\r');\n }\n if (input.contains(\"\\\\t\")) {\n input = input.replace(\"\\\\t\", \"\\\\\\\\\\\\t\");\n }\n if (input.contains('\n')) {\n input = input.replace('\n', '\\\\n');\n }\n if (input.contains('\b')) {\n input = input.replace('\b', '\\\\b');\n }\n if (input.contains('\f')) {\n input = input.replace('\f', '\\\\f');\n }\n return input;\n }\n def name = 'response';\n def result = params.text;\n def json = '{ \"name\": \"' + name + '\",' +\n '\"dataAsMap\": { \"completion\": \"' + escape(result) +\n '\"}}';\n return json;\n \n "
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+Starting in OpenSearch 2.12, you can use the default `escape` function directly in the `post_process_function`:
+
+```json
+"post_process_function": " \n def name = 'response';\n def result = params.text;\n def json = '{ \"name\": \"' + name + '\",' +\n '\"dataAsMap\": { \"completion\": \"' + escape(result) +\n '\"}}';\n return json;"
+```
+{% include copy-curl.html %}
+
+Note the connector ID; you'll use it to register the model.
+
+Register the Cohere Command model:
+
+```json
+POST /_plugins/_ml/models/_register?deploy=true
+{
+ "name": "Cohere command model",
+ "function_name": "remote",
+ "description": "Cohere command model",
+ "connector_id": "your_connector_id"
+}
+```
+{% include copy-curl.html %}
+
+Note the model ID; you'll use it in the following steps.
+
+Test the model:
+
+```json
+POST /_plugins/_ml/models/your_model_id/_predict
+{
+ "parameters": {
+ "inputs": "What is the weather like in Seattle?"
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the LLM completion:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "response",
+ "dataAsMap": {
+ "completion": """It is difficult to provide a comprehensive answer without a specific location or time frame in mind.
+
+As an AI language model, I have no access to real-time data or the ability to provide live weather reports. Instead, I can offer some general information about Seattle's weather, which is known for its mild, wet climate.
+
+Located in the Pacific Northwest region of the United States, Seattle experiences a maritime climate with cool, dry summers and mild, wet winters. While it is best known for its rainy days, Seattle's annual rainfall is actually less than New York City and Boston.
+
+Would you like me to provide more details on Seattle's weather? Or, if you have a specific date or location in mind, I can try to retrieve real-time or historical weather information for you."""
+ }
+ }
+ ],
+ "status_code": 200
+ }
+ ]
+}
+```
+
+## Step 2: Configure conversational search
+
+Create a search pipeline containing a RAG processor:
+
+```json
+PUT /_search/pipeline/my-conversation-search-pipeline-cohere
+{
+ "response_processors": [
+ {
+ "retrieval_augmented_generation": {
+ "tag": "Demo pipeline",
+ "description": "Demo pipeline Using Cohere",
+ "model_id": "your_model_id_created_in_step1",
+ "context_field_list": [
+ "text"
+ ],
+ "system_prompt": "You are a helpful assistant",
+ "user_instructions": "Generate a concise and informative answer in less than 100 words for the given question"
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+To run a conversational search, specify its parameters in the `generative_qa_parameters` object:
+
+```json
+GET /qa_demo/_search?search_pipeline=my-conversation-search-pipeline-cohere
+{
+ "query": {
+ "match": {
+ "text": "What's the population increase of New York City from 2021 to 2023?"
+ }
+ },
+ "size": 1,
+ "_source": [
+ "text"
+ ],
+ "ext": {
+ "generative_qa_parameters": {
+ "llm_model": "bedrock/claude",
+ "llm_question": "What's the population increase of New York City from 2021 to 2023?",
+ "context_size": 5,
+ "timeout": 15
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the model's answer:
+
+```json
+{
+ "took": 1,
+ "timed_out": false,
+ "_shards": {
+ "total": 1,
+ "successful": 1,
+ "skipped": 0,
+ "failed": 0
+ },
+ "hits": {
+ "total": {
+ "value": 6,
+ "relation": "eq"
+ },
+ "max_score": 9.042081,
+ "hits": [
+ {
+ "_index": "qa_demo",
+ "_id": "2",
+ "_score": 9.042081,
+ "_source": {
+ "text": """Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."""
+ }
+ }
+ ]
+ },
+ "ext": {
+ "retrieval_augmented_generation": {
+ "answer": "The population of the New York City metro area increased by about 210,000 from 2021 to 2023. The 2021 population was 18,823,000, and in 2023 it was 18,937,000. The average growth rate is 0.23% yearly."
+ }
+ }
+}
+```
\ No newline at end of file
diff --git a/_ml-commons-plugin/tutorials/generate-embeddings.md b/_ml-commons-plugin/tutorials/generate-embeddings.md
new file mode 100644
index 0000000000..92b62b9fe8
--- /dev/null
+++ b/_ml-commons-plugin/tutorials/generate-embeddings.md
@@ -0,0 +1,336 @@
+---
+layout: default
+title: Generating embeddings
+parent: Tutorials
+nav_order: 5
+---
+
+# Generating embeddings for arrays of objects
+
+This tutorial illustrates how to generate embeddings for arrays of objects.
+
+Replace the placeholders beginning with the prefix `your_` with your own values.
+{: .note}
+
+## Step 1: Register an embedding model
+
+For this tutorial, you will use the [Amazon Bedrock Titan Embedding model](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html).
+
+First, follow the [Amazon Bedrock Titan blueprint example](https://github.com/opensearch-project/ml-commons/blob/2.x/docs/remote_inference_blueprints/bedrock_connector_titan_embedding_blueprint.md) to register and deploy the model.
+
+Test the model, providing the model ID:
+
+```json
+POST /_plugins/_ml/models/your_embedding_model_id/_predict
+{
+ "parameters": {
+ "inputText": "hello world"
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains inference results:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "sentence_embedding",
+ "data_type": "FLOAT32",
+ "shape": [ 1536 ],
+ "data": [0.7265625, -0.0703125, 0.34765625, ...]
+ }
+ ],
+ "status_code": 200
+ }
+ ]
+}
+```
+
+## Step 2: Create an ingest pipeline
+
+Follow the next set of steps to create an ingest pipeline for generating embeddings.
+
+### Step 2.1: Create a k-NN index
+
+First, create a k-NN index:
+
+```json
+PUT my_books
+{
+ "settings" : {
+ "index.knn" : "true",
+ "default_pipeline": "bedrock_embedding_foreach_pipeline"
+ },
+ "mappings": {
+ "properties": {
+ "books": {
+ "type": "nested",
+ "properties": {
+ "title_embedding": {
+ "type": "knn_vector",
+ "dimension": 1536
+ },
+ "title": {
+ "type": "text"
+ },
+ "description": {
+ "type": "text"
+ }
+ }
+ }
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+### Step 2.2: Create an ingest pipeline
+
+Then create an inner ingest pipeline to generate an embedding for one array element.
+
+This pipeline contains three processors:
+
+- `set` processor: The `text_embedding` processor is unable to identify the `_ingest._value.title` field. You must copy `_ingest._value.title` to a non-existing temporary field so that the `text_embedding` processor can process it.
+- `text_embedding` processor: Converts the value of the temporary field to an embedding.
+- `remove` processor: Removes the temporary field.
+
+To create such a pipeline, send the following request:
+
+```json
+PUT _ingest/pipeline/bedrock_embedding_pipeline
+{
+ "processors": [
+ {
+ "set": {
+ "field": "title_tmp",
+ "value": "{{_ingest._value.title}}"
+ }
+ },
+ {
+ "text_embedding": {
+ "model_id": your_embedding_model_id,
+ "field_map": {
+ "title_tmp": "_ingest._value.title_embedding"
+ }
+ }
+ },
+ {
+ "remove": {
+ "field": "title_tmp"
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+Create an ingest pipeline with a `foreach` processor that will apply the `bedrock_embedding_pipeline` to each element of the `books` array:
+
+```json
+PUT _ingest/pipeline/bedrock_embedding_foreach_pipeline
+{
+ "description": "Test nested embeddings",
+ "processors": [
+ {
+ "foreach": {
+ "field": "books",
+ "processor": {
+ "pipeline": {
+ "name": "bedrock_embedding_pipeline"
+ }
+ },
+ "ignore_failure": true
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+### Step 2.3: Simulate the pipeline
+
+First, you'll test the pipeline on an array that contains two book objects, both with a `title` field:
+
+```json
+POST _ingest/pipeline/bedrock_embedding_foreach_pipeline/_simulate
+{
+ "docs": [
+ {
+ "_index": "my_books",
+ "_id": "1",
+ "_source": {
+ "books": [
+ {
+ "title": "first book",
+ "description": "This is first book"
+ },
+ {
+ "title": "second book",
+ "description": "This is second book"
+ }
+ ]
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+The response contains generated embeddings for both objects in their `title_embedding` fields:
+
+```json
+{
+ "docs": [
+ {
+ "doc": {
+ "_index": "my_books",
+ "_id": "1",
+ "_source": {
+ "books": [
+ {
+ "title": "first book",
+ "title_embedding": [-1.1015625, 0.65234375, 0.7578125, ...],
+ "description": "This is first book"
+ },
+ {
+ "title": "second book",
+ "title_embedding": [-0.65234375, 0.21679688, 0.7265625, ...],
+ "description": "This is second book"
+ }
+ ]
+ },
+ "_ingest": {
+ "_value": null,
+ "timestamp": "2024-05-28T16:16:50.538929413Z"
+ }
+ }
+ }
+ ]
+}
+```
+
+Next, you'll test the pipeline on an array that contains two book objects, one with a `title` field and one without:
+
+```json
+POST _ingest/pipeline/bedrock_embedding_foreach_pipeline/_simulate
+{
+ "docs": [
+ {
+ "_index": "my_books",
+ "_id": "1",
+ "_source": {
+ "books": [
+ {
+ "title": "first book",
+ "description": "This is first book"
+ },
+ {
+ "description": "This is second book"
+ }
+ ]
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+The response contains generated embeddings for the object that contains the `title` field:
+
+```json
+{
+ "docs": [
+ {
+ "doc": {
+ "_index": "my_books",
+ "_id": "1",
+ "_source": {
+ "books": [
+ {
+ "title": "first book",
+ "title_embedding": [-1.1015625, 0.65234375, 0.7578125, ...],
+ "description": "This is first book"
+ },
+ {
+ "description": "This is second book"
+ }
+ ]
+ },
+ "_ingest": {
+ "_value": null,
+ "timestamp": "2024-05-28T16:19:03.942644042Z"
+ }
+ }
+ }
+ ]
+}
+```
+### Step 2.4: Test data ingestion
+
+Ingest one document:
+
+```json
+PUT my_books/_doc/1
+{
+ "books": [
+ {
+ "title": "first book",
+ "description": "This is first book"
+ },
+ {
+ "title": "second book",
+ "description": "This is second book"
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+Get the document:
+
+```json
+GET my_books/_doc/1
+```
+{% include copy-curl.html %}
+
+The response contains the generated embeddings:
+
+```json
+{
+ "_index": "my_books",
+ "_id": "1",
+ "_version": 1,
+ "_seq_no": 0,
+ "_primary_term": 1,
+ "found": true,
+ "_source": {
+ "books": [
+ {
+ "description": "This is first book",
+ "title": "first book",
+ "title_embedding": [-1.1015625, 0.65234375, 0.7578125, ...]
+ },
+ {
+ "description": "This is second book",
+ "title": "second book",
+ "title_embedding": [-0.65234375, 0.21679688, 0.7265625, ...]
+ }
+ ]
+ }
+}
+```
+
+You can also ingest several documents in bulk and test the generated embeddings by calling the Get Document API:
+
+```json
+POST _bulk
+{ "index" : { "_index" : "my_books" } }
+{ "books" : [{"title": "first book", "description": "This is first book"}, {"title": "second book", "description": "This is second book"}] }
+{ "index" : { "_index" : "my_books" } }
+{ "books" : [{"title": "third book", "description": "This is third book"}, {"description": "This is fourth book"}] }
+```
+{% include copy-curl.html %}
\ No newline at end of file
diff --git a/_ml-commons-plugin/tutorials/index.md b/_ml-commons-plugin/tutorials/index.md
new file mode 100644
index 0000000000..4479d0878f
--- /dev/null
+++ b/_ml-commons-plugin/tutorials/index.md
@@ -0,0 +1,26 @@
+---
+layout: default
+title: Tutorials
+has_children: true
+has_toc: false
+nav_order: 140
+---
+
+# Tutorials
+
+Using the OpenSearch machine learning (ML) framework, you can build various applications, from implementing conversational search to building your own chatbot. To learn more, explore the following ML tutorials:
+
+- **Semantic search**:
+ - [Generating embeddings for arrays of objects]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/generate-embeddings/)
+ - [Semantic search using byte-quantized vectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/semantic-search-byte-vectors/)
+
+- **Conversational search**:
+ - [Conversational search using the Cohere Command model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/conversational-search-cohere/)
+
+- **Reranking search results**:
+ - [Reranking search results using the Cohere Rerank model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/reranking-cohere/)
+
+- **Agents and tools**:
+ - [Retrieval-augmented generation (RAG) chatbot]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/rag-chatbot/)
+ - [RAG with a conversational flow agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/rag-conversational-agent/)
+ - [Build your own chatbot]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/build-chatbot/)
\ No newline at end of file
diff --git a/_ml-commons-plugin/tutorials/rag-chatbot.md b/_ml-commons-plugin/tutorials/rag-chatbot.md
new file mode 100644
index 0000000000..5dddded23a
--- /dev/null
+++ b/_ml-commons-plugin/tutorials/rag-chatbot.md
@@ -0,0 +1,346 @@
+---
+layout: default
+title: RAG chatbot
+parent: Tutorials
+nav_order: 50
+---
+
+# RAG chatbot
+
+One of the known limitations of large language models (LLMs) is that their knowledge base only contains information from the period of time during which they were trained. LLMs have no knowledge of recent events or of your internal data. You can augment the LLM knowledge base by using retrieval-augmented generation (RAG).
+
+This tutorial illustrates how to build your own chatbot using [agents and tools](https://opensearch.org/docs/latest/ml-commons-plugin/agents-tools/index/) and RAG. RAG supplements the LLM knowledge base with information contained in OpenSearch indexes.
+
+Replace the placeholders beginning with the prefix `your_` with your own values.
+{: .note}
+
+## Prerequisite
+
+Meet the prerequisite and follow Step 1 of the [RAG with a conversational flow agent tutorial]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/rag-conversational-agent/) to set up the `test_population_data` knowledge base index, which contains US city population data.
+
+Note the embedding model ID; you'll use it in the following steps.
+
+## Step 1: Set up a knowledge base
+
+First, create an ingest pipeline:
+
+```json
+PUT /_ingest/pipeline/test_tech_news_pipeline
+{
+ "description": "text embedding pipeline for tech news",
+ "processors": [
+ {
+ "text_embedding": {
+ "model_id": "your_text_embedding_model_id",
+ "field_map": {
+ "passage": "passage_embedding"
+ }
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+Next, create an index named `test_tech_news`, which contains recent tech news:
+
+```json
+PUT test_tech_news
+{
+ "mappings": {
+ "properties": {
+ "passage": {
+ "type": "text"
+ },
+ "passage_embedding": {
+ "type": "knn_vector",
+ "dimension": 384
+ }
+ }
+ },
+ "settings": {
+ "index": {
+ "knn.space_type": "cosinesimil",
+ "default_pipeline": "test_tech_news_pipeline",
+ "knn": "true"
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+Ingest data into the index:
+
+```json
+POST _bulk
+{"index":{"_index":"test_tech_news"}}
+{"c":"Apple Vision Pro is a mixed-reality headset developed by Apple Inc. It was announced on June 5, 2023, at Apple's Worldwide Developers Conference, and pre-orders began on January 19, 2024. It became available for purchase on February 2, 2024, in the United States.[10] A worldwide launch has yet to be scheduled. The Vision Pro is Apple's first new major product category since the release of the Apple Watch in 2015.[11]\n\nApple markets the Vision Pro as a \"spatial computer\" where digital media is integrated with the real world. Physical inputs—such as motion gestures, eye tracking, and speech recognition—can be used to interact with the system.[10] Apple has avoided marketing the device as a virtual reality headset, along with the use of the terms \"virtual reality\" and \"augmented reality\" when discussing the product in presentations and marketing.[12]\n\nThe device runs visionOS,[13] a mixed-reality operating system derived from iOS frameworks using a 3D user interface; it supports multitasking via windows that appear to float within the user's surroundings,[14] as seen by cameras built into the headset. A dial on the top of the headset can be used to mask the camera feed with a virtual environment to increase immersion. The OS supports avatars (officially called \"Personas\"), which are generated by scanning the user's face; a screen on the front of the headset displays a rendering of the avatar's eyes (\"EyeSight\"), which are used to indicate the user's level of immersion to bystanders, and assist in communication.[15]"}
+{"index":{"_index":"test_tech_news"}}
+{"passage":"LLaMA (Large Language Model Meta AI) is a family of autoregressive large language models (LLMs), released by Meta AI starting in February 2023.\n\nFor the first version of LLaMA, four model sizes were trained: 7, 13, 33, and 65 billion parameters. LLaMA's developers reported that the 13B parameter model's performance on most NLP benchmarks exceeded that of the much larger GPT-3 (with 175B parameters) and that the largest model was competitive with state of the art models such as PaLM and Chinchilla.[1] Whereas the most powerful LLMs have generally been accessible only through limited APIs (if at all), Meta released LLaMA's model weights to the research community under a noncommercial license.[2] Within a week of LLaMA's release, its weights were leaked to the public on 4chan via BitTorrent.[3]\n\nIn July 2023, Meta released several models as Llama 2, using 7, 13 and 70 billion parameters.\n\nLLaMA-2\n\nOn July 18, 2023, in partnership with Microsoft, Meta announced LLaMA-2, the next generation of LLaMA. Meta trained and released LLaMA-2 in three model sizes: 7, 13, and 70 billion parameters.[4] The model architecture remains largely unchanged from that of LLaMA-1 models, but 40% more data was used to train the foundational models.[5] The accompanying preprint[5] also mentions a model with 34B parameters that might be released in the future upon satisfying safety targets.\n\nLLaMA-2 includes both foundational models and models fine-tuned for dialog, called LLaMA-2 Chat. In further departure from LLaMA-1, all models are released with weights, and are free for many commercial use cases. However, due to some remaining restrictions, the description of LLaMA as open source has been disputed by the Open Source Initiative (known for maintaining the Open Source Definition).[6]\n\nIn November 2023, research conducted by Patronus AI, an artificial intelligence startup company, compared performance of LLaMA-2, OpenAI's GPT-4 and GPT-4-Turbo, and Anthropic's Claude2 on two versions of a 150-question test about information in SEC filings (e.g. Form 10-K, Form 10-Q, Form 8-K, earnings reports, earnings call transcripts) submitted by public companies to the agency where one version of the test required the generative AI models to use a retrieval system to locate the specific SEC filing to answer the questions while the other version provided the specific SEC filing to the models to answer the question (i.e. in a long context window). On the retrieval system version, GPT-4-Turbo and LLaMA-2 both failed to produce correct answers to 81% of the questions, while on the long context window version, GPT-4-Turbo and Claude-2 failed to produce correct answers to 21% and 24% of the questions respectively.[7][8]"}
+{"index":{"_index":"test_tech_news"}}
+{"passage":"Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon via a single API, along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI. Using Amazon Bedrock, you can easily experiment with and evaluate top FMs for your use case, privately customize them with your data using techniques such as fine-tuning and Retrieval Augmented Generation (RAG), and build agents that execute tasks using your enterprise systems and data sources. Since Amazon Bedrock is serverless, you don't have to manage any infrastructure, and you can securely integrate and deploy generative AI capabilities into your applications using the AWS services you are already familiar with."}
+```
+{% include copy-curl.html %}
+
+## Step 2: Prepare an LLM
+
+Follow [step 2 of the RAG with a conversational flow agent tutorial]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/rag-conversational-agent/#step-2-prepare-an-llm) to configure the Amazon Bedrock Claude model.
+
+Note the model ID; you'll use it in the following steps.
+
+## Step 3: Create an agent
+
+For this tutorial, you will create an agent of the `conversational` type.
+
+Both the `conversational_flow` and `conversational` agents support conversation history.
+
+The `conversational_flow` and `conversational` agents differ in the following ways:
+
+- A `conversational_flow` agent runs tools sequentially, in a predefined order.
+- A `conversational` agent dynamically chooses which tool to run next.
+
+In this tutorial, the agent includes two tools: One provides recent population data, and the other contains tech news.
+
+The agent has the following parameters:
+
+- `"max_iteration": 5`: The agent runs the LLM a maximum of five times.
+- `"response_filter": "$.completion"`: Needed to retrieve the LLM answer from the Amazon Bedrock Claude model response.
+- `"doc_size": 3` (in `population_data_knowledge_base`): Specifies to return the top three documents.
+
+Create an agent with the preceding specifications:
+
+```json
+POST _plugins/_ml/agents/_register
+{
+ "name": "Chat Agent with RAG",
+ "type": "conversational",
+ "description": "this is a test agent",
+ "llm": {
+ "model_id": "your_llm_model_id",
+ "parameters": {
+ "max_iteration": 5,
+ "response_filter": "$.completion"
+ }
+ },
+ "memory": {
+ "type": "conversation_index"
+ },
+ "tools": [
+ {
+ "type": "VectorDBTool",
+ "name": "population_data_knowledge_base",
+ "description": "This tool provides population data of US cities.",
+ "parameters": {
+ "input": "${parameters.question}",
+ "index": "test_population_data",
+ "source_field": [
+ "population_description"
+ ],
+ "model_id": "your_text_embedding_model_id",
+ "embedding_field": "population_description_embedding",
+ "doc_size": 3
+ }
+ },
+ {
+ "type": "VectorDBTool",
+ "name": "tech_news_knowledge_base",
+ "description": "This tool provides recent tech news.",
+ "parameters": {
+ "input": "${parameters.question}",
+ "index": "test_tech_news",
+ "source_field": [
+ "passage"
+ ],
+ "model_id": "your_text_embedding_model_id",
+ "embedding_field": "passage_embedding",
+ "doc_size": 2
+ }
+ }
+ ],
+ "app_type": "chat_with_rag"
+}
+```
+{% include copy-curl.html %}
+
+Note the agent ID; you'll use it in the next step.
+
+## Step 4: Test the agent
+
+The `conversational` agent supports a `verbose` option. You can set `verbose` to `true` to obtain detailed steps.
+
+Alternatively, you can call the [Get Message Traces API](ml-commons-plugin/api/memory-apis/get-message-traces/):
+
+```json
+GET _plugins/_ml/memory/message/message_id/traces
+```
+{% include copy-curl.html %}
+
+### Start a conversation
+
+Ask a question related to tech news:
+
+```json
+POST _plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "What's vision pro",
+ "verbose": true
+ }
+}
+```
+{% include copy-curl.html %}
+
+In the response, note that the agent runs the `tech_news_knowledge_base` tool to obtain the top two documents. The agent then passes these documents as context to the LLM. The LLM uses the context to produce the answer:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "memory_id",
+ "result": "eLVSxI0B8vrNLhb9nxto"
+ },
+ {
+ "name": "parent_interaction_id",
+ "result": "ebVSxI0B8vrNLhb9nxty"
+ },
+ {
+ "name": "response",
+ "result": """{
+ "thought": "I don't have enough context to answer the question directly. Let me check the tech_news_knowledge_base tool to see if it can provide more information.",
+ "action": "tech_news_knowledge_base",
+ "action_input": "{\"query\":\"What's vision pro\"}"
+}"""
+ },
+ {
+ "name": "response",
+ "result": """{"_index":"test_tech_news","_source":{"passage":"Apple Vision Pro is a mixed-reality headset developed by Apple Inc. It was announced on June 5, 2023, at Apple\u0027s Worldwide Developers Conference, and pre-orders began on January 19, 2024. It became available for purchase on February 2, 2024, in the United States.[10] A worldwide launch has yet to be scheduled. The Vision Pro is Apple\u0027s first new major product category since the release of the Apple Watch in 2015.[11]\n\nApple markets the Vision Pro as a \"spatial computer\" where digital media is integrated with the real world. Physical inputs—such as motion gestures, eye tracking, and speech recognition—can be used to interact with the system.[10] Apple has avoided marketing the device as a virtual reality headset, along with the use of the terms \"virtual reality\" and \"augmented reality\" when discussing the product in presentations and marketing.[12]\n\nThe device runs visionOS,[13] a mixed-reality operating system derived from iOS frameworks using a 3D user interface; it supports multitasking via windows that appear to float within the user\u0027s surroundings,[14] as seen by cameras built into the headset. A dial on the top of the headset can be used to mask the camera feed with a virtual environment to increase immersion. The OS supports avatars (officially called \"Personas\"), which are generated by scanning the user\u0027s face; a screen on the front of the headset displays a rendering of the avatar\u0027s eyes (\"EyeSight\"), which are used to indicate the user\u0027s level of immersion to bystanders, and assist in communication.[15]"},"_id":"lrU8xI0B8vrNLhb9yBpV","_score":0.6700683}
+{"_index":"test_tech_news","_source":{"passage":"Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon via a single API, along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI. Using Amazon Bedrock, you can easily experiment with and evaluate top FMs for your use case, privately customize them with your data using techniques such as fine-tuning and Retrieval Augmented Generation (RAG), and build agents that execute tasks using your enterprise systems and data sources. Since Amazon Bedrock is serverless, you don\u0027t have to manage any infrastructure, and you can securely integrate and deploy generative AI capabilities into your applications using the AWS services you are already familiar with."},"_id":"mLU8xI0B8vrNLhb9yBpV","_score":0.5604863}
+"""
+ },
+ {
+ "name": "response",
+ "result": "Vision Pro is a mixed-reality headset developed by Apple that was announced in 2023. It uses cameras and sensors to overlay digital objects and information on the real world. The device runs an operating system called visionOS that allows users to interact with windows and apps in a 3D environment using gestures, eye tracking, and voice commands."
+ }
+ ]
+ }
+ ]
+}
+```
+
+You can trace the detailed steps by using the Get Traces API:
+
+```
+GET _plugins/_ml/memory/message/ebVSxI0B8vrNLhb9nxty/traces
+```
+{% include copy-curl.html %}
+
+Ask a question related to the population data:
+
+```json
+POST _plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "What's the population of Seattle 2023",
+ "verbose": true
+ }
+}
+```
+{% include copy-curl.html %}
+
+In the response, note that the agent runs the `population_data_knowledge_base` tool to obtain the top three documents. The agent then passes these documents as context to the LLM. The LLM uses the context to produce the answer:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "memory_id",
+ "result": "l7VUxI0B8vrNLhb9sRuQ"
+ },
+ {
+ "name": "parent_interaction_id",
+ "result": "mLVUxI0B8vrNLhb9sRub"
+ },
+ {
+ "name": "response",
+ "result": """{
+ "thought": "Let me check the population data tool to find the most recent population estimate for Seattle",
+ "action": "population_data_knowledge_base",
+ "action_input": "{\"city\":\"Seattle\"}"
+}"""
+ },
+ {
+ "name": "response",
+ "result": """{"_index":"test_population_data","_source":{"population_description":"Chart and table of population level and growth rate for the Seattle metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Seattle in 2023 is 3,519,000, a 0.86% increase from 2022.\\nThe metro area population of Seattle in 2022 was 3,489,000, a 0.81% increase from 2021.\\nThe metro area population of Seattle in 2021 was 3,461,000, a 0.82% increase from 2020.\\nThe metro area population of Seattle in 2020 was 3,433,000, a 0.79% increase from 2019."},"_id":"BxF5vo0BubpYKX5ER0fT","_score":0.65775126}
+{"_index":"test_population_data","_source":{"population_description":"Chart and table of population level and growth rate for the Seattle metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Seattle in 2023 is 3,519,000, a 0.86% increase from 2022.\\nThe metro area population of Seattle in 2022 was 3,489,000, a 0.81% increase from 2021.\\nThe metro area population of Seattle in 2021 was 3,461,000, a 0.82% increase from 2020.\\nThe metro area population of Seattle in 2020 was 3,433,000, a 0.79% increase from 2019."},"_id":"7DrZvo0BVR2NrurbRIAE","_score":0.65775126}
+{"_index":"test_population_data","_source":{"population_description":"Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."},"_id":"AxF5vo0BubpYKX5ER0fT","_score":0.56461215}
+"""
+ },
+ {
+ "name": "response",
+ "result": "According to the population data tool, the population of Seattle in 2023 is approximately 3,519,000 people, a 0.86% increase from 2022."
+ }
+ ]
+ }
+ ]
+}
+```
+
+### Continue a conversation
+
+To continue a previous conversation, provide its conversation ID in the `memory_id` parameter:
+
+```json
+POST _plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "What's the population of Austin 2023, compared with Seattle",
+ "memory_id": "l7VUxI0B8vrNLhb9sRuQ",
+ "verbose": true
+ }
+}
+```
+{% include copy-curl.html %}
+
+In the response, note that the `population_data_knowledge_base` doesn't return the population of Seattle. Instead, the agent learns the population of Seattle by referencing historical messages:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "memory_id",
+ "result": "l7VUxI0B8vrNLhb9sRuQ"
+ },
+ {
+ "name": "parent_interaction_id",
+ "result": "B7VkxI0B8vrNLhb9mxy0"
+ },
+ {
+ "name": "response",
+ "result": """{
+ "thought": "Let me check the population data tool first",
+ "action": "population_data_knowledge_base",
+ "action_input": "{\"city\":\"Austin\",\"year\":2023}"
+}"""
+ },
+ {
+ "name": "response",
+ "result": """{"_index":"test_population_data","_source":{"population_description":"Chart and table of population level and growth rate for the Austin metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Austin in 2023 is 2,228,000, a 2.39% increase from 2022.\\nThe metro area population of Austin in 2022 was 2,176,000, a 2.79% increase from 2021.\\nThe metro area population of Austin in 2021 was 2,117,000, a 3.12% increase from 2020.\\nThe metro area population of Austin in 2020 was 2,053,000, a 3.43% increase from 2019."},"_id":"BhF5vo0BubpYKX5ER0fT","_score":0.69129956}
+{"_index":"test_population_data","_source":{"population_description":"Chart and table of population level and growth rate for the Austin metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Austin in 2023 is 2,228,000, a 2.39% increase from 2022.\\nThe metro area population of Austin in 2022 was 2,176,000, a 2.79% increase from 2021.\\nThe metro area population of Austin in 2021 was 2,117,000, a 3.12% increase from 2020.\\nThe metro area population of Austin in 2020 was 2,053,000, a 3.43% increase from 2019."},"_id":"6zrZvo0BVR2NrurbRIAE","_score":0.69129956}
+{"_index":"test_population_data","_source":{"population_description":"Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."},"_id":"AxF5vo0BubpYKX5ER0fT","_score":0.61015373}
+"""
+ },
+ {
+ "name": "response",
+ "result": "According to the population data tool, the population of Austin in 2023 is approximately 2,228,000 people, a 2.39% increase from 2022. This is lower than the population of Seattle in 2023 which is approximately 3,519,000 people, a 0.86% increase from 2022."
+ }
+ ]
+ }
+ ]
+}
+```
\ No newline at end of file
diff --git a/_ml-commons-plugin/tutorials/rag-conversational-agent.md b/_ml-commons-plugin/tutorials/rag-conversational-agent.md
new file mode 100644
index 0000000000..86fe38416a
--- /dev/null
+++ b/_ml-commons-plugin/tutorials/rag-conversational-agent.md
@@ -0,0 +1,838 @@
+---
+layout: default
+title: RAG chatbot with a conversational flow agent
+parent: Tutorials
+nav_order: 40
+---
+
+# RAG chatbot with a conversational flow agent
+
+This tutorial explains how to use a conversational flow agent to build a retrieval-augmented generation (RAG) application with your OpenSearch data as a knowledge base.
+
+Replace the placeholders beginning with the prefix `your_` with your own values.
+{: .note}
+
+An alternative way to build RAG conversational search is to use a RAG pipeline. For more information, see [Conversational search using the Cohere Command model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/tutorials/conversational-search-cohere/).
+
+## Prerequisite
+
+In this tutorial, you'll build a RAG application that provides an OpenSearch [k-NN index]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/) as a knowledge base for a large language model (LLM). For data retrieval, you'll use [semantic search]({{site.url}}{{site.baseurl}}/search-plugins/semantic-search/). For a comprehensive semantic search tutorial, see [Neural search tutorial]({{site.url}}{{site.baseurl}}/search-plugins/neural-search-tutorial/).
+
+First, you'll need to update your cluster settings. If you don't have a dedicated machine learning (ML) node, set `"plugins.ml_commons.only_run_on_ml_node": false`. To avoid triggering a native memory circuit breaker, set `"plugins.ml_commons.native_memory_threshold"` to 100%:
+
+```json
+PUT _cluster/settings
+{
+ "persistent": {
+ "plugins.ml_commons.only_run_on_ml_node": false,
+ "plugins.ml_commons.native_memory_threshold": 100,
+ "plugins.ml_commons.agent_framework_enabled": true
+ }
+}
+```
+{% include copy-curl.html %}
+
+## Step 1: Prepare the knowledge base
+
+Use the following steps to prepare the knowledge base that will supplement the LLM's knowledge.
+
+### Step 1.1: Register a text embedding model
+
+Register a text embedding model that will translate text into vector embeddings:
+
+```json
+POST /_plugins/_ml/models/_register
+{
+ "name": "huggingface/sentence-transformers/all-MiniLM-L12-v2",
+ "version": "1.0.1",
+ "model_format": "TORCH_SCRIPT"
+}
+```
+{% include copy-curl.html %}
+
+Note the text embedding model ID; you'll use it in the following steps.
+
+As an alternative, you can get the model ID by calling the [Get Task API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/tasks-apis/get-task/):
+
+```json
+GET /_plugins/_ml/tasks/your_task_id
+```
+{% include copy-curl.html %}
+
+Deploy the model:
+
+```json
+POST /_plugins/_ml/models/your_text_embedding_model_id/_deploy
+```
+{% include copy-curl.html %}
+
+Test the model:
+
+```json
+POST /_plugins/_ml/models/your_text_embedding_model_id/_predict
+{
+ "text_docs":[ "today is sunny"],
+ "return_number": true,
+ "target_response": ["sentence_embedding"]
+}
+```
+{% include copy-curl.html %}
+
+For more information about using models within your OpenSearch cluster, see [Pretrained models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/).
+
+### Step 1.2: Create an ingest pipeline
+
+Create an ingest pipeline with a text embedding processor, which can invoke the model created in the previous step to generate embeddings from text fields:
+
+```json
+PUT /_ingest/pipeline/test_population_data_pipeline
+{
+ "description": "text embedding pipeline",
+ "processors": [
+ {
+ "text_embedding": {
+ "model_id": "your_text_embedding_model_id",
+ "field_map": {
+ "population_description": "population_description_embedding"
+ }
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+For more information about ingest pipelines, see [Ingest pipelines]({{site.url}}{{site.baseurl}}/ingest-pipelines/).
+
+### Step 1.3: Create a k-NN index
+
+Create a k-NN index specifying the ingest pipeline as a default pipeline:
+
+```json
+PUT test_population_data
+{
+ "mappings": {
+ "properties": {
+ "population_description": {
+ "type": "text"
+ },
+ "population_description_embedding": {
+ "type": "knn_vector",
+ "dimension": 384
+ }
+ }
+ },
+ "settings": {
+ "index": {
+ "knn.space_type": "cosinesimil",
+ "default_pipeline": "test_population_data_pipeline",
+ "knn": "true"
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+For more information about k-NN indexes, see [k-NN index]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/).
+
+### Step 1.4: Ingest data
+
+Ingest test data into the k-NN index:
+
+```json
+POST _bulk
+{"index": {"_index": "test_population_data"}}
+{"population_description": "Chart and table of population level and growth rate for the Ogden-Layton metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\nThe current metro area population of Ogden-Layton in 2023 is 750,000, a 1.63% increase from 2022.\nThe metro area population of Ogden-Layton in 2022 was 738,000, a 1.79% increase from 2021.\nThe metro area population of Ogden-Layton in 2021 was 725,000, a 1.97% increase from 2020.\nThe metro area population of Ogden-Layton in 2020 was 711,000, a 2.16% increase from 2019."}
+{"index": {"_index": "test_population_data"}}
+{"population_description": "Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."}
+{"index": {"_index": "test_population_data"}}
+{"population_description": "Chart and table of population level and growth rate for the Chicago metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Chicago in 2023 is 8,937,000, a 0.4% increase from 2022.\\nThe metro area population of Chicago in 2022 was 8,901,000, a 0.27% increase from 2021.\\nThe metro area population of Chicago in 2021 was 8,877,000, a 0.14% increase from 2020.\\nThe metro area population of Chicago in 2020 was 8,865,000, a 0.03% increase from 2019."}
+{"index": {"_index": "test_population_data"}}
+{"population_description": "Chart and table of population level and growth rate for the Miami metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Miami in 2023 is 6,265,000, a 0.8% increase from 2022.\\nThe metro area population of Miami in 2022 was 6,215,000, a 0.78% increase from 2021.\\nThe metro area population of Miami in 2021 was 6,167,000, a 0.74% increase from 2020.\\nThe metro area population of Miami in 2020 was 6,122,000, a 0.71% increase from 2019."}
+{"index": {"_index": "test_population_data"}}
+{"population_description": "Chart and table of population level and growth rate for the Austin metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Austin in 2023 is 2,228,000, a 2.39% increase from 2022.\\nThe metro area population of Austin in 2022 was 2,176,000, a 2.79% increase from 2021.\\nThe metro area population of Austin in 2021 was 2,117,000, a 3.12% increase from 2020.\\nThe metro area population of Austin in 2020 was 2,053,000, a 3.43% increase from 2019."}
+{"index": {"_index": "test_population_data"}}
+{"population_description": "Chart and table of population level and growth rate for the Seattle metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Seattle in 2023 is 3,519,000, a 0.86% increase from 2022.\\nThe metro area population of Seattle in 2022 was 3,489,000, a 0.81% increase from 2021.\\nThe metro area population of Seattle in 2021 was 3,461,000, a 0.82% increase from 2020.\\nThe metro area population of Seattle in 2020 was 3,433,000, a 0.79% increase from 2019."}
+```
+{% include copy-curl.html %}
+
+## Step 2: Prepare an LLM
+
+This tutorial uses the [Amazon Bedrock Claude model](https://aws.amazon.com/bedrock/claude/) for conversational search. You can also use other LLMs. For more information about using externally hosted models, see [Connecting to externally hosted models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/index/).
+
+### Step 2.1: Create a connector
+
+Create a connector for the Claude model:
+
+```json
+POST /_plugins/_ml/connectors/_create
+{
+ "name": "BedRock Claude instant-v1 Connector ",
+ "description": "The connector to BedRock service for claude model",
+ "version": 1,
+ "protocol": "aws_sigv4",
+ "parameters": {
+ "region": "us-east-1",
+ "service_name": "bedrock",
+ "anthropic_version": "bedrock-2023-05-31",
+ "max_tokens_to_sample": 8000,
+ "temperature": 0.0001,
+ "response_filter": "$.completion"
+ },
+ "credential": {
+ "access_key": "your_aws_access_key",
+ "secret_key": "your_aws_secret_key",
+ "session_token": "your_aws_session_token"
+ },
+ "actions": [
+ {
+ "action_type": "predict",
+ "method": "POST",
+ "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-instant-v1/invoke",
+ "headers": {
+ "content-type": "application/json",
+ "x-amz-content-sha256": "required"
+ },
+ "request_body": "{\"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature}, \"anthropic_version\":\"${parameters.anthropic_version}\" }"
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+Note the connector ID; you'll use it to register the model.
+
+### Step 2.2: Register the model
+
+Register the Claude model hosted on Amazon Bedrock:
+
+```json
+POST /_plugins/_ml/models/_register
+{
+ "name": "Bedrock Claude Instant model",
+ "function_name": "remote",
+ "description": "Bedrock Claude instant-v1 model",
+ "connector_id": "your_LLM_connector_id"
+}
+```
+{% include copy-curl.html %}
+
+Note the LLM model ID; you'll use it in the following steps.
+
+### Step 2.3: Deploy the model
+
+Deploy the Claude model:
+
+```json
+POST /_plugins/_ml/models/your_LLM_model_id/_deploy
+```
+{% include copy-curl.html %}
+
+### Step 2.4: Test the model
+
+To test the model, send a Predict API request:
+
+```json
+POST /_plugins/_ml/models/your_LLM_model_id/_predict
+{
+ "parameters": {
+ "prompt": "\n\nHuman: how are you? \n\nAssistant:"
+ }
+}
+```
+{% include copy-curl.html %}
+
+## Step 3: Register an agent
+
+OpenSearch provides the following agent types: `flow`, `conversational_flow`, and `conversational`. For more information about agents, see [Agents]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/index/#agents).
+
+You will use a `conversational_flow` agent in this tutorial. The agent consists of the following:
+
+- Meta info: `name`, `type`, and `description`.
+- `app_type`: Differentiates between application types.
+- `memory`: Stores user questions and LLM responses as a conversation so that an agent can retrieve conversation history from memory and continue the same conversation.
+- `tools`: Defines a list of tools to use. The agent will run these tools sequentially.
+
+To register an agent, send the following request:
+
+```json
+POST /_plugins/_ml/agents/_register
+{
+ "name": "population data analysis agent",
+ "type": "conversational_flow",
+ "description": "This is a demo agent for population data analysis",
+ "app_type": "rag",
+ "memory": {
+ "type": "conversation_index"
+ },
+ "tools": [
+ {
+ "type": "VectorDBTool",
+ "name": "population_knowledge_base",
+ "parameters": {
+ "model_id": "your_text_embedding_model_id",
+ "index": "test_population_data",
+ "embedding_field": "population_description_embedding",
+ "source_field": [
+ "population_description"
+ ],
+ "input": "${parameters.question}"
+ }
+ },
+ {
+ "type": "MLModelTool",
+ "name": "bedrock_claude_model",
+ "description": "A general tool to answer any question",
+ "parameters": {
+ "model_id": "your_LLM_model_id",
+ "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\nContext:\n${parameters.population_knowledge_base.output:-}\n\n${parameters.chat_history:-}\n\nHuman:${parameters.question}\n\nAssistant:"
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+OpenSearch responds with an agent ID:
+
+```json
+{
+ "agent_id": "fQ75lI0BHcHmo_czdqcJ"
+}
+```
+
+Note the agent ID; you'll use it in the next step.
+
+## Step 4: Run the agent
+
+You'll run the agent to analyze the increase in Seattle's population. When you run this agent, the agent will create a new conversation. Later, you can continue this conversation by asking other questions.
+
+### Step 4.1: Start a new conversation
+
+First, start a new conversation by asking the LLM a question:
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "what's the population increase of Seattle from 2021 to 2023?"
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the answer generated by the LLM:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "memory_id",
+ "result": "gQ75lI0BHcHmo_cz2acL"
+ },
+ {
+ "name": "parent_message_id",
+ "result": "gg75lI0BHcHmo_cz2acZ"
+ },
+ {
+ "name": "bedrock_claude_model",
+ "result": """ Based on the context given:
+- The metro area population of Seattle in 2021 was 3,461,000
+- The current metro area population of Seattle in 2023 is 3,519,000
+- So the population increase of Seattle from 2021 to 2023 is 3,519,000 - 3,461,000 = 58,000"""
+ }
+ ]
+ }
+ ]
+}
+```
+
+The response contains the following fields:
+
+- `memory_id` is the identifier for the memory (conversation) that groups all messages within a single conversation. Note this ID; you'll use it in the next step.
+- `parent_message_id` is the identifier for the current message (one question/answer) between the human and the LLM. One memory can contain multiple messages.
+
+To obtain memory details, call the [Get Memory API](ml-commons-plugin/api/memory-apis/get-memory/):
+
+```json
+GET /_plugins/_ml/memory/gQ75lI0BHcHmo_cz2acL
+```
+{% include copy-curl.html %}
+
+To obtain all messages within a memory, call the [Get Messages API](ml-commons-plugin/api/memory-apis/get-message/):
+
+```json
+GET /_plugins/_ml/memory/gQ75lI0BHcHmo_cz2acL/messages
+```
+{% include copy-curl.html %}
+
+To obtain message details, call the [Get Message API](ml-commons-plugin/api/memory-apis/get-message/):
+
+```json
+GET /_plugins/_ml/memory/message/gg75lI0BHcHmo_cz2acZ
+```
+{% include copy-curl.html %}
+
+For debugging purposes, you can obtain trace data for a message by calling the [Get Message Traces API](ml-commons-plugin/api/memory-apis/get-message-traces/):
+
+```json
+GET /_plugins/_ml/memory/message/gg75lI0BHcHmo_cz2acZ/traces
+```
+{% include copy-curl.html %}
+
+### 4.2 Continue a conversation by asking new questions
+
+To continue the same conversation, provide the memory ID from the previous step.
+
+Additionally, you can provide the following parameters:
+
+- `message_history_limit`: Specify how many historical messages you want included in the new question/answer round for an agent.
+- `prompt`: Use this parameter to customize the LLM prompt. For example, the following example adds a new instruction `always learn useful information from chat history`
+and a new parameter `next_action`:
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "What's the population of New York City in 2023?",
+ "next_action": "then compare with Seattle population of 2023",
+ "memory_id": "gQ75lI0BHcHmo_cz2acL",
+ "message_history_limit": 5,
+ "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\nContext:\n${parameters.population_knowledge_base.output:-}\n\n${parameters.chat_history:-}\n\nHuman:always learn useful information from chat history\nHuman:${parameters.question}, ${parameters.next_action}\n\nAssistant:"
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the answer generated by the LLM:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "memory_id",
+ "result": "gQ75lI0BHcHmo_cz2acL"
+ },
+ {
+ "name": "parent_message_id",
+ "result": "wQ4JlY0BHcHmo_cz8Kc-"
+ },
+ {
+ "name": "bedrock_claude_model",
+ "result": """ Based on the context given:
+- The current metro area population of New York City in 2023 is 18,937,000
+- The current metro area population of Seattle in 2023 is 3,519,000
+- So the population of New York City in 2023 (18,937,000) is much higher than the population of Seattle in 2023 (3,519,000)"""
+ }
+ ]
+ }
+ ]
+}
+```
+
+If you know which tool the agent should use to execute a particular Predict API request, you can specify the tool when executing the agent. For example, if you want to translate the preceding answer into Chinese, you don't need to retrieve any data from the knowledge base. To run only the Claude model, specify the `bedrock_claude_model` tool in the `selected_tools` parameter:
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "Translate last answer into Chinese?",
+ "selected_tools": ["bedrock_claude_model"]
+ }
+}
+```
+{% include copy-curl.html %}
+
+The agent will run the tools one by one in the new order defined in `selected_tools`.
+{: .note}
+
+## Configuring multiple knowledge bases
+
+You can configure multiple knowledge bases for an agent. For example, if you have both product description and comment data, you can configure the agent with the following two tools:
+
+```json
+{
+ "name": "My product agent",
+ "type": "conversational_flow",
+ "description": "This is an agent with product description and comments knowledge bases.",
+ "memory": {
+ "type": "conversation_index"
+ },
+ "app_type": "rag",
+ "tools": [
+ {
+ "type": "VectorDBTool",
+ "name": "product_description_vectordb",
+ "parameters": {
+ "model_id": "your_embedding_model_id",
+ "index": "product_description_data",
+ "embedding_field": "product_description_embedding",
+ "source_field": [
+ "product_description"
+ ],
+ "input": "${parameters.question}"
+ }
+ },
+ {
+ "type": "VectorDBTool",
+ "name": "product_comments_vectordb",
+ "parameters": {
+ "model_id": "your_embedding_model_id",
+ "index": "product_comments_data",
+ "embedding_field": "product_comment_embedding",
+ "source_field": [
+ "product_comment"
+ ],
+ "input": "${parameters.question}"
+ }
+ },
+ {
+ "type": "MLModelTool",
+ "description": "A general tool to answer any question",
+ "parameters": {
+ "model_id": "{{llm_model_id}}",
+ "prompt": "\n\nHuman:You are a professional product recommendation engine. You will always recommend product based on the given context. If you don't have enough context, you will ask Human to provide more information. If you don't see any related product to recommend, just say we don't have such product. \n\n Context:\n${parameters.product_description_vectordb.output}\n\n${parameters.product_comments_vectordb.output}\n\nHuman:${parameters.question}\n\nAssistant:"
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+When you run the agent, the agent will query product description and comment data and then send the query results and the question to the LLM.
+
+To query a specific knowledge base, specify it in `selected_tools`. For example, if the question relates only to product comments, you can retrieve information only from `product_comments_vectordb`:
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "What feature people like the most for Amazon Echo Dot",
+ "selected_tools": ["product_comments_vectordb", "MLModelTool"]
+ }
+}
+```
+{% include copy-curl.html %}
+
+## Running queries on an index
+
+Use `SearchIndexTool` to run any OpenSearch query on any index.
+
+### Setup: Register an agent
+
+```json
+POST /_plugins/_ml/agents/_register
+{
+ "name": "Demo agent",
+ "type": "conversational_flow",
+ "description": "This agent supports running any search query",
+ "memory": {
+ "type": "conversation_index"
+ },
+ "app_type": "rag",
+ "tools": [
+ {
+ "type": "SearchIndexTool",
+ "parameters": {
+ "input": "{\"index\": \"${parameters.index}\", \"query\": ${parameters.query} }"
+ }
+ },
+ {
+ "type": "MLModelTool",
+ "description": "A general tool to answer any question",
+ "parameters": {
+ "model_id": "your_llm_model_id",
+ "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\n Context:\n${parameters.SearchIndexTool.output:-}\n\nHuman:${parameters.question}\n\nAssistant:"
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+### Run a BM25 query
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "what's the population increase of Seattle from 2021 to 2023?",
+ "index": "test_population_data",
+ "query": {
+ "query": {
+ "match": {
+ "population_description": "${parameters.question}"
+ }
+ },
+ "size": 2,
+ "_source": "population_description"
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+### Exposing only the `question` parameter
+
+To expose only the `question` parameter, define the agent as follows:
+
+```json
+POST /_plugins/_ml/agents/_register
+{
+ "name": "Demo agent",
+ "type": "conversational_flow",
+ "description": "This is a test agent support running any search query",
+ "memory": {
+ "type": "conversation_index"
+ },
+ "app_type": "rag",
+ "tools": [
+ {
+ "type": "SearchIndexTool",
+ "parameters": {
+ "input": "{\"index\": \"${parameters.index}\", \"query\": ${parameters.query} }",
+ "index": "test_population_data",
+ "query": {
+ "query": {
+ "match": {
+ "population_description": "${parameters.question}"
+ }
+ },
+ "size": 2,
+ "_source": "population_description"
+ }
+ }
+ },
+ {
+ "type": "MLModelTool",
+ "description": "A general tool to answer any question",
+ "parameters": {
+ "model_id": "your_llm_model_id",
+ "prompt": "\n\nHuman:You are a professional data analyst. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\n Context:\n${parameters.SearchIndexTool.output:-}\n\nHuman:${parameters.question}\n\nAssistant:"
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+Now you can run the agent specifying only the `question` parameter:
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "what's the population increase of Seattle from 2021 to 2023?"
+ }
+}
+```
+{% include copy-curl.html %}
+
+### Run a neural search query
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "what's the population increase of Seattle from 2021 to 2023??",
+ "index": "test_population_data",
+ "query": {
+ "query": {
+ "neural": {
+ "population_description_embedding": {
+ "query_text": "${parameters.question}",
+ "model_id": "your_embedding_model_id",
+ "k": 10
+ }
+ }
+ },
+ "size": 2,
+ "_source": ["population_description"]
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+To expose the `question` parameter, see [Exposing only the `question` parameter](#exposing-only-the-question-parameter).
+
+### Run a hybrid search query
+
+Hybrid search combines keyword and neural search to improve search relevance. For more information, see [Hybrid search]({{site.url}}{{site.baseurl}}/search-plugins/hybrid-search/).
+
+Configure a search pipeline:
+
+```json
+PUT /_search/pipeline/nlp-search-pipeline
+{
+ "description": "Post processor for hybrid search",
+ "phase_results_processors": [
+ {
+ "normalization-processor": {
+ "normalization": {
+ "technique": "min_max"
+ },
+ "combination": {
+ "technique": "arithmetic_mean",
+ "parameters": {
+ "weights": [
+ 0.3,
+ 0.7
+ ]
+ }
+ }
+ }
+ }
+ ]
+ }
+```
+{% include copy-curl.html %}
+
+Run an agent with a hybrid query:
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "what's the population increase of Seattle from 2021 to 2023??",
+ "index": "test_population_data",
+ "query": {
+ "_source": {
+ "exclude": [
+ "population_description_embedding"
+ ]
+ },
+ "size": 2,
+ "query": {
+ "hybrid": {
+ "queries": [
+ {
+ "match": {
+ "population_description": {
+ "query": "${parameters.question}"
+ }
+ }
+ },
+ {
+ "neural": {
+ "population_description_embedding": {
+ "query_text": "${parameters.question}",
+ "model_id": "your_embedding_model_id",
+ "k": 10
+ }
+ }
+ }
+ ]
+ }
+ }
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+To expose the `question` parameter, see [Exposing only the `question` parameter](#exposing-only-the-question-parameter).
+
+### Natural language query
+
+The `PPLTool` can translate a natural language query (NLQ) to [Piped Processing Language (PPL)]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) and execute the generated PPL query.
+
+#### Setup
+
+Before you start, go to the OpenSearch Dashboards home page, select `Add sample data`, and then add `Sample eCommerce orders`.
+
+
+#### Step 1: Register an agent with the PPLTool
+
+
+The `PPLTool` has the following parameters:
+
+- `model_type` (Enum): `CLAUDE`, `OPENAI`, or `FINETUNE`.
+- `execute` (Boolean): If `true`, executes the generated PPL query.
+- `input` (String): You must provide the `index` and `question` as inputs.
+
+For this tutorial, you'll use Bedrock Claude, so set the `model_type` to `CLAUDE`:
+
+```json
+POST /_plugins/_ml/agents/_register
+{
+ "name": "Demo agent for NLQ",
+ "type": "conversational_flow",
+ "description": "This is a test flow agent for NLQ",
+ "memory": {
+ "type": "conversation_index"
+ },
+ "app_type": "rag",
+ "tools": [
+ {
+ "type": "PPLTool",
+ "parameters": {
+ "model_id": "your_ppl_model_id",
+ "model_type": "CLAUDE",
+ "execute": true,
+ "input": "{\"index\": \"${parameters.index}\", \"question\": ${parameters.question} }"
+ }
+ },
+ {
+ "type": "MLModelTool",
+ "description": "A general tool to answer any question",
+ "parameters": {
+ "model_id": "your_llm_model_id",
+ "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\n Context:\n${parameters.PPLTool.output:-}\n\nHuman:${parameters.question}\n\nAssistant:"
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+### Step 2: Run the agent with an NLQ
+
+Run the agent:
+
+```json
+POST /_plugins/_ml/agents/your_agent_id/_execute
+{
+ "parameters": {
+ "question": "How many orders do I have in last week",
+ "index": "opensearch_dashboards_sample_data_ecommerce"
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the answer generated by the LLM:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "memory_id",
+ "result": "sqIioI0BJhBwrVXYeYOM"
+ },
+ {
+ "name": "parent_message_id",
+ "result": "s6IioI0BJhBwrVXYeYOW"
+ },
+ {
+ "name": "MLModelTool",
+ "result": " Based on the given context, the number of orders in the last week is 3992. The data shows a query that counts the number of orders where the order date is greater than 1 week ago. The query result shows the count as 3992."
+ }
+ ]
+ }
+ ]
+}
+```
+
+For more information, obtain trace data by calling the [Get Message Traces API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/memory-apis/get-message-traces/):
+
+```json
+GET _plugins/_ml/memory/message/s6IioI0BJhBwrVXYeYOW/traces
+```
+{% include copy-curl.html %}
\ No newline at end of file
diff --git a/_ml-commons-plugin/tutorials/reranking-cohere.md b/_ml-commons-plugin/tutorials/reranking-cohere.md
new file mode 100644
index 0000000000..412180066f
--- /dev/null
+++ b/_ml-commons-plugin/tutorials/reranking-cohere.md
@@ -0,0 +1,344 @@
+---
+layout: default
+title: Reranking with Cohere Rerank
+parent: Tutorials
+nav_order: 30
+---
+
+# Reranking search results using the Cohere Rerank model
+
+A [reranking pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/reranking-search-results/) can rerank search results, providing a relevance score for each document in the search results with respect to the search query. The relevance score is calculated by a cross-encoder model.
+
+This tutorial illustrates how to use the [Cohere Rerank](https://docs.cohere.com/reference/rerank-1) model in a reranking pipeline.
+
+Replace the placeholders beginning with the prefix `your_` with your own values.
+{: .note}
+
+## Step 1: Register a Cohere Rerank model
+
+Create a connector for the Cohere Rerank model:
+
+```json
+POST /_plugins/_ml/connectors/_create
+{
+ "name": "cohere-rerank",
+ "description": "The connector to Cohere reanker model",
+ "version": "1",
+ "protocol": "http",
+ "credential": {
+ "cohere_key": "your_cohere_api_key"
+ },
+ "parameters": {
+ "model": "rerank-english-v2.0"
+ },
+ "actions": [
+ {
+ "action_type": "predict",
+ "method": "POST",
+ "url": "https://api.cohere.ai/v1/rerank",
+ "headers": {
+ "Authorization": "Bearer ${credential.cohere_key}"
+ },
+ "request_body": "{ \"documents\": ${parameters.documents}, \"query\": \"${parameters.query}\", \"model\": \"${parameters.model}\", \"top_n\": ${parameters.top_n} }",
+ "pre_process_function": "connector.pre_process.cohere.rerank",
+ "post_process_function": "connector.post_process.cohere.rerank"
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+Use the connector ID from the response to register a Cohere Rerank model:
+
+```json
+POST /_plugins/_ml/models/_register?deploy=true
+{
+ "name": "cohere rerank model",
+ "function_name": "remote",
+ "description": "test rerank model",
+ "connector_id": "your_connector_id"
+}
+```
+{% include copy-curl.html %}
+
+Note the model ID in the response; you'll use it in the following steps.
+
+Test the model by calling the Predict API:
+
+```json
+POST _plugins/_ml/models/your_model_id/_predict
+{
+ "parameters": {
+ "query": "What is the capital of the United States?",
+ "documents": [
+ "Carson City is the capital city of the American state of Nevada.",
+ "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
+ "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.",
+ "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."
+ ],
+ "top_n": 4
+ }
+}
+```
+
+To ensure compatibility with the rerank pipeline, the `top_n` value must be the same as the length of the `documents` list.
+{: .important}
+
+You can customize the number of top documents returned in the response by providing the `size` parameter. For more information, see [Step 2.3](#step-23-test-the-reranking).
+
+OpenSearch responds with the inference results:
+
+```json
+{
+ "inference_results": [
+ {
+ "output": [
+ {
+ "name": "similarity",
+ "data_type": "FLOAT32",
+ "shape": [
+ 1
+ ],
+ "data": [
+ 0.10194652
+ ]
+ },
+ {
+ "name": "similarity",
+ "data_type": "FLOAT32",
+ "shape": [
+ 1
+ ],
+ "data": [
+ 0.0721122
+ ]
+ },
+ {
+ "name": "similarity",
+ "data_type": "FLOAT32",
+ "shape": [
+ 1
+ ],
+ "data": [
+ 0.98005307
+ ]
+ },
+ {
+ "name": "similarity",
+ "data_type": "FLOAT32",
+ "shape": [
+ 1
+ ],
+ "data": [
+ 0.27904198
+ ]
+ }
+ ],
+ "status_code": 200
+ }
+ ]
+}
+```
+
+The response contains four `similarity` objects. For each `similarity` object, the `data` array contains a relevance score for each document with respect to the query. The `similarity` objects are provided in the order of the input documents; the first object pertains to the first document. This differs from the default output of the Cohere Rerank model, which orders documents by relevance score. The document order is changed in the `connector.post_process.cohere.rerank` post-processing function in order to make the output compatible with a reranking pipeline.
+
+## Step 2: Configure a reranking pipeline
+
+Follow these steps to configure a reranking pipeline.
+
+### Step 2.1: Ingest test data
+
+Send a bulk request to ingest test data:
+
+```json
+POST _bulk
+{ "index": { "_index": "my-test-data" } }
+{ "passage_text" : "Carson City is the capital city of the American state of Nevada." }
+{ "index": { "_index": "my-test-data" } }
+{ "passage_text" : "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan." }
+{ "index": { "_index": "my-test-data" } }
+{ "passage_text" : "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district." }
+{ "index": { "_index": "my-test-data" } }
+{ "passage_text" : "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states." }
+```
+{% include copy-curl.html %}
+
+### Step 2.2: Create a reranking pipeline
+
+Create a reranking pipeline with the Cohere Rerank model:
+
+```json
+PUT /_search/pipeline/rerank_pipeline_cohere
+{
+ "description": "Pipeline for reranking with Cohere Rerank model",
+ "response_processors": [
+ {
+ "rerank": {
+ "ml_opensearch": {
+ "model_id": "your_model_id_created_in_step1"
+ },
+ "context": {
+ "document_fields": ["passage_text"]
+ }
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+### Step 2.3: Test the reranking
+
+To limit the number of returned results, you can specify the `size` parameter. For example, set `"size": 2` to return the top two documents:
+
+```json
+GET my-test-data/_search?search_pipeline=rerank_pipeline_cohere
+{
+ "query": {
+ "match_all": {}
+ },
+ "size": 4,
+ "ext": {
+ "rerank": {
+ "query_context": {
+ "query_text": "What is the capital of the United States?"
+ }
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the two most relevant documents:
+
+```json
+{
+ "took": 0,
+ "timed_out": false,
+ "_shards": {
+ "total": 1,
+ "successful": 1,
+ "skipped": 0,
+ "failed": 0
+ },
+ "hits": {
+ "total": {
+ "value": 4,
+ "relation": "eq"
+ },
+ "max_score": 0.98005307,
+ "hits": [
+ {
+ "_index": "my-test-data",
+ "_id": "zbUOw40B8vrNLhb9vBif",
+ "_score": 0.98005307,
+ "_source": {
+ "passage_text": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district."
+ }
+ },
+ {
+ "_index": "my-test-data",
+ "_id": "zrUOw40B8vrNLhb9vBif",
+ "_score": 0.27904198,
+ "_source": {
+ "passage_text": "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."
+ }
+ },
+ {
+ "_index": "my-test-data",
+ "_id": "y7UOw40B8vrNLhb9vBif",
+ "_score": 0.10194652,
+ "_source": {
+ "passage_text": "Carson City is the capital city of the American state of Nevada."
+ }
+ },
+ {
+ "_index": "my-test-data",
+ "_id": "zLUOw40B8vrNLhb9vBif",
+ "_score": 0.0721122,
+ "_source": {
+ "passage_text": "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan."
+ }
+ }
+ ]
+ },
+ "profile": {
+ "shards": []
+ }
+}
+```
+
+To compare these results to results without reranking, run the search without a reranking pipeline:
+
+```json
+GET my-test-data/_search
+{
+ "query": {
+ "match_all": {}
+ },
+ "ext": {
+ "rerank": {
+ "query_context": {
+ "query_text": "What is the capital of the United States?"
+ }
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+The first document in the response pertains to Carson City, which is not the capital of the United States:
+
+```json
+{
+ "took": 0,
+ "timed_out": false,
+ "_shards": {
+ "total": 1,
+ "successful": 1,
+ "skipped": 0,
+ "failed": 0
+ },
+ "hits": {
+ "total": {
+ "value": 4,
+ "relation": "eq"
+ },
+ "max_score": 1,
+ "hits": [
+ {
+ "_index": "my-test-data",
+ "_id": "y7UOw40B8vrNLhb9vBif",
+ "_score": 1,
+ "_source": {
+ "passage_text": "Carson City is the capital city of the American state of Nevada."
+ }
+ },
+ {
+ "_index": "my-test-data",
+ "_id": "zLUOw40B8vrNLhb9vBif",
+ "_score": 1,
+ "_source": {
+ "passage_text": "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan."
+ }
+ },
+ {
+ "_index": "my-test-data",
+ "_id": "zbUOw40B8vrNLhb9vBif",
+ "_score": 1,
+ "_source": {
+ "passage_text": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district."
+ }
+ },
+ {
+ "_index": "my-test-data",
+ "_id": "zrUOw40B8vrNLhb9vBif",
+ "_score": 1,
+ "_source": {
+ "passage_text": "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."
+ }
+ }
+ ]
+ }
+}
+```
\ No newline at end of file
diff --git a/_ml-commons-plugin/tutorials/semantic-search-byte-vectors.md b/_ml-commons-plugin/tutorials/semantic-search-byte-vectors.md
new file mode 100644
index 0000000000..7061d3cb5a
--- /dev/null
+++ b/_ml-commons-plugin/tutorials/semantic-search-byte-vectors.md
@@ -0,0 +1,314 @@
+---
+layout: default
+title: Semantic search using byte vectors
+parent: Tutorials
+nav_order: 10
+---
+
+# Semantic search using byte-quantized vectors
+
+This tutorial illustrates how to build a semantic search using the [Cohere Embed model](https://docs.cohere.com/reference/embed) and byte-quantized vectors. For more information about using byte-quantized vectors, see [Lucene byte vector]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-vector/#lucene-byte-vector).
+
+The Cohere Embed v3 model supports several `embedding_types`. For this tutorial, you'll use the `INT8` type to encode byte-quantized vectors.
+
+The Cohere Embed v3 model supports several input types. This tutorial uses the following input types:
+
+- `search_document`: Use this input type when you have text (in the form of documents) that you want to store in a vector database.
+- `search_query`: Use this input type when structuring search queries to find the most relevant documents in your vector database.
+
+For more information about input types, see the [Cohere documentation](https://docs.cohere.com/docs/embed-api#the-input_type-parameter).
+
+In this tutorial, you will create two models:
+
+- A model used for ingestion, whose `input_type` is `search_document`
+- A model used for search, whose `input_type` is `search_query`
+
+Replace the placeholders beginning with the prefix `your_` with your own values.
+{: .note}
+
+## Step 1: Create an embedding model for ingestion
+
+Create a connector for the Cohere model, specifying the `search_document` input type:
+
+```json
+POST /_plugins/_ml/connectors/_create
+{
+ "name": "Cohere embedding connector with int8 embedding type for ingestion",
+ "description": "Test connector for Cohere embedding model",
+ "version": 1,
+ "protocol": "http",
+ "credential": {
+ "cohere_key": "your_cohere_api_key"
+ },
+ "parameters": {
+ "model": "embed-english-v3.0",
+ "embedding_types": ["int8"],
+ "input_type": "search_document"
+ },
+ "actions": [
+ {
+ "action_type": "predict",
+ "method": "POST",
+ "headers": {
+ "Authorization": "Bearer ${credential.cohere_key}",
+ "Request-Source": "unspecified:opensearch"
+ },
+ "url": "https://api.cohere.ai/v1/embed",
+ "request_body": "{ \"model\": \"${parameters.model}\", \"texts\": ${parameters.texts}, \"input_type\":\"${parameters.input_type}\", \"embedding_types\": ${parameters.embedding_types} }",
+ "pre_process_function": "connector.pre_process.cohere.embedding",
+ "post_process_function": "\n def name = \"sentence_embedding\";\n def data_type = \"FLOAT32\";\n def result;\n if (params.embeddings.int8 != null) {\n data_type = \"INT8\";\n result = params.embeddings.int8;\n } else if (params.embeddings.uint8 != null) {\n data_type = \"UINT8\";\n result = params.embeddings.uint8;\n } else if (params.embeddings.float != null) {\n data_type = \"FLOAT32\";\n result = params.embeddings.float;\n }\n \n if (result == null) {\n return \"Invalid embedding result\";\n }\n \n def embedding_list = new StringBuilder(\"[\");\n \n for (int m=0; m
+
-The following example creates a monitor that runs at 12:10 PM Pacific Time on the 1st day of every month.
-#### Example request
+To specify a time zone, you can do so by including a [cron expression]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/) with a time zone name in the `schedule` section of your request. The following example creates a monitor that runs at 12:10 PM Pacific Time on the first day of each month.
+#### Example request
```json
{
"type": "monitor",
@@ -321,15 +316,18 @@ The following example creates a monitor that runs at 12:10 PM Pacific Time on th
}]
}
```
+{% include copy-curl.html %}
+
-For a full list of time zone names, refer to [Wikipedia](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). The Alerting plugin uses the Java [TimeZone](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/TimeZone.html) class to convert a [`ZoneId`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/ZoneId.html) to a valid time zone.
+For a full list of time zone names, see [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). The Alerting plugin uses the Java [TimeZone](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/TimeZone.html) class to convert a [`ZoneId`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/ZoneId.html) to a valid time zone.
---
## Bucket-level monitors
-Bucket-level monitors categorize results into buckets separated by fields. The monitor then runs your script with each bucket's results and evaluates whether to trigger an alert. For more information about bucket-level and query-level monitors, see [Creating monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/).
+Bucket-level monitors categorize results into buckets separated by fields. The monitor then runs the script with each bucket's results and evaluates whether to trigger an alert. For more information about bucket-level and query-level monitors, see [Creating monitors]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/monitors/).
+#### Example request
```json
POST _plugins/_alerting/monitors
{
@@ -444,7 +442,15 @@ POST _plugins/_alerting/monitors
]
}
```
+{% include copy-curl.html %}
+
++ Select to expand example response +
+ {: .text-delta} + #### Example response - ```json { "_id": "vd5k2GsBlQ5JUWWFxhsP", @@ -253,13 +247,14 @@ To learn more about using backend roles to limit access, see [(Advanced) Limit a } } ``` +{% include copy-curl.html %} -If you want to specify a time zone, you can do so by including a [cron expression]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/) with a time zone name in the `schedule` section of your request. +
+
+
+---
+
## Document-level monitors
Introduced 2.0
{: .label .label-purple }
@@ -601,17 +613,18 @@ To retrieve any available findings, send a GET request without any path paramete
```json
GET /_plugins/_alerting/findings/_search?
```
-
+{% include copy-curl.html %}
To retrieve metadata for an individual document finding entry, you can search for the finding by its `findingId` as follows:
```json
GET /_plugins/_alerting/findings/_search?findingId=gKQhj8WJit3BxjGfiOXC
```
+{% include copy-curl.html %}
The response returns the number of individual finding entries in the `total_findings` field.
-To get more specific results in a findings search, you can use any of the optional path parameters that are defined in the following table:
+To get more specific results in a findings search, you can use any of the optional path parameters defined in the following table.
Path parameter | Description | Usage
:--- | :--- : :---
@@ -624,10 +637,9 @@ Path parameter | Description | Usage
### Create a document-level monitor
-You can create a document-level monitor with a POST request that provides the monitor details in the request body.
-At a minimum, you need to provide the following details: specify the queries or combinations by tag with the `inputs` field, a valid trigger condition, and provide the notification message in the `action` field.
+You can create a document-level monitor with a POST request that provides the monitor details in the request body. At a minimum, you need to provide the following details: specify the queries or combinations by tag with the `inputs` field, a valid trigger condition, and provide the notification message in the `action` field.
-The following table shows the syntax to use for each trigger option:
+The following table provides the syntax to use for each trigger option.
Trigger options | Definition | Syntax
:--- | :--- : :---
@@ -635,10 +647,8 @@ Tag | Creates alerts for documents that match a multiple query with this tag app
Query by name | Creates alerts for documents matched or returned by the named query. | `query[name=+ Select to expand example response +
+ {: .text-delta} + #### Example response ```json { @@ -584,6 +590,12 @@ POST _plugins/_alerting/monitors } } ``` +{% include copy-curl.html %} + +
+ ?if_seq_no=3&if_primary_term=1
}
}
```
+{% include copy-curl.html %}
+
---
## Get monitor
-#### Request
+Retrieve the details of a specific monitor using the following request.
+#### Example request
```
GET _plugins/_alerting/monitors/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_id": "Q9aXOmkBC25HCRGmzfw-", @@ -900,20 +919,30 @@ PUT _plugins/_alerting/monitors/
+
}
}
```
+{% include copy-curl.html %}
+
---
@@ -980,17 +1011,23 @@ GET _plugins/_alerting/monitors/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_id": "Q9aXOmkBC25HCRGmzfw-", @@ -972,7 +1001,9 @@ GET _plugins/_alerting/monitors/
+ /stats/
}
}
```
+{% include copy-curl.html %}
+
---
## Delete monitor
-#### Request
+Delete a monitor using the following request.
+#### Example request
```
DELETE _plugins/_alerting/monitors/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_nodes": { @@ -1161,20 +1198,30 @@ GET _plugins/_alerting/
+
"_primary_term": 1
}
```
+{% include copy-curl.html %}
+
---
## Search monitors
-#### Request
+Query and retrieve information about existing monitors based on specific criteria, such as the monitor name, using the following request.
+#### Example request
```json
GET _plugins/_alerting/monitors/_search
{
@@ -1209,9 +1259,16 @@ GET _plugins/_alerting/monitors/_search
}
}
```
+{% include copy-curl.html %}
-#### Example response
++ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_index": ".opensearch-scheduled-jobs", @@ -1191,14 +1238,17 @@ DELETE _plugins/_alerting/monitors/
+
---
@@ -1296,15 +1355,20 @@ GET _plugins/_alerting/monitors/_search
You can add the optional `?dryrun=true` parameter to the URL to show the results of a run without actions sending any message.
-
-#### Request
-
+#### Example request
```json
POST _plugins/_alerting/monitors/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "took": 17, @@ -1288,7 +1345,9 @@ GET _plugins/_alerting/monitors/_search } } ``` +{% include copy-curl.html %} +
+ /_execute
}
}
```
+{% include copy-curl.html %}
+
+
---
## Get alerts
-Returns an array of all alerts.
+Return an array of all alerts.
#### Path parameters
@@ -1345,14 +1412,20 @@ The following table lists the available path parameters. All path parameters are
| `monitorId` | String | Filters by monitor ID.
| `workflowIds` | String | Allows for monitoring the status of chained alerts from multiple workflows within a single dashboard. Available in OpenSearch 2.9 or later.
-#### Request
-
+#### Example request
```json
GET _plugins/_alerting/monitors/alerts
```
+{% include copy-curl.html %}
+
-#### Response
++ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "monitor_name": "logs", @@ -1321,12 +1385,15 @@ POST _plugins/_alerting/monitors/
+
---
## Acknowledge alert
-[After getting your alerts](#get-alerts), you can acknowledge any number of active alerts in one call. If the alert is already in an ERROR, COMPLETED, or ACKNOWLEDGED state, it appears in the `failed` array.
-
-
-#### Request
+[After getting your alerts](#get-alerts), you can acknowledge any number of active alerts in one call. If the alert is already in an `ERROR`, `COMPLETED`, or `ACKNOWLEDGED` state, it appears in the `failed` array.
+#### Example request
```json
POST _plugins/_alerting/monitors/+ Select to expand example response +
+ {: .text-delta} +#### Example response ```json { "alerts": [ @@ -1406,25 +1479,34 @@ GET _plugins/_alerting/monitors/alerts "totalAlerts": 1 } ``` +{% include copy-curl.html %} + +
+ /_acknowledge/alerts
"failed": []
}
```
+{% include copy-curl.html %}
+
+
---
## Create destination
-#### Requests
+Define and configure various destinations for receiving alert notifications using the following request. These destinations can be of different types, such as Slack, custom webhooks, or email, and are used to specify where and how alerts should be delivered.
+#### Example request
```json
POST _plugins/_alerting/destinations
{
@@ -1489,9 +1575,16 @@ POST _plugins/_alerting/destinations
// The email_account_id and email_group_id will be the document IDs of the email_account and email_group you have created.
```
+{% include copy-curl.html %}
-#### Example response
++ Select to expand example response +
+ {: .text-delta} + + +#### Example response ```json { "success": [ @@ -1433,13 +1515,17 @@ POST _plugins/_alerting/monitors/
+
---
## Update destination
-When updating a destination, you can optionally include `seq_no` and `primary_term` as URL parameters. If these numbers don't match the existing destination or the destination doesn't exist, the Alerting plugin throws an error. OpenSearch increments the version number and the sequence number automatically (see the example response).
-
-#### Request
+When updating a destination, you can optionally include `seq_no` and `primary_term` as URL parameters. If these numbers do not match the existing destination or the destination doesn't exist, the Alerting plugin throws an error. OpenSearch increments the version number and the sequence number automatically (see the example response).
+#### Example request
```json
PUT _plugins/_alerting/destinations/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_id": "nO-yFmkB8NzS6aXjJdiI", @@ -1508,16 +1601,17 @@ POST _plugins/_alerting/destinations } } ``` +{% include copy-curl.html %} +
+ ?if_seq_no=3&if_primary_term
}
}
```
+{% include copy-curl.html %}
+
---
## Get destination
-Retrieve one destination.
-
-#### Requests
+Retrieve one destination using the following request.
+#### Example request
```json
GET _plugins/_alerting/destinations/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_id": "pe-1FmkB8NzS6aXjqvVY", @@ -1556,22 +1657,30 @@ PUT _plugins/_alerting/destinations/
+
]
}
```
+{% include copy-curl.html %}
+
---
## Get destinations
-Retrieve all destinations.
-
-#### Requests
+Retrieve all destinations using the following request.
+#### Example request
```json
GET _plugins/_alerting/destinations
```
+{% include copy-curl.html %}
-#### Example response
++ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "totalDestinations": 1, @@ -1601,22 +1710,30 @@ GET _plugins/_alerting/destinations/
+
---
## Delete destination
-#### Request
+Remove a specific destination from the alerting system using the following request.
+#### Example request
```
DELETE _plugins/_alerting/destinations/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "totalDestinations": 1, @@ -1646,20 +1763,30 @@ GET _plugins/_alerting/destinations ] } ``` +{% include copy-curl.html %} +
+
"_primary_term": 1
}
```
+{% include copy-curl.html %}
+
+
+
---
## Create email account
-#### Request
+Set up a new email account for sending alert notifications using the following request.
+
+#### Example request
```json
POST _plugins/_alerting/destinations/email_accounts
{
@@ -1692,6 +1825,14 @@ POST _plugins/_alerting/destinations/email_accounts
"method": "ssl"
}
```
+{% include copy-curl.html %}
+
+
++ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_index": ".opendistro-alerting-config", @@ -1677,11 +1804,17 @@ DELETE _plugins/_alerting/destinations/
+
## Update email account
When updating an email account, you can optionally include `seq_no` and `primary_term` as URL parameters. If these numbers don't match the existing email account or the email account doesn't exist, the Alerting plugin throws an error. OpenSearch increments the version number and the sequence number automatically (see the example response).
-#### Request
+#### Example request
```json
PUT _plugins/_alerting/destinations/email_accounts/+ Select to expand example response +
+ {: .text-delta} #### Example response ```json @@ -1710,12 +1851,15 @@ POST _plugins/_alerting/destinations/email_accounts } } ``` +{% include copy-curl.html %} + +
+ ?if_seq_no=
}
}
```
+{% include copy-curl.html %}
+
+
## Get email account
-#### Request
+Retrieve the details of a specific email account configured for alerting purposes using the following request.
+
+#### Example request
```json
GET _plugins/_alerting/destinations/email_accounts/+ Select to expand example response +
+ {: .text-delta} + #### Example response ```json { @@ -1752,10 +1905,15 @@ PUT _plugins/_alerting/destinations/email_accounts/
+
}
}
```
+{% include copy-curl.html %}
+
+
## Delete email account
-#### Request
+Remove an existing email account configuration from the alerting system using the following request.
+
+#### Example request
```
DELETE _plugins/_alerting/destinations/email_accounts/+ Select to expand example response +
+ {: .text-delta} + #### Example response ```json { @@ -1783,15 +1950,28 @@ GET _plugins/_alerting/destinations/email_accounts/
+
"_primary_term" : 2
}
```
+{% include copy-curl.html %}
+
+
## Search email account
-#### Request
+Retrieve information about the configured email accounts used for email-based alerting using the following request.
+#### Example request
```json
POST _plugins/_alerting/destinations/email_accounts/_search
{
@@ -1829,9 +2013,16 @@ POST _plugins/_alerting/destinations/email_accounts/_search
}
}
```
+{% include copy-curl.html %}
-#### Example response
++ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_index" : ".opendistro-alerting-config", @@ -1809,11 +1989,15 @@ DELETE _plugins/_alerting/destinations/email_accounts/
+
---
## Create email group
-#### Request
+Define a new group of email recipients for alerts using the following request.
+#### Example request
```json
POST _plugins/_alerting/destinations/email_groups
{
@@ -1889,9 +2084,16 @@ POST _plugins/_alerting/destinations/email_groups
}]
}
```
+{% include copy-curl.html %}
-#### Example response
++ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "took" : 8, @@ -1873,13 +2064,17 @@ POST _plugins/_alerting/destinations/email_accounts/_search } } ``` +{% include copy-curl.html %} + +
+
## Update email group
When updating an email group, you can optionally include `seq_no` and `primary_term` as URL parameters. If these numbers don't match the existing email group or the email group doesn't exist, the Alerting plugin throws an error. OpenSearch increments the version number and the sequence number automatically (see the example response).
-#### Request
-
+#### Example request
```json
PUT _plugins/_alerting/destinations/email_groups/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_id" : "email_group_id", @@ -1909,13 +2111,15 @@ POST _plugins/_alerting/destinations/email_groups } } ``` +{% include copy-curl.html %} + +
+ ?if_seq_no=16&i
}
}
```
+{% include copy-curl.html %}
+
+
## Get email group
-#### Request
+Retrieve the details of a specific email group destination using the following request, passing the ID of the email group you want to fetch.
+
+#### Example request
```json
GET _plugins/_alerting/destinations/email_groups/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_id" : "email_group_id", @@ -1952,10 +2164,15 @@ PUT _plugins/_alerting/destinations/email_groups/
+
}
}
```
+{% include copy-curl.html %}
+
+
## Delete email group
-#### Request
+Remove an existing email group from the list of destinations for alerts using the following request.
+
+#### Example request
```
DELETE _plugins/_alerting/destinations/email_groups/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_id" : "email_group_id", @@ -1984,15 +2209,28 @@ GET _plugins/_alerting/destinations/email_groups/
+
"_primary_term" : 2
}
```
+{% include copy-curl.html %}
+
+
## Search email group
-#### Request
+Query and retrieve information about existing email groups used for alerting purposes, enabling you to filter and sort the results based on various criteria. An example is shown in the following request.
+#### Example request
```json
POST _plugins/_alerting/destinations/email_groups/_search
{
@@ -2030,9 +2272,16 @@ POST _plugins/_alerting/destinations/email_groups/_search
}
}
```
+{% include copy-curl.html %}
-#### Example response
++ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "_index" : ".opendistro-alerting-config", @@ -2010,11 +2248,15 @@ DELETE _plugins/_alerting/destinations/email_groups/
+
+
+## Create comment
+This is an experimental feature and is not recommended for use in a production environment.
+{: .warning}
+
+Add comments to a specific alert, providing additional context or notes related to that alert, using the following request.
+
+#### Example request
+```json
+POST _plugins/_alerting/comments/+ Select to expand example response +
+ {: .text-delta} + +#### Example response ```json { "took" : 7, @@ -2075,4 +2324,203 @@ POST _plugins/_alerting/destinations/email_groups/_search } } ``` ---- +{% include copy-curl.html %} + +
+
+
+## Update comment
+This is an experimental feature and is not recommended for use in a production environment.
+{: .warning}
+
+Modify the content of a previously added comment associated with an alert using the following request.
+
+#### Example request
+
+```json
+PUT _plugins/_alerting/comments/+ Select to expand example response +
+ {: .text-delta} + +#### Example response +```json +{ + "_id": "0U6aBJABVWc3FrmWer9s", + "_seq_no": 7, + "_primary_term": 2, + "comment": { + "entity_id": "vCZkA5ABWTh3kzuBEL_9", + "entity_type": "alert", + "content": "sample comment", + "created_time": 1718064151148, + "last_updated_time": null, + "user": "admin" + } +} +``` +{% include copy-curl.html %} + +
+
+
+## Search comment
+This is an experimental feature and is not recommended for use in a production environment.
+{: .warning}
+
+Query and retrieve existing comments associated with alerts using the following request.
+
+#### Example request
+```json
+GET _plugins/_alerting/comments/_search
+{
+ "query": {
+ "match_all" : {}
+ }
+}
+```
+{% include copy-curl.html %}
+
+
++ Select to expand example response +
+ {: .text-delta} + +#### Example response +```json +{ + "_id": "0U6aBJABVWc3FrmWer9s", + "_seq_no": 8, + "_primary_term": 3, + "comment": { + "entity_id": "vCZkA5ABWTh3kzuBEL_9", + "entity_type": "alert", + "content": "sample updated comment", + "created_time": 1718064151148, + "last_updated_time": 1718064745485, + "user": "admin" + } +} +``` +{% include copy-curl.html %} + +
+
+
+## Delete comment
+This is an experimental feature and is not recommended for use in a production environment.
+{: .warning}
+
+Remove a specific comment associated with an alert using the following request.
+
+#### Example request
+```json
+DELETE _plugins/_alerting/comments/+ Select to expand example response +
+ {: .text-delta} + +#### Example response +```json +{ + "took": 14, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": ".opensearch-alerting-comments-history-2024.06.10-1", + "_id": "xE5tBJABVWc3FrmWRL5i", + "_version": 1, + "_seq_no": 3, + "_primary_term": 2, + "_score": 1, + "_source": { + "entity_id": "vCZkA5ABWTh3kzuBEL_9", + "entity_type": "alert", + "content": "a different sample comment", + "created_time": 1718061188191, + "last_updated_time": null, + "user": "admin" + } + }, + { + "_index": ".opensearch-alerting-comments-history-2024.06.10-1", + "_id": "0U6aBJABVWc3FrmWer9s", + "_version": 3, + "_seq_no": 9, + "_primary_term": 3, + "_score": 1, + "_source": { + "entity_id": "vCZkA5ABWTh3kzuBEL_9", + "entity_type": "alert", + "content": "sample updated comment", + "created_time": 1718064151148, + "last_updated_time": 1718064745485, + "user": "admin" + } + } + ] + } +} +``` +{% include copy-curl.html %} + +
+
+
diff --git a/_observing-your-data/alerting/comments.md b/_observing-your-data/alerting/comments.md
new file mode 100644
index 0000000000..fb0630685c
--- /dev/null
+++ b/_observing-your-data/alerting/comments.md
@@ -0,0 +1,26 @@
+---
+layout: default
+title: Adding comments
+nav_order: 35
+parent: Alerting
+has_children: false
+redirect_from:
+ - /monitoring-plugins/alerting/comments/
+---
+
+# Adding comments
+
+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/6999).
+{: .warning}
+
+When an alert is generated, add comments to share information about its root cause and facilitate resolution. Comments are enabled by setting `plugins.alerting.comments_enabled` to `true` using the [`cluster/settings` API]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/settings/).
+
+Comments can be accessed through the alerts table view by selecting the comment icon within an alert's row. From there, comments can be added, edited, or deleted. An Alerting Comments API is also available for programmatic comment management. For more information, see [Alerting API]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/).
+
+## Viewing comment authors
+
+If the Security plugin is installed, then the comment's author is displayed. Otherwise, `Unknown` is displayed.
+
+## Assigning permissions
+
+Comment permissions are determined by the backend roles associated with the alert. These backend roles are inherited from the monitor that generated the alert. For more information about how to limit access based on backend roles, see [Limit access by backend role]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/security/#advanced-limit-access-by-backend-role).
diff --git a/_observing-your-data/alerting/cron.md b/_observing-your-data/alerting/cron.md
index b37d13e576..11a91609dd 100644
--- a/_observing-your-data/alerting/cron.md
+++ b/_observing-your-data/alerting/cron.md
@@ -63,4 +63,4 @@ Every three hours on the first day of every other month:
## API
-For an example of how to use a custom cron expression in an API call, see the [create monitor API operation]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/api#request-1).
+For an example of how to use a custom cron expression in an API call, see the [create monitor API operation]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/api/#example-request-2).
diff --git a/_observing-your-data/alerting/index.md b/_observing-your-data/alerting/index.md
index a78c4cb93f..2b5ced8d1c 100644
--- a/_observing-your-data/alerting/index.md
+++ b/_observing-your-data/alerting/index.md
@@ -45,7 +45,7 @@ Deleted | The monitor or trigger associated with this alert was deleted while th
You can follow these basic steps to create an alert monitor:
1. In the **OpenSearch Plugins** main menu, choose **Alerting**.
-1. Choose **Create monitor**. See [Monitors]({{site.url}}{{site.baseurl}}/observing-your-data/notifications/index/) for more information about the monitor types.
+1. Choose **Create monitor**. See [Monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/monitors/) for more information about the monitor types.
1. Enter the **Monitor details**, including monitor type, method, and schedule.
1. Select a data source from the dropdown list.
1. Define the metrics in the Query section.
@@ -53,4 +53,4 @@ You can follow these basic steps to create an alert monitor:
1. Add an action. See [Actions]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/actions/) for more information about actions.
1. Select **Create**.
-Learn more about creating specific monitor types in their respective documentation.
\ No newline at end of file
+Learn more about creating specific monitor types in their respective documentation.
diff --git a/_observing-your-data/alerting/per-cluster-metrics-monitors.md b/_observing-your-data/alerting/per-cluster-metrics-monitors.md
index 99984a16a5..baea9c626b 100644
--- a/_observing-your-data/alerting/per-cluster-metrics-monitors.md
+++ b/_observing-your-data/alerting/per-cluster-metrics-monitors.md
@@ -115,6 +115,9 @@ The `script` parameter points the `source` to the Painless script `for (cluster
```
The dashboards interface supports the selection of clusters to be monitored and the desired API. A view of the interface is shown in the following image.
+The following [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/) are required in order to create a cross-cluster monitor through the dashboards UI: `cluster:admin/opensearch/alerting/remote/indexes/get`, `indices:admin/resolve/index`, `cluster:monitor/health`, and `indices:admin/mappings/get`.
+{: .note}
+
+ Select to expand example response +
+ {: .text-delta} + +#### Example response +```json +{ + "_id": "0U6aBJABVWc3FrmWer9s" +} +``` +{% include copy-curl.html %} + +
### Limitations
diff --git a/_observing-your-data/alerting/per-query-bucket-monitors.md b/_observing-your-data/alerting/per-query-bucket-monitors.md
index d4fe0ff9d7..cb08c49478 100644
--- a/_observing-your-data/alerting/per-query-bucket-monitors.md
+++ b/_observing-your-data/alerting/per-query-bucket-monitors.md
@@ -15,6 +15,9 @@ Per bucket monitors are a type of alert monitor that can be used to identify and
Both monitor types support querying remote indexes using the same `cluster-name:index-name` pattern used by [cross-cluster search](https://opensearch.org/docs/latest/security/access-control/cross-cluster-search/) or by using OpenSearch Dashboards 2.12 or later.
+The following [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/) are required in order to create a cross-cluster monitor through the dashboards UI: `cluster:admin/opensearch/alerting/remote/indexes/get`, `indices:admin/resolve/index`, `cluster:monitor/health`, and `indices:admin/mappings/get`.
+{: .note}
+
## Creating a per query or per bucket monitor
diff --git a/_observing-your-data/alerting/settings.md b/_observing-your-data/alerting/settings.md
index c50636b3ae..c7eefe528b 100644
--- a/_observing-your-data/alerting/settings.md
+++ b/_observing-your-data/alerting/settings.md
@@ -51,12 +51,20 @@ Setting | Default | Description
`plugins.alerting.alert_history_max_age` | 30d | The oldest document to store in the `.opendistro-alert-history-
-The following example request configures a Prometheus data source with no authentication:
+The following image displays an observability dashboard that visualizes metric data from the OpenSearch index using OTel queries.
+
+
+
+---
+
+## Configuring Prometheus to send metric data to OpenSearch
+
+You must first create a connection from [Prometheus](https://prometheus.io/) to OpenSearch using the [SQL plugin](https://github.com/opensearch-project/sql). You can then configure a connection to Prometheus by using the `_datasources` API endpoint.
+
+The following example shows a request that configures a Prometheus data source without any authentication:
```json
POST _plugins/_query/_datasources
@@ -34,8 +43,9 @@ POST _plugins/_query/_datasources
}
}
```
+{% include copy-curl.html %}
-The following example request configures a Prometheus data source with AWS SigV4 authentication:
+The following example shows how to configure a Prometheus data source using AWS Signature Version 4 authentication:
```json
POST _plugins/_query/_datasources
@@ -51,48 +61,143 @@ POST _plugins/_query/_datasources
}
}
```
+{% include copy-curl.html %}
+
+After configuring the connection, you can view Prometheus metrics in OpenSearch Dashboards by going to the **Observability** > **Metrics** page, as shown in the following image.
+
+
+
+### Developer resources
+
+See the following developer resources for sample code, articles, tutorials, and API references:
+
+* [Datasource Settings](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst), which contains information about authentication and authorization of data source APIs.
+* [Prometheus Connector](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/prometheus_connector.rst), which contains configuration information.
+* [Simple Schema for Observability](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema/observability), which contains information about the OTel schema and ingest pipeline.
+* [OTel Metrics Source](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-metrics-source), which contains information about the Data Prepper metrics pipeline and ingestion.
+
+---
+
+## Experimenting with OpenTelemetry Metrics in the OpenSearch demo environment
+
+The OpenSearch [`opentelemetry-demo` repository](https://github.com/opensearch-project/opentelemetry-demo) provides a practical demonstration of collecting, processing, and visualizing metric data through **OpenTelemetry Metrics** from OpenTelemetry and using the **Metrics** tool in OpenSearch Dashboards.
+
+### Visualizing OTel metrics in OpenSearch
+
+To visualize OTel metric data in OpenSearch, follow these steps:
+
+1. Install the [`opentelemetry-demo` repository](https://github.com/opensearch-project/opentelemetry-demo). See the [Getting Started](https://github.com/opensearch-project/opentelemetry-demo/blob/main/tutorial/GettingStarted.md) guide for instructions.
+2. Collect the OTel signals, including metric signals. See the [OTel Collector](https://opentelemetry.io/docs/collector/) guide for instructions.
+3. Configure the OTel pipeline to emit metric signals. See the [OTel Collector Pipeline](https://github.com/opensearch-project/opentelemetry-demo/tree/main/src/otelcollector) guide for instructions.
+
+#### Example YAML config file
+
+```yaml
+ service:
+ extensions: [basicauth/client]
+ pipelines:
+ traces:
+ receivers: [otlp]
+ processors: [batch]
+ exporters: [otlp, debug, spanmetrics, otlp/traces, opensearch/traces]
+ metrics:
+ receivers: [otlp, spanmetrics]
+ processors: [filter/ottl, transform, batch]
+ exporters: [otlphttp/prometheus, otlp/metrics, debug]
+ logs:
+ receivers: [otlp]
+ processors: [batch]
+ exporters: [otlp/logs, opensearch/logs, debug]
+```
+{% include copy-curl.html %}
+
+4. Configure the [Data Prepper pipeline](https://github.com/opensearch-project/opentelemetry-demo/blob/main/src/dataprepper/pipelines.yaml) to emit the collected metric signals into the OpenSearch metrics index.
+
+#### Example YAML config file
+
+```yaml
+ otel-metrics-pipeline:
+ workers: 8
+ delay: 3000
+ source:
+ otel_metrics_source:
+ health_check_service: true
+ ssl: false
+ buffer:
+ bounded_blocking:
+ buffer_size: 1024 # max number of records the buffer accepts
+ batch_size: 1024 # max number of records the buffer drains after each read
+ processor:
+ - otel_metrics:
+ calculate_histogram_buckets: true
+ calculate_exponential_histogram_buckets: true
+ exponential_histogram_max_allowed_scale: 10
+ flatten_attributes: false
+ sink:
+ - opensearch:
+ hosts: ["https://opensearch-node1:9200"]
+ username: "admin"
+ password: "my_%New%_passW0rd!@#"
+ insecure: true
+ index_type: custom
+ template_file: "templates/ss4o_metrics.json"
+ index: ss4o_metrics-otel-%{yyyy.MM.dd}
+ bulk_size: 4
+```
+{% include copy-curl.html %}
-After configuring the connection from Prometheus to OpenSearch, Prometheus metrics are displayed in Dashboards in the **Observability** > **Metrics analytics** window, as shown in the following image.
+5. Ingest metric data into OpenSearch. As the demo starts generating data, the metric signals will be added to the OpenSearch index that supports the OpenTelemetry Metrics schema format.
+6. On the **Metrics** page, choose `Otel-Index` from the **Data sources** dropdown menu and `Simple Schema for Observability Index` from the **OTel index** dropdown menu. A visualization is displayed, as shown in the following image.
-
+
-* For more information about authentication and authorization of data source APIs, see [data source documentation on GitHub](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst).
-* For more information about the Prometheus connector, see the [Prometheus Connector](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/prometheus_connector.rst) GitHub page.
+---
-## Creating visualizations based on metrics
+## Visualizing metrics in remote clusters
+Introduced 2.14
+{: .label .label-purple }
+
+You can view metrics from remote OpenSearch clusters by using the **Metrics** tool. Select the database icon on the upper-right toolbar and choose a cluster from the **DATA SOURCES** dropdown menu, as shown in the following image. You can switch from a local cluster to a remote cluster.
+
+
+
+You can also view metric visualizations from other sources alongside local metric visualizations. From the **DATA SOURCES** dropdown menu, choose the remote metric visualization to add it to the group of visualizations already shown on the dashboard. An example dashboard is shown in the following image.
-You can create visualizations based on Prometheus metrics and other metrics collected by your OpenSearch cluster.
+
-To create a visualization, do the following:
+To learn about multi-cluster support for data sources, see [Enable OpenSearch Dashboards to support multiple OpenSearch clusters](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1388).
-1. In **Observability** > **Metrics analytics** > **Available Metrics**, select the metrics you would like to include in your visualization.
-1. These visualizations can now be saved.
-1. From the **Metrics analytics** window, select **Save**.
-1. When prompted for a **Custom operational dashboards/application**, choose one of the available options.
-1. Optionally, you can edit the predefined name values under the **Metric Name** fields to suit your needs.
-1. Select **Save**.
+## Creating visualizations based on custom metrics
-The following image shows an example of the visualizations that are displayed in the **Observability** > **Metrics analytics** window.
+You can create visualizations using the metric data collected by your OpenSearch cluster, including Prometheus metrics and custom metrics.
-
+To create these visualizations, follow these steps:
-## Defining PPL queries for use with Prometheus
+1. From the OpenSearch Dashboards main menu, navigate to **Observability** > **Metrics** > **Available Metrics**.
+2. Choose the metrics to add to your visualization and then select **Save**.
+3. When prompted for a **Custom operational dashboards/application**, choose one of the listed options. You can edit the predefined name values in the **Metric Name** field.
+4. Select **Save** to save your visualization. An example visualization is shown in the following image.
-You can define [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index) (PPL) queries against metrics collected by Prometheus. The following example shows a metric-selecting query with specific dimensions:
+
+
+## Defining PPL queries for Prometheus metrics
+
+You can define [Piped Processing Language (PPL)]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) queries to interact with metrics collected by Prometheus. The following is an example PPL query for a Prometheus metric:
```
source = my_prometheus.prometheus_http_requests_total | stats avg(@value) by span(@timestamp,15s), handler, code
```
+{% include copy-curl.html %}
+
+### Creating a custom visualization based on the PPL query
-Additionally, you can create a custom visualization by performing the following steps:
+To create a custom visualization based on the PPL query, follow these steps:
-1. From the **Events Analytics** window, enter your PPL query and select **Refresh**. The **Explorer page** is now displayed.
-1. From the **Explorer page**, select **Save**.
-1. When prompted for a **Custom operational dashboards/application**, choose one of the available options.
-1. Optionally, you can edit the predefined name values under the **Metric Name** fields to suit your needs.
-1. Optionally, you can choose to save the visualization as a metric.
-1. Select **Save**.
+1. From the **Logs** page, select > **Event Explorer**.
+2. On the **Explorer** page, enter your PPL query and select **Run**. Then select **Save**.
+3. When prompted to choose a **Custom Operational Dashboards/Application**, select one of the listed options. Optionally, you can edit the predefined name values in the **Metric Name** fields and can choose to save the visualization as a metric.
+5. Select **Save** to save your custom visualization.
-Note: Only queries that include a time-series visualization and stats/span can be saved as a metric, as shown in the following image.
+Only queries that include a time-series visualization and statistics or span information can be saved as a metric, as shown in the following image.
-
+
diff --git a/_observing-your-data/query-insights/top-n-queries.md b/_observing-your-data/query-insights/top-n-queries.md
index 44469fa64b..e6dadf33c5 100644
--- a/_observing-your-data/query-insights/top-n-queries.md
+++ b/_observing-your-data/query-insights/top-n-queries.md
@@ -9,23 +9,28 @@ nav_order: 65
Monitoring the top N queries in query insights features can help you gain real-time insights into the top queries with high latency within a certain time frame (for example, the last hour).
-## Getting started
+## Configuring top N query monitoring
-To enable monitoring of the top N queries, configure the following [dynamic settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#dynamic-settings):
+You can configure top N query monitoring by the following metric types:
-- `search.insights.top_queries.latency.enabled`: Set to `true` to [enable monitoring of the top N queries](#enabling-the-top-n-queries-feature).
-- `search.insights.top_queries.latency.window_size`: [Configure the window size](#configuring-window-size).
-- `search.insights.top_queries.latency.top_n_size`: [Specify the value of n](#configuring-the-value-of-n).
+- `latency`
+- `cpu`
+- `memory`
-It's important to exercise caution when enabling this feature because it can consume system resources.
-{: .important}
+Each metric has a set of corresponding settings:
+
+- `search.insights.top_queries.| Space type | -Distance function (d) | -OpenSearch score | -
|---|---|---|
| l1 | -\[ d(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^n |x_i - y_i| \] | -\[ score = {1 \over 1 + d } \] | -
| l2 | -\[ d(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^n (x_i - y_i)^2 \] | -\[ score = {1 \over 1 + d } \] | -
| \[ d(\mathbf{x}, \mathbf{y}) = max(|x_i - y_i|) \] | -\[ score = {1 \over 1 + d } \] | -|
| cosinesimil | -\[ d(\mathbf{x}, \mathbf{y}) = 1 - cos { \theta } = 1 - {\mathbf{x} · \mathbf{y} \over \|\mathbf{x}\| · \|\mathbf{y}\|}\]\[ = 1 - - {\sum_{i=1}^n x_i y_i \over \sqrt{\sum_{i=1}^n x_i^2} · \sqrt{\sum_{i=1}^n y_i^2}}\] - where \(\|\mathbf{x}\|\) and \(\|\mathbf{y}\|\) represent the norms of vectors x and y respectively. | -nmslib and Faiss:\[ score = {1 \over 1 + d } \] Lucene:\[ score = {2 - d \over 2}\] |
-
| innerproduct (supported for Lucene in OpenSearch version 2.13 and later) | -\[ d(\mathbf{x}, \mathbf{y}) = - {\mathbf{x} · \mathbf{y}} = - \sum_{i=1}^n x_i y_i \]
- Lucene: - \[ d(\mathbf{x}, \mathbf{y}) = {\mathbf{x} · \mathbf{y}} = \sum_{i=1}^n x_i y_i \] - |
- \[ \text{If} d \ge 0, \] \[score = {1 \over 1 + d }\] \[\text{If} d < 0, score = −d + 1\]
- Lucene: - \[ \text{If} d > 0, score = d + 1 \] \[\text{If} d \le 0\] \[score = {1 \over 1 + (-1 · d) }\] - |
-
-You can use the **Quick select** settings to specify an exact window of time.
-* Select either **Last** or **Next** in the first dropdown list to set the window of time behind the current setting or ahead of the current setting.
-* Select a number in the second dropdown list to define a value for the range.
-* Select a unit of time in the third dropdown list. Available options are seconds, minutes, hours, days, weeks, months, and years.
-Select the **Apply** button to apply the range of dates to the graph. Information on the graph changes accordingly.
+You can use the **Quick select** settings to specify a date range:
+* Select either **Last** or **Next** from the first dropdown menu to set the date range to either behind the current setting or ahead of the current setting.
+* Select a number from the second dropdown menu to define a value for the range.
+* Select a unit of time from the third dropdown menu. Available options are seconds, minutes, hours, days, weeks, months, and years.
+* Select the **Apply** button to apply the time range to the graph. The graph is immediately updated.
+
+An example window is shown in the following image.
-You can use the left and right arrows to move the window of time behind the current range of dates or ahead of the current range of dates. When you use these arrows, the start date and end date appear in the date range field. You can then select each one to set an absolute, relative, or current date and time. For absolute and relative changes, select the **Update** button to apply the changes.
+You can use the left and right arrows in the upper-left corner to shift the time range backward or forward, respectively. When you use these arrows, the start date and end date appear in the date range field. You can then select each one to set an absolute, relative, or current date and time. For absolute and relative changes, select the **Update** button to apply the changes.
+
+An example window is shown in the following image.
-As an alternative, you can select an option in the **Commonly used** section (see the preceding image of the calendar dropdown list) to conveniently set a window of time. Options include date ranges such as **Today**, **Yesterday**, **this week**, and **week to date**.
+As an alternative, you can select an option in the **Commonly used** section (see the preceding image of the calendar dropdown menu) to conveniently set a date range. Options include **Today**, **Yesterday**, **this week**, and **week to date**.
-When one of the commonly used windows of time is selected, you can select the **Show dates** label in the date range field to populate the range of dates. Following that, you can select either the start date or end date to specify by an absolute, relative, or current date and time setting. For absolute and relative changes, select the **Update** button to apply the changes.
+When a commonly used date range is selected, you can select the **Show dates** label in the date range field to populate the ranges. You can then select either the start date or end date to specify an absolute, relative, or current date and time setting. For absolute and relative changes, select the **Update** button to apply the changes.
-As one more alternative, you can select an option from the **Recently used date ranges** section to go back to a previous setting.
+You can also select an option from the **Recently used date ranges** section to revert to a previous setting.
---
-## The Alerts list
-The Alerts list displays all findings according to the time when the alert was triggered, the alert's trigger name, the detector that triggered the alert, the alert status, and alert severity.
-Use the **Alert severity** dropdown list to filter the list of alerts by severity. Use the **Status** dropdown list to filter the list by alert status.
+## Alerts list
+
+The **Alerts list** displays all alerts, with two tabs for different types of alerts:
+
+- **Findings**: The **Alerts list** displays all findings according to the time when the alert was triggered, the alert's trigger name, the detector that triggered the alert, the alert status, and the alert severity.
+- **Correlations**: The **Alerts list** displays all correlations, including the correlation rule and time window, the alert's trigger name, the correlation rule name that triggered the alert, the alert status, and the alert severity.
+Use the **Alert severity** dropdown menu to filter the list of alerts by severity. Use the **Status** dropdown menu to filter the list by alert status.
diff --git a/_security/access-control/api.md b/_security/access-control/api.md
index bea080a078..63717d621a 100644
--- a/_security/access-control/api.md
+++ b/_security/access-control/api.md
@@ -815,18 +815,24 @@ Creates, updates, or deletes multiple roles in a single call.
PATCH _plugins/_security/api/roles
[
{
- "op": "replace", "path": "/role1/index_permissions/0/fls", "value": ["test1", "test2"]
+ "op": "replace", "path": "/role1/index_permissions/0/fls", "value": ["myfield*", "~myfield1"]
},
{
"op": "remove", "path": "/role1/index_permissions/0/dls"
},
{
- "op": "add", "path": "/role2/cluster_permissions", "value": ["manage_snapshots"]
+ "op": "add", "path": "/role2/cluster_permissions/-", "value": {
+ "index_patterns": ["test_index"],
+ "allowed_actions": ["indices:data/read/scroll/clear"]
+ }
}
]
```
{% include copy-curl.html %}
+You can use `-` to insert a new permission to the end of the array of permissions.
+{: .note}
+
#### Example response
```json
@@ -1005,6 +1011,98 @@ PATCH _plugins/_security/api/rolesmapping
}
```
+---
+
+## Allowlist
+
+### Get allowlist
+
+Retrieves the current `allowlist` configuration.
+
+#### Request
+
+```json
+GET _plugins/_security/api/allowlist
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+```json
+{
+ "config" : {
+ "enabled" : true,
+ "requests" : {
+ "/_cat/nodes" : [
+ "GET"
+ ],
+ "/_cat/indices" : [
+ "GET"
+ ],
+ "/_plugins/_security/whoami" : [
+ "GET"
+ ]
+ }
+ }
+}
+```
+
+### Create allowlist
+
+Creates an `allowlist` configuration.
+
+#### Request
+
+```json
+PUT _plugins/_security/api/allowlist
+{
+ "enabled": true,
+ "requests": {
+ "/_cat/nodes": ["GET"],
+ "/_cat/indices": ["GET"],
+ "/_plugins/_security/whoami": ["GET"]
+ }
+}
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+```json
+{
+ "status":"OK",
+ "message":"'config' updated."
+}
+```
+
+### Update allowlist
+
+Updates an `allowlist` configuration.
+
+#### Request
+
+```json
+PATCH _plugins/_security/api/allowlist
+[
+ {
+ "op": "add",
+ "path": "/config/requests",
+ "value": {
+ "/_cat/nodes": ["POST"]
+ }
+ }
+]
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+```json
+{
+ "status":"OK",
+ "message":"Resource updated."
+}
+```
---
diff --git a/_security/access-control/field-masking.md b/_security/access-control/field-masking.md
index c672e75d04..188def5a6c 100644
--- a/_security/access-control/field-masking.md
+++ b/_security/access-control/field-masking.md
@@ -12,7 +12,7 @@ redirect_from:
If you don't want to remove fields from a document using [field-level security]({{site.url}}{{site.baseurl}}/security/access-control/field-level-security/), you can mask their values. Currently, field masking is only available for string-based fields and replaces the field's value with a cryptographic hash.
-Field masking works alongside field-level security on the same per-role, per-index basis. You can allow certain roles to see sensitive fields in plain text and mask them for others. A search result with a masked field might look like this:
+Field masking works alongside field-level security on the same per-role, per-index basis. You can allow certain roles to see sensitive fields in plain text and mask them for others. A search result with a masked field might look like the following:
```json
{
@@ -28,19 +28,20 @@ Field masking works alongside field-level security on the same per-role, per-ind
```
-## Set the salt
+## Set the salt setting
-You set the salt (a random string used to hash your data) in `opensearch.yml`:
+You can set the salt (a random string used to hash your data) in `opensearch.yml` using the optional `plugins.security.compliance.salt` setting. The salt value must fullfil the following requirements:
+
+- Must be at least 32 characters.
+- Use only ASCII characters.
+
+The following example shows a salt value:
```yml
plugins.security.compliance.salt: abcdefghijklmnopqrstuvqxyz1234567890
```
-Property | Description
-:--- | :---
-`plugins.security.compliance.salt` | The salt to use when generating the hash value. Must be at least 32 characters. Only ASCII characters are allowed. Optional.
-
-Setting the salt is optional, but we highly recommend it.
+Although setting the salt is optional, it is highly recommended.
## Configure field masking
@@ -78,7 +79,14 @@ See [Create role]({{site.url}}{{site.baseurl}}/security/access-control/api/#crea
By default, the Security plugin uses the BLAKE2b algorithm, but you can use any hashing algorithm that your JVM provides. This list typically includes MD5, SHA-1, SHA-384, and SHA-512.
-To specify a different algorithm, add it after the masked field:
+You can override the default algorithm in `opensearch.yml` using the option default masking algorithm setting `plugins.security.masked_fields.algorithm.default`, as shown in the following example:
+
+```yml
+plugins.security.masked_fields.algorithm.default: SHA-256
+```
+.
+
+To specify a different algorithm, add it after the masked field in `roles.yml`, as shown in the following:
```yml
someonerole:
@@ -95,7 +103,7 @@ someonerole:
## (Advanced) Pattern-based field masking
-Rather than creating a hash, you can use one or more regular expressions and replacement strings to mask a field. The syntax is `