|
|||||||
| Fast, Scalable Full-text Search | -Application and Infrastructure Monitoring | -Security and Event Information Management | -Operational Health Tracking | +Fast, scalable full-text search | +Application and infrastructure monitoring | +Security and event information management | +Operational health tracking |
| Help users find the right information within your application, website, or data lake catalog. | -Easily store and analyze log data, and set automated alerts for underperformance. | +Easily store and analyze log data, and set automated alerts for performance issues. | Centralize logs to enable real-time security monitoring and forensic analysis. | -Use observability logs, metrics, and traces to monitor your applications and business in real time. | +Use observability logs, metrics, and traces to monitor your applications in real time. |
+## Manage your data source
-3. On the **Configure Amazon S3 data source** page, enter the required **Data source details**, **AWS Glue authentication details**, **AWS Glue index store details**, and **Query permissions**. An example UI is shown in the following image.
-
- 
-
-4. Select the **Review Configuration** button and verify the details.
-5. Select the **Connect to Amazon S3** button.
-
-## Manage your Amazon S3 data source
-
-Once you've connected your Amazon S3 data source, you can explore your data through the **Manage data sources** tab. The following steps guide you through using this functionality:
+To manage your data source, follow these steps:
1. On the **Manage data sources** tab, choose a date source from the list.
-2. On that data source's page, you can manage the data source, choose a use case, and manage access controls and configurations. An example UI is shown in the following image.
-
- 
-
-3. (Optional) Explore the Amazon S3 use cases, including querying your data and optimizing query performance. Go to **Next steps** to learn more about each use case.
+2. On the page for the data source, you can manage the data source, choose a use case, and configure access controls.
+3. (Optional) Explore the Amazon S3 use cases, including querying your data and optimizing query performance. Refer to the [**Next steps**](#next-steps) section to learn more about each use case.
## Limitations
-This feature is still under development, including the data integration functionality. For real-time updates, see the [developer documentation on GitHub](https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md#limitations).
+This feature is currently under development, including the data integration functionality. For up-to-date information, refer to the [developer documentation on GitHub](https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md#limitations).
## Next steps
- Learn about [querying your data in Data Explorer]({{site.url}}{{site.baseurl}}/dashboards/management/query-data-source/) through OpenSearch Dashboards.
-- Learn about ways to [optimize the query performance of your external data sources]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/), such as Amazon S3, through Query Workbench.
+- Learn about [optimizing the query performance of your external data sources]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/), such as Amazon S3, through Query Workbench.
- Learn about [Amazon S3 and AWS Glue Data Catalog](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/s3glue_connector.rst) and the APIS used with Amazon S3 data sources, including configuration settings and query examples.
- Learn about [managing your indexes]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards.
-
diff --git a/_dashboards/management/accelerate-external-data.md b/_dashboards/management/accelerate-external-data.md
index 00e4600ffd..6d1fa030e4 100644
--- a/_dashboards/management/accelerate-external-data.md
+++ b/_dashboards/management/accelerate-external-data.md
@@ -12,55 +12,37 @@ Introduced 2.11
{: .label .label-purple }
-Query performance can be slow when using external data sources for reasons such as network latency, data transformation, and data volume. You can optimize your query performance by using OpenSearch indexes, such as a skipping index or a covering index. A _skipping index_ uses skip acceleration methods, such as partition, minimum and maximum values, and value sets, to ingest and create compact aggregate data structures. This makes them an economical option for direct querying scenarios. A _covering index_ ingests all or some of the data from the source into OpenSearch and makes it possible to use all OpenSearch Dashboards and plugin functionality. See the [Flint Index Reference Manual](https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md) for comprehensive guidance on this feature's indexing process.
+Query performance can be slow when using external data sources for reasons such as network latency, data transformation, and data volume. You can optimize your query performance by using OpenSearch indexes, such as a skipping index or a covering index.
+
+A _skipping index_ uses skip acceleration methods, such as partition, minimum and maximum values, and value sets, to ingest and create compact aggregate data structures. This makes them an economical option for direct querying scenarios.
+
+A _covering index_ ingests all or some of the data from the source into OpenSearch and makes it possible to use all OpenSearch Dashboards and plugin functionality. See the [Flint Index Reference Manual](https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md) for comprehensive guidance on this feature's indexing process.
## Data sources use case: Accelerate performance
To get started with the **Accelerate performance** use case available in **Data sources**, follow these steps:
1. Go to **OpenSearch Dashboards** > **Query Workbench** and select your Amazon S3 data source from the **Data sources** dropdown menu in the upper-left corner.
-2. From the left-side navigation menu, select a database. An example using the `http_logs` database is shown in the following image.
-
- 
-
+2. From the left-side navigation menu, select a database.
3. View the results in the table and confirm that you have the desired data.
4. Create an OpenSearch index by following these steps:
- 1. Select the **Accelerate data** button. A pop-up window appears. An example is shown in the following image.
-
- 
-
+ 1. Select the **Accelerate data** button. A pop-up window appears.
2. Enter your details in **Select data fields**. In the **Database** field, select the desired acceleration index: **Skipping index** or **Covering index**. A _skipping index_ uses skip acceleration methods, such as partition, min/max, and value sets, to ingest data using compact aggregate data structures. This makes them an economical option for direct querying scenarios. A _covering index_ ingests all or some of the data from the source into OpenSearch and makes it possible to use all OpenSearch Dashboards and plugin functionality.
-
-5. Under **Index settings**, enter the information for your acceleration index. For information about naming, select **Help**. Note that an Amazon S3 table can only have one skipping index at a time. An example is shown in the following image.
-
- 
+5. Under **Index settings**, enter the information for your acceleration index. For information about naming, select **Help**. Note that an Amazon S3 table can only have one skipping index at a time.
### Define skipping index settings
-1. Under **Skipping index definition**, select the **Add fields** button to define the skipping index acceleration method and choose the fields you want to add. An example is shown in the following image.
-
- 
-
+1. Under **Skipping index definition**, select the **Add fields** button to define the skipping index acceleration method and choose the fields you want to add.
2. Select the **Copy Query to Editor** button to apply your skipping index settings.
-3. View the skipping index query details in the table pane and then select the **Run** button. Your index is added to the left-side navigation menu containing the list of your databases. An example is shown in the following image.
-
- 
+3. View the skipping index query details in the table pane and then select the **Run** button. Your index is added to the left-side navigation menu containing the list of your databases.
### Define covering index settings
-1. Under **Index settings**, enter a valid index name. Note that each Amazon S3 table can have multiple covering indexes. An example is shown in the following image.
-
- 
-
-2. Once you have added the index name, define the covering index fields by selecting `(add fields here)` under **Covering index definition**. An example is shown in the following image.
-
- 
-
+1. Under **Index settings**, enter a valid index name. Note that each Amazon S3 table can have multiple covering indexes.
+2. Once you have added the index name, define the covering index fields by selecting `(add fields here)` under **Covering index definition**.
3. Select the **Copy Query to Editor** button to apply your covering index settings.
-4. View the covering index query details in the table pane and then select the **Run** button. Your index is added to the left-side navigation menu containing the list of your databases. An example UI is shown in the following image.
-
- 
+4. View the covering index query details in the table pane and then select the **Run** button. Your index is added to the left-side navigation menu containing the list of your databases.
## Limitations
-This feature is still under development, so there are some limitations. For real-time updates, see the [developer documentation on GitHub](https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md#limitations).
+This feature is still under development, so there are some limitations. For real-time updates, refer to the [developer documentation on GitHub](https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md#limitations).
diff --git a/_dashboards/management/connect-prometheus.md b/_dashboards/management/connect-prometheus.md
new file mode 100644
index 0000000000..ed5545bc56
--- /dev/null
+++ b/_dashboards/management/connect-prometheus.md
@@ -0,0 +1,53 @@
+---
+layout: default
+title: Connecting Prometheus to OpenSearch
+parent: Data sources
+nav_order: 20
+---
+
+# Connecting Prometheus to OpenSearch
+Introduced 2.16
+{: .label .label-purple }
+
+This documentation covers the key steps to connect Prometheus to OpenSearch using the OpenSearch Dashboards interface, including setting up the data source connection, modifying the connection details, and creating an index pattern for the Prometheus data.
+
+## Prerequisites and permissions
+
+Before connecting a data source, ensure you have met the [Prerequisites]({{site.url}}{{site.baseurl}}/dashboards/management/data-sources/#prerequisites) and have the necessary [Permissions]({{site.url}}{{site.baseurl}}/dashboards/management/data-sources/#permissions).
+
+## Create a Prometheus data source connection
+
+A data source connection specifies the parameters needed to connect to a data source. These parameters form a connection string for the data source. Using OpenSearch Dashboards, you can add new **Prometheus** data source connections or manage existing ones.
+
+Follow these steps to connect your data source:
+
+1. From the OpenSearch Dashboards main menu, go to **Management** > **Data sources** > **New data source** > **Prometheus**.
+
+2. From the **Configure Prometheus data source** section:
+
+ - Under **Data source details**, provide a title and optional description.
+ - Under **Prometheus data location**, enter the Prometheus URI.
+ - Under **Authentication details**, select the appropriate authentication method from the dropdown list and enter the required details:
+ - **Basic authentication**: Enter a username and password.
+ - **AWS Signature Version 4**: Specify the **Region**, select the OpenSearch service from the **Service Name** list (**Amazon OpenSearch Service** or **Amazon OpenSearch Serverless**), and enter the **Access Key** and **Secret Key**.
+ - Under **Query permissions**, choose the role needed to search and index data. If you select **Restricted**, an additional field will become available to configure the required role.
+
+3. Select **Review Configuration** > **Connect to Prometheus** to save your settings. The new connection will appear in the list of data sources.
+
+## Modify a data source connection
+
+To modify a data source connection, follow these steps:
+
+1. Select the desired connection from the list on the **Data sources** main page. This will open the **Connection Details** window.
+2. Within the **Connection Details** window, edit the **Title** and **Description** fields. Select the **Save changes** button to apply the changes.
+3. To update the **Authentication Method**, choose the method from the dropdown list and enter any necessary credentials. Select **Save changes** to apply the changes.
+ - To update the **Basic authentication** authentication method, select the **Update stored password** button. Within the pop-up window, enter the updated password and confirm it and select **Update stored password** to save the changes. To test the connection, select the **Test connection** button.
+ - To update the **AWS Signature Version 4** authentication method, select the **Update stored AWS credential** button. Within the pop-up window, enter the updated access and secret keys and select **Update stored AWS credential** to save the changes. To test the connection, select the **Test connection** button.
+
+## Delete a data source connection
+
+To delete the data source connection, select the {::nomarkdown}
{:/} icon.
+
+## Create an index pattern
+
+After creating a data source connection, the next step is to create an index pattern for that data source. For more information and a tutorial on index patterns, refer to [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/).
diff --git a/_dashboards/management/data-sources.md b/_dashboards/management/data-sources.md
index fdd4edc150..62d3a5aab2 100644
--- a/_dashboards/management/data-sources.md
+++ b/_dashboards/management/data-sources.md
@@ -13,67 +13,37 @@ This documentation focuses on using the OpenSeach Dashboards interface to connec
## Prerequisites
-The first step in connecting your data sources to OpenSearch is to install OpenSearch and OpenSearch Dashboards on your system. You can follow the installation instructions in the [OpenSearch documentation]({{site.url}}{{site.baseurl}}/install-and-configure/index/) to install these tools.
+The first step in connecting your data sources to OpenSearch is to install OpenSearch and OpenSearch Dashboards on your system. Refer to the [installation instructions]({{site.url}}{{site.baseurl}}/install-and-configure/index/) for information.
Once you have installed OpenSearch and OpenSearch Dashboards, you can use Dashboards to connect your data sources to OpenSearch and then use Dashboards to manage data sources, create index patterns based on those data sources, run queries against a specific data source, and combine visualizations in one dashboard.
Configuration of the [YAML files]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/#configuration-file) and installation of the `dashboards-observability` and `opensearch-sql` plugins is necessary. For more information, see [OpenSearch plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/).
-## Permissions
-
-To work with data sources in OpenSearch Dashboards, make sure that the user has been assigned the correct cluster-level [data source permission]({{site.url}}{{site.baseurl}}/security/access-control/permissions#data-source-permissions).
-
-
-
-## Create a data source connection
-
-A data source connection specifies the parameters needed to connect to a data source. These parameters form a connection string for the data source. Using Dashboards, you can add new data source connections or manage existing ones.
-
-The following steps guide you through the basics of creating a data source connection:
-
-1. From the OpenSearch Dashboards main menu, select **Management** > **Data sources** > **Create data source connection**. The UI is shown in the following image.
+To securely store and encrypt data source connections in OpenSearch, you must add the following configuration to the `opensearch.yml` file on all the nodes:
- 
+`plugins.query.datasources.encryption.masterkey: "YOUR_GENERATED_MASTER_KEY_HERE"`
-2. Create the data source connection by entering the appropriate information into the **Connection Details** and **Authentication Method** fields.
-
- - Under **Connection Details**, enter a title and endpoint URL. For this tutorial, use the URL `http://localhost:5601/app/management/opensearch-dashboards/dataSources`. Entering a description is optional.
+The key must be 16, 24, or 32 characters. You can use the following command to generate a 24-character key:
- - Under **Authentication Method**, select an authentication method from the dropdown list. Once an authentication method is selected, the applicable fields for that method appear. You can then enter the required details. The authentication method options are:
- - **No authentication**: No authentication is used to connect to the data source.
- - **Username & Password**: A basic username and password are used to connect to the data source.
- - **AWS SigV4**: An AWS Signature Version 4 authenticating request is used to connect to the data source. AWS Signature Version 4 requires an access key and a secret key.
- - For AWS Signature Version 4 authentication, first specify the **Region**. Next, select the OpenSearch service in the **Service Name** list. The options are **Amazon OpenSearch Service** and **Amazon OpenSearch Serverless**. Lastly, enter the **Access Key** and **Secret Key** for authorization.
+`openssl rand -hex 12`
- After you have populated the required fields, the **Test connection** and **Create data source** buttons become active. You can select **Test connection** to confirm that the connection is valid.
+Generating 12 bytes results in a hexadecimal string that is 12 * 2 = 24 characters.
+{: .note}
-3. Select **Create data source** to save your settings. The connection is created. The active window returns to the **Data sources** main page, and the new connection appears in the list of data sources.
-
-4. To delete a data source connection, select the checkbox to the left of the data source **Title** and then select the **Delete 1 connection** button. Selecting multiple checkboxes for multiple connections is supported. An example UI is shown in the following image.
-
- 
-
-### Modify a data source connection
-
-To make changes to a data source connection, select a connection in the list on the **Data sources** main page. The **Connection Details** window opens.
-
-To make changes to **Connection Details**, edit one or both of the **Title** and **Description** fields and select **Save changes** in the lower-right corner of the screen. You can also cancel changes here. To change the **Authentication Method**, choose a different authentication method, enter your credentials (if applicable), and then select **Save changes** in the lower-right corner of the screen. The changes are saved.
-
-When **Username & Password** is the selected authentication method, you can update the password by choosing **Update stored password** next to the **Password** field. In the pop-up window, enter a new password in the first field and then enter it again in the second field to confirm. Select **Update stored password** in the pop-up window. The new password is saved. Select **Test connection** to confirm that the connection is valid.
+## Permissions
-When **AWS SigV4** is the selected authentication method, you can update the credentials by selecting **Update stored AWS credential**. In the pop-up window, enter a new access key in the first field and a new secret key in the second field. Select **Update stored AWS credential** in the pop-up window. The new credentials are saved. Select **Test connection** in the upper-right corner of the screen to confirm that the connection is valid.
+To work with data sources in OpenSearch Dashboards, you must be assigned the correct cluster-level [data source permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions#data-source-permissions).
-To delete the data source connection, select the delete icon ({::nomarkdown}{:/}).
+## Types of data streams
-## Create an index pattern
+To configure data sources through OpenSearch Dashboards, go to **Management** > **Dashboards Management** > **Data sources**. This flow can be used for OpenSearch data stream connections. See [Configuring and using multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/).
-Once you've created a data source connection, you can create an index pattern for the data source. An _index pattern_ is a template that OpenSearch uses to create indexes for data from the data source. See [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/) for more information and a tutorial.
+Alternatively, if you are running OpenSearch Dashboards 2.16 or later, go to **Management** > **Data sources**. This flow can be used to connect Amazon Simple Storage Service (Amazon S3) and Prometheus. See [Connecting Amazon S3 to OpenSearch]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/) and [Connecting Prometheus to OpenSearch]({{site.url}}{{site.baseurl}}/dashboards/management/connect-prometheus/) for more information.
## Next steps
- Learn about [managing index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/) through OpenSearch Dashboards.
- Learn about [indexing data using Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards.
- Learn about how to connect [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/).
-- Learn about how to [connect OpenSearch and Amazon S3 through OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/).
-- Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) tool, which gives you the flexibility to use various data ingestion methods and connect data from the Dashboards UI.
-
+- Learn about how to connect [OpenSearch and Amazon S3]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/) and [OpenSearch and Prometheus]({{site.url}}{{site.baseurl}}/dashboards/management/connect-prometheus/) using the OpenSearch Dashboards interface.
+- Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) plugin, which gives you the flexibility to use various data ingestion methods and connect data to OpenSearch Dashboards.
diff --git a/_dashboards/management/query-data-source.md b/_dashboards/management/query-data-source.md
index f1496b3e17..a3392c073e 100644
--- a/_dashboards/management/query-data-source.md
+++ b/_dashboards/management/query-data-source.md
@@ -11,7 +11,7 @@ has_children: false
Introduced 2.11
{: .label .label-purple }
-This tutorial guides you through using the **Query data** use case for querying and visualizing your Amazon Simple Storage Service (Amazon S3) data using OpenSearch Dashboards.
+This tutorial guides you through using the **Query data** use case for querying and visualizing your Amazon Simple Storage Service (Amazon S3) data using OpenSearch Dashboards.
## Prerequisites
@@ -22,15 +22,9 @@ You must be using the `opensearch-security` plugin and have the appropriate role
To get started, follow these steps:
1. On the **Manage data sources** page, select your data source from the list.
-2. On the data source's detail page, select the **Query data** card. This option takes you to the **Observability** > **Logs** page, which is shown in the following image.
-
- 
-
+2. On the data source's detail page, select the **Query data** card. This option takes you to the **Observability** > **Logs** page.
3. Select the **Event Explorer** button. This option creates and saves frequently searched queries and visualizations using [Piped Processing Language (PPL)]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) or [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/), which connects to Spark SQL.
-4. Select the Amazon S3 data source from the dropdown menu in the upper-left corner. An example is shown in the following image.
-
- 
-
+4. Select the Amazon S3 data source from the dropdown menu in the upper-left corner.
5. Enter the query in the **Enter PPL query** field. Note that the default language is SQL. To change the language, select PPL from the dropdown menu.
6. Select the **Search** button. The **Query Processing** message is shown, confirming that your query is being processed.
7. View the results, which are listed in a table on the **Events** tab. On this page, details such as available fields, source, and time are shown in a table format.
@@ -40,10 +34,7 @@ To get started, follow these steps:
To create visualizations, follow these steps:
-1. On the **Explorer** page, select the **Visualizations** tab. An example is shown in the following image.
-
- 
-
+1. On the **Explorer** page, select the **Visualizations** tab.
2. Select **Index data to visualize**. This option currently only creates [acceleration indexes]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/), which give you views of the data visualizations from the **Visualizations** tab. To create a visualization of your Amazon S3 data, go to **Discover**. See the [Discover documentation]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/) for information and a tutorial.
## Use Query Workbench with your Amazon S3 data source
@@ -53,14 +44,12 @@ To create visualizations, follow these steps:
To use Query Workbench with your Amazon S3 data, follow these steps:
1. From the OpenSearch Dashboards main menu, select **OpenSearch Plugins** > **Query Workbench**.
-2. From the **Data Sources** dropdown menu in the upper-left corner, choose your Amazon S3 data source. Your data begins loading the databases that are part of your data source. An example is shown in the following image.
-
- 
-
+2. From the **Data Sources** dropdown menu in the upper-left corner, choose your Amazon S3 data source. Your data begins loading the databases that are part of your data source.
3. View the databases listed in the left-side navigation menu and select a database to view its details. Any information about acceleration indexes is listed under **Acceleration index destination**.
4. Choose the **Describe Index** button to learn more about how data is stored in that particular index.
5. Choose the **Drop index** button to delete and clear both the OpenSearch index and the Amazon S3 Spark job that refreshes the data.
-6. Enter your SQL query and select **Run**.
+6. Enter your SQL query and select **Run**.
+
## Next steps
- Learn about [accelerating the query performance of your external data sources]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/).
diff --git a/_data-prepper/pipelines/configuration/processors/obfuscate.md b/_data-prepper/pipelines/configuration/processors/obfuscate.md
index 8d6bf901da..96b03e7405 100644
--- a/_data-prepper/pipelines/configuration/processors/obfuscate.md
+++ b/_data-prepper/pipelines/configuration/processors/obfuscate.md
@@ -62,6 +62,13 @@ When run, the `obfuscate` processor parses the fields into the following output:
Use the following configuration options with the `obfuscate` processor.
+
+
| Parameter | Required | Description |
| :--- | :--- | :--- |
| `source` | Yes | The source field to obfuscate. |
diff --git a/_data-prepper/pipelines/configuration/sinks/s3.md b/_data-prepper/pipelines/configuration/sinks/s3.md
index d1413f6ffc..3ff266cccf 100644
--- a/_data-prepper/pipelines/configuration/sinks/s3.md
+++ b/_data-prepper/pipelines/configuration/sinks/s3.md
@@ -173,14 +173,14 @@ When you provide your own Avro schema, that schema defines the final structure o
In cases where your data is uniform, you may be able to automatically generate a schema. Automatically generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and all keys must be present in all events in order to automatically generate a working schema. Automatically generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control which data is included in the automatically generated schema.
-Avro fields should use a null [union](https://avro.apache.org/docs/current/specification/#unions) because this will allow missing values. Otherwise, all required fields must be present for each event. Use non-nullable fields only when you are certain they exist.
+Avro fields should use a null [union](https://avro.apache.org/docs/1.10.2/spec.html#Unions) because this will allow missing values. Otherwise, all required fields must be present for each event. Use non-nullable fields only when you are certain they exist.
Use the following options to configure the codec.
Option | Required | Type | Description
:--- | :--- | :--- | :---
-`schema` | Yes | String | The Avro [schema declaration](https://avro.apache.org/docs/current/specification/#schema-declaration). Not required if `auto_schema` is set to true.
-`auto_schema` | No | Boolean | When set to `true`, automatically generates the Avro [schema declaration](https://avro.apache.org/docs/current/specification/#schema-declaration) from the first event.
+`schema` | Yes | String | The Avro [schema declaration](https://avro.apache.org/docs/1.2.0/spec.html#schemas). Not required if `auto_schema` is set to true.
+`auto_schema` | No | Boolean | When set to `true`, automatically generates the Avro [schema declaration](https://avro.apache.org/docs/1.2.0/spec.html#schemas) from the first event.
### `ndjson` codec
diff --git a/_data-prepper/pipelines/configuration/sources/opensearch.md b/_data-prepper/pipelines/configuration/sources/opensearch.md
index 7cc0b9a36a..a7ba965729 100644
--- a/_data-prepper/pipelines/configuration/sources/opensearch.md
+++ b/_data-prepper/pipelines/configuration/sources/opensearch.md
@@ -135,7 +135,7 @@ Option | Required | Type | Description
### Scheduling
-The `scheduling` configuration allows the user to configure how indexes are reprocessed in the source based on the the `index_read_count` and recount time `interval`.
+The `scheduling` configuration allows the user to configure how indexes are reprocessed in the source based on the `index_read_count` and recount time `interval`.
For example, setting `index_read_count` to `3` with an `interval` of `1h` will result in all indexes being reprocessed 3 times, 1 hour apart. By default, indexes will only be processed once.
diff --git a/_field-types/index.md b/_field-types/index.md
index 7a7e816ada..e9250f409d 100644
--- a/_field-types/index.md
+++ b/_field-types/index.md
@@ -12,43 +12,77 @@ redirect_from:
# Mappings and field types
-You can define how documents and their fields are stored and indexed by creating a _mapping_. The mapping specifies the list of fields for a document. Every field in the document has a _field type_, which defines the type of data the field contains. For example, you may want to specify that the `year` field should be of type `date`. To learn more, see [Supported field types]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/).
+Mappings tell OpenSearch how to store and index your documents and their fields. You can specify the data type for each field (for example, `year` as `date`) to make storage and querying more efficient.
-If you're just starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings.
+While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps to maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations.
-For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers.
+For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer instead of both being interpreted as integers by dynamic mapping.
-This section provides an example for how to create an index mapping and how to add a document to it that will get ip_range validated.
-
-#### Table of contents
-1. TOC
-{:toc}
-
-
----
## Dynamic mapping
When you index a document, OpenSearch adds fields automatically with dynamic mapping. You can also explicitly add fields to an index mapping.
-#### Dynamic mapping types
+### Dynamic mapping types
Type | Description
:--- | :---
-null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values.
-boolean | OpenSearch accepts `true` and `false` as boolean values. An empty string is equal to `false.`
-float | A single-precision 32-bit floating point number.
-double | A double-precision 64-bit floating point number.
-integer | A signed 32-bit number.
-object | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`.
-array | Arrays in OpenSearch can only store values of one type, such as an array of just integers or strings. Empty arrays are treated as though they are fields with no values.
-text | A string sequence of characters that represent full-text values.
-keyword | A string sequence of structured characters, such as an email address or ZIP code.
+`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if the field has no value.
+`boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.`
+`float` | A single-precision, 32-bit floating-point number.
+`double` | A double-precision, 64-bit floating-point number.
+`integer` | A signed 32-bit number.
+`object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`.
+`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements---not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, which may contain complex objects or other arrays, can also be used for advanced data modeling.
+`text` | A string sequence of characters that represent full-text values.
+`keyword` | A string sequence of structured characters, such as an email address or ZIP code.
date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date.
numeric detection string | If disabled, OpenSearch may automatically process numeric values as strings when they should be processed as numbers. When enabled, OpenSearch can process strings into `long`, `integer`, `short`, `byte`, `double`, `float`, `half_float`, `scaled_float`, and `unsigned_long`. Default is disabled.
+### Dynamic templates
+
+Dynamic templates are used to define custom mappings for dynamically added fields based on the data type, field name, or field path. They allow you to define a flexible schema for your data that can automatically adapt to changes in the structure or format of the input data.
+
+You can use the following syntax to define a dynamic mapping template:
+
+```json
+PUT index
+{
+ "mappings": {
+ "dynamic_templates": [
+ {
+ "fields": {
+ "mapping": {
+ "type": "short"
+ },
+ "match_mapping_type": "string",
+ "path_match": "status*"
+ }
+ }
+ ]
+ }
+}
+```
+{% include copy-curl.html %}
+
+This mapping configuration dynamically maps any field with a name starting with `status` (for example, `status_code`) to the `short` data type if the initial value provided during indexing is a string.
+
+### Dynamic mapping parameters
+
+The `dynamic_templates` support the following parameters for matching conditions and mapping rules. The default value is `null`.
+
+Parameter | Description |
+----------|-------------|
+`match_mapping_type` | Specifies the JSON data type (for example, string, long, double, object, binary, Boolean, date) that triggers the mapping.
+`match` | A regular expression used to match field names and apply the mapping.
+`unmatch` | A regular expression used to exclude field names from the mapping.
+`match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`.
+`path_match` | Allows you to match nested field paths using a regular expression.
+`path_unmatch` | Excludes nested field paths from the mapping using a regular expression.
+`mapping` | The mapping configuration to apply.
+
## Explicit mapping
-If you know exactly what your field data types need to be, you can specify them in your request body when creating your index.
+If you know exactly which field data types you need to use, then you can specify them in your request body when creating your index, as shown in the following example request:
```json
PUT sample-index1
@@ -62,8 +96,9 @@ PUT sample-index1
}
}
```
+{% include copy-curl.html %}
-### Response
+#### Response
```json
{
"acknowledged": true,
@@ -71,8 +106,9 @@ PUT sample-index1
"index": "sample-index1"
}
```
+{% include copy-curl.html %}
-To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method:
+To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method, as shown in the following example request:
```json
POST sample-index1/_mapping
@@ -84,84 +120,29 @@ POST sample-index1/_mapping
}
}
```
+{% include copy-curl.html %}
You cannot change the mapping of an existing field, you can only modify the field's mapping parameters.
{: .note}
----
-## Mapping example usage
+## Mapping parameters
-The following example shows how to create a mapping to specify that OpenSearch should ignore any documents with malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You accomplish this by setting the `ignore_malformed` parameter to `true`.
+Mapping parameters are used to configure the behavior of index fields. See [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) for more information.
-### Create an index with an `ip` mapping
+## Mapping limit settings
-To create an index, use a PUT request:
+OpenSearch has certain mapping limits and settings, such as the settings listed in the following table. Settings can be configured based on your requirements.
-```json
-PUT /test-index
-{
- "mappings" : {
- "properties" : {
- "ip_address" : {
- "type" : "ip",
- "ignore_malformed": true
- }
- }
- }
-}
-```
-
-You can add a document that has a malformed IP address to your index:
-
-```json
-PUT /test-index/_doc/1
-{
- "ip_address" : "malformed ip address"
-}
-```
-
-This indexed IP address does not throw an error because `ignore_malformed` is set to true.
-
-You can query the index using the following request:
-
-```json
-GET /test-index/_search
-```
+| Setting | Default value | Allowed value | Type | Description |
+|-|-|-|-|-|
+| `index.mapping.nested_fields.limit` | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. |
+| `index.mapping.nested_objects.limit` | 10,000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created in a single document. |
+| `index.mapping.total_fields.limit` | 1,000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. |
+| `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. |
+| `index.mapping.field_name_length.limit` | 50,000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. |
+| `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be dynamically added to a mapping. |
-The response shows that the `ip_address` field is ignored in the indexed document:
-
-```json
-{
- "took": 14,
- "timed_out": false,
- "_shards": {
- "total": 1,
- "successful": 1,
- "skipped": 0,
- "failed": 0
- },
- "hits": {
- "total": {
- "value": 1,
- "relation": "eq"
- },
- "max_score": 1,
- "hits": [
- {
- "_index": "test-index",
- "_id": "1",
- "_score": 1,
- "_ignored": [
- "ip_address"
- ],
- "_source": {
- "ip_address": "malformed ip address"
- }
- }
- ]
- }
-}
-```
+---
## Get a mapping
@@ -170,14 +151,16 @@ To get all mappings for one or more indexes, use the following request:
```json
GET