|
|||||||
| Fast, Scalable Full-text Search | -Application and Infrastructure Monitoring | -Security and Event Information Management | -Operational Health Tracking | +Fast, scalable full-text search | +Application and infrastructure monitoring | +Security and event information management | +Operational health tracking |
| Help users find the right information within your application, website, or data lake catalog. | -Easily store and analyze log data, and set automated alerts for underperformance. | +Easily store and analyze log data, and set automated alerts for performance issues. | Centralize logs to enable real-time security monitoring and forensic analysis. | -Use observability logs, metrics, and traces to monitor your applications and business in real time. | +Use observability logs, metrics, and traces to monitor your applications in real time. |
{:/} icon.
+
+## Create an index pattern
+
+After creating a data source connection, the next step is to create an index pattern for that data source. For more information and a tutorial on index patterns, refer to [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/).
diff --git a/_dashboards/management/data-sources.md b/_dashboards/management/data-sources.md
index fdd4edc150..62d3a5aab2 100644
--- a/_dashboards/management/data-sources.md
+++ b/_dashboards/management/data-sources.md
@@ -13,67 +13,37 @@ This documentation focuses on using the OpenSeach Dashboards interface to connec
## Prerequisites
-The first step in connecting your data sources to OpenSearch is to install OpenSearch and OpenSearch Dashboards on your system. You can follow the installation instructions in the [OpenSearch documentation]({{site.url}}{{site.baseurl}}/install-and-configure/index/) to install these tools.
+The first step in connecting your data sources to OpenSearch is to install OpenSearch and OpenSearch Dashboards on your system. Refer to the [installation instructions]({{site.url}}{{site.baseurl}}/install-and-configure/index/) for information.
Once you have installed OpenSearch and OpenSearch Dashboards, you can use Dashboards to connect your data sources to OpenSearch and then use Dashboards to manage data sources, create index patterns based on those data sources, run queries against a specific data source, and combine visualizations in one dashboard.
Configuration of the [YAML files]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/#configuration-file) and installation of the `dashboards-observability` and `opensearch-sql` plugins is necessary. For more information, see [OpenSearch plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/).
-## Permissions
-
-To work with data sources in OpenSearch Dashboards, make sure that the user has been assigned the correct cluster-level [data source permission]({{site.url}}{{site.baseurl}}/security/access-control/permissions#data-source-permissions).
-
-
-
-## Create a data source connection
-
-A data source connection specifies the parameters needed to connect to a data source. These parameters form a connection string for the data source. Using Dashboards, you can add new data source connections or manage existing ones.
-
-The following steps guide you through the basics of creating a data source connection:
-
-1. From the OpenSearch Dashboards main menu, select **Management** > **Data sources** > **Create data source connection**. The UI is shown in the following image.
+To securely store and encrypt data source connections in OpenSearch, you must add the following configuration to the `opensearch.yml` file on all the nodes:
- 
+`plugins.query.datasources.encryption.masterkey: "YOUR_GENERATED_MASTER_KEY_HERE"`
-2. Create the data source connection by entering the appropriate information into the **Connection Details** and **Authentication Method** fields.
-
- - Under **Connection Details**, enter a title and endpoint URL. For this tutorial, use the URL `http://localhost:5601/app/management/opensearch-dashboards/dataSources`. Entering a description is optional.
+The key must be 16, 24, or 32 characters. You can use the following command to generate a 24-character key:
- - Under **Authentication Method**, select an authentication method from the dropdown list. Once an authentication method is selected, the applicable fields for that method appear. You can then enter the required details. The authentication method options are:
- - **No authentication**: No authentication is used to connect to the data source.
- - **Username & Password**: A basic username and password are used to connect to the data source.
- - **AWS SigV4**: An AWS Signature Version 4 authenticating request is used to connect to the data source. AWS Signature Version 4 requires an access key and a secret key.
- - For AWS Signature Version 4 authentication, first specify the **Region**. Next, select the OpenSearch service in the **Service Name** list. The options are **Amazon OpenSearch Service** and **Amazon OpenSearch Serverless**. Lastly, enter the **Access Key** and **Secret Key** for authorization.
+`openssl rand -hex 12`
- After you have populated the required fields, the **Test connection** and **Create data source** buttons become active. You can select **Test connection** to confirm that the connection is valid.
+Generating 12 bytes results in a hexadecimal string that is 12 * 2 = 24 characters.
+{: .note}
-3. Select **Create data source** to save your settings. The connection is created. The active window returns to the **Data sources** main page, and the new connection appears in the list of data sources.
-
-4. To delete a data source connection, select the checkbox to the left of the data source **Title** and then select the **Delete 1 connection** button. Selecting multiple checkboxes for multiple connections is supported. An example UI is shown in the following image.
-
- 
-
-### Modify a data source connection
-
-To make changes to a data source connection, select a connection in the list on the **Data sources** main page. The **Connection Details** window opens.
-
-To make changes to **Connection Details**, edit one or both of the **Title** and **Description** fields and select **Save changes** in the lower-right corner of the screen. You can also cancel changes here. To change the **Authentication Method**, choose a different authentication method, enter your credentials (if applicable), and then select **Save changes** in the lower-right corner of the screen. The changes are saved.
-
-When **Username & Password** is the selected authentication method, you can update the password by choosing **Update stored password** next to the **Password** field. In the pop-up window, enter a new password in the first field and then enter it again in the second field to confirm. Select **Update stored password** in the pop-up window. The new password is saved. Select **Test connection** to confirm that the connection is valid.
+## Permissions
-When **AWS SigV4** is the selected authentication method, you can update the credentials by selecting **Update stored AWS credential**. In the pop-up window, enter a new access key in the first field and a new secret key in the second field. Select **Update stored AWS credential** in the pop-up window. The new credentials are saved. Select **Test connection** in the upper-right corner of the screen to confirm that the connection is valid.
+To work with data sources in OpenSearch Dashboards, you must be assigned the correct cluster-level [data source permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions#data-source-permissions).
-To delete the data source connection, select the delete icon ({::nomarkdown}{:/}).
+## Types of data streams
-## Create an index pattern
+To configure data sources through OpenSearch Dashboards, go to **Management** > **Dashboards Management** > **Data sources**. This flow can be used for OpenSearch data stream connections. See [Configuring and using multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/).
-Once you've created a data source connection, you can create an index pattern for the data source. An _index pattern_ is a template that OpenSearch uses to create indexes for data from the data source. See [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/) for more information and a tutorial.
+Alternatively, if you are running OpenSearch Dashboards 2.16 or later, go to **Management** > **Data sources**. This flow can be used to connect Amazon Simple Storage Service (Amazon S3) and Prometheus. See [Connecting Amazon S3 to OpenSearch]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/) and [Connecting Prometheus to OpenSearch]({{site.url}}{{site.baseurl}}/dashboards/management/connect-prometheus/) for more information.
## Next steps
- Learn about [managing index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/) through OpenSearch Dashboards.
- Learn about [indexing data using Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards.
- Learn about how to connect [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/).
-- Learn about how to [connect OpenSearch and Amazon S3 through OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/).
-- Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) tool, which gives you the flexibility to use various data ingestion methods and connect data from the Dashboards UI.
-
+- Learn about how to connect [OpenSearch and Amazon S3]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/) and [OpenSearch and Prometheus]({{site.url}}{{site.baseurl}}/dashboards/management/connect-prometheus/) using the OpenSearch Dashboards interface.
+- Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) plugin, which gives you the flexibility to use various data ingestion methods and connect data to OpenSearch Dashboards.
diff --git a/_field-types/index.md b/_field-types/index.md
index 7a7e816ada..e9250f409d 100644
--- a/_field-types/index.md
+++ b/_field-types/index.md
@@ -12,43 +12,77 @@ redirect_from:
# Mappings and field types
-You can define how documents and their fields are stored and indexed by creating a _mapping_. The mapping specifies the list of fields for a document. Every field in the document has a _field type_, which defines the type of data the field contains. For example, you may want to specify that the `year` field should be of type `date`. To learn more, see [Supported field types]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/).
+Mappings tell OpenSearch how to store and index your documents and their fields. You can specify the data type for each field (for example, `year` as `date`) to make storage and querying more efficient.
-If you're just starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings.
+While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps to maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations.
-For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers.
+For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer instead of both being interpreted as integers by dynamic mapping.
-This section provides an example for how to create an index mapping and how to add a document to it that will get ip_range validated.
-
-#### Table of contents
-1. TOC
-{:toc}
-
-
----
## Dynamic mapping
When you index a document, OpenSearch adds fields automatically with dynamic mapping. You can also explicitly add fields to an index mapping.
-#### Dynamic mapping types
+### Dynamic mapping types
Type | Description
:--- | :---
-null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values.
-boolean | OpenSearch accepts `true` and `false` as boolean values. An empty string is equal to `false.`
-float | A single-precision 32-bit floating point number.
-double | A double-precision 64-bit floating point number.
-integer | A signed 32-bit number.
-object | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`.
-array | Arrays in OpenSearch can only store values of one type, such as an array of just integers or strings. Empty arrays are treated as though they are fields with no values.
-text | A string sequence of characters that represent full-text values.
-keyword | A string sequence of structured characters, such as an email address or ZIP code.
+`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if the field has no value.
+`boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.`
+`float` | A single-precision, 32-bit floating-point number.
+`double` | A double-precision, 64-bit floating-point number.
+`integer` | A signed 32-bit number.
+`object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`.
+`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements---not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, which may contain complex objects or other arrays, can also be used for advanced data modeling.
+`text` | A string sequence of characters that represent full-text values.
+`keyword` | A string sequence of structured characters, such as an email address or ZIP code.
date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date.
numeric detection string | If disabled, OpenSearch may automatically process numeric values as strings when they should be processed as numbers. When enabled, OpenSearch can process strings into `long`, `integer`, `short`, `byte`, `double`, `float`, `half_float`, `scaled_float`, and `unsigned_long`. Default is disabled.
+### Dynamic templates
+
+Dynamic templates are used to define custom mappings for dynamically added fields based on the data type, field name, or field path. They allow you to define a flexible schema for your data that can automatically adapt to changes in the structure or format of the input data.
+
+You can use the following syntax to define a dynamic mapping template:
+
+```json
+PUT index
+{
+ "mappings": {
+ "dynamic_templates": [
+ {
+ "fields": {
+ "mapping": {
+ "type": "short"
+ },
+ "match_mapping_type": "string",
+ "path_match": "status*"
+ }
+ }
+ ]
+ }
+}
+```
+{% include copy-curl.html %}
+
+This mapping configuration dynamically maps any field with a name starting with `status` (for example, `status_code`) to the `short` data type if the initial value provided during indexing is a string.
+
+### Dynamic mapping parameters
+
+The `dynamic_templates` support the following parameters for matching conditions and mapping rules. The default value is `null`.
+
+Parameter | Description |
+----------|-------------|
+`match_mapping_type` | Specifies the JSON data type (for example, string, long, double, object, binary, Boolean, date) that triggers the mapping.
+`match` | A regular expression used to match field names and apply the mapping.
+`unmatch` | A regular expression used to exclude field names from the mapping.
+`match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`.
+`path_match` | Allows you to match nested field paths using a regular expression.
+`path_unmatch` | Excludes nested field paths from the mapping using a regular expression.
+`mapping` | The mapping configuration to apply.
+
## Explicit mapping
-If you know exactly what your field data types need to be, you can specify them in your request body when creating your index.
+If you know exactly which field data types you need to use, then you can specify them in your request body when creating your index, as shown in the following example request:
```json
PUT sample-index1
@@ -62,8 +96,9 @@ PUT sample-index1
}
}
```
+{% include copy-curl.html %}
-### Response
+#### Response
```json
{
"acknowledged": true,
@@ -71,8 +106,9 @@ PUT sample-index1
"index": "sample-index1"
}
```
+{% include copy-curl.html %}
-To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method:
+To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method, as shown in the following example request:
```json
POST sample-index1/_mapping
@@ -84,84 +120,29 @@ POST sample-index1/_mapping
}
}
```
+{% include copy-curl.html %}
You cannot change the mapping of an existing field, you can only modify the field's mapping parameters.
{: .note}
----
-## Mapping example usage
+## Mapping parameters
-The following example shows how to create a mapping to specify that OpenSearch should ignore any documents with malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You accomplish this by setting the `ignore_malformed` parameter to `true`.
+Mapping parameters are used to configure the behavior of index fields. See [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) for more information.
-### Create an index with an `ip` mapping
+## Mapping limit settings
-To create an index, use a PUT request:
+OpenSearch has certain mapping limits and settings, such as the settings listed in the following table. Settings can be configured based on your requirements.
-```json
-PUT /test-index
-{
- "mappings" : {
- "properties" : {
- "ip_address" : {
- "type" : "ip",
- "ignore_malformed": true
- }
- }
- }
-}
-```
-
-You can add a document that has a malformed IP address to your index:
-
-```json
-PUT /test-index/_doc/1
-{
- "ip_address" : "malformed ip address"
-}
-```
-
-This indexed IP address does not throw an error because `ignore_malformed` is set to true.
-
-You can query the index using the following request:
-
-```json
-GET /test-index/_search
-```
+| Setting | Default value | Allowed value | Type | Description |
+|-|-|-|-|-|
+| `index.mapping.nested_fields.limit` | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. |
+| `index.mapping.nested_objects.limit` | 10,000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created in a single document. |
+| `index.mapping.total_fields.limit` | 1,000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. |
+| `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. |
+| `index.mapping.field_name_length.limit` | 50,000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. |
+| `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be dynamically added to a mapping. |
-The response shows that the `ip_address` field is ignored in the indexed document:
-
-```json
-{
- "took": 14,
- "timed_out": false,
- "_shards": {
- "total": 1,
- "successful": 1,
- "skipped": 0,
- "failed": 0
- },
- "hits": {
- "total": {
- "value": 1,
- "relation": "eq"
- },
- "max_score": 1,
- "hits": [
- {
- "_index": "test-index",
- "_id": "1",
- "_score": 1,
- "_ignored": [
- "ip_address"
- ],
- "_source": {
- "ip_address": "malformed ip address"
- }
- }
- ]
- }
-}
-```
+---
## Get a mapping
@@ -170,14 +151,16 @@ To get all mappings for one or more indexes, use the following request:
```json
GET