Add metadata fields for mappings (content gap initiative)

_field-types/index.md

@@ -14,23 +14,17 @@
 
 You can define how documents and their fields are stored and indexed by creating a _mapping_. The mapping specifies the list of fields for a document. Every field in the document has a _field type_, which defines the type of data the field contains. For example, you may want to specify that the `year` field should be of type `date`. To learn more, see [Supported field types]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/).
 
-If you're just starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings.
+If you're starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings.
 
 For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers.
 
-This section provides an example for how to create an index mapping and how to add a document to it that will get ip_range validated.
+This documentation provides an example for how to create an index mapping and how to add a document to it that will get `ip_range` validated.
 
-#### Table of contents
-1. TOC
-{:toc}
-
-
----
 ## Dynamic mapping
 
 When you index a document, OpenSearch adds fields automatically with dynamic mapping. You can also explicitly add fields to an index mapping.
 
-#### Dynamic mapping types
+### Dynamic mapping types
 
 Type | Description
 :--- | :---
@@ -63,7 +57,7 @@
 }
 ```
 
-### Response
+#### Response
 ```json
 {
     "acknowledged": true,
@@ -88,6 +82,42 @@
 You cannot change the mapping of an existing field, you can only modify the field's mapping parameters. 
 {: .note}
 
+## Mapping parameters
+
+Mapping parameters are used to configure the behavior of fields in an index. The following table lists commonly used mapping parameters.
+
+Parameter | Description
+:--- | :---
+`analyzer` | Specifies the analyzer used to analyze string fields.
+`boost` | Specifies a field-level query time to boost.
+`coerce` | Tries to convert the value to the specified data type.
+`copy_to` | Copies the values of this field to another field.
+`doc_values` | Specifies whether the field should be stored on disk to make sorting and aggregation faster.
+`dynamic` | Determines whether new fields should be added dynamically.
+`enabled` | Specifies whether the field is enabled or disabled. 
+`format` | Specifies the date format for date fields. 
+`ignore_above` | Skips indexing values that are longer than the specified length.
+`ignore_malformed` | Specifies whether malformed values should be ignored. 
+`index` | Specifies whether the field should be indexed.
+`index_options` | Specifies what information should be stored in the index for scoring purposes.
+
+## Mapping limit settings
+
+OpenSearch has certain limits or settings related to mappings, such as the settings listed in the following table. Settings can be configured based on your requirements. 
+
+| Setting | Default value | Allowed value | Type | Description |
+|-|-|-|-|-|
+| index.mapping.nested_fields.limit | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. |
+| index.mapping.nested_objects.limit | 10000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created within a single document. |
+| index.mapping.total_fields.limit | 1000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. |
+| index.mapping.depth.limit | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. |
+| index.mapping.field_name_length.limit | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. |
+| index.mapper.dynamic | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. |
+
+## Runtime fields
+
+You can define fields at query time, rather than at index time, by using runtime fields. this can be useful for creating fields based on the values of other fields, or for performing transformations on data during the query process. Runtime fields are defined in the query itself and do not affect the underlying data in the index.
+
 ---
 ## Mapping example usage
 
@@ -171,7 +201,7 @@
 GET <index>/_mapping
 ```
 
-In the above request, `<index>` may be an index name or a comma-separated list of index names. 
+In the previous request, `<index>` may be an index name or a comma-separated list of index names. 
 
 To get all mappings for all indexes, use the following request:
 
@@ -220,3 +250,27 @@
   }
 }
 ```
+
+## Delete a mapping
+
+The syntax for deleting a mapping depends on whether you want to delete the entire mapping for an index or the mapping for a specific field. The syntax for deleting a mapping is as follows:
+
+```json
+DELETE /<index_name>/_mapping
+DELETE /<field_name/_mapping>
+```
+
+For example, to delete the entire mapping for the `sample-index1` index, you can use the following commands:
+
+```json
+<insert command>
+```
+
+If you want to delete the mapping for a specific field, you can <insert instructional text> For example, to delete the mapping for the `year` field, use the following command:
+
+```json
+<insert command>
+```
+
+Deleting a field mapping will remove the mapping definition for that field across all indexes or the specified index. It will not delete the actual data stored in those fields.
+{: .note}

_field-types/metadata-fields/field-names.md

@@ -0,0 +1,18 @@
+---
+layout: default
+title: Field names
+nav_order: 10
+has_children: false
+parent: Metadata fields
+---
+
+# Field names
+
+The `field_names` field indexes the names of fields within a document that contain non-null values. This field support the `exists` query, which identifies documents with or without non-null values for a specified field. 
+
+The `field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `field_names`.
+
+## Mapping example
+
+<SME: Please provide a mapping example.>
+

_field-types/metadata-fields/id.md

@@ -0,0 +1,84 @@
+---
+layout: default
+title: ID
+nav_order: 20
+has_children: false
+parent: Metadata fields
+---
+
+# ID
+
+Each document has an `_id` field that uniquely identifies it. This field is indexed, allowing documents to be retrieved either through the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/).
+
+The following examples creates an index `test-index1` and add two documents with different `_id` values:
+
+```json
+PUT test-index1/_doc/1
+{
+  "text": "Document with ID 1"
+}
+
+PUT test-index1/_doc/2?refresh=true
+{
+  "text": "Document with ID 2"
+}
+```
+{% include copy-curl.html %}
+
+Now, you can query the documents using the `_id` field:
+
+```json
+GET test-index1/_search
+{
+  "query": {
+    "terms": {
+      "_id": ["1", "2"]
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The following response shows that this query returns both documents with `_id` values of `1` and `2`.
+
+```json
+{
+  "took": 10,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 2,
+      "relation": "eq"
+    },
+    "max_score": 1,
+    "hits": [
+      {
+        "_index": "test-index1",
+        "_id": "1",
+        "_score": 1,
+        "_source": {
+          "text": "Document with ID 1"
+        }
+      },
+      {
+        "_index": "test-index1",
+        "_id": "2",
+        "_score": 1,
+        "_source": {
+          "text": "Document with ID 2"
+        }
+      }
+    ]
+  }
+  ```
+  {% include copy-curl.html %}
+
+## Querying on the `_id` field
+
+While the `_id` field is accessible in various queries, it is restricted from use in aggregations, sorting, and scripting. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the content of the `_id` field into another field that has `doc_values` enabled.

_field-types/metadata-fields/ignored.md

@@ -0,0 +1,69 @@
+---
+layout: default
+title: Ignored
+nav_order: 15
+has_children: false
+parent: Metadata fields
+---
+
+# Ignored
+
+The `_ignored` field indexes and stores the name of fields within a document that were ignored during the indexing process due to being malformed. This functionality is enabled with the `ignore_malformed` setting is turned on in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/#mapping-example-usage). 
+
+The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understadning issues related to malformed data in your documents. 
+
+You can query the `_ignored` field using `term`, `terms`, and `exists` queries, and the results will be in the search hits.
+
+The `_ignored` field is only populated when the `ignore_malformed` setting is enabled in your index mapping. If `ignore_malformed` is set to `false` (the default value), malformed fields will cause the entire document to be rejected, and the `_ignored` field will not be populated.
+{: .note}
+
+For example, the following query will retrieve all documents that have at least one field that was ignored during indexing:
+
+```json
+GET _search
+{
+  "query": {
+    "exists": {
+      "field": "_ignored"
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+Similarly, you can use a term query to find documents where a specific field, such as created_at, was ignored:
+
+```json
+GET _search
+{
+  "query": {
+    "term": {
+      "_ignored": "created_at"
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+#### Reponse 
+
+```json
+{
+  "took": 51,
+  "timed_out": false,
+  "_shards": {
+    "total": 45,
+    "successful": 45,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 0,
+      "relation": "eq"
+    },
+    "max_score": null,
+    "hits": []
+  }
+}
+```

_field-types/metadata-fields/index-metadata.md

@@ -0,0 +1,74 @@
+---
+layout: default
+title: Index
+nav_order: 25
+has_children: false
+parent: Metadata fields
+---
+
+# Index
+
+When querying across multiple indexes, you may need to filter results based on the index a document was indexed into. The `index` field matches documents based on their index. 
+
+The following example creates two indexes, `products` and `customers` and adds a document to each index:
+
+```json
+PUT products/_doc/1
+{
+  "name": "Widget X"
+}
+
+PUT customers/_doc/2?refresh=true
+{
+  "name": "John Doe"
+}
+```
+{% include copy-curl.html %}
+
+Now, you can query both indexes and filter the results using the `_index` field:
+
+```json
+GET products,customers/_search
+{
+  "query": {
+    "terms": {
+      "_index": ["products", "customers"]
+    }
+  },
+  "aggs": {
+    "index_groups": {
+      "terms": {
+        "field": "_index",
+        "size": 10
+      }
+    }
+  },
+  "sort": [
+    {
+      "_index": {
+        "order": "desc"
+      }
+    }
+  ],
+  "script_fields": {
+    "index_name": {
+      "script": {
+        "lang": "painless",
+        "source": "doc['_index'].value"
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+In this example:
+
+- The `query` section uses a `terms` query to match documents from the `products` and `customers` indexes.
+- The `aggs` section performs a `terms` aggregation on the `_index` field, grouping the results by index.
+- The `sort` section sorts the results by the `_index` field in ascending order.
+- The `script_fields` section adds a new field `index_name` to the search results that contains the value of the `_index` field for each document.
+
+## Querying on the `_index` field
+
+<SME: Please provide information necessary for users to understand how this works for them.>

_field-types/metadata-fields/index.md

@@ -0,0 +1,21 @@
+--
+layout: default
+title: Metadata fields
+nav_order: 90
+has_children: true
+has_toc: false
+---
+
+# Metadata fields
+
+OpenSearch has built-in metadata fields that provide information about the documents in an index. These fields can be accessed or used in queries as needed.
+
+Metadata fields | Description
+:--- | :---
+`field_names` | The fields within the document that hold non-empty or non-null values.   
+`_ignored` | The fields in the document that were disregarded during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting.
+`_id` |  The unique identifier assigned to each individual document. 
+`_index` | The specific index within the OpenSearch database where the document is stored and organized.
+`_meta` | Stores custom metadata or additional information specific to the application or use case.
+`_routing` | Allows you to specify a custom value that determines the shard assignment for the document within the OpenSearch cluster.
+`_source` | Contains the original JSON representation of the document's data.

_field-types/metadata-fields/meta.md

@@ -0,0 +1,8 @@
+---
+layout: default
+title: Meta
+nav_order: 30
+has_children: false
+parent: Metadata fields
+---
+