Merge branch 'main' into cluster-apis

.github/vale/styles/OpenSearch/SubstitutionErroCaseSensitive.yml

@@ -0,0 +1,8 @@
+extends: substitution
+message: "Use '%s' instead of '%s'. Note the correct capitalization."
+ignorecase: false
+level: error
+action:
+  name: replace
+swap:
+  'Retrieval-Augmented Generation': retrieval-augmented generation

.github/vale/styles/OpenSearch/SubstitutionsError.yml

@@ -37,6 +37,7 @@ swap:
   'pre-trained': pretrained
   'premigration': pre-migration
   're-enable': reenable
+  'retrieval augmented generation': retrieval-augmented generation
   'screen shot': screenshot
   'sample request': example request
   'sample response': example response

.github/vale/tests/test-style-pos.md

@@ -76,6 +76,8 @@ This sentence tests splling.
 
 This sentence tests substitution error by using the word indices.
 
+This sentence tests substitution case-sensitive error by using the word Retrieval-Augmented Generation.
+
 This sentence tests substitution suggestion due to its nature.
 
 This Table | tests capitalization

.vale.ini

@@ -61,6 +61,7 @@ OpenSearch.SpacingSlash = YES
 OpenSearch.SpacingWords = YES
 OpenSearch.Spelling = YES
 OpenSearch.StackedHeadings = YES
+OpenSearch.SubstitutionsErrorCaseSensitive = YES
 OpenSearch.SubstitutionsError = YES
 OpenSearch.SubstitutionsSuggestion = YES
 OpenSearch.TableHeadings = YES

TERMS.md

@@ -654,6 +654,8 @@ Copy of a primary shard. Helps improve performance when using indexes across mul
 
 Use as a synonym for repository, on second and subsequent use.
 
+**retrieval-augmented generation (RAG)**
+
 **RPM Package Manager (RPM)**
 
 Formerly known as RedHat Package Manager. An open-source package management system for use with Linux distributions.

_data-prepper/pipelines/configuration/sources/kafka.md

@@ -55,7 +55,7 @@ Option | Required | Type | Description
 `auto_commit` | No | Boolean | When `false`, the consumer's offset will not be periodically committed to Kafka in the background. Default is `false`.
 `commit_interval` | No | Integer | When `auto_commit` is set to `true`, sets how frequently, in seconds, the consumer offsets are auto-committed to Kafka through Kafka's `auto.commit.interval.ms` option. Default is `5s`.
 `session_timeout` | No | Integer | The amount of time during which the source detects client failures when using Kafka's group management features, which can be used to balance the data stream. Default is `45s`.
-`auto_offset_reset` | No | String | Automatically resets the offset to an earlier or the latest offset through Kafka's `auto.offset.reset` option. Default is `latest`.
+`auto_offset_reset` | No | String | Automatically resets the offset to an earlier or the latest offset through Kafka's `auto.offset.reset` option. Default is `earliest`.
 `thread_waiting_time` | No | Integer | The amount of time that threads wait for the preceding thread to complete its task and to signal the next thread. The Kafka consumer API poll timeout value is set to half of this setting. Default is `5s`.
 `max_partition_fetch_bytes` | No | Integer | Sets the maximum limit in megabytes for max data returns from each partition through Kafka's `max.partition.fetch.bytes` setting. Default is `1mb`.
 `heart_beat_interval` | No | Integer | The expected amount of time between heartbeats to the consumer coordinator when using Kafka's group management facilities through Kafka's `heartbeat.interval.ms` setting. Default is `5s`.

_data-prepper/pipelines/configuration/sources/pipeline.md

@@ -12,7 +12,7 @@ Use the `pipeline` sink to read from another pipeline.
 
 ## Configuration options
 
-The `pipeline` sink supports the following configuration options.
+The `pipeline` source supports the following configuration options.
 
 | Option | Required | Type   | Description                            |
 |:-------|:---------|:-------|:---------------------------------------|

_ml-commons-plugin/remote-models/connectors.md

@@ -289,7 +289,11 @@ POST /_plugins/_ml/connectors/_create
 
 ## Updating connector credentials
 
-In some cases, you may need to update credentials, like `access_key`, that you use to connect to externally hosted models. You can update credentials without undeploying the model by providing the new credentials in the following request:
+In some cases, you may need to update credentials, such as `access_key`, used to connect to externally hosted models. To do this without undeploying the model, provide the new credentials in an update request.
+
+### Connector for a specific model
+
+To update credentials for a connector linked to a specific model, provide the new credentials in the following request:
 
 ```json
 PUT /_plugins/_ml/models/<model_id>
@@ -301,6 +305,21 @@ PUT /_plugins/_ml/models/<model_id>
   }
 }
 ```
+{% include copy-curl.html %}
+
+### Standalone connector
+
+To update credentials for a standalone connector, provide the new credentials in the following request:
+
+```json
+PUT /_plugins/_ml/connectors/<connector_id>
+{
+  "credential": {
+    "openAI_key": "YOUR NEW OPENAI KEY"
+  }
+}
+```
+{% include copy-curl.html %}
 
 ## Next steps
 

_query-dsl/specialized/template.md

@@ -0,0 +1,101 @@
+---
+layout: default
+title: Template
+parent: Specialized queries
+nav_order: 70
+---
+
+# Template query
+Introduced 2.19
+{: .label .label-purple }
+
+Use a `template` query to create search queries that contain placeholder variables. Placeholders are specified using the `"${variable_name}"` syntax (note that the variables must be enclosed in quotation marks). When you submit a search request, these placeholders remain unresolved until they are processed by search request processors. This approach is particularly useful when your initial search request contains data that needs to be transformed or generated at runtime.
+
+For example, you might use a template query when working with the [ml_inference search request processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/ml-inference-search-request/), which converts text input into vector embeddings during the search process. The processor will replace the placeholders with the generated values before the final query is executed.
+
+For a complete example, see [Query rewriting using template queries]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/template-query/).
+
+## Example
+
+The following example shows a template k-NN query with a `"vector": "${text_embedding}"` placeholder. The placeholder `"${text_embedding}"` will be replaced with embeddings generated by the `ml_inference` search request processor from the `text` input field:
+
+```json
+GET /template-knn-index/_search?search_pipeline=my_knn_pipeline
+{
+  "query": {
+    "template": {
+      "knn": {
+        "text_embedding": {
+          "vector": "${text_embedding}", // Placeholder for the vector field
+          "k": 2
+        }
+      }
+    }
+  },
+  "ext": {
+    "ml_inference": {
+      "text": "sneakers" // Input text for the ml_inference processor
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+To use a template query with a search request processor, you need to configure a search pipeline. The following is an example configuration for the `ml_inference` search request processor. The `input_map` maps document fields to model inputs. In this example, the `ext.ml_inference.text` source field in a document is mapped to the `inputText` field---the expected input field for the model. The `output_map` maps model outputs to document fields. In this example, the `embedding` output field from the model is mapped to the `text_embedding` destination field in your document:
+
+```json
+PUT /_search/pipeline/my_knn_pipeline
+{
+  "request_processors": [
+    {
+      "ml_inference": {
+        "model_id": "Sz-wFZQBUpPSu0bsJTBG",
+        "input_map": [
+          {
+            "inputText": "ext.ml_inference.text" // Map input text from the request
+          }
+        ],
+        "output_map": [
+          {
+            "text_embedding": "embedding" // Map output to the placeholder
+          }
+        ]
+      }
+    }
+  ]
+}
+```
+{% include copy-curl.html %}
+
+After the `ml_inference` search request processor runs, the search request is rewritten. The `vector` field contains the embeddings generated by the processor, and the `text_embedding` field contains the processor output:
+
+```json
+GET /template-knn-1/_search
+{
+  "query": {
+    "template": {
+      "knn": {
+        "text_embedding": {
+          "vector": [0.6328125, 0.26953125, ...], 
+          "k": 2
+        }
+      }
+    }
+  },
+  "ext": {
+    "ml_inference": {
+      "text": "sneakers",
+      "text_embedding": [0.6328125, 0.26953125, ...] 
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+## Limitations
+
+Template queries require at least one search request processor in order to resolve placeholders. Search request processors must be configured to produce the variables expected in the pipeline.
+
+## Next steps
+
+- For a complete example, see [Template queries]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/template-query/).

_search-plugins/knn/knn-index.md

@@ -49,6 +49,8 @@ Starting with k-NN plugin version 2.17, you can use `byte` vectors with the `fai
 
 Starting with k-NN plugin version 2.16, you can use `binary` vectors with the `faiss` engine to reduce the amount of required storage space. For more information, see [Binary vectors]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-vector#binary-vectors).
 
+Starting with k-NN plugin version 2.19, you can use `binary` vectors with the `lucene` engine.
+
 ## SIMD optimization for the Faiss engine
 
 Starting with version 2.13, the k-NN plugin supports [Single Instruction Multiple Data (SIMD)](https://en.wikipedia.org/wiki/Single_instruction,_multiple_data) processing if the underlying hardware supports SIMD instructions (AVX2 on x64 architecture and Neon on ARM64 architecture). SIMD is supported by default on Linux machines only for the Faiss engine. SIMD architecture helps boost overall performance by improving indexing throughput and reducing search latency. Starting with version 2.18, the k-NN plugin supports AVX512 SIMD instructions on x64 architecture. 
@@ -155,9 +157,12 @@ The IVF algorithm requires a training step. To create an index that uses IVF, yo
 
 ### Supported Lucene methods
 
-Method name | Requires training | Supported spaces | Description
+Method name | Requires training | Supported spaces                                                                | Description
 :--- | :--- |:--------------------------------------------------------------------------------| :---
-`hnsw` | false | l2, cosinesimil, innerproduct (supported in OpenSearch 2.13 and later) | Hierarchical proximity graph approach to approximate k-NN search.
+`hnsw` | false | l2, cosinesimil, innerproduct (supported in OpenSearch 2.13 and later), hamming | Hierarchical proximity graph approach to approximate k-NN search.
+
+The `hamming` space type is supported for binary vectors in OpenSearch version 2.19 and later. For more information, see [Binary k-NN vectors]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-vector#binary-vectors).
+{: .note}
 
 #### HNSW parameters
 

_search-plugins/querqy/index.md

@@ -1,7 +1,8 @@
 ---
 layout: default
 title: Querqy
-parent: Search relevance
+parent: Query rewriting
+grand_parent: Search relevance
 has_children: false
 redirect_from:
   - /search-plugins/querqy/

_search-plugins/search-relevance/index.md

@@ -18,4 +18,4 @@ OpenSearch provides the following search relevance features:
 
 - [Reranking search results]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/reranking-search-results/) using a cross-encoder reranker. 
 
-- Rewriting queries using [Querqy]({{site.url}}{{site.baseurl}}/search-plugins/querqy/).
+- [Query rewriting]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/query-rewriting/).

_search-plugins/search-relevance/query-rewriting.md

@@ -0,0 +1,18 @@
+---
+layout: default
+title: Query rewriting
+parent: Search relevance
+has_children: true
+nav_order: 70
+has_toc: false
+---
+
+# Query rewriting
+
+Query rewriting is the process of transforming or modifying a user query before it is executed. The goal of query rewriting is to improve search accuracy, relevance, or performance by addressing issues such as misspellings, synonyms, ambiguous terms, or inefficient query structure. Query rewriting is commonly used in search systems to enhance the quality of search results.
+
+You can perform query rewriting in OpenSearch using the following features:
+
+- [Template queries]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/template-query/) 
+
+- [Querqy]({{site.url}}{{site.baseurl}}/search-plugins/querqy/)