Merge branch 'main' into 7248-scripts-in-update-bulk-operations-docum…

…entation

.github/PULL_REQUEST_TEMPLATE.md

@@ -7,6 +7,9 @@ _List any issues this PR will resolve, e.g. Closes [...]._
 ### Version
 _List the OpenSearch version to which this PR applies, e.g. 2.14, 2.12--2.14, or all._
 
+### Frontend features
+_If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional._ 
+
 ### Checklist
 - [ ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the [Developers Certificate of Origin](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).
 For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

_about/version-history.md

@@ -9,6 +9,7 @@ permalink: /version-history/
 
 OpenSearch version | Release highlights | Release date  
 :--- | :--- | :--- 
+[2.15.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.15.0.md) | Includes parallel ingestion processing, SIMD support for exact search, and the ability to disable doc values for the k-NN field. Adds wildcard and derived field types. Improves performance for single-cardinality aggregations, rolling upgrades to remote-backed clusters, and more metrics for top N queries. For a full list of release highlights, see the Release Notes. | 25 June 2024
 [2.14.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.14.0.md) | Includes performance improvements to hybrid search and date histogram queries with multi-range traversal, ML model integration within the Ingest API, semantic cache for LangChain applications, low-level vector query interface for neural sparse queries, and improved k-NN search filtering. Provides an experimental tiered cache feature. For a full list of release highlights, see the Release Notes. | 14 May 2024
 [2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md) | Makes agents and tools and the OpenSearch Assistant Toolkit generally available. Introduces vector quantization within OpenSearch. Adds LLM guardrails and hybrid search with aggregations. Adds the Bloom filter skipping index for Apache Spark data sources, I/O-based admission control, and the ability to add an alerting cluster that manages all alerting tasks. For a full list of release highlights, see the Release Notes. | 2 April 2024
 [2.12.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.12.0.md) | Makes concurrent segment search and conversational search generally available. Provides an experimental OpenSearch Assistant Toolkit, including agents and tools, workflow automation, and OpenSearch Assistant for OpenSearch Dashboards UI. Adds a new match-only text field, query insights to monitor top N queries, and k-NN search on nested fields. For a full list of release highlights, see the Release Notes. | 20 February 2024

_config.yml

@@ -5,9 +5,9 @@ baseurl: "/docs/latest" # the subpath of your site, e.g. /blog
 url: "https://opensearch.org" # the base hostname & protocol for your site, e.g. http://example.com
 permalink: /:path/
 
-opensearch_version: '2.14.0'
-opensearch_dashboards_version: '2.14.0'
-opensearch_major_minor_version: '2.14'
+opensearch_version: '2.15.0'
+opensearch_dashboards_version: '2.15.0'
+opensearch_major_minor_version: '2.15'
 lucene_version: '9_10_0'
 
 # Build settings

_data-prepper/pipelines/configuration/sources/http-source.md → _data-prepper/pipelines/configuration/sources/http.md

@@ -1,14 +1,16 @@
 ---
 layout: default
-title: http_source
+title: http
 parent: Sources
 grand_parent: Pipelines
 nav_order: 5
+redirect_from:
+  - /data-prepper/pipelines/configuration/sources/http-source/
 ---
 
-# http_source
+# http
 
-`http_source` is a source plugin that supports HTTP. Currently, `http_source` only supports the JSON UTF-8 codec for incoming requests, such as `[{"key1": "value1"}, {"key2": "value2"}]`. The following table describes options you can use to configure the `http_source` source.
+The `http` plugin accepts HTTP requests from clients. Currently, `http` only supports the JSON UTF-8 codec for incoming requests, such as `[{"key1": "value1"}, {"key2": "value2"}]`. The following table describes options you can use to configure the `http` source.
 
 Option | Required | Type | Description
 :--- | :--- | :--- | :---
@@ -19,6 +21,7 @@ request_timeout | No | Integer | The request timeout, in milliseconds. Default v
 thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`.
 max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`.
 max_pending_requests | No | Integer | The maximum allowed number of tasks in the `ScheduledThreadPool` work queue. Default value is `1024`.
+max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single HTTP request. Default value is `10mb`.
 authentication | No | Object | An authentication configuration. By default, this creates an unauthenticated server for the pipeline. This uses pluggable authentication for HTTPS. To use basic authentication define the `http_basic` plugin with a `username` and `password`. To provide customer authentication, use or create a plugin that implements [ArmeriaHttpAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/1.2.0/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/ArmeriaHttpAuthenticationProvider.java).
 ssl | No | Boolean | Enables TLS/SSL. Default value is false.
 ssl_certificate_file | Conditionally | String | SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. Amazon S3 path example `s3://<bucketName>/<path>`. Required if `ssl` is set to true and `use_acm_certificate_for_ssl` is set to false.
@@ -35,7 +38,7 @@ Content will be added to this section.--->
 
 ## Metrics
 
-The `http_source` source includes the following metrics.
+The `http` source includes the following metrics.
 
 ### Counters
 

_data-prepper/pipelines/configuration/sources/otel-logs-source.md

@@ -21,7 +21,8 @@ You can configure the `otel_logs_source` source with the following options.
 | Option | Type | Description |
 | :--- | :--- | :--- |
 | port | int | Represents the port that the `otel_logs_source` source is running on. Default value is `21892`. |
-| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support an unframed gRPC request with an HTTP idiomatic path to a configurable path. The path should start with `/`, and its length should be at least 1. The `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain a `${pipelineName}` placeholder, which is replaced with the pipeline name. If the value is empty and `unframed_requests` is `true`, then the path that the source provides is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | 
+| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support an unframed gRPC request with an HTTP idiomatic path to a configurable path. The path should start with `/`, and its length should be at least 1. The `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain a `${pipelineName}` placeholder, which is replaced with the pipeline name. If the value is empty and `unframed_requests` is `true`, then the source provides the path `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. |
+| max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`.
 | request_timeout | int | Represents the request timeout duration in milliseconds. Default value is `10000`. |
 | health_check_service | Boolean | Enables the gRPC health check service under `grpc.health.v1/Health/Check`. Default value is `false`. |
 | proto_reflection_service | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default value is `false`. |

_data-prepper/pipelines/configuration/sources/otel-metrics-source.md

@@ -19,6 +19,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto
 unframed_requests | No | Boolean | Enables requests not framed using the gRPC wire protocol.
 thread_count | No | Integer | The number of threads to keep in the `ScheduledThreadPool`. Default value is `200`.
 max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`.
+max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`.
 ssl | No | Boolean | Enables connections to the OpenTelemetry source port over TLS/SSL. Default value is `true`.
 sslKeyCertChainFile | Conditionally | String | File-system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`.
 sslKeyFile | Conditionally | String | File-system path or Amazon S3 path to the security key (for example, `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if `ssl` is set to `true`.

_data-prepper/pipelines/configuration/sources/otel-trace-source.md

@@ -24,6 +24,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto
 unframed_requests | No | Boolean | Enable requests not framed using the gRPC wire protocol.
 thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`.
 max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`.
+max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`.
 ssl | No | Boolean | Enables connections to the OTel source port over TLS/SSL. Defaults to `true`.
 sslKeyCertChainFile | Conditionally | String | File system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`.
 sslKeyFile | Conditionally | String | File system path or Amazon S3 path to the security key (for example, `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if `ssl` is set to `true`.

_data/versions.json

@@ -1,10 +1,11 @@
 {
-  "current": "2.14",
+  "current": "2.15",
   "all": [
-    "2.14",
+    "2.15",
     "1.3"
   ],
   "archived": [
+    "2.14",
     "2.13",
     "2.12",
     "2.11",
@@ -23,7 +24,7 @@
     "1.1",
     "1.0"
     ],
-  "latest": "2.14"
+  "latest": "2.15"
 }
 
 

_layouts/default.html

@@ -165,9 +165,9 @@
         <div id="main-content" class="main-content" role="main">
           {% if page.section == "opensearch" %}
             {% if site.doc_version == "supported" %}            
-              <p class="supported-version-warning">This is an earlier version of the OpenSearch documentation. For the latest version, see the <a href="{{ site.url }}/docs{{ page.url }}">current documentation</a>. For information about OpenSearch version maintenance, see <a href="https://opensearch.org/releases.html">Release Schedule and Maintenance Policy</a>.</p>
+              <p class="supported-version-warning">You're viewing version {{site.opensearch_major_minor_version}} of the OpenSearch documentation. For the latest version, see the <a href="{{ site.url }}/docs{{ page.url }}">current documentation</a>. For information about OpenSearch version maintenance, see <a href="https://opensearch.org/releases.html">Release Schedule and Maintenance Policy</a>.</p>
             {% elsif site.doc_version == "unsupported" %}
-              <p class="unsupported-version-warning">This version of the OpenSearch documentation is no longer maintained. For the latest version, see the <a href="{{ site.url }}/docs{{ page.url }}">current documentation</a>. For information about OpenSearch version maintenance, see <a href="https://opensearch.org/releases.html">Release Schedule and Maintenance Policy</a>.</p>
+              <p class="unsupported-version-warning">You're viewing version {{site.opensearch_major_minor_version}} of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the <a href="{{ site.url }}/docs{{ page.url }}">current documentation</a>. For information about OpenSearch version maintenance, see <a href="https://opensearch.org/releases.html">Release Schedule and Maintenance Policy</a>.</p>
             {% endif %}
           {% endif %}
           {% if site.heading_anchors != false %}

_observing-your-data/metricsanalytics.md → _observing-your-data/prometheusmetrics.md

@@ -2,8 +2,6 @@
 layout: default
 title: Metric analytics
 nav_order: 40
-redirect_from:
-  - /observing-your-data/metricsanalytics/
 ---
 
 # Metric analytics

_security/authentication-backends/jwt.md

@@ -235,7 +235,7 @@ openid_auth_domain:
   transport_enabled: true
   order: 0
   http_authenticator:
-    type: openid
+    type: openid # use the OpenID Connect domain, since JWT is part of this authentication.
     challenge: false
     config:
       subject_key: preferred_username

_security/configuration/configuration.md

@@ -11,7 +11,7 @@ redirect_from:
 
 One of the first steps when setting up the Security plugin is deciding which authentication backend to use. The role played by the backend in authentication is covered in [steps 2 and 3 of the authentication flow]({{site.url}}{{site.baseurl}}/security/authentication-backends/authc-index/#authentication-flow). The plugin has an internal user database, but many people prefer to use an existing authentication backend, such as an LDAP server, or some combination of the two.
 
-The primary file used to configure an authentication and authorization backend is `config/opensearch-security/config.yml`. This file defines how the Security plugin retrieves user credentials, how it verifies the credentials, and how it fetches additional roles when the backend selected for authentication and authorization supports this feature. This topic provides a basic overview of the configuration file and its requirements for setting up security. For information about configuring a specific backend, see [Authentication backends]({{site.url}}{{site.baseurl}}/security/authentication-backends/authc-index/).
+The primary file used to configure the authentication and authorization backend is `/usr/share/opensearch/config/opensearch-security/config.yml`. This file defines how the Security plugin retrieves user credentials, how the plugin verifies the credentials, and how the plugin fetches additional roles when the backend selected for authentication and authorization supports this feature. This topic provides a basic overview of the configuration file and its requirements for setting up security. For information about configuring a specific backend, see [Authentication backends]({{site.url}}{{site.baseurl}}/security/authentication-backends/authc-index/).
 
 The `config.yml` file includes three main parts:
 

release-notes/opensearch-documentation-release-notes-2.15.0.md

@@ -0,0 +1,42 @@
+# OpenSearch Documentation Website 2.15.0 Release Notes
+
+The OpenSearch 2.15.0 documentation includes the following additions and updates.
+
+## New documentation for 2.15.0
+
+- Alerts in correlations feature documentation [#7410](https://github.com/opensearch-project/documentation-website/pull/7410)
+- Add documentations for batch ingestion feature [#7408](https://github.com/opensearch-project/documentation-website/pull/7408)
+-  Changed VisBuilder status from experimental to GA [#7405](https://github.com/opensearch-project/documentation-website/pull/7405)
+- Add documentation for innerHit on knn nested field [#7404](https://github.com/opensearch-project/documentation-website/pull/7404)
+- AD Enhancements in Version 2.15 [#7388](https://github.com/opensearch-project/documentation-website/pull/7388)
+- Add connector tool [#7384](https://github.com/opensearch-project/documentation-website/pull/7384)
+- Add remote guardrails model support [#7377](https://github.com/opensearch-project/documentation-website/pull/7377)
+- Update documentation of ml inference processors to support for local models [#7368](https://github.com/opensearch-project/documentation-website/pull/7368)
+- Trace analytics update [#7362](https://github.com/opensearch-project/documentation-website/pull/7362)
+- Add doc for alerting comments  [#7360](https://github.com/opensearch-project/documentation-website/pull/7360)
+- Add documentation related to removal of source and recovery source in k-NN performance tuning section [#7359](https://github.com/opensearch-project/documentation-website/pull/7359)
+- Added documentation for new default workflow templates [#7346](https://github.com/opensearch-project/documentation-website/pull/7346)
+- Mark docrep to remote migration as GA and modify settings names [#7342](https://github.com/opensearch-project/documentation-website/pull/7342)
+- Add documentation for the new setting of cardinality aggregation dynamic pruning [#7341](https://github.com/opensearch-project/documentation-website/pull/7341)
+- Add documentation for wildcard field type [#7339](https://github.com/opensearch-project/documentation-website/pull/7339)
+- Update document for handle SageMaker throttling [#7331](https://github.com/opensearch-project/documentation-website/pull/7331)
+- Add documentation related to new settings for segment upload timeout [#7330](https://github.com/opensearch-project/documentation-website/pull/7330)
+- Add documentation of derived fields [#7329](https://github.com/opensearch-project/documentation-website/pull/7329)
+- [MDS] Add security analytics, alerting, feature anaywhere in the multiple data source document [#7328](https://github.com/opensearch-project/documentation-website/pull/7328)
+- Add document for top n queries improvements in 2.15 [#7326](https://github.com/opensearch-project/documentation-website/pull/7326)
+- Update the integration page to reflect new integration catalog features [#7324](https://github.com/opensearch-project/documentation-website/pull/7324)
+- Add doc for neural-sparse-query-two-phase-processor. [#7306](https://github.com/opensearch-project/documentation-website/pull/7306)
+- Add documentation for Indices Request Cache Overview and its settings [#7288](https://github.com/opensearch-project/documentation-website/pull/7288)
+- Added documentation for Reindex workflow step [#7271](https://github.com/opensearch-project/documentation-website/pull/7271)
+- Document optional clear_status query parameter for Delete Workflow API [#7268](https://github.com/opensearch-project/documentation-website/pull/7268)
+- Update field-masking.md. Configure default masking algorithm. [#7162](https://github.com/opensearch-project/documentation-website/pull/7162)
+- add documentation for use compound file setting [#7092](https://github.com/opensearch-project/documentation-website/pull/7092)
+- Added documentation for managed identity support in repository-azure plugin [#7068](https://github.com/opensearch-project/documentation-website/pull/7068)
+
+## In progress documentation for 2.15.0
+
+- Initial UBI documentation  [#7284](https://github.com/opensearch-project/documentation-website/pull/7284)
+
+## Documentation for 2.15.0 experimental features
+
+- Add remote state publication [#7364](https://github.com/opensearch-project/documentation-website/pull/7364)