diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index bbf3b8d035..21b6fbfea6 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -7,6 +7,9 @@ _List any issues this PR will resolve, e.g. Closes [...]._ ### Version _List the OpenSearch version to which this PR applies, e.g. 2.14, 2.12--2.14, or all._ +### Frontend features +_If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional._ + ### Checklist - [ ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the [Developers Certificate of Origin](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). diff --git a/_about/version-history.md b/_about/version-history.md index 6a0938541a..0d6d844951 100644 --- a/_about/version-history.md +++ b/_about/version-history.md @@ -9,6 +9,7 @@ permalink: /version-history/ OpenSearch version | Release highlights | Release date :--- | :--- | :--- +[2.15.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.15.0.md) | Includes parallel ingestion processing, SIMD support for exact search, and the ability to disable doc values for the k-NN field. Adds wildcard and derived field types. Improves performance for single-cardinality aggregations, rolling upgrades to remote-backed clusters, and more metrics for top N queries. For a full list of release highlights, see the Release Notes. | 25 June 2024 [2.14.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.14.0.md) | Includes performance improvements to hybrid search and date histogram queries with multi-range traversal, ML model integration within the Ingest API, semantic cache for LangChain applications, low-level vector query interface for neural sparse queries, and improved k-NN search filtering. Provides an experimental tiered cache feature. For a full list of release highlights, see the Release Notes. | 14 May 2024 [2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md) | Makes agents and tools and the OpenSearch Assistant Toolkit generally available. Introduces vector quantization within OpenSearch. Adds LLM guardrails and hybrid search with aggregations. Adds the Bloom filter skipping index for Apache Spark data sources, I/O-based admission control, and the ability to add an alerting cluster that manages all alerting tasks. For a full list of release highlights, see the Release Notes. | 2 April 2024 [2.12.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.12.0.md) | Makes concurrent segment search and conversational search generally available. Provides an experimental OpenSearch Assistant Toolkit, including agents and tools, workflow automation, and OpenSearch Assistant for OpenSearch Dashboards UI. Adds a new match-only text field, query insights to monitor top N queries, and k-NN search on nested fields. For a full list of release highlights, see the Release Notes. | 20 February 2024 diff --git a/_config.yml b/_config.yml index e5cce4e34f..be015cec06 100644 --- a/_config.yml +++ b/_config.yml @@ -5,9 +5,9 @@ baseurl: "/docs/latest" # the subpath of your site, e.g. /blog url: "https://opensearch.org" # the base hostname & protocol for your site, e.g. http://example.com permalink: /:path/ -opensearch_version: '2.14.0' -opensearch_dashboards_version: '2.14.0' -opensearch_major_minor_version: '2.14' +opensearch_version: '2.15.0' +opensearch_dashboards_version: '2.15.0' +opensearch_major_minor_version: '2.15' lucene_version: '9_10_0' # Build settings diff --git a/_data-prepper/pipelines/configuration/sinks/s3.md b/_data-prepper/pipelines/configuration/sinks/s3.md index 71cb7b1f70..d1413f6ffc 100644 --- a/_data-prepper/pipelines/configuration/sinks/s3.md +++ b/_data-prepper/pipelines/configuration/sinks/s3.md @@ -15,19 +15,20 @@ The `s3` sink uses the following format when batching events: ``` ${pathPrefix}events-%{yyyy-MM-dd'T'HH-mm-ss'Z'}-${currentTimeInNanos}-${uniquenessId}.${codecSuppliedExtension} ``` +{% include copy-curl.html %} -When a batch of objects is written to S3, the objects are formatted similarly to the following: +When a batch of objects is written to Amazon S3, the objects are formatted similarly to the following: ``` my-logs/2023/06/09/06/events-2023-06-09T06-00-01-1686290401871214927-ae15b8fa-512a-59c2-b917-295a0eff97c8.json ``` +{% include copy-curl.html %} - -For more information about how to configure an object, see the [Object key](#object-key-configuration) section. +For more information about how to configure an object, refer to [Object key](#object-key-configuration). ## Usage -The following example creates a pipeline configured with an s3 sink. It contains additional options for customizing the event and size thresholds for which the pipeline sends record events and sets the codec type `ndjson`: +The following example creates a pipeline configured with an `s3` sink. It contains additional options for customizing the event and size thresholds for the pipeline and sets the codec type as `ndjson`: ``` pipeline: @@ -49,10 +50,11 @@ pipeline: ndjson: buffer_type: in_memory ``` +{% include copy-curl.html %} ## IAM permissions -In order to use the `s3` sink, configure AWS Identity and Access Management (IAM) to grant Data Prepper permissions to write to Amazon S3. You can use a configuration similar to the following JSON configuration: +To use the `s3` sink, configure AWS Identity and Access Management (IAM) to grant Data Prepper permissions to write to Amazon S3. You can use a configuration similar to the following JSON configuration: ```json { @@ -69,36 +71,62 @@ In order to use the `s3` sink, configure AWS Identity and Access Management (IAM ] } ``` +{% include copy-curl.html %} + +## Cross-account S3 access + +When Data Prepper fetches data from an S3 bucket, it verifies bucket ownership using a [bucket owner condition](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-owner-condition.html). + +By default, the S3 sink does not require `bucket_owners`. If `bucket_owners` is configured and a bucket is not included in one of the mapped configurations, `default_bucket_owner` defaults to the account ID in `aws.sts_role_arn`. You can configure both `bucket_owners` and `default_bucket_owner` and apply the settings together. + +When ingesting data from multiple S3 buckets with different account associations, configure Data Prepper for cross-account S3 access based on the following conditions: + +- For S3 buckets belonging to the same account, set `default_bucket_owner` to that account's ID. +- For S3 buckets belonging to multiple accounts, use a `bucket_owners` map. + +A `bucket_owners` map specifies account IDs for buckets belonging to multiple accounts. For example, in the following configuration, `my-bucket-01` is owned by `123456789012` and `my-bucket-02` is owned by `999999999999`: + +``` +sink: + - s3: + default_bucket_owner: 111111111111 + bucket_owners: + my-bucket-01: 123456789012 + my-bucket-02: 999999999999 +``` +{% include copy-curl.html %} ## Configuration Use the following options when customizing the `s3` sink. -Option | Required | Type | Description -:--- | :--- | :--- | :--- -`bucket` | Yes | String | The name of the S3 bucket to which objects are stored. The `name` must match the name of your object store. -`codec` | Yes | [Codec](#codec) | The codec determining the format of output data. -`aws` | Yes | AWS | The AWS configuration. See [aws](#aws) for more information. -`threshold` | Yes | [Threshold](#threshold-configuration) | Configures when to write an object to S3. -`object_key` | No | Sets the `path_prefix` and the `file_pattern` of the object store. The file pattern is always `events-%{yyyy-MM-dd'T'hh-mm-ss}`. By default, those objects are found inside the root directory of the bucket. The `path_prefix` is configurable. -`compression` | No | String | The compression algorithm to apply: `none`, `gzip`, or `snappy`. Default is `none`. -`buffer_type` | No | [Buffer type](#buffer-type) | Determines the buffer type. -`max_retries` | No | Integer | The maximum number of times a single request should retry when ingesting data to S3. Defaults to `5`. - -## aws +Option | Required | Type | Description +:--- |:---------|:------------------------------------------------| :--- +`bucket` | Yes | String | Specifies the sink's S3 bucket name. Supports dynamic bucket naming using [Data Prepper expressions]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), for example, `test-${/bucket_id}`. If a dynamic bucket is inaccessible and no `default_bucket` is configured, then the object data is dropped. +`default_bucket` | No | String | A static bucket for inaccessible dynamic buckets in `bucket`. +`bucket_owners` | No | Map | A map of bucket names and their account owner IDs for cross-account access. Refer to [Cross-account S3 access](#s3_bucket_ownership). +`default_bucket_owner` | No | String | The AWS account ID for an S3 bucket owner. Refer to [Cross-account S3 access](#s3_bucket_ownership). +`codec` | Yes | [Codec](#codec) | Serializes data in S3 objects. +`aws` | Yes | AWS | The AWS configuration. Refer to [aws](#aws). +`threshold` | Yes | [Threshold](#threshold-configuration) | Condition for writing objects to S3. +`aggregate_threshold` | No | [Aggregate threshold](#threshold-configuration) | A condition for flushing objects with a dynamic `path_prefix`. +`object_key` | No | [Object key](#object-key-configuration) | Sets `path_prefix` and `file_pattern` for object storage. The file pattern is `events-%{yyyy-MM-dd'T'hh-mm-ss}`. By default, these objects are found in the bucket's root directory. `path_prefix` is configurable. +`compression` | No | String | The compression algorithm: Either `none`, `gzip`, or `snappy`. Default is `none`. +`buffer_type` | No | [Buffer type](#buffer-type) | The buffer type configuration. +`max_retries` | No | Integer | The maximum number of retries for S3 ingestion requests. Default is `5`. + +## `aws` Option | Required | Type | Description :--- | :--- | :--- | :--- `region` | No | String | The AWS Region to use for credentials. Defaults to [standard SDK behavior to determine the Region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html). -`sts_role_arn` | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon SQS and Amazon S3. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). +`sts_role_arn` | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon Simple Queue Service (Amazon SQS) and Amazon S3. Defaults to `null`, which uses the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). `sts_header_overrides` | No | Map | A map of header overrides that the IAM role assumes for the sink plugin. -`sts_external_id` | No | String | An STS external ID used when Data Prepper assumes the role. For more information, see the `ExternalId` documentation in the [STS AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) API reference. - - +`sts_external_id` | No | String | An AWS STS external ID used when Data Prepper assumes the role. For more information, refer to the `ExternalId` section under [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) in the AWS STS API reference. ## Threshold configuration -Use the following options to set ingestion thresholds for the `s3` sink. When any of these conditions are met, Data Prepper will write events to an S3 object. +Use the following options to set ingestion thresholds for the `s3` sink. Data Prepper writes events to an S3 object when any of these conditions occur. Option | Required | Type | Description :--- | :--- | :--- | :--- @@ -106,84 +134,77 @@ Option | Required | Type | Description `maximum_size` | No | String | The maximum number of bytes to accumulate before writing an object to S3. Default is `50mb`. `event_collect_timeout` | Yes | String | The maximum amount of time before Data Prepper writes an event to S3. The value should be either an ISO-8601 duration, such as `PT2M30S`, or a simple notation, such as `60s` or `1500ms`. +## Aggregate threshold configuration + +Use the following options to set rules or limits that trigger certain actions or behavior when an aggregated value crosses a defined threshold. + +Option | Required | Type | Description +:--- |:-----------------------------------|:-------| :--- +`flush_capacity_ratio` | No | Float | The percentage of groups to be force-flushed when `aggregate_threshold maximum_size` is reached. The percentage is expressed as a number between `0.0` and `1.0`. Default is `0.5`. +`maximum_size` | Yes | String | The maximum number of bytes to accumulate before force-flushing objects. For example, `128mb`. ## Buffer type -`buffer_type` is an optional configuration that determines how Data Prepper temporarily stores data before writing an object to S3. The default value is `in_memory`. Use one of the following options: +`buffer_type` is an optional configuration that determines how Data Prepper temporarily stores data before writing an object to S3. The default value is `in_memory`. + +Use one of the following options: - `in_memory`: Stores the record in memory. -- `local_file`: Flushes the record into a file on your local machine. This uses your machine's temporary directory. +- `local_file`: Flushes the record into a file on your local machine. This option uses your machine's temporary directory. - `multipart`: Writes using the [S3 multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html). Every 10 MB is written as a part. ## Object key configuration +Use the following options to define how object keys are constructed for objects stored in S3. + Option | Required | Type | Description :--- | :--- | :--- | :--- -`path_prefix` | No | String | The S3 key prefix path to use for objects written to S3. Accepts date-time formatting. For example, you can use `%{yyyy}/%{MM}/%{dd}/%{HH}/` to create hourly folders in S3. The prefix path should end with `/`. By default, Data Prepper writes objects to the root of the S3 bucket. - +`path_prefix` | No | String | The S3 key prefix path to use for objects written to S3. Accepts date-time formatting and dynamic injection of values using [Data Prepper expressions](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/). For example, you can use `/${/my_partition_key}/%{yyyy}/%{MM}/%{dd}/%{HH}/` to create hourly folders in S3 based on the `my_partition_key` value. The prefix path should end with `/`. By default, Data Prepper writes objects to the S3 bucket root. -## codec +## `codec` The `codec` determines how the `s3` source formats data written to each S3 object. -### avro codec +### `avro` codec -The `avro` codec writes an event as an [Apache Avro](https://avro.apache.org/) document. +The `avro` codec writes an event as an [Apache Avro](https://avro.apache.org/) document. Because Avro requires a schema, you may either define the schema or have Data Prepper automatically generate it. Defining your own schema is recommended because this will allow it to be tailored to your particular use case. -Because Avro requires a schema, you may either define the schema yourself, or Data Prepper will automatically generate a schema. -In general, you should define your own schema because it will most accurately reflect your needs. +When you provide your own Avro schema, that schema defines the final structure of your data. Any extra values in any incoming events that are not mapped in the Avro schema will not be included in the final destination. Data Prepper does not allow the use of `include_keys` or `exclude_keys` with a custom schema so as to avoid confusion between a custom Avro schema and the `include_keys` or `exclude_keys` sink configurations. -We recommend that you make your Avro fields use a null [union](https://avro.apache.org/docs/current/specification/#unions). -Without the null union, each field must be present or the data will fail to write to the sink. -If you can be certain that each each event has a given field, you can make it non-nullable. +In cases where your data is uniform, you may be able to automatically generate a schema. Automatically generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and all keys must be present in all events in order to automatically generate a working schema. Automatically generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control which data is included in the automatically generated schema. -When you provide your own Avro schema, that schema defines the final structure of your data. -Therefore, any extra values inside any incoming events that are not mapped in the Arvo schema will not be included in the final destination. -To avoid confusion between a custom Arvo schema and the `include_keys` or `exclude_keys` sink configurations, Data Prepper does not allow the use of the `include_keys` or `exclude_keys` with a custom schema. - -In cases where your data is uniform, you may be able to automatically generate a schema. -Automatically generated schemas are based on the first event received by the codec. -The schema will only contain keys from this event. -Therefore, you must have all keys present in all events in order for the automatically generated schema to produce a working schema. -Automatically generated schemas make all fields nullable. -Use the sink's `include_keys` and `exclude_keys` configurations to control what data is included in the auto-generated schema. +Avro fields should use a null [union](https://avro.apache.org/docs/current/specification/#unions) because this will allow missing values. Otherwise, all required fields must be present for each event. Use non-nullable fields only when you are certain they exist. +Use the following options to configure the codec. Option | Required | Type | Description :--- | :--- | :--- | :--- `schema` | Yes | String | The Avro [schema declaration](https://avro.apache.org/docs/current/specification/#schema-declaration). Not required if `auto_schema` is set to true. `auto_schema` | No | Boolean | When set to `true`, automatically generates the Avro [schema declaration](https://avro.apache.org/docs/current/specification/#schema-declaration) from the first event. - -### ndjson codec - -The `ndjson` codec writes each line as a JSON object. +### `ndjson` codec -The `ndjson` codec does not take any configurations. +The `ndjson` codec writes each line as a JSON object. The `ndjson` codec does not take any configurations. +### `json` codec -### json codec - -The `json` codec writes events in a single large JSON file. -Each event is written into an object within a JSON array. +The `json` codec writes events in a single large JSON file. Each event is written into an object within a JSON array. +Use the following options to configure the codec. Option | Required | Type | Description :--- | :--- | :--- | :--- `key_name` | No | String | The name of the key for the JSON array. By default this is `events`. +### `parquet` codec -### parquet codec - -The `parquet` codec writes events into a Parquet file. -When using the Parquet codec, set the `buffer_type` to `in_memory`. +The `parquet` codec writes events into a Parquet file. When using the codec, set `buffer_type` to `in_memory`. -The Parquet codec writes data using the Avro schema. -Because Parquet requires an Avro schema, you may either define the schema yourself, or Data Prepper will automatically generate a schema. -However, we generally recommend that you define your own schema so that it can best meet your needs. +The `parquet` codec writes data using the schema. Because Parquet requires an Avro schema, you may either define the schema yourself or have Data Prepper automatically generate it. Defining your own schema is recommended because this will allow it to be tailored to your particular use case. -For details on the Avro schema and recommendations, see the [Avro codec](#avro-codec) documentation. +For more information about the Avro schema, refer to [Avro codec](#avro-codec). +Use the following options to configure the codec. Option | Required | Type | Description :--- | :--- | :--- | :--- @@ -192,7 +213,7 @@ Option | Required | Type | Description ### Setting a schema with Parquet -The following example shows you how to configure the `s3` sink to write Parquet data into a Parquet file using a schema for [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html#flow-log-records): +The following example pipeline shows how to configure the `s3` sink to write Parquet data into a Parquet file using a schema for [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html#flow-log-records): ``` pipeline: @@ -235,4 +256,4 @@ pipeline: event_collect_timeout: PT15M buffer_type: in_memory ``` - +{% include copy-curl.html %} diff --git a/_data-prepper/pipelines/configuration/sources/http-source.md b/_data-prepper/pipelines/configuration/sources/http.md similarity index 88% rename from _data-prepper/pipelines/configuration/sources/http-source.md rename to _data-prepper/pipelines/configuration/sources/http.md index b41855cdc2..06933edc1c 100644 --- a/_data-prepper/pipelines/configuration/sources/http-source.md +++ b/_data-prepper/pipelines/configuration/sources/http.md @@ -1,14 +1,16 @@ --- layout: default -title: http_source +title: http parent: Sources grand_parent: Pipelines nav_order: 5 +redirect_from: + - /data-prepper/pipelines/configuration/sources/http-source/ --- -# http_source +# http -`http_source` is a source plugin that supports HTTP. Currently, `http_source` only supports the JSON UTF-8 codec for incoming requests, such as `[{"key1": "value1"}, {"key2": "value2"}]`. The following table describes options you can use to configure the `http_source` source. +The `http` plugin accepts HTTP requests from clients. Currently, `http` only supports the JSON UTF-8 codec for incoming requests, such as `[{"key1": "value1"}, {"key2": "value2"}]`. The following table describes options you can use to configure the `http` source. Option | Required | Type | Description :--- | :--- | :--- | :--- @@ -19,6 +21,7 @@ request_timeout | No | Integer | The request timeout, in milliseconds. Default v thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`. max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`. max_pending_requests | No | Integer | The maximum allowed number of tasks in the `ScheduledThreadPool` work queue. Default value is `1024`. +max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single HTTP request. Default value is `10mb`. authentication | No | Object | An authentication configuration. By default, this creates an unauthenticated server for the pipeline. This uses pluggable authentication for HTTPS. To use basic authentication define the `http_basic` plugin with a `username` and `password`. To provide customer authentication, use or create a plugin that implements [ArmeriaHttpAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/1.2.0/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/ArmeriaHttpAuthenticationProvider.java). ssl | No | Boolean | Enables TLS/SSL. Default value is false. ssl_certificate_file | Conditionally | String | SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. Amazon S3 path example `s3:///`. Required if `ssl` is set to true and `use_acm_certificate_for_ssl` is set to false. @@ -35,7 +38,7 @@ Content will be added to this section.---> ## Metrics -The `http_source` source includes the following metrics. +The `http` source includes the following metrics. ### Counters diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md index 58d8a2b059..068369efaf 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -21,7 +21,8 @@ You can configure the `otel_logs_source` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | | port | int | Represents the port that the `otel_logs_source` source is running on. Default value is `21892`. | -| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support an unframed gRPC request with an HTTP idiomatic path to a configurable path. The path should start with `/`, and its length should be at least 1. The `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain a `${pipelineName}` placeholder, which is replaced with the pipeline name. If the value is empty and `unframed_requests` is `true`, then the path that the source provides is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | +| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support an unframed gRPC request with an HTTP idiomatic path to a configurable path. The path should start with `/`, and its length should be at least 1. The `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain a `${pipelineName}` placeholder, which is replaced with the pipeline name. If the value is empty and `unframed_requests` is `true`, then the source provides the path `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | +| max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. | request_timeout | int | Represents the request timeout duration in milliseconds. Default value is `10000`. | | health_check_service | Boolean | Enables the gRPC health check service under `grpc.health.v1/Health/Check`. Default value is `false`. | | proto_reflection_service | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default value is `false`. | diff --git a/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md b/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md index 0301963538..bea74a96d3 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md @@ -19,6 +19,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto unframed_requests | No | Boolean | Enables requests not framed using the gRPC wire protocol. thread_count | No | Integer | The number of threads to keep in the `ScheduledThreadPool`. Default value is `200`. max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`. +max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. ssl | No | Boolean | Enables connections to the OpenTelemetry source port over TLS/SSL. Default value is `true`. sslKeyCertChainFile | Conditionally | String | File-system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`. sslKeyFile | Conditionally | String | File-system path or Amazon S3 path to the security key (for example, `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if `ssl` is set to `true`. diff --git a/_data-prepper/pipelines/configuration/sources/otel-trace-source.md b/_data-prepper/pipelines/configuration/sources/otel-trace-source.md index 137592bbe8..1be7864c33 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-trace-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-trace-source.md @@ -24,6 +24,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto unframed_requests | No | Boolean | Enable requests not framed using the gRPC wire protocol. thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`. max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`. +max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. ssl | No | Boolean | Enables connections to the OTel source port over TLS/SSL. Defaults to `true`. sslKeyCertChainFile | Conditionally | String | File system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`. sslKeyFile | Conditionally | String | File system path or Amazon S3 path to the security key (for example, `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if `ssl` is set to `true`. diff --git a/_data/versions.json b/_data/versions.json index 969f93681b..0c99ed871e 100644 --- a/_data/versions.json +++ b/_data/versions.json @@ -1,10 +1,11 @@ { - "current": "2.14", + "current": "2.15", "all": [ - "2.14", + "2.15", "1.3" ], "archived": [ + "2.14", "2.13", "2.12", "2.11", @@ -23,7 +24,7 @@ "1.1", "1.0" ], - "latest": "2.14" + "latest": "2.15" } diff --git a/_layouts/default.html b/_layouts/default.html index 8ba6bd4703..d4d40d8cc4 100755 --- a/_layouts/default.html +++ b/_layouts/default.html @@ -165,9 +165,9 @@
{% if page.section == "opensearch" %} {% if site.doc_version == "supported" %} -

This is an earlier version of the OpenSearch documentation. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.

+

You're viewing version {{site.opensearch_major_minor_version}} of the OpenSearch documentation. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.

{% elsif site.doc_version == "unsupported" %} -

This version of the OpenSearch documentation is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.

+

You're viewing version {{site.opensearch_major_minor_version}} of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.

{% endif %} {% endif %} {% if site.heading_anchors != false %} diff --git a/_observing-your-data/metricsanalytics.md b/_observing-your-data/prometheusmetrics.md similarity index 99% rename from _observing-your-data/metricsanalytics.md rename to _observing-your-data/prometheusmetrics.md index 7c31e1cc33..85448ca358 100644 --- a/_observing-your-data/metricsanalytics.md +++ b/_observing-your-data/prometheusmetrics.md @@ -2,8 +2,6 @@ layout: default title: Metric analytics nav_order: 40 -redirect_from: - - /observing-your-data/metricsanalytics/ --- # Metric analytics diff --git a/_security/authentication-backends/jwt.md b/_security/authentication-backends/jwt.md index afcd4c78ee..b6b08388b5 100644 --- a/_security/authentication-backends/jwt.md +++ b/_security/authentication-backends/jwt.md @@ -235,7 +235,7 @@ openid_auth_domain: transport_enabled: true order: 0 http_authenticator: - type: openid + type: openid # use the OpenID Connect domain, since JWT is part of this authentication. challenge: false config: subject_key: preferred_username diff --git a/_security/configuration/configuration.md b/_security/configuration/configuration.md index d4f6a47cde..2a038b7fb9 100755 --- a/_security/configuration/configuration.md +++ b/_security/configuration/configuration.md @@ -11,7 +11,7 @@ redirect_from: One of the first steps when setting up the Security plugin is deciding which authentication backend to use. The role played by the backend in authentication is covered in [steps 2 and 3 of the authentication flow]({{site.url}}{{site.baseurl}}/security/authentication-backends/authc-index/#authentication-flow). The plugin has an internal user database, but many people prefer to use an existing authentication backend, such as an LDAP server, or some combination of the two. -The primary file used to configure an authentication and authorization backend is `config/opensearch-security/config.yml`. This file defines how the Security plugin retrieves user credentials, how it verifies the credentials, and how it fetches additional roles when the backend selected for authentication and authorization supports this feature. This topic provides a basic overview of the configuration file and its requirements for setting up security. For information about configuring a specific backend, see [Authentication backends]({{site.url}}{{site.baseurl}}/security/authentication-backends/authc-index/). +The primary file used to configure the authentication and authorization backend is `/usr/share/opensearch/config/opensearch-security/config.yml`. This file defines how the Security plugin retrieves user credentials, how the plugin verifies the credentials, and how the plugin fetches additional roles when the backend selected for authentication and authorization supports this feature. This topic provides a basic overview of the configuration file and its requirements for setting up security. For information about configuring a specific backend, see [Authentication backends]({{site.url}}{{site.baseurl}}/security/authentication-backends/authc-index/). The `config.yml` file includes three main parts: diff --git a/release-notes/opensearch-documentation-release-notes-2.15.0.md b/release-notes/opensearch-documentation-release-notes-2.15.0.md new file mode 100644 index 0000000000..5f7ab9b049 --- /dev/null +++ b/release-notes/opensearch-documentation-release-notes-2.15.0.md @@ -0,0 +1,42 @@ +# OpenSearch Documentation Website 2.15.0 Release Notes + +The OpenSearch 2.15.0 documentation includes the following additions and updates. + +## New documentation for 2.15.0 + +- Alerts in correlations feature documentation [#7410](https://github.com/opensearch-project/documentation-website/pull/7410) +- Add documentations for batch ingestion feature [#7408](https://github.com/opensearch-project/documentation-website/pull/7408) +- Changed VisBuilder status from experimental to GA [#7405](https://github.com/opensearch-project/documentation-website/pull/7405) +- Add documentation for innerHit on knn nested field [#7404](https://github.com/opensearch-project/documentation-website/pull/7404) +- AD Enhancements in Version 2.15 [#7388](https://github.com/opensearch-project/documentation-website/pull/7388) +- Add connector tool [#7384](https://github.com/opensearch-project/documentation-website/pull/7384) +- Add remote guardrails model support [#7377](https://github.com/opensearch-project/documentation-website/pull/7377) +- Update documentation of ml inference processors to support for local models [#7368](https://github.com/opensearch-project/documentation-website/pull/7368) +- Trace analytics update [#7362](https://github.com/opensearch-project/documentation-website/pull/7362) +- Add doc for alerting comments [#7360](https://github.com/opensearch-project/documentation-website/pull/7360) +- Add documentation related to removal of source and recovery source in k-NN performance tuning section [#7359](https://github.com/opensearch-project/documentation-website/pull/7359) +- Added documentation for new default workflow templates [#7346](https://github.com/opensearch-project/documentation-website/pull/7346) +- Mark docrep to remote migration as GA and modify settings names [#7342](https://github.com/opensearch-project/documentation-website/pull/7342) +- Add documentation for the new setting of cardinality aggregation dynamic pruning [#7341](https://github.com/opensearch-project/documentation-website/pull/7341) +- Add documentation for wildcard field type [#7339](https://github.com/opensearch-project/documentation-website/pull/7339) +- Update document for handle SageMaker throttling [#7331](https://github.com/opensearch-project/documentation-website/pull/7331) +- Add documentation related to new settings for segment upload timeout [#7330](https://github.com/opensearch-project/documentation-website/pull/7330) +- Add documentation of derived fields [#7329](https://github.com/opensearch-project/documentation-website/pull/7329) +- [MDS] Add security analytics, alerting, feature anaywhere in the multiple data source document [#7328](https://github.com/opensearch-project/documentation-website/pull/7328) +- Add document for top n queries improvements in 2.15 [#7326](https://github.com/opensearch-project/documentation-website/pull/7326) +- Update the integration page to reflect new integration catalog features [#7324](https://github.com/opensearch-project/documentation-website/pull/7324) +- Add doc for neural-sparse-query-two-phase-processor. [#7306](https://github.com/opensearch-project/documentation-website/pull/7306) +- Add documentation for Indices Request Cache Overview and its settings [#7288](https://github.com/opensearch-project/documentation-website/pull/7288) +- Added documentation for Reindex workflow step [#7271](https://github.com/opensearch-project/documentation-website/pull/7271) +- Document optional clear_status query parameter for Delete Workflow API [#7268](https://github.com/opensearch-project/documentation-website/pull/7268) +- Update field-masking.md. Configure default masking algorithm. [#7162](https://github.com/opensearch-project/documentation-website/pull/7162) +- add documentation for use compound file setting [#7092](https://github.com/opensearch-project/documentation-website/pull/7092) +- Added documentation for managed identity support in repository-azure plugin [#7068](https://github.com/opensearch-project/documentation-website/pull/7068) + +## In progress documentation for 2.15.0 + +- Initial UBI documentation [#7284](https://github.com/opensearch-project/documentation-website/pull/7284) + +## Documentation for 2.15.0 experimental features + +- Add remote state publication [#7364](https://github.com/opensearch-project/documentation-website/pull/7364)