From a8ad33f7632994a4c08127a67ea4042268890073 Mon Sep 17 00:00:00 2001 From: David Venable Date: Tue, 25 Jun 2024 10:55:48 -0500 Subject: [PATCH] Adds the max_request_length documentation for appropriate Data Prepper sources (#7441) * Adds the max_request_length documentation. Also, updates and corrects the http source documentation to use the correct name. Signed-off-by: David Venable * Update _data-prepper/pipelines/configuration/sources/http.md Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/otel-metrics-source.md Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/otel-trace-source.md Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/http.md Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/http.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/otel-metrics-source.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _data-prepper/pipelines/configuration/sources/otel-trace-source.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --------- Signed-off-by: David Venable Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower --- .../configuration/sources/{http-source.md => http.md} | 11 +++++++---- .../configuration/sources/otel-logs-source.md | 3 ++- .../configuration/sources/otel-metrics-source.md | 1 + .../configuration/sources/otel-trace-source.md | 1 + 4 files changed, 11 insertions(+), 5 deletions(-) rename _data-prepper/pipelines/configuration/sources/{http-source.md => http.md} (88%) diff --git a/_data-prepper/pipelines/configuration/sources/http-source.md b/_data-prepper/pipelines/configuration/sources/http.md similarity index 88% rename from _data-prepper/pipelines/configuration/sources/http-source.md rename to _data-prepper/pipelines/configuration/sources/http.md index b41855cdc2..06933edc1c 100644 --- a/_data-prepper/pipelines/configuration/sources/http-source.md +++ b/_data-prepper/pipelines/configuration/sources/http.md @@ -1,14 +1,16 @@ --- layout: default -title: http_source +title: http parent: Sources grand_parent: Pipelines nav_order: 5 +redirect_from: + - /data-prepper/pipelines/configuration/sources/http-source/ --- -# http_source +# http -`http_source` is a source plugin that supports HTTP. Currently, `http_source` only supports the JSON UTF-8 codec for incoming requests, such as `[{"key1": "value1"}, {"key2": "value2"}]`. The following table describes options you can use to configure the `http_source` source. +The `http` plugin accepts HTTP requests from clients. Currently, `http` only supports the JSON UTF-8 codec for incoming requests, such as `[{"key1": "value1"}, {"key2": "value2"}]`. The following table describes options you can use to configure the `http` source. Option | Required | Type | Description :--- | :--- | :--- | :--- @@ -19,6 +21,7 @@ request_timeout | No | Integer | The request timeout, in milliseconds. Default v thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`. max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`. max_pending_requests | No | Integer | The maximum allowed number of tasks in the `ScheduledThreadPool` work queue. Default value is `1024`. +max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single HTTP request. Default value is `10mb`. authentication | No | Object | An authentication configuration. By default, this creates an unauthenticated server for the pipeline. This uses pluggable authentication for HTTPS. To use basic authentication define the `http_basic` plugin with a `username` and `password`. To provide customer authentication, use or create a plugin that implements [ArmeriaHttpAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/1.2.0/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/ArmeriaHttpAuthenticationProvider.java). ssl | No | Boolean | Enables TLS/SSL. Default value is false. ssl_certificate_file | Conditionally | String | SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. Amazon S3 path example `s3:///`. Required if `ssl` is set to true and `use_acm_certificate_for_ssl` is set to false. @@ -35,7 +38,7 @@ Content will be added to this section.---> ## Metrics -The `http_source` source includes the following metrics. +The `http` source includes the following metrics. ### Counters diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md index 58d8a2b059..068369efaf 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -21,7 +21,8 @@ You can configure the `otel_logs_source` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | | port | int | Represents the port that the `otel_logs_source` source is running on. Default value is `21892`. | -| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support an unframed gRPC request with an HTTP idiomatic path to a configurable path. The path should start with `/`, and its length should be at least 1. The `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain a `${pipelineName}` placeholder, which is replaced with the pipeline name. If the value is empty and `unframed_requests` is `true`, then the path that the source provides is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | +| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support an unframed gRPC request with an HTTP idiomatic path to a configurable path. The path should start with `/`, and its length should be at least 1. The `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain a `${pipelineName}` placeholder, which is replaced with the pipeline name. If the value is empty and `unframed_requests` is `true`, then the source provides the path `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | +| max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. | request_timeout | int | Represents the request timeout duration in milliseconds. Default value is `10000`. | | health_check_service | Boolean | Enables the gRPC health check service under `grpc.health.v1/Health/Check`. Default value is `false`. | | proto_reflection_service | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default value is `false`. | diff --git a/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md b/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md index 0301963538..bea74a96d3 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-metrics-source.md @@ -19,6 +19,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto unframed_requests | No | Boolean | Enables requests not framed using the gRPC wire protocol. thread_count | No | Integer | The number of threads to keep in the `ScheduledThreadPool`. Default value is `200`. max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`. +max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. ssl | No | Boolean | Enables connections to the OpenTelemetry source port over TLS/SSL. Default value is `true`. sslKeyCertChainFile | Conditionally | String | File-system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`. sslKeyFile | Conditionally | String | File-system path or Amazon S3 path to the security key (for example, `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if `ssl` is set to `true`. diff --git a/_data-prepper/pipelines/configuration/sources/otel-trace-source.md b/_data-prepper/pipelines/configuration/sources/otel-trace-source.md index 137592bbe8..1be7864c33 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-trace-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-trace-source.md @@ -24,6 +24,7 @@ proto_reflection_service | No | Boolean | Enables a reflection service for Proto unframed_requests | No | Boolean | Enable requests not framed using the gRPC wire protocol. thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`. max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is `500`. +max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. ssl | No | Boolean | Enables connections to the OTel source port over TLS/SSL. Defaults to `true`. sslKeyCertChainFile | Conditionally | String | File system path or Amazon Simple Storage Service (Amazon S3) path to the security certificate (for example, `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if `ssl` is set to `true`. sslKeyFile | Conditionally | String | File system path or Amazon S3 path to the security key (for example, `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if `ssl` is set to `true`.