diff --git a/docs/en/aws-deploy-elastic-serverless-forwarder.asciidoc b/docs/en/aws-deploy-elastic-serverless-forwarder.asciidoc index abe7cd25..2563b11b 100644 --- a/docs/en/aws-deploy-elastic-serverless-forwarder.asciidoc +++ b/docs/en/aws-deploy-elastic-serverless-forwarder.asciidoc @@ -225,7 +225,7 @@ For `elasticsearch` the following arguments are supported: * `args.es_datastream_name`: Name of data stream or index where logs should be forwarded to. Lambda supports automatic routing of various {aws} service logs to the corresponding data streams for further processing and storage in the {es} cluster. It supports automatic routing of `aws.cloudtrail`, `aws.cloudwatch_logs`, `aws.elb_logs`, `aws.firewall_logs`, `aws.vpcflow`, and `aws.waf` logs. For other log types, if using data streams, you can optionally set its value in the configuration file according to the naming convention for data streams and available integrations. If the `es_datastream_name` is not specified and it cannot be matched with any of the above {aws} services, then the value will be set to `logs-generic-default`. In versions **v0.29.1** and below, this configuration parameter was named `es_index_or_datastream_name`. Rename the configuration parameter to `es_datastream_name` in your `config.yaml` file on the S3 bucket to continue using it in the future version. The older name `es_index_or_datastream_name` is deprecated as of version **v0.30.0**. The related backward compatibility code is removed from version **v1.0.0**. * `args.batch_max_actions`: (Optional) Maximum number of actions to send in a single bulk request. Default value: 500. * `args.batch_max_bytes`: (Optional) Maximum size in bytes to send in a single bulk request. Default value: 10485760 (10MB). - * `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate. + * `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate. For `logstash` the following arguments are supported: @@ -234,7 +234,7 @@ For `logstash` the following arguments are supported: * `args.password`: (Optional) Password of the {ls} instance to connect to. Mandatory if HTTP Basic authentication is enabled in {ls}. * `args.max_batch_size`: (Optional) Maximum number of events to send in a single HTTP(s) request. Default value: 500 * `args.compression_level`: (Optional) The GZIP compression level for HTTP(s) requests towards {ls}. It can be any integer value between 1 (minimum compression, best performance, highest amount of bytes sent) and 9 (maximum compression, worst performance, lowest amount of bytes sent). Default value: 1 - * `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate. + * `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate. [discrete] [[aws-serverless-forwarder-define-deploy-parameters]] @@ -276,20 +276,20 @@ These parameters define the permissions required in order to access the associat [discrete] === Network -The way to attach the Elastic Serverless Forwarder to a specific {aws} VPC is by defining both the security groups IDs and subnet IDs belonging to the {aws} VPC. The limit is related to the https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html[CloudFormation VPCConfig property] + +To attach the Elastic Serverless Forwarder to a specific {aws} VPC, specify the security group IDs and subnet IDs that belong to the {aws} VPC. This requirement is related to the https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html[CloudFormation VPCConfig property]. These are the parameters: -- `ElasticServerlessForwarderSecurityGroups`: Add a comma delimited list of security group IDs to attach to the forwarder. -- `ElasticServerlessForwarderSubnets`: Add a comma delimited list of subnet IDs for to the forwarder. +- `ElasticServerlessForwarderSecurityGroups`: Add a comma delimited list of security group IDs to attach to the forwarder. +- `ElasticServerlessForwarderSubnets`: Add a comma delimited list of subnet IDs for the forwarder. Both parameters are required in order to attach the Elastic Serverless Forwarder to a specific {aws} VPC. Leave both parameters blank if you don't want the forwarder to belong to any specific {aws} VPC. +If the Elastic Serverless Forwarder is attached to a VPC, you need to https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html[create VPC endpoints] for S3 and SQS, and for *every* service you define as an input for the forwarder. S3 and SQS VPC endpoints are always required for reading the `config.yaml` uploaded to S3 and managing the _Continuing queue_ and the _Replay queue_, regardless of the <> used. If you use <>, you need to create a VPC endpoint for EC2, too. -If the Elastic Serverless Forwarder is attached to a VPC, you need to https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html[create VPC Endpoints] for S3 and SQS, and for *every* service you define as an input for the forwarder. S3 and SQS VPC Endpoints are always required for reading the `config.yaml` uploaded to S3 and managing the _Continuing queue_ and the _Replay queue_, regardless of the <> used. If you use <> you need to create a VPC Endpoint for EC2 as well. - -NOTE: Refer to documentation for https://www.elastic.co/guide/en/cloud/current/ec-traffic-filtering-vpc.html[AWS PrivateLink traffic filters] in order to find your VPC endpoint ID, and the hostname to use in the `config.yml` in order to access your Elasticsearch cluster over PrivateLink. +NOTE: Refer to the {cloud}/ec-traffic-filtering-vpc.html[AWS PrivateLink traffic filters] documentation to find your VPC endpoint ID and the hostname to use in the `config.yml` in order to access your Elasticsearch cluster over PrivateLink. [discrete] [[aws-serverless-forwarder-deploy-sar]] diff --git a/docs/en/aws-elastic-serverless-forwarder.asciidoc b/docs/en/aws-elastic-serverless-forwarder.asciidoc index 456e0540..48d9b3bb 100644 --- a/docs/en/aws-elastic-serverless-forwarder.asciidoc +++ b/docs/en/aws-elastic-serverless-forwarder.asciidoc @@ -33,11 +33,11 @@ Elastic Serverless Forwarder ensures <> file to configure the service for each input and output type, including information such as SQS queue ARN (Amazon Resource Number) and {es} or {ls} connection details. You can create multiple input sections within the configuration file to map different inputs to specific log types. -There is no need to define a specific input in the <> for the _Contuining queue_ and the _Replay queue. +There is no need to define a specific input in the <> for the _Continuing queue_ and the _Replay queue_. The forwarder also supports writing directly to an index, alias, or custom data stream. This enables existing {es} users to re-use index templates, ingest pipelines, or dashboards that are already created and connected to other processes. @@ -86,58 +86,58 @@ You can set up a separate SQS queue for each type of log. The config parameter f [discrete] [[aws-serverless-forwarder-at-least-once-delivery]] = At-least-once delivery -The Elastic Serverless Forwarder ensures at-least-once delivery of the forwarder messages recurring to the `Continuing queue` and `Replay queue`. +The Elastic Serverless Forwarder ensures at-least-once delivery of the forwarded messages recurring to the _Continuing queue_ and _Replay queue_. [discrete] [[aws-serverless-forwarder-at-least-once-delivery-continuing-queue]] == Continuing queue -The Elastic Serverless Forwarder can run for a maximum amount of time of 15 minutes. Different inputs can trigger the Elastic Serverless Forwarder, with different payload sizes for each execution trigger. The size of the payload impacts on the number of events to be forwarded. Additional impacts are given by configuring <>, <> and <>. +The Elastic Serverless Forwarder can run for a maximum amount of time of 15 minutes. Different inputs can trigger the Elastic Serverless Forwarder, with different payload sizes for each execution trigger. The size of the payload impacts the number of events to be forwarded. Configuration settings may also impact the number of events, such as <>, <>, and <>. -The `Continuing queue` takes a role in ensuring at-least-once delivery for the scenario when the maximum amount of time of 15 minutes is not sufficient to forward all the events resulting from a single execution of the Elastic Serverless Forwarder. +The _Continuing queue_ helps ensure at-least-once delivery when the maximum amount of time (15 minutes) is not sufficient to forward all the events resulting from a single execution of the Elastic Serverless Forwarder. -For this a grace period of two minutes before reaching the 15 minutes timeout is reserved to handle the remaining events that are left to be processed. -At the beginning of this grace period events forwarding is halted. The rest of the time is dedicated to send a copy of the original messages that contains the remaining events to the `Continuing queue`. -This mechanism removes the necessity to handle partial processing of the trigger at inputs level, that's not always possible (for example in the case of <>), and that will require to force the users to abide to specific configuration of the {aws} resource used as inputs. +For this scenario, a grace period of two minutes is reserved at the end of the 15 minute timeout to handle any remaining events that have not been processed. +At the beginning of this grace period, event forwarding is halted. The remaining time is dedicated to sending a copy of the original messages that contains the remaining events to the _Continuing queue_. +This mechanism removes the need to handle partial processing of the trigger at the input level, which is not always possible (for example in the case of <>) or desirable because it forces users to conform to a specific configuration of the {aws} resources used as inputs. -The messages in the `Continuing queue` contain metadata related to the offsets where to restart forwarding the events from, and a reference to the original input. +Each message in the _Continuing queue_ contains metadata related to the last offset processed and a reference to the original input. -NOTE: You can remove a specific input as trigger of the Elastic Serverless Forwarder, but it is important to not remove its definition from <> as well, before ensuring that all the events generated while the input was still a trigger are fully processed, including the ones in the messages copied to the `Continuing queue`. The handling of the messages in the `Continuing queue` requires a lookup of the original input in the `config.yml`. +NOTE: You can remove a specific input as a trigger of the Elastic Serverless Forwarder. However, before removing its definition from the <>, ensure that all the events generated while the input was still a trigger are fully processed, including the ones in the messages copied to the _Continuing queue_. The handling of the messages in the _Continuing queue_ requires a lookup of the original input in the `config.yml`. -In the unlikely scenario that the Elastic Serverless Forwarder exceeds its maximum allocated execution time and is forcefully terminated, the `Continuing queues` will not be properly populated with a copy of the messages left to be processed, either all of them or only a portion of them. In this scenario the messages might or not might be lost according to the specific {aws} resource used as input and its specific configuration. +In the unlikely scenario that the Elastic Serverless Forwarder exceeds its maximum allocated execution time and is forcefully terminated, the _Continuing queue_ will not be properly populated with a copy of the messages left to be processed. In this scenario, all or a portion of the messages might be lost depending on the specific {aws} resource used as input and its configuration. -An {aws} SQS https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html[Dead Letter Queue] is created for the `Continuing Queue`. +An {aws} SQS https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html[Dead Letter Queue] is created for the _Continuing queue_. -When the Elastic Serverless Forwarder is triggered by the `Continuing Queue` and in the unlikely scenario that it exceeds its maximum allocated execution time and is forcefully terminated, all the messages in the payload that triggered the Elastic Serverless Forwarder execution will not be deleted from the `Continuing Queue` and will trigger another Elastic Serverless Forwarder execution. The `Continuing Queue` is configured for a number of 3 maximum receives before a message is sent to the DLQ. +When the Elastic Serverless Forwarder is triggered by the _Continuing queue_, in the unlikely scenario that it exceeds its maximum allocated execution time and is forcefully terminated, the messages in the payload that triggered the Elastic Serverless Forwarder execution are not deleted from the _Continuing queue_ and another Elastic Serverless Forwarder execution is triggered. The _Continuing queue_ is configured for a number of 3 maximum receives before a message is sent to the DLQ. [discrete] [[aws-serverless-forwarder-at-least-once-delivery-replay-queue]] == Replay queue The Elastic Serverless Forwarder forwards events to the outputs defined for a specific input. Events to be forwarded are grouped in batches that can be configured according to the specific output. -Failures can happen when forwarding events to an output. According to the output type the granularity of the failure can either be for the whole batch of events, or for single events in the batch. -There are multiple reasons for a failure to happen, including, but not limited, network connectivity, the output service being unavailable or under stress. +Failures can happen when forwarding events to an output. Depending on the output type, the granularity of the failure can either be for the whole batch of events, or for single events in the batch. +There are multiple reasons for a failure to happen, including, but not limited, network connectivity or the output service being unavailable or under stress. -The `Replay queue` takes a role in ensuring at-least-once delivery for the scenario when a failure in forwarding an event happens. +The _Replay queue_ helps ensure at-least-once delivery when a failure in forwarding an event happens. -For this, after any time a batch of events is forwarded, a copy of all the events in the batch that failed to be forwarded is sent to the `Replay queue`. Each message sent to the `Replay queue` contain exactly one event that failed to be forwarded. +For this scenario, after a batch of events is forwarded, a copy of all the events in the batch that failed to be forwarded is sent to the _Replay queue_. Each message sent to the _Replay queue_ contains exactly one event that failed to be forwarded. -It is possible to enable the `Replay queue` as a trigger of the Elastic Serverless Forwarder in order to forward again the events in the queue. +It is possible to enable the _Replay queue_ as a trigger of the Elastic Serverless Forwarder in order to forward the events in the queue again. -NOTE: It is left to the user to enable and disable the `Replay queue` as trigger of the Elastic Serverless Forwarder, according to the specific reason of why the forwarding failures happened. In most of the cases it is relevant to resolve the underlying issue causing the failures that populates the `Replay queue`, before trying to forward again the events in the queue. According to the nature and the impact of the issue, forwarding the events again without solving the issue before, will produce the outcome of new failures and events going back to the `Replay queue`. In some scenarios, like the output service being under stress, it is indeed suggested to disable the `Replay queue` as trigger of the Elastic Serverless Forwarder, since keeping forwarding the events could indeed worsen the issue. +NOTE: Before enabling or disabling the _Replay queue_ as a trigger of the Elastic Serverless Forwarder, consider the specific reason why the forwarding failures occurred. In most cases, you should resolve the underlying issue causing the failures before trying to forward the events in the queue again. Depending on the nature and impact of the issue, forwarding the events again without solving the problem may produce new failures and events going back to the _Replay queue_. In some scenarios, like the output service being under stress, it is recommended that you disable the _Replay queue_ as a trigger of the Elastic Serverless Forwarder, since continuing to forward the events could worsen the issue. -When the Elastic Serverless Forwarder is triggered by the `Replay queue`, if no event fails to be forwarded for a specific execution no message is explicitly deleted, the Elastic Serverless Forwarded execution succeeds and the messages in the trigger payload will be removed automatically from the `Replay queue`. -In the case any of the events fails again to be forwarded, all the messages in the trigger payload that contain events that didn't fail are deleted, and a specific expected exception is raised. This marks the Elastic Serverless Forwarded execution as failed with the outcome of having only the messages that failed again to go back to the `Replay queue`. +When the Elastic Serverless Forwarder is triggered by the _Replay queue_ and all events are successfully forwarded, the Elastic Serverless Forwarded execution succeeds, and the messages in the trigger payload are removed automatically from the _Replay queue_. -The messages in the `Replay queue` contain metadata with the references to the original input and the original output of the events. +However, if any events fail again to be forwarded, any messages in the trigger payload that contain events that didn't fail are deleted, and a specific expected exception is raised. The Elastic Serverless Forwarder execution is marked as failed, and any failed messages are sent back to the the _Replay queue_. -NOTE: You can remove a specific input as trigger of the Elastic Serverless Forwarder, but it is important to not remove its definition from <> as well, before ensuring that all the events failed to be ingested while the input was still a trigger are fully processed. The handling of the messages in the `Replay queue` requires a lookup of the original input and output in the `config.yml`. +The messages in the _Replay queue_ contain metadata with references to the original input and the original output of the events. +NOTE: You can remove a specific input as a trigger of the Elastic Serverless Forwarder. However, before removing its definition from <>, ensure that all the events that failed to be ingested while the input was still a trigger are fully processed. The handling of the messages in the _Replay queue_ requires a lookup of the original input and output in the `config.yml`. -An {aws} SQS https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html[Dead Letter Queue] is created for the `Replay Queue`. +An {aws} SQS https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html[Dead Letter Queue (DLQ)] is created for the _Replay queue_. -The same message can go back to the `Replay queue` up to three times, after that it will reach the configured number of 3 maximum receives, and it will be sent to the DLQ. -The same message can go back to the `Replay queue` either because it contains an event that failed again to be forwarded, according to the planned design, or in the unlikely scenario that the Elastic Serverless Forwarder triggered by the queue exceeds its maximum allocated execution time and is forcefully terminated. In this scenario the messages will not be lost and will eventually be sent to the DQL. +The same message can go back to the _Replay queue_ up to three times. After reaching the configured number of 3 maximum receives, the message will be sent to the DLQ. +The same message can go back to the _Replay queue_ either because it contains an event that failed again to be forwarded, according to the planned design, or in the unlikely scenario that the Elastic Serverless Forwarder triggered by the queue exceeds its maximum allocated execution time and is forcefully terminated. In this scenario the messages will not be lost and will eventually be sent to the DQL. [discrete] [[aws-serverless-forwarder-get-started]]