Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Edit ESF docs added by PR #588 #610

Merged
merged 3 commits into from
Feb 13, 2024
Merged

Edit ESF docs added by PR #588 #610

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
285b8cf
Edit ESF docs added by PR #588
dedemorton Feb 6, 2024
c1a1eb5
Resolve open questions
dedemorton Feb 8, 2024
bcc7d64
lint fix
Feb 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions docs/en/aws-deploy-elastic-serverless-forwarder.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ For `elasticsearch` the following arguments are supported:
* `args.es_datastream_name`: Name of data stream or index where logs should be forwarded to. Lambda supports automatic routing of various {aws} service logs to the corresponding data streams for further processing and storage in the {es} cluster. It supports automatic routing of `aws.cloudtrail`, `aws.cloudwatch_logs`, `aws.elb_logs`, `aws.firewall_logs`, `aws.vpcflow`, and `aws.waf` logs. For other log types, if using data streams, you can optionally set its value in the configuration file according to the naming convention for data streams and available integrations. If the `es_datastream_name` is not specified and it cannot be matched with any of the above {aws} services, then the value will be set to `logs-generic-default`. In versions **v0.29.1** and below, this configuration parameter was named `es_index_or_datastream_name`. Rename the configuration parameter to `es_datastream_name` in your `config.yaml` file on the S3 bucket to continue using it in the future version. The older name `es_index_or_datastream_name` is deprecated as of version **v0.30.0**. The related backward compatibility code is removed from version **v1.0.0**.
* `args.batch_max_actions`: (Optional) Maximum number of actions to send in a single bulk request. Default value: 500.
* `args.batch_max_bytes`: (Optional) Maximum size in bytes to send in a single bulk request. Default value: 10485760 (10MB).
* `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate.
* `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate.

For `logstash` the following arguments are supported:

Expand All @@ -234,7 +234,7 @@ For `logstash` the following arguments are supported:
* `args.password`: (Optional) Password of the {ls} instance to connect to. Mandatory if HTTP Basic authentication is enabled in {ls}.
* `args.max_batch_size`: (Optional) Maximum number of events to send in a single HTTP(s) request. Default value: 500
* `args.compression_level`: (Optional) The GZIP compression level for HTTP(s) requests towards {ls}. It can be any integer value between 1 (minimum compression, best performance, highest amount of bytes sent) and 9 (maximum compression, worst performance, lowest amount of bytes sent). Default value: 1
* `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate.
* `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate.

[discrete]
[[aws-serverless-forwarder-define-deploy-parameters]]
Expand Down Expand Up @@ -276,20 +276,20 @@ These parameters define the permissions required in order to access the associat

[discrete]
=== Network
The way to attach the Elastic Serverless Forwarder to a specific {aws} VPC is by defining both the security groups IDs and subnet IDs belonging to the {aws} VPC. The limit is related to the https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html[CloudFormation VPCConfig property]

To attach the Elastic Serverless Forwarder to a specific {aws} VPC, specify the security group IDs and subnet IDs that belong to the {aws} VPC. This requirement is related to the https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html[CloudFormation VPCConfig property].

These are the parameters:

- `ElasticServerlessForwarderSecurityGroups`: Add a comma delimited list of security group IDs to attach to the forwarder.
- `ElasticServerlessForwarderSubnets`: Add a comma delimited list of subnet IDs for to the forwarder.
- `ElasticServerlessForwarderSecurityGroups`: Add a comma delimited list of security group IDs to attach to the forwarder.
- `ElasticServerlessForwarderSubnets`: Add a comma delimited list of subnet IDs for the forwarder.

Both parameters are required in order to attach the Elastic Serverless Forwarder to a specific {aws} VPC.
Leave both parameters blank if you don't want the forwarder to belong to any specific {aws} VPC.

If the Elastic Serverless Forwarder is attached to a VPC, you need to https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html[create VPC endpoints] for S3 and SQS, and for *every* service you define as an input for the forwarder. S3 and SQS VPC endpoints are always required for reading the `config.yaml` uploaded to S3 and managing the continuing queue and the replay queue, regardless of the <<aws-serverless-forwarder-inputs>> used. If you use <<aws-serverless-forwarder-inputs-cloudwatch>>, you need to create a VPC endpoint for EC2, too.

If the Elastic Serverless Forwarder is attached to a VPC, you need to https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html[create VPC Endpoints] for S3 and SQS, and for *every* service you define as an input for the forwarder. S3 and SQS VPC Endpoints are always required for reading the `config.yaml` uploaded to S3 and managing the _Continuing queue_ and the _Replay queue_, regardless of the <<aws-serverless-forwarder-inputs>> used. If you use <<aws-serverless-forwarder-inputs-cloudwatch>> you need to create a VPC Endpoint for EC2 as well.

NOTE: Refer to documentation for https://www.elastic.co/guide/en/cloud/current/ec-traffic-filtering-vpc.html[AWS PrivateLink traffic filters] in order to find your VPC endpoint ID, and the hostname to use in the `config.yml` in order to access your Elasticsearch cluster over PrivateLink.
NOTE: Refer to the {cloud}/ec-traffic-filtering-vpc.html[AWS PrivateLink traffic filters] documentation to find your VPC endpoint ID and the hostname to use in the `config.yml` in order to access your Elasticsearch cluster over PrivateLink.

[discrete]
[[aws-serverless-forwarder-deploy-sar]]
Expand Down Expand Up @@ -543,9 +543,9 @@ continuing-queue:

| `s3-config-file` | Set this value to the location of your forwarder configuration file in S3 URL format: `s3://bucket-name/config-file-name`. This will populate the `S3_CONFIG_FILE` environment variable for the forwarder.

| `continuing-queue.batch_size` | Set this value above the default (`10`) if you experience ingestion delays in your output *and* `ApproximateNumberOfMessagesVisible` and `ApproximateAgeOfOldestMessage` SQS CloudWatch metrics for the _Continuing queue_ keep increasing *and* the average execution time of the forwarder is below 14 minutes. This will increase the number of messages the forwarder will process in a single execution for the _Continuing queue_.
| `continuing-queue.batch_size` | Set this value above the default (`10`) if you experience ingestion delays in your output *and* `ApproximateNumberOfMessagesVisible` and `ApproximateAgeOfOldestMessage` SQS CloudWatch metrics for the continuing queue keep increasing *and* the average execution time of the forwarder is below 14 minutes. This will increase the number of messages the forwarder will process in a single execution for the continuing queue.

| `continuing-queue.batching_window_in_second` | Set this value above the default (`0`) if you experience ingestion delays in your output *and* `ApproximateNumberOfMessagesVisible` and `ApproximateAgeOfOldestMessage` SQS CloudWatch metrics for the _Continuing queue_ keep increasing *and* the average execution time of the forwarder is below 14 minutes. This will increase the number of messages the forwarder will process in a single execution for the _Continuing queue_.
| `continuing-queue.batching_window_in_second` | Set this value above the default (`0`) if you experience ingestion delays in your output *and* `ApproximateNumberOfMessagesVisible` and `ApproximateAgeOfOldestMessage` SQS CloudWatch metrics for the continuing queue keep increasing *and* the average execution time of the forwarder is below 14 minutes. This will increase the number of messages the forwarder will process in a single execution for the continuing queue.

|===

Expand Down
Loading
Loading