-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
updated docs #588
updated docs #588
Changes from 4 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -79,6 +79,26 @@ The forwarder can ingest logs contained within the payload of an Amazon SQS body | |||||
|
||||||
You can set up a separate SQS queue for each type of log. The config parameter for {es} output `es_datastream_name` is mandatory. If this value is set to an {es} data stream, the type of log must be correctly defined with configuration parameters. A single configuration file can have many input sections, pointing to different SQS queues that match specific log types. | ||||||
|
||||||
[discrete] | ||||||
[[at-least-once-delivery]] | ||||||
== At-least-once delivery | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this section might be easier to digest if we split it in two, providing details for the two queues in two separate sections/sub-sections (e.g "More details about the Replay queue" or "Replay queue internals).
I would also add a note about the recent changes we made, removing |
||||||
|
||||||
Elastic Serverless Forwarder ensures "at-least-once" delivery of the forwarded message. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this statement should be in the overview, where the Replay queue and Continuing queue are introduced for the first time. |
||||||
|
||||||
The `Continuing queue` and `Replay queue` are programmatically populated by Elastic Serverless Forwarder. | ||||||
This happens for the `Continuing quque` in the last two minutes of the lambda execution, if any message in the original trigger is left to be processed before that time (including the original triggers and the continuing and replay queues themselves). | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Hmm does There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. no, |
||||||
In the case of the `Replay queue` at any time an event is failed to be ingested (excluding an event already routed to the `Replay queue` itself, in case it fails again) | ||||||
|
||||||
In the case of the `Continuing queue` Elastic Serverless Forwarder can fail to process the messages only if the lambda timeouts or if some unexpected exception is raised during the lambda execution. In all the other cases the original message that triggered the lambda is deleted from the `Continuing queue`, regardless the lambda processed it or not. If the message was not processed we send a copy of the original message to the `Continuing queue`: this has the effect of zeroing the number of the max retries and the mechanism is in place to enforce at-least-once delivery. It should be pretty rare to have messages in the continuing queue DLQ unless the lambda keeps timing out. | ||||||
|
||||||
In the case of the `Replay queue` the same considerations regarding lambda timeouts and unexpected exceptions apply. | ||||||
When processing messages from the `Replay queue`, at the end of the lambda execution, processed replayed messages that didn't fail to be ingested again will be deleted. Messages that fail to be ingested again won't be deleted, and a specific expected exception will be raised. This will mark the lambda as failed. This has the outcome of having only the messages that failed again to go back to the `Replay queue`. Being the same messages the retries count will increase, if it reaches the max value they will end up in the `Replay DLQ` as managed automatically by the {aws} SQS service. | ||||||
|
||||||
The corresponding DLQs are handled automatically by the SQS service in {aws}: there's the option to set the max retries of a queue, ie, the total amount of times the same message in a queue will trigger the lambda if the lambda won't successfully process the message. Once this limit is reached the message will go to the corresponding DLQ. | ||||||
|
||||||
For both `Continuing queue` and `Replay queue`, the max retries value is set to 3 | ||||||
|
||||||
|
||||||
[discrete] | ||||||
[[aws-serverless-forwarder-get-started]] | ||||||
= Get started | ||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be worth explaining why it is not possible and add a link to the AWS docs.
As additional note, I do not think this information is enough to close out #488 as there are more pieces involved in making ESF work with PrivateLink, apart from the VPC settings (e.g VPC endpoints)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
VPC endpoints are documented here
Let's move the content to this section! (and add the requirement for an EC2 VPC endpoint if cloudwatch logs are used as input)
I will add a link to https://www.elastic.co/guide/en/cloud/current/ec-traffic-filtering-vpc.html as well
do you have any hint about what @Udayel mentioned in #488?
I remember it was something related to a version bump of the ESF boto dependency, but I never hit the issue and I'm not sure if it still applies