-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add dead letter index for ES outputs #733
Conversation
If you plan to merge it with 729 pr, please update changelog and include this PR - as per comment #729 (comment) |
@emilioalvap I saw you tested the situation for when both data streams fail with a processor. Does it also happen if the dead letter index does not exist or if we pass an empty string |
@constanca-m As for an index that doesn't exist, that is usually handled by ES auto create policy, which tbh it's not something I've ever come across to be disabled. As for the empty value, it's the default value if not defined, so it should skip the dead letter index and continue straight into replay queue, as it is now. |
What does this PR do?
Depends on #729.
Closes https://github.com/elastic/enhancements/issues/21378.
Adds
es_dead_letter_index
option to ES outputs, with a similar mechanism as to mimic filebeat's non_indexable_policy.After failure to ingest on a bulk() request, ES output will now try to ingest to the provided dead_letter_index before queueing the failed event on the replay queue.
Checklist
CHANGELOG.md