Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dead letter index for ES outputs #733

Merged
merged 18 commits into from
Jul 11, 2024

Conversation

emilioalvap
Copy link
Contributor

@emilioalvap emilioalvap commented Jul 3, 2024

What does this PR do?

Depends on #729.
Closes https://github.com/elastic/enhancements/issues/21378.

Adds es_dead_letter_index option to ES outputs, with a similar mechanism as to mimic filebeat's non_indexable_policy.

After failure to ingest on a bulk() request, ES output will now try to ingest to the provided dead_letter_index before queueing the failed event on the replay queue.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.md

@emilioalvap emilioalvap added the enhancement New feature or request label Jul 3, 2024
@emilioalvap emilioalvap requested review from gizas and constanca-m July 3, 2024 09:47
shippers/es.py Outdated Show resolved Hide resolved
@gizas
Copy link
Contributor

gizas commented Jul 9, 2024

If you plan to merge it with 729 pr, please update changelog and include this PR - as per comment #729 (comment)

@constanca-m
Copy link
Contributor

@emilioalvap I saw you tested the situation for when both data streams fail with a processor. Does it also happen if the dead letter index does not exist or if we pass an empty string as a value for it?

@emilioalvap
Copy link
Contributor Author

@constanca-m As for an index that doesn't exist, that is usually handled by ES auto create policy, which tbh it's not something I've ever come across to be disabled.
Even then with auto_create disabled, my understanding is that the _bulk request will return an index not found response (404) and report it back as an error for each individual action, like a mapping error or processor rejection would. So it would behave the same way as those types of errors and end up in the replay queue.

As for the empty value, it's the default value if not defined, so it should skip the dead letter index and continue straight into replay queue, as it is now.

@emilioalvap emilioalvap merged commit 1962405 into elastic:main Jul 11, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants