Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow different actions based off of different S3 error codes #3804

Open
rhys-evans opened this issue Dec 4, 2023 · 0 comments
Open

Allow different actions based off of different S3 error codes #3804

rhys-evans opened this issue Dec 4, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@rhys-evans
Copy link

Is your feature request related to a problem? Please describe.
At present data prepper seems to continually retry access to a S3 bucket, no matter what the error returned may be. This means that when using SQS messages may be continually returned to the queue and ultimately backlog the entire pipeline.

Describe the solution you'd like
I would like the ability to specify the action to be taken based off of the message returned from S3, when attempting to access the object.

Eg: a 403 being returned may be retry x times and if still failing put message back onto the queue, which with DLQ enabled should get the message into the DLQ. A no object found should result in a message delete and a log line created, eg object x not found in bucket y, message deleted

Note I haven't thought of all the failure scenarios available

Describe alternatives you've considered (Optional)

Additional context
Sourced from #3726

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants