You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
At present data prepper seems to continually retry access to a S3 bucket, no matter what the error returned may be. This means that when using SQS messages may be continually returned to the queue and ultimately backlog the entire pipeline.
Describe the solution you'd like
I would like the ability to specify the action to be taken based off of the message returned from S3, when attempting to access the object.
Eg: a 403 being returned may be retry x times and if still failing put message back onto the queue, which with DLQ enabled should get the message into the DLQ. A no object found should result in a message delete and a log line created, eg object x not found in bucket y, message deleted
Note I haven't thought of all the failure scenarios available
Describe alternatives you've considered (Optional)
Is your feature request related to a problem? Please describe.
At present data prepper seems to continually retry access to a S3 bucket, no matter what the error returned may be. This means that when using SQS messages may be continually returned to the queue and ultimately backlog the entire pipeline.
Describe the solution you'd like
I would like the ability to specify the action to be taken based off of the message returned from S3, when attempting to access the object.
Eg: a 403 being returned may be retry x times and if still failing put message back onto the queue, which with DLQ enabled should get the message into the DLQ. A no object found should result in a message delete and a log line created, eg
object x not found in bucket y, message deleted
Note I haven't thought of all the failure scenarios available
Describe alternatives you've considered (Optional)
Additional context
Sourced from #3726
The text was updated successfully, but these errors were encountered: