-
Notifications
You must be signed in to change notification settings - Fork 372
Decouple concurrency and message batch size in SQS listener #379
Decouple concurrency and message batch size in SQS listener #379
Comments
Somewhat related: #166 |
I submitted a PR to support a per-queue |
Hi! This PR is really very good. Thanks. I'm just waiting to have it in my project - waiting for release 🔢 . I would like to give some suggestions based on my personal feeling when working with spring-cloud-aws, especially with SQS. Indeed I miss the maxConcurrency feature and it will be great to have it ASAP.
Sorry if this message is in wrong place. |
Hi @arseniir. I would like to ask you about your suggestion, because I have a situation that I have not been able to resolve, and I don't know if it is the same. I want to implement concurrency in the SqsListeners. I read an AWS documentation about that, but it's a bit old, so I don't know if I need to change my implementation, or if there is a way to set the concurrency attributes in the config class. Link: https://aws.amazon.com/blogs/developer/using-amazon-sqs-with-spring-boot-and-spring-jms/ The following is the implementation that I currently have, but I don't know how to set the concurrency attributes Config class:
Consumer class:
Do you know if there is already a way to do this? |
@juanledesma84 linked article doesn't use Spring Cloud AWS but amazon-sqs-java-messaging-lib |
For FIFO queues the AsynchronousMessageListener groups messages with same messageGroupId into so called MessageGroups. The MessageExecutor (renamed to MessageGroupExecutor) handles the messages within those groups sequentially. Messages from non-FIFO queues are handled as before with the only difference that they are also wrapped in a MessageGroup. Each separate message belongs to its own MessageGroup. Fixes spring-attic/spring-cloud-aws#387 Fixes spring-attic/spring-cloud-aws#379 Fixes spring-attic/spring-cloud-aws#530 Fixes spring-attic/spring-cloud-aws#756
For FIFO queues the AsynchronousMessageListener groups messages with same messageGroupId into so called MessageGroups. The MessageExecutor (renamed to MessageGroupExecutor) handles the messages within those groups sequentially. Messages from non-FIFO queues are handled as before with the only difference that they are also wrapped in a MessageGroup. Each separate message belongs to its own MessageGroup. Fixes spring-attic/spring-cloud-aws#387 Fixes spring-attic/spring-cloud-aws#379 Fixes spring-attic/spring-cloud-aws#530 Fixes spring-attic/spring-cloud-aws#756 Closes spring-attic/spring-cloud-aws#746 Co-authored-by: Tristan Baumbusch <[email protected]>
@maciejwalkowiak Doesn't really seem like awspring/spring-cloud-aws#40 addresses this. Seems like awspring/spring-cloud-aws#23 is probably the continuation of this? |
For FIFO queues the AsynchronousMessageListener groups messages with same messageGroupId into so called MessageGroups. The MessageExecutor (renamed to MessageGroupExecutor) handles the messages within those groups sequentially. Messages from non-FIFO queues are handled as before with the only difference that they are also wrapped in a MessageGroup. Each separate message belongs to its own MessageGroup. Fixes spring-attic/spring-cloud-aws#387 Fixes spring-attic/spring-cloud-aws#379 Fixes spring-attic/spring-cloud-aws#530 Fixes spring-attic/spring-cloud-aws#756 Closes spring-attic/spring-cloud-aws#746 Co-authored-by: Tristan Baumbusch <[email protected]>
For FIFO queues the AsynchronousMessageListener groups messages with same messageGroupId into so called MessageGroups. The MessageExecutor (renamed to MessageGroupExecutor) handles the messages within those groups sequentially. Messages from non-FIFO queues are handled as before with the only difference that they are also wrapped in a MessageGroup. Each separate message belongs to its own MessageGroup. Fixes spring-attic/spring-cloud-aws#387 Fixes spring-attic/spring-cloud-aws#379 Fixes spring-attic/spring-cloud-aws#530 Fixes spring-attic/spring-cloud-aws#756 Closes spring-attic/spring-cloud-aws#746 Co-authored-by: Tristan Baumbusch <[email protected]>
Enhancement
The
SimpleMessageListenerContainer
has a basic concurrency model that works as follows. While the queue is running:maxNumberOfMessages
messages from SQSThis approach is simple but has the disadvantage of coupling message processing concurrency and message batch size. There is no easy way to request multiple messages at once but only process one at a time. (This can potentially be achieved by configuring a custom task executor, but that doesn't work well with more than one queue because all queues share the same executor.) Likewise, there is no way to handle more than 10 messages at a time (the maximum SQS batch size).
As an enhancement, I would like to request support for the following use cases on a per-queue basis:
maxConcurrency < maxNumberOfMessages
: Request n messages at a time but limit concurrent processing to m < n.maxConcurrency >= maxNumberOfMessages
: Request n messages at a time and allow concurrent processing of up to m >= n total messages.The text was updated successfully, but these errors were encountered: