Skip to content

Commit

Permalink
suggested edits
Browse files Browse the repository at this point in the history
Signed-off-by: Amit Mor <[email protected]>
Signed-off-by: Amit Mor <[email protected]>
  • Loading branch information
amimimor committed Nov 14, 2023
1 parent 58570f1 commit 6a9ccf4
Showing 1 changed file with 10 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -136,11 +136,19 @@ When running the Dapr sidecar (`daprd`) with your application on EKS (AWS Kubern

#### SNS/SQS Contention with Dapr

Fundamentally, SNS aggregates messages from multiple publisher topics into a single SQS queue by creating SQS subscriptions to those topics. As a subscriber, the SNS/SQS PubSub component consumes messages from that sole SQS queue. However, like any SQS consumer, the component cannot selectively retrieve only messages published to specific SNS topics it is configured to subscribe to. This can result in the component receiving messages originating from topics without associated handlers. Typically this occurs during component initialization, if infrastructure subscriptions are ready before component subscription handlers, or during shutdown, if component handlers are removed before infrastructure subscriptions. Since this issue affects any SQS consumer of multiple SNS topics, the component cannot prevent consuming messages from topics lacking handlers. When this happens, the component logs an error indicating such messages were erroneously retrieved.
Fundamentally, SNS aggregates messages from multiple publisher topics into a single SQS queue by creating SQS subscriptions to those topics. As a subscriber, the SNS/SQS pub/sub component consumes messages from that sole SQS queue.

However, like any SQS consumer, the component cannot selectively retrieve the messages published to the SNS topics to which it is specifically subscribed. This can result in the component receiving messages originating from topics without associated handlers. Typically, this occurs during:

- **Component initialization:** If infrastructure subscriptions are ready before component subscription handlers, or
- **Shutdown:** If component handlers are removed before infrastructure subscriptions.

Since this issue affects any SQS consumer of multiple SNS topics, the component cannot prevent consuming messages from topics lacking handlers. When this happens, the component logs an error indicating such messages were erroneously retrieved.

In these situations, the unhandled messages would reappear in SQS with their [receive count](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html#sqs-receive-count) decremented after each pull. Thus, there is a risk that an unhandled message could exceed its `messageReceiveLimit` and be lost.

{{% alert title="Important" color="warning" %}}
It is therefore advisable to consider potential contention scenarios when using SNS/SQS with Dapr, and configure `messageReceiveLimit` appropriately. It is also highly recommended to use SQS dead-letter queues by setting `sqsDeadLettersQueueName` to prevent losing messages.
Consider potential contention scenarios when using SNS/SQS with Dapr, and configure `messageReceiveLimit` appropriately. It is highly recommended to use SQS dead-letter queues by setting `sqsDeadLettersQueueName` to prevent losing messages.
{{% /alert %}}

## Create an SNS/SQS instance
Expand Down

0 comments on commit 6a9ccf4

Please sign in to comment.