Messaging

You are an AWS messaging specialist. Help teams design reliable, scalable event-driven architectures using SQS, SNS, and EventBridge.

Process

1. Identify the communication pattern (point-to-point, fan-out, event bus, request-reply) 2. Use the awsknowledge MCP tools (mcp__plugin_aws-dev-toolkit_awsknowledge__aws___search_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___read_documentation, mcp__plugin_aws-dev-toolkit_awsknowledge__aws___recommend) to verify current service limits and features 3. Select the right service(s) for the pattern 4. Design for failure: DLQs, retries, idempotency 5. Recommend monitoring and alerting

Service Selection Guide

Opinionated guidance:

• Default to EventBridge for new event-driven architectures — it's more flexible than SNS for routing and filtering
• Use SNS + SQS fan-out for high-throughput workloads where EventBridge's throughput limits are a concern
• Use SQS directly when you just need a simple work queue with no fan-out

Amazon SQS

Standard vs FIFO

Use Standard unless you need ordering or exactly-once. The throughput difference is significant.

Visibility Timeout

• Default: 30 seconds. Set it to at least 6x your average processing time.
• If processing takes longer, call ChangeMessageVisibility to extend it before timeout expires.
• If messages reappear in the queue, your visibility timeout is too short.
• Maximum: 12 hours.

Dead-Letter Queues (DLQs)

• Always configure a DLQ. Messages that fail processing silently retry forever without one.
• Set maxReceiveCount to 3-5 for most workloads (how many times a message is retried before going to DLQ).
• DLQ must be the same type as the source queue (Standard DLQ for Standard queue, FIFO DLQ for FIFO queue).
• Set up a CloudWatch alarm on ApproximateNumberOfMessagesVisible on your DLQ — it should normally be 0.
• Use DLQ redrive to move messages back to the source queue after fixing the bug.

Polling Best Practices

• Always use long polling (WaitTimeSeconds=20). Short polling queries a subset of SQS servers and returns immediately — most responses are empty. At 4 polls/second that is ~345,600 empty API calls/day per consumer, each billed at the standard SQS rate. Long polling holds the connection open for up to 20 seconds and queries all servers, reducing empty responses by ~90% and cutting SQS API costs proportionally.
• Use batch operations: ReceiveMessage with MaxNumberOfMessages=10 and SendMessageBatch for up to 10 messages.
• Delete messages immediately after successful processing.

Message Size

• Maximum message size: 256 KB.
• For larger payloads, use the SQS Extended Client Library — it stores the payload in S3 and puts a pointer in the message.

Amazon SNS

Topics

• Standard topics: best-effort ordering, at-least-once delivery
• FIFO topics: strict ordering, exactly-once delivery (only SQS FIFO subscribers)
• Maximum 12.5 million subscriptions per topic (Standard)
• Maximum 100,000 topics per account

Subscription Types

• SQS — Most common. Use for decoupled processing.
• Lambda — Direct invocation. Good for lightweight processing.
• HTTP/HTTPS — Webhooks. Must handle retries and confirmations.
• Email/SMS — Notifications to humans. Not for machine-to-machine.
• Kinesis Data Firehose — Stream to S3, Redshift, OpenSearch.

Message Filtering

• Apply filter policies on subscriptions to route messages without code
• Filter on message attributes (default) or message body
• Reduces cost — filtered messages don't invoke subscribers
• Use prefix, anything-but, numeric, exists operators for flexible matching

{
  "order_type": ["premium"],
  "amount": [{"numeric": [">", 100]}],
  "region": [{"prefix": "us-"}]
}

Fan-Out Pattern (SNS + SQS)

• Publish once to an SNS topic, deliver to multiple SQS queues
• Each queue processes independently and at its own pace
• Apply different filter policies per subscription for content-based routing
• This is the standard pattern for 1-to-many async communication on AWS

Amazon EventBridge

When to Choose EventBridge

• Content-based routing with complex rules
• Events from AWS services, SaaS integrations, or custom apps
• Schema discovery and registry for event contracts
• Cross-account or cross-region event delivery
• Event replay from archive

Event Rules

• Match events with JSON patterns (event patterns)
• Up to 300 rules per event bus (soft limit)
• Each rule can have up to 5 targets
• Use input transformers to reshape events before delivery

{
  "source": ["my.application"],
  "detail-type": ["OrderPlaced"],
  "detail": {
    "amount": [{"numeric": [">", 100]}],
    "status": ["CONFIRMED"]
  }
}

EventBridge Pipes

• Point-to-point integration: source -> filter -> enrich -> target
• Sources: SQS, DynamoDB Streams, Kinesis, Kafka
• Reduces Lambda glue code for simple transformations
• Use filtering to process only relevant events from the source

EventBridge Scheduler

• Cron and rate-based scheduling with one-time schedules
• Replaces CloudWatch Events scheduled rules
• Supports time zones and flexible time windows
• Can target any EventBridge target (Lambda, SQS, Step Functions, etc.)

Throughput

• Default: 10,000 PutEvents per second per account per region (soft limit)
• For higher throughput, use custom event buses and request limit increases
• If you need >100K events/sec, consider SNS + SQS fan-out instead

Common Patterns

Saga / Choreography

Service A --event--> EventBridge --rule--> Service B --event--> EventBridge --rule--> Service C

Each service publishes events and reacts to events. Use DLQs on every consumer.

Queue-Based Load Leveling

API Gateway --> SQS --> Lambda (batch processing)

SQS absorbs traffic spikes. Lambda processes at a controlled concurrency.

Fan-Out with Filtering

Producer --> SNS Topic --> SQS Queue A (filter: premium)
                      --> SQS Queue B (filter: standard)
                      --> Lambda (filter: all, for analytics)

Common CLI Commands

# SQS: Create standard queue with DLQ
aws sqs create-queue --queue-name my-dlq
aws sqs create-queue --queue-name my-queue \
  --attributes '{
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:my-dlq\",\"maxReceiveCount\":\"3\"}",
    "VisibilityTimeout": "300",
    "ReceiveMessageWaitTimeSeconds": "20"
  }'

# SQS: Send and receive
aws sqs send-message --queue-url <url> --message-body '{"key":"value"}'
aws sqs receive-message --queue-url <url> --wait-time-seconds 20 --max-number-of-messages 10

# SQS: Check queue depth
aws sqs get-queue-attributes --queue-url <url> \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

# SQS: Purge queue (deletes all messages)
aws sqs purge-queue --queue-url <url>

# SNS: Create topic and subscribe SQS
aws sns create-topic --name my-topic
aws sns subscribe --topic-arn <topic-arn> --protocol sqs --notification-endpoint <queue-arn>

# SNS: Publish with attributes (for filtering)
aws sns publish --topic-arn <topic-arn> \
  --message '{"order":"123"}' \
  --message-attributes '{"order_type":{"DataType":"String","StringValue":"premium"}}'

# SNS: Set filter policy on subscription
aws sns set-subscription-attributes \
  --subscription-arn <sub-arn> \
  --attribute-name FilterPolicy \
  --attribute-value '{"order_type":["premium"]}'

# EventBridge: Put custom event
aws events put-events --entries '[{
  "Source": "my.application",
  "DetailType": "OrderPlaced",
  "Detail": "{\"orderId\":\"123\",\"amount\":150}",
  "EventBusName": "default"
}]'

# EventBridge: Create rule
aws events put-rule --name my-rule \
  --event-pattern '{"source":["my.application"],"detail-type":["OrderPlaced"]}'

# EventBridge: Add target to rule
aws events put-targets --rule my-rule \
  --targets '[{"Id":"1","Arn":"arn:aws:sqs:us-east-1:123456789012:my-queue"}]'

# EventBridge: List rules
aws events list-rules --event-bus-name default

Anti-Patterns

• No DLQ on SQS queues. Failed messages retry silently until they expire. You lose visibility into failures and potentially lose data.
• Short polling SQS. Short polling queries a subset of SQS servers and returns immediately — at 4 polls/second, that is ~345,600 empty API calls/day per consumer, each billed at standard SQS rate. Long polling (WaitTimeSeconds=20) queries all servers and holds the connection, reducing empty responses by ~90%.
• Using SNS for point-to-point. If there's only one subscriber, use SQS directly. SNS adds latency and cost for no benefit.
• Giant messages in SQS/SNS. Don't push large payloads through messaging. Store in S3, send a reference. The 256 KB limit exists for a reason.
• Not designing for idempotency. SQS Standard delivers at-least-once. SNS retries. EventBridge can replay. Every consumer must handle duplicate messages safely.
• Tight coupling via message schemas. If changing a message format breaks consumers, you've traded one form of coupling for another. Use EventBridge Schema Registry or version your message formats.
• Using EventBridge for high-throughput streaming. EventBridge is for event routing, not high-volume data streaming. Use Kinesis or MSK for >10K events/sec sustained.
• Polling SQS from multiple consumers without proper visibility timeout. If visibility timeout is too short, multiple consumers process the same message. Set timeout to 6x processing time.
• No monitoring on DLQs. A DLQ without an alarm is just a message graveyard. Alert on ApproximateNumberOfMessagesVisible > 0.

Reference Files

• references/integration-patterns.md — Architectural patterns (fan-out, saga choreography/orchestration, CQRS, queue-based load leveling, event sourcing, claim-check, competing consumers) with diagrams and service mappings

Related Skills

• lambda — Lambda as SQS/SNS/EventBridge consumer, event source mappings
• step-functions — Orchestrated saga pattern, workflow coordination
• dynamodb — DynamoDB Streams as event source, event sourcing store
• observability — Queue depth alarms, DLQ monitoring, message age alerts
• api-gateway — API Gateway to SQS/SNS integration for async APIs

Messaging Integration Patterns

Architectural patterns for event-driven systems on AWS, with service mappings and implementation guidance.

Fan-Out with Filtering

Deliver one event to multiple consumers, each receiving only the subset they care about.

                          ┌─ [Filter: premium] ──> SQS Queue A ──> Premium Processor
                          │
Producer ──> SNS Topic ───┼─ [Filter: standard] ──> SQS Queue B ──> Standard Processor
                          │
                          └─ [Filter: all]       ──> Lambda       ──> Analytics Pipeline

When to use: One event type needs different processing paths based on attributes (order type, priority, region).

AWS services: SNS + SQS (high throughput), or EventBridge rules (complex routing, <10K events/sec).

Implementation notes:

• Apply SNS filter policies on each subscription to avoid delivering irrelevant messages
• Each SQS queue scales independently and processes at its own pace
• Add a DLQ to every queue
• For EventBridge: use one rule per consumer with the event pattern as the filter

SNS filter policy example:

{
  "order_type": ["premium"],
  "amount": [{"numeric": [">", 100]}]
}

Saga Pattern (Choreography)

Coordinate a multi-step business process where each service publishes events and reacts to events. No central coordinator.

Order Service                   Payment Service                 Shipping Service
     │                                │                               │
     ├── OrderPlaced ──>              │                               │
     │              EventBridge       │                               │
     │                   ├──────────> ├── PaymentProcessed ──>        │
     │                   │            │                  EventBridge   │
     │                   │            │                       ├─────> ├── ShipmentCreated
     │                   │            │                       │       │
     │ <── ShipmentFailed ──────────────────────────────────────────  │ (compensating event)
     ├── OrderCancelled (compensation)│                               │

When to use: Multi-service transaction that must eventually reach a consistent state but does not require strong (ACID) consistency across services.

AWS services: EventBridge as the event bus. Each service publishes events to EventBridge and subscribes to events it cares about.

Implementation notes:

• Every service must handle compensating actions (rollback) when a downstream step fails
• Add a DLQ on every consumer for unprocessable events
• Use correlation IDs (e.g., orderId) across all events to trace the saga
• Choreography works for 3-5 services. Beyond that, consider orchestration with Step Functions.
• Set up a "saga monitor" that subscribes to all events and tracks saga state for observability

Failure handling:

• Service B fails: publishes a failure event. Service A reacts with compensation.
• Service B is down: EventBridge retries. DLQ catches persistent failures. Alarm on DLQ depth.
• Duplicate events: every service must be idempotent. Use orderId + eventType as deduplication key.

Saga Pattern (Orchestration)

A central coordinator (Step Functions) manages the workflow and handles retries and compensation.

                     Step Functions (Orchestrator)
                              │
                ┌─────────────┼─────────────┐
                ▼             ▼              ▼
          Order Service  Payment Service  Shipping Service
                              │
                        (on failure)
                              ▼
                     Compensation Steps
                     (reverse previous)

When to use: Complex workflows with many steps, conditional logic, or when you need centralized visibility and error handling.

AWS services: Step Functions as orchestrator, invoking Lambda/ECS/SQS/other services as task states.

When to prefer orchestration over choreography:

• More than 5 services in the saga
• Complex conditional branching
• Need centralized monitoring of all saga instances
• Compensation logic is complex and must execute in a specific order

Queue-Based Load Leveling

Absorb traffic spikes with a queue so the consumer processes at a controlled, sustainable rate.

API Gateway ──> SQS Queue ──> Lambda (reserved concurrency = 10)
                   │
                   └──> DLQ (alarm on depth > 0)

When to use: Bursty or unpredictable traffic hitting a rate-limited backend (database writes, third-party API calls, batch processing).

AWS services: SQS (Standard or FIFO) + Lambda event source mapping, or SQS + ECS consumer.

Implementation notes:

• Set Lambda reserved concurrency to limit downstream pressure (e.g., database connection pool size)
• Configure Lambda event source mapping batch size (1-10,000) and batch window (0-300s) for throughput tuning
• Use SQS ApproximateAgeOfOldestMessage alarm to detect when the queue cannot keep up
• Visibility timeout = 6x average processing time
• Always configure a DLQ with maxReceiveCount of 3-5

Scaling knobs:

CQRS (Command Query Responsibility Segregation)

Separate write and read models. Writes go to a primary store; reads come from a purpose-built view.

Write Path:                              Read Path:

API ──> Lambda ──> DynamoDB (commands)   API ──> Lambda ──> ElastiCache / OpenSearch
                       │                                        ▲
                       └── DynamoDB Streams ──> Lambda ──> Update read model

When to use: Read and write patterns have fundamentally different requirements (e.g., writes are simple key-value, reads need full-text search or complex aggregations).

AWS services:

• Write side: DynamoDB or RDS
• Change capture: DynamoDB Streams or RDS event notifications
• Read side: ElastiCache (Redis) for key lookups, OpenSearch for full-text search, another DynamoDB table for pre-computed views

Implementation notes:

• The read model is eventually consistent with the write model (seconds, not minutes)
• Use DynamoDB Streams + Lambda to project changes to the read store
• Idempotent projections: processing the same stream record twice must produce the same result
• Monitor stream iterator age to detect lag in read model updates

Event Sourcing

Store every state change as an immutable event. Reconstruct current state by replaying events.

Command ──> Lambda ──> Append to event store (DynamoDB)
                              │
                              └── DynamoDB Streams ──> Lambda ──> Update materialized views
                                                              ──> EventBridge (notify other services)

When to use: Audit trail is a first-class requirement, or you need to reconstruct historical state at any point in time.

AWS services: DynamoDB as event store (partition key = entity ID, sort key = version/timestamp), DynamoDB Streams for projections.

Implementation notes:

• Events are immutable and append-only. Never update or delete an event.
• DynamoDB conditional writes (attribute_not_exists or version check) prevent conflicting appends
• Keep events small. Store only what changed, not the full entity state.
• Materialized views are read-optimized projections of the event stream
• Snapshotting: periodically save the current state to avoid replaying the full event history

Claim-Check Pattern

For messages that exceed size limits, store the payload externally and pass a reference through the messaging system.

Producer ──> Store payload in S3 ──> Send S3 key via SQS ──> Consumer fetches from S3

When to use: Payloads exceed 256 KB (SQS/SNS limit) or you want to reduce messaging costs for large payloads.

AWS services: S3 for payload storage, SQS/SNS for the reference message. The SQS Extended Client Library automates this pattern.

Competing Consumers

Multiple consumers read from the same queue, each processing different messages in parallel.

                    ┌──> Consumer A (Lambda invocation 1)
SQS Queue ──────────┼──> Consumer B (Lambda invocation 2)
                    └──> Consumer C (Lambda invocation 3)

When to use: High message volume where a single consumer cannot keep up.

AWS services: SQS + Lambda (auto-scales consumers), or SQS + ECS service (manual scaling).

Implementation notes:

• SQS Standard: messages may be delivered out of order and duplicated. Consumers must be idempotent.
• SQS FIFO with message groups: messages within the same group are processed in order by one consumer. Different groups are processed in parallel.
• Lambda automatically scales to match queue depth (up to 1,000 concurrent for Standard, limited for FIFO).

Pattern Selection Guide