Event Driven Architecture

Event-Driven Architecture

When to Use

• Design event-driven integration between bounded contexts or microservices
• Choose pub/sub vs point-to-point, topics vs queues, and broker capabilities
• Model domain events, integration events, and command vs event semantics
• Apply event sourcing or CQRS at architecture level (not framework tutorials only)
• Define schemas, compatibility rules, and event schema versioning strategy
• Specify ordering, partition keys, and delivery guarantees (at-least-once, etc.)
• Design idempotent consumers, deduplication, and exactly-once tradeoffs
• Implement transactional outbox or inbox for reliable publish/consume
• Decide choreography vs orchestration and saga compensation at pattern level
• Operate dead-letter queues, replay, and stream reprocessing safely
• Distinguish stream processing (Kafka Streams, Flink) from discrete business events

When NOT to Use

• Implement microservice code, gRPC/REST APIs, or twelve-factor deployables only → microservices-developer
• Design enterprise iPaaS hubs, canonical models, B2B partner APIs, or API gateway programs only → enterprise-integration-api-developer
• Map X12/EDIFACT segments, AS2/VAN, or EDI partner certification → edi-engineer
• Build generic application features without event/messaging architecture → senior-software-engineer
• Operate classified air-gapped pipelines, ATO evidence, cleared promotion → classified-software-devsecops-engineer
• Formulate VRP, MIP, scheduling, or optimization solvers → operations-research-algorithm-developer
• CI/CD YAML, GitOps, and release automation only → devops
• Landing zone, VPC, and managed cloud provisioning → cloud-engineer

Related skills

Core Workflows

1. Scope and event boundaries

Define event types, producers/consumers, sync vs async boundaries, and non-goals.

See `references/event_driven_architecture_scope.md`.

2. Messaging and brokers

Select patterns, topics/queues, partitioning, and broker fit (Kafka, Pulsar, SNS/SQS, etc.).

See `references/messaging_patterns_and_brokers.md`.

3. Event sourcing and CQRS

When to use write models, projections, snapshots, and read-model consistency.

See `references/event_sourcing_and_cqrs.md`.

4. Reliability, idempotency, outbox

Delivery semantics, deduplication, outbox/inbox, and failure handling.

See `references/reliability_idempotency_outbox.md`.

5. Orchestration, choreography, sagas

Coordinate long-running flows without turning the bus into a distributed monolith.

See `references/orchestration_choreography_sagas.md`.

6. Schema governance and operations

Version events, operate DLQs, replay, and observability for event pipelines.

See `references/schema_governance_operations.md`.

Outputs

• Event catalog — event name, schema version, producer, consumers, SLAs, PII classification
• Context diagram — services, topics/queues, sync fallbacks, trust zones
• Consistency note — outbox/saga/choreography choice with compensation and idempotency keys
• Partitioning and ordering spec — keys, guarantees, hot-partition risks
• Schema compatibility matrix — backward/forward rules, deprecation timeline
• Operations runbook — DLQ drain, replay procedure, lag alerts, poison-message handling

Principles

• Prefer explicit contracts (schemas, AsyncAPI/CloudEvents metadata) over implicit JSON blobs
• Design consumers idempotent by default; treat exactly-once as a bounded, measured goal
• Keep commands and events distinct—events are facts; commands request work
• Avoid chatty orchestration over the bus; sagas compensate, they do not hide missing boundaries
• Make failure observable—correlation ID, structured errors, DLQ with actionable payloads
• Replay is a product feature—document ordering, side effects, and deduplication before reprocessing

Event-driven architecture scope

Table of contents

1. Role focus 2. Event taxonomy 3. Typical deliverables 4. Decision boundaries 5. Anti-patterns

Role focus

Event taxonomy

Rules:

• Events are immutable facts (past tense); commands are intent (imperative).
• Do not publish large payloads when an id + schema version suffices.
• Classify PII/regulated fields in the catalog; avoid broadcasting secrets.

Typical deliverables

• Event catalog (name, version, owner, schema registry id, retention)
• Producer/consumer matrix with delivery expectation and idempotency key
• Sequence diagrams for critical flows (happy path + compensation)
• ADR: sync vs async, broker choice, choreography vs orchestration
• Non-functional table: throughput, lag SLO, ordering needs, replay policy

Decision boundaries

Prefer event-driven integration when:

• Multiple subscribers need the same fact without tight coupling
• Peak load must be absorbed (buffering) or systems have different availability
• Audit trail and temporal queries benefit from append-only history
• Teams can own schemas and evolve consumers independently

Prefer sync (RPC/REST) when:

• Strong immediate consistency required on user-facing path
• Simple request/response with one consumer and low latency budget
• Operation is a query with no side effect to broadcast

Prefer event sourcing when:

• Complete history is a product requirement (finance, compliance, dispute)
• Temporal queries and rebuild-from-history are common
• Team can invest in projections, snapshots, and operational complexity

Anti-patterns

• Event notification as integration database — giant payloads replicated everywhere
• Distributed monolith via events — all services must deploy together for schema changes
• Missing ownership — no team owns topic schema or consumer lag
• Fire-and-forget without DLQ — poison messages block partitions indefinitely
• Synchronous chains disguised as events — request-reply over topics without timeouts

Event sourcing and CQRS

Table of contents

1. Concepts 2. When to apply 3. Write model and event store 4. Read models and projections 5. Consistency and snapshots

Concepts

Event sourcing answers what happened. CQRS answers how we query efficiently.

When to apply

Strong fit:

• Audit, dispute, regulatory replay requirements
• Complex domain with rich history (orders, ledger, workflows)
• Multiple read shapes over same write stream (reports, search, APIs)

Weak fit:

• Simple CRUD with one read/write shape
• Team lacks ops maturity for projections and replay drills
• Strong cross-aggregate ACID on every user click

CQRS without event sourcing is valid—multiple read DBs fed by integration events.

Write model and event store

Aggregate rules:

• Commands validate against current aggregate state (loaded from events or snapshot)
• Emit one or more domain events per successful command
• Enforce invariants inside aggregate boundary—no cross-aggregate transactions

Event store options:

• Dedicated event store (EventStoreDB, custom append log)
• Kafka topic as log (with compaction for keyed aggregates)
• RDBMS events table with monotonic version per aggregate id

Versioning: each event carries aggregateId, version, eventType, schemaVersion.

Read models and projections

Idempotent projection handlers:

• Key by (aggregateId, version) or event id
• Store last processed position in inbox/offset table
• Design projections to tolerate at-least-once delivery

Consistency and snapshots

• Eventual consistency between write and read is normal—UI must handle stale reads or use read-your-writes patterns
• Snapshots every N events reduce replay time; snapshot + tail events on load
• Global ordering across aggregates is usually unnecessary; per-aggregate version is enough
• Deletion/GDPR: legal hold and tombstone strategies must be designed upfront—not bolted on

Integration with microservices:

• Publish integration events at bounded context edge; do not expose raw internal event store to all consumers
• See microservices-developer for service ownership; this skill covers pattern choice

Messaging patterns and brokers

Table of contents

1. Pattern selection 2. Pub/sub vs point-to-point 3. Partitioning and ordering 4. Broker comparison 5. Kafka-oriented notes

Pattern selection

Pub/sub vs point-to-point

Pub/sub (topics):

• Log-based retention (Kafka, Pulsar) enables replay and multiple consumer groups
• Ordering is per partition only—choose partition key deliberately
• Consumers in same group share load; different groups read independently

Point-to-point (queues):

• Message removed (or hidden) after ack—typical for task queues
• Competing consumers scale horizontally; use visibility timeout and idempotency
• Dead-letter after max receives (SQS) or retry policy (Rabbit)

Partitioning and ordering

Hot partitions: monitor skew; split keys or salt high-volume tenants.

Broker comparison

Pick broker based on retention, ordering, ops model, and team skill—not hype.

Kafka-oriented notes

When users mention Kafka events:

• Topic = contract boundary; partition = parallelism + ordering unit
• Consumer group = scaling unit; lag per partition is the alert signal
• Compaction for keyed changelog topics (CQRS projections), not all domain topics
• Headers for correlation id, schema id, trace context, idempotency key
• Prefer Schema Registry (Avro/Protobuf/JSON Schema) over undocumented JSON

Stream processing vs discrete events:

• Discrete business events — one fact per message (OrderPlaced)
• Stream processing — continuous transforms, windows, joins (Kafka Streams, Flink)
• Use streams for analytics/enrichment; use domain events for integration contracts

Orchestration, choreography, and sagas

Table of contents

1. Choreography vs orchestration 2. Saga pattern 3. Compensation 4. State machines and timeouts 5. Anti-patterns

Choreography vs orchestration

Choose choreography when:

• Few steps, clear event chain, each service owns compensation
• Flow rarely changes; observability via correlation id is enough

Choose orchestration when:

• Many steps, timeouts, human tasks, or conditional branches
• Need central view of saga state for support and audits
• Saga orchestration with explicit state machine (Temporal, Camunda, custom)

Event choreography is not “no design”—document the expected event chain and failure branches.

Saga pattern

A saga is a sequence of local transactions coordinated by events or an orchestrator.

Rules:

• Each step has a compensating action (semantic undo, not always DB rollback)
• Sagas are long-running—design for partial completion
• Never assume ACID across services; use eventual consistency

Compensation

Compensations are business operations, not generic rollbacks.

Document in saga spec:

• Which steps are retryable vs non-retryable
• Pivot transaction (point of no return) after which compensation changes
• Idempotency keys for compensate commands

State machines and timeouts

Orchestrator should track:

• Current step, started_at, deadline
• Correlation id, business id (orderId)
• Failure reason and last compensating step attempted

Timeouts:

• Per-step SLA; escalate to human or compensating path
• Use delay queues or scheduler (not busy-loop polling topics)
• Distinguish technical failure (retry) vs business rejection (compensate)

Observability:

• Emit SagaStarted, SagaStepCompleted, SagaFailed, SagaCompensated integration events
• Trace span per step with same correlation id

Anti-patterns

• God topic — every service listens to everything
• Synchronous saga — blocking HTTP chain labeled “async”
• Missing compensation — forward-only flows in money-moving domains
• Orchestrator in the database — stored procedures driving cross-service flow
• Infinite retry on business validation errors (poison without DLQ)

For enterprise hub-style orchestration (iPaaS), see enterprise-integration-api-developer.

Reliability, idempotency, and outbox

Table of contents

1. Delivery guarantees 2. Idempotent consumer 3. Exactly-once semantics 4. Transactional outbox 5. Inbox pattern 6. Failure handling

Delivery guarantees

Brokers usually provide at-least-once. Exactly-once processing is an application concern.

Idempotent consumer

Idempotency key sources:

• Business key: paymentId, orderId + operation
• Broker metadata: message id, offset (weaker for redelivery across topics)
• Event envelope: eventId UUID (preferred for integration events)

Implementation checklist:

1. Persist processed keys in dedup table (TTL ≥ max redelivery window) 2. Make downstream writes upserts or compare-and-set on version 3. Return success on duplicate key (do not double-charge, double-ship) 4. Log duplicate at debug/metric—not error storm

Side-effectful operations: use outbox + single-writer or external idempotent APIs (Stripe idempotency keys).

Exactly-once semantics

True end-to-end exactly-once across services is expensive and rare.

Pragmatic approach:

• Kafka transactional produce + idempotent producer (broker scope)
• Consume-transform-produce with transactions (stream apps)
• Outbox + idempotent consumer for cross-service (most common)

Document which layer provides which guarantee in the ADR.

Transactional outbox

Problem: DB commit and message publish must not diverge.

Pattern:

1. In same DB transaction: update business rows + insert row into outbox table 2. Relay process polls outbox and publishes to broker 3. Mark outbox row published (or delete) after broker ack

Relay options:

• Polling publisher (simple; watch DB load)
• Log-based CDC (Debezium) reading outbox table or WAL
• In-process relay (only if crash safety understood)

Ordering: publish in outbox insertion order per aggregate if consumers depend on it.

Inbox pattern

For incoming messages:

• Store message id in inbox in same transaction as domain update
• Skip processing if id already present
• Complements idempotency when handler has multiple steps

Failure handling

Retry storms: cap retries; use circuit breaker on publisher relay.

Ordering + retry: retries may reorder—consumers must tolerate or use partition stickiness.

Schema governance and operations

Table of contents

1. Schema design 2. Event schema versioning 3. Registry and compatibility 4. Dead-letter queues 5. Replay and reprocessing 6. Observability

Schema design

Envelope fields (recommended):

• eventId, eventType, schemaVersion, occurredAt
• correlationId, causationId, producer
• payload (versioned body)

Conventions:

• Use past-tense event names; version in type or separate field (order.placed.v1)
• Prefer Avro/Protobuf/JSON Schema over schemaless JSON for integration topics
• Document required vs optional fields and default semantics
• Align with CloudEvents where teams already standardize on it

Event schema versioning

Strategies:

1. Single topic, multiple schema versions in registry (Confluent BACKWARD/FULL) 2. New topic per major version (orders.v1 → orders.v2) for clean cutover 3. Dual-write during migration; consumers upgrade before producer drops old fields

Publish deprecation timeline in event catalog; never break consumers without notice.

Registry and compatibility

CI gates:

• Schema diff on PR
• Compatibility check against registry rules
• Contract tests for sample payloads

Dead-letter queues

When to DLQ:

• Deserialization failure (poison schema)
• Business rule rejection after max retries
• Handler bug (fix forward, then replay)

DLQ operations:

• Retain original headers, payload, stack trace, failure reason
• Alert on DLQ depth threshold
• Drain procedure: fix consumer → replay to main topic or dedicated replay topic
• Access control on DLQ (may contain PII)

Kafka: separate topic orders.dlq or use retry topics with backoff tiers.

Replay and reprocessing

Use cases:

• New projection/read model
• Bug fix in consumer logic
• Regulatory audit export

Checklist before replay:

1. Confirm idempotency and dedup window cover replay window 2. Define offset/time range and ordering expectations 3. Isolate side effects (emails, charges)—use dry-run or sandbox 4. Rate-limit replay to protect downstream DBs 5. Metric: replay_lag, duplicate_suppressed_count

Never replay production charges without explicit business approval.

Observability

Tracing: propagate traceparent in headers; link producer span to consumer span.

Dashboards per event type—not only per broker cluster.