Software Architecture Design

Software Architecture Design — Quick Reference

Use this skill for system-level design decisions rather than implementation details within a single service or component.

Quick Reference

When to Use This Skill

Invoke when working on:

• System decomposition: Deciding between monolith, modular monolith, microservices
• Architecture patterns: Event-driven, CQRS, layered, hexagonal, serverless
• Data architecture: Consistency models, sharding, replication, CQRS patterns
• Scalability design: Load balancing, caching strategies, database scaling
• Resilience patterns: Circuit breakers, retries, bulkheads, graceful degradation
• API contracts: Service boundaries, versioning, integration patterns
• Architecture decisions: ADRs, tradeoff analysis, technology selection
• Migration planning: Monolith decomposition, strangler fig, database separation

When NOT to Use This Skill

Use other skills instead for:

• Single-service implementation (routes, controllers, business logic) → software-backend
• API endpoint design (REST conventions, GraphQL schemas) → dev-api-design
• Security implementation (auth, encryption, OWASP) → software-security-appsec
• Frontend component architecture → software-frontend
• Database query optimization → data-sql-optimization

Decision Tree: Choosing Architecture Pattern

Project needs: [New System or Major Refactor]
    ├─ Single team, evolving domain?
    │   ├─ Start simple → Modular Monolith (clear module boundaries)
    │   └─ Need rapid iteration → Layered Architecture
    │
    ├─ Multiple teams, clear bounded contexts?
    │   ├─ Independent deployment critical → Microservices
    │   └─ Shared data model → Modular Monolith with service modules
    │
    ├─ Event-driven workflows?
    │   ├─ Asynchronous processing → Event-Driven Architecture (Kafka, queues)
    │   └─ Complex state machines → Saga pattern + Event Sourcing
    │
    ├─ Variable/unpredictable load?
    │   ├─ Pay-per-use model → Serverless (AWS Lambda, Cloudflare Workers)
    │   └─ Batch processing → Serverless + queues
    │
    └─ High consistency requirements?
        ├─ Strong ACID guarantees → Monolith or Modular Monolith
        └─ Distributed data → CQRS + Event Sourcing

Decision Factors:

• Team size threshold: <10 developers → modular monolith typically outperforms microservices (operational overhead)
• Team structure (Conway's Law) — architecture mirrors org structure
• Deployment independence needs
• Consistency vs availability tradeoffs (CAP theorem)
• Operational maturity (monitoring, orchestration)

See references/modern-patterns.md for detailed pattern descriptions.

Output Guidelines

The references in this skill are background knowledge for you — absorb the patterns and present them as your own expertise. Do not cite internal reference file names (e.g., "from data-architecture-patterns.md") in user-facing output. Users don't know these files exist.

Every architecture recommendation should include:

• Concrete technology picks: Name specific technologies (e.g., "Temporal.io for workflow orchestration", "Socket.io with Redis adapter") rather than staying abstract. The user needs to make build decisions, not just understand patterns.
• What NOT to build: Explicitly call out what to defer or avoid. Premature scope is the #1 architecture mistake — help the user avoid it.
• Team and process alignment: How does this architecture map to team structure? What ownership model does it imply? Include CODEOWNERS, deployment ownership, and on-call boundaries where relevant.
• Success metrics: How will the team know the architecture is working? Include measurable indicators (deploy frequency, lead time, error rates, MTTR).
• Focused length: Aim for depth on the 3–5 decisions that matter most rather than exhaustive coverage of every concern. A recommendation that's too long to read is a recommendation that won't be followed.

Workflow (System-Level)

Use this workflow when a user asks for architecture recommendations, decomposition, or major platform decisions.

1. Clarify: problem statement, non-goals, constraints, and success metrics 2. Capture quality attributes: availability, latency, throughput, durability, consistency, security, compliance, cost 3. Propose 2–3 candidate architectures and compare tradeoffs 4. Define boundaries: bounded contexts, ownership, APIs/events, integration contracts 5. Decide data strategy: storage, consistency model, schema evolution, migrations 6. Design for operations: SLOs, failure modes, observability, deployment, DR, incident playbooks 7. Call out scope limits: what NOT to build yet, what to defer, what to buy vs build 8. Document decisions: write ADRs for key tradeoffs and irreversible choices

Preferred deliverables (pick what fits the request):

• Architecture blueprint: assets/planning/architecture-blueprint.md
• Decision record: assets/planning/adr-template.md
• Pattern deep dives: references/modern-patterns.md, references/scalability-reliability-guide.md

2026 Considerations

Load only when the question explicitly involves current trends, vendor-specific constraints, or "what's the latest thinking on X?"

• references/architecture-trends-2026.md — Platform engineering, data mesh, composable architecture, AI-native systems
• data/sources.json — 60 curated resources organized by category:
• platform_engineering_2026 — IDP trends, AI-platform convergence, Backstage
• optional_ai_architecture — RAG patterns, multi-agent design, MCP/A2A protocols
• modern_architecture_2025 — Data mesh, composable architecture, continuous architecture

If live web access is available, consult 2–3 authoritative sources from data/sources.json and fold findings into the recommendation. If not, answer with durable patterns and explicitly state assumptions that could change (vendor limits, pricing, managed-service capabilities).

Navigation

Core References

Read at most 2–3 references per question — pick the ones most relevant to the specific ask. Do not read all of them.

Templates

Planning & Documentation (assets/planning/):

• architecture-blueprint.md — Service blueprint (dependencies, SLAs, data flows, resilience, security, observability)
• adr-template.md — Architecture Decision Record for tradeoff analysis

Architecture Patterns (assets/patterns/):

• microservices-template.md — Microservices design (API contracts, resilience, deployment, testing)
• event-driven-template.md — Event-driven architecture (event schemas, saga patterns, event sourcing)

Operations (assets/operations/):

• scalability-checklist.md — Scalability checklist (DB scaling, caching, load testing, auto-scaling, DR)

Related Skills

• software-backend — Backend engineering, API implementation, data layer
• software-frontend — Frontend architecture, micro-frontends, state management
• dev-api-design — REST, GraphQL, gRPC design patterns
• ops-devops-platform — CI/CD, deployment strategies, IaC
• qa-observability — Monitoring, tracing, alerting, SLOs
• software-security-appsec — Threat modeling, auth, secure design
• data-sql-optimization — Database design, optimization, indexing
• docs-codebase — Architecture documentation, C4 diagrams, ADRs

System Scalability Checklist

Use this checklist when designing for horizontal scalability and high availability.

Load Estimation

• [ ] Current load: [N] requests/second, [N] concurrent users
• [ ] Peak load: [N] requests/second (expected during [event/time])
• [ ] Growth projection: [X]% yearly growth
• [ ] Target capacity: Support [N]x current load

Scalability Dimensions

Horizontal Scaling (Preferred)

• [ ] Stateless services: All application servers are stateless
• [ ] Session storage: Use Redis/Memcached for distributed sessions
• [ ] File storage: Use object storage (S3/GCS) instead of local filesystem
• [ ] Auto-scaling: Configure based on CPU/memory/RPS metrics
• [ ] Load balancer: Layer 7 (application-aware) load balancing

Vertical Scaling (Limited)

• [ ] Database instance: Right-sized for current + 6 months growth
• [ ] Cache instance: Sized for working set + 20% headroom
• [ ] Max capacity: Identified vertical scaling limit

Database Scalability

Read Scaling

• [ ] Read replicas: [N] replicas for read-heavy workloads
• [ ] Read/write splitting: Route reads to replicas, writes to primary
• [ ] Connection pooling: PgBouncer/ProxySQL to limit connections
• [ ] Query optimization: Queries < [10ms] with proper indexing

Write Scaling

• [ ] Sharding strategy: [Hash-based / Range-based / Geographic]
• Shard key: [user_id / tenant_id / region]
• Number of shards: [N] (plan for [10x] growth)
• [ ] Write-ahead logging: Asynchronous replication for replicas
• [ ] Bulk operations: Batch inserts/updates to reduce round trips

Caching Strategy

• [ ] Cache layers:
• L1: In-memory application cache ([Caffeine / Guava])
• L2: Distributed cache ([Redis / Memcached])
• L3: CDN for static assets ([CloudFront / Fastly])
• [ ] Cache hit ratio: Target > [90]%
• [ ] TTL strategy: Balance freshness vs load ([5min] for hot data, [1h] for warm)
• [ ] Cache eviction: LRU/LFU policy configured
• [ ] Cache warming: Pre-populate on deployment
• [ ] Cache invalidation: Event-driven invalidation for updates

API Gateway

• [ ] Rate limiting:
• Per-user: [N] req/min
• Per-IP: [N] req/min
• Burst allowance: [N] requests
• [ ] Request throttling: Queue requests during spikes
• [ ] Response compression: Gzip/Brotli enabled
• [ ] API versioning: Support [N] concurrent versions

Asynchronous Processing

• [ ] Message queue: [Kafka / RabbitMQ / AWS SQS]
• Throughput: [N] messages/second
• Retention: [X] days
• [ ] Worker pools: [N] workers per queue
• [ ] Backpressure: Reject requests when queue length > [N]
• [ ] Dead letter queue: For failed message handling

Content Delivery

• [ ] CDN: CloudFront/Fastly for static assets
• [ ] Edge caching: Cache-Control headers configured
• [ ] Image optimization: WebP format, lazy loading
• [ ] Asset bundling: Minified and bundled CSS/JS

Data Storage Patterns

Hot/Warm/Cold Data

• [ ] Hot data: Last [7] days in primary DB (fast access)
• [ ] Warm data: Last [30] days in read replicas
• [ ] Cold data: Older than [30] days archived to S3/Glacier
• [ ] Archival strategy: Automated data lifecycle policies

Data Partitioning

• [ ] Time-based partitioning: Partition by month/year for time-series data
• [ ] Hash partitioning: Distribute by hash(user_id) for even distribution
• [ ] List partitioning: Partition by region/tenant for isolation

Connection Management

• [ ] Database connection pool:
• Min connections: [N]
• Max connections: [N]
• Connection timeout: [Xms]
• [ ] HTTP keep-alive: Reuse connections to upstream services
• [ ] Circuit breaker: Prevent cascade failures

Observability for Scalability

Key Metrics

• [ ] Golden signals:
• Latency: p50, p95, p99 response times
• Traffic: Requests per second
• Errors: Error rate (4xx, 5xx)
• Saturation: CPU, memory, disk, network usage
• [ ] Capacity metrics:
• Database connections used/available
• Queue depth and processing rate
• Cache hit/miss ratio
• Thread pool utilization

Alerts

• [ ] CPU > [70]% for [5] minutes → Scale out
• [ ] Memory > [80]% for [5] minutes → Investigate memory leak
• [ ] Database connections > [80]% → Add replicas
• [ ] Queue depth > [1000] → Add workers
• [ ] Error rate > [1]% → Page on-call

Load Testing

• [ ] Baseline test: Measure current performance under typical load
• [ ] Stress test: Identify breaking point (max capacity before failure)
• [ ] Spike test: Test behavior under sudden 10x traffic spike
• [ ] Soak test: Run at [2x] typical load for [24] hours
• [ ] Tools: [k6 / JMeter / Gatling / Locust]

Load Test Scenarios

• [ ] Scenario 1: [N] users browsing product catalog
• [ ] Scenario 2: [N] users checking out simultaneously
• [ ] Scenario 3: [N] API clients polling for updates
• [ ] Target performance: p95 < [200ms], error rate < [0.1]%

Cost Optimization

• [ ] Right-sizing: Use smallest instance that meets SLA
• [ ] Reserved instances: [70]% reserved, [30]% on-demand/spot
• [ ] Auto-scaling policies:
• Scale up: When CPU > [70]% for [2] minutes
• Scale down: When CPU < [30]% for [10] minutes
• Min instances: [N], Max instances: [N]
• [ ] Database optimization: Remove unused indexes, optimize slow queries
• [ ] CDN optimization: Increase cache TTL where possible

Geographic Distribution

• [ ] Multi-region deployment:
• Primary region: [us-east-1]
• Secondary region: [eu-west-1]
• Failover: Automatic DNS failover
• [ ] Data locality: Store user data in nearest region (GDPR compliance)
• [ ] Latency optimization: < [100ms] within region, < [300ms] cross-region

Disaster Recovery

• [ ] RTO (Recovery Time Objective): [X] hours
• [ ] RPO (Recovery Point Objective): [X] minutes
• [ ] Backup frequency: Database backup every [X] hours
• [ ] Failover testing: Quarterly DR drills
• [ ] Data replication: Asynchronous cross-region replication

Security at Scale

• [ ] DDoS protection: CloudFlare/AWS Shield enabled
• [ ] Rate limiting: Per-IP and per-user limits
• [ ] WAF rules: Block common attack patterns
• [ ] Certificate management: Auto-renewal with Let's Encrypt/ACM

Deployment Strategy

• [ ] Blue-green deployment: Zero-downtime releases
• [ ] Canary releases: [10]% traffic to new version initially
• [ ] Feature flags: Toggle features without deployment
• [ ] Rollback plan: Automated rollback if error rate > [X]%

Checklist Before Production

• [ ] Load tested at [3x] expected peak traffic
• [ ] Auto-scaling policies validated
• [ ] Database read replicas configured
• [ ] Caching strategy implemented and tested
• [ ] Monitoring and alerts configured
• [ ] Disaster recovery plan documented and tested
• [ ] Cost monitoring and budgets set
• [ ] Security review completed
• [ ] Runbooks created for common incidents
• [ ] On-call rotation established

Event-Driven Architecture Template

Use this template when designing event-driven systems with asynchronous communication.

Event Schema Definition

Event: [EventName]

{
  "eventId": "uuid",
  "eventType": "[EventName]",
  "eventVersion": "1.0",
  "timestamp": "ISO8601",
  "source": "[ServiceName]",
  "correlationId": "uuid",
  "payload": {
    // Event-specific data
  },
  "metadata": {
    "userId": "string",
    "traceId": "string"
  }
}

Trigger: [When is this event published?] Consumers: [Which services consume this event?] Retention: [How long to keep in event store?]

Event Catalog

Event Broker Configuration

• Platform: [Kafka / RabbitMQ / AWS EventBridge / Google Pub/Sub]
• Topics/Queues:
• orders.placed - Order creation events
• payments.processed - Payment completion events
• inventory.updated - Stock level changes
• Partitioning strategy: [By customer ID / By order ID / Round-robin]
• Replication factor: [3 for production]
• Retention period: [7 days for event replay]

Producer Configuration

Service: [ProducerServiceName]

Events Published:

• [EventName]: Published when [business event occurs]
• Partition key: [customer_id / entity_id]
• Ordering guarantee: [Yes/No]
• Retry policy: [3 retries with exponential backoff]

Reliability Patterns:

• Outbox pattern: Write to database + outbox table atomically
• At-least-once delivery: Idempotent event production
• Schema validation: Validate against schema registry before publishing
• Failure handling: Store failed events in dead letter queue

Code Example:

async def publish_event(event: OrderPlaced):
    # Transactional outbox pattern
    async with db.transaction():
        await db.orders.create(event.order)
        await db.outbox.insert({
            "event_id": event.id,
            "event_type": "OrderPlaced",
            "payload": event.to_json(),
            "status": "pending"
        })

    # Asynchronous event publishing (handled by outbox processor)
    await event_broker.publish(
        topic="orders.placed",
        key=event.customer_id,
        value=event.to_json(),
        headers={"trace_id": trace_id}
    )

Consumer Configuration

Service: [ConsumerServiceName]

Events Consumed:

• [EventName]: From [ProducerService]
• Consumer group: [service-name]-[event-name]
• Concurrency: [N parallel workers]
• Max retries: [5 with exponential backoff]
• Dead letter queue: [event-name].dlq

Processing Patterns:

• Idempotency: Check event ID before processing
• Ordering: Process events in order within partition
• Error handling: Retry transient errors, DLQ for permanent failures
• Acknowledgment: Acknowledge only after successful processing

Code Example:

@consumer(topic="orders.placed", group="payment-service-orders")
async def process_order_placed(event: OrderPlaced):
    # Idempotency check
    if await db.processed_events.exists(event.id):
        logger.info(f"Event {event.id} already processed")
        return

    try:
        # Business logic
        payment = await payment_service.process_payment(event.order)

        # Mark as processed
        await db.processed_events.insert(event.id)

        # Publish downstream event
        await publish_event(PaymentProcessed(payment))

    except RetryableError as e:
        logger.warning(f"Retrying event {event.id}: {e}")
        raise  # Will be retried
    except PermanentError as e:
        logger.error(f"Sending event {event.id} to DLQ: {e}")
        await dlq.send(event, error=str(e))

Event Sourcing

Use when: Need complete audit trail and event replay capabilities.

Aggregate: [AggregateName]

Command Handlers:

• CreateOrder: Validates and produces OrderCreated event
• UpdateOrderStatus: Produces OrderStatusUpdated event

Event Store:

• Storage: [PostgreSQL event_store table / EventStoreDB / Kafka topic]
• Schema:

  CREATE TABLE events (
      event_id UUID PRIMARY KEY,
      aggregate_id UUID NOT NULL,
      aggregate_type VARCHAR(100),
      event_type VARCHAR(100),
      event_data JSONB,
      event_version INT,
      created_at TIMESTAMP,
      INDEX(aggregate_id, event_version)
  );

Snapshot Strategy:

• Create snapshot every [100] events
• Store in separate snapshots table
• Load latest snapshot + subsequent events for aggregate rebuild

Saga Pattern (Distributed Transactions)

Saga: [SagaName]

Orchestration Approach: [Choreography / Orchestrator]

Steps:

1. OrderService → Publish OrderPlaced 2. PaymentService → Process payment → Publish PaymentProcessed or PaymentFailed 3. InventoryService → Reserve stock → Publish InventoryReserved or InventoryUnavailable 4. ShippingService → Schedule shipment → Publish ShipmentScheduled

Compensation (Rollback):

• If PaymentFailed → Publish OrderCancelled
• If InventoryUnavailable → Publish PaymentRefunded → OrderCancelled

State Machine:

[Order Created] → [Payment Processing] → [Inventory Reserved] → [Shipment Scheduled] → [Order Completed]
      ↓                    ↓                       ↓
[Order Cancelled] ← [Payment Failed] ← [Inventory Unavailable]

Schema Evolution

Versioning Strategy: [Backward compatible / Forward compatible / Full compatibility]

Schema Registry: [Confluent Schema Registry / AWS Glue Schema Registry]

Version Migration:

• v1 → v2 Changes:
• Added optional field: customer_email
• Deprecated field: customer_address (use shipping_address instead)
• Consumer compatibility: Old consumers can read v2 events (ignore new fields)
• Producer compatibility: New producers can emit v1 events if needed

Monitoring & Observability

Metrics:

• Event publish rate and latency
• Consumer lag (events behind current offset)
• Processing errors and retry count
• Dead letter queue size

Alerts:

• Consumer lag > [1000 events]
• Error rate > [5%]
• DLQ messages > [100]
• Event processing latency p99 > [500ms]

Distributed Tracing:

• Propagate trace IDs across events
• Track event flow: Producer → Broker → Consumer → Downstream events

Testing

Unit Tests:

• Event schema validation
• Idempotency logic
• Compensation logic (saga rollback)

Integration Tests:

• End-to-end event flow
• Consumer error handling and retries
• Dead letter queue behavior

Chaos Testing:

• Simulate broker downtime
• Duplicate event delivery
• Out-of-order event delivery
• Consumer crash mid-processing

Security

• Encryption: TLS for data in transit, encryption at rest for event store
• Authorization: ACLs for topic access (producers/consumers)
• Audit: Log all event publications and consumptions
• PII handling: Encrypt sensitive fields in event payload

Cost Optimization

• Retention policy: Delete events older than [X days]
• Compaction: Use log compaction for entity snapshots
• Resource allocation: Right-size broker and consumer resources
• Batch processing: Batch consume events to reduce overhead

Disaster Recovery

• Event replay: Re-process events from timestamp/offset
• Backup: Regular snapshots of event store
• Cross-region replication: Mirror events to DR region
• RTO/RPO: Recovery time objective [X hours], Recovery point objective [X minutes]

Microservices Architecture Template

Use this template when designing a microservices-based system.

Service Definition

• Service name: [ServiceName]
• Bounded context: [Domain boundary this service owns]
• Team owner: [Team responsible for this service]

Service Responsibilities

• Core capabilities:
• [Primary business capability 1]
• [Primary business capability 2]
• Data ownership:
• [Entities this service owns]
• [Events this service publishes]

API Contract

REST Endpoints

GET    /api/v1/[resource]
POST   /api/v1/[resource]
PUT    /api/v1/[resource]/{id}
DELETE /api/v1/[resource]/{id}

Events Published

• [EventName]: Published when [trigger condition]
• [EventName]: Published when [trigger condition]

Events Consumed

• [EventName]: From [SourceService], triggers [action]

Dependencies

Upstream Services (Calls)

• [ServiceName]: For [purpose], timeout: [Xms], circuit breaker threshold: [N failures]

Downstream Services (Called by)

• [ServiceName]: Expects [SLA], provides [data/functionality]

Data Storage

• Database type: [PostgreSQL / MongoDB / Cassandra / etc.]
• Schema approach: [Database-per-service / Shared tables / etc.]
• Replication: [Primary-replica / Multi-master]
• Backup strategy: [Automated daily / Point-in-time recovery]

Resilience

• Timeouts:
• Database queries: [Xms]
• External API calls: [Xms]
• Internal service calls: [Xms]
• Retry policy: Exponential backoff, max [N] retries
• Circuit breaker: Open after [N] failures, half-open after [X] seconds
• Rate limiting: [N] requests per second per client
• Fallback behavior: [Return cached data / Default response / Graceful degradation]

Observability

• Metrics:
• Request rate, latency (p50, p95, p99)
• Error rate (4xx, 5xx)
• Dependency health
• Business metrics: [specific to service]
• Distributed tracing: Jaeger/OpenTelemetry with trace IDs
• Logging:
• Structured JSON logs
• Log level: INFO in production, DEBUG in dev
• Key fields: trace_id, user_id, service_name, timestamp
• Health checks:
• Liveness: /health/live (basic ping)
• Readiness: /health/ready (dependencies check)

Deployment

• Container: Docker image, registry: [ECR / Docker Hub / etc.]
• Orchestration: Kubernetes
• Scaling policy:
• Min replicas: [N]
• Max replicas: [N]
• Scale trigger: CPU > [X]% or Memory > [X]% or RPS > [N]
• Deployment strategy: Rolling update / Canary / Blue-green
• Rollback plan: Automated rollback if error rate > [X]%

Security

• Authentication: JWT tokens, validated via [Auth service / API Gateway]
• Authorization: Role-based access control (RBAC)
• Service-to-service auth: mTLS via service mesh
• Secrets: Stored in [AWS Secrets Manager / Vault / Kubernetes Secrets]
• Input validation: All user inputs validated and sanitized
• Rate limiting: Per-user and per-IP limits

Testing

• Unit tests: Coverage target: 80%+
• Integration tests: Test API contracts and database interactions
• Contract tests: Pact/Spring Cloud Contract for upstream/downstream
• Load tests: Target [N] RPS at p95 < [Xms]
• Chaos testing: Simulate dependency failures, network latency

Communication Patterns

• Synchronous: REST/gRPC for request-response
• Asynchronous: Kafka/RabbitMQ for events
• Idempotency: All write operations support idempotency keys
• Message format: JSON for REST, Protobuf for gRPC, Avro for Kafka

Cost Optimization

• Resource allocation:
• CPU request: [Xm], limit: [Xm]
• Memory request: [XMi], limit: [XMi]
• Auto-scaling: Scale down during off-peak hours
• Data retention: [X days] for logs, [X days] for metrics

Migration Plan

• Phase 1: [Extract service from monolith / Build new service]
• Phase 2: [Gradual traffic migration / Feature flag rollout]
• Phase 3: [Full cutover / Decommission old system]
• Rollback criteria: [Error rate / Latency / Business metrics]

ADR References

• [ADR-001]: [Decision about technology choice]
• [ADR-002]: [Decision about data model]

Architecture Decision Record (ADR)

ADR ID: ADR-YYYY-MM-DD-XXX Title: [Short decision title] Status: Proposed | Accepted | Rejected | Superseded (link) Date: YYYY-MM-DD Owner: [Team / person] Deciders: [Names / roles] Scope: [Service / domain / product area]

---

Core

1. Context

Problem statement: What are we deciding, and why now?

Goals:

• [Goal 1]
• [Goal 2]

Non-goals:

• [Non-goal 1]
• [Non-goal 2]

Constraints (REQUIRED):

• Regulatory/compliance:
• Latency/SLO:
• Data residency:
• Platform/runtime:
• Team/operational maturity:

Assumptions (mark unvalidated):

• [Assumption] ([Inference] if not verified)

2. Decision Drivers (What Matters Most)

Rank the drivers to make tradeoffs explicit.

3. Options Considered

4. Decision

We choose: [Option X]

Why (1–5 bullets max):

• [Reason]

5. Architecture Impact (Implementation-Ready)

Boundaries and contracts

• Public APIs/contracts affected:
• Backward compatibility plan:
• Schema evolution strategy:

Data and consistency

• Source of truth:
• Consistency model (strong/eventual/mixed):
• Migration strategy (expand/contract, dual writes, backfill):

Failure modes and resilience

• Known failure modes:
• Timeouts/retries/backoff policy:
• Idempotency strategy:
• Degradation plan (what still works when dependencies fail):

Security

• Threat model summary:
• AuthN/AuthZ model:
• Secret and key management:
• Audit logging requirements:

Observability

• SLIs/SLOs:
• Metrics/traces/logs to add:
• Dashboards and alerts:

Cost and capacity

• Expected traffic/load:
• Cost model (drivers, main spend areas):
• Capacity plan (limits, scaling triggers):

6. Rollout, Validation, and Rollback

Rollout plan

• Feature flag / staged rollout:
• Data migration steps:
• Runbook updates:

Validation plan

• Tests to add (unit/integration/contract):
• Load/perf tests:
• Chaos/failure injection (if applicable):

Rollback plan

• How to revert code:
• How to revert data (or forward-fix):
• Timebox for rollback decision:

7. Consequences

Positive

• [Benefit]

Negative / tradeoffs

• [Cost or risk]

Follow-ups

• [Task] (owner, due date)

8. Links

• Design doc:
• Diagram(s):
• Tickets/epics:
• Related ADRs:

---

Optional: AI/Automation

Include only if this ADR affects AI/automation features or AI-assisted workflows.

AI Risk and Safety

• User impact of wrong output (harm model):
• Human override path and audit trail:
• Data handling (PII/PHI, retention, residency):
• Abuse cases (prompt injection, data exfiltration) [Inference]

Evaluation and Quality Gates

• Offline evaluation set definition:
• Online metrics and guardrails:
• Rollback triggers for quality regression:

Operations and Cost

• Latency/cost targets:
• Rate limiting and quotas:
• Observability for model/tool calls:

---

References

• ADR concept and template rationale: https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions

Service Architecture Blueprint

Use this when drafting a new service or major redesign.

• Service name / domain: Bounded context, upstream/downstream dependencies
• Non-negotiable requirements: Latency/throughput targets, availability %, data retention, compliance
• Architecture diagram link: C4 level 2/3 diagrams, sequence for critical paths
• Data flows: Read/write paths, schemas, partitioning/replication strategy
• Integration contracts: APIs/events with SLAs, versioning, idempotency rules, consumers/providers
• Resilience: Timeouts, retries, circuit breakers, bulkheads, backpressure, graceful degradation
• Security: AuthN/AuthZ model, secrets handling, ingress/egress policies, audit requirements
• Deployability: CI/CD steps, canary/blue-green, rollback, migration plan
• Observability: Golden signals, dashboards, alert rules, trace spans, log fields
• Risks & assumptions: Unknowns, kill criteria, contingency plans

API Gateway & Service Mesh Patterns

Deep reference for API gateway architectures, service mesh implementation (Istio, Linkerd, Envoy), sidecar patterns, and service-to-service communication. Use when designing inter-service communication, implementing traffic management, or choosing between gateway, mesh, or hybrid topologies.

Contents

• API Gateway Patterns
• Service Mesh Architecture
• Technology Comparison
• mTLS and Service Identity
• Observability Through Mesh
• Gateway vs Mesh vs Both
• Implementation Patterns
• Anti-Patterns
• Decision Framework
• Cross-References

---

API Gateway Patterns

Core Gateway Responsibilities

Gateway Topology Patterns

Pattern 1: Single Gateway

┌────────┐     ┌──────────────┐     ┌──────────┐
│ Client │────▶│   Gateway    │────▶│ Service A│
└────────┘     │              │────▶│ Service B│
               │              │────▶│ Service C│
               └──────────────┘     └──────────┘

Best for: Small teams, <10 services, uniform client needs.

Pattern 2: Backend-for-Frontend (BFF)

┌──────────┐     ┌───────────────┐
│  Mobile  │────▶│  Mobile BFF   │───▶ Services
└──────────┘     └───────────────┘

┌──────────┐     ┌───────────────┐
│   Web    │────▶│   Web BFF     │───▶ Services
└──────────┘     └───────────────┘

┌──────────┐     ┌───────────────┐
│ Partner  │────▶│ Partner API   │───▶ Services
│   API    │     │   Gateway     │
└──────────┘     └───────────────┘

Best for: Different client types with distinct data needs.

Pattern 3: Federated Gateway

┌────────┐     ┌───────────────┐     ┌──────────────┐
│ Client │────▶│ Edge Gateway  │────▶│ Team A GW    │───▶ Team A services
└────────┘     │ (auth, rate   │────▶│ Team B GW    │───▶ Team B services
               │  limiting)    │────▶│ Team C GW    │───▶ Team C services
               └───────────────┘     └──────────────┘

Best for: Large organizations, multiple teams owning their own gateway configuration.

Rate Limiting Patterns

// Token bucket rate limiting (typical gateway configuration)
// Kong rate-limiting plugin configuration
const rateLimitConfig = {
  plugin: 'rate-limiting',
  config: {
    minute: 60,              // 60 requests per minute
    hour: 1000,              // 1000 requests per hour
    policy: 'redis',         // Use Redis for distributed counting
    fault_tolerant: true,    // Allow traffic if Redis is down
    hide_client_headers: false,
    redis_host: 'redis',
    redis_port: 6379,
  },
};

// Response headers
// X-RateLimit-Limit: 60
// X-RateLimit-Remaining: 45
// X-RateLimit-Reset: 1640000000

Request Aggregation

// BFF aggregation pattern — single client request, multiple backend calls
app.get('/api/dashboard', async (req, res) => {
  const userId = req.user.id;

  // Parallel fetch from multiple services
  const [profile, orders, notifications, recommendations] = await Promise.all([
    userService.getProfile(userId),
    orderService.getRecent(userId, { limit: 5 }),
    notificationService.getUnread(userId),
    recommendationService.getForUser(userId),
  ]);

  // Aggregate into client-optimized response
  res.json({
    user: { name: profile.name, avatar: profile.avatar },
    recentOrders: orders.map(o => ({ id: o.id, status: o.status, total: o.total })),
    unreadCount: notifications.length,
    recommendations: recommendations.slice(0, 3),
  });
});

---

Service Mesh Architecture

Core Concepts

A service mesh is a dedicated infrastructure layer for service-to-service communication. It uses sidecar proxies deployed alongside each service to handle networking concerns.

┌─────────────────────────────────────────────────┐
│                Control Plane                     │
│  (Configuration, certificates, policies)         │
└───────────┬──────────────┬──────────────┬────────┘
            │              │              │
     ┌──────▼──────┐ ┌────▼────────┐ ┌──▼──────────┐
     │  Service A  │ │  Service B  │ │  Service C  │
     │  ┌────────┐ │ │  ┌────────┐ │ │  ┌────────┐ │
     │  │  App   │ │ │  │  App   │ │ │  │  App   │ │
     │  └───┬────┘ │ │  └───┬────┘ │ │  └───┬────┘ │
     │  ┌───▼────┐ │ │  ┌───▼────┐ │ │  ┌───▼────┐ │
     │  │Sidecar │◄├─┼─▶│Sidecar │◄├─┼─▶│Sidecar │ │
     │  │Proxy   │ │ │  │Proxy   │ │ │  │Proxy   │ │
     │  └────────┘ │ │  └────────┘ │ │  └────────┘ │
     └─────────────┘ └────────────┘ └──────────────┘
           Data Plane (proxies handle all traffic)

Sidecar Proxy Responsibilities

Control Plane vs Data Plane

---

Technology Comparison

Gateway Comparison

Service Mesh Comparison

Selection Quick Guide

Service mesh selection:
  ├─ < 10 services → No mesh needed. Use library-based patterns.
  ├─ 10-50 services, want simplicity → Linkerd
  ├─ 10-50 services, need advanced traffic mgmt → Istio
  ├─ AWS-native, managed preference → App Mesh
  ├─ Performance-critical, eBPF available → Cilium
  └─ > 50 services, multi-cluster → Istio (with careful tuning)

---

mTLS and Service Identity

Mutual TLS in Service Mesh

Service A                                         Service B
┌──────────┐                                     ┌──────────┐
│          │── 1. TLS handshake (client cert) ──▶│          │
│  App     │◀─ 2. TLS handshake (server cert) ──│  App     │
│          │── 3. Encrypted traffic ────────────▶│          │
└──────────┘                                     └──────────┘

Both sides present certificates issued by the mesh CA.
Identity is based on service account, not network address.

Identity-Based Authorization

# Istio AuthorizationPolicy
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: orders-service
  rules:
    # Allow only payment-service and api-gateway to call orders
    - from:
        - source:
            principals:
              - cluster.local/ns/production/sa/payment-service
              - cluster.local/ns/production/sa/api-gateway
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/orders/*"]
    # Deny everything else (implicit deny)

Certificate Management

Recommended: Use mesh-managed CA for most deployments. Integrate with external CA (Vault, AWS ACM PCA) for enterprise compliance requirements.

---

Observability Through Mesh

Distributed Tracing

Service mesh proxies automatically inject trace headers and emit spans.

Client → [Gateway span] → [Service A span] → [Service B span] → [Database span]
                                                    │
                                              [Service C span]

Each proxy adds its own span without application code changes.

Trace header propagation:

Mesh-Level Metrics

Standard metrics emitted by sidecar proxies (no application instrumentation needed):

Golden Signals Dashboard

For each service, track:
  1. Latency     — P50, P95, P99 response time
  2. Traffic     — Requests per second
  3. Errors      — 5xx rate as percentage of total
  4. Saturation  — CPU, memory, connection pool usage

Service mesh provides signals 1-3 automatically.
Signal 4 requires application-level metrics.

Service Topology Visualization

Mesh provides automatic service dependency mapping:

Tools:
  - Kiali (Istio) — Service graph, health status, traffic flow
  - Linkerd Viz — Dashboard with golden metrics per service
  - Hubble (Cilium) — Network flow visibility
  - Jaeger/Tempo — Distributed trace visualization

---

Gateway vs Mesh vs Both

Comparison Matrix

Recommended Architecture

┌──────────────┐
│   Internet   │
└──────┬───────┘
       │
┌──────▼───────┐  ← North-south boundary
│ API Gateway  │  Auth, rate limiting, TLS termination, routing
│ (Kong/Envoy) │
└──────┬───────┘
       │
┌──────▼───────────────────────────────┐
│         Service Mesh (Istio/Linkerd) │  ← East-west boundary
│                                      │  mTLS, retries, circuit breaking,
│  ┌─────┐   ┌─────┐   ┌─────┐       │  observability, traffic management
│  │Svc A│◄─▶│Svc B│◄─▶│Svc C│       │
│  └─────┘   └─────┘   └─────┘       │
└──────────────────────────────────────┘

When You Do NOT Need a Service Mesh

Alternatives to mesh for small deployments:

• Library-based retries and circuit breaking (p-retry, opossum)
• Application-level mTLS (cert-manager + application config)
• OpenTelemetry SDK for observability (no mesh needed)

---

Implementation Patterns

Canary Deployment via Mesh

# Istio: Route 95% to v1, 5% to v2
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: orders
spec:
  hosts:
    - orders
  http:
    - route:
        - destination:
            host: orders
            subset: v1
          weight: 95
        - destination:
            host: orders
            subset: v2
          weight: 5
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: orders
spec:
  host: orders
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

Retry and Timeout Configuration

# Istio retry policy
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: orders
spec:
  hosts:
    - orders
  http:
    - timeout: 3s
      retries:
        attempts: 3
        perTryTimeout: 1s
        retryOn: 5xx,reset,connect-failure,retriable-4xx
      route:
        - destination:
            host: orders

Circuit Breaker Configuration

# Istio circuit breaker
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: orders
spec:
  host: orders
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

---

Anti-Patterns

1. Mesh for Everything

[FAIL] Deploying a service mesh for 3 services because "everyone uses Istio"
       → Massive operational overhead for minimal benefit

[PASS] Start with library-based patterns. Adopt mesh when you have
       10+ services and measurable networking pain.

2. Gateway as Service Bus

[FAIL] Putting business logic, data transformation, and orchestration in the gateway
       → Gateway becomes a bottleneck and single point of failure for logic

[PASS] Gateway handles cross-cutting concerns only (auth, rate limiting, routing).
       Business logic stays in services.

3. Ignoring Sidecar Resource Overhead

[FAIL] Deploying mesh without accounting for sidecar CPU/memory per pod
       → Resource exhaustion, unexpected costs, latency increase

[PASS] Budget 50-128MB memory and 0.1-0.5 CPU per sidecar proxy.
       Monitor mesh overhead as a first-class metric.

4. No Mesh Bypass for Debugging

[FAIL] No way to bypass the mesh for troubleshooting
       → Cannot isolate whether issues are mesh-related or application-related

[PASS] Maintain ability to disable sidecar injection per namespace or pod.
       Have runbooks for mesh-bypass debugging.

5. mTLS Without Identity Policies

[FAIL] Enabling mTLS but no authorization policies → all services can call all services
       → mTLS only encrypts traffic, it does not authorize

[PASS] Pair mTLS with explicit AuthorizationPolicy rules.
       Default deny, explicit allow per service pair.

---

Decision Framework

Do You Need an API Gateway?

1. Do you expose APIs to external clients?              [Yes → You need a gateway]
2. Do you need centralized auth for external traffic?   [Yes → Gateway]
3. Do you need rate limiting for public APIs?            [Yes → Gateway]
4. Do you have multiple BFF needs (mobile, web, partner)? [Yes → Multiple gateways]
5. Internal-only services?                               [Gateway optional, mesh sufficient]

Do You Need a Service Mesh?

1. How many services?                    [< 10 → No, 10-50 → Maybe, > 50 → Yes]
2. Do you need mTLS between services?    [Yes → +2]
3. Do you need advanced traffic management? [Yes → +2]
4. Is observability a gap today?          [Yes → +1]
5. Does the team have K8s experience?     [Yes → +1, No → -2]
6. Are you running on Kubernetes?         [Yes → +1, No → -2]

Score: 0-2 → Library-based patterns
Score: 3-4 → Linkerd (simpler mesh)
Score: 5+  → Istio or Cilium (full mesh)

Gateway Selection

Choosing a gateway:
  ├─ AWS-native, managed preference → AWS API Gateway
  ├─ Multi-cloud, plugin extensibility → Kong
  ├─ Performance-critical, already using Envoy → Envoy as gateway
  ├─ Kubernetes auto-discovery → Traefik
  ├─ Simple reverse proxy, proven → NGINX
  └─ GraphQL federation → Apollo Router

---

Cross-References

• modern-patterns.md — Service mesh and microservices overview
• scalability-reliability-guide.md — Load balancing, circuit breakers, resilience
• migration-modernization-guide.md — Strangler fig pattern with gateway routing
• data-architecture-patterns.md — Service-to-service data patterns
• ../software-backend/SKILL.md — Backend service implementation
• ../../ops-devops-platform/SKILL.md — Kubernetes, CI/CD, deployment strategies
• ../../software-security-appsec/SKILL.md — Zero-trust security, mTLS

Architecture Trends (2026)

Use this reference when the user explicitly asks for "current" or "2026" guidance, or when the system design depends on ecosystem maturity (managed services, tooling, compliance, cost).

Platform Engineering and Internal Developer Platforms (IDPs)

Goal: reduce cognitive load and standardize delivery via self-service "golden paths".

Common building blocks:

• Service catalog and ownership (systems, components, dependencies)
• Templates/scaffolding ("paved roads") for new services and common workflows
• Self-service provisioning (IaC APIs, opinionated modules)
• Policy as code (security, compliance, FinOps guardrails)
• Built-in observability defaults (logs/metrics/traces, dashboards, alerts)

When to use:

• Multiple product teams with recurring platform needs
• Frequent service creation or consistent compliance requirements
• High operational overhead and inconsistent delivery practices

Avoid:

• Building a portal without paved roads (catalog without outcomes)
• Platform team as a ticket queue (no true self-service)

Data Mesh (Analytics and Data Product Architecture)

Goal: scale analytics by shifting ownership to domain teams and standardizing interoperability.

Core ideas:

• Domain-owned data products with SLAs (freshness, latency, schema stability)
• Federated governance (standards + tooling, not a central bottleneck)
• Contracts and versioning for schemas and semantic definitions

When to use:

• Cross-domain analytics is slowed by central data bottlenecks
• Multiple domains need to publish reliable datasets to many consumers

Avoid:

• Rebranding a data lake as data mesh without ownership and contracts
• Uncontrolled schema changes without consumer communication

Composable Architecture (Packaged Business Capabilities)

Goal: assemble business capabilities quickly via well-defined contracts.

Typical characteristics:

• API-first capability components with clear ownership
• Event-driven coordination for cross-capability workflows
• Composition layer (workflow engine, orchestration, or integration platform)

When to use:

• You need to rapidly combine capabilities across products or teams
• You have a stable set of reusable domain capabilities

Avoid:

• Tight coupling through shared databases or shared internal libraries

Continuous Architecture and Fitness Functions

Goal: keep architecture aligned with reality through automation and regular review.

Practices:

• "Just-enough" upfront design, iterate based on feedback and risk
• Fitness functions: automated checks that enforce architectural constraints (dependency rules, SLO budgets, cost gates)
• ADRs for irreversible tradeoffs, revisited when assumptions change

When to use:

• Any long-lived product where architectural drift is a risk
• Systems with explicit constraints (latency, compliance, cost)

Edge-First and Hybrid Edge/Cloud

Goal: meet latency, bandwidth, or offline requirements via local processing.

Common patterns:

• Edge caching and request shaping (CDN, edge gateways)
• Edge validation and filtering (reduce bandwidth to cloud)
• Hybrid pipelines (edge aggregation, cloud analytics and long-term storage)

When to use:

• Real-time UX needs, constrained networks, IoT/OT environments

Avoid:

• Splitting logic across edge/cloud without clear data ownership and observability

AI-Native System Architecture (RAG, Tools, Agents)

Use when LLMs are part of the product or internal platform.

RAG and tool patterns:

• Retrieval as a bounded subsystem (indexing, access control, evaluation)
• Tool gateway layer (rate limits, authZ, auditing, allowlists)
• Async orchestration for slow and failure-prone steps (queues, workflows)

Production requirements that are easy to miss:

• Evaluation and regression testing (golden sets, drift checks)
• Observability tailored to AI (prompt/response logging policy, safety filters, cost tracking)
• Security (prompt injection, data exfiltration, tool abuse, multi-tenant isolation)

Anti-patterns:

• Using a vector store as the source of truth for business data
• Shipping agents without termination conditions and without audit logs

Data Architecture Patterns

Comprehensive guide to data architecture patterns for distributed systems: CQRS, event sourcing, data mesh, polyglot persistence, saga patterns, and consistency models. Use when designing data flows across service boundaries, choosing storage strategies, or managing distributed state.

Contents

• CQRS Implementation Patterns
• Event Sourcing
• Data Mesh
• Polyglot Persistence
• Saga Patterns
• Consistency Models
• Anti-Patterns
• Decision Framework
• Cross-References

---

CQRS Implementation Patterns

When to Use CQRS

CQRS Variants

Variant 1: Same Database, Separate Models

Simplest form. Single database with distinct read/write code paths.

// Write side — normalized, validated
class OrderCommandHandler {
  async createOrder(cmd: CreateOrderCommand): Promise<void> {
    const order = Order.create(cmd.customerId, cmd.items);
    await this.writeRepo.save(order);          // normalized tables
    await this.eventBus.publish(order.events);  // domain events
  }
}

// Read side — denormalized, optimized for queries
class OrderQueryHandler {
  async getCustomerDashboard(customerId: string): Promise<Dashboard> {
    // Single query against a denormalized view or materialized table
    return this.readRepo.getDashboard(customerId);
  }
}

Variant 2: Separate Databases

Write DB (normalized, ACID) + Read DB (denormalized, optimized). Sync via events.

┌──────────────┐     events     ┌──────────────────┐
│  Write DB    │ ──────────────▶│  Read DB          │
│  (Postgres)  │                │  (Elasticsearch / │
│  normalized  │                │   Redis / Mongo)  │
└──────────────┘                └──────────────────┘

Variant 3: CQRS + Event Sourcing

Write side stores events as source of truth. Read side projects events into queryable models.

Synchronization Strategies

Projection Patterns

// Event handler that maintains a read model
class OrderProjection {
  async handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
    await this.readDb.customerOrders.upsert({
      customerId: event.customerId,
      orderId: event.orderId,
      status: 'created',
      total: event.total,
      itemCount: event.items.length,
      createdAt: event.timestamp,
    });
  }

  async handleOrderShipped(event: OrderShippedEvent): Promise<void> {
    await this.readDb.customerOrders.update(
      { orderId: event.orderId },
      { status: 'shipped', shippedAt: event.timestamp }
    );
  }

  // Rebuild: replay all events from the event store
  async rebuild(): Promise<void> {
    await this.readDb.customerOrders.truncate();
    const events = await this.eventStore.readAll('Order');
    for (const event of events) {
      await this.handle(event);
    }
  }
}

---

Event Sourcing

Core Concepts

Event sourcing stores every state change as an immutable event. Current state is derived by replaying events.

Event Store (append-only log):
  [OrderCreated] → [ItemAdded] → [ItemAdded] → [OrderPaid] → [OrderShipped]

Current State = fold(initialState, events)

When to Use Event Sourcing

Event Store Design

interface DomainEvent {
  eventId: string;          // UUID, globally unique
  aggregateId: string;      // Entity this event belongs to
  aggregateType: string;    // e.g., 'Order', 'Account'
  eventType: string;        // e.g., 'OrderCreated', 'ItemAdded'
  version: number;          // Monotonic per aggregate
  timestamp: string;        // ISO 8601
  data: Record<string, unknown>;  // Event payload
  metadata: {
    correlationId: string;  // Trace through the system
    causationId: string;    // Which command caused this
    userId?: string;        // Who triggered it
  };
}

Storage options:

Snapshots

Snapshots prevent replaying thousands of events for every state reconstruction.

class OrderAggregate {
  private version = 0;
  private state: OrderState;
  private static SNAPSHOT_INTERVAL = 100;

  async load(aggregateId: string): Promise<void> {
    // 1. Load latest snapshot (if any)
    const snapshot = await this.snapshotStore.getLatest(aggregateId);
    if (snapshot) {
      this.state = snapshot.state;
      this.version = snapshot.version;
    }

    // 2. Replay events AFTER the snapshot
    const events = await this.eventStore.readFrom(
      aggregateId,
      this.version + 1
    );
    for (const event of events) {
      this.apply(event);
    }
  }

  async save(newEvents: DomainEvent[]): Promise<void> {
    await this.eventStore.append(this.aggregateId, newEvents, this.version);
    this.version += newEvents.length;

    // 3. Create snapshot periodically
    if (this.version % OrderAggregate.SNAPSHOT_INTERVAL === 0) {
      await this.snapshotStore.save({
        aggregateId: this.aggregateId,
        version: this.version,
        state: this.state,
      });
    }
  }
}

Schema Evolution for Events

Events are immutable once stored. Handle schema changes with:

1. Upcasting — Transform old events to new shape at read time 2. Versioned event types — OrderCreated_v2 with migration logic 3. Weak schema — Keep events loosely typed, validate at projection

// Upcaster example
function upcast(event: DomainEvent): DomainEvent {
  if (event.eventType === 'OrderCreated' && !event.data.currency) {
    return { ...event, data: { ...event.data, currency: 'USD' } };
  }
  return event;
}

---

Data Mesh

Principles

Data mesh treats data as a product, owned by domain teams rather than a centralized data team.

Data Product Structure

orders-domain/
├── data-products/
│   ├── order-events/           # Real-time event stream
│   │   ├── schema.avro
│   │   ├── slo.yaml            # Freshness, completeness, accuracy
│   │   └── README.md           # Discovery documentation
│   ├── order-metrics/          # Aggregated metrics
│   │   ├── schema.sql
│   │   └── slo.yaml
│   └── order-snapshots/        # Periodic full snapshots
│       ├── schema.parquet
│       └── slo.yaml
├── pipelines/                  # Domain-owned transformations
└── contracts/                  # Input/output contracts

Data Product Quality Checklist

• [ ] Schema is versioned and published to a registry
• [ ] SLOs defined: freshness (<N minutes), completeness (>99%), accuracy
• [ ] Documentation: purpose, owner, schema, access patterns, known limitations
• [ ] Discoverability: registered in data catalog
• [ ] Access control: RBAC or attribute-based access
• [ ] Monitoring: alerting on SLO violations

Federated Governance Model

---

Polyglot Persistence

Choosing the Right Database Per Service

Decision Criteria

Choosing a database:
  ├─ What is the access pattern?
  │   ├─ Key-value lookups → Redis, DynamoDB
  │   ├─ Complex joins → PostgreSQL
  │   ├─ Free-text search → Elasticsearch
  │   └─ Graph traversal → Neo4j
  ├─ What are the consistency needs?
  │   ├─ Strong ACID → PostgreSQL, MySQL
  │   └─ Eventual consistency OK → DynamoDB, Cassandra
  ├─ What is the write volume?
  │   ├─ < 10K writes/sec → PostgreSQL handles it
  │   └─ > 100K writes/sec → Kafka, Cassandra, DynamoDB
  └─ What is the data lifecycle?
      ├─ Short-lived (cache) → Redis with TTL
      ├─ Append-only (logs) → Kafka, ClickHouse
      └─ Long-lived (records) → PostgreSQL, S3

Operational Considerations

Rule of thumb: Start with one database (usually PostgreSQL). Add specialized stores only when PostgreSQL demonstrably cannot meet a specific access pattern or performance requirement.

---

Saga Patterns

Orchestration vs Choreography

Orchestration Example

// Saga orchestrator — manages the multi-step order process
class OrderSaga {
  async execute(orderId: string): Promise<SagaResult> {
    const steps: SagaStep[] = [
      {
        name: 'reserveInventory',
        execute: () => this.inventoryService.reserve(orderId),
        compensate: () => this.inventoryService.release(orderId),
      },
      {
        name: 'processPayment',
        execute: () => this.paymentService.charge(orderId),
        compensate: () => this.paymentService.refund(orderId),
      },
      {
        name: 'shipOrder',
        execute: () => this.shippingService.schedule(orderId),
        compensate: () => this.shippingService.cancel(orderId),
      },
    ];

    const completedSteps: SagaStep[] = [];

    for (const step of steps) {
      try {
        await step.execute();
        completedSteps.push(step);
      } catch (error) {
        // Compensate in reverse order
        for (const completed of completedSteps.reverse()) {
          await completed.compensate();
        }
        return { status: 'failed', failedStep: step.name, error };
      }
    }

    return { status: 'completed' };
  }
}

Choreography Example

// Each service subscribes to events and publishes next steps

// Inventory service
class InventoryEventHandler {
  async handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
    try {
      await this.inventoryRepo.reserve(event.orderId, event.items);
      await this.eventBus.publish(new InventoryReservedEvent(event.orderId));
    } catch (error) {
      await this.eventBus.publish(new InventoryReservationFailedEvent(
        event.orderId, error.message
      ));
    }
  }

  // Compensation
  async handlePaymentFailed(event: PaymentFailedEvent): Promise<void> {
    await this.inventoryRepo.release(event.orderId);
  }
}

// Payment service
class PaymentEventHandler {
  async handleInventoryReserved(event: InventoryReservedEvent): Promise<void> {
    try {
      await this.paymentProcessor.charge(event.orderId);
      await this.eventBus.publish(new PaymentProcessedEvent(event.orderId));
    } catch (error) {
      await this.eventBus.publish(new PaymentFailedEvent(
        event.orderId, error.message
      ));
    }
  }
}

Saga State Machine

OrderSaga States:
  [Pending] → ReserveInventory
       ├─ Success → [InventoryReserved] → ProcessPayment
       │       ├─ Success → [PaymentProcessed] → ScheduleShipping
       │       │       ├─ Success → [Completed]
       │       │       └─ Failure → RefundPayment → ReleaseInventory → [Failed]
       │       └─ Failure → ReleaseInventory → [Failed]
       └─ Failure → [Failed]

---

Consistency Models

Comparison

Read-Your-Writes Implementation

// Pattern: Write to primary, read from primary for the writing user

class UserProfileService {
  async updateProfile(userId: string, data: ProfileData): Promise<void> {
    await this.primaryDb.users.update(userId, data);

    // Set a "read-from-primary" marker with short TTL
    await this.cache.set(`read-primary:${userId}`, '1', 'EX', 5);
  }

  async getProfile(userId: string, requestingUserId: string): Promise<Profile> {
    // If the requesting user just wrote, read from primary
    const readFromPrimary = await this.cache.get(`read-primary:${requestingUserId}`);

    if (readFromPrimary || userId === requestingUserId) {
      return this.primaryDb.users.findById(userId);
    }

    // Otherwise, read from replica (faster, eventually consistent)
    return this.replicaDb.users.findById(userId);
  }
}

Conflict Resolution Strategies

---

Anti-Patterns

1. Shared Database Across Services

[FAIL] Service A and Service B both read/write to the same tables
       → Tight coupling, schema changes break both services, no independent deployment

[PASS] Each service owns its tables/schema; share data via APIs or events

2. Event Sourcing Everything

[FAIL] Using event sourcing for simple CRUD entities (user profiles, settings)
       → Unnecessary complexity, painful schema evolution

[PASS] Use event sourcing for domains with complex state transitions, audit needs,
       or temporal queries. Use plain CRUD elsewhere.

3. Distributed Transactions (2PC)

[FAIL] Two-phase commit across microservices
       → Blocks on slowest participant, single point of failure, poor availability

[PASS] Use sagas with compensating transactions for cross-service consistency

4. CQRS Without Justification

[FAIL] Applying CQRS to a simple CRUD app with uniform read/write patterns
       → Doubles the code surface, adds sync complexity for no benefit

[PASS] Start with a single model. Split only when read/write patterns diverge
       or independent scaling is needed.

5. Ignoring Event Ordering

[FAIL] Consuming events without partition keys → out-of-order processing
       → Inventory goes negative, payments double-charged

[PASS] Use aggregate ID as partition key. Process events per-partition in order.
       Handle idempotency for at-least-once delivery.

6. Fat Events

[FAIL] Events carrying the entire entity state (100+ fields)
       → Tight coupling, bandwidth waste, hard to evolve

[PASS] Events carry only what changed plus correlation IDs.
       Consumers query for additional data if needed.

---

Decision Framework

Should You Use CQRS?

1. Are read and write models structurally different?       [Yes → +2, No → 0]
2. Is read:write ratio > 10:1?                             [Yes → +2, No → 0]
3. Do you need independent scaling of reads vs writes?     [Yes → +2, No → 0]
4. Is the team experienced with event-driven systems?      [Yes → +1, No → -1]
5. Is this a greenfield project?                           [Yes → +1, No → -1]

Score: 0–3 → Stick with single model
Score: 4–6 → Consider CQRS (same-DB variant first)
Score: 7+  → Strong candidate for full CQRS

Should You Use Event Sourcing?

1. Is audit trail a legal/business requirement?            [Yes → +3, No → 0]
2. Do you need temporal queries ("state at time T")?       [Yes → +2, No → 0]
3. Is the domain event-driven by nature?                   [Yes → +2, No → 0]
4. Can the team handle eventual consistency?               [Yes → +1, No → -2]
5. Are events the natural language of the domain?          [Yes → +1, No → 0]

Score: 0–2 → Use traditional CRUD + audit log
Score: 3–5 → Consider event sourcing for key aggregates
Score: 6+  → Strong candidate for event sourcing

Orchestration vs Choreography

Process involves:
  ├─ 2–3 services, simple flow → Choreography
  ├─ 4+ services, complex branching → Orchestration
  ├─ Need single view of process state → Orchestration
  ├─ Services owned by different teams → Choreography (less central control)
  └─ Strict SLA on completion time → Orchestration (easier to monitor)

---

Cross-References

• modern-patterns.md — Architecture pattern overview including CQRS and event-driven
• scalability-reliability-guide.md — CAP theorem, database scaling, caching strategies
• ../software-backend/references/database-patterns.md — PostgreSQL-specific patterns, connection pooling, migrations
• ../software-backend/references/message-queues-background-jobs.md — BullMQ, Kafka, message broker comparison
• ../../assets/patterns/event-driven-template.md — Event-driven architecture template with saga patterns

Migration & Modernization Guide

Step-by-step patterns for migrating from monoliths to microservices, decomposing databases, and incrementally modernizing legacy systems. Use when planning major refactors, strangler fig migrations, or database decomposition strategies.

Contents

• Strangler Fig Pattern
• Database Decomposition
• Feature Flags for Migration
• Risk Assessment Framework
• Parallel Running and Shadow Traffic
• Migration Path: Monolith to Microservices
• Anti-Patterns
• Decision Framework
• Cross-References

---

Strangler Fig Pattern

Core Concept

Incrementally replace pieces of a legacy system by routing traffic to new implementations while the old system remains operational. Named after strangler fig trees that grow around a host tree.

Phase 1: Identify                Phase 2: Intercept              Phase 3: Replace
┌────────────────┐              ┌────────────────┐              ┌────────────────┐
│   Monolith     │              │   Facade/Proxy │              │   Facade/Proxy │
│  ┌──────────┐  │              │       │        │              │       │        │
│  │ Feature A │  │              │   ┌───┴───┐    │              │   ┌───┴───┐    │
│  │ Feature B │  │              │   │       │    │              │   │       │    │
│  │ Feature C │  │              │ Old A  New B   │              │ New A  New B   │
│  └──────────┘  │              │ Old C          │              │ New C          │
└────────────────┘              └────────────────┘              └────────────────┘

Step-by-Step Implementation

Step 1: Add a routing facade

// API Gateway or reverse proxy routes requests
// Start with 100% traffic to monolith

// nginx.conf or API gateway config
// location /api/orders {
//   proxy_pass http://monolith:3000;   # All traffic to monolith initially
// }

Step 2: Identify extraction candidates

Rank modules by these criteria:

Step 3: Build the new service

// New service implements the SAME interface as the monolith feature
// This is critical — callers should not know about the migration

// New orders-service (standalone)
app.post('/api/orders', async (req, res) => {
  // New implementation with same API contract
  const order = await createOrder(req.body);
  res.json(order);
});

Step 4: Gradual traffic shift

# Canary routing: shift traffic incrementally
# Week 1: 5% to new service
# Week 2: 25% to new service
# Week 3: 50% to new service
# Week 4: 100% to new service (if metrics are green)

# Example: Istio VirtualService
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: orders
spec:
  hosts:
    - orders.example.com
  http:
    - route:
        - destination:
            host: orders-new
          weight: 25
        - destination:
            host: monolith
          weight: 75

Step 5: Decommission the old code

After 100% traffic runs on the new service with stable metrics for at least 2 weeks:

• [ ] Remove old code path from monolith
• [ ] Remove old database tables (after data migration verification)
• [ ] Update documentation and runbooks
• [ ] Remove feature flags related to migration

---

Database Decomposition

Shared Database Problem

[FAIL] Multiple services sharing one database:
  ┌─────────┐  ┌─────────┐  ┌─────────┐
  │Service A│  │Service B│  │Service C│
  └────┬────┘  └────┬────┘  └────┬────┘
       │            │            │
       └────────────┼────────────┘
                    │
            ┌───────▼───────┐
            │  Shared DB    │
            │  (everything) │
            └───────────────┘

Problems:
- Schema changes break multiple services
- No independent deployment
- Performance contention
- Unclear data ownership

Decomposition Strategies

Strategy 1: Database View Layer

Intermediate step. Create views that simulate per-service schemas.

-- Service A sees only its data through a view
CREATE VIEW service_a_orders AS
SELECT id, customer_id, total, status, created_at
FROM orders
WHERE department = 'retail';

-- Service B sees different columns
CREATE VIEW service_b_orders AS
SELECT id, warehouse_id, shipping_status, tracking_number
FROM orders
WHERE shipping_status IS NOT NULL;

Strategy 2: Schema-Per-Service (Same Instance)

Each service gets its own schema within the same database instance.

-- Separate schemas, same PostgreSQL instance
CREATE SCHEMA orders_service;
CREATE SCHEMA inventory_service;
CREATE SCHEMA payments_service;

-- Each service's migrations target its own schema
-- Cross-schema access is explicitly forbidden (enforce via permissions)
REVOKE ALL ON SCHEMA orders_service FROM inventory_user;

Strategy 3: Database-Per-Service (Full Separation)

Each service has its own database instance.

┌─────────┐       ┌─────────┐       ┌─────────┐
│Service A│       │Service B│       │Service C│
└────┬────┘       └────┬────┘       └────┬────┘
     │                 │                 │
┌────▼────┐       ┌────▼────┐       ┌────▼────┐
│  DB A   │       │  DB B   │       │  DB C   │
└─────────┘       └─────────┘       └─────────┘

Data Synchronization During Decomposition

CDC Example with Debezium:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "monolith-db",
    "database.port": "5432",
    "database.dbname": "monolith",
    "table.include.list": "public.orders,public.order_items",
    "topic.prefix": "migration",
    "slot.name": "orders_migration"
  }
}

Join Elimination Strategies

When decomposing a shared database, you lose cross-table joins. Solutions:

---

Feature Flags for Migration

Migration-Specific Flag Patterns

// Flag: route traffic to new vs old implementation
const flags = {
  'orders-service-v2': {
    type: 'percentage',
    value: 25,  // 25% of traffic to new service
    metadata: {
      migration: 'orders-extraction',
      startDate: '2026-01-15',
      targetCompletion: '2026-03-01',
      rollbackPlan: 'Set to 0%, revert proxy config',
    },
  },
};

// Usage in routing layer
async function handleOrderRequest(req: Request): Promise<Response> {
  if (featureFlags.isEnabled('orders-service-v2', { userId: req.userId })) {
    return newOrdersService.handle(req);
  }
  return monolith.handle(req);
}

Flag Lifecycle for Migrations

Phase 1: Create flag (default: OFF)
Phase 2: Enable for internal users (testing)
Phase 3: Enable for 5% of traffic (canary)
Phase 4: Ramp to 25%, 50%, 75%, 100%
Phase 5: Remove flag + old code path (cleanup)

IMPORTANT: Set a cleanup deadline. Stale flags are tech debt.

Comparison Read Pattern

// During migration: read from BOTH, compare results, return old
async function getOrder(orderId: string): Promise<Order> {
  const oldResult = await monolith.getOrder(orderId);

  if (featureFlags.isEnabled('orders-comparison-reads')) {
    try {
      const newResult = await newService.getOrder(orderId);
      if (!deepEqual(oldResult, newResult)) {
        logger.warn('Migration mismatch', {
          orderId,
          diff: generateDiff(oldResult, newResult),
        });
        metrics.increment('migration.comparison.mismatch');
      } else {
        metrics.increment('migration.comparison.match');
      }
    } catch (error) {
      metrics.increment('migration.comparison.error');
    }
  }

  return oldResult; // Always return the trusted source during migration
}

---

Risk Assessment Framework

Migration Risk Matrix

Risk Mitigation Checklist

• [ ] Runbook written for rollback (with specific steps, not "revert changes")
• [ ] Monitoring dashboards ready before migration starts
• [ ] Alerts configured for error rate increase, latency spikes
• [ ] Data reconciliation script ready to compare old vs new
• [ ] Communication plan for stakeholders (downtime windows, expected behavior changes)
• [ ] Rollback tested in staging environment
• [ ] Feature flag wired for instant traffic revert
• [ ] Incremental rollout plan with go/no-go criteria at each stage

Go/No-Go Criteria Per Phase

---

Parallel Running and Shadow Traffic

Shadow Traffic Pattern

Route a copy of production traffic to the new service without affecting users.

┌────────┐     ┌──────────────┐     ┌────────────┐
│ Client │────▶│  Proxy/LB    │────▶│  Monolith  │──▶ Response to client
└────────┘     └──────┬───────┘     └────────────┘
                      │ (async copy)
                      └──────────────▶┌─────────────┐
                                      │ New Service  │──▶ Response discarded
                                      └─────────────┘     (logged + compared)

Implementation

// Shadow traffic middleware
async function shadowTraffic(req: Request, res: Response, next: NextFunction) {
  // 1. Process the real request normally
  next();

  // 2. Asynchronously send a copy to the new service (fire-and-forget)
  if (featureFlags.isEnabled('shadow-traffic-orders')) {
    setImmediate(async () => {
      try {
        const shadowStart = Date.now();
        const shadowResponse = await newService.mirror(req);
        const shadowLatency = Date.now() - shadowStart;

        metrics.histogram('shadow.latency', shadowLatency);

        // Compare responses (logged, not returned to client)
        if (res.locals.responseBody) {
          const match = deepEqual(res.locals.responseBody, shadowResponse);
          metrics.increment(match ? 'shadow.match' : 'shadow.mismatch');
        }
      } catch (error) {
        metrics.increment('shadow.error');
        // Failures in shadow traffic NEVER affect the real response
      }
    });
  }
}

Parallel Running Checklist

• [ ] Shadow traffic does NOT mutate production data
• [ ] Shadow requests are clearly marked (header: X-Shadow: true)
• [ ] New service has separate database/storage from production
• [ ] Comparison results are aggregated into dashboards
• [ ] Shadow traffic can be turned off instantly via feature flag
• [ ] Load from shadow traffic is accounted for in capacity planning

---

Migration Path: Monolith to Microservices

Recommended Progression

Stage 1: Modular Monolith
  └─ Enforce module boundaries within the monolith
  └─ Define clear APIs between modules
  └─ Separate schemas per module (same DB)
  └─ Duration: 2-6 months

Stage 2: Extract First Service
  └─ Pick the module with least coupling
  └─ Use strangler fig + shadow traffic
  └─ Establish service infrastructure (CI/CD, monitoring, service mesh)
  └─ Duration: 1-3 months

Stage 3: Extract Core Services
  └─ Extract 2-4 more services in parallel (different teams)
  └─ Introduce async messaging for cross-service communication
  └─ Decompose database per service
  └─ Duration: 3-9 months

Stage 4: Steady State
  └─ New features built as services by default
  └─ Remaining monolith handles only legacy features
  └─ Monolith shrinks over time (or stays as a module)

Case Study Pattern: E-Commerce Migration

Key principle: Extract the easiest services first to build team confidence and infrastructure. Save the hardest (most coupled, most critical) for last.

---

Migration Plan Binding

Migration plans are only executable when architecture, controls, and rollout mechanics are bound together in a single coherent plan:

• Target-state decisions must be connected to pilot-provider rollout sequencing, degraded modes, cutover controls, readiness checks, and decommission sequencing.
• Architecture proposals alone are not enough — the plan needs explicit stages, open-gap tracking, and clear acknowledgement of what is still unresolved.
• Roadmaps, trackers, and architecture docs must stay synchronized in the same delivery cycle. When they drift, they become contradictory instructions for the next implementation session.
• Distinguish verified current state, selected target state, and unresolved gaps in all migration documentation. Flattening them together makes docs unsafe to trust.

Migration Plan Checklist

• [ ] Target architecture documented with explicit boundaries
• [ ] Pilot-provider or canary rollout sequence defined
• [ ] Degraded mode behavior specified (what happens when new path fails)
• [ ] Cutover controls identified (feature flags, traffic routing, kill switches)
• [ ] Readiness checks listed (what must be true before each phase advances)
• [ ] Decommission sequencing planned (what gets removed, when, and by whom)
• [ ] Open gaps explicitly tracked with owners
• [ ] Roadmap, architecture doc, and gap tracker synchronized

---

Anti-Patterns

1. Big Bang Rewrite

[FAIL] "Let's rewrite the entire monolith as microservices over 6 months"
       → Delayed value delivery, high risk, second-system effect, team burnout

[PASS] Incremental extraction with strangler fig. Ship value every 2-4 weeks.
       Each extraction is independently valuable and rollback-safe.

2. Premature Decomposition

[FAIL] Splitting into 20 microservices before understanding domain boundaries
       → Distributed monolith, wrong service boundaries, expensive to fix

[PASS] Start with a modular monolith. Let boundaries emerge from real usage.
       Extract services only when you have clear, stable bounded contexts.

3. Shared Database Migration

[FAIL] Extracting services but keeping the shared database
       → Services are coupled at the data layer, no independent deployment

[PASS] Database decomposition is part of the migration plan, not an afterthought.
       Use the view layer or schema-per-service as intermediate steps.

4. No Rollback Plan

[FAIL] "We'll figure out rollback if something goes wrong"
       → Panic during incidents, data loss, extended outages

[PASS] Write the rollback runbook BEFORE starting migration.
       Test rollback in staging. Include data reconciliation steps.

5. Ignoring Data Migration

[FAIL] Extracting the service but leaving historical data in the monolith
       → Split brain, queries return partial results, reporting breaks

[PASS] Plan data migration as a first-class concern. Include:
       - Historical data transfer
       - Data format transformation
       - Validation and reconciliation
       - Cutover strategy

6. Migrating Without Observability

[FAIL] Extracting services without distributed tracing or unified logging
       → Impossible to debug issues across service boundaries

[PASS] Set up observability BEFORE the first extraction:
       - Distributed tracing (OpenTelemetry)
       - Centralized logging with correlation IDs
       - Service-level dashboards and alerts

---

Decision Framework

Should You Migrate at All?

1. Are deployment bottlenecks hurting business velocity?    [Yes → +2, No → 0]
2. Do different parts of the system need different scaling? [Yes → +2, No → 0]
3. Are multiple teams blocked by monolith coupling?         [Yes → +2, No → 0]
4. Is the monolith technically healthy (tests, CI, docs)?   [Yes → +1, No → -1]
5. Does the team have distributed systems experience?       [Yes → +1, No → -2]
6. Is there executive buy-in for a multi-quarter effort?    [Yes → +1, No → -2]

Score: 0–3 → Improve the monolith (modularize, add tests, fix CI)
Score: 4–6 → Start with modular monolith, plan first extraction
Score: 7+  → Begin migration with strangler fig approach

Extraction Order Prioritization

Score each module (1-5 per criterion), extract in descending total order:

---

Cross-References

• modern-patterns.md — Architecture patterns overview (microservices, modular monolith)
• data-architecture-patterns.md — CQRS, event sourcing, saga patterns for distributed data
• scalability-reliability-guide.md — Scaling strategies post-migration
• api-gateway-service-mesh.md — Service mesh and gateway patterns for microservices
• ../software-backend/SKILL.md — Service-level implementation patterns
• ../../ops-devops-platform/SKILL.md — CI/CD and deployment for microservices

Core Architecture Questions

Use these questions to frame any design discussion:

• Domain:
• What problem is this system solving?
• What are the core domain concepts and invariants?
• Boundaries:
• How should responsibilities be split across services or modules?
• What must stay together for strong consistency?
• Data:
• What data is stored, where, and in what shape?
• What are the consistency and durability requirements?
• Workload:
• What are the expected read/write patterns?
• How does traffic scale over time (burst vs steady)?
• Failure:
• What happens when dependencies fail or degrade?
• What are the recovery paths and fallbacks?

---

Pattern: Layered Architecture

Use when the system is not yet highly distributed, or when clarity and separation of concerns are more important than aggressive scalability.

Typical layering:

• Presentation/API layer: HTTP or messaging interfaces, authentication, request validation.
• Application/service layer: orchestration, use cases, workflows.
• Domain layer: core business logic and invariants.
• Infrastructure layer: databases, queues, external services, file storage.

Checklist:

• Clear direction of dependencies (outer layers depend on inner, not vice versa).
• Domain code does not depend on transport or storage details.
• Cross-cutting concerns (logging, metrics, security) handled via composition, not duplication.

---

Pattern: Service Decomposition

Use when deciding between monolith, modular monolith, and microservices.

• Monolith:
• Use when the team is small and the problem space is still evolving
• Focus on modular boundaries inside a single deployable
• Modular monolith:
• Use when you need strong internal boundaries and clear ownership but still want a single deployment unit
• Microservices:
• Use when you have clear bounded contexts, strong ownership boundaries, and operational maturity

Decomposition heuristics:

• Group code by domain boundary, not by technical layer only
• Avoid splitting entities that must be updated transactionally
• Prefer a small number of well-designed services over many tiny ones

---

Pattern: Data and Consistency

Use when designing storage and consistency behavior.

• Choose the primary source of truth for each piece of data
• Decide consistency model:
• Strong: transactions and immediate consistency; fewer consumers, higher coupling
• Eventual: asynchronous updates; requires idempotency and reconciliation
• Avoid writing to multiple sources in a single request without a clear strategy:
• Use a "single writer" or orchestrator service
• Use outbox patterns for publishing events reliably

Checklist:

• Data ownership is clear for each service
• Failure modes for partial writes are understood and handled
• Migrations and schema evolution have a plan (backwards compatibility where needed)

---

Pattern: Request-Driven vs Event-Driven

Use this pattern to decide between synchronous and asynchronous flows.

• Request-driven (synchronous):
• Good for user-facing APIs needing immediate feedback
• Latency and availability of dependencies directly affect the caller
• Event-driven (asynchronous):
• Good for decoupling producers and consumers
• Suited to background work, aggregations, notifications

Guidelines:

• Keep request paths shallow for user interactions; offload heavy work via events or queues
• In event-driven flows, design idempotent handlers and clear retry behaviors

---

Pattern: Security Architecture

Use when: Designing secure system architectures with proper security layers and controls.

IMPORTANT: For comprehensive security patterns, see ../software-security-appsec/SKILL.md which covers:

• Defense in depth and security boundaries
• Zero trust architecture
• Threat modeling (STRIDE framework)
• Authentication & Authorization architecture
• Data protection at rest and in transit
• Secure design principles

Architecture-Specific Security Patterns:

• Security Boundaries: Define trust boundaries between layers (client → API → services → data)
• Defense in Depth: Layer security controls (authentication → authorization → validation → encryption)
• Least Privilege: Each service should have minimum necessary permissions
• Fail Securely: Systems should default to deny, not allow on errors
• API Gateway Pattern: Centralized authentication, rate limiting, logging
• Service Mesh: mTLS between services, zero trust networking
• Secret Management: Centralized secret storage (AWS Secrets Manager, Vault)

Quick Example - Security Layers:

Client
  ↓ HTTPS + JWT
API Gateway (Auth, Rate Limit, Logging)
  ↓ Service-to-Service Auth
Backend Services (Authorization, Validation)
  ↓ Encrypted Connection
Database (Encrypted at Rest, Row-Level Security)

Checklist:

• Authentication at entry points
• Authorization on every service call
• Input validation at boundaries
• Encryption in transit (TLS)
• Encryption at rest for sensitive data
• Centralized logging for security events
• Rate limiting and DDoS protection
• Regular security audits and penetration testing

---

External Resources

See data/sources.json for 42 curated references on:

• Architecture pattern catalogs (AWS, Azure, Google Cloud, Martin Fowler, microservices.io)
• Distributed systems and scalability guides (ByteByteGo, System Design)
• ADR templates and best practices (GitHub ADR org, AWS, Microsoft)
• Scalability and reliability (Google SRE Book, CAP theorem, resilience patterns)
• Event-driven architecture (AWS, Martin Fowler, CQRS, Saga patterns)
• Observability (OpenTelemetry, distributed tracing, SLI/SLO/SLA)
• API design (REST, GraphQL, gRPC, API Gateway patterns)
• Security architecture (OWASP, Zero Trust, Service Mesh security)
• Essential books (Designing Data-Intensive Applications, Building Microservices, Release It!)