Microservices Developer

Microservices Developer

When to Use

• Decompose a monolith or greenfield system into bounded contexts and service boundaries
• Design REST, gRPC, or event contracts between services with clear ownership
• Choose sync vs async communication and document failure semantics
• Implement resilience—timeouts, retries with jitter, circuit breakers, bulkheads, load shedding
• Enforce database-per-service (or schema-per-service) and avoid shared mutable stores
• Apply saga, outbox, or idempotent consumers for cross-service consistency
• Containerize services with twelve-factor config, health checks, and graceful shutdown
• Add observability—correlation/trace IDs, RED metrics, structured logs, trace propagation
• Plan API versioning, deprecation, and backward compatibility at gateway or mesh edge
• Introduce contract tests or consumer-driven contract checks between teams

When NOT to Use

• Operate Kubernetes clusters, Helm platform add-ons, or cluster SRE only → platform-engineer, cluster-deployment-engineer
• Define org-wide SLO programs, error budgets, and PRR gates → site-reliability-engineer
• Design enterprise iPaaS, canonical enterprise models, or B2B integration hubs → enterprise-integration-api-developer
• Build monolith features, general RFCs, or stack-agnostic code review without service split → senior-software-engineer
• Implement CI/CD pipelines, GitOps, or release automation only → devops
• Provision VPC, managed cloud services, or landing zones → cloud-engineer, infrastructure-engineer
• Gate production builds and artifact promotion policy → build-validator
• Profile p99 latency and run load/soak tests as the main task → performance-engineer
• Classified air-gapped pipelines, ATO evidence, cleared promotion → classified-software-devsecops-engineer

Related skills

Core Workflows

1. Scope and boundaries

Map domains, define service APIs, and document non-goals.

See `references/microservices_developer_scope.md` and `references/service_boundaries_and_design.md`.

2. Communication and contracts

Pick sync/async patterns; define schemas, errors, and versioning.

See `references/communication_sync_async.md`.

3. Resilience and reliability

Apply timeouts, retries, breakers, and failure isolation per dependency.

See `references/resilience_and_reliability.md`.

4. Data, events, and consistency

Own data per service; use outbox/saga where cross-service invariants matter.

See `references/data_consistency_and_events.md`.

5. Operate, test, and ship

Observability, contract tests, twelve-factor deploy, gateway compatibility.

See `references/observability_testing_deployment.md`.

Outputs

• Service map — contexts, APIs, data ownership, sync/async edges
• Contract draft — OpenAPI/proto/event schema with error model and versioning note
• Resilience table — per-dependency timeout, retry, breaker, fallback
• Consistency note — saga/outbox/idempotency choice with failure compensation
• Runbook snippet — health checks, dashboards, rollback triggers

Principles

• Prefer fewer, cohesive services over fine-grained chatter; split on change cadence and team boundaries
• Fail fast with explicit timeouts; never unbounded blocking across the network
• Design for partial failure—degrade features, do not cascade outages
• Make contracts testable before production coupling multiplies

Communication: sync and async

Table of contents

1. When to use sync 2. When to use async 3. Hybrid patterns 4. Gateway and mesh

When to use sync

HTTP/REST or gRPC when the caller needs an immediate outcome or strong read-after-write.

Rules:

• Set end-to-end deadline shorter than client timeout
• Propagate correlation ID and trace context on every hop
• Return 429/503 with Retry-After when overloaded; document retry safety

When to use async

Message bus, log, or outbox when decoupling, buffering, or fan-out matters.

Rules:

• Design idempotent consumers (event_id, business key)
• Define ordering scope (partition key = aggregate id)
• Plan poison message handling—DLQ, replay tooling, alerts

Hybrid patterns

Choreography vs orchestration (sagas):

• Choreography — each service reacts to events; fewer central failures; harder to debug
• Orchestration — coordinator drives steps; clearer state machine; coordinator is critical path

Pick orchestration when steps, timeouts, and compensation are complex.

Gateway and mesh

Version APIs with URL prefix (/v2/) or header (Accept-Version); never break consumers without deprecation window.

For OpenAPI governance at enterprise scale → enterprise-integration-api-developer.

Data consistency and events

Table of contents

1. Data ownership 2. Consistency models 3. Transactional outbox 4. Sagas

Data ownership

Shared database is a temporary migration state only—document exit criteria.

Consistency models

Avoid two-phase commit (2PC) across services unless platform mandates and team can operate it.

Transactional outbox

Problem: DB commit and message publish must not diverge.

Pattern:

1. Business row + outbox row in same DB transaction 2. Relay process polls outbox → publishes to broker → marks sent 3. Consumers idempotent on event_id

Alternative: CDC (Debezium) from DB log—ops-heavy, strong for analytics pipelines.

Sagas

Long-running business processes across services.

Choreography saga:

• Each service listens and emits events
• Compensation events (PaymentFailed → OrderCancelled)
• Requires clear event catalog and idempotency

Orchestration saga:

• Coordinator stores state machine (PENDING_PAYMENT, COMPLETED)
• Calls participants with timeouts; runs compensating calls on failure
• Easier tracing; coordinator must be HA

Idempotency keys on every participant command.

Duplicate saga/outbox depth at enterprise integration layer → enterprise-integration-api-developer.

Microservices developer scope

Table of contents

1. Role focus 2. Typical deliverables 3. Decision boundaries 4. Anti-patterns

Role focus

Typical deliverables

• Context map and service catalog (name, owner, SLA tier)
• Sequence diagrams for critical cross-service flows
• OpenAPI, protobuf, or event schema with compatibility rules
• Resilience matrix per downstream dependency
• Data ownership matrix (system of record, read models, caches)
• Contract-test plan (provider/consumer, Pact or schema-diff in CI)

Decision boundaries

Split a service when:

• Different scaling or availability profiles (e.g., read-heavy catalog vs write-heavy orders)
• Independent release cadence and team ownership
• Clear bounded context with stable ubiquitous language
• Regulatory or blast-radius isolation

Keep together when:

• High-frequency chatty calls with shared transactions
• No stable domain seam; split would only add latency
• Team cannot operate two deployables safely yet

Anti-patterns

• Distributed monolith — many services, one shared database, coordinated deploys
• Synchronous chains — A→B→C→D on user-facing path without budgets
• Leaky ownership — multiple writers to the same tables
• Retry storms — retries without jitter on overloaded dependencies
• Contractless coupling — breaking schema changes without consumer notice

Observability, testing, and deployment

Table of contents

1. Observability 2. Testing strategy 3. Twelve-factor deployment 4. API versioning and rollout

Observability

Three pillars per service:

Dashboards: one row per service SLO; dependency health; breaker state if exposed.

Alerts: user-journey burn rate, not only CPU; page on missing heartbeats for critical workers.

Trace and SLO program design → site-reliability-engineer. Pipeline metrics → devops.

Testing strategy

Contract testing (conceptual):

• Consumer-driven contracts — consumer defines expected interactions; provider verifies before release
• Schema registry — Avro/Protobuf/JSON Schema compatibility checks (BACKWARD, FULL)
• Breaking change gate — diff OpenAPI/proto in PR; fail on incompatible without major version

Test doubles for dependencies: use wire mocks or test containers; avoid shared staging for unit-level contracts.

Twelve-factor deployment

Container build and promotion gates → devops, build-validator.

API versioning and rollout

Backward compatible changes: add optional fields, new endpoints; never rename or change type in place.

Rollout: blue/green or canary at gateway/mesh; feature flags inside service for risky logic.

Cutover planning and change tiers → deployment-strategist.

Resilience and reliability

Table of contents

1. Timeouts and deadlines 2. Retries 3. Circuit breakers and bulkheads 4. Load shedding and fallbacks

Timeouts and deadlines

Never use infinite waits. Document per-dependency budgets in a table:

Dependency | p99 latency | Timeout | Notes
-----------|-------------|---------|------
payments   | 120ms       | 300ms   | retry idempotent GET only
inventory  | 80ms        | 200ms   | breaker after 50% errors

Retries

Retry only when:

• Operation is idempotent (safe key, dedup store, or read)
• Failure is transient (timeouts, connection reset, 503, gRPC UNAVAILABLE)

Do not retry: 4xx business errors, validation failures, or when breaker is open.

Circuit breakers and bulkheads

Circuit breaker states: closed → open (fail fast) → half-open (probe).

• Open after error rate or consecutive failures exceed threshold
• Half-open allows limited probes before closing again

Bulkhead isolates resources:

• Separate thread pools / connection pools per dependency
• Queue depth limits; reject early under pressure

Bulkhead + breaker prevent one slow dependency from starving the service.

Load shedding and fallbacks

Log degraded mode with metric degraded=true for SRE visibility.

Reliability program ownership (SLO burn, PRR) → site-reliability-engineer.

Service boundaries and design

Table of contents

1. Bounded contexts 2. Context mapping 3. Service API design 4. Decomposition workflow

Bounded contexts

A bounded context is a model boundary where terms and rules are consistent.

Context mapping

Relationships between contexts (from strategic DDD):

Document on a context map diagram before cutting services.

Service API design

REST (HTTP):

• Resource-oriented URLs; use POST for commands when actions are not CRUD
• Standard error envelope: code, message, details, trace_id
• Pagination: cursor preferred for large sets; stable sort keys
• Idempotency: Idempotency-Key header on mutating operations

gRPC:

• Prefer for internal high-throughput, typed contracts
• Define deadlines on every call; propagate metadata (traceparent, tenant)
• Use UNAVAILABLE / DEADLINE_EXCEEDED for retry decisions; not for business errors

Events:

• Name events in past tense (OrderPlaced); include schema version
• Partition keys preserve per-entity ordering
• Consumers must be idempotent (event_id dedup)

Decomposition workflow

1. Identify core domains and supporting/generic subdomains 2. Draw context map; mark ACLs where legacy exists 3. List transactions that must stay atomic vs can be eventual 4. Prototype strangler route: extract read path or async boundary first 5. Define contract tests before splitting databases 6. Migrate data with dual-write or CDC only with explicit cutover plan

For enterprise-wide integration hubs and canonical models → enterprise-integration-api-developer.