Senior System Architecture

Senior System Architecture

When to Use

• Define or review cross-service/system boundaries and contracts
• Compare architectural options with explicit trade-offs and NFR impact
• Author or review ADRs for decisions that are hard to reverse
• Run architecture review before major launches or vendor commitments
• Plan migration off legacy systems (strangler, parallel run, cutover)
• Establish architecture principles, standards, and exception process

When NOT to Use

• Single-team service RFC and implementation slices → senior-software-engineer
• Lakehouse, mesh, or enterprise data modeling → data-architect
• VPC, Kubernetes, Terraform, or CI/CD build → infrastructure-engineer, devops
• IDP, golden paths, Backstage → platform-engineer
• Milestones, RAID, steering status → technical-program-manager
• Release cutover tactics only → deployment-strategist
• Security control catalog and enterprise GRC → cybersecurity
• LLM/RAG system design → ai-engineer
• Business strategy, issue trees, steerCo cases → business-consultant
• Applied AI / LLM solution architecture → applied-ai-architect-commercial-enterprise
• Customer-facing solution design, RFP/RFI, PoC scope, deal integration → solutions-architect

Related skills

Core Workflows

1. Frame the decision

Capture before drawing boxes:

• Business outcome and measurable success criteria
• Constraints: budget, timeline, compliance, existing estate
• Non-goals (explicit scope cuts)
• Stakeholders and decision owner
• Reversibility: one-way door vs two-way door

One-way doors require ADR + architecture review. Two-way doors can stay in team RFC.

2. Model the system (C4-lite)

Minimum views for reviewers:

1. Context — actors, external systems, trust zones 2. Containers — deployable units, data stores, queues, who owns each 3. Critical path — sequence diagram for highest-risk flows only

Label every arrow: sync/async, protocol, auth model, and failure behavior.

See `references/integration_patterns.md`.

3. Define NFRs and quality attributes

For each capability, specify targets (not vague "high availability"):

See `references/nfr_quality_attributes.md`.

4. Evaluate options

Present at least two viable options plus "do nothing / minimal change":

Recommend one; document rejected options and why.

See `references/adr_template.md`.

5. Architecture review

Before build or contract signature:

1. Problem and constraints restated in one paragraph 2. Diagrams current; contracts versioned (OpenAPI, event schema) 3. Failure modes: partial outage, dependency down, poison messages 4. Data: ownership, retention, PII, migration path 5. Observability: golden signals per container 6. Security: authz boundaries, secrets, blast radius 7. Rollout and rollback linked to deployment-strategist if multi-phase

See `references/architecture_review.md`.

6. Evolution and migration

For legacy replacement:

1. Identify capability slices that can move independently 2. Prefer strangler over big-bang when risk is high 3. Define parity criteria before cutover 4. Plan dual-write / dual-read duration and reconciliation 5. Deprecate old path with telemetry proving zero traffic

See `references/migration_evolution.md`.

When to load references

• ADR format and bar → references/adr_template.md
• Review checklist → references/architecture_review.md
• Sync, events, sagas, APIs → references/integration_patterns.md
• SLOs, capacity, DR, cost → references/nfr_quality_attributes.md
• Strangler and cutover → references/migration_evolution.md

ADR template

Table of contents

1. When to write an ADR 2. Template 3. Quality bar

When to write an ADR

Write an ADR when:

• Multiple teams depend on the decision
• Reversal cost is high (data model, public API, vendor, network topology)
• The decision sets a precedent others will copy

Skip ADR for reversible team-local choices; use a short RFC in senior-software-engineer instead.

Template

# ADR-NNN: [Short title]

**Status:** Proposed | Accepted | Deprecated | Superseded by ADR-XXX
**Date:** YYYY-MM-DD
**Deciders:** [names/roles]

## Context

What forces the decision? Include constraints and prior art.

## Decision

We will [clear statement]. Because [primary reason].

## Consequences

### Positive
- ...

### Negative / trade-offs
- ...

### Risks and mitigations
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|

## Alternatives considered

### [Alternative A]
- Pros: ...
- Cons: ...
- Why rejected: ...

## Compliance with principles

Reference org principles (e.g., prefer managed services, event-first for async).

## Follow-up

- [ ] Tasks, owners, dates
- [ ] Metrics to validate decision in production

Quality bar

Reject or send back ADRs that:

• Recommend without comparing alternatives
• Omit operational ownership (who runs it at 3 a.m.)
• Ignore security, cost, or migration
• Use "best practice" without tying to constraints

Supersede; do not silently edit accepted ADRs—link forward to the new record.

Architecture review

Table of contents

1. Review types 2. Pre-read package 3. Checklist 4. Outcomes

Review types

Pre-read package

Send 24h before review:

• Problem statement (≤ 200 words)
• C4 context + container diagrams
• ADR draft or link
• NFR table with targets
• Open questions list

Checklist

Scope and fit

• [ ] Solves stated problem; non-goals explicit
• [ ] Fits architecture principles or documents exception
• [ ] Build vs buy justified with TCO, not preference

Boundaries and contracts

• [ ] Clear ownership per container
• [ ] APIs/events versioned; backward compatibility plan
• [ ] No inappropriate shared databases across teams

Reliability

• [ ] Single points of failure identified
• [ ] Timeouts, retries, idempotency on external calls
• [ ] Bulkheads / rate limits where needed
• [ ] DR story matches stated RPO/RTO

Security and privacy

• [ ] Trust zones and authn/authz per boundary
• [ ] Secrets not in app config plaintext
• [ ] PII flow and retention documented
• [ ] Blast radius of compromise bounded

Operability

• [ ] Logs, metrics, traces per golden signals
• [ ] Runbooks or playbooks for top failure modes
• [ ] Feature flags or safe rollout path

Data

• [ ] System of record identified
• [ ] Consistency model explicit (sync vs eventual)
• [ ] Migration/reconciliation if dual systems

Cost and scale

• [ ] Rough capacity model (orders of magnitude)
• [ ] Cost drivers named (egress, storage, licensed seats)

Outcomes

Record one of:

• Approved — proceed; note conditions
• Approved with actions — fix listed items before build
• Revise and resubmit — major gaps
• Deferred — need spike or more data

Assign action owners and dates in the review notes.

Integration patterns

Table of contents

1. Choosing a pattern 2. Sync integration 3. Async and events 4. Distributed workflows 5. Anti-patterns

Choosing a pattern

Sync integration

REST/GraphQL/gRPC:

• Version APIs; never break without deprecation window
• Set timeouts < client patience; retry only idempotent ops
• Use correlation IDs across calls
• Circuit break when dependency error rate spikes

BFF:

• Own aggregation and response shaping for one client class
• Do not own business rules that belong in domain services
• Cache with explicit TTL and invalidation rules

Async and events

Event contract:

• Schema registry or documented JSON/Avro/Protobuf
• eventType, schemaVersion, occurredAt, correlationId, idempotencyKey
• Define ordering guarantees (per partition key only)

Consumer rules:

• Idempotent processing (natural key or dedup store)
• Poison message: DLQ + alert + replay tooling
• At-least-once is default; design for duplicates

Outbox pattern:

• Write business row + outbox in same DB transaction
• Publisher reads outbox—avoids dual-write races

Distributed workflows

Saga (orchestrated or choreographed):

• Each step compensates or marks terminal failure
• Document partial failure: what the user sees
• Avoid long synchronous chains across services

CQRS (when justified):

• Separate write model and read projections
• Accept eventual consistency on reads; document lag SLO

Anti-patterns

• Distributed monolith — many services, one database, coupled deploys
• Chatty sync mesh — N×M point-to-point without gateway or events
• Events as command bus — fire-and-forget without consumer contract
• Shared library for domain logic — hides boundaries; use contracts instead

For service-level boundary heuristics, see senior-software-engineer references.

Migration and evolution

Table of contents

1. Strategies 2. Strangler fig pattern 3. Dual-write and dual-read 4. Cutover and rollback 5. Deprecation

Strategies

Default to strangler for customer-facing systems with uptime SLOs.

Strangler fig pattern

1. Put facade (gateway, router, or API layer) in front of legacy 2. Route new capabilities to new system only 3. Migrate read paths first when safe (often lower risk) 4. Migrate writes with reconciliation 5. Retire legacy module when traffic and data parity proven

Track per-slice: % traffic, % data migrated, defect rate vs legacy.

Dual-write and dual-read

Dual-write:

• Write to old and new; reconcile on mismatch
• Limit duration; define max drift window
• Use idempotency keys on both sides

Dual-read:

• Read new; compare to old in shadow (log diffs only)
• Promote new read when diff rate below threshold

Reconciliation job:

• Scheduled compare + quarantine bad rows
• Human playbook for irreconcilable records

Cutover and rollback

Coordinate with deployment-strategist:

• Go criteria: parity metrics, error rate, support ticket volume
• Rollback criteria: SLO burn, data corruption signal, failed reconciliation
• Freeze window: communicate to support and CS

Run dress rehearsal on production-like data volume when possible.

Deprecation

1. Announce timeline (internal + external if API) 2. Metric: callers still on old path 3. Sunset with 410/structured error after window 4. Archive data per retention policy 5. Remove code and infra; close ADR as superseded

Document lessons learned; update architecture principles if gaps found.

NFR and quality attributes

Table of contents

1. NFR worksheet 2. Availability and reliability 3. Performance and scale 4. Security and compliance 5. Cost and sustainability

NFR worksheet

Copy per capability:

| NFR | Target | Measurement | Owner |
|---|---|---|---|
| Availability | e.g. 99.95% / 30d | SLO dashboard | Platform |
| Latency (p99) | e.g. 500ms | APM | Service team |
| Throughput | e.g. 2k RPS | Load test + prod | Service team |
| Durability | RPO 15m, RTO 2h | DR test annual | Infra |
| Data retention | 7y audit logs | Policy + job | Compliance |
| Cost | $X / month at peak | FinOps tag | Eng + Finance |

If target is unknown, schedule a spike—do not ship "TBD" to production.

Availability and reliability

Error budget mindset:

• Pick SLO below 100%; budget funds velocity vs reliability debates
• Multi-AZ is baseline for stateless; stateful needs explicit failover design
• Dependencies inherit your SLO—map critical path vendors

Failure design:

• Graceful degradation (read-only mode, cached responses)
• Health checks: liveness vs readiness
• Chaos or game days for tier-1 paths annually

Performance and scale

Capacity sketch:

peak_RPS × payload_bytes × fan_out = egress_pressure
active_users × writes_per_day = storage_growth_per_year

Load test the critical path at 2× expected peak before launch.

Watch: connection pools, thread pools, GC pauses, cold starts (serverless).

Security and compliance

Align with cybersecurity / information-security-engineer for control mapping.

Architecture must state:

• Data classification per store
• Encryption in transit and at rest
• Identity: human vs machine, least privilege, break-glass
• Audit logging for sensitive actions

Cost and sustainability

• Tag resources by service/team for chargeback
• Right-size before multi-region expansion
• Prefer managed services when operational cost > license delta
• Document cost of scale (egress, index size, per-seat SaaS)

For FinOps implementation detail, pair with infrastructure-engineer / devops.