Mission Critical

Mission-Critical Systems

When to Use

• Classify criticality tiers (mission-critical, business-critical, important) with explicit impact criteria
• Set availability, integrity, and continuity objectives (uptime, RTO/RPO, MTPD, data loss tolerance)
• Map dependencies, failure domains, and blast radius for severe-impact services
• Select architecture patterns—active-active, geo-redundancy, quorum, deterministic execution, fail-safe defaults
• Define change and release governance—hold points, CAB tiers, emergency change, rollback criteria
• Tailor monitoring, alerting, and escalation to criticality (SLOs alone are insufficient for Tier 0)
• Align engineering with contractual SLAs, sector regulation, and continuity obligations (generic framing)
• Produce criticality registers, architecture decision records, and governance briefs for leadership

When NOT to Use

• Own SLI/SLO definitions, error-budget policy, burn-rate alerting, and toil reduction → site-reliability-engineer
• Establish failure-prevention culture, HRO principles, stop-the-line, and defect-escape metrics → zero-tolerance-for-failure
• Run live incident war room, containment, and forensic preservation → incident-responder
• Design backup/immutability, ransomware recovery, and resilience test evidence → cyber-resilience-engineer
• Own enterprise BCM program, BIA facilitation, and regulatory BCM policy → bcm-disaster-recovery-specialist
• Enterprise security reference architecture and control frameworks without tiering lens → enterprise-security-architect
• Greenfield integration patterns and ADRs without criticality classification → senior-system-architecture
• Cleared program ATO, inspection, and personnel security without operational tiering → classified-cyber-security-senior-manager
• CI compile/lint gates without release governance for critical services → build-validator

Related skills

Core Workflows

1. Scope and criticality tiering

Define what “mission-critical” means in context; classify services and data; document drivers.

See `references/mission_critical_scope_and_tiering.md`.

2. Objectives: availability, integrity, continuity

Set measurable targets and exclusions; align tiers to RTO/RPO/MTPD and integrity classes.

See `references/objectives_availability_integrity_continuity.md`.

3. Architecture patterns for critical systems

Choose redundancy, partitioning, determinism, and degradation modes for Tier 0/1 workloads.

See `references/architecture_patterns_for_critical_systems.md`.

4. Dependencies and blast radius

Map upstream/downstream, shared fate, control planes, and containment boundaries.

See `references/dependencies_and_blast_radius.md`.

5. Change control and release governance

Tier releases, evidence gates, emergency change, and rollback for high-criticality paths.

See `references/change_control_and_release_governance.md`.

6. Operations, monitoring, and escalation

Observability depth, synthetic probes, runbooks, and escalation matched to impact.

See `references/operations_monitoring_and_escalation.md`.

Outputs

• Criticality register — tier, owner, impact narrative, objectives, last review
• Objectives sheet — availability, integrity, RTO/RPO/MTPD per tier with measurement method
• Dependency / blast-radius map — failure domains, shared components, isolation notes
• Architecture decision pack — pattern choice, trade-offs, degradation and fail-safe behavior
• Release governance matrix — change type × tier → gates, approvers, evidence, rollback
• Operations brief — alert routes, escalation tree, runbook index, drill calendar hooks

Principles

• Tier before tooling — criticality drives architecture and governance; do not retrofit labels after build
• Integrity equals availability — corruption or unauthorized change can be worse than downtime
• Contain blast radius — shared control planes and “single throat to choke” are Tier 0 risks
• Govern change proportionally — higher tier → more evidence, independence, and rollback discipline
• Measure what customers and regulators care about — pair SRE SLIs with tier-specific continuity proofs
• Pair with peers — reliability math (SRE), prevention culture (zero-tolerance), recovery (resilience), incidents (IR)

Architecture patterns for critical systems

Table of contents

1. Pattern selection 2. Redundancy and active-active 3. Geographic and zone isolation 4. Determinism and overload behavior 5. Fail-safe, fail-closed, degradation 6. Control plane vs data plane 7. Anti-patterns

Pattern selection

Choose patterns from tier, failure modes, and integrity class:

Document decisions in ADRs; pair senior-system-architecture for general NFR templates.

Redundancy and active-active

Mission-critical default: prefer proven failover over theoretical multi-master unless conflict resolution is specified and tested.

Geographic and zone isolation

• Failure domains — account, region, AZ, cluster, rack; map on blast-radius diagram
• Traffic steering — health-based routing; avoid flapping during partial degradation
• Data residency — sovereignty constraints as hard requirements (generic regulatory framing)
• Clocks and ordering — use logical clocks or external coordination where wall-clock unsafe

Pair cloud-engineer for regional placement; you own tier fit and continuity proof.

Determinism and overload behavior

Tier 0/1 systems should behave predictably under stress:

Avoid “best effort” background jobs on Tier 0 paths without isolation.

Fail-safe, fail-closed, degradation

Align with zero-tolerance-for-failure:

Define degradation levels (L0 full → L3 minimal) with customer-visible behavior and rollback triggers.

Control plane vs data plane

Never assume “internal only” control planes are low tier if Tier 0 services depend on them.

Anti-patterns

• Stretch cluster across regions without split-brain design
• Shared database for unrelated Tier 0 domains
• Silent fail-open on auth or validation for availability
• Chaos in production without tier approval and rollback (site-reliability-engineer coordinates)
• Undocumented manual runbooks as the only failover path

Change control and release governance

Table of contents

1. Governance principles 2. Change categories 3. Tiered release gates 4. Emergency change 5. Rollback and verification 6. Interfaces with SRE and prevention

Governance principles

High-criticality change management optimizes for predictable outcomes, not maximum deploy frequency:

• Evidence before production — tests, peer review, security review proportionate to tier
• Independence — separate build, verify, approve roles for Tier 0 (zero-tolerance-for-failure)
• Reversibility — every Tier 0/1 change has tested rollback or forward-fix with same rigor
• Audit trail — who approved, what artifact hash, what window, what validation ran
• Freeze alignment — honor error-budget freeze and regulatory/contractual blackouts

Change categories

Tag every change record with service tier and blast-radius estimate.

Tiered release gates

Illustrative gate matrix—adapt to org process:

Hold points: pipeline must stop until human approval for Tier 0 production deploy.

Emergency change

Allowed when active harm exceeds delay risk:

1. Declare emergency change ID linked to incident if applicable 2. Minimum approvers — named roster (not “any manager”) 3. Scope limit — smallest change that mitigates; defer unrelated fixes 4. Real-time comms — ops bridge + stakeholder awareness 5. PIR within 48–72h — retroactive evidence, test gap, tier appropriateness

Document why normal path was unsafe; repeated emergencies signal tier or architecture debt.

Rollback and verification

Interfaces with SRE and prevention

Your deliverable: release governance matrix mapping change type × tier → gates, owners, evidence.

Dependencies and blast radius

Table of contents

1. Dependency mapping scope 2. Blast radius concepts 3. Mapping workflow 4. Shared fate and choke points 5. Containment strategies 6. Outputs and maintenance

Dependency mapping scope

For each Tier 0/1 service, document:

Mark direction (sync/async), criticality of path, and fallback if dependency fails.

Blast radius concepts

Tier 0 design goal: limit blast radius even when availability SLO is momentarily met (e.g., corrupt downstream).

Mapping workflow

1. Anchor on Tier 0/1 service and primary user journeys 2. Walk the request path — ingress → app → data → async consumers 3. Walk the change path — commit → build → deploy → config → secrets 4. Walk the ops path — metrics, logs, traces, paging, runbooks 5. Label each node with tier, owner, and max outage if node fails alone 6. Identify cycles — mutual dependencies (A→B→A) are high risk 7. Validate with game day or tabletop; update quarterly minimum

Use dependency truth from CMDB/service catalog; challenge “unknown” edges for Tier 0.

Shared fate and choke points

Flag components that concentrate risk:

For each choke point: mitigation (shard, isolate, dual-vendor, break-glass) or accepted risk with sponsor sign-off.

Containment strategies

Pair cyber-resilience-engineer when dependency loss is attack-driven (IdP, logging, backup).

Outputs and maintenance

Trigger refresh on: new Tier 0 promotion, major vendor change, merger integration, or post-incident learning.

Mission-critical scope and tiering

Table of contents

1. Purpose and boundaries 2. Criticality tiers 3. Classification criteria 4. Sector and contractual drivers 5. Stakeholders and RACI 6. Deliverables and review cadence

Purpose and boundaries

Mission-critical systems engineering frames how services are classified, designed, operated, and governed when outage or corruption causes severe harm—safety, legal/regulatory breach, large financial loss, or loss of essential public function.

This skill is a design and governance lens. It complements:

You own tiering, objectives, and proportional controls—not incident command, BCM policy ownership, or SRE math alone.

Criticality tiers

Use a small, enforced taxonomy (example—adapt names to org glossary):

Rules:

• One tier per service in the register; subcomponents may differ only with documented decomposition
• Tier 0/1 require named service owner, deputy, and executive sponsor
• Down-tiering needs written rationale and risk acceptance—not budget convenience

Classification criteria

Score each dimension; highest sustained score drives tier (document overrides):

Anti-patterns: labeling everything Tier 0; tiering by team prestige; ignoring integrity and covert change scenarios.

Sector and contractual drivers

Frame drivers generically—do not invent customer-specific obligations:

Extract contractual SLAs, audit clauses, and notification timelines into the criticality register as constraints, not legal advice.

Stakeholders and RACI

Deliverables and review cadence

Escalate unresolved tier disputes to architecture review with explicit residual risk statement.

Objectives: availability, integrity, continuity

Table of contents

1. Objective families 2. Availability targets 3. Integrity and authenticity 4. Continuity: RTO, RPO, MTPD 5. Measurement and exclusions 6. Tier-to-objective mapping

Objective families

Mission-critical engineering treats three families as co-equal:

Pair SRE SLIs (site-reliability-engineer) with tier-specific continuity proofs—error budgets alone do not satisfy Tier 0 obligations.

Availability targets

Define per user journey or control loop, not per server:

• Target — e.g., 99.95% monthly for Tier 1 payment initiation (illustrative; set per contract)
• Window — calendar month vs rolling 28d; align with SLA text
• Scope — regions, channels, customer segments included
• Dependencies — explicit upstream exclusions (identity, network) documented

Degraded modes: specify minimum viable function when full feature set is unavailable (read-only, queue-and-replay, manual fallback).

Integrity and authenticity

For Tier 0/1, specify:

Integrity SLO examples: max undetected ledger divergence duration; max time to detect unauthorized config drift.

Continuity: RTO, RPO, MTPD

Document RTA (actual recovery in tests) vs RTO (target)—gaps drive backlog.

Measurement and exclusions

Report budget burn (SRE) and continuity margin (time to MTPD breach) on one leadership slide for Tier 0/1.

Tier-to-objective mapping

Illustrative defaults—replace with org standards:

Every Tier 0/1 row links to last successful drill date and open gap count.

Operations, monitoring, and escalation

Table of contents

1. Observability by tier 2. Alerting philosophy 3. Escalation and command 4. Runbooks and drills 5. Regulatory and contractual operations 6. Metrics for leadership

Observability by tier

Tier 0/1 require defense in depth beyond default infra monitoring:

Pair implementation detail with site-reliability-engineer; you define what must exist per tier.

Alerting philosophy

Anti-pattern: hundreds of low-signal pages that train on-call to ignore Tier 0.

Escalation and command

Define escalation tree per Tier 0/1 service:

Distinguish technical incident command (incident-responder, incident-management-engineer) from business crisis (bcm-disaster-recovery-specialist).

Document communication cadence inputs (status, ETA, customer impact)—not full comms ownership.

Runbooks and drills

Failed drill = open risk until remediated or risk-accepted at sponsor level.

Regulatory and contractual operations

Generic operational hooks (not legal advice):

Map clocks to monitoring and paging—“we would learn too late” is a design defect.

Metrics for leadership

Present one page per quarter: tier summary, top risks, investment asks—pair financial BCM views when needed.