Zero Tolerance For Failure

Zero-Tolerance for Failure

When to Use

• Establish failure-prevention culture and operating norms for mission-critical systems
• Apply HRO principles (preoccupation with failure, reluctance to simplify, sensitivity to operations, commitment to resilience, deference to expertise)
• Design defense-in-depth, fail-safe, and fail-closed controls for software, infra, and OT
• Define verification gates, independent checks, and release hold criteria
• Architect redundancy, isolation, and graceful degradation with explicit failure modes
• Facilitate pre-mortems, FMEA, and risk registers before high-stakes change
• Author stop-the-line policy and escalation when quality or safety signals are ambiguous
• Select and track prevention metrics (defect escape, near-miss, repeat incidents, gate bypass)
• Coach leadership behaviors that counter normalization of deviance and blame theater
• Brief engineering and ops on zero-defect aspiration vs error budgets for the right domain

When NOT to Use

• Own SLI/SLO definitions, error-budget policy, and burn-rate alerting → site-reliability-engineer
• Run live incident war room, SEV classification, and status communications → incident-management-engineer
• Design backup/immutability, RTO/RPO recovery, and ransomware restore architecture → cyber-resilience-engineer
• Enforce CI compile/lint/test gates without broader prevention program → build-validator
• Facilitate sprint ceremonies, backlog grooming, or team agile transformation → agile coaching skills
• Issue HR warnings, performance plans, or legal disciplinary guidance → escalate to HR/legal
• Own classified ATO/accreditation packages without operational excellence deliverables → classified-cyber-security-senior-manager (pair for cleared context)
• Produce ADRs and integration patterns without failure-prevention lens → senior-system-architecture (pair for architecture)

Related skills

Core Workflows

1. Scope, limits, and charter

Clarify what “zero tolerance” means in context—aspiration, gates, and metrics—without perfectionism traps.

See `references/zero_tolerance_scope_and_limits.md`.

2. HRO principles and operating mindset

Embed high-reliability organization behaviors in teams that face rare, catastrophic failure.

See `references/high_reliability_organization_principles.md`.

3. Prevention, verification, and gates

Layer independent checks, hold points, and evidence before irreversible change.

See `references/prevention_verification_and_gates.md`.

4. Redundancy, degradation, and fail-safe design

Specify failure modes, safe defaults, and degraded operation—not only happy path.

See `references/redundancy_degradation_and_fail_safe.md`.

5. Pre-mortem, FMEA, and risk registers

Surface latent failures before launch; maintain living risk and mitigation traceability.

See `references/pre_mortem_fmea_and_risk_registers.md`.

6. Leadership, culture, and metrics

Measure prevention; reinforce stop-the-line; reduce normalization of deviance.

See `references/leadership_culture_and_metrics.md`.

Outputs

• Failure-prevention charter — scope, principles, RACI, interfaces with SRE/IR/QA
• Gate catalog — hold points, owners, evidence required, bypass rules and audit trail
• Design review pack — fail-safe/fail-closed decisions, degradation modes, verification plan
• FMEA / pre-mortem record — failure modes, causes, controls, residual risk, owners
• Stop-the-line policy — triggers, authority, duration, comms, and restart criteria
• Metrics dashboard brief — defect escape, near-miss, repeat incidents, gate effectiveness
• Leadership playbook — behaviors, rituals, and anti-patterns (learning vs blame)

Principles

• Prevent over punish — optimize systems and norms; avoid blame theater and hidden workarounds
• Fail closed by default — ambiguous safety or auth states deny; document explicit fail-open exceptions
• Independent verification — separation between build, check, and approve for high-criticality change
• Deference to expertise — elevate domain experts at the boundary; leaders ask, not override silently
• Measure escapes and near-misses — lagging severity alone rewards luck; track what almost failed
• Stop-the-line is a gift — halting bad change is success; normalize escalation without career penalty
• Pair with peers — reliability math (SRE), incidents (IR program), recovery (resilience), builds (CI)

High-reliability organization principles

Table of contents

1. HRO overview 2. Five principles in practice 3. Behaviors by role 4. Normalization of deviance 5. Classified and safety-critical overlays 6. Assessment checklist

HRO overview

High-reliability organizations operate in domains where failures are rare but catastrophic—aviation, nuclear, healthcare, military systems, large-scale cloud control planes.

HRO success depends on collective mindfulness, not heroic individuals. This reference translates Weick/Sutcliffe-style HRO ideas into engineering and operations rituals.

Five principles in practice

Behaviors by role

Normalization of deviance

Normalization of deviance occurs when repeated workaround becomes “how we do it” until an accident exposes the gap.

Run a norms audit quarterly: interview operators and reviewers; compare written gates to observed practice.

Classified and safety-critical overlays

Do not substitute HRO language for legal or accreditation obligations—coordinate with compliance owners.

Assessment checklist

• [ ] Near-miss mechanism exists and is used without retaliation
• [ ] Design reviews record dissent and unresolved risks
• [ ] Stop-the-line authority is named and exercised in last 12 months
• [ ] Tier-0 changes have independent verification (not author-only)
• [ ] Repeat incidents trigger system fixes, not repeat training alone
• [ ] Executives receive escape/near-miss trends, not only outage counts

Score maturity 1–4: ad hoc → defined gates → measured escapes → continuous norm reinforcement.

Leadership, culture, and metrics

Table of contents

1. Leadership behaviors 2. Stop-the-line authority 3. Metrics that matter 4. Dashboards and reviews 5. Anti-metrics and gaming 6. Communication templates

Leadership behaviors

Stop-the-line authority

Define in policy:

Success metric: stop-the-line events that prevented customer or safety impact—celebrate, do not penalize.

Metrics that matter

Not sufficient alone: raw incident count (luck), deployment frequency without escapes, individual “error counts” for blame.

Dashboards and reviews

Pair with SRE reliability review (site-reliability-engineer)—prevention metrics explain why SLOs broke, not only that they broke.

Anti-metrics and gaming

Communication templates

Near-miss report (short):

What almost happened:
What prevented impact:
What could fail next time:
Proposed system fix / owner / date:

Stop-the-line notice:

Halted: [change/deploy]
Reason: [signal]
Owner: [name]
Next step: [review time / criteria to resume]

Executive brief (monthly):

Defect escapes (tier-0/1): [n] trend
Near-miss reports: [n] trend
Repeat incidents: [n]
Gate bypasses: [n] with top reasons
Stop-the-line events: [n] outcomes
Top systemic actions: [list]

Keep narratives learning-oriented—escalate individual conduct to HR outside this skill.

Pre-mortem, FMEA, and risk registers

Table of contents

1. When to run structured risk work 2. Pre-mortem workflow 3. FMEA basics 4. Risk register pattern 5. Linking to gates and metrics 6. Facilitation tips

When to run structured risk work

Skip heavyweight FMEA for low-tier experiments—still document residual risk and owner.

Pre-mortem workflow

Goal: Imagine future failure, then prevent it before launch.

1. Frame — objective, scope, success criteria, irreversible steps 2. Assume failure — “It is 6 months later; we failed badly. Why?” 3. Silent brainstorm — causes, blind spots, org failures (5–10 min) 4. Share and cluster — themes: technical, process, people, external 5. Mitigate — controls, gates, tests, owners, dates 6. Residual risk — accept, defer with date, or stop launch 7. Record — store in risk register; link to change ticket

Outputs: mitigation backlog, new gates, test additions, stop-the-line triggers.

FMEA basics

Failure Mode and Effects Analysis scores:

Prioritize rows with high S even if RPN moderate—catastrophic rare events matter in HRO contexts.

Risk register pattern

Maintain a living register (spreadsheet or ticket epic):

Accepted high residual risk requires named executive or delegated authority and expiry—no permanent “accepted” without review.

Linking to gates and metrics

After launch, track did any pre-mortem scenario occur? — calibrate facilitation quality.

Facilitation tips

• Invite skeptics and operators, not only builders
• Use premortem.google style anonymity if hierarchy suppresses speech
• Separate learning from go/no-go decision in the meeting record
• Time-box; assign owners before adjourn
• Re-run when scope changes materially—do not shelf the doc

Do not use pre-mortem output for performance punishment—undermines HRO reporting culture.

Prevention, verification, and gates

Table of contents

1. Defense in depth 2. Verification layers 3. Gate catalog pattern 4. Independent checks 5. Bypass and emergency change 6. Evidence and audit trail

Defense in depth

Layer controls so no single failure defeats safety or integrity:

Map layers to tier (crown jewels vs standard services)—do not apply maximum depth everywhere without cost trade-off documentation.

Verification layers

Rule: Tier-0/1 requires at least one independent layer beyond author self-check before production.

Gate catalog pattern

Document each hold point in a gate catalog:

Example gates: architecture review, threat model, ops readiness, DR fit check (cyber-resilience-engineer), classified change board (classified-cyber-security-senior-manager).

Independent checks

Pair automated CI gates (build-validator) with human judgment for context CI cannot see.

Bypass and emergency change

Track bypass rate and repeat bypass reasons—chronic bypass indicates broken gates, not heroic ops.

Evidence and audit trail

Store for each gate passage:

• Ticket/link to change record
• Test results, scan reports, review notes
• Approver identity and timestamp
• Residual risks accepted with owner and date

Evidence must be durable and searchable for inspections—not only chat threads.

Redundancy, degradation, and fail-safe design

Table of contents

1. Failure mode thinking 2. Fail-safe vs fail-closed vs fail-open 3. Redundancy patterns 4. Graceful degradation 5. OT and classified considerations 6. Design review prompts

Failure mode thinking

For each component, document:

Use FMEA (references/pre_mortem_fmea_and_risk_registers.md) for structured scoring on high-criticality systems.

Fail-safe vs fail-closed vs fail-open

Default: Prefer fail-closed for security and integrity; fail-safe for safety; document any fail-open with risk owner and monitoring.

Redundancy patterns

Redundancy without tested failover is wishful thinking—schedule exercises; pair cyber-resilience-engineer for recovery proof.

Graceful degradation

Define degraded modes explicitly:

Requirements:

• Deterministic defaults — feature flags and config default to safer mode
• Backpressure — shed load before unbounded queue growth
• Cascading failure containment — timeouts, bulkheads, circuit breakers
• Observability — metric per degradation mode; alert on unintended entry

OT and classified considerations

Design review prompts

• [ ] Unsafe failure states identified and eliminated where possible
• [ ] Fail-closed/default-deny documented for auth and data paths
• [ ] Degraded modes defined with monitoring and runbooks
• [ ] Redundancy tested in last 12 months (or justified exception)
• [ ] Single points of failure listed with accepted risk owner
• [ ] Rollback/automation path exists for deploy-induced failure

Capture decisions in ADR or architecture pack—pair senior-system-architecture for NFR traceability.

Zero-tolerance scope and limits

Table of contents

1. Purpose and boundaries 2. Zero-defect aspiration vs error budgets 3. Perfectionism and blame traps 4. In scope vs out of scope 5. Domains of application 6. Charter elements

Purpose and boundaries

Zero-tolerance for failure (operational excellence lens) means building systems and culture where preventable failures are designed out, caught early, and learned from—without claiming literal zero defects everywhere.

This skill sits between:

• Reliability operations (site-reliability-engineer) — SLOs, error budgets, capacity
• Incident program (incident-management-engineer) — SEV, on-call, postmortem process
• Recovery engineering (cyber-resilience-engineer) — backup, RTO/RPO, restore tests
• Build validation (build-validator) — automated CI gates

Own prevention philosophy, verification depth, HRO behaviors, and prevention metrics—not paging policy, restore architecture, or lint rules alone.

Zero-defect aspiration vs error budgets

Decision rule: If failure implies harm, compromise, or regulatory breach, bias to prevention, gates, and fail-closed design. If failure implies degraded UX or revenue within agreed SLO, pair with error budgets and mitigation—still track escapes.

Perfectionism and blame traps

Escalate HR or legal disciplinary questions out of this skill—stay on engineering and operational norms.

In scope vs out of scope

Domains of application

Charter elements

1. Scope — tiers/systems covered; explicit exclusions 2. Principles — fail-closed, independent verification, stop-the-line 3. Governance — who can bypass gates; audit requirements 4. Interfaces — SRE, IR, QA, security, BCM 5. Metrics — defect escape, near-miss, repeat incidents, gate bypass rate 6. Review cadence — quarterly norm check; after material escape or near-catastrophe

Store charter and gate definitions in a durable, version-controlled repository accessible to builders and reviewers.