Cyber Resilience Engineer

Cyber Resilience Engineer

When to Use

• Define RTO/RPO and recovery tiers for production, security, and identity services
• Design backup, restore, and immutability architecture (object lock, air-gap, WORM, vault isolation)
• Map critical service dependencies and failure domains for continuity during attacks
• Author attack-scenario playbooks (ransomware, wiper, IdP loss, logging blind spot, cloud control-plane)
• Plan and run resilience tests—restore drills, game days, chaos/failure injection with pass/fail evidence
• Align engineering deliverables with NIST CSF Recover and BCM/IR interfaces
• Produce resilience metrics and engineering briefs for leadership and audit consumers

When NOT to Use

• Own enterprise BCM program, BIA facilitation, and regulatory BCM policy → bcm-disaster-recovery-specialist
• Lead active incident war room, containment, and forensic preservation → incident-responder
• Define SEV matrices, paging policy, and status-page program → incident-management-engineer
• Operate SLIs, SLOs, error budgets, and capacity toil without recovery design → site-reliability-engineer
• Execute ticket-level snapshots and restores without resilience architecture → cloud-system-administrator
• Control-by-control audit evidence and framework mapping only → compliance-specialist
• Board-level security strategy and risk appetite without engineering recovery → chief-information-security-officer
• Broad security program ownership without resilience engineering → cybersecurity
• Greenfield cloud landing zone without recovery lens → cloud-engineer (pair for placement; you own RTO/RPO fit)

Related skills

Core Workflows

1. Scope and engineering charter

Clarify resilience boundaries, ownership, and interfaces with BCM, IR, SRE, and platform teams.

See `references/cyber_resilience_scope.md`.

2. Recovery objectives and tiers

Set RTO/RPO, tiering, and recovery strategies with explicit trade-offs and test hooks.

See `references/recovery_objectives_and_tiers.md`.

3. Backup, restore, and immutability

Design backup topology, isolation, encryption, and restore validation for cyber events.

See `references/backup_restore_and_immutability.md`.

4. Resilience testing and chaos

Plan game days, restore drills, and controlled failure injection with evidence and remediation.

See `references/resilience_testing_and_chaos.md`.

5. Attack scenarios and playbooks

Engineer playbooks for ransomware, destructive malware, identity and logging loss, and cloud control-plane failure.

See `references/attack_scenarios_and_playbooks.md`.

6. Metrics, reporting, and governance

Track RTA vs RTO, restore success, test coverage, and NIST CSF Recover alignment for stakeholders.

See `references/metrics_reporting_and_governance.md`.

Outputs

• Resilience architecture — tiers, dependencies, backup/immutability design, failure domains
• RTO/RPO register — per service with strategy, owner, last test, and known gaps
• Playbook pack — attack-scenario sequences with decision gates and IR handoffs
• Test report — scope, RTA/RPO achieved, integrity checks, findings, remediation backlog
• Chaos/game-day summary — hypothesis, blast radius, controls validated, follow-ups
• Resilience dashboard brief — KPIs, trend, top risks (engineering lens; not legal advice)

Principles

• Engineer for compromise — assume attacker presence; prefer rebuild and immutable recovery paths
• Test restores, not jobs — backup success ≠ recoverable; measure RTA and data integrity
• Recover security first — identity, logging, and detection before convenience workloads
• Separate roles — resilience engineering complements BCM policy and IR command
• Evidence by design — every tier and playbook links to a test or exercise with dated results

Attack scenarios and playbooks

Table of contents

1. Playbook structure 2. Ransomware and extortion 3. Destructive malware / wiper 4. Identity platform compromise 5. Logging and detection loss 6. Cloud control-plane and SaaS loss 7. Supply chain and golden image 8. Decision gates

Playbook structure

Each engineering playbook includes:

Keep playbooks executable—commands, console paths, automation job names, not policy prose.

Ransomware and extortion

Triggers: encryption notices, mass file renames, backup encryption, BEC + deployment anomaly.

Engineering sequence:

1. Activate bridge; IR leads containment (incident-responder) 2. Isolate segments; disable risky automation and remote tools 3. Assess backup plane—last clean immutable copy; assume prod untrusted 4. Decision gate: decrypt vs rebuild (default rebuild unless IR/legal approves pay/decrypt) 5. Provision greenfield environment from gold IaC/images 6. Restore data from immutable copy after malware-free verification 7. Rotate all secrets, API keys, certs; force password reset if IdP touched 8. Re-enable logging and EDR before broad user access 9. MVC cutover; phased user return; monitor for re-entry

Document RTO impact if rebuild path chosen—set expectations early.

Destructive malware / wiper

Triggers: boot failures, MBR/GPT damage, mass deletes, firmware alerts.

Differences from ransomware:

• Often no negotiation; backups targeted first
• Faster priority on offline/immutable copies and gold images
• Hardware reimage may dominate RTA

Sequence emphasizes spare capacity (hardware pool, cloud quota), IaC rebuild, and tape/air-gap restore paths.

Identity platform compromise

Triggers: impossible travel admin actions, MFA bypass reports, rogue federation apps, golden SAML cert.

Resilience actions:

1. Enable break-glass local accounts (pre-staged, tested quarterly) 2. Revoke sessions and refresh tokens globally 3. Disable compromised federation apps and OAuth grants 4. Restore IdP config from immutable config backup or vendor export 5. Re-issue MFA enrollment; prioritize tier-0 admins 6. Validate conditional access and PAM before restoring SaaS SSO

Coordinate with information-security-engineer for hardening; IR for attribution.

Logging and detection loss

Triggers: SIEM ingest drop, log forwarder silence, EDR cloud disconnect.

Impact: Extended blind spot—recovery priority after IdP, parallel with network isolation.

Actions:

1. Queue/buffer on endpoints where possible 2. Failover to secondary region or warm SIEM 3. Restore parsers from versioned config backup 4. Backfill from archives after ingest restored (watch RPO) 5. Document detection gap window for IR and compliance consumers

Cloud control-plane and SaaS loss

Triggers: regional API outage, identity provider API failure, mistaken org-wide deny policy.

Actions:

1. Execute alternate admin path (break-glass account, support ticket, secondary region) 2. Run DNS and traffic failover per cloud-engineer architecture 3. Activate manual runbooks for critical security SaaS (export APIs if available) 4. Track vendor status; record RTA against RTO 5. Post-incident: add synthetic canaries for control-plane dependencies

Maintain offline copies of runbooks and contact trees—assume IdP unavailable.

Supply chain and golden image

Triggers: compromised CI/CD, malicious package, trojaned AMI/container base.

Actions:

1. Halt deployments; freeze artifact registry 2. Identify last known-good image digest and IaC commit 3. Rebuild pipelines from clean runners 4. Redeploy from signed gold images only 5. Expand scanning and provenance (SBOM) in backlog

Link to IR for scope; resilience owns rebuild path and image promotion gates.

Decision gates

Record gate decisions with timestamp and approver in incident ticket.

Backup, restore, and immutability

Table of contents

1. Design goals 2. Backup topology 3. Immutability patterns 4. Encryption and keys 5. Restore validation 6. SaaS and cloud-native 7. Anti-patterns

Design goals

• Recoverability under ransomware, insider, and operator error
• Integrity — detect tampering before restore
• Isolation — backup plane not reachable from compromised production credentials
• Provable RPO — measurable lag, not assumed schedule success
• Evidence — audit-ready test and restore logs

Backup topology

3-2-1-1-0 variant for cyber: 3 copies, 2 media types, 1 offsite/immutable, 1 offline or logically air-gapped, 0 unverified restores.

Document backup admin as separate identity from production admin; no standing shared roles.

Immutability patterns

Retention: align legal hold and regulatory minimums with BCM (bcm-disaster-recovery-specialist); engineering sets technical retention and lock duration.

Encryption and keys

• Encrypt backups with CMK or HSM-backed keys; separate key admin from backup admin
• Document key recovery procedure—KMS loss is tier-0 event
• Rotate keys on schedule; test restore after rotation
• For ransomware: assume prod keys compromised—vault keys and offline copies use different trust path

Restore validation

Every tier-1+ system needs a restore test profile:

Types of tests:

• Tabletop — walkthrough only
• Technical restore — isolated environment, no prod cutover
• Failover — production switch with rollback plan
• Full game day — multi-team, comms, metrics

Failed tests → remediation ticket with priority from tier.

SaaS and cloud-native

Maintain a SaaS recovery register:

For cloud-engineer patterns: multi-region, cross-account backups, private endpoints for backup APIs, deny delete via org policy.

Control-plane loss: pre-stage runbooks for DNS, identity, and console access via alternate account and out-of-band credentials.

Anti-patterns

• Single admin role over prod and backups
• Immutable backup with same SSO session as production
• Restore test only at file level, never app consistency
• Relying on vendor "backup included" without restore proof
• Keeping backups on same domain as ransomware entry
• Skipping restore after major version upgrade of backup software

Pair operational execution with cloud-system-administrator; you own design and test criteria.

Cyber resilience scope

Table of contents

1. Purpose and boundaries 2. In scope vs out of scope 3. Stakeholders and interfaces 4. NIST CSF Recover alignment 5. Deliverables and lifecycle 6. Maturity model

Purpose and boundaries

Cyber resilience engineering designs how the organization withstands, adapts to, and recovers from cyber disruptions while preserving critical security and business functions. It sits between:

• BCM/DR program (bcm-disaster-recovery-specialist) — policy, BIA, regulatory BCM, crisis comms cadence
• Incident response (incident-responder) — live containment, investigation, evidence
• Reliability engineering (site-reliability-engineer) — steady-state SLOs, capacity, toil reduction

You own how systems are built and tested to recover, not who declares incidents or signs BCM policy.

In scope vs out of scope

Stakeholders and interfaces

Document RACI in the resilience charter: who approves tier changes, who authorizes chaos in production, who signs test exceptions.

NIST CSF Recover alignment

Map engineering work to Recover categories (CSF 2.0):

Pair with Identify (asset inventory) and Protect (immutable backups) implementations from information-security-engineer where controls are missing.

Deliverables and lifecycle

1. Charter — scope, owners, interfaces, escalation 2. Architecture — tiers, backup design, regions, isolation 3. Registers — RTO/RPO, dependencies, SaaS recovery contacts 4. Playbooks — per attack scenario with decision trees 5. Test calendar — restore, tabletop, chaos; linked evidence store 6. Metrics — RTA, test pass rate, coverage by tier 7. Annual refresh — after major architecture, M&A, or material incident

Store evidence (reports, screenshots, RTA logs) in a durable, access-controlled repository—not only in chat.

Maturity model

Target level 3+ for tier-0/1 security services (IdP, logging, KMS) before claiming "cyber resilient."

Metrics, reporting, and governance

Table of contents

1. KPI framework 2. Operational metrics 3. Test and exercise metrics 4. Incident and RTA tracking 5. Reporting audiences 6. Governance cadence 7. NIST CSF Recover evidence map

KPI framework

Operational metrics

Leading indicators (automate where possible):

• Backup job success vs verified restore readiness
• Replication lag (seconds/minutes) vs RPO
• Object-lock deny events (attempted deletes)
• IdP/SIEM/EDR synthetic availability checks
• Gold image age and vulnerability scan status
• Secrets rotation failures blocking recovery

Dashboards should separate prod vs security plane vs backup plane health.

Test and exercise metrics

Per quarter report:

Link each metric to evidence artifact (report ID, ticket).

Incident and RTA tracking

During real events capture:

• Detection to mobilization time
• Mobilization to recovery start
• RTA per service vs documented RTO
• Effective RPO (timestamp of restored data)
• MVC duration — time at degraded capability
• Decision gates invoked (G1–G5 from playbooks)

Post-incident: compare to last drill; if RTA >2× drill RTA, schedule targeted retest.

Reporting audiences

Keep reports factual—distinguish tested vs assumed recovery.

Governance cadence

Change control: RTO/RPO tightening needs service owner approval; loosening needs risk acceptance.

NIST CSF Recover evidence map

Maintain an evidence index (control → artifact → date) for audit requests; control interpretation stays with compliance peers.

Recovery objectives and tiers

Table of contents

1. Definitions 2. Tiering model 3. Setting RTO and RPO 4. Security-critical service targets 5. Recovery strategies 6. Activation and sequencing 7. Exceptions and risk acceptance

Definitions

RTO/RPO are engineering contracts—require service owner sign-off and BCM alignment (bcm-disaster-recovery-specialist).

Tiering model

Cyber rule: tier-0/1 security services (IdP, SIEM hot, KMS, secrets) default to tier-0/1 even if business app is tier-2—logging and identity gaps extend attacker dwell time.

Setting RTO and RPO

1. Ingest MTPD and process criticality from BCM BIA 2. Subtract mobilization (assessment, bridge stand-up, approvals)—often 30–120 min 3. Set RTO ≤ remaining budget for restore work (not full performance) 4. Set RPO from replication lag, backup frequency, and legal log minimums 5. Validate with restore test or game day; record RTA vs RTO 6. Automate monitoring of backup lag and replication drift as leading indicators

Document MVC per service: what "recovered" means (e.g., SIEM ingesting 80% sources, not full historical search).

Security-critical service targets

Indicative starting points—calibrate to your BIA:

Recovery strategies

After compromise, restore-in-place requires IR approval (incident-responder) and integrity proof—otherwise rebuild.

Activation and sequencing

Declare resilience/DR mode when triggers met (customize in playbooks):

• Tier-0 outage >50% of RTO without credible ETA
• Ransomware/wiper affecting production or backup plane
• IdP or MFA total loss
• Cloud control-plane or identity API widespread failure
• Loss of central logging beyond RPO for tier-0/1

Security-first recovery order (adjust per scenario):

1. Command bridge and roles (incident-responder) 2. Network isolation and safe paths 3. IdP / MFA / break-glass 4. KMS, secrets, PKI 5. Logging and EDR visibility 6. Critical apps and data 7. Full performance and backlog catch-up

Exceptions and risk acceptance

Every gap needs: risk ID, owner, compensating control, review date.

Examples requiring explicit acceptance:

• RPO >24 h for dev SIEM with no prod dependency
• No immutable backup for tier-1 DB (document attack scenario impact)
• Single-region IdP without warm standby

Review exceptions after every material incident and failed test.

Resilience testing and chaos

Table of contents

1. Test taxonomy 2. Planning a game day 3. Restore drills 4. Chaos and failure injection 5. Safety guardrails 6. Evidence and reporting 7. Continuous improvement

Test taxonomy

Distinguish reliability chaos (latency, pod kill) from cyber resilience tests (backup delete attempt, IdP unavailable, logging pipeline stopped).

Planning a game day

1. Objective — e.g., "restore IdP and SIEM within RTO after simulated ransomware" 2. Scope — systems, environments, explicit non-goals 3. Hypothesis — what should work; what you expect to fail 4. Success criteria — RTA, RPO, MVC met; comms milestones 5. Roles — resilience lead, IR liaison, SRE, app owners, scribe 6. Schedule — inject timeline, go/no-go checkpoints 7. Rollback — abort criteria and technical steps 8. Approvals — change advisory, leadership, customer impact

Coordinate calendar with bcm-disaster-recovery-specialist to avoid duplicate exercises.

Restore drills

Minimum drill script:

1. Select target from tier register (rotate coverage) 2. Provision isolated network and accounts 3. Restore from immutable copy where applicable 4. Run integrity and smoke tests 5. Measure RTA and effective RPO 6. Record gaps (automation, runbook, staffing, tooling) 7. File remediation; link to next quarter's drill

For ransomware simulation, prefer:

• Restore to greenfield VPC/subscription
• No reuse of prod AD/connectors until IR clearance
• Validate golden image hash and IaC revision

Chaos and failure injection

Security-relevant experiments (examples):

Use blast-radius budgets with SRE (site-reliability-engineer): max error rate, max duration, automatic abort.

Production chaos requires:

• Feature flags or kill switches
• On-call aware and bridge optional
• Customer impact assessment
• Post-experiment stability soak

Safety guardrails

• Never inject destructive malware or real encryption in prod
• No production restore into prod without change window and IR awareness
• Separate test credentials from prod; revoke after exercise
• Document stop conditions: customer SLA breach, data corruption signal, leadership call
• Legal/compliance review for scenarios touching regulated data

Evidence and reporting

Capture per exercise:

• Date, scope, tier, scenario ID
• Participants and roles
• RTA, RPO achieved vs targets
• Pass/fail per validation check
• Screenshots/logs (redacted) in evidence repository
• Findings with severity and owners
• Retest date for failed items

See `references/metrics_reporting_and_governance.md` for KPI definitions.

Continuous improvement

• Feed failures into architecture backlog (not only runbook edits)
• Update playbooks within 5 business days of material gap
• Re-run failed scenarios before claiming closure
• Share blameless summary with IR and BCM
• After real incidents, compare RTA to last drill—update tests if drift >25%