Secrets Vault Manager

Secrets Vault Manager

Tier: POWERFUL Category: Engineering Domain: Security / Infrastructure / DevOps

---

Overview

Production secret infrastructure management for teams running HashiCorp Vault, cloud-native secret stores, or hybrid architectures. This skill covers policy authoring, auth method configuration, automated rotation, dynamic secrets, audit logging, and incident response.

Distinct from env-secrets-manager which handles local .env file hygiene and leak detection. This skill operates at the infrastructure layer — Vault clusters, cloud KMS, certificate authorities, and CI/CD secret injection.

When to Use

• Standing up a new Vault cluster or migrating to a managed secret store
• Designing auth methods for services, CI runners, and human operators
• Implementing automated credential rotation (database, API keys, certificates)
• Auditing secret access patterns for compliance (SOC 2, ISO 27001, HIPAA)
• Responding to a secret leak that requires mass revocation
• Integrating secrets into Kubernetes workloads or CI/CD pipelines

---

HashiCorp Vault Patterns

Architecture Decisions

Auth Methods

AppRole — Machine-to-machine authentication for services and batch jobs.

# Enable AppRole
path "auth/approle/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Application-specific role
vault write auth/approle/role/payment-service \
  token_ttl=1h \
  token_max_ttl=4h \
  secret_id_num_uses=1 \
  secret_id_ttl=10m \
  token_policies="payment-service-read"

Kubernetes — Pod-native authentication via service account tokens.

vault write auth/kubernetes/role/api-server \
  bound_service_account_names=api-server \
  bound_service_account_namespaces=production \
  policies=api-server-secrets \
  ttl=1h

OIDC — Human operator access via SSO provider (Okta, Azure AD, Google Workspace).

vault write auth/oidc/role/engineering \
  bound_audiences="vault" \
  allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
  user_claim="email" \
  oidc_scopes="openid,profile,email" \
  policies="engineering-read" \
  ttl=8h

Secret Engines

Policy Design

Follow least-privilege with path-based granularity:

# payment-service-read policy
path "secret/data/production/payment/*" {
  capabilities = ["read"]
}

path "database/creds/payment-readonly" {
  capabilities = ["read"]
}

# Deny access to admin paths explicitly
path "sys/*" {
  capabilities = ["deny"]
}

Policy naming convention: {service}-{access-level} (e.g., payment-service-read, api-gateway-admin).

---

Cloud Secret Store Integration

Comparison Matrix

When to Use Which

• AWS Secrets Manager: RDS/Aurora credential rotation out of the box. Best when fully on AWS.
• Azure Key Vault: Certificate management strength. Required for Azure AD integrated workloads.
• GCP Secret Manager: Simplest API surface. Best for GKE-native workloads with Workload Identity.
• HashiCorp Vault: Multi-cloud, dynamic secrets, PKI, transit encryption. Best for complex or hybrid environments.

SDK Access Patterns

Principle: Always fetch secrets at startup or via sidecar — never bake into images or config files.

# AWS Secrets Manager pattern
import boto3, json

def get_secret(secret_name, region="us-east-1"):
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])

# GCP Secret Manager pattern
from google.cloud import secretmanager

def get_secret(project_id, secret_id, version="latest"):
    client = secretmanager.SecretManagerServiceClient()
    name = f"projects/{project_id}/secrets/{secret_id}/versions/{version}"
    response = client.access_secret_version(request={"name": name})
    return response.payload.data.decode("UTF-8")

# Azure Key Vault pattern
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

def get_secret(vault_url, secret_name):
    credential = DefaultAzureCredential()
    client = SecretClient(vault_url=vault_url, credential=credential)
    return client.get_secret(secret_name).value

---

Secret Rotation Workflows

Rotation Strategy by Secret Type

Database Credential Rotation (Dual-Account)

1. Two database accounts exist: app_user_a and app_user_b 2. Application currently uses app_user_a 3. Rotation rotates app_user_b password, updates secret store 4. Application switches to app_user_b on next credential fetch 5. After grace period, app_user_a password is rotated 6. Cycle repeats

API Key Rotation (Overlap Window)

1. Generate new API key with provider 2. Store new key in secret store as current, move old to previous 3. Deploy applications — they read current 4. After all instances restarted (or TTL expired), revoke previous 5. Monitoring confirms zero usage of old key before revocation

---

Dynamic Secrets

Dynamic secrets are generated on-demand with automatic expiration. Prefer dynamic secrets over static credentials wherever possible.

Database Dynamic Credentials (Vault)

# Configure database engine
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/app" \
  allowed_roles="app-readonly,app-readwrite" \
  username="vault_admin" \
  password="<admin-password>"

# Create role with TTL
vault write database/roles/app-readonly \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl=1h \
  max_ttl=24h

Cloud IAM Dynamic Credentials

Vault can generate short-lived AWS IAM credentials, Azure service principal passwords, or GCP service account keys — eliminating long-lived cloud credentials entirely.

SSH Certificate Authority

Replace SSH key distribution with a Vault-signed certificate model:

1. Vault acts as SSH CA 2. Users/machines request signed certificates with short TTL (30 min) 3. SSH servers trust the CA public key — no authorized_keys management 4. Certificates expire automatically — no revocation needed for normal operations

---

Audit Logging

What to Log

Anomaly Detection Signals

• Secret accessed from new IP/CIDR range
• Access volume spike (>3x baseline for a path)
• Off-hours access for human auth methods
• Service accessing secrets outside its policy scope (denied requests)
• Multiple failed auth attempts from single source
• Token created with unusually long TTL

Compliance Reporting

Generate periodic reports covering:

1. Access inventory — Which identities accessed which secrets, when 2. Rotation compliance — Secrets overdue for rotation 3. Policy drift — Policies modified since last review 4. Orphaned secrets — Secrets with no recent access (>90 days)

Use audit_log_analyzer.py to parse Vault or cloud audit logs for these signals.

---

Emergency Procedures

Secret Leak Response (Immediate)

Time target: Contain within 15 minutes of detection.

1. Identify scope — Which secret(s) leaked, where (repo, log, error message, third party) 2. Revoke immediately — Rotate the compromised credential at the source (provider API, Vault, cloud SM) 3. Invalidate tokens — Revoke all Vault tokens that accessed the leaked secret 4. Audit blast radius — Query audit logs for usage of the compromised secret in the exposure window 5. Notify stakeholders — Security team, affected service owners, compliance (if PII/regulated data) 6. Post-mortem — Document root cause, update controls to prevent recurrence

Vault Seal Operations

When to seal: Active security incident affecting Vault infrastructure, suspected key compromise.

Sealing stops all Vault operations. Use only as last resort.

Unseal procedure: 1. Gather quorum of unseal key holders (Shamir threshold) 2. Or confirm auto-unseal KMS key is accessible 3. Unseal via vault operator unseal or restart with auto-unseal 4. Verify audit devices reconnected 5. Check active leases and token validity

See references/emergency_procedures.md for complete playbooks.

---

CI/CD Integration

Vault Agent Sidecar (Kubernetes)

Vault Agent runs alongside application pods, handles authentication and secret rendering:

# Pod annotation for Vault Agent Injector
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "api-server"
  vault.hashicorp.com/agent-inject-secret-db: "database/creds/app-readonly"
  vault.hashicorp.com/agent-inject-template-db: |
    {{- with secret "database/creds/app-readonly" -}}
    postgresql://{{ .Data.username }}:{{ .Data.password }}@db:5432/app
    {{- end }}

External Secrets Operator (Kubernetes)

For teams preferring declarative GitOps over agent sidecars:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: api-credentials
  data:
    - secretKey: api-key
      remoteRef:
        key: secret/data/production/api
        property: key

GitHub Actions OIDC

Eliminate long-lived secrets in CI by using OIDC federation:

- name: Authenticate to Vault
  uses: hashicorp/vault-action@v2
  with:
    url: https://vault.example.com
    method: jwt
    role: github-ci
    jwtGithubAudience: https://vault.example.com
    secrets: |
      secret/data/ci/deploy api_key | DEPLOY_API_KEY ;
      secret/data/ci/deploy db_password | DB_PASSWORD

---

Anti-Patterns

---

Tools

---

Cross-References

• env-secrets-manager — Local .env file hygiene, leak detection, drift awareness
• senior-secops — Security operations, incident response, threat modeling
• ci-cd-pipeline-builder — Pipeline design where secrets are consumed
• docker-development — Container secret injection patterns
• helm-chart-builder — Kubernetes secret management in Helm charts

Cloud Secret Store Reference

Provider Comparison

Feature Matrix

Decision Guide

Choose AWS Secrets Manager when:

• Fully on AWS
• Need native RDS/Aurora/Redshift rotation
• Using ECS/EKS with native AWS IAM integration
• Cross-account secret sharing via resource policies

Choose Azure Key Vault when:

• Azure-primary workloads
• Certificate lifecycle management is critical (built-in CA integration)
• Need HSM-backed key protection (Premium SKU)
• Azure AD conditional access integration required

Choose GCP Secret Manager when:

• GCP-primary workloads
• Using GKE with Workload Identity
• Want simplest API surface (few concepts, fast to integrate)
• Cost-sensitive (generous free tier)

Choose HashiCorp Vault when:

• Multi-cloud or hybrid environments
• Dynamic secrets (database, cloud IAM, SSH) are primary use case
• Need transit encryption, PKI, or SSH CA
• Regulatory requirement for self-hosted secret management

AWS Secrets Manager

Access Patterns

import boto3
import json
from botocore.exceptions import ClientError

def get_secret(secret_name, region="us-east-1"):
    """Retrieve secret from AWS Secrets Manager."""
    client = boto3.client("secretsmanager", region_name=region)
    try:
        response = client.get_secret_value(SecretId=secret_name)
    except ClientError as e:
        code = e.response["Error"]["Code"]
        if code == "ResourceNotFoundException":
            raise ValueError(f"Secret {secret_name} not found")
        elif code == "DecryptionFailureException":
            raise RuntimeError("KMS decryption failed — check key permissions")
        raise
    if "SecretString" in response:
        return json.loads(response["SecretString"])
    return response["SecretBinary"]

Rotation with Lambda

# rotation_lambda.py — skeleton for custom rotation
def lambda_handler(event, context):
    secret_id = event["SecretId"]
    step = event["Step"]
    token = event["ClientRequestToken"]
    client = boto3.client("secretsmanager")

    if step == "createSecret":
        # Generate new credentials
        new_password = generate_password()
        client.put_secret_value(
            SecretId=secret_id,
            ClientRequestToken=token,
            SecretString=json.dumps({"password": new_password}),
            VersionStages=["AWSPENDING"],
        )
    elif step == "setSecret":
        # Apply new credentials to the target service
        pending = get_secret_version(client, secret_id, "AWSPENDING", token)
        apply_credentials(pending)
    elif step == "testSecret":
        # Verify new credentials work
        pending = get_secret_version(client, secret_id, "AWSPENDING", token)
        test_connection(pending)
    elif step == "finishSecret":
        # Mark AWSPENDING as AWSCURRENT
        client.update_secret_version_stage(
            SecretId=secret_id,
            VersionStage="AWSCURRENT",
            MoveToVersionId=token,
            RemoveFromVersionId=get_current_version(client, secret_id),
        )

IAM Policy for Secret Access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:production/api/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "us-east-1"
        }
      }
    }
  ]
}

Cross-Account Access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::987654321098:role/shared-secret-reader"},
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringEquals": {
          "secretsmanager:VersionStage": "AWSCURRENT"
        }
      }
    }
  ]
}

Azure Key Vault

Access Patterns

from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.keyvault.secrets import SecretClient

def get_secret(vault_url, secret_name, use_managed_identity=True):
    """Retrieve secret from Azure Key Vault."""
    if use_managed_identity:
        credential = ManagedIdentityCredential()
    else:
        credential = DefaultAzureCredential()
    client = SecretClient(vault_url=vault_url, credential=credential)
    return client.get_secret(secret_name).value

def list_secrets(vault_url):
    """List all secret names (not values)."""
    credential = DefaultAzureCredential()
    client = SecretClient(vault_url=vault_url, credential=credential)
    return [s.name for s in client.list_properties_of_secrets()]

RBAC vs Access Policies

RBAC (recommended):

• Uses Azure AD roles (Key Vault Secrets User, Key Vault Secrets Officer)
• Managed at subscription/resource group/vault level
• Audit via Azure AD activity logs

Access Policies (legacy):

• Per-vault configuration
• Object ID based
• No inheritance from resource group

# Assign RBAC role
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee <service-principal-id> \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault>

Certificate Management

Azure Key Vault has first-class certificate management with automatic renewal:

# Create certificate with auto-renewal
az keyvault certificate create \
  --vault-name my-vault \
  --name api-tls \
  --policy @cert-policy.json

# cert-policy.json
{
  "issuerParameters": {"name": "Self"},
  "keyProperties": {"keyType": "RSA", "keySize": 2048},
  "lifetimeActions": [
    {"action": {"actionType": "AutoRenew"}, "trigger": {"daysBeforeExpiry": 30}}
  ],
  "x509CertificateProperties": {
    "subject": "CN=api.example.com",
    "validityInMonths": 12
  }
}

GCP Secret Manager

Access Patterns

from google.cloud import secretmanager

def get_secret(project_id, secret_id, version="latest"):
    """Retrieve secret from GCP Secret Manager."""
    client = secretmanager.SecretManagerServiceClient()
    name = f"projects/{project_id}/secrets/{secret_id}/versions/{version}"
    response = client.access_secret_version(request={"name": name})
    return response.payload.data.decode("UTF-8")

def create_secret(project_id, secret_id, secret_value):
    """Create a new secret with initial version."""
    client = secretmanager.SecretManagerServiceClient()
    parent = f"projects/{project_id}"

    # Create the secret resource
    secret = client.create_secret(
        request={
            "parent": parent,
            "secret_id": secret_id,
            "secret": {"replication": {"automatic": {}}},
        }
    )

    # Add a version with the secret value
    client.add_secret_version(
        request={
            "parent": secret.name,
            "payload": {"data": secret_value.encode("UTF-8")},
        }
    )
    return secret.name

Workload Identity for GKE

Eliminate service account key files by binding Kubernetes service accounts to GCP IAM:

# Create IAM binding
gcloud iam service-accounts add-iam-policy-binding \
  secret-accessor@my-project.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:my-project.svc.id.goog[namespace/ksa-name]"

# Annotate Kubernetes service account
kubectl annotate serviceaccount ksa-name \
  --namespace namespace \
  iam.gke.io/gcp-service-account=secret-accessor@my-project.iam.gserviceaccount.com

IAM Policy

# Grant secret accessor role to a service account
gcloud secrets add-iam-policy-binding my-secret \
  --member="serviceAccount:my-app@my-project.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Cross-Cloud Patterns

Abstraction Layer

When operating multi-cloud, create a thin abstraction that normalizes secret access:

# secret_client.py — cross-cloud abstraction
class SecretClient:
    def __init__(self, provider, **kwargs):
        if provider == "aws":
            self._client = AWSSecretClient(**kwargs)
        elif provider == "azure":
            self._client = AzureSecretClient(**kwargs)
        elif provider == "gcp":
            self._client = GCPSecretClient(**kwargs)
        elif provider == "vault":
            self._client = VaultSecretClient(**kwargs)
        else:
            raise ValueError(f"Unknown provider: {provider}")

    def get(self, key):
        return self._client.get(key)

    def set(self, key, value):
        return self._client.set(key, value)

Migration Strategy

When migrating between providers:

1. Dual-write phase — Write to both old and new store simultaneously 2. Dual-read phase — Read from new store, fallback to old 3. Cut-over — Read exclusively from new store 4. Cleanup — Remove secrets from old store after grace period

Secret Synchronization

For hybrid setups (e.g., Vault as primary, cloud SM for specific workloads):

• Use Vault's cloud secret engines to generate cloud-native credentials dynamically
• Or use External Secrets Operator to sync from Vault into cloud-native stores
• Never manually copy secrets between stores — always automate

Caching and Performance

Client-Side Caching

All three cloud providers support caching SDKs:

• AWS: aws-secretsmanager-caching-python — caches with configurable TTL
• Azure: Built-in HTTP caching in SDK, or use Azure App Configuration
• GCP: No official caching library — implement in-process cache with TTL

Caching Rules

1. Cache TTL should be shorter than rotation period (e.g., cache 5 min if rotating every 30 days) 2. Implement cache invalidation on secret version change events 3. Never cache secrets to disk — in-memory only 4. Log cache hits/misses for debugging rotation issues

Compliance Mapping

Emergency Procedures Reference

Secret Leak Response Playbook

Severity Classification

P0/P1 Response Procedure

Phase 1: Contain (0-15 minutes)

1. Identify the leaked secret

• What credential was exposed? (type, scope, permissions)
• Where was it exposed? (repo, log, error page, third-party service)
• When was it first exposed? (commit timestamp, log timestamp)
• Is the exposure still active? (repo public? log accessible?)

2. Revoke immediately

• Database password: ALTER ROLE app_user WITH PASSWORD 'new_password';
• API key: Regenerate via provider console/API
• Vault token: vault token revoke <token>
• AWS access key: aws iam delete-access-key --access-key-id <key>
• Cloud service account: Delete and recreate key
• TLS certificate: Revoke via CA, generate new certificate

3. Remove exposure

• Public repo: Remove file, force-push to remove from history, request GitHub cache purge
• Build logs: Delete log artifacts, rotate CI/CD secrets
• Error page: Deploy fix to suppress secret in error output
• Third-party: Contact vendor for log purge if applicable

4. Deploy new credentials

• Update secret store with rotated credential
• Restart affected services to pick up new credential
• Verify services are healthy with new credential

Phase 2: Assess (15-60 minutes)

5. Audit blast radius

• Query Vault/cloud SM audit logs for the compromised credential
• Check for unauthorized usage during the exposure window
• Review network logs for suspicious connections from unknown IPs
• Check if the compromised credential grants access to other secrets (privilege escalation)

6. Notify stakeholders

• Security team (always)
• Service owners for affected systems
• Compliance team if regulated data was potentially accessed
• Legal if customer data may have been compromised
• Executive leadership for P0 incidents

Phase 3: Recover (1-24 hours)

7. Rotate adjacent credentials

• If the leaked credential could access other secrets, rotate those too
• If a Vault token leaked, check what policies it had — rotate everything accessible

8. Harden against recurrence

• Add pre-commit hook to detect secrets (e.g., gitleaks, detect-secrets)
• Review CI/CD pipeline for secret masking
• Audit who has access to the source of the leak

Phase 4: Post-Mortem (24-72 hours)

9. Document incident

• Timeline of events
• Root cause analysis
• Impact assessment
• Remediation actions taken
• Preventive measures added

Response Communication Template

SECURITY INCIDENT — SECRET EXPOSURE
Severity: P0/P1
Time detected: YYYY-MM-DD HH:MM UTC
Secret type: [database password / API key / token / certificate]
Exposure vector: [public repo / build log / error output / other]
Status: [CONTAINED / INVESTIGATING / RESOLVED]

Immediate actions taken:
- [ ] Credential revoked at source
- [ ] Exposure removed
- [ ] New credential deployed
- [ ] Services verified healthy
- [ ] Audit log review in progress

Blast radius assessment: [PENDING / COMPLETE — no unauthorized access / COMPLETE — unauthorized access detected]

Next update: [time]
Incident commander: [name]

Vault Seal/Unseal Procedures

Understanding Seal Status

Vault uses a seal mechanism to protect the encryption key hierarchy. When sealed, Vault cannot decrypt any data or serve any requests.

Sealed State:
  Vault process running → YES
  API responding → YES (503 Sealed)
  Serving secrets → NO
  All active leases → FROZEN (not revoked)
  Audit logging → NO

Unsealed State:
  Vault process running → YES
  API responding → YES (200 OK)
  Serving secrets → YES
  Active leases → RESUMING
  Audit logging → YES

When to Seal Vault (Emergency Only)

Seal Vault when:

• Active intrusion on Vault infrastructure is confirmed
• Vault server compromise is suspected (unauthorized root access)
• Encryption key material may have been extracted
• Regulatory/legal hold requires immediate data access prevention

Do NOT seal for:

• Routine maintenance (use graceful shutdown instead)
• Single-node issues in HA cluster (let standby take over)
• Suspected secret leak (revoke the secret, don't seal Vault)

Seal Procedure

# Seal a single node
vault operator seal

# Seal all nodes (HA cluster)
# Seal each node individually — leader last
vault operator seal -address=https://vault-standby-1:8200
vault operator seal -address=https://vault-standby-2:8200
vault operator seal -address=https://vault-leader:8200

Impact of sealing:

• All active client connections dropped immediately
• All token and lease timers paused
• Applications lose secret access — prepare for cascading failures
• Monitoring will fire alerts for sealed state

Unseal Procedure (Shamir Keys)

Requires a quorum of key holders (e.g., 3 of 5).

# Each key holder provides their unseal key
vault operator unseal <key-1>
vault operator unseal <key-2>
vault operator unseal <key-3>
# Vault unseals after reaching threshold

Operational checklist after unseal: 1. Verify health: vault status shows Sealed: false 2. Check audit devices: vault audit list — confirm all enabled 3. Check auth methods: vault auth list 4. Verify HA status: vault operator raft list-peers 5. Check lease count: monitor vault.expire.num_leases 6. Verify applications reconnecting (check application logs)

Unseal Procedure (Auto-Unseal)

If using cloud KMS auto-unseal, Vault unseals automatically on restart:

# Restart Vault service
systemctl restart vault

# Verify unseal (should happen within seconds)
vault status

If auto-unseal fails:

• Check cloud KMS key permissions (IAM role may have been modified)
• Check network connectivity to cloud KMS endpoint
• Check KMS key status (not disabled, not scheduled for deletion)
• Check Vault logs: journalctl -u vault -f

Mass Credential Rotation Procedure

When a broad compromise requires rotating many credentials simultaneously.

Pre-Rotation Checklist

• [ ] Identify all credentials in scope
• [ ] Map credential dependencies (which services use which credentials)
• [ ] Determine rotation order (databases before applications)
• [ ] Prepare rollback plan for each credential
• [ ] Notify all service owners
• [ ] Schedule maintenance window if zero-downtime not possible
• [ ] Stage new credentials in secret store (but don't activate yet)

Rotation Order

1. Infrastructure credentials — Database root passwords, cloud IAM admin keys 2. Service credentials — Application database users, API keys 3. Integration credentials — Third-party API keys, webhook secrets 4. Human credentials — Force password reset, revoke SSO sessions

Rollback Plan

For each credential, document:

• Previous value (store in sealed emergency envelope or HSM)
• How to revert (specific command or API call)
• Verification step (how to confirm old credential works)
• Maximum time to rollback (SLA)

Vault Recovery Procedures

Lost Unseal Keys

If unseal keys are lost and auto-unseal is not configured:

1. If Vault is currently unsealed: Enable auto-unseal immediately, then reseal/unseal with KMS 2. If Vault is sealed: Data is irrecoverable without keys. Restore from Raft snapshot backup 3. Prevention: Store unseal keys in separate, secure locations (HSMs, safety deposit boxes). Use auto-unseal for production.

Raft Cluster Recovery

Single node failure (cluster still has quorum):

# Remove failed peer
vault operator raft remove-peer <failed-node-id>

# Add replacement node
# (new node joins via retry_join in config)

Loss of quorum (majority of nodes failed):

# On a surviving node with recent data
vault operator raft join -leader-ca-cert=@ca.crt https://surviving-node:8200

# If no node survives, restore from snapshot
vault operator raft snapshot restore /backups/latest.snap

Root Token Recovery

If root token is lost (it should be revoked after initial setup):

# Generate new root token (requires unseal key quorum)
vault operator generate-root -init
# Each key holder provides their key
vault operator generate-root -nonce=<nonce> <unseal-key>
# After quorum, decode the encoded token
vault operator generate-root -decode=<encoded-token> -otp=<otp>

Best practice: Generate a root token only when needed, complete the task, then revoke it:

vault token revoke <root-token>

Incident Severity Escalation Matrix

HashiCorp Vault Architecture & Patterns Reference

Architecture Overview

Vault operates as a centralized secret management service with a client-server model. All secrets are encrypted at rest and in transit. The seal/unseal mechanism protects the master encryption key.

Core Components

┌─────────────────────────────────────────────────┐
│                   Vault Cluster                  │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐   │
│  │  Leader    │  │ Standby   │  │ Standby   │   │
│  │  (active)  │  │ (forward) │  │ (forward) │   │
│  └─────┬─────┘  └─────┬─────┘  └─────┬─────┘   │
│        │               │               │         │
│  ┌─────┴───────────────┴───────────────┴─────┐   │
│  │            Raft Storage Backend            │   │
│  └───────────────────────────────────────────┘   │
│                                                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────────┐   │
│  │ Auth     │  │ Secret   │  │ Audit        │   │
│  │ Methods  │  │ Engines  │  │ Devices      │   │
│  └──────────┘  └──────────┘  └──────────────┘   │
└─────────────────────────────────────────────────┘

Storage Backend Selection

High Availability Setup

Raft Cluster Configuration

Minimum 3 nodes for production (tolerates 1 failure). 5 nodes for critical workloads (tolerates 2 failures).

# vault-config.hcl (per node)
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr = "https://vault-2.internal:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-3.internal:8200"
  }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file  = "/opt/vault/tls/vault.key"
}

api_addr     = "https://vault-1.internal:8200"
cluster_addr = "https://vault-1.internal:8201"

Auto-Unseal with AWS KMS

Eliminates manual unseal key management. Vault encrypts its master key with the KMS key.

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal"
}

Requirements:

• IAM role with kms:Encrypt, kms:Decrypt, kms:DescribeKey permissions
• KMS key must be in the same region or accessible cross-region
• KMS key should have restricted access — only Vault nodes

Auto-Unseal with Azure Key Vault

seal "azurekeyvault" {
  tenant_id  = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  vault_name = "vault-unseal-kv"
  key_name   = "vault-unseal-key"
}

Auto-Unseal with GCP KMS

seal "gcpckms" {
  project    = "my-project"
  region     = "global"
  key_ring   = "vault-keyring"
  crypto_key = "vault-unseal-key"
}

Namespaces (Enterprise)

Namespaces provide tenant isolation within a single Vault cluster. Each namespace has independent policies, auth methods, and secret engines.

root/
├── dev/           # Development environment
│   ├── auth/
│   └── secret/
├── staging/       # Staging environment
│   ├── auth/
│   └── secret/
└── production/    # Production environment
    ├── auth/
    └── secret/

OSS alternative: Use path-based isolation with strict policies. Prefix all paths with environment name (e.g., secret/data/production/...).

Policy Patterns

Templated Policies

Use identity-based templates for scalable policy management:

# Allow entities to manage their own secrets
path "secret/data/{{identity.entity.name}}/*" {
  capabilities = ["create", "read", "update", "delete"]
}

# Read shared config for the entity's group
path "secret/data/shared/{{identity.groups.names}}/*" {
  capabilities = ["read"]
}

Sentinel Policies (Enterprise)

Enforce governance rules beyond path-based access:

# Require MFA for production secret writes
import "mfa"

main = rule {
  request.path matches "secret/data/production/.*" and
  request.operation in ["create", "update", "delete"] and
  mfa.methods.totp.valid
}

Policy Hierarchy

1. Global deny — Explicit deny on sys/*, auth/token/create-orphan 2. Environment base — Read access to environment-specific paths 3. Service-specific — Scoped to exact paths the service needs 4. Admin override — Requires MFA, time-limited, audit-heavy

Secret Engine Configuration

KV v2 (Versioned Key-Value)

# Enable with custom config
vault secrets enable -path=secret -version=2 kv

# Configure version retention
vault write secret/config max_versions=10 cas_required=true delete_version_after=90d

Check-and-Set (CAS): Prevents accidental overwrites. Client must supply the current version number to update.

Database Engine

# Enable and configure PostgreSQL
vault secrets enable database

vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/app?sslmode=require" \
  allowed_roles="app-readonly,app-readwrite" \
  username="vault_admin" \
  password="INITIAL_PASSWORD"

# Rotate the root password (Vault manages it from now on)
vault write -f database/rotate-root/postgres

# Create a read-only role
vault write database/roles/app-readonly \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl=1h \
  max_ttl=24h

PKI Engine (Certificate Authority)

# Enable PKI engine
vault secrets enable -path=pki pki
vault secrets tune -max-lease-ttl=87600h pki

# Generate root CA
vault write -field=certificate pki/root/generate/internal \
  common_name="Example Root CA" \
  ttl=87600h > root_ca.crt

# Enable intermediate CA
vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int

# Generate intermediate CSR
vault write -field=csr pki_int/intermediate/generate/internal \
  common_name="Example Intermediate CA" > intermediate.csr

# Sign with root CA
vault write -field=certificate pki/root/sign-intermediate \
  csr=@intermediate.csr format=pem_bundle ttl=43800h > intermediate.crt

# Set signed certificate
vault write pki_int/intermediate/set-signed certificate=@intermediate.crt

# Create role for leaf certificates
vault write pki_int/roles/web-server \
  allowed_domains="example.com" \
  allow_subdomains=true \
  max_ttl=2160h

Transit Engine (Encryption-as-a-Service)

vault secrets enable transit

# Create encryption key
vault write -f transit/keys/payment-data \
  type=aes256-gcm96

# Encrypt data
vault write transit/encrypt/payment-data \
  plaintext=$(echo "sensitive-data" | base64)

# Decrypt data
vault write transit/decrypt/payment-data \
  ciphertext="vault:v1:..."

# Rotate key (old versions still decrypt, new encrypts with latest)
vault write -f transit/keys/payment-data/rotate

# Rewrap ciphertext to latest key version
vault write transit/rewrap/payment-data \
  ciphertext="vault:v1:..."

Performance and Scaling

Performance Replication (Enterprise)

Primary cluster replicates to secondary clusters in other regions. Secondaries handle read traffic locally.

Performance Standbys (Enterprise)

Standby nodes serve read requests without forwarding to the leader, reducing leader load.

Response Wrapping

Wrap sensitive responses in a single-use token — the recipient unwraps exactly once:

# Wrap a secret (TTL = 5 minutes)
vault kv get -wrap-ttl=5m secret/data/production/db-creds

# Recipient unwraps
vault unwrap <wrapping_token>

Batch Tokens

For high-throughput workloads (Lambda, serverless), use batch tokens instead of service tokens. Batch tokens are not persisted to storage, reducing I/O.

Monitoring and Health

Key Metrics

Health Check Endpoint

# Returns 200 if initialized, unsealed, and active
curl -s https://vault.internal:8200/v1/sys/health

# Status codes:
# 200 — initialized, unsealed, active
# 429 — unsealed, standby
# 472 — disaster recovery secondary
# 473 — performance standby
# 501 — not initialized
# 503 — sealed

Disaster Recovery

Backup

# Raft snapshot (includes all data)
vault operator raft snapshot save backup-$(date +%Y%m%d).snap

# Schedule daily backups via cron
0 2 * * * /usr/local/bin/vault operator raft snapshot save /backups/vault-$(date +\%Y\%m\%d).snap

Restore

# Restore from snapshot (causes brief outage)
vault operator raft snapshot restore backup-20260320.snap

DR Replication (Enterprise)

Secondary cluster in standby. Promote on primary failure:

# On DR secondary
vault operator generate-root -dr-token
vault write sys/replication/dr/secondary/promote dr_operation_token=<token>