Managing Secrets

Managing Secrets

Secure storage, rotation, and delivery of secrets (API keys, database credentials, TLS certificates) for applications and infrastructure.

When to Use This Skill

Use when:

• Storing API keys, database credentials, or encryption keys
• Implementing secret rotation (manual or automatic)
• Syncing secrets from external stores to Kubernetes
• Setting up dynamic secrets (database, cloud providers)
• Scanning code for leaked secrets
• Implementing zero-knowledge patterns
• Meeting compliance requirements (SOC 2, ISO 27001, PCI DSS)

Quick Decision Frameworks

Framework 1: Choosing a Secret Store

Decision Tree:

• Kubernetes? → External Secrets Operator (ESO) with chosen backend
• Single cloud? → Cloud-native (AWS/GCP/Azure)
• Multi-cloud/on-prem? → HashiCorp Vault
• GitOps? → SOPS or Sealed Secrets

Framework 2: Static vs. Dynamic Secrets

Framework 3: Kubernetes Secret Delivery

HashiCorp Vault Fundamentals

Core Components

• Secrets Engines: KV v2 (static), Database (dynamic), AWS, PKI, SSH
• Auth Methods: Kubernetes, JWT/OIDC, AppRole, LDAP
• Policies: HCL-based access control (least privilege)
• Leases: TTL for secrets, auto-renewal, auto-revocation

Static Secrets (KV v2)

# Create secret
vault kv put secret/myapp/config api_key=sk_live_EXAMPLE

# Read secret
vault kv get secret/myapp/config

# List versions
vault kv metadata get secret/myapp/config

Dynamic Database Credentials

# Configure PostgreSQL
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb"

# Create role
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\"..." \
  default_ttl="1h"

# Generate credentials
vault read database/creds/app-role

For detailed Vault architecture, see references/vault-architecture.md.

Kubernetes Integration

External Secrets Operator (ESO)

Syncs secrets from 30+ providers to Kubernetes Secrets.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      auth:
        kubernetes:
          role: "app-role"

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: secret/data/database/config

Vault Secrets Operator (VSO)

Kubernetes-native Vault integration with automatic lease renewal.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/app-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    name: dynamic-db-creds

For ESO vs CSI vs VSO comparison, see references/kubernetes-integration.md.

Secret Rotation Patterns

Pattern 1: Versioned Static Secrets (Blue/Green)

1. Create new secret version in Vault 2. Update staging environment 3. Monitor for errors (24-48 hours) 4. Gradual production rollout (10% → 50% → 100%) 5. Revoke old secret (after 7 days)

Pattern 2: Dynamic Database Credentials

Vault auto-generates credentials with short TTL:

• App fetches credentials from Vault
• Vault automatically renews lease (at 67% of TTL)
• On expiration, Vault revokes access
• On renewal failure, app requests new credentials

Pattern 3: TLS Certificate Rotation

Using cert-manager + Vault PKI:

• cert-manager requests certificate from Vault
• Automatically renews before expiration (default: 67% of duration)
• Updates Kubernetes Secret on renewal
• Optional pod restart (via Reloader)

For detailed rotation workflows, see references/rotation-patterns.md.

Multi-Language Integration

Python (hvac)

import hvac

client = hvac.Client(url='https://vault.example.com')
client.auth.kubernetes(role='app-role', jwt=jwt)

# Fetch dynamic credentials
response = client.secrets.database.generate_credentials(name='postgres-role')
username = response['data']['username']
password = response['data']['password']

Go (Vault API)

import vault "github.com/hashicorp/vault/api"

client, _ := vault.NewClient(vault.DefaultConfig())
k8sAuth, _ := auth.NewKubernetesAuth("app-role")
client.Auth().Login(context.Background(), k8sAuth)

secret, _ := client.Logical().Read("database/creds/postgres-role")

TypeScript (node-vault)

import vault from 'node-vault';

const client = vault({ endpoint: 'https://vault.example.com' });
await client.kubernetesLogin({ role: 'app-role', jwt });

const response = await client.read('database/creds/postgres-role');

For complete examples, see examples/dynamic-db-credentials/.

Secret Scanning

Pre-Commit Hooks (Gitleaks)

# Install Gitleaks
brew install gitleaks

# Run on staged files
gitleaks protect --staged --verbose

Pre-commit hook prevents secrets from being committed. For setup, see examples/secret-scanning/pre-commit.

CI/CD Integration

# GitHub Actions
- name: Run Gitleaks
  uses: gitleaks/gitleaks-action@v2

Remediation Workflow

When a secret is leaked: 1. Rotate immediately (within 1 hour) 2. Revoke at provider 3. Remove from Git history (BFG Repo-Cleaner) 4. Force push (notify team) 5. Audit access (who had access during leak window) 6. Document incident

For detailed remediation, see references/secret-scanning.md.

Zero-Knowledge Patterns

Client-Side Encryption (E2EE)

User password → PBKDF2 → encryption key → encrypt secret → send to server

Server stores only encrypted blobs (cannot decrypt).

Shamir's Secret Sharing

Split secret into N shares, require M to reconstruct (e.g., 3 of 5).

# Initialize Vault with Shamir shares
vault operator init -key-shares=5 -key-threshold=3

# Unseal requires 3 of 5 key shares
vault operator unseal <KEY_1>
vault operator unseal <KEY_2>
vault operator unseal <KEY_3>

For implementations, see references/zero-knowledge.md.

Library Recommendations (2025)

Secret Stores

Secret Scanning

Client Libraries

Common Workflows

Workflow 1: Vault + ESO on Kubernetes

1. Install Vault (Helm chart) 2. Initialize and unseal Vault 3. Enable Kubernetes auth 4. Install External Secrets Operator 5. Create SecretStore (Vault connection) 6. Create ExternalSecret (secret mapping)

For step-by-step guide, see examples/vault-eso-setup/.

Workflow 2: Dynamic Database Credentials

1. Enable database secrets engine 2. Configure database connection 3. Create role with TTL 4. App fetches credentials 5. Vault auto-renews lease

For implementation, see examples/dynamic-db-credentials/.

Workflow 3: Secret Scanning Remediation

1. Gitleaks detects secret 2. Block commit (pre-commit hook) 3. Developer removes secret 4. Developer stores in Vault 5. Developer references Vault path

For setup, see examples/secret-scanning/.

Integration with Related Skills

• auth-security: OAuth client secrets, JWT signing keys
• databases-*: Dynamic database credentials
• deploying-applications: Container registry credentials
• observability: Grafana/Datadog API keys
• infrastructure-as-code: Cloud provider credentials

Security Best Practices

1. Never commit secrets to Git (use Gitleaks pre-commit hook) 2. Use dynamic secrets where possible 3. Rotate secrets regularly (quarterly for static, hourly for dynamic) 4. Implement least privilege (Vault policies, RBAC) 5. Enable audit logging 6. Encrypt at rest (Vault storage, etcd encryption) 7. Use short TTLs (< 24 hours for dynamic secrets) 8. Monitor failed access attempts

Common Pitfalls

Secrets in Environment Variables

Environment variables visible in process lists. Solution: Use file-based secrets (Kubernetes volumes, CSI driver).

Hardcoded Secrets in Manifests

Base64 is not encryption. Solution: Use External Secrets Operator.

No Secret Rotation

Stale credentials increase breach risk. Solution: Use dynamic secrets or automate rotation.

Root Token in Production

Unlimited permissions. Solution: Use auth methods with least privilege policies.

For Detailed Information, See

• references/vault-architecture.md - Vault internals, HA setup, policies
• references/kubernetes-integration.md - ESO, CSI driver, VSO comparison
• references/rotation-patterns.md - Detailed rotation workflows
• references/secret-scanning.md - Gitleaks, remediation procedures
• references/zero-knowledge.md - E2EE, Shamir's secret sharing
• references/cloud-providers.md - AWS, GCP, Azure secret managers
• examples/vault-eso-setup/ - Complete Kubernetes setup
• examples/dynamic-db-credentials/ - Multi-language examples
• examples/secret-scanning/ - Pre-commit hooks, CI/CD
• scripts/setup_vault.sh - Automated Vault installation

Cloud Provider Secret Managers

Quick reference for AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault.

Table of Contents

1. Comparison Matrix 2. AWS Secrets Manager 3. GCP Secret Manager 4. Azure Key Vault 5. Kubernetes Integration (ESO)

Comparison Matrix

AWS Secrets Manager

CLI Commands

# Create secret
aws secretsmanager create-secret \
  --name prod/database/credentials \
  --secret-string '{"username":"admin","password":"secret123"}'

# Retrieve secret
aws secretsmanager get-secret-value \
  --secret-id prod/database/credentials \
  --query SecretString --output text

# Update secret
aws secretsmanager update-secret \
  --secret-id prod/database/credentials \
  --secret-string '{"username":"admin","password":"newsecret456"}'

# Enable automatic rotation
aws secretsmanager rotate-secret \
  --secret-id prod/database/credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:rotate-secret

IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/*"
    }
  ]
}

Python SDK (boto3)

import boto3
import json

client = boto3.client('secretsmanager', region_name='us-east-1')

# Get secret
response = client.get_secret_value(SecretId='prod/database/credentials')
secret = json.loads(response['SecretString'])

print(f"Username: {secret['username']}")
print(f"Password: {secret['password']}")

GCP Secret Manager

gcloud Commands

# Create secret
echo -n "secret-password" | gcloud secrets create db-password \
  --data-file=- \
  --replication-policy=automatic

# Add version
echo -n "new-password" | gcloud secrets versions add db-password \
  --data-file=-

# Access secret
gcloud secrets versions access latest --secret=db-password

# Delete secret
gcloud secrets delete db-password

IAM Binding

gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:app@project.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Python SDK

from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()

# Access secret
name = "projects/my-project/secrets/db-password/versions/latest"
response = client.access_secret_version(request={"name": name})
secret = response.payload.data.decode('UTF-8')

print(f"Secret: {secret}")

Azure Key Vault

Azure CLI Commands

# Create Key Vault
az keyvault create \
  --name my-keyvault \
  --resource-group my-rg \
  --location eastus

# Set secret
az keyvault secret set \
  --vault-name my-keyvault \
  --name db-password \
  --value "secret123"

# Get secret
az keyvault secret show \
  --vault-name my-keyvault \
  --name db-password \
  --query value --output tsv

# Enable soft-delete
az keyvault update \
  --name my-keyvault \
  --enable-soft-delete true \
  --retention-days 90

Access Policy

az keyvault set-policy \
  --name my-keyvault \
  --spn <service-principal-id> \
  --secret-permissions get list

Python SDK

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

credential = DefaultAzureCredential()
client = SecretClient(vault_url="https://my-keyvault.vault.azure.net", credential=credential)

# Get secret
secret = client.get_secret("db-password")
print(f"Secret: {secret.value}")

Kubernetes Integration (ESO)

AWS Secrets Manager + ESO

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: app-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
  target:
    name: db-credentials
  data:
  - secretKey: username
    remoteRef:
      key: prod/database/credentials
      property: username
  - secretKey: password
    remoteRef:
      key: prod/database/credentials
      property: password

GCP Secret Manager + ESO

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secrets
spec:
  provider:
    gcpsm:
      projectID: "my-project-123"
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production-cluster
          serviceAccountRef:
            name: app-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secrets
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: db-password
      version: latest

Azure Key Vault + ESO

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-secrets
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-keyvault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: app-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-secrets
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: db-password

Kubernetes Secret Integration

Comprehensive guide to integrating secret management with Kubernetes using External Secrets Operator, Secrets Store CSI Driver, and Vault Secrets Operator.

Table of Contents

1. Integration Approaches 2. External Secrets Operator (ESO) 3. Secrets Store CSI Driver 4. Vault Secrets Operator (VSO) 5. Comparison Matrix 6. Best Practices

Integration Approaches

Native Kubernetes Secrets (Baseline)

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=  # base64("admin")
  password: cGFzc3dvcmQ=  # base64("password")

Problems:

• Base64 is NOT encryption
• Secrets stored in etcd (encrypted at rest if enabled)
• Manual rotation required
• No audit trail
• Secrets in Git (if committed)

Solution: Use external secret stores + sync operators.

External Secret Stores

┌─────────────────────────────────────────────────────┐
│         External Secret Store Integrations          │
├─────────────────────────────────────────────────────┤
│                                                      │
│  ┌────────────────────────────────────────┐         │
│  │   External Secrets Operator (ESO)      │         │
│  │  ├── Syncs from 30+ providers          │         │
│  │  ├── Creates Kubernetes Secrets        │         │
│  │  └── Polling-based refresh             │         │
│  └────────────────────────────────────────┘         │
│                                                      │
│  ┌────────────────────────────────────────┐         │
│  │   Secrets Store CSI Driver             │         │
│  │  ├── Mounts secrets as files           │         │
│  │  ├── No Kubernetes Secret creation     │         │
│  │  └── Watch-based refresh               │         │
│  └────────────────────────────────────────┘         │
│                                                      │
│  ┌────────────────────────────────────────┐         │
│  │   Vault Secrets Operator (VSO)         │         │
│  │  ├── Vault-specific CRDs               │         │
│  │  ├── Dynamic secret renewal            │         │
│  │  └─ Kubernetes-native experience       │         │
│  └────────────────────────────────────────┘         │
│                                                      │
└─────────────────────────────────────────────────────┘

External Secrets Operator (ESO)

Installation

# Helm installation
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace

# Verify installation
kubectl get pods -n external-secrets-system

Core Resources

SecretStore (Namespace-scoped)

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-role"
          serviceAccountRef:
            name: app-sa

ClusterSecretStore (Cluster-wide)

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-global
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "cluster-admin-role"
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets-system

ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h  # Poll every hour
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials  # Kubernetes Secret name
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: secret/data/database/config
      property: username
  - secretKey: password
    remoteRef:
      key: secret/data/database/config
      property: password

Multi-Provider Examples

AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: app-sa

GCP Secret Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secrets
spec:
  provider:
    gcpsm:
      projectID: "my-project-123"
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production-cluster
          serviceAccountRef:
            name: app-sa

Azure Key Vault

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-secrets
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-vault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: app-sa

Advanced Features

Secret Templating

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-config
spec:
  secretStoreRef:
    name: vault-backend
  target:
    name: app-config
    template:
      type: Opaque
      data:
        config.yaml: |
          database:
            url: postgresql://{{ .username }}:{{ .password }}@postgres:5432/mydb
          api_key: {{ .api_key }}
  data:
  - secretKey: username
    remoteRef:
      key: secret/data/database/config
      property: username
  - secretKey: password
    remoteRef:
      key: secret/data/database/config
      property: password
  - secretKey: api_key
    remoteRef:
      key: secret/data/api/keys
      property: stripe_key

DataFrom (Fetch All Keys)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: all-secrets
spec:
  secretStoreRef:
    name: vault-backend
  target:
    name: all-secrets
  dataFrom:
  - extract:
      key: secret/data/myapp/config  # Fetch all keys from this path

PushSecret (Sync TO External Store)

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: push-credentials
spec:
  refreshInterval: 10m
  secretStoreRefs:
  - name: vault-backend
    kind: SecretStore
  selector:
    secret:
      name: local-secret  # Kubernetes Secret to push
  data:
  - match:
      secretKey: username
      remoteRef:
        remoteKey: secret/data/synced/credentials
        property: username

Secrets Store CSI Driver

Installation

# Install CSI driver
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system

# Install Vault provider
kubectl apply -f https://raw.githubusercontent.com/hashicorp/vault-csi-provider/main/deployment/vault-csi-provider.yaml

SecretProviderClass (Vault)

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: production
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "app-role"
    vaultSkipTLSVerify: "false"
    objects: |
      - objectName: "db-username"
        secretPath: "secret/data/database/config"
        secretKey: "username"
      - objectName: "db-password"
        secretPath: "secret/data/database/config"
        secretKey: "password"

Pod with CSI Volume

apiVersion: v1
kind: Pod
metadata:
  name: app
  namespace: production
spec:
  serviceAccountName: app-sa
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: secrets-store
      mountPath: "/mnt/secrets"
      readOnly: true
    env:
    - name: DB_USERNAME_FILE
      value: /mnt/secrets/db-username
    - name: DB_PASSWORD_FILE
      value: /mnt/secrets/db-password
  volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-database"

Sync to Kubernetes Secret (Optional)

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  secretObjects:
  - secretName: db-credentials
    type: Opaque
    data:
    - objectName: db-username
      key: username
    - objectName: db-password
      key: password
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "app-role"
    objects: |
      - objectName: "db-username"
        secretPath: "secret/data/database/config"
        secretKey: "username"
      - objectName: "db-password"
        secretPath: "secret/data/database/config"
        secretKey: "password"

Auto-Rotation Detection

Application code to watch for file changes:

import time
from pathlib import Path
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class SecretReloader(FileSystemEventHandler):
    def on_modified(self, event):
        if event.src_path == "/mnt/secrets/db-password":
            print("Secret rotated, reloading database connection...")
            reload_database_connection()

observer = Observer()
observer.schedule(SecretReloader(), "/mnt/secrets", recursive=False)
observer.start()

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
observer.join()

Vault Secrets Operator (VSO)

Installation

# Helm installation
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --namespace vault-secrets-operator-system \
  --create-namespace

VaultConnection

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault-connection
  namespace: production
spec:
  address: "https://vault.example.com:8200"
  skipTLSVerify: false
  caCertSecretRef: vault-ca-cert

VaultAuth

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: production
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: app-role
    serviceAccount: app-sa

VaultStaticSecret (KV v2)

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: api-keys
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: secret
  path: myapp/api-keys
  type: kv-v2
  refreshAfter: 1h
  destination:
    create: true
    name: api-keys

VaultDynamicSecret (Database)

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/postgres-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    create: true
    name: dynamic-db-creds
  rolloutRestartTargets:
  - kind: Deployment
    name: app

Automatic Renewal:

• VSO renews lease at 67% of TTL
• On renewal failure, requests new credentials
• Optionally triggers pod restart (rolloutRestartTargets)

VaultPKISecret (TLS Certificates)

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  name: app-tls
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: pki
  role: web-server
  commonName: app.example.com
  altNames:
  - www.app.example.com
  ttl: 24h
  destination:
    create: true
    name: app-tls-cert
  rolloutRestartTargets:
  - kind: Deployment
    name: app

Comparison Matrix

When to Use Each

External Secrets Operator (ESO):

• Multi-cloud environments
• Need to support multiple secret stores
• Static secrets with hourly refresh acceptable
• Team familiar with Kubernetes operators

Secrets Store CSI Driver:

• Need file-based secret delivery
• Automatic rotation without pod restart
• TLS certificates (frequent rotation)
• Applications that watch files for changes

Vault Secrets Operator (VSO):

• Vault-centric infrastructure
• Dynamic secrets (database, cloud)
• Automatic lease renewal required
• Prefer Kubernetes-native CRDs

Best Practices

1. Namespace Isolation

Use namespace-scoped SecretStores:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production  # Isolated to production namespace
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret/production"  # Namespace-specific path
      auth:
        kubernetes:
          role: "production-app-role"
          serviceAccountRef:
            name: app-sa

2. Least Privilege Service Accounts

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-secret-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["db-credentials"]  # Specific secret only
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-secret-reader-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-sa
roleRef:
  kind: Role
  name: app-secret-reader
  apiGroup: rbac.authorization.k8s.io

3. Refresh Interval Tuning

# Frequent rotation (TLS certs)
refreshInterval: 10m

# Moderate rotation (API keys)
refreshInterval: 1h

# Infrequent rotation (static configs)
refreshInterval: 6h

4. Monitoring and Alerting

Prometheus Metrics (ESO):

- external_secrets_sync_calls_total
- external_secrets_sync_calls_error
- external_secrets_status_condition

Alerts:

- alert: ExternalSecretSyncFailure
  expr: external_secrets_sync_calls_error > 0
  for: 5m
  annotations:
    summary: "ExternalSecret sync failing"

5. Encryption at Rest (etcd)

Enable etcd encryption for Kubernetes Secrets:

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <BASE64_ENCODED_SECRET>
  - identity: {}

# kube-apiserver flag
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

6. Secret Versioning

Track secret versions for rollback:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: versioned-secret
  annotations:
    external-secrets.io/secret-version: "v2"
spec:
  secretStoreRef:
    name: vault-backend
  target:
    name: app-config
  data:
  - secretKey: api_key
    remoteRef:
      key: secret/data/api-keys
      version: 2  # Specific version

7. Testing Secret Rotation

# Update secret in Vault
vault kv put secret/myapp/config api_key=new_key_v2

# Wait for refresh interval
sleep 3600

# Verify Kubernetes Secret updated
kubectl get secret db-credentials -o jsonpath='{.data.api_key}' | base64 -d

# Check pod logs for reload
kubectl logs -f deployment/app

8. Disaster Recovery

Backup SecretStore configurations:

# Export all ExternalSecrets
kubectl get externalsecrets -A -o yaml > externalsecrets-backup.yaml

# Export all SecretStores
kubectl get secretstores -A -o yaml > secretstores-backup.yaml

# Restore
kubectl apply -f externalsecrets-backup.yaml
kubectl apply -f secretstores-backup.yaml

Secret Rotation Patterns

Detailed workflows for rotating static secrets, dynamic credentials, and TLS certificates.

Table of Contents

1. Why Rotate Secrets 2. Versioned Static Secrets 3. Dynamic Database Credentials 4. TLS Certificate Rotation 5. Cloud Provider Credentials 6. Automation Scripts

Why Rotate Secrets

Security Benefits:

• Limits blast radius of compromised credentials
• Reduces window of unauthorized access
• Meets compliance requirements (SOC 2, ISO 27001, PCI DSS)
• Detects dormant credential usage

Industry Standards:

• PCI DSS: Rotate passwords every 90 days
• SOC 2: Document rotation policy and evidence
• ISO 27001: Regular credential review and rotation
• NIST: Recommend rotation on suspicion of compromise

Versioned Static Secrets

Pattern: Blue/Green Rotation

For third-party API keys with no auto-rotation support.

Steps:

1. Create New Secret Version

# Vault KV v2 (versioned)
vault kv put secret/api-keys/stripe \
  key=sk_live_NEW_KEY_v2 \
  created_at="2025-12-03T10:00:00Z" \
  rotated_by="ops-team"

# Verify version created
vault kv metadata get secret/api-keys/stripe
# current_version: 2

2. Update Staging Environment

# Update Kubernetes Secret in staging
kubectl set env deployment/api-service -n staging \
  STRIPE_API_KEY=sk_live_NEW_KEY_v2

# Or update ExternalSecret to fetch version 2
kubectl patch externalsecret stripe-key -n staging --type merge -p '
spec:
  data:
  - secretKey: key
    remoteRef:
      key: secret/data/api-keys/stripe
      property: key
      version: 2
'

3. Monitor for Errors (24-48 Hours)

# Check application logs
kubectl logs -f deployment/api-service -n staging | grep -i "stripe\|error"

# Monitor error rates
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~\"5..\"}[5m])"

# Verify API calls succeed
curl https://api.stripe.com/v1/charges \
  -u sk_live_NEW_KEY_v2: \
  -d amount=100 \
  -d currency=usd \
  -d source=tok_test

4. Gradual Production Rollout

# Update 10% of pods
kubectl patch deployment api-service -n production -p '
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
'
kubectl set env deployment/api-service -n production STRIPE_API_KEY=sk_live_NEW_KEY_v2

# Wait 1 hour, check metrics
sleep 3600
kubectl top pods -n production -l app=api-service

# Update remaining pods
kubectl rollout status deployment/api-service -n production

5. Revoke Old Secret (After 7 Days)

# Revoke at provider
# Stripe: Dashboard → API Keys → Revoke old key

# Delete old version from Vault
vault kv metadata delete secret/api-keys/stripe -versions=1

# Document rotation
vault kv put secret/rotation-log/stripe-2025-12 \
  old_key=sk_live_OLD_KEY_v1 \
  new_key=sk_live_NEW_KEY_v2 \
  rotated_at="2025-12-03T10:00:00Z" \
  revoked_at="2025-12-10T10:00:00Z"

Automation Script

#!/bin/bash
# rotate-static-secret.sh

SECRET_PATH=$1  # e.g., "secret/api-keys/stripe"
NEW_VALUE=$2

echo "Rotating secret: $SECRET_PATH"

# Write new version
vault kv put "$SECRET_PATH" \
  key="$NEW_VALUE" \
  created_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  rotated_by="$(whoami)"

# Get new version number
NEW_VERSION=$(vault kv metadata get -format=json "$SECRET_PATH" | jq -r '.current_version')
echo "New version: $NEW_VERSION"

# Update staging
kubectl set env deployment/api-service -n staging \
  API_KEY="$NEW_VALUE"

echo "Secret rotated. Monitor staging for 24-48 hours before production rollout."

Dynamic Database Credentials

Pattern: Automatic Lease Renewal

Vault auto-generates credentials with short TTL (1 hour).

Initial Setup:

# 1. Enable database engine
vault secrets enable database

# 2. Configure PostgreSQL connection
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb?sslmode=require" \
  username="vault-admin" \
  password="vault-admin-password"

# 3. Create role with TTL
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Application Integration (Python):

import hvac
import time
import threading
from sqlalchemy import create_engine
from sqlalchemy.pool import NullPool

class VaultDatabaseClient:
    def __init__(self, vault_url, role):
        self.client = hvac.Client(url=vault_url)
        self.role = role
        self.lease_id = None
        self.engine = None

        # Authenticate
        with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
            jwt = f.read()
        self.client.auth.kubernetes(role='app-role', jwt=jwt)

        # Get initial credentials
        self._refresh_credentials()

        # Start renewal thread
        threading.Thread(target=self._renewal_loop, daemon=True).start()

    def _refresh_credentials(self):
        response = self.client.secrets.database.generate_credentials(name=self.role)
        username = response['data']['username']
        password = response['data']['password']
        self.lease_id = response['lease_id']
        self.lease_duration = response['lease_duration']

        # Create new database engine
        if self.engine:
            self.engine.dispose()
        self.engine = create_engine(
            f"postgresql://{username}:{password}@postgres:5432/mydb",
            poolclass=NullPool  # Don't pool connections (credentials rotate)
        )

        print(f"Credentials refreshed. Lease ID: {self.lease_id}, TTL: {self.lease_duration}s")

    def _renewal_loop(self):
        while True:
            # Renew at 67% of lease duration
            renewal_time = self.lease_duration * 0.67
            time.sleep(renewal_time)

            try:
                # Attempt renewal
                self.client.sys.renew_lease(self.lease_id)
                print(f"Lease renewed: {self.lease_id}")
            except Exception as e:
                print(f"Renewal failed: {e}. Requesting new credentials...")
                self._refresh_credentials()

    def query(self, sql):
        with self.engine.connect() as conn:
            return conn.execute(sql).fetchall()

# Usage
db_client = VaultDatabaseClient('https://vault.example.com', 'app-role')
users = db_client.query("SELECT * FROM users")

Kubernetes with VSO:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/app-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    create: true
    name: dynamic-db-creds
  rolloutRestartTargets:
  - kind: Deployment
    name: app  # Auto-restart on credential change

Monitoring:

# Check active leases
vault list sys/leases/lookup/database/creds/app-role

# View lease details
vault lease lookup database/creds/app-role/<LEASE_ID>

# Force revoke (emergency)
vault lease revoke database/creds/app-role/<LEASE_ID>

# Revoke all leases for role
vault lease revoke -prefix database/creds/app-role

TLS Certificate Rotation

Pattern: cert-manager + Vault PKI

Automatic certificate issuance and renewal.

Vault PKI Setup:

# 1. Enable PKI engine
vault secrets enable pki
vault secrets tune -max-lease-ttl=8760h pki

# 2. Generate root CA
vault write pki/root/generate/internal \
  common_name=example.com \
  ttl=8760h

# 3. Configure URLs
vault write pki/config/urls \
  issuing_certificates="https://vault.example.com/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com/v1/pki/crl"

# 4. Create role
vault write pki/roles/web-server \
  allowed_domains=example.com \
  allow_subdomains=true \
  max_ttl=72h \
  key_type=rsa \
  key_bits=2048

cert-manager Integration:

# 1. Issuer (Vault-backed)
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: production
spec:
  vault:
    server: https://vault.example.com
    path: pki/sign/web-server
    auth:
      kubernetes:
        role: cert-manager
        mountPath: /v1/auth/kubernetes
        secretRef:
          name: cert-manager-vault-token
          key: token

---
# 2. Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: app-tls
  namespace: production
spec:
  secretName: app-tls-secret
  duration: 24h
  renewBefore: 8h  # Renew 8 hours before expiration (67% of 24h)
  issuerRef:
    name: vault-issuer
  dnsNames:
  - app.example.com
  - www.app.example.com

Automatic Renewal:

cert-manager automatically: 1. Monitors certificate expiration 2. Requests new certificate 8 hours before expiry 3. Updates Kubernetes Secret (app-tls-secret) 4. Triggers pod reload (via Reloader or similar)

Pod Reload with Reloader:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  annotations:
    reloader.stakater.com/auto: "true"  # Auto-reload on Secret change
spec:
  template:
    spec:
      containers:
      - name: app
        volumeMounts:
        - name: tls
          mountPath: /etc/tls
      volumes:
      - name: tls
        secret:
          secretName: app-tls-secret

Cloud Provider Credentials

Pattern: AWS IAM with Vault

Vault AWS Engine Setup:

# 1. Enable AWS engine
vault secrets enable aws

# 2. Configure root credentials
vault write aws/config/root \
  access_key=AKIAIOSFODNN7EXAMPLE \
  secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  region=us-east-1

# 3. Create role with inline policy
vault write aws/roles/s3-access \
  credential_type=iam_user \
  policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
EOF
  default_ttl=15m \
  max_ttl=1h

Application Integration:

import boto3
import hvac
import time
import threading

class VaultAWSClient:
    def __init__(self, vault_url, aws_role):
        self.client = hvac.Client(url=vault_url)
        self.aws_role = aws_role
        self.lease_id = None

        # Authenticate with Vault
        with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
            jwt = f.read()
        self.client.auth.kubernetes(role='app-role', jwt=jwt)

        # Get initial AWS credentials
        self._refresh_credentials()

        # Start renewal thread
        threading.Thread(target=self._renewal_loop, daemon=True).start()

    def _refresh_credentials(self):
        response = self.client.secrets.aws.generate_credentials(name=self.aws_role)
        self.access_key = response['data']['access_key']
        self.secret_key = response['data']['secret_key']
        self.lease_id = response['lease_id']
        self.lease_duration = response['lease_duration']

        # Update boto3 session
        self.session = boto3.Session(
            aws_access_key_id=self.access_key,
            aws_secret_access_key=self.secret_key
        )
        self.s3 = self.session.client('s3')

        print(f"AWS credentials refreshed. Lease: {self.lease_id}, TTL: {self.lease_duration}s")

    def _renewal_loop(self):
        while True:
            renewal_time = self.lease_duration * 0.67
            time.sleep(renewal_time)

            try:
                self.client.sys.renew_lease(self.lease_id)
                print(f"Lease renewed: {self.lease_id}")
            except Exception as e:
                print(f"Renewal failed. Requesting new credentials...")
                self._refresh_credentials()

    def upload_file(self, bucket, key, file_path):
        self.s3.upload_file(file_path, bucket, key)

# Usage
aws_client = VaultAWSClient('https://vault.example.com', 's3-access')
aws_client.upload_file('my-bucket', 'data.csv', '/tmp/data.csv')

Automation Scripts

Comprehensive Rotation Script

#!/usr/bin/env python3
"""
rotate_secrets.py

Automated secret rotation script for multiple secret types.

Usage:
  python rotate_secrets.py --type static --path secret/api-keys/stripe --value sk_live_NEW
  python rotate_secrets.py --type database --role app-role --ttl 1h
"""

import argparse
import hvac
import time
from datetime import datetime

class SecretRotator:
    def __init__(self, vault_url, vault_token):
        self.client = hvac.Client(url=vault_url, token=vault_token)

    def rotate_static(self, path, new_value):
        """Rotate static secret with versioning."""
        print(f"Rotating static secret: {path}")

        # Write new version
        self.client.secrets.kv.v2.create_or_update_secret(
            path=path.replace('secret/data/', '').replace('secret/', ''),
            secret={'key': new_value},
            mount_point='secret'
        )

        # Get metadata
        metadata = self.client.secrets.kv.v2.read_secret_metadata(
            path=path.replace('secret/data/', '').replace('secret/', ''),
            mount_point='secret'
        )

        version = metadata['data']['current_version']
        print(f"✓ New version created: {version}")
        print(f"✓ Previous version available for rollback")

        return version

    def rotate_database(self, role, ttl='1h'):
        """Generate new dynamic database credentials."""
        print(f"Generating new database credentials for role: {role}")

        response = self.client.secrets.database.generate_credentials(name=role)

        print(f"✓ Username: {response['data']['username']}")
        print(f"✓ Lease ID: {response['lease_id']}")
        print(f"✓ TTL: {response['lease_duration']}s")

        return response

    def revoke_old_version(self, path, versions_to_keep=2):
        """Delete old versions of static secret."""
        print(f"Cleaning up old versions: {path}")

        metadata = self.client.secrets.kv.v2.read_secret_metadata(
            path=path.replace('secret/data/', '').replace('secret/', ''),
            mount_point='secret'
        )

        current_version = metadata['data']['current_version']
        versions_to_delete = list(range(1, current_version - versions_to_keep + 1))

        if versions_to_delete:
            self.client.secrets.kv.v2.delete_secret_versions(
                path=path.replace('secret/data/', '').replace('secret/', ''),
                versions=versions_to_delete,
                mount_point='secret'
            )
            print(f"✓ Deleted versions: {versions_to_delete}")
        else:
            print("✓ No old versions to delete")

def main():
    parser = argparse.ArgumentParser(description='Rotate secrets in Vault')
    parser.add_argument('--type', required=True, choices=['static', 'database'])
    parser.add_argument('--path', help='Secret path (for static secrets)')
    parser.add_argument('--value', help='New secret value (for static secrets)')
    parser.add_argument('--role', help='Database role name (for dynamic secrets)')
    parser.add_argument('--ttl', default='1h', help='TTL for dynamic secrets')
    parser.add_argument('--vault-url', default='https://vault.example.com')
    parser.add_argument('--vault-token', required=True)

    args = parser.parse_args()

    rotator = SecretRotator(args.vault_url, args.vault_token)

    if args.type == 'static':
        if not args.path or not args.value:
            parser.error("--path and --value required for static secrets")
        rotator.rotate_static(args.path, args.value)

        # Wait 7 days before cleanup (manual trigger)
        print("\nRun the following command in 7 days to clean up old versions:")
        print(f"  python rotate_secrets.py --type cleanup --path {args.path} --vault-token <TOKEN>")

    elif args.type == 'database':
        if not args.role:
            parser.error("--role required for database secrets")
        rotator.rotate_database(args.role, args.ttl)

if __name__ == '__main__':
    main()

Validation Script

#!/bin/bash
# validate_rotation.sh

set -e

SECRET_PATH=$1
EXPECTED_VERSION=$2

echo "Validating secret rotation: $SECRET_PATH"

# Check Vault
CURRENT_VERSION=$(vault kv metadata get -format=json "$SECRET_PATH" | jq -r '.current_version')

if [ "$CURRENT_VERSION" -eq "$EXPECTED_VERSION" ]; then
  echo "✓ Vault version correct: $CURRENT_VERSION"
else
  echo "✗ Vault version mismatch. Expected: $EXPECTED_VERSION, Got: $CURRENT_VERSION"
  exit 1
fi

# Check Kubernetes Secret (if using ESO)
K8S_SECRET_NAME=$(echo "$SECRET_PATH" | sed 's/\//-/g')
K8S_VERSION=$(kubectl get secret "$K8S_SECRET_NAME" -o jsonpath='{.metadata.annotations.external-secrets\.io/secret-version}')

if [ "$K8S_VERSION" -eq "$EXPECTED_VERSION" ]; then
  echo "✓ Kubernetes Secret synced: $K8S_VERSION"
else
  echo "✗ Kubernetes Secret not synced. Expected: $EXPECTED_VERSION, Got: $K8S_VERSION"
  exit 1
fi

echo "✓ Rotation validated successfully"

Secret Scanning and Remediation

Comprehensive guide to detecting, preventing, and remediating leaked secrets using Gitleaks and other tools.

Table of Contents

1. Why Scan for Secrets 2. Gitleaks 3. Pre-Commit Hooks 4. CI/CD Integration 5. Remediation Workflow 6. Alternative Tools

Why Scan for Secrets

The Problem:

• 10M+ secrets exposed on GitHub in 2024 (GitGuardian)
• Average breach cost: $4.45M (IBM 2025)
• 63% of breaches from leaked credentials (Verizon DBIR)
• 95% preventable with secret scanning

Common Leak Sources:

• Git commits (hardcoded passwords, API keys)
• Environment files (.env, config.json)
• Configuration files (application.yml, settings.py)
• CI/CD logs (exposed secrets in build output)
• Docker images (secrets baked into layers)

Gitleaks

High-performance secret scanner with customizable rules.

Installation

# macOS
brew install gitleaks

# Linux
curl -sSfL https://raw.githubusercontent.com/gitleaks/gitleaks/master/scripts/install.sh | sh

# Docker
docker pull ghcr.io/gitleaks/gitleaks:latest

Basic Usage

# Scan current repository
gitleaks detect --verbose

# Scan specific directory
gitleaks detect --source /path/to/repo --verbose

# Scan Git history
gitleaks detect --log-opts "--all" --verbose

# Protect mode (scan uncommitted changes)
gitleaks protect --staged --verbose

# Generate report
gitleaks detect --report-format json --report-path gitleaks-report.json

Configuration (.gitleaks.toml)

title = "Gitleaks Configuration"

[extend]
useDefault = true

[[rules]]
id = "generic-api-key"
description = "Generic API Key"
regex = '''(?i)(api[_-]?key|apikey)['"` ]*[:=]['"` ]*[a-zA-Z0-9]{20,}'''
tags = ["key", "API"]

[[rules]]
id = "aws-access-key"
description = "AWS Access Key"
regex = '''AKIA[0-9A-Z]{16}'''
tags = ["AWS", "key"]

[[rules]]
id = "private-key"
description = "Private Key"
regex = '''-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----'''
tags = ["key", "private"]

[[rules]]
id = "jwt"
description = "JSON Web Token"
regex = '''eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}'''
tags = ["JWT"]

[allowlist]
description = "Allowlist for test files and examples"
paths = [
  '''.*/test/.*''',
  '''.*/examples/.*''',
  '''.*_test\.go''',
  '''.*\.md'''
]
regexes = [
  '''sk_test_''',  # Stripe test keys
  '''pk_test_''',  # Stripe test keys
  '''EXAMPLE_API_KEY''',  # Placeholder examples
]

Exit Codes

gitleaks detect
echo $?

# 0: No secrets found
# 1: Secrets detected
# 2: Error occurred

Pre-Commit Hooks

Install pre-commit

# Install pre-commit framework
pip install pre-commit

# Or via Homebrew
brew install pre-commit

.pre-commit-config.yaml

repos:
- repo: https://github.com/gitleaks/gitleaks
  rev: v8.18.1
  hooks:
  - id: gitleaks
    name: Gitleaks
    description: Detect secrets in staged files
    entry: gitleaks protect --staged --redact
    language: system
    pass_filenames: false

Manual Hook (.git/hooks/pre-commit)

#!/bin/bash
# .git/hooks/pre-commit

echo "Running Gitleaks secret scan..."

# Run Gitleaks on staged files
gitleaks protect --staged --verbose --redact

if [ $? -ne 0 ]; then
  echo ""
  echo "❌ Secret detected! Commit blocked."
  echo "To fix:"
  echo "  1. Remove the secret from your code"
  echo "  2. Store it in Vault: vault kv put secret/myapp/config api_key=<YOUR_KEY>"
  echo "  3. Reference it in code: vault.read('secret/data/myapp/config')"
  echo ""
  echo "To bypass (NOT recommended): git commit --no-verify"
  exit 1
fi

echo "✅ No secrets detected. Proceeding with commit."
exit 0

Install hook:

chmod +x .git/hooks/pre-commit

Husky (for Node.js projects)

{
  "husky": {
    "hooks": {
      "pre-commit": "gitleaks protect --staged --verbose"
    }
  }
}

npx husky install
npx husky add .husky/pre-commit "gitleaks protect --staged --verbose"

CI/CD Integration

GitHub Actions

# .github/workflows/secret-scan.yml
name: Secret Scanning

on:
  push:
    branches: [main, develop]
  pull_request:

jobs:
  gitleaks:
    name: Gitleaks Secret Scan
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        fetch-depth: 0  # Full history for comprehensive scan

    - name: Run Gitleaks
      uses: gitleaks/gitleaks-action@v2
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}  # For commercial use

    - name: Upload results (if secrets found)
      if: failure()
      uses: actions/upload-artifact@v4
      with:
        name: gitleaks-report
        path: gitleaks-report.sarif

GitLab CI

# .gitlab-ci.yml
gitleaks:
  stage: test
  image: ghcr.io/gitleaks/gitleaks:latest
  script:
  - gitleaks detect --verbose --report-format json --report-path gitleaks-report.json
  artifacts:
    when: on_failure
    paths:
    - gitleaks-report.json
  allow_failure: false

Jenkins

pipeline {
  agent any
  stages {
    stage('Secret Scan') {
      steps {
        sh 'gitleaks detect --verbose --report-format json --report-path gitleaks-report.json'
      }
    }
  }
  post {
    always {
      archiveArtifacts artifacts: 'gitleaks-report.json', allowEmptyArchive: true
    }
  }
}

CircleCI

# .circleci/config.yml
version: 2.1

jobs:
  gitleaks:
    docker:
    - image: ghcr.io/gitleaks/gitleaks:latest
    steps:
    - checkout
    - run:
        name: Run Gitleaks
        command: gitleaks detect --verbose
    - store_artifacts:
        path: gitleaks-report.json

workflows:
  version: 2
  build:
    jobs:
    - gitleaks

Remediation Workflow

When a Secret is Leaked

Immediate Actions (within 1 hour):

1. Rotate the Exposed Secret

# PRIORITY 1: Create new secret in Vault
vault kv put secret/api-keys/stripe \
  key=sk_live_NEW_KEY_AFTER_LEAK \
  rotated_reason="Leaked to Git on 2025-12-03" \
  previous_key=sk_live_OLD_KEY_LEAKED

2. Revoke at Provider

# Stripe: Dashboard → API Keys → Revoke
# AWS: IAM → Delete access key
# GitHub: Settings → Developer settings → Revoke token

3. Update Applications

# Update all environments immediately
kubectl set env deployment/api-service -n staging STRIPE_API_KEY=sk_live_NEW_KEY_AFTER_LEAK
kubectl set env deployment/api-service -n production STRIPE_API_KEY=sk_live_NEW_KEY_AFTER_LEAK

Git History Cleanup (within 24 hours):

4. Remove from Git History (BFG Repo-Cleaner)

# Install BFG
brew install bfg

# Clone fresh copy
git clone --mirror https://github.com/org/repo.git repo.git
cd repo.git

# Create file with secrets to remove
cat > secrets.txt <<EOF
sk_live_OLD_KEY_LEAKED
AKIA1234567890ABCDEF
-----BEGIN RSA PRIVATE KEY-----
EOF

# Remove secrets
bfg --replace-text secrets.txt .

# Clean up
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Force push (CAUTION: Coordinate with team)
git push --force --all
git push --force --tags

Alternative: filter-repo

pip install git-filter-repo

# Remove specific file
git filter-repo --invert-paths --path config/secrets.json

# Remove specific pattern
echo "sk_live_" > secrets-pattern.txt
git filter-repo --replace-text secrets-pattern.txt

5. Audit Access

# Check Vault audit logs
vault audit list

# Read audit log
cat /vault/logs/audit.log | jq 'select(.request.path == "secret/data/api-keys/stripe")'

# GitHub: Who cloned during leak window?
# Check repository Insights → Traffic → Git clones

6. Document Incident

# Incident Report: Secret Leak 2025-12-03

## Summary
- **Date**: 2025-12-03 10:15 UTC
- **Secret**: Stripe API key (sk_live_OLD_KEY_LEAKED)
- **Exposure**: Committed to main branch, pushed to GitHub
- **Discovered**: Gitleaks CI/CD scan
- **Resolved**: Secret rotated within 30 minutes

## Timeline
- 10:15: Secret committed (commit abc123)
- 10:20: CI/CD pipeline detected leak, blocked merge
- 10:25: Secret rotated in Vault
- 10:30: Old key revoked at Stripe
- 11:00: Git history rewritten (BFG)
- 11:15: Force push completed

## Root Cause
- Developer accidentally committed .env file
- Pre-commit hook not installed locally

## Prevention
- [ ] Enforce pre-commit hooks (CI check)
- [ ] Add .env to .gitignore (template)
- [ ] Developer training on secret management
- [ ] Mandatory .gitleaks.toml in all repos

## Impact
- No unauthorized API usage detected
- No customer data accessed
- Leak window: 15 minutes (before rotation)

False Positive Handling

Allowlist in .gitleaks.toml:

[allowlist]
description = "Allowlist for test files and examples"

# Ignore test files
paths = [
  '''.*/test/.*''',
  '''.*/examples/.*''',
  '''.*\.md''',  # Documentation
]

# Ignore test keys and placeholders
regexes = [
  '''sk_test_[a-zA-Z0-9]{24}''',  # Stripe test keys
  '''pk_test_[a-zA-Z0-9]{24}''',
  '''EXAMPLE_[A-Z_]+''',  # Placeholders
  '''YOUR_API_KEY_HERE''',
]

# Ignore specific commits (emergency override)
commits = [
  "abc123def456",  # Migration commit with test data
]

Inline Ignore:

# gitleaks:allow
api_key = "sk_test_THIS_IS_A_TEST_KEY_FOR_EXAMPLES"

Alternative Tools

TruffleHog

Deep Git history scanning with entropy detection.

# Installation
pip install truffleHog

# Scan repository
trufflehog git https://github.com/org/repo.git

# Scan since specific commit
trufflehog git https://github.com/org/repo.git --since-commit abc123

# JSON output
trufflehog git https://github.com/org/repo.git --json

detect-secrets (Yelp)

Baseline-based scanning for Python projects.

# Installation
pip install detect-secrets

# Create baseline
detect-secrets scan > .secrets.baseline

# Audit baseline (interactive)
detect-secrets audit .secrets.baseline

# Scan for new secrets
detect-secrets scan --baseline .secrets.baseline

git-secrets (AWS)

Prevents committing AWS credentials.

# Installation
brew install git-secrets

# Install hooks
git secrets --install
git secrets --register-aws

# Scan repository
git secrets --scan

Comparison Matrix

Recommendation: Use Gitleaks for most projects (fast, accurate, easy CI/CD integration).

Best Practices

1. Layer Defenses

• Pre-commit hooks (local)
• CI/CD scanning (remote)
• Periodic full repository scans

2. Rotate Immediately

• Assume leaked = compromised
• Rotate within 1 hour
• Revoke at provider

3. Educate Developers

• Secret management training
• Code review checklist
• Security champions

4. Monitor Audit Logs

• Track secret access patterns
• Alert on unusual activity
• Regular audit reviews

5. Automate Response

• Auto-rotate on detection
• Auto-revoke leaked credentials
• Incident tickets

HashiCorp Vault Architecture

Comprehensive guide to Vault internals, high availability setup, policies, and production deployment patterns.

Table of Contents

1. Architecture Overview 2. Secrets Engines 3. Authentication Methods 4. Policies and Access Control 5. Storage Backends 6. High Availability Setup 7. Audit Logging 8. Production Deployment 9. Common Pitfalls

Architecture Overview

Core Components

┌─────────────────────────────────────────────────────────┐
│              HashiCorp Vault Architecture                │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  ┌────────────────────────────────────────────┐         │
│  │         API Layer (HTTP/HTTPS)             │         │
│  │  ├── RESTful API                           │         │
│  │  ├── CLI (vault command)                   │         │
│  │  └── UI (Web interface)                    │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │      Authentication Methods                │         │
│  │  ├── Kubernetes (service accounts)         │         │
│  │  ├── JWT/OIDC (external identity)          │         │
│  │  ├── AppRole (machines, CI/CD)             │         │
│  │  ├── LDAP/Active Directory                 │         │
│  │  └── Token (direct auth)                   │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │         Core (Policy Engine)               │         │
│  │  ├── Request routing                       │         │
│  │  ├── Policy evaluation (HCL)               │         │
│  │  ├── Lease management                      │         │
│  │  └── Token generation                      │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │         Secrets Engines                    │         │
│  │  ├── KV v2 (versioned key-value)           │         │
│  │  ├── Database (dynamic credentials)        │         │
│  │  ├── AWS (dynamic IAM)                     │         │
│  │  ├── PKI (TLS certificates)                │         │
│  │  └── SSH (dynamic certificates)            │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │      Barrier (Encryption Layer)            │         │
│  │  ├── AES-256-GCM encryption                │         │
│  │  ├── Unsealing mechanism                   │         │
│  │  └── Master key protection                 │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │      Storage Backend                       │         │
│  │  ├── Consul (HA, recommended)              │         │
│  │  ├── etcd (Kubernetes native)              │         │
│  │  ├── S3 (cost-effective, no HA)            │         │
│  │  └── PostgreSQL (relational)               │         │
│  └────────────────────────────────────────────┘         │
│                                                          │
└─────────────────────────────────────────────────────────┘

Data Flow

1. Request: Client sends request (CLI, API, SDK) 2. Authentication: Vault validates identity (token, K8s SA, etc.) 3. Authorization: Policy engine evaluates permissions 4. Secrets Engine: Generates or retrieves secret 5. Encryption: Barrier encrypts data 6. Storage: Encrypted data written to backend 7. Response: Decrypted secret returned to client

Secrets Engines

KV v2 (Versioned Key-Value)

Static secrets with versioning and soft deletes.

Enable:

vault secrets enable -path=secret kv-v2

Write Secret:

vault kv put secret/myapp/config \
  api_key=sk_live_123 \
  database_url=postgresql://localhost/mydb

Read Secret:

vault kv get secret/myapp/config
vault kv get -version=2 secret/myapp/config  # Specific version

Versioning:

vault kv metadata get secret/myapp/config  # View all versions
vault kv undelete -versions=2 secret/myapp/config  # Restore deleted version
vault kv destroy -versions=1 secret/myapp/config  # Permanently delete

Use Cases:

• API keys (third-party services)
• OAuth client secrets
• Encryption keys (KEK)
• Configuration values

Database Secrets Engine (Dynamic Credentials)

Auto-generates database credentials with TTL.

Enable:

vault secrets enable database

Configure PostgreSQL:

vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role,readonly-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb?sslmode=require" \
  username="vault-admin" \
  password="vault-admin-password"

Create Role:

vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Generate Credentials:

vault read database/creds/app-role
# Output:
# Key                Value
# ---                -----
# lease_id           database/creds/app-role/abc123
# lease_duration     1h
# username           v-k8s-app-role-xyz789
# password           A1b2C3d4E5f6

Rotation:

# Rotate root credentials (Vault admin password)
vault write -force database/rotate-root/postgres

Supported Databases:

• PostgreSQL, MySQL, MongoDB
• MSSQL, Oracle, Cassandra
• Elasticsearch, InfluxDB
• Redshift, Snowflake

AWS Secrets Engine (Dynamic IAM)

Auto-generates AWS IAM credentials.

Enable:

vault secrets enable aws

Configure:

vault write aws/config/root \
  access_key=AKIAIOSFODNN7EXAMPLE \
  secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  region=us-east-1

Create Role:

vault write aws/roles/app-role \
  credential_type=iam_user \
  policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
EOF

Generate Credentials:

vault read aws/creds/app-role
# Output:
# Key                Value
# ---                -----
# lease_id           aws/creds/app-role/def456
# lease_duration     15m
# access_key         AKIAI44QH8DHBEXAMPLE
# secret_key         je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

TTL: Default 15 minutes, auto-revoked on expiration.

PKI Secrets Engine (TLS Certificates)

Issues TLS certificates with automatic renewal.

Enable:

vault secrets enable pki
vault secrets tune -max-lease-ttl=8760h pki

Generate Root CA:

vault write pki/root/generate/internal \
  common_name=example.com \
  ttl=8760h

Configure URLs:

vault write pki/config/urls \
  issuing_certificates="https://vault.example.com/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com/v1/pki/crl"

Create Role:

vault write pki/roles/web-server \
  allowed_domains=example.com \
  allow_subdomains=true \
  max_ttl=72h

Issue Certificate:

vault write pki/issue/web-server \
  common_name=app.example.com \
  ttl=24h
# Returns: certificate, issuing_ca, private_key

Authentication Methods

Kubernetes Auth

Authenticate using Kubernetes service account tokens.

Enable:

vault auth enable kubernetes

Configure:

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  token_reviewer_jwt="<service-account-jwt>" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Create Role:

vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app-sa \
  bound_service_account_namespaces=production \
  policies=app-policy \
  ttl=1h

Login (from pod):

vault write auth/kubernetes/login \
  role=app-role \
  jwt=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

JWT/OIDC Auth

Authenticate using external identity providers (Okta, Auth0, Google).

Enable:

vault auth enable oidc

Configure:

vault write auth/oidc/config \
  oidc_discovery_url="https://accounts.google.com" \
  oidc_client_id="<client-id>" \
  oidc_client_secret="<client-secret>" \
  default_role="default"

Create Role:

vault write auth/oidc/role/default \
  bound_audiences="<client-id>" \
  allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
  user_claim="email" \
  policies=default

AppRole Auth

Machine authentication for CI/CD, applications.

Enable:

vault auth enable approle

Create Role:

vault write auth/approle/role/ci-role \
  secret_id_ttl=10m \
  token_num_uses=10 \
  token_ttl=20m \
  token_max_ttl=30m \
  secret_id_num_uses=40 \
  policies=ci-policy

Get RoleID:

vault read auth/approle/role/ci-role/role-id
# role_id: 12345678-1234-1234-1234-123456789012

Generate SecretID:

vault write -f auth/approle/role/ci-role/secret-id
# secret_id: 87654321-4321-4321-4321-210987654321

Login:

vault write auth/approle/login \
  role_id=12345678-1234-1234-1234-123456789012 \
  secret_id=87654321-4321-4321-4321-210987654321

Policies and Access Control

Policy Syntax (HCL)

# Read access to KV v2 secrets
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}

# Write access to specific secret
path "secret/data/myapp/config" {
  capabilities = ["create", "update", "read"]
}

# Dynamic database credentials
path "database/creds/app-role" {
  capabilities = ["read"]
}

# Deny all other paths
path "*" {
  capabilities = ["deny"]
}

Capabilities:

• create: Create new secrets
• read: Read existing secrets
• update: Update existing secrets
• delete: Delete secrets
• list: List secret paths
• sudo: Admin operations
• deny: Explicitly deny access

Policy Templates

Dynamic policies using templates:

# Policy for service accounts in specific namespace
path "secret/data/{{identity.entity.aliases.auth_kubernetes_abc123.metadata.service_account_namespace}}/*" {
  capabilities = ["read"]
}

Writing Policies

# Write policy from file
vault policy write app-policy app-policy.hcl

# Write policy inline
vault policy write readonly-policy - <<EOF
path "secret/data/*" {
  capabilities = ["read", "list"]
}
EOF

# List policies
vault policy list

# Read policy
vault policy read app-policy

Least Privilege Example

# Application policy (minimal permissions)
path "secret/data/myapp/config" {
  capabilities = ["read"]
}

path "database/creds/app-role" {
  capabilities = ["read"]
}

path "auth/token/renew-self" {
  capabilities = ["update"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

Storage Backends

Consul (Recommended for HA)

Pros:

• High availability
• Automatic failover
• Service discovery
• Health checking

Cons:

• Requires separate Consul cluster
• More complex setup

Configuration:

storage "consul" {
  address = "consul.example.com:8500"
  path    = "vault/"
  token   = "<consul-token>"
}

etcd (Kubernetes Native)

Pros:

• Kubernetes native
• HA support
• Widely deployed

Cons:

• Performance at scale
• Requires etcd cluster

Configuration:

storage "etcd" {
  address  = "https://etcd.example.com:2379"
  etcd_api = "v3"
  path     = "vault/"
  ha_enabled = "true"
}

S3 (Cost-Effective)

Pros:

• Low cost
• Managed service
• Unlimited storage

Cons:

• No HA support
• Higher latency

Configuration:

storage "s3" {
  bucket     = "vault-storage"
  region     = "us-east-1"
  access_key = "AKIAIOSFODNN7EXAMPLE"
  secret_key = "wJalrXUtnFEMI/K7MDENG"
}

PostgreSQL

Pros:

• Relational database
• Familiar tooling
• HA with replication

Cons:

• Not officially recommended
• Performance tuning needed

Configuration:

storage "postgresql" {
  connection_url = "postgres://vault:password@postgres:5432/vault?sslmode=require"
  ha_enabled     = "true"
}

High Availability Setup

HA Architecture

┌────────────────────────────────────────────────┐
│         Vault HA Cluster (3 nodes)             │
├────────────────────────────────────────────────┤
│                                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────┐ │
│  │   vault-0    │  │   vault-1    │  │vault-│ │
│  │   (Active)   │  │  (Standby)   │  │  2   │ │
│  │              │  │              │  │(Stand│ │
│  └──────┬───────┘  └──────┬───────┘  └──┬───┘ │
│         │                 │              │     │
│         └─────────────────┼──────────────┘     │
│                           │                    │
│                           ▼                    │
│              ┌─────────────────────┐           │
│              │   Storage Backend   │           │
│              │   (Consul/etcd)     │           │
│              └─────────────────────┘           │
│                                                 │
└────────────────────────────────────────────────┘

Key Concepts:

• One active node (handles requests)
• Multiple standby nodes (ready for failover)
• Automatic leader election
• Storage backend coordinates HA

Kubernetes Deployment (HA)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
spec:
  serviceName: vault
  replicas: 3
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15
        args:
        - "server"
        - "-config=/vault/config/vault.hcl"
        ports:
        - containerPort: 8200
          name: api
        - containerPort: 8201
          name: cluster
        volumeMounts:
        - name: config
          mountPath: /vault/config
        - name: data
          mountPath: /vault/data
      volumes:
      - name: config
        configMap:
          name: vault-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

vault.hcl:

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-0"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/vault/tls/tls.crt"
  tls_key_file  = "/vault/tls/tls.key"
}

api_addr = "https://vault-0.vault:8200"
cluster_addr = "https://vault-0.vault:8201"

ui = true

Unsealing in HA

All nodes must be unsealed independently:

# Unseal each node (requires 3 of 5 key shares)
kubectl exec vault-0 -- vault operator unseal <KEY_SHARE_1>
kubectl exec vault-0 -- vault operator unseal <KEY_SHARE_2>
kubectl exec vault-0 -- vault operator unseal <KEY_SHARE_3>

kubectl exec vault-1 -- vault operator unseal <KEY_SHARE_1>
kubectl exec vault-1 -- vault operator unseal <KEY_SHARE_2>
kubectl exec vault-1 -- vault operator unseal <KEY_SHARE_3>

kubectl exec vault-2 -- vault operator unseal <KEY_SHARE_1>
kubectl exec vault-2 -- vault operator unseal <KEY_SHARE_2>
kubectl exec vault-2 -- vault operator unseal <KEY_SHARE_3>

Auto-Unseal (Recommended for Production):

Using cloud KMS (AWS, GCP, Azure):

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abc-def-ghi"
}

Audit Logging

Enable Audit Device

# File audit device
vault audit enable file file_path=/vault/logs/audit.log

# Syslog audit device
vault audit enable syslog tag="vault" facility="AUTH"

# Socket audit device
vault audit enable socket address="logstash.example.com:9090" socket_type="tcp"

Audit Log Format

JSON format with request and response details:

{
  "time": "2025-12-03T10:15:30Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:abc123...",
    "accessor": "hmac-sha256:def456...",
    "display_name": "kubernetes-production-app-sa",
    "policies": ["app-policy", "default"]
  },
  "request": {
    "operation": "read",
    "path": "database/creds/app-role",
    "remote_address": "10.244.1.5"
  },
  "response": {
    "secret": {
      "lease_id": "database/creds/app-role/xyz789"
    }
  }
}

Logged Information:

• Authentication details
• Request path and operation
• Response (secrets are HMAC-hashed)
• Remote IP address
• Timestamp

Production Deployment

Checklist

• [ ] TLS enabled (never run Vault without TLS in production)
• [ ] HA setup (3+ nodes with Consul/etcd/Raft storage)
• [ ] Auto-unseal (AWS KMS, GCP KMS, Azure Key Vault)
• [ ] Audit logging (multiple devices for redundancy)
• [ ] Backup strategy (storage backend snapshots)
• [ ] Monitoring (Prometheus metrics, health checks)
• [ ] Policies reviewed (least privilege, no root tokens)
• [ ] Secrets rotation (quarterly for static, automatic for dynamic)
• [ ] Disaster recovery (tested restore procedure)

Resource Requirements

Development:

• 1 CPU, 512 MB RAM
• Single node
• File storage backend

Production (Small - <1000 secrets):

• 2 CPU, 2 GB RAM per node
• 3 nodes (HA)
• Consul/etcd storage

Production (Large - >10,000 secrets):

• 4 CPU, 8 GB RAM per node
• 5+ nodes (HA)
• Dedicated storage cluster
• Load balancer

Monitoring

Prometheus Metrics:

# vault.hcl
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = true
}

Key Metrics:

• vault_core_unsealed: Vault seal status (1 = unsealed)
• vault_core_active: Active node (1 = active, 0 = standby)
• vault_token_count: Total active tokens
• vault_secret_lease_creation: Secret lease generation rate
• vault_runtime_alloc_bytes: Memory usage

Common Pitfalls

Pitfall 1: Running Without TLS

Problem: Secrets transmitted in plaintext.

Solution:

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0  # NEVER set to 1 in production
  tls_cert_file = "/vault/tls/tls.crt"
  tls_key_file  = "/vault/tls/tls.key"
}

Pitfall 2: Using Root Token

Problem: Unlimited permissions, no audit trail.

Solution: Delete root token after initial setup, use auth methods.

# Revoke root token
vault token revoke <ROOT_TOKEN>

Pitfall 3: No Backup Strategy

Problem: Data loss on storage backend failure.

Solution: Regular snapshots of storage backend.

# Consul snapshot
consul snapshot save backup.snap

# etcd snapshot
ETCDCTL_API=3 etcdctl snapshot save backup.db

Pitfall 4: Unsealing Delays

Problem: Manual unsealing after restart delays recovery.

Solution: Use auto-unseal with cloud KMS.

Pitfall 5: Single Audit Device

Problem: Audit log failure blocks all requests.

Solution: Configure multiple audit devices (file + syslog).

vault audit enable file file_path=/vault/logs/audit.log
vault audit enable syslog tag="vault"

Pitfall 6: No Lease Renewal

Problem: Dynamic secrets expire, causing app failures.

Solution: Implement lease renewal in application code.

import time
import threading

def renew_lease(client, lease_id, lease_duration):
    renewal_time = lease_duration * 0.67  # Renew at 67% of TTL
    time.sleep(renewal_time)
    client.sys.renew_lease(lease_id)

Pitfall 7: Hardcoded Policies in Code

Problem: Policy changes require code deployment.

Solution: Store policies as files, reference in Vault.

vault policy write app-policy /path/to/app-policy.hcl