Terraform Skill

Terraform Operational Traps

Failure patterns from real deployments. Every item caused an incident. Organized as: exact error → root cause → copy-paste fix.

Provisioner traps (symptom → fix)

docker: not found in remote-exec

cloud-init still installing Docker when provisioner SSHs in.

provisioner "remote-exec" {
  inline = [
    "cloud-init status --wait || true",
    "which docker || { echo 'FATAL: Docker not ready'; exit 1; }",
  ]
}

rsync: connection unexpectedly closed in local-exec

Terraform holds its SSH connection open; local-exec rsync opens a second one that gets rejected. Never use local-exec for file transfer to remote. Use tarball + file provisioner:

provisioner "local-exec" {
  command = "tar czf /tmp/src.tar.gz --exclude=node_modules --exclude=.git -C ${path.module}/../../.. myproject"
}
provisioner "file" {
  source      = "/tmp/src.tar.gz"
  destination = "/tmp/src.tar.gz"
}
provisioner "remote-exec" {
  inline = ["tar xzf /tmp/src.tar.gz -C /data/ && rm -f /tmp/src.tar.gz"]
}

macOS BSD tar: --exclude must come BEFORE the source argument.

cloud-init status shows "running" forever

apt-get -y does not suppress debconf dialogs. Packages like iptables-persistent block on TTY prompts.

- |
    echo iptables-persistent iptables-persistent/autosave_v4 boolean true | debconf-set-selections
    echo iptables-persistent iptables-persistent/autosave_v6 boolean true | debconf-set-selections
    DEBIAN_FRONTEND=noninteractive apt-get install -y iptables-persistent

Known offenders: iptables-persistent, postfix, mysql-server, wireshark-common.

EACCES: permission denied in container logs, container Restarting

Host volume dirs are root-owned; container runs as non-root (uid 1001). Fix before docker compose up:

mkdir -p /data/myapp/data /data/myapp/logs
chown -R 1001:1001 /data/myapp/data /data/myapp/logs

Find UID: grep adduser.*-u or USER in Dockerfile.

Provisioner fails but no diagnostic output

set -e exits on first error, hiding subsequent docker logs output. Use set -u without -e, put one verification gate at the end:

provisioner "remote-exec" {
  inline = [
    "set -u",
    "docker compose up -d",
    "sleep 15",
    "docker logs myapp --tail 20 2>&1 || true",
    "docker ps --format 'table {{.Names}}\\t{{.Status}}' || true",
    "docker ps --filter name=myapp --format '{{.Status}}' | grep -q healthy || exit 1",
  ]
}

Container Restarting — database tables missing

DB migrations not in provisioner. PostgreSQL docker-entrypoint-initdb.d only runs on empty data dir. Explicitly create DB + run migrations:

# After postgres healthy:
docker exec pg psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname='mydb'" | grep -q 1 \
  || docker exec pg psql -U postgres -c "CREATE DATABASE mydb;"

# Idempotent migrations:
for f in migrations/*.sql; do
  VER=$(basename $f)
  APPLIED=$($PSQL -tAc "SELECT 1 FROM schema_migrations WHERE version='$VER'" | tr -d ' ')
  [ "$APPLIED" = "1" ] && continue
  { echo 'BEGIN;'; cat $f; echo 'COMMIT;'; } | $PSQL
  $PSQL -tAc "INSERT INTO schema_migrations(version) VALUES ('$VER') ON CONFLICT DO NOTHING"
done

docker compose build ignores env var override

Compose reads build args from .env file, not shell env. VAR=x docker compose build does NOT work.

# WRONG
DOCKER_WITH_PROXY_MODE=disabled docker compose build

# RIGHT
grep -q DOCKER_WITH_PROXY_MODE .env || echo 'DOCKER_WITH_PROXY_MODE=disabled' >> .env
docker compose build

TLS handshake fails: Invalid format for Authorization header

Caddy DNS-01 ACME needs a Cloudflare API Token (cfut_ prefix, 40+ chars, Bearer auth). A Global API Key (37 hex chars, X-Auth-Key auth) causes HTTP 400 Code:6003. Production may appear to work because it has cached certificates; fresh environments fail on first cert request.

# Verify token format before deploy:
TOKEN=$(grep CLOUDFLARE_API_TOKEN .env | cut -d= -f2)
echo "$TOKEN" | grep -q "^cfut_" || echo "FATAL: needs API Token, not Global Key"

Create scoped token via API:

curl -s "https://api.cloudflare.com/client/v4/user/tokens" -X POST \
  -H "X-Auth-Email: $CF_EMAIL" -H "X-Auth-Key: $CF_GLOBAL_KEY" \
  -d '{"name":"caddy-dns-acme","policies":[{"effect":"allow",
    "resources":{"com.cloudflare.api.account.zone.<ZONE_ID>":"*"},
    "permission_groups":[
      {"id":"4755a26eedb94da69e1066d98aa820be","name":"DNS Write"},
      {"id":"c8fed203ed3043cba015a93ad1616f1f","name":"Zone Read"}]}]}'

TLS fails on staging but works on production — hardcoded domains

Caddyfile or compose has literal domain names. Staging Caddy loads production config, tries to get certs for domains it doesn't own → ACME fails.

Caddyfile: Use {$VAR} — Caddy evaluates env vars at startup.

# WRONG
example.com { tls { dns cloudflare {env.CLOUDFLARE_API_TOKEN} } }

# RIGHT
{$LOBEHUB_DOMAIN} { tls { dns cloudflare {env.CLOUDFLARE_API_TOKEN} } }

Compose: Use ${VAR:?required} — fail-fast if unset.

# WRONG
- APP_URL=https://example.com

# RIGHT
- APP_URL=${APP_URL:?APP_URL is required}

Pass the env var to the gateway container so Caddy can read it:

environment:
  - LOBEHUB_DOMAIN=${LOBEHUB_DOMAIN:?LOBEHUB_DOMAIN is required}
  - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN:?required for DNS-01 TLS}

OAuth login fails: Social sign in failed

Casdoor init_data.json contains hardcoded redirect URIs. --createDatabase=true only applies init_data on first-ever DB creation — not on restarts. Fix via SQL in provisioner:

# Replace production domain with staging in existing Casdoor DB
$PSQL -c "UPDATE application SET redirect_uris = REPLACE(redirect_uris,
  'example.com', 'staging.example.com')
  WHERE name='lobechat'
  AND redirect_uris LIKE '%example.com%'
  AND redirect_uris NOT LIKE '%staging.example.com%';"

Also check AUTH_CASDOOR_ISSUER — it must match the Casdoor subdomain (auth.staging.example.com), not the app root domain.

Multi-environment isolation

Before creating a second environment, grep .tf files for hardcoded names. See references/multi-env-isolation.md for the complete matrix.

Will fail on apply (globally unique):

DNS duplication trap: Two environments creating A records for the same name in the same Cloudflare zone → two independent record IDs → DNS round-robin → ~50% traffic to wrong instance. Fix: use subdomain isolation (staging.example.com) or separate zones. Remember to create DNS records for ALL subdomains Caddy serves (e.g., auth.staging, minio.staging).

Snapshot cross-contamination: Unfiltered data "alicloud_ecs_snapshots" returns ALL account snapshots. New env inherits old 100GB snapshot, fails creating 40GB disk. Gate with variable:

locals {
  latest_snapshot_id = var.enable_snapshot_recovery && length(local.available_snapshots) > 0
    ? local.available_snapshots[0].snapshot_id : null
}

Do NOT add count to the data source — changes its state address, causes drift.

Pre-deploy validation

Run a validation script before terraform apply to catch configuration errors locally. This eliminates the deploy→discover→fix→redeploy cycle.

Key checks (see references/pre-deploy-validation.md): 1. terraform validate — syntax 2. No hardcoded domains in Caddyfiles or compose files 3. Required env vars present (LOBEHUB_DOMAIN, CLAUDE4DEV_DOMAIN, CLOUDFLARE_API_TOKEN, APP_URL, etc.) 4. Cloudflare API Token format (not Global API Key) 5. DNS records exist for all Caddy-served domains 6. Casdoor issuer URL matches auth.* subdomain 7. SSH private key exists

Integrate into Makefile: make pre-deploy ENV=staging before make apply.

Zero-to-deployment

Fresh disks expose every implicit dependency. See references/zero-to-deploy-checklist.md.

Key items that break provisioners on fresh instances: 1. Directories: mkdir -p /data/{svc1,svc2} in cloud-init — file provisioner fails if target dir missing 2. Databases: Explicit CREATE DATABASE — PG init scripts only run on empty data dir 3. Migrations: Tracked in schema_migrations table, applied idempotently 4. Provisioner ordering: depends_on between resources sharing Docker networks 5. Memory: Stop non-critical containers during Docker build on small instances (≤8GB) 6. Domain parameterization: Every domain in Caddyfile/compose must be {$VAR} / ${VAR:?required} 7. Credential format: Caddy needs API Token (cfut_), not Global API Key

Multi-Environment Isolation Checklist

When creating a second Terraform environment (staging, lab, etc.) in the same cloud account alongside production, every item below must be verified. Skip one and you get silent name collisions or cross-contamination.

Terraform state isolation

Two environments MUST use different state paths. Same OSS/S3 bucket is fine — different prefix isolates completely:

# production
backend "oss" {
  bucket = "myproject-terraform-state"
  prefix = "environments/production"
}

# staging
backend "oss" {
  bucket = "myproject-terraform-state"  # same bucket OK
  prefix = "environments/staging"       # different prefix = isolated state
}

Verification: terraform state list in one environment must show ZERO resources from the other.

Resource naming collision matrix

Grep every .tf file for hardcoded names. Every globally-unique resource will collide.

Must rename (apply will fail)

Should rename (won't fail but causes confusion)

Pattern: Use a module name variable

# production main.tf
module "app" {
  source = "../../modules/spot-with-data-disk"
  name   = "production-spot"  # flows to instance_name, disk_name, snapshot_policy_name
}

# staging main.tf
module "app" {
  source = "../../modules/spot-with-data-disk"
  name   = "staging-spot"     # all child resource names auto-isolated
}

DNS record isolation

The duplication trap

Two Terraform environments creating A records for @ (root) in the same Cloudflare zone:

• Each gets its own Cloudflare record ID (independent)
• Cloudflare now has TWO A records for the same domain
• DNS round-robins between the two IPs
• ~50% of traffic goes to the wrong instance

Correct patterns

Pattern A: Subdomain isolation (recommended for staging/lab):

# Production: root domain records
resource "cloudflare_dns_record" "prod" {
  name = "@"  # example.com
}

# Staging: subdomain records only
resource "cloudflare_dns_record" "staging" {
  name = "staging"  # staging.example.com
}

Pattern B: Separate zones (for fully independent deployments): Each environment gets its own domain/zone. No shared Cloudflare zone IDs.

Pattern C: One environment owns DNS (production): Only production has DNS resources. Other environments access via IP only.

Destroy safety

When one environment is destroyed:

• Its DNS records are deleted (by their specific Cloudflare record IDs)
• Other environments' DNS records are NOT affected
• Verify before destroy: Compare DNS record IDs between environments:

  terraform state show 'cloudflare_dns_record.app["root"]' | grep "^id"

IDs must be different.

Shared resources (safe to share)

These are referenced but NOT managed by the second environment:

Makefile pattern for multi-environment

ENV ?= production
ENV_DIR := environments/$(ENV)

init: ; cd $(ENV_DIR) && terraform init
plan: ; cd $(ENV_DIR) && terraform plan -out=tfplan
apply: ; cd $(ENV_DIR) && terraform apply tfplan
drift: ; cd $(ENV_DIR) && terraform plan -detailed-exitcode

Usage: make plan ENV=staging

Pre-Deploy Validation Pattern

Run before terraform apply to catch configuration errors locally. Eliminates the deploy→discover→fix→redeploy cycle that wastes hours.

Why this matters

Every hardcoded value becomes a bug when creating a second environment. Production accumulates implicit state over time (cached TLS certs, manually created databases, hand-edited configs). Fresh instances expose all of these as failures. A pre-deploy script catches them before they reach the remote.

Validation categories

1. Terraform syntax

terraform validate

2. Hardcoded domains

# Caddyfiles: should use {$VAR} not literal domains
grep -v "^#" gateway/conf.d/*.caddy | grep -c "example\.com" # should be 0

# Compose: should use ${VAR:?required} not literal domains
grep -v "^#" docker-compose.production.yml | grep -c "example\.com" # should be 0

3. Required env vars

Check that every ${VAR:?required} in compose has a matching entry in .env:

for VAR in LOBEHUB_DOMAIN CLAUDE4DEV_DOMAIN CLOUDFLARE_API_TOKEN APP_URL AUTH_URL; do
  grep -q "^$VAR=" .env || echo "FAIL: $VAR missing"
done

4. Cloudflare credential format

Caddy's Cloudflare plugin uses Bearer auth. Global API Keys (37 hex chars) fail with Invalid format for Authorization header.

TOKEN=$(grep CLOUDFLARE_API_TOKEN .env | cut -d= -f2)
echo "$TOKEN" | grep -qE "^cfut_|^[A-Za-z0-9_-]{40,}$" || echo "FAIL: looks like Global API Key, not API Token"

5. DNS ↔ Caddy consistency

Every domain Caddy serves needs a DNS record. Check live resolution:

for DOMAIN in staging.example.com auth.staging.example.com; do
  curl -sf "https://dns.google/resolve?name=$DOMAIN&type=A" | python3 -c \
    "import sys,json; d=json.load(sys.stdin); exit(0 if d.get('Answer') else 1)" \
    || echo "FAIL: $DOMAIN not resolving"
done

6. Casdoor issuer consistency

AUTH_CASDOOR_ISSUER must point to auth.<domain>, not the app's root domain:

ISSUER=$(grep AUTH_CASDOOR_ISSUER .env | cut -d= -f2)
DOMAIN=$(grep LOBEHUB_DOMAIN .env | cut -d= -f2)
[ "$ISSUER" = "https://auth.$DOMAIN" ] || echo "FAIL: issuer should be https://auth.$DOMAIN"

7. SSH key exists

[ -f ~/.ssh/id_ed25519 ] || echo "FAIL: SSH key not found"

Makefile integration

pre-deploy:
	@./scripts/validate-env.sh $(ENV)

# Enforce: plan requires pre-deploy to pass
plan: pre-deploy
	cd $(ENV_DIR) && terraform plan -out=tfplan

Anti-pattern: deploy-and-pray

The opposite of pre-deploy validation is the "deploy and see what breaks" cycle: 1. terraform apply → fails 2. SSH in to debug → discover error 3. Fix locally → commit → re-apply → fails differently 4. Repeat 5-10 times

Each cycle takes 3-5 minutes (plan + apply + provisioner). Pre-deploy catches 80% of issues in <5 seconds locally.

Zero-to-Deployment Checklist

A fresh instance with an empty data disk exposes every implicit dependency that production silently relies on. This checklist covers everything that must be explicitly created before services will start.

Pre-flight: cloud-init must handle

These run at OS boot, before Terraform provisioners:

• [ ] Mount data disk: Format if new (blkid check), mount to /data, add to fstab
• [ ] Create service directories: mkdir -p /data/{service1,service2,...} — file provisioners fail if target dir doesn't exist
• [ ] Install Docker + Compose: Curl installer, enable systemd service
• [ ] Configure swap: fallocate on data disk (NOT system disk)
• [ ] SSH hardening: key-only auth, no password root login
• [ ] Firewall: UFW + DOCKER-USER iptables chain
• [ ] Debconf preseed: For any package with interactive prompts (iptables-persistent, etc.)
• [ ] Signal readiness: Write timestamp to /data/cloud-init.log

Provisioner ordering

Terraform provisioners execute in declaration order within a resource, but resources execute in parallel unless depends_on is set.

lobehub_deploy ──────────────────→ channel_sync (depends_on lobehub)
                                 → casdoor_sync (depends_on lobehub)
                                 → minio_sync (depends_on lobehub)

claude4dev_deploy (depends_on lobehub_deploy)
  ├─ wait for cloud-init
  ├─ upload source (tarball via file provisioner)
  ├─ upload .env (staging variant)
  ├─ start stateful (postgres, redis) --no-recreate
  ├─ run DB migrations
  ├─ build stateless images
  ├─ fix volume permissions
  ├─ start stateless (relay, api, frontend, gateway)
  └─ verify health

Database bootstrap

PostgreSQL databases

PostgreSQL docker-entrypoint-initdb.d scripts only run when the data directory is empty (first-ever start). On subsequent starts — even if a database doesn't exist — init scripts are skipped.

Fix: Explicitly create databases in provisioner:

# Wait for postgres healthy
sleep 10
# Create database if missing (idempotent)
docker exec my-postgres psql -U postgres -tc \
  "SELECT 1 FROM pg_database WHERE datname='mydb'" | grep -q 1 \
  || docker exec my-postgres psql -U postgres -c "CREATE DATABASE mydb;"

Schema migrations

Migrations must be idempotent. Track applied versions:

PSQL='docker compose exec -T postgres psql -v ON_ERROR_STOP=1 -U myuser -d mydb'

# Create tracking table
$PSQL -tAc "CREATE TABLE IF NOT EXISTS schema_migrations (
  version TEXT PRIMARY KEY,
  applied_at TIMESTAMPTZ DEFAULT now()
)"

# Apply each migration file in order
for f in migrations/*.sql; do
  VER=$(basename $f)
  APPLIED=$($PSQL -tAc "SELECT 1 FROM schema_migrations WHERE version='$VER'" | tr -d ' ')
  if [ "$APPLIED" = "1" ]; then
    echo "Skip: $VER"
  else
    echo "Apply: $VER"
    { echo 'BEGIN;'; cat $f; echo 'COMMIT;'; } | $PSQL
    $PSQL -tAc "INSERT INTO schema_migrations(version) VALUES ('$VER') ON CONFLICT DO NOTHING"
  fi
done

Docker build on remote

Proxy mode

Docker Compose reads build args from .env via ${VAR:-default}. Command-line env vars do NOT override .env values for compose interpolation.

# WRONG: compose still reads DOCKER_WITH_PROXY_MODE from .env
DOCKER_WITH_PROXY_MODE=disabled docker compose build myapp

# RIGHT: modify .env so compose reads the correct value
grep -q DOCKER_WITH_PROXY_MODE .env || echo 'DOCKER_WITH_PROXY_MODE=disabled' >> .env
docker compose build myapp

Memory management

Building Docker images while 10+ containers run can OOM on small instances (8GB). Strategy:

# Stop non-critical containers to free RAM
cd /data/other-project && docker compose stop search-engine analytics-db || true

# Build (memory-intensive)
cd /data/myproject && docker compose build myapp

# Restart stopped containers
cd /data/other-project && docker compose up -d search-engine analytics-db || true

Volume permissions

Containers running as non-root need writable volume directories:

# Before docker compose up:
mkdir -p data-dir logs-dir
chown -R 1001:1001 data-dir logs-dir  # match container UID

Find the UID from the Dockerfile:

RUN adduser -S myuser -u 1001 -G mygroup
USER myuser  # runs as uid 1001

Environment-specific .env files

Production .env contains production URLs. Staging needs its own .env with:

Pattern: Create .env.staging alongside .env. In Terraform:

locals {
  env_src = "${local.repo}/.env.staging"  # staging-specific
}

provisioner "file" {
  source      = local.env_src
  destination = "${local.deploy_dir}/.env"
}

Rsync must exclude .env files (otherwise production .env overwrites staging .env):

--exclude=.env --exclude='.env.*'

Verification template

After all services start, verify in the provisioner (not ad-hoc SSH):

sleep 20
echo '=== Service logs ==='
docker logs my-critical-service --tail 20 2>&1 || true
echo '=== All containers ==='
docker ps --format 'table {{.Names}}\t{{.Status}}' 2>&1 || true
# Final gate (only line that can fail)
docker ps --filter name=my-critical-service --format '{{.Status}}' | grep -q healthy \
  || { echo 'FATAL: service unhealthy'; exit 1; }