K8s

k8s Cluster Operations — joelclaw on Talos

Architecture

Mac Mini (localhost ports)
  └─ Colima/Lima port forwarding (grpc for host-published service ports; avoid separate persistent autossh tunnels)
      └─ Colima VM (8 CPU, 24 GiB, 100 GiB, VZ framework, aarch64)
          └─ Docker 29.x + buildx (joelclaw-builder, docker-container driver)
              └─ Talos v1.12.4 container (joelclaw-controlplane-1, 18 GiB cap)
                  └─ k8s v1.35.0 (single node, Flannel CNI)
                      └─ joelclaw namespace (privileged PSA)

⚠️ Talos has NO shell. No bash, no /bin/sh, nothing. You cannot docker exec into the Talos container. Use talosctl for node operations and the Colima VM (ssh lima-colima) for host-level operations like modprobe.

Colima Stability Rules (2026-03-17 incident)

`nestedVirtualization: true` is unstable on M4 Pro under load. It causes the Colima VM to silently crash during Docker builds/pushes. Each crash:

• Kills the Talos container mid-operation
• Corrupts Redis AOF (if caught mid-write) → crash-loop on restart
• Breaks Lima socket forwarding → docker CLI on macOS disconnects
• Creates stale k8s pods that re-pull images → amplifies pressure

Recovery from Colima crash-loop: 1. colima stop && colima start — basic restart 2. If Redis crash-loops: redis-check-aof --fix (see Redis AOF Recovery below) 3. If Restate has stuck invocations: purge PVC or kill via admin API 4. If native Docker socket dead: use SSH tunnel ssh -L /tmp/docker.sock:/var/run/docker.sock

Docker image builds should use the buildx container builder (docker buildx build --builder joelclaw-builder) to isolate build IO from k8s workloads.

Redis AOF Recovery

If Redis crash-loops after a VM restart with Bad file format reading the append only file:

# 1. Scale down Redis (or use a temp pod if StatefulSet can't mount PVC concurrently)
kubectl -n joelclaw apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: redis-fix
  namespace: joelclaw
spec:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  containers:
    - name: fix
      image: redis:7-alpine
      command: ["sh", "-c", "cd /data/appendonlydir && echo y | redis-check-aof --fix *.incr.aof && redis-check-aof *.incr.aof"]
      volumeMounts:
        - name: data
          mountPath: /data
  restartPolicy: Never
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-redis-0
EOF
# 2. Wait, check logs, then clean up
kubectl -n joelclaw logs redis-fix
kubectl -n joelclaw delete pod redis-fix --force
# 3. Restart Redis
kubectl -n joelclaw delete pod redis-0

For port mappings, recovery procedures, and cluster recreation steps, read references/operations.md.

Reboot-heal persistence rule (2026-04-15 incident)

infra/k8s-reboot-heal.sh runs under launchd as a fresh process every interval. Any recovery marker that only lives in shell memory dies at the end of that tick.

That means flannel/event-healing state must be persisted on disk. Canonical path:

~/.local/state/k8s-reboot-heal.env

Persist at least:

• COLIMA_START_EPOCH
• RECOVERY_START_EPOCH
• LAST_FLANNEL_RESTART_EPOCH
• COLIMA_UNHEALTHY_STREAK
• LAST_COLIMA_UNHEALTHY_EPOCH
• LAST_COLIMA_FORCE_CYCLE_EPOCH
• LAST_COLIMA_FAILED_RECOVERY_EPOCH

Why this matters: kubelet FailedCreatePodSandBox events mentioning missing subnet.env can stay recent for minutes after the first repair. If the healer forgets that it already restarted flannel, the next launchd tick can bounce flannel again and knock healthy services like Typesense back into 503 warmup for no good reason. The extra failed-recovery marker also stops the system from counting a one-minute green flash as success and then force-cycling Colima again when the control path collapses.

Kubeconfig / Talos Endpoint Contract (2026-05-30 incident)

Current operator access uses Colima/Lima's directly published TCP ports:

• Kubernetes API: https://127.0.0.1:6443
• Talos API: 127.0.0.1:50000

Do not rewrite kubeconfig/Talos back to the older manual tunnel ports (16443 / 15000) unless explicitly testing KUBE_OPERATOR_MODE=ssh. After the 2026-05-30 reboot, grpc publishing kept service ports open but could leave Colima SSH and the Docker socket dead after load; a live ssh port-forwarder test did the opposite and dropped the service ports. The current availability-first default remains Colima portForwarder: grpc, but completion audits must verify both service ports and Docker/limactl instead of trusting either mode by vibes.

Fix/verify:

talosctl config endpoint 127.0.0.1:50000
talosctl config node 10.5.0.2
kubectl config set-cluster joelclaw --server=https://127.0.0.1:6443 --insecure-skip-tls-verify=true
kubectl config use-context admin@joelclaw
kubectl get nodes --request-timeout=10s
talosctl -e 127.0.0.1:50000 -n 10.5.0.2 version --client=false

infra/kube-operator-access.sh now defaults to KUBE_OPERATOR_MODE=direct and runs as a boring launchd monitor that keeps configs pointed at 6443 / 50000. It still has an ssh mode for deliberate fallback testing.

Durable recovery rule (ADR-0244)

A Colima restart is not recovery.

After any colima start / force-cycle, the system only counts recovery as real if a post-restart stability window stays healthy across repeated passes for:

• Colima SSH
• Docker socket
• Kubernetes API
• Typesense localhost health
• Inngest localhost health

If those regress during the verification window, classify the event as a failed recovery, capture proof artifacts, and stop repeated force-cycles for the configured hold period. The point is durability, not healer theatre.

Quick Health Check

kubectl get pods -n joelclaw                          # all pods
curl -s localhost:3111/api/inngest                     # system-bus-worker → 200
curl -s localhost:7880/                                # LiveKit → "OK"
curl -s localhost:8108/health                          # Typesense → {"ok":true}
curl -s localhost:8288/health                          # Inngest → {"status":200}
curl -s localhost:9070/deployments                     # Restate admin → deployments list
curl -s localhost:9627/xrpc/_health                    # PDS → {"version":"..."}
kubectl exec -n joelclaw redis-0 -- redis-cli ping     # → PONG
joelclaw restate cron status                           # Dkron scheduler → healthy via temporary CLI tunnel

Services

Restate / Firecracker runtime notes

• deployment/restate-worker is intentionally privileged and mounts /dev/kvm (hostPath type "" — optional).
• PVC firecracker-images at /tmp/firecracker-test stores kernel, rootfs, and snapshot artifacts.
• When nestedVirtualization is OFF: /dev/kvm absent, microvm DAG handler fails, but shell/infer/noop handlers work normally.
• When nestedVirtualization is ON: Firecracker one-shot exec works (create workspace ext4 → write command → boot VM → guest executes → poweroff → read results).
• Restate retry caps: dagWorker maxAttempts=5, dagOrchestrator maxAttempts=3. Prevents journal poisoning.
• Restate journal purge (if stuck invocations block work): scale down Restate, mount PVC with temp pod, rm -rf /restate-data/*, scale back up, re-register worker.
• Re-register worker: curl -X POST http://localhost:9070/deployments -H 'content-type: application/json' -d '{"uri":"http://restate-worker:9080"}'

⚠️ PDS port trap: Docker maps 9627→3000 (host→container). NodePort must be 3000 to match the container-side port. If set to 9627, traffic won't route.

Rule: NodePort value = Docker's container-side port, not host-side.

Agent Runner (Cold k8s Jobs)

Status: local sandbox remains the default/live path; the k8s backend is now code-landed and opt-in, but still needs supervised rollout before calling it earned runtime.

The agent runner executes sandboxed story runs as isolated k8s Jobs. Jobs are created dynamically via @joelclaw/agent-execution/job-spec — no static manifests.

Runtime Image Contract

See k8s/agent-runner.yaml for the full specification.

Required components:

• Git (checkout, diff, commit)
• Bun runtime
• runner-installed agent tooling (currently claude and/or other installed CLIs)
• /workspace working directory
• runtime entrypoint at /app/packages/agent-execution/src/job-runner.ts

Configuration via environment variables:

• Request metadata: WORKFLOW_ID, REQUEST_ID, STORY_ID, SANDBOX_PROFILE, BASE_SHA, EXECUTION_BACKEND, JOB_NAME, JOB_NAMESPACE
• Repo materialization: REPO_URL, REPO_BRANCH, optional HOST_REQUESTED_CWD
• Agent identity: AGENT_NAME, AGENT_MODEL, AGENT_VARIANT, AGENT_PROGRAM
• Execution config: SESSION_ID, TIMEOUT_SECONDS
• Task prompt: TASK_PROMPT_B64 (base64-encoded)
• Verification: VERIFICATION_COMMANDS_B64 (base64-encoded JSON array)
• Callback path: RESULT_CALLBACK_URL, RESULT_CALLBACK_TOKEN

Expected behavior: 1. Decode task from TASK_PROMPT_B64 2. Materialize repo from REPO_URL / REPO_BRANCH at BASE_SHA 3. Execute the requested AGENT_PROGRAM 4. Run verification commands (if set) 5. Print SandboxExecutionResult markers to stdout and POST the same result to /internal/agent-result 6. Exit 0 (success) or non-zero (failure)

Current truthful limit:

• pi remains local-backend only for now; do not pretend the pod runner can execute pi story runs yet.

Job Lifecycle

import { generateJobSpec, generateJobDeletion } from "@joelclaw/agent-execution";

// 1. Generate Job spec
const spec = generateJobSpec(request, {
  runtime: {
    image: "ghcr.io/joelhooks/agent-runner:latest",
    imagePullPolicy: "Always",
    command: ["bun", "run", "/app/packages/agent-execution/src/job-runner.ts"],
  },
  namespace: "joelclaw",
  imagePullSecret: "ghcr-pull",
  resultCallbackUrl: "http://host.docker.internal:3111/internal/agent-result",
  resultCallbackToken: process.env.OTEL_EMIT_TOKEN,
});

// 2. Apply to cluster (via kubectl or k8s client library)
// 3. Job runs → Pod materializes repo, executes agent, posts SandboxExecutionResult callback
// 4. Host worker can recover the same terminal result from log markers if callback delivery fails
// 5. Job auto-deletes after TTL (default: 5 minutes)

// Cancel a running Job
const deletion = generateJobDeletion("req-xyz");
// kubectl delete job ${deletion.name} -n ${deletion.namespace}

Resource Defaults

• CPU: 500m request, 2 limit
• Memory: 1Gi request, 4Gi limit
• Active deadline: 1 hour
• TTL after completion: 5 minutes
• Backoff limit: 0 (no retries)

Security

• Non-root execution (UID 1000, GID 1000)
• No privilege escalation
• All capabilities dropped
• RuntimeDefault seccomp profile
• Control plane toleration for single-node cluster

Verification Commands

# List agent runner Jobs
kubectl get jobs -n joelclaw -l app.kubernetes.io/name=agent-runner

# Check Job status
kubectl describe job <job-name> -n joelclaw

# View logs
kubectl logs job/<job-name> -n joelclaw

# Check for stale Jobs (should be auto-deleted by TTL)
kubectl get jobs -n joelclaw --show-all

Current State

• ✅ Job spec generator (packages/agent-execution/src/job-spec.ts)
• ✅ Runtime contract (k8s/agent-runner.yaml)
• ✅ Tests (packages/agent-execution/__tests__/job-spec.test.ts)
• ⏳ Runtime image not yet built (Story 3)
• ⏳ Hot-image CronJob not yet implemented (Story 4)
• ⏳ Warm-pool scheduler not yet implemented (Story 5)
• ⏳ Restate integration not yet wired (Story 6)

NAS NFS Access from k8s (ADR-0088 Phase 2.5)

k8s pods can mount NAS storage over NFS via a LAN route through the Colima bridge.

How it works

k8s pod → Talos container (10.5.0.x) → Docker NAT → Colima VM
  → ip route 192.168.1.0/24 via 192.168.64.1 dev col0
  → macOS host (IP forwarding enabled) → LAN → NAS (192.168.1.163)

Root cause of prior failures: VZ framework's shared networking on eth0 doesn't properly forward LAN-bound traffic. The fix routes LAN traffic through col0 (Colima bridge → macOS host) instead.

Route persistence

The LAN route is set in two places for reliability: 1. Colima provision script (~/.colima/default/colima.yaml) — runs on colima start (cold boot) 2. k8s-reboot-heal (~/Code/joelhooks/joelclaw/infra/k8s-reboot-heal.sh) — reasserts the route during reboot recovery ticks

Both execute: ip route replace 192.168.1.0/24 via 192.168.64.1 dev col0

Duplicate tunnel ownership is a bug (2026-04-16)

com.joel.colima-tunnel is deprecated. Colima/Lima already forwards the docker-published host ports for joelclaw-controlplane-1, so a second autossh daemon on those same ports is not redundancy — it's interference.

Rules:

• com.joel.colima is the only boot/start helper for the VM; it must not keep a periodic StartInterval
• com.joel.colima-tunnel should be absent from /Library/LaunchDaemons/; install-critical-launchdaemons.sh removes it instead of reinstalling it
• do not run a second autossh daemon on ports Colima/Lima already publishes for joelclaw-controlplane-1 (3838, 6379, 7880, 7881, 8108, 8288, 8289, 9627, 64784)
• do not kill generic ssh listeners on those host ports; that can kill Lima's own forwarders
• infra/colima-tunnel.sh is now only a deprecated compatibility stub so stale launchd installs exit cleanly instead of fighting Lima
• com.joel.kube-operator-access is now a direct-mode monitor, not a persistent tunnel. It keeps kubectl/talosctl aimed at Colima/Lima's published loopback ports: 6443 for kube-apiserver and 50000 for Talos
• the old SSH tunnel mode (16443 -> 10.5.0.2:6443, 15000 -> 10.5.0.2:50000) is fallback-only via KUBE_OPERATOR_MODE=ssh; do not leave it crash-looping under launchd
• once the daemon is installed, kubectl should use https://127.0.0.1:6443 and talosctl should use 127.0.0.1:50000
• com.joel.k8s-reboot-heal must use the same JSON status check; a plain colima status false-negative can force-cycle the VM and retrigger the flannel/NAS failure cascade during reboot recovery
• do not trust status output alone when deciding to cycle Colima; if the Docker socket or Colima SSH path is still healthy, treat the VM as alive and keep your hands off it
• a Colima force-cycle now requires confirmed evidence; one ugly observation is not enough to panic-cycle the VM
• confirmation can come from consecutive launchd ticks or from a short rapid-confirmation window when both the Docker socket and Colima SSH path stay down long enough to prove a severe collapse
• after any Colima force-cycle, honor the persisted cooldown in ~/.local/state/k8s-reboot-heal.env so Talos and workload warmup can finish before another escalation is even considered
• if the host path is still down but escalation is not yet earned, bail out early and mark the tick failed; do not pretend downstream kube/NAS repair steps are actionable without Colima host access
• reboot recovery is not healthy until the NAS route 192.168.1.0/24 via 192.168.64.1 dev col0 exists again and NFS is reachable from the Colima VM
• flannel can be "Running" while kubelet still reports failed to load flannel 'subnet.env' file; treat recent FailedCreatePodSandBox events with that message as a restart signal for the flannel pod

Available PVs

Mounting NAS in a pod

volumes:
  - name: nas
    persistentVolumeClaim:
      claimName: nas-nvme
containers:
  - volumeMounts:
      - name: nas
        mountPath: /nas
        # Optional: subPath for specific dir
        subPath: typesense

Rules

• Always use IP (192.168.1.163), never hostname (three-body). DNS doesn't resolve from inside k8s.
• Always use `nfsvers=3,tcp,resvport,noatime` mount options. NFSv4 has issues with Asustor ADM.
• NAS unavailability degrades gracefully with soft mount option — returns errors, doesn't hang pods.
• NFS write performance: ~660 MiB/s over 10GbE with jumbo frames. Good for sequential I/O (backups, snapshots). Latency-sensitive workloads (Redis, active Typesense indexes) stay on local SSD.
• If NFS mount fails after Colima restart: verify the route exists: colima ssh -- ip route | grep 192.168.1.0

Verify connectivity

# From Colima VM
colima ssh -- timeout 2 bash -c "echo > /dev/tcp/192.168.1.163/2049" && echo "NFS OK"

# From k8s pod
kubectl run nfs-test --image=busybox --restart=Never -n joelclaw \
  --overrides='{"spec":{"tolerations":[{"key":"node-role.kubernetes.io/control-plane","operator":"Exists","effect":"NoSchedule"}],"containers":[{"name":"t","image":"busybox","command":["sh","-c","ls /nas && echo OK"],"volumeMounts":[{"name":"n","mountPath":"/nas"}]}],"volumes":[{"name":"n","persistentVolumeClaim":{"claimName":"nas-nvme"}}]}}'
kubectl logs nfs-test -n joelclaw && kubectl delete pod nfs-test -n joelclaw --force

Deploy Commands

# Manifests (redis, typesense, inngest, dkron)
kubectl apply -f ~/Code/joelhooks/joelclaw/k8s/

# Restate runtime
kubectl apply -f ~/Code/joelhooks/joelclaw/k8s/restate.yaml
kubectl apply -f ~/Code/joelhooks/joelclaw/k8s/firecracker-pvc.yaml
kubectl rollout status statefulset/restate -n joelclaw
~/Code/joelhooks/joelclaw/k8s/publish-restate-worker.sh
curl -fsS http://localhost:9070/deployments

# Dkron phase-1 scheduler (ClusterIP API + CLI-managed short-lived tunnel access)
kubectl apply -f ~/Code/joelhooks/joelclaw/k8s/dkron.yaml
kubectl rollout status statefulset/dkron -n joelclaw
joelclaw restate cron status
joelclaw restate cron sync-tier1        # seed/update ADR-0216 tier-1 jobs

# system-bus worker (build + push GHCR + apply + rollout wait)
~/Code/joelhooks/joelclaw/k8s/publish-system-bus-worker.sh

# LiveKit (Helm + reconcile patches)
~/Code/joelhooks/joelclaw/k8s/reconcile-livekit.sh joelclaw

# AIStor (Helm operator + objectstore)
# Defaults to isolated `aistor` namespace to avoid service-name collisions with legacy `joelclaw/minio`.
# Cutover override (explicit only): AISTOR_OBJECTSTORE_NAMESPACE=joelclaw AISTOR_ALLOW_JOELCLAW_NAMESPACE=true
~/Code/joelhooks/joelclaw/k8s/reconcile-aistor.sh

# PDS (Helm) — always patch NodePort to 3000
# (export current values first if the release already exists)
helm get values bluesky-pds -n joelclaw > /tmp/pds-values-live.yaml 2>/dev/null || true
helm upgrade --install bluesky-pds nerkho/bluesky-pds \
  -n joelclaw -f /tmp/pds-values-live.yaml
kubectl patch svc bluesky-pds -n joelclaw --type='json' \
  -p='[{"op":"replace","path":"/spec/ports/0/nodePort","value":3000}]'

Auto Deploy (GitHub Actions)

• Workflow: .github/workflows/system-bus-worker-deploy.yml
• Trigger: push to main touching packages/system-bus/** or worker deploy files
• Behavior:
• builds/pushes ghcr.io/joelhooks/system-bus-worker:${GITHUB_SHA} + :latest
• runs deploy job on self-hosted runner
• updates k8s deployment image + waits for rollout + probes worker health
• If deploy job is queued forever, check that a self-hosted runner is online on the Mac Mini.

GHCR push 403 Forbidden

Cause: GITHUB_TOKEN (default Actions token) does not have packages:write scope for this repo. A dedicated PAT is required.

Fix already applied: Workflow uses secrets.GHCR_PAT (not secrets.GITHUB_TOKEN) for the GHCR login step. The PAT is stored in:

• GitHub repo secrets as GHCR_PAT (set via GitHub UI)
• agent-secrets as ghcr_pat (secrets lease ghcr_pat)

If this breaks again: PAT may have expired. Regenerate at github.com → Settings → Developer settings → PATs, update both stores.

Local fallback (bypass GHA entirely):

DOCKER_CONFIG_DIR=$(mktemp -d)
echo '{"credsStore":""}' > "$DOCKER_CONFIG_DIR/config.json"
export DOCKER_CONFIG="$DOCKER_CONFIG_DIR"
secrets lease ghcr_pat | docker login ghcr.io -u joelhooks --password-stdin
~/Code/joelhooks/joelclaw/k8s/publish-system-bus-worker.sh

Note: publish-system-bus-worker.sh uses gh auth token internally — if gh auth is stale, use the Docker login above before running the script, or patch it to use secrets lease ghcr_pat directly.

Resilience Rules (ADR-0148)

1. NEVER use `kubectl port-forward` for persistent service exposure. All long-lived operator surfaces MUST use NodePort + Docker port mappings. The narrow exception is a CLI-managed, short-lived tunnel for an otherwise in-cluster-only control surface (for example joelclaw restate cron * tunneling to dkron-svc). Port-forwards silently die on idle/restart/pod changes, so do not leave them running. 2. All workloads MUST have liveness + readiness + startup probes. Missing probes = silent hangs that never recover. 3. After any Docker/Colima/node restart: remove control-plane taint, uncordon node, verify flannel, check all pods reach Running. 4. PVC reclaimPolicy is Delete — deleting a PVC = permanent data loss. Never delete PVCs without backup. 5. `firecracker-images` is stateful runtime data. Treat it like a real runtime PVC: kernel, rootfs, and snapshot loss will break the microVM path. 6. Colima VM disk is limited (19GB). Monitor with colima ssh -- df -h /. Alert at >80%. 7. All launchd plists MUST set PATH including `/opt/homebrew/bin`. Colima shells to limactl, kubectl/talosctl live in homebrew. launchd's default PATH is /usr/bin:/bin:/usr/sbin:/sbin — no homebrew. The canonical PATH for infra plists is: /opt/homebrew/bin:/Users/joel/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin. Discovered Feb 2026: missing PATH caused 6 days of silent recovery failures. 8. Shell scripts run by launchd MUST export PATH at the top. Even if the plist sets EnvironmentVariables, belt-and-suspenders — add export PATH="/opt/homebrew/bin:..." to the script itself.

Current Probe Gaps (fix when touching these services)

• Typesense: missing liveness probe (hangs won't be detected)
• Bluesky PDS: missing readiness and startup probes
• system-bus-worker: missing startup probe

Danger Zones

1. Stale SSH mux socket after Colima restart — When Colima restarts (disk resize, crash recovery, colima stop && start), the SSH port changes but the mux socket (~/.colima/_lima/colima/ssh.sock) caches the old connection. Symptoms: kubectl port-forward fails with "tls: internal error", kubectl get nodes may intermittently work then fail. Fix: rm -f ~/.colima/_lima/colima/ssh.sock && pkill -f "ssh.*colima", then re-establish tunnels with ssh -o ControlPath=none. Always verify SSH port with colima ssh-config | grep Port after restart. 2. Adding Docker port mappings — can be hot-added without cluster recreation via hostconfig.json edit. See references/operations.md for the procedure. 3. Inngest legacy host alias in manifests — old container-host alias may still appear in legacy configs. Worker uses connect mode, so it usually still works, but prefer explicit Talos/Colima hostnames. 4. Colima zombie state — colima status reports "Running" but docker socket / SSH tunnels are dead. All k8s ports unresponsive. colima start is a no-op. Only colima restart recovers. Detect with: ssh -F ~/.colima/_lima/colima/ssh.config lima-colima "docker info" — if that fails while colima status passes, it's a zombie. The heal script handles this automatically. 5. Talos container has NO shell — No bash, no /bin/sh. Cannot docker exec into it. Kernel modules like br_netfilter must be loaded at the Colima VM level: ssh lima-colima "sudo modprobe br_netfilter". 6. AIStor service-name collision — if AIStor objectstore is deployed in joelclaw, it can claim svc/minio and break legacy MinIO assumptions. Keep AIStor objectstore in isolated namespace (aistor) unless intentionally cutting over. 7. AIStor operator webhook SSA conflict — repeated helm upgrade can fail on MutatingWebhookConfiguration caBundle ownership conflict. Current mitigation in this cluster: set operators.object-store.webhook.enabled=false in k8s/aistor-operator-values.yaml. 8. MinIO pinned tag trap — minio/minio:RELEASE.2025-10-15T17-29-55Z is not available on Docker Hub in this environment (ErrImagePull). Legacy fallback currently relies on minio/minio:latest. 9. `restate-worker` privilege is intentional. Do not “harden” away /dev/kvm, privileged: true, or the unconfined seccomp profile unless you are simultaneously changing the Firecracker runtime contract. 10. Dkron service-name collision — never create a bare svc/dkron. Kubernetes injects DKRON_* env vars into pods, which collides with Dkron's own config parsing. Use dkron-peer and dkron-svc. 11. Dkron PVC permissions — upstream dkron/dkron:latest currently needs root on the local-path PVC. Non-root hardening caused permission denied under /data/raft/snapshots/permTest and CrashLoopBackOff. 12. Typesense host access must be a real service contract — after the 2026-04-19 rebuild, OTEL emit hung because host code still targeted localhost:8108 while typesense had been restored as ClusterIP only. The fix was to make k8s/typesense.yaml a NodePort service on 8108 again so host worker + CLI writes have a stable path without reviving a launchd port-forward sidecar. 13. docs-api restore also needs `docs-api-env` — the manifest depends on secret docs-api-env with key PDF_BRAIN_API_TOKEN. The token lives in agent-secrets as pdf_brain_api_token; recreate the k8s secret before applying k8s/docs-api.yaml on a rebuilt cluster or the Deployment will stay broken. 14. knowledge search can fail right after a rebuild even when Typesense is healthy — if system_knowledge is missing you will see 404 {"message":"Collection not found"}. The CLI now auto-heals this on first joelclaw knowledge search by recreating the collection and re-syncing ADRs + skills, but an explicit joelclaw knowledge sync is still the blunt proof command. 15. PDS rebuilds are a two-step restore, not just a Helm install — recreate bluesky-pds-secrets, reinstall the bluesky-pds Helm release, force the service nodePort back to 3000, then recreate Joel's account if the PVC was wiped. The new account returns a fresh DID, so update the pds_joel_did secret afterward or host dual-write will keep authenticating against a dead repo. 16. PDS session auth is handle-first in practice — on the rebuilt PDS, com.atproto.server.createSession succeeded against joel.pds.panda.tail7af24.ts.net but rejected the raw DID. packages/system-bus/src/lib/pds.ts now resolves the handle from pds_joel_did via describeRepo before it asks for a session, which keeps the dual-write path aligned with reality. 17. Typesense "Too many open files" masquerades as raft ERROR — during large HNSW reindexes (e.g. docs_chunks_v2 ~223k rows), Typesense exhausts its 1024 default FD limit and starts logging Fail to open /proc/self/fd: Too many open files. The pod stays Running but 503s everything, and the external symptom is identical to a raft leader-election failure. Fix: wrap the container command with ulimit -n 1048576 before exec-ing /opt/typesense-server. Canonical form lives in k8s/typesense.yaml since 2026-04-19. 18. Talos container memory cap is separate from the Colima VM size — Docker can leave joelclaw-controlplane-1 at a 4 GiB cap even when the VM has 24 GiB. Every k8s pod inside Talos shares that cap. Under Typesense otel_events / docs_chunks_v2 load the Talos container can peg and make Typesense plus operator access look dead together. Live fix without restart: ssh -F ~/.colima/_lima/colima/ssh.config lima-colima "sudo docker update --restart unless-stopped --memory=18g --memory-swap=18g joelclaw-controlplane-1". infra/k8s-reboot-heal.sh now reasserts this cap on each recovery tick. 19. *`joelclaw.com/api/docs/ 502 can be a host-port exposure failure, not a docs-api pod failure** — the public web route proxies to https://panda.tail7af24.ts.net/api/docs, and Tailscale Funnel serves /api/docs by proxying Panda-local localhost:3838. If kubectl get pod -n joelclaw -l app=docs-api is healthy but lsof -nP -iTCP:3838 -sTCP:LISTEN is empty, Funnel returns bare 502s and Vercel propagates them. Fast proof: curl -i https://panda.tail7af24.ts.net/api/docs/health fails while kubectl port-forward -n joelclaw svc/docs-api 3838:3838 makes it pass. Temporary repair: start that port-forward; durable repair is restoring the Colima/Lima host-published NodePort path for 3838, not declaring the pod healthy and moving on. 20. **Root cause pattern for closed NodePorts after Colima churn** — if most host-published ports are closed and only manual kubectl port-forward listeners exist, inspect ~/.colima/_lima/_networks/user-v2/usernet.user-v2.stderr.log. Lima 2.0.3 can panic in pkg/networks/usernet/gvproxy.go (Failed to get FD via socketinvalid argument, index out of range) and leave host publishing half-dead. The 2026-05-30 recovery also found the inverse failure with portForwarder: grpc: service ports stayed open while Colima SSH and the Docker socket died; switching live to portForwarder: ssh restored Docker briefly but cancelled service port forwarding. The current mitigation is Colima portForwarder: grpc plus direct operator ports (6443 / 50000), with Docker/SSH treated as a separate health gate rather than assumed healthy. After a Colima force-cycle, joelclaw-controlplane-1 may remain Exited (255) unless its Docker restart policy is unless-stopped. Durable recovery sequence: clean stale usernet/SSH state, colima stop --force && colima start, docker start joelclaw-controlplane-1, docker update --restart unless-stopped --memory=18g --memory-swap=18g joelclaw-controlplane-1, ssh lima-colima "sudo modprobe br_netfilter", then restart flannel if pods fail with missing /run/flannel/subnet.env. Verify localhost:3838, 8108, 8288, 9070, 9627, 6443, and 50000 are real Docker/Lima listeners, not ad hoc kubectl port-forward` stand-ins.

Key Files

Troubleshooting

Read references/operations.md for:

• Recovery after Colima restart
• Recovery after Mac reboot
• Flannel br_netfilter crash fix
• Full cluster recreation (nuclear option)
• Caddy/Tailscale HTTPS proxy details
• All port mapping details with explanation

k8s Operations Reference

Port Mapping Details

Traffic path: Mac:port → Lima SSH tunnel → Docker port map → Talos NodePort → Pod

Port Mapping Rule

NodePort must equal the Docker container-side port. Docker maps hostPort:containerPort. The Talos node receives traffic on containerPort, and NodePort listens on the node at that same value.

For symmetric mappings (6379:6379), NodePort=6379 works. For PDS (9627:3000), NodePort must be 3000.

Important exception: 3111 is no longer a Talos/Docker port mapping. The host worker is canonical on localhost:3111; leaving a stale 3111/tcp Docker binding on joelclaw-controlplane-1 will block the host worker and produce Unable to reach SDK URL failures in Inngest.

Inspecting Docker Port Mappings

ssh -F ~/.colima/_lima/colima/ssh.config lima-colima \
  "docker inspect joelclaw-controlplane-1 --format '{{json .HostConfig.PortBindings}}'" \
  | python3 -m json.tool

Adding Ports Without Cluster Recreation

To hot-add Docker port mappings to the Talos container (preserves all PVCs/data):

# 1. Stop the container
docker stop joelclaw-controlplane-1

# 2. Stop Docker in Colima VM
colima ssh -- sudo systemctl stop docker.socket
colima ssh -- sudo systemctl stop docker

# 3. Edit hostconfig.json (add to PortBindings)
CONTAINER_ID=$(docker inspect joelclaw-controlplane-1 --format '{{.Id}}')
CONFIG=/var/lib/docker/containers/$CONTAINER_ID/hostconfig.json
colima ssh -- sudo python3 -c "
import json
with open('$CONFIG') as f: config = json.load(f)
config['PortBindings']['NEW_PORT/tcp'] = [{'HostIp': '0.0.0.0', 'HostPort': 'NEW_PORT'}]
with open('$CONFIG', 'w') as f: json.dump(config, f)
"

# 4. Also update config.v2.json ExposedPorts
CONFIG_V2=/var/lib/docker/containers/$CONTAINER_ID/config.v2.json
colima ssh -- sudo python3 -c "
import json
with open('$CONFIG_V2') as f: config = json.load(f)
config['Config']['ExposedPorts']['NEW_PORT/tcp'] = {}
with open('$CONFIG_V2', 'w') as f: json.dump(config, f)
"

# 5. Restart Docker + container
colima ssh -- sudo systemctl start docker.socket
colima ssh -- sudo systemctl start docker
docker start joelclaw-controlplane-1

# 6. Remove control-plane taint (returns after Docker restart)
kubectl taint nodes joelclaw-controlplane-1 \
  node-role.kubernetes.io/control-plane:NoSchedule- || true

# 7. Convert the k8s service to NodePort
kubectl patch svc SERVICE_NAME -n joelclaw --type='json' -p='[
  {"op": "replace", "path": "/spec/type", "value": "NodePort"},
  {"op": "replace", "path": "/spec/ports/0/nodePort", "value": NEW_PORT}
]'

NEVER use `kubectl port-forward` for persistent services. All services must be NodePort with Docker port mappings.

Recovery Procedures

After Colima Restart

colima status                    # Verify VM running
# Talos container should auto-start (Docker restart policy)
# If not:
ssh -F ~/.colima/_lima/colima/ssh.config lima-colima \
  "docker start joelclaw-controlplane-1"
# Wait 30-60s, then verify:
kubectl get pods -n joelclaw

Colima Zombie State Recovery

Symptoms: colima status says Running, but all k8s ports (8288, 6379, 8108, etc.) refuse connections. kubectl gets connection refused. Docker socket is dead.

Detection (what the heal script does):

# colima status returns 0 (lies)
colima status && echo "claims running"

# But docker inside VM is unreachable (truth)
ssh -F ~/.colima/_lima/colima/ssh.config lima-colima "docker info" 2>/dev/null
# ^ fails = zombie

Fix: colima start is a no-op in zombie state. Must use colima restart:

colima restart

# Then standard post-restart recovery:
kubectl taint nodes joelclaw-controlplane-1 node-role.kubernetes.io/control-plane:NoSchedule- || true
kubectl uncordon joelclaw-controlplane-1 || true

# Load br_netfilter at VM level (NOT inside Talos — Talos has no shell)
ssh -F ~/.colima/_lima/colima/ssh.config lima-colima "sudo modprobe br_netfilter"

# If flannel is CrashLoopBackOff, force-delete the pod
kubectl get pods -n kube-system | grep flannel
kubectl delete pod -n kube-system <flannel-pod> --force --grace-period=0

# Clean zombie pods in joelclaw namespace
kubectl get pods -n joelclaw --field-selector=status.phase=Unknown \
  -o name | xargs -r kubectl delete -n joelclaw --force --grace-period=0

# Taint can reappear — check again after 5s
sleep 5
kubectl taint nodes joelclaw-controlplane-1 node-role.kubernetes.io/control-plane:NoSchedule- || true

Root cause pattern: Colima/Lima forwarding can fail in two different ways. The old manual SSH tunnel ports (16443 / 15000) are obsolete, but the 2026-05-30 reboot incident also showed portForwarder: grpc can keep TCP service ports open while leaving Colima SSH and the Docker socket dead after load; a live switch to portForwarder: ssh restored Docker briefly but cancelled the k8s service forwards. The current availability-first profile uses Colima portForwarder: grpc with direct localhost service ports (6443 / 50000); separate autossh daemons stay deprecated. The heal script (infra/k8s-reboot-heal.sh) treats Docker socket or Colima SSH health as proof the VM is alive, and the operator daemon defaults to direct published ports instead of long-lived manual tunnels.

After Mac Reboot

Colima starts via launchd (com.joel.colima) through infra/colima-start.sh. Treat it as a boot/startup helper, not a periodic babysitter: it should run at load, return success if the profile is already running, otherwise call colima start ..., then exit. If the installed plist still has StartInterval 300, that is stale and should be reinstalled from the repo-managed plist because re-running colima start every five minutes against an already-running VM adds churn and muddies collapse diagnosis. com.joel.colima-tunnel is deprecated and should be absent from /Library/LaunchDaemons/; Colima/Lima already owns the docker-published host ports for joelclaw-controlplane-1, so a second autossh daemon on the same ports just creates duplicate ownership and host-path fights. com.joel.typesense-portforward is also deprecated; Typesense is already exposed through the controlplane container, so a separate kubectl port-forward daemon on 8108 only adds churn. com.joel.kube-operator-access defaults to direct mode and keeps local config pointed at Colima/Lima's published ports: 6443 for kube-apiserver and 50000 for the Talos API. Wait ~60s for full stack: VM → Docker → Talos → k8s → pods. Worker auto-starts via com.joel.system-bus-worker.

Resource invariant first: the stable Colima profile is currently cpu: 8, memory: 24, disk: 100, portForwarder: grpc (see ~/.colima/default/colima.yaml). If the profile drifts down to 4/8, Docker can refuse to restart joelclaw-controlplane-1 with range of CPUs is from 0.01 to 4.00, leaving the whole cluster down after reboot. The repo-managed infra/launchd/com.joel.colima.plist must stay aligned at 8 / 24 / 100 / grpc so boot automation does not reintroduce the drift or the ssh-mode service-port failure. The Talos container cap must also be reasserted to 18g after churn.

Boot-safe critical daemons (ADR-0240): if Panda needs the host control plane to survive headless reboots, install the critical system daemons once:

sudo ~/Code/joelhooks/joelclaw/infra/install-critical-launchdaemons.sh

Compatibility alias if old notes still point at it:

sudo ~/Code/joelhooks/joelclaw/infra/install-headless-bootstrap.sh

This installs the repo-managed plists directly into /Library/LaunchDaemons/, removes stale ~/Library/LaunchAgents copies for the critical labels, deletes the superseded com.joel.headless-bootstrap bridge, kills known manual nohup fallbacks, kills stale manual operator tunnels on 16443 and 15000, and bootstraps these services in the system domain using UserName=joel where needed:

• com.joel.colima
• com.joel.k8s-reboot-heal
• com.joel.kube-operator-access
• com.joel.agent-secrets
• com.joel.system-bus-worker
• com.joel.gateway
• com.joelclaw.agent-mail

com.joel.typesense-portforward is deprecated and should be absent from /Library/LaunchDaemons/; k8s/typesense.yaml now exposes Typesense as NodePort 8108, and Colima/Lima publishes that host port through the controlplane container. A separate kubectl port-forward svc/typesense 8108:8108 daemon only adds churn.

com.joel.kube-operator-access is the canonical kubectl/talos operator plane. In normal mode it does not open a manual tunnel; it monitors direct Colima/Lima published ports and rewrites ~/.kube/config / ~/.talos/config toward https://127.0.0.1:6443 and 127.0.0.1:50000. The old tunnel shape (16443 / 15000) is available only for deliberate fallback testing with KUBE_OPERATOR_MODE=ssh. If launchd shows the service crash-looping with SSH exit 255, stop it from poisoning kubeconfig and return to direct mode.

infra/colima-proof.sh is the evidence-first substrate harness. Use it to capture incident-scoped artifacts under ~/.local/share/colima-proof/incidents/<incident_id>/ and emit OTEL under source=infra, component=colima-proof before destructive recovery erases the failure state. The reboot healer now uses it on failure edges, hold states, force-cycle boundaries, and post-invariant outcomes. For the first non-destructive discriminator, run recover-usernet --restart-mode none --verify-wait-secs 15: it resets Lima user-v2 control state, writes pre/post snapshots, and records a verdict artifact that says whether the intervention actually supported H1-usernet or not.

com.joelclaw.agent-mail is launched via infra/agent-mail-daemon.sh, not by hardcoding the third-party checkout path into the plist. The daemon script expects the joelclaw-managed joelhooks/mcp_agent_mail fork; a legacy on-disk directory name is acceptable only if that checkout's origin remote points at joelhooks/mcp_agent_mail.

Do not try to bootstrap user LaunchAgents into user/$UID from a system daemon anymore. That ADR-0239 bridge was not earned on Panda; launchctl bootstrap user/501 ... kept failing with Input/output error.

⚠️ launchd PATH requirement: The Colima plist MUST include EnvironmentVariables with PATH containing /opt/homebrew/bin. Colima internally shells to limactl which is a Homebrew formula. Without this, launchd recovery silently fails (Feb 2026 incident: 6 days of silent failures). Same applies to k8s-reboot-heal.sh — it exports PATH at the top as belt-and-suspenders.

launchctl print system/com.joel.kube-operator-access | rg 'state =|pid =|last exit code'
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'
talosctl config info | rg 'Endpoints|Nodes|127.0.0.1:50000|10.5.0.2'
kubectl get nodes
curl -k https://127.0.0.1:6443/readyz?verbose
kubectl get pods -n joelclaw
curl localhost:8288/health

If kubectl fails with connection refused after Colima is up, verify Talos container state and start it manually:

ssh -F ~/.colima/_lima/colima/ssh.config lima-colima \
  "docker ps -a --format '{{.Names}}\t{{.Status}}'"
ssh -F ~/.colima/_lima/colima/ssh.config lima-colima \
  "docker start joelclaw-controlplane-1"

Then verify single-node scheduling is enabled (the control-plane taint AND SchedulingDisabled may return after reboot):

kubectl taint nodes joelclaw-controlplane-1 \
  node-role.kubernetes.io/control-plane:NoSchedule- || true
kubectl uncordon joelclaw-controlplane-1 || true

If node shows "shutting down" in conditions after a Talos container restart, the container needs a full docker restart (not just start). The shutdown state is sticky from the previous unclean stop.

Finally, if pods are Unknown, restart flannel and stale pods:

kubectl get pods -n kube-system | grep kube-flannel
kubectl logs -n kube-system <kube-flannel-pod-name> --tail=80
ssh -F ~/.colima/_lima/colima/ssh.config lima-colima \
  "sudo modprobe br_netfilter"
kubectl delete pod -n kube-system <kube-flannel-pod-name>

For stale Unknown workloads in joelclaw, delete the pod and let the controller recreate it:

kubectl delete pod -n joelclaw <pod-name> --force --grace-period=0

If the host worker still won't come back after k8s is healthy, inspect the Talos container port map and remove any stale 3111/tcp binding:

DOCKER_HOST=unix:///Users/joel/.colima/default/docker.sock \
  docker inspect joelclaw-controlplane-1 --format '{{json .HostConfig.PortBindings}}' | python3 -m json.tool

If 3111/tcp is present, remove it with the same hostconfig/config.v2 edit flow used for hot port changes, then restart Docker + the Talos container. localhost:3111 must be free before worker-supervisor can bind.

Reboot Hardening

Ensure the Talos container restart policy is persistent in the Colima VM:

ssh -F ~/.colima/_lima/colima/ssh.config lima-colima \
  "docker update --restart unless-stopped --memory=18g --memory-swap=18g joelclaw-controlplane-1"
ssh -F ~/.colima/_lima/colima/ssh.config lima-colima \
  "docker inspect joelclaw-controlplane-1 --format '{{.HostConfig.RestartPolicy.Name}} {{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'"

Expected inspect output starts with unless-stopped and memory values around 19327352832 bytes.

Use the boot-safe critical daemon installer instead of user LaunchAgent symlinks:

sudo ~/Code/joelhooks/joelclaw/infra/install-critical-launchdaemons.sh

This installs com.joel.k8s-reboot-heal as a system LaunchDaemon for common reboot races (Colima stopped, Talos stopped, taint restored, flannel unhealthy) and installs com.joel.colima as the one-shot infra/colima-start.sh startup helper with --port-forwarder grpc.

Script path: ~/Code/joelhooks/joelclaw/infra/k8s-reboot-heal.sh Logs: ~/.local/log/k8s-reboot-heal.log

Flannel br_netfilter Crash

Symptoms: Flannel pods crash, stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory

Root cause: Talos-in-Docker shares Colima VM kernel. br_netfilter must load in the VM.

ssh -F ~/.colima/_lima/colima/ssh.config lima-colima "sudo modprobe br_netfilter"
# Wait for Flannel to auto-recover or delete the pod

The --config-patch at cluster creation (machine.kernel.modules: [{name: br_netfilter}]) prevents this on fresh clusters.

Full Cluster Recreation

When: Adding new port mappings (Docker ports are immutable), or unrecoverable corruption.

Before destroying: Back up Helm values and any data:

helm get values livekit-server -n joelclaw > /tmp/livekit-values-backup.yaml
helm get values bluesky-pds -n joelclaw > /tmp/pds-values-backup.yaml

# 1. Destroy
talosctl cluster destroy --name joelclaw

# 2. Ensure DOCKER_HOST is set
export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"

# 3. Write kernel module patch
cat > /tmp/talos-patch.yaml << 'EOF'
machine:
  kernel:
    modules:
      - name: br_netfilter
EOF

# 4. Create with ALL port mappings (add new ones here)
talosctl cluster create docker \
  --name joelclaw \
  --cpus-controlplanes "2.0" \
  --memory-controlplanes "4GiB" \
  --exposed-ports "3111:3111/tcp,6379:6379/tcp,7880:7880/tcp,7881:7881/tcp,8108:8108/tcp,8288:8288/tcp,8289:8289/tcp,9627:3000/tcp" \
  --workers 0 \
  --config-patch @/tmp/talos-patch.yaml \
  --subnet "10.5.0.0/24"

# 5. Fix kubeconfig context
kubectl config use-context admin@joelclaw-1

# 6. Get the talosctl endpoint port (auto-assigned)
TALOS_PORT=$(talosctl config info 2>&1 | grep Endpoints | awk -F: '{print $NF}')

# 7. Allow low NodePorts
talosctl -e 127.0.0.1:$TALOS_PORT -n 10.5.0.2 patch machineconfig --patch @- <<'PATCH'
cluster:
  apiServer:
    extraArgs:
      service-node-port-range: "1-65535"
PATCH

# 8. Remove control-plane taint (single node)
kubectl taint nodes joelclaw-controlplane-1 \
  node-role.kubernetes.io/control-plane:NoSchedule-

# 9. Install local-path-provisioner (Talos has no built-in storage)
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.30/deploy/local-path-storage.yaml
kubectl patch storageclass local-path \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# 10. Set privileged PSA
kubectl label namespace local-path-storage \
  pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl label namespace joelclaw \
  pod-security.kubernetes.io/enforce=privileged --overwrite

# 11. Deploy core services
kubectl apply -f ~/Code/joelhooks/joelclaw/k8s/

# 12. Deploy LiveKit (Helm + reconcile patches)
~/Code/joelhooks/joelclaw/k8s/reconcile-livekit.sh joelclaw

# 13. Deploy PDS (Helm) — NodePort MUST be 3000
helm install bluesky-pds nerkho/bluesky-pds \
  -n joelclaw -f /tmp/pds-values-backup.yaml
kubectl patch svc bluesky-pds -n joelclaw --type='json' \
  -p='[{"op":"replace","path":"/spec/ports/0/nodePort","value":3000}]'

# 14. Restart worker to reconnect
launchctl kickstart -k gui/$(id -u)/com.joel.system-bus-worker

Caddy HTTPS Proxy (Tailscale)

Caddyfile: ~/.local/caddy/Caddyfile TLS certs: ~/.local/certs/panda.tail7af24.ts.net.{crt,key}

Tailscale Funnel: panda.tail7af24.ts.net:443 → localhost:3111 (public internet webhooks).

Talos-Specific Commands

# Dashboard (live TUI)
talosctl -e 127.0.0.1:64785 -n 10.5.0.2 dashboard

# Kubelet logs
talosctl -e 127.0.0.1:64785 -n 10.5.0.2 logs kubelet

# Machine config
talosctl -e 127.0.0.1:64785 -n 10.5.0.2 get machineconfig -o yaml

# Config info (endpoints, cert expiry)
talosctl config info

Note: The talosctl endpoint port (64785) is auto-assigned at cluster creation and changes on recreation. Check talosctl config info for current value.

Verifying launchd PATH

After editing any infra launchd plist, verify it has PATH:

# Check all infra plists have PATH
for f in com.joel.colima com.joel.k8s-reboot-heal com.joel.system-bus-worker com.joel.gateway com.joel.caddy; do
  HAS_PATH=$(grep -c "/opt/homebrew/bin" ~/Library/LaunchAgents/$f.plist 2>/dev/null || echo "MISSING")
  echo "$f: $HAS_PATH"
done

Expected: all show 1 or higher. If any show 0 or MISSING, fix before deploying.

Helm Repos

nerkho   https://charts.nerkho.ch    # Bluesky PDS
livekit  https://helm.livekit.io     # LiveKit server

Secrets (agent-secrets)

launchd Services

Known Issues

1. Inngest `--sdk-url http://host.k3d.internal:3111` — Stale k3d hostname in ~/Code/joelhooks/joelclaw/k8s/inngest.yaml. Doesn't resolve in Talos. Works anyway because worker uses connect mode (INNGEST_DEV=0), not polling. Fix: update manifest to remove or replace with valid hostname.

2. Stale kubeconfig context — admin@joelclaw (old cluster) still in ~/.kube/config. Points to dead port 63324. Active context is admin@joelclaw-1. Clean up: kubectl config delete-context admin@joelclaw.

3. PDS data loss on recreation — PDS uses local-path PVC. Cluster destroy = data gone. Back up sqlite files before recreation if needed: kubectl cp joelclaw/bluesky-pds-xxx:/pds /tmp/pds-backup/.

4. No metrics-server — kubectl top doesn't work. Install if needed: kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml.

5. `serveHost` in serve.ts — Host worker must advertise a callback URL reachable from the Inngest pod. On Panda that is the host worker's Tailscale URL (INNGEST_SERVE_HOST=http://100.93.201.72:3111); do not use host.lima.internal or host.docker.internal unless a live pod-to-host probe proves it reaches macOS. Cluster worker should leave INNGEST_SERVE_HOST unset/empty in connect mode.

6. Control-plane taint can reappear after reboot — Single-node workloads may remain Pending with untolerated taint(s) until node-role.kubernetes.io/control-plane:NoSchedule is removed again.

8. Colima zombie state — colima status lies (returns 0) when VM tunnels are dead. Only real connectivity check (SSH + docker info) detects it. colima restart (not start) is the only fix. The heal script handles this automatically. See "Colima Zombie State Recovery" above.

9. Talos container has no shell — No bash, no /bin/sh, no busybox. Cannot docker exec into joelclaw-controlplane-1. Use talosctl for node operations. Kernel modules must be loaded at the Colima VM level via SSH: ssh lima-colima "sudo modprobe br_netfilter".

7. LiveKit hostNetwork probe target — With hostNetwork: true, probing pod IP (10.5.0.2) can fail even while LiveKit serves on 127.0.0.1:7880, causing CrashLoopBackOff (exit code 0, then kubelet SIGTERM on failed liveness/startup checks). Keep probe host pinned to loopback and use Recreate strategy for single-node hostPort scheduling:

kubectl patch deployment livekit-server -n joelclaw --type='strategic' -p '{
  "spec":{
    "strategy":{"type":"Recreate"},
    "template":{"spec":{"containers":[{"name":"livekit-server",
      "startupProbe":{"httpGet":{"host":"127.0.0.1","path":"/","port":"http","scheme":"HTTP"}},
      "livenessProbe":{"httpGet":{"host":"127.0.0.1","path":"/","port":"http","scheme":"HTTP"}},
      "readinessProbe":{"httpGet":{"host":"127.0.0.1","path":"/","port":"http","scheme":"HTTP"}}
    }]}}
  }
}'
kubectl rollout restart deployment/livekit-server -n joelclaw