Rch

RCH — Remote Compilation Helper

rch transparently offloads compilation commands to remote workers via a Claude Code PreToolUse hook. The daemon picks the fastest healthy worker, rsync's the workspace, runs the build, syncs artifacts back, and exits with the worker's exit code.

This skill is the operational layer agents use when something about that pipeline isn't working — and the much more common case where it thinks it's working but is silently falling back to local execution. The skill is built around a single principle: self-resolve before asking the human. Every recovery path here is one the agent can run on its own.

Don't re-learn the command surface here. rch --help, rch doctor, and the machine surfaces (--json, --schema, --help-json, --capabilities — see MACHINE_INTROSPECTION.md) self-describe every subcommand, flag, and env var. Env-var knobs and config precedence: CONFIGURATION.md. This skill carries the triage doctrine and recovery playbook routing.

Tested against rch v1.0.18; concepts apply to v1.0.16+.

---

Read This First

When a build feels slow, run one thing:

RCH_VISIBILITY=verbose <your-command> 2>&1 | grep -E '^\[RCH\]'

The summary line is a contract:

If you can't see why offload isn't happening, prove the path works in isolation before doing anything else:

rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check --workspace --all-targets

If that prints [RCH] remote <worker> (...), the offload pipeline is healthy. The problem is upstream of rch exec — usually the hook classifier or the agent's invocation form. If it also fails, follow RECOVERY_PLAYBOOKS.md.

---

Fast Triage Order

Run in this order and stop at the first failing stage:

1. Availability — rch check, rch status --workers --jobs, rch workers probe --all, rch queue 2. Config + socket consistency — rch config show --sources, rch --json config get general.socket_path, rch --json daemon status 3. Hook integration — rch hook status, rch agents status, rch hook install (idempotent) 4. Command classification + path closure — rch diagnose --dry-run "<your-command>" 5. Remote compile proof — the rch exec probe above 6. If sync fails or storage looks bad, inspect the worker directly:

ssh ubuntu@<host> 'df -h / /tmp && free -h && cat /proc/pressure/memory && cat /proc/pressure/io'
ssh ubuntu@<host> 'du -sh /tmp/rch-* /tmp/rch_target_* 2>/dev/null | sort -h'

Always check both / and /tmp on the worker before deciding what to fix. End-to-end verify: rch self-test --all; comprehensive checks: rch doctor (--fix --dry-run previews auto-fixes).

---

Quick Fixes

Debugging fail-opens: RCH_VISIBILITY=verbose shows the summary line; RCH_LOG_LEVEL=debug surfaces which fail-open path was taken. All other env knobs (priority, env allowlist, SSH keepalives, compression, profiles): rch --help + CONFIGURATION.md + SSH_TUNING.md.

---

Anti-Asking Rules

These are the questions agents historically ask the human that they should just answer themselves. The answer is in this skill or trivially derivable. Do not ask. Do.

• "Can I restart the daemon?" — Yes. rch daemon restart -y drains in-flight builds gracefully. It's the documented upgrade path.
• "Can I clean up /tmp/rch_target_*?" — If sudo lsof +D <dir> is empty, yes. See DISK_AND_PRESSURE.md. If non-empty, never.
• "Should I fix the chown on the worker?" — If the symptom matches the Permission denied recipe, yes. It's documented.
• "Should I disable an unreachable worker and continue?" — Yes. rch workers disable <id> --reason "..." --drain -y, then proceed with what's healthy.
• "Should I reinstall the hook?" — If rch hook status says missing, yes. rch hook install is idempotent.
• "Should I sync the toolchain to the workers?" — If RCH-E205 or "toolchain missing on X" appears, yes. rch workers sync-toolchain --all.
• "The cooldown is blocking my retry — should I delete it?" — No. Wait auto_start_cooldown_secs. If you really need to bypass, use rch daemon start directly (it's not gated by the hook autostart cooldown).
• "Can I drop the corrupt telemetry db?" — Yes. TELEMETRY_RECOVERY.md. Telemetry is derived data.
• "Should I recover SSH keys from a sibling host?" — If the keys are missing on this host but reachable on another, yes. SSH_KEY_RECOVERY.md Step 3.

When in genuine doubt, capture the escalation packet (Playbook end of RECOVERY_PLAYBOOKS.md) and surface that — not a wall of text — to the human.

---

Reference Index

Everything below ships in the skill. Read whichever is relevant.

Recognising what's wrong:

• FAIL_OPEN.md — every [RCH] local (...) reason mapped to a self-fix
• ERROR_CODES.md — full RCH-Exxx catalog with skill-doc cross-refs
• TROUBLESHOOTING.md — diagnostic flow + common errors

Solving specific failure classes:

• RECOVERY_PLAYBOOKS.md — symptom → fix in ≤90s, organized as 12 lettered playbooks
• SSH_KEY_RECOVERY.md — when workers.toml references keys this host doesn't have
• PATH_DEPENDENCIES.md — multi-repo workspaces, closure planner, [path_topology]
• DISK_AND_PRESSURE.md — RCH-E210..217 + the sbh handoff
• TELEMETRY_RECOVERY.md — corrupt ~/.local/share/rch/telemetry/telemetry.db
• SELF_HEALING.md — autostart cooldown, daemon supervision, [self_healing]
• SSH_TUNING.md — ControlMaster, keepalives, retry classification

Operating in fleets and swarms:

• MULTI_AGENT_CONTENTION.md — TOCTOU, fleet deploy races, autostart cooldown sharing
• OPERATIONS.md — full runbook + worker fleet lifecycle
• WORKERS.md — worker config, drain/disable/enable, deploy
• CONFIGURATION.md — config precedence, env vars, runtime paths
• HOOKS.md — hook protocol, install, test
• MACHINE_INTROSPECTION.md — --json, --schema, --help-json, --capabilities

Automation scripts (in `scripts/`):

• auto_recover.sh — heuristic, dry-run-by-default fleet recovery
• worker_disk_triage.sh — read-only mount-aware disk report per worker
• protocol_test.sh — directly probe the hook protocol with synthetic input
• multi_agent_safety.sh — flock wrapper for fleet/setup operations
• mine_rch_history.sh — find prior agent sessions that hit a given failure
• diagnose-rch.sh — comprehensive end-to-end diagnostic (the original)

Templates and project docs:

• assets/workers-template.toml
• Source: <https://github.com/Dicklesworthstone/remote_compilation_helper>

---

Adjacent Skills

• `sbh` — disk-pressure defense for AI coding workloads. Use when RCH-E210/211/215/216 fires.
• `agent-mail` — file reservations and messaging between agents. Use before rch fleet deploy or any worker config edit in a swarm.
• `ntm` / `vibing-with-ntm` — multi-agent tmux orchestration; common parent context for agents that hit rch failures.
• `cass` — search prior agent sessions; the skill ships scripts/mine_rch_history.sh as a fallback when cass index has dead pointers.

---

Reading Output: TUI vs Hook

rch itself, when invoked with no subcommand, runs in PreToolUse hook mode (reads JSON from stdin, writes JSON to stdout). Don't run bare rch from a terminal expecting help — use rch --help. Bare TUIs are at rch dashboard (terminal) and rch web (browser); both block your session.

RCH Configuration Reference

Contents

• Precedence and File Locations
• Main Config (`~/.config/rch/config.toml`)
• Workers Config (`~/.config/rch/workers.toml`)
• Environment Variables
• Hook Configuration (Claude Code)
• Validation and Diagnostics
• Runtime Data Paths

Precedence and File Locations

RCH resolves settings in this order (highest to lowest):

1. CLI flags (--json, --verbose, etc.) 2. Environment variables (RCH_*) 3. Profile defaults (RCH_PROFILE) 4. .env / .rch.env 5. Project config (.rch/config.toml) 6. User config (~/.config/rch/config.toml) 7. Built-in defaults

Primary files:

• User config: ~/.config/rch/config.toml
• Worker config: ~/.config/rch/workers.toml
• Project override: .rch/config.toml
• Optional transfer excludes: .rchignore

---

Main Config (~/.config/rch/config.toml)

[general]
enabled = true
force_local = false
force_remote = false
log_level = "info"                    # trace, debug, info, warn, error, off
socket_path = "~/.cache/rch/rch.sock" # default resolves from runtime/cache path

[compilation]
confidence_threshold = 0.85
min_local_time_ms = 2000
remote_speedup_threshold = 1.2
build_slots = 4
test_slots = 8
check_slots = 2
build_timeout_sec = 300
test_timeout_sec = 1800
bun_timeout_sec = 600
external_timeout_enabled = true

[transfer]
compression_level = 3
remote_base = "/tmp/rch"
adaptive_compression = true
verify_artifacts = false
exclude_patterns = [
  "target/",
  ".git/objects/",
  "node_modules/",
]

[selection]
strategy = "fair_fastest"

[output]
visibility = "summary"                # none, summary, verbose
first_run_complete = true

[self_healing]
hook_starts_daemon = true
daemon_installs_hooks = true
auto_start_timeout_secs = 3

Socket path default behavior:

• First choice: $XDG_RUNTIME_DIR/rch.sock
• Fallback: ~/.cache/rch/rch.sock
• Last resort: /tmp/rch.sock

---

Workers Config (~/.config/rch/workers.toml)

[[workers]]
id = "worker-name"
host = "203.0.113.20"
user = "ubuntu"
identity_file = "~/.ssh/id_ed25519"
total_slots = 16
priority = 100
tags = ["rust", "bun", "fast"]

Slot guidance:

• Start with ~2x physical CPU cores for mixed workloads.
• Reduce slots if workers hit CPU steal, swap pressure, or I/O saturation.
• Increase priority for faster/more reliable workers.

---

Environment Variables

Common overrides:

---

Hook Configuration (Claude Code)

Location: ~/.claude/settings.json

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "/absolute/path/to/rch"
          }
        ]
      }
    ]
  }
}

Recommended management commands:

rch hook install
rch hook status
rch hook uninstall

---

Validation and Diagnostics

rch config show --sources
rch config validate
rch config lint
rch config doctor
rch check

---

Runtime Data Paths

Worker Disk & Resource Pressure

Contents

• Pressure Surfaces RCH Tracks
• The First Mistake: `/` vs `/tmp` Confusion
• Inspect First, Delete Second
• Hand Off to `sbh`
• When `sbh` Doesn't Help (RCH-E215 / RCH-E216 / RCH-E217)
• Memory Pressure (RCH-E214)
• I/O Pressure (RCH-E213)
• Telemetry Lag (RCH-E212)
• Preventive Hygiene
• Triage Cheat Sheet

Disk pressure on workers is the single biggest source of "rch was working, now it isn't" bug reports. This file is the canonical playbook — and the explicit handoff protocol to the sbh (Storage Ballast Handler) skill, which exists precisely to defend against this.

---

Pressure Surfaces RCH Tracks

The daemon collects telemetry from each worker and exposes it as a stable enum. Visible codes:

These appear in:

• rch --json status --workers — at .data.daemon.workers[], with flat fields:

pressure_state, pressure_reason_code, pressure_confidence, pressure_disk_free_gb, pressure_disk_total_gb, pressure_disk_free_ratio, pressure_disk_io_util_pct, pressure_memory_pressure, pressure_telemetry_age_secs, pressure_telemetry_fresh

• rch --json workers probe --all — .data[].error includes pressure-related text when probe surfaces it
• [RCH] remote <worker> failed [RCH-E2xx] summary line

---

The First Mistake: / vs /tmp Confusion

By default RCH stages remote builds under [transfer] remote_base = "/tmp/rch", but workspace mirrors live under /data/projects (canonical root) which is on /. If rch status warns about pressure on a worker, check both filesystems separately — fixing the wrong one wastes time.

ssh ubuntu@<host> 'df -h / /tmp && free -h && cat /proc/pressure/memory && cat /proc/pressure/io'

Interpretation:

• /tmp hot, / fine → stale /tmp/rch_target_* or /tmp/rch-* artifact dirs
• / hot, /tmp fine → bloated target_* trees inside /data/projects
• both hot → sbh reclaim, then targeted cleanup

---

Inspect First, Delete Second

Before removing anything:

ssh ubuntu@<host> 'du -sh /tmp/rch-* /tmp/rch_target_* 2>/dev/null | sort -h | tail'
ssh ubuntu@<host> 'find /data/projects -maxdepth 2 -type d \( -name "target_rch_*" -o -name "target_*" -o -name "target-*" -o -name target \) -exec du -sh {} + 2>/dev/null | sort -h | tail -n 20'

Then verify the candidate is inactive (no open files):

ssh ubuntu@<host> 'sudo lsof +D /tmp/rch_target_<name> 2>/dev/null | head'
ssh ubuntu@<host> 'sudo lsof +D /data/projects/<repo>/target_rch_<name> 2>/dev/null | head'

Only when lsof is empty is the directory safe to clean.

---

Hand Off to sbh

The sbh skill is the right tool for sustained disk pressure on workers. Quote from its description: "Disk-pressure defense for AI coding workloads. Use when: disk full, low space, ballast, cleanup, scan artifacts."

Pattern: detect with rch, remediate with sbh.

# Detect from rch — the actual JSON path is .data.daemon.workers[]
# and pressure fields are FLAT (e.g., .pressure_state, .pressure_reason_code).
rch --json status --workers | jq -r '
  .data.daemon.workers[]
  | select(.pressure_state != "healthy")
  | "\(.id)\t\(.pressure_state)\t\(.pressure_reason_code)"'

# Get host for a given worker id (so you can ssh to it)
worker_host() {
  rch --json workers list \
    | jq -r --arg id "$1" '.data.workers[] | select(.id == $id) | .host'
}

# For each pressure-flagged worker, hand off to sbh on that host
rch --json status --workers \
  | jq -r '.data.daemon.workers[] | select(.pressure_state != "healthy") | .id' \
  | while read -r w; do
      h="$(worker_host "$w")"
      ssh "ubuntu@$h" 'sbh status --json' 2>/dev/null \
        || echo "($w / $h) — sbh not installed or ssh failed"
    done

If sbh is installed on the worker, it can:

• Drop ballast files
• Scan and clean artifact dirs (incremental compilation, doctests, stale incremental/)
• Report what it freed

If sbh is not on a worker yet, you should not write rm -rf of your own. Either install sbh (one-line bootstrap) or escalate. The combination of (a) building under-resourced disks and (b) one agent's rm -rf colliding with another agent's active build has historically been the worst class of incidents in this fleet.

---

When sbh Doesn't Help (RCH-E215 / RCH-E216 / RCH-E217)

• `RCH-E215 Disk reclaim failed` — sbh ran but couldn't free enough. Inspect the largest residents that sbh refused to touch (often /var/log/journal or ~/.cargo/registry). For ~/.cargo/registry, cargo cache --autoclean (if cargo-cache installed) is safer than blanket deletion.
• `RCH-E216 Insufficient headroom for build reservation` — the worker's free space is below the build reservation watermark even after reclaim. Either raise reservation budget (config) or steer the build elsewhere with tags / rch workers drain <id>.
• `RCH-E217 Active build protection prevented reclaim` — sbh refused to touch a path that was actively being written. Wait for the build, then retry. Don't override; this guard exists to avoid breaking other agents' builds.

---

Memory Pressure (RCH-E214)

Memory pressure is usually transient (a big test process). Quick diagnose:

ssh ubuntu@<host> 'free -h && cat /proc/pressure/memory && ps -eo pid,user,rss,cmd --sort=-rss | head -15'

If a runaway cargo test is the culprit, use the process-triage skill (pt) — its job is exactly this. Don't kill -9 blindly; another agent's active build might be the largest process.

---

I/O Pressure (RCH-E213)

If you see this code repeatedly without disk pressure, the worker is likely under contention from multiple parallel builds. Either:

• Lower the worker's total_slots to reduce concurrency
• Route work to a less-loaded worker via tags
• Wait — RCH-E213 clears as soon as I/O drops

---

Telemetry Lag (RCH-E212)

Disk pressure telemetry stale or missing. The worker isn't reporting fresh pressure data. Either:

• rch-wkr on the worker is dead → ssh in and check, then rch fleet deploy --workers <id> --verify
• The worker just rebooted; wait one telemetry tick (~30s)
• Daemon is wedged — rch daemon restart -y (after checking rch queue for active builds)

---

Preventive Hygiene

In ~/.config/rch/config.toml on every host, keep these excludes generous:

[transfer]
exclude_patterns = [
  "target/",
  "target_*/",
  "target-*/",
  ".cargo-target/",
  ".cargo-target-*/",
  ".rch-target-*/",
  "node_modules/",
  ".git/objects/",
  "dist/",
  ".next/",
]

Reload after editing:

rch daemon reload
rch config show --sources | grep -A8 transfer

Use ${TMPDIR:-/tmp} rather than hardcoded /tmp for any agent-injected target dir:

rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check

(This is what rch itself uses since v1.0.17 commit 0f4158d.)

---

Triage Cheat Sheet

# 1. Quick view: pressure across the fleet
rch --json status --workers \
  | jq -r '.data.daemon.workers[]
           | "\(.id)  state=\(.pressure_state)  free_gb=\(.pressure_disk_free_gb)  io_util=\(.pressure_disk_io_util_pct)  mem=\(.pressure_memory_pressure)"'

# 2. Drill down on one worker (use the .host field from `rch workers list`)
ssh ubuntu@<host> 'df -h / /tmp && free -h && cat /proc/pressure/memory && cat /proc/pressure/io'

# 3. Inventory artifact directories
ssh ubuntu@<host> 'du -sh /tmp/rch-* /tmp/rch_target_* /data/projects/*/target* 2>/dev/null | sort -h | tail'

# 4. Verify candidate is inactive
ssh ubuntu@<host> 'sudo lsof +D <candidate-dir> 2>/dev/null | head'

# 5. Hand to sbh
ssh ubuntu@<host> 'sbh status --json'        # what's the situation?
ssh ubuntu@<host> 'sbh reclaim --auto'       # if available

# 6. If sbh insufficient — drain and investigate manually
rch workers drain <id> -y
# ... investigate ...
rch workers enable <id>

RCH Error Code Catalog

Contents

• Live Catalog
• Categories
• High-Frequency Codes (with the right reaction)
• Cross-References
• Schema Discovery

RCH ships a stable error catalog of 94 codes in the RCH-Exxx namespace. Every user-visible failure that is expected and explainable carries one of these codes. They appear in:

• [RCH] remote <worker> failed [RCH-Exxx] <summary> (build env failures)
• [RCH] local (dependency preflight RCH-Exxx: <remediation>) (closure planner)
• rch doctor --json (.checks[].code)
• rch --json responses on errors (.error.code)
• Daemon log lines

Treat the code as the stable handle. Don't grep for the human-readable summary, which can be reworded between releases.

---

Live Catalog

The authoritative catalog is shipped with the binary. Always prefer this over what's quoted below:

rch schema export -o /tmp/rch-schemas
jq -r '.errors[] | "\(.code) | \(.message)"' /tmp/rch-schemas/error-codes.json | sort

Per-code remediation steps:

jq '.errors[] | select(.code=="RCH-E210") | {code, message, remediation}' /tmp/rch-schemas/error-codes.json

---

Categories

---

High-Frequency Codes (with the right reaction)

These are the codes agents actually see in practice. The rest are in the schema export.

Configuration

Network

Worker

Build

Transfer

Internal

---

Cross-References

• Path-dep family (RCH-E013–E024): PATH_DEPENDENCIES.md
• Disk-pressure family (RCH-E210–E217): DISK_AND_PRESSURE.md + sbh skill
• SSH family (RCH-E100–E109): SSH_TUNING.md
• Selection family (RCH-E200–E209): FAIL_OPEN.md
• Daemon/internal (RCH-E500–E509): SELF_HEALING.md + OPERATIONS.md

---

Schema Discovery

For agents that need to consume the catalog programmatically (e.g., to build a remediation table at runtime):

rch schema list                  # human-readable
rch schema export -o ./schemas   # writes api-response, api-error, error-codes
rch --schema config lint         # JSON Schema for one command's output
rch --capabilities               # full capability description

Every command also accepts --help-json to dump its argument tree as JSON.

Detecting and Responding to RCH Fail-Open

Contents

• Golden Rule
• The Fail-Open Surface
• Fail-Open Reasons (and What To Do)
• Detection Snippets
• Force the Issue
• "Don't Ask the Human" Rules

RCH's most expensive failure mode for agents is silent fall-back to local execution. The build "succeeded" — but it ran on the local machine, slowly, while the worker fleet sat idle. If you don't notice, you bake hours of extra latency into every iteration.

This file is the canonical guide for: (1) how to see a fail-open, (2) what each fail-open reason means, and (3) what to do about it before asking the human.

---

Golden Rule

Never say "build done" until you've checked stderr for `[RCH] local (...)`.

If you see that string, the build did not run remotely. It might still be a correct build, but RCH chose to fall back, and the parenthetical reason is a contract telling you exactly why.

---

The Fail-Open Surface

rch exec and the PreToolUse hook print exactly one summary line on stderr at the end of every routed compilation. The visibility is controlled by [output] visibility = "summary"|"verbose"|"none" (env: RCH_VISIBILITY).

There are five summary forms:

To force a summary banner without changing config:

RCH_VISIBILITY=verbose cargo check

---

Fail-Open Reasons (and What To Do)

Every reason in the parens comes from one of two sources:

1. Hook decision points in rch/src/hook.rs::process_hook and run_exec — short, hand-written reason strings 2. Daemon selection reasons (SelectionReason enum in rch-common/src/types.rs) — stable machine reasons from worker selection

Hook-decision fail-opens

Daemon-decision fail-opens (selection reasons)

These come from the daemon's SelectionReason enum. The [RCH] local (...) summary uses the human Display form (verbatim from Display for SelectionReason in rch-common/src/types.rs). Machine-readable JSON output (rch --json) uses the snake_case tag instead. Match either when you grep — the human form is what appears on stderr.

Two more variants — affinity_pinned and affinity_fallback — are success paths (a worker was assigned via affinity), not fail-opens, so they never appear in [RCH] local (...) output.

"Build" succeeded but nothing went remote

If the command exited 0, the agent often calls the work done. Check the summary line first. A common pathology:

   Compiling foo v0.1.0
    Finished `dev` profile in 38.41s
[RCH] local (all workers at capacity)

That's a 38-second build that should have been 2 seconds remote. Queueing is on by default; if you still see this, bump RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120 so the next iteration waits longer for a slot rather than burning local CPU.

---

Detection Snippets

Single-shot check after a build:

# Show the last RCH summary line from this shell's stderr capture
grep -E '^\[RCH\] (remote|local)' /tmp/rch_last_run.stderr | tail -1

Wrapper that runs a build and asserts remote:

out=$(RCH_VISIBILITY=summary cargo check 2>&1)
if grep -qE '^\[RCH\] local' <<<"$out"; then
  echo "RCH FELL BACK TO LOCAL:" >&2
  grep -E '^\[RCH\] local' <<<"$out" >&2
  exit 99
fi
echo "$out" | tail -3

JSON-mode check (for scripted agents):

rch --json check | jq -r '.data.status'   # ready | degraded | unhealthy

---

Force the Issue

When you absolutely need to know whether remote works at all (for example, before deciding to file an issue), bypass the hook entirely:

rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check --workspace --all-targets

If that prints [RCH] remote <worker> (...), the offload path is healthy and the failures are coming from the hook classifier (or some shell wrapper around your command). If it also prints [RCH] local (...), follow the reason above.

---

"Don't Ask the Human" Rules

The skill exists so agents resolve these things autonomously. Concrete cases where the answer is always do, never ask:

• "daemon unavailable" → rch daemon start && rch --json daemon status. If the auto-start cooldown is active, wait or remove ${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.cooldown and try again.
• "all workers unreachable" → rch workers probe --all and inspect SSH errors. Don't ask "should I fix this?" — fix what's fixable (key permissions, missing host entries, dead daemon).
• "all workers at capacity" → queueing is already on by default; bump RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120 to wait longer, or raise total_slots if the fleet is genuinely undersized.
• "toolchain missing on X" → rch workers sync-toolchain --all. The skill is designed for you to do this without confirmation.
• Stale unix socket detected (auto-start log message) → already handled by try_auto_start_daemon. Don't manually rm the socket; just retry.
• Permission denied on `/data/projects/<repo>` over rsync → if the repo on the worker is owned by root: ssh ubuntu@<host> 'sudo chown -R ubuntu:ubuntu /data/projects/<repo> && sudo chmod 775 /data/projects/<repo>'. This is a known recovery; don't escalate.

When in genuine doubt — escalate after collecting rch doctor --json, rch --json daemon status, rch --json workers probe --all, and the failing command's stderr. That packet lets the human resolve in one round trip.

Hook Integration

Contents

• Execution Flow
• Installation and Status
• Hook Protocol (Current)
• What Gets Intercepted
• Quick Hook Tests
• Performance and Safety Notes

Execution Flow

Claude Code PreToolUse -> rch (no subcommand = hook mode)
                               |
                               +-- Non-Bash tool -> allow unchanged
                               +-- Bash non-compilation -> allow unchanged
                               +-- Bash compilation -> allow with modified command:
                                   "rch exec -- <original command>"

Important behavior:

• Hook returns quickly (classification path), then remote execution happens via rch exec -- ....
• On unsafe/unavailable remote conditions, hook fails open and allows local execution.
• For successful or failed remote runs, RCH preserves command semantics by rewriting to true or exit <code> as needed.

---

Installation and Status

rch hook install
rch hook status
rch hook test
rch hook uninstall

For multi-agent installs:

rch agents list
rch agents status
rch agents install-hook claude-code
rch agents uninstall-hook claude-code

Installed Claude settings entry:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "/absolute/path/to/rch" }
        ]
      }
    ]
  }
}

---

Hook Protocol (Current)

Input on stdin

{
  "tool_name": "Bash",
  "tool_input": {
    "command": "cargo build --release",
    "description": "Build project"
  },
  "session_id": "optional"
}

Output on stdout

1. Allow unchanged command: empty stdout 2. Allow with command rewrite (transparent interception):

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "allow",
    "updatedInput": {
      "command": "rch exec -- cargo build --release"
    }
  }
}

3. Deny command (rare, policy-level):

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "reason text"
  }
}

---

What Gets Intercepted

Common intercepted command families:

• Rust: cargo build, cargo check, cargo clippy, cargo doc, cargo test, cargo nextest run, cargo bench, rustc
• Bun/TypeScript: bun test, bun typecheck
• C/C++: gcc, g++, clang, clang++
• Build systems: make, cmake --build, ninja, meson compile

Commonly not intercepted:

• Local-mutating package commands (cargo install, cargo clean, bun install, bun add, etc.)
• Interactive/dev commands (bun run, bun dev, bun build, bunx)
• Piped/redirected/backgrounded shell forms where deterministic offload is unsafe

---

Quick Hook Tests

# Direct protocol test (compilation command)
printf '%s\n' \
  '{"tool_name":"Bash","tool_input":{"command":"cargo build --release"}}' | rch

# Direct protocol test (non-compilation command should usually return empty stdout)
printf '%s\n' \
  '{"tool_name":"Bash","tool_input":{"command":"ls -la"}}' | rch

# Built-in integration test
rch hook test

Debugging:

RCH_LOG_LEVEL=debug rch hook test
RCH_LOG_LEVEL=debug rch diagnose "cargo test --workspace"

---

Performance and Safety Notes

• Non-compilation decisions target sub-millisecond latency.
• Compilation decisions target low-millisecond latency.
• Hook mode always keeps stdout protocol-clean; diagnostics go to stderr.
• If parsing/config/daemon selection fails, RCH allows local execution instead of blocking.

Machine-Readable Surfaces (For Agents)

Contents

• Three Levels of Discovery
• Per-Command JSON Mode
• Unified Response Envelope
• Output Format Modes
• Useful jq Recipes
• Wire-Level Hook Protocol
• Built-In Robot Docs
• Where Agents Trip Up

rch is built for agents. Every command returns structured output, every command exposes its schema, and the whole CLI is queryable as JSON. This file is the canonical guide for using those surfaces — so an agent can discover capability instead of guessing.

---

Three Levels of Discovery

1. --capabilities — what does this rch know how to do?

rch --capabilities

Returns a JSON object describing version, build info, supported runtimes (rust, bun, node), available subcommands, and feature flags. Use this once at session start to confirm you're talking to the rch you expect.

2. --help-json — full CLI tree as JSON

rch --help-json                  # entire CLI
rch --help-json workers          # one subcommand subtree
rch --help-json workers probe

Parse this to drive an agent that needs to construct flags it hasn't seen before.

3. --schema — JSON Schema for a specific command's output

rch --schema config lint
rch --schema workers list
rch --schema daemon status

Validate the JSON you get back, or use it to build typed clients.

---

Per-Command JSON Mode

Every subcommand accepts --json (and -F json|toon):

rch --json check                          # quick health
rch --json daemon status                  # daemon state
rch --json workers list                   # configured workers
rch --json workers probe --all            # connectivity
rch --json status --workers --jobs        # full status with workers and jobs
rch --json queue                          # build backlog
rch --json hook status                    # hook install state across agents
rch --json agents status                  # agent detection result
rch --json self-test --all                # end-to-end verification
rch --json speedscore --all               # composite score per worker
rch --json fleet status                   # fleet deploy state
rch --json fleet history --limit 20       # deployment timeline
rch --json doctor                         # diagnostic report
rch --json config show --sources          # effective config + provenance
rch --json config get general.socket_path # one value with source
rch --json config diff                    # delta from defaults
rch --json diagnose --dry-run "<cmd>"     # explain routing decision

stdout is always data-only. Diagnostics go to stderr. Exit code 0 means success.

---

Unified Response Envelope

Every JSON response follows this shape:

{
  "kind": "ok" | "error",
  "command": "workers.probe",
  "data":  { ... },        // present when kind=ok
  "error": { ... },        // present when kind=error
  "elapsed_ms": 123,
  "request_id": "...",
  "version": "1.0.18"
}

Schema: rch schema export -o ./schemas produces:

• api-response.schema.json — the success envelope
• api-error.schema.json — the error envelope
• error-codes.json — the full RCH-Exxx catalog

Errors carry error.code (RCH-Exxx), error.message, and error.remediation (an array of strings). Use the code as the stable handle.

---

Output Format Modes

RCH_OUTPUT_FORMAT=json rch status     # JSON (implies --json)
RCH_OUTPUT_FORMAT=toon rch status     # TOON (compact text-overlay format)
TOON_DEFAULT_FORMAT=toon rch --json status   # Switch JSON-flagged calls to TOON

For agent pipelines, JSON is universally safest. TOON is useful for terminals.

NO_COLOR=1 and FORCE_COLOR=1 work as expected.

---

Useful jq Recipes

These jq paths reflect the actual response shapes in rch v1.0.18. Each path was verified against live output, not assumed.

# Daemon health summary
rch --json check | jq -r '.data.status'   # "ready" | "degraded" | "unhealthy"

# Daemon version (NOT in 'rch --json daemon status' — that endpoint is minimal)
rch --json status | jq -r '.data.daemon.daemon.version'

# Worker IDs that are reachable. `rch --json workers probe --all` returns
# .data as a flat array, not nested under .workers.
rch --json workers probe --all \
  | jq -r '.data[] | select(.status == "ok") | .id'

# Workers that aren't healthy (any non-"ok" status surfaces an error string)
rch --json workers probe --all \
  | jq -r '.data[] | select(.status != "ok") | "\(.id) [\(.status)] \(.error // "")"'

# Workers under pressure. Pressure fields live FLAT on each worker record
# under .data.daemon.workers[] inside `rch --json status`.
rch --json status --workers \
  | jq -r '.data.daemon.workers[]
           | select(.pressure_state != "healthy")
           | "\(.id) [\(.pressure_state)] \(.pressure_reason_code)"'

# Active builds (lives in `rch --json queue`, NOT `daemon status`)
rch --json queue | jq -r '.data.active_builds[]? | "\(.id) \(.worker_id) \(.project_id)"'

# Queue depth
rch --json queue | jq '.data.active_builds | length'

# Configured workers (canonical shape: .data.workers[].{id, host, user, total_slots, priority, tags})
rch --json workers list | jq -r '.data.workers[] | "\(.id)\t\(.host)\t\(.tags|join(","))"'

# Hook install state across detected agents
rch --json hook status | jq -r '.data.agents[] | "\(.agent)\t\(.status)"'

# All known error codes for a category (after `rch schema export -o ./schemas`)
jq -r '.errors[] | select(.category == "transfer") | "\(.code)\t\(.message)"' schemas/error-codes.json

---

Wire-Level Hook Protocol

rch is itself a Claude Code PreToolUse hook. You can hand-craft requests to it (useful for tests):

printf '%s\n' \
  '{"tool_name":"Bash","tool_input":{"command":"cargo build --release"}}' \
  | rch

Three response shapes (full text in references/HOOKS.md):

• Empty stdout → allow unchanged
• {"hookSpecificOutput": {"permissionDecision": "allow", "updatedInput": {"command": "rch exec -- ..."}}} → allow with rewrite
• {"hookSpecificOutput": {"permissionDecision": "deny", "permissionDecisionReason": "..."}} → block

Drive a deeper protocol probe with the skill's scripts/protocol_test.sh.

---

Built-In Robot Docs

rch --help                       # human help
rch --help-json                  # everything as JSON
rch schema list                  # what schemas are available
rch schema export -o ./schemas   # write them to disk

For comparison, cass robot-docs guide is the cass equivalent (used by the cass skill).

---

Where Agents Trip Up

• Forgetting `--json`. Human-readable rch output is nice but reformats. Always use --json (or --schema) when piping into other tools.
• Conflating "exit 0" with "build was remote". It isn't. See FAIL_OPEN.md.
• Bare `rch dashboard` / `rch web`. Both launch interactive UIs that block your session. Don't run them from automation.
• Bare `rch tui`-like commands. RCH does not currently ship a tui subcommand; the dashboard is rch dashboard. The general anti-pattern is the same: anything interactive blocks.
• Reading `--json` output with grep instead of jq. Field names are stable. Use jq.

Multi-Agent Contention

Contents

• Same Project on Same Worker (TOCTOU)
• All Workers Busy / Backlog Pressure
• Daemon Stop / Restart Coordination
• Worker Setup Storms
• Concurrent Edits in the Project Tree During Sync
• Stale Daemon Across rch CLI Upgrade
• Auto-Start Cooldown
• SSH ControlMaster Poisoning
• Cancellation Storms
• Coordination Stack Cheat Sheet

When 22 Claude Max accounts and 11 GPT Pro accounts are all driving rch on the same machine and into the same worker fleet, the failure modes are no longer about correctness — they're about contention. This file catalogs the patterns and the cooperation rules.

---

Same Project on Same Worker (TOCTOU)

Symptom: Two agents kick off cargo build for the same project at almost the same time. One succeeds; the other gets RCH-E305 Remote working directory error or sees the rsync target locked.

Root cause: Two builds racing in the same target/ checkout on the worker.

Status: Largely fixed in v1.0.16 (commit fbea95f, then dbf9682 for the final TOCTOU-race close in rchd/src/selection.rs). The daemon now:

• Excludes workers already running an active build for the same project from selection
• Atomically claims the active-build slot after slot reservation

What you can still do wrong: bypass the daemon. If you ssh worker 'cd /data/projects/foo && cargo build' outside rch exec, you defeat the guard. Always go through rch exec (or let the hook route the command).

If you see the symptom anyway:

1. Confirm both agents are routing through the daemon, not direct SSH. Active builds live in rch --json queue, not daemon status:

   rch --json queue | jq -r '.data.active_builds[]? | "\(.id) \(.worker_id) \(.project_id)"'

2. Look for the same project_id repeated across active builds. 3. If reproducible, capture rch doctor --json and rch daemon logs -n 200; this is an upstream regression worth filing.

---

All Workers Busy / Backlog Pressure

Symptom: [RCH] local (all workers at capacity) or [RCH] local (all worker circuits open) start showing up under heavy swarm load. (Snake-case tags in JSON output: all_workers_busy, all_circuits_open.)

Self-fix (per-agent, no human needed):

Queue-when-busy is on by default in current rch — agents already wait for a slot rather than falling open immediately. The env var RCH_QUEUE_WHEN_BUSY exists primarily to disable this (set =0) for benchmarking. If you still see [RCH] local (all workers at capacity), the wait timed out — bump it:

export RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120   # default is shorter

This buys more time for slots to free up before falling back. For unbounded waiting, raise it further; for queue-or-die behavior, combine with RCH_QUEUE_WHEN_BUSY=1 (explicit) and a high timeout.

Fleet-wide tuning:

rch queue --watch                                                          # see backlog live
rch --json status --workers --jobs | jq '.data.daemon.workers[] | {id, used: .used_slots, total: .total_slots}'
rch workers list --speedscore                                              # spread across workers

If aggregate slot capacity is the bottleneck, raise total_slots on the fastest workers (~/.config/rch/workers.toml) and rch daemon reload. A reasonable starting point is 2x physical cores.

If a single worker is hot and others are cold, check tags / runtime gates that may be funneling everyone to one worker.

---

Daemon Stop / Restart Coordination

Risk: One agent runs rch daemon restart -y while five others have active builds against the daemon. The restart drops their connections.

Cooperation rule:

• Never rch daemon stop or rch daemon restart while builds are active. The CLI prompts for confirmation; agents in --yes mode skip the prompt — don't.
• Before any restart, check:

  rch --json queue | jq '.data.active_builds | length'

If the active list isn't empty, drain first or wait.

If you must restart (e.g., new binary deploy), use:

rch fleet drain --all -y          # gracefully stop accepting new jobs fleet-wide (workers, not local)
rch daemon restart -y             # local daemon
rch fleet enable --all            # re-enable workers

Use Agent Mail file reservations on ~/.config/rch/{config.toml,workers.toml} and the daemon socket path before destructive ops.

---

Worker Setup Storms

Symptom: Six agents all hit rch workers setup --all at once. They step on each other deploying rch-wkr binaries and toolchains; one or more workers end up half-installed.

Cooperation rule:

rch workers setup and rch fleet deploy are idempotent but not multi-writer safe under heavy concurrency. Wrap them with a host-local flock so only one runs at a time. The skill ships scripts/multi_agent_safety.sh as a wrapper:

.claude/skills/rch/scripts/multi_agent_safety.sh rch workers setup --all

Or use Agent Mail file reservations:

file_reservation_paths(project_key, agent_name,
  paths=["~/.config/rch/workers.toml", "~/.config/rch/config.toml"],
  ttl_seconds=600, exclusive=true,
  reason="rch fleet deploy")

---

Concurrent Edits in the Project Tree During Sync

Symptom: rsync error mid-transfer — file changed/disappeared mid-stream. Or RCH-E406 Transfer checksum mismatch.

Root cause: Another agent is editing files while your build is being shipped to the worker.

Mitigations:

• Add target_*/, target-*/, and other artifact-shaped patterns to [transfer] exclude_patterns so build outputs don't get picked up by the upload (cf. recent split between upload and retrieval excludes in transfer.rs).
• For multi-agent projects, take an Agent Mail reservation on the source files before you touch them, so other agents stage their work elsewhere.
• Consider a pre-build snapshot (git stash --include-untracked plus a re-apply) for deeply concurrent workspaces. Only with explicit user authorization (per AGENTS.md rules).

---

Stale Daemon Across rch CLI Upgrade

Symptom: You ran rch update (or installed a new build) and now CLI features behave oddly while the daemon is still running the old version.

Self-fix:

rch --version
rch --json daemon status | jq '.data.version'
# If they don't match:
rch daemon restart -y
rch --json daemon status | jq '.data.version'   # Confirm equal

rch fleet verify checks worker binary hashes too:

rch fleet verify

If a worker is on an old rch-wkr, deploy:

rch fleet deploy --canary 25 --canary-wait 60 --verify
# or for a single worker:
rch fleet deploy --worker <id> --verify

To update both the local rch CLI and the fleet binaries in one shot: rch update --fleet.

---

Auto-Start Cooldown

try_auto_start_daemon in rch/src/hook.rs writes a cooldown timestamp to ${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.cooldown after each spawn attempt. Subsequent invocations within [self_healing] auto_start_cooldown_secs (default 30s) get AutoStartError::CooldownActive and fall open.

Implication for swarms: Right after a daemon crash, only the first agent gets to restart it. Everyone else falls open until the cooldown expires (30 seconds by default — that's a long time for a swarm; lower it via [self_healing] auto_start_cooldown_secs = 5 if you accept the trade-off of more spawn churn on a flapping daemon).

What to do: Don't manually delete the cooldown file in a tight loop. Let it expire (a few seconds), or rch daemon start explicitly (which doesn't go through the hook autostart path).

If something is genuinely wedged and the cooldown is hiding a real failure to spawn:

ls -la "${XDG_RUNTIME_DIR:-/tmp}/rch/"     # see lock + cooldown
cat "${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.cooldown"
rch daemon start                           # bypasses hook cooldown

If rch daemon start itself fails, you have a real problem — check rch daemon logs -n 200 and which rchd.

---

SSH ControlMaster Poisoning

SshOptions::default().control_master = false since the recent fix (commit 464a25b). Stale local control sockets had been poisoning otherwise healthy connections.

If you (or another agent) explicitly opted into ControlMaster via RCH_SSH_CONTROL_PERSIST_SECS and you start seeing intermittent RCH-E100/RCH-E105 errors:

ls ~/.ssh/control-*  ~/.ssh/cm-*  /run/user/$(id -u)/ssh-*  2>/dev/null
# If stale sockets exist, remove them (with user authorization for /run path)
ssh -O check ubuntu@<host> 2>&1 || true
ssh -O exit ubuntu@<host> 2>&1 || true

Then unset RCH_SSH_CONTROL_PERSIST_SECS for the next swarm run. See SSH_TUNING.md.

---

Cancellation Storms

If a parent agent issues rch cancel --all --yes while sub-agents are mid-build, expect:

• RCH-E320 graceful cancel signal dispatched
• RCH-E321 escalated to forced kill after timeout (if the sub-agent's command ignored SIGTERM)
• RCH-E323 post-cancel cleanup errors (rare; usually file-handle related)
• RCH-E324 slots not released after cancel (rare; daemon log will show it)

Best practice in swarms:

• Prefer rch cancel <build-id> over --all. Sub-agents tell you their build IDs in the summary line.
• If you must --all, broadcast it via Agent Mail first so sub-agents know to expect failures.

---

Coordination Stack Cheat Sheet

RCH Operations

Contents

• Baseline Runbook
• Worker Fleet Lifecycle
• Fleet Deploy/Rollback
• Path-Dependency and Multi-Repo Notes
• Transfer Stability (Rsync/Artifact Churn)
• Queue and Cancellation Operations
• Anti-Patterns
• Debug Command Pack

Baseline Runbook

Use this sequence for most production incidents.

1) Confirm current posture

rch check
rch status --workers --jobs
rch workers probe --all
rch queue

2) Validate config and daemon wiring

rch config show --sources
rch --json config get general.socket_path
rch --json daemon status
rch config doctor

3) Validate hook routing

rch hook status
rch agents status
rch hook test

4) Validate offload path directly

rch diagnose "cargo check --workspace --all-targets"
rch exec -- env CARGO_TARGET_DIR=/tmp/rch_target_<name> cargo check --workspace --all-targets

If step 4 succeeds, RCH infrastructure is healthy and remaining failures are project/toolchain specific.

5) If workers show storage pressure, inspect the right filesystem

RCH pressure warnings often come from / while the immediate churn lives in /tmp. Check both before deciding whether the host actually needs intervention:

ssh ubuntu@<host> 'df -h / /tmp'
ssh ubuntu@<host> 'free -h && cat /proc/pressure/memory && cat /proc/pressure/io'

Then inspect the usual large artifact surfaces:

ssh ubuntu@<host> 'du -sh /tmp/rch-* /tmp/rch_target_* 2>/dev/null | sort -h'
ssh ubuntu@<host> 'find /data/projects -maxdepth 2 -type d \( -name "target_rch_*" -o -name "target_*" -o -name "target-*" -o -name target \) -exec du -sh {} + 2>/dev/null | sort -h | tail -n 20'

Before removing anything, verify the candidate is inactive:

ssh ubuntu@<host> 'sudo lsof +D /tmp/rch_target_<name>'
ssh ubuntu@<host> 'sudo lsof +D /data/projects/<repo>/target_rch_<name>'

Only treat empty lsof results as a low-risk stale-artifact cleanup signal.

6) If rch exec fails at sync time, verify remote mirror ownership

When the canonical worker mirror under /data/projects/<repo> is owned by root or another account, rsync fails with Permission denied or Operation not permitted.

Check:

ssh ubuntu@<host> "stat -c '%U:%G %a %n' /data/projects/<repo>"

Fix:

ssh ubuntu@<host> 'sudo chown -R ubuntu:ubuntu /data/projects/<repo> && sudo chmod 775 /data/projects/<repo>'

After the fix, rerun:

rch diagnose --dry-run "cargo build --release"
rch exec -- cargo build --release

---

Worker Fleet Lifecycle

Discovery and setup

rch workers discover
rch workers discover --probe
rch workers discover --add --yes
rch workers setup --all

Runtime management

rch workers list --speedscore
rch workers capabilities --refresh
rch workers benchmark
rch workers drain <worker> -y
rch workers enable <worker>
rch workers disable <worker> --reason "maintenance" --drain -y

Toolchain/binary synchronization

rch workers sync-toolchain --all
rch workers deploy-binary --all

---

Fleet Deploy/Rollback

rch fleet status
rch fleet deploy --verify
rch fleet deploy --canary 25 --canary-wait 60 --verify
rch fleet rollback --verify
rch fleet history --limit 20

For large fleets:

• Prefer canary first, then full rollout.
• Use --dry-run before disruptive operations.
• Use --drain-first if workers are heavily loaded.

---

Path-Dependency and Multi-Repo Notes

RCH now supports dependency-closure planning and canonical topology handling, but path-based workspaces still require worker-accessible sibling repos.

Recommended checks:

rch diagnose --dry-run "cargo test --workspace"
rch exec -- env CARGO_TARGET_DIR=/tmp/rch_target_<name> cargo test --workspace --no-fail-fast

If remote path dependencies are missing:

• Ensure required sibling repos exist on worker hosts under canonical project roots.
• Re-run rch workers setup --all and then retry the rch exec -- ... command.

---

Transfer Stability (Rsync/Artifact Churn)

If sync fails due to active artifact churn, extend transfer excludes:

[transfer]
exclude_patterns = [
  "target/",
  "target_*/",
  "target-*/",
  ".cargo-target/",
  ".cargo-target-*/",
]

Then reload daemon config:

rch daemon reload
rch config show --sources

Operational note:

• If you need a manual target dir for Rust builds, prefer /tmp/rch_target_<name>.
• If the working tree itself cannot sync because the remote canonical mirror is broken, either repair ownership on the worker or temporarily build from a clean directory under /data/projects, not /tmp, because RCH canonical-root normalization expects /data/projects.

---

Queue and Cancellation Operations

rch queue
rch queue --watch
rch cancel <build-id>
rch cancel --all --yes

Use cancellation when builds are wedged or backlog pressure is starving high-priority work.

---

Anti-Patterns

---

Debug Command Pack

RCH_LOG_LEVEL=debug rch diagnose "cargo build --release"
RCH_LOG_LEVEL=debug rch check
rch doctor --json > /tmp/rch-doctor.json
rch --json workers probe --all > /tmp/rch-workers-probe.json
rch daemon logs -n 200

Path Dependencies, Workspaces, and Topology

Contents

• Mental Model
• [The [path_topology] Section](#the-path_topology-section)
• Closure Planner
• Common Failure Patterns
• Strategies
• Diagnostic Commands

The single largest class of "RCH worked yesterday, fails today" incidents is multi-repo workspace topology: a Cargo.toml somewhere has path = "../../other-repo", and either RCH can't decide what to ship or the worker doesn't have the sibling repo at the right place.

This file is the canonical guide. It also covers the [path_topology] config section, which lets you change the canonical roots away from the hard-coded defaults.

---

Mental Model

RCH plans a dependency closure before shipping anything. Three things have to line up:

1. Workspace expansion. RCH walks cargo metadata from the entry manifest, follows every local path = "...", and promotes nested path-deps to their enclosing workspace root. 2. Canonical-root containment. Every root in the closure must live under [path_topology] canonical_root (default /data/projects). If any escapes, RCH refuses to ship the closure (RCH-E016) and falls open. 3. Worker-side mirror. Each closure root must already exist on the chosen worker under the canonical root, with the SSH user able to write to it.

If any of these breaks, you'll see one of:

• [RCH] local (dependency preflight RCH-E0XX: <remediation>) — closure planner refused
• [RCH] local (all workers failed repo convergence checks) — snake-case tag all_workers_failed_convergence; workers couldn't bring repos to needed state
• RCH-E405 Permission denied during file transfer — mirror perms broken
• rsync error mentioning a path under the canonical root that doesn't exist on the worker

---

The [path_topology] Section

Defaults:

[path_topology]
# canonical_root = "/data/projects"   # Where canonical project paths live
# alias_root     = "/dp"              # Optional symlink alias (must point at canonical_root)

Override in ~/.config/rch/config.toml (host-wide) or .rch/config.toml (project-local). Env-var overrides:

export RCH_CANONICAL_PROJECT_ROOT=/srv/projects
export RCH_ALIAS_PROJECT_ROOT=/p

Verify:

rch --json config get path_topology.canonical_root
rch --json config get path_topology.alias_root
rch config show --sources | grep -A2 path_topology

After changing, rch daemon reload (no restart needed for path topology in v1.0.18+).

Why operators change it

• macOS workers where /data doesn't exist
• Multi-tenant boxes where /srv/projects/<tenant> is the right anchor
• Sandbox/test rigs where everything lives under /tmp/rch-fixtures
• Worker mirrors that already exist under a different prefix

Topology error exit codes (visible in worker logs)

When the daemon probes a worker's topology, the probe script can exit with:

Reproduce on the worker:

ssh ubuntu@<host> 'ls -ld /data/projects && [ -L /dp ] && readlink /dp'

---

Closure Planner

The planner (rch_common::dependency_closure_planner) runs before transfer and emits:

DependencyClosurePlan {
  state: Ready | FailOpen,
  entry_manifest_path,
  workspace_root,
  sync_actions: [DependencySyncAction { package_root, manifest_path, package_name, risk, metadata }],
  issues: [DependencyPlanIssue { code, message, risk, diagnostics }],
}

Inspect with:

rch diagnose --dry-run "cargo build --release"   # human form
rch --json diagnose --dry-run "cargo build --release" | jq '.data.dependency_closure'

Risk classes:

• Low — workspace member, in canonical root
• Medium — transitive path dep, in canonical root
• High — risky symlink hop, accepted but flagged
• Critical — outside canonical root or cyclic — closure becomes FailOpen

Recent fixes you should know about

• v1.0.17 (29d0d63): Path-deps that live inside an enclosing workspace are now promoted to the workspace root for transfer. Before, you'd see redundant transfers per crate.
• v1.0.16 (cb80c59): Nested path dependencies promote to the enclosing Cargo workspace root.
• v1.0.16 (61e95d1): Dev-only path dependencies ([dev-dependencies]) are excluded from the runtime closure. If you depend on one for production code, declare it in [dependencies].
• v1.0.16 (877c800): Symlink targets are accepted as valid dependency-scope candidates.
• v1.0.16 (61e95d1): Fail-open fallback is narrowed to policy violations only. A generic cargo metadata error no longer causes silent local fallback — it now surfaces as RCH-E017/E019 and you'll see the reason explicitly.
• v1.0.18 (24580cd): rch diagnose and rch exec now use the configured [path_topology] instead of the compiled-in defaults. If you were on v1.0.17 or earlier and set custom roots, upgrade.

---

Common Failure Patterns

"input resolves outside canonical root"

A path entered rch exec (cwd, target dir, or path-dep) that isn't under the canonical root.

pwd                                   # Where am I?
rch --json config get path_topology   # What's canonical?
rch diagnose --dry-run "cargo check"  # Show normalization decisions

Fix: move the workspace under the canonical root, or extend canonical_root to a parent that contains it.

RCH-E014 Path dependency declared but target dir not found

A Cargo.toml references path = "../sibling" but ../sibling doesn't exist on the host or the worker.

# Show all path-deps in the entry manifest
cargo metadata --no-deps --format-version 1 | jq -r '.packages[].dependencies[] | select(.source==null) | "\(.name) -> \(.path)"'

Fix on the host: clone the sibling under the canonical root. Fix on the worker: same — make sure each sibling repo is checked out at the matching canonical path. Or, if the dep is only used for dev, move it to [dev-dependencies] so the runtime closure excludes it.

RCH-E016 Path dependency violates canonical-root topology

The path-dep target exists, but it resolves outside the canonical root (often via a symlink chain).

realpath ../sibling

Fix: re-anchor the sibling under the canonical root, or update canonical_root to a common ancestor.

all workers failed repo convergence checks (tag: all_workers_failed_convergence)

Daemon picked candidate workers but none could converge their copy of the closure repos to the required state. Check that each worker has every closure root present and writable:

ssh ubuntu@<host> "for r in /data/projects/{repo_a,repo_b,sibling}; do test -w \$r && echo OK \$r || echo FAIL \$r; done"

If a sibling is missing on the worker:

ssh ubuntu@<host> "git clone --depth 1 git@github.com:org/sibling.git /data/projects/sibling"
ssh ubuntu@<host> "sudo chown -R ubuntu:ubuntu /data/projects/sibling && sudo chmod 775 /data/projects/sibling"
rch workers probe <id>

Permission denied during rsync to /data/projects/<repo>

The mirror was created (or modified) as root and the SSH user can't write to it. Recovery:

ssh ubuntu@<host> 'sudo chown -R ubuntu:ubuntu /data/projects/<repo> && sudo chmod 775 /data/projects/<repo>'

If you see this often, audit who's pushing changes — usually it's a sudo git pull somewhere.

Symlink target not detected (pre-v1.0.16)

If you're on rch ≤ v1.0.15 and your sibling is a symlink, you'll see scope-validation failures. Upgrade the rch CLI: rch update (or rch update --fleet to also update worker binaries).

---

Strategies

• Co-locate. Put every interlinked repo under the same canonical root. This is the fastest setup.
• Workspace-first. Prefer a top-level [workspace] Cargo manifest with members = ["repo_a", "repo_b"] over standalone crates with reciprocal path = deps. Workspaces are first-class.
• Excludes. If a sibling is huge and you only need the build artifacts of a published crate version, move it from path-dep to a normal version-pinned dep.
• Tags. When some workers don't have a sibling and you don't want to mirror it everywhere, tag the projects' workers (tags = ["my-app"]) and require that tag in the project's .rch/config.toml.

---

Diagnostic Commands

rch diagnose --dry-run "cargo test --workspace"           # full pipeline preview
rch --json diagnose --dry-run "cargo test --workspace" | jq '.data.dependency_closure'
RCH_LOG_LEVEL=debug rch exec -- cargo check               # see closure decisions live
rch --json workers probe --all | jq '.data[] | {id, status, last_error}'

For deep debugging of the closure planner output, the dependency_closure_plan field is JSON-serializable (rch_common::dependency_closure_planner::DependencyClosurePlan).

Recovery Playbooks (Symptom → Fix in ≤ 90 Seconds)

Contents

• Playbook A: "My build feels slow"
• Playbook B: "Daemon unavailable"
• Playbook C: "All workers unreachable"
• Playbook D: "All workers at capacity"
• Playbook E: "All workers failed preflight"
• Playbook F: "Permission denied" during sync
• Playbook G: Worker disk pressure
• Playbook H: Hook isn't being called
• Playbook I: Hook timeout / slow classification
• Playbook J: Worker version drift after upgrade
• Playbook K: `rch self-test` won't finish
• Playbook L: TOML/config edit broke things
• When the Playbook Doesn't Apply

Each playbook is structured as: observed signal → one-shot diagnostic → ordered fix attempts → verification. Run them in order. Don't skip the diagnostic. Don't ask the human if the fix is in the playbook.

For unknown symptoms or a stuck loop, fall through to the bottom of this file ("When the Playbook Doesn't Apply") for the escalation packet.

---

Playbook A: "My build feels slow"

Signal: Build wall time is much higher than expected; you suspect rch fell back to local.

# Diagnostic
RCH_VISIBILITY=verbose cargo check 2>&1 | grep -E '^\[RCH\]'

Fix attempts:

1. If you see [RCH] local (...) — open references/FAIL_OPEN.md and look up the parenthetical reason. Apply the matching self-fix. 2. If you see no [RCH] line at all — the hook isn't intercepting. Run rch hook status. If not installed: rch hook install. If installed but not firing: rch hook test. 3. If you see [RCH] remote ... — RCH is doing what it can; the slowness is real. Inspect rch speedscore --all and consider whether the project warrants a faster worker.

Verify:

rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check --workspace --all-targets 2>&1 | tail -5

The summary line should say [RCH] remote <worker> (...).

---

Playbook B: "Daemon unavailable"

Signal: [RCH] local (daemon unavailable) or RCH-E500 / RCH-E502.

# Diagnostic
rch --json daemon status 2>&1 | head -20
ls -la "${XDG_RUNTIME_DIR:-/tmp}/rch/" 2>&1

Fix attempts:

1. If which rchd is empty → daemon binary missing. Install rch (see project README). 2. If autostart cooldown is recent → wait 5–10 seconds and retry the original command. 3. If autostart lock is held by no process → it's stale. Ask user authorization, then rm "${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.lock". 4. Foreground spawn to surface the real error: rch daemon start. Read its stderr. 5. If rch daemon start succeeds but the hook still doesn't see the daemon, check socket consistency: rch --json config get general.socket_path and rch --json daemon status | jq '.data.socket_path' must match. If they don't: rch daemon restart -y.

Verify: rch check returns ready.

---

Playbook C: "All workers unreachable"

Signal: [RCH] local (all_workers_unreachable) or RCH-E100 / RCH-E101 / RCH-E108.

# Diagnostic
rch --json workers probe --all | jq '.data[] | {id, status, last_error}'

Fix attempts (per failing worker):

1. SSH directly: ssh -v -i <identity_file> ubuntu@<host> 'echo OK'. The first error you see is the real one. 2. Auth error → check identity_file permissions (chmod 600), key on agent (ssh-add -l), and authorized_keys on the worker. 3. Connection refused → sshd not running on worker, or wrong port. 4. DNS / network unreachable → host moved or networking broken. 5. Host key changed → with explicit user authorization, refresh the entry. Compare fingerprints first.

Verify: rch workers probe <id> returns ok.

---

Playbook D: "All workers at capacity"

Signal: [RCH] local (all workers at capacity) (snake-case tag in JSON: all_workers_busy) or RCH-E204 repeatedly.

Fix attempts (in order, escalating):

1. Verify queueing is on — RCH_QUEUE_WHEN_BUSY is already enabled by default in current rch (only set =0 to disable). If you still see all workers at capacity, queueing didn't help — the wait timed out. 2. Bump the wait timeout for the next invocation: RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120 <your-command>. 3. Check whether one worker is hot and others are cold (uneven distribution): rch --json status --workers | jq '.data.daemon.workers[] | {id, used: .used_slots, total: .total_slots}'. 4. If aggregate capacity is the problem, raise total_slots on top workers in ~/.config/rch/workers.toml, then rch daemon reload. 5. Watch the backlog drain: rch queue --watch (this is an interactive polling view, not a TUI — Ctrl-C exits cleanly).

Verify: Next rch exec lands [RCH] remote <worker> (...).

---

Playbook E: "All workers failed preflight"

Signal: [RCH] local (all workers failed preflight checks) (snake-case tag: all_workers_failed_preflight) or RCH-E013..024 / RCH-E205 / RCH-E305.

# Diagnostic
rch diagnose --dry-run "<the command>" 2>&1 | head -50
RCH_LOG_LEVEL=debug rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_diag" cargo check 2>&1 | tail -40

Fix attempts:

1. If a path-dep error (RCH-E013..016) → see PATH_DEPENDENCIES.md for the exact code. 2. If RCH_TOPOLOGY_ERR_CANONICAL_NOT_DIRECTORY or _ALIAS_NOT_SYMLINK in worker stderr → fix the topology on the worker (/data/projects should be a directory; /dp should be a symlink to it). 3. If RCH-E205 Worker missing toolchain → rch workers sync-toolchain --all. 4. If RCH-E305 Remote working dir error → typically mirror perms broken; see Playbook F.

Verify: rch diagnose --dry-run "<command>" reports Ready for the closure plan.

---

Playbook F: "Permission denied" during sync

Signal: [RCH] local (remote execution failed) followed by stderr lines mentioning Permission denied or Operation not permitted under /data/projects/<repo>.

# Diagnostic
ssh ubuntu@<worker> "stat -c '%U:%G %a %n' /data/projects/<repo>"

Fix:

ssh ubuntu@<worker> 'sudo chown -R ubuntu:ubuntu /data/projects/<repo> && sudo chmod 775 /data/projects/<repo>'
rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check

If you see this on multiple repos, audit who's doing sudo git clone or running things as root in /data/projects.

---

Playbook G: Worker disk pressure

Signal: [RCH] remote <worker> failed [RCH-E210] (or E211/E215/E216/E217); or rch status calls out a worker as critical.

See the dedicated DISK_AND_PRESSURE.md. TL;DR:

rch --json status --workers | jq '.data.daemon.workers[] | select(.pressure_state != "healthy") | {id, pressure_state, pressure_reason_code, pressure_disk_free_gb}'
ssh ubuntu@<worker> 'df -h / /tmp && free -h && cat /proc/pressure/memory'
ssh ubuntu@<worker> 'sbh status --json'      # let sbh handle it

If sbh isn't installed on the worker: install it, or escalate. Don't rm -rf build artifacts blindly — other agents might be mid-build.

---

Playbook H: Hook isn't being called

Signal: Builds run locally; [RCH] summary lines never appear; the hook seems silent.

# Diagnostic
rch hook status --json
rch agents status --json
which rch
cat ~/.claude/settings.json 2>/dev/null | jq '.hooks.PreToolUse'

Fix attempts:

1. Hook not installed → rch hook install (or rch agents install-hook claude-code). 2. Hook command path is wrong → rch hook install re-resolves to the current absolute path. 3. The Claude Code session predates the hook install → restart Claude Code (the harness only loads hooks on startup). 4. Hook installed but fires for a different agent → confirm rch agents list --json includes the agent you're running under.

Verify:

rch hook test
printf '%s\n' '{"tool_name":"Bash","tool_input":{"command":"cargo check"}}' | rch

The second should produce a JSON updatedInput rewriting to rch exec -- cargo check.

---

Playbook I: Hook timeout / slow classification

Signal: Claude Code reports the hook timed out, or the hook is logging classification budget warnings.

# Diagnostic
RCH_LOG_LEVEL=debug printf '%s\n' '{"tool_name":"Bash","tool_input":{"command":"cargo build"}}' | rch 2>&1 | grep -iE 'budget|classif'

Likely causes:

• Massive Cargo.toml/metadata and the closure preflight is slow → cache should warm; if not, the cache may be invalid: rm -rf ~/.cache/rch/classify_cache_v* (with explicit user authorization).
• A misbehaving CI invocation is running rch in a loop and starving the daemon.

If the hook's compilation decision exceeds 5ms, that's a budget regression worth filing.

---

Playbook J: Worker version drift after upgrade

Signal: New rch features behave inconsistently across workers; rch fleet status shows mixed versions.

rch fleet status --json    # exact JSON shape varies by version; inspect first
rch fleet verify           # human-readable comparison of installed binaries

Fix:

rch fleet deploy --canary 25 --canary-wait 60 --verify
# observe output, then
rch fleet deploy --verify           # full rollout
rch fleet verify                    # confirm uniform

If rollback is needed: rch fleet rollback --verify.

For a single worker, rch fleet deploy --worker <id> --verify (deploy, single host).

---

Playbook K: rch self-test won't finish

Signal: rch self-test --all runs forever or returns no output for minutes.

# Diagnostic
rch self-test --worker <id> --timeout 120 --debug 2>&1 | tail -30
rch self-test history --limit 5 --json

Fix attempts:

1. Try a single worker with --timeout 120 --debug. If that works, the --all mode is hitting a slow worker — narrow down with rch speedscore --all. 2. If self-test hangs against any single worker, that worker has a deeper problem. rch workers probe <id>, then drain it (rch workers drain <id>) and continue without it. 3. Capture a full doctor report: rch doctor --json > /tmp/rch-doctor.json. The pre-v1.0.16 hang bug bd-w5r9 is fixed; if you reproduce on current rch, escalate with the doctor output.

---

Playbook L: TOML/config edit broke things

Signal: Things were working; you edited ~/.config/rch/config.toml or workers.toml; now nothing works.

rch config validate
rch config doctor
rch config show --sources
rch config diff                      # what differs from defaults

If rch config validate flags an issue, fix the indicated line and rch daemon reload. If you can't see what's wrong, git diff of the config (if you keep it under version control) is your friend. Otherwise rch config init writes a clean baseline you can compare against.

---

When the Playbook Doesn't Apply

Capture the escalation packet before pinging the human:

mkdir -p /tmp/rch-escalation && cd /tmp/rch-escalation
rch doctor --json                         > doctor.json 2>&1 || true
rch --json daemon status                  > daemon-status.json 2>&1 || true
rch --json workers probe --all            > workers-probe.json 2>&1 || true
rch --json status --workers --jobs        > status.json 2>&1 || true
rch --json queue                          > queue.json 2>&1 || true
rch --json config show --sources          > config.json 2>&1 || true
rch --json hook status                    > hook-status.json 2>&1 || true
rch --json agents status                  > agents-status.json 2>&1 || true
rch --json self-test --all --timeout 120  > selftest.json 2>&1 || true
rch daemon logs -n 500                    > daemon.log 2>&1 || true
{ echo "rch=$(rch --version)"; echo "rchd=$(rchd --version 2>/dev/null || true)"; \
  echo "daemon-reported-version=$(rch --json status 2>/dev/null | jq -r '.data.daemon.daemon.version // ""')"; } > versions.txt
ls -lah

Then surface a one-paragraph synthesis to the human: what symptom you saw, what you tried (with playbook letter), where the packet lives. Don't ship a wall of text; the packet is the data, your message is the signal.

Self-Healing: Daemon Autostart, Cooldown, Hook Bootstrap

Contents

• [What's in [self_healing]](#whats-in-self_healing)
• What `try_auto_start_daemon` Actually Does
• Symptoms and What They Mean
• Hook Re-installation (`daemon_installs_hooks`)
• Self-Test as Continuous Verification
• Auto-Start Knobs You Can Tune
• Agent Operating Rules

RCH ships a self-healing layer so the hook can recover from a crashed daemon without operator intervention. Understanding how it works lets agents avoid step-on-step interactions and lets you diagnose when self-healing is masking a real bug.

---

What's in [self_healing]

[self_healing]
hook_starts_daemon = true       # Hook will spawn rchd if socket is unreachable
daemon_installs_hooks = true    # Daemon will reinstall missing PreToolUse hooks
auto_start_timeout_secs = 3     # How long to wait for the socket after spawn (default: 3)
auto_start_cooldown_secs = 30   # Minimum gap between consecutive autostart attempts (default: 30)

(Field names match rch_common::SelfHealingConfig. Defaults verified against default_autostart_cooldown_secs() and default_autostart_timeout_secs() in rch-common/src/types.rs at time of writing.)

If hook_starts_daemon = false, you're back to manual: every rch exec that finds the socket missing falls open with [RCH] local (daemon unavailable).

---

What try_auto_start_daemon Actually Does

Code path: rch/src/hook.rs::try_auto_start_daemon. Step-by-step:

1. Bail if disabled. Err(AutoStartError::Disabled) if hook_starts_daemon = false. 2. Probe an existing socket. If the socket file exists, do a 300ms connect + ping:

• Healthy: return Ok(()). We're done.
• Stale: remove the socket and continue.

3. Check cooldown. Read ${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.cooldown. If the elapsed seconds since the last attempt is less than auto_start_cooldown_secs, bail with Err(AutoStartError::CooldownActive(elapsed, required)). 4. Acquire cross-process lock. Open-with-create-new on ${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.lock. If another process holds it, bail with Err(AutoStartError::LockHeld). 5. Write cooldown stamp. Persist current epoch seconds to the cooldown file. 6. Find the binary. which_rchd_path() checks the directory of the current rch executable first, then PATH. Fail with BinaryNotFound if absent. 7. Spawn. nohup <rchd> with stdio nulled. Hand off to background. 8. Wait for socket. Up to auto_start_timeout_secs, polling for the socket to come up. On timeout: Err(AutoStartError::Timeout). 9. Drop the lock (RAII via AutoStartLock). Cooldown stamp persists.

So, in steady state, only one agent at a time gets to spawn the daemon, and after either success or failure no agent retries for auto_start_cooldown_secs.

---

Symptoms and What They Mean

Many agents see [RCH] local (daemon unavailable) simultaneously

Either:

• The daemon crashed and one agent is currently in the cooldown/lock window, holding off the others — wait auto_start_cooldown_secs and try again
• rchd binary is missing or not in PATH — which rchd && rchd --version

Diagnose:

ls -la "${XDG_RUNTIME_DIR:-/tmp}/rch/"
cat "${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.cooldown" 2>/dev/null   # epoch seconds
date +%s                                                                 # compare
which rchd

Repeated cooldown-active errors but daemon never starts

rchd is failing to come up at all (broken binary, port in use, OOM at startup, missing $HOME for state, etc.). The cooldown is hiding the real spawn failure. Bypass the autostart path:

rch daemon start                 # foreground spawn — surfaces the real error
# or
rchd --foreground                # depending on rchd flags
journalctl --user -u rch.daemon -n 100 2>/dev/null    # if running as systemd user unit
rch daemon logs -n 200

LockHeld perpetually

A previous rch invocation crashed while holding the lock. The lock file is ${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.lock. If no process exists for it (fuser returns nothing), it's safe to remove. Prefer asking the user before doing so — it's a cleanup operation, not strictly an "rch" command. If authorized:

fuser "${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.lock" 2>/dev/null && echo "still held" || rm "${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.lock"

Socket exists but rch check says not ready

A previous rchd died without cleanup. try_auto_start_daemon will detect a stale socket on next run and remove it. To force the issue:

rch daemon restart -y
rch --json daemon status | jq '.'

---

Hook Re-installation (daemon_installs_hooks)

When daemon_installs_hooks = true, the daemon notices on startup whether the PreToolUse hook is registered for any detected agent (Claude Code, Codex, Gemini, etc.). If it's missing for the agent that owns the user session, it will reinstall.

This is mostly invisible. To inspect what the daemon thinks:

rch agents list --json
rch agents status
rch hook status --json

If you've explicitly uninstalled the hook (rch hook uninstall), set daemon_installs_hooks = false to avoid re-install on next daemon start.

---

Self-Test as Continuous Verification

rch self-test runs the full classifier → daemon → worker → transfer → exec → result loop end-to-end. Schedule it (or just run it before believing the system is healthy):

rch self-test --all                               # one round on every worker
rch self-test --worker css                        # single worker
rch self-test --worker css --debug --timeout 600  # debug build, longer timeout
rch self-test --scheduled                         # use scheduled config
rch self-test status                              # last run + schedule
rch self-test history --limit 10

If rch self-test --all ever hangs indefinitely, that's the symptom of a known prior bug (bd-w5r9) — kill it and capture rch doctor --json. Recent versions are timeout-bounded; if you reproduce on v1.0.18+, escalate.

---

Auto-Start Knobs You Can Tune

For test rigs or environments where you want autostart off:

[self_healing]
hook_starts_daemon = false

For environments where the autostart cooldown is too aggressive (rare):

[self_healing]
auto_start_cooldown_secs = 2
auto_start_timeout_secs = 5

Verify after editing:

rch config show --sources | grep -A6 self_healing
rch daemon reload

---

Agent Operating Rules

• Do trust the autostart path on first failure. Re-issue the original rch exec once after auto_start_cooldown_secs.
• Do check which rchd if autostart never recovers.
• Don't delete hook_autostart.cooldown in a tight loop — that's defeating a safety. Either wait or use rch daemon start directly (which doesn't go through the cooldown path).
• Don't kill rchd to "fix" things while builds are active. Use rch daemon restart -y only after confirming rch queue is empty.
• Don't ask the human "should I restart the daemon?" — the daemon's own self-healing is the right answer for transient failures. Only escalate after rch doctor --json plus rch daemon logs -n 200 show an actionable upstream problem.

SSH Key Recovery on a Fresh Host

Contents

• Diagnostic Signature
• Step 1 — Confirm the keys are actually missing
• Step 2 — Search exhaustively before giving up
• Step 3 — Recover from a sibling host (if reachable)
• Step 4 — If recovery isn't possible, disable the affected workers and continue
• Step 5 — Avoid the recurrence
• What NOT to Do

This is the single most common cause of "all workers unhealthy" in the wild: a workers.toml checked in (or copied) from another machine references SSH identity files that don't exist on the host you're on right now. Every probe returns RCH-E100/RCH-E101. Agents historically stall here and ask the human.

Don't ask. Recover. Here's the playbook.

---

Diagnostic Signature

Probing 9 worker(s)...
  vmi1149989 ubuntu@212.90.121.76... ✗ Connection failed:
    RCH-E100
      × SSH connection failed for ubuntu@212.90.121.76
      help: SSH Troubleshooting:
            1. Verify host/user/key in workers.toml.
            2. Try a manual connection with verbose logs:
            ssh -vvv -i "/home/ubuntu/.ssh/contabo_vps_ed25519"
                       ubuntu@212.90.121.76
            Run 'rch doctor' for comprehensive SSH diagnostics.

…repeated for every worker, and ls -la ~/.ssh/ shows the referenced key file is missing.

---

Step 1 — Confirm the keys are actually missing

# Pull every identity_file referenced by workers.toml
grep -h '^[[:space:]]*identity_file' ~/.config/rch/workers.toml \
  | sed 's/.*=[[:space:]]*"\(.*\)"/\1/' | sort -u | while read -r p; do
      r="${p/#~/$HOME}"
      printf '%s -> %s\n' "$p" "$([[ -r "$r" ]] && echo OK || echo MISSING)"
    done

Anything tagged MISSING is a candidate.

---

Step 2 — Search exhaustively before giving up

The key may exist on this host under a different path:

# Common locations
for k in $(grep -h identity_file ~/.config/rch/workers.toml | sed 's/.*"\([^"]*\)"/\1/' | xargs -n1 basename | sort -u); do
  echo "## $k"
  find ~/.ssh /root/.ssh 2>/dev/null -name "$k*"
  # Then broader (only if needed; this is slower):
  # sudo find / -xdev -name "$k*" -not -path '*/proc/*' -not -path '*/sys/*' 2>/dev/null
done

Also check ssh-agent for in-memory keys (their fingerprints compared to authorized_keys on the worker can confirm a working key exists even if the file does not):

ssh-add -l
ssh-add -L           # the actual public keys

If a key is in the agent, you don't strictly need the file — rch will still fail because it expects a file path, but you can use this to:

• copy the matching private key from another host you can reach
• or, with explicit user authorization, generate a new key and add it to the workers' authorized_keys

---

Step 3 — Recover from a sibling host (if reachable)

If you have any host that does have the keys (typically the original developer machine or another worker), grab them via SSH:

# Replace <SOURCE> with the host name and adjust paths
SOURCE=trj
for k in contabo_vps_ed25519 thinkstation2_ed25519; do
  if ssh "$SOURCE" "test -r ~/.ssh/$k"; then
    scp "$SOURCE:~/.ssh/$k" "~/.ssh/$k"
    scp "$SOURCE:~/.ssh/${k}.pub" "~/.ssh/${k}.pub" 2>/dev/null || true
    chmod 600 "~/.ssh/$k"
    chmod 644 "~/.ssh/${k}.pub" 2>/dev/null || true
  else
    echo "$k not on $SOURCE either"
  fi
done
rch workers probe --all

---

Step 4 — If recovery isn't possible, disable the affected workers and continue

Don't let one missing key block all work. Disable just the unreachable workers and proceed with whatever's left:

# Disable any worker whose probe surfaces a key/auth error.
# Probe response shape: .data is a flat array of {id, host, status, latency_ms?, error?}
rch --json workers probe --all \
  | jq -r '.data[] | select(.status != "ok" and (.error // "" | test("Permission denied \\(publickey|no such identity|identity_file"))) | .id' \
  | while read -r id; do
      [[ -z "$id" ]] && continue
      echo "disabling $id (missing key)"
      rch workers disable "$id" --reason "missing ssh key on host" --drain -y
    done

# Now re-check
rch workers probe --all
rch check

When the keys are recovered later, re-enable:

rch workers enable <id>

---

Step 5 — Avoid the recurrence

Bake a check into the host's provisioning. Two ergonomic options:

Option A — pre-flight in the shell rc

In ~/.bashrc (or equivalent), after any rch-related work:

rch_keys_check() {
  local missing=0
  while read -r p; do
    p="${p/#~/$HOME}"
    [[ -r "$p" ]] || { echo "missing rch key: $p"; missing=1; }
  done < <(grep -h '^[[:space:]]*identity_file' ~/.config/rch/workers.toml 2>/dev/null \
             | sed 's/.*=[[:space:]]*"\(.*\)"/\1/')
  return $missing
}

Then rch_keys_check && rch check.

Option B — rch config doctor upstream

rch config validate doesn't currently check identity_file existence. This is filed as upstream UX feedback (see "Upstream Bugs to File" in RECOVERY_PLAYBOOKS.md). When that lands, rch config doctor will surface missing keys directly.

---

What NOT to Do

• Do not delete `~/.config/rch/workers.toml` to "start over". You'll lose every host/priority/tag that was carefully configured. If you must reset, copy it first: cp ~/.config/rch/workers.toml{,.bak}.
• Do not run `rch workers discover --add --yes` blindly to "rediscover" — it will add freshly-discovered hosts but won't remove the broken ones, leaving a doubly-confused config.
• Do not pause and ask the human unless you've tried Steps 1–4 and the keys are genuinely lost. The first three steps are seconds; only Step 3 might be slow if SCP transfer is large.
• Do not generate a brand-new key and try to push it to authorized_keys on every worker without explicit user authorization. That's a privilege change, not a recovery.

SSH Tuning for RCH

Contents

• Defaults Worth Knowing
• ControlMaster Default Is OFF
• Keepalives for Long Builds
• Connect / Command Timeouts
• Authentication Failure Triage
• Retryable vs Fatal Transport Errors
• Known-Hosts Policy
• End-to-End Probe Recipes
• Multi-Hop / Jump Host
• Quick Knob Cheat Sheet

Most "intermittent worker failure" reports trace to SSH transport quirks: stale ControlMaster sockets, broken keepalives, or auth misconfiguration. RCH gives you knobs for all of them.

---

Defaults Worth Knowing

rch_common::ssh::SshOptions::default():

connect_timeout       = 10s
command_timeout       = 300s
server_alive_interval = None   (OpenSSH default; keepalive disabled)
control_persist_idle  = None   (uses ControlPersist=yes when control_master is true)
control_master        = false  (default OFF — see history below)
known_hosts           = Add    (add unknown hosts to ~/.ssh/known_hosts)

MAX output capture per command: 10 MB (stdout/stderr each). Larger output is truncated to prevent OOM.

---

ControlMaster Default Is OFF

Recent change: commit 464a25b flipped the default to control_master = false because stale local control sockets were poisoning otherwise healthy connections — particularly painful for multi-agent fleets where one terminating session would orphan a master and the next agent would see hangs.

Opt-in if you really want connection reuse on a single-agent box:

export RCH_SSH_CONTROL_PERSIST_SECS=60       # enables persistence with 60s idle

If you opt in, expect to manually clean stale sockets occasionally:

ls /run/user/$(id -u)/openssh-* ~/.ssh/control-* ~/.ssh/cm-* 2>/dev/null
ssh -O check ubuntu@<host> 2>&1 || true
ssh -O exit ubuntu@<host> 2>&1 || true

A symptom signature for ControlMaster poisoning: probes succeed (rch workers probe --all), but rch exec -- ... hangs at "syncing" or "executing" with no progress for ~30s before failing.

---

Keepalives for Long Builds

cargo build --release for a big workspace can sit idle on the SSH channel while the worker does heavy CPU work and emit nothing for minutes. NAT/firewall idle timers can drop the connection.

export RCH_SSH_SERVER_ALIVE_INTERVAL_SECS=15

This sets ServerAliveInterval=15, equivalent to a heartbeat every 15 seconds. Symptom this fixes: RCH-E105 SSH session terminated unexpectedly mid-build with no other apparent cause.

---

Connect / Command Timeouts

Connect: 10s default. Command: 300s default. The command timeout is the upper bound on a single SSH command (probe, mkdir, exec wrapper). A real build runs through the workers' own pipeline and is bounded by [compilation] build_timeout_sec, not the SSH command timeout.

To raise across the board, set [compilation] build_timeout_sec = 1800 for a 30-minute ceiling.

For ad-hoc one-shot tuning of the daemon's own SSH command timeouts, you'll need to edit the source — there's no env knob today. (If you find yourself wanting one, that's a code change worth filing.)

---

Authentication Failure Triage

RCH-E101 SSH authentication failed is almost always one of:

1. Wrong key in workers config: rch --json config get to inspect the worker's identity_file. 2. Permissions on the key: chmod 600 ~/.ssh/<key>. 3. Agent doesn't have the key: ssh-add ~/.ssh/<key> (if you use ssh-agent). 4. Key not authorized on the worker: ssh -i <key> ubuntu@<host> true reproduces.

RCH-E102 SSH key not found or invalid format — the path in identity_file doesn't exist or isn't a private key. Check ls -la <path> and head -1 <path> (PEM vs OpenSSH).

RCH-E103 SSH host key verification failed — the worker's host key changed (rebuild?). Compare the new fingerprint with what's stored:

ssh-keygen -F <host>             # what we know
ssh-keyscan <host> 2>/dev/null   # what the host claims now

If you trust the new fingerprint, remove the old entry (with explicit user authorization) and let KnownHostsPolicy::Add re-add on next probe.

---

Retryable vs Fatal Transport Errors

is_retryable_transport_error_text (in rch_common/ssh_utils.rs) classifies SSH errors. Retryable signatures include:

• connection reset
• broken pipe
• connection refused
• temporary failure
• transient DNS errors

Fatal:

• auth errors
• host key mismatch
• "no such file" inside the remote command (those are application errors, not transport)

The daemon retries retryable errors with backoff. If you keep seeing the same error on retry, treat it as fatal and triage with rch workers probe <id> and direct ssh -v.

---

Known-Hosts Policy

Three modes (rch_common::ssh::KnownHostsPolicy):

• Strict — production-style, fails closed on unknown
• Add (default) — auto-adds first time
• AcceptAll — testing only

There's no env override today; this is set by the daemon's SSH session builder. If you want strict, you'd have to use a wrapper. For most agent fleets, Add is fine.

---

End-to-End Probe Recipes

Single worker:

rch workers probe css --json
ssh -v -i <identity_file> ubuntu@<host> 'echo OK; df -h / /tmp; free -h'

All workers, parallel:

rch --json workers probe --all | jq '.data[] | {id, status, latency_ms, last_error}'

If a probe says ok but rch exec fails:

RCH_LOG_LEVEL=debug rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_probe" cargo check --quiet 2>&1 | tail -50

Look for the first SSH-shaped error in that tail. The structured remediation field of RCH-E1xx codes will point you to the right knob.

---

Multi-Hop / Jump Host

If your workers sit behind a bastion, set it in ~/.ssh/config and refer to the alias as the worker host:

Host worker-css
  HostName 10.0.0.5
  User ubuntu
  ProxyJump bastion.example.com
  IdentityFile ~/.ssh/id_ed25519_workers

Then [[workers]] uses host = "worker-css". The recent 464a25b fix specifically improved alias-based path topology resolution, so multi-hop aliases work cleanly in v1.0.16+.

---

Quick Knob Cheat Sheet

Telemetry Database Corruption Recovery

Contents

• Symptom Signatures
• Recovery (Safe)
• Verify Integrity Before Acting
• "Telemetry protocol version mismatch"
• Prevention
• When the Recovery Doesn't Stick

The rch telemetry SQLite database lives at ~/.local/share/rch/telemetry/telemetry.db. Like any SQLite file under heavy concurrent write load, it can occasionally corrupt — most often after host crashes, OOM-kills, or disk-full incidents.

This is one of the cliffs agents previously fell off (cass evidence: 30+ incidents under "Telemetry database integrity check failed"). The recovery is mechanical and safe — but the skill never previously documented it.

---

Symptom Signatures

Any of:

• [RCH-E507] Metrics collection error recurring in rch daemon logs
• Telemetry database integrity check failed log lines
• Telemetry protocol version mismatch (related; less common)
• rch speedscore --all returns empty or stale data
• rch self-test history returns empty even though you know runs happened
• database disk image is malformed in daemon stderr

rch doctor may or may not catch this (depends on version).

---

Recovery (Safe)

Telemetry is purely derived data. Losing it loses historical SpeedScores and build durations, but nothing operationally critical.

# 1. Stop the daemon (drains in-flight builds)
rch daemon stop -y

# 2. Move the corrupt db aside (don't delete; you might want it for forensics)
mv ~/.local/share/rch/telemetry/telemetry.db ~/.local/share/rch/telemetry/telemetry.db.broken-$(date +%s)
mv ~/.local/share/rch/telemetry/telemetry.db-wal ~/.local/share/rch/telemetry/telemetry.db-wal.broken-$(date +%s) 2>/dev/null || true
mv ~/.local/share/rch/telemetry/telemetry.db-shm ~/.local/share/rch/telemetry/telemetry.db-shm.broken-$(date +%s) 2>/dev/null || true

# 3. Restart — the daemon recreates the schema on first write
rch daemon start
sleep 2
rch check

Verify:

ls -la ~/.local/share/rch/telemetry/    # new telemetry.db should exist
rch --json daemon status | jq '.data.version'

You'll lose:

• SpeedScore history (rch speedscore --history) — rebuilds on subsequent self-tests
• Self-test history (rch self-test history) — same

You will NOT lose:

• Worker config
• Daemon config
• Hook installs
• Active builds (the daemon stop drains them gracefully)

---

Verify Integrity Before Acting

Before assuming the db is broken, check it directly:

sqlite3 ~/.local/share/rch/telemetry/telemetry.db 'PRAGMA integrity_check;'

If that prints ok, the corruption is elsewhere — probably the daemon process is wedged on a different bug. Capture diagnostics and consider Playbook B in RECOVERY_PLAYBOOKS.md instead.

If it prints any error or hangs, the db is genuinely corrupt — proceed with the move-aside recovery.

---

"Telemetry protocol version mismatch"

This is different — it means a worker is running a rch-wkr whose telemetry schema doesn't match the daemon's. The fix is to redeploy the worker binary:

rch fleet status                    # confirm version drift
rch fleet deploy --canary 25 --canary-wait 60 --verify
rch fleet deploy --verify           # full

If only one worker is affected:

rch fleet deploy --worker <id> --verify

---

Prevention

• Don't kill the daemon while builds are in flight (use rch daemon stop -y, which drains).
• Watch disk pressure on the host (not just on workers — the host runs the daemon and the telemetry db). df -h ~/.local/share should never be near full.
• If you're upgrading rch on the host, restart the daemon afterwards (rch daemon restart -y). The daemon_installs_hooks self-healing covers one direction, but version drift on the daemon binary is independent.

---

When the Recovery Doesn't Stick

If you move the file aside and the new db corrupts again within minutes, you have a deeper issue — disk failure, kernel-level bug, or another process writing to the same file. Capture:

journalctl -k --since "1 hour ago" | grep -iE 'i/o|sata|nvme|memory'
sudo dmesg | tail -50
df -h ~/.local/share/rch/telemetry/
mount | grep "$(stat -c %m ~/.local/share/rch/telemetry/)"

…and escalate. This is filesystem-level; the rch skill has done what it can.

RCH Troubleshooting

Contents

• Diagnostic Flow
• Common Errors
• Debug Mode
• Safe Reset Sequence
• Reading `rch status` Output Correctly
• Daemon Version Drift After Upgrade
• Telemetry Corruption
• "Why did my command run locally?" (Silent Fail-Open)
• See Also

Diagnostic Flow

Compilation running locally instead of remotely?
│
├─ Quick health gate:
│  $ rch check
│  │
│  ├─ Not ready/degraded?
│  │   ├─ Check daemon:
│  │   │  $ rch --json daemon status
│  │   │
│  │   ├─ Check workers:
│  │   │  $ rch workers probe --all
│  │   │
│  │   └─ Check hook install:
│  │      $ rch hook status
│  │
│  └─ Ready?
│      continue below
│
└─ Ready but behavior is wrong?
   ├─ Socket alignment:
   │  $ rch --json config get general.socket_path
   │  $ rch --json daemon status
   │
   ├─ Explain routing decision:
   │  $ rch diagnose "cargo build --release"
   │
   ├─ Validate hook protocol path:
   │  $ rch hook test
   │
   └─ Force direct offload proof:
      $ rch exec -- cargo check --workspace --all-targets

---

Common Errors

Daemon not running / check says not ready

Cause: daemon process absent or startup failure.

rch daemon start
rch --json daemon status
rch daemon logs -n 200

Socket mismatch between config and daemon

Cause: general.socket_path differs from active daemon socket.

rch --json config get general.socket_path
rch --json daemon status
# then align and restart:
rch daemon restart -y

"No workers available" / probe failures

Cause: no workers configured, SSH/auth failures, or workers are disabled/drained.

rch workers list
rch workers probe --all
rch workers discover --probe
rch workers discover --add --yes
rch workers setup --all

"rustup: not found" / "cargo: not found" on worker

Cause: missing toolchain on one or more workers.

rch workers sync-toolchain --all
rch workers capabilities --refresh

If still failing, SSH to the specific worker and validate rustup, cargo, and PATH.

Hook not intercepting

Cause: hook missing, wrong binary path, or command classified as local.

rch hook status
rch hook install
rch hook test
rch diagnose "cargo build --release"

Sync/transfer fails under active target churn

Cause: build artifacts changing during rsync.

# Add target-like excludes in ~/.config/rch/config.toml [transfer].exclude_patterns
rch daemon reload
rch config show --sources

Also inspect the worker directly:

ssh ubuntu@<host> 'df -h / /tmp'
ssh ubuntu@<host> 'du -sh /tmp/rch-* /tmp/rch_target_* 2>/dev/null | sort -h'

If cleanup is needed, verify inactivity first:

ssh ubuntu@<host> 'sudo lsof +D /tmp/rch_target_<name>'

If the directory is inactive, prefer targeted stale-artifact cleanup over broad cache deletion.

Sync fails with Permission denied or Operation not permitted inside /data/projects/<repo>

Cause: the canonical mirror on the worker is not writable by the SSH user. This commonly happens when a repo under /data/projects was created or updated as root.

Check:

ssh ubuntu@<host> "stat -c '%U:%G %a %n' /data/projects/<repo>"

Fix:

ssh ubuntu@<host> 'sudo chown -R ubuntu:ubuntu /data/projects/<repo> && sudo chmod 775 /data/projects/<repo>'

Then retry:

rch exec -- cargo check --workspace --all-targets

rch exec fails open for workdirs outside /data/projects

Cause: canonical-root normalization rejects workdirs outside the configured project root.

Symptoms include errors mentioning input resolves outside canonical root.

Fix:

pwd
rch diagnose --dry-run "cargo build --release"

Then run the build from a workspace under /data/projects. If you need a clean copy for testing, stage it under /data/projects/<temp-repo> instead of /tmp/<temp-repo>.

Worker shows storage pressure even after cleanup

Cause: telemetry lag, large ballast allocation, or active live build churn.

Check:

rch status --workers --jobs
ssh ubuntu@<host> 'df -h / /tmp && free -h'
ssh ubuntu@<host> 'journalctl -u sbh -n 50 --no-pager'

Interpretation:

• If df is healthy but rch status still warns, give telemetry a minute and refresh.
• If /tmp is healthy but / is still low, inspect large project target_* trees under /data/projects.
• If sbh is active but repeatedly logging scan channel saturated or scan timed out, inspect stale build artifacts and verify the host is running the current sbh binary and the narrowed worker config.

Path dependency missing remotely (../.../Cargo.toml)

Cause: required sibling repositories are not available in worker topology.

rch diagnose --dry-run "cargo test --workspace"
rch exec -- env CARGO_TARGET_DIR=/tmp/rch_target_<name> cargo check --workspace --all-targets

Then ensure sibling repos exist on workers under canonical roots and retry.

---

Debug Mode

RCH_LOG_LEVEL=debug rch check
RCH_LOG_LEVEL=debug rch diagnose "cargo test --workspace"
RCH_LOG_LEVEL=debug rch exec -- cargo check --workspace --all-targets

Protocol-level hook test:

RCH_LOG_LEVEL=debug printf '%s\n' \
  '{"tool_name":"Bash","tool_input":{"command":"cargo check"}}' | rch

---

Safe Reset Sequence

rch daemon restart -y
rch config validate
rch config doctor
rch workers probe --all
rch hook status
rch hook test
rch check

If still failing, capture artifacts for escalation:

rch doctor --json > /tmp/rch-doctor.json
rch --json daemon status > /tmp/rch-daemon-status.json
rch --json workers probe --all > /tmp/rch-workers-probe.json

---

Reading rch status Output Correctly

rch status (and rch check) can simultaneously show ✓ RCH is ready (9/9 workers healthy) AND a list of [warning] Circuit opened for worker '<id>' alerts. The alerts are informational — circuit breakers are self-healing once the worker is healthy and the half-open probe succeeds. Don't over-react.

Wrong: "I see warnings — better restart the daemon and reload the config."

Right: rch workers probe --all && rch status --workers --jobs — the alert clears within the next status refresh.

If a circuit doesn't auto-clear after 60 seconds and the underlying probe is healthy, then there's a real bug; capture rch --json daemon status | jq '.data.circuit_breakers' and rch daemon logs -n 100.

---

Daemon Version Drift After Upgrade

Symptom: New rch CLI features behave inconsistently; rch --version differs from the daemon's reported version.

Self-fix (this is safe — never ask first):

The daemon's running version is reported by rch --json status at .data.daemon.daemon.version (the rch --json daemon status endpoint deliberately returns only running/socket/uptime — not version). Compare:

rch --version | awk '{print $2}'
rch --json status | jq -r '.data.daemon.daemon.version'
# If they differ:
rch daemon restart -y                                    # drains in-flight builds gracefully
rch --json status | jq -r '.data.daemon.daemon.version'  # confirm equal

rch daemon restart -y is the documented upgrade path. It drains active builds before stopping. The -y skips the interactive prompt — but it does not skip the drain.

If a worker shows mismatched binary version after a host upgrade:

rch fleet status              # human-readable per-worker status
rch fleet verify              # compare installed vs expected
rch fleet deploy --canary 25 --canary-wait 60 --verify
rch fleet deploy --verify

---

Telemetry Corruption

Symptom: Recurring RCH-E507, Telemetry database integrity check failed, empty rch speedscore --history, daemon log lines mentioning database disk image is malformed.

Self-fix: See references/TELEMETRY_RECOVERY.md. Short version: stop daemon, move ~/.local/share/rch/telemetry/telemetry.db* aside, restart. Telemetry is derived data; you lose history but nothing operational.

---

"Why did my command run locally?" (Silent Fail-Open)

Symptom: rch hook status says installed; rch exec works in isolation; but a particular cargo build invocation runs locally without the rch wrapper. No [RCH] local (...) line appears because RCH_VISIBILITY=none is set, or because the hook never engaged at all.

Self-fix:

1. Force visibility: RCH_VISIBILITY=verbose <your-command>. If you now see [RCH] local (...), follow references/FAIL_OPEN.md to map the reason to a fix. 2. If still no [RCH] line, the hook never fired. Probe the protocol directly:

   .claude/skills/rch/scripts/protocol_test.sh "<your-command>"

If stdout is empty, the classifier is rejecting your command. Common causes: shell pipe (cargo build | tee log), backgrounded with &, env-prefixed in an unusual form. Restructure or use rch exec -- <cmd> directly. 3. If the hook fires but the command still runs locally, the rewrite isn't being honored — check that ~/.claude/settings.json has the right hook command path (rch hook install re-resolves it).

See references/FAIL_OPEN.md for the full taxonomy.

---

See Also

• references/FAIL_OPEN.md — the canonical guide for [RCH] local (...) reasons
• references/ERROR_CODES.md — the full RCH-Exxx catalog
• references/PATH_DEPENDENCIES.md — multi-repo workspace problems
• references/MULTI_AGENT_CONTENTION.md — TOCTOU, fleet deploy races, autostart cooldown
• references/DISK_AND_PRESSURE.md — RCH-E210..217 + sbh handoff
• references/SELF_HEALING.md — autostart cooldown, daemon supervision
• references/SSH_KEY_RECOVERY.md — host-doesn't-have-the-key recovery
• references/SSH_TUNING.md — ControlMaster, keepalives, retry semantics
• references/TELEMETRY_RECOVERY.md — corrupt telemetry.db recovery
• references/MACHINE_INTROSPECTION.md — JSON/schema/capability surfaces
• references/RECOVERY_PLAYBOOKS.md — symptom→fix in ≤90s
• scripts/auto_recover.sh — heuristic, dry-run-by-default recovery
• scripts/worker_disk_triage.sh — read-only disk report per worker
• scripts/protocol_test.sh — probe the hook protocol directly
• scripts/multi_agent_safety.sh — flock wrapper for fleet ops
• scripts/mine_rch_history.sh — search prior incidents in agent session history