Pueue Job Orchestration

Pueue Job Orchestration

Universal CLI telemetry layer and job management — every command routed through pueue gets precise timing, exit code capture, full stdout/stderr logs, environment snapshots, and callback-on-completion.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

Overview

Pueue is a Rust CLI tool for managing shell command queues. It provides:

• Daemon persistence - Survives SSH disconnects, crashes, reboots
• Disk-backed queue - Auto-resumes after any failure
• Group-based parallelism - Control concurrent jobs per group
• Easy failure recovery - Restart failed jobs with one command
• Full telemetry - Timing, exit codes, stdout/stderr logs, env snapshots per task

When to Route Through Pueue

When to Use This Skill

Use this skill when the user mentions:

Quick Reference

Check Status

# Local
pueue status

# Remote (BigBlack)
ssh bigblack "~/.local/bin/pueue status"

Queue a Job

# Local (with working directory)
pueue add -w ~/project -- python long_running_script.py

# Local (simple)
pueue add -- python long_running_script.py

# Remote (BigBlack)
ssh bigblack "~/.local/bin/pueue add -w ~/project -- uv run python script.py"

# With group (for parallelism control)
pueue add --group p1 --label "BTCUSDT@1000" -w ~/project -- python populate.py --symbol BTCUSDT

Monitor Jobs

pueue follow <id>         # Watch job output in real-time
pueue log <id>            # View completed job output
pueue log <id> --full     # Full output (not truncated)

Manage Jobs

pueue restart <id>         # ⚠ Creates NEW task (see warning below)
pueue restart --in-place <id>  # Restarts task in-place (no new ID)
pueue restart --all-failed # ⚠ Restarts ALL failed across ALL groups
pueue kill <id>            # Kill running job
pueue clean                # Remove completed jobs from list
pueue reset                # Clear all jobs (use with caution)

CRITICAL WARNING — `pueue restart` semantics:

pueue restart <id> does NOT restart the task in-place. It creates a brand new task with a new ID, copying the command from the original. The original stays as Done/Failed. This causes exponential task growth when used in loops or by autonomous agents. In a 2026-03-04 incident, agents calling pueue restart on failed tasks grew 60 jobs to ~12,800.

• Use `--in-place` if you truly need to restart: pueue restart --in-place <id>
• Verify before restart: Read pueue log <id> to check if the failure is persistent (missing data, bad args) — retrying will never help
• Never use `--all-failed` without --group filter — it restarts every failed task across ALL groups

Host Configuration

Core Workflows

1. Queue Single Remote Job

# Step 1: Verify daemon is running
ssh bigblack "~/.local/bin/pueue status"

# Step 2: Queue the job
ssh bigblack "~/.local/bin/pueue add --label 'my-job' -- cd ~/project && uv run python script.py"

# Step 3: Monitor progress
ssh bigblack "~/.local/bin/pueue follow <id>"

2. Batch Job Submission (Multiple Symbols)

For rangebar cache population or similar batch operations:

# Use the pueue-populate.sh script
ssh bigblack "cd ~/rangebar-py && ./scripts/pueue-populate.sh setup"   # One-time
ssh bigblack "cd ~/rangebar-py && ./scripts/pueue-populate.sh phase1"  # Queue Phase 1
ssh bigblack "cd ~/rangebar-py && ./scripts/pueue-populate.sh status"  # Check progress

3. Configure Parallelism Groups

# Create groups with different parallelism limits
pueue group add fast      # Create 'fast' group
pueue parallel 4 --group fast  # Allow 4 parallel jobs

pueue group add slow
pueue parallel 1 --group slow  # Sequential execution

# Queue jobs to specific groups
pueue add --group fast -- echo "fast job"
pueue add --group slow -- echo "slow job"

4. Handle Failed Jobs

# Check what failed
pueue status | grep Failed

# View error output FIRST (distinguish transient vs persistent)
pueue log <id>

# Restart specific job IN-PLACE (no new task created)
pueue restart --in-place <id>

# ⚠ NEVER blindly restart all failed — classify failures first
# pueue restart --all-failed  # DANGEROUS: no group filter, creates duplicates

Failure classification before restart: Exit code 1 (app error) may be persistent (missing data, bad args — will never succeed). Exit code 137 (OOM) or 143 (SIGTERM) may be transient. Always read pueue log before restarting.

Troubleshooting

Priority Scheduling (--priority)

Higher priority number = runs first when a queue slot opens:

# Urgent validation (runs before queued lower-priority jobs)
pueue add --priority 10 -- python validate_critical.py

# Normal compute (default priority is 0)
pueue add -- python train_model.py

# Low-priority background task
pueue add --priority -5 -- python cleanup_logs.py

Priority only affects queued jobs waiting for an open slot. Running jobs are not preempted.

Per-Task Environment Override (pueue env)

Inject or override environment variables on stashed or queued tasks:

# Create a stashed job
JOB_ID=$(pueue add --stashed --print-task-id -- python train.py)

# Set environment variables (NOTE: separate args, NOT KEY=VALUE)
pueue env set "$JOB_ID" BATCH_SIZE 64
pueue env set "$JOB_ID" LEARNING_RATE 0.001

# Enqueue when ready
pueue enqueue "$JOB_ID"

Syntax: pueue env set <id> KEY VALUE -- the key and value are separate positional arguments.

Constraint: Only works on stashed/queued tasks. Cannot modify environment of running tasks.

Relationship to mise.toml `[env]`: mise [env] remains the SSoT for default environment. Use pueue env set only for one-off overrides (e.g., hyperparameter sweeps) without modifying config files.

Blocking Wait (pueue wait)

Block until tasks complete -- simpler than polling loops for scripts:

# Wait for specific task
pueue wait 42

# Wait for all tasks in a group
pueue wait --group mygroup

# Wait for ALL tasks across all groups
pueue wait --all

# Wait quietly (no progress output)
pueue wait 42 --quiet

# Wait for tasks to reach a specific status
pueue wait --status queued

Script Integration Pattern

# Queue -> wait -> process results
TASK_ID=$(pueue add --print-task-id -- python etl_pipeline.py)
pueue wait "$TASK_ID" --quiet
EXIT_CODE=$(pueue status --json | jq -r ".tasks[\"$TASK_ID\"].status.Done.result" 2>/dev/null)
if [ "$EXIT_CODE" = "Success" ]; then
    echo "Pipeline succeeded"
    pueue log "$TASK_ID" --full
else
    echo "Pipeline failed"
    pueue log "$TASK_ID" --full >&2
fi

---

Companion CLI Tools (bigblack)

Four lightweight CLI tools complement pueue for monitoring and notifications. All installed on bigblack at /usr/local/bin/ (noti, mprocs) or via apt (ntfy, task-spooler).

noti — Wrap Any Command with Telegram Notification

Best for one-off "notify me when this finishes" workflows. Native Telegram support.

# Wrap any command — sends Telegram when it finishes
noti -g mise run kintsugi:catchup

# With execution time in message
noti -g -e python long_script.py

# Custom title/message
noti -g -t "Deploy done" -m "bigblack updated" mise run deploy:bigblack

Config: ~/.config/noti/noti.yaml + env vars NOTI_TELEGRAM_TOKEN, NOTI_TELEGRAM_CHAT_ID, NOTI_DEFAULT=telegram in ~/.bashrc.

ntfy — Push Notifications from Scripts/Callbacks

Best for programmatic notifications from scripts, pueue callbacks, or curl one-liners.

# One-liner from any script
ntfy pub --title "Backfill done" odb-alerts "BTCUSDT@500 complete"

# With priority and tags
ntfy pub --priority high --tags "warning" odb-alerts "Job failed"

# curl variant (works anywhere, no binary needed)
curl -d "Backup finished" ntfy.sh/my-topic

# Subscribe to topic (poll mode)
ntfy sub --poll odb-alerts

Pueue callback integration — add to ~/.config/pueue/pueue.yml:

callback: 'ntfy pub --title "pueue: {{label}}" --tags "{{status}}" odb-alerts "Task #{{id}} {{status}} ({{command}})"'

mprocs — Multi-Process TUI

Best for interactive SSH sessions watching multiple processes side-by-side.

# Watch multiple pueue jobs in split panes
mprocs "pueue follow 3" "pueue follow 7" "pueue follow 8"

# Monitor services
mprocs "journalctl -fu opendeviationbar-sidecar" "journalctl -fu opendeviationbar-kintsugi"

Requires interactive TTY (SSH directly, not via scripts).

task-spooler (tsp) — Simplest Job Queue

Best for quick ad-hoc sequential/parallel tasks without pueue group setup.

# Queue tasks (default: 1 parallel)
tsp python train.py
tsp python validate.py

# Set parallel slots
tsp -S 3

# List queue
tsp -l

# View job output
tsp -c 0

# Combine with noti
tsp bash -c 'python train.py && noti -g -m "training done"'

Tool Selection Guide

Need to...
├── Get notified when a command finishes? → noti -g <command>
├── Send notification from a script/callback? → ntfy pub topic "msg"
├── Watch multiple logs/processes live? → mprocs "cmd1" "cmd2"
├── Quick sequential queue (no groups needed)? → tsp <command>
└── Persistent queue with groups, priorities? → pueue add <command>

---

Deep-Dive References

---

Pueue vs Temporal: When to Use Which

For structured, repeatable workflows needing durability and dedup, consider Temporal (pip install temporalio, brew install temporal):

Recommendation: Keep pueue for ad-hoc work. Migrate structured pipelines to Temporal when dedup and retry policies matter. See devops-tools:distributed-job-safety for invariants that apply to both.

---

Related

• Hook: itp-hooks/posttooluse-reminder.ts - Reminds to use Pueue for detected long-running commands
• Reference: Pueue GitHub
• Companion tools: noti (command wrapper → Telegram), ntfy (pub-sub notifications), mprocs (multi-process TUI), task-spooler (minimal job queue)
• SOTA Alternative: Temporal — durable workflow orchestration with built-in dedup, retry, visibility
• Issue: rangebar-py#77 - Original implementation
• Issue: rangebar-py#88 - Production deployment lessons

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure. 2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match. 3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

Skill: Pueue Job Orchestration

Callback Hooks & Scheduling

Callback Hooks (Completion Notifications)

Pueue fires a callback command on every task completion. Configure in pueue.yml:

daemon:
  callback: 'curl -s -X POST https://hooks.example.com/pueue -d ''{"id":{{id}},"result":"{{result}}","exit_code":{{exit_code}},"command":"{{command}}"}'''
  callback_log_lines: 10 # Lines of stdout/stderr available in {{output}}

Template Variables (14 total, Handlebars syntax)

Production Examples

# File-based sentinel (for script polling)
callback: "echo '{{id}}:{{result}}:{{exit_code}}' >> /tmp/pueue-completions.log"

# Telegram notification
callback: "curl -s 'https://api.telegram.org/bot${BOT_TOKEN}/sendMessage?chat_id=${CHAT_ID}&text=Job%20{{id}}%20{{result}}%20(exit%20{{exit_code}})'"

# Conditional alert (only on failure)
callback: "/bin/bash -c 'if [ \"{{result}}\" != \"Success\" ]; then echo \"FAILED: {{command}}\" | mail -s \"Pueue Alert\" user@example.com; fi'"

Config File Location (Platform Difference)

See Pueue Config Reference for all settings.

---

Delayed Scheduling (--delay)

Queue a job that starts after a specified delay:

# Relative time
pueue add --delay 3h -- python heavy_computation.py

# Natural language
pueue add --delay "next wednesday 5pm" -- python weekly_report.py

# RFC 3339
pueue add --delay "2026-03-01T02:00:00" -- python overnight_batch.py

Stashed + Delay Combo

Create stashed jobs that auto-enqueue at a future time:

# Stash now, auto-enqueue in 2 hours
pueue add --stashed --delay 2h -- python populate_cache.py

Patterns

Skill: Pueue Job Orchestration

Claude Code + Pueue Integration Patterns

Patterns for using pueue as a telemetry and orchestration layer within Claude Code sessions.

Background Task Execution

Queue a task, continue working, poll results later:

# Queue and capture task ID
TASK_ID=$(pueue add --print-task-id -w ~/project -- python train_model.py)
echo "Queued as task $TASK_ID"

# ... continue other work ...

# Check status via JSON (structured for AI consumption)
pueue status --json | jq ".tasks[\"$TASK_ID\"]"

# Get results when done
pueue log "$TASK_ID" --full

Synchronous Wrapper Pattern

Claude Code expects synchronous stdout. This pattern wraps pueue to behave synchronously while capturing telemetry:

TASK_ID=$(pueue add --print-task-id -w "$(pwd)" -- <original-command>) && \
  pueue wait "$TASK_ID" --quiet && \
  pueue log "$TASK_ID" --full

What you get: The command runs through pueue (timing, exit code, log persistence, env snapshot) but stdout/stderr flows back to Claude Code as if the command ran directly.

Structured Monitoring via JSON

pueue status --json returns structured data ideal for AI agent consumption:

# All tasks with full metadata
pueue status --json | jq '.tasks | to_entries[] | {
  id: .key,
  command: .value.command,
  status: .value.status,
  group: .value.group,
  label: .value.label
}'

# Filter by group
pueue status --json | jq '[.tasks | to_entries[] | select(.value.group == "mygroup")]'

# Count by status
pueue status --json | jq '.tasks | to_entries | group_by(.value.status | keys[0]) | map({status: .[0].value.status | keys[0], count: length})'

Callback as Completion Signal

Use a sentinel file pattern for scripts that need to react to task completion:

# In pueue.yml
daemon:
  callback: "echo '{{id}}:{{result}}:{{exit_code}}' >> /tmp/pueue-completions.log"

Then poll the sentinel:

# Wait for specific task ID to appear in completions
while ! grep -q "^${TASK_ID}:" /tmp/pueue-completions.log 2>/dev/null; do
  sleep 1
done
RESULT=$(grep "^${TASK_ID}:" /tmp/pueue-completions.log | cut -d: -f2)

Note: pueue wait is simpler for most cases. Use callbacks when you need to react asynchronously or trigger external systems.

Batch Test/Build Orchestration

Run tests or builds across multiple packages using groups and DAGs:

# Create group with parallelism limit
pueue group add tests --parallel 4

# Queue test suite
for pkg in pkg-a pkg-b pkg-c pkg-d pkg-e; do
  pueue add --group tests --label "test:$pkg" -w ~/project \
    -- uv run pytest "packages/$pkg/tests/" -x
done

# Wait for all tests
pueue wait --group tests

# Check results
pueue status --json | jq '[.tasks | to_entries[] |
  select(.value.group == "tests") |
  {label: .value.label, result: .value.status.Done.result}]'

Build DAG with --after

# Build dependencies first, then dependents
BUILD_A=$(pueue add --print-task-id --group build -- make -C lib-a)
BUILD_B=$(pueue add --print-task-id --group build -- make -C lib-b)

# App depends on both libs
pueue add --group build --after "$BUILD_A" "$BUILD_B" \
  --label "build:app" -- make -C app

PreToolUse Hook Auto-Wrapping

The pretooluse-pueue-wrap-guard.ts hook (in itp-hooks) silently rewrites non-trivial Bash commands to run through pueue. This provides invisible telemetry — Claude Code operates normally while every non-trivial command gets timing, logs, and exit code capture.

How It Works

1. Claude Code generates a bash command (e.g., python train.py) 2. The PreToolUse hook intercepts the command before execution 3. If non-trivial, it rewrites to the synchronous wrapper pattern 4. Claude Code sees stdout/stderr as if the command ran directly 5. Pueue captures full telemetry in the background

Skip Tier (No Wrapping)

Read-only commands, pueue commands, interactive commands, and very short commands pass through without wrapping.

Escape Hatch

Add # PUEUE-SKIP comment to any command to bypass auto-wrapping:

python populate_cache.py --phase 1  # PUEUE-SKIP

Telemetry Query Patterns

After accumulating tasks, query the telemetry:

# Average runtime by label prefix
pueue status --json | jq '
  [.tasks | to_entries[] |
   select(.value.status.Done) |
   {label: .value.label,
    duration_s: ((.value.status.Done.end | split(".")[0] | fromdate) -
                 (.value.start | split(".")[0] | fromdate))}] |
  group_by(.label | split(":")[0]) |
  map({prefix: .[0].label | split(":")[0],
       avg_s: (map(.duration_s) | add / length),
       count: length})'

# Failed tasks with exit codes
pueue status --json | jq '[.tasks | to_entries[] |
  select(.value.status.Done.result != "Success") |
  {id: .key, label: .value.label, result: .value.status.Done.result}]'

Skill: Pueue Job Orchestration

ClickHouse Parallelism Tuning (pueue + ClickHouse)

When using pueue to orchestrate ClickHouse queries, the interaction between pueue parallelism and ClickHouse's thread scheduler determines actual throughput.

The Thread Soft Limit

ClickHouse has a concurrent_threads_soft_limit_ratio_to_cores setting (default: 2). On a 32-core machine, this means ClickHouse allows 64 concurrent execution threads total, regardless of how many queries are running.

Each query requests max_threads threads (default: auto = nproc = 32 on a 32-core machine). With 8 parallel queries each requesting 32 threads (= 256 requested), ClickHouse throttles to 64 actual threads. The queries get ~8 effective threads each, not 32.

Right-Size max_threads Per Query

Anti-pattern: Letting each query request 32 threads when it only gets 8 effective threads. This creates scheduling overhead for no benefit.

Fix: Set --max_threads to match the effective thread count:

# In the job wrapper script:
clickhouse-client --max_threads=8 --multiquery < "$SQL_FILE"

This reduces thread scheduling overhead and allows higher pueue parallelism without oversubscription.

Parallelism Sizing Formula

effective_threads_per_query = concurrent_threads_soft_limit / pueue_parallel_slots
concurrent_threads_soft_limit = nproc * concurrent_threads_soft_limit_ratio_to_cores

# Example: 32-core machine, ratio=2, soft_limit=64
# 8 pueue slots  -> 8 effective threads/query  -> ~55% CPU (baseline)
# 16 pueue slots -> 4 effective threads/query  -> ~87% CPU (1.5-1.8x throughput)
# 24 pueue slots -> 2-3 effective threads/query -> ~95% CPU (diminishing returns)

Decision Matrix

Live Tuning (No Restart Required)

Pueue parallelism can be changed live -- running jobs finish with old settings, new jobs use the new limit:

# Check current
pueue group | grep mygroup

# Bump up
pueue parallel 16 -g mygroup

# Monitor for 2-3 minutes, then check
uptime                    # Load average
free -h                   # Memory
vmstat 1 3                # Swap (si/so = 0?)
clickhouse-client --query "SELECT count() FROM system.query_log
    WHERE event_time > now() - INTERVAL 5 MINUTE
    AND type = 'ExceptionWhileProcessing'"  # Errors = 0?

Skill: Pueue Job Orchestration

Environment Variables & Secrets for Pueue Jobs

Preferred Pattern: python-dotenv for Pueue Job Secrets

Pueue jobs run in clean shells without .bashrc, .zshrc, or mise activation. This means mise.toml [env] variables are invisible to pueue jobs. The most portable solution is python-dotenv:

Architecture

mise.toml           -> Task definitions only (no [env] for secrets)
.env                -> Secrets (gitignored, loaded by python-dotenv at runtime)
scripts/backfill.sh -> Pueue orchestrator (just `cd $PROJECT_DIR` for dotenv)

Implementation

1. Project `.env` (gitignored):

# .env -- loaded by python-dotenv at runtime
API_KEY=sk-abc123
DATABASE_URL=postgresql://localhost/mydb

2. Python entry point -- call load_dotenv() early:

from dotenv import load_dotenv
load_dotenv()  # Auto-loads .env from cwd

import os
API_KEY = os.getenv("API_KEY")  # Works everywhere

3. Pueue job -- just needs cd to project root:

# The only requirement: cwd must contain .env
pueue add -- bash -c 'cd ~/project && uv run python my_script.py'

Why This Beats Alternatives

Cross-reference: See distributed-job-safety skill -- G-15, AP-16

---

Integration with rangebar-py

The rangebar-py project has Pueue integration scripts:

Phase-Based Execution

# Phase 1: 1000 dbps (fast, 4 parallel)
./scripts/pueue-populate.sh phase1

# Phase 2: 250 dbps (moderate, 2 parallel)
./scripts/pueue-populate.sh phase2

# Phase 3: 500, 750 dbps (3 parallel)
./scripts/pueue-populate.sh phase3

# Phase 4: 100 dbps (resource intensive, 1 at a time)
./scripts/pueue-populate.sh phase4

Evolution Log

Convention: Reverse chronological order (newest on top, oldest at bottom). Prepend new entries.

---

2026-02-26: Initial Evolution Log

Status: Skill is in use and maintained. Track improvements here.

Purpose

This evolution log tracks updates to the skill. Each entry should note:

• What changed (content, structure, tooling)
• Why it changed (bug fix, feature request, best practice)
• Files affected

How to Use

1. When updating SKILL.md or references, add an entry here with the date 2. Keep entries reverse-chronological (newest first) 3. Link to ADRs or GitHub issues when relevant 4. Reference specific line changes when helpful

---

Skill: Pueue Job Orchestration

Installation Guide

macOS (Local)

brew install pueue
pueued -d  # Start daemon

Linux (BigBlack)

# Download from GitHub releases (see https://github.com/Nukesor/pueue/releases for latest)
curl -sSL https://raw.githubusercontent.com/terrylica/rangebar-py/main/scripts/setup-pueue-linux.sh | bash

# Or manually:
# SSoT-OK: Version from GitHub releases page
PUEUE_VERSION="v4.0.2"
curl -sSL "https://github.com/Nukesor/pueue/releases/download/${PUEUE_VERSION}/pueue-x86_64-unknown-linux-musl" -o ~/.local/bin/pueue
curl -sSL "https://github.com/Nukesor/pueue/releases/download/${PUEUE_VERSION}/pueued-x86_64-unknown-linux-musl" -o ~/.local/bin/pueued
chmod +x ~/.local/bin/pueue ~/.local/bin/pueued

# Start daemon
~/.local/bin/pueued -d

Systemd Auto-Start (Linux)

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/pueued.service << 'EOF'
[Unit]
Description=Pueue Daemon
After=network.target

[Service]
ExecStart=%h/.local/bin/pueued -v
Restart=on-failure

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user enable --now pueued

macOS Auto-Start (launchd)

Auto-start the pueue daemon on login. Create the plist at ~/Library/LaunchAgents/com.nukesor.pueued.plist:

# Generate the launchd plist (standard Apple plist format)  # SSoT-OK
cat > ~/Library/LaunchAgents/com.nukesor.pueued.plist << 'PLIST'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.nukesor.pueued</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/homebrew/bin/pueued</string>
        <string>-v</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/pueued.stdout.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/pueued.stderr.log</string>
</dict>
</plist>
PLIST

Then load the agent:

# Load (starts immediately + on login)
launchctl load ~/Library/LaunchAgents/com.nukesor.pueued.plist

# Unload
launchctl unload ~/Library/LaunchAgents/com.nukesor.pueued.plist

# Check status
launchctl list | grep pueued

Linux equivalent: Use systemd -- see pueued --systemd or create a user service in ~/.config/systemd/user/.

Skill: Pueue Job Orchestration

Production Lessons (Issue #88)

Battle-tested patterns from real production deployments.

Dependency Chaining with --after

Pueue supports automatic job dependency resolution via --after. This is critical for post-processing pipelines where steps must run sequentially after batch jobs complete.

Key flags:

• --after <id>... -- Start job only after ALL specified jobs succeed. If any dependency fails, this job fails too.
• --print-task-id (or -p) -- Return only the numeric job ID (for scripting).

Pattern: Capturing job IDs for dependency wiring

# Capture job IDs during batch submission
JOB_IDS=()
for symbol in BTCUSDT ETHUSDT; do
    job_id=$(cd /path/to/project && pueue add --print-task-id --group mygroup \
        --label "${symbol}@250" \
        -- uv run python scripts/process.py --symbol "$symbol")
    JOB_IDS+=("$job_id")
done

# Chain post-processing after ALL batch jobs
optimize_id=$(pueue add --print-task-id --group mygroup \
    --label "optimize-table" \
    --after "${JOB_IDS[@]}" \
    -- clickhouse-client --query "OPTIMIZE TABLE mydb.mytable FINAL")

# Chain validation after optimize
pueue add --group mygroup \
    --label "validate" \
    --after "$optimize_id" \
    -- uv run python scripts/validate.py

Result in pueue status:

Job 0  BTCUSDT@250    Running
Job 1  ETHUSDT@250    Running
Job 2  optimize-table Queued  Deps: 0, 1
Job 3  validate       Queued  Deps: 2

When to use `--after`:

• Post-processing steps (OPTIMIZE TABLE, validation scripts, cleanup)
• Multi-stage pipelines where Stage N depends on Stage N-1
• Verification jobs that should only run after data is fully written

Anti-pattern: Manual waiting

# BAD: Manual polling or instructions to "run this after that finishes"
postprocess_all() {
    queue_repopulation_jobs
    echo "Run 'pueue wait --group postfix' then run optimize manually"  # NO!
}

# GOOD: Automatic dependency chain
postprocess_all() {
    queue_repopulation_jobs  # captures JOB_IDS
    pueue add --after "${JOB_IDS[@]}" -- optimize_command
    pueue add --after "$optimize_id" -- validate_command
}

Mise Task to Pueue Pipeline Integration

Pattern for mise run commands that build pueue DAGs:

# .mise/tasks/cache.toml
["cache:postprocess-all"]
description = "Full post-fix pipeline via pueue: repopulate -> optimize -> detect (auto-chained)"
run = "./scripts/pueue-populate.sh postprocess-all"

The shell script captures pueue job IDs and chains them with --after. Mise provides the entry point; pueue provides the execution engine with dependency resolution.

Forensic Audit Before Deployment

ALWAYS audit the remote host before mutating anything:

# 1. Pueue job state
ssh host 'pueue status'
ssh host 'pueue status --json | python3 -c "import json,sys; d=json.load(sys.stdin); print(sum(1 for t in d[\"tasks\"].values() if \"Running\" in str(t[\"status\"])))"'

# 2. Database state (ClickHouse example)
ssh host 'clickhouse-client --query "SELECT symbol, threshold, count(), countIf(volume < 0) FROM mytable GROUP BY ALL"'

# 3. Checkpoint state
ssh host 'ls -la ~/.cache/myapp/checkpoints/'
ssh host 'cat ~/.cache/myapp/checkpoints/latest.json'

# 4. System resources
ssh host 'uptime && free -h && df -h /home'

# 5. Installed version
ssh host 'cd ~/project && git log --oneline -1'

Force-Refresh vs Checkpoint Resume

Decision matrix for restarting killed/failed jobs:

PATH Gotcha: Rust Not in PATH via uv run

On remote hosts, uv run maturin develop may fail because ~/.cargo/bin is not in uv run's PATH:

# FAILS: rustc not found
ssh host 'cd ~/project && uv run maturin develop --uv'

# WORKS: Prepend cargo bin to PATH
ssh host 'cd ~/project && PATH="$HOME/.cargo/bin:$PATH" uv run maturin develop --uv'

For pueue jobs that need Rust compilation:

pueue add -- env PATH="/home/user/.cargo/bin:$PATH" uv run maturin develop

Per-Year (Epoch) Parallelization -- DEFAULT STRATEGY

This is the default approach for all multi-year cache population. Never queue a monolithic multi-year job when epoch boundaries exist. A single DOGEUSDT@500 job estimated 22 days; per-year splits brought it to ~3-4 days with 4 parallel cores.

When a processing pipeline has natural reset boundaries (yearly, monthly, etc.) where processor state resets, each epoch becomes an independent processing unit. This enables massive speedup by splitting a multi-year sequential job into concurrent per-year pueue jobs.

Why it's safe (three isolation layers):

Pattern: Per-symbol pueue groups

Give each symbol (or job family) its own pueue group for independent parallelism control:

# Create per-symbol groups
pueue group add btc-yearly --parallel 4
pueue group add eth-yearly --parallel 4
pueue group add shib-yearly --parallel 4

# Queue per-year jobs
for year in 2019 2020 2021 2022 2023 2024 2025 2026; do
    pueue add --group btc-yearly \
        --label "BTC@250:${year}" \
        -- uv run python scripts/process.py \
        --symbol BTCUSDT --threshold 250 \
        --start-date "${year}-01-01" --end-date "${year}-12-31"
done

# Chain post-processing after ALL groups complete
ALL_JOB_IDS=($(pueue status --json | jq -r \
    '.tasks | to_entries[] | select(.value.group | test("-yearly$")) | .value.id'))
pueue add --after "${ALL_JOB_IDS[@]}" \
    --label "optimize-table:final" \
    -- clickhouse-client --query "OPTIMIZE TABLE mydb.mytable FINAL"

When to use per-year vs sequential:

Critical rules:

1. Working directory: Use pueue add -w ~/project (preferred) or cd ~/project && pueue add -- SSH cwd defaults to $HOME, not the project directory. Jobs fail instantly with No such file or directory if this is missed. Note: on macOS, -w /tmp resolves to /private/tmp (symlink). 2. First year uses domain-specific effective start date, not 01-01 3. Last year uses actual latest available date as end 4. Chain OPTIMIZE TABLE FINAL after ALL year-jobs via --after 5. Memory budget: each job peaks independently -- with 61 GB total, 4-5 concurrent jobs at 5 GB each are safe 6. No `--force-refresh` on per-year jobs when other year-jobs for the same symbol are running -- it deletes cached bars by date range and can conflict with concurrent writes.

Pipeline Monitoring (Group-Based Phase Detection)

For multi-group pipelines, monitor job phases by group completion, not hardcoded job IDs. Job IDs change when jobs are removed, re-queued, or split into per-year jobs.

Anti-pattern: Hardcoded job IDs in monitors

# WRONG: Breaks when jobs are removed/re-queued
job14=$(echo "$JOBS" | grep "^14|")
if [ "$(echo "$job14" | cut -d'|' -f2)" = "Done" ]; then
    echo "Phase 1 complete"
fi

Correct pattern: Dynamic group detection

get_job_status() {
    ssh host "pueue status --json 2>/dev/null" | jq -r \
        '.tasks | to_entries[] |
         "\(.value.id)|\(.value.status | if type == "object" then keys[0] else . end)|\(.value.label // "-")|\(.value.group)"'
}

group_all_done() {
    local group="$1"
    local group_jobs
    group_jobs=$(echo "$JOBS" | grep "|${group}$" || true)
    [ -z "$group_jobs" ] && return 1
    echo "$group_jobs" | grep -qE "\|(Running|Queued)\|" && return 1
    return 0
}

# Detect phase transitions by group name
SEEN_GROUPS=""
for group in $(echo "$JOBS" | cut -d'|' -f4 | sort -u); do
    if group_all_done "$group" && [[ "$SEEN_GROUPS" != *"|${group}|"* ]]; then
        echo "GROUP COMPLETE: $group"
        run_integrity_checks "$group"
        SEEN_GROUPS="${SEEN_GROUPS}|${group}|"
    fi
done

Integrity checks at phase boundaries:

Run automated validation when a group finishes, before starting the next phase:

run_integrity_checks() {
    local phase="$1"
    # Check 1: Data corruption (negative values, out-of-bounds)
    ssh host 'clickhouse-client --query "SELECT ... countIf(value < 0) ... HAVING count > 0"'
    # Check 2: Duplicate rows
    ssh host 'clickhouse-client --query "SELECT ... count(*) - uniqExact(key) as dupes HAVING dupes > 0"'
    # Check 3: Coverage gaps (NULL required fields)
    ssh host 'clickhouse-client --query "SELECT ... countIf(field IS NULL) ... HAVING missing > 0"'
    # Check 4: System resources (load, memory)
    ssh host 'uptime && free -h'
}

Monitoring as a background loop:

POLL_INTERVAL=300  # 5 minutes
while true; do
    JOBS=$(get_job_status)
    # Count statuses, detect failures, detect group completions
    # Run integrity checks at phase boundaries
    # Exit when all jobs complete
    sleep "$POLL_INTERVAL"
done

Skill: Pueue Job Orchestration

Pueue Configuration Reference

Complete pueue.yml configuration derived from verified source code (settings.rs) and empirical testing on macOS + Linux.

Config File Location

Override with environment variable: PUEUE_CONFIG_PATH=/path/to/pueue.yml

[shared] Settings

[client] Settings

[daemon] Settings

Shell Command Defaults

[profiles] Section

Profiles override settings when invoked with pueue -p <profile>:

profiles:
  gpu-heavy:
    daemon:
      pause_group_on_failure: true
      callback: "echo 'GPU job {{id}} {{result}}' >> /tmp/gpu-completions.log"
    client:
      dark_mode: true
    shared: {}

Usage: pueue -p gpu-heavy status

Example Complete Config

shared:
  use_unix_socket: true

client:
  restart_in_place: false
  read_local_logs: true
  show_confirmation_questions: false
  dark_mode: false
  status_time_format: "%H:%M:%S"
  status_datetime_format: "%Y-%m-%d\n%H:%M:%S"

daemon:
  pause_group_on_failure: false
  pause_all_on_failure: false
  compress_state_file: true
  callback: "echo '{{id}}:{{result}}:{{exit_code}}' >> /tmp/pueue-completions.log"
  callback_log_lines: 10
  env_vars: {}

profiles:
  production:
    daemon:
      pause_group_on_failure: true
      compress_state_file: true
    client:
      show_confirmation_questions: true
    shared: {}

Source

Derived from `pueue_lib/src/settings.rs` and `pueue_lib/src/setting_defaults.rs` in the pueue repository.

Skill: Pueue Job Orchestration

State File Management & Bulk Submission

State Bloat (CRITICAL)

Pueue stores ALL task metadata in a single state.json file. This file grows with every completed task and is read/written on EVERY pueue add call. Neglecting state hygiene is the #1 cause of slow job submission in large sweeps.

The State Bloat Anti-Pattern

Symptom: pueue add takes 1-2 seconds instead of <100ms.

Root cause: Pueue serializes/deserializes the entire state file on every operation. With 50K+ completed tasks, state.json grows to 80-100MB. Each pueue add becomes 80MB read + 80MB write = 160MB I/O.

Benchmarks (pueue v4, NVMe SSD, 32-core Linux):

Key insight: Parallelism does NOT help when state is bloated -- the pueue daemon serializes all operations through a mutex. The 455ms at P16 is WORSE per-operation than 1,300ms sequential because of lock contention overhead. Clean first, then parallelize.

Pre-Submission Clean (Mandatory Pattern)

Before any bulk submission (>100 jobs), clean completed tasks:

# ALWAYS clean before bulk submission
pueue clean -g mygroup 2>/dev/null || true

# Verify state is manageable
STATE_FILE="$HOME/.local/share/pueue/state.json"
STATE_SIZE=$(stat -c%s "$STATE_FILE" 2>/dev/null || stat -f%z "$STATE_FILE" 2>/dev/null || echo 0)
if [ "$STATE_SIZE" -gt 52428800 ]; then  # 50MB
    echo "WARNING: state.json is $(( STATE_SIZE / 1048576 ))MB -- running extra clean"
    pueue clean 2>/dev/null || true
fi

Periodic Clean During Long Sweeps

For sweeps with 100K+ jobs, clean periodically between submission batches:

BATCH_SIZE=5000
POS=0
while [ "$POS" -lt "$TOTAL" ]; do
    # Submit batch
    tail -n +$((POS + 1)) "$CMDFILE" | head -n "$BATCH_SIZE" | \
        xargs -P16 -I{} bash -c '{}' 2>/dev/null || true
    POS=$((POS + BATCH_SIZE))

    # Prevent state bloat between batches
    pueue clean -g mygroup 2>/dev/null || true
done

Compressed State File

Reduce I/O for state persistence with zstd compression:

# In pueue.yml
daemon:
  compress_state_file: true

Compression ratio: ~10:1 (from pueue source code).

When to enable:

• I/O-constrained hosts (spinning disks, NFS mounts)
• Large task histories (hundreds of completed tasks)
• Defense-in-depth alongside periodic pueue clean

Note: Compression helps I/O performance. pueue clean reduces data volume. They are complementary, not alternatives.

---

Bulk Submission with xargs -P (High-Throughput Pattern)

For large job counts (1K+), submitting one pueue add at a time via SSH is prohibitively slow. Use a batch command file fed through xargs -P for parallel submission.

Why Not GNU Parallel?

CRITICAL: Many Linux hosts (including Ubuntu/Debian) ship with moreutils `parallel`, NOT GNU Parallel. They share the binary name /usr/bin/parallel but are completely different tools:

Detection:

if parallel --version 2>&1 | grep -q 'GNU'; then
    echo "GNU Parallel available"
else
    echo "moreutils parallel (or none) -- use xargs -P instead"
fi

Safe default: Always use xargs -P -- it's POSIX standard and available everywhere.

Batch Command File Pattern

Step 1: Generate commands file (one pueue add per line):

# gen_commands.sh -- generates commands.txt
for SQL_FILE in /tmp/sweep_sql/*.sql; do
    echo "pueue add -g p1 -- /tmp/run_job.sh '${SQL_FILE}' '${LOG_FILE}'"
done > /tmp/commands.txt
echo "Generated $(wc -l < /tmp/commands.txt) commands"

Step 2: Feed via xargs -P (parallel submission):

# Submit in batches with periodic state cleanup
BATCH=5000
P=16
TOTAL=$(wc -l < /tmp/commands.txt)
POS=0

while [ "$POS" -lt "$TOTAL" ]; do
    tail -n +$((POS + 1)) /tmp/commands.txt | head -n "$BATCH" | \
        xargs -P"$P" -I{} bash -c '{}' 2>/dev/null || true
    POS=$((POS + BATCH))

    # Clean between batches to prevent state bloat
    pueue clean -g p1 2>/dev/null || true

    QUEUED=$(pueue status -g p1 --json 2>/dev/null | python3 -c \
        "import json,sys; d=json.load(sys.stdin); print(sum(1 for t in d.get('tasks',{}).values() if 'Queued' in str(t.get('status',''))))" 2>/dev/null || echo "?")
    echo "Batch: ${POS}/${TOTAL} | Queued: ${QUEUED}"
done

Crash Recovery with Skip-Done

For idempotent resubmission after SSH drops or crashes:

# Build done-set from existing JSONL output
declare -A DONE_SET
for logfile in /tmp/sweep_*.jsonl; do
    while IFS= read -r config_id; do
        DONE_SET["${config_id}"]=1
    done < <(jq -r '.feature_config // empty' "$logfile" 2>/dev/null | sort -u)
done

# Generate commands, skipping completed configs
for SQL_FILE in /tmp/sweep_sql/*.sql; do
    CONFIG_ID=$(basename "$SQL_FILE" .sql)
    if [ "${DONE_SET[${CONFIG_ID}]+_}" ]; then
        continue  # Already completed
    fi
    echo "pueue add -g p1 -- /tmp/run_job.sh '${SQL_FILE}' '${LOG_FILE}'"
done > /tmp/commands.txt

Requirements: bash 4+ for associative arrays (declare -A).

---

Two-Tier Architecture (300K+ Jobs)

For sweeps exceeding 10K queries, the single-tier "pueue add per query" pattern is unusable -- pueue add has 148ms overhead per call even with clean state (= 8+ hours for 196K jobs). The fix is eliminating pueue add at the query level entirely.

Architecture

macOS (local)
  mise run gen:generate   -> N SQL files
  mise run gen:submit-all -> rsync + queue M pueue units
  mise run gen:collect    -> scp + validate JSONL

BigBlack (remote)
  pueue group p1 (parallel=1)   <- sequential units (avoid log contention)
    |-- Unit 1: submit_unit.sh pattern1 BTCUSDT 750
    |     \-- xargs -P16 -> K queries (direct clickhouse-client, no pueue add)
    |-- Unit 2: submit_unit.sh pattern1 BTCUSDT 1000
    |     \-- xargs -P16 -> K queries
    \-- ... (M total units)

Key Principles

When to Use Each Tier

Shell Script Safety (set -euo pipefail)

Skipped Config NDJSON Pattern

Configs with 0 signals after feature filtering produce 1 JSONL line (skipped entry), not N barrier lines. This is correct behavior, not data loss.

When validating line counts:

expected_lines = (N_normal x barriers_per_query) + (N_skipped x 1) + (N_error x 1)

Example: 95 normal configs x 3 barriers + 5 skipped x 1 = 290 lines (not 300).

comm -23 for Large Skip-Done Sets (100K+)

For done-sets exceeding 10K entries, comm -23 (sorted set difference) is O(N+M) vs grep-per-file O(N\*M):

# Build sorted done-set from JSONL
python3 -c "
import json
seen = set()
for line in open('\${LOG_FILE}'):
    try:
        d = json.loads(line)
        fc = d.get('feature_config','')
        if fc: seen.add(fc)
    except: pass
for s in sorted(seen): print(s)
" > /tmp/done.txt

# Build sorted all-configs, compute set difference
ls \${DIR}/*.sql | xargs -n1 basename | sed 's/\.sql$//' | sort > /tmp/all.txt
comm -23 /tmp/all.txt /tmp/done.txt > /tmp/todo.txt

# Submit remaining via xargs
cat /tmp/todo.txt | while read C; do echo "\${DIR}/\${C}.sql"; done | \
    xargs -P16 -I{} bash /tmp/wrapper.sh {} \${LOG} \${SYM} \${THR} \${GIT}