
Harness Engineering Playbook
- 314 installs
- 25 repo stars
- Updated June 28, 2026
- broomva/harness-engineering-skill
harness-engineering-playbook is a Claude Code skill that bootstraps OpenAI Harness Engineering artifacts and deterministic agent harness commands for developers running repeated autonomous coding workflows in a repositor
About
harness-engineering-playbook from broomva/harness-engineering-skill operationalizes OpenAI Harness Engineering practices for repositories that agents touch repeatedly. The skill baselines an existing repo, then bootstraps AGENTS.md, PLANS.md, Makefile-style deterministic entrypoints, architecture boundary rules, and entropy-control checks through a Python harness_wizard.py init command with baseline, control, or full profiles. It maps all nine Harness Engineering practices to concrete artifacts, runs harness audit checks that treat missing practices as blocking gaps, and iterates after real agent runs. Shell fallback scripts such as bootstrap_harness.sh scaffold templates when the wizard is unavailable. The repository is migrating to broomva/skills under the same skill name. Developers reach for harness-engineering-playbook when agent runs suffer from setup drift, flaky tests, missing trace IDs, or unclear documentation boundaries.
- Harness pipeline scaffolding
- Trigger and stage configuration
- Cloud and artifact integrations
- Approval and quality gate patterns
- Rollback and promotion workflows
Harness Engineering Playbook by the numbers
- 314 all-time installs (skills.sh)
- Ranked #327 of 1,435 DevOps & CI/CD skills by installs in the Skillselion catalog
- Data as of Jul 28, 2026 (Skillselion catalog sync)
npx skills add https://github.com/broomva/harness-engineering-skill --skill harness-engineering-playbookAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 314 |
|---|---|
| repo stars | ★ 25 |
| Last updated | June 28, 2026 |
| Repository | broomva/harness-engineering-skill ↗ |
How do you bootstrap Harness Engineering for agent repos?
Configure Harness pipelines, triggers, and deployment stages while integrating cloud providers and quality gates into a repeatable delivery workflow.
Who is it for?
Engineering teams adopting OpenAI Harness Engineering patterns who need scripted smoke, test, lint, and entropy controls for autonomous agent runs.
Skip if: Teams looking to configure Harness.io CD pipelines, cloud deployment stages, or one-off scripts without agent-oriented repository guardrails.
When should I use this skill?
A developer asks to bootstrap harness engineering, add AGENTS.md guardrails, or audit a repo for agent-ready deterministic workflows.
What you get
AGENTS.md, PLANS.md, Makefile harness commands, architecture boundary docs, and passing harness audit reports.
- AGENTS.md
- PLANS.md
- deterministic harness Makefile commands
By the numbers
- Applies 9 OpenAI Harness Engineering practices to repository artifacts
- Offers 3 bootstrap profiles: baseline, control, and full
Files
Harness Engineering Playbook
Use this skill to operationalize the practices from OpenAI's Harness Engineering guide in a repo that agents can run against repeatedly and safely.
What To Load
- Use
references/openai-harness-practices.mdfor the full practice-to-artifact mapping. - Use
references/rollout-checklist.mdfor phased adoption in active repos. - Use
references/wizard-cli.mdfor Typer wizard command flows. - Use
assets/templates/when creating or updating harness files.
Inputs
- Target repository path.
- Existing command surface (
make,npm,cargo,pytest, etc.). - Existing CI workflows and branch protections.
Workflow
1. Baseline the repo and detect existing workflows. 2. Bootstrap harness artifacts and templates. 3. Apply all nine Harness Engineering practices. 4. Run harness audit checks and repair gaps. 5. Iterate after real agent runs.
Step 1: Baseline The Repo
- Identify language/toolchain and canonical entrypoints.
- Inventory existing checks, scripts, and CI jobs.
- Record current pain points for agent runs: setup drift, unclear docs, flaky tests, missing trace IDs, slow loops.
Use a short baseline note inside PLANS.md so decisions remain durable.
Step 2: Bootstrap Harness Artifacts
Preferred entrypoint:
python3 scripts/harness_wizard.py init <repo-path> --profile controlProfiles:
baseline: only core harness artifacts.control: baseline + control-system primitives.full: control + entropy controls (nightly audit + entropy checks).
Direct shell fallback:
Run:
./scripts/bootstrap_harness.sh <repo-path>This script installs safe defaults from assets/templates/:
AGENTS.mdPLANS.mddocs/ARCHITECTURE.mddocs/OBSERVABILITY.mdMakefile.harness(+-include Makefile.harnessinMakefile)scripts/audit_harness.shscripts/harness/{smoke,test,lint,typecheck}.sh.github/workflows/harness.yml
By default, existing files are not overwritten. Pass --force to replace template-managed files.
Step 3: Apply The Nine Practices
Implement each practice directly in repo artifacts.
1. Make Easy To Do Hard Thing
- Ensure hard, high-value tasks are one command away (
make smoke,make check,make ci). - Keep setup and cleanup scripted.
- Make smoke checks cheap enough for frequent use.
2. Communicate Actionable Constraints With Compact Docs
- Keep
AGENTS.mdshort, concrete, and command-first. - Document non-obvious constraints and guardrails.
- Keep docs close to code and update with behavior changes.
3. Structure Codebase With Strict Boundaries And Flow
- Define module boundaries in
docs/ARCHITECTURE.md. - Parse and validate data at boundaries; use typed contracts for internal flow.
- Prefer one abstraction per module and one clear ownership path.
4. Build Observability In From Day 1
- Emit structured logs/events with correlation IDs.
- Capture key transitions in long-running workflows.
- Define minimum observable fields in
docs/OBSERVABILITY.md.
5. Optimize For Agent Flow, Not Human Flow
- Treat context as a first-class system dependency.
- Use
PLANS.mdfor multi-step/multi-hour tasks. - Front-load durable context (scope, constraints, checkpoints) so restarts stay cheap.
6. Bring Your Own Harness
- Standardize repo-local wrappers (
Makefile.harness,scripts/harness/). - Wrap local infra actions in deterministic scripts.
- Make agent behavior reproducible across machines and runs.
7. Prototype In Natural Language First
- Draft logic and tests in prose before coding.
- Review edge cases in prose and lock acceptance criteria.
- Translate approved prose into code and tests.
8. Invest In Static Analysis And Linting
- Pin formatter/linter/typechecker versions where practical.
- Enforce checks in both local workflow and CI.
- Run static checks before long tests to shorten failure loops.
9. Manage Entropy
- Add periodic audits for docs drift, flaky checks, and dead scripts.
- Keep templates synchronized with real workflows.
- Remove stale abstractions quickly to keep agent context clean.
For a detailed artifact matrix, load references/openai-harness-practices.md.
Step 4: Validate
Run:
python3 scripts/harness_wizard.py audit <repo-path>Treat any MISSING or FAIL result as blocking before calling harness setup complete.
Step 5: Iterate On Real Runs
- Observe one full agent run from clean checkout to merged change.
- Patch harness gaps immediately.
- Re-run audit.
- Keep
AGENTS.md,PLANS.md, and architecture docs aligned with current behavior.
Adaptation Rules
- Preserve existing project conventions and replace templates incrementally.
- Do not overwrite user-authored files without explicit approval.
- Keep command names stable; change internals behind wrappers.
- Favor deterministic, scriptable workflows over ad-hoc interactive steps.
interface:
display_name: "Harness Engineering Playbook"
short_description: "OpenAI harness patterns for agent workflows"
default_prompt: "Analyze this repository and set up harness engineering workflows, docs, and automation following OpenAI Harness Engineering principles."
name: Harness CI
on:
push:
branches: [main]
pull_request:
jobs:
harness:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# Add language/runtime setup steps as needed for this repository.
# Examples:
# - actions/setup-node@v4
# - actions/setup-python@v5
# - dtolnay/rust-toolchain@stable
- name: Run harness pipeline
run: make ci
name: Nightly Harness Audit
on:
schedule:
- cron: "0 4 * * *"
workflow_dispatch:
jobs:
audit:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# Add runtime setup steps required for this repository.
- name: Baseline harness audit
run: scripts/audit_harness.sh .
- name: Entropy check
run: scripts/harness/entropy_check.sh
AGENTS.md
Replace placeholder text and keep this file compact, command-first, and operational.
Project Overview
- Project:
<project-name> - Primary runtime(s):
<runtime> - Main entrypoint(s):
<entrypoints>
Harness Commands
Run from repository root:
| Goal | Command |
|---|---|
| Fast sanity check | make smoke |
| Static checks | make check |
| Full test suite | make test |
| CI-equivalent local run | make ci |
Constraints And Guardrails
- Prefer deterministic scripts over interactive/manual steps.
- Keep command names stable (
smoke,check,test,ci). - Update docs and scripts in the same change when workflow behavior changes.
- Avoid side effects outside the repo unless explicitly required.
Architecture Boundaries
- Parse and validate external data at boundaries.
- Keep internal data models typed and normalized.
- Keep each module focused on one responsibility.
- Document boundary ownership in
docs/ARCHITECTURE.md.
Observability Expectations
- Include
trace_idandrun_idin long-running workflow logs. - Emit structured event names for major transitions (start, step, success, failure).
- Keep event fields stable for querying and alerting.
- Maintain field definitions in
docs/OBSERVABILITY.md.
Execution Plans
- For tasks expected to exceed ~30 minutes, create/update
PLANS.mdbefore coding. - Track scope, constraints, milestones, and verification steps.
- Update status checkpoints during execution and after major decisions.
Static Analysis And Quality Gates
- Run
make checkbeforemake test. - Run
make cibefore pushing large refactors. - Treat lint/type failures as blocking.
Entropy Management
- Remove stale scripts/docs quickly.
- Keep templates and real workflows in sync.
- Run periodic harness audits:
scripts/audit_harness.sh .
Architecture
Purpose
Describe the system's primary responsibilities and boundaries.
Boundaries
| Boundary | Input | Output | Owner |
|---|---|---|---|
| Example: API Layer | HTTP request | DTO | api module |
| Example: Domain Layer | DTO | domain model | core module |
| Example: Persistence Layer | domain model | persisted record | store module |
Data Shape Contracts
- Parse and validate external data at boundaries.
- Convert to internal typed models before crossing module boundaries.
- Keep boundary transformation logic centralized and testable.
Module Ownership Rules
- One primary responsibility per module.
- No cross-layer shortcuts without explicit architecture update.
- New modules require ownership and boundary documentation.
Execution Flow
1. Entry: 2. Boundary parse/validate: 3. Core execution: 4. Persistence/output: 5. Event/log emission:
Refactor Checklist
- [ ] Boundary contracts unchanged or versioned.
- [ ] Ownership map still accurate.
- [ ] Integration tests cover boundary paths.
- [ ] Documentation updated in same change.
Actuators
Define the actions agents are allowed to perform to move the system toward setpoints.
Actuation Surface
- Code edits
- Test and build execution
- Script/template updates
- CI workflow adjustments
- Documentation updates
Safety Boundaries
- Protected branches/rules:
- Restricted commands:
- Approval-required actions:
Action Catalog
| Action | Preconditions | Postconditions | Rollback |
|---|---|---|---|
| patch code | tests defined | checks green | revert commit |
| update harness docs | doc owner review | docs aligned | restore prior doc |
| tune CI workflow | CI dry run | stable runtime | revert workflow |
Control System Model
Purpose
Use this document to keep the repository's autonomous development loop explicit and stable.
System Definition
- Setpoint:
- Plant:
- Controller:
- Actuators:
- Sensors:
- Feedback channels:
- Disturbances:
Maturity Targets
- Stability target:
- Adaptation target:
- Recovery target:
Review Cadence
- Weekly harness review owner:
- Monthly architecture review owner:
- Entropy cleanup cadence:
Controller
Describe the policy and logic that decides corrective actions.
Control Policy
- Primary control objective:
- Secondary objectives:
- Priority order:
Control Inputs
- Required signals:
- Input freshness constraints:
- Input confidence thresholds:
Control Actions
- Tighten constraints (docs/scripts/gates)
- Adjust evaluation scope
- Escalate to human review
- Trigger refactor cleanup
Escalation Rules
- Escalation trigger:
- Escalation owner:
- Maximum autonomous retries:
Entropy Management
Define recurring cleanup actions that prevent harness drift.
Drift Sources
- Stale docs after workflow changes
- Dead scripts no longer called by CI
- Flaky tests ignored over time
- Inconsistent logging field names
Entropy Controls
- Weekly harness audit
- Monthly docs/script alignment review
- Periodic flaky-test triage
- Architectural boundary checks after refactors
Required Commands
scripts/harness/entropy_check.shscripts/audit_harness.sh .
Ownership
- Primary owner:
- Backup owner:
- Review cadence:
Feedback Loop
Define how observations produce corrective actions.
Loop Steps
1. Measure: capture sensor outputs. 2. Compare: compute error against setpoints. 3. Decide: choose control action. 4. Act: apply change. 5. Verify: re-measure and close the loop.
Control Frequency
- Fast loop (per change):
- Daily loop:
- Weekly loop:
Error Budget Policy
- Error budget metric:
- Budget window:
- Budget exhaustion response:
Sensors
List the signals used to evaluate whether the system is on target.
Required Sensors
- CI results (lint/typecheck/test/smoke)
- Structured runtime events
- Trace spans for long workflows
- Regression eval outcomes
- Review outcomes (requested changes, approval lag)
Signal Contracts
| Sensor | Required Fields | Sampling | Storage |
|---|---|---|---|
| harness events | trace_id, run_id, status, duration_ms | always | logs/traces |
| CI checks | check_name, status, duration_ms | always | CI provider |
| eval runs | task_id, pass_fail, score, runtime | per run | eval store |
Sensor Gaps
- Missing signals:
- Noisy/unreliable signals:
- Planned remediation:
Setpoints
Define numeric targets for the autonomous development loop.
Core Setpoints
| Metric | Target | Alert Threshold | Owner |
|---|---|---|---|
| PR pass@1 | |||
| Time to actionable failure | |||
| Merge cycle time | |||
| Revert rate | |||
| Human intervention rate |
Constraints
- Required quality gates:
- Security constraints:
- Cost/runtime constraints:
Stability
Track whether the development loop remains stable under normal and disturbed conditions.
Stability Indicators
- Check pass consistency over time
- Low variance in cycle time
- Bounded retry counts
- Controlled regression rate
Disturbance Scenarios
| Scenario | Expected Behavior | Recovery Target |
|---|---|---|
| dependency upgrade | temporary check failures | recover within 1 day |
| major feature branch | higher variance | recover within sprint |
| infra outage | degraded CI signal | recover when infra restored |
Stabilization Playbook
- Reconfirm setpoints.
- Reduce surface area of active change.
- Enforce stricter checks temporarily.
- Run entropy cleanup.
Observability
Goal
Make agent and harness workflows diagnosable without reproducing locally.
Required Event Fields
timestamplevelevent_nametrace_idrun_idstep_idcomponentstatusduration_ms
Event Taxonomy
harness.startharness.step.startharness.step.finishharness.step.failharness.check.passharness.check.fail
Logging Rules
- Emit structured logs for machine parsing.
- Keep field names stable over time.
- Include enough context to replay failures.
- Redact secrets and personally identifiable values.
Metrics
- Smoke-check duration
- Check failure rate (lint/type/test)
- Retry count per run
- Time-to-first-actionable-error
Alerting
- Alert on repeated harness failures in CI.
- Alert on missing observability fields in critical events.
- Alert on regression in smoke-check runtime budget.
version: 1
setpoints:
pr_pass_at_1:
target: 0.70
alert_below: 0.55
merge_cycle_time_hours:
target: 24
alert_above: 48
revert_rate:
target: 0.03
alert_above: 0.08
human_intervention_rate:
target: 0.20
alert_above: 0.40
time_to_actionable_failure_minutes:
target: 10
alert_above: 30
measurements:
- name: pr_pass_at_1
source: ci
- name: merge_cycle_time_hours
source: scm
- name: revert_rate
source: scm
- name: human_intervention_rate
source: review
- name: time_to_actionable_failure_minutes
source: ci
.PHONY: smoke test lint typecheck check ci
smoke:
@./scripts/harness/smoke.sh
test:
@./scripts/harness/test.sh
lint:
@./scripts/harness/lint.sh
typecheck:
@./scripts/harness/typecheck.sh
check: lint typecheck
ci: smoke check test
PLANS.md
Use this file for multi-step work where durable context matters.
Objective
- Outcome:
- Why it matters:
- Non-goals:
Constraints
- Runtime/tooling constraints:
- Security/compliance constraints:
- Performance/reliability constraints:
Context Snapshot
- Relevant files/modules:
- Existing commands/workflows:
- Known risks:
Execution Plan
1. Step:
- Expected output:
- Verification:
2. Step:
- Expected output:
- Verification:
3. Step:
- Expected output:
- Verification:
Checkpoints
- [ ] Baseline captured
- [ ] Implementation complete
- [ ] Static checks passed
- [ ] Tests passed
- [ ] Docs updated
Decision Log
- Date:
- Decision:
- Reason:
- Alternatives considered:
Final Verification
- Commands run:
- Key outputs:
- Follow-up tasks:
#!/usr/bin/env bash
set -euo pipefail
usage() {
cat <<'EOF'
Usage: scripts/audit_harness.sh [repo_path]
Audit a repository for baseline harness engineering artifacts.
EOF
}
target_path="${1:-.}"
if [ "$target_path" = "-h" ] || [ "$target_path" = "--help" ]; then
usage
exit 0
fi
if [ ! -d "$target_path" ]; then
echo "error: target path does not exist: $target_path" >&2
exit 1
fi
target_path=$(cd "$target_path" && pwd)
failures=0
ok() {
echo "[ok] $1"
}
fail() {
echo "[missing] $1"
failures=$((failures + 1))
}
check_file() {
local relative="$1"
if [ -f "$target_path/$relative" ]; then
ok "$relative"
else
fail "$relative"
fi
}
check_contains() {
local relative="$1"
local pattern="$2"
local label="$3"
local full="$target_path/$relative"
if [ ! -f "$full" ]; then
fail "$label (file missing: $relative)"
return
fi
if grep -Eq "$pattern" "$full"; then
ok "$label"
else
fail "$label"
fi
}
echo "Auditing harness artifacts in: $target_path"
echo
check_file "AGENTS.md"
check_file "PLANS.md"
check_file "docs/ARCHITECTURE.md"
check_file "docs/OBSERVABILITY.md"
check_file "Makefile.harness"
check_file "scripts/audit_harness.sh"
check_file "scripts/harness/smoke.sh"
check_file "scripts/harness/test.sh"
check_file "scripts/harness/lint.sh"
check_file "scripts/harness/typecheck.sh"
check_file ".github/workflows/harness.yml"
echo
check_contains "AGENTS.md" "Harness Commands" "AGENTS.md: Harness Commands section"
check_contains "AGENTS.md" "Execution Plans" "AGENTS.md: Execution Plans section"
check_contains "docs/ARCHITECTURE.md" "Boundaries" "ARCHITECTURE.md: boundary guidance"
check_contains "docs/OBSERVABILITY.md" "Required Event Fields" "OBSERVABILITY.md: required fields"
check_contains "Makefile.harness" "^smoke:" "Makefile.harness: smoke target"
check_contains "Makefile.harness" "^test:" "Makefile.harness: test target"
check_contains "Makefile.harness" "^lint:" "Makefile.harness: lint target"
check_contains "Makefile.harness" "^typecheck:" "Makefile.harness: typecheck target"
check_contains "Makefile.harness" "^ci:" "Makefile.harness: ci target"
check_contains ".github/workflows/harness.yml" "make ci" "CI workflow executes make ci"
echo
if [ "$failures" -gt 0 ]; then
echo "Harness audit failed: $failures issue(s) detected."
exit 1
fi
echo "Harness audit passed."
#!/usr/bin/env bash
set -euo pipefail
root_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
cd "$root_dir"
failures=0
check_exists() {
local rel="$1"
if [ -e "$rel" ]; then
echo "[ok] $rel"
else
echo "[missing] $rel"
failures=$((failures + 1))
fi
}
check_not_contains() {
local rel="$1"
local pattern="$2"
local label="$3"
if [ ! -f "$rel" ]; then
echo "[missing] $label (file missing: $rel)"
failures=$((failures + 1))
return
fi
if grep -En "$pattern" "$rel" >/dev/null 2>&1; then
echo "[drift] $label"
failures=$((failures + 1))
else
echo "[ok] $label"
fi
}
echo "Entropy check: $root_dir"
echo
check_exists "AGENTS.md"
check_exists "PLANS.md"
check_exists "docs/ARCHITECTURE.md"
check_exists "docs/OBSERVABILITY.md"
check_exists "Makefile.harness"
check_not_contains "AGENTS.md" "<project-name>|<runtime>|<entrypoints>" "AGENTS.md placeholders removed"
check_not_contains "docs/ARCHITECTURE.md" "^# Architecture$" "ARCHITECTURE customized"
check_not_contains "docs/OBSERVABILITY.md" "^# Observability$" "OBSERVABILITY customized"
echo
if [ "$failures" -gt 0 ]; then
echo "Entropy check failed: $failures issue(s)."
exit 1
fi
echo "Entropy check passed."
#!/usr/bin/env bash
set -euo pipefail
root_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
if [ -n "${HARNESS_LINT_CMD:-}" ]; then
cd "$root_dir"
eval "$HARNESS_LINT_CMD"
exit 0
fi
if [ -f "$root_dir/Cargo.toml" ] && command -v cargo >/dev/null 2>&1; then
cd "$root_dir"
cargo clippy --all-targets --all-features -- -D warnings
exit 0
fi
if [ -f "$root_dir/package.json" ] && command -v node >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then
cd "$root_dir"
if node -e 'const p=require("./package.json"); process.exit(p.scripts&&p.scripts.lint?0:1)' >/dev/null 2>&1; then
npm run -s lint
exit 0
fi
fi
if [ -f "$root_dir/pyproject.toml" ]; then
cd "$root_dir"
if command -v ruff >/dev/null 2>&1; then
ruff check .
exit 0
fi
if command -v flake8 >/dev/null 2>&1; then
flake8 .
exit 0
fi
fi
echo "No default lint command detected."
echo "Set HARNESS_LINT_CMD or customize scripts/harness/lint.sh."
exit 1
#!/usr/bin/env bash
set -euo pipefail
root_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
if [ -n "${HARNESS_SMOKE_CMD:-}" ]; then
cd "$root_dir"
eval "$HARNESS_SMOKE_CMD"
exit 0
fi
if [ -f "$root_dir/Cargo.toml" ] && command -v cargo >/dev/null 2>&1; then
cd "$root_dir"
cargo check --quiet
exit 0
fi
if [ -f "$root_dir/package.json" ] && command -v node >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then
cd "$root_dir"
if node -e 'const p=require("./package.json"); process.exit(p.scripts&&p.scripts.smoke?0:1)' >/dev/null 2>&1; then
npm run -s smoke
exit 0
fi
if node -e 'const p=require("./package.json"); process.exit(p.scripts&&p.scripts.build?0:1)' >/dev/null 2>&1; then
npm run -s build
exit 0
fi
fi
if [ -f "$root_dir/pyproject.toml" ] && command -v pytest >/dev/null 2>&1; then
cd "$root_dir"
pytest -q -k smoke || pytest -q -k "not integration and not e2e"
exit 0
fi
echo "No default smoke command detected."
echo "Set HARNESS_SMOKE_CMD or customize scripts/harness/smoke.sh."
exit 1
#!/usr/bin/env bash
set -euo pipefail
root_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
if [ -n "${HARNESS_TEST_CMD:-}" ]; then
cd "$root_dir"
eval "$HARNESS_TEST_CMD"
exit 0
fi
if [ -f "$root_dir/Cargo.toml" ] && command -v cargo >/dev/null 2>&1; then
cd "$root_dir"
cargo test --quiet
exit 0
fi
if [ -f "$root_dir/package.json" ] && command -v node >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then
cd "$root_dir"
if node -e 'const p=require("./package.json"); process.exit(p.scripts&&p.scripts.test?0:1)' >/dev/null 2>&1; then
npm run -s test
exit 0
fi
fi
if [ -f "$root_dir/pyproject.toml" ] && command -v pytest >/dev/null 2>&1; then
cd "$root_dir"
pytest -q
exit 0
fi
echo "No default test command detected."
echo "Set HARNESS_TEST_CMD or customize scripts/harness/test.sh."
exit 1
#!/usr/bin/env bash
set -euo pipefail
root_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
if [ -n "${HARNESS_TYPECHECK_CMD:-}" ]; then
cd "$root_dir"
eval "$HARNESS_TYPECHECK_CMD"
exit 0
fi
if [ -f "$root_dir/Cargo.toml" ] && command -v cargo >/dev/null 2>&1; then
cd "$root_dir"
cargo check --quiet
exit 0
fi
if [ -f "$root_dir/package.json" ] && command -v node >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then
cd "$root_dir"
if node -e 'const p=require("./package.json"); process.exit(p.scripts&&p.scripts.typecheck?0:1)' >/dev/null 2>&1; then
npm run -s typecheck
exit 0
fi
if node -e 'const p=require("./package.json"); process.exit(p.scripts&&p.scripts.tsc?0:1)' >/dev/null 2>&1; then
npm run -s tsc
exit 0
fi
fi
if [ -f "$root_dir/pyproject.toml" ]; then
cd "$root_dir"
if command -v pyright >/dev/null 2>&1; then
pyright
exit 0
fi
if command -v mypy >/dev/null 2>&1; then
mypy .
exit 0
fi
fi
echo "No default typecheck command detected."
echo "Set HARNESS_TYPECHECK_CMD or customize scripts/harness/typecheck.sh."
exit 1
OpenAI Harness Practices Mapping
This file maps each practice from OpenAI's Harness Engineering guidance to concrete repo artifacts.
Sources
- Harness Engineering: https://openai.com/index/harness-engineering/
- Using
PLANS.mdfor multi-hour tasks: https://cookbook.openai.com/articles/plan-driven-workflow - Data-shape boundary design reference: https://matklad.github.io/2023/08/17/types-are-parse-don-t-validate.html
- Boundary-first architecture reference: https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html
Practice Matrix
| Practice | What To Implement | Required Artifacts | Verification |
|---|---|---|---|
| 1. Make easy to do hard thing | Single-command wrappers for high-value tasks. | Makefile.harness, scripts/harness/*.sh | make smoke, make check, and make ci run without manual prep. |
| 2. Communicate actionable constraints with compact docs | Command-first guardrails and operational constraints. | AGENTS.md | Agent can execute common tasks without guessing undocumented behavior. |
| 3. Structure codebase with strict boundaries and flow | Clear boundaries, typed contracts, boundary parsing. | docs/ARCHITECTURE.md | Data transformations happen at edges; internals are simpler and traceable. |
| 4. Build observability in from day 1 | Structured events/logs and correlation IDs. | docs/OBSERVABILITY.md | Every critical transition has traceable identifiers and stable fields. |
| 5. Optimize for agent flow, not human flow | Durable, resumable planning context. | PLANS.md | Long tasks remain reproducible after interruptions or handoffs. |
| 6. Bring your own harness | Repo-local deterministic workflows (no hidden UI/manual steps). | Makefile.harness, scripts/harness/ | Same commands work in local and CI environments. |
| 7. Prototype in natural language first | Prose-first logic drafts before code. | PLANS.md sections for behavior/testing intent | First implementation pass has fewer reworks and edge-case misses. |
| 8. Invest in static analysis and linting | Fast-fail checks before expensive runs. | Makefile.harness, CI workflow | Lint/typecheck break builds early; test time is spent on validated code. |
| 9. Manage entropy | Scheduled audits and drift control. | scripts/audit_harness.sh, CI integration | Harness docs/scripts stay aligned with repo behavior over time. |
Non-Negotiables
1. Keep command entrypoints stable (make smoke, make check, make ci). 2. Keep docs compact and executable, not narrative-heavy. 3. Keep scripts deterministic and machine-readable. 4. Keep architecture boundaries explicit and reviewed during refactors. 5. Keep observability fields stable to support aggregation and replay.
Wizard Entry Point
Use scripts/harness_wizard.py as the stable orchestration layer for bootstrap, primitive upgrades, and auditing.
Harness Rollout Checklist
Use this staged checklist when integrating the harness into an existing repository with active development.
Phase 0: Baseline
- [ ] Record current build/test/lint/typecheck entrypoints.
- [ ] Identify flaky checks and long-running hot spots.
- [ ] Confirm required environments (local, CI, containers, services).
- [ ] Create a starter entry in
PLANS.mdwith scope and constraints.
Phase 1: Bootstrap
- [ ] Run
python3 scripts/harness_wizard.py init <repo-path> --profile control. - [ ] Verify generated files are present.
- [ ] Customize template placeholders for project-specific commands.
- [ ] Confirm
MakefileincludesMakefile.harness.
Phase 2: Practice Alignment
- [ ] Validate all nine practices against real workflows.
- [ ] Tighten
AGENTS.mdso high-probability tasks are one command each. - [ ] Update
docs/ARCHITECTURE.mdwith concrete module boundaries. - [ ] Add observability identifiers in logs/events and document them.
- [ ] Make static analysis and type checks mandatory before full test runs.
Phase 3: Automation + Entropy Control
- [ ] Enable
.github/workflows/harness.yml(or equivalent CI job). - [ ] Run
python3 scripts/harness_wizard.py audit <repo-path>in CI. - [ ] Add periodic review cadence for docs/scripts drift.
- [ ] Remove stale scripts and outdated docs to keep context clean.
Exit Criteria
- [ ] New contributors can run harness commands without extra tribal knowledge.
- [ ] Agent runs are reproducible from clean checkout.
- [ ] Core workflows are observable and debuggable.
- [ ] Harness audit passes consistently.
Wizard CLI
This skill ships a Typer-based CLI wizard:
scripts/harness_wizard.py
Use it as the primary interface for bootstrapping and evolving repositories.
Quick Start
python3 scripts/harness_wizard.py init <repo-path> --profile control
python3 scripts/harness_wizard.py status <repo-path>
python3 scripts/harness_wizard.py audit <repo-path>Commands
init
Initialize a repository with harness and control-system structures.
python3 scripts/harness_wizard.py init <repo-path> --profile baseline
python3 scripts/harness_wizard.py init <repo-path> --profile control
python3 scripts/harness_wizard.py init <repo-path> --profile full
python3 scripts/harness_wizard.py init <repo-path> --profile full --forceProfiles:
baseline: AGENTS/PLANS/docs/Makefile/scripts/CI core harness.control: baseline + control primitives (docs/control/*, metrics yaml).full: control + entropy controls (entropy_check.sh, nightly workflow).
audit
Run baseline harness audit wrapper:
python3 scripts/harness_wizard.py audit <repo-path>status
Show baseline and primitive coverage:
python3 scripts/harness_wizard.py status <repo-path>primitive list
List all available control primitives and associated files:
python3 scripts/harness_wizard.py primitive listprimitive add
Add selected primitives to an existing repo incrementally:
python3 scripts/harness_wizard.py primitive add setpoint sensors --repo <repo-path>
python3 scripts/harness_wizard.py primitive add entropy --repo <repo-path>#!/usr/bin/env bash
set -euo pipefail
usage() {
cat <<'EOF'
Usage: audit_harness.sh [repo_path]
Audit a repository for baseline harness engineering artifacts.
EOF
}
target_path="${1:-.}"
if [ "$target_path" = "-h" ] || [ "$target_path" = "--help" ]; then
usage
exit 0
fi
if [ ! -d "$target_path" ]; then
echo "error: target path does not exist: $target_path" >&2
exit 1
fi
target_path=$(cd "$target_path" && pwd)
failures=0
ok() {
echo "[ok] $1"
}
fail() {
echo "[missing] $1"
failures=$((failures + 1))
}
check_file() {
local relative="$1"
if [ -f "$target_path/$relative" ]; then
ok "$relative"
else
fail "$relative"
fi
}
check_contains() {
local relative="$1"
local pattern="$2"
local label="$3"
local full="$target_path/$relative"
if [ ! -f "$full" ]; then
fail "$label (file missing: $relative)"
return
fi
if grep -Eq "$pattern" "$full"; then
ok "$label"
else
fail "$label"
fi
}
echo "Auditing harness artifacts in: $target_path"
echo
check_file "AGENTS.md"
check_file "PLANS.md"
check_file "docs/ARCHITECTURE.md"
check_file "docs/OBSERVABILITY.md"
check_file "Makefile.harness"
check_file "scripts/audit_harness.sh"
check_file "scripts/harness/smoke.sh"
check_file "scripts/harness/test.sh"
check_file "scripts/harness/lint.sh"
check_file "scripts/harness/typecheck.sh"
check_file ".github/workflows/harness.yml"
echo
check_contains "AGENTS.md" "Harness Commands" "AGENTS.md: Harness Commands section"
check_contains "AGENTS.md" "Execution Plans" "AGENTS.md: Execution Plans section"
check_contains "docs/ARCHITECTURE.md" "Boundaries" "ARCHITECTURE.md: boundary guidance"
check_contains "docs/OBSERVABILITY.md" "Required Event Fields" "OBSERVABILITY.md: required fields"
check_contains "Makefile.harness" "^smoke:" "Makefile.harness: smoke target"
check_contains "Makefile.harness" "^test:" "Makefile.harness: test target"
check_contains "Makefile.harness" "^lint:" "Makefile.harness: lint target"
check_contains "Makefile.harness" "^typecheck:" "Makefile.harness: typecheck target"
check_contains "Makefile.harness" "^ci:" "Makefile.harness: ci target"
check_contains ".github/workflows/harness.yml" "make ci" "CI workflow executes make ci"
echo
if [ "$failures" -gt 0 ]; then
echo "Harness audit failed: $failures issue(s) detected."
exit 1
fi
echo "Harness audit passed."
#!/usr/bin/env bash
set -euo pipefail
usage() {
cat <<'EOF'
Usage: bootstrap_harness.sh [repo_path] [--force]
Install harness templates into a target repository.
Arguments:
repo_path Target repository path (default: current directory)
--force Overwrite existing template-managed files
EOF
}
target_path="."
force=0
while [ $# -gt 0 ]; do
case "$1" in
--force)
force=1
;;
-h|--help)
usage
exit 0
;;
*)
if [ "$target_path" != "." ]; then
echo "error: multiple repo paths provided" >&2
usage
exit 1
fi
target_path="$1"
;;
esac
shift
done
if [ ! -d "$target_path" ]; then
echo "error: target path does not exist: $target_path" >&2
exit 1
fi
target_path=$(cd "$target_path" && pwd)
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
skill_dir=$(cd "$script_dir/.." && pwd)
template_dir="$skill_dir/assets/templates"
if [ ! -d "$template_dir" ]; then
echo "error: template directory missing: $template_dir" >&2
exit 1
fi
copy_template() {
local relative="$1"
local source="$template_dir/$relative"
local destination="$target_path/$relative"
if [ ! -f "$source" ]; then
echo "[error] missing template: $relative" >&2
exit 1
fi
mkdir -p "$(dirname "$destination")"
if [ -f "$destination" ] && [ "$force" -ne 1 ]; then
echo "[skip] $relative (exists)"
return 0
fi
cp "$source" "$destination"
echo "[write] $relative"
}
templates=(
"AGENTS.md"
"PLANS.md"
"docs/ARCHITECTURE.md"
"docs/OBSERVABILITY.md"
"Makefile.harness"
"scripts/audit_harness.sh"
"scripts/harness/smoke.sh"
"scripts/harness/test.sh"
"scripts/harness/lint.sh"
"scripts/harness/typecheck.sh"
".github/workflows/harness.yml"
)
for relative in "${templates[@]}"; do
copy_template "$relative"
done
makefile="$target_path/Makefile"
if [ ! -f "$makefile" ]; then
cat > "$makefile" <<'EOF'
-include Makefile.harness
EOF
echo "[write] Makefile"
elif ! grep -Eq '(^|[[:space:]])-?include[[:space:]]+Makefile\.harness([[:space:]]|$)' "$makefile"; then
cat >> "$makefile" <<'EOF'
# Harness engineering targets
-include Makefile.harness
EOF
echo "[update] Makefile (+ include Makefile.harness)"
else
echo "[skip] Makefile already includes Makefile.harness"
fi
chmod +x \
"$target_path/scripts/audit_harness.sh" \
"$target_path/scripts/harness/smoke.sh" \
"$target_path/scripts/harness/test.sh" \
"$target_path/scripts/harness/lint.sh" \
"$target_path/scripts/harness/typecheck.sh"
echo
echo "Bootstrap complete."
echo "Next:"
echo " 1) Customize commands in scripts/harness/*.sh"
echo " 2) Update AGENTS.md and docs/* placeholders"
echo " 3) cd \"$target_path\" && scripts/audit_harness.sh ."
#!/usr/bin/env python3
"""Harness Engineering wizard CLI.
This command-line tool bootstraps and upgrades repositories so they are
harness/control-system ready for autonomous agent workflows.
"""
from __future__ import annotations
import subprocess
from enum import Enum
from pathlib import Path
from typing import Dict, Iterable, List, Tuple
import typer
app = typer.Typer(help="Harness engineering wizard for repository setup and control primitives.")
primitive_app = typer.Typer(help="Add or inspect control-system primitives.")
app.add_typer(primitive_app, name="primitive")
SCRIPT_DIR = Path(__file__).resolve().parent
SKILL_DIR = SCRIPT_DIR.parent
TEMPLATE_DIR = SKILL_DIR / "assets" / "templates"
BOOTSTRAP_SCRIPT = SCRIPT_DIR / "bootstrap_harness.sh"
AUDIT_SCRIPT = SCRIPT_DIR / "audit_harness.sh"
BASELINE_FILES: Tuple[str, ...] = (
"AGENTS.md",
"PLANS.md",
"docs/ARCHITECTURE.md",
"docs/OBSERVABILITY.md",
"Makefile.harness",
"scripts/audit_harness.sh",
"scripts/harness/smoke.sh",
"scripts/harness/test.sh",
"scripts/harness/lint.sh",
"scripts/harness/typecheck.sh",
".github/workflows/harness.yml",
)
class Primitive(str, Enum):
loop = "loop"
setpoint = "setpoint"
sensors = "sensors"
controller = "controller"
actuators = "actuators"
feedback = "feedback"
stability = "stability"
entropy = "entropy"
class Profile(str, Enum):
baseline = "baseline"
control = "control"
full = "full"
PRIMITIVE_FILES: Dict[Primitive, Tuple[str, ...]] = {
Primitive.loop: ("docs/control/CONTROL_SYSTEM.md",),
Primitive.setpoint: (
"docs/control/SETPOINTS.md",
"evals/control-loop-metrics.yaml",
),
Primitive.sensors: ("docs/control/SENSORS.md",),
Primitive.controller: ("docs/control/CONTROLLER.md",),
Primitive.actuators: ("docs/control/ACTUATORS.md",),
Primitive.feedback: ("docs/control/FEEDBACK_LOOP.md",),
Primitive.stability: ("docs/control/STABILITY.md",),
Primitive.entropy: (
"docs/control/ENTROPY.md",
"scripts/harness/entropy_check.sh",
".github/workflows/nightly-harness-audit.yml",
),
}
CONTROL_PROFILE: Tuple[Primitive, ...] = (
Primitive.loop,
Primitive.setpoint,
Primitive.sensors,
Primitive.controller,
Primitive.actuators,
Primitive.feedback,
Primitive.stability,
)
FULL_PROFILE: Tuple[Primitive, ...] = CONTROL_PROFILE + (Primitive.entropy,)
def _resolve_repo(repo_path: Path) -> Path:
repo = repo_path.expanduser().resolve()
if not repo.exists() or not repo.is_dir():
typer.secho(f"error: repo path does not exist: {repo}", fg=typer.colors.RED, err=True)
raise typer.Exit(code=2)
return repo
def _run(script: Path, args: List[str]) -> None:
if not script.exists():
typer.secho(f"error: script not found: {script}", fg=typer.colors.RED, err=True)
raise typer.Exit(code=2)
command = [str(script), *args]
result = subprocess.run(command, check=False)
if result.returncode != 0:
raise typer.Exit(code=result.returncode)
def _copy_template(relative_path: str, repo: Path, force: bool) -> str:
source = TEMPLATE_DIR / relative_path
target = repo / relative_path
if not source.exists():
typer.secho(f"error: missing template file: {source}", fg=typer.colors.RED, err=True)
raise typer.Exit(code=2)
target.parent.mkdir(parents=True, exist_ok=True)
if target.exists() and not force:
return "skip"
target.write_bytes(source.read_bytes())
if target.suffix == ".sh":
target.chmod(0o755)
return "write"
def _add_primitives(repo: Path, primitives: Iterable[Primitive], force: bool) -> None:
for primitive in primitives:
typer.secho(f"\n[{primitive.value}]")
for relative_path in PRIMITIVE_FILES[primitive]:
state = _copy_template(relative_path, repo, force)
verb = "write" if state == "write" else "skip "
typer.echo(f" [{verb}] {relative_path}")
def _check_exists(repo: Path, relative_path: str) -> bool:
return (repo / relative_path).exists()
def _primitive_status(repo: Path, primitive: Primitive) -> Tuple[int, int]:
files = PRIMITIVE_FILES[primitive]
present = sum(1 for rel in files if _check_exists(repo, rel))
return present, len(files)
@app.command()
def init(
repo_path: Path = typer.Argument(Path("."), help="Target repository path."),
profile: Profile = typer.Option(
Profile.control,
"--profile",
"-p",
help="Setup profile: baseline, control, or full.",
),
force: bool = typer.Option(False, "--force", help="Overwrite existing files."),
) -> None:
"""Initialize harness artifacts and optionally apply control primitives."""
repo = _resolve_repo(repo_path)
typer.secho(f"Initializing harness in {repo}", fg=typer.colors.CYAN)
args = [str(repo)]
if force:
args.append("--force")
_run(BOOTSTRAP_SCRIPT, args)
if profile == Profile.baseline:
return
primitives = CONTROL_PROFILE if profile == Profile.control else FULL_PROFILE
_add_primitives(repo, primitives, force=force)
typer.secho("\nInitialization complete.", fg=typer.colors.GREEN)
@app.command()
def audit(
repo_path: Path = typer.Argument(Path("."), help="Target repository path."),
) -> None:
"""Run baseline harness audit."""
repo = _resolve_repo(repo_path)
_run(AUDIT_SCRIPT, [str(repo)])
@app.command()
def status(
repo_path: Path = typer.Argument(Path("."), help="Target repository path."),
) -> None:
"""Show harness and primitive coverage status."""
repo = _resolve_repo(repo_path)
typer.secho(f"Harness status for {repo}", fg=typer.colors.CYAN)
typer.echo()
baseline_present = sum(1 for rel in BASELINE_FILES if _check_exists(repo, rel))
baseline_total = len(BASELINE_FILES)
typer.echo(f"baseline: {baseline_present}/{baseline_total}")
for rel in BASELINE_FILES:
mark = "OK " if _check_exists(repo, rel) else "MISS"
typer.echo(f" [{mark}] {rel}")
typer.echo()
typer.echo("control primitives:")
for primitive in Primitive:
present, total = _primitive_status(repo, primitive)
mark = "OK " if present == total else "PARTIAL" if present > 0 else "MISS"
typer.echo(f" [{mark}] {primitive.value}: {present}/{total}")
@primitive_app.command("list")
def primitive_list() -> None:
"""List available control primitives and their files."""
for primitive in Primitive:
typer.echo(f"{primitive.value}")
for rel in PRIMITIVE_FILES[primitive]:
typer.echo(f" - {rel}")
@primitive_app.command("add")
def primitive_add(
primitives: List[Primitive] = typer.Argument(..., help="Primitive names to add."),
repo_path: Path = typer.Option(Path("."), "--repo", "-r", help="Target repository path."),
force: bool = typer.Option(False, "--force", help="Overwrite existing primitive files."),
) -> None:
"""Add specific control primitives to a repository."""
repo = _resolve_repo(repo_path)
_add_primitives(repo, primitives, force=force)
typer.secho("\nPrimitive update complete.", fg=typer.colors.GREEN)
if __name__ == "__main__":
app()
MIT License
Copyright (c) 2026 Carlos Escobar (BroomVA)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
harness-engineering-skill (DEPRECATED — migrated to broomva/skills monorepo)
Status: This repository is in a 6-month deprecation window (until 2026-11-25). The skill has migrated to the broomva/skills monorepo as a Tier-2 vendored skill, renamed toharness-engineering-playbookper ecosystem norm (drops the-skillsuffix).
New install command
npx skills add broomva/skills --skill harness-engineering-playbookSkill home (new)
broomva/skills/skills/harness-engineering-playbook
Why migrated?
Per the Broomva skills packaging strategy, Tier-2 skills consolidate into the broomva/skills monorepo. This reduces repo sprawl, aligns with the 2026 ecosystem norm (anthropics/skills, vercel-labs/skills), and enables atomic cross-skill refactors.
After 2026-11-25
This repository will be archived (read-only on GitHub permanently). All updates ship from broomva/skills.
License
MIT — unchanged.
Related skills
How it compares
Pick harness-engineering-playbook over generic CI templates when the goal is agent-repeatable repo harnesses rather than cloud pipeline orchestration.
FAQ
Does harness-engineering-playbook configure Harness.io CI/CD?
harness-engineering-playbook implements OpenAI Harness Engineering repository practices, not Harness.io pipeline products. The skill bootstraps AGENTS.md, deterministic command surfaces, architecture boundaries, and entropy audits for autonomous coding agent workflows.
What bootstrap profiles does harness-engineering-playbook offer?
harness-engineering-playbook exposes baseline, control, and full profiles via harness_wizard.py init. Baseline adds core harness artifacts, control adds control-system primitives, and full adds entropy controls including nightly audit and entropy checks.