But

GitButler CLI Skill

Use GitButler CLI (but) as the default version-control interface.

Non-Negotiable Rules

1. Use but for all write operations. Never run git add, git commit, git push, git checkout, git merge, git rebase, git stash, or git cherry-pick. If the user says a git write command, translate it to but and run that. 2. After mutations, read the returned output for the updated workspace state — it replaces a follow-up but status -fv. 3. Never chain but mutations with && or ;. Each mutation can reassign CLI IDs, so the second command may silently target the wrong file or commit. Run one mutation, read the returned workspace state, and take fresh IDs from it. 4. Use CLI IDs from but diff / but status -fv / but show; never hardcode IDs. 5. Do not run but status or but status -fv as routine preflight for selected dirty-file or hunk commits. Start with but diff; use but status -fv when existing branch, stack, commit, conflict, or history state matters. 6. For "commit these selected changes on a new branch", prefer one command: but commit <branch> -c -m "<msg>" --changes <ids>.

Choose Inspection By Task

Start with the narrowest inspection that answers the task. Avoid ritual status checks.

# Selected dirty files/hunks:
but diff

# Branch/stack/commit/conflict/history state:
but status -fv

# Details for one known branch or commit:
but show <id>

Do not run plain but status and then but status -fv; that is usually a redundant round-trip.

Perform mutations with IDs from diff, status -fv, or show:

but <mutation> ...

Command Patterns

• Commit: but commit <branch> -m "<msg>" --changes <id>,<id>
• but commit -a is accepted as a no-op compatibility flag; GitButler already includes uncommitted changes by default.
• Commit + create branch: but commit <branch> -c -m "<msg>" --changes <id>
• Amend: but amend <commit-id> --changes <file-or-hunk-id>,<file-or-hunk-id>
• Uncommit and show resulting dirty diff: but uncommit <commit-id> --diff
• Insert empty commit: but commit empty [-m "<msg>"] [<target>]
• Reorder commits: but move <source-commit-id> <target-commit-id> (commit IDs, not branch names)
• Stack branches: but move <branch-name-or-id> <target-branch-name-or-id> (branch names or branch CLI IDs)
• Tear off a branch: but move <branch-name-or-id> zz (zz = unassigned; branch name or branch CLI ID)
• Push: but push <branch-name> — always specify the branch; bare but push pushes ALL branches when run non-interactively
• Pull: but pull --check then but pull

Task Recipes

Update workspace from main

For "get latest from main", "update/sync this workspace", or "pull main":

1. but status -fv 2. but pull --check 3. If clean, but pull 4. but status -fv

but pull updates applied branches onto the latest target branch (usually main). Do not use raw git pull or git rebase.

Commit selected files or hunks

1. but diff — use this first for selective dirty commits. It shows file and hunk IDs for uncommitted changes. 2. Use file IDs when whole files belong in the commit. Use hunk IDs when only part of a file belongs. Do not run plain but status first. 3. For a new branch, use one command: but commit <branch> -c -m "<msg>" --changes <id1>,<id2>. For an existing branch, omit -c: but commit <branch> -m "<msg>" --changes <id1>,<id2>. Omit IDs you don't want committed. Creating a new branch with -c does not require a prior but branch or but status -fv. 4. Check the returned status for remaining uncommitted changes. If the file still appears as unassigned or assigned to another branch after commit, it may be dependency-locked. See "Stacked dependency / commit-lock recovery" below.

Edge case: if wanted and unwanted edits are in the same diff hunk, GitButler cannot split that hunk by ID. Only when the task requires keeping part of that hunk uncommitted, temporarily edit the working tree to isolate the wanted lines, commit with --changes, then restore the leftover lines so they remain uncommitted.

Amend into existing commit

1. but status -fv (or but show <branch-id>) 2. Locate file/hunk IDs and target commit ID. 3. but amend <commit-id> --changes <file-or-hunk-id>,<file-or-hunk-id>; use one command for multiple files/hunks that belong in the same commit.

Reorder commits

but move supports both commit reordering and branch stack operations. Use commit IDs when reordering commits.

1. but status -fv 2. but move <commit-a> <commit-b> — uses commit IDs like c3, c5 3. Refresh IDs from the returned status if you need to keep editing history.

Stack existing branches

To make one existing branch depend on (stack on top of) another, use top-level move:

but move feature/frontend feature/backend

This moves the frontend branch on top of the backend branch in one step.

DO NOT use uncommit + branch delete + branch new -a to stack existing branches. That approach fails because git branch names persist even after but branch delete. Always use but move <branch> <target-branch>.

To unstack (make a stacked branch independent again):

but move feature/logging zz

Note: branch stack/tear-off operations use branch names (like feature/frontend) or branch CLI IDs, while commit reordering uses commit IDs (like c3). Do NOT use but undo to unstack — it may revert more than intended and lose commits.

Stacked dependency / commit-lock recovery

A dependency lock occurs when a file was originally committed on branch A, but you're trying to commit changes to it on branch B. Symptoms:

• but commit succeeds but the file still appears in unassignedChanges in the returned status
• The file still shows as "unassigned" in the status output

Recovery: Stack your branch on the dependency branch, then commit:

1. but status -fv — identify which branch originally owns the file (check commit history). 2. but move <your-branch-name> <dependency-branch-name> — stack your branch on the dependency. Uses full branch names, not CLI IDs. 3. but status -fv — the file should now be committable. Commit it. 4. but commit <branch> -m "<msg>" --changes <id>

If `but move <branch> <target-branch>` fails: Do NOT try uncommit, squash, or undo to work around it — these will leave the workspace in a worse state. Instead, re-run but status -fv to confirm both branches still exist and are applied, then retry with exact branch names from the status output.

Resolve conflicts after reorder/move

NEVER use `git add`, `git commit`, `git checkout --theirs`, `git checkout --ours`, or any git write commands during resolution. Only use but resolve commands and edit files directly with the Edit tool.

If but move causes conflicts (conflicted commits in status):

1. but status -fv — find commits marked as conflicted. 2. but resolve <commit-id> — enter resolution mode. This puts conflict markers in the files. 3. Read the conflicted files to see the <<<<<<< / ======= / >>>>>>> markers. 4. Edit the files to resolve conflicts by choosing the correct content and removing markers. 5. but resolve finish — finalize. Do NOT run this without editing the files first. 6. Repeat for any remaining conflicted commits.

Common mistakes: Do NOT use but amend on conflicted commits (it won't work). Do NOT skip step 4 — you must actually edit the files to remove conflict markers before finishing.

Git-to-But Map

Notes

• Prefer explicit IDs over file paths for mutations.
• --changes accepts comma-separated values (--changes a1,b2) or repeated flags (--changes a1 --changes b2), not space-separated.
• Avoid plain but status in write flows. It is a compact human overview; agents usually need but diff or but status -fv next, so starting with plain status adds a redundant round-trip.
• Read-only git inspection (git log, git blame, git show --stat) is allowed.
• After a successful mutation, trust the workspace state it printed. Re-run but status -fv only if that output lacks the ID you need or files changed since.
• Use but show <branch-id> to see commit details for a branch, including per-commit file changes and line counts.
• Per-commit file counts: but status does NOT include per-commit file counts. Use but show <branch-id> or git show --stat <commit-hash> to get them.
• Avoid --help probes; use this skill and references/reference.md first. Only use --help after a command fails or required syntax is missing from the installed references.
• Run but skill check only when command behavior diverges from this skill, not as routine preflight.
• If but prints an AGENT ACTION REQUIRED skill warning, run the suggested command once, then reload/use the GitButler skill. If it repeats, report it instead of retrying.
• For command syntax and flags: references/reference.md
• For workspace model: references/concepts.md
• For workflow examples: references/examples.md

GitButler CLI Skill

Agent skill for working with the GitButler CLI (but command) in workspace mode.

Installation

Install this skill using the GitButler CLI:

but skill install    # Prompts for scope and format

The command prompts you to choose local (repository) or global (home directory) scope first, then prompts you to select a skill format (Agent Skills / .agents/skills, Claude Code, OpenCode, Codex, GitHub Copilot, Cursor, or Windsurf) and installs to the appropriate location. When run outside a git repository, local scope is unavailable and the default install location is global (home directory), but you can still install to a custom location with --path using an absolute or ~ path.

Options:

• --path <path> - Install to a custom path (outside a repository, relative paths require --global)
• --global - Install globally (in home directory) instead of current repository
• --detect - Auto-detect installation location from existing installation (useful for updates)

In non-interactive mode, use --path or --detect.

Requirements:

• GitButler CLI installed (refer to the docs for installation instructions)
• An AI assistant with skills support such as Agent Skills / .agents/skills, Claude Code, OpenCode, Codex, GitHub Copilot, Cursor, or Windsurf
• Repository initialized with GitButler: but setup (only for local installs)

Updating:

To update the skill to the latest version, use the --detect flag to automatically detect and update your existing installation:

but skill install --detect

Alternatively, re-run the install command and select the same location:

but skill install

This will overwrite the existing skill files with the latest version.

Skill Structure

The skill directory contains both distributable skill files and development documentation:

crates/but/skill/
├── SKILL.md                   ← Skill entry point (INSTALLED)
├── README.md                  ← This file - development docs (NOT installed)
├── RESEARCH.md                ← Tier 4 testing research and strategy (NOT installed)
├── eval/                      ← Tier 4 integration eval harness (NOT installed)
└── references/                ← Additional skill documentation (INSTALLED)
    ├── reference.md           - Command reference
    ├── concepts.md            - Deep concepts
    └── examples.md            - Workflow examples

What gets installed: The but skill install command only copies the distributable files to the user's system:

• SKILL.md - Main skill entry point
• references/ - All reference documentation files

What stays in the repository: Development documentation remains in the source tree and is not installed:

• README.md - This file (development and maintenance docs)
• RESEARCH.md - Testing strategy and learnings
• eval/ - Tier 4 integration test harness and scenarios

When This Skill Is Invoked

Claude automatically invokes this skill when:

• Checking version control state (status, diffs, commits)
• Starting new work (should create branch/stack for each task)
• After making code changes (should commit them to the right branches)
• Committing work (when logical units complete)
• Editing history (amend, squash, move changes)
• Any git-like operation

Progressive Disclosure

Claude loads files on-demand:

1. SKILL.md - Always loaded when skill activates (lean overview) 2. references/reference.md - Loaded when detailed command syntax needed 3. references/concepts.md - Loaded when deeper understanding required 4. references/examples.md - Loaded when workflow examples needed

Files in references/ directory are only loaded when explicitly referenced, keeping context lean while providing comprehensive documentation when needed.

Key Design Principles

Trigger-Rich Description

The YAML description field contains all triggering information so Claude knows when to use this skill before loading the body.

Lean Entry Point

SKILL.md is kept under 150 lines as a "table of contents" that points to detailed materials.

Domain Separation

Separate files by domain (commands, concepts, examples) so Claude only loads relevant context.

Active Language

Uses directive language ("do this") rather than passive ("this might happen").

Maintaining This Skill

When to Update SKILL.md

• New high-level workflow patterns
• Changes to core concepts
• Updates to quick reference commands

When to Update REFERENCE.md

• New but commands
• Changed command syntax
• New flags or options

When to Update CONCEPTS.md

• New conceptual models
• Changes to workspace behavior
• New architectural patterns

When to Update EXAMPLES.md

• New workflow patterns
• Common user questions
• Real-world scenarios

Line Count Guideline

Keep SKILL.md at or under 250 lines. Split content into reference files if approaching the limit.

Testing the Skill

Test that Claude:

1. Invokes skill when starting new work 2. Creates branches before making changes 3. Commits changes to the right branches at logical points 4. Uses but commands instead of git

For automated integration testing against real disposable repositories, use the Tier 4 harness in eval/: Node must satisfy the promptfoo engine range documented in eval/README.md (repo baseline: lts/jod).

cd crates/but/skill/eval
pnpm install --ignore-workspace
pnpm approve-builds --ignore-workspace
pnpm run eval
# Optional: run the same harness with Codex instead of Claude
pnpm run eval:codex

References

• Agent Skills Best Practices
• Claude Code Skills Documentation
• GitButler CLI Documentation

GitButler CLI Key Concepts

Deep dive into GitButler's conceptual model and philosophy.

The Workspace Model

Traditional Git: Serial Branching

main ──┬── feature-a (checkout here, work, commit, checkout back)
       └── feature-b (checkout here, work, commit, checkout back)

• Work on ONE branch at a time
• Switch contexts with git checkout
• Changes are isolated by branch

GitButler: Parallel Stacks

workspace (gitbutler/workspace)
  ├─ feature-a (applied, merged into workspace)
  ├─ feature-b (applied, merged into workspace)
  └─ feature-c (unapplied, not in workspace)

• Work on MULTIPLE branches simultaneously
• No context switching - all applied branches merged in working directory
• Changes are ASSIGNED to branches, not isolated by checkout

Key Implications

1. No `git checkout`: You don't switch between branches. All applied branches exist simultaneously in your workspace.

2. The `gitbutler/workspace` branch: A merge commit containing all applied stacks. Don't interact with it directly - use but commands.

3. Applied vs Unapplied: Control which branches are active:

• Applied branches: In your working directory
• Unapplied branches: Exist but not active
• Use but apply/but unapply to control

CLI IDs: Short Identifiers

Every object gets a short, human-readable CLI ID shown in but status. IDs are generated per-session and are unique across all entity types (no two objects share an ID) — always read them from but status.

Commits:    1b, 8f, c2     (short hex prefixes of the SHA, long enough to be unique)
Branches:   fe, bu, ui     (unique 2–3 char substring of the branch name, e.g. "fe" from "feature-x";
                             falls back to auto-generated ID if no unique substring exists)
Files:      g0, h0, i0     (auto-generated, 2–3 chars)
Hunks:      j0, k1, l2     (auto-generated, 2–3 chars)
Stacks:     m0, n0          (auto-generated, 2–3 chars)

Why? Git commit SHAs are long (40 chars). CLI IDs are short (2-3 chars) and unique within your current workspace context.

Usage: Pass these IDs as arguments to commands:

but commit <branch-id> -m "message"      # Commit to branch
but amend <commit-id> --changes <file-or-hunk-id>,<file-or-hunk-id>  # Amend file(s) or hunk(s) into commit
but rub <commit-id> <commit-id>          # Squash commits

Parallel vs Stacked Branches

Parallel Branches (Independent Work)

Create with but branch new <name>:

main ──┬── api-endpoint (independent)
       └── ui-update    (independent)

Use when:

• Tasks don't depend on each other
• Can be merged independently
• No shared code between them

Example: Adding a new API endpoint and updating button styles are independent.

Stacked Branches (Dependent Work)

To stack an existing branch on top of another: but move <child-branch-name> <parent-branch-name>.

To create a new stacked branch from scratch: but branch new <name> -a <anchor> — only use this when the child branch doesn't exist yet.

main ── authentication ── user-profile ── settings-page
        (base)            (stacked)       (stacked)

Use when:

• Feature B needs code from Feature A
• Building incrementally on previous work
• Creating a series of related changes

Example: User profile page needs authentication to be implemented first.

Stacking two existing branches: If both branches already exist and you need to make one depend on the other, use top-level move:

but move feature/frontend feature/backend
# Now frontend is stacked on top of backend — both in the same stack

To tear off a branch from a stack:

but move feature/frontend zz

Dependency tracking: GitButler automatically tracks which changes depend on which commits. A dependent change can only be committed to the stack that contains the commits it depends on.

The but rub Philosophy

but rub is the core primitive operation: "rub two things together" to perform an action.

What Happens Based on Types

The operation performed depends on what you combine:

zz is a special target meaning "unassigned" (no branch).

Higher-Level Conveniences

These commands are wrappers around but rub:

• but amend = explicitly amend uncommitted files/hunks into a known commit
• but squash = Multiple but rub <commit> <commit> operations
• but move = commit move/reorder with position control, plus branch stack/tear-off (<branch> <target-branch> and <branch> zz)

Why this design? One powerful primitive is easier to understand and maintain than many specialized commands. Once you understand but rub, you understand the editing model.

Dependency Tracking

GitButler tracks dependencies between changes automatically.

How It Works

Commit C1: Added function foo()
Commit C2: Added function bar()
Uncommitted: Call to foo() in new code

The uncommitted change depends on C1 (because it calls foo()).

Implications:

1. Can't commit this change to a stack that doesn't contain C1 2. but absorb will automatically amend it into C1 (or a commit after C1) 3. If you try to move the change, GitButler prevents invalid operations

Why This Matters

Prevents you from creating broken states:

• Can't move dependent code away from its dependencies
• Can't commit changes to the wrong stack
• Ensures each branch remains independently functional

Empty Commits as Placeholders

You can create empty commits:

but commit empty --before c3
but commit empty --after c3

Use cases:

1. Mark future work: Create empty commit as placeholder for changes you'll make 2. Organize history: Add semantic markers in commit history

Example workflow:

but commit empty --before c5 -m "TODO: Add error handling"
# Later, amend the error handling changes into the placeholder
but amend <empty-commit-id> --changes <file-id>

Operation History (Oplog)

Every operation in GitButler is recorded in the oplog (operation log).

What Gets Recorded

• Branch creation/deletion
• Commits
• Rub/squash/move operations
• Push/pull operations

Using Oplog

but oplog                      # View history
but undo                       # Undo last operation
but redo                       # Redo last undone operation
but oplog list --since <snapshot-id>
but oplog list --snapshot
but oplog snapshot -m "known good"
but oplog restore <snapshot-id>  # Restore to specific point

Think of it as "git reflog" but for all GitButler operations, not just branch movements.

Safety net: Made a mistake? but undo it. Experimented and want to go back? but oplog restore to earlier snapshot.

Applied vs Unapplied Branches

Branches can be in two states:

Applied Branches

• Active in your workspace
• Merged into gitbutler/workspace
• Changes visible in working directory
• Can make changes and commit

Unapplied Branches

• Exist but not active
• Not in working directory
• Can't make changes (must apply first)
• Useful for temporarily setting aside work

Controlling State

but apply <branch-name>    # Make branch active
but unapply <id>           # Make branch inactive

Use cases:

• Unapply branches causing conflicts
• Focus on subset of work (unapply others)
• Temporarily set aside work without deleting

Conflict Resolution Mode

When but pull causes conflicts, affected commits are marked as conflicted.

Resolution Workflow

1. Identify: but status shows conflicted commits 2. Enter mode: but resolve <commit-id> 3. Fix conflicts: Edit files, remove conflict markers 4. Check: but resolve status shows remaining conflicts 5. Finalize: but resolve finish or but resolve cancel

During Resolution

• You're in a special mode focused on that commit
• Other GitButler operations are limited
• but status shows you're in resolution mode
• Must finish or cancel before continuing normal work

Read-Only Git Commands

Git commands that don't modify state are safe to use:

Safe (read-only):

• git log - View history
• git diff - See changes (but prefer but diff — it supports CLI IDs)
• git show - View commits
• git blame - See line history
• git reflog - View reference log

Don't use in a GitButler workspace:

• git status - Misleading: shows merged workspace state, not individual stacks; missing CLI IDs that agents need
• git commit - Commits to the workspace merge commit, not your branch
• git checkout - Breaks workspace model
• git rebase - Conflicts with GitButler's management
• git merge - Use but merge instead

Rule of thumb: If it reads, it's fine. If it writes, use but instead.

GitButler CLI Workflow Examples

Real-world examples of common workflows.

Note on CLI IDs: Examples below use illustrative IDs like bu, c3, a1 to keep commands readable. In practice, always read actual IDs from `but status -fv` — they are generated per-session and will differ from these examples. Branch IDs are derived from unique substrings of the branch name (e.g., fe from feature-x), commit IDs use short hex prefixes (e.g., 1b, 8f), and file/hunk/stack IDs are auto-generated (e.g., g0, h0). All IDs are unique across entity types.

Example 1: Starting Independent Parallel Work

Scenario: Need to work on two independent features: a new API endpoint and UI styling updates.

# 1. Check current state
but status -fv

# 2. Create two independent (parallel) branches
but branch new api-endpoint
but branch new ui-styling

# 3. Make changes to multiple files
# (edit api/users.js and components/Button.svelte)

# 4. Check what's unassigned
but status -fv

# 5. Commit specific files directly using --changes (recommended for agents)
# Use CLI ID values from but status -fv output (e.g., branch IDs and file IDs)
# For multiple IDs, use one comma-separated argument or repeat --changes.
but commit <api-branch-id> -m "Add user details endpoint" --changes <api-file-id>
but commit <ui-branch-id> -m "Update button hover styles" --changes <ui-file-id>

# Follow-up fix that belongs in a commit you just made? Amend it in.
# Each mutation returns updated workspace state — take fresh IDs from it before the next command.
# but amend <api-commit-id> --changes <api-fix-file-id>,<api-fix-hunk-id>

# 6. Create pull requests (auto-pushes the branches)
but pr new <api-branch-id>
but pr new <ui-branch-id>

Why parallel branches? The API endpoint and UI styling are independent - neither depends on the other. They can be reviewed and merged separately.

Example 2: Building Stacked Features

Scenario: Need to add authentication, then build a user profile page that requires auth.

# 1. Check current state and update
but pull
but status -fv

# 2. Create base branch for authentication
but branch new add-authentication

# 3. Implement auth and commit
# (edit auth/login.js, auth/middleware.js)
but status -fv
but commit bu -m "Add JWT authentication" --changes <file-ids>

# 4. Create stacked branch anchored on authentication
but branch new user-profile -a bu

# 5. Implement profile page (depends on auth)
# (edit pages/profile.js)
but status -fv
but commit bv -m "Add user profile page" --changes <file-ids>

# 6. Push both branches explicitly (maintains stack relationship)
but push add-authentication
but push user-profile

Result: Two PRs where user-profile PR depends on authentication PR. GitHub/GitLab shows the dependency.

Example 3: Using Absorb Instead of New Commits

Scenario: Made a small typo fix that should be part of the last commit, not a new commit.

# 1. Check current commits and unassigned changes
but status -fv

# Output shows:
# Branch: feature-x (bu)
# Commits:
#   c3: Implement feature logic
#   c2: Add feature tests
# Uncommitted:
#   a1: fix-typo.js

# 2. Preview what absorb would do (recommended first step)
but absorb a1 --dry-run    # Shows where a1 would be absorbed

# 3. Absorb the specific file into appropriate commit
but absorb a1    # Absorb just this file + get updated status

# GitButler analyzes the change and amends it into c3
# (because the typo is in code from c3)

Targeted vs blanket absorb:

but absorb a1    # Absorb specific file (recommended)
but absorb bu    # Absorb all changes assigned to branch bu
but absorb       # Absorb ALL uncommitted changes (use with caution)

Why absorb? Keeps history clean. Small fixes belong in the commits they fix, not as separate "fix typo" commits.

Example 4: Reorganizing Commit History

Scenario A: Squashing Commits

Situation: Made 5 small WIP commits, want to combine into one logical commit.

# Before:
# c5: More tweaks
# c4: Fix another thing
# c3: Fix tests
# c2: Adjust logic
# c1: Initial implementation

# Squash all commits in branch
but squash bu

# Or squash specific range
but squash c2..c5    # Squashes c2, c3, c4, c5 into one

# Or squash specific commits
but squash c2 c3 c4    # Squashes these three

Scenario B: Moving Files Between Commits

Situation: A file was committed in the wrong commit, need to move it.

# 1. See which files are in which commits
but status -fv

# Output shows:
# c3: api.js, utils.js
# c2: config.js

# 2. Move utils.js from c3 to c2
but rub a2 c2    # File a2 (utils.js) → commit c2 + get updated status

Scenario C: Moving Commit to Different Branch

Situation: Committed to wrong branch, need to move commit.

# 1. Check current state
but status -fv

# Output:
# Branch: feature-a (bu)
#   c3: This should be in feature-b!
#   c2: Correct commit

# 2. Create or identify target branch
but branch new feature-b    # Creates branch bv

# 3. Move the commit
but move c3 bv    # Move c3 to top of branch bv

Example 5: Stacking Existing Branches

Scenario: Two independent branches exist, but one now depends on the other. Stack them.

# 1. Check current state — two independent branches in separate stacks
but status -fv

# Output:
# Stack 1: feature/backend (bu) — 2 commits
# Stack 2: feature/frontend (bv) — 1 commit

# 2. Frontend now depends on backend API — stack frontend on backend
#    IMPORTANT: Prefer full branch NAMES here; branch CLI IDs are also accepted
but move feature/frontend feature/backend

# Result: Both branches are now in the same stack:
# Stack 1: feature/backend → feature/frontend (stacked)

# 3. Continue working — commits go to the right branch
but status -fv
but commit bu -m "Add caching layer" --changes <id>   # To backend
but commit bv -m "Add dialog component" --changes <id> # To frontend

Key point: branch stack moves use branch names (like feature/frontend) or branch CLI IDs. Commit reordering still uses commit IDs.

Example 6: Conflict Resolution

Scenario: After but pull, conflicts appear in a commit.

# 1. Pull updates
but pull

# Output:
# Conflict in commit c3 on branch feature-x

# 2. Check status
but status -fv

# Output:
# Branch: feature-x (bu)
#   c3: Add validation (CONFLICTED)

# 3. Enter resolution mode
but resolve c3

# Output:
# Entering resolution mode for commit c3
# Fix conflicts in: api/users.js, api/validation.js

# 4. Read each conflicted file and edit to resolve
# IMPORTANT: You MUST edit the files — do NOT just run `but resolve finish`
# NEVER use `git add`, `git checkout --theirs/--ours`, or any git write command — just edit the files directly with the Edit tool, then `but resolve finish`
cat api/users.js           # Read to see conflict markers
# (edit to remove <<<<<<< ======= >>>>>>> markers and keep correct content)

# 5. Check progress
but resolve status

# Output:
# Remaining conflicts:
#   api/validation.js

# 6. Continue fixing...
# (resolve last conflict)

# 7. Finalize
but resolve finish

# Back to normal workspace mode

Example 7: Complete Feature Development Workflow

Scenario: Building a complete feature from start to finish.

# 1. Update to latest
but pull

# 2. Create branch for feature
but branch new user-dashboard

# 3. Make initial changes
# (create dashboard.js, add routes)

# 4. Check status and gather file IDs
but status -fv

# 5. First commit
but commit bu -m "Add dashboard route and basic layout" --changes <file-ids>

# 6. Continue iterating
# (add widgets, styling)
but commit bu -m "Add dashboard widgets" --changes <file-ids>
but commit bu -m "Style dashboard components" --changes <file-ids>

# 7. Make small fix
# (fix typo in widget)
but absorb a1    # Absorb specific file into appropriate commit

# 8. Clean up if needed
but squash bu    # Combine all commits (optional)

# 9. Create pull request (auto-pushes the branch)
but pr new bu

# Output:
# Created PR #123: https://github.com/org/repo/pull/123

# 10. After PR is merged, update
but pull

Example 8: Working with Applied/Unapplied Branches

Scenario: Have 3 branches, but two are causing conflicts. Temporarily unapply them.

# 1. Check active branches
but status -fv

# Output:
# Applied branches:
#   bu: feature-a
#   bv: feature-b
#   bw: feature-c

# 2. Conflicts between feature-b and feature-c
# Unapply them temporarily
but unapply bv
but unapply bw

# 3. Focus on feature-a
# (make changes, commit)
but commit bu -m "Complete feature-a" --changes <file-ids>

# 4. Create PR for feature-a (auto-pushes)
but pr new bu

# 5. Reapply other branches
but apply feature-b
but apply feature-c

# 6. Deal with their conflicts now
but resolve ...

Example 9: Fixing History Before Pushing

Scenario: Made several commits, realized you need to reword messages and reorder.

# 1. Current state
but status -fv

# Output:
# Branch: feature-x (bu)
#   c5: final commit
#   c4: WIP
#   c3: Fix stuff
#   c2: Another fix
#   c1: Initial

# 2. Reword commit messages
but reword c4 -m "Add validation logic"
but reword c3 -m "Fix edge case in parser"
but reword c2 -m "Update error messages"

# 3. Move c5 to be earlier
but move c5 c3    # Move c5 before c3

# 4. Squash similar commits
but squash c2 c3    # Combine error handling commits

# Output:
# Branch: feature-x (bu)
#   c4: Add validation logic
#   c3: final commit
#   c2: Fix edge case in parser and update error messages
#   c1: Initial

# 5. Push clean history
but push feature-x

Example 10: Daily Development Workflow

Typical day working with GitButler:

# Morning: Start day
but pull    # Get latest from team

# Start new task
but branch new fix-auth-bug  # Create branch for today's work

# Work and commit iteratively
# (make changes)
but status -fv              # Check changes
but commit bu -m "Identify auth bug source" --changes <file-ids>
# (make more changes)
but commit bu -m "Fix token expiration handling" --changes <file-ids>
# (small fix to existing code)
but absorb a1              # Absorb specific fix into appropriate commit

# Mid-day: Start urgent fix on different branch
but branch new hotfix-login  # Parallel branch for urgent work
# (make fix)
but commit bv -m "Fix login redirect loop" --changes <file-ids>
but pr new bv      # Push and create PR immediately

# Back to original work
# (continue working on bu, auth bug fix)
but commit bu -m "Add tests for token handling" --changes <file-ids>

# End of day: Clean up and create PR
but squash bu    # Combine into clean history
but pr new bu      # Push and create PR

# After PR review: Make requested changes
# (make changes based on feedback)
but absorb <file-id>    # Absorb specific changes into commits
# Or absorb all changes for this branch:
but absorb bu          # Absorb all changes assigned to bu
but push fix-auth-bug   # Push updated history

Example 11: Recovering from Mistakes

Scenario: Made changes you didn't mean to, need to undo.

Undo Last Operation

# Made a mistake
but squash bu    # Oops! Didn't mean to squash

# Undo it
but undo         # Reverts the squash

Restore to Earlier Point

# View operation history
but oplog

# Output:
# s5: squash branch bu
# s4: commit bu "message"
# s3: amend a1 into c2
# s2: create branch bu
# s1: pull from remote

# Restore to before squash
but oplog restore s4

Discard Uncommitted Changes

# Changed a file but want to discard
but status -fv

# Output:
# Unassigned:
#   a1: bad-changes.js

# Discard it
but discard a1

Tips and Tricks

Quick Status Check

but status -fv    # File-centric view for quick overview

Preview Before Doing

but absorb <file-id> --dry-run  # See where specific file would be absorbed
but push my-feature --dry-run   # See what would be pushed

Auto-completion

eval "$(but completions zsh)"     # Add to ~/.zshrc
eval "$(but completions bash)"    # Add to ~/.bashrc

Viewing History

but show bu       # Show all commits in branch
git log bu               # Traditional git log (read-only, still works)

GitButler CLI Command Reference

Agent-focused reference for useful but commands.

Contents

• Inspection - status, show, diff
• Branching - branch new, apply, unapply, branch delete, pick
• Committing - commit, absorb
• Editing History - rub, squash, amend, move, uncommit, reword, discard
• Conflict Resolution - resolve
• Remote Operations - push, pull, pr, merge
• Workspace Maintenance - clean
• History & Undo - undo, oplog
• Setup & Configuration - setup, teardown, config, update, skill, gui
• Selected Options

Inspection (Understanding State)

but status

Overview of branch, stack, commit, and workspace state. Use this when you need existing branch/stack/commit/conflict context. For selected dirty-file or hunk commits, start with but diff instead.

but status              # Compact human overview; avoid as routine preflight for write tasks
but status -fv          # File-centric view with full commit details
but status --verbose    # Detailed information
but status --upstream   # Show upstream relationship

Shows:

• Applied/unapplied branches in workspace
• Unassigned and assigned changes
• Commits on each stack
• CLI IDs to use in other commands

but show <id>

Details about a commit or branch.

but show <id>           # Show details
but show <id> --verbose # Show with full messages and file details

but diff [target]

Display diff for file, branch, stack, or commit.

but diff                # Diff for entire workspace; best first command for selective dirty commits
but diff <file-id>      # Diff for specific file
but diff <branch-id>    # Diff for all changes in branch
but diff <commit-id>    # Diff for specific commit

Hunk IDs: For uncommitted changes, but diff shows each hunk with an ID (e.g., e8, j0). Pass these IDs to but commit --changes for fine-grained, hunk-level commits.

Branching

but branch

List all branches (default when no subcommand).

but branch              # List branches
but branch list [filter]  # Filter branches by name (case-insensitive substring)
but branch list --no-ahead  # Skip commits-ahead calculation (faster)
but branch list --no-check  # Skip clean-merge check (faster)
but branch list -r      # Show only remote branches
but branch list -l      # Show only local branches
but branch list -a      # Show all branches (not just active + 20 most recent)
but branch list --empty  # Include empty branches
but branch list --review  # Fetch and display review information

but branch new [name]

Create a new branch.

but branch new                      # Generated branch name
but branch new feature              # Independent branch (parallel work)
but branch new feature -a <anchor>  # Stacked branch (dependent work)

Use parallel branches for independent tasks. Use stacked branches when work depends on another branch.

For "commit these selected changes on a new branch", prefer but commit <branch> -c -m "message" --changes <ids> instead of a separate but branch new or preflight but status -fv.

but apply <branch-name>

Activate a branch in the workspace.

but apply feature-branch  # Activate branch in workspace

Applied branches are merged into gitbutler/workspace and visible in working directory.

but unapply <id>

Deactivate a branch from the workspace.

but unapply <id>         # Deactivate branch from workspace

The identifier can be a CLI ID pointing to a stack or branch, or a branch name. If a branch is specified, the entire stack containing that branch will be unapplied.

but branch delete <id>

Delete a branch.

but branch delete <id>
but branch -d <id>      # Short form

but branch show <id>

Show commits ahead of base for a branch.

but branch show <id>
but branch show <id> -f       # Show files modified in each commit with line counts
but branch show <id> --ai     # Generate AI summary of branch changes
but branch show <id> --check  # Check if branch merges cleanly into upstream
but branch show <id> -r       # Fetch and display review information

but pick <source> [target]

Cherry-pick commits from unapplied branches into applied branches.

but pick <commit-sha> <branch>       # Pick specific commit into branch
but pick <cli-id> <branch>           # Pick using CLI ID (e.g., "c5")
but pick <unapplied-branch>          # Interactive commit selection from branch
but pick <commit-sha>                # Auto-select target if only one branch

The source can be:

• A commit SHA (full or short)
• A CLI ID from but status
• An unapplied branch name (shows interactive commit picker)

If no target is specified and multiple branches exist, prompts for selection interactively.

Committing

but commit [branch]

Commit changes to a branch.

but commit <branch> -m "message"         # Commit ALL uncommitted changes to branch
but commit <branch> -am "message"        # Accepted Git muscle-memory form; -a is a no-op
but commit <branch> -m "message" --changes <id>,<id>  # Commit specific files or hunks by CLI ID
but commit <branch> -m "message" --changes <id> --changes <id>  # Alternative: repeat flag
but commit <branch> --message-file msg.txt  # Read commit message from file
but commit <branch> -c -m "message"      # Create new branch (or use existing) and commit
but commit <branch> -n -m "message"      # Bypass git commit hooks (pre-commit, commit-msg, post-commit)
but commit empty                         # Insert empty commit at top of first branch
but commit empty -m "message"            # Insert empty commit with message
but commit empty <target>                # Insert empty commit before target
but commit empty --before <target>       # Insert empty commit before target
but commit empty --after <target>        # Insert empty commit after target

Important: Plain but commit <branch> -m commits ALL uncommitted changes to the branch. Use --changes to commit only specific files or hunks.

Committing specific files or hunks: Start with but diff for selective dirty commits, then use --changes (or -p) with comma-separated CLI IDs to commit only those files or hunks:

• File IDs from but diff or but status -fv: commits entire files
• Hunk IDs from but diff: commits individual hunks
• --changes takes one argument per flag. Use --changes a1,b2 or --changes a1 --changes b2, not --changes a1 b2.

Creating branches on commit: Use -c / --create to create a new branch for the commit. If the branch name matches an existing branch, that branch is used instead.

Example: but commit my-branch -m "Fix bug" --changes ab,cd commits files/hunks ab and cd.

Example new branch: but commit feature/contact-form -c -m "Validate contact form input" --changes ab,cd creates feature/contact-form and commits only those selected file or hunk IDs.

To commit specific hunks from a file with multiple changes, use but diff to see hunk IDs, then specify them individually.

Edge case: if wanted and unwanted edits are in the same hunk, GitButler cannot split that hunk by ID. Only when the task requires keeping part of that hunk uncommitted, temporarily edit the working tree to isolate the wanted lines, commit with --changes, then restore the leftover lines so they remain uncommitted.

If only one branch is applied, you can omit the branch ID.

but absorb [source]

Automatically amend uncommitted changes into existing commits.

but absorb <file-id>          # Absorb specific file (recommended)
but absorb <branch-id>        # Absorb all changes assigned to this branch
but absorb                    # Absorb ALL uncommitted changes (use with caution)
but absorb --dry-run          # Preview without making changes
but absorb <file-id> --dry-run  # Preview specific file absorption

Recommendation: Prefer targeted absorb (but absorb <file-id>) over absorbing everything. Running but absorb without arguments absorbs ALL uncommitted changes across all branches, which may not be what you want.

Logic:

• Changes amended into topmost commit of their branch
• Changes depending on specific commit amended into that commit
• Uses smart matching to find appropriate commits

Editing History

but rub <source> <dest>

Universal editing primitive that does different operations based on types.

but rub <file> <commit>      # Amend file into commit
but rub <commit> <commit>    # Squash commits together
but rub <commit> <branch>    # Move commit to branch
but rub <commit> zz          # Undo commit to unassigned
but rub zz <commit>          # Amend all unassigned changes into commit
but rub <file-in-commit> zz  # Uncommit specific file from its commit
but rub <file-in-commit> <commit>  # Move file from one commit to another

The core "rub two things together" operation. zz is a special target meaning "unassigned" (no branch).

but squash <commits>

Squash commits together.

but squash <c1> <c2> <c3>    # Squash multiple commits (into last)
but squash <c1>..<c4>        # Squash a range
but squash <branch>          # Squash all commits in branch into bottom-most
but squash <branch> -d       # Squash and drop source commit messages (keep target's)
but squash <branch> -m "msg" # Squash with a new commit message
but squash <branch> -i       # Squash with AI-generated commit message

but amend <commit> --changes <file>[,<file>...]

Amend one or more files/hunks into a specific commit. Use when you know exactly which commit the change belongs to.

but amend <commit-id> --changes <file-id>,<hunk-id>

When to use `amend` vs `absorb`:

• but amend - You know the target commit; explicit control
• but absorb - Let GitButler auto-detect the target; smart matching based on dependencies

Convenience wrapper around rub for amending uncommitted files or hunks into a known commit.

but move <source> <target>

Move commits or branches to a different location.

but move <commit> <target-commit>            # Move before target commit
but move <commit>,<commit> <target-commit>   # Move multiple commits before target
but move <commit> <target-commit> --after    # Move after target commit
but move <commit> <branch>                   # Move commit to top of branch
but move <branch> <target-branch>            # Stack branch on top of target branch
but move <branch> zz                          # Tear off (unstack) branch

--after is valid only for commit-to-commit moves.

but uncommit <source>

Uncommit changes back to unassigned changes.

but uncommit <commit-id>      # Uncommit entire commit
but uncommit <file-id>        # Uncommit specific file from its commit
but uncommit <commit-id> --diff  # Also show resulting dirty diff with hunk IDs
but uncommit <commit-id> -d   # Discard committed changes instead of moving to unassigned
but uncommit <file-id> --discard  # Discard committed file changes completely

Use --diff when you plan to recommit selected files or hunks immediately after uncommitting.

but reword <id>

Reword commit message or rename branch.

but reword <id>               # Interactive editor
but reword <id> -m "new"      # Non-interactive
but reword <id> --fix-formatting  # Format to 72-char wrapping

but discard <id>

Discard uncommitted changes.

but discard <file-id>         # Discard file changes
but discard <hunk-id>         # Discard hunk changes

Conflict Resolution

When commits have conflicts (shown in but status — look for commits marked as conflicted):

but resolve <commit>

Enter resolution mode for a conflicted commit.

but resolve <commit-id>

but resolve status

Show remaining conflicted files.

but resolve status

but resolve finish

Finalize conflict resolution.

but resolve finish

but resolve cancel

Cancel conflict resolution and return to workspace mode.

but resolve cancel
but resolve cancel --force

Workflow:

1. but status — identify conflicted commits (marked as conflicted in the output) 2. but resolve <commit-id> — enter resolution mode for the conflicted commit 3. Edit the conflicted files — remove <<<<<<<, =======, >>>>>>> markers and keep the correct content 4. but resolve status — verify no conflicts remain 5. but resolve finish — finalize and return to normal mode 6. If multiple commits are conflicted, repeat steps 2-5 for each one

Important: Never use git add, git commit, or other git write commands during conflict resolution. Only use but resolve commands and edit files directly.

Remote Operations

but push <branch>

Push a branch to remote. Always specify which branch to push: without one, but push prompts for a selection in interactive terminals and pushes ALL branches with unpushed commits otherwise. Accepts a full branch name or a branch CLI ID — prefer the name; it stays valid across mutations.

but push <branch-name>             # Push specific branch
but push <branch-name> --dry-run   # Preview what would be pushed
but push <branch-name> -s          # Skip force push protection checks
but push <branch-name> --no-hooks  # Bypass pre-push hooks (--no-verify also works)

Force push is enabled by default with protection checks. Use -s only when intentionally skipping those checks.

but pull

Update applied branches onto the latest target branch changes (usually main). Use this for "get latest from main" in a GitButler workspace.

but pull                      # Fetch and rebase applied branches
but pull --check              # Check if can merge cleanly (no changes)

Run but pull --check first, then but pull if clean. Do not use raw git pull or git rebase.

but pr

Create and manage pull requests.

but pr new <branch-id>        # Push branch and create PR (recommended)
but pr new <branch-id> -F pr_message.txt    # Use file: first line is title, rest is description
but pr new <branch-id> -m "Title..."        # Inline message: first line is title, rest is description
but pr new <branch-id> -t     # Use default content (commit message), skip prompts
but pr new <branch-id> --draft  # Create as draft
but pr new <branch-id> --no-hooks  # Bypass pre-push hooks (--no-verify also works)
but pr new <branch-id> -s     # Skip force-push protection checks
but pr --draft                # Top-level draft flag
but pr auto-merge <selector>  # Enable auto-merge
but pr set-draft <selector>   # Mark review as draft
but pr set-ready <selector>   # Mark review as ready

Key behavior: but pr new automatically pushes the branch to remote before creating the PR. No need to run but push first. Force push and pre-push hooks run by default. Use --no-hooks to bypass pre-push hooks when needed.

Selectors for auto-merge, set-draft, and set-ready can be branch names, branch IDs, stack IDs, or numeric review IDs, comma-separated.

In non-interactive environments, use --message (-m), --file (-F), or --default (-t) to avoid editor prompts. The -t flag uses the commit message as title/description for single-commit branches; for multi-commit branches it falls back to the branch name as the title.

Note: For stacked branches, the custom message (-m or -F) only applies to the selected branch. Dependent branches in the stack will use default messages (commit title/description).

Requires forge integration to be configured via but config forge auth.

but merge <branch>

Merge branch into local target branch.

but merge <branch-id>

Merges into local target branch, then runs but pull to update.

Workspace Maintenance

but clean

Remove empty branches from the workspace.

but clean                   # Delete all empty branches
but clean --dry-run         # Preview which branches would be deleted
but clean --pull            # Pull latest changes first, then clean
but clean --include-upstream # Also remove branches with upstream-only commits

A branch is considered empty if it has no local commits and no assigned changes. Branches with upstream-only commits are preserved by default unless --include-upstream is used.

The entire operation is a single oplog entry — use but undo to restore all deleted branches.

History & Undo

but undo / but redo

Undo or redo operations.

but undo
but redo

but oplog

View operation history.

but oplog
but oplog list --since <snapshot-id>
but oplog list --snapshot
but oplog snapshot -m "known good"

Shows all operations with snapshot IDs.

but oplog restore <snapshot>

Restore to a specific oplog snapshot.

but oplog restore <snapshot-id>

Setup & Configuration

but setup

Initialize GitButler in current git repository.

but setup
but setup --init              # Also initialize a new git repo if none exists

Converts regular git repo to use GitButler workspace model. Use --init in non-interactive environments (CI/CD) to ensure a git repository exists before setup.

but teardown

Exit GitButler mode and return to normal git workflow.

but teardown

but config

View and manage GitButler configuration.

but config
but config user               # Also: forge, target, metrics, ui, ai
but config ai openai          # Also: anthropic, ollama, lmstudio, openrouter

but update check

Manage GitButler CLI and app updates.

but update check
but update install
but update install [nightly|release|0.18.7]

but skill

Manage installed GitButler skill files.

but skill check
but skill check --update
but skill install --detect

but gui [path]

Open the GitButler desktop app for a project directory.

but gui                     # Open the current directory in the app
but gui ../other-repo       # Open a specific project directory
but gui --new-window        # Open the current project in a new app window
but gui -n ../other-repo    # Short flag for opening another project in a new window

Selected Options

Useful to agents:

• -C, --current-dir <PATH> - Run as if started in different directory
• -h, --help - Show help for command. Avoid routine help probes; use this reference first.

External commands (PATH helpers)

Important: Not available for Windows yet

Similar to Git, if <command> is not a built-in but command and but-<command> exists on PATH, but runs that executable instead (for example but forecast … invokes but-forecast …).

Restriction: <command> must consist of characters in the set [a-zA-Z_-]

Getting More Help

but --help                    # List all commands
but <subcommand> --help       # Detailed help for specific command

Use help only after a command fails or the installed references do not contain the syntax you need.

Full documentation: <https://docs.gitbutler.com/cli-overview>

Skill File Evaluation Research

How do we know if our skill file is "good"? How do we prove changes improve it? This document synthesizes industry research into a concrete strategy for quantitatively testing the but CLI skill.

The Problem

Every change to SKILL.md is gut-feel based. We can read it and think "this seems clearer," but we have no way to measure whether agents actually behave better after a change. We need:

1. Metrics — What to measure 2. Test cases — What scenarios to test 3. Infrastructure — How to run tests reproducibly 4. Regression tracking — How to know if a change helped or hurt

Current Direction

As of February 7, 2026, this project is operating in a Tier 4-first mode:

• Tier 4 integration tests are the main source of truth for skill quality
• Tier 1/2/3 are secondary tools for diagnostics and faster iteration
• A Tier 4 smoke run is expected for skill-file changes

Industry Landscape

How Others Test Agent Tool Use

Berkeley Function-Calling Leaderboard (BFCL) — The most comprehensive public benchmark for LLM tool use. Evaluates across dimensions: tool selection (did it pick the right tool?), parameter accuracy (right arguments?), relevance detection (correctly refusing irrelevant tools), and multi-turn chaining. Uses AST comparison of generated function calls against ground truth.

SWE-bench — Tests end-to-end coding agent performance on real GitHub issues. Binary pass/fail based on whether the agent's patch passes the test suite. Key insight: test the outcome (did the task succeed?), not just individual tool calls.

MCP-Bench — Benchmarks tool-using LLM agents across tool appropriateness (right tool for the subtask) and parameter accuracy (correct/complete arguments). Uses prompt shuffling to control for ordering bias in evaluation.

DeepEval Tool Correctness — Deterministic metric: correctness = correctly_used_tools / total_tools_called. Evaluates tool selection, input parameters, and output accuracy. Can layer LLM-as-judge for optimality assessment.

How Anthropic Recommends Testing Skills

From Anthropic's skill authoring best practices:

Create evaluations BEFORE writing extensive documentation. This ensures your Skill solves real problems rather than documenting imagined ones.

Their recommended evaluation-driven development loop:

1. Identify gaps — Run Claude on representative tasks without a skill. Document specific failures 2. Create evaluations — Build scenarios that test these gaps 3. Establish baseline — Measure performance without the skill 4. Write minimal instructions — Only enough to pass evaluations 5. Iterate — Run evals, compare against baseline, refine

Evaluation structure they suggest:

{
  "skills": ["but"],
  "query": "Commit the auth changes to the feature branch",
  "files": ["src/auth.rs"],
  "expected_behavior": [
    "Runs but status --format json to check workspace state",
    "Uses but commit with --changes flag for specific files",
    "Includes --format json and --status-after flags"
  ]
}

They also recommend testing with all model tiers (Haiku, Sonnet, Opus) since skill effectiveness varies by model capability.

Available Frameworks

promptfoo + Claude Agent SDK Integration

promptfoo has a dedicated Claude Agent SDK provider that runs agents with full tool access:

providers:
  - id: anthropic:claude-agent-sdk
    config:
      model: claude-sonnet-4-5-20250929
      working_dir: ./test-repo
      append_allowed_tools: ['Bash', 'Read', 'Edit']
      permission_mode: acceptEdits
      max_turns: 10
      max_budget_usd: 0.25
      ask_user_question:
        behavior: first_option  # Auto-answer prompts

tests:
  - vars:
      prompt: "Commit my auth changes"
    assert:
      - type: contains
        value: "but commit"
      - type: not-contains
        value: "git commit"
      - type: cost
        threshold: 0.25
      - type: javascript
        value: |
          const text = String(output).toLowerCase();
          return text.includes('--format json') && text.includes('--status-after');

Metrics That Matter

Based on industry standards and our specific skill file, here are the metrics to track:

Core Metrics

Derived Metrics

What the Industry Measures (for reference)

From the academic survey literature:

• Invocation accuracy — Correct decision on whether to call a tool at all
• Tool selection accuracy — Correct tool from available set
• Parameter accuracy — Correct argument names and values
• Hallucination rate — Invented tools or parameters
• Steps to completion — Number of tool calls to finish
• Error recovery rate — Recoveries / errors encountered

Testing Architecture

Tier 1: Static Analysis (Zero-cost, instant)

Validate skill file structure without calling any LLM. Run as part of cargo test.

What to check:

• YAML frontmatter is valid and meets Anthropic's constraints (name <=64 chars, description <=1024 chars)
• All referenced files exist (references/reference.md, etc.)
• Code examples are internally consistent (every mutation command example includes --format json --status-after)
• No contradictions between SKILL.md and reference files
• Translation table covers all commands mentioned in reference.md
• Line count stays under 250 (our budget)
• Description field contains key trigger words

Implementation: Rust tests in crates/but or a simple script that parses SKILL.md.

Tier 2: Single-Turn Tool Selection (Low-cost, fast)

Test whether Claude picks the right first command given a user prompt and the skill file as context. Uses the Anthropic API with mock tool definitions.

Pattern:

Input: skill_file + user_prompt → Model → First tool call
Score: Does the tool call match expectations?

Example test cases:

Implementation options:

1. promptfoo (recommended) — YAML config, built-in assertions, HTML reports 2. Custom Python script — Maximum control, use anthropic SDK 3. Claude CLI — claude -p "prompt" --output-format json

Tier 3: Multi-Turn Workflow (Medium-cost, comprehensive)

Test complete workflows with mock tool execution. This is the highest-signal tier.

Pattern:

1. Provide skill file as system context
2. Send user prompt
3. Model returns tool call → feed mock result
4. Model returns next tool call → feed mock result
5. Repeat until model finishes
6. Score the entire command sequence

Example scenarios:

Scenario: Basic commit flow

User: "I just finished implementing auth. Commit it."
Expected sequence:
  1. but status --format json           (check state)
  2. but commit <branch> -m "..." --changes <id>,<id> --format json --status-after  (commit)
Assertions:
  - Step 1 happens before step 2
  - Commit includes --changes (not bare commit)
  - Commit includes --format json --status-after
  - No git commands used

Scenario: New feature workflow

User: "Add a dark mode feature"
Expected sequence:
  1. but status --format json           (check state)
  2. but branch new dark-mode    (create branch)
  3. [file edits happen]
  4. but commit ... --changes ... --format json --status-after
Assertions:
  - Branch created before any commits
  - Commit targets the new branch

Scenario: Multiple independent features

User: "Add API endpoint and update UI styling"
Expected:
  - Two parallel branches created
  - Files staged to appropriate branches
  - Separate commits per branch

Mock tool execution:

The key insight from the research: you don't need to run real commands. Mock the tool results:

def mock_bash(command: str) -> str:
    if "but status --format json" in command:
        return json.dumps({
            "unassignedChanges": [
                {"cliId": "a1", "filePath": "src/auth.rs", "changeType": "modified"}
            ],
            "stacks": [{"cliId": "su", "branches": [{"cliId": "bu", "name": "main"}]}]
        })
    if "but commit" in command:
        return json.dumps({"result": {"commitId": "abc123"}, "status": {...}})
    if "but branch new" in command:
        return json.dumps({"result": {"branchId": "bv", "name": "..."}})
    return "unknown command"

This gives full control, deterministic scoring, and low cost (can use Sonnet/Haiku).

Tier 3 Implementation Notes

Key insight: Tier 3 tests the skill file, not the `but` CLI. No but binary runs. No git repo exists. The mock handlers return canned JSON that looks like but status --format json output. You're measuring whether SKILL.md teaches the model correctly — complementary to Tier 1's structural validation.

                    ┌─────────────┐
  SKILL.md ───────► │  LLM (API)  │ ◄──── user prompt
  (system context)  └──────┬──────┘
                           │
                      tool_use: "but status --format json"
                           │
                    ┌──────▼──────┐
                    │ Mock handler │ ──► canned JSON
                    └──────┬──────┘
                           │
                      tool_use: "but commit ... --changes a1 --format json --status-after"
                           │
                    Score: did the command sequence follow SKILL.md rules?

Tier 3 remains useful for cheap, deterministic diagnostics, but this project gates on Tier 4 integration.

Current Tier 4 scenario set (7 scenarios):

Tier 4: Integration (High-cost, realistic)

Run Claude Code against a real test repository with the latest but binary and skill files. Unlike Tier 3's mocks, this tests the full stack: skill file → agent behavior → actual CLI execution → real repo state changes.

What makes this different from Tier 3:

Current Harness Implementation

The current Tier 4 harness lives in crates/but/skill/eval/ and uses:

• providers/but-integration.ts for real Agent SDK execution with Bash hook traces
• promptfooconfig.yaml for scenario data
• assertions/but-assertions.ts for shared assertion functions (file://...:functionName)
• setup-fixture.sh for disposable repo setup and skill installation

How to Run

cd crates/but/skill/eval

# One run (PR smoke)
pnpm run eval

# Repeated run (nightly/pre-release)
pnpm run eval:repeat

# View report UI
pnpm run view

Field Learnings (Observed February 7, 2026)

Running the real Tier 4 harness surfaced a few practical issues that are not obvious from design alone.

Measured baselines from live runs:

• 2026-02-07 eval eval-5T4-2026-02-07T17:57:51: 4/5 pass (80%) on the initial suite.
• 2026-02-07 eval eval-8oX-2026-02-07T19:26:09: 1/7 pass (14.29%) after adding stricter assertions plus amend/reorder scenarios.
• 2026-02-07 eval eval-qCp-2026-02-07T19:28:35: 2/7 pass (28.57%) after iterative scenario tuning.
• 2026-02-07 eval eval-lG2-2026-02-07T19:32:49: 2/7 pass (28.57%) after adding repo-state ordering validation for reorder.

What this indicates:

• Most current failures are real policy gaps, not harness bugs: the agent still leaks into raw git writes (git add, git commit, git push, git checkout) under several prompts.
• Amend and reorder scenarios are representative for but because they exercise explicit history-editing commands (amend, move/rub) where fallback behavior is currently weak.

Implementation fixes required for stable Tier 4 runs:

1. promptfoo `javascript` assertion return shape is version-sensitive.

• In promptfoo 0.119.x, returning { pass, reason } from type: javascript caused failures.
• Returning a plain boolean stabilized assertions across runs.

2. Do not mutate `process.env` globally in a concurrent provider.

• Parallel test execution caused cross-test races when setting E2E_TEST_APP_DATA_DIR globally.
• Fix: pass a per-invocation env object into SDK query/options and child processes.

3. Canonicalize fixture paths before `but setup`.

• but setup stored /private/var/... while later status checks used /var/..., causing "Setup required" lookup failures.
• Fix: normalize fixture path with pwd -P in setup-fixture.sh.

4. Keep fixture support files out of Git status.

• .but-data/ and installed .claude/skills/ content polluted but status --format json and changed CLI IDs.
• Fix: add .but-data/, .claude/, .tmp/ to .git/info/exclude in each fixture.

5. Fixture cleanup should be best-effort.

• Rare ENOTEMPTY races during directory deletion can fail otherwise-successful evals.
• Fix: treat cleanup errors as non-fatal in provider finally blocks.

Behavioral takeaway:

• Tier 4 correctly catches real regressions that Tier 3 mocks can miss (for example, model fallback to raw git push despite skill intent).
• Keep the git-synonym scenario as a required gate for future skill revisions.

Cost and Cadence

• ~$0.10-0.50 per scenario (real Claude Code turns against a real repo)
• Full suite of 7 scenarios × 3 repeats is roughly in the same order of magnitude (typically low single-digit USD to low double-digit USD depending on turn count)
• PRs touching crates/but/skill/: run Tier 4 smoke once (--repeat 1)
• Nightly or pre-release: run Tier 4 with repeats (--repeat 3 or higher)
• Keep Tier 2/3 as supplemental diagnostics, not primary gates

Why TypeScript, Not Rust

This project is a Rust codebase, but Tier 4 evals are TypeScript. The reason is practical:

• The Claude Agent SDK (which provides PostToolUse hooks for command trace capture) only exists in TypeScript and Python — there is no Rust SDK
• Without hooks, you're limited to std::process::Command::new("claude").arg("-p") which gives you final output but no command trace — strictly less informative
• The promptfoo ecosystem (YAML configs, --repeat, HTML reports, assertions) is JS/TS-native
• The eval harness is ~80 LOC of glue code calling external processes — Rust's strengths (performance, safety) don't apply here

For Tier 3 (mock tool execution), Rust is viable since it just calls the Anthropic API directly — no Agent SDK needed. The tradeoff is losing promptfoo's reporting infrastructure. For Tier 4, TypeScript is the clear choice.

Recommended Strategy

Tier 4-First Rollout

1. Keep Tier 4 as the default evaluator for skill changes. 2. Treat a 7-scenario Tier 4 smoke run (`--repeat 1`) as the PR gate for changes under crates/but/skill/. 3. Run repeated Tier 4 (`--repeat 3+`) nightly or pre-release to catch stochastic regressions. 4. Track the key Tier 4 metrics over time: pass rate, git-command leakage rate, --format json and --status-after compliance, and cost per scenario.

Supplemental Layers (Optional)

1. Use Tier 2/Tier 3 only for targeted debugging when Tier 4 fails and root cause is unclear. 2. Use Tier 1 static checks as cheap hygiene, not as confidence signals for behavior. 3. Retain at least one git-synonym scenario ("git push") as a hard regression detector.

Cross-Model Policy

1. Primary model for gating: Sonnet (current operational model). 2. Secondary model sweeps: Haiku/Opus on-demand when major skill rewrites land. 3. Do not block on secondary models by default unless product policy requires it.

Framework Recommendation

Use promptfoo + custom provider as the primary Tier 4 harness. Reasons:

• YAML configuration (low barrier, version-controllable)
• Built-in assertions (contains, not-contains, javascript, llm-rubric, cost, latency)
• --repeat N for statistical significance
• HTML report generation
• Can run from CLI or CI
• Open source

Important nuance: promptfoo's vanilla anthropic provider is single-turn and insufficient for workflow traces. For Tier 4, prefer a custom provider (file://dist/providers/but-integration.js) that wraps Claude Agent SDK hooks and captures Bash command traces.

Use `promptfoo eval` with the custom provider for standard runs, and claude -p only for ad-hoc reproduction.

Scoring Rubric

For each test, produce a composite score:

Aggregate skill score = weighted average across all test cases.

Track this score over time. Every SKILL.md change should improve or maintain the score.

Statistical Considerations

LLM outputs are non-deterministic. To get meaningful signal:

• Run each test at least 3-5 times (promptfoo --repeat flag)
• Use temperature 0 for eval runs to reduce variance
• Report mean + std deviation for each metric
• Need ~30 runs per variant at temperature>0 for A/B testing
• Watch for prompt sensitivity — small wording changes can have outsized effects

Key References

• Anthropic Skill Authoring Best Practices
• Anthropic Advanced Tool Use
• Claude Code Headless Mode / Agent SDK CLI
• Claude Code CLI Reference — full flag reference (--max-turns, --max-budget-usd, --allowedTools, --output-format, etc.)
• Claude Agent SDK Overview — TypeScript/Python SDK with hooks, sessions, permissions
• Claude Agent SDK Hooks — PreToolUse/PostToolUse for intercepting tool calls
• promptfoo: Evaluate Coding Agents
• promptfoo: Claude Agent SDK Provider
• promptfoo: Custom Providers — callApi() interface for custom execution loops
• Berkeley Function-Calling Leaderboard
• DeepEval Tool Correctness
• LLM Agent Evaluation Survey (KDD 2025)
• Claude's Context Engineering Secrets
• SWE-bench
• MCP-Bench