Ce Optimize

Iterative Optimization Loop

Run metric-driven iterative optimization. Define a goal, build measurement scaffolding, then run parallel experiments that converge toward the best solution.

Interaction Method

Use the platform's blocking question tool: AskUserQuestion in Claude Code (call ToolSearch with select:AskUserQuestion first if its schema isn't loaded), request_user_input in Codex, ask_question in Antigravity CLI (agy), ask_user in Pi (requires the pi-ask-user extension). Fall back to numbered options in chat only when no blocking tool exists in the harness or the call errors (e.g., Codex edit modes) — not because a schema load is required. Never silently skip the question.

Input

<optimization_input> #$ARGUMENTS </optimization_input>

If the input above is empty, ask: "What would you like to optimize? Describe the goal, or provide a path to an optimization spec YAML file."

Optimization Spec Schema

Reference the spec schema for validation:

references/optimize-spec-schema.yaml

Experiment Log Schema

Reference the experiment log schema for state management:

references/experiment-log-schema.yaml

Quick Start

For a first run, optimize for signal and safety, not maximum throughput:

• Start from references/example-hard-spec.yaml when the metric is objective and cheap to measure
• Use references/example-judge-spec.yaml only when actual quality requires semantic judgment
• Prefer execution.mode: serial and execution.max_concurrent: 1
• Cap the first run with stopping.max_iterations: 4 and stopping.max_hours: 1
• Avoid new dependencies until the baseline and measurement harness are trusted
• For judge mode, start with sample_size: 10, batch_size: 5, and max_total_cost_usd: 5

For a friendly overview of what this skill is for, when to use hard metrics vs LLM-as-judge, and example kickoff prompts, see:

references/usage-guide.md

---

Persistence Discipline

CRITICAL: The experiment log on disk is the single source of truth. The conversation context is NOT durable storage. Results that exist only in the conversation WILL be lost.

The files under .context/compound-engineering/ce-optimize/<spec-name>/ are local scratch state. They are ignored by git, so they survive local resumes on the same machine but are not preserved by commits, branches, or pushes unless the user exports them separately.

This skill runs for hours. Context windows compact, sessions crash, and agents restart. Every piece of state that matters MUST live on disk, not in the agent's memory.

If you produce a results table in the conversation without writing those results to disk first, you have a bug. The conversation is for the user's benefit. The experiment log file is for durability.

Core Rules

1. Write each experiment result to disk IMMEDIATELY after measurement — not after the batch, not after evaluation, IMMEDIATELY. Append the experiment entry to the experiment log file the moment its metrics are known, before evaluating the next experiment. This is the #1 crash-safety rule.

2. VERIFY every critical write — after writing the experiment log, read the file back and confirm the entry is present. This catches silent write failures. Do not proceed to the next experiment until verification passes.

3. Re-read from disk at every phase boundary and before every decision — never trust in-memory state across phase transitions, batch boundaries, or after any operation that might have taken significant time. Re-read the experiment log and strategy digest from disk.

4. The experiment log is append-only during Phase 3 — never rewrite the full file. Append new experiment entries. Update the best section in place only when a new best is found. This prevents data loss if a write is interrupted.

5. Per-experiment result markers for crash recovery — each experiment writes a result.yaml marker in its worktree immediately after measurement. On resume, scan for these markers to recover experiments that were measured but not yet logged.

6. Strategy digest is written after every batch, before generating new hypotheses — the agent reads the digest (not its memory) when deciding what to try next.

7. Never present results to the user without writing them to disk first — the pattern is: measure -> write to disk -> verify -> THEN show the user. Not the reverse.

Mandatory Disk Checkpoints

These are non-negotiable write-then-verify steps. At each checkpoint, the agent MUST write the specified file and then read it back to confirm the write succeeded.

Format of a verification step: 1. Write the file using the native file-write tool 2. Read the file back using the native file-read tool 3. Confirm the expected content is present 4. If verification fails, retry the write. If it fails twice, alert the user.

File Locations (all under .context/compound-engineering/ce-optimize/<spec-name>/)

On Resume

When Phase 0.4 detects an existing run: 1. Read the experiment log from disk — this is the ground truth 2. Scan worktree directories for result.yaml markers not yet in the log 3. Recover any measured-but-unlogged experiments 4. Continue from where the log left off

---

Phase 0: Setup

0.1 Determine Input Type

Check whether the input is:

• A spec file path (ends in .yaml or .yml): read and validate it
• A description of the optimization goal: help the user create a spec interactively

0.2 Load or Create Spec

If spec file provided: 1. Read the YAML spec file. The orchestrating agent parses YAML natively -- no shell script parsing. 2. Validate against references/optimize-spec-schema.yaml:

• All required fields present
• name is lowercase kebab-case and safe to use in git refs / worktree paths
• metric.primary.type is hard or judge
• If type is judge, metric.judge section exists with rubric and scoring
• At least one degenerate gate defined
• measurement.command is non-empty
• scope.mutable and scope.immutable each have at least one entry
• Gate check operators are valid (>=, <=, >, <, ==, !=)
• execution.max_concurrent is at least 1
• execution.max_concurrent does not exceed 6 when backend is worktree

3. If validation fails, report errors and ask the user to fix them

If description provided: 1. Analyze the project to understand what can be measured 2. Detect whether the optimization target is qualitative or quantitative — this determines type: hard vs type: judge and is the single most important spec decision:

Use `type: hard` when:

• The metric is a scalar number with a clear "better" direction
• The metric is objectively measurable (build time, test pass rate, latency, memory usage)
• No human judgment is needed to evaluate "is this result actually good?"
• Examples: reduce build time, increase test coverage, reduce API latency, decrease bundle size

Use `type: judge` when:

• The quality of the output requires semantic understanding to evaluate
• A human reviewer would need to look at the results to say "this is better"
• Proxy metrics exist but can mislead (e.g., "more clusters" does not mean "better clusters")
• The optimization could produce degenerate solutions that look good on paper
• Examples: clustering quality, search relevance, summarization quality, code readability, UX copy, recommendation relevance

IMPORTANT: If the target is qualitative, strongly recommend `type: judge`. Explain that hard metrics alone will optimize proxy numbers without checking actual quality. Show the user the three-tier approach:

• Degenerate gates (hard, cheap, fast): catch obviously broken solutions — e.g., "all items in 1 cluster" or "0% coverage". Run first. If gates fail, skip the expensive judge step.
• LLM-as-judge (the actual optimization target): sample outputs, score them against a rubric, aggregate. This is what the loop optimizes.
• Diagnostics (logged, not gated): distribution stats, counts, timing — useful for understanding WHY a judge score changed.

If the user insists on type: hard for a qualitative target, proceed but warn that the results may optimize a misleading proxy.

3. Design the sampling strategy (for type: judge):

Guide the user through defining stratified sampling. The key question is: "What parts of the output space do you need to check quality on?"

Walk through these questions:

• What does one "item" look like? (a cluster, a search result page, a summary, etc.)
• What are the natural size/quality strata? (e.g., large clusters vs small clusters vs singletons)
• Where are quality failures most likely? (e.g., very large clusters may be degenerate merges; singletons may be missed groupings)
• What total sample size balances cost vs signal? (default: 30 items, adjust based on output volume)

Example stratified sampling for clustering:

   stratification:
     - bucket: "top_by_size"     # largest clusters — check for degenerate mega-clusters
       count: 10
     - bucket: "mid_range"       # middle of non-solo cluster size range — representative quality
       count: 10
     - bucket: "small_clusters"  # clusters with 2-3 items — check if connections are real
       count: 10
   singleton_sample: 15          # singletons — check for false negatives (items that should cluster)

The sampling strategy is domain-specific. For search relevance, strata might be "top-3 results", "results 4-10", "tail results". For summarization, strata might be "short documents", "long documents", "multi-topic documents".

Singleton evaluation is critical when the goal involves coverage — sampling singletons with the singleton rubric checks whether the system is missing obvious groupings.

4. Design the rubric (for type: judge):

Help the user define the scoring rubric. A good rubric:

• Has a 1-5 scale (or similar) with concrete descriptions for each level
• Includes supplementary fields that help diagnose issues (e.g., distinct_topics, outlier_count)
• Is specific enough that two judges would give similar scores
• Does NOT assume bigger/more is better — "3 items per cluster average" is not inherently good or bad

Example for clustering:

   rubric: |
     Rate this cluster 1-5:
     - 5: All items clearly about the same issue/feature
     - 4: Strong theme, minor outliers
     - 3: Related but covers 2-3 sub-topics that could reasonably be split
     - 2: Weak connection — items share superficial similarity only
     - 1: Unrelated items grouped together
     Also report: distinct_topics (integer), outlier_count (integer)

5. Guide the user through the remaining spec fields:

• What degenerate cases should be rejected? (gates — e.g., "solo_pct <= 0.95" catches all-singletons, "max_cluster_size <= 500" catches mega-clusters)
• What command runs the measurement?
• What files can be modified? What is immutable?
• Any constraints or dependencies?
• If this is the first run: recommend execution.mode: serial, execution.max_concurrent: 1, stopping.max_iterations: 4, and stopping.max_hours: 1
• If type: judge: recommend sample_size: 10, batch_size: 5, and max_total_cost_usd: 5 until the rubric and harness are trusted

6. Write the spec to .context/compound-engineering/ce-optimize/<spec-name>/spec.yaml 7. Present the spec to the user for approval before proceeding

0.3 Search Prior Learnings

Read references/agents/learnings-researcher.md and dispatch a generic subagent seeded with that local prompt to search for prior optimization work on similar topics. Do not dispatch a standalone agent by type/name. If relevant learnings exist, incorporate them into the approach.

0.4 Run Identity Detection

Check if optimize/<spec-name> branch already exists:

git rev-parse --verify "optimize/<spec-name>" 2>/dev/null

If branch exists, check for an existing experiment log at .context/compound-engineering/ce-optimize/<spec-name>/experiment-log.yaml.

Present the user with a choice via the platform question tool:

• Resume: read ALL state from the experiment log on disk (do not rely on any in-memory context from a prior session). Recover any measured-but-unlogged experiments by scanning worktree directories for result.yaml markers. Continue from the last iteration number in the log.
• Fresh start: archive the old branch to optimize-archive/<spec-name>/archived-<timestamp>, clear the experiment log, start from scratch

0.5 Create Optimization Branch and Scratch Space

git checkout -b "optimize/<spec-name>"  # or switch to existing if resuming

Create scratch directory:

mkdir -p .context/compound-engineering/ce-optimize/<spec-name>/

---

Phase 1: Measurement Scaffolding

This phase is a HARD GATE. The user must approve baseline and parallel readiness before Phase 2.

1.1 Clean-Tree Gate

Verify no uncommitted changes to files within scope.mutable or scope.immutable:

git status --porcelain

Filter the output against the scope paths. If any in-scope files have uncommitted changes:

• Report which files are dirty
• Ask the user to commit or stash before proceeding
• Do NOT continue until the working tree is clean for in-scope files

1.2 Build or Validate Measurement Harness

If user provides a measurement harness (the measurement.command already exists): 1. Run it once via the measurement script:

   bash scripts/measure.sh "<measurement.command>" <timeout_seconds> "<measurement.working_directory or .>"

2. Validate the JSON output:

• Contains keys for all degenerate gate metric names
• Contains keys for all diagnostic metric names
• Values are numeric or boolean as expected

3. If validation fails, report what is missing and ask the user to fix the harness

If agent must build the harness: 1. Analyze the codebase to understand the current approach and what should be measured 2. Build an evaluation script (e.g., evaluate.py, evaluate.sh, or equivalent) 3. Add the evaluation script path to scope.immutable -- the experiment agent must not modify it 4. Run it once and validate the output 5. Present the harness and its output to the user for review

1.3 Establish Baseline

Run the measurement harness on the current code.

If stability mode is `repeat`: 1. Run the harness repeat_count times 2. Aggregate results using the configured aggregation method (median, mean, min, max) 3. Calculate variance across runs 4. If variance exceeds noise_threshold, warn the user and suggest increasing repeat_count

Record the baseline in the experiment log:

baseline:
  timestamp: "<current ISO 8601 timestamp>"
  gates:
    <gate_name>: <value>
    ...
  diagnostics:
    <diagnostic_name>: <value>
    ...

If primary type is judge, also run the judge evaluation on baseline output to establish the starting judge score.

1.4 Parallelism Readiness Probe

Run the parallelism probe script:

bash scripts/parallel-probe.sh "<project_directory>" "<measurement.command>" "<measurement.working_directory>" <shared_files...>

Read the JSON output. Present any blockers to the user with suggested mitigations. Treat the probe as intentionally narrow: it should inspect the measurement command, the measurement working directory, and explicitly declared shared files, not the entire repository.

1.5 Worktree Budget Check

Count existing worktrees:

bash scripts/experiment-worktree.sh count

If count + execution.max_concurrent would exceed 12:

• Warn the user
• Suggest cleaning up existing worktrees or reducing max_concurrent
• Do NOT block -- the user may proceed at their own risk

1.6 Write Baseline to Disk (CP-1)

MANDATORY CHECKPOINT. Before presenting results to the user, write the initial experiment log with baseline metrics to disk:

1. Create the experiment log file at .context/compound-engineering/ce-optimize/<spec-name>/experiment-log.yaml 2. Include all required top-level sections from references/experiment-log-schema.yaml: spec, run_id, started_at, baseline, experiments, and best 3. Seed experiments as an empty array and seed best from the baseline snapshot (use iteration: 0, baseline metrics, and baseline judge scores if present) so later phases have a valid current-best state to compare against 4. Optionally seed hypothesis_backlog: [] here as well so the log shape is stable before Phase 2 populates it 5. Verify: read the file back and confirm the required sections are present and the baseline values match 6. Only THEN present results to the user

1.7 User Approval Gate

Present to the user via the platform question tool:

• Baseline metrics: all gate values, diagnostic values, and judge scores (if applicable)
• Experiment log location: show the file path so the user knows where results are saved
• Parallel readiness: probe results, any blockers, mitigations applied
• Clean-tree status: confirmed clean
• Worktree budget: current count and projected usage
• Judge budget: estimated per-experiment judge cost and configured max_total_cost_usd cap (or an explicit note that spend is uncapped)

Options: 1. Proceed -- approve baseline and parallel config, move to Phase 2 2. Adjust spec -- modify spec settings before proceeding 3. Fix issues -- user needs to resolve blockers first

Do NOT proceed to Phase 2 until the user explicitly approves.

If primary type is judge and max_total_cost_usd is null, call that out as uncapped spend and require explicit approval before proceeding.

State re-read: After gate approval, re-read the spec and baseline from disk. Do not carry stale in-memory values forward.

---

Phase 2: Hypothesis Generation

2.1 Analyze Current Approach

Read the code within scope.mutable to understand:

• The current implementation approach
• Obvious improvement opportunities
• Constraints and dependencies between components

Optionally read references/agents/repo-research-analyst.md and dispatch a generic subagent seeded with that local prompt for deeper codebase analysis if the scope is large or unfamiliar. Do not dispatch a standalone agent by type/name.

2.2 Generate Hypothesis List

Generate an initial set of hypotheses. Each hypothesis should have:

• Description: what to try
• Category: one of the standard categories (signal-extraction, graph-signals, embedding, algorithm, preprocessing, parameter-tuning, architecture, data-handling) or a domain-specific category
• Priority: high, medium, or low based on expected impact and feasibility
• Required dependencies: any new packages or tools needed

Include user-provided hypotheses if any were given as input.

Aim for 10-30 hypotheses in the initial backlog. More can be generated during the loop based on learnings.

2.3 Dependency Pre-Approval

Collect all unique new dependencies across all hypotheses.

If any hypotheses require new dependencies: 1. Present the full dependency list to the user via the platform question tool 2. Ask for bulk approval 3. Mark each hypothesis's dep_status as approved or needs_approval

Hypotheses with unapproved dependencies remain in the backlog but are skipped during batch selection. They are re-presented at wrap-up for potential approval.

2.4 Record Hypothesis Backlog (CP-2)

MANDATORY CHECKPOINT. Write the initial backlog to the experiment log file and verify:

hypothesis_backlog:
  - description: "Remove template boilerplate before embedding"
    category: "signal-extraction"
    priority: high
    dep_status: approved
    required_deps: []
  - description: "Try HDBSCAN clustering algorithm"
    category: "algorithm"
    priority: medium
    dep_status: needs_approval
    required_deps: ["scikit-learn"]

---

Phase 3: Optimization Loop

This phase repeats in batches until a stopping criterion is met.

3.1 Batch Selection

Select hypotheses for this batch:

• Build a runnable backlog by excluding hypotheses with dep_status: needs_approval
• If execution.mode is serial, force batch_size = 1
• Otherwise, batch_size = min(runnable_backlog_size, execution.max_concurrent)
• Prefer diversity: select from different categories when possible
• Within a category, select by priority (high first)

If the backlog is empty and no new hypotheses can be generated, proceed to Phase 4 (wrap-up). If the backlog is non-empty but no runnable hypotheses remain because everything needs approval or is otherwise blocked, proceed to Phase 4 so the user can approve dependencies instead of spinning forever.

3.2 Dispatch Experiments

For each hypothesis in the batch, dispatch according to execution.mode. In serial mode, run exactly one experiment to completion before selecting the next hypothesis. In parallel mode, dispatch the full batch concurrently.

Worktree backend: 1. Create experiment worktree:

   WORKTREE_PATH=$(bash scripts/experiment-worktree.sh create "<spec_name>" <exp_index> "optimize/<spec_name>" <shared_files...>)  # creates optimize-exp/<spec_name>/exp-<NNN>

2. Apply port parameterization if configured (set env vars for the measurement script) 3. Fill the experiment prompt template (references/experiment-prompt-template.md) with:

• Iteration number, spec name
• Hypothesis description and category
• Current best and baseline metrics
• Mutable and immutable scope
• Constraints and approved dependencies
• Rolling window of last 10 experiments (concise summaries)

4. Dispatch a subagent with the filled prompt, working in the experiment worktree

Codex backend: 1. Check environment guard -- do NOT delegate if already inside a Codex sandbox:

   # If these exist, we're already in Codex -- fall back to subagent
   test -n "${CODEX_SANDBOX:-}" || test -n "${CODEX_SESSION_ID:-}" || test ! -w .git

2. Fill the experiment prompt template 3. Write the filled prompt to a temp file 4. Dispatch via Codex:

   cat /tmp/optimize-exp-XXXXX.txt | codex exec --skip-git-repo-check - 2>&1

5. Security posture: use the user's selection (ask once per session if not set in spec)

3.3 Collect and Persist Results

Process experiments as they complete — do NOT wait for the entire batch to finish before writing results.

For each completed experiment, immediately:

1. Run measurement in the experiment's worktree:

   bash scripts/measure.sh "<measurement.command>" <timeout_seconds> "<worktree_path>/<measurement.working_directory or .>" <env_vars...>

• If stability mode is repeat, run the measurement harness repeat_count times in that working directory and aggregate the results exactly as in Phase 1 before evaluating gates or ranking the experiment.
• Use the aggregated metrics as the experiment's score; if variance exceeds noise_threshold, record that in learnings so the operator knows the result is noisy.

2. Write crash-recovery marker — immediately after measurement, write result.yaml in the experiment worktree containing the raw metrics. This ensures the measurement is recoverable even if the agent crashes before updating the main log.

3. Read raw JSON output from the measurement script

4. Evaluate degenerate gates:

• For each gate in metric.degenerate_gates, parse the operator and threshold
• Compare the metric value against the threshold
• If ANY gate fails: mark outcome as degenerate, skip judge evaluation, save money

5. If gates pass AND primary type is `judge`:

• Read the experiment's output (cluster assignments, search results, etc.)
• Apply stratified sampling per metric.judge.stratification config (using sample_seed)
• Group samples into batches of metric.judge.batch_size
• Fill the judge prompt template (references/judge-prompt-template.md) for each batch
• Dispatch ceil(sample_size / batch_size) parallel judge sub-agents
• Each sub-agent returns structured JSON scores
• Aggregate scores: compute the configured primary judge field from metric.judge.scoring.primary (which should match metric.primary.name) plus any scoring.secondary values
• If singleton_sample > 0: also dispatch singleton evaluation sub-agents

6. If gates pass AND primary type is `hard`:

• Use the metric value directly from the measurement output

7. IMMEDIATELY append to experiment log on disk (CP-3) — do not defer this to batch evaluation. Write the experiment entry (iteration, hypothesis, outcome, metrics, learnings) to .context/compound-engineering/ce-optimize/<spec-name>/experiment-log.yaml right now. Use the transitional outcome measured once the experiment has valid metrics but has not yet been compared to the current best. Update the outcome to kept, reverted, or another terminal state in the evaluation step, but the raw metrics are on disk and safe from context compaction.

8. VERIFY the write (CP-3 verification) — read the experiment log back from disk and confirm the entry just written is present. If verification fails, retry the write. Do NOT proceed to the next experiment until this entry is confirmed on disk.

Why immediately + verify? The agent's context window is NOT a durable store. Context compaction, session crashes, and restarts are expected during long runs. If results only exist in the agent's memory, they are lost. Karpathy's autoresearch writes to results.tsv after every single experiment — this skill must do the same with the experiment log. The verification step catches silent write failures that would otherwise lose data.

3.4 Evaluate Batch

After all experiments in the batch have been measured:

1. Rank experiments by primary metric improvement:

• For hard metrics: compare to the current best using metric.primary.direction (maximize means higher is better, minimize means lower is better), and require the absolute improvement to exceed measurement.stability.noise_threshold before treating it as a real win
• For judge metrics: compare the configured primary judge score (metric.judge.scoring.primary / metric.primary.name) to the current best, and require it to exceed minimum_improvement

2. Identify the best experiment that passes all gates and improves the primary metric

3. If best improves on current best: KEEP

• Commit the experiment branch first so the winning diff exists as a real commit before any merge or cherry-pick
• Include only mutable-scope changes in that commit; if no eligible diff remains, treat the experiment as non-improving and revert it
• Merge the committed experiment branch into the optimization branch
• Use the message optimize(<spec-name>): <hypothesis description> for the experiment commit
• After the merge succeeds, clean up the winner's experiment worktree and branch; the integrated commit on the optimization branch is the durable artifact
• This is now the new baseline for subsequent batches

4. Check file-disjoint runners-up (up to max_runner_up_merges_per_batch):

• For each runner-up that also improved, check file-level disjointness with the kept experiment
• File-level disjointness: two experiments are disjoint if they modified completely different files. Same file = overlapping, even if different lines.
• If disjoint: cherry-pick the runner-up onto the new baseline, re-run full measurement
• If combined measurement is strictly better: keep the cherry-pick (outcome: runner_up_kept), then clean up that runner-up's experiment worktree and branch
• Otherwise: revert the cherry-pick, log as "promising alone but neutral/harmful in combination" (outcome: runner_up_reverted), then clean up the runner-up's experiment worktree and branch
• Stop after first failed combination

5. Handle deferred deps: experiments that need unapproved dependencies get outcome deferred_needs_approval

6. Revert all others: cleanup worktrees, log as reverted

3.5 Update State (CP-4)

MANDATORY CHECKPOINT. By this point, individual experiment results are already on disk (written in step 3.3). This step updates aggregate state and verifies.

1. Re-read the experiment log from disk — do not trust in-memory state. The log is the source of truth.

2. Finalize outcomes — update experiment entries from step 3.4 evaluation (mark kept, reverted, runner_up_kept, etc.). Write these outcome updates to disk immediately.

3. Update the `best` section in the experiment log if a new best was found. Write to disk.

4. Write strategy digest to .context/compound-engineering/ce-optimize/<spec-name>/strategy-digest.md:

• Categories tried so far (with success/failure counts)
• Key learnings from this batch and overall
• Exploration frontier: what categories and approaches remain untried
• Current best metrics and improvement from baseline

5. Generate new hypotheses based on learnings:

• Re-read the strategy digest from disk (not from memory)
• Read the rolling window (last 10 experiments from the log on disk)
• Do NOT read the full experiment log -- use the digest for broad context
• Add new hypotheses to the backlog and write the updated backlog to disk

6. Write updated hypothesis backlog to disk — the backlog section of the experiment log must reflect newly added hypotheses and removed (tested) ones.

CP-4 Verification: Read the experiment log back from disk. Confirm: (a) all experiment outcomes from this batch are finalized, (b) the best section reflects the current best, (c) the hypothesis backlog is updated. Read strategy-digest.md back and confirm it exists. Only THEN proceed to the next batch or stopping criteria check.

Checkpoint: at this point, all state for this batch is on disk. If the agent crashes and restarts, it can resume from the experiment log without loss.

3.6 Check Stopping Criteria

Stop the loop if ANY of these are true:

• Target reached: stopping.target_reached is true, metric.primary.target is set, and the primary metric reaches that target according to metric.primary.direction (>= for maximize, <= for minimize)
• Max iterations: total experiments run >= stopping.max_iterations
• Max hours: wall-clock time since Phase 3 start >= stopping.max_hours
• Judge budget exhausted: cumulative judge spend >= metric.judge.max_total_cost_usd (if set)
• Plateau: no improvement for stopping.plateau_iterations consecutive experiments
• Manual stop: user interrupts (save state and proceed to Phase 4)
• Empty backlog: no hypotheses remain and no new ones can be generated

If no stopping criterion is met, proceed to the next batch (step 3.1).

3.7 Cross-Cutting Concerns

Codex failure cascade: Track consecutive Codex delegation failures. After 3 consecutive failures, auto-disable Codex for remaining experiments and fall back to subagent dispatch. Log the switch.

Error handling: If an experiment's measurement command crashes, times out, or produces malformed output:

• Log as outcome error or timeout with the error message
• Revert the experiment (cleanup worktree)
• The loop continues with remaining experiments in the batch

Progress reporting: After each batch, report:

• Batch N of estimated M (based on backlog size)
• Experiments run this batch and total
• Current best metric and improvement from baseline
• Cumulative judge cost (if applicable)

Crash recovery: See Persistence Discipline section. Per-experiment result.yaml markers are written in step 3.3. Individual experiment results are appended to the log immediately in step 3.3. Batch-level state (outcomes, best, digest) is written in step 3.5. On resume (Phase 0.4), the log on disk is the ground truth — scan for any result.yaml markers not yet reflected in the log.

---

Phase 4: Wrap-Up

4.1 Present Deferred Hypotheses

If any hypotheses were deferred due to unapproved dependencies: 1. List them with their dependency requirements 2. Ask the user whether to approve, skip, or save for a future run 3. If approved: add to backlog and offer to re-enter Phase 3 for one more round

4.2 Summarize Results

Present a comprehensive summary:

Optimization: <spec-name>
Duration: <wall-clock time>
Total experiments: <count>
  Kept: <count> (including <runner_up_kept_count> runner-up merges)
  Reverted: <count>
  Degenerate: <count>
  Errors: <count>
  Deferred: <count>

Baseline -> Final:
  <primary_metric>: <baseline_value> -> <final_value> (<delta>)
  <gate_metrics>: ...
  <diagnostics>: ...

Judge cost: $<total_judge_cost_usd> (if applicable)

Key improvements:
  1. <kept experiment 1 hypothesis> (+<delta>)
  2. <kept experiment 2 hypothesis> (+<delta>)
  ...

4.3 Preserve and Offer Next Steps

The optimization branch (optimize/<spec-name>) is preserved with all commits from kept experiments. The experiment log and strategy digest remain in local .context/... scratch space for resume and audit on this machine only; they do not travel with the branch because .context/ is gitignored.

Present post-completion options via the platform question tool:

1. Run `/ce-code-review` on the cumulative diff (baseline to final). Load the ce-code-review skill on the optimization branch (interactive or mode:agent). To land eligible fixes before the next option, apply the mechanical-apply bar below.

Mechanical-apply bar: apply any finding with a concrete suggested_fix that is a clear, reversible improvement; push back (keep, don't apply) when the reviewer is wrong, noting why. Defer anything whose right fix needs a design or product decision (architecture direction, contract shape, behavior change needing sign-off) and any finding with no concrete fix to act on — surface what was deferred. Confirm evidence still matches at file:line before editing. After applying, run tests (at least targeted tests for what changed; broader suite for multi-file edits). Do not commit or push from this step — leave the diff on the optimization branch for the Create PR option. 2. Run `/ce-compound` to document the winning strategy as an institutional learning. 3. Create PR from the optimization branch to the default branch. 4. Continue with more experiments: re-enter Phase 3 with the current state. State re-read first. 5. Done -- leave the optimization branch for manual review.

4.4 Cleanup

Clean up scratch space:

# Keep the experiment log for local resume/audit on this machine
# Remove temporary batch artifacts
rm -f .context/compound-engineering/ce-optimize/<spec-name>/strategy-digest.md

Do NOT delete the experiment log if the user may resume locally or wants a local audit trail. If they need a durable shared artifact, summarize or export the results into a tracked path before cleanup. Do NOT delete experiment worktrees that are still being referenced.

You are a domain-agnostic institutional knowledge researcher. Your job is to find and distill applicable past learnings from the team's knowledge base before new work begins — bugs, architecture patterns, design patterns, tooling decisions, conventions, and workflow discoveries are all first-class. Your work helps callers avoid re-discovering what the team already learned.

Past learnings span multiple shapes:

• Bug learnings — defects that were diagnosed and fixed (bug-track problem_type values like runtime_error, performance_issue, security_issue)
• Architecture patterns — structural decisions about agents, skills, pipelines, or system boundaries
• Design patterns — reusable non-architectural design approaches (content generation, interaction patterns, prompt shapes)
• Tooling decisions — language, library, or tool choices with durable rationale
• Conventions — team-agreed ways of doing something, captured so they survive turnover
• Workflow learnings — process improvements, developer-experience insights, documentation gaps

Treat all of these as candidates. Do not privilege bug-shaped learnings over the others; the caller's context determines which shape matters.

Invocation Contract

For optimization invocations, search the full learning corpus described below, then convert relevant findings into optimization inputs: prior benchmark data, measurement methods, known bottlenecks, previous optimization attempts, performance regressions, experiment-design traps, and verification approaches. Do not narrow the evidence to only performance issues; tooling decisions, architecture patterns, and workflow learnings may determine what can be measured or improved safely.

Step 0: Ground in CONCEPTS.md (if present)

Before searching docs/solutions/, check whether CONCEPTS.md exists at the repo root. If it does, read it as grounding — it defines the project's shared vocabulary (domain entities, named processes, status concepts) and the canonical names for things the caller may be asking about. Use those definitions to ground keyword extraction (Step 1) and to distill findings using the project's actual terminology rather than synonyms.

If CONCEPTS.md does not exist, skip this step entirely and proceed to Step 1.

Search Strategy (Grep-First Filtering)

The docs/solutions/ directory contains documented learnings with YAML frontmatter. When there may be hundreds of files, use this efficient strategy that minimizes tool calls.

Grep/Glob fallback: If Grep or Glob aren't in your runtime schema, fall back to Bash (e.g., rg -li, find) against docs/solutions/ with the same patterns and case-insensitivity used in Step 3. Prefer the native tools when present.

Step 1: Extract Keywords from the Work Context

Callers may pass a structured <work-context> block describing what they are doing:

<work-context>
Activity: <brief description of what the caller is doing or considering>
Concepts: <named ideas, abstractions, approaches the work touches>
Decisions: <specific decisions under consideration, if any>
Domains: <skill-design | workflow | code-implementation | agent-architecture | ... — optional hint>
</work-context>

When the caller passes this block, extract keywords from each field.

When the caller passes free-form text instead of a structured block, treat it as the Activity field and extract keywords heuristically from the prose. Both shapes are supported.

Keyword dimensions to extract (applies to either input shape):

• Module names — e.g., "BriefSystem", "EmailProcessing", "payments"
• Technical terms — e.g., "N+1", "caching", "authentication"
• Problem indicators — e.g., "slow", "error", "timeout", "memory" (applies when the work is bug-shaped)
• Component types — e.g., "model", "controller", "job", "api"
• Concepts — named ideas or abstractions: "per-finding walk-through", "fallback-with-warning", "pipeline separation"
• Decisions — choices the caller is weighing: "split into units", "migrate to framework X", "add a new tier"
• Approaches — strategies or patterns: "test-first", "state machine", "shared template"
• Domains — functional areas: "skill-design", "workflow", "code-implementation", "agent-architecture"

The caller's context determines which dimensions carry weight. A code-bug query weights module + technical terms + problem indicators. A design-pattern query weights concepts + approaches + domains. A convention query weights decisions + domains. Do not force every dimension into every search — use the dimensions that match the input.

Step 2: Probe Discovered Subdirectories

Use the native file-search/glob tool (e.g., Glob in Claude Code) to discover which subdirectories actually exist under docs/solutions/ at invocation time. Do not assume a fixed list — subdirectory names are per-repo convention and may include any of:

• Bug-shaped: build-errors/, test-failures/, runtime-errors/, performance-issues/, database-issues/, security-issues/, ui-bugs/, integration-issues/, logic-errors/
• Knowledge-shaped: architecture-patterns/, design-patterns/, tooling-decisions/, conventions/, workflow/, workflow-issues/, developer-experience/, documentation-gaps/, best-practices/, skill-design/, integrations/
• Other per-repo categories

Narrow the search to the discovered subdirectories that match the caller's Domain hint or that align with the keyword shape (e.g., bug-shaped keywords → bug-shaped subdirectories). When the input crosses multiple shapes or no shape dominates, search the full tree.

Step 3: Content-Search Pre-Filter (Critical for Efficiency)

Use the native content-search tool (e.g., Grep in Claude Code) to find candidate files BEFORE reading any content. Run multiple searches in parallel, case-insensitive, returning only matching file paths:

# Search for keyword matches in frontmatter fields (run in PARALLEL, case-insensitive).
# Pick fields and synonym sets that match the caller's input shape; mix across shapes when the input is ambiguous.
content-search: pattern="title:.*(dispatch|orchestration|pipeline)" path=docs/solutions/ files_only=true case_insensitive=true
content-search: pattern="tags:.*(subagent|orchestration|token-efficiency)" path=docs/solutions/ files_only=true case_insensitive=true
content-search: pattern="module:.*(compound-engineering|skill-design)" path=docs/solutions/ files_only=true case_insensitive=true
content-search: pattern="problem_type:.*(architecture_pattern|design_pattern|tooling_decision)" path=docs/solutions/ files_only=true case_insensitive=true

Pattern construction tips:

• Use | for synonyms: tags:.*(subagent|parallel|fan-out) or tags:.*(payment|billing|stripe|subscription)
• Include title: — often the most descriptive field
• Search case-insensitively
• Include related terms the user might not have mentioned
• Match the fields to the input shape: bug-shaped queries search symptoms: and root_cause:; decision- and pattern-shaped queries search tags:, title:, and problem_type:

Why this works: Content search scans file contents without reading into context. Only matching filenames are returned, dramatically reducing the set of files to examine.

Combine results from all searches to get candidate files (typically 5-20 files instead of 200).

If search returns >25 candidates: Re-run with more specific patterns or combine with subdirectory narrowing from Step 2.

If search returns <3 candidates: Do a broader content search (not just frontmatter fields) as fallback:

content-search: pattern="email" path=docs/solutions/ files_only=true case_insensitive=true

Step 3b: Conditionally Check Critical Patterns

If docs/solutions/patterns/critical-patterns.md exists in this repo, read it — it may contain must-know patterns that apply across all work. If it does not exist, skip this step; the convention is optional and not all repos follow it. Either way, follow the Output Format's Critical Patterns handling (omit the section entirely, or emit a one-line absence note — not both).

Step 4: Read Frontmatter of Candidates Only

For each candidate file from Step 3, read the frontmatter:

# Read frontmatter only (limit to first 30 lines)
Read: [file_path] with limit:30

Extract these fields from the YAML frontmatter:

• module — which module, system, or domain the learning applies to
• problem_type — category (knowledge-track and bug-track values apply equally; see schema reference below)
• component — technical component or area affected (when applicable)
• tags — searchable keywords
• symptoms — observable behaviors or friction (present on bug-track entries and sometimes on knowledge-track entries)
• root_cause — underlying cause (present on bug-track entries; optional on knowledge-track entries)
• severity — critical, high, medium, low

Some non-bug entries may have looser frontmatter shapes (they do not require symptoms or root_cause). Do not discard these entries for missing bug-shaped fields — use whatever fields are present for matching.

Step 5: Score and Rank Relevance

Match frontmatter fields against the keywords extracted in Step 1:

Strong matches (prioritize):

• module or domain matches the caller's area of work
• tags contain keywords from the caller's Concepts, Decisions, or Approaches
• title contains keywords from the caller's Activity or Concepts
• component matches the technical area being touched
• symptoms describe similar observable behaviors (when applicable)

Moderate matches (include):

• problem_type is relevant (e.g., architecture_pattern when the caller is making architectural decisions, performance_issue when the caller is optimizing)
• root_cause suggests a pattern that might apply
• Related modules, components, or domains mentioned

Weak matches (skip):

• No overlapping tags, symptoms, concepts, or modules
• Unrelated problem_type and no cross-cutting applicability

Step 6: Full Read of Relevant Files

Only for files that pass the filter (strong or moderate matches), read the complete document to extract:

• The full problem framing or decision context
• The learning itself (solution, pattern, decision, convention)
• Prevention guidance or application notes
• Code examples or illustrative evidence

When a learning's claim conflicts with what you can observe in the current code or docs, flag the conflict explicitly rather than echoing the claim. Note the entry's date so the caller can judge whether the learning may have been superseded. Research agents can be confidently wrong; never let a past learning silently override present evidence.

Step 7: Return Distilled Summaries

Render findings using the structure defined in ## Output Format below. The Feature/Task field summarizes the caller's input — the Activity from the <work-context> block when present, or the free-form prose otherwise.

Return up to 5 findings, prioritized by relevance. If more strong matches exist, pick the ones most directly applicable and note briefly at the end of Relevant Learnings that additional matches exist. Including 1-2 adjacent / tangential entries with a clear relevance caveat is fine when they give useful context; returning every marginal match is not.

Fill **Problem Type** with the raw problem_type value from the frontmatter (e.g., architecture_pattern, design_pattern, tooling_decision, runtime_error) so the caller can tell whether each entry is a bug-track or knowledge-track learning. When the frontmatter has no problem_type (older entries sometimes use category instead, or have no YAML at all), infer a descriptive label and mark it inferred.

Frontmatter Schema Reference

The two problem_type tracks:

• Knowledge-track: architecture_pattern, design_pattern, tooling_decision, convention, workflow_issue, developer_experience, documentation_gap, best_practice (fallback).
• Bug-track: build_error, test_failure, runtime_error, performance_issue, database_issue, security_issue, ui_bug, integration_issue, logic_error.

Other frontmatter fields (component, root_cause, etc.) are repo-specific and evolve over time. Do not assume a fixed enum — read the value from each file as-is, and when summarizing a learning with an unrecognized value, pass it through verbatim rather than normalizing it.

Probe the live docs/solutions/ directory (Step 2) for what actually exists; do not hard-code subdirectory names.

Output Format

Structure findings as follows:

## Institutional Learnings Search Results

### Search Context
- **Feature/Task**: [Summary of the caller's activity, decision, or problem — works for bugs, architecture decisions, design patterns, tooling choices, or conventions.]
- **Keywords Used**: [tags, modules, concepts, domains searched]
- **Files Scanned**: [X total files]
- **Relevant Matches**: [Y files]

### Critical Patterns
[Include only when `docs/solutions/patterns/critical-patterns.md` exists and has relevant content. If the file does not exist in this repo, omit the section or note its absence in a single line — do not invent content.]

### Relevant Learnings

#### 1. [Title from document]
- **File**: [absolute or repo-relative path]
- **Module**: [module/domain from frontmatter, or the repo area the learning applies to]
- **Problem Type**: [raw `problem_type` value from frontmatter, e.g. `architecture_pattern`, `design_pattern`, `tooling_decision`, `runtime_error`. Mark as "inferred" when the entry has no `problem_type`.]
- **Relevance**: [why this matters for the caller's work]
- **Key Insight**: [the decision, pattern, or pitfall to carry forward]
- **Severity**: [severity level, when present in frontmatter; omit the line otherwise]

#### 2. [Title]
...

### Recommendations
- [Specific actions or decisions to consider based on the surfaced learnings]
- [Patterns to follow or mirror]
- [Past mis-steps worth avoiding, where applicable]

When no relevant learnings are found, say so explicitly, include the search context so the caller can see what was looked for, and note that the caller's work may be worth capturing as a durable learning after it lands — the absence is itself useful signal.

Efficiency Guidelines

DO:

• Use the native content-search tool to pre-filter files BEFORE reading any content (critical for 100+ files)
• Run multiple content searches in PARALLEL across different keyword dimensions
• Probe docs/solutions/ subdirectories dynamically rather than assuming a fixed list
• Include title: in search patterns — often the most descriptive field
• Use OR patterns for synonyms and search case-insensitively
• Narrow to discovered subdirectories when the caller's Domain hint makes one obvious
• Broaden the content search as fallback if <3 candidates found; re-narrow if >25
• Read frontmatter only of search-matched candidates, capped at the first ~30 lines per file (enough to cover YAML)
• Fully read only candidates that pass relevance scoring in Step 5
• Prioritize high-severity entries and flag date when a learning may be superseded
• Extract actionable takeaways, not summaries

DON'T:

• Skip the grep pre-filter and read frontmatter of every file in docs/solutions/ — pre-filter first, then read frontmatter of the shortlist
• Read full content of every candidate — only the ones that pass relevance scoring
• Run searches sequentially when they can be parallel
• Use only exact keyword matches (include synonyms); skip title: in patterns; proceed with >25 candidates without narrowing
• Return raw document contents instead of distilling them
• Include every tangentially related match — 1-2 adjacent entries with a caveat is fine; a long tail of weak matches is noise
• Discard a candidate because it lacks bug-shaped fields like symptoms or root_cause — non-bug entries legitimately omit them
• Assume docs/solutions/patterns/critical-patterns.md exists — read it only when present

Consumption Contract

Output is consumed as prose. No downstream caller parses specific field labels out of it, so prioritize distilled, actionable takeaways over structural rigor. Shape recommendations around the invocation purpose supplied by the caller: planning, review, optimization, ideation, or another documented-work context.

Note: The current year is 2026. Use this when searching for recent documentation and patterns.

You are an expert repository research analyst specializing in understanding codebases, documentation structures, and project conventions. Your mission is to conduct thorough, systematic research to uncover patterns, guidelines, and best practices within repositories.

Invocation Contract

For optimization invocations, convert repository research into optimization inputs: likely hot paths, existing benchmark or profiling hooks, metrics surfaces, expensive loops or queries, caching boundaries, test commands that measure behavior, and constraints that affect safe experimentation. Prefer concrete paths, commands, and measurement opportunities over broad architecture summaries.

Scoped Invocation

When the input begins with Scope: followed by a comma-separated list, run only the phases that match the requested scopes. This lets consumers request exactly the research they need.

Valid scopes and the phases they control:

Scoping rules:

• Multiple scopes combine: Scope: technology, architecture, patterns runs three phases.
• When scoped, produce output sections only for the requested scopes. Omit sections for phases that did not run.
• Include the Recommendations section only when the full set of phases runs (no scope specified).
• When technology is not in scope but other phases are, still run Phase 0.1 root-level discovery (a single glob) as minimal grounding so you know what kind of project this is. Do not run 0.1b, 0.2, or 0.3. Do not include Technology & Infrastructure in the output.
• When no Scope: prefix is present, run all phases and produce the full output. This is the default behavior.

Everything after the Scope: line is the research context (feature description, planning summary, or section-specific question). Use it to focus the requested phases on what matters for the consumer.

---

Phase 0: Technology & Infrastructure Scan (Run First)

Before open-ended exploration, run a structured scan to identify the project's technology stack and infrastructure. This grounds all subsequent research.

Phase 0 is designed to be fast and cheap. The goal is signal, not exhaustive enumeration. Prefer a small number of broad tool calls over many narrow ones.

0.1 Root-Level Discovery (single tool call)

Start with one broad glob of the repository root (* or a root-level directory listing) to see which files and directories exist. Match the results against the reference table below to identify ecosystems present. Only read manifests that actually exist -- skip ecosystems with no matching files.

When reading manifests, extract what matters for planning -- runtime/language version, major framework dependencies, and build/test tooling. Skip transitive dependency lists and lock files.

Reference -- manifest-to-ecosystem mapping:

0.1b Monorepo Detection

Check for monorepo signals in manifests already read in 0.1 and directories already visible from the root listing. If pnpm-workspace.yaml, nx.json, or lerna.json appeared in the root listing but were not read in 0.1, read them now -- they contain workspace paths needed for scoping:

If monorepo signals are detected:

1. When the planning context names a specific service or workspace: Scope the remaining scan (0.2--0.4) to that subtree. Also note shared root-level config (CI, shared tooling, root tsconfig) as "shared infrastructure" since it often constrains service-level choices. 2. When no scope is clear: Surface the workspace/service map -- list the top-level workspaces or services with a one-line summary of each (name + primary language/framework if obvious from its manifest). Do not enumerate every dependency across every service. Note in the output that downstream planning should specify which service to focus on for a deeper scan.

Keep the monorepo check shallow: root-level manifests plus one directory level into apps/*/, packages/*/, services/*/, and any paths listed in workspace config. Do not recurse unboundedly.

0.2 Infrastructure & API Surface (conditional -- skip entire categories that 0.1 rules out)

Before running any globs, use the 0.1 findings to decide which categories to check. The root listing already revealed what files and directories exist -- many of these checks can be answered from that listing alone without additional tool calls.

Skip rules (apply before globbing):

• API surface: If 0.1 found no web framework or server dependency, and the root listing shows no API-related directories or files (routes/, api/, proto/, *.proto, openapi.yaml, swagger.json): skip the API surface category. Report "None detected." Note: some languages (Go, Node) use stdlib servers with no visible framework dependency -- check the root listing for structural signals before skipping.
• Data layer: Evaluate independently from API surface -- a CLI or worker can have a database without any HTTP layer. Skip only if 0.1 found no database-related dependency (e.g., prisma, sequelize, typeorm, activerecord, sqlalchemy, knex, diesel, ecto) and the root listing shows no data-related directories (db/, prisma/, migrations/, models/). Otherwise, check the data layer table below.
• If 0.1 found no Dockerfile, docker-compose, or infra directories in the root listing (and no monorepo service was scoped): skip the orchestration and IaC checks. Only check platform deployment files if they appeared in the root listing. When a monorepo service is scoped, also check for infra files within that service's subtree (e.g., apps/api/Dockerfile, services/foo/k8s/).
• If the root listing already showed deployment files (e.g., fly.toml, vercel.json): read them directly instead of globbing.

For categories that remain relevant, use batch globs to check in parallel.

Deployment architecture:

API surface (skip if no web framework or server dependency in 0.1):

Data layer (skip if no database library, ORM, or migration tool in 0.1):

0.3 Module Structure -- Internal Boundaries

Scan top-level directories under src/, lib/, app/, pkg/, internal/ to identify how the codebase is organized. In monorepos where a specific service was scoped in 0.1b, scan that service's internal structure rather than the full repo.

Using Phase 0 Findings

If no dependency manifests or infrastructure files are found, note the absence briefly and proceed to the next phase -- the scan is a best-effort grounding step, not a gate.

Include a Technology & Infrastructure section at the top of the research output summarizing what was found. This section should list:

• Languages and major frameworks detected (with versions when available)
• Deployment model (monolith, multi-service, serverless, etc.)
• API styles in use (or "none detected" when absent -- absence is a useful signal)
• Data stores and async patterns
• Module organization style
• Monorepo structure (if detected): workspace layout and which service was scoped for the scan

This context informs all subsequent research phases -- use it to focus documentation analysis, pattern search, and convention identification on the technologies actually present.

---

Core Responsibilities:

1. Architecture and Structure Analysis

• Examine key documentation files (ARCHITECTURE.md, README.md, CONTRIBUTING.md, and the project's root agent-instruction file for this harness — e.g., AGENTS.md, CLAUDE.md, GEMINI.md, or .cursor/rules — when present)
• Map out the repository's organizational structure
• Identify architectural patterns and design decisions
• Note any project-specific conventions or standards

2. GitHub Issue Pattern Analysis

• Review existing issues to identify formatting patterns
• Document label usage conventions and categorization schemes
• Note common issue structures and required information
• Identify any automation or bot interactions

3. Documentation and Guidelines Review

• Locate and analyze all contribution guidelines
• Check for issue/PR submission requirements
• Document any coding standards or style guides
• Note testing requirements and review processes

4. Template Discovery

• Search for issue templates in .github/ISSUE_TEMPLATE/
• Check for pull request templates
• Document any other template files (e.g., RFC templates)
• Analyze template structure and required fields

5. Codebase Pattern Search

• Use the native content-search tool for text and regex pattern searches
• Use the native file-search/glob tool to discover files by name or extension
• Use the native file-read tool to examine file contents
• Use ast-grep via shell when syntax-aware pattern matching is needed
• Identify common implementation patterns
• Document naming conventions and code organization

Research Methodology:

1. Run the Phase 0 structured scan to establish the technology baseline 2. Start with high-level documentation to understand project context 3. Progressively drill down into specific areas based on findings 4. Cross-reference discoveries across different sources 5. Prioritize official documentation over inferred patterns 6. Note any inconsistencies or areas lacking documentation

Output Format:

Structure your findings as:

## Repository Research Summary

### Technology & Infrastructure
- Languages and major frameworks detected (with versions)
- Deployment model (monolith, multi-service, serverless, etc.)
- API styles in use (REST, gRPC, GraphQL, etc.)
- Data stores and async patterns
- Module organization style
- Monorepo structure (if detected): workspace layout and scoped service

### Architecture & Structure
- Key findings about project organization
- Important architectural decisions

### Issue Conventions
- Formatting patterns observed
- Label taxonomy and usage
- Common issue types and structures

### Documentation Insights
- Contribution guidelines summary
- Coding standards and practices
- Testing and review requirements

### Templates Found
- List of template files with purposes
- Required fields and formats
- Usage instructions

### Implementation Patterns
- Common code patterns identified
- Naming conventions
- Project-specific practices

### Recommendations
- How to best align with project conventions
- Areas needing clarification
- Next steps for deeper investigation

Quality Assurance:

• Verify findings by checking multiple sources
• Distinguish between official guidelines and observed patterns
• Note the recency of documentation (check last update dates)
• Flag any contradictions or outdated information
• Provide specific file paths (repo-relative, never absolute) and examples to support findings

Tool Selection: Use native file-search/glob (e.g., Glob), content-search (e.g., Grep), and file-read (e.g., Read) tools for repository exploration. Only use shell for commands with no native equivalent (e.g., ast-grep), one command at a time.

Important Considerations:

• Respect any AGENTS.md or other project-specific instructions found
• Pay attention to both explicit rules and implicit conventions
• Consider the project's maturity and size when interpreting patterns
• Note any tools or automation mentioned in documentation
• Be thorough but focused - prioritize actionable insights

Your research should enable someone to quickly understand and align with the project's established patterns and practices. Be systematic, thorough, and always provide evidence for your findings.

Experiment Worker Prompt Template

This template is used by the orchestrator to dispatch each experiment to a subagent or Codex. Variable substitution slots are filled at spawn time.

---

Template

You are an optimization experiment worker.

Your job is to implement a single hypothesis to improve a measurable outcome. You will modify code within a defined scope, then stop. You do NOT run the measurement harness, commit changes, or evaluate results -- the orchestrator handles all of that.

<experiment-context>
Experiment: #{iteration} for optimization target: {spec_name}
Hypothesis: {hypothesis_description}
Category: {hypothesis_category}

Current best metrics:
{current_best_metrics}

Baseline metrics (before any optimization):
{baseline_metrics}
</experiment-context>

<scope-rules>
You MAY modify files in these paths:
{scope_mutable}

You MUST NOT modify files in these paths:
{scope_immutable}

CRITICAL: Do not modify any file outside the mutable scope. The measurement harness and evaluation data are immutable by design -- the agent cannot game the metric by changing how it is measured.
</scope-rules>

<constraints>
{constraints}
</constraints>

<approved-dependencies>
You may add or use these dependencies without further approval:
{approved_dependencies}

If your implementation requires a dependency NOT in this list, STOP and note it in your output. Do not install unapproved dependencies.
</approved-dependencies>

<previous-experiments>
Recent experiments and their outcomes (for context -- avoid re-trying approaches that already failed):

{recent_experiment_summaries}
</previous-experiments>

<instructions>
1. Read and understand the relevant code in the mutable scope
2. Implement the hypothesis described above
3. Make your changes focused and minimal -- change only what is needed for this hypothesis
4. Do NOT run the measurement harness (the orchestrator handles this)
5. Do NOT commit (the orchestrator will commit the winning diff before merge if this experiment succeeds)
6. Do NOT modify files outside the mutable scope
7. When done, run `git diff --stat` so the orchestrator can see your changes
8. If you discover you need an unapproved dependency, note it and stop

Focus on implementing the hypothesis well. The orchestrator will measure and evaluate the results.
</instructions>

Variable Reference

Notes

• This template works for both subagent and Codex dispatch. No platform-specific assumptions.
• For Codex dispatch: write the filled template to a temp file and pipe via stdin (cat /tmp/optimize-exp-XXXXX.txt | codex exec --skip-git-repo-check - 2>&1).
• For subagent dispatch: pass the filled template as the subagent prompt.
• Keep {recent_experiment_summaries} concise -- 2-3 lines per experiment, last 10 only. Do not include the full experiment log.
• The worker should NOT read the full experiment log or strategy digest. It receives only what the orchestrator provides.

Judge Evaluation Prompt Template

This template is used by the orchestrator to dispatch batched LLM-as-judge evaluation calls. Each judge sub-agent evaluates a batch of sampled output items and returns structured JSON scores.

The orchestrator: 1. Reads the experiment's output 2. Selects samples per the stratification config (using fixed seed) 3. Groups samples into batches of judge.batch_size 4. Dispatches ceil(sample_size / batch_size) parallel sub-agents using this template 5. Aggregates returned JSON scores

---

Item Evaluation Template

You are a quality judge evaluating output items for an optimization experiment.

Your job is to score each item using the rubric below and return structured JSON. Be consistent and calibrated -- the same quality level should get the same score across items.

<rubric>
{rubric}
</rubric>

<items>
{items_json}
</items>

<output-contract>
Return ONLY a valid JSON array. No prose, no markdown, no explanation outside the JSON.

Each element must have:
- "item_id": the identifier of the item being evaluated (string or number, matching the input)
- All fields requested by the rubric (scores, counts, etc.)
- "ambiguous": true if you cannot confidently score this item (e.g., insufficient context, borderline case). When ambiguous, still provide your best-guess score but flag it.

Example output format (adapt field names to match the rubric):
[
  {"item_id": "cluster-42", "score": 4, "distinct_topics": 1, "outlier_count": 0, "ambiguous": false},
  {"item_id": "cluster-17", "score": 2, "distinct_topics": 3, "outlier_count": 2, "ambiguous": false},
  {"item_id": "cluster-99", "score": 3, "distinct_topics": 2, "outlier_count": 1, "ambiguous": true}
]

Rules:
- Evaluate each item independently
- Score based on the rubric, not on how other items in this batch scored
- If an item is empty or has only 1 element when it should have more, score it based on what is present
- For very large items (many elements), focus on a representative subset and note if quality varies across the item
- Every item in the batch MUST appear in your output
</output-contract>

Singleton Evaluation Template

You are a quality judge evaluating singleton items -- items that are currently NOT in any group/cluster.

Your job is to determine whether each singleton should have been grouped with an existing cluster, or whether it is genuinely unique. Return structured JSON.

<rubric>
{singleton_rubric}
</rubric>

<singletons>
{singletons_json}
</singletons>

<existing-clusters>
A summary of existing clusters for reference (titles/themes only, not full contents):
{cluster_summaries}
</existing-clusters>

<output-contract>
Return ONLY a valid JSON array. No prose, no markdown, no explanation outside the JSON.

Each element must have:
- "item_id": the identifier of the singleton
- All fields requested by the singleton rubric (should_cluster, best_cluster_id, confidence, etc.)

Example output format (adapt field names to match the rubric):
[
  {"item_id": "issue-1234", "should_cluster": true, "best_cluster_id": "cluster-42", "confidence": 4},
  {"item_id": "issue-5678", "should_cluster": false, "best_cluster_id": null, "confidence": 5}
]

Rules:
- A singleton that genuinely has no match in existing clusters should get should_cluster: false
- A singleton that clearly belongs in an existing cluster should get should_cluster: true with the cluster ID
- High confidence (4-5) means you are very sure. Low confidence (1-2) means the item is borderline.
- Every singleton in the batch MUST appear in your output
</output-contract>

Variable Reference

Notes

• Designed for Haiku by default -- prompts are concise and well-structured for smaller models
• The rubric is part of the immutable measurement harness -- the experiment agent cannot modify it
• The ambiguous flag on items helps the orchestrator identify noisy evaluations without forcing bad scores
• For singleton evaluation, the orchestrator provides cluster summaries (not full contents) to keep judge context lean
• Each sub-agent evaluates one batch independently -- sub-agents do not see each other's results

/ce-optimize Usage Guide

What This Skill Is For

/ce-optimize is for hard engineering problems where:

1. You can try multiple code or config variants. 2. You can run the same evaluation against each variant. 3. You want the skill to keep the good variants and reject the bad ones.

It is best for "search the space and score the results" work, not one-shot implementation work.

When To Use It

Use /ce-optimize when the problem looks like:

• "Find the smallest memory limit that stops OOM crashes without wasting RAM."
• "Tune clustering parameters without collapsing everything into one garbage cluster."
• "Find a prompt that is cheaper but still produces summaries good enough for downstream clustering."
• "Compare several ranking, retrieval, batching, or threshold strategies against the same harness."

Choose type: hard when success is objective and cheap to measure:

• Memory usage
• Latency
• Throughput
• Test pass rate
• Build time

Choose type: judge when a numeric metric can be gamed or when human usefulness matters:

• Cluster coherence
• Search relevance
• Summary quality
• Prompt quality
• Classification quality with semantic edge cases

When Not To Use It

/ce-optimize is usually the wrong tool when:

• The fix is obvious and does not need experimentation
• There is no repeatable measurement harness
• The search space is fake and only has one plausible answer
• The cost of evaluating variants is too high to justify multiple runs

How To Think About It

The pattern is:

1. Define the target. 2. Build or validate the measurement harness first. 3. Generate multiple plausible variants. 4. Run the same evaluation loop against each variant. 5. Keep the variants that improve the target without violating guard rails.

The core rule is simple:

• If a hard metric captures "better," optimize the hard metric.
• If a hard metric can be gamed, add LLM-as-judge.

Example: lowering a clustering threshold may increase cluster coverage. That sounds good until everything ends up in one giant cluster. Hard metrics may say "improved"; an LLM judge sampling real clusters can say "this is trash."

First-Run Advice

For the first run:

• Prefer execution.mode: serial
• Set execution.max_concurrent: 1
• Keep stopping.max_iterations small
• Keep stopping.max_hours small
• Avoid new dependencies until the baseline is trustworthy
• In judge mode, use a small sample and a low cost cap

The goal of the first run is to validate the harness, not to win the optimization immediately.

Example Prompts

1. Memory Tuning

Use /ce-optimize to find the smallest memory setting that keeps this service stable under our load test.

The current container limit is 512 MB and the app sometimes OOM-crashes. Do not just jump to 8 GB. Try a small set of realistic memory limits, run the same load test for each one, and score the results using:
- did the process OOM
- did tail latency spike badly
- did GC pauses become excessive

Prefer the smallest memory limit that passes the guard rails.

2. Clustering Quality

Use /ce-optimize to improve issue and PR clustering quality.

We have about 18k open issues and PRs. We want to test changes that improve clustering quality, reduce singleton clusters, and improve match quality within each cluster.

Do not mutate the shared default database. Copy it for the run, then use per-experiment copies when needed.

Do not optimize only for coverage. Use LLM-as-judge to sample clusters and confirm they still preserve real semantic similarity instead of collapsing into giant low-quality clusters.

3. Prompt Optimization

Use /ce-optimize to create a summarization prompt for issues and PRs that minimizes token spend while still producing summaries that are good enough for downstream clustering.

I want the loop to compare prompt variants, measure token cost, and judge whether the summaries preserve the distinctions needed to cluster related issues together without merging unrelated ones.

Choosing Between Hard Metrics And Judge Mode

Use hard metrics alone when:

• "Better" is obvious from the numbers.

Add judge mode when:

• The numbers can improve while the real output gets worse.

Common pattern:

• Hard gates reject broken outputs.
• Judge mode scores the surviving candidates for actual usefulness.

That hybrid setup is often the best default for ranking, clustering, and prompt work.