Autoresearch
Run an autonomous git-backed experiment loop that changes code, measures a metric you define, and keeps or discards each iteration until you stop it.
Overview
autoresearch is an agent skill most often used in Operate (also Build, Ship) that runs an autonomous git-backed loop of code changes, experiments, and keep-or-discard decisions against a metric you define in Setup.
Install
npx skills add https://github.com/github/awesome-copilot --skill autoresearchWhat is this skill?
- Interactive Setup phase before any autonomous loop starts
- Requires a measurable metric and explicit scope constraints—no metric means do not use
- Autonomous cycle: modify code, run experiments, measure, keep or discard via git
- Inspired by Karpathy autoresearch, generalized beyond ML to any programming outcome
- Explicitly excludes one-shot fixes, simple bug patches, and code review without measurement
- Generalized from Karpathy autoresearch to any programming task with a measurable outcome
Adoption & trust: 2.4k installs on skills.sh; 34.6k GitHub stars; 1/3 security scanners passed (skills.sh audits).
What problem does it solve?
You know the code could be faster or better but lack time to manually try dozens of variants and track which changes actually improved a measurable outcome.
Who is it for?
Solo builders with a git repo, terminal access, and a quantifiable target such as latency, throughput, test pass rate, or benchmark score.
Skip if: One-shot tasks, simple bug fixes, code review without metrics, or work where success cannot be measured objectively.
When should I use this skill?
USE FOR autonomous improvement, iterative optimization, experiment loop, performance tuning, or autonomous coding loop; DO NOT USE FOR one-shot tasks, simple bug fixes, or code review without a measurable metric.
What do I get? / Deliverables
You finish Setup with a clear metric and constraints, then the agent iterates autonomously—measuring each attempt and retaining only improvements in git until you interrupt the loop.
- Kept git commits that improved the target metric
- Discarded experiment attempts with recorded measurements
- Documented goal, metric, and constraints from Setup
Recommended Skills
Journey fit
Spans multiple journey phases - primary shelf plus alternate fits below.
Continuous measure-and-keep loops belong on Operate–iterate, though the same pattern tunes performance during Build and Ship when a metric is already defined. iterate is the shelf for hill-climbing improvement cycles that autonomously modify the repo until interrupted.
Where it fits
Sweep caching strategies until API p95 latency drops under your stated budget.
Run repeated benchmark variants before release and keep only commits that improve the score.
Leave an overnight loop optimizing a hot path while you focus on support and roadmap work.
Clarify success metrics and forbidden files during Setup before any autonomous edits begin.
How it compares
Use this structured experiment loop instead of asking the agent for a single speculative patch when optimization needs trial-and-error hill climbing.
Common Questions / FAQ
Who is autoresearch for?
It is for developers who want autonomous iterative optimization on real codebases and can define how to measure success before the loop runs.
When should I use autoresearch?
Use it during Build when tuning implementation quality, during Ship when chasing performance targets, and during Operate when iteratively improving production-adjacent code with benchmarks.
Is autoresearch safe to install?
It requires git write history and terminal command execution—review the Security Audits panel on this Prism page and use branch isolation before letting the loop run unsupervised.
SKILL.md
READMESKILL.md - Autoresearch
# Autoresearch: Autonomous Iterative Experimentation An autonomous experimentation loop for any programming task. You define the goal and how to measure it; the agent iterates autonomously -- modifying code, running experiments, measuring results, and keeping or discarding changes -- until interrupted. This skill is inspired by [Karpathy's autoresearch](https://github.com/karpathy/autoresearch), generalized from ML training to **any programming task with a measurable outcome**. --- ## Agent Behavior Rules 1. **DO** guide the user through the Setup phase interactively before starting the loop. 2. **DO** establish a baseline measurement before making any changes. 3. **DO** commit every experiment attempt before running it (so it can be reverted cleanly). 4. **DO** keep a results log (TSV) tracking every experiment. 5. **DO** revert changes that do not improve the metric (git reset to last known good). 6. **DO** run autonomously once the loop starts -- never pause to ask "should I continue?". 7. **DO NOT** modify files the user marked as out-of-scope. 8. **DO NOT** skip the measurement step -- every experiment must be measured. 9. **DO NOT** keep changes that regress the metric unless the user explicitly allowed trade-offs. 10. **DO NOT** install new dependencies or make environment changes unless the user approved it. --- ## Phase 1: Setup (Interactive) Before any experimentation begins, work with the user to establish these parameters. Ask the user directly for each item. Do not assume or skip any. ### 1.1 Define the Goal Ask the user: > **What are you trying to improve or optimize?** > > Examples: execution time, memory usage, binary size, test pass rate, code coverage, > API response latency, throughput, error rate, benchmark score, build time, bundle size, > lines of code, cyclomatic complexity, etc. Record the user's answer as the **goal**. ### 1.2 Define the Metric Ask the user: > **How do we measure success? What exact command produces the metric?** > > I need: > 1. **The command** to run (e.g., `dotnet test`, `npm run benchmark`, `time ./build.sh`, `pytest --tb=short`) > 2. **How to extract the metric** from the output (e.g., a regex pattern, a specific line, a JSON field) > 3. **Direction**: Is lower better or higher better? > > Example: "Run `dotnet test --logger trx`, count passing tests. Higher is better." > Example: "Run `hyperfine './my-program'`, extract mean time. Lower is better." Record: - `METRIC_COMMAND`: the command to run - `METRIC_EXTRACTION`: how to extract the numeric metric from output - `METRIC_DIRECTION`: `lower_is_better` or `higher_is_better` ### 1.3 Define the Scope Ask the user: > **Which files or directories am I allowed to modify?** > > And which files are OFF LIMITS (read-only)? Record: - `IN_SCOPE_FILES`: files/dirs the agent may edit - `OUT_OF_SCOPE_FILES`: files/dirs that must not be modified ### 1.4 Define Constraints Ask the user: > **Are there any constraints I should respect?** > > Examples: > - Time budget per experiment (e.g., "each run should take < 2 minutes") > - No new dependencies > - Must keep all ex