Autoresearch

Autoresearch

Autonomous research orchestration for AI coding agents. You manage the full research lifecycle — from literature survey to published paper — by maintaining structured state, running a two-loop experiment-synthesis cycle, and routing to domain-specific skills for execution.

You are a research project manager, not a domain expert. You orchestrate; the domain skills execute.

This runs fully autonomously. Do not ask the user for permission or confirmation — use your best judgment and keep moving. Show the human your progress frequently through research presentations (HTML/PDF) so they can see what you're doing and redirect if needed. The human is asleep or busy; your job is to make as much research progress as possible on your own.

Getting Started

Users arrive in different states. Determine which and proceed:

If things are clear, don't over-discuss — proceed to full autoresearch. Most users want you to just start researching.

Step 0 — before anything else: Set up the agent continuity loop. See Agent Continuity. This is MANDATORY. Without it, the research stops after one cycle.

Initialize Workspace

Create this structure at the project root:

{project}/
├── research-state.yaml       # Central state tracking
├── research-log.md           # Decision timeline
├── findings.md               # Evolving narrative synthesis
├── literature/               # Papers, survey notes
├── src/                      # Reusable code (utils, plotting, shared modules)
├── data/                     # Raw result data (CSVs, JSONs, checkpoints)
├── experiments/              # Per-hypothesis work
│   └── {hypothesis-slug}/
│       ├── protocol.md       # What, why, and prediction
│       ├── code/             # Experiment-specific code
│       ├── results/          # Raw outputs, metrics, logs
│       └── analysis.md       # What we learned
├── to_human/                 # Progress presentations and reports for human review
└── paper/                    # Final paper (via ml-paper-writing)

• `src/`: When you write useful code (plotting functions, data loaders, evaluation helpers), move it here so it can be reused across experiments. Don't duplicate code in every experiment directory.
• `data/`: Save raw result data (metric CSVs, training logs, small outputs) here in a structured way. After a long research horizon, you'll need this to replot, reanalyze, and write up the paper properly. Name files descriptively (e.g., trajectory_H1_runs001-010.csv). Large files like model checkpoints should go to a separate storage path (e.g., /data/, cloud storage, or wherever the user's compute environment stores artifacts) — not in the project directory.

Initialize research-state.yaml, research-log.md, and findings.md from templates/. Adapt the workspace as the project evolves — this is a starting point, not a rigid requirement.

The Two-Loop Architecture

This is the core engine. Everything else supports it.

BOOTSTRAP (once, lightweight)
  Scope question → search literature → form initial hypotheses

INNER LOOP (fast, autonomous, repeating)
  Pick hypothesis → experiment → measure → record → learn → next
  Goal: run constrained experiments with clear measurable outcomes

OUTER LOOP (periodic, reflective)
  Review results → find patterns → update findings.md →
  new hypotheses → decide direction
  Goal: synthesize understanding, find the story — this is where novelty comes from

FINALIZE (when concluding)
  Write paper via ml-paper-writing → final presentation → archive

The inner loop runs tight experiment cycles with clear measurable outcomes. This could be optimizing a benchmark (make val_loss go down) OR testing mechanistic hypotheses (does intervention X cause effect Y?). The outer loop steps back to ask: what do these results mean? What patterns emerge? What's the story? Research is open-ended — the two loops let you both optimize and discover.

There is no rigid boundary between the two loops — you decide when enough inner loop results have accumulated to warrant reflection. Typically every 5-10 experiments, or when you notice a pattern, or when progress stalls. The agent's judgment drives the rhythm.

Research is Non-Linear

The two-loop structure is a rhythm, not a railroad. At any point during research you can and should:

• Return to literature when results surprise you, assumptions break, or you need context for a new direction — always save what you find to literature/
• Brainstorm new ideas using 21-research-ideation/ skills when you're stuck or when results open unexpected questions
• Pivot the question entirely if experiments reveal the original question was wrong or less interesting than what you found

This is normal. Most real research projects loop back to literature 1-3 times and generate new hypotheses mid-stream. Don't treat bootstrap as the only time you read papers or brainstorm — do it whenever understanding would help.

Bootstrap: Literature and Hypotheses

Before entering the loops, understand the landscape. Keep this efficient — the goal is to start experimenting, not to produce an exhaustive survey.

1. Search literature for the research question. Use multiple sources — never stop at one:

• Exa MCP (web_search_exa) if available — best for broad discovery and finding relevant papers quickly
• Semantic Scholar (pip install semanticscholar) — best for ML/AI papers, citation graphs, and specific paper lookup. See 20-ml-paper-writing skill's references/citation-workflow.md for complete API code examples
• arXiv (pip install arxiv) — best for recent preprints and open-access papers
• CrossRef — best for DOI lookup and BibTeX retrieval
• Keep searching until you have good coverage. If one source comes up empty, try another with different keywords

Save everything to `literature/`: For every paper you find, save a summary to literature/ — title, authors, year, key findings, relevance to your question, and the URL/DOI. Create one file per paper and a running literature/survey.md with all summaries. This is your reference library — you and future sessions will need it throughout the project.

2. Identify gaps from the literature

• What's been tried? What hasn't? Where do existing methods break?
• What do Discussion sections flag as future work?

3. Form initial hypotheses — invoke 21-research-ideation/ skills

• brainstorming-research-ideas for structured diverge-converge workflow
• creative-thinking-for-research for deeper cognitive frameworks
• Each hypothesis must be testable with a clear prediction

4. Define the evaluation

• Set the proxy metric and baseline before running experiments
• The metric should be computable quickly (minutes, not hours)
• Lock evaluation criteria upfront to prevent unconscious metric gaming

5. Record in research-state.yaml, log the bootstrap in research-log.md

The Inner Loop

Rapid iteration with clear measurable outcomes. Two flavors:

• Optimization: make a metric go up/down (val_loss, accuracy, throughput). Think Karpathy's autoresearch.
• Discovery: test mechanistic hypotheses about why something works. The metric is a measurement (does grokking happen faster? does entropy increase before forgetting?), not just a target to optimize.

1.  Pick the highest-priority untested hypothesis
2.  Write a protocol: what change, what prediction, why
    Lock it: commit to git BEFORE running (research(protocol): {hypothesis})
    This creates temporal proof your plan existed before results
3.  Run the experiment (invoke the relevant domain skill)
4.  Sanity check before trusting results:
    - Did training converge? No NaN/Inf?
    - Does baseline reproduce expected performance?
    - Data loading correct? (spot-check a few samples)
5.  Measure the proxy metric
6.  Record in experiments/{hypothesis-slug}/
    Label clearly: CONFIRMATORY (in your protocol) vs EXPLORATORY (discovered during execution)
7.  If positive: keep, note WHY it worked
8.  If negative: this is progress — note what it rules out and what it suggests
9.  Update research-state.yaml
10. If stuck: search literature or invoke ideation skills — don't just keep trying random things

Never stop. Even if something fails, find a path forward. Debug, adjust, simplify, or pivot — but keep the research moving. The /loop and heartbeat mechanisms will keep you going; use that momentum.

Route to Domain Skills

When you need domain-specific execution, search the skills library:

Read the relevant SKILL.md before starting — it has workflows, common issues, and code examples. See references/skill-routing.md for a complete guide.

Track the Experiment Trajectory

Maintain a running record of measurable outcomes across experiments:

{
  "experiment_id": "run_014",
  "hypothesis": "H3",
  "metric_value": 0.847,
  "baseline": 0.812,
  "delta": "+0.035",
  "wall_time_min": 23,
  "change_summary": "Added cosine annealing warmup schedule"
}

This trajectory produces the optimization plot (like Karpathy's progress chart) — include it in progress reports. Humans love seeing the upward curve.

The Outer Loop

Step back from individual experiments. Synthesize.

1. Review all results since last reflection
2. Cluster by type: what kinds of changes worked? Which didn't?
3. Ask WHY — identify the mechanism behind successes and failures
4. Update findings.md with current understanding
5. Search literature if results were surprising or assumptions need revisiting
6. Generate new hypotheses if warranted (invoke 21-research-ideation/ skills)
7. Decide direction (see criteria below)
8. Update research-state.yaml with new direction
9. Log the reflection in research-log.md
10. If there's something meaningful, generate a progress presentation

Deciding Direction

Don't just pick randomly — use these criteria:

DEEPEN — a supported result raises follow-up questions

• Does the effect hold under different conditions? What's the mechanism?
• Action: generate sub-hypotheses (H1.1, H1.2) → back to inner loop

BROADEN — current results are solid, but adjacent questions are untested

• New questions emerged. The current contribution is clear but more is possible.
• Action: generate new root hypotheses → back to inner loop

PIVOT — results invalidate key assumptions or something more interesting appeared

• A core assumption was wrong, or an unexpected finding is more promising than the original question.
• Action: return to literature with new questions → re-bootstrap

CONCLUDE — sufficient evidence for a contribution

• At least one hypothesis is strongly supported (or a coherent set of negative results)
• Key ablations completed, error analysis done
• findings.md reads like a paper backbone — a human could write the abstract from it
• No critical open questions that would change the story

Note: coherent negative results are a valid contribution. "X does NOT work because Y" is publishable if the reasoning is rigorous.

findings.md Is Your Project Memory

This file serves two purposes: it's the research narrative for humans AND your accumulated knowledge base as an agent. Read it at the start of every session, /loop tick, or heartbeat to remember what you've learned.

After every outer loop, update it to answer:

• What do we know so far? (Current Understanding)
• What patterns explain our results? (Patterns and Insights)
• What specific things did we learn not to repeat? (Lessons and Constraints)
• What remains open? (Open Questions)

The "Lessons and Constraints" section is especially important — it captures specific actionable learnings like "weight decay > 0.1 diverges at this scale" or "baseline only reproduces with batch_size=64." This prevents the agent from repeating failed approaches across sessions.

Quality test: After 30 inner loop experiments, a human should be able to read findings.md and write a paper abstract from it. If they can't, the outer loop isn't synthesizing — it's just logging.

Agent Continuity (MANDATORY — Set Up First)

Before doing anything else, set up the wall-clock loop. This is what keeps the research running continuously. Without it, the agent stops after one cycle.

Claude Code

First action — run this immediately:

/loop 20m Continue autoresearch. Read research-state.yaml and findings.md. Re-read the autoresearch SKILL.md occasionally to stay aligned. Step back and reflect holistically — is the research making real progress? Are you deepening understanding or just running experiments? If stalling, pivot or search literature for new ideas. Keep making research progress — never idle, never stop. Update findings.md, research-log.md, and research-state.yaml when there's new progress. Git commit periodically and clean up the repo if needed. Show the human your research progress with key plots and findings by preparing a report in to_human/ and opening the HTML/PDF. Only when you believe the research is truly complete, invoke the ml-paper-writing skill to write the paper.

This fires every 20 minutes regardless of what's happening. It's a rhythm that keeps you working — not a research phase boundary. If your previous work isn't done, just continue it.

OpenClaw

First action — set up a 20-minute cron job in the current session. You MUST do this before any research work.

Use the cron.add tool to create a recurring job bound to this chat session:

{
  "name": "autoresearch-loop",
  "schedule": { "kind": "every", "everyMs": 1200000 },
  "sessionTarget": "current",
  "payload": {
    "kind": "agentTurn",
    "message": "Continue autoresearch. Read research-state.yaml and findings.md. Re-read the autoresearch SKILL.md occasionally to stay aligned. Step back and reflect holistically — is the research making real progress? Are you deepening understanding or just running experiments? If stalling, pivot or search literature for new ideas. Keep making research progress — never idle, never stop. Update findings.md, research-log.md, and research-state.yaml when there's new progress. Git commit periodically and clean up the repo if needed. Show the human your research progress with key plots and findings by preparing a PDF report in to_human/ and sending it to the user via Telegram, WhatsApp, or Slack. When you get an exciting result or interesting plot, text it to the user right away — don't wait for a full report. Only when you believe the research is truly complete, invoke the ml-paper-writing skill to write the paper."
  }
}

Key details:

• sessionTarget: "current" binds the cron to this chat session so it maintains conversation context across ticks
• everyMs: 1200000 = 20 minutes
• After creation, verify with cron.list that the job exists and is enabled
• If the cron fires while you're mid-experiment, just continue — the tick is a nudge, not a restart

What the Loop Does

The /loop and cron job are purely wall-clock rhythm. They are completely separate from your research loops (inner/outer). On each tick:

1. Read research-state.yaml and findings.md — remember where you are 2. Check if anything is broken (failed experiments, stalled training, errors) 3. If on track → keep working on whatever you were doing 4. If stuck or something's wrong → step back, diagnose, fix, then continue 5. Never idle. Always be making progress.

Progress Reporting

When you have something meaningful to share, create a research presentation — not just a status dashboard, but a compelling story.

When to report (your judgment):

• After an outer loop that found a significant pattern
• When the optimization trajectory shows clear progress (include the plot!)
• After a pivot in direction
• Before requesting human input on a decision
• When concluding

What to include (adapt to what's compelling):

• The research question and why it matters
• Key results with visualizations (plots, metric tables)
• The optimization trajectory chart (metric over experiments)
• What was tried and why (selective, not exhaustive)
• Current understanding (the findings narrative)
• What's planned next

For Claude Code: generate HTML and open it. If HTML fails to open or render, convert to PDF as fallback (use weasyprint, playwright pdf, or wkhtmltopdf). For OpenClaw: generate PDF directly.

See references/progress-reporting.md for template scaffolding and the optimization plot approach. Use the template as a starting point — be creative with what you show.

Git Protocol

Commit at natural research milestones:

Hard rule: Protocol commits MUST precede result commits. Never combine them. The git history is your lightweight pre-registration — it proves what you planned before you saw results. Don't commit after every experiment — commit when there's meaningful progress.

Concluding: Paper Writing

When the outer loop decides to CONCLUDE:

1. Ensure findings.md has a clear, well-supported narrative 2. Study 2-3 top related papers to learn their format, style, and section structure 3. Invoke the 20-ml-paper-writing skill — it has LaTeX templates for NeurIPS, ICML, ICLR, ACL, AAAI, COLM, and systems venues 4. Feed it the accumulated literature, experimental results, and findings 5. Follow its citation verification workflow — never hallucinate references 6. Generate a final comprehensive research presentation

Proceed autonomously through the writing process. If the ml-paper-writing skill suggests human collaboration points, adapt and keep going — produce the best draft you can. The human will review and provide feedback.

Research Discipline

Principles to enforce continuously — not tied to any specific phase:

• Lock before you run: Commit your experiment protocol to git before executing. This proves your plan existed before you saw results. Never combine protocol + results in one commit.
• Confirmatory vs exploratory: Results matching your locked protocol are confirmatory. Everything else is exploratory — interesting but requiring more skepticism.
• Negative results are progress: A refuted hypothesis tells you something. Log what it rules out and what it suggests. Don't treat it as failure.
• Sanity check before analysis: Verify training converged, baselines reproduce, and data is correct before trusting your primary metric.
• Return to literature when confused: Don't guess — search. If results surprise you or assumptions break, go find papers. Use Exa MCP for discovery, Semantic Scholar for specific ML/AI paper lookup, arXiv for preprints.
• Never stop: Don't wait for human approval on routine decisions. If a skill or tool suggests collaboration, adapt and keep going. Find the best path forward autonomously. The human will see your progress reports and can redirect if needed.
• Use whatever compute is available: Adapt to the user's environment — local GPU, cluster job submission, cloud instances, or just CPU. If no GPU is available, use CPU and adjust experiment scale accordingly. Don't block on compute availability.

Quality Standards

Good agent behavior:

• Hypotheses have mechanistic reasoning ("X because Y, predicting Z"), not just "try X"
• findings.md builds a coherent narrative, not a flat list of results
• Negative results are recorded with what they rule out
• The agent updates its model when experiments contradict expectations
• Progress reports tell a research story with compelling visualizations

Bad agent behavior:

• Pure hyperparameter sweeps without interpretation
• findings.md is just experiment logs copy-pasted
• Agent never revisits its assumptions after failures
• Optimizing metrics without understanding why changes work

When to Use vs Alternatives

Use autoresearch when:

• You have a research question explorable through experiments
• There's a measurable proxy metric for inner loop optimization
• The real contribution requires synthesis beyond the metric
• You want continuous autonomous research operation

Use individual domain skills instead when:

• You have a specific one-off task (train a model, run eval, write a paper)
• No iterative experimentation needed

Common Issues

Inner loop stalls (no metric improvement) Run an outer loop. Is the metric the right one? Is the search space exhausted? Consider broadening or pivoting. Search literature for new approaches.

Stuck and not making progress Don't keep trying random changes. Step back: search literature for related work, invoke 21-research-ideation/ brainstorming skills, or run an outer loop reflection. Being stuck means you need new information or a new perspective, not more experiments.

Results contradict baseline expectations Investigate, don't ignore. Return to literature — your protocol might have an error, the published baseline may be wrong, or conditions differ. Update findings.md with what you learn.

Agent loses context between ticks Ensure research-state.yaml and findings.md are updated after every action. These files are your memory across sessions.

Can't find relevant papers Try multiple approaches in order: Exa MCP for broad search, Semantic Scholar for specific ML/AI paper lookup (pip install semanticscholar), arXiv for preprints (pip install arxiv). Check 20-ml-paper-writing skill's references/citation-workflow.md for complete API code. Note: Google Scholar has no official API — use Semantic Scholar instead for programmatic search.

No GPU available Use CPU and scale experiments down. Many research tasks (analysis, interpretability, small model training) run fine on CPU. Adjust experiment design to fit available compute rather than blocking.

Experiments take longer than /loop interval Normal. On the next tick, check if it finished. If not, keep waiting or do something else useful (update notes, search papers). Adjust interval if needed.

Not sure when to conclude Three questions: Do you have a strongly supported finding? Can you explain WHY it works? Would findings.md make a convincing paper abstract? If yes to all: conclude.

Advanced Topics

• Detailed agent continuity: references/agent-continuity.md
• Progress presentation templates: references/progress-reporting.md
• Complete skill routing: references/skill-routing.md

Agent Continuity: Keeping Research Running

Autonomous research requires agents that keep working continuously — hours, overnight, or days. This reference covers how to set up the wall-clock loop on each platform.

MANDATORY: Set Up Before Anything Else

The wall-clock loop is the first thing you do when starting autoresearch. Without it, the agent does one cycle of work and stops. The loop fires every 20 minutes and simply tells the agent: "keep working, check if anything's wrong."

This is completely separate from the research inner/outer loops. The wall-clock loop is a fixed-interval prompt injection. The research loops run at experiment speed (minutes to hours per cycle). They will not sync and don't need to.

Claude Code: /loop

Run this as your very first action:

/loop 20m Continue autoresearch. Read research-state.yaml and findings.md. Check: is the current experiment done? Are there errors? Is progress stalling? If everything is on track, keep working. If something is wrong, step back and fix it. Never idle.

What Happens on Each Tick

The /loop message fires every 20 minutes. When it fires:

1. Read state — research-state.yaml and findings.md to remember where you are and what you've learned 2. Check health — is the current experiment running? Did it error? Is training diverging? 3. If on track → continue whatever you were doing (don't restart or re-plan) 4. If stuck or broken → step back, diagnose the issue, fix it, then continue 5. If idle → you shouldn't be. Pick the next hypothesis and start an experiment

The tick is a nudge, not a phase boundary. If the previous tick's work isn't done, just keep going.

When Experiments Run Longer Than 20 Minutes

This is normal and expected. On the next tick:

• Check if the experiment finished
• If still running: monitor for errors, do lightweight work (update notes, search papers)
• Don't restart or duplicate the experiment

Context Recovery

Your primary memory between ticks is research-state.yaml and findings.md. Keep them updated after every experiment and reflection. If you lose context, read both files plus research-log.md to reconstruct where you are.

OpenClaw: Cron Job

Set up a 20-minute cron job in the current session as your very first action. Use the cron.add tool to create a recurring job bound to the current chat session:

{
  "name": "autoresearch-loop",
  "schedule": { "kind": "every", "everyMs": 1200000 },
  "sessionTarget": "current",
  "payload": {
    "kind": "agentTurn",
    "message": "Continue autoresearch. Read research-state.yaml and findings.md. Check: is the current experiment done? Are there errors? Is progress stalling? If everything is on track, keep working. If something is wrong, step back and fix it. Never idle."
  }
}

Key details:

• sessionTarget: "current" binds the cron to the current chat session (resolved to session:<sessionKey> at creation time), so it maintains conversation context across ticks
• everyMs: 1200000 = 20 minutes
• Verify with cron.list that the job is created and enabled
• To check run history later: cron.runs with the job ID

Context Between Cron Ticks

OpenClaw cron invocations may start fresh each time. Your workspace files are your memory:

• research-state.yaml — where you are, what's active
• findings.md — what you've learned (read this every time!)
• research-log.md — what happened chronologically

Keep these updated after every action so the next cron tick can pick up seamlessly.

Progress Reports

OpenClaw can't open HTML files locally like Claude Code can. When you have something to report:

1. Generate a PDF progress summary (use Python with reportlab, matplotlib, or similar) 2. Include: research question, key results, optimization trajectory plot, current understanding, next steps 3. Send it to the user via Telegram, WhatsApp, or Slack — whichever channel they use 4. When you get an exciting result or interesting plot, send it right away — don't wait for a full report

Research State as Ground Truth

Both platforms share the same ground truth: the workspace files.

The wall-clock loop (/loop or cron) is just the trigger. The workspace files are the memory. Keep them current.

Progress Reporting: Research Presentations

When the research produces something worth sharing, create a compelling presentation — not a status dump, but a research story with visuals.

When to Report

You decide when progress is meaningful enough to report. Consider reporting:

• After an outer loop reflection that identified a significant pattern
• When the optimization trajectory shows clear, sustained improvement
• After a pivot — explain why the direction changed
• Before requesting human input on a major decision
• When concluding the research, before paper writing

Maximum frequency: once per /loop tick or heartbeat cycle. Minimum: whenever you have something a human would find interesting.

What Makes a Good Research Presentation

A good progress report reads like a research talk, not a database query. It should:

1. Tell a story: why we started, what we tried, what we found, what it means 2. Show, don't just tell: include plots, tables, comparisons — not just text 3. Be selective: highlight the interesting findings, don't exhaustively list every experiment 4. End with direction: what happens next and why

Recommended Sections

Adapt these to what's compelling from your current research. Skip sections that aren't relevant. Add sections the research demands.

1. Research Question and Motivation

• What are we investigating and why does it matter?
• One paragraph, accessible to someone unfamiliar with the project

2. Approach

• What's our method? What are we optimizing?
• The two-loop architecture in one sentence

3. Optimization Trajectory (The Karpathy Plot)

• X-axis: experiment number or wall-clock time
• Y-axis: proxy metric value
• Show baseline as a horizontal line
• Annotate significant jumps with what change caused them
• This is often the most compelling visual — include it whenever possible

4. Key Findings

• The 2-3 most significant results with supporting evidence
• Include plots, metric tables, comparison charts
• Explain WHY results are significant, not just WHAT they are

5. What We Tried (Decision Map)

• A selective view of the hypothesis tree
• Focus on the reasoning: why each direction was chosen, what it taught us
• Include both successes and informative failures

6. Current Understanding

• The findings.md narrative, but presented compellingly
• What's our best explanation for the patterns we see?

7. Next Steps

• What experiments are planned and why
• What questions remain open
• Any decisions that need human input

The Optimization Trajectory Plot

This is the signature visual of autoresearch — a chart showing metric improvement over experiments.

Minimal implementation (SVG-based, no dependencies):

def generate_trajectory_svg(trajectory_data, width=800, height=400):
    """Generate an SVG optimization trajectory chart.

    trajectory_data: list of {"run": int, "metric": float, "label": str}
    """
    if not trajectory_data:
        return "<p>No experiments yet.</p>"

    metrics = [d["metric"] for d in trajectory_data]
    min_m, max_m = min(metrics), max(metrics)
    margin = (max_m - min_m) * 0.1 or 0.1
    y_min, y_max = min_m - margin, max_m + margin

    padding = 60
    plot_w = width - 2 * padding
    plot_h = height - 2 * padding
    n = len(trajectory_data)

    def x_pos(i):
        return padding + (i / max(n - 1, 1)) * plot_w

    def y_pos(v):
        return padding + plot_h - ((v - y_min) / (y_max - y_min)) * plot_h

    # Build SVG
    svg = f'<svg width="{width}" height="{height}" xmlns="http://www.w3.org/2000/svg">'
    svg += f'<rect width="{width}" height="{height}" fill="#1a1a2e" rx="8"/>'

    # Grid lines
    for i in range(5):
        y = padding + i * plot_h / 4
        val = y_max - i * (y_max - y_min) / 4
        svg += f'<line x1="{padding}" y1="{y}" x2="{width-padding}" y2="{y}" stroke="#333" stroke-dasharray="4"/>'
        svg += f'<text x="{padding-8}" y="{y+4}" fill="#888" text-anchor="end" font-size="11">{val:.3f}</text>'

    # Baseline line
    baseline = trajectory_data[0]["metric"]
    by = y_pos(baseline)
    svg += f'<line x1="{padding}" y1="{by}" x2="{width-padding}" y2="{by}" stroke="#ff6b6b" stroke-dasharray="6" opacity="0.7"/>'
    svg += f'<text x="{width-padding+5}" y="{by+4}" fill="#ff6b6b" font-size="10">baseline</text>'

    # Data line
    points = " ".join(f"{x_pos(i)},{y_pos(d['metric'])}" for i, d in enumerate(trajectory_data))
    svg += f'<polyline points="{points}" fill="none" stroke="#4ecdc4" stroke-width="2"/>'

    # Data points
    for i, d in enumerate(trajectory_data):
        cx, cy = x_pos(i), y_pos(d["metric"])
        svg += f'<circle cx="{cx}" cy="{cy}" r="4" fill="#4ecdc4"/>'

    # Title
    svg += f'<text x="{width/2}" y="24" fill="#eee" text-anchor="middle" font-size="14" font-weight="bold">Optimization Trajectory</text>'
    svg += f'<text x="{width/2}" y="{height-10}" fill="#888" text-anchor="middle" font-size="11">Experiment Run</text>'
    svg += '</svg>'
    return svg

Embed the SVG output directly in the HTML report. Annotate significant jumps with brief labels.

HTML Presentation Template

Use templates/progress-presentation.html as a starting point. It provides:

• Clean, dark-themed styling suitable for research presentations
• Responsive layout
• Section scaffolding matching the recommended structure
• Placeholder for the trajectory chart

Replace placeholder content with your actual research data. Add, remove, or rearrange sections as the research demands. The template is a scaffold, not a constraint.

Claude Code

Generate the HTML, then show it to the human:

open to_human/progress-001.html

OpenClaw

Generate a PDF version. Options:

• Use Python weasyprint to convert HTML to PDF
• Use matplotlib to generate plots directly as PDF
• Create a simple markdown → PDF pipeline

Note the PDF path in HEARTBEAT.md so the human knows to look at it.

Presentation Quality Tips

• One insight per section — don't overload
• Label axes and units on all plots
• Use color consistently — one color for improvements, another for baselines
• Include confidence intervals or error bars where meaningful
• Show the trajectory early — it's the hook that tells the reader "this is working"
• End with a clear next step — the human should know what happens next without asking

Skill Routing: When to Use Which Domain Skill

The autoresearch skill orchestrates — domain skills execute. This reference maps research activities to the skills library.

Routing Principle

When you encounter a domain-specific task during research, search the skills library for the right tool. Read the SKILL.md of the relevant skill before starting — it contains workflows, common issues, and production-ready code examples.

Complete Routing Map

Data and Preprocessing

Model Architecture and Training

Fine-tuning

Post-training (RL / Alignment)

Interpretability

Distributed Training

Evaluation

Inference and Serving

Experiment Tracking

Optimization Techniques

Safety and Alignment

Infrastructure

Agents and RAG

Prompt Engineering and Structured Output

Multimodal

Observability

Emerging Techniques

Research Output

Common Research Workflows

"I need to fine-tune a model and evaluate it"

1. Pick fine-tuning skill based on needs (Unsloth for speed, Axolotl for flexibility) 2. Use lm-evaluation-harness for standard benchmarks 3. Track with W&B or MLflow

"I need to understand what the model learned"

1. Use TransformerLens for circuit-level analysis 2. Train SAEs with SAELens for feature-level understanding 3. Run interventions with NNsight or Pyvene

"I need to do RL training"

1. Start with TRL for standard PPO/DPO 2. Use GRPO skill for DeepSeek-R1 style training 3. Scale with OpenRLHF if needed

"I need to run experiments on cloud GPUs"

1. Modal for quick serverless runs 2. SkyPilot for multi-cloud optimization 3. Lambda Labs for dedicated instances

Finding Skills

If you're not sure which skill to use:

# Search by keyword in skill names
ls */*/SKILL.md | head -20

# Search skill descriptions for a keyword
grep -l "keyword" */*/SKILL.md

Or search the repository's README.md which lists all skills with descriptions.

Research Findings

Research Question

<!-- What are we trying to discover? One clear sentence. -->

Current Understanding

<!-- Updated after each outer loop cycle. What do we know so far? What patterns explain our results? What's the mechanism? This section should read like the core argument of a paper. -->

Key Results

<!-- Significant experimental findings. Include metrics, comparisons, and brief interpretation. Link to experiment directories for full details. -->

Patterns and Insights

<!-- What emerges across multiple experiments? What types of changes consistently work or fail? Why? -->

Lessons and Constraints

<!-- Specific actionable learnings that should guide future experiments. Things you tried that didn't work and WHY, so you don't repeat them. Constraints you discovered about the problem space.

Examples:

• Weight decay > 0.1 causes training instability at 125M param scale
• SwiGLU and RoPE improvements stack because they're orthogonal (FFN vs positional)
• Baseline only reproduces published numbers with batch_size=64, not 32
• Sleep phases before memorization completion hurt — model needs memories to consolidate -->

Open Questions

<!-- What remains unanswered? What would strengthen or challenge our current understanding? -->

Optimization Trajectory

<!-- Summary of inner loop progress. How has the metric evolved? Note inflection points and what caused them. -->

Research Log

Chronological record of research decisions and actions. Append-only.

<!-- Entry types: bootstrap — initial scoping, literature search, hypothesis formation inner-loop — experiment run and result outer-loop — synthesis, reflection, direction decision pivot — change in research direction report — progress presentation generated conclude — decision to finalize and write paper

Example entries: | 1 | 2026-03-15 | bootstrap | Searched Semantic Scholar + arXiv for efficient transformer architectures. Found 8 relevant papers. Gap: no systematic comparison of GLU variants on small models. Formed 3 hypotheses. Baseline: NanoGPT 5-min run, val_loss=4.82. | | 2 | 2026-03-15 | inner-loop | H1 run_001: swapped ReLU for SwiGLU in FFN. 5-min training run. val_loss=4.61 (baseline 4.82, delta -0.21). Kept. | | 3 | 2026-03-15 | inner-loop | H1 run_002: increased FFN hidden dim from 4x to 5.3x to match SwiGLU param count. val_loss=4.58 (-0.03 vs run_001). Marginal — SwiGLU benefit mostly from gating, not extra params. | | 4 | 2026-03-15 | inner-loop | H1 run_003: tried GEGLU instead of SwiGLU. val_loss=4.63. Slightly worse than SwiGLU. SwiGLU wins for this scale. | | 5 | 2026-03-15 | inner-loop | H2 run_004: replaced learned positional embeddings with RoPE. val_loss=4.55 (-0.06 vs SwiGLU baseline). Promising — stacks with SwiGLU. | | 6 | 2026-03-15 | inner-loop | H2 run_005: RoPE + SwiGLU combined. val_loss=4.41 (-0.41 vs original baseline). Best so far. | | 7 | 2026-03-16 | outer-loop | Reviewed 5 runs. Pattern: gating mechanisms (SwiGLU) and rotary embeddings (RoPE) give independent gains that stack. Combined improvement ~9%. But WHY do they stack? Hypothesis: they operate on orthogonal aspects (FFN expressiveness vs positional encoding). Direction: DEEPEN — test if adding RMSNorm also stacks independently. | | 8 | 2026-03-16 | inner-loop | H3 run_006: replaced LayerNorm with RMSNorm. val_loss=4.39 (-0.02). Small gain. Stacks but diminishing returns on normalization. | | 9 | 2026-03-17 | outer-loop | 8 runs complete. Optimization plateau around val_loss=4.38. The easy architectural wins (SwiGLU, RoPE) are captured. Searched literature on training dynamics — found papers on warmup schedules at small scale. Direction: BROADEN — shift from architecture to training recipe. | | 10 | 2026-03-17 | report | Generated progress-001.html with trajectory plot showing 9% improvement from architectural changes. |

Example entries (discovery-type research — understanding grokking): | 1 | 2026-03-20 | bootstrap | Searched literature on grokking and delayed generalization. Found Nanda et al. progress measures, Grokfast spectral filtering. Gap: no connection to memory consolidation theory from neuroscience. 3 hypotheses formed. | | 2 | 2026-03-20 | inner-loop | H1 run_001: trained modular addition transformer to memorization (100% train acc, 0% test). Steps to memorize: 1200. Baseline established. | | 3 | 2026-03-20 | inner-loop | H1 run_002: continued training with standard weight decay. Grokking at step 48000. Measured progress measure throughout — sharp transition at step 44000. | | 4 | 2026-03-20 | inner-loop | H1 run_003: inserted "sleep phase" at step 20000 (elevated weight decay + oscillatory LR for 500 steps). Grokking now at step 31000. 35% acceleration. | | 5 | 2026-03-20 | inner-loop | H1 run_004: sleep phase at step 10000. Grokking at step 27000. Earlier sleep = earlier grokking. | | 6 | 2026-03-20 | inner-loop | H1 run_005: sleep phase at step 5000 (before full memorization). Grokking at step 38000. Too early hurts — model hadn't memorized enough for consolidation to work. | | 7 | 2026-03-21 | outer-loop | Reviewed 5 runs. Clear pattern: sleep phases accelerate grokking but only AFTER memorization is complete. This matches memory consolidation theory exactly — you need memories formed before consolidation can reorganize them. Searched for neural slow-wave sleep literature. The weight decay + oscillatory LR during sleep phases mimics synaptic downscaling. Direction: DEEPEN — sweep sleep timing relative to memorization completion. | | 8 | 2026-03-21 | inner-loop | H1.1 run_006-010: swept sleep insertion at 80%, 100%, 120%, 150%, 200% of memorization step. Sweet spot at 110-120%. Consistent across 3 seeds. | | 9 | 2026-03-22 | outer-loop | 10 runs complete. The story is clear: neural networks "dream to learn" just like brains — consolidation after encoding, not during. Grokfast achieves similar acceleration through a different mechanism (gradient spectral filtering). Next: compare gradient spectra during our sleep phases vs Grokfast filtering to see if they converge on the same signal. Direction: BROADEN. | | 10 | 2026-03-22 | report | Generated progress-001.html with sleep timing vs grokking step plot. Key visual: sweet spot curve mirrors neuroscience memory consolidation window. | -->