Experiment Code

Experiment Code

Generate and iteratively improve ML experiment code for research papers.

Input

• $0 — Task: generate, improve, debug, plot
• $1 — Research plan, idea description, or error message

References

• Experiment prompts and patterns: ~/.claude/skills/experiment-code/references/experiment-prompts.md
• Code patterns (error handling, repair, hill-climbing): ~/.claude/skills/experiment-code/references/code-patterns.md

Action: generate

Generate initial experiment code following this structure:

1. Plan experiments first — List all runs needed (hyperparameter sweeps, ablations, baselines) 2. Write self-contained code — All code in project directory, no external imports from reference repos 3. Include proper logging — Save results to JSON, print intermediate metrics 4. Generate figures — At minimum Figure_1.png and Figure_2.png

Mandatory Structure

project/
├── experiment.py      # Main experiment script
├── plot.py            # Visualization script
├── notes.txt          # Experiment descriptions and results
├── run_1/             # Results from run 1
│   └── final_info.json
├── run_2/
└── ...

Constraints

• No placeholder code (pass, ..., raise NotImplementedError)
• Must use actual datasets (not toy data unless explicitly requested)
• PyTorch or scikit-learn preferred (no TensorFlow/Keras)
• Each run uses: python experiment.py --out_dir=run_i

Action: improve

Improve existing experiment code: 1. Read current code and results 2. Reflect on what worked and what didn't 3. Apply targeted edits (prefer small edits over full rewrites) 4. Re-run and compare scores 5. Keep the best-performing code variant

Action: debug

Fix experiment code errors: 1. Read the error message (truncate to last 1500 chars if very long) 2. Identify the root cause 3. Apply minimal fix 4. Up to 4 retry attempts before changing approach

Action: plot

Generate publication-quality plots from experiment results: 1. Read all run_*/final_info.json files 2. Generate comparison plots with proper labels 3. Use the figure-generation skill for styling

Rules

• Always plan experiments before writing code
• After each run, document results in notes.txt
• Include print statements explaining what results show
• Method MUST not get 0% accuracy — verify accuracy calculations
• Use seeds for reproducibility
• Before each experiment include a print statement explaining exactly what the results are meant to show

Related Skills

• Upstream: experiment-design, algorithm-design
• Downstream: data-analysis, backward-traceability
• See also: code-debugging, paper-to-code

Experiment Code Patterns Reference

Pattern 1: Experiment Execution Loop (AI-Scientist)

MAX_ITERS = 4      # Max fix attempts per run
MAX_RUNS = 5       # Max experiment runs
MAX_STDERR_OUTPUT = 1500  # Truncate stderr

def perform_experiments(idea, folder_name, coder, baseline_results):
    current_iter = 0
    run = 1
    next_prompt = initial_prompt.format(...)

    while run < MAX_RUNS + 1:
        if current_iter >= MAX_ITERS:
            break
        coder_out = coder.run(next_prompt)
        if "ALL_COMPLETED" in coder_out:
            break
        return_code, next_prompt = run_experiment(folder_name, run)
        if return_code == 0:
            run += 1
            current_iter = 0
        current_iter += 1

def run_experiment(folder_name, run_num, timeout=7200):
    command = ["python", "experiment.py", f"--out_dir=run_{run_num}"]
    result = subprocess.run(command, cwd=cwd, stderr=subprocess.PIPE,
                           text=True, timeout=timeout)
    if result.returncode != 0:
        stderr_output = result.stderr[-MAX_STDERR_OUTPUT:]
        next_prompt = f"Run failed with the following error {stderr_output}"
    else:
        results = json.load(open(f"run_{run_num}/final_info.json"))
        results = {k: v["means"] for k, v in results.items()}
        next_prompt = f"Run {run_num} completed. Results: {results}"
    return result.returncode, next_prompt

Pattern 2: Hill-Climbing Code Optimization (AgentLaboratory)

def solve(self):
    num_attempts = 0
    best_pkg = None
    top_score = None

    while True:
        model_resp = query_model(
            system_prompt=self.system_prompt(),
            prompt=f"History: {self.history_str()}\nEnter a command: ",
            temp=1.0
        )
        cmd_str, code_lines, prev_code_ret, should_execute_code, score = \
            self.process_command(model_resp)

        if score is not None:
            if top_score is None or score > top_score:
                best_pkg = copy(code_lines), copy(prev_code_ret), ...
                top_score = score

        if num_attempts >= self.min_gen_trials and top_score is not None:
            break
        num_attempts += 1

    # Keep best code variant
    if top_score > self.best_codes[-1][1]:
        self.best_codes.append((copy(self.code_lines), copy(top_score), ...))
        self.best_codes.sort(key=lambda x: x[1], reverse=True)
        if len(self.best_codes) >= self.max_codes:
            self.best_codes.pop(-1)
            self.code_reflect = self.reflect_code()

Pattern 3: Initial Code Generation with Error History (AgentLaboratory)

def gen_initial_code(self):
    num_attempts = 0
    error_hist = []

    while True:
        if num_attempts == 0:
            err_hist = ""
        else:
            err = f"Previous command: {model_resp}. Error: {cmd_str}. " \
                  f"Do not repeat this error."
            error_hist.append(err)
            if len(error_hist) == 5:
                error_hist.pop(0)
            err_hist = "Error history:\n" + "\n".join(error_hist) + \
                      "\nDO NOT REPEAT THESE."

        model_resp = query_model(
            system_prompt=self.system_prompt(),
            prompt=f"{err_hist}\nUse ```REPLACE to create initial code: ",
            temp=1.0
        )
        cmd_str, code_lines, prev_code_ret, should_execute_code, score = \
            self.process_command(model_resp)
        if score is not None:
            break
        num_attempts += 1

    return code_lines, prev_code_ret, score

Pattern 4: Code Reflection for Improvement (AgentLaboratory)

def reflect_code(self):
    code_strs = "\n\n".join([
        f"Code variant:\n{code}\nScore: {score}"
        for code, score, _ in self.best_codes
    ])

    prompt = f"""Please reflect on ideas for how to improve your current code.
Examine the provided code and think very specifically (with precise ideas)
on how to improve performance, which methods to use, how to improve
generalization on the test set with line-by-line examples."""

    return query_model(prompt=prompt, system_prompt=system + code_strs)

Pattern 5: Self-Contained Project Structure (AI-Researcher)

project/
├── data/
│   └── data_loader.py
├── model/
│   └── model.py
├── training/
│   └── trainer.py
├── testing/
│   └── evaluator.py
├── run_training_testing.py   # Entry point
├── config.yaml
└── requirements.txt

Common Error Prevention Checklist

- Import everything you use
- Reflect on code before writing to catch bugs
- Use actual command names (EDIT, REPLACE), not the word COMMAND
- Under no circumstances use tensorflow or keras (use pytorch/sklearn)
- Make sure not to produce placeholder code
- Use seeds for reproducibility
- Include proper logging (print statements before results)
- Save results to JSON for downstream use

Experiment Code Prompts Reference

Extracted from AI-Scientist, AgentLaboratory, and AI-Researcher.

1. Initial Experiment Prompt (AI-Scientist)

Your goal is to implement the following idea: {title}.
The proposed experiment is as follows: {idea}.
You are given a total of up to {max_runs} runs to complete the necessary experiments. You do not need to use all {max_runs}.

First, plan the list of experiments you would like to run. For example, if you are sweeping over a specific hyperparameter, plan each value you would like to test for each run.

Note that we already provide the vanilla baseline results, so you do not need to re-run it.

For reference, the baseline results are as follows:

{baseline_results}

After you complete each change, we will run the command `python experiment.py --out_dir=run_i` where i is the run number and evaluate the results.
YOUR PROPOSED CHANGE MUST USE THIS COMMAND FORMAT, DO NOT ADD ADDITIONAL COMMAND LINE ARGS.
You can then implement the next thing on your list.

2. Post-Run Success Prompt (AI-Scientist)

Run {run_num} completed. Here are the results:
{results}

Decide if you need to re-plan your experiments given the result (you often will not need to).

Someone else will be using `notes.txt` to perform a writeup on this in the future.
Please include *all* relevant information for the writeup on Run {run_num}, including an experiment description and the run number. Be as verbose as necessary.

Then, implement the next thing on your list.
We will then run the command `python experiment.py --out_dir=run_{run_num + 1}`.
YOUR PROPOSED CHANGE MUST USE THIS COMMAND FORMAT, DO NOT ADD ADDITIONAL COMMAND LINE ARGS.
If you are finished with experiments, respond with 'ALL_COMPLETED'.

3. Error Handling Prompt (AI-Scientist)

Run failed with the following error {stderr_output}

Parameters: MAX_ITERS = 4, MAX_RUNS = 5, MAX_STDERR_OUTPUT = 1500

4. Plot Generation Prompt (AI-Scientist)

Great job! Please modify `plot.py` to generate the most relevant plots for the final writeup.

In particular, be sure to fill in the "labels" dictionary with the correct names for each run that you want to plot.

Only the runs in the `labels` dictionary will be plotted, so make sure to include all relevant runs.

We will be running the command `python plot.py` to generate the plots.

5. ML Engineer System Prompt (AgentLaboratory)

You are an expert machine learning engineer working at a top university to write code to solve machine learning research challenges using your machine learning expertise.

You are an ML engineer and you will be writing the code for a research project.
Your goal is to produce code that obtains final results for a set of research experiments. You should aim for simple code to collect all results, not complex code. You should integrate the provided literature review and the plan to make sure you are implementing everything outlined in the plan.

Make sure you do not write functions, only loose code.
You should also try generating at least two figures to showcase the results, titled Figure_1.png and Figure_2.png.
Your method MUST not get 0% accuracy. If it does, you have done something wrong and must correct this.
Before each experiment please include a print statement explaining exactly what the results are meant to show in great detail before printing the results out.

6. Code Editing Commands (AgentLaboratory)

REPLACE — Full code rewrite

<entire new code>

EDIT — Line-range replacement

<new lines to replace lines N through M (inclusive)>

Rules:

• Single command per turn
• Code is tested before replacing; errors prevent the change
• Prefer EDIT over REPLACE for incremental changes

7. Code Reflection Prompt (AgentLaboratory)

Please reflect on the following sets of code:
{code_variants_with_scores}
and come up with generalizable insights that will help you improve your performance on this benchmark.

Error Reflection:

This is your code: {code_str}

Your code returned the following error {code_return}. Please provide a detailed reflection on why this error was returned, which lines in the code caused this error, and exactly (line by line) how you hope to fix this in the next update. This step is mostly meant to reflect in order to help your future self fix the error better. Do not provide entirely new code but provide suggestions on how to fix the bug using LINE EDITS.

8. Reward Model / Scoring Prompt (AgentLaboratory)

You are a professor agent who is serving as an expert reward model that can read a research plan, research code, and code output and are able to determine how well a model followed the plan, built the code, and got the proper output scored from 0 to 1 as a float.

You must structure your score exactly in the following way:

<score here (float 0-1)>

9. Code Repair Prompts (AgentLaboratory)

Repair via REPLACE

You are an automated code repair tool.
Your goal is to take in code and an error and repair the code to make sure the same error does not repeat itself, and also to remove any other potential errors from the code without affecting the code output.
Your output should match the original code as closely as possible.
You must wrap the code in: ```python <code here> ```

Repair via EDIT

You are an automated code repair tool.
[Same goal as above]
Please use the code editing tool to fix this code.
Your output should look like: ```EDIT N M <new lines> ```

10. AI-Researcher: Project Structure Constraints

OBJECTIVE:
Create a self-contained, well-organized implementation in the project directory.

CODE INTEGRATION PRINCIPLES:
1. Self-Contained Project
   - ALL code must reside within the project directory
   - NO direct imports from reference codebases
   - Reference code must be thoughtfully integrated

2. Code Adaptation Guidelines
   - Study reference implementations thoroughly
   - Rewrite and adapt code to fit your project's architecture
   - Document the origin and modifications of adapted code

IMPORTANT:
- No placeholder code (pass, ..., raise NotImplementedError)
- Must use actual datasets (not toy data)
- Must generate figures (Figure_1.png, Figure_2.png minimum)
- PyTorch or scikit-learn only