Reproduce

Reproduce: paper reproduction from scratch

Reproducing an ML paper often means filling gaps the authors didn't ship, training scripts, hyperparameter tables, augmentation specifics, exact dataset splits. This skill walks seven stages from "I have an arxiv link" to "I have a replication run with measurable delta vs the paper's number."

Each stage has a separate reference file under references/ so this overview stays scannable.

When to run

The user just said any of:

• "reproduce / implement / replicate / re-run paper X"
• pasted an arxiv URL with reproduction intent ("can you redo this", "let's try this approach")
• pointed at an OpenReview / proceedings link with the same intent
• said "the paper has no code, can we build it"

Workflow

Walk them in order. Each stage has its own success criteria; do not advance to the next until the current one passes.

Working directory layout

For each paper reproduction, set up a dedicated workspace:

repro/<paper-arxiv-id>/
├── paper.md              # structured extract from stage 1
├── inventory.md          # what exists / missing from stage 2
├── gaps_filled.md        # hyperparam table with provenance from stage 3
├── code/                 # implementation from stage 4 (or cloned + extended)
├── data/                 # dataset symlinks or actual data from stage 5
├── dataset_substitution.md  # if a public dataset stood in for a private one
├── smoke_logs/           # outputs from stage 6
└── results.md            # replication outcomes from stage 7

This keeps reproductions self-contained and easy to revisit later.

Cross-references

• After stage 3, hand the gap analysis off to the paper-verification skill for a round-trip check ("did I really capture every hyperparam the paper mentions").
• Stage 4 implementation should be committed in small, reviewable pieces: each commit references the paper section that justified the filled value.
• Stage 6 smoke failures route to the /phd-skills:debug skill, not to ad-hoc fixes.
• Stage 7 launches go through the /phd-skills:launch checklist before any multi-hour run.
• Stage 7 comparisons go through the /phd-skills:compare skill at the paper's reported epochs (never current-vs-final).

Output

For each reproduction, the final artifact is results.md with absolute deltas (not just %) and one of three labels per metric:

• [matched within 0.X pp]: within the paper's reported variance
• [gap, hypothesis: ...]: measurable underperformance, with a stated hypothesis for the cause
• [fundamental disagreement, see X]: the result and the paper's claim are inconsistent in a way that needs investigation, not just more compute

If the workspace is on a public repo, link the workspace README from the project's main reproduction-tracking doc.

Stage 1: Paper acquisition

Fetch the paper into a structured local extract before doing anything else. Working from a polished local copy is much faster than re-scrolling arxiv on every hyperparam question.

Inputs

• arxiv URL or arxiv ID (e.g. 2508.12345, https://arxiv.org/abs/2508.12345)
• OpenReview URL or proceedings URL (CVPR / NeurIPS / ICLR / ICML / etc.)
• direct PDF URL as fallback

Steps

1. Prefer the HTML version

ArXiv's HTML rendering (https://arxiv.org/html/<id>) is far easier to parse than the PDF. Try it first via the Tavily MCP tool:

mcp__tavily__tavily_extract({
  urls: ["https://arxiv.org/html/2508.12345v1"],
  extract_depth: "advanced"
})

If HTML is missing or broken, fall back to:

• mcp__tavily__tavily_extract on the abstract page (/abs/<id>) to get bibliographic info
• The pdf skill (anthropic-office-skills:pdf) on the PDF URL for full content

2. Cross-check with HuggingFace papers

Use the HF papers tool to find paper metadata, related papers, and any community-maintained reproduction notes:

mcp__claude_ai_Hugging_Face__paper_search({
  query: "<paper title or arxiv id>"
})

This often surfaces:

• official author code repo (when arxiv didn't link it)
• community reproductions (huggingface spaces, gradio demos)
• related papers cited later

3. Capture into repro/<id>/paper.md

Structured extract with these sections (copy verbatim from paper, do not paraphrase yet):

# <Paper Title>

**arxiv**: <id>
**venue**: <conference/journal/year>
**authors**: <full author list>
**code**: <official repo url, "none" if not provided>
**abstract**: <verbatim>

## Method (verbatim from paper sections 3-4)

<copy-paste relevant sections>

## Hyperparameter tables (verbatim)

<every table that lists training hyperparameters, dataset stats, eval setup>

## Algorithm boxes (verbatim)

<every numbered algorithm or pseudocode block>

## Loss formulation (with equation numbers)

<every equation referenced from the method, with its number>

## Results tables (verbatim)

<the main result table + ablations>

## Caveats / negative results / mentioned-but-not-shown

<scan the paper for "we found", "in our experiments", "with X we observed", these often
hide critical implementation details>

4. Note what's missing

At the bottom of paper.md, add an # Open questions section listing every gap you noticed during the read:

• "Optimizer betas not specified, only said 'AdamW with default settings'"
• "Augmentation pipeline order ambiguous: does cutout happen before or after normalize?"
• "Eval protocol references appendix B which is not in the arxiv version"

These become inputs to stage 3 (gap analysis).

Success criteria

• paper.md exists with all six sections populated
• the # Open questions list has at least 3-5 items (most papers have many; if you found zero, you didn't read carefully enough)
• you can answer "what does this paper actually do" in one paragraph from paper.md alone

If the paper has an official code repo, note its URL but do not clone yet, that's stage 2.

Stage 2: Existing code discovery + inventory

Most papers ship some code. Almost none ship complete training pipelines. The goal of this stage is to know exactly what you have and exactly what you need to build.

Discover code repos

1. From the paper's claimed link

If paper.md lists an official repo URL, use it. Verify the URL actually works:

gh repo view <owner>/<repo>

2. Search if no official link

gh search repos "<paper title fragment>" --language python --sort stars --limit 10
gh search repos "<arxiv id>" --limit 10

Sometimes authors release code under a different name. Check the README of high-star matches for an arxiv reference back to your paper.

3. Papers With Code

Manually visit https://paperswithcode.com/paper/<paper-slug>. PWC often lists multiple implementations including community reproductions that may be more complete than the official one.

Clone and inventory

1. Shallow clone

cd repro/ < arxiv-id > /
git clone --depth 1 < repo-url > code/

Shallow clone is fine, you don't need history, you need current state.

2. Inventory what's there

Walk the repo and answer each:

Save this table to repro/<arxiv-id>/inventory.md.

3. Compare against gaps from paper.md

Look at the # Open questions from stage 1. Cross-reference each with the inventory:

• some open questions get resolved by reading the code (the paper said "default settings" but the code shows the defaults)
• some open questions remain (no code, or code-vs-paper mismatch)

The remaining open questions are the inputs to stage 3.

4. Sanity-check the model definition

Even if the model code is present, do a quick read-through:

• does the architecture in code match figure 2 / figure 3 of the paper?
• are the layer dimensions consistent with table 1?
• are there any flags / features in the code that aren't mentioned in the paper?

Note any mismatches in inventory.md. The code is usually the source of truth, but if the paper's claim depends on a feature absent from the code, that's important.

Success criteria

• inventory.md exists with the full table
• every "missing" component has a path in the planned implementation (i.e. you know where you'll create it in stage 4)
• the open-questions list from paper.md has been triaged: each item is now either "resolved by code at <path>" or carried forward to gap analysis

If the inventory shows the paper has a complete training repo, you may be able to skip directly to stage 5 (dataset). But verify completeness with one smoke before assuming.

Stage 3: Gap analysis

The paper says "trained for 100 epochs with AdamW." That sentence hides at least 8 hyperparameters. This stage extracts every value needed to launch a training run, with a provenance tag for each so you know how confident you are.

What to extract

For each missing component identified in stage 2, fill in:

Optimizer block

Batch size block

Schedule block

Augmentation pipeline

Reconstruct the _order_ of augmentations, not just the list. Order matters (random crop before vs after color jitter changes statistics). For each:

augmentation:
  - { name: RandomResizedCrop, size: 224, scale: [0.08, 1.0] } # [paper §4.2]
  - { name: RandomHorizontalFlip, p: 0.5 } # [paper §4.2]
  - { name: ColorJitter, brightness: 0.4, contrast: 0.4 } # [paper §4.2]
  - { name: RandAugment, N: 2, M: 9 } # [code, default config]
  - { name: ToTensor } # [framework]
  - { name: Normalize, mean: imagenet, std: imagenet } # [framework]

If the paper says "standard ImageNet augmentations," that's a [guess: venue convention] tag, note the assumption explicitly.

Loss formulation

For each loss term in the paper's main equation, fill in:

loss:
  - name: CrossEntropy
    weight: 1.0
    label_smoothing: 0.1 # [paper eq 3, λ_smooth = 0.1]
  - name: KLDivergenceWithTeacher
    weight: 0.5 # [paper eq 4, β = 0.5]
    temperature: 4.0 # [paper §4.1]

If the paper's equation has a coefficient, find and tag every coefficient. Equation numbers help when you re-read later.

Tricks

The "tricks" section often hides the most reproduction-critical details. Check for:

• label smoothing
• mixup / cutmix (probability + alpha)
• stochastic depth (drop_path_rate)
• exponential moving average (decay rate, update freq)
• gradient checkpointing
• mixed precision (fp16 / bf16)
• distillation temperature (if applicable)
• specific weight init scheme (trunc_normal, kaiming, etc.)

Each gets a row in gaps_filled.md with provenance.

Provenance tags

Every value should have one of:

• [paper §X]: explicit in the paper text or table
• [code: <path>:<lineno>]: found in the official code
• [venue convention]: guessed from common practice in the venue (e.g. "ImageNet augs" at CVPR are usually a specific recipe)
• [framework default]: explicitly using the framework's default
• [guess: assumption stated]: your best guess; document the reasoning

The fewer [guess] tags, the higher confidence the reproduction will replicate.

Round-trip check

After filling in gaps_filled.md, hand off to the existing paper-verification skill:

"Verify that gaps_filled.md is consistent with paper.md, every hyperparam I claimed [paper §X] for must actually be in section X."

This catches transcription errors and over-confident attributions before they propagate to stage 4.

Output

repro/<arxiv-id>/gaps_filled.md with:

• All seven blocks above (optimizer, batch, schedule, augmentation, loss, tricks, plus any architecture-specific block)
• Every value provenance-tagged
• Round-trip-verified against paper.md

Success criteria

• Every component identified as "missing" in stage 2's inventory now has values in gaps_filled.md
• < 30% of values are [guess] tagged (more than that means the paper is genuinely under-specified and the reproduction will be approximate)
• Round-trip with paper-verification raises no contradictions

Stage 4: Implementation

Now write the code that fills the gaps. The goal: minimum new code, maximum reuse of the cloned repo.

Bootstrap the environment

Use uv for fast, reproducible Python envs:

cd repro/ < arxiv-id > /code/
uv venv
source .venv/bin/activate
uv pip install torch numpy transformers wandb pyyaml

Add any paper-specific deps (timm, einops, datasets, accelerate, etc.) one at a time as you hit imports.

Reuse before you write

Before writing any new file:

1. Check the cloned repo for an existing scaffold. Many repos have scripts/ or tools/ with starter training code, even if incomplete, it gives you the right import/config conventions. 2. Match the existing repo's idioms:

• if the repo uses argparse, don't introduce hydra
• if configs are YAML, don't introduce TOML
• if the repo uses PyTorch Lightning, don't write a raw loop

3. Vendor missing pieces from timm / transformers / accelerate rather than rolling your own.

Write the missing pieces

For each "missing" component in inventory.md, create _one file per concern_:

• train.py: the training loop (most commonly missing)
• configs/<paper-name>-default.yaml: the hyperparam config matching gaps_filled.md
• optim_factory.py: optimizer + schedule construction (if not already in repo)
• losses.py: loss functions matching the paper's equations (if not already)
• eval.py: eval protocol (if not already, or if existing doesn't match the paper)

Keep files small. A 200-line training loop is suspicious.

Config-first

Write the config file first, before the training loop. The config is the contract between gaps_filled.md and the training code. Every value in the config should appear in gaps_filled.md with provenance.

# configs/repro-default.yaml
optimizer:
  name: AdamW
  lr: 3e-4
  weight_decay: 0.05
  betas: [0.9, 0.999]
  eps: 1e-8

schedule:
  epochs: 100
  warmup_epochs: 5
  warmup_type: linear
  decay: cosine

batch:
  per_device: 64
  global: 512 # validated at runtime

# ... rest matching gaps_filled.md

Commit per gap

This is the discipline that pays off later. Each commit fills one gap and references its provenance:

git add configs/repro-default.yaml
git commit -m "config: optimizer block from paper §4.1 + hyperparam table

Optimizer is AdamW (lr=3e-4, wd=0.05, betas=default).
betas tag is [framework default] since paper says 'default AdamW settings'.
"

git add losses.py
git commit -m "losses: cross-entropy + KL-divergence-with-teacher from paper eq 3-4

Implements L = CE(student, label) + 0.5 * KL(student || teacher) at T=4.
Coefficients from paper §4.1.
"

Three benefits:

1. when the run fails or the result diverges, you can git log the implementation to see which gap might be wrong 2. when the paper authors release more code later, you can diff per-commit 3. when someone reviews the reproduction, each commit is self-explanatory

Imports

Match the cloned repo's import style. If it uses absolute imports from a top-level package, do the same. If it uses relative imports inside subpackages, do the same. Import-style mismatches break the run silently when the cloned modules try to find each other.

What NOT to do

• Don't refactor the cloned code "to make it cleaner" before reproducing the result. Reproduce first, refactor never (or in a separate branch after the result lands).
• Don't add type hints, docstrings, or tests to the cloned code unless they were missing in a way that broke imports. Reproduction is not a code review.
• Don't fix the cloned code's bugs unless they prevent the run. The paper's result was produced by the code-as-written, including its bugs. "Fixing" a bug may explain a non-replication.
• Don't skip writing train.py because "the user can write it." The reproduction is incomplete without an executable launch.

Output

A working training launch:

python train.py --config configs/repro-default.yaml --debug-mode

--debug-mode should run for 10 iterations and exit cleanly. If it doesn't, stage 6 will be a mess. Fix it now.

Success criteria

• train.py exists and accepts --config
• the config in configs/repro-default.yaml matches gaps_filled.md 1:1
• --debug-mode runs end-to-end (10 iters) without error
• every commit references the gap it filled and the paper section that justified the value

Stage 5: Dataset acquisition

The dataset is the hardest part of most reproductions. Public datasets are usually fine; private datasets require a substitution decision that needs to be documented honestly.

Public dataset path

1. Try HuggingFace datasets first

mcp__claude_ai_Hugging_Face__hub_repo_search({
  query: "<dataset name>",
  repo_type: "dataset"
})

If the dataset is on HF Hub, loading it is one line:

from datasets import load_dataset
ds = load_dataset("<org>/<name>", split="train")

This handles caching, format conversion, and version pinning automatically.

2. Fall back to the official source

If not on HF:

• the paper's "data" section usually lists the canonical URL
• check the cloned repo's README for download instructions
• use wget / curl / aria2c for direct download (parallel chunks for large datasets)

3. Verify dataset integrity

Always check:

ls -la data/ < dataset > / | head
du -sh data/ < dataset > /*

# Sample-count sanity check
find data/ -type f -name "*.jpg" < dataset > /train | wc -l
# does this match the paper's reported train size?

If the count is off, you likely have a corrupted download or a different version. Resolve before stage 6.

4. Match split conventions

Papers sometimes use non-standard splits:

• "train" might mean "trainval" combined
• "val" might be the held-out test set the paper actually reports on
• some papers split the canonical val into val + dev_test

Document the split convention you're using in repro/<arxiv-id>/dataset.md:

# Dataset

**Source**: HuggingFace `<org>/<name>` v1.2.0 (commit hash <abc>)
**Train**: 1,281,167 samples (matches paper's table 1)
**Val**: 50,000 samples (paper's "test set", note: paper calls val "test")
**Splits used**:

- training: HF "train" split as-is
- evaluation: HF "validation" split, no further subsplit

Private dataset path (substitution)

If the paper used a proprietary, gated, or paper-specific collected dataset that isn't publicly available:

1. Try harder for legitimate access

• check if it's gated on HF (sometimes a free agreement gets you in)
• email the corresponding author (this often works)
• check if the academic license is available via institutional access
• check OpenReview reviewer guidance for the venue (sometimes "data on request" was promised)

2. Substitute with care

If access is genuinely impossible, pick a _structurally similar_ public dataset:

Selection criteria:

• same task type (classification → classification, not classification → detection)
• same modality (images at similar resolution, text at similar length)
• similar scale (don't substitute a 10-class 1k-image dataset for a 1000-class 1M-image one)
• prefer datasets cited in the paper's _related work_ section: those are explicitly comparable in the authors' framing

Use HF search with task tags:

mcp__claude_ai_Hugging_Face__hub_repo_search({
  query: "<task type> <modality> <key constraints>",
  repo_type: "dataset"
})

3. Document the substitution

Substitution is a real scientific compromise. It must be visible:

# Dataset substitution

**Original dataset (paper)**: ProprietaryDataset-X (not publicly available)
**Substitute**: HuggingFace `<org>/<public-name>`

**Justification**:

- Task: both are <task type>
- Modality: both are <modality>
- Scale: substitute is <X>k samples vs paper's <Y>k (within an order of magnitude)
- Cited in paper related work as comparable: <yes/no>

**Bias caveats**:

- the substitute has <some property> that the paper's dataset did not
- the substitute's class distribution is <skewed/balanced> differently
- the substitute may favor / disadvantage methods that <some sensitivity>

**Expected impact on reproduction**:

- absolute metric numbers will differ from the paper's
- relative comparisons within our reproduction (method A vs method B) are still meaningful
- claims that depend specifically on the original dataset's properties cannot be validated

This document goes into repro/<arxiv-id>/dataset_substitution.md and is referenced from results.md in stage 7.

Symlink, don't copy

For large datasets, symlink into the workspace rather than copying:

ln -s /shared/datasets/imagenet repro/ < arxiv-id > /data/imagenet

Saves disk space and avoids stale duplicates.

Success criteria

• dataset accessible from train.py at the configured path
• sample counts validated against the paper's reported numbers (or substitution documented if not)
• splits documented in dataset.md
• if substituted, dataset_substitution.md is honest about the compromise

Stage 6: Smoke runs

Three gated tiers. Each is cheap; each catches a class of failures the next can't. Do not advance to stage 7 until all three pass.

If any tier fails, route to the /phd-skills:debug skill, do NOT speculate about causes or restart with an ad-hoc fix.

Tier 1: Forward pass smoke (~30 seconds)

The cheapest test. One batch through the model in train and eval mode.

# scripts/smoke_forward.py
model = build_model_from_config("configs/repro-default.yaml")
batch = next(iter(build_dataloader(...)))

# Train mode
model.train()
out_train = model(batch)
loss_train = compute_loss(out_train, batch.label)
print(f"train loss: {loss_train.item()}")
assert torch.isfinite(loss_train), "train loss is NaN/Inf"

loss_train.backward()
total_grad = sum(p.grad.norm() for p in model.parameters() if p.grad is not None)
print(f"total grad norm: {total_grad}")
assert total_grad > 0, "no gradient flowed"

# Eval mode
model.eval()
with torch.no_grad():
    out_eval = model(batch)
print(f"eval output shape: {out_eval.shape}")

Pass criteria:

• training-mode forward produces finite loss
• gradient norm > 0 (gradients are flowing through the network)
• eval-mode forward produces finite output of the expected shape
• the _value_ of the initial training loss is within 2× of what the paper's initial-loss should be (e.g. for cross-entropy on 1000 classes, initial loss ≈ ln(1000) ≈ 6.9)

If the initial loss is wildly off (e.g. 0.001 or 1000), there's a bug in the loss or in the data labels.

Tier 2: Single-step optimizer smoke (~1 minute)

Forward → backward → optimizer step → forward again. Loss should decrease.

optimizer = build_optimizer(model.parameters(), config)
scheduler = build_scheduler(optimizer, config)

batch = next(iter(dataloader))

loss_before = compute_loss(model(batch), batch.label)
optimizer.zero_grad()
loss_before.backward()
optimizer.step()
scheduler.step()

loss_after = compute_loss(model(batch), batch.label)
print(f"loss: {loss_before.item()} -> {loss_after.item()}")
assert loss_after.item() < loss_before.item(), "loss did not decrease after one step"

Pass criteria:

• loss strictly decreased (allow tiny epsilon for fp16 noise)
• no parameter became NaN / Inf during the step
• optimizer state is populated for all parameters that received gradients

If loss doesn't decrease, common causes:

• learning rate too low (loss decrease is below numerical precision)
• gradient is masked by gradient clipping at 0
• wrong loss reduction (sum vs mean)
• model is in eval mode or BN is frozen when it shouldn't be

Tier 3: 20-iteration convergence smoke (~5 minutes)

Real training, but tiny. Watch the trajectory.

losses = []
for step in range(20):
    batch = next(iter(dataloader))
    loss = compute_loss(model(batch), batch.label)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    scheduler.step()
    losses.append(loss.item())
    print(f"step {step}: loss = {loss.item():.4f}")

# Check 5-step rolling window for monotonic-ish decrease
rolling = [sum(losses[i:i+5])/5 for i in range(len(losses)-4)]
trend_down = all(rolling[i+1] <= rolling[i] * 1.05 for i in range(len(rolling)-1))
assert trend_down, f"loss did not trend down over 20 iters: {rolling}"

Pass criteria:

• 5-iter rolling-window average decreases (allow 5% bumps for noise)
• no NaN / Inf at any step
• final loss < initial loss by at least one decimal place

If tier 3 fails after tiers 1-2 pass, common causes:

• learning rate is too high (early loss spike, then NaN)
• batch size is wrong (loss is jittery due to too-small batches)
• data shuffling is broken (training on the same batches repeatedly)
• BN running stats are broken (eval suddenly diverges from train)

What to do on failure

Do NOT restart with a different config and "see if it works." That hides the bug.

Hand off to /phd-skills:debug:

"Tier 2 smoke is failing, loss doesn't decrease after one optimizer step. Loss before: 6.92. Loss after: 6.92 (literally identical to 4 decimal places)."

The debug skill will probe (was the optimizer.step() call effective? did any parameter actually update? is the optimizer's lr non-zero at step 0?) before guessing.

Output

A smoke_logs/ directory with one log file per tier, captured outputs, and a smoke_status.md:

# Smoke status

- Tier 1 (forward): PASS: train loss 6.92 ≈ ln(1000), grad norm 142.3, eval output shape [64, 1000]
- Tier 2 (single step): PASS: loss 6.92 → 6.89
- Tier 3 (20 iter): PASS: rolling avg 6.91 → 6.34

Success criteria

All three tiers pass. Stage 7 (full replication) runs only after this gate.

Stage 7: Replication runs and comparison

The implementation works in smoke. Now run it at full scale and compare honestly to the paper's numbers.

Plan the runs

Use the existing experiment-design skill to plan:

• minimum: one full run reproducing the paper's main result (e.g. their primary baseline)
• ideal: 3 seeds for the main result + 1 run per important ablation
• budget-aware: pick the subset of ablations that the paper itself reports as load-bearing

The plan goes into repro/<arxiv-id>/run_plan.md with one row per planned run:

Launch with the launch checklist

Every full run goes through /phd-skills:launch first. The checklist will:

• diff your config against the previous reference run
• verify dataset and checkpoint paths
• confirm monitoring is set up (wandb tags scoped to this reproduction)
• verify the run name has no internal jargon (use repro-main-seed0, not wave-1)
• record an ETA in your local timezone

Do not skip the checklist for "small" runs. The hooks will warn anyway.

Track progress at the paper's reported epochs

Don't only check the final number. The paper usually reports a metric trajectory or at least mid-training checkpoints. Use /phd-skills:compare to align comparisons:

"Compare repro-main-seed0 to the paper's reported numbers at each of the paper's checkpoints (typically epochs 25, 50, 75, 100)."

Same-epoch comparisons surface convergence problems early, if the paper hits 60% top-1 by epoch 50 and your run is at 45%, that's diagnostic before the run finishes.

Record results honestly

repro/<arxiv-id>/results.md is the final artifact:

# Reproduction results

**Paper claim**: 76.0 ± 0.2 top-1 on ImageNet val (from table 1)

## Our runs

| Run              | Final top-1 | Final top-5 | Wall-clock  | Notes     |
| ---------------- | ----------- | ----------- | ----------- | --------- |
| repro-main-seed0 | 75.7        | 92.4        | 23.4 GPU-hr | clean run |
| repro-main-seed1 | 75.9        | 92.5        | 23.5 GPU-hr | clean run |
| repro-main-seed2 | 75.8        | 92.4        | 23.5 GPU-hr | clean run |

**Mean (3 seeds)**: 75.8 ± 0.1
**Paper claim**: 76.0 ± 0.2
**Delta**: -0.2 ± 0.2

## Verdict

**[matched within 0.3 pp]**, within paper's reported variance. Reproduction successful.

## Ablations

| Ablation | Our delta | Paper delta | Notes                                         |
| -------- | --------- | ----------- | --------------------------------------------- |
| no EMA   | -1.2      | -1.4        | matches direction; slightly smaller magnitude |
| no mixup | -0.8      | -1.1        | matches direction                             |

Three verdict labels

Use exactly one per metric:

• `[matched within X pp]`: your number is within the paper's reported variance (or, if no variance reported, within ±0.5pp on top-1 / ±1pp on more variable metrics)
• `[gap, hypothesis: ...]`: your number is measurably below the paper's, with a stated cause hypothesis (e.g. "we used a smaller batch size and didn't rescale lr; expect ~1pp gap")
• `[fundamental disagreement, see X]`: your number contradicts the paper in a way that can't be explained by config mismatch, points at either a real reproduction failure or a paper claim that doesn't hold up

Be honest. Reproductions that find gaps are more valuable than reproductions that fudge numbers to match.

Cite handoffs

In results.md, cite which skills were used at each stage:

Implementation gaps tracked in gaps_filled.md and verified via the paper-verification skill (round-trip checked 2026-04-30).

Smoke runs ran clean per /phd-skills:reproduce stage 6.

Replication runs launched via /phd-skills:launch checklist (logs in launch_logs/).

Numerical comparison aligned via /phd-skills:compare at paper's reported epochs.

This is also useful documentation for someone reviewing the reproduction later.

Success criteria

• at least one full replication run completes successfully
• results.md exists with verdict labels for each metric the paper reports
• gaps and disagreements have stated hypotheses, not just "we didn't match"
• the workspace is portable: someone else with the same data could rerun your repro-main-seed0 from configs/repro-default.yaml and get the same number