Hypothesis And Ablation Planner

Hypothesis and Ablation Planner

Ablations should test explanations, not merely remove components.

Procedure

1. State the main empirical claim. 2. Decompose the method into causal components. 3. For each component, define:

• claimed mechanism
• minimal ablation
• expected directional effect
• competing explanation

4. Add controls for:

• parameter count
• compute budget
• data leakage/filtering
• prompt/template effects
• preprocessing artifacts
• implementation bias

5. Minimize the ablation set:

• remove redundant studies
• prioritize hypothesis-separating experiments

6. Label each ablation:

• must-have
• useful
• optional

Rules

• Every ablation must answer a specific causal question.
• Avoid ablation explosion.
• If two explanations remain confounded, state it explicitly.
• Prefer interpretable ablations over cosmetic variants.
• “Removing one block” is not sufficient unless tied to a hypothesis.
• Keep training budget and evaluation protocol controlled whenever possible.

Output Format

Final Checks

After the table, briefly state:

• which hypotheses are still confounded
• which ablation is most decisive
• which ablation is cheapest per unit insight