Ai Researcher

AI Researcher

When to Use

• Surveying state-of-the-art models, methods, or benchmarks
• Comparing model families or techniques with fair experimental design
• Designing ablation studies with controlled variables
• Writing research memos or technical reports for stakeholder decisions
• Critiquing methodology in papers or internal experiments
• Planning novel experiments with falsifiable hypotheses
• Reproducing published results and verifying claims

When NOT to Use

• Shipping production LLM features, RAG, or agent systems → ai-engineer
• Enterprise AI policy, regulation, or risk tiering → ai-risk-governance
• Adversarial product testing or jailbreak campaigns → ai-redteam
• Classical ML pipelines, A/B testing, or statistical analysis → data-scientist

Related skills

Core Workflows

1. Research question framing

1. Convert vague ask into falsifiable question 2. Define scope: task, data regime, compute budget, timeline 3. List baselines that must be beaten or matched 4. Specify primary and secondary metrics 5. Document assumptions and out-of-scope items

See `references/research_framing.md` for question templates and hypothesis types.

2. Literature review

Process:

1. Search: arXiv, ACL Anthology, OpenReview, major labs' blogs 2. Screen by relevance, recency, citation quality 3. Extract: problem, method, data, metrics, limitations 4. Synthesize themes and open gaps 5. Cite primary sources; avoid over-relying on secondary summaries

See `references/literature_review.md` for screening matrix and synthesis outline.

3. Experimental design

See `references/experiment_design.md` for power analysis pointers and leakage checks.

4. Benchmarking and analysis

• Use public benchmarks when task-aligned; document train/test contamination risk
• Report compute cost (GPU hours) alongside accuracy
• Separate in-distribution vs stress tests
• Visualize failure modes, not only aggregate scores

See `references/benchmarking.md` for leaderboard caveats and custom eval sets.

5. Research communication

Deliverable types: memo (1–3 pages), technical report, slide deck for decision meeting.

Include: question, method summary, results table, limitations, recommended next step.

See `references/research_writing.md` for memo structure and peer-review checklist.

When to load references

• Question and hypothesis → references/research_framing.md
• Literature survey → references/literature_review.md
• Experiments → references/experiment_design.md
• Benchmarks → references/benchmarking.md
• Writing → references/research_writing.md

Benchmarking

Table of contents

1. Leaderboard caveats 2. Custom eval sets

Leaderboard caveats

• Test set contamination in pretraining
• Tuning budget unfairness
• Metric gaming (format tricks)

Custom eval sets

Build domain-specific eval from:

• Production failure samples (redacted)
• SME-authored gold Q&A
• Adversarial cases from ai-redteam (sanitized)

Report mean, variance, and worst decile.

Experiment design

Table of contents

1. Reproducibility checklist 2. Leakage checks

Reproducibility checklist

• [ ] Random seeds recorded
• [ ] Data version / hash
• [ ] Code commit SHA
• [ ] Hyperparameters logged
• [ ] Hardware noted for timing claims

Leakage checks

• No test labels in feature engineering
• Temporal splits for time-series
• Deduplicate near-duplicate train/test examples

Literature review

Table of contents

1. Screening matrix 2. Synthesis outline

Screening matrix

| Paper | Year | Method | Data | Metric | Limitation | Include? |

Synthesis outline

1. Problem definitions across papers 2. Method families 3. Consistent findings 4. Conflicts and why 5. Open problems 6. Recommendation for this org

Research framing

Table of contents

1. Question template 2. Hypothesis types

Question template

## Research question
[Specific, falsifiable]

## Why now
[Gap or business driver]

## Success criteria
[Metric + threshold]

## Baselines
[What must be matched]

## Out of scope
[Explicit exclusions]

Hypothesis types

Research writing

Table of contents

1. Memo structure 2. Peer review checklist

Memo structure

1. Executive summary (5 sentences) 2. Background 3. Methods tried 4. Results (table + figure) 5. Limitations 6. Recommendation

Peer review checklist

• Claims match evidence
• Baselines fair
• Limitations honest
• Repro steps sufficient for internal replication