Scholar Deep Research

Scholar Deep Research

End-to-end academic research workflow that turns a question into a cited, structured report. Built for depth: multi-source federation, transparent ranking, citation chasing, and a mandatory self-critique pass before the report ships.

When to use

Explicit triggers: "literature review", "research report", "state of the art", "survey the field", "what's known about X", "deep research on Y", "systematic review", "scoping review", "compare papers on Z".

Proactive triggers (use without being asked):

• User asks a factual question whose honest answer is "it depends on the literature"
• User frames a research plan and needs the background section
• User is drafting a paper intro/related-work and hasn't yet scoped prior work
• User proposes a method and asks whether it's novel

Do not use when: a single known paper answers the question, the user wants a tutorial (not a survey), or they're debugging code.

Guiding principles

1. Scripts over vibes. Every search, dedupe, rank, and export step runs through a script in scripts/. The same input should produce the same output. Do not improvise ranking or counting by eye. 2. Sources are federated, not singular. OpenAlex is the primary backbone (free, 240M+ works, no key). arXiv (CS/ML/physics preprints), Crossref (DOI metadata), PubMed (biomedical), DBLP (CS conferences/journals), bioRxiv (life-sci preprints via Europe PMC), and Exa (open-web, requires EXA_API_KEY) fill gaps. Semantic Scholar is also script-driven — build_citation_graph.py --source s2|both is the spine path for Phase 4, with better CS / arXiv / cross-disciplinary coverage than OpenAlex; the two graphs disagree more than you'd expect. The asta MCP tools (mcp__asta__*) and Brave Search are skin — used opportunistically for relevance ranking or non-academic context, never on the critical path. If MCP times out, research continues. 3. State is persistent. Everything goes through research_state.json. Queries ran, papers seen, decisions made, phase progress. Research becomes resumable and auditable. 4. Citations are anchors, not decorations. Every non-trivial claim in the draft carries [^id] where id matches a paper in state. Unanchored claims are treated as hallucinations and fail the gate. 5. Saturation, not exhaustion, is the stop signal. A phase ends when a new round of search adds <20% novel papers AND no new paper has >100 citations. 6. Self-critique is a phase, not a checkbox. Phase 6 reads the draft with adversarial intent. Its output goes into the report appendix.

The 8-phase workflow (Phase 0..7)

Phase 0: Scope       → decompose question, pick archetype, init state
Phase 1: Discovery   → multi-source search, dedupe
Phase 2: Triage      → rank, select top-N for deep read
Phase 3: Deep read   → extract evidence per paper
Phase 4: Chasing     → citation graph (forward + backward)
Phase 5: Synthesis   → cluster by theme, map tensions
Phase 6: Self-critique → adversarial review, gap finding
Phase 7: Report      → render archetype template, export bibliography

Each phase writes to research_state.json before advancing. If the user pauses or a session crashes, the next run reads the state and picks up from the last completed phase.

Phase 0 — Scope

Before searching anything, decompose the question.

1. Restate the question in one sentence. Surface ambiguities. 2. PICO-style decomposition (or equivalent for non-biomedical fields):

• Population / Problem — what system, species, setting, or phenomenon?
• Intervention / Independent var — what method, factor, or manipulation?
• Comparison — against what baseline or alternative?
• Outcome — what is being measured or claimed?

3. Pick an archetype that matches user intent (see references/report_templates.md):

• literature_review — what is known about X (default)
• systematic_review — rigorous PRISMA-lite, comparison of many studies on one narrow question
• scoping_review — what has been studied and how (breadth over depth)
• comparative_analysis — X vs Y, head-to-head
• grant_background — narrative background + gap for a proposal

4. Draft keyword clusters — 3-5 Boolean clusters covering synonyms, acronyms, and variant spellings. Include a "negative" cluster (terms to exclude). 5. Initialize state:

   python scripts/research_state.py --state research_state.json init \
     --question "<restated question>" \
     --archetype literature_review

(--state is top-level and applies to every subcommand; init itself takes --question, --archetype, and optional --force.)

When in doubt about archetype, ask the user. The choice shapes everything downstream.

Phase 1 — Discovery

Run searches across all available sources, in parallel where the source can take it. OpenAlex is primary; the others fill gaps.

Where parallelism actually pays off. The right place to fan out is Phase 3 (one agent per paper to read PDFs concurrently — see references/agent_prompts/phase3_deep_read.md). At Phase 1 the bottleneck is the upstream API, not local compute, and parallel fan-out across the same source mostly buys 429s and sticky cooldowns. The skill's bias should be: parallel between different sources, serial within one source. Concretely:

• Parallel-friendly: OpenAlex (polite-pool, very tolerant), Crossref (polite-pool), Exa (paid quota), bioRxiv (Europe PMC).
• Self-serialised (file-locked, automatic): arXiv (≥3s/req), PubMed (≥0.34s/req without NCBI_API_KEY, ≥0.10s with), DBLP (1s buffer to avoid SSL EOF flakes).

The serialised sources use a per-source file lock under ${SCHOLAR_CACHE_DIR:-.scholar_cache}/rate/<source>.lock, so even N parallel search_arxiv.py invocations from the same agent will queue automatically and sleep the right gap — no agent-side coordination required, but parallel calls don't speed those sources up either, just don't error.

# Primary (no API key, always available)
python scripts/search_openalex.py --query "<cluster 1>" --limit 50 --state research_state.json
python scripts/search_openalex.py --query "<cluster 2>" --limit 50 --state research_state.json

# Domain-specific (use when relevant)
python scripts/search_arxiv.py    --query "<cluster>" --limit 50 --state research_state.json  # CS/ML/physics preprints
python scripts/search_dblp.py     --query "<cluster>" --limit 50 --state research_state.json  # CS gold-standard bibliography (no abstracts)
python scripts/search_pubmed.py   --query "<cluster>" --limit 50 --state research_state.json  # biomedical (PubMed)
python scripts/search_biorxiv.py  --query "<cluster>" --limit 50 --state research_state.json  # life-sci preprints (bioRxiv + medRxiv via Europe PMC)
python scripts/search_crossref.py --query "<cluster>" --limit 50 --state research_state.json  # DOI-backed metadata

# Open-web coverage (optional, requires EXA_API_KEY) — finds material the
# scholarly APIs miss: lab sites, institutional PDFs, conference mirrors,
# preprints parked outside arXiv, NGO/government reports.
python scripts/search_exa.py --query "<cluster>" --limit 50 --state research_state.json

# Dedupe across sources (DOI-first, title-similarity fallback)
python scripts/dedupe_papers.py --state research_state.json

MCP enrichment (optional, run if available): call mcp__asta__search_papers_by_relevance and mcp__asta__snippet_search and feed results via scripts/research_state.py ingest. If the MCP call errors or times out, do not retry — move on.

Iterate. Read the state file. Are there keyword gaps? Are there authors appearing 3+ times whose other work you haven't pulled? Run another round. Stop when saturation hits — every source, not just the last one queried:

python scripts/research_state.py saturation --state research_state.json
# Returns { "per_source": {...}, "overall_saturated": true/false, ... }

overall_saturated is true only when every queried source has run at least --min-rounds (default 2) rounds AND each is individually below the new-paper percentage and new-citation thresholds. A source that has been queried only once cannot be declared saturated, which rules out the failure mode where a single quiet source falsely ends discovery. Use --source openalex to check one source in isolation.

Budget caps and broad-topic escape hatches. Phase 1 has two hard caps to prevent runaway agents: SCHOLAR_PHASE1_MAX_ROUNDS (default 10 rounds per source) and SCHOLAR_PHASE1_MAX_REQUESTS_PER_SOURCE (default 20 ingests per source). Hitting either returns phase1_budget_exhausted with a next: hint. For genuinely broad topics that cross subfields (e.g. CS-ML topics with multiple keyword clusters), the saturation thresholds can also fail to converge under the defaults — relax them with SCHOLAR_SATURATION_NEW_PCT (default 20.0), SCHOLAR_SATURATION_MAX_CITATIONS (default 100), and SCHOLAR_SATURATION_NEW_AUTHORS_PCT / SCHOLAR_SATURATION_NEW_VENUES_PCT. These env vars are honored both by python scripts/research_state.py saturation and by the G2 gate, so raising them lets the gate accept "good enough" coverage on topics where the default is unreachable.

Phase 2 — Triage

Rank the deduplicated corpus and pick the top-N for deep reading.

python scripts/rank_papers.py \
  --state research_state.json \
  --question "<phase 0 question>" \
  --alpha 0.4 --beta 0.3 --gamma 0.2 --delta 0.1 \
  --top 20

The formula is transparent — the script prints it and writes the components to state so the report can cite its own methodology:

score = α·relevance + β·log10(citations+1)/3 + γ·recency_decay(half-life=5yr) + δ·venue_prior

Defaults target a literature review. For a scoping review prefer higher α (relevance) and lower β (citations). For a systematic review of a narrow question, lower α and higher β.

Write the top-N selection to state:

python scripts/research_state.py select --state research_state.json --top 20

Triage the selection into deep / skim / defer tiers before advancing. Phase 3 fan-out is the most expensive stage of the workflow; not every selected paper deserves a full agent dispatch:

python scripts/skim_papers.py --state research_state.json \
  --deep-ratio 0.5 --skim-ratio 0.5

Defaults split the top-N evenly: top half → deep (agent dispatch in Phase 3), bottom half → skim (abstract-derived evidence stub auto-filled, depth=shallow). For tighter budgets, use --deep-ratio 0.3 --skim-ratio 0.5 — the remaining 20% gets tier=defer and is removed from selected_ids (still queryable as candidates for citation chase).

The script emits data.deep_tier_preview listing the deep-tier papers by triage_score. Show this to the user before advancing so they can hand-override before agents fan out (re-run with different ratios, or manually re-rank in state). Triage is required before G3 passes — the gate's triage_applied check rejects the advance otherwise.

Optional but recommended — prefetch deep-tier PDFs before agent fan-out:

python scripts/prefetch_pdfs.py --state research_state.json \
  --tier deep --concurrency 4

Fetches every deep-tier paper's PDF into ${SCHOLAR_CACHE_DIR:-.scholar_cache}/pdfs/<id-hash>/ via paper-fetch (with Unpaywall fallback), in parallel waves, and writes pdf_path / pdf_status / pdf_source / pdf_bytes per paper. Phase 3 agents then read the local file directly instead of each running its own download — Agent context stays focused on reading + reasoning, not on retrying paywalls.

Failures land as pdf_status='failed' with a pdf_failure_code (paper_fetch_error, no_open_access_pdf, pdf_download_failed, …); papers without a DOI get pdf_status='no_doi'. Phase 3 agents check pdf_path first and only fall back to extract_pdf.py --doi if the prefetched path is missing. Re-running prefetch is cheap: papers with an existing pdf_path on disk are skipped (pdf_status='cached').

Human-in-loop for paywalled PDFs. When automatic fetch fails (paywall, OA chain exhausted, no DOI), surface a hand-fetch list to the user via --emit-manifest (read-only):

python scripts/prefetch_pdfs.py --state research_state.json --emit-manifest
# Returns { needs_user_download: [{id, doi, title, drop_at, alt_urls}, ...] }

The user downloads each PDF (institutional VPN, ResearchGate, etc.) and drops it at the listed drop_at path (any *.pdf filename in that subdir works). On the next normal prefetch_pdfs.py run, dropped files are auto-absorbed as pdf_source='user_provided' without re-fetching.

Skip prefetch entirely when paper-fetch is not installed AND you don't want Unpaywall traffic — Phase 3 agents will then download per-paper inside their own contexts (slower, noisier, but functionally identical).

Phase 3 — Deep read (parallel agent fan-out)

Phase 3 splits by tier:

• `tier=skim` — apply_triage() already wrote an abstract-derived evidence stub with depth=shallow. No further action needed.
• `tier=deep` — dispatch one agent per paper, in parallel waves of 8–10. Each agent reads the PDF, writes structured evidence back to state, and returns one JSON line. The host's main context never sees the full PDF text.

The agent prompt template lives at references/agent_prompts/phase3_deep_read.md. Load it once, instantiate per paper, and dispatch all N tool_use calls in a single message so they fan out concurrently. Per-agent contract:

• Input: paper_id, doi, pdf_url, abstract, question, state_path
• Action: extract_pdf.py --doi <doi> --output <tmp> → read text → write evidence --depth full
• Output: one line {"paper_id": "...", "status": "ok"|"evidence_unavailable", ...}

The state CLI is exclusive-locked, so N agents writing concurrent evidence calls are serialized automatically — no coordination needed.

# After all wave(s) complete, verify deep-tier coverage:
python scripts/research_state.py advance --state research_state.json \
  --to 4 --check-only

If deep_tier_full_evidence is failing, dispatch a follow-up wave for the missing ids only. If a paper's full text is genuinely unreachable (paywall, exhausted OA chain), the agent should write a depth=shallow record with method starting evidence_unavailable: per the prompt's failure-mode section — that record satisfies depth_marks_valid without inflating the deep-tier coverage count.

Manual fallback (no agents available). Hosts that cannot dispatch parallel agents (some non-CC platforms) can run Phase 3 sequentially in the main session: for each tier=deep paper, extract_pdf.py --doi <doi> then research_state.py evidence --id <pid> --depth full .... Slower and burns more context, but the gate logic is identical.

Phase 4 — Citation chasing

Take the top 5-10 highest-ranked papers and expand the graph.

# Preview the request count first — this is the most expensive command
python scripts/build_citation_graph.py \
  --state research_state.json \
  --seed-top 8 --direction both --depth 1 --dry-run

# Run with an idempotency key so a retry after a network blip is free
python scripts/build_citation_graph.py \
  --state research_state.json \
  --seed-top 8 --direction both --depth 1 \
  --idempotency-key "chase-$(date -u +%Y%m%dT%H%M)"

The script pulls backward references (what did this paper cite?) and forward citations (who cited this paper?), deduplicates against existing state, and writes new candidate papers with discovered_via: citation_chase. Run rank + deep read again on any new high-scoring additions.

Dual backend. --source openalex|s2|both (default both). OpenAlex covers most fields well; Semantic Scholar (S2) has better CS / arXiv / cross-disciplinary coverage. The two graphs disagree more than you'd expect — running both then deduping by id surfaces real coverage gaps. S2 needs a DOI / arXiv id / PMID on each seed (it doesn't accept OpenAlex ids); seeds without one skip the S2 backend. S2_API_KEY env var raises the S2 quota; without it the public quota of ~1 req/s applies.

Idempotency. When --idempotency-key <k> is set, the first successful run writes {response, signature} to .scholar_cache/<hash>.json. A retried run with the same key replays the cached response without re-hitting OpenAlex or re-mutating state. Reusing the same key with different arguments returns idempotency_key_mismatch rather than silently serving stale data. Cache directory: SCHOLAR_CACHE_DIR env var, default .scholar_cache/.

Special case — a highly cited paper has never been challenged. If rank says a paper is top-3 by citations but no critiques appear in the corpus, search explicitly for "<first author> <year>" critique OR limitations OR reanalysis OR failed replication. This is the confirmation-bias backstop.

Phase 5 — Synthesis

No scripts here — this is where the agent earns its keep. Cluster and structure:

1. Thematic clustering. Group the top-N into 3-6 themes that map onto the report outline. Themes should be orthogonal: a paper can be primary to one, secondary to at most one other. 2. Tension map. Where do papers disagree? For each disagreement, note: which papers, on what, and whether the disagreement is empirical (different data), methodological (different tools), or theoretical (different framings). 3. Timeline. When relevant, a chronological arc: seminal paper → consolidation → refinement → current frontier. 4. Venn / gap. What has been studied well, partially, and not at all? The gap is the pivot for Phase 7.

Phase 6 — Self-critique

This is not optional. Load assets/prompts/self_critique.md and run the full checklist against your draft (still unpublished). The checklist covers:

• Single-source claims (any claim backed by only one paper?)
• Citation/recency skew (is the latest-2-years window covered?)
• Venue bias (is the corpus dominated by one journal/venue?)
• Author bias (does one lab dominate the citations?)
• Untested high-citation papers (anyone cite a paper without reading a critique?)
• Contradictions buried (any tension in Phase 5 that got glossed over?)
• Archetype fit (does the structure match the chosen archetype?)
• Unanchored claims (any statement without a [^id] anchor?)

Write findings to research_state.json under self_critique and fix blockers before Phase 7. Findings go into the report appendix verbatim — the reader deserves to see what the research process doubted itself about.

Phase 7 — Report

Render an archetype scaffold from state, then fill the agent-prose slots and validate anchors:

# Generate the scaffold — fills header, themes, tensions, methodology
# appendix, self-critique appendix, and bibliography anchor index from
# state. Leaves `<!-- AGENT: ... -->` placeholders for prose.
python scripts/render_report.py --state research_state.json
# → reports/<slug>_<YYYYMMDD>.md by default; pass --output PATH to override.

# After filling in the prose, lint every [^id] anchor against
# state.papers. Catches typo'd anchors before the report ships.
python scripts/render_report.py --state research_state.json \
  --lint reports/<slug>_<YYYYMMDD>.md

# Export bibliography in the user's preferred format
python scripts/export_bibtex.py --state research_state.json --format bibtex --output refs.bib
python scripts/export_bibtex.py --state research_state.json --format csl-json --output refs.json

The scaffold's body uses [^id] anchors (the paper id from state). The bibliography section at the bottom carries one definition per selected paper. The lint mode flags unknown_anchors_used (typos) and undefined_in_text (anchors with no footnote definition); both are blockers. unused_definitions is a soft signal — selected papers that ended up not cited inline.

Save path convention: reports/<slug>_<YYYYMMDD>.md. The skill does not write outside the working directory unless the user specifies a path.

Report archetype selection

Templates live in assets/templates/<archetype>.md. Load only the one you need.

Scripts reference

All scripts accept --help, --schema, emit a structured JSON envelope on stdout, and use research_state.json as the single source of truth. Every script is idempotent on the state file (network-layer idempotency is P1 work).

CLI contract, env vars, and state schema

Three details that agents discover by running scripts and reading the JSON envelopes — kept out of the body to save context. Load on demand:

• references/cli_contract.md — the success/failure envelope shape, exit codes, --schema introspection, and idempotency cache semantics.
• references/env_vars.md — the trust-boundary env vars (SCHOLAR_*, NCBI_API_KEY, EXA_API_KEY, S2_API_KEY, PAPER_FETCH_SCRIPT). Agents should never set these — surface to the user when a script reports a missing one.
• references/state_schema.md — the research_state.json shape. Prefer python scripts/research_state.py --schema for the live, machine-readable version.

Completion gates

Each phase transition has a gate (G1..G7). Advance ONLY via:

python scripts/research_state.py --state <path> advance          # advance by 1
python scripts/research_state.py --state <path> advance --check-only   # preview only

The gate predicates are enforced in scripts/_gates.py. Direct set --field phase is rejected — the phase field is no longer settable. If the gate fails, the envelope lists the failing checks by name so you know exactly what's missing.

Enrichment with MCP tools

Semantic Scholar coverage is not one of these — it is reached through the script path (build_citation_graph.py --source s2|both) and is a first-class Phase 4 backend, not enrichment. The MCP tools below are the genuine skin layer: they may time out, get renamed, or be absent entirely, and no phase output depends on them.

If the session has asta or Brave Search MCP tools available, use them as enrichment:

• mcp__asta__search_papers_by_relevance — good for dense relevance ranking on top of the script searches
• mcp__asta__get_citations — lighter weight than build_citation_graph.py for spot-checking a single seed paper
• mcp__asta__snippet_search — grep-like search across abstracts
• Brave Search — non-academic sources (blog posts, press releases, pre-print discussion)

Treat MCP tools as unreliable by design — they may timeout or be unavailable. Never place a phase-critical step behind an MCP call. Scripts are the spine; MCP is the skin.

Pitfalls (short list; see references/pitfalls.md for detail)

1. Treating the first page of search results as "the literature" — run multiple keyword clusters and chase citations. 2. Unanchored claims — every non-trivial statement in the report needs a [^id] pointing to a paper in state. 3. Confirmation bias — actively search for critiques of top-cited papers; see Phase 4 special case. 4. Preprint conflation — arXiv/bioRxiv are preprints; tag them as such in the report and weight evidence accordingly. Lint-safe convention: place the anchor and marker separately — [^id] *(preprint)*, not [^id, preprint] (commas inside footnote brackets break Markdown parsing and the render_report.py --lint check). 5. Venue monoculture — if >60% of top-N come from one journal/venue, broaden sources. 6. Author monoculture — same for a single lab or author. 7. Recency collapse — the last 2 years matter for "state of the art" framings; check explicit coverage. 8. Stale MCP tool names — MCP servers rename tools; always list available tools before assuming names. Script paths are stable; MCP names are not. 9. Single-shot search — budget for ≥3 search rounds per cluster, not one. 10. Skipping self-critique — the temptation to ship a clean draft is exactly when Phase 6 catches the most.

Example interaction

A complete walk-through (CRISPR base editing for DMD — Phase 0 question restate through Phase 7 report and bibliography) lives in references/example_run.md. Read it once when you want to see what a healthy run looks like end-to-end; it's not load-bearing for routine sessions.

References

Modular documentation, loaded only when needed:

• references/search_strategies.md — Boolean clusters, PICO, snowballing, saturation math
• references/source_selection.md — which database for which question
• references/quality_assessment.md — CRAAP, journal tier, retraction check, preprint handling
• references/report_templates.md — the 5 archetypes with section-by-section guidance
• references/pitfalls.md — long-form version of the pitfalls list with examples
• references/cli_contract.md — JSON envelope shape, exit codes, --schema introspection, idempotency cache
• references/env_vars.md — trust-boundary configuration (SCHOLAR_*, NCBI_API_KEY, EXA_API_KEY, S2_API_KEY, PAPER_FETCH_SCRIPT)
• references/state_schema.md — research_state.json shape and ID-normalization rules
• references/example_run.md — full end-to-end example (CRISPR base editing for DMD)
• references/agent_prompts/phase3_deep_read.md — per-paper prompt for parallel agent fan-out in Phase 3

Self-critique checklist (Phase 6)

Read your draft with adversarial intent. Write findings to state.self_critique.findings via:

python scripts/research_state.py critique --finding "..."

When all findings are addressed (resolved) or explicitly accepted, write the appendix:

python scripts/research_state.py critique --appendix "$(cat critique_summary.md)"

The appendix is copied verbatim into the report. Don't sanitize it — readers deserve to see what the process doubted itself about.

---

The checklist

Work top to bottom. For each item, write a one-line finding even if the answer is "checked, no issue found, here's why."

1. Unanchored claims

Scan the draft for sentences that make a non-trivial assertion without [^id]. Each one is either:

• a hallucination (remove or anchor)
• a definitional sentence that's OK without an anchor (mark as such)

2. Single-source claims

For every claim with exactly one anchor, ask: would the report still be honest if that one paper turned out to be wrong? If not, find a corroborating anchor or downgrade the claim.

3. Citation skew — venue

Run python scripts/research_state.py query selected and tally venues. If one venue accounts for >60% of selected papers, note it. Either justify (the field really is that concentrated) or broaden.

4. Citation skew — author

Same for first authors and senior (last) authors. >40% from one lab is a flag.

5. Recency coverage

What's the most recent year in the selected set? If the latest 18 months are absent, run a final search round restricted to that window. Re-rank. Re-select.

6. Untested high-citation papers

Identify the top 3 papers by citation count in the selected set. For each, check whether the corpus contains any critique, replication, or failed reproduction. If not, search explicitly:

"<first author> <year>" (critique OR limitation OR replication OR challenge)

Even a "no critique exists" finding is a legitimate result — note it.

7. Buried tensions

Re-read state.tensions. For each tension, find where it appears in the draft. If a tension was documented in Phase 5 but doesn't appear in the body, you buried it. Surface it.

8. Archetype fit

Look at the chosen archetype's structure (in references/report_templates.md). Does the draft actually follow it? A draft that secretly drifted from comparative_analysis to literature_review is a sign you should re-pick.

9. Preprint flagging

Every paper with source: arxiv and no journal DOI should render as [^id, preprint] in the body. Grep for these IDs and check.

10. Methodology appendix

Is the methodology appendix populated? It should include: queries run, sources, dedupe stats, ranking formula, weights, and selection size. If it's a stub, fill it from state.queries and state.ranking.

11. Saturation actually happened

Did Phase 1 actually saturate, or did it stop because the model got tired? Check:

python scripts/research_state.py saturation

If the last round wasn't saturated, run more rounds before publishing.

12. Counter-arguments

Is there a strong counter-argument the report doesn't engage with? If your topic is contested at all, the absence of opposition in the draft is itself suspicious.

13. The "honest reader" test

If the user reads this report and then reads one of the papers it cites, will they feel misled? Find any place where the summary in the draft is significantly more confident than the source paper's own framing.

14. The "ten years from now" test

If a critic reads this report in ten years and the field has moved on, what would embarrass us most? That answer is the gap to mention in the limitations.

---

Appendix template

Once you're done, the appendix you write to state should look something like this:

## Self-critique findings

The Phase 6 self-critique checked the following items. Findings noted, with resolutions:

1. **Unanchored claims** — found 3 sentences without anchors. All resolved (1 removed, 2 anchored to existing papers).
2. **Single-source claims** — 4 claims rested on single papers. 2 corroborated with additional searches; 2 explicitly downgraded ("only one study to date reports this").
3. **Venue skew** — selected papers were 38% Nature/Cell/Science; not flagged as a problem given the field.
4. **Recency** — latest paper in original selection was {{year-1}}. Ran a final search restricted to the last 18 months and added 4 papers.
5. **Untested high-citation papers** — top-cited paper [^id] had no critique in corpus. Targeted search returned a {{year}} commentary [^id2] questioning the {{aspect}}; added to corpus and incorporated into Section X.
6. **Buried tensions** — {{0 / 1 / N}} tensions surfaced in the body; {{0 / 1 / N}} were already there.
7. **Other findings** — {{...}}

Items where no issue was found are listed in the README of state.self_critique for completeness.

The appendix is short, honest, and dated. Future readers know what the report's blind spots were, and that itself is part of the contribution.

{{X}} vs {{Y}}: A Comparative Analysis

Question: {{question}} Date: {{date}} Sources consulted: {{sources}}

---

Executive summary

Verdict: {{one-sentence verdict}}

Confidence: high / medium / low — {{why}}

When the verdict flips: {{edge cases}}

---

1. What is being compared

1.1 {{X}}

Brief definition, intended use, scope. Who proposes/uses it.

1.2 {{Y}}

Same shape.

1.3 What is not being compared

Explicit exclusions to keep the scope honest.

2. Axes of comparison

Each axis: criterion, evidence from corpus, per-axis verdict.

Axis 1: {{e.g., performance / accuracy}}

Per-axis verdict: {{X or Y}} wins on this axis when {{condition}}, but {{caveat}}.

Axis 2: {{e.g., compute cost / sample efficiency}}

Same shape.

Axis 3: {{e.g., robustness / generalization}}

Same shape.

Axis 4: {{e.g., interpretability}}

Same shape.

Axis 5: {{e.g., maturity / community adoption}}

Same shape.

(Add or remove axes to fit the question. Aim for 3-6.)

3. Where they agree

Often overlooked: where do {{X}} and {{Y}} not differ meaningfully? Acknowledging this prevents the report from inventing conflicts.

4. Where they disagree (and why)

For each meaningful disagreement, name:

• The empirical observation
• The methodological reason (different datasets, baselines, hyperparameters)
• The theoretical reason (different framings)

5. Overall recommendation

If the user must pick one today, with no further information:

• Pick {{X}} when {{conditions}}
• Pick {{Y}} when {{conditions}}
• Pick neither (or wait) when {{conditions}}

6. Open questions

What would change the verdict? What study would be most informative?

Appendix A — Methodology

{{search + ranking + dedupe stats}}

Appendix B — Self-critique

{{self_critique.appendix}}

Bibliography

{{rendered from export_bibtex.py}}

Background and Significance: {{topic}}

Note: This document is the background section of a research proposal. It is narrative-first and persuasive — it builds toward "and that's why our work matters." It is NOT a neutral literature review.

---

The problem

One paragraph. State the problem in human terms, with stakes:

• Who is affected?
• What is the scale (numbers, geography, dollars)?
• What is the cost of inaction?

Anchor every empirical claim. [^id1][^id2]

What is known

Two-three paragraphs synthesizing the established science. Use the strongest evidence (meta-analyses, replicated findings, consensus statements). Anchor heavily.

Organize around what we understand:

• Foundational mechanism / principle: {{...}} [^id]
• Established findings: {{...}} [^id1][^id2]
• Where the field has converged: {{...}} [^id1][^id2][^id3]

What has been tried

One paragraph on prior approaches and why they fall short. Be respectful — these are the people who will review your grant.

• Approach 1 [^id]: works for {{case}} but fails when {{case}}
• Approach 2 [^id]: addresses {{aspect}} but {{cost / scope}} prevents {{thing}}
• Approach 3 [^id]: promising but [^id, preprint] not yet validated

What is missing — the gap

This is the pivot of the document. State the gap precisely:

Despite progress on {{X}}, no work to date has {{Y}}. This gap matters because {{Z}}.

The gap should be:

• Specific (not "more research is needed")
• Tractable (you can plausibly close it)
• Important (the reader nods)

Why our approach is positioned to close it

One paragraph hinting at the proposal's approach without launching into the methods. The reader should finish this section saying "yes, and the natural next step is..." — and then turn the page to your Aims.

Bibliography

{{rendered from export_bibtex.py — typically 30-60 references for a grant background}}

{{title}}: A Literature Review

Question: {{question}} Date: {{date}} Sources consulted: {{sources}} Papers in corpus: {{total_papers}} ({{selected_count}} selected for deep read)

---

Executive summary

• {{bullet 1}}
• {{bullet 2}}
• {{bullet 3}}
• {{bullet 4 — optional}}
• {{bullet 5 — optional}}

1. Background

Define key terms, scope, and why this matters. Two-three short paragraphs. Every non-trivial claim is anchored: claim [^id1][^id2].

2. {{theme 1 name}}

Synthesize what the corpus says about this theme.

• Sub-finding A [^id]
• Sub-finding B [^id1][^id2]

If a tension lives in this theme, surface it here:

Tension: {{topic}}. {{position A}} [^id1][^id2] vs {{position B}} [^id3].

Empirical / methodological / theoretical disagreement.

3. {{theme 2 name}}

Same shape.

4. {{theme 3 name}}

Same shape. Add more themes (max ~6) as needed.

5. Synthesis

What does the corpus collectively say? This is the section that earns the report. Don't just summarize — argue. Where do the themes connect? What is the dominant view? Where is the field actually moving?

6. Open questions and gaps

• Gap 1: {{description}} — no studies in corpus address {{specific subquestion}}
• Gap 2: {{description}} — only one paper [^id] tackles this and has not been replicated
• Gap 3: {{description}} — recency: nothing newer than {{year}}

7. Recommendations for further reading

If the user wants to go deeper, start here:

1. {{paper 1}} [^id] — {{why}} 2. {{paper 2}} [^id] — {{why}} 3. {{paper 3}} [^id] — {{why}}

Appendix A — Methodology

Search strategy:

• Sources: {{sources}}
• Clusters: {{clusters}}
• Rounds: {{round count}} per cluster
• Saturation: hit at round {{n}} (new={{pct}}%, max_new_citations={{n}})

Ranking formula:

{{state.ranking.formula}}

Weights: α={{alpha}}, β={{beta}}, γ={{gamma}}, δ={{delta}}, half-life={{half_life}}y

Selection: top {{N}} by score. Selected papers and component scores in state.papers[*].score_components.

Appendix B — Self-critique

{{self_critique.appendix}}

Bibliography

{{rendered from export_bibtex.py --format bibtex}}

{{title}}: A Scoping Review

Question: What has been studied (and how) in {{topic}}? Date: {{date}} Sources consulted: {{sources}} Papers in corpus: {{total_papers}}

---

1. Background

Why scope this field? Who is the audience? What does the user need to plan next?

2. Scope question

Refined scope statement (broader than a PICO; narrow enough to be tractable).

3. Methods

Brief — scoping reviews use broad inclusion. Note minimum exclusion criteria only.

4. Coverage map

A matrix view of the field. Rows = subtopics, columns = methods (or populations, or settings). Cell = paper count.

The empty cells are the gap.

5. Methods inventory

What methods has the field used? Brief description of each, with a representative paper.

• Method A: {{description}}. Representative: [^id]
• Method B: {{description}}. Representative: [^id]

6. Population / setting inventory

Same shape — what populations, models, or settings have been studied?

7. Subtopic narratives

One short paragraph per subtopic, with anchor pointers to the most representative work — not a full review of each.

7.1 {{Subtopic 1}}

{{paragraph with anchors}}

7.2 {{Subtopic 2}}

{{paragraph with anchors}}

8. Research gap

Synthesize the empty cells into a research gap statement. This is the deliverable.

• Gap 1: {{description}}
• Gap 2: {{description}}
• Gap 3: {{description}}

9. Recommendations for future work

• Most tractable next study: {{...}}
• Highest-impact next study: {{...}}
• Methodological recommendation: {{...}}

Appendix A — Methodology

{{search + ranking + dedupe stats}}

Appendix B — Self-critique

{{self_critique.appendix}}

Bibliography

{{rendered from export_bibtex.py}}

{{title}}: A Systematic Review

Question (PICO):

• Population: {{P}}
• Intervention: {{I}}
• Comparator: {{C}}
• Outcome: {{O}}

Date: {{date}} Protocol: This review followed PRISMA-lite guidance. Search and screening were not pre-registered.

---

1. Background and rationale

Why does this question matter? What is the prior state of evidence?

2. Methods

2.1 Search strategy

2.2 Inclusion criteria

• {{criterion 1}}
• {{criterion 2}}
• {{criterion 3}}

2.3 Exclusion criteria

• {{criterion 1}}
• {{criterion 2}}

2.4 Risk-of-bias assessment

For each included study we noted: sample size, pre-registration status, blinding, conflicts of interest, retraction status, and replication status. See extraction table.

3. PRISMA-lite flow

Records identified: {{total_hits}}
        │
        ▼
After dedupe: {{after_dedupe}}
        │
        ▼
Screened (title/abstract): {{after_screen}}
        │
        ▼
Full-text assessed: {{full_text}}
        │
        ├── Excluded ({{n}}):
        │     - {{reason 1}}: n={{n}}
        │     - {{reason 2}}: n={{n}}
        ▼
Included in synthesis: {{included}}

4. Extraction table

5. Synthesis

5.1 Primary outcome

Narrative synthesis. If outcomes are numerical and homogeneous, a meta-analytic note can go here (the skill does not run meta-analyses — flag this for the user).

5.2 Secondary outcomes

5.3 Subgroup observations

6. Quality of evidence

Use a GRADE-style summary if applicable, otherwise narrative:

• High: {{summary}}
• Moderate: {{summary}}
• Low / very low: {{summary}}

7. Conclusions

What does the body of evidence support, with what confidence?

8. Limitations of this review

• {{limitation 1}}
• {{limitation 2}}
• {{from self-critique appendix}}

Appendix A — Methodology details

(See SKILL.md Phase 0-7 description; same content as literature_review template.)

Appendix B — Self-critique

{{self_critique.appendix}}

Bibliography

{{rendered from export_bibtex.py}}

Phase 3 — Parallel Deep-Read Agent Prompt Template

This file is loaded on demand by the host LLM during Phase 3, after skim_papers.py has assigned tiers. The host dispatches one agent per `tier=deep` paper, in parallel waves of 8–10 (see "Wave sizing" below). Each agent runs the prompt below in isolation and writes evidence back to research_state.json via the shared CLI — the host's main context only sees the one-line return.

Why parallel agents (and not a single async script)

Phase 3 is reasoning-heavy, not I/O-heavy: the agent has to read the paper, decide what counts as a finding vs. background, link claims to evidence, and judge limitations. Compressing that into a deterministic Python loop loses the reasoning. Compressing 50 papers' worth of full-text into the host's main context wastes tokens. Parallel agents put each paper's reasoning in its own ~200-token context bubble; only the structured evidence record returns.

Wave sizing

• Default: waves of 8–10 agents per dispatch message.
• Why batch: more than ~10 simultaneous tool_use blocks risk host-side rate limits and back-pressure. Smaller waves also let you abort cheaply if the first 1–2 results look wrong.
• Total cost is roughly linear in deep-tier count, so trim aggressively at triage time (skim_papers.py --deep-ratio 0.3) when budget is tight.

Per-agent prompt (copy-paste; fill in the ${...} placeholders)

You are a Phase 3 deep-read agent for the scholar-deep-research skill.

## Your single paper

paper_id   : ${paper_id}        # e.g. "doi:10.1038/s41586-020-2649-2"
title      : ${title}
doi        : ${doi}             # may be null — fall back to pdf_url
pdf_url    : ${pdf_url}         # may be null
pdf_path   : ${pdf_path}        # may be null. If set AND file exists, use it
                                # directly — prefetch_pdfs.py already pulled
                                # the PDF, no network call needed.
abstract   : ${abstract}        # already in state; use as a fallback if PDF fetch fails

## Research question (Phase 0)

${question}

## What you must do

1. Get the full text. Try in this order, stopping at the first that yields >2000 chars:
   a. **If `pdf_path` is provided AND points at an existing file** (`prefetch_pdfs.py` ran):

python scripts/extract_pdf.py --input '${pdf_path}' --output /tmp/${safe_id}.txt

      No network call — fastest path. **Always prefer this when available.**
   b. `python scripts/extract_pdf.py --doi '${doi}' --output /tmp/${safe_id}.txt`
      (uses the paper-fetch skill's 5-source OA chain when installed)
   c. `python scripts/extract_pdf.py --url '${pdf_url}' --output /tmp/${safe_id}.txt`
   d. If all fail: write evidence_unavailable (see "Failure mode" below) and stop.

   If `pdf_path` is set but the file is missing (cache wiped between prefetch
   and dispatch), fall through to (b). Do **not** silently skip — that path
   is what `pdf_status='failed'` already records, and re-attempting via (b)
   gives the paper one more chance with a different transport.

2. Read the extracted text. Extract per-paper evidence covering:
   - method            : 1 sentence on the experimental/computational approach
   - findings          : 3–5 bullets, each with a section/page anchor where possible
                         (e.g. "ABE7.10 corrects 65% of dystrophin in mdx mice (Fig 3a)")
   - limitations       : what the paper itself acknowledges + what you noticed
   - relevance         : 1–2 sentences on how this moves the question forward

3. Write evidence back to state. **Prefer the JSON path** — it skips
   the multi-quote shell escape dance that bites when findings contain
   single quotes, unicode, or section headers:

echo '${json_payload}' | python scripts/research_state.py \ --state ${state_path} evidence --id '${paper_id}' --from-json -

   Where `${json_payload}` is `{"method": "...", "findings": ["...", ...],
   "limitations": "...", "relevance": "...", "depth": "full"}`. JSON's
   `depth` wins over the `--depth` flag.

   Structured mode is still supported for short single invocations:

python scripts/research_state.py --state ${state_path} evidence \ --id '${paper_id}' --depth full \ --method '${method}' \ --findings '${finding_1}' '${finding_2}' '${finding_3}' \ --limitations '${limitations}' \ --relevance '${relevance}'

   The CLI is exclusive-locked — N agents writing concurrently are serialized
   automatically; no coordination needed.

4. Return EXACTLY one JSON line to the host (no prose):

{"paper_id": "${paper_id}", "status": "ok", "evidence_chars": <int>, "method_brevity": <int>}

## Failure modes

There are two escape hatches that count as valid deep-tier coverage so a
single bad paper does not block the whole workflow. Both keep `depth='shallow'`
and prefix `evidence.method` with a magic string the gate recognises.

### Failure mode A — full text unreachable

Paywall, exhausted OA chain, scanned PDF, dead link. The PDF was *not* read.

python scripts/research_state.py --state ${state_path} evidence \ --id '${paper_id}' --depth shallow \ --method 'evidence_unavailable: ${reason_code}' \ --findings 'No full text available; abstract excerpt: ${abstract_excerpt}' \ --limitations 'Marked evidence_unavailable; do not cite as sole source.' \ --relevance 'Pending source recovery.'

Reason codes: `paywall_no_oa`, `pdf_fetch_failed`, `scanned_no_ocr`, `dead_link`.

Return:

{"paper_id": "${paper_id}", "status": "evidence_unavailable", "reason": "${reason_code}"}

### Failure mode B — PDF read but topic mismatch

The PDF *was* extracted in full and you read it, but the paper turned out to
be off-topic — Phase 2 ranking surfaced it on surface-token overlap (e.g. it
shares words like "evaluation" or "LLM" with the question but is actually
about a different problem). Record what little is usable and tag the
mismatch so the synthesis can treat the paper as a contrast/baseline rather
than a primary source. Do not silently mark it as `depth='full'` — the
relevance flag matters for the report.

python scripts/research_state.py --state ${state_path} evidence \ --id '${paper_id}' --depth shallow \ --method 'topic_mismatch: ${one_sentence_what_paper_is_actually_about}' \ --findings '${useful_observation_1}' '${useful_observation_2}' \ --limitations 'Off-topic vs Phase 0 question; cite as contrast/baseline only, not primary evidence.' \ --relevance '${one_sentence_what_the_paper_can_anchor_in_the_report}'

Return:

{"paper_id": "${paper_id}", "status": "topic_mismatch", "evidence_chars": <int>}

## Constraints

- DO NOT call MCP tools. Phase 3 must run offline-first.
- DO NOT modify any state field other than `papers[<id>].evidence` and `papers[<id>].depth`. The CLI enforces this; do not try to work around it.
- DO NOT chain into Phase 4 (citation chase) or Phase 5 (synthesis). One paper, one evidence record, one return.
- Findings MUST be specific (numbers, conditions, comparisons), not generic ("the authors found that base editing works"). Generic findings are worse than no finding — they silently inflate G4's coverage count without contributing to the report.

Host-side dispatch (single message, multiple Agent tool_use blocks)

After skim_papers.py reports counts.deep, the host LLM:

1. Loads this template once. 2. For each paper_id with tier == "deep", instantiates the prompt with the per-paper substitutions. 3. Sends a single message containing all N tool_use blocks so they fan out in parallel. (In Claude Code: one assistant message with N Agent tool calls; subagent_type general-purpose.) 4. After the wave returns, runs:

python scripts/research_state.py --state ${state_path} advance --to 4 --check-only

If deep_tier_full_evidence is still failing, dispatch a second wave for the missing ids only.

Recovery after a partial wave

If 7/10 agents returned status:"ok" and 3 returned status:"evidence_unavailable" (or timed out), do not retry the failed three immediately. First inspect state.papers[<id>].evidence:

• evidence.method starts with evidence_unavailable: → genuine OA chain failure. Either accept the shallow record or open the URL manually and feed it through extract_pdf.py --input <local.pdf>.
• No evidence field at all → agent crashed before write. Re-dispatch one agent for that id.

The state CLI is idempotent on evidence (re-writing the same id overwrites the record), so re-dispatching is safe.

CLI contract

Every script in scripts/ follows the same agent-native contract. This is the long-form reference; agents typically discover it by running scripts and reading the JSON envelope, but it's documented here for humans, new contributors, and anyone debugging an unexpected response shape.

Stdout is JSON-only

Every script prints exactly one JSON envelope to stdout and exits with a code from the stable vocabulary below. No prose is ever mixed into stdout — diagnostics and progress logs go to stderr.

Success envelope

{
  "ok": true,
  "data": { ... },
  "meta": {
    "request_id": "...",
    "latency_ms": 123,
    "cli_version": "<X.Y.Z, matches scripts/_common.py:VERSION>",
    "schema_version": 1
  }
}

Failure envelope

{
  "ok": false,
  "error": {
    "code": "snake_case_routing_key",
    "message": "human sentence",
    "retryable": true,
    "...extra context fields...": "..."
  },
  "meta": { ... }
}

code is the routing key — a stable, snake_case identifier the agent can match against. message is the human-friendly sentence. retryable tells the agent whether a re-run might succeed without intervention.

Exit codes

Schema introspection

Every script supports --schema, which prints its full parameter schema (types, defaults, choices, required flags, subcommands where applicable) as JSON and exits 0. An agent discovering an unfamiliar script should run `--schema` before `--help` — it is machine-parseable and covers everything --help does.

python scripts/search_openalex.py --schema
python scripts/research_state.py --schema   # includes every subcommand

The top-level schema response carries cli_version so an agent caching a schema can detect drift. Per-subcommand schemas carry meta.{since, tier, dangerous_if} so agents can detect new commands and graduated-safety paired-flag requirements.

Export bibliography exception

export_bibtex.py without --output writes raw BibTeX/RIS/CSL text to stdout for pipe compatibility:

python scripts/export_bibtex.py --state research_state.json --format bibtex > refs.bib

This is the one place where stdout is not the JSON envelope — it's the deliberate TTY/pipe affordance for human users and shell pipelines. Agents that need a structured response should always pass --output <path>; that path returns {"ok": true, "data": {"output": "...", "format": "bibtex", "count": N}} like every other script.

Idempotency on mutating commands

Every mutating command accepts --idempotency-key <k>. The first successful run writes {response, signature} to ${SCHOLAR_CACHE_DIR:-.scholar_cache}/<sha256>.json. A retry with the same key replays the cached response. The same key with different semantic arguments returns idempotency_key_mismatch rather than silently serving stale data. Combining --idempotency-key with --dry-run is rejected at the boundary — a dry run doesn't mutate, so caching it is meaningless.

Environment variables

Trust-boundary configuration. These are set once by the human or orchestrator — never by the agent. CLI flags override env vars where both are present.

Why env-var, not CLI flag

Per the agent-native-design principle "trust is directional," credentials and host-level config belong in higher-trust boundaries than the agent's own argv. The shell profile, a systemd unit, or the orchestrator's env injection is set by a human; the agent inherits it without being able to mint it. This is also why there is no login / auth / token subcommand — auth is delegated, not invoked.

When a script needs an env var that isn't set, it returns a structured envelope (code: missing_env_var or similar) telling the agent which variable to ask the user about — never a silent failure.

Example run

A complete walk-through showing what the 8-phase workflow looks like end-to-end on a single concrete question. Useful as a sanity check that your own run is hitting the expected checkpoints.

User: Give me a deep research report on CRISPR base editing for Duchenne muscular dystrophy.

Agent:
[Phase 0] Restating: "What is the current state and evidence for CRISPR base editing
          as a therapeutic approach for Duchenne muscular dystrophy (DMD)?"
          Archetype: literature_review (default; confirmed — broad overview requested)
          Clusters:
            - ("base editing" OR "adenine base editor" OR "cytosine base editor" OR ABE OR CBE)
            - ("Duchenne muscular dystrophy" OR DMD OR dystrophin)
            - (in vivo OR AAV OR "muscle delivery")
          → research_state.json initialized

[Phase 1] Running OpenAlex + PubMed + arXiv + Crossref across 3 clusters...
          Round 1: 187 hits, 142 unique. Round 2: 94 hits, 31 new.
          Saturation check: new=11%, max_new_citations=23 → SATURATED
          143 unique papers in state.

[Phase 2] Ranking with default weights (literature review)...
          Top 20 selected. Score components written to state.
          Triage: 10 deep / 10 skim. Prefetch fills 9/10 deep PDFs (1 paywalled,
          surfaced to user via --emit-manifest manifest).

[Phase 3] Fetching full text... 17/20 full, 3 abstract-only (flagged shallow).
          Evidence extraction complete.

[Phase 4] Citation chasing on top 8 seeds, depth 1.
          OpenAlex + S2 backends both run. Added 24 candidates after dedupe,
          6 re-scored into top 20.

[Phase 5] Themes: (a) delivery platforms, (b) editing efficiency, (c) off-target/safety,
          (d) pre-clinical outcomes, (e) clinical translation barriers.
          Tensions: AAV serotype optimality (Theme a) — 3 papers disagree.

[Phase 6] Self-critique flagged 2 single-source claims and a recency gap
          (no 2025 paper in top 20). Re-ran focused search; added 4 papers.

[Phase 7] Rendering literature_review template...
          Report: reports/crispr-base-editing-dmd_20260411.md
          Bibliography: reports/crispr-base-editing-dmd_20260411.bib (84 refs)

Things to notice

• Phase 1 took two rounds, not one. Saturation isn't a single search — 11% new on round 2 is what passed the threshold.
• Phase 2 split into deep/skim before fan-out, with a paywall manifest surfaced to the user. The agent did not waste an agent dispatch on the paywalled paper.
• Phase 4 ran both OpenAlex and S2 by default, then deduped — coverage gaps between the two are real, especially for CS-adjacent biomed.
• Phase 6 found a recency gap and looped back to search before declaring done. Self-critique is not a checkbox; it's allowed to push the workflow backwards.
• Final bibliography size (84) > top-N (20): every paper anchored in the report's appendices/methodology — including ones from the citation chase that weren't in the top-20 deep-read pool — gets a bibliography entry.

Pitfalls

Long-form catalog of failure modes. The SKILL.md has the short list; this file is for when something feels off and you want to debug what kind of off it is.

1. First-page fixation

Symptom: the report draws heavily from the first 10 OpenAlex hits, ignores everything past page 1.

Why it happens: search engines are good at returning relevant-looking results; the model anchors on whatever appears first.

Fix: require ≥3 search rounds per cluster, and explicitly check that the top-N selected papers do not all come from the first round of any single cluster.

Detection: in state, look at papers[*].first_seen_round. If 80%+ of selected papers have first_seen_round=1, you fixated.

2. Unanchored claims

Symptom: the draft contains assertions like "it is widely believed that..." or "studies have shown..." without [^id] anchors.

Why it happens: the model has prior knowledge that isn't sourced from the corpus; it leaks in.

Fix: treat any non-trivial claim without an anchor as a hallucination. Either find a paper in state to anchor it, or remove the claim. The Phase 6 self-critique pass catches these.

3. Confirmation bias

Symptom: every cited paper supports the same position; no tensions documented in Phase 5.

Why it happens: keyword choice and citation chasing both reinforce existing framings. If your seed papers all share an assumption, snowballing won't break out.

Fix: in Phase 4, explicitly search for "<top author> critique", "<key term> failed replication", "<consensus claim> challenged". Add a Round of negative-keyword search.

Detection: if state.tensions is empty AND your topic is non-trivial, you almost certainly have confirmation bias.

4. Preprint conflation

Symptom: the report cites an arXiv paper as if it were peer-reviewed.

Why it happens: preprints look like papers. The schema does mark source: arxiv, but the model can forget.

Fix: every citation in the bibliography that has only source: ["arxiv"] and no journal DOI should be marked as a preprint in the body — but anchors and tags are separate. Markdown footnote anchors must be a single token (commas inside the brackets break parsing and the render_report.py --lint check). The lint-compatible convention is: place the anchor first, then the marker as inline italic prose, e.g. [^id] *(preprint)*. Make the absence of *(preprint)* after a preprint id a self-critique check.

5. Venue monoculture

Symptom: >60% of selected papers come from one journal or one conference series.

Why it happens: OpenAlex's relevance scoring + citation prior favors high-prestige venues, which creates a feedback loop.

Fix: in Phase 6, run python scripts/research_state.py query selected | jq '.[] | .venue' | sort | uniq -c | sort -rn (or equivalent). If one venue is >60%, broaden the search clusters or add a different source.

6. Author monoculture

Symptom: one lab or one author appears as first/last author in 5+ selected papers.

Why it happens: snowballing inside an active lab's network. The lab's papers cite each other, and the citation chase amplifies that.

Fix: re-run search with NOT author:<dominant author> in one round, just to surface the alternatives. Then re-rank.

7. Recency collapse

Symptom: saturation hit before the most recent year was well-covered. Top-N has no 2025 papers.

Why it happens: older papers have more citations, so they win the rank. The recency component (γ) is intentionally light, so it can be overwhelmed.

Fix: always run a final search round restricted to the last 18 months: --year-from 2024. Re-dedupe and re-rank.

8. Stale MCP tool names

Symptom: the workflow references mcp__asta__search_papers_by_relevance but the tool is unavailable, or has been renamed.

Why it happens: MCP servers rename tools without warning. The skill's scripts are stable; MCP names are not.

Fix: before using any MCP tool, list the actually-available tools. The pipeline does not depend on MCP — scripts cover all critical paths.

9. Single-shot search

Symptom: one search round, then off to the report.

Why it happens: the model is in a hurry. Saturation didn't fire because the threshold was never tested.

Fix: the completion gate for Phase 1→2 requires state.queries to have ≥3 entries from primary sources. Enforce.

10. Skipping self-critique

Symptom: Phase 6 was logged as "looks good, no findings."

Why it happens: the temptation to ship a clean draft is highest at the moment Phase 6 catches the most.

Fix: Phase 6 must produce findings — either real ones or an explicit "I checked X, Y, Z and found no issues, with reasoning." An empty state.self_critique.findings is itself a flag.

11. Lossy abstract reading

Symptom: evidence section fields are filled, but they don't actually match the paper's findings.

Why it happens: the model read the title + abstract, projected what it thought the paper said, and wrote that into evidence.

Fix: for every paper in the top-N, deep-read the full text (not just abstract) when available, and require evidence fields to cite section numbers or page numbers.

12. Stale facts from training data

Symptom: the report includes a "well-known fact" that turns out to be from the model's training data, not the corpus.

Why it happens: the model is genuinely helpful and adds context. The context isn't from your sources.

Fix: the anchor rule (every claim has [^id]) catches this. If you can't anchor it, you can't claim it.

13. Brittle DOIs

Symptom: the bibliography has DOIs that 404, or the same paper appears twice with slightly different DOIs.

Why it happens: Crossref has multiple DOIs for some works (preprint DOI + journal DOI). OpenAlex sometimes records a placeholder.

Fix: the dedupe script normalizes by lowercase DOI. Run it after every ingest. For papers with multiple DOIs, prefer the journal DOI (the one returned by Crossref).

14. Black-box ranking

Symptom: the user asks why paper X is in the top-N and you can't say.

Why it happens: the ranking formula is buried in script defaults.

Fix: every report has a methodology appendix that prints state.ranking.formula and state.ranking.weights. Per-paper components live in state.papers[id].score_components. Show your work.

Quality Assessment

Reference for Phase 3 (Deep read) and Phase 6 (Self-critique). Load this when judging whether a paper deserves the weight it would carry in the report.

CRAAP test (with adjustments)

The CRAAP framework is from library science but maps cleanly to academic work.

CRAAP is necessary but not sufficient. A CRAAP-perfect paper can still be wrong.

Venue tiers (rough)

The rank_papers.py script uses a small built-in tier-1 list (Nature/Science/Cell/PNAS/NeurIPS/ICML/ICLR/...). Treat it as a starting prior, not a verdict.

Tier-1 signals (in approximate order of evidential weight):

1. Replication by an independent group 2. Multiple cited-by criticisms that failed to overturn the paper 3. Inclusion in a Cochrane / NICE / FDA / consensus document 4. Published in a top-tier venue 5. High citation count (with attention to who is citing)

Citation count alone is a weak signal. A wrong paper can be highly cited (often cited as a counter-example).

Preprint handling

arXiv, bioRxiv, medRxiv, ChemRxiv, SSRN — none are peer-reviewed.

Rules:

• Tag every preprint as preprint in the evidence section.
• A claim in the report should not rest only on preprints unless the entire report is about pre-registered or in-flight work.
• Check whether the preprint has since appeared in a journal — Crossref by author + title or OpenAlex by title is the fastest check.
• Preprints with >100 citations and >12 months in the wild without journal publication deserve a note: "in pre-print since X, not yet peer reviewed."

Retraction check

Before deeply citing a paper:

• Look up the DOI on Retraction Watch (https://retractionwatch.com) — it's slow but authoritative
• Check OpenAlex is_retracted: true flag if available
• Crossref returns a relation: is-corrected-by link when an erratum exists

The cost of citing a retracted paper as if it were valid is high. Spend the 30 seconds.

Conflicts and funding

Note in the evidence section when:

• Funding came from an entity with a direct stake in the result (e.g., drug trial funded by manufacturer)
• Senior author is on the advisory board of a directly-relevant company
• The paper is part of a regulatory dossier, not an independent science paper

This isn't dismissal — it's calibration. A pharma-sponsored Phase 3 trial is still evidence; it's evidence that needs context.

Sample-size sanity

Quick smell tests by field:

These are smells, not failures. A well-controlled n=8 mouse experiment can be informative; a sloppy n=80 one isn't.

When the paper disagrees with the consensus

Two questions to ask:

1. Did the author engage with the prior consensus? If they ignore it, they're either revolutionary or ignorant. Both are possible; the methodology section will tell you which. 2. Did they explain the discrepancy? If they say "X claimed Y; we find Z because of W methodological choice," that's evidence. If they don't mention X at all, they may not know.

A reasonable report can include a heterodox paper, but the surrounding text should acknowledge that it's heterodox.

Quick decision tree for Phase 3

Have full text? ──no──> mark depth=shallow, fall back to abstract
       │
       yes
       │
Sample size sane? ──no──> mark "low_power" in evidence.limitations
       │
       yes
       │
Pre-registered or replicated? ──yes──> evidence weight = high
       │
       no
       │
Single lab, novel finding, hot field? ──yes──> evidence weight = medium
       │                                       (note as "needs replication")
       no
       │
Established methodology, multiple corroborating papers in corpus?
   ──yes──> evidence weight = high
   ──no──>  evidence weight = medium

Report Templates

Five archetypes. Pick one in Phase 0 based on user intent. Each template lives in assets/templates/<archetype>.md. This file explains which to choose and why.

Decision tree

Is the user's question about a single narrow effect with many studies?
  ├── yes ─> systematic_review
  └── no ─>
     │
     Are they asking "what has been studied in this area"?
       ├── yes ─> scoping_review
       └── no ─>
          │
          Is it "X vs Y, which is better/different"?
            ├── yes ─> comparative_analysis
            └── no ─>
               │
               Is the output going into a grant or proposal?
                 ├── yes ─> grant_background
                 └── no ─> literature_review (default)

Archetype profiles

literature_review (default)

Use when: the user wants to understand what's known about a topic, with synthesis and gaps.

Structure: 1. Executive summary (3-5 bullets) 2. Background and definitions 3. Thematic sections (one per Phase 5 theme) 4. Synthesis (what we collectively know) 5. Open questions and gaps 6. Methodology appendix (search, ranking, self-critique findings) 7. Bibliography

Citation style: narrative, with [^id] anchors after each non-trivial claim.

systematic_review

Use when: the question is narrow, many studies exist, and the user needs rigor (PRISMA-lite). Common in medicine, psych, education.

Structure: 1. Background and rationale 2. Question (PICO) 3. Methods (search strategy, inclusion/exclusion, risk of bias) 4. PRISMA-lite flow diagram (descriptive, not the formal one) 5. Extraction table (one row per included study) 6. Synthesis (narrative; meta-analysis only if numerical) 7. Quality of evidence (GRADE-style) 8. Conclusions 9. Bibliography

Citation style: dense, with extraction-table cross-references.

scoping_review

Use when: the user wants to map a field — what topics, what methods, what populations have been studied. Breadth over depth.

Structure: 1. Background and rationale 2. Scope question 3. Methods (broad search; minimal exclusion) 4. Coverage map (matrix of subtopic × method) 5. Methods inventory 6. Population/setting inventory 7. Research gap (what hasn't been studied) 8. Recommendations for future work 9. Bibliography

Citation style: more enumerative than narrative. Tables dominate prose.

comparative_analysis

Use when: "X vs Y" — methods, models, frameworks, treatments, technologies.

Structure: 1. Executive summary with verdict 2. What's being compared (X and Y, scope) 3. Axes of comparison (each with subsection) 4. Per-axis verdict 5. Overall recommendation with caveats 6. When the verdict flips (edge cases) 7. Bibliography

Citation style: every comparison cell needs an anchor. Side-by-side tables are standard.

grant_background

Use when: the output is the "Background and Significance" or "Prior work" section of a research proposal.

Structure: 1. The problem (why it matters, who is affected, scale) 2. What is known (succinct synthesis with anchors) 3. What is missing (the gap — this becomes the proposal's hook) 4. Why our approach is positioned to fill it (one-paragraph segue) 5. Bibliography

Citation style: narrative-first, citation-supporting. Persuasive prose with sources, not a literature dump.

Cross-cutting requirements (all archetypes)

Every report:

• Has a methodology appendix listing the queries run, sources consulted, ranking weights, and dedupe stats. Pull from state.queries and state.ranking.
• Has a self-critique appendix copied verbatim from state.self_critique.appendix.
• Includes preprint flags inline ([^id, preprint]).
• Resolves every [^id] anchor against the bibliography. The host LLM is responsible for this check during Phase 7 — the export script emits entries for state.papers, but does not scan the report body for anchors.
• Saves as reports/<slug>_<YYYYMMDD>.md and writes the path back to state.report_path.

Search Strategies

Reference for Phase 1 (Discovery) and Phase 4 (Citation chasing). Load this when planning queries or when discovery feels unproductive.

Boolean clusters

A query is rarely a single keyword. Build 3-5 clusters of synonyms and join them with AND. Example for "CRISPR base editing for muscular dystrophy":

Cluster A (technique): "base editing" OR "adenine base editor" OR ABE OR
                       "cytosine base editor" OR CBE OR prime editing
Cluster B (disease):   "Duchenne muscular dystrophy" OR DMD OR
                       "Becker muscular dystrophy" OR dystrophinopathy OR
                       dystrophin
Cluster C (delivery):  AAV OR "adeno-associated virus" OR LNP OR
                       "lipid nanoparticle" OR "in vivo delivery"
Negative:              NOT review NOT editorial NOT comment

Each search script accepts one query at a time — run one cluster per call, then dedupe across all of them. Don't pre-AND clusters; that over-constrains and misses cross-references.

PICO (or PICO-style)

For comparative or systematic questions, decompose with PICO:

Non-biomedical translations:

• ML: Task, Method, Baseline, Metric (TMBM)
• Engineering: System, Modification, Reference, Performance
• Social science: Population, Treatment, Control, Effect

PICO matters because it forces you to articulate what counts as an answer — and that constrains your search.

Snowballing

Two flavors. Both are run by build_citation_graph.py:

• Backward snowballing. Pull the references of high-quality seed papers. The best work cites foundational papers you'd otherwise miss with keywords.
• Forward snowballing. Pull the cited-by list. The most recent critiques, replications, and extensions live here.

Special case — citation chasing for criticism. When a paper has very high citations but no critique appears in your corpus, search explicitly:

"<first author> <year>" (critique OR limitations OR reanalysis OR
                          replication OR failed OR flawed OR overstated)

If no critique exists, that's interesting. Note it. Don't assume "uncriticized" = "correct."

Saturation

Discovery ends when adding more search rounds stops adding signal, not when you get tired. The script research_state.py saturation formalizes this:

saturated when:
  (new_papers_in_round / total_in_round) < 20%   AND
  max(citations of papers first seen in round) < 100

The first condition catches "we've seen most of these before." The second catches "but there's still a high-impact paper we missed." Both must hold.

If one cluster saturates and another doesn't, run more rounds on the unsaturated cluster only.

Iteration patterns

A productive search sequence usually looks like:

Round 1: broad keywords on each cluster, limit=50
Round 2: tighter keywords (using terms learned from Round 1 abstracts)
Round 3: author search for repeat-appearing authors (search by author name)
Round 4: citation chase (build_citation_graph) on top 8 seeds
Round 5: targeted gap fills based on Phase 6 self-critique

A keyword you didn't know existed yesterday but appears in 6 abstracts today is a signal. Re-run Round 1 with that keyword added to the cluster.

Source-specific tips

• OpenAlex is the best general-purpose first stop. Free, no key, citation counts included.
• arXiv for CS/ML/physics preprints. Note: papers are unrefereed until cross-listed with a venue.
• Crossref is best for known DOIs — verify metadata, find venue, get bibliographic detail.
• PubMed for biomedical questions. Use MeSH terms when you know them: "CRISPR-Cas Systems"[MeSH].

Common failure modes

• First-page fixation. The first 10 OpenAlex results aren't "the literature." Run multiple rounds and chase citations.
• Acronym blindness. "ABE" means base editor, but also "average bit error" — disambiguate with co-occurring terms.
• English-only. Some fields have important non-English literature. Note this in the report's limitations.
• Recency collapse. Saturation can fire before the most recent year is well-covered. Always run a final query restricted to the last 18 months.

Source Selection

When you have four databases and limited rounds, where do you spend the queries? This is a one-page decision guide.

Default — always run

Add by domain

Add by question type

What about Semantic Scholar / Google Scholar / Web of Science?

• Semantic Scholar is excellent. We expose it as enrichment via the asta MCP tools when available. If those time out, you lose some semantic ranking but the rest of the pipeline keeps going. Treat it as a bonus, not a dependency.
• Google Scholar has no public API (scraping is against ToS and brittle). Skip.
• Web of Science / Scopus require institutional access. Mention in the report appendix as "not consulted" if your user explicitly cares about indexing rigor.
• Brave Search (web search MCP) is for non-academic sources — press releases, blog posts, community discussion of preprints. Only run when the question explicitly needs them.

What each source is BAD at

• OpenAlex: occasional metadata errors (wrong author, wrong year). Verify the top-N before they enter the report.
• arXiv: no citation counts, no peer review.
• Crossref: only catalogs DOI-registered works; misses arXiv preprints and many gray-lit reports. Citation counts are incoming-only (not as semantic as OpenAlex).
• PubMed: biomedical only, and some preprints/non-MEDLINE journals are missing.

Rate limits and politeness

When in doubt, pass --email. It costs nothing and unblocks higher throughput.

research_state.json schema

The state file is the single source of truth for a research run. Every script reads and writes it through research_state.py subcommands or the matching apply_* library functions in that module — no script touches the JSON directly. The shape is versioned via schema_version; loading a state file from an unsupported version returns state_schema_mismatch (exit 4) rather than silently coercing.

Run python scripts/research_state.py --schema for the machine-readable version with every subcommand expanded.

Abbreviated shape

{
  "schema_version": 1,
  "question": "...",
  "archetype": "literature_review",
  "phase": 3,
  "created_at": "...",
  "updated_at": "...",
  "queries": [
    {"source": "openalex", "query": "...", "hits": 42, "new": 30, "round": 1}
  ],
  "papers": {
    "doi:10.1038/nature12373": {
      "id": "doi:10.1038/nature12373",
      "title": "...",
      "authors": ["..."],
      "year": 2013,
      "venue": "Nature",
      "citations": 523,
      "abstract": "...",
      "source": ["openalex", "crossref"],
      "score": 0.81,
      "score_components": {
        "relevance": 0.9,
        "citations": 0.8,
        "recency": 0.6,
        "venue": 1.0
      },
      "selected": true,
      "depth": "full",
      "tier": "deep",
      "triage_score": 0.74,
      "triage_components": {
        "relevance": 0.8,
        "citation_density": 0.6,
        "recency": 0.9,
        "has_pdf": 1.0,
        "abstract_quality": 1.0
      },
      "evidence": {
        "method": "...",
        "findings": ["..."],
        "limitations": "..."
      },
      "discovered_via": "search"
    }
  },
  "triage_complete": true,
  "triage_meta": {
    "weights": {},
    "deep_ratio": 0.5,
    "skim_ratio": 0.5,
    "triaged_at": "..."
  },
  "themes": [{"name": "...", "paper_ids": ["..."]}],
  "tensions": [
    {"topic": "...", "sides": [{"position": "...", "paper_ids": ["..."]}]}
  ],
  "self_critique": {"findings": [], "resolved": [], "appendix": "..."},
  "report_path": "reports/slug_20260411.md"
}

ID normalization

Paper IDs are normalized in priority order: doi:... → openalex:W... → arxiv:... → pmid:.... dedupe_papers.py depends on this ordering, and merging logic prefers the higher-priority ID when the same paper is discovered through multiple sources.

What's settable directly

Only archetype and report_path are settable via research_state.py set --field .... phase is not settable — it advances only through research_state.py advance, which runs the gate predicates in _gates.py. Every collection field (papers, queries, themes, tensions, self_critique) is mutable only through its dedicated subcommand. Widening the SETTABLE_FIELDS whitelist is a security decision — don't do it casually.