Paper Autoraters

Paper Autoraters (App. F.3)

Faithful implementation of the four LLM-as-judge autoraters used in PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).

These are the metrics the paper uses to demonstrate that PaperOrchestra beats single-agent and AI-Scientist-v2 baselines. Use them to:

1. Score a generated paper against a ground-truth paper. 2. Compare two paper-writing pipelines side-by-side. 3. Validate your own host-agent execution of the paper-orchestra pipeline.

The four autoraters

The paper uses Gemini-3.1-Pro and GPT-5 as judges, set to temperature 0.0 (Gemini) or default 1.0 (GPT-5, which doesn't allow temperature adjustment). Use whatever your host LLM is.

Workflow

Citation F1 (compute Precision / Recall / F1 vs ground truth)

This is a two-step procedure:

Step 1: Partition the reference lists into P0 / P1

For both the ground-truth paper AND the generated paper, run the LLM with references/citation-f1-prompt.md:

inputs:
  paper_text:    full paper LaTeX or markdown
  references_str: numbered reference list (e.g., "1. Vaswani et al. (2017)
                  Attention Is All You Need. NeurIPS. 2. He et al. (2016)
                  Deep Residual Learning for Image Recognition. CVPR. ...")

output: JSON {"1": "P0", "2": "P1", "3": "P0", ...}

Save both partitions:

• bench/<paper_id>/gt_partition.json
• bench/<paper_id>/gen_partition.json

Step 2: Resolve references to entity IDs and compute F1

The paper uses Semantic Scholar paper IDs to match references between the two lists. The compute_f1.py script does this deterministically given two input lists:

python skills/paper-autoraters/scripts/compute_f1.py \
    --gt-partition gt_partition.json \
    --gt-refs gt_refs.json \
    --gen-partition gen_partition.json \
    --gen-refs gen_refs.json \
    --out f1_report.json

Where gt_refs.json and gen_refs.json are lists of {ref_num, paper_id, title} produced by your host's S2-resolution pass (the same fuzzy match + S2 verification used by literature-review-agent/scripts/).

Output JSON contains P0 / P1 / overall Precision, Recall, F1.

Literature Review Quality (single paper, 6 axes)

Load references/litreview-quality-prompt.md. Inputs:

• The full paper PDF (or LaTeX/markdown if your host lacks PDF input)
• avg_citation_count for the venue/field (used as the baseline for

citation count anchoring, e.g., 58.52 for CVPR 2025, 59.18 for ICLR 2025 per the paper)

The prompt instructs the model to evaluate ONLY the literature-review function of the paper (Introduction + Related Work / Background sections). It produces a strict JSON output with per-axis scores and justifications.

Critical anti-inflation rules baked into the prompt:

Plus penalty table:

Save the output to litreview_quality_score.json. The score JSON is the same shape used by content-refinement-agent/scripts/score_delta.py, so you can re-use the halt-rule logic to compare iterations.

SxS Overall Paper Quality (side-by-side, full paper)

Load references/sxs-paper-quality-prompt.md. Inputs:

• Two paper PDFs or LaTeX files (call them paper_1 and paper_2)

The prompt produces a JSON with paper_1_holistic_analysis, paper_2_holistic_analysis, comparison_justification, and winner ∈ {paper_1, paper_2, tie}.

To mitigate LLM positional bias (the paper notes this in §5.4), run the comparison twice with the order swapped:

call_1: paper_A → paper_1, paper_B → paper_2  → winner1
call_2: paper_B → paper_1, paper_A → paper_2  → winner2

Final outcome: a win (both calls agree on paper A), tie (one win + one tie, or two ties), or loss (both agree on paper B). The paper uses this exact ordering protocol.

SxS Literature Review Quality (side-by-side, Intro+RW only)

Load references/sxs-litreview-prompt.md. Same input/output shape as the SxS paper quality autorater, but the model is instructed to evaluate only the Introduction and Related Work / Background sections of each paper. Same positional-bias mitigation: run twice, swap order.

Resources

• references/citation-f1-prompt.md — verbatim P0/P1 partition prompt from App. F.3
• references/litreview-quality-prompt.md — verbatim 6-axis litreview rubric from App. F.3
• references/sxs-paper-quality-prompt.md — verbatim SxS paper-quality prompt from App. F.3
• references/sxs-litreview-prompt.md — verbatim SxS litreview prompt from App. F.3
• scripts/compute_f1.py — Precision / Recall / F1 from two partition JSONs

Citation F1 — P0/P1 Partition prompt

Source: arXiv:2604.05018, Appendix F.3, page 58 (verbatim).

Use this as your system message to partition a paper's reference list into P0 (must-cite) and P1 (good-to-cite) categories. Run it independently on both the ground-truth paper and the generated paper, then feed both partitions into scripts/compute_f1.py along with the resolved Semantic Scholar IDs to compute Precision / Recall / F1.

---

You are an expert academic reviewer. Read the following paper text and
analyze its references.
Your goal is to categorize the provided references into two priorities:

Priority Levels

  - P0 (Must-Cite): Core citations strictly necessary for the paper. These
    MUST include:
      - Baselines directly compared against in experiments
      - Datasets the paper utilizes or evaluates on
      - Core methods the paper is directly building upon or modifying
      - Metrics or standard numbers heavily relied upon and cited from
        another paper

  - P1 (Good-To-Have): Supplemental citations. These include:
      - Standard background references covering broad history
      - General related work that is not directly competing or built-upon
      - Minor implementations or utility tools mentioned in passing

Paper Text:
{paper_text}

References List:
{references_str}

Output Format

Please return ONLY a JSON dictionary where the keys are the exact reference
numbers (e.g., "1", "2") and the values are either "P0" or "P1". Example
output:

{{ "1": "P0", "2": "P1", "3": "P0" }}

---

Substitution

The model returns a JSON dict; the host agent saves it as gt_partition.json (for the ground-truth paper) or gen_partition.json (for the generated paper).

How F1 is computed

After both partitions exist, the host agent must resolve every numbered reference to a unique Semantic Scholar paper ID (using the same fuzzy match + S2 verification logic as literature-review-agent/scripts/). Then:

P0_GT  = set of S2 IDs from gt refs flagged P0
P0_Gen = set of S2 IDs from gen refs flagged P0
P0_Precision = |P0_GT ∩ P0_Gen| / |P0_Gen|
P0_Recall    = |P0_GT ∩ P0_Gen| / |P0_GT|
P0_F1        = 2 * P / R / (P + R)

Same for P1. Overall F1 uses the union of P0 and P1.

The deterministic computation lives in scripts/compute_f1.py.

Literature Review Quality Autorater — verbatim prompt

Source: arXiv:2604.05018, Appendix F.3, pages 59–63 (verbatim).

Use this as your system message to score the literature review quality of a single paper draft. Output is a strict JSON object with per-axis scores, penalties, and an overall score. Designed to be conservative — high scores require explicit textual evidence.

---

You are an expert, skeptical academic reviewer agent. Your task is to
rigorously evaluate the quality of the literature review in a draft
research paper PDF.

You must be conservative with scoring. High scores are rare and must be
explicitly justified with concrete evidence from the text. Assume most
drafts are not publication-ready.

Contextual Baseline

The user has provided the average citation count for accepted papers in
this specific field/venue.
Reference Average Citation Count: {avg_citation_count}
Use this number as the baseline for "typical" coverage volume.

Scope

  - Evaluate ONLY the literature-review function of:
    - Introduction
    - Related Work / Background / Literature Review (or equivalent)

  - Ignore methods, experiments, and results except to verify whether the
    literature review correctly sets up the paper's scope and claims.

Process (Follow Strictly)

  1. Identify the paper title.
  2. Locate the Introduction and Related Work sections (or closest
     equivalents).
  3. Identify:
     - The paper's stated research problem
     - Claimed contributions
     - Implied relevant subfields
  4. Estimate citation statistics from the literature review:
     - Approximate number of unique cited works
     - Citation density relative to section length
     - Breadth across relevant sub-areas
     - Volume relative to the Reference Average ({avg_citation_count}).
  5. For each scoring axis, evaluate ONLY what is explicitly written.
     - Do NOT infer author intent.
     - Do NOT reward missing but "expected" knowledge.
  6. Apply anti-inflation rules and penalties.
  7. Produce output strictly in the JSON schema defined below.
     - NO extra text before or after the JSON.
     - All fields must be filled.
     - Use null if information is genuinely unavailable.

Anti-Inflation Rules (Mandatory)

  - Default expectation: overall score between 45-70.
  - Scores > 85 require strong evidence across ALL axes.
  - Scores > 90 are extremely rare and require near-survey-level mastery.
  - If any axis < 50, overall score should rarely exceed 75.
  - If the review is mostly descriptive (paper-by-paper summaries),
    Critical Analysis must be ≤ 60.
  - If novelty is asserted without explicit comparison to close prior
    work, Positioning must be ≤ 60.
  - Sparse or inconsistent citations cap Citation Rigor at ≤ 60.
  - High citation count does NOT automatically imply high quality;
    relevance and synthesis must justify it.

Scoring Scale (Anchors - Do Not Invent New Ones)

  - 0-20  = Unacceptable
  - 21-40 = Weak
  - 41-55 = Adequate but flawed
  - 56-70 = Solid
  - 71-85 = Strong
  - 86-92 = Excellent
  - 93-100 = Exceptional (extremely rare)

Axes (0-100 Each)

Axis 1: Coverage & Completeness
  - Evaluate:
    - Breadth across major relevant threads
    - Inclusion of foundational and recent work
    - Absence of obvious omissions
    - Citation volume relative to the Reference Average
      ({avg_citation_count})
  - Citation count anchors (Relative to Reference Average of
    {avg_citation_count}):
    - Count is < 50% of Reference: Usually narrow or incomplete (cap ≤ 55
      unless field is very small).
    - Count is 50%-80% of Reference: Minimal acceptable coverage.
    - Count is 80%-120% of Reference: Solid breadth if well integrated.
    - Count is > 120% of Reference: Strong evidence of comprehensive
      coverage IF relevance is maintained.

Axis 2: Relevance & Focus
  - Evaluate:
    - Alignment of citations with the research problem
    - Minimal tangents or citation padding
    - Clear scoping and prioritization of literature

Axis 3: Critical Analysis & Synthesis
  - Evaluate:
    - Thematic grouping and comparison of approaches
    - Discussion of tradeoffs, limitations, and open gaps
    - Evidence of synthesis rather than sequential summaries
  - Hard cap: ≤ 60 if the review is mostly descriptive.

Axis 4: Positioning & Novelty Justification
  - Evaluate:
    - Clear, literature-grounded research gap
    - Explicit differentiation from closest related work
    - Motivation for why the gap matters
  - Hard cap: ≤ 60 if novelty claims are vague or unsupported.

Axis 5: Organization & Writing Quality
  - Evaluate:
    - Logical structure, flow, and signposting
    - Clarity and precision of academic language
    - Appropriate subsectioning and definitions

Axis 6: Citation Practices, Density & Scholarly Rigor
  - Evaluate:
    - Whether key claims are supported by citations
    - Credibility and consistency of sources
    - Citation density relative to section length
    - Balance between foundational and recent work
  - Hard caps:
    - Citation count significantly below Reference Average
      ({avg_citation_count}) for a broad problem: ≤ 55
    - High citation count with weak integration: ≤ 65

Penalties (Apply After Axis Scoring)

Apply zero or more penalties:
  - Overclaiming novelty without close comparison: -5 to -15
  - Missing key recent work (if detectable): -5 to -15
  - Mostly descriptive review with weak synthesis: -5 to -10
  - Weak or generic gap statements: -5 to -10
  - Citation dumping or consistency issues: -5 to -10

Optional Positive Adjustment (Rare)

You MAY apply a small positive adjustment (+3 to +7 total points) ONLY IF:
  - Citation count is substantially higher (> 150%) than the Reference
    Average ({avg_citation_count})
  - Citations are relevant and distributed across subtopics
  - Review remains synthesized and focused
  - Critical Analysis score > 60 AND Relevance score > 65
Do NOT apply this adjustment otherwise.

Overall Score
  - Use weighted judgment:
    - Coverage: 20%
    - Relevance: 15%
    - Critical Analysis: 25%
    - Positioning: 25%
    - Organization: 10%
    - Citation Rigor: 5%
  - Then apply penalties and any justified positive adjustment.
  - Sanity-check against anti-inflation rules.

Output Format (Strict JSON Only)

Return exactly the following JSON structure and nothing else:

{{ "paper_title": string | null, "citation_statistics": {{ "estimated_unique_citations": number, "citation_density_assessment": "low" | "appropriate" | "high", "breadth_across_subareas": "narrow" | "moderate" | "broad", "comparison_to_baseline": string, "notes": string }}, "axis_scores": {{ "coverage_and_completeness": {{ "score": number, "justification": string }}, "relevance_and_focus": {{ "score": number, "justification": string }}, "critical_analysis_and_synthesis": {{ "score": number, "justification": string }}, "positioning_and_novelty": {{ "score": number, "justification": string }}, "organization_and_writing": {{ "score": number, "justification": string }}, "citation_practices_and_rigor": {{ "score": number, "justification": string }} }}, "penalties": [ {{ "reason": string, "points_deducted": number }} ], "summary": {{ "strengths": [string], "weaknesses": [string], "top_improvements": [string] }}, "overall_score": number }}

Justification Constraints

  - Each justification: 2-5 sentences, evidence-based.
  - Do NOT quote more than 25 total words from the paper.
  - If evidence is missing, explicitly state: "Not evidenced in the text."

---

Substitution

SxS Literature Review Quality Autorater — verbatim prompt

Source: arXiv:2604.05018, Appendix F.3, pages 64–65 (verbatim).

Use this as your system message to perform a side-by-side preference comparison of just the literature review (Introduction + Related Work) sections of two papers. Run twice with order swapped to mitigate positional bias.

---

You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).

Your task is to perform a Side-by-Side (SxS) comparison of the literature
review sections (Introduction and Related Work) between two academic
papers.

The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.

Critical Evaluation Criteria

  1. Problem Framing And Motivation
     - Which paper introduces the research problem more clearly?
     - Does the introduction explain the importance of the problem and
       the gap in existing work?

  2. Coverage Of Prior Work
     - Which paper provides a more complete and relevant overview of
       prior research?

  3. Organization And Synthesis
     - Which paper organizes related work more effectively (e.g.,
       grouping by themes or approaches)?
     - Does it synthesize prior work rather than simply listing papers?

  4. Positioning Of The Contribution
     - Which paper more clearly explains how its approach differs from
       existing methods?

  5. Writing Quality And Readability
     - Which literature review is clearer, more concise, and easier to
       follow?

Output Format

Return a valid JSON object with the following schema:

{ "paper_1_analysis": "analysis of paper 1", "paper_2_analysis": "analysis of paper 2", "comparison_justification": "comparison reasoning", "winner": "winner of your choice" }

The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".

---

Inputs

This autorater needs only the Introduction and Related Work sections of each paper, NOT the full document. The host agent should:

1. Extract Intro + Related Work from both papers (LaTeX section commands or PDF section detection). 2. Pass them as paper_1 and paper_2. 3. Run twice, swapping order, to mitigate positional bias (see sxs-paper-quality-prompt.md for the protocol).

SxS Overall Paper Quality Autorater — verbatim prompt

Source: arXiv:2604.05018, Appendix F.3, pages 63–64 (verbatim).

Use this as your system message to perform a side-by-side preference comparison between two paper drafts. To mitigate positional bias, run the comparison TWICE with the paper order swapped, then aggregate.

---

You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).
Your task is to perform a Side-by-Side (SxS) holistic comparison of two
academic papers.
The two papers describe the same or highly similar research ideas. Your
evaluation should formulate a holistic judgment that accounts for both
scientific execution and writing quality/presentation.

The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.

Critical Evaluation Criteria

  1. Scientific Depth And Soundness
     - Which paper provides more rigorous technical justifications,
       theoretical foundations, and comprehensive experimental setups?

  2. Technical Execution
     - Within the bounds of the described idea, which paper executes the
       implementation and methodology more innovatively or effectively?

  3. Organization And Logical Flow
     - Which paper presents ideas in a clearer and more coherent order
       from Abstract through Conclusion?
     - Are sections and paragraphs structured logically with smooth
       transitions?

  4. Clarity And Precision Of Writing
     - Which paper explains its ideas more clearly and concisely?
     - Does the writing avoid unnecessary verbosity, ambiguity, or
       repetitive phrasing?

  5. Presentation Of Evidence
     - Which paper integrates figures, tables, and experimental results
       more effectively into the narrative?
     - Are visuals clearly referenced and explained in the text?

  6. Professional Academic Style
     - Which paper maintains a more polished and professional academic
       tone?
     - Does it use precise domain terminology and consistent terminology
       throughout the paper?

Output Format

Return a valid JSON object with the following schema:

{ "paper_1_holistic_analysis": "analysis of paper_1 writing, presentation, and scientific execution", "paper_2_holistic_analysis": "analysis of paper_2 writing, presentation, and scientific execution", "comparison_justification": "comparison reasoning", "winner": "winner of your choice" }

The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".

---

Positional bias mitigation protocol

The paper notes (§5.4): "human preferences correlate strongly with our GPT-5 evaluator for Overall Quality (Pearson r = 0.6458, Spearman ρ = 0.6355). Literature review correlation is lower due to inherent LLM self-bias." To get a robust SxS verdict, run the comparison twice:

Call 1: paper_A → paper_1,  paper_B → paper_2,  result1
Call 2: paper_B → paper_1,  paper_A → paper_2,  result2

normalize both results to "A wins" / "B wins" / "tie", then:

Final outcome:
  - WIN  for A:    A wins in both calls
  - LOSS for A:    B wins in both calls
  - TIE:           one win + one tie, or two ties, or A wins one + B wins one

The paper uses this exact protocol — see §5.2 "(2) SxS Paper Quality" description.