Paper Writing Bench

PaperWritingBench (§3)

Faithful implementation of the PaperWritingBench dataset construction procedure from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §3 and App. C, F.2).

The original benchmark contains 200 papers (100 CVPR 2025 + 100 ICLR 2025). For each paper, the authors reverse-engineer the (I, E) tuple by stripping narrative flow from the original PDF using the three prompts in App. F.2. You can use this skill to reverse-engineer your own benchmark cases from any paper PDF.

What this skill does

Given an existing AI research paper (PDF or markdown extract), produce:

• idea.md (Sparse variant) — high-level concept note, no math, no

experimental results

• idea.md (Dense variant) — detailed technical proposal with LaTeX

equations and variable definitions, but still no experimental results

• experimental_log.md — exhaustive raw experimental setup, numeric data,

and qualitative observations, with all narrative references stripped

These three files form a complete (I, E) input pair for the paper-orchestra pipeline. You can then run the pipeline and compare its output to the original paper using paper-autoraters.

Inputs

• A paper PDF or extracted markdown text. The paper uses MinerU

(Wang et al., 2024) for PDF→markdown extraction; you (the host agent) should use whatever PDF extractor your environment provides.

• For controlled experiments, you may also extract figures separately

(PDFFigures 2.0 in the paper).

Outputs

• bench/<paper_id>/idea_sparse.md — Sparse variant
• bench/<paper_id>/idea_dense.md — Dense variant
• bench/<paper_id>/experimental_log.md — Experimental log

Workflow

For each paper, run three independent LLM calls using the verbatim prompts below:

1. Sparse idea generation

Load references/sparse-idea-prompt.md. Pass the paper text (or markdown extract) as {paper_content}. The prompt instructs the model to:

• Stop extracting at empirical verification (no Experiments / Results / Comparisons)
• Use first-person future tense ("We propose to explore...")
• Avoid LaTeX math; describe components by function
• Anonymize authors and titles

Output: idea_sparse.md with the four sections (Problem Statement, Core Hypothesis, Proposed Methodology high-level, Expected Contribution).

2. Dense idea generation

Load references/dense-idea-prompt.md. Same input. The prompt instructs the model to:

• Preserve mathematical formulations using LaTeX
• Define every variable used in equations
• Include specific architectural choices and dimensions
• Same exclusion zone (no experiments)

Output: idea_dense.md with the four sections (Problem Statement, Core Hypothesis, Proposed Methodology detailed, Expected Contribution).

3. Experimental log generation

Load references/experimental-log-prompt.md. Same input. The prompt instructs the model to:

• Use past-tense persona ("We ran...", "The results were...")
• Strip all references to figure/table numbers
• Deconstruct tables into raw numeric data
• Log figure findings as factual observations
• Anonymize authors

Output: experimental_log.md with sections for Setup, Raw Numeric Data, and Qualitative Observations.

Critical rules from the prompts

These are excerpted from App. F.2. The host agent MUST honor them:

• No citations. None of the three outputs may contain \cite,

reference numbers, or author names from the source paper.

• No URLs. Strip all hyperlinks.
• Anonymize. Author identities, affiliations, acknowledgements all

removed.

• Self-contained. Each file must make sense without the original paper.
• No experimental leakage in idea files. The Sparse and Dense ideas

must stop where empirical verification begins. They describe what will be done, not what was done.

• No table/figure references in experimental log. No "as shown in

Table 1", "see Fig. 5". The downstream paper-orchestra pipeline will generate its own figures and tables — the log must not assume any particular ones exist.

• 100% numeric accuracy in experimental log. This becomes the ground

truth for the section-writing-agent and content-refinement-agent's hallucination check.

How the bench is used

After producing (idea_sparse.md, idea_dense.md, experimental_log.md) for a paper:

1. Pick a variant (Sparse or Dense) — the paper ablates both, with Dense producing more rigorous methodology and Sparse exercising the system's robustness on under-specified inputs. 2. Drop the chosen idea.md, plus experimental_log.md, plus a template.tex for the target conference, plus a conference_guidelines.md, into a paper-orchestra workspace. 3. Run the pipeline. 4. Compare the generated paper against the original using paper-autoraters (citation F1, lit review quality, SxS paper quality).

Resources

• references/bench-overview.md — the 200-paper bench, venue cutoffs, sizes
• references/sparse-idea-prompt.md — verbatim from App. F.2
• references/dense-idea-prompt.md — verbatim from App. F.2
• references/experimental-log-prompt.md — verbatim from App. F.2

PaperWritingBench Overview

Source: arXiv:2604.05018, §3 ("Task and Dataset"), App. C ("Dataset Details").

Composition

ICLR 2025 papers exhibit "higher visual and analytical density than CVPR 2025" (App. C.1) — roughly twice as many figures and tables, and substantially longer experimental logs (mean 2,387 vs 1,530 words).

Idea variant statistics

Sparse ideas are roughly half the length of Dense ideas. Both are kept deliberately constrained — the bench tests whether the pipeline can reconstruct a full paper from sparse human notes, not from a complete draft.

Construction pipeline

Per App. C.2:

1. Sample papers from OpenReview (ICLR) and the CVF Open Access repository (CVPR). 2. Extract with MinerU (Wang et al., 2024) for the markdown body and PDFFigures 2.0 (Clark and Divvala, 2016) for visual entities and captions. Discard incomplete or misparsed samples. 3. Reverse-engineer raw materials with Gemini-3.1-Pro (or any capable LLM in your host) using the three verbatim prompts in this skill's references/:

• sparse-idea-prompt.md
• dense-idea-prompt.md
• experimental-log-prompt.md

4. Anonymize: strip authors, titles, citations, URLs, figure/table references. The result must be self-contained. 5. Inject visual context (App. C.2 "Structured Context Injection"): when generating the experimental log, the paper passes the actual figure images alongside the text to the extraction LLM, so the model can convert visual insights into standalone factual observations (e.g., "training loss converged after 200 epochs"). This avoids the common failure mode of "naive text-only extraction corrupts tabular and mathematical data."

Why two variants

The Dense vs Sparse split tests robustness to input granularity:

• Dense: closer to a real research note from a scientist — preserves

the math, the architectural specifics, the variable definitions. Tests whether the system can faithfully translate detailed technical content into a polished manuscript.

• Sparse: closer to a brainstorming note — high-level concept only.

Tests whether the system can fill in formal mathematical structure from a rough sketch.

The paper's ablation (Table 5) shows Dense wins more on "Overall Paper Quality" (43-56% Dense win vs 18-24% Sparse win) but the gap is much smaller for "Lit Review" (32-40% Dense win vs 33-40% Sparse win) — demonstrating that the literature search step is robust to input density.

Cutoff dates

The paper aligns research cutoff with venue submission deadline:

For other venues, use one month before the stated submission deadline as the cutoff. Encode as YYYY-MM-DD. Months default to day-1 (e.g., "October" → 2024-10-01).

Using the bench with your own pipeline

To benchmark a coding agent's paper-writing skill, run:

1. Pick a paper from your held-out set (NOT in the LLM's training data
   ideally — the paper notes pre-training contamination is an inherent
   risk; use unpublished or recent work to mitigate).
2. Run paper-writing-bench to extract idea_sparse.md / idea_dense.md /
   experimental_log.md.
3. Run paper-orchestra on those inputs (with the venue's template and
   guidelines).
4. Run paper-autoraters to compare the generated paper to the original.

The autoraters skill ships the four metrics from App. F.3:

• Citation F1 (P0 must-cite, P1 good-to-cite) — checks the generated

bibliography against the ground truth.

• Literature Review Quality (6-axis 0-100) — scores Intro + Related Work.
• SxS Overall Paper Quality — full-paper side-by-side comparison.
• SxS Literature Review Quality — Intro+Related Work side-by-side.

Dense Idea Generation — verbatim prompt

Source: arXiv:2604.05018, Appendix F.2, pages 55–56 (verbatim).

Use this as your system message when reverse-engineering a Dense idea.md from an existing paper.

---

You are a Lead Research Scientist planning a new project. Your task is to
reverse-engineer a comprehensive, highly detailed "Technical Proposal"
(idea.md) based on a provided text.

You have been given the text content of a paper ([PAPER CONTENT]). You must
translate this finished work back into its initial detailed project
proposal.

Critical Data Ingestion Rules

  1. Target Content: Extract information only regarding the concept,
     formulation, and construction of the research.
     - Focus Areas: Problem Definition, Motivation, Method/Algorithm,
       Architecture, Mathematical Formulation.

  2. Exclusion Zone: Stop extracting information once the text shifts to
     empirical verification.
     - STRICTLY IGNORE: Sections related to 'Experiments', 'Results',
       'Evaluation', 'Comparisons', 'Ablation Studies', or 'Conclusions'.
     - Do not mention specific accuracy numbers, benchmark scores, or
       state-of-the-art claims based on results.

Instructions

  1. Perspective & Tone:
     - Write in First-Person Future Tense (e.g., "We will define...", "We
       formulate the loss as...", "The architecture will consist of...").
     - Act as if the experiments have not yet happened. You are proposing
       what you plan to build.

  2. Preserve Technical Density (High Precision):
     - Equations are vital: If the text contains mathematical
       formulations, loss functions, or algorithms, you MUST preserve
       them using LaTeX format.
     - Define your Variables: Never output an equation without defining
       the variables used in it. (e.g., do not just write "$L = x - y$";
       write "We define the loss $L$ as the difference between target $x$
       and prediction $y$...").
     - Do not simplify: If the paper describes a specific mechanism (e.g.,
       "multi-head attention with $d=512$"), include that specific detail
       in the plan.

  3. Structure:
     - Problem Statement: The precise gap we are filling.
     - Core Hypothesis: The specific technical novelty we are proposing.
     - Proposed Methodology: The core of the document. A rigorous
       walkthrough of the framework. Include mathematical notation, module
       specifications, and data flow.
     - Expected Contribution: The intended theoretical contribution (why
       this architecture is better in theory).

  4. Formatting:
     - Be self-contained. No citations, no URLs, no references to
       Figure/Table numbers.
     - Fully anonymize authors/titles.

Output Format

Return only the markdown memo in the following structure:

Problem Statement

(Precise definition of the technical problem.)

Core Hypothesis

(The proposed solution.)

Proposed Methodology (Detailed Technical Approach)

(A rigorous breakdown of the methodology. Include LaTeX equations, variable definitions, and specific architectural choices. Do not summarize; specify.)

Expected Contribution

(The intended theoretical or practical value of this architecture.)

[PAPER CONTENT]
{paper_content}
[END PAPER CONTENT]

Experimental Log Generation — verbatim prompt

Source: arXiv:2604.05018, Appendix F.2, page 57 (verbatim).

Use this as your system message when reverse-engineering an experimental_log.md from an existing paper.

---

You are a research scientist who has just completed all experiments. Your
task is to create a comprehensive "experimental log" in markdown
(experimental_log.md).

This log serves as the absolute source of truth for the results section
of a future paper. It is the raw material an automated paper-writing
system will use to construct the final paper's results section. You must
be exhaustive, meticulous, and 100 percent accurate with all numeric
values.

You have been given the text content of a paper ([PAPER CONTENT]). Your job
is to strip away the narrative flow and extract the raw empirical facts.

Core Instructions

  1. Crucial Rule: No References. The output log must be 100 percent
     self-contained. It must NEVER reference a figure or table number
     (e.g., "See Table 1" or "As shown in Fig. 5"). The paper-writing AI
     will not have these; it will only have this log.

  2. Adopt a Past-Tense Persona. Use "We ran...", "We observed...", "The
     results were...". This is a log of what was done.

  3. Deconstruct Tables into Raw Data. This is the most important task.
     All numeric data from tables must be moved into the
     ## 2. Raw Numeric Data (from Tables) section.
     - Do NOT recreate the table.
     - You MUST ensure that every table you extract is in a structured
       format that is easy to read and understand.
     - Be 100 percent accurate. This data is the single source of truth.

  4. Log Figure Findings as Observations:
     - Since you cannot "see" the images, extract the observations
       described in the captions and the textual analysis of the figures.
     - Convert these into factual statements (e.g., "Observation: Training
       loss converged after 200 epochs.").

  5. Anonymize:
     - Be self-contained. No citations, no URLs.
     - Fully anonymize authors/titles.

Output Format

Return only the raw markdown log.

Experimental Log

1. Experimental Setup

[Extract every technical detail required to reproduce the experiment.]

• Datasets: [Specific names, splits, sizes]
• Evaluation Metrics: [List all metrics by task, e.g., "XMR: R@K",

"SCR: Accuracy & MacroF1", "FIC: BLEU-4, METEOR, ROUGE-L, CIDEr"]

• Baselines Compared

[PAPER CONTENT]
{paper_content}
[END PAPER CONTENT]

---

Note on the structure

The paper's example output for Experimental Log uses the structure:

# Experimental Log

## 1. Experimental Setup
* **Datasets:** ...
* **Evaluation Metrics:** ...
* **Baselines Compared:** ...
* **Implementation Details:** ...

## 2. Raw Numeric Data
### Table 1: Performance comparison on Dataset X
| Method | Acc | F1 | Latency |
|--------|-----|-----|---------|
| ...    | ... | ... | ...     |

## 3. Qualitative Observations
* Observation: ...
* Observation: ...

The extract_metrics.py helper in section-writing-agent/scripts/ parses the ## 2. Raw Numeric Data section's markdown tables into structured JSON. Make sure your generated log places tables under that exact heading so the helper can find them.

Sparse Idea Generation — verbatim prompt

Source: arXiv:2604.05018, Appendix F.2, page 54 (verbatim).

Use this as your system message when reverse-engineering a Sparse idea.md from an existing paper.

---

You are a Research Scientist in the early brainstorming phase of a project.
Your task is to write a high-level "Concept Note" (idea.md) based on a
provided text.

You have been given the text content of a paper ([PAPER CONTENT]). You must
distill this into a streamlined, conceptual project proposal.

Critical Data Ingestion Rules

  1. Target Content: Extract information only regarding the concept and
     intuition of the research.
     - Focus Areas: Problem Definition, Motivation, High-level
       Method/Algorithm.

  2. Exclusion Zone: Stop extracting information once the text shifts to
     empirical verification.
     - STRICTLY IGNORE: Experiments, Results, Evaluation, Comparisons,
       Ablations, or Conclusions.

Instructions

  1. Perspective: Write in First-Person Future Tense (e.g., "We propose to
     explore...", "We aim to investigate...").

  2. Enforce Sparsity (High-Level Only):
     - Conceptual Over Mathematical: Do NOT use LaTeX. Do not provide
       formulas. Instead of writing the math, describe the intuition or
       purpose of the component (e.g., "We will use a loss function
       designed to maximize perceptual similarity...").
     - Strategic Logic: Describe the methodology at a "whiteboard" level.
       Avoid hyperparameters (like "$d=512$"). Focus on the flow of data
       and the logic of the modules.
     - Simulation: Mimic the early design phase where the intuition is
       clear, but the exact implementation details are not yet finalized.

  3. Structure:
     - Problem Statement: The gap we are filling.
     - Core Hypothesis: The specific technical novelty.
     - Proposed Methodology: A conceptual description. Focus on strategy
       and logical steps. Describe modules by their function, not their
       math.
     - Expected Contribution: The theoretical value.

  4. Formatting:
     - Be self-contained. No citations, no URLs, no references to
       Figure/Table numbers.
     - Fully anonymize authors/titles.

Output Format

Return only the markdown memo in the following structure:

Problem Statement

(Precise definition of the technical problem.)

Core Hypothesis

(The proposed solution/intuition.)

Proposed Methodology (High-Level Technical Approach)

(A conceptual description of the approach. Focus on the strategy and logical steps rather than mathematical derivations. Describe the modules and their functions.)

Expected Contribution

(The intended theoretical or practical value of this approach.)

[PAPER CONTENT]
{paper_content}
[END PAPER CONTENT]