
Paper Orchestra
- 58 installs
- 628 repo stars
- Updated July 9, 2026
- ar9av/paperorchestra
paper-orchestra is a Claude orchestrator skill that coordinates five agents to turn research materials into a submission-ready LaTeX paper and PDF.
About
The top-level orchestrator for the PaperOrchestra pipeline that turns unstructured research materials into a submission-ready LaTeX manuscript and compiled PDF. It coordinates five specialized agents (outline, plotting, literature-review, section-writing, content-refinement) and runs pre-flight validation and anti-leakage guards. A developer runs it to write a full conference paper from an idea and experimental log.
- Orchestrates the full five-agent PaperOrchestra pipeline
- Turns idea + experiment log into a submission-ready LaTeX PDF
- Coordinates outline, plotting, lit-review, section-writing, refinement
Paper Orchestra by the numbers
- 58 all-time installs (skills.sh)
- Ranked #6,517 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
paper-orchestra capabilities & compatibility
- Capabilities
- orchestration · research · documentation · planning
- Use cases
- orchestration · research · documentation
What paper-orchestra says it does
Orchestrate the full PaperOrchestra (Song et al., 2026, arXiv:2604.05018) five-agent pipeline to turn unstructured research materials
A complete submission package `P = (paper.tex, paper.pdf)` written into `workspace/final/`
npx skills add https://github.com/ar9av/paperorchestra --skill paper-orchestraAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 58 |
|---|---|
| repo stars | ★ 628 |
| Last updated | July 9, 2026 |
| Repository | ar9av/paperorchestra ↗ |
What it does
Run the end-to-end five-agent pipeline that turns an idea and experiment log into a submission-ready paper.
Who is it for?
Producing a complete conference paper from raw research inputs end to end
Skip if: One-off single-section edits or when structured inputs are not yet available (aggregate first)
When should I use this skill?
The user asks to write a paper from experiments, turn an idea and results into a paper, or run the end-to-end pipeline
What you get
Produces workspace/final/paper.tex and paper.pdf plus a full audit trail
- workspace/final/paper.tex
- workspace/final/paper.pdf
- workspace audit trail
By the numbers
- 5-agent pipeline
- 5 pipeline steps (outline, plotting, lit-review, section-writing, refinement)
- Steps 2 and 3 run in parallel
Files
paper-orchestra (Orchestrator)
Top-level driver for the PaperOrchestra pipeline. Read this document and follow the steps below. The detailed prompts and rules live in each sub-skill's SKILL.md and references/ directories — you (the host agent) will load them as you go.
Source paper: Song et al., *PaperOrchestra: A Multi-Agent Framework for
Automated AI Research Paper Writing*, arXiv:2604.05018, 2026.
<https://arxiv.org/pdf/2604.05018>
What this skill produces
A complete submission package P = (paper.tex, paper.pdf) written into workspace/final/, plus a full audit trail under workspace/ (outline, figures, refs, drafts, refinement worklog, provenance snapshot).
Inputs (the (I, E, T, G, F) tuple from the paper)
The workspace MUST contain:
| File | Symbol | Required | Description |
|---|---|---|---|
workspace/inputs/idea.md | I | yes | Idea Summary (Sparse or Dense variant — see references/io-contract.md) |
workspace/inputs/experimental_log.md | E | yes | Experimental Log: setup, raw numeric data, qualitative observations |
workspace/inputs/template.tex | T | yes | LaTeX template for the target conference (with \section{...} commands) |
workspace/inputs/conference_guidelines.md | G | yes | Formatting rules, page limit, mandatory sections |
workspace/inputs/figures/ | F | no | Optional pre-existing figures. If empty, the plotting agent generates everything. |
scripts/init_workspace.py will scaffold this layout. scripts/validate_inputs.py will check it before the pipeline runs.
Pipeline (read references/pipeline.md for the full diagram)
Step 1: Outline ──▶ outline.json (1 LLM call)
Step 2: Plotting ─┐
├──▶ figures/*.png + captions.json (~20-30 calls)
Step 3: Lit Review ─┘ (~20-30 calls)
intro_relwork.tex + refs.bib
Step 4: Section Writing ──▶ drafts/paper.tex (1 LLM call)
Step 5: Content Refine ──▶ final/paper.tex + final/paper.pdf (~5-7 calls, ~3 iters)Step 2 and Step 3 are independent and MUST run in parallel when your host supports parallel sub-agents. If not, run Step 3 first (it has the longer wall time due to Semantic Scholar rate limits) and Step 2 second.
Critical pre-instruction (read once, apply always)
Before any LLM call that writes paper content (outline, intro/related work, section writing, refinement), you MUST prepend the Anti-Leakage Prompt at references/anti-leakage-prompt.md to your system prompt. This is verbatim from Appendix D.4 of the paper and prevents pre-training-data leakage. The paper applies it uniformly across all baselines for fair comparison; we apply it for fidelity and to keep generated papers grounded in the user's inputs.
Step-by-step execution
0. Pre-flight Checks
Before running the pipeline, perform the following quality gates in order:
# 1. Scaffold the workspace
python skills/paper-orchestra/scripts/init_workspace.py --out workspace/
# user drops their inputs into workspace/inputs/
# 2. Validate required files are present and well-formed
python skills/paper-orchestra/scripts/validate_inputs.py --workspace workspace/
# 3. Check input density — idea and experimental log must meet minimum thresholds
python skills/paper-orchestra/scripts/check_idea_density.py \
--idea workspace/inputs/idea.md \
--log workspace/inputs/experimental_log.md
# 4. Cross-validate consistency between idea and experimental log
python skills/paper-orchestra/scripts/validate_consistency.py \
--idea workspace/inputs/idea.md \
--log workspace/inputs/experimental_log.mdIf validate_inputs.py or check_idea_density.py fail (exit code 1 or 2), stop and tell the user what's missing or below threshold — do not proceed until fixed.
validate_consistency.py produces warnings only (exit code 1 = WARN, non-blocking); report warnings to the user but continue.
Before failing on missing inputs, check whether aggregation can supply them:
| Inputs state | Action |
|---|---|
idea.md and experimental_log.md both present and non-empty | Continue to Step 1. |
| Either is missing/empty, and the user mentioned a directory | Load and run agent-research-aggregator with that directory as --search-roots, then re-validate. |
| Either is missing/empty, no directory mentioned | Ask the user: "Your workspace is missing idea.md / experimental_log.md. Do you have a folder with research notes or agent history I can aggregate from? If so, tell me the path — or drop the files manually into workspace/inputs/." |
If validation still fails after aggregation (e.g. template.tex or conference_guidelines.md are missing), stop and tell the user exactly which files remain outstanding.
Also probe the TeX installation (once per workspace, result cached):
python skills/paper-orchestra/scripts/check_tex_packages.py \
--out workspace/tex_profile.jsonThe Section Writing Agent reads tex_profile.json to decide which LaTeX patterns to use (e.g., Figure~\ref{} vs \cref{}, whether to include \usepackage{microtype}, etc.). This eliminates compile-time package failures that previously required iterative manual edits.
1. Outline (Step 1 — 1 LLM call)
Load skills/outline-agent/SKILL.md and follow it. Output: workspace/outline.json. Validate with python skills/outline-agent/scripts/validate_outline.py workspace/outline.json. Halt the pipeline if validation fails — every downstream agent depends on the schema.
2 ∥ 3. Plotting and Literature Review (in parallel)
Parse outline.json. Extract:
outline.plotting_plan→ drives Step 2outline.intro_related_work_plan→ drives Step 3
If your host supports parallel sub-agents (Claude Code's Agent tool with multiple concurrent calls; Cursor's parallel agents; Antigravity's worker pool), spawn two concurrent sub-tasks:
- Sub-task A: load
skills/plotting-agent/SKILL.md, execute the plotting plan,
produce workspace/figures/<figure_id>.png for every entry, plus workspace/figures/captions.json.
- Sub-task B: load
skills/literature-review-agent/SKILL.md, execute the
research strategy, produce workspace/drafts/intro_relwork.tex and workspace/refs.bib.
If your host does not support parallel sub-agents, run Sub-task B first (it has slower wall-clock due to Semantic Scholar QPS limits) then Sub-task A. The artifacts are independent, so order doesn't affect correctness.
3.5. Outline Reconciliation (after Step 3 completes, before Step 4)
Once Step 3 (Literature Review) has produced citation_pool.json and cross_verification_report.json, run the reconciliation step.
Load references/outline-reconciliation.md and follow its prompt. Output: workspace/outline_reconciled.json.
Validate and diff:
python skills/outline-agent/scripts/validate_outline.py workspace/outline_reconciled.json
python skills/paper-orchestra/scripts/diff_outlines.py \
--original workspace/outline.json \
--reconciled workspace/outline_reconciled.json \
--summary workspace/reconciliation_summary.mdIf validation fails, fall back to outline.json for Step 4 and warn the user. Show the user the reconciliation_summary.md (even if no changes — it confirms the outline matched the actual literature).
Skip conditions: citation pool empty, Step 3 failed, or Step 2 is still running and the host cannot issue another call concurrently. See references/outline-reconciliation.md for full skip conditions.
4. Section Writing (Step 4 — ONE single multimodal LLM call)
Load skills/section-writing-agent/SKILL.md and follow it. This is one single call in the paper (App. B: "Section Writing Agent (1 call)") — do not split it per section. The agent receives:
outline_reconciled.json(use this if it exists; fall back tooutline.json)idea.md,experimental_log.mdintro_relwork.tex(already-filled from Step 3 — preserve verbatim)refs.bib(the citation map)conference_guidelines.mdresearch_brief.md(if it exists — read §1–§3 for accumulated pipeline context)- The actual figure image files from
workspace/figures/(multimodal input)
Output: workspace/drafts/paper.tex (a complete LaTeX document).
Then run the deterministic gates:
python skills/section-writing-agent/scripts/orphan_cite_gate.py workspace/drafts/paper.tex workspace/refs.bib
python skills/section-writing-agent/scripts/latex_sanity.py workspace/drafts/paper.tex
python skills/paper-orchestra/scripts/anti_leakage_check.py workspace/drafts/paper.tex
python skills/paper-orchestra/scripts/claim_evidence_gate.py \
--paper workspace/drafts/paper.tex \
--log workspace/inputs/experimental_log.md \
--out workspace/claim_evidence_report.jsonclaim_evidence_gate.py is a WARN gate (exit 1 = warnings, not a hard stop). Report the count of unsupported claims to the user. The content-refinement agent will address them in Step 5.
If any gate fails, the host agent must fix the issue (re-prompting the writing step with the gate's error report) before proceeding.
5. Content Refinement (Step 5 — ~3 iterations, ~5-7 calls)
Load skills/content-refinement-agent/SKILL.md and follow it. The skill implements the loop with strict halt rules from halt-rules.md. Maintain workspace/refinement/worklog.json and snapshot each iteration into workspace/refinement/iter<N>/.
Halt conditions (any one triggers the loop to stop and accept the current best snapshot):
1. Iteration count reaches the cap (default 3, see halt-rules.md). 2. Overall score from the simulated reviewer decreases vs the previous iteration → revert to previous snapshot, halt. 3. Overall score ties but at least one sub-axis decreases while none gain compensatingly (negative net sub-axis change) → revert, halt. 4. Reviewer issues no new actionable weaknesses.
The accepted snapshot is copied to workspace/final/paper.tex.
6. Compile and finalize
cd workspace/final && latexmk -pdf paper.texThen write workspace/provenance.json capturing input file hashes, outline hash, refs hash, figure hashes, and final tex/pdf hashes (helper: scripts/snapshot.py in the orchestrator scripts dir if you want a one-shot; otherwise the host agent computes hashes inline).
Report to the user: the path to workspace/final/paper.pdf, a brief summary of which sections were drafted, citation count, refinement iterations completed, and any gates that failed mid-pipeline.
Workspace layout
See references/io-contract.md. Summary:
workspace/
├── inputs/ # user-provided
│ ├── idea.md
│ ├── experimental_log.md
│ ├── template.tex
│ ├── conference_guidelines.md
│ └── figures/ # optional pre-existing figures
├── outline.json # Step 1 output
├── figures/ # Step 2 output
│ ├── <figure_id>.png
│ └── captions.json
├── refs.bib # Step 3 output
├── drafts/ # Step 3 + Step 4 output
│ ├── intro_relwork.tex
│ └── paper.tex
├── refinement/ # Step 5 working dir
│ ├── worklog.json
│ ├── iter1/
│ ├── iter2/
│ └── iter3/
├── final/ # accepted snapshot + compiled PDF
│ ├── paper.tex
│ └── paper.pdf
└── provenance.json # input/output hashes for reproducibilityCost budget (from paper App. B)
Total: ~60–70 LLM calls per paper, ~40 minutes wall-time on the paper's setup. Budget breakdown:
| Step | Calls |
|---|---|
| Outline | 1 |
| Plotting | ~20–30 |
| Literature Review | ~20–30 |
| Section Writing | 1 |
| Content Refinement | ~5–7 |
Host integration
See references/host-integration.md for per-host invocation details (Claude Code, Cursor, Antigravity, Cline, Aider, OpenCode).
Resources
references/pipeline.md— full step-by-step flow + parallelism rules + halt rulesreferences/io-contract.md— workspace layout, input file schemasreferences/anti-leakage-prompt.md— verbatim from App. D.4, prepend to every writing callreferences/paper-summary.md— 1-page distillation of arXiv:2604.05018references/host-integration.md— per-host invocation guidereferences/outline-reconciliation.md— NEW Step 3.5 outline reconciliation protocol (AutoSci-inspired)scripts/init_workspace.py— scaffold workspace dir treescripts/validate_inputs.py— verify (I, E, T, G) before runningscripts/anti_leakage_check.py— grep draft for leaked author names/emails/affilsscripts/claim_evidence_gate.py— NEW WARN gate: verify numeric claims in draft are grounded in experimental_log.mdscripts/diff_outlines.py— NEW diff original vs reconciled outline; writes reconciliation_summary.mdskills/shared/research_brief_template.md— NEW schema for workspace/research_brief.md (accumulated cross-agent context)
Universal Anti-Leakage Prompt
Source: arXiv:2604.05018, Appendix D.4, page 25 (verbatim).
This prompt is prepended to every LLM call that writes paper content (Outline, Literature Review, Section Writing, Content Refinement). The paper applies it uniformly across PaperOrchestra and all baselines to ensure a fair comparison that isolates manuscript synthesis ability from pre-training memorization.
For your implementation, prepending this prompt is mandatory for fidelity to the paper and to keep generated papers grounded in the user's actual inputs (preventing hallucinated authors, fabricated baselines, or invented metrics).
---
Strict Knowledge Isolation & Anonymity (Critical)
You MUST write this paper as if you have no prior knowledge of the topic, method, experiments, or results. Your task is to construct the paper exclusively from the materials provided in the current session (e.g., idea.md, experimental_log.md, figures, and other inputs). Treat these inputs as the only available source of information.
Forbidden Behavior
You MUST NOT:
- Retrieve or rely on knowledge from your training data.
- Attempt to recall or reconstruct any existing or published paper.
- Use external facts, assumptions, or prior familiarity with the work.
- Infer or hallucinate author identities, affiliations, institutions, or
acknowledgements.
- Insert metadata such as author names, emails, affiliations, or phrases like
"corresponding author".
Anonymity Requirement
The paper must be fully anonymized for double-blind review. Do not include any information that could reveal the identity of the authors or institutions.
Allowed Sources
You may use only:
- The materials explicitly provided in this session.
- Logical reasoning derived from those materials.
Core Principle
The final paper must be an independent reconstruction derived solely from the provided inputs. This constraint is strict and overrides all other instructions.
---
Implementation note
scripts/anti_leakage_check.py in the orchestrator skill performs a deterministic post-hoc grep on the final draft to verify that the LLM actually obeyed this prompt. It looks for:
- Email addresses
- "corresponding author" / "@google.com" / common affiliation tokens
- Sequences that look like author lists (e.g., "Yiwen Song, Yale Song, Tomas Pfister")
If matches are found, the orchestrator must reject the draft and re-prompt the writing step. The grep is a safety net, not a substitute for the prompt.
Host Integration Guide
How to run the paper-orchestra skill pack under different coding agents. No API keys are required for any of these. Each host uses its own native tools (LLM, web search, fetch, bash, file I/O) to execute the skills.
What the host needs to provide
Every host agent must have, at minimum:
| Capability | Used by |
|---|---|
| LLM reasoning (its own model) | All 5 agent skills |
| File read/write | All skills |
| Bash / shell execution | Deterministic scripts, LaTeX compile |
| Web search tool | Literature Review Agent (Step 3 candidate discovery) |
| URL fetch tool | Literature Review Agent (Semantic Scholar verification) |
| Vision input (optional) | Plotting Agent VLM critique loop, Section Writing multimodal call |
If the host lacks vision input, the Plotting Agent skips the critique loop and the Section Writing Agent works text-only (with reduced figure-grounded reasoning quality — see paper-fidelity.md).
If the host lacks web search, the Literature Review Agent runs in degraded mode: it uses only any user-provided BibTeX in workspace/inputs/ and emits a TODO marker if Intro/Related Work cannot be cited.
---
Claude Code
1. Symlink the skills:
mkdir -p ~/.claude/skills
for s in paper-orchestra outline-agent plotting-agent literature-review-agent \
section-writing-agent content-refinement-agent paper-writing-bench \
paper-autoraters; do
ln -sf ~/paper-orchestra/skills/$s ~/.claude/skills/$s
done2. Start a session in your project root. 3. Scaffold a workspace and drop your inputs:
! python ~/paper-orchestra/skills/paper-orchestra/scripts/init_workspace.py --out workspace/4. Ask Claude:
Run the paper-orchestra pipeline on ./workspace.Claude will detect the trigger phrase, load paper-orchestra/SKILL.md, and follow it step by step. Steps 2 and 3 are run in parallel via Claude Code's Agent tool — Claude spawns two concurrent sub-agents (one with subagent_type=general-purpose reading plotting-agent/SKILL.md, one reading literature-review-agent/SKILL.md).
5. Web search uses Claude Code's WebSearch tool. Semantic Scholar fetches use WebFetch against https://api.semanticscholar.org/graph/v1/paper/... (public, no key required).
6. LaTeX compilation uses Claude Code's Bash tool: latexmk -pdf.
---
Cursor
1. Drop the skill markdown files into .cursor/rules/:
mkdir -p .cursor/rules
cp -r ~/paper-orchestra/skills/*/SKILL.md .cursor/rules/2. Use @SKILL.md references in your prompts, or just paste the trigger phrase: "Run paper-orchestra on this workspace." 3. Cursor's web search (@web) handles Step 3 candidate discovery. Its browser tool fetches api.semanticscholar.org URLs. 4. Cursor's parallel agents (Agent panel) run Steps 2 and 3 concurrently when you split them into two tasks.
---
Google Antigravity
1. Antigravity has a worker pool. Configure two workers to run the plotting and literature review steps in parallel. 2. Each worker reads the corresponding SKILL.md from ~/paper-orchestra/skills/. 3. Antigravity's built-in search and fetch tools handle Step 3 networking; no API key configuration needed. 4. Final compile via Antigravity's shell runner.
---
Cline (VS Code extension)
1. Add the skills to Cline's custom instructions or as project-level docs. 2. Cline reads SKILL.md files via the file tool. 3. Web search and fetch are provided by Cline's built-in browser/search tools. 4. Steps 2 and 3 run sequentially (Cline does not yet support parallel sub-agents) — start with Step 3 since it's slower.
---
Aider
1. Aider is text-only and lacks built-in web search or vision. Use the degraded mode: provide a pre-built BibTeX in workspace/inputs/refs.bib and run only the Outline → Section Writing → Content Refinement steps. 2. Aider's /run command executes the deterministic scripts. 3. The Plotting Agent's VLM critique loop is skipped; figures are rendered once with no refinement.
---
OpenCode / generic CLI agent
1. Any agent that can read files, run shell commands, and call an LLM can execute this pipeline. The skills are just markdown. 2. The minimum integration is: cat skills/paper-orchestra/SKILL.md into the system prompt, then let the agent take over.
---
Known issues and workarounds
LaTeX compilation: figures appearing between references
The most common final-layout defect is figures floating into or after the References section. This happens when the Experiments section has many floats and LaTeX cannot place them all before \bibliography{}.
Fix (already encoded in `section-writing-agent/SKILL.md`): the Section Writing Agent must emit \clearpage immediately before \bibliographystyle{...}. If you encounter this in a compiled PDF, edit workspace/final/paper.tex and add \clearpage before the bibliography line, then recompile.
LaTeX compilation: missing style files on basic TeX installations
The NeurIPS 2024 template requires packages (nicefrac, microtype, cleveref, and the T1/Courier fonts via \usepackage[T1]{fontenc}) that are not included in minimal TeX Live distributions (e.g., texlive-2025-basic on macOS). If compilation fails with File '*.sty' not found:
1. Install the full TeX Live scheme: tlmgr install scheme-full (requires admin), or install individual packages: tlmgr install cleveref nicefrac microtype. 2. Alternatively, comment out the missing packages and replace:
\cref{fig:X}→Figure~\ref{fig:X}\cref{tab:Y}→Table~\ref{tab:Y}\texttt{...}uses courier; if courier is missing, use\textit{...}- Remove
\usepackage[T1]{fontenc}and\usepackage{url}if Courier
TFM files are absent (symptom: pcrr7t not loadable).
citation_pool.json key format mismatch
The Literature Review Agent may generate citation keys in its own format (e.g., lewis2020rag) while bibtex_format.py generates canonical keys from author + year + first title word (e.g., lewis2020retrievalaugmented). Running bibtex_format.py after the Lit Review Agent will update the keys in the pool, but the already-written intro_relwork.tex and refs.bib will still use the old keys, causing citation coverage gate failures.
Fix: after running bibtex_format.py --pool ... --out ..., run a key substitution pass over intro_relwork.tex:
import json
with open('workspace/citation_pool.json') as f:
pool = json.load(f)
key_map = {p['key']: p['bibtex_key'] for p in pool['papers']}
with open('workspace/drafts/intro_relwork.tex') as f:
content = f.read()
for old, new in key_map.items():
content = content.replace('{' + old + '}', '{' + new + '}')
with open('workspace/drafts/intro_relwork.tex', 'w') as f:
f.write(content)Then rebuild refs.bib from the pool's bibtex_key fields before running the Section Writing Agent.
authors format in citation_pool.json
If the Literature Review Agent writes authors as plain strings ("authors": ["Alice Smith", "Bob Jones"]), but bibtex_format.py expects dicts ({"name": "Alice Smith"}), bibtex_format.py will raise AttributeError. Fix by running a normalisation pass before bibtex_format.py:
for p in pool['papers']:
if p.get('authors') and isinstance(p['authors'][0], str):
p['authors'] = [{'name': a} for a in p['authors']]---
Verifying your host integration
Run the smoke test on the bundled example:
cd ~/paper-orchestra
python skills/paper-orchestra/scripts/init_workspace.py --out /tmp/po-test/
cp examples/minimal/inputs/* /tmp/po-test/inputs/
python skills/paper-orchestra/scripts/validate_inputs.py --workspace /tmp/po-test/Then ask your host agent to run the pipeline on /tmp/po-test/. A successful run should produce /tmp/po-test/final/paper.pdf after ~15-40 minutes (host-dependent).
I/O Contract
Reference for the orchestrator. Defines the workspace layout and the schema of every input and intermediate artifact.
Workspace layout
workspace/
├── inputs/ # User-provided
│ ├── idea.md # I — Sparse or Dense idea (markdown)
│ ├── experimental_log.md # E — setup, raw numeric data, observations
│ ├── template.tex # T — conference LaTeX template
│ ├── conference_guidelines.md # G — formatting rules, page limit, sections
│ └── figures/ # F — optional pre-existing figures (PNG/PDF)
├── outline.json # Step 1 output
├── figures/ # Step 2 output (generated)
│ ├── <figure_id>.png
│ ├── ...
│ └── captions.json # {figure_id: caption_text, ...}
├── refs.bib # Step 3 output
├── citation_pool.json # Step 3 output (verified S2 metadata)
├── drafts/
│ ├── intro_relwork.tex # Step 3 output
│ └── paper.tex # Step 4 output (then mutated by Step 5)
├── refinement/ # Step 5 working dir
│ ├── worklog.json
│ ├── iter1/{paper.tex,paper.pdf,review.json,score.json}
│ ├── iter2/...
│ └── iter3/...
├── final/ # Accepted snapshot + compiled PDF
│ ├── paper.tex
│ └── paper.pdf
└── provenance.json # Input/output hashes for reproducibilityInput file schemas
idea.md — Idea Summary (I)
Markdown. Two valid variants (the paper distinguishes them in App. C.3).
Sparse variant (high-level concept note, no math):
## Problem Statement
(Precise definition of the technical problem.)
## Core Hypothesis
(The proposed solution / intuition.)
## Proposed Methodology (High-Level Technical Approach)
(Conceptual description; describe modules by function, not their math.)
## Expected Contribution
(Intended theoretical or practical value.)Dense variant (preserves math, equations, variable definitions):
## Problem Statement
## Core Hypothesis
## Proposed Methodology (Detailed Technical Approach)
- includes LaTeX equations, variable definitions, architectural choices
## Expected ContributionThe Outline Agent automatically handles both. Dense produces more rigorous methodology sections; Sparse exercises the system's robustness (per the paper's ablation, App. E).
experimental_log.md — Experimental Log (E)
Markdown. Strict structure required:
# Experimental Log
## 1. Experimental Setup
* **Datasets:** ...
* **Evaluation Metrics:** ...
* **Baselines Compared:** ...
* **Implementation Details:** ...
## 2. Raw Numeric Data
(Tables in markdown format. Section-Writing Agent extracts these into LaTeX
booktabs tables. Use plain markdown table syntax — | col | col | — with no
references to "Table N" or "Figure N".)
## 3. Qualitative Observations
* (factual statements like "training loss converged after 200 epochs",
"method X failed on test case Y", etc.)Critical rules (from App. F.2 Experimental Log Generation prompt):
- No references to figure or table numbers ("See Table 1", "as shown in Fig. 5")
- Past-tense persona ("We ran...", "The results were...")
- Self-contained: no citations, no URLs, no author names
- Numeric values must be 100% accurate — they become the ground truth for
the Refinement Agent's hallucination check
template.tex — LaTeX Template (T)
A conference LaTeX template (CVPR, ICLR, NeurIPS, ICML, etc.) with empty \section{...} placeholders. The Section Writing Agent fills the empty sections in-place; the preamble (\documentclass, \usepackage, etc.) is preserved verbatim.
conference_guidelines.md — Conference Guidelines (G)
Markdown describing:
- Page limit (in pages, integer)
- Mandatory sections (e.g., "must have an Abstract, Introduction, Methods,
Experiments, Conclusion")
- Formatting requirements (single-column vs two-column, font, margins)
- Submission deadline date (used to derive
cutoff_datefor the literature
review and section writing agents)
inputs/figures/ — Pre-existing Figures (F)
Optional. PNG or PDF files. If present, the Plotting Agent will reuse them where the outline plan permits and only generate the missing ones. If empty, the Plotting Agent generates everything from scratch (the paper calls this mode PlotOff; the GT-figure mode is PlotOn).
Intermediate artifact schemas
outline.json
See skills/outline-agent/references/outline-schema.md and the JSON Schema at skills/outline-agent/references/outline_schema.json.
figures/captions.json
{
"fig_framework_overview": "Plain-text caption here, no markdown, no 'Figure N:' prefix.",
"fig_main_results": "..."
}The Section Writing Agent splices these into \caption{...} commands.
citation_pool.json
Internal record produced by the Literature Review Agent of every verified citation, with full Semantic Scholar metadata. Schema:
{
"papers": [
{
"paperId": "abc123...", // S2 unique ID
"bibtex_key": "vaswani2017attention",
"title": "Attention Is All You Need",
"authors": [{"name": "A. Vaswani"}, ...],
"year": 2017,
"venue": "NeurIPS",
"abstract": "...",
"externalIds": {"DOI": "...", "ArXiv": "1706.03762"},
"verified_via": "semantic_scholar",
"match_score": 100, // Levenshtein ratio
"discovered_for": ["intro", "related_work"]
}
],
"cutoff_date": "2024-11-01",
"min_cite_paper_count": 27 // 90% of len(papers), rounded down
}refinement/worklog.json
{
"iterations": [
{
"iter": 1,
"timestamp": "2026-04-09T12:34:56Z",
"review": { "strengths": [...], "weaknesses": [...], "questions": [...] },
"score": { "overall": 67, "axes": {...} },
"actions_taken": ["Rewrote Section 3.2 for clarity", ...],
"decision": "accept"
},
...
],
"halted_because": "iteration_cap_reached" | "overall_decreased" | "tie_with_negative_subaxis_delta" | "no_new_weaknesses",
"best_iter": 2
}provenance.json
{
"created_at": "2026-04-09T12:34:56Z",
"inputs": {
"idea.md": {"sha256": "...", "bytes": 1234},
"experimental_log.md": {"sha256": "...", "bytes": 5678},
"template.tex": {"sha256": "...", "bytes": 9012},
"conference_guidelines.md": {"sha256": "...", "bytes": 345}
},
"outline.json": {"sha256": "..."},
"refs.bib": {"sha256": "...", "n_entries": 59},
"figures": {
"fig_framework_overview.png": {"sha256": "..."},
...
},
"final": {
"paper.tex": {"sha256": "..."},
"paper.pdf": {"sha256": "..."}
},
"skill_versions": {
"paper-orchestra": "0.1.0"
}
}This is an out-of-paper improvement for reproducibility. Optional but recommended.
Outline Reconciliation (Step 3.5)
Inspired by: AutoSci (arXiv:2605.31468) — hypothesis refinement based on early findings.
Why this step exists
The outline is generated in Step 1 from idea.md and experimental_log.md alone — before any literature is actually found. Step 3 (Literature Review) may discover that:
- A Related Work cluster the outline assumed was well-populated returns few or no verified papers
- The baseline comparisons framed in
section_planrely on papers that failed S2/Crossref
verification
- The actual landscape of citations warrants splitting, merging, or re-ordering clusters
- The Introduction strategy should emphasise different foundational context based on what
was found
Without reconciliation the section-writing agent receives an outline whose section_plan may contradict the actual citation_pool.json. The reconciliation step closes this gap with one lightweight LLM call after Step 3 completes, before Step 4 begins.
What gets updated
Only section_plan is updated — never plotting_plan (Step 2 may still be running) and never intro_related_work_plan (Step 3 has already acted on it).
Permitted changes:
- Reframe section claims to match what the verified citation pool actually supports
- Adjust or remove baseline comparisons that cannot be backed by found papers
- Reorder or rename subsections to reflect the actual literature structure
- Add a note to a section that a certain comparison is "concurrent work" rather than
a beaten baseline (per the TIMELINE RULE)
Forbidden changes:
- Adding or removing top-level sections defined in
template.tex - Changing any field in
plotting_plan - Changing any field in
intro_related_work_plan - Inventing new citation hints for papers not in
citation_pool.json
Input / Output
| File | Role |
|---|---|
workspace/outline.json | Read — original outline (Step 1 output) |
workspace/citation_pool.json | Read — what was actually verified (Step 3 output) |
workspace/cross_verification_report.json | Read — confidence tiers (Step 3 output) |
workspace/drafts/intro_relwork.tex | Read — what claims Step 3 actually wrote |
workspace/outline_reconciled.json | Write — updated outline for Step 4 |
If citation_pool.json is absent (Step 3 is still running or failed), skip this step and use outline.json directly in Step 4.
Prompt for the reconciliation call
Load skills/outline-agent/references/prompt.md as the base system prompt, then prepend:
RECONCILIATION MODE — do not regenerate the full outline.
You are reviewing the original outline.json against what was actually found
during the literature review. Your only task is to update the `section_plan`
array so that every claim in each section is backed by the verified citation
pool.
RULES:
1. Output a complete JSON object with the same three top-level keys as outline.json.
Copy `plotting_plan` and `intro_related_work_plan` VERBATIM — do not change a
single character.
2. Update only `section_plan` entries where the original content_bullets reference
papers or comparisons that (a) failed verification, or (b) are post-cutoff
(concurrent work only).
3. For each changed bullet: keep the scientific claim, adjust only the framing
(e.g. "outperforms [X]" → "compares favourably with the concurrent work [X]",
or drop the comparison if [X] is entirely absent from citation_pool.json).
4. Do not add subsections or remove top-level sections.
5. Output raw JSON only — no prose, no markdown fences.Input context to provide:
outline.json(full)citation_pool.json(paper titles + bibtex_keys only — no full abstracts needed)cross_verification_report.json(only thelowandconflicttier entries)
Validation
After the call, validate the output:
python skills/outline-agent/scripts/validate_outline.py workspace/outline_reconciled.jsonIf validation fails, fall back to outline.json for Step 4 and log a warning.
Diff the two files to produce a human-readable reconciliation summary:
python skills/paper-orchestra/scripts/diff_outlines.py \
--original workspace/outline.json \
--reconciled workspace/outline_reconciled.json \
--summary workspace/reconciliation_summary.mdReport the summary to the user before starting Step 4.
When to skip
- Step 3 failed or produced an empty citation pool → skip, use
outline.json cross_verification_report.jsonshows zerolow/conflictentries AND
all section_plan citation hints are already present in citation_pool.json → skip (nothing to reconcile), use outline.json
- Host does not support a 4th parallel call at this stage → skip, use
outline.json
Skipping is safe. Reconciliation is a quality improvement, not a hard gate.
PaperOrchestra — 1-page distillation
Source: Song, Y., Song, Y., Pfister, T., Yoon, J. PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing. arXiv:2604.05018, 2026. <https://arxiv.org/pdf/2604.05018>
Problem
Existing autonomous research-paper writers are tightly coupled to specific experimental loops. They cannot transform unstructured human-provided materials (an idea, an experimental log, a template) into a submission-ready manuscript. Survey-only frameworks (AutoSurvey2, LiRA) lack full-paper synthesis. Single-agent baselines hallucinate citations and produce shallow literature reviews.
Approach
A five-agent pipeline that maps W(I, E, T, G, F) → P = (paper.tex, paper.pdf), where:
I= Idea Summary (Sparse or Dense markdown)E= Experimental Log (raw data, ablations, observations)T= LaTeX templateG= Conference guidelinesF= Optional pre-existing figures
The five agents:
1. Outline Agent (1 LLM call) — synthesizes the inputs into a JSON outline with three sub-plans: a visualization plan, a literature search strategy (macro for Intro + micro for Related Work), and a section-level writing plan with mandatory citation hints for every dataset / metric / baseline. 2. Plotting Agent (~20–30 calls) — executes the visualization plan, using PaperBanana-style few-shot retrieval and a VLM critique loop to iteratively refine generated figures. 3. Literature Review Agent (~20–30 calls) — runs parallel candidate discovery (10 web search workers), then sequential Semantic Scholar verification (1 QPS, Levenshtein > 70 fuzzy title match), dedupes by Semantic Scholar paper ID, and drafts Introduction + Related Work using ≥90% of the verified pool. 4. Section Writing Agent (1 single multimodal call) — drafts Abstract, Methodology, Experiments, Conclusion; extracts numeric values from E into LaTeX booktabs tables; integrates figures from Step 2. 5. Content Refinement Agent (~5–7 calls) — runs an AgentReview-style simulated peer review loop; accepts revisions only if overall score improves OR ties with non-negative net sub-axis change; reverts on decrease or negative tie-break; halts at iteration cap (~3).
Steps 2 and 3 run in parallel.
Engineering details that matter for fidelity
- Universal Anti-Leakage Prompt (App. D.4) prepended to every writing call.
- Citation cutoff: research_cutoff aligned to venue submission deadline
(Nov 2024 for CVPR 2025, Oct 2024 for ICLR 2025). Months default to first day of the month for strict-predates comparison.
- Citation verification: Levenshtein title ratio > 70, year alignment
bonus, must have abstract, must strictly predate cutoff, dedup by S2 paperId.
- Citation density rule: ≥90% of the gathered pool MUST be cited in
Intro + Related Work.
- Refinement safety: agent must ignore reviewer requests for new
experiments and must never explicitly write "limitation" — both prevent reward hacking against the simulated evaluator.
- One single Section Writing call — not chunked per section.
Results (paper §5)
- Side-by-side LLM evaluation: PaperOrchestra wins 88–99% on literature
review quality and 39–86% on overall quality vs single-agent and AI-Scientist-v2 baselines.
- Simulated acceptance rate (ScholarPeer): 84% (CVPR), 81% (ICLR) — close to
human GT rates of 86%/94%.
- Citation P1 Recall: 12.59–13.75% absolute over the strongest baseline.
- Refinement loop alone: +19% (CVPR) and +22% (ICLR) acceptance rate gains.
What this repo implements
This repo is a host-agent-pluggable skill pack that lets any coding agent (Claude Code, Cursor, Antigravity, Cline, Aider, OpenCode) execute the PaperOrchestra pipeline. There are no API keys, no LLM SDKs, no embedded network clients. Each skill is a markdown instruction document plus deterministic local helper scripts. The host agent does all LLM reasoning, web search, and Semantic Scholar fetching using its own native tools.
See pipeline.md, io-contract.md, and host-integration.md for details.
PaperOrchestra Pipeline
Reference for the orchestrator. Source: arXiv:2604.05018, §4 and Fig. 1.
The 5 steps
┌──────────────────────────────────────────────────┐
│ Inputs: I (idea.md), E (experimental_log.md), │
│ T (template.tex), G (guidelines.md), │
│ F (figures/, optional) │
└────────────────────┬─────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Step 1: Outline Agent (1 LLM call) │
│ outline.json = { │
│ plotting_plan, │
│ intro_related_work_plan, │
│ section_plan │
│ } │
└────────────────────┬─────────────────────────────┘
│
┌────────────┴────────────┐
│ PARALLEL │
▼ ▼
┌──────────────────────────┐ ┌──────────────────────────────┐
│ Step 2: Plotting Agent │ │ Step 3: Lit Review Agent │
│ ~20-30 calls │ │ ~20-30 calls │
│ - few-shot retrieval │ │ - parallel candidate │
│ - visual planning │ │ discovery (10 workers) │
│ - render │ │ - sequential S2 verify │
│ - VLM critique loop │ │ (1 QPS, Levenshtein > 70) │
│ - caption │ │ - dedup by S2 paperId │
│ → figures/*.png │ │ - draft Intro + Related Work │
│ → figures/captions.json │ │ (≥90% citation integration)│
│ │ │ → drafts/intro_relwork.tex │
│ │ │ → refs.bib │
└──────────────┬───────────┘ └──────────────┬──────────────┘
└────────────┬─────────────────┘
▼
┌──────────────────────────────────────────────────┐
│ Step 4: Section Writing Agent (1 LLM call) │
│ ONE single multimodal call: │
│ - extracts numeric values from E │
│ - builds booktabs LaTeX tables │
│ - drafts Abstract / Methodology / Experiments / │
│ Conclusion (preserves Intro + Related Work) │
│ - splices figures from F │
│ → drafts/paper.tex │
└────────────────────┬─────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Step 5: Content Refinement Agent (~5-7 calls) │
│ Loop (≤ ~3 iterations): │
│ 1. simulated reviewer scores current draft │
│ 2. apply revision targeting weaknesses │
│ 3. re-score │
│ 4. accept or revert per halt rules │
│ → final/paper.tex + final/paper.pdf │
└──────────────────────────────────────────────────┘Parallelism rules
- Steps 2 and 3 are independent and have no shared state. They MUST run in
parallel when the host supports it. If not, run Step 3 first because its wall-time floor is set by Semantic Scholar's 1 QPS verification limit.
- Within Step 2: figure rendering jobs are independent and can be parallelized
per figure_id. The VLM critique loop within a single figure is sequential (render → critique → redraw).
- Within Step 3: candidate discovery is parallel (10 concurrent web search
workers in the paper, but the host can use whatever its tool supports). Candidate verification via Semantic Scholar must be sequential at ≤1 QPS to respect the public API rate limit. See s2-api-cookbook.md.
Step 5 halt rules (verbatim from content-refinement-agent/references/halt-rules.md)
The refinement loop accepts a revision iff:
ACCEPT if overall_new > overall_prev
ACCEPT if overall_new == overall_prev AND net_subaxis_delta >= 0
REVERT otherwiseThe loop halts when any of the following becomes true:
1. Iteration cap reached (default = 3). 2. Overall score decreased vs previous iteration → revert to previous snapshot. 3. Overall score tied but net sub-axis change is negative → revert, halt. 4. No new actionable weaknesses in the simulated reviewer output.
The "best" snapshot at termination is copied to workspace/final/.
Anti-Leakage prompt
Every LLM call that writes paper content (Outline, Lit Review draft, Section Writing, Refinement) MUST be prefixed with the verbatim Anti-Leakage Prompt at anti-leakage-prompt.md. The Plotting Agent and Caption Generation calls are exempt because they don't generate paper text.
Cost budget per agent (from App. B)
| Agent | LLM calls |
|---|---|
| Outline Agent | 1 |
| Plotting Agent | ~20–30 |
| Hybrid Literature Agent | ~20–30 |
| Section Writing Agent | 1 |
| Content Refinement Agent | ~5–7 |
| Total | ~60–70 |
Mean wall-clock per paper: ~39.6 minutes (paper Table 7) on parallel infrastructure. Sequential execution will be 2-3x slower.
#!/usr/bin/env python3
"""
anti_leakage_check.py — Deterministic post-hoc check that the LLM obeyed the
Universal Anti-Leakage Prompt (arXiv:2604.05018, App. D.4).
Greps a generated LaTeX paper for forbidden artifacts:
- Email addresses (a-z0-9.+_-@a-z0-9.-)
- "corresponding author" / "Correspondence to" phrases
- Common affiliation tokens (Google, OpenAI, Microsoft, DeepMind, FAIR, etc.)
- Author-list patterns (e.g., "FirstName LastName, FirstName LastName, ...")
in the title-block region (before \\begin{document} or first \\section)
Exit codes:
0 no leaks found
1 leaks found (the orchestrator should reject the draft and re-prompt)
This is a safety net, NOT a substitute for the prompt itself.
Usage:
python anti_leakage_check.py path/to/paper.tex
"""
import re
import sys
EMAIL_RE = re.compile(r"[a-zA-Z0-9_.+\-]+@[a-zA-Z0-9\-]+\.[a-zA-Z0-9\-.]+")
CORR_AUTHOR_RE = re.compile(r"corresponding\s+author|correspondence\s+to", re.I)
# Affiliation tokens that appear in real papers' title blocks. These are
# heuristics; false positives are possible if a paper legitimately discusses
# one of these as a topic. The check fires only on the title-block region.
AFFILIATION_TOKENS = [
"Google", "OpenAI", "Microsoft", "DeepMind", "Meta AI", "FAIR",
"Anthropic", "Stanford", "MIT", "Berkeley", "CMU",
"Tsinghua", "Peking University", "Cornell",
]
# An author-list pattern: "Name1, Name2, Name3 and Name4" or with superscripts.
AUTHOR_LIST_RE = re.compile(
r"(?:[A-Z][a-z]+\s+[A-Z][a-z]+(?:\s*\^\d+|\s*\$\^\{\d+\}\$)?(?:\s*,\s*|\s+and\s+)){2,}"
r"[A-Z][a-z]+\s+[A-Z][a-z]+"
)
def get_titleblock(text: str) -> str:
"""Return the region between \\begin{document} (or top) and the first \\section."""
start_m = re.search(r"\\begin\{document\}", text)
start = start_m.end() if start_m else 0
end_m = re.search(r"\\section\b", text[start:])
end = start + end_m.start() if end_m else min(start + 4000, len(text))
return text[start:end]
def check(path: str) -> int:
text = open(path).read()
leaks: list[str] = []
for m in EMAIL_RE.finditer(text):
leaks.append(f" email: {m.group()}")
for m in CORR_AUTHOR_RE.finditer(text):
ctx = text[max(0, m.start()-30):m.end()+30].replace("\n", " ")
leaks.append(f" corresponding-author phrase: ...{ctx}...")
titleblock = get_titleblock(text)
for tok in AFFILIATION_TOKENS:
if re.search(rf"\b{re.escape(tok)}\b", titleblock):
leaks.append(f" affiliation token in title block: {tok}")
m = AUTHOR_LIST_RE.search(titleblock)
if m:
leaks.append(f" author-list-like pattern in title block: {m.group()[:80]}...")
if leaks:
print(f"FAIL: anti-leakage violations in {path}:", file=sys.stderr)
for line in leaks:
print(line, file=sys.stderr)
print("\nThe Anti-Leakage Prompt (App. D.4) forbids these. Re-prompt the",
file=sys.stderr)
print("writing step with explicit instructions to remove these artifacts.",
file=sys.stderr)
return 1
print(f"OK: no anti-leakage violations in {path}")
return 0
def main() -> int:
if len(sys.argv) != 2:
print(__doc__, file=sys.stderr)
return 2
return check(sys.argv[1])
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""
build_pdf.py — Fallback academic PDF builder (ReportLab).
Converts a Markdown research paper into a two-column IEEE-style PDF with:
- Full-width title + abstract on page 1 (proper IEEE layout, no blank first page)
- Two-column body starting below the abstract on the same page
- Clickable in-text citations [N] that jump to the reference entry
- Academic colour scheme (slate / blue)
- Header/footer with short title and page numbers
Usage:
python build_pdf.py --input paper.md --output paper.pdf
python build_pdf.py --input paper.md --output paper.pdf --title-short "Short Title"
python build_pdf.py --input paper.md --output paper.pdf --byline "NeurIPS 2026 Submission"
python build_pdf.py --input paper.md --output paper.pdf --author "Alice Smith" --byline "ICML 2026"
python build_pdf.py --input paper.md --output paper.pdf --demo-diagram
Requires: reportlab >= 4.0
"""
import argparse
import re
import sys
from pathlib import Path
from reportlab.lib import colors
from reportlab.lib.enums import TA_CENTER, TA_JUSTIFY, TA_LEFT
from reportlab.lib.pagesizes import letter
from reportlab.lib.styles import ParagraphStyle
from reportlab.lib.units import inch
from reportlab.platypus import (
BaseDocTemplate,
Frame,
HRFlowable,
NextPageTemplate,
PageTemplate,
Paragraph,
Spacer,
Table,
TableStyle,
FrameBreak,
)
from reportlab.platypus.flowables import AnchorFlowable
# ── Page geometry ─────────────────────────────────────────────────────────────
PAGE_W, PAGE_H = letter # 8.5 × 11 inches
M_TOP = 0.75 * inch
M_BOTTOM = 0.85 * inch
M_LEFT = 0.75 * inch
M_RIGHT = 0.75 * inch
COL_GAP = 0.25 * inch
BODY_W = PAGE_W - M_LEFT - M_RIGHT
COL_W = (BODY_W - COL_GAP) / 2
BODY_H = PAGE_H - M_TOP - M_BOTTOM
# How much vertical space the title+abstract strip gets on page 1.
# 4.5 in comfortably holds ~200-word abstract + title block.
TITLE_STRIP_H = 4.5 * inch
COL_H_P1 = BODY_H - TITLE_STRIP_H # column height on page 1
# ── Brand colours ─────────────────────────────────────────────────────────────
C_DARK = colors.HexColor("#0f172a")
C_BLUE = colors.HexColor("#1d4ed8")
C_NAVY = colors.HexColor("#1e3a5f")
C_RULE = colors.HexColor("#cbd5e1")
C_MUTED = colors.HexColor("#64748b")
C_BG_CODE = colors.HexColor("#f8fafc")
C_BG_BOX = colors.HexColor("#eff6ff")
C_BOX_BDR = colors.HexColor("#bfdbfe")
C_LINK = colors.HexColor("#1d4ed8")
# ── Styles ────────────────────────────────────────────────────────────────────
def make_styles() -> dict:
return {
"title": ParagraphStyle(
"PTitle", fontName="Times-Bold", fontSize=17, leading=21,
textColor=C_DARK, alignment=TA_CENTER, spaceAfter=3,
),
"subtitle": ParagraphStyle(
"PSub", fontName="Times-Italic", fontSize=9.5, leading=12,
textColor=C_MUTED, alignment=TA_CENTER, spaceAfter=4,
),
"abstract_h": ParagraphStyle(
"PAbsH", fontName="Times-Bold", fontSize=9, leading=11,
textColor=C_NAVY, alignment=TA_CENTER, spaceBefore=2, spaceAfter=3,
),
"abstract": ParagraphStyle(
"PAbs", fontName="Times-Roman", fontSize=8.5, leading=11.5,
textColor=C_DARK, alignment=TA_JUSTIFY,
leftIndent=20, rightIndent=20, spaceAfter=4,
),
"section": ParagraphStyle(
"PSec", fontName="Times-Bold", fontSize=10, leading=13,
textColor=C_NAVY, spaceBefore=8, spaceAfter=3, keepWithNext=1,
),
"subsection": ParagraphStyle(
"PSSec", fontName="Times-BoldItalic", fontSize=9, leading=12,
textColor=C_DARK, spaceBefore=5, spaceAfter=2, keepWithNext=1,
),
"body": ParagraphStyle(
"PBody", fontName="Times-Roman", fontSize=9, leading=12,
textColor=C_DARK, alignment=TA_JUSTIFY,
spaceAfter=4, firstLineIndent=12,
),
"body_ni": ParagraphStyle(
"PBodyNI", fontName="Times-Roman", fontSize=9, leading=12,
textColor=C_DARK, alignment=TA_JUSTIFY, spaceAfter=4,
),
"bullet": ParagraphStyle(
"PBul", fontName="Times-Roman", fontSize=9, leading=12,
textColor=C_DARK, alignment=TA_JUSTIFY,
leftIndent=12, firstLineIndent=-8, spaceAfter=2,
),
"code": ParagraphStyle(
"PCode", fontName="Courier", fontSize=7.5, leading=10,
textColor=C_DARK, leftIndent=8, spaceAfter=4,
backColor=C_BG_CODE,
),
"ref": ParagraphStyle(
"PRef", fontName="Times-Roman", fontSize=8, leading=11,
textColor=C_DARK, leftIndent=14, firstLineIndent=-14, spaceAfter=3,
),
"diagram_label": ParagraphStyle(
"PDiag", fontName="Helvetica-Bold", fontSize=8, leading=10,
textColor=C_NAVY, alignment=TA_CENTER,
),
"diagram_body": ParagraphStyle(
"PDiagB", fontName="Helvetica", fontSize=7.5, leading=10,
textColor=C_DARK, alignment=TA_CENTER,
),
"diagram_caption": ParagraphStyle(
"PDiagC", fontName="Times-Italic", fontSize=8, leading=10,
textColor=C_MUTED, alignment=TA_CENTER, spaceBefore=4, spaceAfter=6,
),
}
# ── Header / footer ───────────────────────────────────────────────────────────
def on_page(canvas, doc, short_title: str, byline: str = ""):
canvas.saveState()
y_h = PAGE_H - M_TOP + 8
canvas.setStrokeColor(C_RULE)
canvas.setLineWidth(0.5)
canvas.line(M_LEFT, y_h, PAGE_W - M_RIGHT, y_h)
canvas.setFont("Times-Italic", 7.5)
canvas.setFillColor(C_MUTED)
canvas.drawString(M_LEFT, y_h + 2, short_title)
if byline:
canvas.drawRightString(PAGE_W - M_RIGHT, y_h + 2, byline)
y_f = M_BOTTOM - 12
canvas.line(M_LEFT, y_f, PAGE_W - M_RIGHT, y_f)
canvas.drawCentredString(PAGE_W / 2, y_f - 10, str(doc.page))
canvas.restoreState()
# ── Page templates ────────────────────────────────────────────────────────────
def build_doc(out_path: str, short_title: str,
title_meta: str = "", author_meta: str = "",
byline: str = "") -> BaseDocTemplate:
doc = BaseDocTemplate(
out_path, pagesize=letter,
leftMargin=M_LEFT, rightMargin=M_RIGHT,
topMargin=M_TOP, bottomMargin=M_BOTTOM,
title=title_meta or short_title,
author=author_meta,
)
cb = lambda c, d: on_page(c, d, short_title, byline)
# Page 1: full-width title+abstract strip at top, two columns below
title_y = PAGE_H - M_TOP - TITLE_STRIP_H # y-coordinate (from bottom) of bottom of title frame
f_title = Frame(M_LEFT, title_y, BODY_W, TITLE_STRIP_H, id="title",
showBoundary=0)
f_p1_l = Frame(M_LEFT, M_BOTTOM, COL_W, COL_H_P1, id="p1_left",
showBoundary=0)
f_p1_r = Frame(M_LEFT + COL_W + COL_GAP, M_BOTTOM, COL_W, COL_H_P1, id="p1_right",
showBoundary=0)
# Pages 2+: pure two-column
f_l = Frame(M_LEFT, M_BOTTOM, COL_W, BODY_H, id="left", showBoundary=0)
f_r = Frame(M_LEFT + COL_W + COL_GAP, M_BOTTOM, COL_W, BODY_H, id="right", showBoundary=0)
pt_first = PageTemplate(id="First", frames=[f_title, f_p1_l, f_p1_r], onPage=cb)
pt_twocol = PageTemplate(id="TwoCol", frames=[f_l, f_r], onPage=cb)
doc.addPageTemplates([pt_first, pt_twocol])
return doc
# ── Architecture diagram ──────────────────────────────────────────────────────
def make_arch_diagram(styles: dict, col_width: float) -> list:
S = styles
W = col_width - 8
def cell(title, lines, bg):
rows = [[Paragraph(title, S["diagram_label"])]] + \
[[Paragraph(l, S["diagram_body"])] for l in lines]
t = Table(rows, colWidths=[W - 12])
t.setStyle(TableStyle([
("BACKGROUND", (0, 0), (0, 0), bg),
("BACKGROUND", (0, 1), (-1, -1), colors.white),
("BOX", (0, 0), (-1, -1), 0.75, C_BOX_BDR),
("INNERGRID", (0, 0), (-1, -1), 0.3, C_RULE),
("TOPPADDING", (0, 0), (-1, -1), 2),
("BOTTOMPADDING",(0, 0), (-1, -1), 2),
("LEFTPADDING", (0, 0), (-1, -1), 4),
("RIGHTPADDING", (0, 0), (-1, -1), 4),
]))
return t
warden = cell("Warden: Runtime Monitor", [
"PreToolUse / PostToolUse hooks",
"YAML policy engine • 25+ rules",
"Shell • File • Network • MCP • Prompt",
"Observe mode | Enforce mode",
"SQLite + JSONL audit trail",
], colors.HexColor("#dbeafe"))
cloak = cell("Cloak: Secret Prevention", [
"@@SECRET:name@@ placeholder convention",
"PreToolUse: decloak at execution time",
"sed-wrap scrubs stdout before recording",
"UserPromptSubmit: intercepts pasted keys",
"PostToolUse: scrubs MCP responses",
], colors.HexColor("#fef9c3"))
sweep = cell("Sweep: Secret Cleanup", [
"Gitleaks-powered residue scan",
"Claude • Cursor • Windsurf caches",
"AES-256-CBC vault • Redact / Restore",
"Run on-demand or via Stop hook",
], colors.HexColor("#dcfce7"))
agent_box = Table(
[[Paragraph("IDE / Agent (Claude Code • Cursor • Windsurf)", S["diagram_body"])]],
colWidths=[W - 12],
)
agent_box.setStyle(TableStyle([
("BACKGROUND", (0, 0), (-1, -1), colors.HexColor("#f1f5f9")),
("BOX", (0, 0), (-1, -1), 1.0, C_NAVY),
("ALIGN", (0, 0), (-1, -1), "CENTER"),
("TOPPADDING", (0, 0), (-1, -1), 3),
("BOTTOMPADDING",(0, 0), (-1, -1), 3),
]))
arrow = Paragraph("hooks", S["diagram_body"])
outer = Table(
[[agent_box], [arrow], [warden], [arrow], [cloak], [arrow], [sweep]],
colWidths=[W],
)
outer.setStyle(TableStyle([
("ALIGN", (0, 0), (-1, -1), "CENTER"),
("VALIGN", (0, 0), (-1, -1), "MIDDLE"),
("TOPPADDING", (0, 0), (-1, -1), 2),
("BOTTOMPADDING", (0, 0), (-1, -1), 2),
("LEFTPADDING", (0, 0), (-1, -1), 6),
("RIGHTPADDING", (0, 0), (-1, -1), 6),
]))
return [
Spacer(1, 6),
outer,
Paragraph(
"Figure 1. Prismor Immunity Agent: three-layer defense architecture.",
S["diagram_caption"],
),
Spacer(1, 4),
]
# ── Inline markdown → ReportLab XML ──────────────────────────────────────────
# Citation pattern: [1], [1, 2], [1–3] etc.
_CIT_RE = re.compile(r'\[(\d[\d,\s\-–]*\d|\d)\]')
def md_inline(text: str, linkify_cites: bool = False) -> str:
"""
Convert inline markdown to ReportLab paragraph XML.
Backtick spans processed first to prevent italic regex firing on
underscores inside code text (LD_PRELOAD, NODE_OPTIONS, etc.).
If linkify_cites=True, [N] patterns become internal hyperlinks.
"""
parts = re.split(r'`([^`]+)`', text)
out = []
for idx, part in enumerate(parts):
if idx % 2 == 1:
safe = (part.replace("&", "&")
.replace("<", "<")
.replace(">", ">"))
out.append(f'<font name="Courier" size="8">{safe}</font>')
else:
safe = (part.replace("&", "&")
.replace("<", "<")
.replace(">", ">"))
safe = re.sub(r'\[([^\]]+)\]\([^\)]+\)', r'\1', safe)
safe = re.sub(r'\*\*\*(.+?)\*\*\*', r'<b><i>\1</i></b>', safe)
safe = re.sub(r'\*\*(.+?)\*\*', r'<b>\1</b>', safe)
safe = re.sub(r'(?<!\*)\*(?!\*)(.+?)(?<!\*)\*(?!\*)', r'<i>\1</i>', safe)
if linkify_cites:
# Link [N] → #ref_N and [N, M] → links for each number
def _link_cit(m):
raw = m.group(0) # e.g. "[1, 3]"
nums = re.findall(r'\d+', raw)
if len(nums) == 1:
n = nums[0]
return (f'<link href="#ref_{n}" '
f'color="{C_LINK.hexval()}">{raw}</link>')
# Multiple numbers: link each individually
inner = ", ".join(
f'<link href="#ref_{n}" color="{C_LINK.hexval()}">{n}</link>'
for n in nums
)
return f"[{inner}]"
safe = _CIT_RE.sub(_link_cit, safe)
out.append(safe)
return "".join(out)
# ── Abstract extraction ───────────────────────────────────────────────────────
def extract_abstract(md_text: str) -> str:
"""Pull the abstract block out of the markdown. Returns plain text."""
lines = md_text.splitlines()
in_abs = False
buf = []
for line in lines:
if line.startswith("## ") and line[3:].strip().lower() == "abstract":
in_abs = True
continue
if in_abs:
if line.startswith("## "): # next section
break
if line.strip() in ("---", "***", "___"):
break
buf.append(line.strip())
return " ".join(b for b in buf if b)
# ── ASCII table parser ────────────────────────────────────────────────────────
def _is_ascii_table(lines: list[str]) -> bool:
"""Return True if the block looks like a +---+ style ASCII table."""
content = [l for l in lines if l.strip()]
if len(content) < 2:
return False
sep_lines = [l for l in content if re.match(r'^\s*\+[-+]+\+\s*$', l)]
pipe_lines = [l for l in content if l.strip().startswith("|")]
return len(sep_lines) >= 1 and len(pipe_lines) >= 1
def _col_widths_from_sep(sep_line: str, available_w: float) -> list[float]:
"""Derive proportional column widths from a +---+---+ separator line."""
# Each segment between + markers gives a relative width in chars
segs = re.split(r'\+', sep_line.strip())
segs = [s for s in segs if s] # remove empties at start/end
if not segs:
return []
chars = [len(s) for s in segs]
total = sum(chars)
return [available_w * c / total for c in chars]
def _parse_rows(lines: list[str]) -> list[list[str]]:
"""
Parse an ASCII table into logical rows.
Separator lines (+---+) delimit rows. Content lines (| ... |) between
separators belong to the same logical row; multi-line cells are joined
with a space.
"""
logical_rows: list[list[str]] = []
current_lines: list[list[str]] = [] # list of column-text lists
def flush():
if not current_lines:
return
# Merge multi-line cells column-by-column
num_cols = max(len(r) for r in current_lines)
merged = []
for c in range(num_cols):
parts = []
for row_line in current_lines:
if c < len(row_line):
v = row_line[c].strip()
if v:
parts.append(v)
merged.append(" ".join(parts))
logical_rows.append(merged)
current_lines.clear()
for line in lines:
stripped = line.strip()
if not stripped:
continue
if re.match(r'^\+[-+]+\+$', stripped):
flush()
elif stripped.startswith("|"):
# Split on | and strip whitespace from each cell
cells = [c.strip() for c in stripped.split("|")]
# Remove empty strings that come from leading/trailing |
cells = cells[1:-1] if len(cells) > 2 else cells
current_lines.append(cells)
flush()
return logical_rows
def try_parse_ascii_table(code_lines: list[str],
styles: dict,
col_width: float) -> list | None:
"""
If code_lines is an ASCII table, parse and return ReportLab flowables.
Returns None if the block is not a table.
"""
if not _is_ascii_table(code_lines):
return None
# Derive column widths from the first separator line
sep_line = next((l for l in code_lines if re.match(r'^\s*\+[-+]+\+', l)), None)
col_widths = _col_widths_from_sep(sep_line, col_width - 4) if sep_line else []
rows = _parse_rows(code_lines)
if not rows:
return None
S = styles
# Build Paragraph cells
tbl_style = ParagraphStyle(
"TblCell", fontName="Helvetica", fontSize=7, leading=9,
textColor=C_DARK, alignment=TA_LEFT,
)
hdr_style = ParagraphStyle(
"TblHdr", fontName="Helvetica-Bold", fontSize=7, leading=9,
textColor=colors.white, alignment=TA_LEFT,
)
para_rows = []
for r_idx, row in enumerate(rows):
style = hdr_style if r_idx == 0 else tbl_style
para_rows.append([Paragraph(cell, style) for cell in row])
# Normalise column count
max_cols = max(len(r) for r in para_rows)
for row in para_rows:
while len(row) < max_cols:
row.append(Paragraph("", tbl_style))
# Column widths: use proportional if available, else equal
if col_widths and len(col_widths) == max_cols:
cw = col_widths
else:
cw = [col_width / max_cols] * max_cols
t = Table(para_rows, colWidths=cw, repeatRows=1)
t.setStyle(TableStyle([
# Header row
("BACKGROUND", (0, 0), (-1, 0), C_NAVY),
("TEXTCOLOR", (0, 0), (-1, 0), colors.white),
# Data rows alternating
("BACKGROUND", (0, 1), (-1, -1), colors.white),
("ROWBACKGROUNDS",(0, 1), (-1, -1),
[colors.white, colors.HexColor("#f1f5f9")]),
# Grid
("BOX", (0, 0), (-1, -1), 0.5, C_RULE),
("INNERGRID", (0, 0), (-1, -1), 0.3, C_RULE),
# Padding
("TOPPADDING", (0, 0), (-1, -1), 3),
("BOTTOMPADDING", (0, 0), (-1, -1), 3),
("LEFTPADDING", (0, 0), (-1, -1), 4),
("RIGHTPADDING", (0, 0), (-1, -1), 4),
("VALIGN", (0, 0), (-1, -1), "TOP"),
]))
return [Spacer(1, 4), t, Spacer(1, 6)]
# ── Markdown → flowable list ──────────────────────────────────────────────────
def parse_markdown(md_text: str, styles: dict, col_width: float,
inject_diagram: bool = True,
skip_abstract: bool = False) -> list:
"""
Parse markdown into ReportLab flowables.
skip_abstract=True: skip the ## Abstract section (already in title block).
"""
lines = md_text.splitlines()
S = styles
flowables = []
in_abstract = False
in_refs = False
diagram_injected = False
i = 0
while i < len(lines):
line = lines[i]
# H1 title — skip (rendered in title block)
if line.startswith("# ") and not line.startswith("## "):
i += 1
continue
# Explicit diagram marker (only honored when inject_diagram is set)
if line.strip() == "<!-- ARCH_DIAGRAM -->":
if inject_diagram:
flowables.extend(make_arch_diagram(S, col_width))
diagram_injected = True
i += 1
continue
# Horizontal rule
if line.strip() in ("---", "***", "___"):
if not in_abstract:
flowables.append(HRFlowable(
width="100%", thickness=0.5, color=C_RULE,
spaceAfter=4, spaceBefore=4,
))
i += 1
continue
# H2 section
if line.startswith("## "):
txt = line[3:].strip()
if txt.lower() == "abstract":
if skip_abstract:
in_abstract = True
i += 1
continue
else:
in_abstract = False
flowables.append(Paragraph("Abstract", S["abstract_h"]))
i += 1
continue
# Leaving abstract
in_abstract = False
if txt.lower().startswith("reference"):
in_refs = True
else:
in_refs = False
# Auto-inject diagram before §4
if inject_diagram and not diagram_injected:
m = re.match(r'^(\d+)\.', txt)
if m and int(m.group(1)) >= 4:
flowables.extend(make_arch_diagram(S, col_width))
diagram_injected = True
flowables.append(Paragraph(md_inline(txt), S["section"]))
i += 1
continue
# H3 subsection
if line.startswith("### "):
in_abstract = False
txt = line[4:].strip()
flowables.append(Paragraph(md_inline(txt), S["subsection"]))
i += 1
continue
# H4
if line.startswith("#### "):
txt = line[5:].strip()
flowables.append(Paragraph(f"<b>{md_inline(txt)}</b>", S["body_ni"]))
i += 1
continue
# Skip abstract lines when skip_abstract is True
if in_abstract and skip_abstract:
if line.startswith("## "):
in_abstract = False
continue # re-process this line
i += 1
continue
# Blank line
if not line.strip():
i += 1
continue
# Bullet
if line.strip().startswith(("- ", "* ", "• ")):
in_abstract = False
txt = re.sub(r'^[\s\-\*•]+', '', line)
flowables.append(Paragraph("• " + md_inline(txt, linkify_cites=True),
S["bullet"]))
i += 1
continue
# Numbered list
if re.match(r'^\d+\.\s', line.strip()):
in_abstract = False
txt = re.sub(r'^\d+\.\s+', '', line.strip())
flowables.append(Paragraph("• " + md_inline(txt, linkify_cites=True),
S["bullet"]))
i += 1
continue
# Fenced code block
if line.strip().startswith("```"):
in_abstract = False
i += 1
code_lines = []
while i < len(lines) and not lines[i].strip().startswith("```"):
code_lines.append(lines[i])
i += 1
i += 1
if code_lines:
# If the block looks like an ASCII table, render it properly
tbl = try_parse_ascii_table(code_lines, S, col_width)
if tbl is not None:
flowables.extend(tbl)
else:
xml = "<br/>".join(
f'<font name="Courier" size="7">'
+ l.replace("&", "&").replace("<", "<").replace(">", ">")
+ '</font>'
for l in code_lines[:25]
)
flowables.append(Paragraph(xml, S["code"]))
continue
# Reference line: [N] Author...
if in_refs and re.match(r'^\[(\d+)\]', line.strip()):
m = re.match(r'^\[(\d+)\]', line.strip())
ref_n = m.group(1)
# Insert an anchor so citations can jump here
flowables.append(AnchorFlowable(f"ref_{ref_n}"))
flowables.append(Paragraph(md_inline(line.strip()), S["ref"]))
i += 1
continue
# Normal body text
if line.strip() and not in_abstract:
flowables.append(Paragraph(md_inline(line.strip(), linkify_cites=True),
S["body"]))
i += 1
return flowables
# ── Title block (fills the full-width title frame on page 1) ─────────────────
def build_title_block(styles: dict, title_text: str, abstract_text: str,
byline: str = "") -> list:
S = styles
flows = [
Spacer(1, 0.08 * inch),
Paragraph(md_inline(title_text), S["title"]),
Spacer(1, 0.04 * inch),
HRFlowable(width="55%", thickness=1.5, color=C_BLUE,
hAlign="CENTER", spaceBefore=2, spaceAfter=3),
]
if byline:
flows.append(Paragraph(byline, S["subtitle"]))
if abstract_text:
flows += [
Paragraph("Abstract", S["abstract_h"]),
Paragraph(md_inline(abstract_text), S["abstract"]),
]
flows += [Spacer(1, 0.06 * inch)]
return flows
# ── Main ──────────────────────────────────────────────────────────────────────
def main() -> int:
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--input", required=True, help="Input Markdown file")
p.add_argument("--output", required=True, help="Output PDF path")
p.add_argument("--title", default=None,
help="Full paper title (read from first # heading if omitted)")
p.add_argument("--title-short", dest="title_short", default=None,
help="Short title for header")
p.add_argument("--byline", default="",
help="Optional byline / venue strap shown under the title and "
"in the right side of the page header "
"(e.g. 'NeurIPS 2026 Submission' or 'Acme Corp · April 2026')")
p.add_argument("--author", default="",
help="Author string written into PDF metadata")
p.add_argument("--demo-diagram", dest="demo_diagram", action="store_true",
help="Insert the legacy Prismor Immunity Agent demo "
"architecture diagram (auto-injected before §4 and on "
"<!-- ARCH_DIAGRAM --> markers). Off by default.")
args = p.parse_args()
md_path = Path(args.input)
if not md_path.exists():
print(f"ERROR: {md_path} not found", file=sys.stderr)
return 1
md_text = md_path.read_text(encoding="utf-8")
# Extract title
title = args.title
if not title:
m = re.search(r'^# (.+)', md_text, re.MULTILINE)
title = m.group(1).strip() if m else md_path.stem
short_title = args.title_short or (title[:60] + "..." if len(title) > 60 else title)
# Extract abstract separately so it goes into the full-width title frame
abstract_text = extract_abstract(md_text)
styles = make_styles()
doc = build_doc(args.output, short_title,
title_meta=title, author_meta=args.author,
byline=args.byline)
# Title block (full-width): title + abstract
title_block = build_title_block(styles, title, abstract_text,
byline=args.byline)
# Body flowables: skip the abstract (already in title block)
# FrameBreak pushes remaining title-frame space into the two-column area
body_flows = (
[FrameBreak()] # end title frame, start left column on page 1
+ [NextPageTemplate("TwoCol")]
+ parse_markdown(
md_text, styles, col_width=COL_W,
inject_diagram=args.demo_diagram,
skip_abstract=True,
)
)
story = title_block + body_flows
doc.build(story)
size_kb = Path(args.output).stat().st_size // 1024
print(f"PDF written: {args.output} ({size_kb} KB)")
return 0
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""
check_idea_density.py — Pre-flight quality assessment for PaperOrchestra inputs.
Exit codes:
0 — PASS: all checks passed
1 — FAIL: one or more checks failed
2 — File not found
"""
import argparse
import re
import sys
def tokenize(text: str) -> set:
"""Return a set of lowercase word tokens from text."""
return set(re.findall(r"[a-z]+", text.lower()))
def jaccard(set_a: set, set_b: set) -> float:
"""Compute Jaccard similarity between two token sets."""
if not set_a and not set_b:
return 0.0
intersection = len(set_a & set_b)
union = len(set_a | set_b)
return intersection / union if union > 0 else 0.0
def count_words(text: str) -> int:
"""Count whitespace-delimited words."""
return len(text.split())
def count_markdown_tables(text: str) -> int:
"""Count markdown table separator lines (|---|)."""
return len(re.findall(r"^\|[-| :]+\|", text, re.MULTILINE))
def count_numeric_values(text: str) -> int:
r"""Count numeric values matching -?\d+\.?\d*."""
return len(re.findall(r"-?\d+\.?\d*", text))
def has_hypothesis_signal(text: str) -> tuple:
"""Return (found: bool, matched_word: str)."""
signals = ["hypothesis", "propose", "method", "approach", "contribution"]
lower = text.lower()
for signal in signals:
if signal in lower:
return True, signal
return False, ""
def main():
parser = argparse.ArgumentParser(
description="Pre-flight density check for idea.md and experimental_log.md"
)
parser.add_argument("--idea", required=True, help="Path to idea.md")
parser.add_argument("--log", required=True, help="Path to experimental_log.md")
parser.add_argument(
"--min-idea-words",
type=int,
default=50,
help="Minimum word count for idea.md (default: 50)",
)
parser.add_argument(
"--min-tables",
type=int,
default=1,
help="Minimum number of markdown tables in experimental_log.md (default: 1)",
)
parser.add_argument(
"--min-numeric-values",
type=int,
default=5,
help="Minimum numeric values in experimental_log.md (default: 5)",
)
args = parser.parse_args()
# Load files
try:
with open(args.idea, "r", encoding="utf-8") as f:
idea_text = f.read()
except FileNotFoundError:
print(f"[ERROR] File not found: {args.idea}", file=sys.stderr)
sys.exit(2)
try:
with open(args.log, "r", encoding="utf-8") as f:
log_text = f.read()
except FileNotFoundError:
print(f"[ERROR] File not found: {args.log}", file=sys.stderr)
sys.exit(2)
failures = []
# Check 1: idea.md word count
idea_words = count_words(idea_text)
if idea_words >= args.min_idea_words:
print(f"[PASS] idea.md: {idea_words} words (min: {args.min_idea_words})")
else:
print(
f"[FAIL] idea.md: {idea_words} words (min: {args.min_idea_words})"
)
failures.append(f"idea.md word count {idea_words} < {args.min_idea_words}")
# Check 2: idea.md hypothesis signal
found, matched = has_hypothesis_signal(idea_text)
if found:
print(f'[PASS] idea.md: contains hypothesis signal ("{matched}")')
else:
print(
"[FAIL] idea.md: no hypothesis signal found "
"(need one of: hypothesis, propose, method, approach, contribution)"
)
failures.append("idea.md missing hypothesis signal")
# Check 3: experimental_log.md table count
# Prose-style logs are valid as long as they contain enough numeric values
# (checked next). A missing table is therefore a warning, not a hard fail.
table_count = count_markdown_tables(log_text)
if table_count >= args.min_tables:
print(
f"[PASS] experimental_log.md: {table_count} tables found "
f"(min: {args.min_tables})"
)
else:
print(
f"[WARN] experimental_log.md: {table_count} markdown tables found "
f"(min: {args.min_tables}). Prose-format logs are accepted when "
f"numeric values meet the threshold — see Check 4."
)
# Check 4: experimental_log.md numeric values
numeric_count = count_numeric_values(log_text)
if numeric_count >= args.min_numeric_values:
print(
f"[PASS] experimental_log.md: {numeric_count} numeric values "
f"(min: {args.min_numeric_values})"
)
else:
print(
f"[FAIL] experimental_log.md: {numeric_count} numeric values "
f"(min: {args.min_numeric_values})"
)
failures.append(
f"experimental_log.md numeric values {numeric_count} < {args.min_numeric_values}"
)
# Check 5: Jaccard similarity (not duplicates)
idea_tokens = tokenize(idea_text)
log_tokens = tokenize(log_text)
similarity = jaccard(idea_tokens, log_tokens)
if similarity < 0.5:
print(
f"[PASS] idea.md / experimental_log.md not duplicates "
f"(Jaccard: {similarity:.2f})"
)
else:
print(
f"[FAIL] idea.md / experimental_log.md appear to be duplicates "
f"(Jaccard: {similarity:.2f} >= 0.50)"
)
failures.append(
f"idea.md and experimental_log.md too similar (Jaccard {similarity:.2f})"
)
# Summary
print()
if not failures:
print("Pre-flight density check: PASS")
sys.exit(0)
else:
print(f"Pre-flight density check: FAIL ({len(failures)} check(s) failed)")
for f in failures:
print(f" - {f}")
sys.exit(1)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
check_tex_packages.py — Probe the local TeX installation for packages used
by common conference templates, and write tex_profile.json so downstream
agents can select the correct LaTeX patterns at generation time rather than
discovering failures during compilation.
Without this probe, the pipeline learns which packages are missing only after
a compile failure — requiring manual edits and re-runs. Running this script
once at pipeline startup eliminates that iteration.
Checks:
cleveref → if missing: use Figure~\\ref{} instead of \\cref{}
nicefrac → if missing: use a/b instead of \\nicefrac{a}{b}
microtype → if missing: omit \\usepackage{microtype}
fontenc → if missing: omit \\usepackage[T1]{fontenc} (avoids pcrr8t error)
url → if missing: omit \\usepackage{url}
booktabs → required for tables; warns if absent
natbib → required for plainnat bibliography style
hyperref → optional; common in many templates
times → optional; some templates request Times fonts
lmodern → fallback font package
Output: workspace/tex_profile.json
{
"available": ["booktabs", "natbib", ...],
"missing": ["cleveref", "nicefrac", ...],
"use_cleveref": false,
"use_nicefrac": false,
"use_microtype": false,
"use_t1_fontenc": false,
"tex_binary": "/Library/TeX/texbin/pdflatex",
"checked_at": "2026-04-10T12:00:00"
}
Usage:
python check_tex_packages.py --out workspace/tex_profile.json
python check_tex_packages.py --out workspace/tex_profile.json \\
--tex-bin /Library/TeX/texbin/pdflatex
"""
import argparse
import datetime
import json
import os
import shutil
import subprocess
import sys
import tempfile
PACKAGES_TO_CHECK: list[tuple[str, str]] = [
# (package_name, load_option) — empty string means no option
("cleveref", "capitalize"),
("nicefrac", ""),
("microtype", ""),
("url", ""),
("booktabs", ""),
("natbib", ""),
("hyperref", ""),
("fontenc", "T1"),
("times", ""),
("lmodern", ""),
]
TEX_BINARY_CANDIDATES = [
"pdflatex",
"/Library/TeX/texbin/pdflatex",
"/usr/local/bin/pdflatex",
"/usr/bin/pdflatex",
"/opt/homebrew/bin/pdflatex",
]
def find_tex_binary(hint: str | None) -> str | None:
candidates = ([hint] if hint else []) + TEX_BINARY_CANDIDATES
for c in candidates:
if shutil.which(c) or (os.path.isabs(c) and os.path.isfile(c)):
return c
return None
def probe_package(tex_binary: str, package: str, option: str = "") -> bool:
"""Compile a minimal .tex document that loads the package. Returns True if OK."""
if option:
use_line = f"\\usepackage[{option}]{{{package}}}"
else:
use_line = f"\\usepackage{{{package}}}"
minimal = (
"\\documentclass{article}\n"
f"{use_line}\n"
"\\begin{document}x\\end{document}\n"
)
with tempfile.TemporaryDirectory() as tmpdir:
tex_path = os.path.join(tmpdir, "probe.tex")
with open(tex_path, "w") as f:
f.write(minimal)
try:
result = subprocess.run(
[tex_binary, "-interaction=nonstopmode", "-halt-on-error",
"-output-directory", tmpdir, tex_path],
capture_output=True,
timeout=20,
)
return result.returncode == 0
except (subprocess.TimeoutExpired, FileNotFoundError):
return False
def build_fallback_profile(reason: str) -> dict:
"""Profile assuming nothing is available — used when pdflatex not found."""
return {
"available": [],
"missing": [pkg for pkg, _ in PACKAGES_TO_CHECK],
"use_cleveref": False,
"use_nicefrac": False,
"use_microtype": False,
"use_t1_fontenc": False,
"tex_binary": None,
"error": reason,
"checked_at": datetime.datetime.utcnow().isoformat(),
}
def main() -> int:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--out", required=True, help="Output tex_profile.json path")
p.add_argument("--tex-bin", help="Path to pdflatex (auto-detected if omitted)")
args = p.parse_args()
tex_binary = find_tex_binary(args.tex_bin)
if not tex_binary:
print("WARN: pdflatex not found — writing fallback profile (all missing)",
file=sys.stderr)
profile = build_fallback_profile("pdflatex not found in PATH or known locations")
os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True)
with open(args.out, "w") as f:
json.dump(profile, f, indent=2)
return 1
print(f"TeX binary: {tex_binary}")
print("Probing packages...")
available: list[str] = []
missing: list[str] = []
for pkg, opt in PACKAGES_TO_CHECK:
ok = probe_package(tex_binary, pkg, option=opt)
label = f"[{opt}]{pkg}" if opt else pkg
status = "ok" if ok else "MISSING"
print(f" {label:30s} {status}")
(available if ok else missing).append(pkg)
profile = {
"available": available,
"missing": missing,
"use_cleveref": "cleveref" in available,
"use_nicefrac": "nicefrac" in available,
"use_microtype": "microtype" in available,
"use_t1_fontenc": "fontenc" in available,
"tex_binary": tex_binary,
"checked_at": datetime.datetime.utcnow().isoformat(),
}
os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True)
with open(args.out, "w") as f:
json.dump(profile, f, indent=2)
print(f"\nProfile written → {args.out}")
print(f" use_cleveref = {profile['use_cleveref']}")
print(f" use_nicefrac = {profile['use_nicefrac']}")
print(f" use_microtype = {profile['use_microtype']}")
print(f" use_t1_fontenc = {profile['use_t1_fontenc']}")
if missing:
print(f" missing pkgs = {', '.join(missing)}")
return 0
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""
claim_evidence_gate.py — Verify that quantitative claims in a paper draft
are grounded in the experimental log (AutoSci-inspired claim-evidence gate).
Analogous to orphan_cite_gate.py for citations: this gate extracts numeric
claims from the LaTeX draft and checks each against experimental_log.md.
Claims that cannot be corroborated are flagged as UNSUPPORTED.
This is a WARN gate (like validate_consistency.py), not a hard-stop gate:
exit 0 — PASS: all extracted claims corroborated, or no claims extracted
exit 1 — WARN: one or more claims could not be corroborated
exit 2 — ERROR: input file missing or unreadable
Run during content-refinement Step 0 (pre-refinement integrity gate), after
ai_failure_modes checks.
Usage:
python claim_evidence_gate.py \\
--paper workspace/drafts/paper.tex \\
--log workspace/inputs/experimental_log.md \\
--out workspace/claim_evidence_report.json
Output JSON:
{
"supported": [ {claim, value, context, evidence_snippet} ],
"unsupported": [ {claim, value, context} ],
"uncertain": [ {claim, value, context, reason} ],
"summary": {
"total": N,
"supported": N, "unsupported": N, "uncertain": N
}
}
"""
import argparse
import json
import os
import re
import sys
from dataclasses import dataclass, asdict
# ── numeric claim patterns ────────────────────────────────────────────────────
# These patterns capture quantitative claims that typically appear in results:
# - percentage improvements / accuracies e.g. "improves by 3.2%"
# - absolute metric values in result context e.g. "achieves 87.4 mAP"
# - ratio/fold improvements e.g. "2.5× faster"
# - comparison operators with numbers e.g. "outperforms X by 5.1"
# Patterns are intentionally broad; false positives are marked UNCERTAIN.
CLAIM_PATTERNS: list[re.Pattern] = [
# percentage: "by 3.2%", "of 87.4%", "achieves 92.1%"
re.compile(
r"(?:by|of|achieves?|improves?\s+(?:by|to)|gains?|reduces?\s+(?:by|to)|"
r"increases?\s+(?:by|to)|decreases?\s+(?:by|to)|accuracy|f1|recall|"
r"precision|score|performance)\s+([+-]?\d+\.?\d*)\s*%",
re.IGNORECASE,
),
# ratio: "2.5× faster", "3× more", "×1.8"
re.compile(r"(\d+\.?\d*)\s*[×x]\s*(?:faster|slower|more|less|better|worse)", re.IGNORECASE),
re.compile(r"[×x]\s*(\d+\.?\d*)", re.IGNORECASE),
# absolute metric with label: "87.4 mAP", "0.923 AUC", "12.3 BLEU"
re.compile(r"(\d+\.?\d*)\s+(?:mAP|AUC|BLEU|ROUGE|CIDEr|FID|IS|top-\d+|WER|CER|IoU)",
re.IGNORECASE),
# "outperforms / exceeds / surpasses ... by N"
re.compile(
r"(?:outperforms?|exceeds?|surpasses?|beats?|better\s+than|"
r"superior\s+to|lags?\s+behind)\s+[^.]{0,60}?\s+by\s+([+-]?\d+\.?\d*)",
re.IGNORECASE,
),
# LaTeX table cells: numbers in tabular environments (heuristic)
re.compile(r"\\textbf\{(\d+\.?\d*)\}", re.IGNORECASE),
re.compile(r"(\d{2,3}\.\d{1,2})\s*(?:\\\\|&|\})", re.IGNORECASE),
]
# Patterns that indicate a sentence is in a related-work / prior-work context
# (these numbers belong to cited papers, not our claims — mark as UNCERTAIN)
PRIOR_WORK_CONTEXT = re.compile(
r"(?:previous|prior|existing|baselines?|compared\s+to|cite|cited|"
r"et\s+al\.|\\cite\{|\\citet\{|\\citep\{|concurrent|related)",
re.IGNORECASE,
)
# Minimum number of characters around a match to extract as context snippet
CONTEXT_WINDOW = 120
# ── helpers ───────────────────────────────────────────────────────────────────
@dataclass
class Claim:
value: str
context: str
pattern_id: int
is_prior_work: bool = False
def strip_latex_commands(text: str) -> str:
"""Remove common LaTeX markup to reduce false-positive matches."""
text = re.sub(r"\\(?:label|ref|cite[tp]?|footnote|url|href)\{[^}]*\}", " ", text)
text = re.sub(r"\\(?:begin|end)\{[^}]*\}", " ", text)
text = re.sub(r"%.*$", "", text, flags=re.MULTILINE) # strip comments
return text
def _extract_sentence(text: str, match_start: int, match_end: int) -> str:
"""
Return the sentence(s) most immediately containing the match.
Uses sentence boundaries (. ! ?) rather than a fixed char window for the
prior-work detection, so adjacent sections don't bleed in.
"""
# Find the sentence start: last sentence-ending punctuation before the match
before = text[:match_start]
sent_start = max(
before.rfind(". "),
before.rfind(".\n"),
before.rfind("! "),
before.rfind("? "),
before.rfind("\n\n"),
)
sent_start = sent_start + 1 if sent_start >= 0 else 0
# Find the sentence end: next sentence-ending punctuation after the match
after = text[match_end:]
ends = [after.find(". "), after.find(".\n"), after.find("! "), after.find("? "),
after.find("\n\n")]
ends = [e for e in ends if e >= 0]
sent_end = match_end + (min(ends) + 1 if ends else len(after))
return text[sent_start:sent_end].replace("\n", " ").strip()
def extract_claims(tex: str) -> list[Claim]:
"""Extract all quantitative claims from LaTeX source."""
clean = strip_latex_commands(tex)
seen_values: set[str] = set()
claims: list[Claim] = []
for pid, pat in enumerate(CLAIM_PATTERNS):
for m in pat.finditer(clean):
val = m.group(1).strip()
if not val or val in seen_values:
continue
# Skip very small numbers that are likely formatting (0, 1, 2…)
try:
if float(val) < 0.5:
continue
except ValueError:
pass
# Sentence-bounded context for prior-work detection (prevents
# adjacent-section bleed where "Previous methods..." in Related Work
# falsely flags numbers in the Results section)
sentence = _extract_sentence(clean, m.start(), m.end())
is_prior = bool(PRIOR_WORK_CONTEXT.search(sentence))
# Wider context for the human-readable context snippet
start = max(0, m.start() - CONTEXT_WINDOW)
end = min(len(clean), m.end() + CONTEXT_WINDOW)
ctx = clean[start:end].replace("\n", " ").strip()
claims.append(Claim(value=val, context=ctx, pattern_id=pid, is_prior_work=is_prior))
seen_values.add(val)
return claims
def build_number_index(log_text: str) -> set[str]:
"""
Extract all numeric strings from experimental_log.md for O(1) lookup.
Returns a set of string representations (e.g. "87.4", "3.2", "2.5").
"""
nums: set[str] = set()
for m in re.finditer(r"\b(\d+\.?\d*)\b", log_text):
nums.add(m.group(1))
return nums
def find_evidence(value: str, log_text: str) -> str | None:
"""
Return a snippet from the log containing the given numeric value, or None.
Matches whole decimal numbers (e.g. "87.4" matches "87.4" but not "87.41").
"""
pat = re.compile(r"(?<!\d)" + re.escape(value) + r"(?!\d)")
m = pat.search(log_text)
if not m:
return None
start = max(0, m.start() - 80)
end = min(len(log_text), m.end() + 80)
return log_text[start:end].replace("\n", " ").strip()
# ── main ──────────────────────────────────────────────────────────────────────
def main() -> int:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--paper", required=True, help="Path to paper.tex (draft)")
p.add_argument("--log", required=True, help="Path to experimental_log.md")
p.add_argument("--out", required=True, help="Output path for claim_evidence_report.json")
args = p.parse_args()
for path in (args.paper, args.log):
if not os.path.exists(path):
print(f"ERROR: file not found: {path}", file=sys.stderr)
return 2
with open(args.paper) as f:
tex = f.read()
with open(args.log) as f:
log_text = f.read()
claims = extract_claims(tex)
log_nums = build_number_index(log_text)
supported: list[dict] = []
unsupported: list[dict] = []
uncertain: list[dict] = []
for c in claims:
evidence = find_evidence(c.value, log_text)
if c.is_prior_work:
uncertain.append({
"claim": c.context[:200],
"value": c.value,
"context": c.context,
"reason": "appears in prior-work / citation context — not our claim",
})
elif evidence is not None:
supported.append({
"claim": c.context[:200],
"value": c.value,
"context": c.context,
"evidence_snippet": evidence,
})
elif c.value in log_nums:
# Number is in the log but not in matching sentence context — weak support
supported.append({
"claim": c.context[:200],
"value": c.value,
"context": c.context,
"evidence_snippet": f"[number {c.value!r} found in log, no local snippet]",
})
else:
unsupported.append({
"claim": c.context[:200],
"value": c.value,
"context": c.context,
})
report = {
"supported": supported,
"unsupported": unsupported,
"uncertain": uncertain,
"summary": {
"total": len(claims),
"supported": len(supported),
"unsupported": len(unsupported),
"uncertain": len(uncertain),
},
}
os.makedirs(os.path.dirname(os.path.abspath(args.out)) or ".", exist_ok=True)
with open(args.out, "w") as f:
json.dump(report, f, indent=2, ensure_ascii=False)
s = report["summary"]
print(f"Claim-evidence gate: {s['total']} claims extracted")
print(f" supported: {s['supported']}")
print(f" unsupported: {s['unsupported']}")
print(f" uncertain: {s['uncertain']}")
if unsupported:
print("\nUNSUPPORTED claims (not found in experimental_log.md):")
for item in unsupported:
print(f" [{item['value']}] {item['claim'][:120]}")
print(f"\nWARN: {len(unsupported)} unsupported claim(s) — review before submission.")
print(f"Full report: {args.out}")
return 1
print(f"PASS — all extracted claims are corroborated by experimental_log.md")
return 0
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""
diff_outlines.py — Produce a human-readable summary of changes between
outline.json (Step 1) and outline_reconciled.json (Step 3.5).
Only diffs the `section_plan` array; ignores `plotting_plan` and
`intro_related_work_plan` (those are not permitted to change).
Exit codes:
0 identical section_plans (nothing was reconciled)
1 differences found (summary written to --summary)
2 input error
Usage:
python diff_outlines.py \\
--original workspace/outline.json \\
--reconciled workspace/outline_reconciled.json \\
--summary workspace/reconciliation_summary.md
"""
import argparse
import json
import os
import sys
from difflib import unified_diff
def load(path: str) -> dict:
try:
with open(path) as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError) as e:
print(f"ERROR: {path}: {e}", file=sys.stderr)
sys.exit(2)
def section_text(plan: list) -> str:
lines = []
for sec in plan:
lines.append(f"## {sec.get('title', '(untitled)')}")
for sub in sec.get("subsections", []):
lines.append(f" ### {sub}")
for bullet in sec.get("content_bullets", []):
lines.append(f" - {bullet}")
lines.append("")
return "\n".join(lines)
def main() -> int:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--original", required=True)
p.add_argument("--reconciled", required=True)
p.add_argument("--summary", required=True)
args = p.parse_args()
orig = load(args.original)
recon = load(args.reconciled)
orig_text = section_text(orig.get("section_plan", []))
recon_text = section_text(recon.get("section_plan", []))
if orig_text == recon_text:
print("OK: section_plan unchanged — nothing to reconcile.")
# Write empty summary
os.makedirs(os.path.dirname(os.path.abspath(args.summary)) or ".", exist_ok=True)
with open(args.summary, "w") as f:
f.write("# Outline Reconciliation Summary\n\nNo changes — "
"section_plan is identical to original outline.\n")
return 0
diff_lines = list(unified_diff(
orig_text.splitlines(keepends=True),
recon_text.splitlines(keepends=True),
fromfile="outline.json (original)",
tofile="outline_reconciled.json",
lineterm="",
))
# Count sections that contain any changed lines
# Track current section as we walk the diff; mark it when a +/- appears
changed_sections: set[str] = set()
current_section = "(unknown)"
for line in diff_lines:
stripped = line.rstrip("\n")
# Context or changed section-header lines
if stripped.startswith((" ##", "+##", "-##", " ##")):
current_section = stripped.lstrip("+-").strip()
elif stripped.startswith(("+", "-")) and not stripped.startswith(("+++", "---")):
changed_sections.add(current_section)
summary_lines = [
"# Outline Reconciliation Summary",
"",
f"**Changed sections:** {len(changed_sections)}",
"",
"## Diff (section_plan only)",
"",
"```diff",
] + diff_lines + ["```", ""]
os.makedirs(os.path.dirname(os.path.abspath(args.summary)) or ".", exist_ok=True)
with open(args.summary, "w") as f:
f.write("\n".join(summary_lines))
print(f"Reconciliation summary written to {args.summary}")
print(f" {len(changed_sections)} section(s) changed")
return 1
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""
init_workspace.py — Scaffold a paper-orchestra workspace.
Creates the directory tree expected by the orchestrator and writes a stub
README inside `inputs/` listing the four required input files.
Usage:
python init_workspace.py --out /path/to/workspace/
"""
import argparse
import os
import sys
import textwrap
# Minimal fallback template.tex written when the user hasn't provided one.
# Covers the sections required by most conference guidelines and is
# compatible with both single- and double-column document classes.
FALLBACK_TEMPLATE = textwrap.dedent(r"""
\documentclass{article}
% Minimal template scaffolded by init_workspace.py.
% Replace \documentclass{article} with your target conference class
% (e.g. \documentclass[10pt,twocolumn]{article} for double-column venues).
% The Section Writing Agent will fill the TODO placeholders.
% The Literature Review Agent fills Introduction and Related Work.
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{geometry}
\geometry{margin=1in}
\title{TODO: Title}
\author{Anonymous Authors}
\date{}
\begin{document}
\maketitle
\begin{abstract}
TODO: Abstract.
\end{abstract}
\section{Introduction}
TODO: Introduction. (To be filled by literature-review-agent.)
\section{Related Work}
TODO: Related Work. (To be filled by literature-review-agent.)
\section{Method}
TODO: Method. (To be filled by section-writing-agent.)
\section{Experiments}
TODO: Experiments. (To be filled by section-writing-agent.)
\section{Conclusion}
TODO: Conclusion. (To be filled by section-writing-agent.)
\bibliographystyle{plain}
\bibliography{refs}
\end{document}
""").lstrip()
WORKSPACE_DIRS = [
"inputs",
"inputs/figures",
"figures",
"drafts",
"refinement",
"final",
"cache", # S2 verification cache (s2_cache.json written here)
]
INPUTS_README = textwrap.dedent("""\
# Inputs
The paper-orchestra pipeline expects the four files below in this
directory before you start. Optional figures go in `figures/`.
## Required
- `idea.md` — Idea Summary (Sparse or Dense; see io-contract.md)
- `experimental_log.md` — Setup, raw numeric data, qualitative observations
- `template.tex` — LaTeX template for the target conference
- `conference_guidelines.md` — Page limit, mandatory sections, formatting rules
## Optional
- `figures/` — Pre-existing figures (PNG/PDF). If empty, the
plotting agent generates everything from scratch.
See `skills/paper-orchestra/references/io-contract.md` in the repo for the
exact schemas of each file.
""")
def main() -> int:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--out", required=True, help="Path to the workspace directory to create")
p.add_argument("--force", action="store_true",
help="Allow scaffolding into a non-empty directory")
args = p.parse_args()
out = os.path.abspath(args.out)
if os.path.exists(out) and os.listdir(out) and not args.force:
print(f"ERROR: {out} exists and is non-empty. Use --force to overlay.", file=sys.stderr)
return 1
for sub in WORKSPACE_DIRS:
os.makedirs(os.path.join(out, sub), exist_ok=True)
inputs_readme = os.path.join(out, "inputs", "README.md")
if not os.path.exists(inputs_readme):
with open(inputs_readme, "w") as f:
f.write(INPUTS_README)
# Write a fallback template.tex if none exists yet.
# Users can overwrite it with their conference-specific class at any time.
template_path = os.path.join(out, "inputs", "template.tex")
if not os.path.exists(template_path):
with open(template_path, "w") as f:
f.write(FALLBACK_TEMPLATE)
print("INFO: wrote a minimal fallback template.tex to inputs/. "
"Replace \\documentclass{article} with your target conference class if needed.")
print(f"Workspace scaffolded at: {out}")
print("Next: drop your idea.md, experimental_log.md, and")
print("conference_guidelines.md into the inputs/ subdirectory, then run:")
print(f" python {os.path.dirname(__file__)}/validate_inputs.py --workspace {out}")
return 0
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""
validate_consistency.py — Cross-validate idea.md vs experimental_log.md.
Checks for semantic consistency between the stated research idea and the
experimental results. Exits with code 1 (WARN) if issues are found, 0 (PASS)
otherwise.
Note: file-not-found errors should be caught upstream by check_idea_density.py.
This script assumes files exist.
Exit codes:
0 — PASS: all checks passed (warnings may still be printed at PASS level)
1 — WARN: one or more consistency issues found (non-blocking)
"""
import argparse
import re
import sys
def extract_metric_names(log_text: str) -> list:
"""
Extract metric names from markdown table headers in experimental_log.md.
Heuristic: words appearing before ':' or '|' in table header rows,
that look like metric names (capitalized or standard ML metric patterns).
Also extracts column headers from markdown tables.
"""
metrics = set()
# Extract table header rows (rows before |---|)
# A table header is a row of |...|...|..| followed by a separator row
table_header_pattern = re.compile(
r"^\|(.+)\|\s*$\n\|[-| :]+\|", re.MULTILINE
)
for match in table_header_pattern.finditer(log_text):
header_row = match.group(1)
# Split by | and clean each cell
cells = [c.strip() for c in header_row.split("|") if c.strip()]
for cell in cells:
# Keep cells that look like metric names:
# - Contains letters
# - Reasonable length (2-30 chars)
# - Not purely numeric
if re.match(r"^[A-Za-z][\w\s\-@%]*$", cell) and 2 <= len(cell) <= 30:
metrics.add(cell.strip())
# Also extract words before ':' that appear in context suggesting metrics
metric_colon_pattern = re.compile(
r"\b([A-Z][a-zA-Z0-9\-@%]{1,20})\s*:", re.MULTILINE
)
for match in metric_colon_pattern.finditer(log_text):
candidate = match.group(1)
# Filter to likely metric names (avoid common non-metric words)
non_metrics = {
"Note", "Warning", "Error", "Table", "Figure", "Section",
"Results", "Setup", "Config", "Method", "Model", "Dataset",
"Baseline", "Experiment", "Run", "Step", "Epoch", "Loss",
}
if candidate not in non_metrics:
metrics.add(candidate)
return list(metrics)
def extract_dataset_names(log_text: str) -> list:
"""
Extract dataset/benchmark names from experimental_log.md.
Heuristic: capitalized words (3+ chars, title-case or all-caps) appearing
near keywords: dataset, on, benchmark, evaluated, trained on, tested on.
"""
datasets = set()
# Look for patterns like "on CIFAR-10", "evaluated on ImageNet", etc.
dataset_context_pattern = re.compile(
r"(?:dataset|benchmark|evaluated\s+on|trained\s+on|tested\s+on|on\s+the)\s+"
r"([A-Z][A-Za-z0-9\-_]{2,30}(?:\s*[-/]\s*[A-Za-z0-9]+)?)",
re.IGNORECASE,
)
for match in dataset_context_pattern.finditer(log_text):
candidate = match.group(1).strip()
# Must start with capital letter and be title-case or contain digits
if re.match(r"^[A-Z]", candidate) and len(candidate) >= 3:
datasets.add(candidate)
# Also look for all-caps acronyms that are likely dataset names (3-10 chars)
allcaps_pattern = re.compile(r"\b([A-Z]{3,10}(?:-[A-Z0-9]+)?)\b")
# Only in table contexts
table_lines = [
line for line in log_text.split("\n") if "|" in line and "---" not in line
]
for line in table_lines:
for match in allcaps_pattern.finditer(line):
candidate = match.group(1)
# Filter out common non-dataset all-caps words
exclude = {
"GPU", "CPU", "RAM", "SSD", "API", "LLM", "NLP", "CV", "ML",
"DL", "AI", "SGD", "RNN", "CNN", "GAN", "VAE", "BERT",
"TODO", "FIXME", "NOTE", "MAX", "MIN", "AVG", "STD",
}
if candidate not in exclude:
datasets.add(candidate)
return list(datasets)
def count_nonzero_numerics_per_table(log_text: str) -> dict:
"""
For each markdown table, count how many non-zero numeric values appear.
Returns dict mapping table_index (1-based) to count of non-zero values.
"""
results = {}
# Split into table blocks
table_pattern = re.compile(
r"((?:^\|.+\|\s*$\n)+)", re.MULTILINE
)
for i, match in enumerate(table_pattern.finditer(log_text), start=1):
table_text = match.group(1)
numerics = re.findall(r"-?\d+\.?\d*", table_text)
nonzero = [n for n in numerics if float(n) != 0.0]
results[i] = len(nonzero)
return results
def main():
parser = argparse.ArgumentParser(
description="Cross-validate idea.md vs experimental_log.md for consistency"
)
parser.add_argument("--idea", required=True, help="Path to idea.md")
parser.add_argument("--log", required=True, help="Path to experimental_log.md")
args = parser.parse_args()
# Load files (file-not-found handled by check_idea_density.py upstream)
try:
with open(args.idea, "r", encoding="utf-8") as f:
idea_text = f.read()
except FileNotFoundError:
print(f"[ERROR] File not found: {args.idea}", file=sys.stderr)
print("Run check_idea_density.py first to validate file existence.")
sys.exit(1)
try:
with open(args.log, "r", encoding="utf-8") as f:
log_text = f.read()
except FileNotFoundError:
print(f"[ERROR] File not found: {args.log}", file=sys.stderr)
print("Run check_idea_density.py first to validate file existence.")
sys.exit(1)
warnings = []
idea_lower = idea_text.lower()
# Check 1: metric names from log appear in idea
metrics = extract_metric_names(log_text)
if metrics:
matched_metrics = [m for m in metrics if m.lower() in idea_lower]
if matched_metrics:
print(
f"[PASS] Metric alignment: {len(matched_metrics)}/{len(metrics)} "
f"metrics from log found in idea.md "
f"(e.g., {matched_metrics[:3]})"
)
else:
print(
f"[WARN] Metric alignment: 0/{len(metrics)} metrics from "
f"experimental_log.md appear in idea.md"
)
print(f" Metrics detected in log: {metrics[:10]}")
print(
" The idea should mention what it's measuring. "
"Verify the idea describes the same evaluation as the experiments."
)
warnings.append("No metrics from experimental_log.md found in idea.md")
else:
print(
"[WARN] Metric extraction: no metric names detected in "
"experimental_log.md tables — check table formatting"
)
warnings.append("Could not extract metric names from experimental_log.md")
# Check 2: dataset names from log appear in idea (>=50%)
datasets = extract_dataset_names(log_text)
if datasets:
matched_datasets = [d for d in datasets if d.lower() in idea_lower]
coverage = len(matched_datasets) / len(datasets)
if coverage >= 0.5:
print(
f"[PASS] Dataset alignment: {len(matched_datasets)}/{len(datasets)} "
f"datasets from log found in idea.md ({coverage:.0%} coverage)"
)
else:
print(
f"[WARN] Dataset alignment: {len(matched_datasets)}/{len(datasets)} "
f"datasets from log found in idea.md ({coverage:.0%} coverage, need >=50%)"
)
missing = [d for d in datasets if d.lower() not in idea_lower]
print(f" Datasets in log not mentioned in idea: {missing[:5]}")
warnings.append(
f"Only {coverage:.0%} of datasets from log mentioned in idea.md"
)
else:
print(
"[WARN] Dataset extraction: no dataset names detected in "
"experimental_log.md — check that dataset names appear in table context"
)
warnings.append("Could not extract dataset names from experimental_log.md")
# Check 3: no table has all-zero results
table_nonzero = count_nonzero_numerics_per_table(log_text)
if table_nonzero:
all_zero_tables = [
idx for idx, count in table_nonzero.items() if count == 0
]
if all_zero_tables:
print(
f"[WARN] Zero-result tables: table(s) {all_zero_tables} contain "
f"only zero or no numeric values — verify results are present"
)
warnings.append(
f"Tables {all_zero_tables} have no non-zero numeric results"
)
else:
total_tables = len(table_nonzero)
print(
f"[PASS] Result coverage: all {total_tables} table(s) contain "
f"at least one non-zero numeric value"
)
else:
# No tables found — this should be caught by check_idea_density.py
print(
"[WARN] No tables found in experimental_log.md — "
"run check_idea_density.py to verify table count"
)
warnings.append("No tables found in experimental_log.md")
# Summary
print()
if not warnings:
print("Consistency check: PASS")
sys.exit(0)
else:
print(f"Consistency check: WARN ({len(warnings)} issue(s) found — non-blocking)")
for w in warnings:
print(f" - {w}")
print(
"\nThese are warnings, not errors. The pipeline will continue. "
"Address warnings before submission."
)
sys.exit(1)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
validate_inputs.py — Verify that a paper-orchestra workspace has the four
required input files in the correct place and minimally well-formed.
This is a deterministic structural check. It does NOT call an LLM, does NOT
talk to the network, and does NOT validate semantic content — that is the
job of the Outline Agent itself.
Exit codes:
0 all checks passed
1 one or more required inputs missing or malformed
Usage:
python validate_inputs.py --workspace /path/to/workspace/
"""
import argparse
import os
import re
import sys
REQUIRED_INPUTS = [
"idea.md",
"experimental_log.md",
"template.tex",
"conference_guidelines.md",
]
def check_file_exists(path: str) -> list[str]:
if not os.path.isfile(path):
return [f"MISSING: {path}"]
if os.path.getsize(path) == 0:
return [f"EMPTY: {path}"]
return []
def check_idea_md(path: str) -> list[str]:
errors = check_file_exists(path)
if errors:
return errors
text = open(path).read()
required_headings = ["Problem Statement", "Core Hypothesis"]
missing = [h for h in required_headings if not re.search(rf"^#+\s*{re.escape(h)}", text, re.M)]
if missing:
return [f"WARN: idea.md missing recommended headings: {missing}"]
return []
def check_experimental_log(path: str) -> list[str]:
errors = check_file_exists(path)
if errors:
return errors
text = open(path).read()
if not re.search(r"^##\s+1\.?\s*Experimental Setup", text, re.M):
return ["WARN: experimental_log.md missing '## 1. Experimental Setup' heading"]
if not re.search(r"^##\s+2\.?\s*Raw Numeric Data", text, re.M):
return ["WARN: experimental_log.md missing '## 2. Raw Numeric Data' heading"]
# Anti-leakage check on log itself: should not reference Figure N or Table N
leaks = re.findall(r"(?:see|in|from)\s+(?:Figure|Fig\.|Table|Tab\.)\s*\d+", text, re.I)
if leaks:
return [f"ERROR: experimental_log.md contains figure/table references "
f"({leaks[:3]}...). Per App. F.2 the log must be self-contained."]
return []
def check_template(path: str) -> list[str]:
errors = check_file_exists(path)
if errors:
return errors
text = open(path).read()
if "\\documentclass" not in text:
return [f"ERROR: {path} missing \\documentclass — not a LaTeX document"]
if not re.search(r"\\section\s*\{", text):
return [f"WARN: {path} has no \\section{{...}} commands — outline agent "
f"will have no skeleton to fill"]
return []
def check_guidelines(path: str) -> list[str]:
errors = check_file_exists(path)
if errors:
return errors
text = open(path).read().lower()
out = []
if "page" not in text:
out.append("WARN: conference_guidelines.md does not mention 'page' — "
"page limit unclear")
if "deadline" not in text and "cutoff" not in text and "submission" not in text:
out.append("WARN: conference_guidelines.md does not mention a deadline / "
"cutoff — literature review agent will not be able to scope citations")
return out
def main() -> int:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--workspace", required=True, help="Path to the workspace directory")
args = p.parse_args()
ws = os.path.abspath(args.workspace)
inputs = os.path.join(ws, "inputs")
if not os.path.isdir(inputs):
print(f"ERROR: {inputs} does not exist. Run init_workspace.py first.",
file=sys.stderr)
return 1
all_problems: list[str] = []
checks = {
"idea.md": check_idea_md,
"experimental_log.md": check_experimental_log,
"template.tex": check_template,
"conference_guidelines.md": check_guidelines,
}
for fname, fn in checks.items():
problems = fn(os.path.join(inputs, fname))
for p_ in problems:
all_problems.append(p_)
figs = os.path.join(inputs, "figures")
if os.path.isdir(figs):
n_figs = len([f for f in os.listdir(figs) if f.lower().endswith((".png", ".pdf", ".jpg", ".jpeg"))])
print(f"INFO: {n_figs} pre-existing figure(s) in inputs/figures/")
else:
print("INFO: no inputs/figures/ — plotting agent will generate everything")
if not all_problems:
print("OK: all 4 required inputs present and well-formed.")
return 0
fatal = [p for p in all_problems if p.startswith("ERROR") or p.startswith("MISSING") or p.startswith("EMPTY")]
warn = [p for p in all_problems if p.startswith("WARN")]
for p_ in fatal:
print(p_, file=sys.stderr)
for p_ in warn:
print(p_)
return 1 if fatal else 0
if __name__ == "__main__":
sys.exit(main())