Ref Verify

ref-verify — Reference Hallucination Guard

The specific failure this skill prevents: LLMs describe paper content from recalled training data rather than from what the abstract actually says. A paper gets attributed findings it doesn't contain, or cited for claims that appear nowhere in the text. The fix is one rule applied consistently:

Every content statement about a paper must come from a live-fetched abstract, quoted or paraphrased verbatim. If you cannot fetch the abstract, say so explicitly — never fill the gap with recalled description.

---

Mode Decision

Pick the mode before doing any work. The choice controls cost and depth.

User provides DOI(s) for sanity check?
  └─ ≤10 refs → Quick Screen all
  └─ >10 refs → Quick Screen all; Full Audit only MISMATCH/DEAD results

User says "find papers on X" or "cite papers supporting claim Y"?
  └─ Full Audit (searching from scratch requires content verification)

User says "verify/check my reference list" or pre-submission audit?
  └─ ≤5 refs  → Full Audit all
  └─ >5 refs  → Quick Screen all first; Full Audit MISMATCH/DEAD + any ref
                 cited for a specific factual claim

User is writing inline and adds a single citation from memory?
  └─ Quick Screen minimum; Full Audit if citing for a specific claim

The expensive part is Full Audit (5-layer, abstract fetch). Quick Screen costs ~5s per paper. Only escalate to Full Audit when the task genuinely requires content verification.

---

Executable Engine

Use this as a skill-level workflow. Do not build or require MCP for this path. The CLI is an execution engine the skill can call from the terminal; the manual protocol remains the fallback and the source of truth for layers the CLI does not yet cover.

CLI Availability Check

Before a DOI-backed check, see whether the executable engine is available:

ref-verify --help

npx skills add does not pip-install the Python CLI. If the console script is unavailable, do not treat that as a verification result.

If the console script is unavailable but the repository source is present, use:

PYTHONPATH=src python3 -m ref_verify.cli --help

If both commands fail, follow the manual fallback protocol below. Do not pretend the CLI ran, and do not invent a result from memory.

CLI-first workflow

Use the CLI first when the user provides a DOI or asks whether a DOI-backed abstract supports a specific factual claim. Current CLI scope is CrossRef metadata verification plus DOI-bound abstract claim checks. Claim checks use CrossRef first, then DOI-bound OpenAlex, Semantic Scholar, and PubMed fallback when CrossRef has no abstract.

CrossRef metadata screen:

ref-verify verify-doi <doi> \
  --title "<provided title>" \
  --first-author <provided-first-author-last-name> \
  --year <provided-year> \
  --json

Source-checkout fallback:

PYTHONPATH=src python3 -m ref_verify.cli verify-doi <doi> \
  --title "<provided title>" \
  --first-author <provided-first-author-last-name> \
  --year <provided-year> \
  --json

verify-doi exits 0 only for PASS; WARN and REJECT return non-zero. Supplying only a DOI checks that CrossRef resolves but is not enough to pass metadata verification.

Route the result:

• PASS: report the metadata match, then continue Layer 4 DOI landing-page

resolution and Layer 5 retraction checks when the selected mode requires them.

• WARN: report the missing or weak metadata, then continue manual fallback for

the missing layers instead of silently accepting the reference.

• REJECT: stop using that citation as verified; report the mismatch and ask

for a corrected DOI or citation unless the user explicitly wants alternatives.

Single claim check against a DOI abstract:

ref-verify check-claim <doi> --claim "<specific factual claim>" --json

Source-checkout fallback:

PYTHONPATH=src python3 -m ref_verify.cli check-claim <doi> --claim "<specific factual claim>" --json

By default, check-claim uses CrossRef first. If CrossRef has no abstract, it tries DOI-bound OpenAlex, Semantic Scholar, and PubMed fallback sources. Use --source crossref, --source openalex, --source semantic-scholar, or --source pubmed for source-specific debugging. Explicit non-CrossRef source selection bypasses CrossRef, so it can isolate an OpenAlex, Semantic Scholar, or PubMed failure.

Route the result:

• ACCEPT: quote the CLI evidence and continue any required manual DOI

resolution, second-source existence, and retraction layers.

• WARN or PARTIAL: report what the selected abstract source did and did not

support, then continue manual fallback when more layers are required.

• UNVERIFIABLE: no trusted DOI-bound abstract evidence was available from the

attempted CLI sources. Continue the manual fallback chain below instead of treating the claim as rejected or supported.

For JSON output, use abstract_source, source_attempts, and error_code to decide the next step:

• CLAIM_SUPPORTED: explicit abstract support found.
• CLAIM_NOT_EXPLICIT: an abstract was available, but the claim was not

explicitly supported.

• CLAIM_AMBIGUOUS: numeric evidence or context exists, but binding is

ambiguous.

• NO_ABSTRACT: attempted DOI-bound sources did not provide abstract text.
• DOI_NOT_FOUND: selected source did not find a DOI-bound record.
• DOI_MISMATCH: the primary or explicitly selected DOI-bound record did not

match the requested DOI.

• SOURCE_API_ERROR, SOURCE_TIMEOUT, SOURCE_RATE_LIMITED, SOURCE_UNSUPPORTED: source lookup

failed, timed out, was rate-limited, or could not be used.

The CLI does not replace Layer 4 DOI landing-page resolution, two-source existence checks, or retraction checks. Continue the manual protocol for those layers when the selected mode requires them.

---

Quick Screen — metadata + DOI sanity check

Use when the user provides a DOI or full citation and wants a sanity check.

1. Hit CrossRef: https://api.crossref.org/works/{DOI} 2. Compare returned title + first-author last name against what user provided 3. Fetch https://doi.org/{DOI} — confirm it resolves and lands on the right paper 4. Report one line per reference:

Smith et al. (2021) 10.1234/example — PASS (title/author match, DOI resolves)
Jones (2019) 10.5678/other — MISMATCH (CrossRef: Jones & Lee 2019, not Jones alone)
Kim (2023) 10.9999/fake — DEAD DOI

Escalate to Full Audit if: DOI resolves to a different paper, any field mismatches, or user is citing for a specific factual claim.

---

Full Audit — for literature search or pre-submission check

Use when: searching for papers to support a claim, or doing a final citation sweep.

Run all five layers per paper. The layers are ordered by what they catch — don't skip forward.

Layer 1 — Existence

Search two sources independently:

• CrossRef: https://api.crossref.org/works?query.bibliographic={title+author}&rows=5
• Semantic Scholar: https://api.semanticscholar.org/graph/v1/paper/search?query={title+author}&fields=title,authors,year,externalIds,abstract&limit=5
• arXiv for preprints: https://export.arxiv.org/api/query?search_query=ti:{title}&max_results=3

A paper is confirmed only if titles essentially match and first-author last name agrees across two sources.

• Two-source hit → CONFIRMED
• One-source → SINGLE-SOURCE ⚠ — proceed with caution, note in output
• Zero → NOT FOUND ✗ — stop; report clearly; do not invent a substitute

Layer 2 — Metadata

Extract from confirmed sources and compare: title, all authors (last names), year, journal full name, DOI, volume/pages (mark [NOT IN SOURCE] if absent). If any field differs between sources, show both — do not silently pick one.

Layer 3 — Content Traceability ← most important layer

This is where the skill's core value lies. The goal is not just "does this paper exist" but "does this paper actually contain the claim being attributed to it."

Fetch the abstract using this priority order: 1. CrossRef raw JSON: https://api.crossref.org/works/{DOI} — check the abstract field 2. OpenAlex: https://api.openalex.org/works/doi:{DOI}?mailto=verify@ref-verify.local — reconstruct abstract_inverted_index 3. Semantic Scholar: append &fields=abstract to your S2 DOI lookup 4. Open-access fallback: https://api.unpaywall.org/v2/{DOI}?email=verify@ref-verify.local — check is_oa and oa_locations 5. arXiv fallback for preprints: https://export.arxiv.org/api/query?id_list={arxiv_id} 6. PubMed Central for life/bio papers: https://www.ncbi.nlm.nih.gov/pmc/articles/{PMCID}/

After fetching, check: does the abstract contain the specific claim being cited?

• Abstract explicitly contains the claim (quote it verbatim) → CONTENT: SUPPORTED
• Abstract is about the topic but doesn't make the specific claim → CONTENT: PARTIAL — quote what it actually says
• Abstract contradicts the claim → CONTENT: CONTRADICTED — do not use this citation
• Abstract not accessible after trying all 5 sources → CONTENT: UNVERIFIABLE — user must check full text

The rule that cannot be relaxed: if you describe what a paper "shows" or "demonstrates" or "reports," you must quote or directly paraphrase the fetched abstract text. Summarizing from memory is not permitted even if you feel confident.

Layer 4 — DOI Resolution

Fetch https://doi.org/{DOI}. Confirm the landing page matches the expected paper. A 403 (bot-blocked) from a URL slug containing the title and volume is not a dead link — note it as paywalled. A redirect to an unrelated page is a critical failure.

Layer 5 — Retraction

Search "{first author last name}" "{journal name}" retraction and check the DOI landing page for retraction banners. A retracted paper must not be used as a primary source.

---

Output Format

Quick Screen: one line per reference (see above).

Full Audit: one card per paper, then a summary table.

REFERENCE AUDIT
────────────────────────────────────────────────
Paper:   [Title from live source — not from memory]
DOI:     [DOI] — [✓ Resolves | ✗ Dead | ✗ Wrong paper | ⚠ Paywalled-403]
Authors: [Full list from CrossRef/OpenAlex/S2]
Year:    [Year] — Source: CrossRef | S2 | arXiv
Journal: [Full name]

EXISTENCE:  ✓ Confirmed (sources) | ⚠ Single-source | ✗ Not found
METADATA:   ✓ Consistent | ⚠ Discrepancy: [field: value-A vs value-B]
CONTENT:    ✓ Supported — "[verbatim abstract excerpt]"
            ⚠ Partial — abstract says: "[what it actually says]"
            ✗ Contradicted | — Unverifiable (tried CrossRef/OpenAlex/S2/Unpaywall/arXiv/PubMed)
RETRACTION: ✓ None found | ✗ Retracted

VERDICT: ACCEPT | WARN | REJECT
Reason: [one sentence — what's missing or wrong]
────────────────────────────────────────────────

CONTENT field must show either a verbatim excerpt or an explicit "Unverifiable" — never a summary written from memory.

ACCEPT: two-source confirmed, DOI resolves to right paper, content supported by fetched abstract, no retraction. WARN: solvable issue — single source, partial content match, or abstract inaccessible after trying all fallbacks. Safe to use if user verifies the flagged item. REJECT: DOI dead or resolves to wrong paper, paper not found anywhere, content contradicted, or retraction confirmed.

Summary table after all cards:

SUMMARY
────────────────────────────────────────────────
1. Smith et al. (2021)  — ACCEPT
2. Kim & Park (2019)    — WARN (abstract unverifiable; try PMC or institutional access)
3. Zhang (2023)         — REJECT (DOI resolves to different paper)
────────────────────────────────────────────────
X / Y verified.  Z need attention.

---

Anti-Hallucination Rules

• Never recall a DOI from memory — fetch from CrossRef or S2.
• Never describe paper content without a fetched abstract to quote from.
• Never fill in missing metadata by guessing or pattern-matching.
• If two sources disagree, show both — do not choose silently.
• If the abstract is inaccessible after all five fallback sources, mark UNVERIFIABLE and stop — do not substitute a description from memory.

---

Edge Cases

Preprint vs. published: record both DOIs; prefer published for citation; note if title changed between versions.

Author name variants: "J. Smith" vs "John Smith" — flag but do not merge; let user confirm.

Conference proceedings: volume/pages often absent from CrossRef; mark [NOT IN SOURCE], not guessed.

S2 rate limiting: wait 2s and retry once; if still failing, use CrossRef as primary and note single-source limitation.

What happened

The skill returned WARN or REJECT but the citation was correct.

Prompt used:

(paste the prompt you gave the agent)

Skill output:

(paste the VERDICT and the specific flag/warning)

Why the flag was wrong:

(explain what the correct metadata or content is, with a source)

Evidence:

• CrossRef record: (URL or paste)
• doi.org result: (URL)
• Abstract source: (CrossRef / S2 / PubMed link)

Expected verdict: ACCEPT / WARN (lower severity)

What problem does this solve?

(describe a real failure mode or gap, not an abstract improvement)

Proposed change

(what should the skill do differently?)

Tradeoffs

• Does this add cost (more API calls, longer output)?
• Does this change the Quick Screen / Full Audit balance?
• Does it affect agents other than Claude Code?

Example

(if possible, show a before/after of what the output would look like)

What happened

The skill returned ACCEPT or WARN but the citation was actually wrong.

Prompt used:

(paste the prompt you gave the agent)

Skill output:

(paste the VERDICT and CONTENT section)

What was actually wrong:

• [ ] Wrong DOI (resolves to different paper)
• [ ] Wrong authors
• [ ] Wrong year
• [ ] Content not in abstract (hallucinated description)
• [ ] Near-miss (right number, wrong context)
• [ ] Retracted paper not flagged
• [ ] Other:

Evidence (CrossRef / DOI / abstract text):

(paste the live CrossRef or doi.org result that shows the error)

Expected verdict: WARN / REJECT

What this changes

(one paragraph — what problem does this PR solve?)

Type of change

• [ ] Bug fix — skill was producing wrong verdicts
• [ ] New API source (e.g. Retraction Watch, IEEE Xplore)
• [ ] Trigger description improvement
• [ ] New test case in evals/evals.json
• [ ] Documentation

Evidence

For skill changes: show a before/after example with a real DOI.

Before (without this change):
[verdict or behavior]

After (with this change):
[verdict or behavior]

For trigger changes: show which queries now correctly trigger (or don't) that didn't before.

Checklist

• [ ] The core rule is preserved (verbatim abstract traceability, UNVERIFIABLE instead of guessing)
• [ ] Quick Screen and Full Audit remain distinct modes
• [ ] SKILL.md is under 500 lines
• [ ] If adding a test case: the DOI is real and independently verifiable
• [ ] CHANGELOG.md updated under [Unreleased]

Agent Usage Contract

This document defines how an AI agent should call ref-verify when checking citation claims.

ref-verify is a verifier, not a claim extractor. The agent is responsible for extracting DOI-bound numeric claim candidates from the draft, chat session, markdown file, or other working context. ref-verify then checks those candidates against DOI-bound abstracts.

When To Use

Use ref-verify when a citation includes:

• a DOI, and
• a numeric claim about that DOI's paper.

Examples:

• This paper reports 95% accuracy.
• The study included 12 patients.
• The device survived 5000 cycles.
• Samples were incubated at 37 °C.

Do not use ref-verify to judge paper quality, field consensus, full-text claims, table values, figure values, or complex statistical interpretation.

Agent Workflow

1. Extract candidate {doi, claim} pairs from the working context. 2. Keep only claims that contain explicit numeric evidence candidates. 3. Write a JSONL file with one object per candidate. 4. Run:

ref-verify check-file claims.jsonl --json

5. Treat only verdict == "ACCEPT" as verified. 6. Treat every other result as not verified for citation support. 7. Do not rewrite WARN, PARTIAL, REJECT, UNVERIFIABLE, or failed > 0 into acceptance.

Input Contract

Preferred agent input is JSONL:

{"id":"claim-1","doi":"10.xxxx/example-a","claim":"This paper reports 95% accuracy."}
{"id":"claim-2","doi":"10.xxxx/example-b","claim":"The study included 12 patients.","note":"draft paragraph 4"}

Required fields:

• doi
• claim

Optional fields:

• id: stable identifier for mapping results back to the draft.
• source: one of auto, crossref, openalex, semantic-scholar, or pubmed.
• note: any caller context. It is preserved in JSON output.

CSV is supported for user-created files, but agents should prefer JSONL because it avoids CSV quoting ambiguity.

Exit Code Contract

• Exit 0: command completed and every row was ACCEPT.
• Exit 2: command completed, but one or more rows were not accepted.
• Exit 1: input or runtime failure prevented normal batch processing.

Agents must inspect JSON output even when the exit code is non-zero. Exit 2 can still contain useful row-level results.

JSON Output Contract

check-file --json returns:

{
  "summary": {
    "total": 2,
    "accept": 1,
    "warn": 1,
    "reject": 0,
    "partial": 0,
    "unverifiable": 1,
    "failed": 1
  },
  "results": [
    {
      "row_number": 1,
      "id": "claim-1",
      "doi": "10.xxxx/example-a",
      "claim": "This paper reports 95% accuracy.",
      "verdict": "ACCEPT",
      "status": "SUPPORTED",
      "reason": "...",
      "evidence": "...",
      "abstract_source": "crossref",
      "source_attempts": [],
      "error_code": "CLAIM_SUPPORTED"
    },
    {
      "row_number": 2,
      "id": "claim-2",
      "doi": "10.xxxx/example-b",
      "claim": "The study included 12 patients.",
      "verdict": "WARN",
      "status": "UNVERIFIABLE",
      "reason": "Row could not be checked: ...",
      "evidence": "",
      "abstract_source": null,
      "source_attempts": [],
      "error_code": "ROW_CHECK_ERROR"
    }
  ]
}

Summary categories are diagnostic counts, not mutually exclusive buckets. For example, a failed row can also be WARN and UNVERIFIABLE.

Agents should route results by error_code, verdict, and status.

Common routing:

• CLAIM_SUPPORTED: verified citation claim.
• CLAIM_AMBIGUOUS: numeric evidence exists, but binding is ambiguous.
• CLAIM_NOT_EXPLICIT: abstract does not explicitly support the claim.
• NO_ABSTRACT: no trusted DOI-bound abstract evidence was available.
• DOI_NOT_FOUND: selected source did not find a DOI-bound record.
• DOI_MISMATCH: selected DOI-bound record did not match the requested DOI.
• SOURCE_API_ERROR, SOURCE_TIMEOUT, SOURCE_RATE_LIMITED, SOURCE_UNSUPPORTED: source lookup failed, timed out, was rate-limited, or could not be used.
• ROW_CHECK_ERROR: one row could not be checked, but other rows may still have results.

Agent Must Not

• Do not accept a claim because the DOI exists.
• Do not accept a claim because a related number appears under a different subject.
• Do not accept a claim when verdict is WARN or REJECT.
• Do not accept a claim when status is PARTIAL or UNVERIFIABLE.
• Do not treat failed > 0 as harmless.
• Do not fill missing abstract evidence from memory or model knowledge.
• Do not infer full-text, table, or figure support from an abstract-only check.

Minimal Agent Prompt

Extract DOI-bound numeric citation claims from this draft.
Write claims.jsonl with {id, doi, claim, note}.
Run ref-verify check-file claims.jsonl --json.
Treat only verdict ACCEPT as verified.
Report every WARN, PARTIAL, REJECT, UNVERIFIABLE, or failed result as not verified.
Do not use memory to fill missing abstract evidence.

Safe Interpretation

ref-verify is a conservative citation guard. It does not prove that a paper is good, important, unretracted, or that the full paper supports a broader statement. It only checks whether the DOI-bound abstract explicitly supports the submitted numeric claim.

Changelog

All notable changes to ref-verify will be documented here.

Format follows Keep a Changelog. Versioning follows Semantic Versioning.

---

Unreleased

[1.2.0] — 2026-06-08

Added

• Added ref-verify check-file for JSONL and CSV DOI/claim batch checks.
• Added fixture-backed numeric claim eval coverage for repeated-use workflows.
• Added DOI-bound OpenAlex abstract fallback before Semantic Scholar and PubMed.
• Added a CLI regression corpus and manual Live Smoke ship gate for release-readiness checks.
• Added English and Korean scope guidance that explains what the tool verifies, what it does not verify, and how to interpret non-ACCEPT verdicts.

Fixed

• Fixed composite scientific units such as MV/m being misread as numerator-only units.
• Added numeric claim support for common physical-science units such as eV, Ω·cm, S/m, and MPa.
• Treated estimated to be <value> as a reported numeric value while keeping predictive estimated to exceed frames conservative.
• Prevented comparative evidence such as >220 °C from accepting an exact 220 °C claim.
• Allowed physical measurement conditions such as 1.7 eV in the temperature range and 5 S/m at 1 kHz without relaxing count-claim scope guards.
• Separated Semantic Scholar 429 rate limits into SOURCE_RATE_LIMITED and retried once before marking the source unavailable.
• Clarified that npx skills add installs the agent skill but does not pip-install the Python CLI.
• Fixed comma-clause splitting so current-study result sentences can bind a number to a subject across descriptive commas when no same-unit competing value is present.
• Recognized claim-side up to <value> comparators for percentage and unit/count claims while keeping exact-claim guards conservative.
• Treated temperature measurements followed by physical range/field conditions as measurement context, and allowed generic Measurements... sentences to inherit subject context from the immediately preceding sentence.

[1.1.2] — 2026-06-08

Changed

• Added release automation guardrails for CI, wheel smoke testing, manual live API smoke checks, and PyPI trusted publishing.
• Updated GitHub Actions workflows to current Node runtime-compatible action versions.

[1.1.1] — 2026-06-08

Changed

• Updated Python packaging metadata to the current SPDX license format.
• Clarified that zero runtime dependencies means zero third-party Python packages; CLI verification still requires outbound HTTPS access to public academic APIs.
• Clarified that the Python package is the CLI engine only. Install the agent skill from GitHub with npx skills add.

[1.1.0] — 2026-06-07

Added

• Python package scaffold with zero third-party Python runtime dependencies.
• ref-verify verify-doi CLI for CrossRef-backed DOI metadata checks.
• ref-verify check-claim CLI for abstract-grounded claim support checks.
• Machine-readable JSON output for downstream manuscript preflight, MCP, and Zotero integrations.
• Offline unit tests for DOI metadata comparison, CrossRef parsing, claim support verdicts, and CLI output.

Changed

• Documented the executable engine path alongside the existing agent skill workflow.
• Updated the skill instructions to prefer the CLI when it is installed, while keeping the manual verification protocol as fallback.

[1.0.0] — 2026-06-01

Added

• 5-layer verification protocol: Existence → Metadata → Content Traceability → DOI Resolution → Retraction Check
• Two-mode design: Quick Screen (seconds per paper, for DOI spot-checks) and Full Audit (abstract fetch + claim verification, for search tasks and pre-submission review)
• Content traceability rule: every content statement must come from a live-fetched abstract quoted verbatim — never from training data recall
• Open-access fallback chain: CrossRef JSON → Semantic Scholar → Unpaywall → arXiv → PubMed, in order
• Near-miss detection: evaluates whether the abstract supports the specific claim being cited, not just whether the paper exists
• Automatic mode selection: decision tree based on task type (search vs. spot-check vs. audit)
• Structured verdicts: ACCEPT / WARN / REJECT with explicit per-layer evidence
• Trigger description optimized for Claude Code, Cursor, and Codex auto-detection
• Evaluation suite: 3 test cases with real-world hallucination examples from materials science literature

Verified catches

• Content hallucination: AI described paper content not present in the CrossRef abstract (Nemat-Nasser 2002)
• Wrong DOI: citation resolved to different paper, different authors, wrong year (Carpi 2011)
• Near-miss: "500% strain" in abstract was a measurement condition, not an actuation result (Kofod 2003)

Code of Conduct

Our Pledge

We are committed to making participation in this project a respectful, harassment-free experience for everyone, regardless of background or experience level.

Our Standards

Expected behavior:

• Use clear, factual language — this project is about verification, not advocacy
• Accept corrections gracefully — if a test case shows a bug, that's valuable information
• Focus feedback on the skill behavior, not on other contributors

Unacceptable behavior:

• Harassment, personal attacks, or discriminatory language
• Deliberate misrepresentation of test results or failure cases
• Submitting fabricated hallucination examples as real catches

Enforcement

Instances of unacceptable behavior may be reported to the maintainers via GitHub Issues (mark as conduct). All reports will be reviewed and investigated. Maintainers have the right to remove comments, close issues, or ban contributors who violate these standards.

Attribution

Adapted from the Contributor Covenant, version 2.1.

Contributing to ref-verify

Thank you for helping improve citation verification. Contributions of all kinds are welcome — new failure modes, broader API coverage, improved trigger descriptions, and additional test cases.

---

Ways to contribute

Report a hallucination case

If ref-verify missed a real error (false negative) or flagged something incorrectly (false positive), open an issue with:

• The prompt you used
• What the skill returned
• What the correct result should have been
• The DOI or paper title involved

These are the most valuable contributions. Real failure cases are how the skill improves.

Add a test case

Test cases live in evals/evals.json. A good test case:

• Uses a real DOI that can be independently verified
• Tests a specific failure mode (wrong author, hallucinated content, near-miss, retracted paper)
• Has a clear expected_output description

See the existing three cases for format reference.

Run verification locally

Before opening a pull request, run the source tests:

PYTHONPATH=src python3 -m unittest discover -s tests -v
python3 -m py_compile src/ref_verify/*.py tests/*.py scripts/*.py

For release or packaging changes, also build and smoke-test the package:

python3 -m pip install --upgrade build twine
python3 -m build --sdist --wheel --outdir dist .
python3 -m twine check dist/*
version="$(python3 -c 'import re; print(re.search(r"^version = \"([^\"]+)\"", open("pyproject.toml", encoding="utf-8").read(), re.M).group(1))')"
python3 scripts/package_smoke.py --wheel dist/ref_verify-*.whl --expected-version "$version"

The live API smoke workflow is manual because it calls public academic APIs and can fail when an upstream service is slow or unavailable.

The PyPI publish workflow uses trusted publishing. Before publishing from a GitHub Release, configure PyPI Trusted Publisher for this repository and the GitHub environment named pypi; otherwise the release build can pass and the final publish step will still fail.

Improve the skill

SKILL.md is the skill itself — the instructions the agent follows. Improvements should:

• Solve a documented problem (link to an issue or test case)
• Not add scope beyond citation verification
• Keep the two-mode design intact (Quick Screen and Full Audit)
• Preserve the core rule: every content statement must be verbatim from a fetched abstract

Extend API coverage

Currently covers: CrossRef, Semantic Scholar, Unpaywall, arXiv, PubMed.

Additions worth considering: Retraction Watch API, DOAJ for open-access status, IEEE Xplore for conference papers, bioRxiv for life-science preprints.

---

Submitting changes

1. Fork the repository 2. Create a branch: git checkout -b fix/description-near-miss or feat/retraction-watch-api 3. Make your change 4. Test it: run the skill on the evals/evals.json cases and verify outputs look correct 5. Open a pull request — use the template provided

Pull requests that include a new test case or a documented before/after example are much easier to review and merge.

---

What not to change

• Do not weaken the core rule (verbatim abstract traceability)
• Do not merge Quick Screen and Full Audit into a single mode
• Do not add verification for non-academic sources (web pages, blog posts) — that is a different problem

---

Questions

Open a GitHub Discussion for anything that isn't a bug or a concrete feature request.

Batch Claim Workflow Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add ref-verify check-file so users can verify many DOI-bound numeric claims from JSONL or CSV while preserving the existing conservative single-claim verdict logic.

Architecture: Add src/ref_verify/batch.py for parsing, validation, result aggregation, and text/JSON-friendly batch models. Keep DOI lookup and claim verification in src/ref_verify/cli.py by extracting the current check-claim execution into a reusable helper, then call that helper once per row. Add deterministic fixture-backed eval tests so batch workflow improves real use without broadening claim acceptance.

Tech Stack: Python standard library only (argparse, csv, json, dataclasses, pathlib), existing unittest tests, existing CrossrefClient, abstract source clients, and ClaimSupportResult.

---

File Map

• Create src/ref_verify/batch.py: input format detection, JSONL/CSV parsing, row validation, batch result aggregation, text rendering.
• Modify src/ref_verify/cli.py: add check-file parser, extract reusable claim execution, wire batch command, preserve single-claim behavior.
• Create tests/test_batch.py: direct unit tests for parser, validation, summary, and text rendering.
• Modify tests/test_cli.py: CLI-level tests using fake clients for check-file JSONL/CSV, exit codes, JSON output, and malformed input.
• Create tests/fixtures/numeric_claim_eval.jsonl: local abstract/claim eval fixtures across domains.
• Modify tests/test_numeric_claim.py: load fixture evals and verify expected statuses without live APIs.
• Modify README.md, README.ko.md, CHANGELOG.md: document batch usage after code lands.

Task 1: Batch Parser And Models

Files:

• Create: src/ref_verify/batch.py
• Test: tests/test_batch.py

• [ ] Step 1: Write failing parser tests

Add tests/test_batch.py:

import json
import tempfile
import unittest
from pathlib import Path

from ref_verify.batch import (
    BatchInputError,
    BatchRowResult,
    BatchSummary,
    ClaimInputRow,
    detect_format,
    parse_claim_file,
    summarize_results,
)


class BatchParserTests(unittest.TestCase):
    def test_detect_format_from_extension(self):
        self.assertEqual(detect_format(Path("claims.jsonl"), None), "jsonl")
        self.assertEqual(detect_format(Path("claims.csv"), None), "csv")
        self.assertEqual(detect_format(Path("claims.txt"), "jsonl"), "jsonl")

    def test_unknown_format_is_rejected(self):
        with self.assertRaisesRegex(BatchInputError, "Unsupported input format"):
            detect_format(Path("claims.txt"), None)

    def test_parse_jsonl_rows(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.jsonl"
            path.write_text(
                "\n".join(
                    [
                        json.dumps(
                            {
                                "id": "c1",
                                "doi": "10.1000/a",
                                "claim": "This paper reports 95% accuracy.",
                                "source": "crossref",
                                "note": "draft",
                            }
                        ),
                        json.dumps(
                            {
                                "doi": "10.1000/b",
                                "claim": "This study included 12 patients.",
                            }
                        ),
                    ]
                ),
                encoding="utf-8",
            )

            rows = parse_claim_file(path, None)

        self.assertEqual(
            rows,
            [
                ClaimInputRow(
                    row_number=1,
                    id="c1",
                    doi="10.1000/a",
                    claim="This paper reports 95% accuracy.",
                    source="crossref",
                    note="draft",
                ),
                ClaimInputRow(
                    row_number=2,
                    id=None,
                    doi="10.1000/b",
                    claim="This study included 12 patients.",
                    source="auto",
                    note=None,
                ),
            ],
        )

    def test_parse_csv_rows(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.csv"
            path.write_text(
                "id,doi,claim,source\n"
                "c1,10.1000/a,This paper reports 95% accuracy.,crossref\n",
                encoding="utf-8",
            )

            rows = parse_claim_file(path, None)

        self.assertEqual(rows[0].id, "c1")
        self.assertEqual(rows[0].doi, "10.1000/a")
        self.assertEqual(rows[0].source, "crossref")

    def test_missing_required_field_is_rejected_with_row_number(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.jsonl"
            path.write_text(json.dumps({"doi": "10.1000/a"}) + "\n", encoding="utf-8")

            with self.assertRaisesRegex(BatchInputError, "line 1.*claim"):
                parse_claim_file(path, None)

    def test_invalid_source_is_rejected(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.jsonl"
            path.write_text(
                json.dumps(
                    {
                        "doi": "10.1000/a",
                        "claim": "This paper reports 95% accuracy.",
                        "source": "wikipedia",
                    }
                )
                + "\n",
                encoding="utf-8",
            )

            with self.assertRaisesRegex(BatchInputError, "line 1.*source"):
                parse_claim_file(path, None)

    def test_summarize_results_counts_verdicts_and_statuses(self):
        results = [
            BatchRowResult(row=ClaimInputRow(1, "a", "10.1000/a", "claim a", "auto", None), payload={"verdict": "ACCEPT", "status": "SUPPORTED"}),
            BatchRowResult(row=ClaimInputRow(2, "b", "10.1000/b", "claim b", "auto", None), payload={"verdict": "WARN", "status": "PARTIAL"}),
            BatchRowResult(row=ClaimInputRow(3, "c", "10.1000/c", "claim c", "auto", None), payload={"verdict": "WARN", "status": "UNVERIFIABLE"}),
        ]

        summary = summarize_results(results)

        self.assertEqual(
            summary,
            BatchSummary(total=3, accept=1, warn=2, reject=0, partial=1, unverifiable=1, failed=0),
        )

• [ ] Step 2: Run tests to confirm failure

Run:

PYTHONPATH=src python3 -m unittest tests.test_batch -v

Expected: import failure because ref_verify.batch does not exist.

• [ ] Step 3: Implement parser and model helpers

Create src/ref_verify/batch.py:

from __future__ import annotations

import csv
import json
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Literal

BatchFormat = Literal["jsonl", "csv"]
_VALID_SOURCES = {"auto", "crossref", "semantic-scholar", "pubmed"}


class BatchInputError(ValueError):
    pass


@dataclass(frozen=True)
class ClaimInputRow:
    row_number: int
    id: str | None
    doi: str
    claim: str
    source: str = "auto"
    note: str | None = None

    def to_dict(self) -> dict[str, Any]:
        return asdict(self)


@dataclass(frozen=True)
class BatchRowResult:
    row: ClaimInputRow
    payload: dict[str, Any]

    def to_dict(self) -> dict[str, Any]:
        result = {
            "row_number": self.row.row_number,
            "id": self.row.id,
            "doi": self.row.doi,
            "claim": self.row.claim,
        }
        if self.row.note is not None:
            result["note"] = self.row.note
        result.update(self.payload)
        return result


@dataclass(frozen=True)
class BatchSummary:
    total: int
    accept: int
    warn: int
    reject: int
    partial: int
    unverifiable: int
    failed: int

    def to_dict(self) -> dict[str, int]:
        return asdict(self)


def detect_format(path: Path, explicit_format: str | None) -> BatchFormat:
    if explicit_format in ("jsonl", "csv"):
        return explicit_format
    if explicit_format is not None:
        raise BatchInputError(f"Unsupported input format: {explicit_format}")
    suffix = path.suffix.lower()
    if suffix == ".jsonl":
        return "jsonl"
    if suffix == ".csv":
        return "csv"
    raise BatchInputError("Unsupported input format; use .jsonl, .csv, or --format")


def parse_claim_file(path: Path, explicit_format: str | None) -> list[ClaimInputRow]:
    batch_format = detect_format(path, explicit_format)
    try:
        if batch_format == "jsonl":
            return _parse_jsonl(path)
        return _parse_csv(path)
    except OSError as exc:
        raise BatchInputError(f"Could not read input file: {exc}") from exc


def summarize_results(results: list[BatchRowResult]) -> BatchSummary:
    accept = warn = reject = partial = unverifiable = failed = 0
    for result in results:
        verdict = str(result.payload.get("verdict", ""))
        status = str(result.payload.get("status", ""))
        if verdict == "ACCEPT":
            accept += 1
        if verdict == "WARN":
            warn += 1
        if verdict == "REJECT":
            reject += 1
        if status == "PARTIAL":
            partial += 1
        if status == "UNVERIFIABLE":
            unverifiable += 1
        if verdict == "ERROR":
            failed += 1
    return BatchSummary(
        total=len(results),
        accept=accept,
        warn=warn,
        reject=reject,
        partial=partial,
        unverifiable=unverifiable,
        failed=failed,
    )


def batch_payload(results: list[BatchRowResult]) -> dict[str, Any]:
    return {
        "summary": summarize_results(results).to_dict(),
        "results": [result.to_dict() for result in results],
    }


def render_batch_text(results: list[BatchRowResult]) -> str:
    summary = summarize_results(results)
    lines = [
        (
            f"Summary: total={summary.total} accept={summary.accept} "
            f"warn={summary.warn} reject={summary.reject} "
            f"partial={summary.partial} unverifiable={summary.unverifiable}"
        )
    ]
    for result in results:
        payload = result.payload
        label = str(payload.get("verdict", "WARN"))
        row_id = result.row.id or f"row-{result.row.row_number}"
        lines.extend(
            [
                "",
                f"{label}  {row_id}  {result.row.doi}",
                f"Claim: {result.row.claim}",
                f"Reason: {payload.get('reason', '')}",
            ]
        )
        evidence = payload.get("evidence")
        if evidence:
            lines.append(f"Evidence: {evidence}")
        error_code = payload.get("error_code")
        if error_code:
            lines.append(f"Error code: {error_code}")
    return "\n".join(lines)


def _parse_jsonl(path: Path) -> list[ClaimInputRow]:
    rows: list[ClaimInputRow] = []
    with path.open("r", encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, start=1):
            stripped = line.strip()
            if not stripped:
                continue
            try:
                raw = json.loads(stripped)
            except json.JSONDecodeError as exc:
                raise BatchInputError(f"Invalid JSON on line {line_number}: {exc.msg}") from exc
            if not isinstance(raw, dict):
                raise BatchInputError(f"Invalid row on line {line_number}: expected object")
            rows.append(_row_from_mapping(raw, line_number=line_number, row_label="line"))
    return rows


def _parse_csv(path: Path) -> list[ClaimInputRow]:
    rows: list[ClaimInputRow] = []
    with path.open("r", encoding="utf-8", newline="") as handle:
        reader = csv.DictReader(handle)
        if reader.fieldnames is None:
            raise BatchInputError("CSV input is missing a header row")
        for row_number, raw in enumerate(reader, start=2):
            rows.append(_row_from_mapping(raw, line_number=row_number, row_label="line"))
    return rows


def _row_from_mapping(raw: dict[str, Any], *, line_number: int, row_label: str) -> ClaimInputRow:
    doi = _required_string(raw, "doi", line_number, row_label)
    claim = _required_string(raw, "claim", line_number, row_label)
    source = _optional_string(raw, "source") or "auto"
    if source not in _VALID_SOURCES:
        raise BatchInputError(f"Invalid source on {row_label} {line_number}: {source}")
    return ClaimInputRow(
        row_number=line_number,
        id=_optional_string(raw, "id"),
        doi=doi,
        claim=claim,
        source=source,
        note=_optional_string(raw, "note"),
    )


def _required_string(raw: dict[str, Any], field: str, line_number: int, row_label: str) -> str:
    value = raw.get(field)
    if not isinstance(value, str) or not value.strip():
        raise BatchInputError(f"Missing required field on {row_label} {line_number}: {field}")
    return value.strip()


def _optional_string(raw: dict[str, Any], field: str) -> str | None:
    value = raw.get(field)
    if value is None:
        return None
    if not isinstance(value, str):
        return str(value)
    stripped = value.strip()
    return stripped or None

• [ ] Step 4: Run parser tests

Run:

PYTHONPATH=src python3 -m unittest tests.test_batch -v

Expected: all BatchParserTests pass.

• [ ] Step 5: Commit

git add src/ref_verify/batch.py tests/test_batch.py
git commit -m "Add batch claim input parsing"

Task 2: Shared Single-Claim Execution

Files:

• Modify: src/ref_verify/cli.py
• Test: tests/test_cli.py

• [ ] Step 1: Add a regression test for existing single-claim behavior

Add this test near existing check-claim tests in tests/test_cli.py:

    def test_check_claim_still_outputs_same_json_shape_after_helper_extraction(self):
        record = PaperRecord(
            doi="10.1000/helper",
            title="Helper extraction",
            authors=["Lee"],
            year=2024,
            abstract="The model achieved 95% accuracy.",
            source="fixture",
        )
        output = io.StringIO()

        with redirect_stdout(output):
            exit_code = main(
                [
                    "check-claim",
                    "10.1000/helper",
                    "--claim",
                    "The model achieved 95% accuracy.",
                    "--json",
                ],
                client=FakeClient(record),
                abstract_clients=[],
            )

        payload = json.loads(output.getvalue())
        self.assertEqual(exit_code, 0)
        self.assertEqual(payload["verdict"], "ACCEPT")
        self.assertEqual(payload["status"], "SUPPORTED")
        self.assertEqual(payload["error_code"], "CLAIM_SUPPORTED")
        self.assertEqual(payload["abstract_source"], "crossref")
        self.assertIn("source_attempts", payload)

• [ ] Step 2: Run the regression test before refactor

Run:

PYTHONPATH=src python3 -m unittest tests.test_cli.CliTests.test_check_claim_still_outputs_same_json_shape_after_helper_extraction -v

Expected: PASS before refactor.

• [ ] Step 3: Extract shared helper in `cli.py`

Modify src/ref_verify/cli.py so _check_claim delegates to this helper:

def _run_claim_check(
    doi: str,
    claim: str,
    source: str,
    client: CrossrefClient,
    fallback_clients: Sequence[AbstractSourceClient],
) -> dict:
    lookup_doi = normalize_doi(doi)
    selected_clients = _select_abstract_clients(fallback_clients, source)
    if source in ("auto", "crossref"):
        fetched = client.fetch_work(lookup_doi)
        lookup_result = lookup_abstract(lookup_doi, fetched, selected_clients)
    else:
        lookup_result = lookup_selected_abstract(lookup_doi, selected_clients)
    if lookup_result.error_code == "DOI_MISMATCH":
        result = ClaimSupportResult(
            status="UNVERIFIABLE",
            verdict="WARN",
            reason="Fetched DOI does not match the requested DOI.",
            evidence="",
            paper=lookup_result.record,
            claim=claim,
        )
        return _claim_payload(result, lookup_result)

    result = check_claim_support(lookup_result.record, claim)
    return _claim_payload(result, lookup_result)

Then replace _check_claim body with:

def _check_claim(
    args: argparse.Namespace,
    client: CrossrefClient,
    fallback_clients: Sequence[AbstractSourceClient],
) -> int:
    payload = _run_claim_check(args.doi, args.claim, args.source, client, fallback_clients)
    _emit(payload, as_json=args.json)
    return 0 if payload.get("verdict") == "ACCEPT" else 2

• [ ] Step 4: Run targeted CLI tests

Run:

PYTHONPATH=src python3 -m unittest tests.test_cli.CliTests.test_check_claim_still_outputs_same_json_shape_after_helper_extraction tests.test_cli.CliTests.test_check_claim_normalizes_prefixed_doi_before_fetching -v

Expected: PASS.

• [ ] Step 5: Commit

git add src/ref_verify/cli.py tests/test_cli.py
git commit -m "Extract reusable claim check execution"

Task 3: JSONL check-file CLI

Files:

• Modify: src/ref_verify/cli.py
• Modify: tests/test_cli.py

• [ ] Step 1: Write failing JSONL CLI tests

Add these tests to CliTests in tests/test_cli.py:

    def test_check_file_jsonl_outputs_json_summary(self):
        record = PaperRecord(
            doi="10.1000/batch",
            title="Batch paper",
            authors=["Lee"],
            year=2024,
            abstract="The model achieved 95% accuracy.",
            source="fixture",
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.jsonl"
            path.write_text(
                json.dumps(
                    {
                        "id": "c1",
                        "doi": "10.1000/batch",
                        "claim": "The model achieved 95% accuracy.",
                    }
                )
                + "\n",
                encoding="utf-8",
            )
            output = io.StringIO()

            with redirect_stdout(output):
                exit_code = main(
                    ["check-file", str(path), "--json"],
                    client=FakeClient(record),
                    abstract_clients=[],
                )

        payload = json.loads(output.getvalue())
        self.assertEqual(exit_code, 0)
        self.assertEqual(payload["summary"]["total"], 1)
        self.assertEqual(payload["summary"]["accept"], 1)
        self.assertEqual(payload["results"][0]["id"], "c1")
        self.assertEqual(payload["results"][0]["verdict"], "ACCEPT")

    def test_check_file_jsonl_exits_two_when_any_claim_warns(self):
        record = PaperRecord(
            doi="10.1000/batch-warn",
            title="Batch warning",
            authors=["Lee"],
            year=2024,
            abstract="The model achieved 90% accuracy.",
            source="fixture",
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.jsonl"
            path.write_text(
                json.dumps(
                    {
                        "doi": "10.1000/batch-warn",
                        "claim": "The model achieved 95% accuracy.",
                    }
                )
                + "\n",
                encoding="utf-8",
            )
            output = io.StringIO()

            with redirect_stdout(output):
                exit_code = main(
                    ["check-file", str(path), "--json"],
                    client=FakeClient(record),
                    abstract_clients=[],
                )

        payload = json.loads(output.getvalue())
        self.assertEqual(exit_code, 2)
        self.assertEqual(payload["summary"]["warn"], 1)
        self.assertEqual(payload["results"][0]["status"], "PARTIAL")

Also add imports at the top of tests/test_cli.py:

import tempfile
from pathlib import Path

• [ ] Step 2: Run tests to confirm failure

Run:

PYTHONPATH=src python3 -m unittest tests.test_cli.CliTests.test_check_file_jsonl_outputs_json_summary tests.test_cli.CliTests.test_check_file_jsonl_exits_two_when_any_claim_warns -v

Expected: FAIL because check-file is not registered.

• [ ] Step 3: Wire `check-file` command

Modify imports in src/ref_verify/cli.py:

from pathlib import Path

from ref_verify.batch import (
    BatchInputError,
    BatchRowResult,
    batch_payload,
    parse_claim_file,
    render_batch_text,
)

Add parser branch in _build_parser():

    check_file = subparsers.add_parser("check-file", help="Check claims from a JSONL or CSV file")
    check_file.add_argument("path")
    check_file.add_argument("--format", choices=("jsonl", "csv"))
    check_file.add_argument("--json", action="store_true")

Add dispatch in main():

        if args.command == "check-file":
            return _check_file(args, lookup_client, fallback_clients)

Add _check_file:

def _check_file(
    args: argparse.Namespace,
    client: CrossrefClient,
    fallback_clients: Sequence[AbstractSourceClient],
) -> int:
    try:
        rows = parse_claim_file(Path(args.path), args.format)
    except BatchInputError as exc:
        _emit({"error": str(exc)}, as_json=args.json)
        return 1

    results = [
        BatchRowResult(
            row=row,
            payload=_run_claim_check(row.doi, row.claim, row.source, client, fallback_clients),
        )
        for row in rows
    ]
    payload = batch_payload(results)
    if args.json:
        _emit(payload, as_json=True)
    else:
        print(render_batch_text(results))
    summary = payload["summary"]
    return 0 if summary["total"] == summary["accept"] else 2

• [ ] Step 4: Run JSONL CLI tests

Run:

PYTHONPATH=src python3 -m unittest tests.test_cli.CliTests.test_check_file_jsonl_outputs_json_summary tests.test_cli.CliTests.test_check_file_jsonl_exits_two_when_any_claim_warns -v

Expected: PASS.

• [ ] Step 5: Commit

git add src/ref_verify/cli.py tests/test_cli.py
git commit -m "Add JSONL batch claim CLI"

Task 4: CSV, Format Override, And Human Output

Files:

• Modify: tests/test_cli.py
• Modify: src/ref_verify/batch.py
• Modify: src/ref_verify/cli.py

• [ ] Step 1: Write CLI tests for CSV, format override, and invalid input

Add these tests to CliTests:

    def test_check_file_csv_outputs_human_summary(self):
        record = PaperRecord(
            doi="10.1000/csv",
            title="CSV paper",
            authors=["Lee"],
            year=2024,
            abstract="The experiment was conducted at 37 °C.",
            source="fixture",
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.csv"
            path.write_text(
                "id,doi,claim\n"
                "temp,10.1000/csv,The experiment was conducted at 37 °C.\n",
                encoding="utf-8",
            )
            output = io.StringIO()

            with redirect_stdout(output):
                exit_code = main(
                    ["check-file", str(path)],
                    client=FakeClient(record),
                    abstract_clients=[],
                )

        text = output.getvalue()
        self.assertEqual(exit_code, 0)
        self.assertIn("Summary: total=1 accept=1", text)
        self.assertIn("ACCEPT  temp  10.1000/csv", text)
        self.assertIn("Claim: The experiment was conducted at 37 °C.", text)

    def test_check_file_accepts_format_override(self):
        record = PaperRecord(
            doi="10.1000/override",
            title="Override paper",
            authors=["Lee"],
            year=2024,
            abstract="The study included 12 patients.",
            source="fixture",
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.txt"
            path.write_text(
                json.dumps(
                    {
                        "doi": "10.1000/override",
                        "claim": "The study included 12 patients.",
                    }
                )
                + "\n",
                encoding="utf-8",
            )
            output = io.StringIO()

            with redirect_stdout(output):
                exit_code = main(
                    ["check-file", str(path), "--format", "jsonl", "--json"],
                    client=FakeClient(record),
                    abstract_clients=[],
                )

        payload = json.loads(output.getvalue())
        self.assertEqual(exit_code, 0)
        self.assertEqual(payload["summary"]["accept"], 1)

    def test_check_file_invalid_input_returns_json_error(self):
        record = PaperRecord(
            doi="10.1000/error",
            title="Error paper",
            authors=["Lee"],
            year=2024,
            abstract="The model achieved 95% accuracy.",
            source="fixture",
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "claims.jsonl"
            path.write_text("{bad json}\n", encoding="utf-8")
            output = io.StringIO()

            with redirect_stdout(output):
                exit_code = main(
                    ["check-file", str(path), "--json"],
                    client=FakeClient(record),
                    abstract_clients=[],
                )

        payload = json.loads(output.getvalue())
        self.assertEqual(exit_code, 1)
        self.assertIn("Invalid JSON on line 1", payload["error"])

• [ ] Step 2: Run tests

Run:

PYTHONPATH=src python3 -m unittest tests.test_cli.CliTests.test_check_file_csv_outputs_human_summary tests.test_cli.CliTests.test_check_file_accepts_format_override tests.test_cli.CliTests.test_check_file_invalid_input_returns_json_error -v

Expected: PASS.

• [ ] Step 3: Commit

git add src/ref_verify/batch.py src/ref_verify/cli.py tests/test_cli.py
git commit -m "Add CSV and human batch output"

Task 5: Fixture-Backed Numeric Eval

Files:

• Create: tests/fixtures/numeric_claim_eval.jsonl
• Modify: tests/test_numeric_claim.py

• [ ] Step 1: Add eval fixture

Create tests/fixtures/numeric_claim_eval.jsonl:

{"id":"materials-accept","domain":"materials","abstract":"The actuator demonstrated actuation strain above 120% under cyclic loading.","claim":"The actuator demonstrated actuation strain above 100%.","expected_status":"SUPPORTED","why":"Same subject, percent unit, and comparator entailment are explicit."}
{"id":"biomed-accept","domain":"biomedicine","abstract":"The study enrolled 12 patients with confirmed disease.","claim":"The study enrolled 12 patients.","expected_status":"SUPPORTED","why":"Patient count is explicit and subject-bound."}
{"id":"ml-accept","domain":"machine-learning","abstract":"The classifier achieved 95% accuracy on the held-out test set.","claim":"The classifier achieved 95% accuracy.","expected_status":"SUPPORTED","why":"Accuracy percentage is explicit and subject-bound."}
{"id":"chemistry-accept","domain":"chemistry","abstract":"Samples were incubated at 37 °C for 2 h before analysis.","claim":"Samples were incubated at 37 °C.","expected_status":"SUPPORTED","why":"Temperature and subject are explicit in the same clause."}
{"id":"general-wrong-subject","domain":"general-science","abstract":"Group A included 12 patients, while group B included 20 patients.","claim":"Group B included 12 patients.","expected_status":"PARTIAL","why":"The number exists, but it belongs to the wrong subject."}
{"id":"materials-wrong-comparator","domain":"materials","abstract":"The device survived up to 3000 cycles before failure.","claim":"The device survived at least 5000 cycles.","expected_status":"PARTIAL","why":"The unit matches, but comparator and value do not support the claim."}
{"id":"ml-multiple-numbers","domain":"machine-learning","abstract":"Model A achieved 95% accuracy, whereas Model B achieved 88% accuracy.","claim":"Model B achieved 95% accuracy.","expected_status":"PARTIAL","why":"Same sentence contains the number, but subject binding would be wrong."}

• [ ] Step 2: Add fixture test

Append to tests/test_numeric_claim.py:

import json
from pathlib import Path


class NumericClaimEvalFixtureTests(unittest.TestCase):
    def test_numeric_claim_eval_fixture(self):
        fixture = Path(__file__).parent / "fixtures" / "numeric_claim_eval.jsonl"
        with fixture.open("r", encoding="utf-8") as handle:
            rows = [json.loads(line) for line in handle if line.strip()]

        self.assertGreaterEqual(len(rows), 7)
        self.assertEqual(
            {row["domain"] for row in rows},
            {"materials", "biomedicine", "machine-learning", "chemistry", "general-science"},
        )

        for row in rows:
            with self.subTest(row=row["id"]):
                result = check_numeric_claim_support(row["abstract"], row["claim"])
                self.assertEqual(result.status, row["expected_status"], row["why"])

If tests/test_numeric_claim.py already imports json, Path, or unittest, merge imports instead of duplicating them.

• [ ] Step 3: Run numeric fixture tests

Run:

PYTHONPATH=src python3 -m unittest tests.test_numeric_claim.NumericClaimEvalFixtureTests -v

Expected: PASS. If a false-accept defense row fails as SUPPORTED, stop and fix numeric_claim.py before continuing.

• [ ] Step 4: Commit

git add tests/fixtures/numeric_claim_eval.jsonl tests/test_numeric_claim.py
git commit -m "Add numeric claim eval fixture"

Task 6: Documentation, Full Verification, And Release Prep

Files:

• Modify: README.md
• Modify: README.ko.md
• Modify: CHANGELOG.md

• [ ] Step 1: Update English README

Add a "Batch claim checks" section to README.md near the existing CLI usage:

````markdown

Batch claim checks

Use check-file when you have many DOI/claim pairs from a draft, literature note, or AI-agent output.

JSONL:

ref-verify check-file claims.jsonl
ref-verify check-file claims.jsonl --json

CSV:

ref-verify check-file claims.csv

Each row must include doi and claim. Optional fields are id, source, and note.

Batch mode reuses the same conservative check-claim engine. ACCEPT means the abstract explicitly supports the numeric claim. WARN, PARTIAL, REJECT, or UNVERIFIABLE means the claim should not be treated as verified. ````

• [ ] Step 2: Update Korean README

Add the Korean equivalent to README.ko.md:

````markdown

여러 claim 한 번에 확인하기

초안, 리서치 메모, AI 에이전트 출력처럼 DOI/claim 쌍이 여러 개 있을 때는 check-file을 사용합니다.

JSONL:

ref-verify check-file claims.jsonl
ref-verify check-file claims.jsonl --json

CSV:

ref-verify check-file claims.csv

각 행에는 doi와 claim이 필요합니다. id, source, note는 선택 필드입니다.

배치 모드는 기존의 보수적인 check-claim 엔진을 그대로 사용합니다. ACCEPT는 abstract가 숫자 claim을 명시적으로 지지한다는 뜻입니다. WARN, PARTIAL, REJECT, UNVERIFIABLE은 검증된 claim으로 취급하면 안 됩니다. ````

• [ ] Step 3: Update changelog

Add to the top of CHANGELOG.md:

## Unreleased

- Add `check-file` batch workflow for JSONL and CSV DOI/claim inputs.
- Add fixture-backed numeric claim eval coverage for repeated-use workflows.

• [ ] Step 4: Run full local verification

Run:

PYTHONPATH=src python3 -m unittest discover -s tests -v
python3 -m py_compile src/ref_verify/*.py tests/*.py scripts/*.py
python3 -m build --sdist --wheel
python3 -m twine check dist/*
python3 scripts/package_smoke.py --wheel dist/ref_verify-*.whl --expected-version "$(python3 -c 'import pathlib, tomllib; print(tomllib.loads(pathlib.Path("pyproject.toml").read_text())["project"]["version"])')"
git diff --check

Expected:

• Unit tests pass.
• py_compile exits 0.
• build creates one source distribution and one wheel.
• twine check passes.
• package smoke passes and confirms SKILL.md is not packaged.
• git diff --check exits 0.

• [ ] Step 5: Commit docs

git add README.md README.ko.md CHANGELOG.md
git commit -m "Document batch claim workflow"

Final Review Checklist

• [ ] check-claim output shape and exit codes are unchanged.
• [ ] check-file calls the same claim execution helper as check-claim.
• [ ] JSONL and CSV parsing use structured parsers, not manual splitting.
• [ ] Unknown file formats fail closed unless --format is provided.
• [ ] Malformed rows return exit 1 and do not get silently skipped.
• [ ] Batch exit 0 only happens when every row is ACCEPT.
• [ ] Human output includes row-level reason/evidence, especially for non-accepted rows.
• [ ] JSON output includes summary counts and row results.
• [ ] Eval fixture includes at least five domains and false-accept defense cases.
• [ ] README English and Korean explain batch mode and conservative verdict interpretation.

Plan Self-Review

• Spec coverage: JSONL, CSV, format override, row validation, exit codes, JSON output, human output, eval fixtures, documentation, and verification are covered by tasks.
• Placeholder scan: no placeholder markers or vague "add tests" steps remain.
• Type consistency: plan uses ClaimInputRow, BatchRowResult, BatchSummary, and BatchInputError consistently from Task 1 through CLI integration.
• Residual risk: the fixture rows may expose existing numeric-claim false accepts. If Task 5 fails, fix the numeric engine before documenting the workflow as complete.

Numeric Claim Verifier Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a conservative, cross-domain numeric claim checker for explicit abstract evidence.

Architecture: Create src/ref_verify/numeric_claim.py to parse numeric claims and evidence clauses, then route numeric claims through it from claim_check.py. Keep existing text fallback and actuation-strain false-accept guards.

Tech Stack: Python 3.10 standard library, unittest, existing dataclass result model.

---

Task 1: Add Numeric Claim Engine Skeleton

Files:

• Create: src/ref_verify/numeric_claim.py
• Create: tests/test_numeric_claim.py

• [ ] Step 1: Write failing tests for percent and unit claim extraction

Add tests that call check_numeric_claim_support() with abstracts and claims:

def test_accepts_subject_matched_percent_claim():
    result = check_numeric_claim_support("Device efficiency reached 95%.", "device efficiency above 90%")
    self.assertEqual(result.status, "SUPPORTED")

def test_accepts_subject_matched_unit_claim():
    result = check_numeric_claim_support("The actuator survived 5000 cycles.", "actuator survived at least 4000 cycles")
    self.assertEqual(result.status, "SUPPORTED")

• [ ] Step 2: Run tests and verify failure

Run: PYTHONPATH=src python3 -m unittest tests.test_numeric_claim -v

Expected: import or missing-function failure.

• [ ] Step 3: Implement minimal public API

Create NumericClaimResult and check_numeric_claim_support(abstract: str, claim: str).

• [ ] Step 4: Run tests and verify pass

Run: PYTHONPATH=src python3 -m unittest tests.test_numeric_claim -v

Expected: all new tests pass.

Task 2: Add Subject Binding and False-Accept Guards

Files:

• Modify: src/ref_verify/numeric_claim.py
• Modify: tests/test_numeric_claim.py

• [ ] Step 1: Write failing tests for mixed-subject sentences

Add cases where the supporting number belongs to the wrong subject:

def test_rejects_wrong_subject_number_in_same_sentence():
    result = check_numeric_claim_support(
        "Device efficiency reached 80%, and response rate was 95%.",
        "device efficiency above 90%",
    )
    self.assertEqual(result.status, "PARTIAL")

• [ ] Step 2: Run tests and verify failure

Run: PYTHONPATH=src python3 -m unittest tests.test_numeric_claim -v

Expected: wrong-subject test fails before guard implementation.

• [ ] Step 3: Require subject terms in the same clause

Split evidence into clauses before extracting evidence. Accept only when claim subject terms appear in the clause that owns the numeric value.

• [ ] Step 4: Run tests and verify pass

Run: PYTHONPATH=src python3 -m unittest tests.test_numeric_claim -v

Expected: positive and false-accept tests pass.

Task 3: Route Numeric Claims Through Claim Check

Files:

• Modify: src/ref_verify/claim_check.py
• Modify: tests/test_claim_check.py
• Modify: tests/test_cli.py

• [ ] Step 1: Write failing integration tests

Add check_claim_support() and CLI tests for unit/count claims:

def test_unit_claim_is_supported_when_subject_and_unit_match():
    record = PaperRecord(..., abstract="The actuator survived 5000 cycles.")
    result = check_claim_support(record, "actuator survived at least 4000 cycles")
    self.assertEqual(result.status, "SUPPORTED")

• [ ] Step 2: Run integration tests and verify failure

Run: PYTHONPATH=src python3 -m unittest tests.test_claim_check tests.test_cli -v

Expected: unit/count integration tests fail before routing.

• [ ] Step 3: Call numeric engine before text fallback

In claim_check.py, call check_numeric_claim_support(record.abstract, claim). Convert SUPPORTED to ClaimSupportResult(status="SUPPORTED", verdict="ACCEPT", ...); otherwise continue existing fallback.

• [ ] Step 4: Run integration tests and verify pass

Run: PYTHONPATH=src python3 -m unittest tests.test_claim_check tests.test_cli -v

Expected: all integration tests pass.

Task 4: Update Documentation and Final Verification

Files:

• Modify: README.md
• Modify: README.ko.md
• Modify: tests/test_skill_docs.py

• [ ] Step 1: Write docs assertions

Assert README mentions explicit numeric evidence, percent claims, unit/count claims, and deferred statistical metrics.

• [ ] Step 2: Update README copy

Describe Phase 1 CLI scope as explicit numeric evidence only. Do not imply semantic paper understanding.

• [ ] Step 3: Run full verification

Run:

PYTHONPATH=src python3 -m unittest discover -s tests -v
tmpdir=$(mktemp -d /tmp/ref-verify-install-test.XXXXXX) && python3 -m venv "$tmpdir/venv" && "$tmpdir/venv/bin/python" -m pip install -e . && "$tmpdir/venv/bin/ref-verify" --help

Expected: all tests pass and editable install exposes ref-verify --help.

Release Automation Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add CI, manual live smoke, PyPI publishing guardrails, and package smoke verification.

Architecture: GitHub Actions owns remote automation. A local Python script owns package artifact checks so CI and maintainers can run the same validation. Docs explain the workflows without changing runtime behavior.

Tech Stack: GitHub Actions, Python stdlib, build, twine, pypa/gh-action-pypi-publish.

---

Task 1: Package Smoke Script

Files:

• Create: scripts/package_smoke.py

• [ ] Create a script that accepts --wheel and --expected-version.
• [ ] Create a temporary virtualenv.
• [ ] Install the wheel into that virtualenv.
• [ ] Run ref-verify --help and ref-verify check-claim --help.
• [ ] Import ref_verify.__version__ and compare it to the expected version.
• [ ] Inspect the wheel and fail if SKILL.md is packaged.

Task 2: GitHub Actions Workflows

Files:

• Create: .github/workflows/ci.yml
• Create: .github/workflows/live-smoke.yml
• Create: .github/workflows/publish-pypi.yml

• [ ] CI runs on push and pull request with Python 3.10 and 3.12.
• [ ] CI runs unit tests, byte compilation, build, twine check, and package smoke.
• [ ] Live smoke is manual-only and runs representative DOI checks.
• [ ] PyPI publishing runs only on published v* GitHub releases and uses OIDC trusted publishing.

Task 3: Contributor Documentation

Files:

• Modify: CONTRIBUTING.md
• Modify: README.md
• Modify: README.ko.md

• [ ] Document local verification commands.
• [ ] Explain that live smoke is manual because it hits public APIs.
• [ ] Explain that PyPI publishing requires trusted publisher setup.

Task 4: Verification

• [ ] Run PYTHONPATH=src python3 -m unittest discover -s tests -v.
• [ ] Run python3 -m py_compile src/ref_verify/*.py tests/*.py scripts/*.py.
• [ ] Build wheel and sdist.
• [ ] Run twine check.
• [ ] Run python3 scripts/package_smoke.py --wheel <built wheel> --expected-version 1.1.1.
• [ ] Run git diff --check.

Abstract Source Fallback and Error Verdicts Design

Goal

Make ref-verify check-claim useful across more research domains by reducing false UNVERIFIABLE results caused by missing CrossRef abstracts, and by returning clearer machine-readable reasons when evidence cannot be checked.

This is the next practical layer after the numeric claim verifier. The numeric engine can only help when an abstract is available. In real use, the bigger bottleneck is often source coverage and vague failure reporting.

Positioning

ref-verify remains a conservative abstract-grounded citation guard.

Fallback sources are allowed only to find an abstract for the same DOI. They must not broaden the tool into paper discovery, title search, semantic retrieval, full-text parsing, or claim inference.

Current Problem

The current CLI fetches a CrossRef work and checks its abstract. This creates three practical gaps:

• Many valid DOIs do not expose an abstract through CrossRef.
• UNVERIFIABLE can mean multiple different things: no abstract, DOI mismatch, failed API call, missing record, or unsupported source behavior.
• Agents and downstream tools cannot reliably decide whether to retry, try another source, ask a human, or reject the citation.

Non-Goals

• Do not use LLMs to infer paper meaning.
• Do not parse full text, tables, figures, supplements, or PDFs.
• Do not search by title or author in Phase 1.
• Do not accept an abstract unless it is tied to the requested DOI.
• Do not let fallback abstracts override CrossRef metadata verification.
• Do not add statistical metric support in this phase.

Source Policy

CrossRef remains the primary metadata source. Fallback sources are abstract sources only.

Recommended source order:

1. CrossRef work by DOI. 2. Semantic Scholar paper lookup by DOI. 3. PubMed lookup by DOI, when the DOI maps cleanly to a PubMed record.

The implementation should verify exact API contracts against official documentation before coding. The design requirement is source behavior, not a fixed endpoint string.

DOI Binding Rules

An abstract source can be used only if one of these is true:

• The returned record explicitly contains the normalized requested DOI.
• The source lookup was a DOI-only endpoint or DOI-only query and returns a single unambiguous record whose DOI matches after normalization.

If the returned source has no DOI, multiple records, or a DOI mismatch, the source attempt must be recorded but ignored for claim verification.

No title-based rescue is allowed in this phase because it risks attaching claims to the wrong paper.

Data Model

Add a source-attempt structure that can be serialized in CLI JSON:

AbstractSourceAttempt
- source: crossref | semantic_scholar | pubmed
- status: FOUND | NO_ABSTRACT | NOT_FOUND | DOI_MISMATCH | API_ERROR | TIMEOUT | UNSUPPORTED
- reason: short human-readable explanation
- record_id: optional source record identifier
- doi: optional normalized DOI returned by source
- elapsed_ms: optional timing field

Add an abstract lookup result:

AbstractLookupResult
- record: PaperRecord
- abstract_source: crossref | semantic_scholar | pubmed | none
- attempts: list[AbstractSourceAttempt]
- error_code: optional final error code

PaperRecord may stay as the canonical claim-check input. If so, source metadata should be attached at the CLI/result layer rather than forcing every claim-check unit test to construct source attempts.

Error Codes

Use explicit error codes in JSON output. These codes should be stable enough for agents to branch on.

NO_ABSTRACT
DOI_NOT_FOUND
DOI_MISMATCH
SOURCE_API_ERROR
SOURCE_TIMEOUT
SOURCE_UNSUPPORTED
CLAIM_NOT_EXPLICIT
CLAIM_AMBIGUOUS
CLAIM_SUPPORTED

Mapping:

• CLAIM_SUPPORTED maps to status=SUPPORTED, verdict=ACCEPT, exit 0.
• CLAIM_AMBIGUOUS maps to status=PARTIAL, verdict=WARN, non-zero exit.
• CLAIM_NOT_EXPLICIT maps to status=PARTIAL, verdict=WARN, non-zero exit.
• NO_ABSTRACT, DOI_NOT_FOUND, DOI_MISMATCH, API, timeout, and unsupported-source codes map to status=UNVERIFIABLE, verdict=WARN, non-zero exit.

Do not collapse source errors into CLAIM_NOT_EXPLICIT. A claim cannot be evaluated when no trusted abstract is available.

CLI Behavior

ref-verify check-claim <doi> --claim "<claim>" --json should:

1. Normalize the requested DOI. 2. Fetch CrossRef metadata. 3. Use CrossRef abstract if available and DOI matches. 4. If CrossRef has no abstract, try fallback abstract sources in order. 5. Run the existing claim checker against the selected abstract. 6. Return JSON containing the selected abstract_source, all source_attempts, and the final error_code.

Human-readable output should stay short:

WARN UNVERIFIABLE: no abstract found for DOI after CrossRef, Semantic Scholar, and PubMed attempts.

JSON output should include enough detail for an agent:

{
  "status": "UNVERIFIABLE",
  "verdict": "WARN",
  "error_code": "NO_ABSTRACT",
  "abstract_source": null,
  "source_attempts": [
    {"source": "crossref", "status": "NO_ABSTRACT", "reason": "CrossRef record had no abstract."},
    {"source": "semantic_scholar", "status": "NO_ABSTRACT", "reason": "Record found, but no abstract was provided."},
    {"source": "pubmed", "status": "NOT_FOUND", "reason": "No PubMed record matched the DOI."}
  ]
}

verify-doi Behavior

verify-doi should remain a CrossRef metadata check in this phase.

Fallback abstract sources must not make a bad title, author, year, or DOI match pass. If future versions expose fallback metadata, that must be designed separately with stricter provenance rules.

Conservative Acceptance Rules

Fallback only changes whether an abstract is available. It must not weaken claim acceptance.

The same claim checker, numeric matcher, subject-binding rules, comparator rules, and ambiguity guards apply regardless of abstract source.

If two sources provide different abstracts, use the first DOI-bound source in the configured order and record later sources only if explicitly requested by a diagnostic option. Do not merge abstracts from multiple sources in this phase.

Test Plan

Use fake source clients. Do not depend on live network in tests.

Required cases:

• CrossRef abstract exists: no fallback source is queried.
• CrossRef has metadata but no abstract, Semantic Scholar has DOI-bound abstract: claim check uses Semantic Scholar.
• CrossRef and Semantic Scholar have no abstract, PubMed has DOI-bound abstract: claim check uses PubMed.
• All sources lack abstracts: result is UNVERIFIABLE with error_code=NO_ABSTRACT and all attempts listed.
• A fallback source returns a different DOI: attempt is DOI_MISMATCH, source is ignored, and checking continues.
• A fallback source returns an API error: attempt is API_ERROR, checking continues to the next source.
• A fallback source times out: attempt is TIMEOUT, checking continues to the next source.
• A fallback abstract is available but claim is not explicit: result is claim-level WARN, not source-level UNVERIFIABLE.
• verify-doi output is unchanged except for any explicitly documented error-code field.

Documentation Updates

Update README and README.ko to say:

• check-claim is DOI-only.
• CrossRef is primary.
• Fallback abstract sources can be used when CrossRef has no abstract.
• ACCEPT means explicit abstract support, not paper-level truth.
• WARN and UNVERIFIABLE are expected conservative outcomes, not necessarily tool failures.

Avoid product copy that implies paper understanding or source completeness.

Implementation Shape

Recommended modules:

• src/ref_verify/abstract_lookup.py
• orchestrates source order
• returns AbstractLookupResult
• src/ref_verify/sources.py
• source client protocols and source-attempt models
• src/ref_verify/semantic_scholar.py
• DOI-bound abstract adapter
• src/ref_verify/pubmed.py
• DOI-bound abstract adapter
• src/ref_verify/cli.py
• wires fallback into check-claim

Keep claim_check.py focused on claim support. It should not know how abstracts are fetched.

Rollout Strategy

Phase A: introduce data models and source attempt JSON for CrossRef only.

Phase B: add Semantic Scholar fallback with fake-client tests and one optional live smoke command documented for maintainers.

Phase C: add PubMed fallback after DOI binding behavior is verified.

Phase D: update README examples and skill instructions.

This staging keeps the CLI stable and avoids mixing fallback networking with claim-matching changes.

Risks

• Fallback sources may return stale, truncated, or license-filtered abstracts.
• DOI matching can be inconsistent across sources.
• PubMed DOI mapping can be indirect for some biomedical records.
• More source attempts can make CLI latency worse.
• Detailed error codes can become a compatibility burden if named too casually.

Mitigation:

• DOI-only lookup.
• No abstract merging.
• Stable source-attempt records.
• Timeouts per source.
• Tests for source mismatch and source failure before positive fallback tests.

Open Decisions

Resolved for the first implementation:

• Fallback is enabled by default for check-claim.
• --source crossref|semantic-scholar|pubmed is available for debugging source-specific behavior; explicit non-CrossRef source selection bypasses CrossRef.
• CI tests use fake clients and parser fixtures only; live smoke tests are excluded from CI.
• UNVERIFIABLE remains verdict=WARN; JSON error_code distinguishes source absence from weak evidence.

Still open:

• Whether to add a maintainer-only live smoke command later.
• Whether future versions should introduce a verdict separate from WARN for source absence.

Revision Notes

Implemented decisions:

• Added a CrossRef-first abstract lookup layer with DOI-bound Semantic Scholar and PubMed fallback.
• Kept verify-doi as CrossRef metadata verification only.
• Added abstract_source, source_attempts, and error_code to check-claim JSON output.
• Added explicit source debugging that can isolate Semantic Scholar or PubMed when CrossRef is unavailable.
• Added fake-client flow tests and source parser tests without live network dependency.

Batch Claim Workflow Design

Goal

Move ref-verify from a single-claim demo flow to a repeatable research workflow by adding batch claim checking, fixture-backed evaluation, and human-readable reporting.

This is not a semantic expansion of the verifier. It keeps the current product boundary: verify DOI-bound, abstract-grounded numeric claims conservatively, and refuse to accept claims that require full-text reading, table/figure parsing, or broad paper understanding.

User Problem

Researchers and AI agents rarely need to verify only one citation claim. A realistic draft, literature note, or agent output may contain many DOI/claim pairs:

• "This paper reports 95% accuracy."
• "The study included 12 patients."
• "The material retained performance after 5000 cycles."
• "The experiment was conducted at 37 °C."

The current check-claim command can verify these one at a time, but that is too slow for repeated use and awkward for agent pipelines. The next product step should make the same conservative verifier usable across many claims without widening the acceptance logic.

Non-Goals

• Do not add LLM-based semantic inference.
• Do not parse full text, tables, or figures.
• Do not add p-value, AUC/AUROC, F1, hazard ratio, odds ratio, or 95% CI metric binding in this phase.
• Do not judge paper quality, claim importance, or field consensus.
• Do not automatically rewrite claims into a form that can pass.
• Do not silently treat missing abstracts as failures of the user rather than UNVERIFIABLE results.

Proposed CLI

Add a new command:

ref-verify check-file claims.jsonl
ref-verify check-file claims.csv
ref-verify check-file claims.txt --format jsonl

Supported input formats:

• JSONL: one object per line.
• CSV: one row per claim.

Format detection:

• Infer jsonl from .jsonl.
• Infer csv from .csv.
• Allow --format jsonl or --format csv when the extension is absent or nonstandard.
• Reject unknown formats with exit 1.

Required fields:

• doi
• claim

Optional fields:

• id: caller-provided stable identifier.
• source: auto, crossref, semantic-scholar, or pubmed.
• note: ignored by the verifier but preserved in JSON output when present.

Example JSONL:

{"id":"c1","doi":"10.xxxx/example-a","claim":"This paper reports 95% accuracy."}
{"id":"c2","doi":"10.xxxx/example-b","claim":"This study included 12 patients."}
{"id":"c3","doi":"10.xxxx/example-c","claim":"The material retained performance after 5000 cycles."}

Example CSV:

id,doi,claim
c1,10.xxxx/example-a,This paper reports 95% accuracy.
c2,10.xxxx/example-b,This study included 12 patients.
c3,10.xxxx/example-c,The material retained performance after 5000 cycles.

Output Modes

The default output should be human-readable and compact:

WARN  c2  10.xxxx/example-b
Claim: This study included 12 patients.
Reason: The abstract contains the number, but the subject binding is ambiguous.
Evidence: "...12 patients..."

JSON output should remain available for agents and CI:

ref-verify check-file claims.jsonl --json

JSON shape:

{
  "summary": {
    "total": 3,
    "accept": 1,
    "warn": 2,
    "reject": 0,
    "partial": 2,
    "unverifiable": 0,
    "failed": 0
  },
  "results": [
    {
      "id": "c1",
      "doi": "10.xxxx/example-a",
      "claim": "This paper reports 95% accuracy.",
      "verdict": "ACCEPT",
      "status": "SUPPORTED",
      "reason": "...",
      "evidence": "...",
      "abstract_source": "crossref",
      "error_code": "CLAIM_SUPPORTED",
      "source_attempts": []
    }
  ]
}

Exit Codes

Batch mode should distinguish "the command ran" from "every claim was accepted":

• Exit 0: command completed and every valid row returned ACCEPT.
• Exit 2: command completed, but at least one valid row returned a non-accepting verdict or status, such as WARN, REJECT, PARTIAL, or UNVERIFIABLE.
• Exit 1: command failed due to invalid input, unreadable file, malformed JSON/CSV, missing required fields, or an unexpected runtime error.

Invalid rows should not be silently skipped. If any row is malformed, return exit 1 and include row-level diagnostics. This prevents users from thinking a partial input file was fully checked.

Result Semantics

Reuse the existing single-claim logic for every row. Batch mode must not introduce a second verdict system.

ACCEPT means the existing verifier found explicit abstract evidence for the DOI-bound claim.

WARN is the non-accepting claim verdict currently used for ambiguous or unverifiable claim checks.

REJECT should be preserved if the existing single-claim path returns it for a hard mismatch.

PARTIAL means the command found some relevant numeric evidence but the subject, unit, comparator, or sentence/clause binding is not clear enough to accept.

UNVERIFIABLE means the DOI or abstract evidence could not be checked.

The command may summarize results, but it must not turn WARN, REJECT, PARTIAL, or UNVERIFIABLE into softer words such as "probably supported."

Architecture

Keep src/ref_verify/cli.py as the command entry point, but avoid embedding parsing and rendering logic directly into the command branch.

Add:

• src/ref_verify/batch.py
• parse JSONL and CSV inputs
• validate required fields
• normalize optional row fields
• call the existing claim-check flow row by row
• produce a structured batch result
• tests/test_batch.py
• parser tests
• row validation tests
• exit-code tests through the CLI
• JSON output shape tests
• human-readable output tests

Keep claim verification centralized. If check-file needs the same behavior as check-claim, extract a private helper from cli.py rather than duplicating DOI lookup and abstract-source selection logic.

Evaluation Fixture

Add a small fixture-backed eval set:

tests/fixtures/numeric_claim_eval.jsonl

Each fixture should contain:

• id
• domain
• abstract
• claim
• expected_verdict
• why

The eval should use local abstract text, not live APIs. Live API checks are already covered separately by the manual smoke workflow; deterministic tests need stable fixtures.

Initial domains:

• materials
• biomedicine
• machine learning
• chemistry
• general science

The eval should intentionally include false-accept defense cases:

• Same sentence with multiple subjects and multiple numbers.
• Correct number but wrong subject.
• Correct unit but wrong comparator.
• Claim that requires full-text/table/figure evidence.
• Abstract with numeric evidence but no clear subject binding.

Documentation Updates After Implementation

Update both README.md and README.ko.md with:

• single-claim usage
• batch JSONL usage
• batch CSV usage
• how to read ACCEPT, WARN/PARTIAL, and UNVERIFIABLE
• clear "when not to use" boundaries
• a short statement that PyPI installs the CLI package, not the Codex SKILL.md

Update CHANGELOG.md once the feature lands.

Implementation Order

1. Add fixture-backed eval tests for batch input and numeric claim outcomes. 2. Add batch.py parser and result model helpers. 3. Extract shared single-claim execution from cli.py. 4. Add check-file CLI command for JSONL. 5. Add CSV support. 6. Add human-readable rendering and JSON summary output. 7. Update README files and changelog. 8. Run unit tests, py_compile, package build, twine check, and package smoke.

Risk Review

False Accept Risk

Batch mode can make mistakes more harmful because users may scan summaries instead of individual evidence. The summary must not hide row-level warnings. Human-readable output should show at least the claim, DOI, reason, and evidence for every non-accepted row.

API Rate Risk

Batch mode may trigger many CrossRef, Semantic Scholar, or PubMed calls. Phase 1 should keep the implementation simple, but the CLI should process rows sequentially and avoid parallel live requests. This reduces accidental load and keeps failures understandable.

Input Ambiguity Risk

CSV quoting and multiline claims can become messy. Use Python's csv module, not manual string splitting. JSONL parse errors should report the line number.

Scope Creep Risk

Batch mode may tempt users to ask for broader claim interpretation. The command should call the same conservative engine and preserve existing verdict semantics. It should improve throughput, not loosen acceptance.

Report Misreading Risk

A single summary such as "47 checked" can sound like success. Summaries must include verdict counts, and nonzero warning/unverifiable counts should be visually clear in text output.

Spec Self-Review

• No LLM, full-text, table, figure, or complex statistical metric expansion is included.
• The new command improves repeated use without changing the core verifier boundary.
• Exit codes are explicit and consistent with the existing check-claim behavior.
• The spec separates verdict values from status values so implementation can reuse ClaimSupportResult directly.
• Invalid input handling is fail-closed rather than silently skipping rows.
• Deterministic evals use local abstract fixtures instead of live APIs.
• The main remaining design choice is whether malformed-row diagnostics should be emitted as plain text before JSON parsing errors or as structured JSON when --json is present. Implementation should prefer structured JSON for --json.

Numeric Claim Verifier Design

Goal

Expand ref-verify into a general-purpose research numeric claim verifier while keeping it a conservative citation guard.

Positioning

ref-verify is not a tool that semantically understands papers. It helps AI agents check whether an abstract contains explicit numeric evidence before attaching a citation.

Core Principles

• Do not add LLM semantic inference.
• Do not parse full text, tables, or figures.
• Check only numbers, units, and comparator language explicitly present in the abstract.
• Return ACCEPT only when the subject, numeric value, unit, and comparator clearly match.
• Return PARTIAL/WARN when a number is present but subject binding or context is ambiguous.
• Return UNVERIFIABLE when no abstract evidence is available.

Phase 1 Scope

Phase 1 covers common numeric claims used across research fields:

• Percent claims, such as 95%, above 90%, and below 10%.
• Simple unit and count claims, such as 5000 cycles, 12 patients, 3.2 V, 37 °C, and 10 mg/mL.
• Basic comparators: >, >=, <, <=, at least, more than, below, and up to.
• Subject binding in the same sentence or clause.
• Conservative handling when a sentence mixes multiple subjects or numbers.

Deferred Scope

Phase 2 will handle statistical metrics that require tighter contextual binding:

• p < 0.05
• AUC / AUROC
• F1 score
• hazard ratio
• odds ratio
• 95% CI

Architecture

Create src/ref_verify/numeric_claim.py for numeric extraction and matching. It should extract numeric claims from user claims, extract numeric evidence from abstract clauses, compare values through comparator entailment, and require subject terms to match in the same clause before accepting.

Keep src/ref_verify/claim_check.py as the verdict router. It should call the numeric engine first, preserve existing actuation-strain near-miss guards, and retain the literal text-claim fallback.

Add tests/test_numeric_claim.py for focused numeric behavior across materials, biomedicine, ML, and general science examples. Keep false-accept tests at least as prominent as positive tests.

Verdict Criteria

ACCEPT: subject plus number/unit/comparator clearly match abstract evidence.

PARTIAL/WARN: a number exists, but subject binding is ambiguous, the wrong subject has the supporting value, multiple numbers create mismatch risk, or semantic paraphrase would be needed.

UNVERIFIABLE: no abstract is available or evidence cannot be checked.

Product Copy

ref-verify is not a tool that "understands" papers. It is a conservative citation guard that makes AI agents verify whether an abstract actually contains explicit numeric evidence before citing a paper.

Release Automation Design

Goal

Make ref-verify safer for non-developer users by automatically checking that source tests, package builds, wheel installs, and release publishing paths stay consistent.

Scope

• Add a normal CI workflow for pushes and pull requests.
• Add a manual live-smoke workflow for public academic API checks.
• Add a GitHub Release-triggered PyPI publishing workflow using trusted

publishing.

• Add a local packaging smoke script that installs a built wheel, checks the

ref-verify console script, confirms the package version, and confirms the Python wheel remains CLI-only.

• Document the release checks in contributor-facing docs.

Non-Goals

• Do not publish to PyPI in this change.
• Do not add live API tests to default CI.
• Do not add new runtime dependencies.
• Do not package SKILL.md into the Python wheel.

Design

Default CI uses Python 3.10 and 3.12. It runs unit tests from the source checkout, byte-compiles Python files, builds wheel and sdist artifacts, checks metadata with twine, and runs a wheel install smoke test. This catches the main failure modes that matter for users: broken tests, broken package metadata, missing console scripts, and accidental skill-file packaging drift.

The live-smoke workflow is manual-only because CrossRef, Semantic Scholar, and PubMed availability can be flaky. It verifies representative public-API CLI paths without making default CI dependent on external services.

The PyPI workflow runs only when a GitHub Release is published for a v* tag. It uses GitHub OIDC trusted publishing through pypa/gh-action-pypi-publish, so no long-lived PyPI token needs to be stored in repository secrets.

Success Criteria

• python3 scripts/package_smoke.py --wheel <wheel> --expected-version 1.1.1

passes after a local build.

• GitHub Actions has workflows for CI, manual live smoke, and release publishing.
• README/CONTRIBUTING tell maintainers how these checks relate to installation

and release safety.

CLI regression corpus (ship-gate)

cli_regression.jsonl is a labeled, deterministic regression set for the check-file engine. It complements evals.json (which evaluates skill-level LLM behavior); this one pins machine-checkable verdicts so unit/source/matcher changes can be regression-tested without an LLM in the loop.

Each row carries the claim plus ground-truth labels:

Two invariant classes

SAFETY (release blocker). must_accept rows must stay ACCEPT; must_not_accept rows must never become ACCEPT. This is the tool's core promise — no fabricated, relational, unreachable, or over-accepting claim is waved through, and the one clean supported claim stays green. A break here fails the gate (non-zero exit).

PROGRESS (informational). Gated rows do not yet reach expected_verdict because a fix has not landed. They are reported, not failed, and flip to PASS as their gated_on issue is resolved. This is how the corpus tracks the roadmap.

How to run

PYTHONPATH=src python3 evals/run_cli_regression.py

The same command is also wired into the manual GitHub Actions Live Smoke workflow. It is intentionally not part of every pull-request CI run because it depends on live public APIs.

Exit code is non-zero iff a SAFETY invariant is violated. (Live network: CrossRef / OpenAlex / Semantic Scholar / PubMed. Semantic-Scholar free-tier 429 only affects PROGRESS rows that depend on it, never SAFETY rows.)

What the corpus encodes (snapshot, latest main)

SAFETY: 12/12 ok  |  PROGRESS pending: 0

• Supported happy paths — A1 (>220 °C entails >200 °C), A2

(OpenAlex-reached 30 °C conductivity measurements), A3 (1.7 eV), B2 (200 g sulfur-network synthesis), and E2 (up to 117% actuated strain) ACCEPT and must stay green.

• Never-accept controls (all PASS) — B1 fabricated number, C1/C2 relational,

D1/D2 unreachable (Elsevier / old Nature, abstract-only ceiling), E1 dead DOI. B3 (over-acceptance) is now PARTIAL after #11 — kept must_not_accept so the bug cannot silently regress.

• No current gated false-negatives — if future supported rows are added before

their matcher/source work lands, they should use gated_on and report as PROGRESS rather than failing SAFETY.

The verdict labels for A2/A3/B2 were grounded by fetching the live abstracts (CrossRef + OpenAlex) and confirming the value appears verbatim; no label asserts support that is not in a fetched abstract.

Security Policy

Scope

ref-verify is a skill (prompt instructions) for AI agents. It makes read-only HTTP requests to public academic APIs — CrossRef, Semantic Scholar, Unpaywall, arXiv, and PubMed. It does not execute code, store credentials, or write files.

In scope for security reports:

• The skill instructing an agent to send user data to an unexpected third-party endpoint
• A prompt injection vector in the skill instructions that could be exploited via a malicious paper abstract
• Any behavior that could leak the user's research content to an unintended destination

Out of scope:

• Vulnerabilities in CrossRef, Semantic Scholar, or other upstream APIs
• Rate limiting or API availability issues
• Incorrect verification results (those are bugs, not security issues — use a regular issue)

Reporting

Do not open a public issue for security vulnerabilities. Email the maintainer directly or use GitHub's private vulnerability reporting.

Include:

• A description of the vulnerability
• Steps to reproduce
• The potential impact

You will receive a response within 72 hours.

Prompt injection risk

This skill fetches content from external sources (paper abstracts) and includes it in the agent's context. A maliciously crafted abstract could theoretically contain text designed to manipulate the agent's behavior. The skill mitigates this by:

• Quoting abstract content verbatim (rather than acting on it)
• Only fetching from established academic APIs with stable content policies
• Not executing any content from fetched sources

If you discover a prompt injection vector in fetched abstract content, please report it.