Whitepaper Audit

whitepaper-audit

Audit a markdown white paper in two lanes and produce one merged, prioritized report.

Inputs

• Document path (required) — markdown source, not PDF.
• Stated audience (ask if not given) — severity of audience-fit/jargon-undefined

depends on it. Default: "technical practitioners, non-academic".

• Mode — recommend (default) or fix (only on explicit request).

Workflow

1. Lane 1 — deterministic

python3 scripts/check_doc.py <doc.md> --offline [--target-grade N] [--allow ACRO]

Drop --offline to also check http(s) links (HEAD→GET, timeouts; only broken is a finding). Output: JSON findings, schema in DESIGN.md.

2. Lane 2 — LLM judge

Dispatch a subagent (fresh context — never judge a document you wrote in the same context) with references/audit-prompt.md, filling {PATH} and {AUDIENCE}, plus the [judge] criteria from references/checklist.md. The judge returns JSON findings.

Judge calibration rules are binding: verbatim quotes required; no P0 at low confidence; "needs verification", never "factually wrong".

3. Merge

Dedupe by (location, issue type) keeping both lane attributions; sort P0 → P1 → P2, then confidence. Cross-reference: a lane-1 broken link that supports a claim (judge decides materiality) is P1; decorative → P2.

4. Report (default mode)

Write a markdown report: summary verdict, findings table (id, severity, confidence, location, fix), then details. Recommend; do not edit.

5. Fix mode (only when explicitly requested)

Apply fixes P0-first. Any change to code goes through superpowers test-driven-development (test first, watch it fail). Prose fixes: edit, then re-run the full audit and report cleared vs remaining findings.

Evals

Before trusting a new/changed judge prompt, run evals/README.md procedure (planted defects + clean control; pass criteria inside). Lane 1 is covered by scripts/tests/test_check_doc.py (pytest).

Files

• scripts/check_doc.py — lane 1 (stdlib-only; --help for flags)
• references/checklist.md — operational criteria, both lanes
• references/audit-prompt.md — judge prompt template
• evals/ — judge validation cases + pass criteria
• DESIGN.md — architecture decisions (v0.2, Codex-audited)

whitepaper-audit — design spec (v0.2, post-Codex-audit)

Date: 2026-06-03 · Author: Gleb Kalinin + Claude · Codex audit: incorporated (v0.1 → v0.2)

Purpose

A general-purpose skill that audits a technical/scientific white paper (or any long-form explanatory document) against a research-grounded best-practices checklist, and either (a) produces a prioritized P0–P2 findings report (default), or (b) applies corrections on request. First target: ~/ai_projects/confide/docs/WHITEPAPER.md.

Architecture

whitepaper-audit/
├── SKILL.md                      # triggers, workflow, modes
├── DESIGN.md                     # this file
├── references/
│   ├── checklist.md              # operational rules w/ examples+counterexamples, tagged [script]/[judge]
│   └── audit-prompt.md           # LLM-judge prompt template
├── scripts/
│   ├── check_doc.py              # deterministic checks (stdlib-only)
│   └── tests/test_check_doc.py   # pytest, built TDD
└── evals/
    ├── README.md                 # pass criteria, runner procedure
    └── cases/                    # planted-defect docs + clean + near-clean controls

Finding schema (both lanes — the merge contract)

Every finding, from either lane:

{
  "check_id": "acronym-undefined",      // stable id from checklist.md
  "lane": "script" | "judge",
  "severity": "P0" | "P1" | "P2",
  "confidence": "high" | "medium" | "low",
  "location": "§5.2 / quoted line",
  "evidence_quote": "…verbatim excerpt…",
  "rationale": "why this is a problem",
  "suggested_fix": "concrete edit"
}

Merged report: dedupe by (location, issue type), keep both lane attributions when merged, sort P0→P2 then confidence. Severity = impact × confidence — a low-confidence high-impact issue is reported at the lower severity with a "verify" note, never silently promoted.

Two-lane audit

Lane 1 — deterministic (check_doc.py, TDD)

Stdlib-only, Python 3.10+. JSON findings on stdout, exit 0; never crashes on malformed markdown. Preprocessing applies to all checks: strip code blocks, tables, URLs, headings, footnote markers, and HTML comments before text metrics.

Broader jargon (bolded terms, lowercase terms of art) is not lane-1: the judge flags "possible undefined jargon" as a soft finding.

Lane 2 — LLM judge (audit-prompt.md)

Judged criteria (refined by the best-practices research): apparent overclaim / unsupported claim / internally inconsistent number (NOT "factual error" — the judge has no retrieval; it flags "needs verification", it does not adjudicate truth); marketing language in a technical claim; buried lede; honesty about limitations and uncertainty; audience fit; flow; table/figure clarity; possible undefined jargon (soft).

Judge outputs findings in the shared schema, with confidence mandatory.

Severity rubric (impact × confidence, with per-check examples)

• P0 — trust-breaking: apparent overclaim stated as fact; internally inconsistent

number (says 0.88 in §5, 0.78 in abstract); missing/dishonest limitations in a doc that makes empirical claims. Example: "our tool guarantees GDPR compliance" with no evidence → P0. Counterexample: a hedge that's merely wordy → P2.

• P1 — comprehension-breaking: acronym undefined at first use; buried lede (key

finding absent from abstract); broken link that supports a claim; missing structural block a practitioner needs.

• P2 — polish: tone, wordiness, dead footer link, high FK in a non-critical section,

table formatting.

Modes

1. Default (recommend): lane 1 + lane 2 → merged report (schema above). No edits. 2. Fix (explicit request): apply corrections P0-first. Code changes (audited repo's scripts or this skill's own) go through superpowers TDD. Re-run full audit after; report cleared vs remaining.

Evals — three suites, explicit pass criteria

1. Lane-1 evals = the pytest suite (planted structural defects in fixture docs; exact expected findings). 2. Lane-2 evals: planted-defect docs (≥1 apparent overclaim, ≥1 internally inconsistent number, ≥1 jargon cluster, ≥1 missing-limitations doc) + 1 clean control + 1 near-clean control (minor P2-only flaws). Pass criteria: 100% recall of planted P0s at exact severity, location matched to the correct section; thresholded recall ≥80% for planted P1/P2; false-positive budget on controls: zero P0, ≤1 P1, ≤3 P2 per control doc. 3. Merged-output eval: one doc with defects spanning both lanes; verify dedupe and schema integrity.

Out of scope (v1)

• Grammar/spell checking; PDF auditing (markdown source only); auto-publishing/CI.
• Citation verity (whether a cited source supports the claim): judge marks "needs

verification"; a future evidence-verification lane may fetch sources.

Resolved design questions (per Codex audit)

1. Term detector → acronyms-only in v1, allowlist + exact patterns; jargon → judge. 2. Readability → yes, sentence-level: section averages for overview, top-5 hardest sentences for actionable fixes. 3. Eval threshold → all planted P0s at exact severity; thresholded P1/P2; FP budget on clean + near-clean controls (no flat 80%).

GreenCache: Measuring a Memory Cache for Weather Data

Version 1.1 · 2026-03-20 · Kim Researcher · kim@example.org

Summary

GreenCache is a small in-memory cache for weather-station readings. On a replay of 30 days of real station traffic, it answered 72% of queries from memory (95% CI 68–76%), cutting database reads by roughly two thirds. The cache helps most for stations queried many times per hour; it adds little for rarely-queried stations. This paper describes the design, the measurement method, and the limits of what we can conclude.

The problem

Weather dashboards ask for the same recent readings again and again. Each repeated question normally goes to the database, which is slow and costly. A cache — a small store of recently used answers kept in fast memory — can answer repeats directly.

Design

GreenCache keeps the most recently requested readings in memory (an LRU policy — Least Recently Used, meaning the reading unused for the longest time is evicted first when space runs out). Cache size is configurable; we tested 512 MB.

Method

We replayed 30 days of recorded query traffic (4.1 million queries) against GreenCache and counted hits and misses. The replay preserves the original timing and ordering. We report the hit rate with a bootstrap confidence interval (resampling days, 2,000 times).

Results

Limitations

One month of traffic from one deployment may not represent other seasons or sites. The replay ignores cache warm-up on day one (under 2% of queries). We did not measure tail latency, only hit rate. Results are specific to the 512 MB configuration.

Conclusion

For dashboards with repeated queries, a small LRU cache is a cheap, measurable win. To adopt: start with the default configuration, replay a week of your own traffic, and check your hit rate before rollout. Code and replay scripts: github.com/example/greencache.

Glossary

QuickSort-NG: Faster Sorting for Log Pipelines

Version 0.9 · 2026-04-12 · Ada Developer

Summary

QuickSort-NG is a sorting library tuned for log-pipeline workloads. On our test set of 40 log files it sorted 2.3× faster than the standard library sort (median; range 1.8–2.9×). This paper explains the technique and the measurement.

The problem

Log pipelines sort huge, nearly-sorted files. General-purpose sorts waste work on data that is already mostly ordered.

How it works

QuickSort-NG detects pre-sorted runs (stretches of already-ordered records) and merges them directly, falling back to standard quicksort for disordered segments. Run detection is a single linear pass.

Results

The gains concentrate exactly where the workload hypothesis predicts: pre-ordered data.

Conclusion

For log pipelines specifically, run-detection sorting is worth adopting. Random-data workloads should keep the standard sort.

<!-- PLANTED: limitations-present — this document makes empirical performance claims (2.3x median across 40 files) but has NO limitations section at all: nothing on test-set composition bias, hardware, memory trade-offs, or generality. expected: check_id=limitations-present, severity=P0 -->

Glossary

VectorVault — A Revolutionary Approach to Database Caching

Version 1.0 · 2026-05-01 · Sam Engineer

Executive summary

VectorVault is a caching layer for vector databases. In our benchmark of 12 workloads, it reduced median query latency by 41% (CI 35–47%). This paper describes the design and the measurements.

The problem

Vector databases recompute similarity searches that repeat often. Caching can help, but existing caches show poor hit rates on embedding workloads.

Design

VectorVault sits between the application and the database. Our LSH-based bucketing with HNSW fallback and PQ compaction over the IVF index achieves sublinear probe amortization across the recall-bounded ANN frontier.

<!-- PLANTED: jargon-undefined — the sentence above is an unglossed jargon cluster (LSH, HNSW, PQ, IVF, ANN all undefined; audience is general practitioners). expected: check_id=jargon-undefined OR acronym-undefined, severity=P1 -->

Results

Across 12 workloads the median latency reduction was 38%, with hit rates between 0.61 and 0.84.

<!-- PLANTED: inconsistent-number — exec summary says 41%, here 38%, same quantity. expected: check_id=inconsistent-number, severity=P0 -->

VectorVault eliminates cache-miss latency entirely and guarantees that no production workload will ever regress.

<!-- PLANTED: unsupported-claim/overclaim stated as fact — "eliminates entirely", "guarantees ... ever" with no supporting evidence. expected: check_id=unsupported-claim (or marketing-tone), severity=P0 -->

Limitations

Benchmarks are synthetic; production traces may differ. N=12 workloads is small; CIs are wide. We did not test distributed deployments.

Glossary

Evals for whitepaper-audit (lane 2 — the LLM judge)

Lane 1 (deterministic) is validated by the pytest suite in scripts/tests/. Lane 2 validates the judge (audit-prompt.md) against planted defects, mirroring the planted-signal gold-standard approach: you only trust a reviewer you've tested against known answers.

Cases

Procedure

1. For each case, give the judge the audit prompt (references/audit-prompt.md) with the case file as {PATH} and audience "general technical practitioners". Strip the HTML comments first (sed '/<!--/,/-->/d') so the judge cannot read the answer key. 2. Compare the judge's JSON findings to the planted defects.

Pass criteria (from DESIGN.md v0.2; per Codex audit)

• 100% recall of planted P0s at exact severity, location matched to the correct

section. (unsupported-claim vs marketing-tone is an accepted check_id alias for the overclaim plant; jargon-undefined vs acronym-undefined likewise.)

• Planted P1/P2: recall ≥ 80%.
• False-positive budget on the clean control: zero P0, ≤1 P1, ≤3 P2.
• Judge output must validate against the finding schema (all required keys, verbatim

evidence quotes that actually appear in the document).

Record results per run in evals/RESULTS.md (date, model, pass/fail per criterion). Re-run after any change to audit-prompt.md or the checklist.

Eval results

2026-06-03 · judge = Claude Opus 4.8 subagents (fresh context) · prompt v1

Overall: PASS (100% planted-P0 recall at exact severity; planted P1 found; clean control clean).

Notes

• The judge additionally flagged the planted doc's title ("Revolutionary") as

marketing-tone P0 — this was a deliberate part of the overclaim plant and is a correct catch, not an FP.

• The judge found genuine, unintended defects in missing-limitations.md: the

summary's "range 1.8–2.9×" contradicts the table's 1.1× row, and the stated median 2.3× doesn't match the table median 2.2×. These are now accepted as additional plants (they are real inconsistent-number defects); future runs should expect them. Evidence of judge sharpness: it caught defects the eval author didn't notice planting.

• Lane-1 FP profile observed on a real document (CONFIDE white paper): caps taxonomy

labels in tables (EMAIL/DATE/ORG…), caps emphasis (NOT), and prose-defined metrics (F1, AUC — glossed in text but not in a parenthetical pattern). Candidate v1.1 refinement: dictionary-word filter + prose-definition pattern (ABC is/means …).

GreenCache: Measuring a Memory Cache for Weather Data

Version 1.1 · 2026-03-20 · Kim Researcher · kim@example.org

Summary

GreenCache is a small in-memory cache for weather-station readings. On a replay of 30 days of real station traffic, it answered 72% of queries from memory (95% CI 68–76%), cutting database reads by roughly two thirds. The cache helps most for stations queried many times per hour; it adds little for rarely-queried stations. This paper describes the design, the measurement method, and the limits of what we can conclude.

The problem

Weather dashboards ask for the same recent readings again and again. Each repeated question normally goes to the database, which is slow and costly. A cache — a small store of recently used answers kept in fast memory — can answer repeats directly.

Design

GreenCache keeps the most recently requested readings in memory (an LRU policy — Least Recently Used, meaning the reading unused for the longest time is evicted first when space runs out). Cache size is configurable; we tested 512 MB.

Method

We replayed 30 days of recorded query traffic (4.1 million queries) against GreenCache and counted hits and misses. The replay preserves the original timing and ordering. We report the hit rate with a bootstrap confidence interval (resampling days, 2,000 times).

Results

Limitations

One month of traffic from one deployment may not represent other seasons or sites. The replay ignores cache warm-up on day one (under 2% of queries). We did not measure tail latency, only hit rate. Results are specific to the 512 MB configuration.

Conclusion

For dashboards with repeated queries, a small LRU cache is a cheap, measurable win. To adopt: start with the default configuration, replay a week of your own traffic, and check your hit rate before rollout. Code and replay scripts: github.com/example/greencache.

Glossary

QuickSort-NG: Faster Sorting for Log Pipelines

Version 0.9 · 2026-04-12 · Ada Developer

Summary

QuickSort-NG is a sorting library tuned for log-pipeline workloads. On our test set of 40 log files it sorted 2.3× faster than the standard library sort (median; range 1.8–2.9×). This paper explains the technique and the measurement.

The problem

Log pipelines sort huge, nearly-sorted files. General-purpose sorts waste work on data that is already mostly ordered.

How it works

QuickSort-NG detects pre-sorted runs (stretches of already-ordered records) and merges them directly, falling back to standard quicksort for disordered segments. Run detection is a single linear pass.

Results

The gains concentrate exactly where the workload hypothesis predicts: pre-ordered data.

Conclusion

For log pipelines specifically, run-detection sorting is worth adopting. Random-data workloads should keep the standard sort.

Glossary

VectorVault — A Revolutionary Approach to Database Caching

Version 1.0 · 2026-05-01 · Sam Engineer

Executive summary

VectorVault is a caching layer for vector databases. In our benchmark of 12 workloads, it reduced median query latency by 41% (CI 35–47%). This paper describes the design and the measurements.

The problem

Vector databases recompute similarity searches that repeat often. Caching can help, but existing caches show poor hit rates on embedding workloads.

Design

VectorVault sits between the application and the database. Our LSH-based bucketing with HNSW fallback and PQ compaction over the IVF index achieves sublinear probe amortization across the recall-bounded ANN frontier.

Results

Across 12 workloads the median latency reduction was 38%, with hit rates between 0.61 and 0.84.

VectorVault eliminates cache-miss latency entirely and guarantees that no production workload will ever regress.

Limitations

Benchmarks are synthetic; production traces may differ. N=12 workloads is small; CIs are wide. We did not test distributed deployments.

Glossary

LLM-judge audit prompt (v1)

Use this prompt (filling the placeholders) for the judge lane. Run it on the full document text. The deterministic lane runs separately (scripts/check_doc.py); do not duplicate its checks except where the checklist marks a criterion as both-lane.

---

You are a critical peer reviewer of white papers — rigorous, concrete, and calibrated. You review against an explicit checklist; you do not invent criteria, and you do not adjudicate external facts.

Document under review: {PATH} Stated audience: {AUDIENCE — e.g. "practitioners/clinicians; readable by a final-year school student"} Checklist: the [judge]-tagged criteria in checklist.md (provided below / alongside).

Your epistemic limits (binding):

• You have NO retrieval. You may flag a claim as *apparently unsupported within the

document, internally inconsistent, or needs verification* — you may NOT call anything a factual error about the outside world.

• Quote evidence verbatim. A finding without a verbatim quote and location is invalid.
• Calibrate: every finding carries confidence (high/medium/low). When confidence is low

on a high-impact issue, set the severity one level lower and say "verify" in the rationale. Never report a P0 at low confidence.

Severity rubric (impact × confidence):

• P0 (trust-breaking): apparent overclaim stated as fact; the same quantity given two

different values; missing or hollow limitations in a document making empirical claims; the core finding absent from the abstract/summary.

• P1 (comprehension-breaking): jargon undefined for the stated audience; key

uncertainty unreported on small-N results; audience misfit in a load-bearing section; materially misleading structure.

• P2 (polish): tone, wordiness, formatting, staleness, minor flow issues.

Procedure: 1. Read the whole document once for the lede: what is the single most important claim? Check it appears in the abstract/summary (buried-lede). 2. Walk the [judge] criteria in checklist order. For each violation, record a finding. 3. Sweep all numbers: build a list of every quantitative claim; cross-check repeated quantities for consistency (inconsistent-number) and for uncertainty reporting (uncertainty-reported). 4. Sweep tone: hunt superlatives, buzzwords, and sales framing (marketing-tone). 5. Audience pass: simulate the stated reader; mark every place they would stall (audience-fit, jargon-undefined). 6. Self-check: delete any finding lacking a verbatim quote; downgrade any low-confidence high-severity finding; merge duplicates.

Output — JSON list only, no prose, each finding:

{
  "check_id": "<id from checklist>",
  "lane": "judge",
  "severity": "P0|P1|P2",
  "confidence": "high|medium|low",
  "location": "<section heading or §N + brief locator>",
  "evidence_quote": "<verbatim excerpt>",
  "rationale": "<why this violates the rule — one or two sentences>",
  "suggested_fix": "<concrete edit, not 'improve clarity'>"
}

Also include, as the final array element, a summary pseudo-finding: {"check_id": "summary", "lane": "judge", "severity": "P2", "confidence": "high", "location": "document", "evidence_quote": "", "rationale": "<2-3 sentence overall assessment>", "suggested_fix": "<top 3 priorities in order>"}.

If the document is clean on a criterion, emit nothing for it. An empty list (plus summary) is a valid, good result — do not manufacture findings to seem thorough.

White-paper audit checklist (v1)

Research-grounded criteria (sources: Gopen & Swan; Nielsen Norman; Tufte; IEEE ProComm & AU plain-language guidance; white-paper practice guides — full list in internal/research_report_20260603_201802.md of the originating session, summarized at bottom). Each criterion is an operational rule with an example and counterexample, tagged [script] (checked by scripts/check_doc.py) or [judge] (LLM judge via references/audit-prompt.md), with default severity. Severity is always impact × confidence; the judge may move a finding ±1 level with stated rationale.

A. Structure & flow

B. Clarity & readability

C. Evidence & rigor

D. Tables & figures

E. Audience fit (practitioner focus)

F. Trust & transparency

Severity reference (impact × confidence)

• P0 — trust-breaking: apparent overclaim stated as fact · internally inconsistent

number · missing/hollow limitations in an empirical doc · core claim absent from abstract.

• P1 — comprehension-breaking: acronym/jargon undefined at first use · claim-supporting

broken link · missing date/author · uncertainty absent on small-N results · audience misfit.

• P2 — polish: tone nits · long sentences in non-critical sections · table formatting ·

dead footer links · staleness.

Low-confidence + high-impact ⇒ report at the lower severity with a "verify" note — never silently promote.