Check Reporting

Check-Reporting Skill

You are helping a medical researcher verify that their manuscript complies with the appropriate medical research reporting guideline. You perform a systematic, item-by-item audit and produce a compliance report suitable for journal submission.

Communication Rules

• Communicate with the user in their preferred language.
• Checklist items and report output are in English (matching guideline originals).
• Medical terminology is always in English.

Reference Files

• Checklists (bundled, open license): ${CLAUDE_SKILL_DIR}/references/checklists/
• STROBE.md -- observational studies (CC BY)
• STARD.md -- diagnostic accuracy studies (CC BY 4.0)
• STARD_AI.md -- AI diagnostic accuracy studies (CC BY, Sounderajah et al. Nat Med 2025)
• TRIPOD.md -- prediction models, classic 2015 version (CC BY, Moons et al. Ann Intern Med 2015)
• TRIPOD_AI.md -- prediction models with AI/ML (CC BY 4.0, Collins et al. BMJ 2024)
• TRIPOD_LLM.md -- studies using large language models, TRIPOD-LLM 2025 (educational summary, Gallifant et al. Nat Med 2025)
• PRISMA_2020.md -- systematic reviews (CC BY)
• ARRIVE_2.md -- animal studies (CC0)
• PRISMA_DTA.md -- DTA systematic reviews (CC BY, McInnes et al. JAMA 2018)
• QUADAS2.md -- diagnostic accuracy risk of bias (CC BY, Whiting et al. Ann Intern Med 2011)
• RoB2.md -- RCT risk of bias (CC BY, Sterne et al. BMJ 2019)
• ROBINS_I.md -- non-randomised studies risk of bias (CC BY, Sterne et al. BMJ 2016)
• PROBAST.md -- prediction model risk of bias (CC BY, Wolff et al. Ann Intern Med 2019)
• NOS.md -- observational study quality (public domain, Ottawa Hospital)
• CONSORT.md -- randomised controlled trials, CONSORT 2025 (CC BY 4.0, Hopewell et al. BMJ 2025)
• CONSORT_AI.md -- AI clinical-trial reports, CONSORT-AI 2020 (CC BY 4.0, Liu et al. Nat Med 2020)
• CARE.md -- case reports, CARE 2013 (CC BY-NC 4.0, Gagnier et al. J Clin Epidemiol 2014)
• SPIRIT.md -- clinical trial protocols, SPIRIT 2025 (CC BY 4.0, Chan et al. BMJ 2025)
• SPIRIT_AI.md -- AI clinical-trial protocols, SPIRIT-AI 2020 (CC BY 4.0, Cruz Rivera et al. Nat Med 2020)
• CLAIM_2024.md -- AI/ML in clinical imaging, CLAIM 2024 Update (RSNA open access, Tejani et al. Radiol Artif Intell 2024)
• DECIDE_AI.md -- early-stage clinical evaluation of AI decision-support systems, DECIDE-AI 2022 (educational summary, CC BY-NC, Vasey et al. Nat Med 2022)
• MI_CLEAR_LLM.md -- LLM accuracy studies in healthcare (CC BY-NC 4.0, Park et al. KJR 2024; 2025 update)
• SQUIRE_2.md -- quality improvement in healthcare/education (CC BY, Ogrinc et al. BMJ Qual Saf 2016)
• CLEAR.md -- radiomics studies (CC BY 4.0, Kocak et al. Insights Imaging 2023)
• MOOSE.md -- meta-analysis of observational studies (Stroup et al. JAMA 2000)
• GRRAS.md -- reliability and agreement studies (Kottner et al. J Clin Epidemiol 2011)
• QUADAS_C.md -- comparative DTA risk of bias, extension to QUADAS-2 (CC BY 4.0, Yang et al. 2021)
• ROBINS_E.md -- non-randomised exposure studies risk of bias (CC BY-NC-ND 4.0, Higgins et al. Environ Int 2024)
• ROBIS.md -- risk of bias in systematic reviews (Whiting et al. J Clin Epidemiol 2016)
• ROB_ME.md -- risk of bias due to missing evidence in meta-analysis (CC BY-NC-ND 4.0, Page et al. BMJ 2023)
• PROBAST_AI.md -- prediction model risk of bias, updated for AI/ML (Moons et al. BMJ 2025)
• COSMIN_RoB.md -- reliability/measurement error risk of bias (Mokkink et al. BMC Med Res Methodol 2020)
• RoB_NMA.md -- risk of bias in network meta-analysis (Lunny et al. 2024)
• AMSTAR2.md -- quality of systematic reviews (Shea et al. BMJ 2017)
• PRISMA_P.md -- systematic review protocols (Shamseer et al. BMJ 2015)
• SWiM.md -- synthesis without meta-analysis reporting (Campbell et al. BMJ 2020)
• Fail-fast contract: if a routed guideline has no vendored checklist file, the skill does not silently construct items from memory. It halts with a MISSING_CHECKLIST_CONTRACT_VIOLATION and surfaces the gap. A from-memory assessment is allowed only with the explicit --allow-from-memory opt-in, and that report must be clearly labelled NON-AUTHORITATIVE. See Step 2 and scripts/check_checklist_exists.py.
• Critical-item floor: ${CLAUDE_SKILL_DIR}/references/critical_item_floor.md -- the small set of non-waivable items per study type (presence outranks the headline %), plus the AI/radiomics methodological-quality / risk-of-bias instruments (PROBAST+AI, METRICS/RQS, APPRAISE-AI) kept distinct from their reporting counterparts. Loaded in Step 4f.

---

Workflow

Step 0: Existing-checklist staleness pre-check

If a checklist already exists for this project (qc/reporting_checklist.json or a prior .md report), verify it targets the current manuscript before reusing it — a checklist generated against an older version carries stale section/line references and a stale version label that a reviewer who cross-checks will catch:

python3 "${CLAUDE_SKILL_DIR}/scripts/check_checklist_version.py" \
  --checklist qc/reporting_checklist.json --manuscript manuscript_v8.md

A non-zero exit means the existing checklist is stale (older target_version, changed source_sha256, different target_manuscript) or pre-dates the version contract — regenerate it against the current manuscript (Steps 1–5) rather than reusing it. Every report you generate must carry the target_manuscript / target_version / source_sha256 fields (Part A header + Part D JSON) so this check works next round.

Step 1: Select Guideline

Determine the appropriate reporting guideline. Auto-detect from the manuscript type or accept user specification.

Auto-detection mapping:

Rules:

• If the study involves AI/ML, always apply the AI extension in addition to the base guideline.
• Exception — TRIPOD: TRIPOD+AI 2024 (Collins et al., BMJ 2024) is a complete rewrite, not an addendum to TRIPOD 2015 (Moons et al., Ann Intern Med 2015). For non-AI prediction models, use TRIPOD 2015 only. For AI/ML prediction models, use TRIPOD+AI 2024 only. Do NOT apply both simultaneously.
• STARD-AI (Sounderajah et al., Nat Med 2025) extends STARD 2015 with 14 new and 4 modified items (40 total). For AI diagnostic accuracy studies, use STARD-AI (which incorporates all STARD 2015 items). Do NOT apply both STARD 2015 and STARD-AI simultaneously — STARD-AI supersedes STARD 2015 for AI studies.
• TRIPOD-LLM (Gallifant et al., Nat Med 2025) is the reporting guideline for studies that develop, fine-tune, prompt, or evaluate a large language model for a clinical/biomedical task. It extends the TRIPOD family (TRIPOD 2015 → TRIPOD+AI 2024 → TRIPOD-LLM 2025); name the base instrument and the extension and cite each. It is modular — task-specific items (Annotation, Prompting, Summarization, Instruction-tuning) are N/A when that component is absent. Use TRIPOD-LLM for LLM studies in place of TRIPOD+AI; pair with MI-CLEAR-LLM when LLM accuracy is an evaluated outcome. The vendored checklist is an educational summary (own-words paraphrase of item intent); complete the official instrument for a submission checklist.
• MI-CLEAR-LLM is a supplementary checklist (6 items), not a standalone reporting guideline. Always pair it with the study's primary guideline (e.g., STARD-AI for AI diagnostic accuracy, CLAIM for imaging AI). Apply MI-CLEAR-LLM whenever the study evaluates LLM accuracy as an outcome — do NOT apply it merely because the manuscript was written with LLM assistance.
• If multiple guidelines apply (e.g., a diagnostic accuracy study that is also an AI study), check against all relevant guidelines and merge into one report.
• If the user requests a specific guideline, use that one regardless of auto-detection.

Step 2: Load Checklist

1. Run the fail-fast guard first for every guideline you intend to apply:

   python "${CLAUDE_SKILL_DIR}/scripts/check_checklist_exists.py" --guideline "STARD-AI"

• Exit 0 → the vendored checklist exists; read it from

${CLAUDE_SKILL_DIR}/references/checklists/ and proceed.

• Exit 1 (MISSING_CHECKLIST_CONTRACT_VIOLATION) → the guideline is routed

but no checklist file is vendored. Do not construct items from memory. Halt, report the violation to the user, and stop unless they explicitly opt in (next bullet).

• Exit 2 (UNKNOWN_GUIDELINE) → the name is not recognised; confirm the

correct guideline with the user.

2. No silent fallback. A from-memory checklist is permitted only when the user explicitly accepts it — re-run the guard with --allow-from-memory (exit 0 + a NON-AUTHORITATIVE warning). In that case the output report MUST carry a prominent banner that the assessment was constructed from model knowledge and is not backed by a vendored checklist, and submission_safe must not be asserted on its basis.

Step 3: Scan Manuscript

Read all sections of the manuscript thoroughly: 1. Title and abstract 2. Introduction 3. Methods (all subsections) 4. Results (all subsections) 5. Discussion 6. Tables, figures, and their captions 7. Supplemental materials (if available) 8. References (for registration numbers, protocol references)

Gather context from the full document before starting the item-by-item assessment.

Step 4: Assess Each Item

For every checklist item, determine:

For each item, record:

• Status: PRESENT / PARTIAL / MISSING / N/A
• Location: Section name and paragraph or approximate position (e.g., "Methods, paragraph 3")
• Notes: What was found (if PRESENT/PARTIAL) or what should be added (if MISSING)

Step 4b: Section Boundary Check

In addition to checklist items, verify that:

• Results section contains only factual findings: no interpretation, no "why" explanations,

no prior literature comparisons, no evaluative adjectives without numbers.

• Discussion section does not introduce new data not presented in Results.
• Flag any boundary violation as a separate finding in Part C Action Items with the label

[BOUNDARY].

Step 4c: Registration / Protocol Timing Consistency Check

Applies to: systematic reviews, meta-analyses, and intervention studies with prospective registration (PRISMA 2020, PRISMA-DTA, PRISMA-P, MOOSE, CONSORT, SPIRIT).

Why this step exists: the registration identifier is a single checklist item and can pass Step 4 even when the manuscript is internally inconsistent about when the registration or its amendments occurred relative to the analysis. An undisclosed post-hoc amendment is a common rejection trigger.

Five audit items (summary): (1) registration identifier present in Methods, Abstract, and cover letter; (2) initial registration date precedes — or is explicitly disclosed as post-dating — the extraction milestone; (3) amendment dates appear in Methods, the described change is visible in Methods, analysis was re-run if amendment post-dates the lock, and no amendment post-dates submission; (4) cross-artifact agreement between Methods and the registry record (PROSPERO PDF, ClinicalTrials.gov export) — silent discrepancy is a finding; (5) retrospective-registration disclosure paragraph when evidence suggests post-extraction filing.

Registration-ID format gate: a PROSPERO ID is CRD42 + 9 digits = 14 characters (^CRD42\d{9}$, e.g. CRD42024500001). Run grep -oE 'CRD42[0-9]+' manuscript.md and assert each match is 14 characters long; a 15-character ID (a stray inserted digit) is a transcription error logged as [REGISTRATION-TIMING] (fixable_by_ai: false — verify against the live PROSPERO record, do not guess the correct digit).

Flagging: any failure is logged in Part C Action Items with label [REGISTRATION-TIMING]. fixable_by_ai: false when reconciliation requires an external amendment filing; true only when the fix is a Methods-text insertion of a date already disclosed elsewhere. Part D JSON includes a registration_timing object (registry, id, initial_registration_date, amendments[], timing_consistency, findings[]).

Load-on-demand procedural detail (exact item-by-item procedure, JSON schema, flagging edge cases): ${CLAUDE_SKILL_DIR}/references/step4c_registration_timing.md.

Step 4d: PRISMA Figure 1 Arithmetic & Cross-Reference Audit

Applies to: systematic reviews and meta-analyses using PRISMA 2020 / PRISMA-DTA / PRISMA-P. Triggers when Item 16a (flow diagram) is PRESENT.

Why this step exists: the flow diagram is a single checklist item and can pass Step 4 visually while still containing arithmetic errors (records screened ≠ identified − duplicates; sought-for-retrieval ≠ screened − excluded) or text↔figure number disagreements. Senior MA reviewers commonly require strict PRISMA 2020 diagram conformance and explicit body↔ figure number agreement; reviewers who detect these mismatches lose confidence in the study's data integrity immediately.

Four arithmetic checks: 1. records screened = records identified − duplicates removed 2. records sought-for-retrieval = records screened − records excluded (screening) 3. reports retrieved = sought − reports not retrieved 4. studies included = reports assessed for eligibility − reports excluded (with reasons)

Two cross-reference checks:

• Body text PRISMA numbers (e.g., "315 records identified, 122 duplicates removed,

186 records screened") match Figure 1 box labels 1:1.

• Reasons for exclusion (Methods + Figure legend) agree on counts and category names.

Procedure: 1. Extract numbers from manuscript Results / PRISMA flow paragraph (regex: integers near keywords identified, duplicates, screened, excluded, sought, retrieved, assessed, included). 2. Extract numbers from Figure 1 source — preferred order: (a) analysis/figures/Figure1_PRISMA.md markdown manifest, (b) caption text in manuscript.md, (c) PPTX text run if .pptx exists, (d) manual entry from PNG/SVG. 3. Cross-check `analysis/figures/_figure_manifest.md` (produced by /make-figures): verify that the row whose Type = prisma (or Type = prisma-dta) points at the same file path used as the audit source, and that the row's Critic field is yes or partial (not no). A missing manifest row, mismatched path, or Critic = no flag logs [MANIFEST-XREF] (advisory) — the arithmetic check still runs against the source identified in step 2. Skip this sub-step if _figure_manifest.md does not exist (older projects). 4. Run 4 arithmetic checks; emit PRESENT / MISSING / MISMATCH per equation. 5. Run 2 cross-reference checks; emit PRESENT / MISSING / MISMATCH per number. 6. Output qc/prisma_figure_audit.json and a short table.

Flagging: any MISMATCH or arithmetic failure logs a Part C Action Item with label [PRISMA-FIGURE]. fixable_by_ai: false (numbers must be reconciled by the author).

Load-on-demand procedural detail (exact regex set, JSON schema, edge cases — duplicates handled across databases, citation searching strand, dual-reviewer screening): ${CLAUDE_SKILL_DIR}/references/step4d_prisma_figure_audit.md.

Cross-cutting: integrates with ~/.claude/rules/numerical-safety.md (PRISMA 5-way consistency: text ↔ Figure ↔ extraction CSV ↔ analysis script ↔ supplementary).

Step 4e: Reporting-Framework Naming Audit

Applies to: any manuscript that invokes an AI/extension reporting framework (PROBAST+AI, STARD-AI, TRIPOD+AI, TRIPOD-LLM, CONSORT-AI, SPIRIT-AI, PRISMA-DTA, QUADAS-C).

Why this step exists: a base reporting tool and its extension are distinct instruments with separate citations (manuscript-style-classical §14). Step 1 routes to the right checklist but does not police how the framework is named in prose. The recurring failures are: invoking an extension without ever naming or citing the base instrument it extends; mixing +AI and -AI hyphenation for one family within a single document; coining item labels like "12-AI"; and waving at "recent guidance" instead of naming the framework.

Run the deterministic gate:

python3 "${CLAUDE_SKILL_DIR}/scripts/check_framework_naming.py" \
  --manuscript manuscript.md --out qc/framework_naming.json --strict

Verdicts: BASE_MISSING (extension used, base instrument never named standalone) is a Major and logs [FRAMEWORK-NAMING] in Part C with fixable_by_ai: true (insert the base name + its citation). HYPHEN_MIX, CITE_MISSING, SELF_COINED_LABEL, and VAGUE_GUIDANCE are Minor (fixable_by_ai: true). Part D JSON includes a framework_naming object mirroring the script's claims[].

Step 4f: Critical-item floor cross-check

Applies to: every guideline assessment for which the floor defines a row (load and check only those; do not invent a floor for an unlisted guideline). After the item-by-item table, load ${CLAUDE_SKILL_DIR}/references/critical_item_floor.md and check the small set of non-waivable items for this study type. A MISSING critical item is surfaced as a Critical gap and becomes the report's headline regardless of the overall percentage — a high percentage with a missing critical item (undefined reference standard, no leakage-controlled partition, calibration absent for a prediction model, an unreconciled flow diagram) is not "broadly acceptable."

For AI/ML and radiomics manuscripts, also confirm the chosen methodological-quality / risk-of-bias instrument (PROBAST+AI, METRICS/RQS, APPRAISE-AI) and its non-waivable concerns — a fully reported paper can still be at high risk of bias. For radiomics, the fuller METRICS breakdown (9 categories / 30 weighted items) is in ${CLAUDE_SKILL_DIR}/references/appraisal_tools/METRICS.md (an appraisal reference, not a counted reporting checklist). Keep these distinct from the reporting counterparts (CLEAR, DECIDE-AI), which route through the normal checklist flow. Do not assert a numeric journal desk-reject threshold; the hard signals are a missing critical item and the journal's own required elements.

Step 5: Generate Report

Produce a structured compliance report in two parts.

Part A: Summary

## Reporting Guideline Compliance Report

Manuscript: {title}
Target manuscript file: {manuscript filename, e.g. manuscript_v8.md}
Target version: {version token from the filename or frontmatter, e.g. v8}
Guideline: {name and version}
Date: {YYYY-MM-DD}
Assessed by: Claude (automated pre-screening)

### Summary

| Status | Count | Percentage |
|--------|-------|------------|
| PRESENT | {n} | {%} |
| PARTIAL | {n} | {%} |
| MISSING | {n} | {%} |
| N/A | {n} | {%} |
| **Total** | **{n}** | **100%** |

Overall compliance: {PRESENT count}/{applicable count} ({%})

Critical items (Step 4f): {present}/{total} present.{ if any missing: " Critical gap — " + each MISSING critical item with the section it belongs in. This, not the percentage, is the headline.}

Part B: Item-by-Item Checklist

### Detailed Checklist

| # | Section | Item | Status | Location | Notes |
|---|---------|------|--------|----------|-------|
| 1 | Title/Abstract | {item text} | PRESENT | Title | {notes} |
| 2 | Introduction | {item text} | MISSING | -- | {suggestion} |
| ... | ... | ... | ... | ... | ... |

Part C: Action Items (for MISSING and PARTIAL)

### Action Items (Priority Order)

1. **[MISSING] Item {N}: {item name}**
   - Required: {what needs to be added}
   - Suggested location: {section, paragraph}
   - Example text: "{draft sentence or phrase}"

2. **[PARTIAL] Item {N}: {item name}**
   - Current: {what was found}
   - Needed: {what additional detail is required}
   - Suggested revision: "{draft revision}"

Order action items by: 1. Items most journals enforce strictly (e.g., ethics approval, registration, sample size) 2. Items in the Methods section (easiest to fix) 3. Items in other sections

Part D: Machine-Readable JSON Summary

Append a fenced JSON block at the end of the report. This enables /write-paper Phase 7 and /orchestrate to parse compliance results programmatically. This block MUST be present when invoked with --json flag or when called from /write-paper Phase 7. It SHOULD also be present in standard invocations (appended after Part C).

{
  "check_reporting_version": "1.1",
  "manuscript_title": "...",
  "target_manuscript": "manuscript_v8.md",
  "target_version": "v8",
  "source_sha256": "<first 12 hex chars of sha256 of the manuscript file bytes>",
  "guideline": "STARD-AI",
  "guideline_version": "2025",
  "date": "YYYY-MM-DD",
  "total_items": 40,
  "present": 32,
  "partial": 4,
  "missing": 3,
  "na": 1,
  "compliance_pct": 88.9,
  "action_items": [
    {
      "item_number": 12,
      "section": "Methods",
      "item_name": "Sample size justification",
      "status": "MISSING",
      "suggested_location": "Methods, after participant description",
      "suggested_fix": "Add: 'The sample size was determined based on [rationale]. A minimum of [N] cases was required to achieve [target] precision for the primary endpoint.'",
      "fixable_by_ai": true
    },
    {
      "item_number": 7,
      "section": "Methods",
      "item_name": "Blinding of index test to reference standard",
      "status": "PARTIAL",
      "current_text": "Readers were blinded",
      "needed": "Specify what readers were blinded to (reference standard results, clinical information, other reader results)",
      "suggested_fix": "Expand to: 'Readers interpreted [index test] images blinded to the reference standard results, clinical information, and other readers' assessments.'",
      "fixable_by_ai": true
    }
  ]
}

Field definitions:

• compliance_pct: present / (total_items - na) * 100, rounded to one decimal
• action_items: Array of MISSING and PARTIAL items only (PRESENT and N/A excluded)
• fixable_by_ai: true if the fix involves inserting or expanding text with information available in the manuscript or inferable from context; false if it requires external information (e.g., registration number, IRB approval number, specific protocol details only the author knows)
• suggested_fix: Concrete draft text that can be inserted or used to expand an existing sentence

---

Assessment Standards

Be Strict

• PARTIAL means the item is mentioned but lacks specificity. For example:
• "We used appropriate statistical tests" = PARTIAL (which tests?)
• "We used the Mann-Whitney U test for continuous variables and Fisher's exact test for categorical variables" = PRESENT
• A vague reference does not count as PRESENT. The detail level must match what the guideline expects.

Be Specific in Suggestions

• For MISSING items, provide a draft sentence the user can insert.
• For PARTIAL items, point to the exact gap and suggest specific additions.
• Reference the specific manuscript section where the addition should go.

Common Gaps to Watch For

These items are frequently missing in medical manuscripts:

1. Study registration number (CONSORT, PRISMA, STARD) 2. Registration / amendment date consistency (PRISMA 2020, PRISMA-DTA, CONSORT, SPIRIT) — run Step 4c whenever a registration identifier is present 3. Sample size justification (CONSORT, STROBE, STARD) 4. Missing data handling (all guidelines) 5. Blinding details (CONSORT, STARD) 6. Funding and conflicts of interest (all guidelines) 7. Ethics approval with committee name and approval number (all guidelines) 8. Data availability statement (increasingly required) 9. AI-specific: training/validation/test split details (TRIPOD+AI, CLAIM, STARD-AI) 10. AI-specific: model architecture and hyperparameters (TRIPOD+AI, CLAIM, STARD-AI) 11. AI-specific: failure mode analysis (CLAIM, STARD-AI) 12. AI-specific: fairness/bias assessment (STARD-AI) 13. AI-specific: commercial interests and data/code availability (STARD-AI) 14. Power-aware framing of a null result (STROBE 16a / 18 / 20) — for an observational study whose headline is a non-significant association, a flat "X was not associated with Y" overreads the data when the analysis is not powered to exclude a clinically meaningful effect. Mark item 18/20 PARTIAL unless the manuscript states the precision as an exclusion (e.g., "the 95% CI excluded an eGFR difference larger than ~1.7") or reports a minimum detectable effect — "no effect" vs "could not exclude an effect of size X" are different claims, and a negative conclusion needs the latter. 15. Confounder-selection rationale, not "adjust for everything that differs" (STROBE 16a explicitly asks which confounders were adjusted for and why) — flag a kitchen-sink adjustment set chosen because variables differ in Table 1. The Methods must give a causal rationale (DAG / prior literature) and must not adjust for a mediator or consequence of the outcome (over-adjustment, e.g. serum uric acid in an eGFR model); both an unjustified inclusion and an unjustified omission are item-16a gaps.

---

PRISMA Cascade Arithmetic Auto-Verify

PRISMA 2020 flow diagrams chain a cascade of subtractions (database records → after dedup → title/abstract screened → full-text reviewed → included in synthesis). Off-by-one errors in the prose cascade are a high-frequency reviewer red flag (e.g., 151 + 108 + 39 + 1 + 1 + 4 = 304 followed by a prose summary "305" four lines later).

When PRISMA 2020 or PRISMA-DTA is selected and round-by-round screening TSV artifacts are available, run the cascade auto-verify:

python "${CLAUDE_SKILL_DIR}/scripts/prisma_cascade_check.py" \
    --round1 2_Screening/round1.tsv \
    --round2 2_Screening/round2.tsv \
    --round3 2_Screening/round3_adjudication.tsv \
    --manuscript manuscript.md \
    --out qc/prisma_cascade.json

The script: 1. Reads the round TSVs and counts INCLUDE / EXCLUDE / MAYBE decisions per round. 2. Computes the cascade arithmetic from raw decisions (no prose). 3. Optionally grep the manuscript for matching stage-count claims and emits per-stage drift when the prose disagrees.

Treat any manuscript_drift entry as a P0 blocker — fix the prose to match the computed cascade and re-run.

Submission Checklist Export

Many journals require a filled reporting checklist to be submitted alongside the manuscript. When the user asks for a submission-ready checklist, format the output as:

{Guideline Name} Checklist

Manuscript title: {title}
Date: {YYYY-MM-DD}

| Item # | Checklist Item | Reported on Page # | Reported in Section |
|--------|---------------|-------------------|-------------------|
| 1 | {item text} | {page or N/A} | {section} |
| 2 | {item text} | {page or N/A} | {section} |
| ... | ... | ... | ... |

Page numbers should be filled in by the user after final formatting. Use section names as placeholders.

---

Skill Interactions

---

Error Handling

• If the manuscript file cannot be read, ask the user for the correct path.
• If the study type is ambiguous, ask the user to confirm before selecting a guideline.
• If a checklist item is genuinely unclear in its applicability, mark as N/A with justification.
• This is a pre-screening tool. Always remind the user that final compliance should be verified by all co-authors and ideally by a methodologist.

Language

• Checklist content and compliance report: English
• Communication with user: Match user's preferred language
• Medical terms: English only

Anti-Hallucination

• Never fabricate references. All citations must be verified via /search-lit with confirmed DOI or PMID. Mark unverified references as [UNVERIFIED - NEEDS MANUAL CHECK].
• Never invent clinical definitions, diagnostic criteria, or guideline recommendations. If uncertain, flag with [VERIFY] and ask the user.
• Never fabricate numerical results — compliance percentages, scores, effect sizes, or sample sizes must come from actual data or analysis output.
• If a reporting guideline item, journal policy, or clinical standard is uncertain, state the uncertainty rather than guessing.

---

Gates

METRICS — radiomics methodological-quality appraisal

METhodological RadiomICs Score (METRICS) · EuSoMII-endorsed quality-scoring tool Reference: Kocak B, et al. Insights Imaging 2024;15:8. doi:10.1186/s13244-023-01572-w (CC BY 4.0) Tool / calculator: https://metricsscore.github.io/metrics/METRICS.html

This is an appraisal (methodological-quality) tool, NOT a reporting guideline. It answers

"was the radiomics study done well (low risk of bias)?", which is distinct from "was it

reported?" (that is CLEAR — a reporting checklist). It therefore lives under appraisal_tools/

and does not count toward the reporting-guideline catalog. Use it alongside, not instead of,

the reporting checklist. Educational summary in our own words from the CC BY 4.0 source; complete

the official weighted calculator for a submission-ready score and cite Kocak et al. 2024.

Scope and how it scores

• Covers radiomics research from handcrafted features to fully deep-learning pipelines.
• 30 items across 9 categories. Items carry condition-dependent weights and some categories

are conditional (apply only when that step is present — e.g., manual segmentation, explicit feature selection); the official tool produces a weighted percentage with quality bands.

• Pair with the reporting checklist (CLEAR) and, for prediction-model risk of bias, PROBAST+AI —

a fully reported radiomics paper can still be at high methodological risk of bias.

Categories (9) and the concerns each covers

Non-waivable concerns (surface as a Critical gap if absent)

Consistent with the critical-item floor's appraisal note, the highest-yield METRICS concerns are:

• Feature reproducibility / stability — test–retest, inter-observer/segmentation stability, ICC-based filtering (category 5; segmentation, category 3).
• *Internal and external validation, with multiplicity control* — a single internal split is not

enough for a generalisation claim (categories 6–8).

• Calibration, not discrimination only — a probability that drives a decision must be calibrated

(category 7).

• Leakage-controlled partitioning — feature selection / preprocessing fit on pooled data, or the

same data used for tuning and testing, inflates performance (category 6).

How to use in the report

• This note backs the Step 4f appraisal cross-check (critical_item_floor.md, METRICS/RQS row).

After the reporting item-by-item table, confirm the manuscript's chosen methodological-quality instrument and whether the non-waivable concerns above are met.

• Do not fold the METRICS score into the reporting compliance percentage — keep appraisal

(risk of bias / quality) and reporting (completeness) separate, and do not assert a journal desk-reject threshold from either.

• For the exact item wording, weights, and quality bands, use the official EuSoMII calculator.

AMSTAR 2 Checklist

A MeaSurement Tool to Assess systematic Reviews, version 2 Version: AMSTAR 2 (2017) Source: Shea BJ et al. BMJ 2017;358:j4008. doi: 10.1136/bmj.j4008

Checklist Items (16 items)

Items

---

Overall Confidence Rating

AMSTAR 2 does NOT generate a numerical score. Instead, rate overall confidence:

Critical Domains (7 of 16)

Items 2, 4, 7, 9, 11, 13, 15 are considered critical domains. A flaw in any critical domain results in at least "Low" confidence.

Notes for Assessors

• AMSTAR 2 replaces the original AMSTAR (2007)
• Designed for systematic reviews of interventions (RCTs and/or NRSI)
• Not designed for DTA reviews (use ROBIS or domain-specific tools)
• Cannot be used to assess individual primary studies
• The tool should NOT be used to generate an overall score — use the confidence rating scheme above
• For reviews including NRSI: Item 9 should assess confounding, selection bias, and information bias

ARRIVE 2.0 Checklist — Animal Research

Reference: Percie du Sert N et al. The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research. PLoS Biol. 2020;18(7):e3000410. PMID: 32663219

Website: https://arriveguidelines.org

---

How to Use This Checklist

ARRIVE 2.0 has two tiers:

• Essential 10 (Items 1–10): Non-negotiable. All journals require these. Missing any = REJECTED.
• Recommended (Items 11–21): Best practice. Complete if possible.

For each item: PRESENT / PARTIAL / MISSING

---

ESSENTIAL 10 — Non-Negotiable

Item 1 — Study Design ⚠️ commonly missed

Description: For each experiment, provide brief details of the study design. Required content:

• Type of study design (e.g., parallel group, crossover, factorial, dose-response)
• Number of experimental groups and description of each
• Independent experimental units (e.g., animal, litter, cage)
• For any within-animal design: outline the design clearly

Example: "Animals were randomly allocated to one of three parallel groups: (1) vehicle control (n=10), (2) low-dose treatment (n=10), (3) high-dose treatment (n=10). Each animal was an independent experimental unit."

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 2 — Sample Size ⚠️ commonly missed

Description: Specify the number of animals used and rationale for this number. Required content:

• Total number of animals used AND number per group
• Method of sample size determination:
• Formal power calculation (provide effect size, α, power, formula, software) OR
• Pragmatic reasons (limited number of animals available, pilot study) — state explicitly

Power calculation example: "Sample size was estimated using G*Power 3.1 based on a two-sample t-test, assuming an expected effect size of Cohen's d = 1.2 (derived from [Author Year]), α = 0.05 (two-tailed), and 80% power. The calculation indicated 15 animals per group were required. Accounting for 20% expected attrition, 18 animals per group were enrolled."

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 3 — Inclusion/Exclusion Criteria ⚠️ commonly missed

Description: Describe any criteria used for including or excluding animals or data. Required content:

• Criteria for excluding animals from the experiment (pre-specified, not post-hoc)
• Criteria for excluding data from analysis (e.g., outliers — method for outlier detection)
• Any animals that were excluded and why

Example: "Animals were excluded if: (a) weight loss exceeded 20% at any time point, (b) technical failure of surgical procedure occurred, or (c) infection was detected. Any exclusion was documented in the study log at the time of occurrence. Two animals were excluded (one from Group A due to procedure failure; one from Group C due to infection)."

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 4 — Randomization ⚠️ commonly missed

Description: Describe how animals were allocated to experimental groups. Required content:

• Method of randomization (computer-generated, random number table)
• Type: simple, block, stratified
• Who performed randomization (and whether separate from who conducted experiments)
• If randomization not used, justify

Example: "Animals were randomly allocated to groups using computer-generated random numbers (R version 4.2.0). Block randomization (block size = 6) was used to ensure balanced allocation across experimental days. Allocation was performed by a researcher (initials) not involved in data collection."

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 5 — Blinding ⚠️ commonly missed

Description: Describe who was aware of group allocation during different stages. Required content: State who was blinded (or not) at each stage:

• During the experiment (treatment administration)
• During outcome assessment
• During data analysis

"Outcome assessors were blinded to group allocation during behavioral testing and histological analysis. The researcher administering treatments could not be blinded due to the nature of the procedure; however, outcome assessment was performed by a blinded assessor."

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 6 — Outcome Measures ⚠️ commonly missed

Description: Clearly define all outcome measures used and how they were assessed. Required content:

• Primary outcome: define explicitly (what is measured, how, by whom)
• Secondary outcomes: same level of detail
• Timing of measurements (baseline, Day 7, Day 14, etc.)
• Observer reliability: intra- and inter-observer variability assessed?

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 7 — Statistical Methods ⚠️ commonly missed

Description: Describe the statistical methods used for each analysis. Required content:

• Primary statistical test for primary outcome
• Rationale for choice (parametric vs. non-parametric, based on what?)
• Handling of repeated measures (linear mixed-effects model preferred over repeated ANOVA)
• Multiple comparison correction: which method (Bonferroni, FDR)?
• Significance threshold (α)
• Software: R (version), SPSS (version), GraphPad Prism (version)

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 8 — Experimental Animals ⚠️ commonly missed

Description: Provide details of animals used in the study. Required content:

• Species and strain/substrain (with substrain details — this matters!)
• Source (commercial vendor; in-house breeding)
• Sex (justify if single sex)
• Age and weight at start of experiment (mean ± SD or range)
• Health status (specific pathogen-free, germ-free, defined flora, conventional)
• For transgenic animals: complete genotype

Example: "Male C57BL/6J mice (Jackson Laboratory, Bar Harbor, ME; stock #000664) aged 8–10 weeks (weight: 22.3 ± 1.8 g at enrollment) were used. Only males were used because [biological justification]. All animals were specific pathogen-free."

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 9 — Experimental Procedures ⚠️ commonly missed

Description: For each experiment and each experimental group, including controls, describe the procedures in enough detail to allow replication. Required content:

• Exact procedure steps
• Anesthesia: agent, dose, route, monitoring (temperature, heart rate)
• Analgesia: pre- and post-operative
• Humane endpoints: pre-specified criteria for euthanasia
• Equipment: manufacturer, model, settings
• Any details that could affect reproducibility (time of day, number of animals per session)

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

Item 10 — Results ⚠️ commonly missed

Description: Report the results for each experiment conducted, with a measure of precision (e.g., SD, SEM, 95% CI). Required content:

• All pre-specified outcomes (not just significant ones)
• Effect size with 95% CI
• Exact p-values
• N at each time point (with attrition documented)
• Adverse events (even if none: "No adverse events were observed")
• Individual data points for small groups (dot plots)

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING Location: ___

---

RECOMMENDED ITEMS — Best Practice

Item 11 — Abstract ⚠️ commonly missed

Description: Provide an accurate summary of the research objectives, animal species used, key methods, principal findings, and study conclusions. Required content: Abstract should contain enough information for readers to understand: what species, what procedure, primary result with magnitude, conclusion. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 12 — Background

Description: Explain why the study was done; include sufficient background to inform the reader why this animal model was chosen. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 13 — Objectives

Description: Clearly describe the research question, hypothesis, and the specific objectives of the study. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 14 — Ethical Statement ⚠️ commonly missed

Description: Provide the name of the ethics committee or equivalent, and the approval ID. Required content: "All animal experiments were approved by the Institutional Animal Care and Use Committee of [Institution] (protocol number [XXX])." Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 15 — Housing and Husbandry ⚠️ commonly missed

Description: Describe the conditions under which animals were housed and cared for. Required content: Cage type and dimensions, group or individual housing (N per cage), temperature (°C), humidity (%), light cycle, food and water access (ad libitum or restricted; diet specification), acclimatization period, enrichment. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 16 — Animal Care and Monitoring

Description: Describe the welfare assessments that were carried out before, during, and after the experiment. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 17 — Interpretation

Description: Interpret the results, taking into account the study objectives and hypotheses, current theory, and other relevant studies. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 18 — Generalisability/Translation

Description: Comment on whether, and how, the findings of this study are likely to translate to other species or systems, and the relevance to human biology. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 19 — Limitations ⚠️ commonly missed

Description: Discuss the potential limitations of the study. Required content: At minimum: (a) model limitations relative to human disease, (b) single sex (if applicable), (c) single center/vendor, (d) follow-up duration, (e) sample size for secondary outcomes. Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 20 — Protocol Registration

Description: Provide a statement indicating whether a study protocol was pre-registered. Required content: OSF (osf.io), PROSPERO for systematic reviews, or equivalent. State registration ID or that registration was not done (with justification if exploratory study). Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

Item 21 — Data Availability ⚠️ commonly missed

Description: Provide a statement describing if and where study data are available. Required content: "Data supporting this study are available at [repository/DOI]." or "Data are available upon reasonable request from the corresponding author." Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

Summary

Verdict: [ ] All Essential 10 PRESENT → proceed to submission [ ] Any Essential 10 MISSING → MUST REVISE before submission

CARE Checklist

CAse REports (CARE) guidelines Version: CARE 2013 Source: https://www.care-statement.org Reference: Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D. The CARE guidelines: consensus-based clinical case report guideline development. J Clin Epidemiol 2014;67(1):46-51.

Checklist Items (13 topics)

Title and Key Words

Abstract

Introduction

Patient Information

Clinical Findings

Timeline

Diagnostic Assessment

Therapeutic Intervention

Follow-up and Outcomes

Discussion

Patient Perspective

Informed Consent

---

Applying CARE to common case-report subtypes

CARE covers the single-patient narrative. Two frequent subtypes need additions CARE alone does not name:

• Adverse drug / device / contrast reaction (pharmacovigilance). Beyond item 8 (diagnostic

assessment), report a named causality instrument (Naranjo or WHO-UMC) with score and tier, document dechallenge (withdrawal → resolution) and the exposure-to-onset latency, and consider severity (e.g., Modified Hartwig–Siegel) and preventability (e.g., Schumock–Thornton). Locate the event against a denominator and note safety-reporting. Causality asserted without an instrument, dechallenge, or exclusion of alternatives is a reporting gap.

• Case series (n≥2). CARE is single-patient; a series additionally needs cohort-style methods

(design, setting, case-identification source, eligibility, protocol) and an all-cases summary table. Report counts, not rates — a selected/referral series cannot estimate prevalence — and state selection/ascertainment with the screened pool size. See /write-paper paper_types/case_series.md.

Educational summary of the CARE 2013 checklist (CC BY-NC 4.0). Cite the original guideline (Gagnier et al. 2014) and consult https://www.care-statement.org for the authoritative, full checklist with explanation and elaboration.

CLAIM 2024 Checklist

Checklist for Artificial Intelligence in Medical Imaging Version: CLAIM 2024 Update Source: https://pubs.rsna.org/doi/10.1148/ryai.240300 Reference: Tejani AS, Klontzas ME, Gatti AA, Mongan JT, Moy L, Park SH, Kahn CE Jr. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiol Artif Intell 2024;6(4):e240300.

Note: The 2024 update replaces "ground truth" with "reference standard" and discourages "validation" in favour of "internal/external testing". Each item is answered Yes / No / Not Applicable with the manuscript location cited.

Checklist Items (44 items)

Title and Abstract

Introduction

Methods — Study Design

Methods — Data

Methods — Reference Standard

Methods — Data Partitions

Methods — Testing Data

Methods — Model

Methods — Training

Methods — Evaluation

Results — Data

Results — Model Performance

Discussion

Other Information

---

Educational summary of the CLAIM 2024 Update checklist (© RSNA, open access). Cite the original (Tejani et al., Radiol Artif Intell 2024;6(4):e240300) and consult the RSNA article for the authoritative, full checklist.

CLEAR Checklist

CheckList for EvaluAtion of Radiomics research Version: CLEAR 2023 Source: https://doi.org/10.1186/s13244-023-01415-8 Reference: Kocak B, Baessler B, Bakas S, et al. CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. Insights Imaging. 2023;14(1):75. doi:10.1186/s13244-023-01415-8

Checklist Items (58 items)

Domain 1: Study Design (Items 1-8)

Domain 2: Imaging Data (Items 9-18)

Domain 3: Segmentation (Items 19-24)

Domain 4: Feature Extraction and Processing (Items 25-34)

Domain 5: Modeling (Items 35-44)

Domain 6: Performance Assessment (Items 45-52)

Domain 7: Open Science (Items 53-58)

---

Notes for Assessors

• CLEAR is specifically designed for radiomics studies. For deep learning studies without handcrafted radiomics features, CLAIM 2024 or TRIPOD+AI may be more appropriate.
• Items in Domain 3 (Segmentation) may not apply if the study uses atlas-based or fully automated segmentation without reader involvement.
• Items 17 (phantom data) and 57 (pre-registration) are aspirational best practices; their absence should be noted but may be scored as N/A if justified.
• Item 29 (IBSI compliance) is strongly recommended. Non-IBSI-compliant software should be flagged as a potential limitation.
• For studies combining radiomics with deep learning features, assess both CLEAR (for radiomics components) and CLAIM/TRIPOD+AI (for deep learning components).
• Domain 7 (Open Science) items are increasingly required by journals. Missing items here are more likely to trigger reviewer comments in high-impact journals.
• CLEAR was endorsed by the European Society of Radiology (ESR) and the European Society of Medical Imaging Informatics (EuSoMII).

CONSORT-AI Checklist

Consolidated Standards of Reporting Trials -- Artificial Intelligence Extension Version: CONSORT-AI 2020 (extends CONSORT 2010) Source: https://www.consort-spirit.org · EQUATOR Network Reference: Liu X, Cruz Rivera S, Moher D, et al. Nat Med 2020;26(9):1364-1374. doi:10.1038/s41591-020-1034-x (CC BY 4.0)

Educational summary, authored in our own words from the CC BY 4.0 source. Use the official

CONSORT-AI checklist for a submission-ready form and cite Liu et al. 2020.

Naming and scope (read first)

• CONSORT-AI is an extension of CONSORT 2010, for reports of randomized clinical trials of

interventions that include an AI/ML component. Apply both: every base CONSORT 2010 item plus the AI-specific items below; name and cite both instruments (manuscript-style-classical §14).

• It is the reports counterpart of SPIRIT-AI (trial protocols). For a trial protocol use

SPIRIT-AI; for the completed-trial report use CONSORT-AI.

• The AI items elaborate existing CONSORT items (numbered to match), so assess them alongside the

parent item.

AI-specific extension items

Status each PRESENT / PARTIAL / MISSING / N/A.

Title and Abstract

Introduction — Background and objectives

Methods

Results

Other Information

---

Notes for Assessors

• Apply CONSORT-AI with all base CONSORT 2010 items — the AI items do not replace them.
• Algorithm version (5 (i)) is non-waivable: a trial result tied to an unspecified model version is

not interpretable or reproducible. Mark MISSING if absent.

• Input-data eligibility (4a (ii)) is distinct from participant eligibility — a frequent omission;

a trial can enroll eligible patients yet feed the AI out-of-distribution inputs.

• Human–AI interaction (5 (iv)) and how outputs fed decisions (5 (vi)) determine whether the

trial evaluated the AI as used in practice; vague "the model assisted clinicians" is PARTIAL.

• Performance-error analysis (19) and code accessibility (25) are commonly dropped and are

where AI-specific safety/reproducibility live.

• Use SPIRIT-AI for the trial protocol; CONSORT-AI is for the completed-trial report.

CONSORT 2025 Checklist

Consolidated Standards of Reporting Trials Version: CONSORT 2025 Source: https://www.consort-spirit.org Reference: Hopewell S, Chan AW, Collins GS, et al. CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ 2025;389:e081123 (published simultaneously in BMJ, JAMA, Lancet, Nature Medicine, PLoS Medicine).

Note: CONSORT 2025 supersedes CONSORT 2010. It is a 30-item checklist (seven new items, three revised, one deleted) restructured with a new Open Science section.

Checklist Items (30 items)

Title and Abstract

Open Science

Introduction

Methods

Results

Discussion

---

Educational summary of the CONSORT 2025 checklist (CC BY 4.0). Cite the original statement (Hopewell et al., BMJ 2025) and consult https://www.consort-spirit.org for the authoritative, full checklist with explanation and elaboration.

COSMIN Risk of Bias Assessment Guide

COnsensus-based Standards for the selection of health Measurement INstruments — Risk of Bias tool for reliability and measurement error. Reference: Mokkink LB et al. BMC Medical Research Methodology 2020;20:293. Website: https://www.cosmin.nl

Purpose

The COSMIN Risk of Bias tool assesses the methodological quality of studies on reliability and measurement error of outcome measurement instruments (e.g., questionnaires, imaging measurements, lab tests).

Structure

Two parts:

• Part A: Understanding how the study informs on reliability/measurement error (7 elements of a comprehensive research question)
• Part B: Assessing quality using standards (9 for reliability, 8 for measurement error)

Quality rating per standard: Very Good / Adequate / Doubtful / Inadequate

Overall quality uses the worst-score-counts principle (lowest rating across all standards).

Part A: Elements of a Comprehensive Research Question

Extract these 7 elements from the study:

Components of Outcome Measurement Instruments

Without biological sampling

• Equipment (questionnaire forms, imaging device, software)
• Preparatory actions (training, calibration, patient instructions)
• Collection of raw data (patient completing questions, rater observing)
• Data processing and storage (digitization, image reconstruction)
• Assignment of score (formula, algorithm, clinical judgment)
• Optional further use (conversion to another scale/classification)

With biological sampling

• Equipment (collection tools, containers, lab equipment)
• Preparatory actions (patient prep, equipment calibration)
• Collection of sample (blood draw, biopsy)
• Processing and storage (centrifugation, freezing)
• Determination of value (assay, measurement)
• Optional further use (conversion to classification)

Part B: Standards for Studies on Reliability

Standard 1: Study Design

• Were patients measured independently by raters (inter-rater) or on separate occasions (test-retest)?
• Was the time interval appropriate (short enough to avoid true change, long enough to avoid memory)?

Standard 2: Missing Data

• Were missing data adequately handled?
• Was the percentage of missing data acceptable?

Standard 3: Sample Size

• Was the sample size adequate? (minimum 30 recommended, 50+ preferred)

Standard 4: Patients

• Were patients selected to represent a range of the construct being measured?
• Were patients stable between measurements (for test-retest)?

Standard 5: Independent Measurements

• Were measurements performed independently of each other?
• Were raters blinded to previous measurements or other rater's results?

Standard 6: Similar Conditions

• Were measurement conditions similar across repeated measurements?
• Were there important differences in testing conditions?

Standard 7: Appropriate Statistical Methods

• Was the appropriate reliability parameter calculated (ICC for continuous, kappa for categorical)?
• Was the correct ICC model used (one-way, two-way, agreement vs consistency)?

Standard 8: ICC Model and Formula

• Were the ICC model and formula correctly specified?
• Was the ICC type appropriate for the study design (single measures vs average measures)?

Standard 9: Reporting

• Were sufficient statistics reported (ICC/kappa with 95% CI)?
• Were descriptive statistics reported (means, SDs)?

Part B: Standards for Studies on Measurement Error

Standard 1: Study Design

Same as reliability Standard 1.

Standard 2: Missing Data

Same as reliability Standard 2.

Standard 3: Sample Size

Same as reliability Standard 3.

Standard 4: Patients

Same as reliability Standard 4.

Standard 5: Independent Measurements

Same as reliability Standard 5.

Standard 6: Similar Conditions

Same as reliability Standard 6.

Standard 7: Appropriate Parameter

• Was the appropriate measurement error parameter calculated (SEM, LoA, SDC, or percentage agreement)?

Standard 8: Reporting

• Were sufficient statistics reported (SEM, LoA, or SDC with 95% CI)?
• Were Bland-Altman plots provided where appropriate?

Overall Quality Rating

Uses the worst-score-counts principle:

• Rate each standard as: Very Good / Adequate / Doubtful / Inadequate
• Overall rating = lowest rating across all applicable standards

When to Use

• Systematic reviews of measurement properties of health measurement instruments
• Selecting outcome measurement instruments for clinical trials or research
• Developing core outcome sets (COS)
• Evaluating reliability/agreement of imaging measurements, scoring systems, clinical tests
• Typically used alongside other COSMIN boxes (content validity, structural validity, etc.)

DECIDE-AI Checklist

Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence Version: DECIDE-AI 2022 Source: https://www.equator-network.org · https://www.decide-ai.org Reference: Vasey B, et al. Nat Med 2022;28(5):924-933. doi:10.1038/s41591-022-01772-9

Educational summary, authored in our own words. DECIDE-AI materials are CC BY-NC; this file

paraphrases the intent of each item and copies no verbatim guideline wording. Complete the

official DECIDE-AI checklist for a submission-ready form and cite Vasey et al. 2022.

Naming and scope (read first)

• DECIDE-AI reports the early-stage (live) clinical evaluation of an AI decision-support system —

the development-to-implementation gap between offline model validation (TRIPOD+AI / STARD-AI / CLAIM) and a definitive trial (CONSORT-AI). It is not a model-accuracy guideline; it is about how the AI behaves, is used, and is kept safe in real clinical workflow during first-in-human use.

• It has 17 AI-specific items (28 subitems) plus 10 generic reporting items (study identifiers,

objectives, setting, sample-size rationale, statistics, funding/COI, registration, etc. — apply these like any reporting guideline). The AI-specific items below are the core.

• Distinct from CONSORT-AI (definitive RCT report) and SPIRIT-AI (trial protocol); use DECIDE-AI for

the exploratory/early live-clinical evaluation stage.

AI-specific items (17)

Status each PRESENT / PARTIAL / MISSING / N/A.

Title / Abstract

Introduction

Methods

Results

Discussion

Other

---

Notes for Assessors

• Apply the 10 generic items as well (study identifiers, objectives, setting, sample-size rationale,

statistical methods, funding/COI, registration) — DECIDE-AI assumes standard reporting underneath.

• Human factors / learning curve (7, 14) and human–computer agreement / override (12) are what

make DECIDE-AI distinct from offline-accuracy guidelines; a study that reports only model metrics and no real-use human-interaction data is PARTIAL/MISSING on the core of DECIDE-AI.

• Safety and errors (6/13/16) must run end-to-end: pre-defined → captured → discussed. A safety

claim with no pre-defined error capture is PARTIAL.

• Implementation (5/10) is about workflow integration and actual use, not just deployment intent.
• Modifications (11): silent mid-study model/threshold changes without disclosure undermine the

evaluation — mark MISSING if changes are implied but not reported.

• Use CONSORT-AI for a definitive trial and TRIPOD+AI / STARD-AI / CLAIM for offline model

development/accuracy; DECIDE-AI covers the early live-clinical evaluation between them.

GRRAS Checklist

Guidelines for Reporting Reliability and Agreement Studies Version: GRRAS 2011 Source: https://doi.org/10.1016/j.jclinepi.2010.03.002 Reference: Kottner J, Audige L, Brorson S, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. doi:10.1016/j.jclinepi.2010.03.002

Checklist Items (15 items)

Title and Abstract

Introduction

Methods

Results

Discussion

---

Notes for Assessors

• GRRAS applies to all reliability and agreement studies, including inter-rater, intra-rater, test-retest, and method-comparison designs across any medical discipline.
• Reliability vs Agreement: These are distinct concepts. Reliability (e.g., ICC, kappa) reflects the ability to distinguish between subjects; agreement (e.g., Bland-Altman limits of agreement, SEM) reflects how close scores are on repeated measurements. A study may report one or both.
• Item 4 (Study design): The ICC model choice (one-way random, two-way random, two-way mixed) and type (single measures, average measures) must match the study design. Flag mismatches as a Major Comment.
• Item 7 (Raters): Rater characteristics strongly influence reliability estimates. Studies using expert raters only will overestimate reliability in general clinical practice. Note this if raters are not representative.
• Item 9 (Order of measurements): Randomization of measurement order is critical for avoiding memory and learning effects. Its absence should be flagged as a limitation.
• Item 12: For ICC, report the model, type, and definition (consistency vs absolute agreement). For kappa, specify weighted vs unweighted and the weighting scheme. Bare numbers without these specifications are PARTIAL.
• Common in radiology: inter-reader agreement for measurements (Bland-Altman, ICC), segmentation agreement (Dice, ICC of volumes), and diagnostic classification agreement (kappa, percent agreement).
• GRRAS is complementary to study-type checklists. For example, a diagnostic accuracy study reporting inter-reader agreement should follow STARD for the main analysis and GRRAS for the agreement component.

MI-CLEAR-LLM Checklist — Reporting LLM Accuracy Studies in Healthcare

Reference: Park SH, Suh CH, Lee JH, Kahn CE Jr, Moy L. Minimum Reporting Items for Clear Evaluation of Accuracy Reports of Large Language Models in Healthcare (MI-CLEAR-LLM). Korean J Radiol. 2024;25(10):865-868. PMID: 39344542 Reference (2025 Update): Park SH, Suh CH, Lee JH, Tejani AS, You SC, Kahn CE Jr, Moy L. MI-CLEAR-LLM: 2025 Updates. Korean J Radiol. 2025;26(12):1123-1132. PMID: 41199132

Version: 2025 (expanded from 2024 original) License: CC BY-NC 4.0 (Korean Society of Radiology) Source: https://kjronline.org/DOIx.php?id=10.3348/kjr.2025.1522

Scope: Studies evaluating the accuracy of LLMs in healthcare applications (diagnosis, triage, clinical decision support, medical question answering, etc.). This is NOT for disclosing LLM use in manuscript writing — for that, see ICMJE/COPE policies and the write-paper skill's LLM disclosure feature.

vs. CLAIM 2024: Use CLAIM for imaging AI model validation. Use MI-CLEAR-LLM for studies testing LLM accuracy in healthcare tasks. Both may apply if the study evaluates an LLM integrated with medical imaging interpretation.

---

Item 1 — LLM Identification and Specifications ⚠️ commonly missed

Description: Fully identify the LLM evaluated in the study. Required content:

• Model name and version (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro)
• Manufacturer/developer (e.g., OpenAI, Anthropic, Google)
• Training data cutoff date (if known or disclosed by the developer)
• Date(s) when queries were executed (LLM outputs may change over time due to updates)
• Access mode: web-based chatbot interface, API, or locally deployed open-source model
• For API access: specific API version or endpoint used
• For open-source models: model weights version, quantization method (if applicable), hardware specifications
• Whether retrieval-augmented generation (RAG) or internet access was enabled
• Any fine-tuning or adaptation applied to the base model

Rationale: LLMs are frequently updated without version increments. The same model name may produce different outputs on different dates. Without exact version and date documentation, studies are not reproducible.

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

Item 2 — Stochasticity Handling ⚠️ commonly missed

Description: Document how the inherent randomness of LLM outputs was managed. Required content:

• Number of query attempts per question/task (e.g., each prompt was run 3 times)
• How multiple outputs were synthesized (e.g., majority vote, best-of-N, mean score)
• Reliability analysis across repeated attempts (e.g., agreement rate, Fleiss' kappa)
• Technical parameter settings that control randomness:
• Temperature setting (e.g., temperature = 0 for deterministic output)
• Top-p (nucleus sampling) setting
• Any other sampling parameters (top-k, frequency penalty, etc.)
• If parameters were not controllable (e.g., web chatbot interface): state this explicitly

Rationale: LLM outputs are stochastic by design. A single query attempt is insufficient to characterize model performance. Studies that do not control or report stochasticity parameters produce unreliable and unreproducible results.

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

Item 3 — Full Prompt Text ⚠️ commonly missed

Description: Provide the complete text of all prompts used, preserving exact formatting. Required content:

• Full prompt text including system prompts, user prompts, and any few-shot examples
• Preserve precise spellings, symbols, punctuation, spaces, line breaks, and formatting
• If prompts include clinical data or images: describe the format and structure of input data
• For multi-turn conversations: provide the complete conversation sequence
• For chain-of-thought or structured prompting: document the full prompting strategy
• Recommended location: supplementary materials (if prompts are lengthy)

Rationale: Minor prompt variations can substantially alter LLM outputs. Without exact prompt text, no reader can assess the study methodology or attempt replication.

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

Item 4 — Prompt Execution Details ⚠️ commonly missed

Description: Explain how prompts were operationally used during the study. Required content:

• Whether each query was executed as an independent chat session (new conversation) or within a continuing session
• Whether multiple queries were submitted simultaneously (batch) or sequentially
• If sequentially: whether prior outputs could influence subsequent queries (context carryover)
• Whether any post-processing was applied to LLM outputs (e.g., extracting structured answers from free text, parsing JSON)
• Who executed the queries (researchers, automated scripts, clinical staff)
• For API-based studies: batch size, rate limiting, error handling for failed API calls

Rationale: Chat session context and query ordering can influence LLM responses. Studies must specify whether the LLM had access to context from prior queries, as this affects comparability.

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

Item 5 — Prompt Testing and Optimization ⚠️ commonly missed

Description: Describe the prompt development process. Required content:

• How prompts were created (expert-designed, iteratively refined, generated by another LLM)
• Rationale for specific wording choices (why this prompt structure was chosen)
• Whether prompt optimization/engineering was performed
• If optimized: method used (manual iteration, automated prompt tuning, grid search over prompt variants)
• Dataset used for prompt optimization (must be independent of test data — see Item 6)
• Number of prompt iterations tested
• Whether different prompt strategies were compared (zero-shot vs. few-shot vs. chain-of-thought)

Rationale: Prompt design is a critical methodological choice equivalent to feature engineering in traditional ML. Unreported optimization inflates apparent performance if optimization and test data overlap.

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

Item 6 — Test Data Independence ⚠️ commonly missed

Description: Confirm separation between test data and all other data used in the study. Required content:

• Explicit statement that test data were not used during prompt development or optimization
• Whether test data could have been part of the LLM's training data (data contamination risk)
• If test data were sourced from the internet: provide exact URLs and confirm whether the content was publicly accessible before the LLM's training cutoff date
• For published medical examination questions: acknowledge that these may be in the LLM's training corpus and discuss implications
• Mitigation strategies for data contamination (e.g., using unpublished cases, temporal splits, institution-specific data)

Rationale: If test data were seen during prompt optimization or were part of the LLM's pretraining corpus, performance estimates are inflated. Unlike traditional ML where training data are known, LLM training data are often undisclosed, making contamination assessment particularly challenging.

Status: [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

Notes for Assessors

When to Apply MI-CLEAR-LLM

Apply this checklist when the study's primary aim is to evaluate LLM accuracy, performance, or clinical utility in a healthcare context. Examples:

• LLM answering medical board examination questions
• LLM performing clinical reasoning or differential diagnosis
• LLM triaging patient messages or radiology reports
• LLM extracting structured data from clinical notes
• LLM generating radiology report impressions

Do NOT apply this checklist when:

• The study uses LLMs as a tool in the research pipeline (e.g., NLP preprocessing) but does not evaluate LLM accuracy as an outcome
• The manuscript merely discloses that LLM writing assistance was used

Common Assessment Pitfalls

1. Item 1 — "GPT-4" is insufficient. Require the exact model version (e.g., gpt-4-0613, gpt-4-turbo-2024-04-09) and the access date. OpenAI and other providers frequently update models behind the same name. 2. Item 2 — Single-run studies. If only one query attempt was made per item, mark as PARTIAL at best. The study cannot characterize reliability without repeated measurements. 3. Item 3 — "We used a standardized prompt" is insufficient. The full text must be available, even if in supplementary materials. 4. Item 5 — Absent optimization reporting. Many studies iteratively refine prompts but do not report this process. If the prompt appears highly optimized but no development process is described, flag as MISSING. 5. Item 6 — Published exam questions. Studies using publicly available medical exam questions (USMLE, board exams) should explicitly discuss training data contamination risk. This is a known and well-documented concern.

Relationship to Other Guidelines

• CLAIM 2024: Use alongside MI-CLEAR-LLM if the LLM processes medical images (e.g., vision-language models interpreting radiology images).
• TRIPOD+AI: Use alongside MI-CLEAR-LLM if the LLM is used as a clinical prediction model.
• STARD / STARD-AI: Use alongside MI-CLEAR-LLM if the study evaluates diagnostic accuracy of the LLM against a reference standard.

---

Summary

Verdict: [ ] READY FOR SUBMISSION [ ] NEEDS REVISION

Co-application note: MI-CLEAR-LLM is typically used alongside another guideline (STARD, CLAIM, TRIPOD+AI) depending on the study design. The 6 items here supplement — not replace — the primary reporting guideline.

MOOSE Checklist

Meta-analysis Of Observational Studies in Epidemiology Version: MOOSE 2000 Source: https://doi.org/10.1001/jama.283.15.2008 Reference: Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008-2012. doi:10.1001/jama.283.15.2008

Checklist Items (35 items)

Reporting of Background

Reporting of Search Strategy

Reporting of Methods

Reporting of Results

Reporting of Discussion

Reporting of Conclusions

---

Notes for Assessors

• MOOSE is designed specifically for meta-analyses of observational studies (cohort, case-control, cross-sectional). For meta-analyses of RCTs, use PRISMA 2020.
• MOOSE and PRISMA are complementary. Many journals require adherence to both for observational MAs. When both apply, audit against both checklists separately.
• Items 7-16 (Search Strategy) overlap substantially with PRISMA but include observational-specific items such as handling of confounding and study quality assessment.
• Item 20 (Assessment of confounding) is particularly important for observational MAs, as confounding is the primary threat to validity -- unlike RCTs where randomization addresses this.
• Item 21 (Assessment of study quality) should describe a formal tool. Common tools include NOS (Newcastle-Ottawa Scale), ROBINS-I, or the Joanna Briggs Institute checklists.
• Item 29 (Quantitative assessment of bias) is critical. Publication bias assessment is expected in all meta-analyses, but is especially important for observational studies where positive-result bias is well documented.
• MOOSE was published before the PRISMA era (2009/2020). Some journals now accept PRISMA with MOOSE-specific additions. If in doubt, check journal instructions.

Newcastle-Ottawa Scale (NOS) Assessment Guide

Quality assessment tool for non-randomised studies in meta-analyses. Reference: Wells GA et al. Ottawa Hospital Research Institute.

Structure

NOS uses a "star system" (maximum 9 stars) across 3 categories. Higher stars = higher quality.

Cohort Studies (max 9 stars)

Selection (max 4 stars)

1. Representativeness of the exposed cohort (1 star)

• a) Truly representative of the average [describe] in the community *
• b) Somewhat representative *
• c) Selected group of users
• d) No description

2. Selection of the non-exposed cohort (1 star)

• a) Drawn from the same community as the exposed *
• b) Drawn from a different source
• c) No description

3. Ascertainment of exposure (1 star)

• a) Secure record (e.g., surgical records) *
• b) Structured interview *
• c) Written self-report
• d) No description

4. Demonstration that outcome was not present at start (1 star)

• a) Yes *
• b) No

Comparability (max 2 stars)

5. Comparability of cohorts on the basis of design or analysis (up to 2 stars)

• a) Study controls for [most important factor] *
• b) Study controls for any additional factor *

Outcome (max 3 stars)

6. Assessment of outcome (1 star)

• a) Independent blind assessment *
• b) Record linkage *
• c) Self-report
• d) No description

7. Was follow-up long enough for outcomes to occur? (1 star)

• a) Yes (select adequate follow-up period) *
• b) No

8. Adequacy of follow-up of cohorts (1 star)

• a) Complete follow-up (all subjects accounted for) *
• b) Subjects lost to follow-up unlikely to introduce bias (<=20% lost, or description provided) *
• c) Follow-up rate <80% and no description of those lost
• d) No statement

Case-Control Studies (max 9 stars)

Selection (max 4 stars)

1. Is the case definition adequate? 2. Representativeness of the cases 3. Selection of controls 4. Definition of controls

Comparability (max 2 stars)

5. Comparability of cases and controls (same 2-star system)

Exposure (max 3 stars)

6. Ascertainment of exposure 7. Same method of ascertainment for cases and controls 8. Non-response rate

Interpretation

Alternative thresholds may be pre-specified in the protocol.

When to Use

• Observational cohort studies in intervention or exposure meta-analyses
• Case-control studies
• Simpler alternative to ROBINS-I when full domain-level assessment is not needed
• Note: NOS does not provide domain-level judgments -- only an aggregate score

PRISMA 2020 Checklist

Preferred Reporting Items for Systematic Reviews and Meta-Analyses Version: PRISMA 2020 Source: https://www.prisma-statement.org

Checklist Items (27 items)

Title

Abstract

Introduction

Methods

Results

Discussion

Other Information

---

PRISMA 2020 Flow Diagram

The PRISMA flow diagram (item 16a) should include four phases:

IDENTIFICATION
  Records identified from databases (n = ?)
  Records identified from other sources (n = ?)
      |
      v
  Records removed before screening:
    Duplicate records (n = ?)
    Records marked as ineligible by automation tools (n = ?)
    Records removed for other reasons (n = ?)
      |
      v
SCREENING
  Records screened (n = ?)
    Records excluded (n = ?)
      |
      v
  Reports sought for retrieval (n = ?)
    Reports not retrieved (n = ?)
      |
      v
  Reports assessed for eligibility (n = ?)
    Reports excluded (n = ?), with reasons:
      Reason 1 (n = ?)
      Reason 2 (n = ?)
      Reason 3 (n = ?)
      |
      v
INCLUDED
  Studies included in review (n = ?)
  Reports of included studies (n = ?)

---

Notes for Assessors

• PRISMA 2020 expanded from the original 27-item checklist; several items now have sub-items (e.g., 10a/10b, 13a-13f, 16a/16b).
• Item 7 (full search strategy): the complete strategy for at least one database must be provided, either in the manuscript or supplementary materials.
• Item 15 (certainty assessment): GRADE is the most common framework; mark as MISSING if no certainty assessment is reported.
• Item 24 (registration): PROSPERO is the standard registry for systematic reviews. If not registered, the authors should explicitly state this.
• Item 27 (data availability): increasingly required; check if authors state where extracted data and analytic code can be accessed.
• For systematic reviews of diagnostic test accuracy, also consider PRISMA-DTA extension.
• For reviews that include meta-analysis, items 13d (synthesis model) and 20b (heterogeneity measures) are critical.