Academic Paper Verify

Academic Paper Verification

A systematic skill for verifying the integrity and replicability of an academic research paper. This covers everything from individual coefficient checks to full end-to-end replication.

Overview

Verification proceeds in six phases. Each phase produces structured output. Do not skip phases - earlier phases feed into later ones.

Phase 1: Discovery       -> inventory of all project files, scripts, outputs, paper
Phase 2: Table Audit     -> cross-check every number in every table
Phase 3: Inline Claims   -> verify quantitative claims in paper body text
Phase 4: Code Review     -> audit R scripts for correctness, modeling decisions, data pipeline
Phase 5: Manifest Build  -> create verification_manifest.json linking claims to code
Phase 6: Replication     -> write and run tests/verify_replication.R, fix failures

Before You Start

1. Identify the project root directory. Look for .Rproj files, README, or ask the user. 2. Read references/phase-details.md for the full procedure for each phase. 3. Read references/common-pitfalls.md for known failure modes to watch for.

Phase 1: Discovery

Scan the entire project and build an inventory. You need to know what you're working with before you can verify anything.

Find and catalog:

• All .R and .Rmd scripts (note execution order if a master script exists)
• All output files: .csv, .rds, .tex, .txt, .log in results/, output/, tables/, etc.
• The LaTeX paper file(s): .tex in the root or paper/ or draft/ directory
• Any data files: .csv, .dta, .rds, .xlsx in data/ or similar
• Any configuration or parameter files

Produce: A file inventory printed to the console, organized by type, with notes on what each script appears to do (based on filename and a quick scan of its first ~30 lines).

Key questions to answer in this phase:

• Is there a master script that runs everything in order?
• Where do intermediate outputs land?
• Which scripts produce which tables/figures?
• Are there any scripts that appear unused or orphaned?

Phase 2: Table Audit

This is the most critical phase. Read references/phase-details.md Section 2 for the full procedure.

For every table in the paper:

1. Locate the table in the LaTeX source. Extract every number: coefficients, standard errors, t-statistics, p-values, confidence intervals, sample sizes (N), R-squared, F-statistics, means, medians, percentages - everything.

2. Locate the corresponding R output file that produced this table. This might be a .tex file generated by stargazer, modelsummary, xtable, kableExtra, huxtable, or similar. It could also be a .csv, .rds, or text log.

3. Cross-check every single number. Compare to the R output with appropriate tolerance:

• Coefficients and standard errors: match to the number of decimal places shown
• Sample sizes: must match exactly
• R-squared and similar: match to displayed precision
• Percentages: verify the arithmetic (numerator/denominator)

4. Check for rounding consistency - if a coefficient is 0.0347 in the R output and 0.035 in the paper, that is acceptable rounding. If it is 0.038, that is a discrepancy.

5. Verify that column headers, variable names, and panel labels in the paper match the specification in the code.

6. Check that the number of observations (N) is consistent across all tables that use the same sample. If Table 1 reports N=4,521 and Table 3 uses the same sample but reports N=4,519, that needs explanation.

Produce: A table-by-table verification report. For each table:

• Table number and title
• Source R script and output file
• Number of values checked
• List of any discrepancies with exact locations (paper line number, output file line number)
• PASS/FAIL status

Phase 3: Inline Claims Audit

Read the paper body text (not just tables) and find every quantitative claim. These include:

• "We find a 3.2 percentage point increase..."
• "The effect is significant at the 5% level..."
• "Our sample includes 12,450 observations..."
• "Column 3 of Table 2 shows that..."
• "The coefficient on X is negative and significant..."
• Footnotes with numbers or statistical claims
• Abstract claims about magnitudes and significance

For each claim, trace it back to a specific table cell, figure, or R output. Flag any claim that cannot be traced or that contradicts the evidence.

Produce: A claims checklist with claim text, source location in paper, evidence source, and VERIFIED/UNVERIFIED/DISCREPANCY status.

Phase 4: Code Review

Read every R script in the project, in execution order. This is not just a syntax check - you are auditing the analytical pipeline. Read references/phase-details.md Section 4 and references/common-pitfalls.md for what to look for.

Data Pipeline Verification:

• At every merge, join, filter, subset, or mutate step, check:

(a) How many observations before vs. after the transformation? (b) Do all column names needed downstream still exist? (c) Are key summary statistics (mean, min, max, N) reasonable after the step?

• Flag any joins that could silently drop or duplicate observations
• Flag any filters that might be too aggressive or too permissive
• Check for proper handling of missing values (NA) - are they dropped, imputed, or ignored?
• Verify that panel/time-series data is properly balanced or that imbalance is handled

Modeling Decisions:

• Are the regression specifications consistent with what the paper describes?

(e.g., if the paper says "we control for year fixed effects", is that in the code?)

• Are standard errors clustered as described? (robust, clustered at the right level, etc.)
• Are instrumental variables correctly specified? (first stage, exclusion restriction checks)
• Is the sample restriction for each regression clearly defined and consistent with the paper?
• Are interaction terms, polynomials, or transformations correctly implemented?
• Do subsample analyses actually use the right subsamples?

Robustness and Red Flags:

• Are there hardcoded values that should be computed? (e.g., filter(year > 2005) when the

paper says "post-treatment period" without defining the cutoff)

• Are there commented-out lines that suggest alternative specifications were tried?
• Is there any evidence of p-hacking patterns (many specifications tried, only one reported)?
• Are random seeds set for any stochastic procedures?
• Are there warnings or errors being suppressed?

Produce: A script-by-script review with:

• Script name and purpose
• Data pipeline issues (with line numbers)
• Modeling decision flags (with line numbers)
• Red flags (with line numbers)
• Overall assessment: CLEAN / MINOR ISSUES / MAJOR ISSUES

Phase 5: Build Verification Manifest

Create verification_manifest.json that maps every quantitative claim in the paper to the code that produces it.

Structure:

{
  "paper_file": "paper/main.tex",
  "generated_at": "2026-02-08T12:00:00Z",
  "claims": [
    {
      "id": "T1_R2_C3",
      "type": "coefficient",
      "paper_location": {"file": "paper/main.tex", "line": 234, "context": "Table 1, Row 2, Col 3"},
      "paper_value": "0.035",
      "source_script": "code/02_main_regression.R",
      "source_line": 87,
      "output_file": "results/table1.tex",
      "output_location": {"line": 15, "context": "second coefficient in column 3"},
      "expected_value": "0.0347",
      "tolerance": 0.001,
      "status": "PASS",
      "notes": "Acceptable rounding from 0.0347 to 0.035"
    },
    {
      "id": "BODY_P12_S3",
      "type": "inline_claim",
      "paper_location": {"file": "paper/main.tex", "line": 412, "context": "paragraph 12, sentence 3"},
      "paper_value": "3.2 percentage points",
      "source_script": "code/02_main_regression.R",
      "source_line": 87,
      "output_file": "results/table1.tex",
      "output_location": {"line": 15},
      "expected_value": "0.032",
      "tolerance": 0.001,
      "status": "PASS",
      "notes": "Coefficient 0.0323 reported as 3.2pp"
    }
  ],
  "summary": {
    "total_claims": 142,
    "passed": 139,
    "failed": 2,
    "unverified": 1
  }
}

Every coefficient, standard error, sample size, p-value, summary statistic, and verbal claim should appear in this manifest. Be exhaustive.

Phase 6: Replication Test Suite

Write tests/verify_replication.R that programmatically reruns the analysis and checks results against the manifest.

Read references/replication-script-template.md for the template and structure.

The test script must:

1. Source or rerun each analysis script in the correct order 2. Extract the relevant outputs (coefficients, SEs, N, R-squared, etc.) 3. Compare against the values in verification_manifest.json 4. Use appropriate tolerance for floating-point comparisons 5. Report PASS/FAIL for each claim with clear diagnostics on failure 6. Handle dependencies gracefully (if a data file is missing, report it, do not crash)

After writing the test script: 1. Run it 2. For any failures, diagnose the root cause 3. If the failure is due to a code bug (not a paper-code mismatch), fix the upstream script and document what you fixed 4. Rerun until all tests pass or all remaining failures are genuine paper-code discrepancies 5. Produce a final summary

Produce:

• tests/verify_replication.R - the test script
• tests/replication_results.json - structured test results
• tests/replication_summary.md - human-readable summary of what passed, what failed,

what was fixed, and what remains unresolved

Output Format

At the end of the full verification, produce a consolidated report. Use this structure:

# Paper Verification Report

## Executive Summary
- Total quantitative claims checked: X
- Passed: Y
- Failed: Z
- Unverified: W
- Code issues found: N (M major, K minor)

## Table-by-Table Results
[from Phase 2]

## Inline Claims Results
[from Phase 3]

## Code Review Findings
[from Phase 4]

## Replication Test Results
[from Phase 6]

## Recommendations
[prioritized list of issues to address]

Important Notes

• Never silently skip a number. If you cannot verify a value, mark it UNVERIFIED with

an explanation.

• When in doubt, flag it. False positives are better than missed discrepancies.
• Pay special attention to N (sample sizes) - these are the most common source of

inconsistencies across tables and text.

• If the project uses R packages that produce formatted output (stargazer, modelsummary,

etc.), check the raw model objects too, not just the formatted output.

• If you encounter Stata .do files or Python scripts mixed in, verify those too using

the same principles.

• The user may want you to run this on a subset (e.g., "just check Table 3"). Adapt

accordingly but note what was not checked.

Common Pitfalls in Academic Paper Verification

Known failure modes organized by category. Check for all of these during verification.

Table of Contents

• Numerical Discrepancies
• Data Pipeline Issues
• Econometric / Modeling Issues
• LaTeX and Formatting Issues
• Replicability Issues

---

Numerical Discrepancies

1. Rounding chain errors

The R output shows 0.03468. The author rounds to 0.035. Then in the text they write "approximately 3.5 percentage points." Each rounding step is individually defensible, but the chain can introduce drift. Always compare to the raw value, not to intermediate rounded versions.

2. Stale output files

The code was updated after the last time it was run. The output files in results/ are from an earlier version of the code. The paper includes these stale numbers. Check file modification dates. If the code file is newer than its output file, the output may be stale.

3. Copy-paste across tables

Authors sometimes copy a row or panel from one table to another (e.g., "baseline specification" appears in Tables 2, 3, and 4). If one table was regenerated but the others were not, the "same" row might show different numbers. Cross-check identical rows across tables.

4. Manual overrides in LaTeX

Sometimes the auto-generated .tex table is included via \input{}, but the author then manually edits numbers in the .tex file (e.g., to add significance stars, fix formatting, or "correct" a rounding). Compare the .tex file in the paper directory with the one in the results directory byte-by-byte if possible.

5. Standard error type mismatch

The paper says "clustered standard errors" but the table was generated with heteroskedasticity-robust (HC) standard errors. Coefficients will match but standard errors, t-stats, and p-values will all differ.

---

Data Pipeline Issues

6. Unintended row duplication from joins

A left_join where the right-hand table has duplicate keys silently duplicates rows in the output. The author may not notice because they never check N after the join. This inflates the sample size and biases standard errors downward.

7. Factor level ordering

In R, the reference category for a factor depends on factor level ordering. If the data is read in differently (e.g., different locale, different R version), factor levels may change, shifting which category is the baseline in regressions.

8. Silent NA propagation

If a variable used in a regression has NAs, lm() drops those observations by default. If the paper reports N = 5,000 but the regression actually uses N = 4,800 because 200 observations had NA in one control variable, the paper's N is wrong. Check nobs(model) against the claimed sample size.

9. Encoding issues in data

CSV files with special characters (accents, cyrillic, etc.) may read differently depending on locale settings. This can silently corrupt string variables used for merging or fixed effects.

10. Date parsing ambiguity

Is "01/02/2020" January 2nd or February 1st? Different read_csv locale settings parse this differently. If dates are used for sample restrictions or event studies, this can change results.

---

Econometric / Modeling Issues

11. Fixed effects specification mismatch

The paper says "we include municipality and year fixed effects" but the code uses factor(region) instead of factor(municipality). This is a different specification than described. Or the code uses feols(y ~ x | municipality + year) but the paper says "province fixed effects."

12. Clustering level mismatch

The paper says standard errors are "clustered at the district level" but the code clusters at the municipality level. Or the paper is ambiguous ("clustered at the regional level") and the code reveals which specific regional unit is used.

13. Weight variable issues

If the regression uses weights (lm(..., weights = w)), verify:

• Are the weights the same ones described in the paper?
• Are there zero or negative weights?
• Do the weights change the effective N?

14. Subsample definition drift

The paper says "urban areas" but the code uses urban_flag == 1, and the definition of urban_flag was set earlier in the pipeline using a threshold that may not match the conventional definition. Trace the definition of subsample indicators back to their construction.

15. Instrument validity not checked

For IV/2SLS regressions, the first-stage F-statistic should be reported. If it is not computed in the code, the paper may either fabricate this number or report a different statistic. Check that diagnostic tests match.

16. Winsorization / trimming not documented

Some authors winsorize or trim outliers but do not mention this in the paper, or mention it only in a footnote. Check the code for any quantile() based filtering or variable recoding.

---

LaTeX and Formatting Issues

17. Significance stars inconsistent

The paper footnote says p<0.1, p<0.05, ** p<0.01, but stargazer defaults are

• p<0.1, p<0.05, p<0.01. However, some journals use p<0.05, ** p<0.01,

*** p<0.001. Check that the stars in the table match the footnote definition, and that both match the actual p-values.

18. Parentheses ambiguity

Are the values in parentheses standard errors, t-statistics, confidence intervals, or p-values? The table note should specify, and the code should confirm.

19. Table note vs. actual specification

Table notes like "All regressions include year and region fixed effects" should be verified against the actual code. Sometimes the note is copied from a template and does not match the actual specification in a particular table.

20. Absolute vs. relative values

The paper says "a 5% increase" - is that 5 percentage points (0.05) or 5 percent of the baseline? Check the coefficient, the baseline mean, and the exact wording.

---

Replicability Issues

21. Missing package versions

The code uses packages that have been updated since the analysis was run. Function behavior may have changed. Check for renv.lock or sessionInfo() output. If not available, note that exact replication may not be possible.

22. Random seed not set

Bootstrap standard errors, permutation tests, or simulation-based inference will give different results each time without set.seed(). If the paper reports bootstrapped CIs, check that a seed is set.

23. Platform-dependent results

Some numerical results differ slightly between Windows, Mac, and Linux due to floating-point arithmetic differences. This usually affects only very small digits but can occasionally flip significance at borderline p-values.

24. Proprietary or restricted data

If the raw data is not included in the repository (common with confidential survey data or administrative records), full replication is impossible. Note which steps can be verified and which cannot.

25. Undocumented manual steps

Sometimes the pipeline includes manual steps ("open the Excel file and delete the first two rows", "rename column X to Y"). These break automated replication. Flag any gap in the automated pipeline.

Phase Details Reference

Detailed procedures for each verification phase. The SKILL.md gives the overview; this file has the step-by-step instructions for tricky parts.

Table of Contents

• Section 2: Table Audit Procedure
• Section 4: Code Review Procedure
• Section 5: Manifest Construction
• Section 6: Replication Script Construction

---

Section 2: Table Audit Procedure

Step 2.1: Parse LaTeX Tables

LaTeX tables come in many formats. Here is how to handle the common ones:

Standard tabular environments: Look for \begin{table} ... \end{table} blocks. Inside, find the tabular or tabular* environment. Parse rows by splitting on \\ and columns by splitting on &.

Stargazer output: Stargazer produces .tex files with a specific structure. The coefficient rows alternate with standard error rows (in parentheses). The bottom section contains N, R-squared, and other diagnostics. Key pattern:

variable_name & coeff1 & coeff2 & coeff3 \\
              & (se1)  & (se2)  & (se3)  \\

modelsummary output: Similar structure but may use \multicolumn, \midrule, and different formatting. Pay attention to the gof_map argument which controls which goodness-of-fit statistics appear.

kableExtra / huxtable: These produce more varied output. Look at the raw .tex or .html file rather than trying to parse the R code that generates them.

Manual tables (not auto-generated): These are the highest-risk tables. If a table was created by hand (typing numbers into LaTeX directly), it is the most likely to contain transcription errors. Flag these prominently.

Step 2.2: Extract Numbers Systematically

For each table, create a structured extraction:

Table X: [title]
  Row 1, Col 1: value (type: coefficient)
  Row 1, Col 1 SE: value (type: standard_error)
  Row 1, Col 2: value (type: coefficient)
  ...
  Footer: N = value (type: sample_size)
  Footer: R-squared = value (type: r_squared)

Step 2.3: Match to R Output

For auto-generated tables, the matching is straightforward - compare the .tex file included in the paper to the .tex file in the results directory. Check for:

• Exact match (ideal)
• Date/version differences (the paper might include an older version)
• Manual edits (the paper version might have hand-edited labels or formatting)

For manually constructed tables, you need to find the model object or summary output that produced the numbers. Look for:

• summary(model) output in log files
• coef(), confint(), nobs() calls
• .rds files containing saved model objects (load them and extract)

Step 2.4: Tolerance Rules

Step 2.5: Cross-Table Consistency

After checking individual tables, verify consistency across tables:

• If Tables 1 and 3 use the same sample, N must match
• If Table 2 is a subsample of Table 1, N(Table 2) < N(Table 1)
• Summary statistics in Table 1 should be consistent with regression samples
• If the same variable appears in multiple tables, its coefficient should differ only

if the specification changed (different controls, different sample, etc.)

---

Section 4: Code Review Procedure

Step 4.1: Establish Execution Order

Find the master script (often called main.R, run_all.R, master.R, or _run.R). If none exists, infer execution order from:

• File numbering (01_clean.R, 02_merge.R, 03_analysis.R)
• File dependencies (which scripts load outputs from other scripts?)
• README instructions

Step 4.2: Data Pipeline Walk-Through

For each script, in order, trace the data:

Script: 01_clean_data.R
  Input:  data/raw_survey.csv (N = 15,234 rows, 45 cols)
  Step 1: filter(!is.na(income))     -> N = 14,891 (lost 343, 2.3%)
  Step 2: filter(age >= 18)          -> N = 14,502 (lost 389, 2.6%)
  Step 3: mutate(log_income = ...)   -> N = 14,502 (same, new col added)
  Step 4: left_join(geo_data)        -> N = 14,502 (check: no duplication?)
  Output: data/clean_survey.rds      -> N = 14,502 rows, 47 cols

At every transformation, verify: 1. Observation count: How many rows before and after? Is the loss reasonable? 2. Column existence: Do all columns needed by downstream scripts exist? 3. Summary stats: For key variables, are mean/min/max/NA-count sensible?

Step 4.3: Merge Audit

Merges are the #1 source of silent data corruption. For every merge/join:

• What type? (left_join, inner_join, merge, etc.)
• What are the key columns?
• Is the merge 1:1, 1:m, m:1, or m:m?
• Are there duplicate keys in either dataset? (This causes row multiplication)
• How many rows match vs. don't match?
• Are unmatched rows dropped or kept as NA?

If the code uses merge() without specifying all, all.x, or all.y, flag it - the default behavior might not be what the author intended.

Step 4.4: Regression Specification Audit

For each regression model, verify:

Variables:

• Dependent variable matches paper description
• Independent variables match paper description
• Control variables are all present
• Fixed effects are correctly specified (if using feols, check the FE formula;

if using lm with dummies, check that the right dummies are included)

Standard errors:

• Clustering level matches the paper
• If the paper says "robust standard errors", check for vcov = "HC1" or equivalent
• If using feols, check the vcov argument
• If using coeftest or sandwich, verify the specification

Sample:

• The data subset used for this regression matches what the paper describes
• Any sample restrictions (e.g., "excluding outliers", "balanced panel only") are

correctly implemented

Instrumental variables:

• First-stage specification matches the paper
• Instruments are correctly specified
• Check for weak instrument diagnostics (F-stat > 10 or similar)

Step 4.5: Red Flag Patterns

Watch for these specific patterns:

Hardcoded magic numbers:

# BAD: What is 2005? Why this cutoff?
df <- df %>% filter(year > 2005)

# BETTER: Named and documented
treatment_start_year <- 2005  # Policy enacted in 2005
df <- df %>% filter(year > treatment_start_year)

Silent NA handling:

# This silently drops NAs - is that intentional?
model <- lm(y ~ x, data = df)
# How many observations were dropped due to NAs?

Overwritten objects:

df <- read_csv("data.csv")
df <- df %>% filter(...)
df <- df %>% mutate(...)
# If something goes wrong, you can't recover the original df

Suppressed warnings:

suppressWarnings(...)  # What is being suppressed and why?
options(warn = -1)     # Globally turning off warnings is a red flag

Commented-out alternatives:

# model <- lm(y ~ x + z, data = df)           # Was this tried first?
# model <- lm(y ~ x + z + w, data = df)       # And this?
model <- lm(y ~ x + z + w + v, data = df)     # And this is the "winner"?

---

Section 5: Manifest Construction

Systematic Claim Extraction

Go through the paper linearly (abstract through appendix) and extract claims in order. Use a consistent ID scheme:

• ABS_1, ABS_2: Abstract claims
• T1_R2_C3: Table 1, Row 2, Column 3
• T1_N: Table 1 sample size
• T1_R2: Table 1 R-squared
• F3_POINT_2: Figure 3, data point 2
• BODY_S4_P2_S3: Section 4, Paragraph 2, Sentence 3
• FN_12: Footnote 12
• APP_T_A1_R1_C2: Appendix Table A1, Row 1, Column 2

Linking Claims to Code

For each claim, trace the full chain:

Paper claim -> Table/Figure -> Output file -> R script -> Data source

Every link in the chain must be verified. If any link is broken (e.g., you cannot find which script produces a particular output file), mark the claim as UNVERIFIED and document what is missing.

---

Section 6: Replication Script Construction

Script Architecture

The replication script should be structured as:

# tests/verify_replication.R
# Automated replication verification
# Generated by academic-paper-verify skill

library(jsonlite)
library(testthat)

# --- Configuration ---
manifest <- fromJSON("verification_manifest.json")
tolerance_default <- 1e-3
results <- list()

# --- Helper functions ---
check_value <- function(claim_id, actual, expected, tol = tolerance_default) { ... }
extract_coef <- function(model, var_name) { ... }
extract_se <- function(model, var_name) { ... }
extract_nobs <- function(model) { ... }

# --- Run scripts in order ---
# Each block: source the script, extract values, compare to manifest

# Block 1: Table 1
source("code/02_main_regression.R")
# ... extract and check ...

# Block 2: Table 2
# ... etc ...

# --- Report ---
report <- generate_report(results)
writeLines(report, "tests/replication_summary.md")
write_json(results, "tests/replication_results.json")

Handling Script Dependencies

Many academic R projects are not designed to be re-run cleanly. Common issues:

• Scripts that assume objects are already in the global environment
• Scripts that use setwd() or relative paths that break
• Scripts that read data from absolute paths
• Missing packages

The replication script should handle these gracefully:

• Wrap each source() call in tryCatch
• Check for required objects before proceeding
• Report missing dependencies clearly
• Do not let one failure block the rest of the checks

Tolerance Strategy

Not all values need the same tolerance:

• Values that should be deterministic (same data, same code): tight tolerance (1e-10)
• Values that involve floating-point differences across platforms: moderate (1e-6)
• Values displayed in tables with limited decimal places: match to display precision
• Values involving bootstrapping or simulation: may not match at all without same seed

Replication Script Template

Use this template when building tests/verify_replication.R in Phase 6. Adapt it to the specific project structure.

---

Full Template

###############################################################################
# verify_replication.R
# Automated verification of paper claims against code output
# Generated by academic-paper-verify skill
###############################################################################

# --- Setup -------------------------------------------------------------------
library(jsonlite)

cat("=== Paper Replication Verification ===\n")
cat("Started:", format(Sys.time(), "%Y-%m-%d %H:%M:%S"), "\n\n")

# Load manifest
manifest_path <- "verification_manifest.json"
if (!file.exists(manifest_path)) {
  stop("Manifest not found at: ", manifest_path,
       "\nRun the manifest builder first.")
}
manifest <- fromJSON(manifest_path)
cat("Loaded manifest with", length(manifest$claims$id), "claims to verify.\n\n")

# --- Results collector -------------------------------------------------------
results <- data.frame(
  claim_id = character(),
  type = character(),
  expected = character(),
  actual = character(),
  tolerance = numeric(),
  status = character(),     # PASS, FAIL, ERROR, SKIP
  message = character(),
  stringsAsFactors = FALSE
)

add_result <- function(claim_id, type, expected, actual,
                       tolerance = NA, status, message = "") {
  results <<- rbind(results, data.frame(
    claim_id = claim_id,
    type = type,
    expected = as.character(expected),
    actual = as.character(actual),
    tolerance = tolerance,
    status = status,
    message = message,
    stringsAsFactors = FALSE
  ))
}

# --- Helper functions --------------------------------------------------------
check_numeric <- function(claim_id, actual, expected, tol = 1e-3, type = "value") {
  if (is.null(actual) || is.na(actual)) {
    add_result(claim_id, type, expected, "NA", tol, "ERROR",
               "Could not extract actual value")
    return(FALSE)
  }
  if (abs(as.numeric(actual) - as.numeric(expected)) <= tol) {
    add_result(claim_id, type, expected, actual, tol, "PASS", "")
    return(TRUE)
  } else {
    add_result(claim_id, type, expected, actual, tol, "FAIL",
               paste0("Difference: ", abs(as.numeric(actual) - as.numeric(expected))))
    return(FALSE)
  }
}

check_exact <- function(claim_id, actual, expected, type = "count") {
  if (is.null(actual) || is.na(actual)) {
    add_result(claim_id, type, expected, "NA", 0, "ERROR",
               "Could not extract actual value")
    return(FALSE)
  }
  if (as.character(actual) == as.character(expected)) {
    add_result(claim_id, type, expected, actual, 0, "PASS", "")
    return(TRUE)
  } else {
    add_result(claim_id, type, expected, actual, 0, "FAIL",
               paste0("Expected exactly ", expected, ", got ", actual))
    return(FALSE)
  }
}

safe_source <- function(script_path) {
  cat("  Sourcing:", script_path, "...")
  if (!file.exists(script_path)) {
    cat(" NOT FOUND\n")
    return(FALSE)
  }
  tryCatch({
    source(script_path, local = FALSE)
    cat(" OK\n")
    return(TRUE)
  }, error = function(e) {
    cat(" ERROR:", conditionMessage(e), "\n")
    return(FALSE)
  })
}

safe_extract_coef <- function(model, varname) {
  tryCatch({
    coefs <- coef(model)
    if (varname %in% names(coefs)) return(coefs[[varname]])
    # Try partial matching for interaction terms, transformations
    matches <- grep(varname, names(coefs), fixed = TRUE)
    if (length(matches) == 1) return(coefs[matches])
    return(NA)
  }, error = function(e) return(NA))
}

safe_extract_se <- function(model, varname, vcov_type = NULL) {
  tryCatch({
    if (!is.null(vcov_type)) {
      # Use sandwich / clubSandwich for robust/clustered SEs
      if (requireNamespace("sandwich", quietly = TRUE)) {
        vcov_mat <- sandwich::vcovHC(model, type = vcov_type)
        se <- sqrt(diag(vcov_mat))
      } else {
        se <- sqrt(diag(vcov(model)))
      }
    } else {
      se <- sqrt(diag(vcov(model)))
    }
    if (varname %in% names(se)) return(se[[varname]])
    matches <- grep(varname, names(se), fixed = TRUE)
    if (length(matches) == 1) return(se[matches])
    return(NA)
  }, error = function(e) return(NA))
}

# --- Verification blocks -----------------------------------------------------
# Each block corresponds to one script or one table.
# Adapt this section entirely to the specific project.

cat("\n--- Verifying Table 1 ---\n")
# Example block:
# if (safe_source("code/02_main_regression.R")) {
#   # The script should have created a model object, e.g. model_main
#   if (exists("model_main")) {
#     check_numeric("T1_R1_C1", safe_extract_coef(model_main, "treatment"),
#                   manifest$claims$expected_value[manifest$claims$id == "T1_R1_C1"])
#     check_exact("T1_N", nobs(model_main),
#                 manifest$claims$expected_value[manifest$claims$id == "T1_N"])
#   } else {
#     add_result("T1_ALL", "table", NA, NA, NA, "ERROR",
#                "model_main not found after sourcing script")
#   }
# } else {
#   add_result("T1_ALL", "table", NA, NA, NA, "SKIP",
#              "Script could not be sourced")
# }

# --- Generate reports --------------------------------------------------------
cat("\n\n=== Verification Complete ===\n")
cat("Total claims:", nrow(results), "\n")
cat("PASS: ", sum(results$status == "PASS"), "\n")
cat("FAIL: ", sum(results$status == "FAIL"), "\n")
cat("ERROR:", sum(results$status == "ERROR"), "\n")
cat("SKIP: ", sum(results$status == "SKIP"), "\n")

# Save structured results
write_json(results, "tests/replication_results.json", pretty = TRUE)

# Generate human-readable summary
summary_lines <- c(
  "# Replication Verification Summary",
  "",
  paste("Generated:", format(Sys.time(), "%Y-%m-%d %H:%M:%S")),
  "",
  "## Results",
  "",
  paste("- Total claims checked:", nrow(results)),
  paste("- PASS:", sum(results$status == "PASS")),
  paste("- FAIL:", sum(results$status == "FAIL")),
  paste("- ERROR:", sum(results$status == "ERROR")),
  paste("- SKIP:", sum(results$status == "SKIP")),
  ""
)

if (sum(results$status == "FAIL") > 0) {
  summary_lines <- c(summary_lines,
    "## Failures",
    ""
  )
  fails <- results[results$status == "FAIL", ]
  for (i in seq_len(nrow(fails))) {
    summary_lines <- c(summary_lines,
      paste0("### ", fails$claim_id[i]),
      paste0("- Type: ", fails$type[i]),
      paste0("- Expected: ", fails$expected[i]),
      paste0("- Actual: ", fails$actual[i]),
      paste0("- Message: ", fails$message[i]),
      ""
    )
  }
}

if (sum(results$status == "ERROR") > 0) {
  summary_lines <- c(summary_lines,
    "## Errors",
    ""
  )
  errs <- results[results$status == "ERROR", ]
  for (i in seq_len(nrow(errs))) {
    summary_lines <- c(summary_lines,
      paste0("### ", errs$claim_id[i]),
      paste0("- Message: ", errs$message[i]),
      ""
    )
  }
}

writeLines(summary_lines, "tests/replication_summary.md")
cat("\nResults saved to tests/replication_results.json")
cat("\nSummary saved to tests/replication_summary.md\n")

Adaptation Notes

This template is a skeleton. When building the actual script for a specific project:

1. Replace the example verification block with real blocks for each table and script in the project.

2. Handle the script execution model - some projects are designed so each script is self-contained (reads data, runs analysis, saves output). Others assume a shared global environment built up by running scripts in sequence. Adapt accordingly:

• Self-contained scripts: safe_source() each one independently
• Sequential scripts: source them in order into the global environment

3. Match the SE extraction to the project - if the project uses fixest::feols, use fixest::se() and fixest::coef() instead of base R equivalents. If it uses lfe::felm, use the appropriate extraction methods.

4. Handle package-specific model objects - different packages store results differently:

• lm / glm: coef(), vcov(), nobs()
• fixest::feols: coef(), se(), fitstat() for diagnostics
• plm: coef(), vcov() with appropriate method
• ivreg: coef(), vcov(), check first-stage via summary()
• lfe::felm: $coefficients, $se, $N

5. Add tolerance overrides per claim if needed. Some claims (like bootstrapped CIs) need wider tolerance. Others (like sample sizes) need exact matching.

6. Consider memory - if the project processes large datasets, sourcing all scripts sequentially may run out of memory. Add gc() calls between blocks and consider running in a fresh R session for each major block.