Academic Pdf To Gfm

Academic PDF → GitHub GFM Conversion

A battle-tested workflow for converting academic/research PDF papers into GitHub-renderable GFM markdown with inline figures, mathematically correct LaTeX, and validated output.

Battle-tested on: López de Prado (2026) "How to Use the Sharpe Ratio" — 51 pages, 82 equations, 8 figures.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

Quick Start (3 Steps)

# Step 1: Extract prose (best structure preservation)
uv run --python 3.14 --with pymupdf4llm python3 -c "
import pymupdf4llm
md = pymupdf4llm.to_markdown('paper.pdf')
open('paper-raw.md', 'w').write(md)
"

# Step 2: Extract images
uv run --python 3.14 --with pymupdf python3 references/extract-images.py paper.pdf

# Step 3: Validate math before pushing
node references/validate-math.mjs paper.md

---

CRITICAL: Detect PDF Type First

This determines the entire workflow. Getting it wrong wastes hours.

Type A — Word-Generated PDF (Most Modern Academic Papers)

Signs: Embedded fonts, copyable text, Unicode math chars when you copy-paste (∑, π, α, β, γ, →)

Math encoding: Math is Unicode text in PDF stream — NOT images, NOT glyph maps

Consequence: OCR tools like marker-pdf cannot extract LaTeX — they see text like "γ₄" not \gamma_4. They may return empty output or crash silently.

Required approach:

1. Use pymupdf4llm for prose extraction 2. Manually transcribe all equations from PDF screenshots — there is no shortcut 3. Read each formula visually, write LaTeX by hand

How to confirm: Run marker-pdf — if output is empty or has zero math content, it's Type A.

Type B — LaTeX-Generated PDF

Signs: Computer Modern fonts, precise mathematical spacing, arxiv.org source available

Math encoding: Glyph-mapped — structure is partially extractable

Approach: pymupdf4llm or pdftotext for text. If arxiv source exists, extract directly from .tex (vastly preferred over PDF conversion).

Type C — Scanned/Image PDF

Signs: All pages are raster images, zero copyable text

Approach: OCR pipeline — marker-pdf is best option, or tesseract

---

Tool Comparison

Never trust `marker-pdf` output on Type A/B PDFs — the apparent "success" with empty math sections is the failure mode.

---

Image Extraction

Save references/extract-images.py:

import fitz, os, sys

doc = fitz.open(sys.argv[1])
os.makedirs("references/media", exist_ok=True)
saved = []
for page_num in range(len(doc)):
    for img_idx, img in enumerate(doc[page_num].get_images(full=True)):
        xref = img[0]
        base_image = doc.extract_image(xref)
        img_bytes = base_image["image"]
        if len(img_bytes) < 2048:   # skip icons/logos/watermarks/rules
            continue
        ext = base_image["ext"]
        fname = f"fig-p{page_num+1:02d}-{img_idx+1:02d}.{ext}"
        with open(f"references/media/{fname}", "wb") as f:
            f.write(img_bytes)
        saved.append((page_num+1, fname, base_image.get("width"), base_image.get("height")))
        print(f"Saved: {fname} ({len(img_bytes)//1024}KB, {base_image.get('width')}×{base_image.get('height')})")
doc.close()
print(f"\n{len(saved)} images saved to references/media/")

Naming: fig-p{page:02d}-{idx:02d}.{ext} — page number in name for easy location matching.

Size filter: Skip < 2 KB (captures icons, watermarks, horizontal rules). Review everything ≥ 2 KB — some are decorative but most are figures.

Insert in markdown:

![Figure 1: Variance of Sharpe ratio estimates](./media/fig-p12-01.png)

Place immediately after the nearest section heading or the paragraph that references the figure.

---

GitHub GFM Math Rendering Rules

The $$ vs `math ` Decision — Root Cause

GitHub's Markdown pre-processor runs BEFORE the math renderer. It treats \\ as an escaped backslash and collapses it to \. This breaks LaTeX line breaks in display math.

The rule is simple:

````markdown

BROKEN on GitHub — \\ stripped by pre-processor:

$$ \begin{aligned} a &= b + c \\ d &= e + f \end{aligned} $$

CORRECT on GitHub:

\begin{aligned}
a &= b + c \\
d &= e + f
\end{aligned}

````

````

Display Block Formatting Rules

• $$ must be on its own line — not $$formula$$ on one line
• Blank line required before AND after every $$ block
• Blank line required between consecutive $$ blocks
• These rules do NOT apply to ` `math ` ` blocks

Supported/Unsupported LaTeX

See references/github-math-support-table.md for the full table.

Key things to avoid:

Common Gotchas

• \\[8pt] vertical spacing inside $$ → eaten by pre-processor → move to ` `math ` `
• \frac{1}{T}:\left( → spurious colon after fraction → remove colon
• Pearson vs excess kurtosis: most finance formulas need Pearson (γ₄ = 3 for Gaussian), not excess. Always document the kurtosis convention in the formula comment.
• \begin{pmatrix} with \\ → must use ` `math ` `
• \begin{cases} with multiple rows → must use ` `math ` `

---

GitLab: No Workarounds Needed

Empirically verified 2026-03-15 on GitLab CE 18.9.2. Confirmed by Comrak source code analysis.

GitLab uses the Comrak Rust parser with math_dollars: true. When Comrak encounters $$, it calls handle_dollars which slices the raw input buffer directly and stores it as a NodeMath AST node — CommonMark's backslash handler is never invoked on math content. The raw LaTeX is passed to KaTeX via <span data-math-style="display/inline"> unchanged.

Every GitHub workaround is unnecessary on GitLab:

On GitLab you can write standard LaTeX without any platform-specific workarounds. If you're targeting GitLab (or hosting your own GitLab CE), skip all the \lbrace/\rbrace substitutions and `math ` conversions — plain $$ with standard LaTeX is correct.

GitLab.com Has a Hard 50-Span Per-Page Limit

GitLab.com (SaaS) enforces a limit of 50 total math spans per page (display + inline combined). After the 50th span, all subsequent equations silently fall back to raw LaTeX text. This limit exists to prevent DoS attacks and cannot be overridden on GitLab.com.

Validated on: Sharpe ratio paper (341 spans) — breaks at span 51 on gitlab.com, renders fully on local CE.

The W6 check in `validate-math.mjs` warns when a file exceeds the limit.

Summary: which platform to use:

• GitHub.com: No math span limit. Use \lbrace/\rbrace workarounds (handled by --fix).
• Self-hosted GitLab CE: No limit (disable math_rendering_limits_enabled). No workarounds needed.
• GitLab.com: Only suitable for documents with ≤ 50 math spans.

Self-hosting GitLab CE for Math-Heavy Documents

GitLab CE is free and runs on a single machine. On a 61 GB workstation with slim config:

• Memory footprint: ~3 GB (puma['worker_processes'] = 2, sidekiq['concurrency'] = 5, monitoring disabled)
• Push mirroring to GitHub: free on CE (syncs within 5 min)
• glab CLI: first-party, comparable to gh

# docker-compose.yml — slim GitLab CE
services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    restart: unless-stopped
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://YOUR_IP:8929'
        puma['worker_processes'] = 2
        sidekiq['concurrency'] = 5
        prometheus_monitoring['enable'] = false
        alertmanager['enable'] = false
        node_exporter['enable'] = false
        redis_exporter['enable'] = false
        postgres_exporter['enable'] = false
        gitlab_exporter['enable'] = false
    ports: ["8929:8929", "8922:22"]
    volumes:
      - /srv/gitlab/config:/etc/gitlab
      - /srv/gitlab/logs:/var/log/gitlab
      - /srv/gitlab/data:/var/opt/gitlab

---

Validation Pipeline

Step 1: Install KaTeX Validator

bun add -g katex   # Bun-first per project policy
# or: npm install -g katex

Step 2: Run Before Every Push

# Validate only (exit 1 on errors)
node references/validate-math.mjs your-file.md

# Validate + auto-fix correctable issues
node references/validate-math.mjs your-file.md --fix

The script is at references/validate-math.mjs. It runs two layers:

Layer 1 — KaTeX syntax: parse errors in $, $$, `math ` blocks Layer 2 — GFM structural (issues KaTeX passes but GitHub breaks):

E0 is the most dangerous: a single failing $$ block exposes its $$ delimiters as literal text, creating an orphaned $ that shifts ALL subsequent inline $...$ pairings. One broken equation takes down the entire document.

`\{`/`\}` trap: In $$ blocks, \left\{ becomes \left{ (invalid KaTeX delimiter → "Missing or unrecognized delimiter") and \{...\} set notation becomes invisible grouping. Fix: use \lbrace/\rbrace (letter-based, CommonMark-immune). This affects every equation using set notation like \{\hat{SR}_k\} or \min_T\left\{...\right\}.

Exits code 1 on errors (CI-friendly). Warnings do not block CI but should be reviewed.

Local Preview Tools

# GitHub-accurate hot-reload preview
bun add -g @hyrious/gfm
gfm your-file.md --serve

# Offline binary (gh extension)
gh extension install thiagokokada/gh-gfm-preview
gh gfm-preview your-file.md

VS Code extensions:

• shd101wyy.markdown-preview-enhanced — closest to GitHub rendering
• bierner.markdown-preview-github-styles — GitHub CSS styling

---

Multi-Agent Adversarial Equation Validation

For papers with 10+ equations, use this multi-agent pattern:

Phase 1 — Parallel Extraction

• Agent A: Extract prose with pymupdf4llm, transcribe math from PDF screenshots
• Agent B: Extract and categorize all images

Phase 2 — Parallel Validation

• Agent C: Validate equations against reference implementation (if code/repo exists)
• Agent D: Numerical spot-checks — compute paper's exhibit values, compare

Phase 3 — Discrepancy Handling

• For each discrepancy: write /tmp/paper-discrepancy/eq-{N}.md
• Spawn resolver agents to search online for authoritative third-party sources
• Authority rule: Paper is tentatively more authoritative than code implementation; a third independent source breaks ties

Phase 4 — Guarded Application

• Apply only HIGH-confidence fixes to the markdown
• For MEDIUM-confidence: spawn an independent audit agent before touching the file
• Document all discrepancies even if not fixed — future readers need to know

---

Anti-Patterns

---

References

---

Related Skills

Post-Execution Reflection

After this skill completes, reflect before closing the task:

0. Locate yourself. — Find this SKILL.md's canonical path before editing. 1. What failed? — Fix the instruction that caused it. 2. What worked better than expected? — Promote to recommended practice. 3. What drifted? — Fix any script, reference, or dependency that no longer matches reality. 4. Log it. — Evolution-log entry with trigger, fix, and evidence.

Do NOT defer. The next invocation inherits whatever you leave behind.

GitHub GFM Math Support Table

Complete reference for LaTeX commands supported/unsupported on GitHub's GFM renderer (powered by MathJax with KaTeX-compatible subset).

Last verified: 2026-03 (López de Prado 2026 paper conversion, 82 equations)

Official Open Bug Reports

Gap: No existing FOSS tool detects the pre-processor stripping issues. They all run AFTER the markdown layer. validate-math.mjs in this skill is the only validator that simulates this layer statically.

---

Environments

---

Display Math Delimiters

---

Common Math Commands

Fractions, Roots, Sums

Operators and Functions

Accents and Decorators

Greek Letters

Arrows and Relations

Delimiters

---

Commands to Avoid

---

The $$ vs `math ` Decision Tree

```` Does the equation contain any of:

• \\ (line breaks)
• \begin{aligned} with multiple rows
• \begin{pmatrix}, \begin{bmatrix} (matrices)
• \begin{cases} with multiple cases
• \\[8pt] or other vertical spacing

? ├── Yes → Use ``math ... ` └── No → Use $$ ... $$ ```

When in doubt, use ` ```math ``` ` — it is always safe. The only reason to prefer $$ is that it renders slightly faster and is more universally supported in non-GitHub renderers (VS Code, Jupyter, etc.).

---

Display Block Formatting Rules

# REQUIRED: blank line before and after each $$ block

$$
E = mc^2
$$

# REQUIRED: blank line between consecutive $$ blocks

$$
a + b = c
$$

$$
d + e = f
$$

# WRONG: consecutive blocks without blank lines

$$
a + b = c
$$

$$
d + e = f   ← GitHub collapses this into the first block
$$

---

KaTeX vs GitHub Rendering Differences

KaTeX validation (node validate-math.mjs) catches parse errors but NOT GitHub pre-processor issues.

After KaTeX passes, also check:

1. All multi-line blocks (\\ present) use `math ` not $$ 2. All $$ blocks have blank lines before and after 3. No \boxed{} usage 4. No \operatorname{} usage — use \text{} instead 5. No \begin{align} — use \begin{aligned}

PDF Type Detection Guide

Identifying the PDF type is the most important first step. Using the wrong extraction approach wastes hours.

Detection Checklist

Run these quick checks before choosing any tool:

Test 1: Copy-Paste Check (30 seconds)

Open the PDF in Preview (macOS) or any PDF viewer. Copy a formula or math expression. Paste into a text editor.

• You see Unicode math symbols (∑, π, α, β, γ, →, ≤, ≥, ∈): → Type A (Word-generated)
• You see garbled glyphs or nothing: → possibly Type B (LaTeX) or Type C (scanned)
• Copy is impossible (nothing selectable): → Type C (scanned/image)

Test 2: Font Inspection (2 minutes)

In macOS Preview: File → Properties → Fonts, or use:

pdffonts paper.pdf

• Embedded fonts like "TimesNewRoman", "Arial", "Calibri": → Type A (Word/Office)
• Computer Modern, Latin Modern, CM-Super: → Type B (LaTeX/TeX)
• No fonts listed: → Type C (scanned images)

Test 3: marker-pdf Smoke Test (1 minute)

uv run --python 3.14 --with marker-pdf marker_single paper.pdf /tmp/marker-out --output_format markdown
cat /tmp/marker-out/*.md | head -50

• Output has LaTeX math ($\frac{...}$, \sum, etc.): → probably Type B or C
• Output has zero math or replaces formulas with Unicode text: → Type A (marker-pdf's Unicode bug)
• Output is empty / crash with torch error: → Type A or Type C with incompatible paper size

Test 4: arxiv Check (10 seconds)

Look for the paper on arxiv.org. If found:

• Download the .tar.gz source: https://arxiv.org/src/{arxiv-id}
• Extract and check for .tex files
• If .tex exists: extract directly from source — skip PDF entirely

---

Type A: Word-Generated PDF (Most Common for Recent Academic Papers)

Examples: SSRN papers, working papers, journal submissions from MS Word

Identification

• Microsoft-style fonts (Times New Roman, Calibri, Arial, Georgia)
• Unicode math symbols visible when copy-pasted
• pdffonts shows no math-specific fonts
• marker-pdf returns empty math sections or Unicode text instead of LaTeX

What Works

Math Extraction Approach

There is no automated solution. You must:

1. Read the PDF page by page 2. Screenshot each page containing equations (macOS: Cmd+Shift+4) 3. Write LaTeX manually by inspecting the visual formula 4. Cross-reference against any code implementations if available

Tools that help:

• Read PDF as screenshots within Claude Code using the Read tool on PNG files
• Use reference implementations (Python code, Julia notebooks) to verify formulas
• Search online for the specific formula + paper title to find existing LaTeX typesettings

Efficiency Tips for Manual Transcription

• Work top-to-bottom: prose first, then formulas in order
• Number equations as you go — reference numbers make validation easier
• For recurring symbols, decide convention early (e.g., $\hat{S}$ vs $\tilde{S}$)
• Use aligned inside $$ for single-line display (no \\), `math ` for multi-line

---

Type B: LaTeX-Generated PDF

Examples: ArXiv preprints, ACM/IEEE proceedings, Springer/Elsevier journals

Identification

• Computer Modern or Latin Modern fonts
• Precise mathematical spacing (kerning around operators)
• Text is selectable but copy-pasted math may show glyph codes not Unicode
• arxiv.org source download yields .tex files

What Works

Math Extraction: Prefer Source

If arxiv source is available, skip PDF:

# Download source
curl -L "https://arxiv.org/src/2412.12345" -o paper-source.tar.gz
tar -xzf paper-source.tar.gz
ls *.tex

# Convert .tex to markdown (approximate)
pandoc main.tex -o paper.md --mathjax

Warning: Pandoc .tex → markdown conversion is imperfect for complex papers. Use it as a starting point and clean up.

---

Type C: Scanned/Image PDF

Examples: Older papers, scans of physical documents, digitized theses

Identification

• Zero selectable text
• pdffonts returns nothing
• All pages are raster images at constant DPI

What Works

marker-pdf Pipeline

# Install (requires GPU or CPU inference)
pip install marker-pdf

# Convert single file
marker_single paper.pdf /tmp/marker-out --output_format markdown

# If torch error on Apple Silicon:
PYTORCH_ENABLE_MPS_FALLBACK=1 marker_single paper.pdf /tmp/marker-out

# Check output quality
cat /tmp/marker-out/*.md | wc -l   # should be > 100 lines for a real paper

Quality check: marker-pdf quality varies. Always review the output equations against the original PDF screenshots. Expect 80-90% accuracy on clean scans; less on degraded documents.

---

Decision Flowchart

Can you copy-paste text from the PDF?
├── No → TYPE C (scanned) → use marker-pdf
└── Yes
    ├── Copy-pasted math shows Unicode symbols (∑ π α)?
    │   └── Yes → TYPE A (Word) → pymupdf4llm + manual math transcription
    └── No
        ├── Is paper on arxiv with .tex source?
        │   └── Yes → extract from .tex (bypass PDF entirely)
        └── No → TYPE B (LaTeX) → pymupdf4llm + marker-pdf attempt