Firecrawl Research Patterns

Firecrawl Research Patterns

Programmatic patterns for using self-hosted Firecrawl in research workflows — search, scrape, route academic papers, run recursive deep research, and persist raw results for future re-analysis. Also covers self-hosted deployment, health checks, and recovery.

For archiving AI chat conversations (ChatGPT/Gemini shares), see Skill(gh-tools:research-archival).

---

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

FIRST — TodoWrite Task Templates

MANDATORY: Select and load the appropriate template before any research work.

Intent routing — AI chat share URLs (chatgpt / gemini / claude)

AI chat share URLs (chatgpt.com/share/*, chat.openai.com/share/*, gemini.google.com/share/*, g.co/gemini/share/*, claude.ai/share/*, claude.ai/chat/*) can be processed by either this skill or Skill(gh-tools:research-archival). Pick by intent, not URL pattern:

Both paths share the same Firecrawl backend. research-archival calls Firecrawl too — it adds an archival layer on top. There is no scraping capability gap between the two; the difference is what happens to the bytes after they come back.

WebFetch limitation, regardless of intent: Claude Code hard-blocks WebFetch against chatgpt.com. Use Firecrawl (this skill, any port) or Jina Reader instead. Verified 2026-05-27.

Empirical note (2026-05-27): port 3003 successfully scrapes ChatGPT shares — curl :3003/scrape?url=...&name=... returned a 75 KB / 1,734-line markdown for a real ChatGPT share via the Caddy two-step pattern (see Section 5). Earlier guidance that said "route AI chat shares out" was overcautious and contradicted Section 5's port table.

Template A — Single Firecrawl Search + Persist

1. Health check — GET http://littleblack.tail0f299b.ts.net:3002/ (expect 200 + {"message":"Firecrawl API",...}; NEVER use /v1/health — it 404s)
2. Execute search — POST /v1/search with query, limit, scrapeOptions
3. Persist raw results — save each result page to docs/research/corpus/ with frontmatter
4. Update corpus index — append entries to docs/research/corpus-index.jsonl
5. Extract findings — summarize key learnings from raw corpus files

Template B — Academic Paper Retrieval + Persist

1. Identify source — classify URL/DOI per academic-paper-routing.md decision tree
2. Route to scraper — arxiv direct HTML, Semantic Scholar API, Firecrawl, or Jina Reader
3. Scrape content — execute fetch with appropriate method and timeout
4. Persist raw result — save to docs/research/corpus/ with academic-specific frontmatter
5. Update corpus index — append entry to corpus-index.jsonl
6. Summarize paper — extract key claims, methods, results from raw corpus file

Template C — Full Recursive Deep Research with Corpus

1. Health check — GET http://littleblack.tail0f299b.ts.net:3002/ (expect 200 + Firecrawl banner; NEVER /v1/health — it 404s)
2. Initialize parameters — set breadth (default 4), depth (default 2), concurrency (default 2)
3. Generate search queries — LLM generates N queries from topic + prior learnings
4. Execute searches — Firecrawl /v1/search for each query via p-limit(concurrency)
5. Persist raw results — save ALL scraped pages to docs/research/corpus/ with provenance
6. Extract learnings — LLM extracts key findings + follow-up questions per result set
7. Recurse — for each follow-up, recurse with breadth=ceil(breadth/2), depth=depth-1
8. Base case — depth=0, return accumulated learnings
9. Synthesize report — LLM generates final markdown from all learnings
10. Write session report — save to docs/research/sessions/ with corpus file references
11. Update corpus index — append all new entries to corpus-index.jsonl

Template D — Corpus Review / Re-Analysis

1. Inventory corpus — read docs/research/corpus-index.jsonl, filter by session/topic/date
2. Read raw files — load matching corpus files from docs/research/corpus/
3. Re-analyze — extract new insights with current context/questions
4. Update session report — amend or create new session report in docs/research/sessions/

Template E — Image-Rich Paper with Inline Figures

Use when paper contains architecture diagrams, result plots, attention maps, or any critical visual content.

1. Scrape text — use port 3003 (preferred, preserves absolute image URLs) or Jina fallback
2. Detect figures — scan scraped markdown for ![alt](URL) patterns with .png/.jpg/.svg
3. Extract figure URLs — for arXiv: probe https://arxiv.org/html/{id}v{n}/x{N}.png until 404
4. Keep URLs inline — DO NOT rewrite to local relative paths (breaks GitHub rendering)
5. Ensure inline embedding — markdown body must have ![Figure N](absolute-url) for each figure
6. Catalog in frontmatter — add figure_count and figure_urls list (all absolute URLs)
7. Save corpus file — GFM markdown with inline absolute URLs renders on GitHub without hosting
8. Update corpus-index.jsonl — include has_figures: true, figure_count, figure_urls

---

Section 1 — Programmatic Firecrawl Usage

Instance: Self-hosted on littleblack — Debian 12 (bookworm), kernel 6.1.0-31, hostname kab, login user yca, RTX 2080 Ti, 62 GiB RAM. No API key required for any Firecrawl endpoint.

MagicDNS preflight (run before relying on bare littleblack):

# macOS — does the OS resolver know about the bare name?
dscacheutil -q host -a name littleblack | grep -q '^ip_address'  && echo OK || echo MISSING

# Cross-platform — does any path resolve?
getent hosts littleblack 2>/dev/null || ping -c1 -W1 littleblack 2>&1 | head -1

If preflight returns MISSING / "cannot resolve", use the FQDN row. SSH happens to work because ~/.ssh/config hard-codes the FQDN under the Host littleblack alias — that's an SSH-only shortcut, not a system-wide DNS facility. Bare littleblack over HTTP fails silently as HTTP 000 when the resolver doesn't have it; the failure mode is invisible without ping/dscacheutil. Confirmed broken on m3max (this Mac) as of 2026-05-27.

SSH (for ops, not API calls): ssh littleblack — defined in ~/.ssh/config as HostName littleblack.tail0f299b.ts.net, User yca, IdentityFile ~/.ssh/id_ed25519_zerotier_np.

Why fetch() Instead of @mendable/firecrawl-js SDK

The official SDK uses jiti for dynamic imports, which is incompatible with Bun's module resolution. Direct fetch() calls are simpler, more reliable, and have zero dependencies.

Two Endpoints

See api-endpoint-reference.md for full request/response contracts.

Quick Examples

Use the FQDN base URL — works on every tailnet-attached client regardless of MagicDNS resolver state. Pull from $FIRECRAWL_BASE env var if your project sets one, otherwise hard-code the FQDN:

const FIRECRAWL_BASE =
  process.env.FIRECRAWL_BASE ?? "http://littleblack.tail0f299b.ts.net:3002";

Search (returns multiple results with markdown):

const res = await fetch(`${FIRECRAWL_BASE}/v1/search`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    query: "mixture of experts scaling laws",
    limit: 5,
    scrapeOptions: { formats: ["markdown"] },
  }),
});
const { data } = await res.json(); // data: [{ url, markdown, metadata }]

Scrape (single URL):

const res = await fetch(`${FIRECRAWL_BASE}/v1/scrape`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://arxiv.org/abs/2401.12345",
    formats: ["markdown"],
    waitFor: 3000, // ms — for JS-heavy pages
  }),
});
const { data } = await res.json(); // data: { markdown, metadata }

Error Handling

// Always set a timeout
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 15_000);

try {
  const res = await fetch(url, { ...opts, signal: controller.signal });
  if (!res.ok) throw new Error(`Firecrawl: ${res.status} ${res.statusText}`);
  const json = await res.json();
  if (!json.data || (Array.isArray(json.data) && json.data.length === 0)) {
    // Empty results — not an error, but no content to process
  }
} finally {
  clearTimeout(timeoutId);
}

Health Check

There is no `/v1/health` endpoint on this Firecrawl build. Probing it returns HTTP 404 (Express's HTML error page), which looks like a service-down signal but isn't. Use the root / endpoint, which returns HTTP 200 with {"message":"Firecrawl API","documentation_url":"https://docs.firecrawl.dev"}. Confirmed 2026-05-27 against ports 3002 / FQDN / IP.

// Quick health check before starting a research session.
// Uses the Tailscale FQDN — works regardless of MagicDNS resolver state.
const FIRECRAWL_BASE = "http://littleblack.tail0f299b.ts.net:3002";
const res = await fetch(`${FIRECRAWL_BASE}/`);
if (!res.ok) {
  throw new Error(
    `Firecrawl unreachable (${res.status}) — see self-hosted-operations.md and self-hosted-troubleshooting.md`,
  );
}
const banner = await res.json();
if (banner.message !== "Firecrawl API") {
  throw new Error(
    `Unexpected root response: ${JSON.stringify(banner).slice(0, 200)}`,
  );
}

For a true end-to-end probe (proves the full search/scrape stack works, not just the HTTP listener), POST /v1/scrape against https://example.com and check success: true:

curl -s --max-time 15 -X POST \
  "http://littleblack.tail0f299b.ts.net:3002/v1/scrape" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}' \
  | python3 -c "import sys, json; d=json.load(sys.stdin); print('OK' if d.get('success') else 'FAIL')"

---

Section 2 — Academic Paper Routing

Route paper retrieval to the most effective method based on source. Full decision tree in academic-paper-routing.md.

Quick Reference

DOI Resolution

// DOI → publisher URL → route to appropriate scraper
const res = await fetch(`https://doi.org/${doi}`, { redirect: "follow" });
const publisherUrl = res.url; // e.g., https://dl.acm.org/doi/10.1145/...
// Then route publisherUrl through the decision tree above

---

Section 3 — Recursive Research Protocol

The iterative search → extract → recurse → synthesize pattern. Full step-by-step protocol in recursive-research-protocol.md.

Algorithm Overview

deepResearch(topic, breadth=4, depth=2, concurrency=2):
   1. Generate N search queries (N = breadth) from topic + prior learnings
   2. For each query (via p-limit concurrency):
      a. Firecrawl /v1/search → get results
      b. PERSIST each raw result to docs/research/corpus/
      c. Extract learnings + follow-up questions
   3. For each follow-up question:
      → Recurse with breadth=ceil(breadth/2), depth=depth-1
   4. Base case: depth=0 → return accumulated learnings
   5. Synthesize final report from all learnings
   6. Write session report to docs/research/sessions/

Default Parameters (from working implementation)

Token Budget

Each search returns up to 5 pages. Trim each page to ~25,000 tokens before LLM processing:

function trimToTokenLimit(text: string, maxTokens: number): string {
  if (!text) return "";
  const estimatedTokens = Math.ceil(text.length / 3.5);
  if (estimatedTokens <= maxTokens) return text;
  const maxChars = Math.floor(maxTokens * 3.5 * 0.8);
  return text.slice(0, maxChars);
}

Partial Failure Principle

Partial results are better than total failure. If a query fails, log it and continue with remaining queries. Never abort the entire research session because one query timed out.

---

Section 4 — Raw Corpus Persistence

Critical principle: Every Firecrawl-scraped page must be persisted in its original raw markdown with provenance metadata. Synthesized reports reference these originals but never replace them.

Full format specification in corpus-persistence-format.md.

Directory Layout

{project-root}/
├── docs/research/
│   ├── corpus/                              # Raw scraped pages (committed)
│   │   └── YYYY-MM-DD-{slug}.md             # One file per scraped URL
│   ├── sessions/                            # Research session reports (committed)
│   │   └── YYYY-MM-DD-{topic-slug}.md       # Synthesized report with corpus refs
│   └── corpus-index.jsonl                   # Append-only registry (committed)

Corpus File Frontmatter

---
source_url: https://arxiv.org/html/2401.12345
scraped_at: "2026-02-25T14:30:00Z"
scraper: firecrawl
firecrawl_endpoint: /v1/search
search_query: "mixture of experts scaling"
result_index: 2
research_session: "2026-02-25-moe-scaling"
depth_level: 1
claude_code_uuid: SESSION_UUID
content_tokens_approx: 4200
---
[RAW MARKDOWN FROM FIRECRAWL — NEVER MODIFIED]

Key Rules

1. Content below --- is the exact markdown Firecrawl returned — no summarization, trimming, or reformatting 2. One file per URL per scrape — if the same URL is scraped in multiple sessions, each gets its own timestamped file 3. File naming: YYYY-MM-DD-{slug}.md where slug is kebab-case from page title or URL path (max 60 chars) 4. Session reports in docs/research/sessions/ reference corpus files by relative path

Corpus Index (JSONL)

{
  "url": "https://arxiv.org/html/2401.12345",
  "file": "corpus/2026-02-25-moe-scaling-arxiv-2401-12345.md",
  "scraped_at": "2026-02-25T14:30:00Z",
  "session": "2026-02-25-moe-scaling",
  "tokens": 4200,
  "scraper": "firecrawl"
}

Why This Matters

• LLM re-analysis: Future sessions can re-read raw corpus files and extract different insights with better prompts or newer models
• No information loss: Synthesis drops details; raw files preserve everything Firecrawl captured
• Deduplication awareness: The JSONL index lets agents skip URLs already in the corpus
• Git-friendly: Markdown files diff cleanly, JSONL is append-only

---

Section 5 — Self-Hosted Operations

The Firecrawl instance runs on littleblack (Debian 12, RTX 2080 Ti, hostname kab). System uptime is in the 100+ day range; Firecrawl is stable on this host. No API key needed. For the full access matrix (Tailscale FQDN / IP / MagicDNS, same-LAN, legacy ZeroTier), see Section 1 "Instance". Section 5 examples use the Tailscale FQDN (littleblack.tail0f299b.ts.net) since it works on every tailnet-attached client regardless of resolver state — substitute any path from the Section 1 table when appropriate.

When to use which port:

Two-step pattern — port 3003 and 3004 do not return markdown directly. They scrape, save to Caddy-served storage, and return a JSON pointer. You then fetch the markdown from the returned Caddy URL. (Discovered 2026-05-27 — earlier snippets that ran a single curl :3003/scrape?... and treated the response body as the scraped content were silently wrong: that body is {"url":"...","file":"..."}, not markdown.)

BASE="http://littleblack.tail0f299b.ts.net"   # FQDN — works without MagicDNS
URL="https://chatgpt.com/share/<id>"          # or any JS-rendered page
NAME="chatgpt-metric-stack-2026-05-27"        # slug — NO whitespace or special chars

# URL-encode the target (avoid Python's trailing newline — use end='')
ENC=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1], safe=''), end='')" "$URL")

# Step 1 — POST scrape request, get JSON pointer
SCRAPE_JSON=$(curl -s --max-time 90 "${BASE}:3003/scrape?url=${ENC}&name=${NAME}")
echo "$SCRAPE_JSON"
# → {"url":"http://172.25.236.1:8080/<NAME>-<timestamp>.md","file":"<NAME>-<timestamp>.md"}

# Step 2 — extract Caddy URL, rewrite host to FQDN (the JSON returns the legacy ZeroTier IP),
# then fetch the actual markdown
FILE=$(echo "$SCRAPE_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin)['file'])")
curl -s --max-time 30 "${BASE}:8080/${FILE}" -o "/tmp/${FILE}"
wc -c "/tmp/${FILE}"   # sanity-check that content actually arrived

The JSON response embeds the legacy ZeroTier IP (http://172.25.236.1:8080/...) — do NOT follow that URL directly if ZeroTier isn't reachable from your client. Always reconstruct the Caddy URL using your preferred host base (${BASE}:8080/${FILE}), as shown above.

Shell-quoting trap (zsh/bash): the & in ?url=X&name=Y is fine inside double quotes, but if you splice $(...) command substitution mid-URL, any trailing newline from Python's print() becomes %0A in the encoded URL and the server rejects the malformed target silently. Always use end='' in the encoder or pipe through tr -d '\n'.

Cloudflare-bypass wrapper (port 3004) follows the same POST → Caddy two-step:

curl -s --max-time 90 "${BASE}:3004/scrape-cf?url=${ENC}&name=${NAME}"
# → same JSON shape; same Caddy GET to retrieve the markdown

Health probes — none of these services expose a /v1/health or /health endpoint. Probe the root and inspect the response body for the service's identity string:

BASE="http://littleblack.tail0f299b.ts.net"

# Port 3002 — Firecrawl API
# Healthy: HTTP 200, body contains '"message":"Firecrawl API"'
curl -s --max-time 4 "${BASE}:3002/" | grep -q '"Firecrawl API"' && echo "3002 OK" || echo "3002 DOWN"

# Port 3003 — Scraper wrapper
# Healthy: HTTP 400, body contains 'Usage: /scrape?url=' (service up, rejects missing params)
curl -s --max-time 4 "${BASE}:3003/" | grep -q 'Usage: /scrape' && echo "3003 OK" || echo "3003 DOWN"

# Port 3004 — Cloudflare bypass wrapper
# Healthy: HTTP 200, body contains '"service":"cloudflare-bypass-scraper"'
curl -s --max-time 4 "${BASE}:3004/" | grep -q 'cloudflare-bypass-scraper' && echo "3004 OK" || echo "3004 DOWN"

# Port 8080 — Caddy
# Healthy: HTTP 200 (directory listing)
curl -s --max-time 4 -o /dev/null -w '%{http_code}\n' "${BASE}:8080/" | grep -q '^200$' && echo "8080 OK" || echo "8080 DOWN"

# Real end-to-end probe — proves /v1/scrape works against a known-good URL
curl -s --max-time 15 -X POST "${BASE}:3002/v1/scrape" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}' \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK' if d.get('success') else 'FAIL')"

Do not probe /v1/health, /health, or /v0/health on port 3002 — all three return HTTP 404 (Express's HTML error page), which looks like a service-down signal but isn't. Confirmed 2026-05-27.

For architecture diagrams, health checks, recovery commands, and deployment details, see:

• Self-Hosted Operations — Architecture, health checks, recovery commands
• Self-Hosted Bootstrap Guide — Fresh installation (7 steps)
• Self-Hosted Best Practices — Docker restart policies, monitoring
• Self-Hosted Troubleshooting — Symptom-based diagnosis

---

Section 6 — Image and Figure Capture

Text-only scrapers (Jina, direct Firecrawl) capture prose but lose architecture diagrams, result plots, and attention maps. For image-rich papers, always capture figures.

When to Capture Images

Capture figures when the paper contains any of:

• Architecture diagrams (model structure, attention patterns)
• Benchmark/result comparison plots
• Qualitative examples (generated outputs, visualizations)
• Algorithm flowcharts or pseudocode figures

arXiv HTML Figure URL Discovery

arXiv HTML papers store figures at sequential absolute URLs (x1.png, x2.png, ...). Probe to discover all figure URLs — do NOT download them locally:

ARXIV_ID="2312.00752"
ARXIV_VER="v2"
BASE_URL="https://arxiv.org/html/${ARXIV_ID}${ARXIV_VER}"
FIGURE_URLS=()

# Probe sequential URLs until 404 — collect absolute URLs only
for i in $(seq 1 50); do
  url="${BASE_URL}/x${i}.png"
  status=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  if [ "$status" != "200" ]; then
    echo "Stopped at x${i}.png (${status}) — found ${#FIGURE_URLS[@]} figures"
    break
  fi
  FIGURE_URLS+=("$url")
  echo "Found: $url"
done

The collected absolute URLs go directly into the markdown body and frontmatter — no local copies needed.

Inline Figure Embedding (GFM)

Each figure must appear inline in the corpus markdown as an absolute URL so GitHub renders it in-place:

## Key Figures

![Figure 1 — Mamba SSM architecture](https://arxiv.org/html/2312.00752v2/x1.png)

![Figure 2 — Selective scan mechanism](https://arxiv.org/html/2312.00752v2/x2.png)

![Figure 3 — Performance vs sequence length](https://arxiv.org/html/2312.00752v2/x3.png)

Never rewrite to relative paths like ./figures/x1.png — relative paths break on GitHub unless images are committed to the same repo.

Extracting Existing Inline URLs from Scraped Markdown

When port 3003 (Playwright) already embedded absolute URLs in the scraped markdown, extract them for the frontmatter catalog:

CORPUS_FILE="docs/research/corpus/2026-03-13-mamba-ssm.md"

# Extract all absolute image URLs already in the markdown
grep -oE 'https://[^)]+\.(png|jpg|svg|gif|webp)' "$CORPUS_FILE" | sort -u

These URLs are already inline — just copy them into the frontmatter figure_urls list.

Frontmatter for Image-Rich Papers

The YAML frontmatter catalogs all figure source URLs for provenance. The markdown body embeds them inline:

---
source_url: https://arxiv.org/html/2312.00752v2
scraped_at: "2026-03-13T00:00:00Z"
scraper: firecrawl-port3003
tags: [ssm, state-space-model, mamba, sequence-modeling]
content_tokens_approx: 4200
has_figures: true
figure_count: 12
figure_urls:
  - https://arxiv.org/html/2312.00752v2/x1.png
  - https://arxiv.org/html/2312.00752v2/x2.png
  - https://arxiv.org/html/2312.00752v2/x3.png
  - https://arxiv.org/html/2312.00752v2/x4.png
  - https://arxiv.org/html/2312.00752v2/x5.png
---

Corpus Index Entry with Figures

{
  "url": "https://arxiv.org/html/2312.00752v2",
  "file": "corpus/2026-03-13-mamba-ssm.md",
  "scraped_at": "2026-03-13T00:00:00Z",
  "session": "2026-03-13-mamba-ssm",
  "scraper": "firecrawl-port3003",
  "has_figures": true,
  "figure_count": 12,
  "figure_urls": [
    "https://arxiv.org/html/2312.00752v2/x1.png",
    "https://arxiv.org/html/2312.00752v2/x2.png"
  ]
}

Port 3003 vs Jina Reader: Empirical Comparison (arXiv)

Validated on arXiv:2312.00752v2 (Mamba paper) — both scrapers running, same URL:

Verdict: Firecrawl (port 3002/3003) gets 17% more bytes, 2.1× more lines, 22% more words, 1 extra figure vs Jina. Port 3002 and 3003 produce identical markdown (3003 just wraps 3002 and saves to Caddy). Both emit absolute inline figure URLs — no URL reconstruction needed from either scraper.

Note on the earlier session timeout: The March 2026 session failure was machine downtime (littleblack was offline), not a routing issue. When littleblack is up, port 3003 reaches arxiv.org fine.

Recommended arXiv workflow:

1. Port 3003 (preferred) — more complete content, figures inline, saves to Caddy 2. Jina Reader (fallback when littleblack is down) — 17% less content but still gets absolute figure URLs 3. Probe loop to build figure_urls frontmatter catalog regardless of scraper used 4. For human-readable math on GitHub: Pandoc from arXiv LaTeX source (see below)

Math Rendering: Empirically Validated Approaches

Validated on arXiv:2312.00752v2 (Mamba paper), March 2026.

Firecrawl/Jina Math Output: Unreadable on GitHub

Both Firecrawl (port 3002/3003) and Jina Reader extract math by doubling content — each equation appears as a Unicode render followed immediately by raw LaTeX source, packed into markdown table cells with \displaystyle prefixes and \\bm{} escaping. Example from the empirical test:

|     | h′​(t)\\displaystyle h^{\\prime}(t) | \=𝑨​h​(t)+𝑩​x​(t)\\displaystyle=\\bm{A}h(t)+\\bm{B}x(t) |     | (1a) |

No $...$ delimiters — GitHub cannot render this as math. The raw LaTeX portion is parseable by an LLM (equations are present), but the output is completely unreadable to humans on GitHub.

For LLM consumption: Firecrawl's doubled content is sufficient — the LaTeX source is embedded and an LLM can extract it.

For human-readable GitHub rendering: Use Pandoc from the arXiv LaTeX source tarball (see below).

Pandoc from arXiv LaTeX Source (Human-Readable Math)

Produces proper $inline$ and `math ` display blocks that GitHub's MathJax/KaTeX renders natively:

ARXIV_ID="2312.00752"

# Download arXiv LaTeX source tarball
curl -L "https://arxiv.org/src/${ARXIV_ID}" -o "${ARXIV_ID}-src.tar.gz"
mkdir -p "${ARXIV_ID}-src"
tar xzf "${ARXIV_ID}-src.tar.gz" -C "${ARXIV_ID}-src/"

# Find main .tex entry point and section files
ls "${ARXIV_ID}-src/"*.tex
ls "${ARXIV_ID}-src/src/"*.tex 2>/dev/null  # some papers put sections in src/

# Option A: Convert individual section files (safer — avoids macro parse errors)
pandoc "${ARXIV_ID}-src/src/background.tex" \
  --to gfm+tex_math_dollars \
  --wrap=none \
  -o "${ARXIV_ID}-background.md"

# Option B: Convert full main.tex (may fail on custom macros like \iftoggle)
pandoc "${ARXIV_ID}-src/main.tex" \
  --to gfm+tex_math_dollars \
  --wrap=none \
  -o "${ARXIV_ID}-pandoc.md"

Install: brew install pandoc. Works on any arXiv paper that publishes LaTeX source (most do).

Pandoc output quality (empirically validated):

• Inline math: $x(t) \in \R \mapsto y(t) \in \R$ ✅ GitHub renders
• Display math: `math\n\begin{align}\nh'(t) &= \A h(t) + \B x(t)\n\end{align}\n` ✅ GitHub renders
• Custom macros (\A, \B, \R, \dt, \dA, \dB): ⚠️ undefined in KaTeX — macros pass through as-is and may partially fail on GitHub without the preamble's \newcommand definitions

Handling custom macros: Prepend the \newcommand block from main.tex preamble to the output:

````bash

Extract custom macro definitions from preamble

grep '\\newcommand\|\\renewcommand\|\\def ' "${ARXIV_ID}-src/main.tex" > macros.tex

Pandoc does not read preamble macros — include them explicitly in a math block at the top:

echo '``math' > preamble-block.md cat macros.tex >> preamble-block.md echo '``' >> preamble-block.md

cat preamble-block.md "${ARXIV_ID}-pandoc.md" > "${ARXIV_ID}-with-macros.md" ````

Known Pandoc parse errors on arXiv LaTeX:

---

Anti-Patterns

---

References

• API Endpoint Reference — /v1/search and /v1/scrape contracts
• Academic Paper Routing — Decision tree for paper sources
• Recursive Research Protocol — Step-by-step recursive pattern
• Corpus Persistence Format — Raw content archival format + directory layout
• Self-Hosted Operations — Architecture, health checks, recovery
• Self-Hosted Bootstrap Guide — Fresh installation guide
• Self-Hosted Best Practices — Docker restart policies, monitoring
• Self-Hosted Troubleshooting — Symptom-based diagnosis and recovery

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure. 2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match. 3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

Academic Paper Routing

Decision tree for choosing the best retrieval method based on paper source. Optimized for content quality and reliability.

---

Routing Table

---

Source-Specific Patterns

arxiv.org

arxiv provides multiple access paths. Prefer HTML over PDF for LLM consumption.

arxiv.org/abs/2401.12345      → metadata page (abstract, authors)
arxiv.org/html/2401.12345     → full HTML paper (preferred for LLM)
arxiv.org/pdf/2401.12345      → PDF (less useful for text extraction)

Primary: Port 3003 (Firecrawl wrapper) — empirically gets 17% more content than Jina:

curl "http://littleblack:3003/scrape?url=https://arxiv.org/html/2401.12345&name=paper-slug"
# Returns: {"url":"http://littleblack:8080/paper-slug-TIMESTAMP.md","file":"..."}

Fallback (when littleblack is down): Jina Reader:

curl -s "https://r.jina.ai/https://arxiv.org/html/2401.12345" -o paper.md

Empirically validated (arXiv:2312.00752v2, Mamba paper, March 2026):

• Port 3003: 99,104 bytes, 1,267 lines, 13 figures (absolute inline URLs ✅)
• Jina Reader: 84,832 bytes, 596 lines, 12 figures (absolute inline URLs ✅)
• Both emit absolute figure URLs — no URL reconstruction needed
• The earlier session timeout was machine downtime, not a routing issue — port 3003 reaches arxiv.org fine when littleblack is online

Math rendering gap (empirically validated): Both Jina and Firecrawl double all equations — each equation appears as Unicode render + raw LaTeX source in the same table cell with \displaystyle prefixes, no $...$ delimiters. Unreadable on GitHub for humans; LaTeX is still parseable by LLMs. For human-readable GFM math, use Pandoc from the arXiv LaTeX source tarball (--to gfm+tex_math_dollars) — produces proper $inline$ and `math ` blocks GitHub renders, but paper-specific custom macros (\A, \B, \R, etc.) need the preamble's \newcommand definitions prepended (see Section 6 of SKILL.md).

arXiv Figure URL Pattern

arXiv HTML papers store figures at sequential absolute URLs (x1.png, x2.png, …). The correct approach is to keep these URLs inline in the markdown body and catalog them in the YAML frontmatter — do NOT download to local paths (relative paths break on GitHub).

# Probe sequential URLs to discover figure_count — collect absolute URLs for frontmatter
ARXIV_ID="2401.12345"
BASE="https://arxiv.org/html/${ARXIV_ID}/"
FIGURE_URLS=()

for i in $(seq 1 50); do
  url="${BASE}x${i}.png"
  http_code=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  if [ "$http_code" = "404" ]; then
    echo "Found ${#FIGURE_URLS[@]} figures (stopped at x${i}.png)"
    break
  fi
  FIGURE_URLS+=("$url")
done

# Embed inline in GFM corpus markdown (renders on GitHub without hosting):
for i in "${!FIGURE_URLS[@]}"; do
  echo "![Figure $((i+1))](${FIGURE_URLS[$i]})"
done

Frontmatter catalog (YAML, inside the corpus .md file):

has_figures: true
figure_count: 12
figure_urls:
  - https://arxiv.org/html/2401.12345/x1.png
  - https://arxiv.org/html/2401.12345/x2.png
  - https://arxiv.org/html/2401.12345/x3.png

Notes:

• Files are x1.png, x2.png, … (sequential, 1-indexed); first 404 means no more figures
• Some papers use .svg or .jpg; probe .png first, then alternatives
• Version suffix: https://arxiv.org/html/2401.12345v2/ for a specific version
• Port 3003 already embeds these as inline absolute URLs — just extract them with grep -oE 'https://arxiv.org/html/[^)]+\.png'

Fallback: If /html/ is unavailable (older papers), use Firecrawl to scrape /abs/:

const res = await fetch("http://littleblack:3002/v1/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: `https://arxiv.org/abs/${arxivId}`,
    formats: ["markdown"],
  }),
});

Semantic Scholar

API-first approach for structured metadata. Free tier: 100 requests/5 minutes.

// Search by title
const res = await fetch(
  `https://api.semanticscholar.org/graph/v1/paper/search?query=${encodeURIComponent(title)}&limit=5&fields=title,abstract,url,year,authors,citationCount`,
);
const { data } = await res.json();

// Get by paper ID (S2 ID, DOI, arxiv ID, etc.)
const paper = await fetch(
  `https://api.semanticscholar.org/graph/v1/paper/${paperId}?fields=title,abstract,url,year,authors,references,citations`,
);

Fallback: If API rate-limited or paper not indexed, search via Firecrawl:

const res = await fetch("http://littleblack:3002/v1/search", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    query: `"${paperTitle}" site:semanticscholar.org`,
    limit: 3,
    scrapeOptions: { formats: ["markdown"] },
  }),
});

Conference Proceedings (NeurIPS, ICML, ICLR)

These use JS-rendered pages. Always use waitFor:

const res = await fetch("http://littleblack:3002/v1/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: proceedingsUrl,
    formats: ["markdown"],
    waitFor: 2000,
  }),
});

IEEE Xplore / ACM Digital Library

Heavy JS SPAs that require extended wait times:

const res = await fetch("http://littleblack:3002/v1/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: ieeeOrAcmUrl,
    formats: ["markdown"],
    waitFor: 3000, // Critical — page won't render without this
  }),
});

Note: Paywalled content may return only abstract + metadata. For full text, check if the author has a preprint on arxiv or their personal website.

Author Blogs / Personal Websites

Static HTML — Jina Reader is faster and cleaner than Firecrawl:

curl -s "https://r.jina.ai/https://author-blog.com/post-about-paper"

Or via WebFetch in Claude Code:

WebFetch(url="https://r.jina.ai/https://author-blog.com/post", prompt="Extract full content")

---

DOI Resolution

DOIs redirect to the publisher's canonical URL. Resolve first, then route:

// Follow redirects to get the publisher URL
const res = await fetch(`https://doi.org/${doi}`, { redirect: "follow" });
const publisherUrl = res.url;

// Route based on publisher domain
if (publisherUrl.includes("arxiv.org")) {
  // → arxiv path
} else if (publisherUrl.includes("dl.acm.org")) {
  // → ACM DL path with waitFor: 3000
} else if (publisherUrl.includes("ieeexplore.ieee.org")) {
  // → IEEE path with waitFor: 3000
} else {
  // → Generic Firecrawl scrape
}

---

Preprint vs Published Version Detection

When a paper exists in multiple locations:

1. Prefer arxiv HTML — free, structured, no paywalls 2. Check Semantic Scholar for citation metadata + links to all versions 3. Use published version only when arxiv version is significantly outdated (check version dates)

// Semantic Scholar returns all known versions
const paper = await fetch(
  `https://api.semanticscholar.org/graph/v1/paper/search?query=${title}&fields=externalIds,url`,
);
// externalIds: { ArXiv: "2401.12345", DOI: "10.1145/...", ... }

---

Citation Extraction

For extracting references from a paper's bibliography:

1. Semantic Scholar API — best for structured citation data:

const refs = await fetch(
  `https://api.semanticscholar.org/graph/v1/paper/${paperId}/references?fields=title,authors,year,externalIds&limit=100`,
);

1. Firecrawl scrape of references section — when API doesn't have the paper

---

Complement to Existing Routing

This table extends Skill(gh-tools:research-archival) URL routing, which covers:

• ChatGPT share URLs → Jina Reader
• Gemini share URLs → Firecrawl
• Claude artifacts → Jina Reader

This skill adds academic-specific routing. The two are complementary — use research-archival for AI chat conversations, this skill for academic papers and research content.

API Endpoint Reference

Firecrawl self-hosted API contracts for the two endpoints used in research workflows, plus health check.

Base URL: http://littleblack:3002 (Tailscale primary, no API key needed; legacy ZeroTier fallback at 172.25.236.1:3002)

---

POST /v1/search

Combined search + scrape. Searches the web for a query and returns scraped markdown for each result.

Request

{
  "query": "mixture of experts scaling laws",
  "limit": 5,
  "scrapeOptions": {
    "formats": ["markdown"]
  }
}

Response (200 OK)

{
  "success": true,
  "data": [
    {
      "url": "https://example.com/page1",
      "markdown": "# Page Title\n\nContent...",
      "metadata": {
        "title": "Page Title",
        "description": "Meta description",
        "sourceURL": "https://example.com/page1"
      }
    }
  ]
}

Error Responses

curl Example

curl -s -X POST http://littleblack:3002/v1/search \
   -H "Content-Type: application/json" \
   -d '{
      "query": "transformer attention mechanism",
      "limit": 3,
      "scrapeOptions": { "formats": ["markdown"] }
   }' | jq '.data[].url'

fetch() Example

async function firecrawlSearch(
  query: string,
  limit = 5,
): Promise<SearchResult> {
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), 15_000);

  try {
    const res = await fetch("http://littleblack:3002/v1/search", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        query,
        limit,
        scrapeOptions: { formats: ["markdown"] },
      }),
      signal: controller.signal,
    });

    if (!res.ok) {
      throw new Error(
        `Firecrawl search failed: ${res.status} ${res.statusText}`,
      );
    }

    return await res.json();
  } finally {
    clearTimeout(timeoutId);
  }
}

---

POST /v1/scrape

Single URL scrape. Fetches a specific URL and returns its content as markdown.

Request

{
  "url": "https://arxiv.org/abs/2401.12345",
  "formats": ["markdown"],
  "waitFor": 3000
}

When to use `waitFor`: JS-heavy SPAs (IEEE Xplore, ACM DL, NeurIPS proceedings). Static pages (arxiv, blogs) don't need it.

Response (200 OK)

{
  "success": true,
  "data": {
    "markdown": "# Paper Title\n\nAbstract...",
    "metadata": {
      "title": "Paper Title",
      "description": "Abstract text",
      "sourceURL": "https://arxiv.org/abs/2401.12345"
    }
  }
}

curl Example

# Simple static page
curl -s -X POST http://littleblack:3002/v1/scrape \
   -H "Content-Type: application/json" \
   -d '{"url":"https://arxiv.org/abs/2401.12345","formats":["markdown"]}' \
   | jq -r '.data.markdown'

# JS-heavy page (wait for rendering)
curl -s -X POST http://littleblack:3002/v1/scrape \
   -H "Content-Type: application/json" \
   -d '{"url":"https://dl.acm.org/doi/10.1145/12345","formats":["markdown"],"waitFor":3000}' \
   | jq -r '.data.markdown'

fetch() Example

async function firecrawlScrape(
  url: string,
  waitFor?: number,
): Promise<ScrapeResult> {
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), 30_000);

  try {
    const res = await fetch("http://littleblack:3002/v1/scrape", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url,
        formats: ["markdown"],
        ...(waitFor ? { waitFor } : {}),
      }),
      signal: controller.signal,
    });

    if (!res.ok) {
      throw new Error(
        `Firecrawl scrape failed: ${res.status} ${res.statusText}`,
      );
    }

    return await res.json();
  } finally {
    clearTimeout(timeoutId);
  }
}

---

GET /v1/health

Health check endpoint. Use before starting a research session.

Response (200 OK)

{
  "status": "ok"
}

curl Example

curl -sf http://littleblack:3002/v1/health && echo "Firecrawl OK" || echo "Firecrawl UNHEALTHY"

fetch() Example

async function checkFirecrawlHealth(): Promise<boolean> {
  try {
    const res = await fetch("http://littleblack:3002/v1/health", {
      signal: AbortSignal.timeout(5_000),
    });
    return res.ok;
  } catch {
    return false;
  }
}

---

Self-Hosted Specifics

The wrapper at :3003 saves markdown to disk and returns a file URL. For programmatic research workflows, prefer the direct API at :3002 — it gives you full control over the response.

Corpus Persistence Format

Defines how raw Firecrawl output is saved for future Claude Code sessions to re-read and re-analyze.

Design follows existing cc-skills patterns:

• YAML frontmatter + raw markdown body from Skill(gh-tools:research-archival)
• NDJSON append-only registry from Skill(devops-tools:session-chronicle)

---

Directory Layout

{project-root}/
├── docs/research/
│   ├── corpus/                              # Raw scraped pages (committed to git)
│   │   ├── 2026-02-25-moe-scaling-arxiv-2401-12345.md
│   │   ├── 2026-02-25-switch-transformer-google.md
│   │   └── ...
│   ├── sessions/                            # Synthesized research reports (committed)
│   │   ├── 2026-02-25-moe-scaling.md
│   │   └── ...
│   └── corpus-index.jsonl                   # Append-only registry (committed)

---

Raw Corpus File Format

Each file in docs/research/corpus/ = one Firecrawl-scraped URL, preserved exactly as returned.

File Naming

YYYY-MM-DD-{slug}.md

• YYYY-MM-DD — date of scrape
• slug — kebab-case derived from page title or URL path (max 60 chars)

Examples:

• 2026-02-25-moe-scaling-arxiv-2401-12345.md
• 2026-02-25-switch-transformer-google-research.md
• 2026-02-25-expert-parallelism-deepspeed-docs.md

YAML Frontmatter

---
source_url: https://arxiv.org/html/2401.12345
scraped_at: "2026-02-25T14:30:00Z"
scraper: firecrawl
firecrawl_endpoint: /v1/search
search_query: "mixture of experts scaling"
result_index: 2
research_session: "2026-02-25-moe-scaling"
depth_level: 1
claude_code_uuid: SESSION_UUID
content_tokens_approx: 4200
---

Body Content

Everything below the closing --- is the exact markdown Firecrawl returned. Rules:

1. Never modify — no summarization, no trimming, no reformatting 2. No added headers — don't prepend # Title if Firecrawl didn't include one 3. Preserve whitespace — keep original line breaks, spacing, formatting 4. Include artifacts — if Firecrawl returned table markdown, code blocks, etc., keep them

One File Per Scrape

If the same URL is scraped in multiple sessions:

• Each scrape gets its own timestamped file
• The corpus index tracks all versions
• Deduplication is the _index's_ job, not the file system's

This preserves temporal snapshots — content at a URL may change between scrapes.

---

Corpus Index Format

docs/research/corpus-index.jsonl — append-only NDJSON, one line per scraped page.

Schema

{
  "url": "https://arxiv.org/html/2401.12345",
  "file": "corpus/2026-02-25-moe-scaling-arxiv-2401-12345.md",
  "scraped_at": "2026-02-25T14:30:00Z",
  "session": "2026-02-25-moe-scaling",
  "tokens": 4200,
  "scraper": "firecrawl"
}

Usage

Claude Code can query the index to find relevant corpus files:

# Find all corpus files for a session
grep '"session":"2026-02-25-moe-scaling"' docs/research/corpus-index.jsonl | jq -r '.file'

# Check if a URL is already in the corpus
grep '"url":"https://arxiv.org/html/2401.12345"' docs/research/corpus-index.jsonl

# Count corpus entries per session
jq -r '.session' docs/research/corpus-index.jsonl | sort | uniq -c | sort -rn

Append Pattern

import { appendFileSync } from "node:fs";

function appendToCorpusIndex(entry: CorpusIndexEntry): void {
  const line = JSON.stringify(entry) + "\n";
  appendFileSync("docs/research/corpus-index.jsonl", line);
}

---

Session Report Format

Synthesized reports in docs/research/sessions/YYYY-MM-DD-{topic-slug}.md.

Structure

---
topic: "Mixture of Experts Scaling Laws"
started_at: "2026-02-25T14:00:00Z"
completed_at: "2026-02-25T15:30:00Z"
breadth: 4
depth: 2
total_queries: 12
queries_succeeded: 10
queries_failed: 2
corpus_files: 35
total_tokens_scraped: 147000
claude_code_uuid: SESSION_UUID
---

# Mixture of Experts Scaling Laws

## Summary

[Synthesized findings organized by theme...]

## Key Findings

1. Finding 1 (from [source](../corpus/2026-02-25-moe-scaling-arxiv.md))
2. Finding 2 (from [source](../corpus/2026-02-25-switch-transformer.md))

## Open Questions

- Question that couldn't be fully answered
- Area needing more research

## Sources

| #   | Title                 | Corpus File                                                                                               | Tokens |
| --- | --------------------- | --------------------------------------------------------------------------------------------------------- | ------ |
| 1   | Scaling MoE Models... | [corpus/2026-02-25-moe-scaling-arxiv-2401-12345.md](../corpus/2026-02-25-moe-scaling-arxiv-2401-12345.md) | 4200   |
| 2   | Switch Transformer... | [corpus/2026-02-25-switch-transformer-google.md](../corpus/2026-02-25-switch-transformer-google.md)       | 6100   |
| 3   | Expert Parallelism    | [corpus/2026-02-25-expert-parallelism-deepspeed.md](../corpus/2026-02-25-expert-parallelism-deepspeed.md) | 3800   |

## Failed Queries

- "MoE training stability RLHF" — timeout
- "expert routing load balance GPU memory" — no results

Source References

Every finding in the report should link to its source corpus file using relative paths. This lets any future Claude Code session:

1. Read the synthesized report for a quick overview 2. Drill into specific corpus files for full original content 3. Re-analyze raw sources with different questions or newer models

---

Initialization

When starting the first research session in a project:

mkdir -p docs/research/corpus docs/research/sessions
touch docs/research/corpus-index.jsonl

Add to .gitignore if raw corpus files would be too large:

# Uncomment if corpus files are too large for git
# docs/research/corpus/

By default, commit everything — corpus files are markdown and diff cleanly.

---

Consistency with Existing Patterns

Evolution Log

Convention: Reverse chronological order (newest on top, oldest at bottom). Prepend new entries. Refer to releases by date, not by version tag — semantic-release owns the version SSoT (see .releaserc.yml).

---

2026-05-27 (b): Antifragile reconciliation of the morning's URL-routing guard

Trigger: A live session later the same day invoked the skill on a chatgpt.com/share/* URL and hit a contradiction — the morning's URL-routing guard said "route AI chat shares out to Skill(gh-tools:research-archival), this skill cannot handle them," while Section 5's port-routing table explicitly listed Gemini/ChatGPT shares → Port 3003 (Needs JS rendering). The operator (Claude) had to make a judgment call mid-flow, chose Section 5, and port 3003 returned a 75 KB / 1,734-line scrape successfully. Section 5 was right; the guard was overcautious.

Root cause: When the URL-routing guard was introduced in the morning's patch to make AI-chat-share routing visible at the top of the templates section, it was framed as a hand-off ("Templates A–E are for research-grade source material, not AI chat transcripts") instead of an intent split. Both skills wrap the same Firecrawl backend; the difference is what happens to the bytes after they come back (raw file vs. frontmatter + GH issue + provenance). The original line-11 reference to gh-tools:research-archival was a _suggestion_, but the guard upgraded it to _exclusion_ without empirical justification.

Fix 1 — Intent-based routing: Replaced the URL-pattern hand-off table with an intent-decision table. Operator picks based on what output they want (read-only conversation text vs. full archival pipeline), not based on the URL string. Both rows are valid uses of the same backend.

Fix 2 — Documented the port-3003 → Caddy two-step: The skill's Section 5 example showed curl :3003/scrape?url=...&name=... as if it returned markdown directly. It does not. It returns JSON of the shape {"url": "<caddy-url>", "file": "<filename>"} — a pointer. The operator must then GET the Caddy URL to retrieve the actual markdown. Added the two-step bash snippet, plus a note that the JSON's url field embeds the legacy ZeroTier IP and should be reconstructed against the operator's preferred host base.

Fix 3 — Shell-quoting trap: Documented that python3 -c '... print(...)' inside command substitution leaves a trailing \n which becomes %0A in the URL-encoded payload and is silently rejected by the wrapper. Recommend print(..., end='').

Files modified:

• SKILL.md — replaced "URL-routing guard" section (now "Intent routing — AI chat share URLs") and Section 5 port-3003/3004 bash block.

Validation evidence: The triggering session's port-3003 invocation against https://chatgpt.com/share/6a168eb9-b118-83e8-8397-2a4ef1a93a5c returned 75,353 bytes / 1,734 lines of markdown via the Caddy two-step. Cannot be retroactively reproduced without re-scraping; the live trace from 2026-05-27T07:10:40Z is the audit record.

---

2026-05-27 (a): Three broken-instruction bugs from the prior MINOR release

Trigger: A diagnostic session caught — and the very next invocation of the skill demonstrated — three documented-but-unfixed bugs that survived the prior MINOR release:

1. /v1/health does not exist on this Firecrawl build. Probing returns HTTP 404 (Express HTML error page) which looks like service-down but isn't. 2. Bare littleblack hostname was labeled "Preferred" in the access table but doesn't resolve over HTTP on the m3max client (MagicDNS isn't pushing the search suffix to the system resolver; SSH works only because ~/.ssh/config hard-codes the FQDN). 3. Templates A–E had no entry-point guard against AI chat-share URLs.

Fix: Replaced all /v1/health references with GET / (returns 200 + Firecrawl banner). Demoted bare hostname to "Conditional" with dscacheutil/getent preflight; promoted Tailscale FQDN to "Preferred". Added URL-routing guard at the top of templates section. (The guard's framing was over-strict — see entry 2026-05-27 (b) above for the reconciliation.)

Files modified: SKILL.md.

---

2026-03-02: Merged firecrawl-self-hosted into this skill

What: Absorbed firecrawl-self-hosted skill — its SKILL.md condensed into self-hosted-operations.md reference, and its 3 reference docs (bootstrap-guide, best-practices, troubleshooting) moved here.

Why: The two skills covered the same service (self-hosted Firecrawl). Consolidation eliminates skill discovery friction — one skill for all Firecrawl concerns.

Files added:

• references/self-hosted-operations.md (new — condensed from old SKILL.md)
• references/self-hosted-bootstrap-guide.md (moved + renamed)
• references/self-hosted-best-practices.md (moved + renamed)
• references/self-hosted-troubleshooting.md (moved + renamed)

Files modified:

• SKILL.md — added self-hosted triggers, Section 5, updated references, removed scope boundary note

---

2026-02-26: Initial Evolution Log

Status: Skill is in use and maintained. Track improvements here.

Purpose

This evolution log tracks updates to the skill. Each entry should note:

• What changed (content, structure, tooling)
• Why it changed (bug fix, feature request, best practice)
• Files affected

How to Use

1. When updating SKILL.md or references, add an entry here with the date 2. Keep entries reverse-chronological (newest first) 3. Link to ADRs or GitHub issues when relevant 4. Reference specific line changes when helpful

---

Recursive Research Protocol

Step-by-step protocol for the iterative search → extract → recurse → synthesize pattern. Extracted from the working deep-research Pi extension.

---

Parameters

Query Budget Estimation

Total queries at each depth level: breadth queries. Each recursion halves breadth.

Depth 1: breadth queries (e.g., 4)
Depth 2: breadth * ceil(breadth/2) queries (e.g., 4 * 2 = 8)
Depth 3: breadth * ceil(breadth/2) * ceil(ceil(breadth/2)/2) queries

For breadth=4, depth=2: approximately 4 + 8 = 12 total search queries. For breadth=4, depth=3: approximately 4 + 8 + 8 = 20 total search queries.

---

Protocol Steps

Step 1: Health Check

Verify Firecrawl is reachable before starting. A failed health check saves minutes of wasted API calls.

try {
  await fetch("http://littleblack:3002/v1/health", {
    signal: AbortSignal.timeout(5_000),
  });
} catch {
  // Abort — see self-hosted-operations.md and troubleshooting.md references
}

If health check fails, do NOT proceed. Report the failure and suggest checking the Firecrawl deployment.

Step 2: Generate Search Queries

Given the research topic and any prior learnings, generate N search queries (N = breadth).

Input: Topic string + accumulated learnings array Output: Array of { query, researchGoal } objects

const queries = await generateSerpQueries(topic, breadth, priorLearnings);
// Returns: [{ query: "mixture of experts scaling", researchGoal: "understand scaling laws" }, ...]

The LLM generates diverse queries that avoid duplicating prior learnings. Each query has an explicit researchGoal used to focus follow-up recursion.

Step 3: Execute Searches

For each query, call Firecrawl /v1/search with concurrency control via p-limit.

import pLimit from "p-limit";

const limit = pLimit(concurrency); // default: 2

const results = await Promise.all(
  queries.map((q) =>
    limit(async () => {
      const searchResult = await firecrawlSearch(
        "http://littleblack:3002",
        q.query,
        { timeout: 15_000, limit: 5 },
      );
      return { query: q, data: searchResult.data ?? [] };
    }),
  ),
);

Step 4: Persist Raw Results

CRITICAL: Save each scraped page to docs/research/corpus/ BEFORE any LLM processing. This ensures raw content survives even if the session is interrupted.

For each search result page:

1. Generate filename: YYYY-MM-DD-{slug}.md 2. Write file with YAML frontmatter + raw markdown body 3. Append entry to docs/research/corpus-index.jsonl

See corpus-persistence-format.md for the exact file format.

Step 5: Extract Learnings

For each set of search results, pass the scraped content to an LLM to extract:

• Key learnings: Factual findings, data points, conclusions
• Follow-up questions: Gaps in understanding that warrant deeper investigation

// Trim each page to fit in context window
const trimmedContents = contents.map((c) => trimToTokenLimit(c, 25_000));

const extracted = await processSerpResult(
   query,
   trimmedContents,
   numLearnings: 3,    // Extract up to 3 learnings per result set
   numFollowUp: breadth / 2, // Generate follow-up questions for next depth
);
// Returns: { learnings: string[], followUpQuestions: string[] }

Step 6: Recurse

For each follow-up question, recurse with halved breadth and decremented depth.

const newBreadth = Math.ceil(breadth / 2);
const newDepth = depth - 1;

if (newDepth > 0) {
   const nextQuery = `Previous research goal: ${researchGoal}
Follow-up research directions: ${followUpQuestions.join("\n- ")}`;

   return researchLoop(nextQuery, newBreadth, newDepth, allLearnings, ...);
}

Why halve breadth: Deeper levels explore narrower sub-topics. Halving breadth prevents exponential query explosion while maintaining focus.

Step 7: Base Case

When depth = 0, return accumulated learnings without further recursion.

if (depth === 0) {
  return { learnings: allLearnings, visitedUrls: allUrls };
}

Step 8: Early Stopping

Stop recursion early when all new learnings duplicate prior ones (no new information being discovered):

const newLearnings = extracted.learnings.filter(
  (l) => !priorLearnings.some((p) => similarity(l, p) > 0.9),
);
if (newLearnings.length === 0) {
  // No new information — stop recursing this branch
  return { learnings: allLearnings, visitedUrls: allUrls };
}

Step 9: Synthesize Final Report

Pass all accumulated learnings to an LLM for a structured markdown report.

const report = await writeFinalReport(topic, allLearnings, visitedUrls);

The report should:

• Organize learnings by theme/subtopic
• Include a Sources section referencing raw corpus files by relative path
• Highlight areas of consensus and disagreement across sources
• Note gaps where information was unavailable

Step 10: Write Session Report

Save the synthesized report to docs/research/sessions/YYYY-MM-DD-{topic-slug}.md.

The session report includes a Sources table linking to raw corpus files:

## Sources

| #   | Title              | Corpus File                                                                           | Tokens |
| --- | ------------------ | ------------------------------------------------------------------------------------- | ------ |
| 1   | Scaling MoE...     | [corpus/2026-02-25-moe-scaling-arxiv.md](../corpus/2026-02-25-moe-scaling-arxiv.md)   | 4200   |
| 2   | Switch Transformer | [corpus/2026-02-25-switch-transformer.md](../corpus/2026-02-25-switch-transformer.md) | 6100   |

---

Handling Partial Failures

The protocol is designed to tolerate failures at every level:

Principle: At every level, partial results are returned rather than throwing. The queriesFailed array tracks what didn't work.

---

Deduplication

Results are deduplicated at the learning and URL level:

return {
  learnings: [...new Set(results.flatMap((r) => r.learnings))],
  visitedUrls: [...new Set(results.flatMap((r) => r.visitedUrls))],
};

The corpus index (corpus-index.jsonl) enables cross-session deduplication — check if a URL was already scraped before re-scraping.

---

Visualization

Topic: "mixture of experts scaling"
│
├─ Depth 1 (breadth=4)
│  ├─ Query 1: "MoE scaling laws" → 5 pages → 3 learnings
│  ├─ Query 2: "switch transformer efficiency" → 5 pages → 2 learnings
│  ├─ Query 3: "expert parallelism GPU" → 5 pages → 3 learnings
│  └─ Query 4: "MoE vs dense models" → 5 pages → 2 learnings
│     │
│     └─ Depth 2 (breadth=2, per follow-up from each Query)
│        ├─ Follow-up 1a: "MoE load balancing" → 5 pages → 2 learnings
│        ├─ Follow-up 1b: "expert dropout" → 5 pages → 1 learning
│        ├─ Follow-up 2a: "MoE inference cost" → 5 pages → 2 learnings
│        └─ ... (more follow-ups)
│
└─ Synthesize: 15+ learnings → Final Report
   └─ Corpus: 20-40 raw markdown files preserved

Firecrawl Best Practices (Empirically Verified)

1. Always Use restart: unless-stopped

Docker default is no restart policy. Containers WILL stop on SIGINT/SIGTERM and not recover.

Anti-pattern:

services:
  api:
    image: firecrawl/api
    # Missing restart policy = container dies and stays dead

Correct:

services:
  api:
    image: firecrawl/api
    restart: unless-stopped # Auto-restart on crash or signal

2. Use YAML Anchors for Consistency

Don't repeat restart: unless-stopped for each service. Use anchors:

x-common-service: &common-service
  restart: unless-stopped
  logging:
    driver: "json-file"
    options:
      max-size: "1G"
      max-file: "4"

services:
  api:
    <<: *common-service
    # ...

3. Verify After docker compose up

ALWAYS verify restart policies after docker compose up -d:

docker inspect --format "{{.Name}}: {{.HostConfig.RestartPolicy.Name}}" \
  $(docker ps -a --filter "name=firecrawl" -q)

4. Use systemd for Non-Docker Services

For Bun scripts and Caddy, use systemd with Restart=always:

[Service]
Restart=always
RestartSec=5

5. Monitor with Health Checks

Add periodic health check to catch silent failures:

# Add to crontab
*/5 * * * * curl -sf http://localhost:3002/health || systemctl --user restart firecrawl

Firecrawl Bootstrap: Fresh Installation

Prerequisites

• Debian/Ubuntu server with Docker
• Tailscale network membership (tailnet: terrylica.github)
• Domain or static IP (optional, for public access)

Step 1: Clone Repository

cd ~
git clone https://github.com/mendableai/firecrawl.git
cd firecrawl

Step 2: Configure docker-compose.yaml

CRITICAL: Add restart policy to prevent shutdown on signals:

x-common-service: &common-service
  networks:
    - backend
  restart: unless-stopped # <-- ADD THIS
  logging:
    driver: "json-file"
    options:
      max-size: "1G"
      max-file: "4"

Apply to all services using the anchor:

services:
  api:
    <<: *common-service
    # ...
  playwright-service:
    <<: *common-service
    # ...
  redis:
    <<: *common-service
    # ...
  rabbitmq:
    <<: *common-service
    # ...

Step 3: Environment Variables

Create .env from template:

cp .env.example .env

Minimal required settings:

# .env
NUM_WORKERS_PER_QUEUE=2
PORT=3002
HOST=0.0.0.0
REDIS_URL=redis://redis:6379
REDIS_RATE_LIMIT_URL=redis://redis:6379

Step 4: Start Services

docker compose up -d

Step 5: Verify Restart Policies

docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" \
  $(docker ps -a --filter "name=firecrawl" -q)

All should show unless-stopped.

Step 6: Optional - Scraper Wrapper

Create ~/firecrawl-scraper.ts:

import { serve } from "bun";
import { $ } from "bun";

const FIRECRAWL_API = "http://localhost:3002";
const OUTPUT_DIR = "/home/kab/firecrawl-output";

serve({
  port: 3003,
  async fetch(req) {
    const url = new URL(req.url);

    if (url.pathname === "/health") {
      return new Response("OK", { status: 200 });
    }

    if (url.pathname === "/scrape") {
      const targetUrl = url.searchParams.get("url");
      const name = url.searchParams.get("name") || "scraped";

      if (!targetUrl) {
        return Response.json(
          { error: "url parameter required" },
          { status: 400 },
        );
      }

      const response = await fetch(`${FIRECRAWL_API}/v1/scrape`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          url: targetUrl,
          formats: ["markdown"],
          waitFor: 5000,
        }),
      });

      const data = await response.json();
      const markdown = data?.data?.markdown;

      if (!markdown) {
        return Response.json(
          { error: "No markdown returned" },
          { status: 500 },
        );
      }

      const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
      const filename = `${name}-${timestamp}.md`;
      const filepath = `${OUTPUT_DIR}/${filename}`;

      await Bun.write(filepath, markdown);

      return Response.json({
        url: `http://littleblack:8080/${filename}`,
        file: filename,
      });
    }

    return new Response("Not Found", { status: 404 });
  },
});

Create systemd user service ~/.config/systemd/user/firecrawl-scraper.service:

[Unit]
Description=Firecrawl Scraper Wrapper
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/kab
ExecStart=/home/kab/.bun/bin/bun run firecrawl-scraper.ts
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

Enable:

systemctl --user daemon-reload
systemctl --user enable --now firecrawl-scraper

Step 7: Optional - Caddy File Server

Download Caddy from GitHub releases (latest version).

# Download and extract (check releases for current version)
wget https://github.com/caddyserver/caddy/releases/download/v<version>/caddy_<version>_linux_amd64.tar.gz  # SSoT-OK
tar xzf caddy_*.tar.gz
chmod +x caddy

Create systemd user service ~/.config/systemd/user/caddy-firecrawl.service:

[Unit]
Description=Caddy Firecrawl File Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/kab
ExecStart=/home/kab/caddy file-server --root /home/kab/firecrawl-output --listen :8080 --browse
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

Enable:

systemctl --user daemon-reload
systemctl --user enable --now caddy-firecrawl

Firecrawl Self-Hosted Operations

Deployment, health checks, recovery, and best practices for the self-hosted Firecrawl instance.

Host: littleblack (Tailscale: littleblack.tail0f299b.ts.net, legacy ZeroTier: 172.25.236.1). All 5 containers up 5+ weeks, stable. Source: <https://github.com/mendableai/firecrawl>

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                    littleblack (Tailscale)                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐      │
│  │   Client     │───▶│ Scraper      │───▶│ Firecrawl    │      │
│  │   (curl)     │    │ Wrapper :3003│    │ API :3002    │      │
│  └──────────────┘    └──────────────┘    └──────────────┘      │
│         │                   │                   │               │
│         │                   │                   ▼               │
│         │                   │            ┌──────────────┐       │
│         │                   │            │ Playwright   │       │
│         │                   │            │ Service      │       │
│         │                   │            └──────────────┘       │
│         │                   │                   │               │
│         │                   ▼                   ▼               │
│         │            ┌──────────────┐    ┌──────────────┐       │
│         │            │ Caddy :8080  │    │ Redis        │       │
│         │            │ (files)      │    │ RabbitMQ     │       │
│         ▼            └──────────────┘    └──────────────┘       │
│  ┌──────────────┐                                               │
│  │ Output URL   │◀── http://littleblack:8080/NAME-TS.md       │
│  └──────────────┘                                               │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Quick Reference

When to Use Which Port

Usage

Recommended: Wrapper Endpoint (port 3003)

curl "http://littleblack:3003/scrape?url=URL&name=NAME"

Returns:

{
  "url": "http://littleblack:8080/NAME-TIMESTAMP.md",
  "file": "NAME-TIMESTAMP.md"
}

Direct API (Advanced)

curl -s -X POST http://littleblack:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"URL","formats":["markdown"],"waitFor":5000}' \
  | jq -r '.data.markdown'

Health Checks

Quick Status

# All containers running?
ssh littleblack 'docker ps --filter "name=firecrawl" --format "{{.Names}}: {{.Status}}"'

# API responding?
ssh littleblack 'curl -s -o /dev/null -w "%{http_code}" http://localhost:3002/v1/scrape'
# Expected: 401 (no payload) or 200 (with payload)

# Wrapper responding?
curl -s -o /dev/null -w "%{http_code}" "http://littleblack:3003/health"

Detailed Status

# systemd services (services run under kab user, not yca SSH user)
ssh littleblack "sudo systemctl --user -M kab@ status firecrawl-scraper caddy-firecrawl"

# Docker container details
ssh littleblack 'docker ps -a --filter "name=firecrawl" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'

# Logs (live)
ssh littleblack "sudo journalctl --user -M kab@ -u firecrawl-scraper -u caddy-firecrawl -f"

Note: Firecrawl services run under the kab user on littleblack. The SSH user is yca. Always use sudo systemctl --user -M kab@ — plain systemctl --user targets the SSH user and sees no services.

Recovery Commands Cheatsheet

# Full restart (all services)
ssh littleblack 'cd ~/firecrawl && docker compose restart'
ssh littleblack 'sudo systemctl --user -M kab@ restart firecrawl-scraper caddy-firecrawl'

# Check everything
ssh littleblack 'docker ps --filter "name=firecrawl" && sudo systemctl --user -M kab@ status firecrawl-scraper caddy-firecrawl --no-pager'

# Logs (last 100 lines)
ssh littleblack 'docker logs firecrawl-api-1 --tail 100'
ssh littleblack 'sudo journalctl --user -M kab@ -u firecrawl-scraper --no-pager -n 100'

# Force recreate with new config
ssh littleblack 'cd ~/firecrawl && docker compose up -d --force-recreate'

# Verify restart policies
ssh littleblack 'docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'

Cloudflare Bypass (Port 3004)

For sites that block Playwright-based scraping (Cloudflare challenge pages), use the curl-impersonate bypass service:

curl "http://littleblack:3004/scrape-cf?url=URL&name=NAME"

This uses curl-impersonate to mimic a real browser TLS fingerprint, bypassing Cloudflare's bot detection. Use when port 3003 returns a Cloudflare challenge instead of page content.

Files Reference

Related Guides

• Self-Hosted Bootstrap Guide — 7-step fresh installation
• Self-Hosted Best Practices — Docker restart policies, health monitoring
• Self-Hosted Troubleshooting — Symptom-based diagnosis and recovery

External References

• Firecrawl Official Docs - API reference
• Docker Compose Restart - Policy options

Firecrawl Troubleshooting

Symptom: API Container Stopped

Root Cause: Docker restart policy was no (default). Container received SIGINT and didn't restart.

Diagnosis:

# Check container status
ssh littleblack 'docker ps -a --filter "name=firecrawl"'

# Check restart policy
ssh littleblack 'docker inspect --format "{{.Name}}: {{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'

Fix: Add restart: unless-stopped to ALL services in docker-compose.yaml:

# ~/firecrawl/docker-compose.yaml
x-common-service: &common-service
  networks:
    - backend
  restart: unless-stopped # CRITICAL: Add this line
  logging:
    driver: "json-file"
    options:
      max-size: "1G"
      max-file: "4"

services:
  playwright-service:
    <<: *common-service
    # ... rest of config

  api:
    <<: *common-service
    # ... rest of config

  redis:
    <<: *common-service
    # ... rest of config

  rabbitmq:
    <<: *common-service
    # ... rest of config

Apply Fix:

ssh littleblack 'cd ~/firecrawl && docker compose up -d --force-recreate'

Verify:

ssh littleblack 'docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'
# All should show: RestartPolicy=unless-stopped

Symptom: Scraper Wrapper Not Responding

Diagnosis:

ssh littleblack "systemctl --user status firecrawl-scraper"

Fix:

ssh littleblack "systemctl --user restart firecrawl-scraper"

Symptom: Caddy File Server Down

Diagnosis:

ssh littleblack "systemctl --user status caddy-firecrawl"
curl -I http://littleblack:8080/

Fix:

ssh littleblack "systemctl --user restart caddy-firecrawl"

Symptom: Tailscale Unreachable

Diagnosis:

# From local machine
tailscale ping littleblack

# Check Tailscale status
tailscale status

Fix: Re-authorize device in Tailscale admin console if needed.