Pdf Brain Ingest

PDF Brain Ingest v2 (ADR-0234)

Staged artifact-chain pipeline with durable NAS storage, nomic embeddings, and workload queue orchestration.

Pipeline v2 Architecture

PDF (source, immutable)
  → Stage 1: CONVERT — opendataloader-pdf → {docId}.md (NAS artifact)
  → Stage 2: CLASSIFY + SUMMARIZE — taxonomy + LLM summary → {docId}.meta.json (NAS artifact)
  → Stage 3: CHUNK — markdown-native headings, no overlap → {docId}.chunks.jsonl (NAS artifact)
  → Stage 4: INDEX — upsert to docs + docs_chunks_v2 (nomic-embed-text-v1.5, 768-dim)

Key properties:

• Durable: artifacts on NAS RAID5, survive reboots/crashes
• Resumable: each stage checks for existing artifacts, skips if present
• Recoverable: re-run any stage from existing artifacts without re-extracting
• Observable: OTEL event per stage per book

Artifacts dir: /Volumes/three-body/docs-artifacts/{docId}/

• {docId}.md — structured markdown extraction
• {docId}.meta.json — taxonomy, summary, metadata
• {docId}.chunks.jsonl — chunk records, one per line

Core Workflow

1) Preflight

joelclaw status
joelclaw docs status

Status now shows both v1 and v2 collection stats plus artifact availability.

2) Single File Ingest (v1 pipeline)

joelclaw docs add "/absolute/path/to/file.pdf"
joelclaw docs add "/absolute/path/to/file.pdf" --title "Title" --tags "tag1,tag2" --category programming

3) Single File v2 Reindex (artifact pipeline)

joelclaw docs reindex-v2 "/absolute/path/to/file.pdf"
joelclaw docs reindex-v2 "/absolute/path/to/file.pdf" --title "Title" --skip-existing

Fires docs/reindex-v2.requested → 4-stage artifact pipeline → NAS artifacts + docs_chunks_v2.

4) Batch Reindex (full library)

# Reindex all PDFs from NAS /Volumes/three-body/books/
joelclaw docs batch-reindex --skip-existing

# Reindex from existing Typesense docs collection
joelclaw docs batch-reindex --from-collection --skip-existing

Fires docs/reindex-batch.requested → scans NAS/collection → dispatches individual reindex-v2 events in batches of 10, concurrency 3.

--skip-existing skips books that already have all 3 artifacts on NAS (default true).

5) Monitor Progress

# Artifact count on NAS
ls /Volumes/three-body/docs-artifacts/ | wc -l

# v2 collection chunk count
joelclaw docs status

# OTEL events from pipeline
joelclaw otel search "docs.reindex" --hours 4
joelclaw o11y session system-bus --hours 4

# Individual run trace
joelclaw runs --count 10
joelclaw run <run-id>

6) Inspect Artifacts

# Read the extracted markdown
joelclaw docs markdown <doc-id>

# Read the summary + taxonomy metadata
joelclaw docs summary <doc-id>

# Or directly on NAS
cat /Volumes/three-body/docs-artifacts/<docId>/<docId>.md
cat /Volumes/three-body/docs-artifacts/<docId>/<docId>.meta.json | jq
wc -l /Volumes/three-body/docs-artifacts/<docId>/<docId>.chunks.jsonl

7) Retrieval from v2

# Search uses docs_chunks_v2 (nomic 768-dim) by default
joelclaw docs search "distributed consensus" --limit 8

# Context expansion
joelclaw docs context <chunk-id> --mode snippet-window
joelclaw docs context <chunk-id> --mode parent-section
joelclaw docs context <chunk-id> --mode section-neighborhood

8) Coverage Reconcile

joelclaw docs reconcile --sample 20

9) Recovery

If the batch stalls or books fail:

• Inngest retries each step automatically (default retry policy)
• --skip-existing means re-firing the batch only processes unfinished books
• Check OTEL for errors: joelclaw otel list --level error --hours 4
• Individual retry: joelclaw docs reindex-v2 "/path/to/failed.pdf"

Extraction Details

• Primary: opendataloader-pdf v2.0.0 (Java-based, #1 in benchmarks, 0.90 accuracy)
• Fallback: pypdf (basic text extraction if Java unavailable)
• Requires: Java 11+ (OpenJDK 25 installed on panda, PATH configured in worker start.sh)
• Speed: ~2.5s per book on M4 Pro

Embedding Model

• v2: nomic-embed-text via ollama (GPU-accelerated on M4 Pro, 768-dim, retrieval-tuned). Pre-computed at ingest time, stored as raw float[] vectors. ~150x faster than Typesense CPU auto-embed.
• v1: ts/all-MiniLM-L12-v2 — 384-dim, general-purpose, Typesense auto-embed (legacy, still in docs_chunks)
• Ollama runs on panda at localhost:11434. System-bus-worker (host process) embeds at ingest time.

Chunking Strategy (ADR-0234)

• Markdown-native heading detection (# markers, not heuristics)
• Recursive splitting within sections exceeding target tokens
• No overlap (arxiv R100-0 finding: 45% higher precision)
• Two-level hierarchy: section chunks + snippet sub-chunks
• heading_path derived from actual markdown heading levels
• Context inheritance: retrieval_text includes [DOC: title] [SUMMARY: ...] [PATH: heading > path] [CONCEPTS: ...]

Acquisition Pipeline (aa-book → ingest)

joelclaw send pipeline/book.download -d '{
  "query": "designing data-intensive applications",
  "format": "pdf",
  "reason": "library expansion"
}'

Downloads via aa-book → NAS backup → fires docs/ingest.requested for immediate processing.

Inngest Events

PDF-Brain Ingest Operator Guide (Joelclaw)

Date: 2026-02-23 Scope: Ingest new pdf/md files into the docs + docs_chunks memory pipeline on joelclaw.

0) aa-book End-to-End Acquisition (Recommended for books)

Queue acquisition + inference + download + ingest in one durable workflow:

joelclaw send pipeline/book.download -d '{
  "query": "designing data-intensive applications",
  "format": "pdf",
  "reason": "memory backfill"
}'

Current runtime behavior:

• aa-book download keeps the local file in outputDir for docs ingest.
• The workflow then attempts a non-fatal NAS backup to /volume1/home/joel/books/<year>/....
• docs/ingest.requested prefers the local filePath when present, so NAS outages do not block ingest.

Optional direct MD5 (skip search/inference):

joelclaw send pipeline/book.download -d '{
  "md5": "0123456789abcdef0123456789abcdef"
}'

1) Preflight

joelclaw status
joelclaw inngest status
joelclaw docs status

If worker registration looks stale:

joelclaw inngest sync-worker --restart

2) Ingest a Single File

Queue one file:

joelclaw docs add "/absolute/path/to/file.pdf"
joelclaw docs add "/absolute/path/to/file.md"

Optional metadata:

joelclaw docs add "/absolute/path/to/file.pdf" \
  --title "Readable Title" \
  --tags "manifest,catalog-fill" \
  --category programming

Notes:

• Use absolute paths.
• Supported file types are pdf, md, txt.
• Ingest writes document records into docs and hierarchical section/snippet chunks into docs_chunks.

3) Monitor Runs

joelclaw runs --count 20 --hours 1
joelclaw run <run-id>

Healthy run shape for docs-ingest:

• validate-file
• extract-text
• classify-taxonomy
• ensure-docs-collections
• upsert-document
• chunk-and-index
• emit-completed
• cleanup-text-artifact
• Finalization

4) Verify Index + Retrieval

joelclaw docs list --limit 20
joelclaw docs show <doc-id>
joelclaw docs search "your query"
joelclaw docs context <chunk-id> --mode snippet-window

5) Reconcile Coverage Against Manifest

Use dual coverage to avoid false missing churn:

joelclaw docs reconcile --sample 20

This reports:

• path_exact coverage: exact nas_path match.
• content_equivalent coverage: category-insensitive book path equivalence.

Interpretation:

• If content_equivalent > path_exact, missing churn is mostly path-variant noise.
• falseMissingChurn sample shows paths that look missing by exact path but are content-equivalent matches.

6) Path Alias Behavior (Important)

Docs now store both:

• nas_path (canonical)
• nas_paths[] (all known aliases for same doc id/content)

Result:

• Re-ingesting the same content via a different absolute path appends to nas_paths[].
• Canonical nas_path stays stable, preventing category/path flip-flop.

7) OTEL Checks

Core ingest telemetry:

joelclaw otel search "docs.file.validated" --hours 1
joelclaw otel search "docs.taxonomy.classified" --hours 1
joelclaw otel search "docs.chunks.indexed" --hours 1
joelclaw otel search "docs.path.aliases.updated" --hours 24

Timeout signal (taxonomy subprocess guard):

joelclaw otel search "docs.taxonomy.classify.timeout" --hours 24

8) Troubleshooting

Stuck running at validate step with NAS paths

Symptom in run trace:

• validate-file retries/fails with EINTR: interrupted system call, open '/Volumes/three-body/...

Find quickly:

joelclaw otel search "EINTR: interrupted system call" --hours 1

Mitigation:

• Requeue affected files once NAS path is stable.
• Avoid blasting high-concurrency retries on flaky mounts.
• Keep observing run queue and OTEL until the file validates cleanly.

Finalization/network anomalies

joelclaw run <run-id>
joelclaw logs worker --lines 120
joelclaw otel search "docs." --hours 1

9) Useful Maintenance Commands

joelclaw docs enrich <doc-id>
joelclaw docs reindex --doc <doc-id>
joelclaw docs reindex

Use reindex to normalize earlier runs after taxonomy/path logic changes.