Skill Template

Book SFT Pipeline

A complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.

When to Activate

Activate this skill when:

• Building fine-tuning datasets from literary works
• Creating author-voice or style-transfer models
• Preparing training data for Tinker or similar SFT platforms
• Designing text segmentation pipelines for long-form content
• Training small models (8B or less) on limited data

Core Concepts

The Three Pillars of Book SFT

1. Intelligent Segmentation Text chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.

2. Diverse Instruction Generation Use multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.

3. Style Over Content The goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.

Pipeline Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    ORCHESTRATOR AGENT                           │
│  Coordinates pipeline phases, manages state, handles failures   │
└──────────────────────┬──────────────────────────────────────────┘
                       │
       ┌───────────────┼───────────────┬───────────────┐
       ▼               ▼               ▼               ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│  EXTRACTION  │ │ SEGMENTATION │ │  INSTRUCTION │ │   DATASET    │
│    AGENT     │ │    AGENT     │ │    AGENT     │ │   BUILDER    │
│ ePub → Text  │ │ Text → Chunks│ │ Chunks →     │ │ Pairs →      │
│              │ │ 150-400 words│ │ Prompts      │ │ JSONL        │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
                       │
       ┌───────────────┴───────────────┐
       ▼                               ▼
┌──────────────┐               ┌──────────────┐
│   TRAINING   │               │  VALIDATION  │
│    AGENT     │               │    AGENT     │
│ LoRA on      │               │ AI detector  │
│ Tinker       │               │ Originality  │
└──────────────┘               └──────────────┘

Phase 1: Text Extraction

Critical Rules

1. Always source ePub over PDF - OCR errors become learned patterns 2. Use paragraph-level extraction - Extract from <p> tags to preserve breaks 3. Remove front/back matter - Copyright and TOC pollute the dataset

# Extract text from ePub paragraphs
from epub2 import EPub
from bs4 import BeautifulSoup

def extract_epub(path):
    book = EPub(path)
    chapters = []
    for item in book.flow:
        html = book.get_chapter(item.id)
        soup = BeautifulSoup(html, 'html.parser')
        paragraphs = [p.get_text().strip() for p in soup.find_all('p')]
        chapters.append('\n\n'.join(p for p in paragraphs if p))
    return '\n\n'.join(chapters)

Phase 2: Intelligent Segmentation

Smaller Chunks + Overlap

Smaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).

def segment(text, min_words=150, max_words=400):
    paragraphs = text.split('\n\n')
    chunks, buffer, buffer_words = [], [], 0
    
    for para in paragraphs:
        words = len(para.split())
        if buffer_words + words > max_words and buffer_words >= min_words:
            chunks.append('\n\n'.join(buffer))
            # Keep last paragraph for overlap
            buffer = [buffer[-1], para] if buffer else [para]
            buffer_words = sum(len(p.split()) for p in buffer)
        else:
            buffer.append(para)
            buffer_words += words
    
    if buffer:
        chunks.append('\n\n'.join(buffer))
    return chunks

Expected Results

For an 86,000-word book:

• Old method (250-650 words): ~150 chunks
• New method (150-400 + overlap): ~300 chunks
• With 2 variants per chunk: 600+ training examples

Phase 3: Diverse Instruction Generation

The Key Insight

Using a single prompt template causes memorization. Diverse templates teach the underlying style.

SYSTEM_PROMPTS = [
    "You are an expert creative writer capable of emulating specific literary styles.",
    "You are a literary writer with deep knowledge of classic prose styles.",
    "You are a creative writer skilled at emulating distinctive authorial voices.",
    "You write prose that captures the essence of modernist literature.",
    "You are a talented writer who can channel classic American authors.",
]

PROMPT_TEMPLATES = [
    "Write a passage in the style of {author}: {desc}",
    "Channel {author}'s voice to write about: {desc}",
    "In {author}'s distinctive prose style, describe: {desc}",
    "Write this scene as {author} would have: {desc}",
    "Using {author}'s repetitive technique, describe: {desc}",
    "Capture the rhythm of {author} in this passage: {desc}",
    "Write like {author}: {desc}",
    "In the voice of {author}, write: {desc}",
    "This is a literary exercise. Write like {author}: {desc}",
    "Can you write in {author}'s style? {desc}",
]

Instruction Generation

INSTRUCTION_PROMPT = """Describe what is happening in this excerpt in 2-3 sentences.
Focus on: characters present, actions, emotions, setting.
Do NOT quote the text directly.

Excerpt:
{text}
"""

# Use a fast, cheap LLM (e.g., Gemini Flash)
instruction = llm_call(INSTRUCTION_PROMPT.format(text=chunk))

Phase 4: Dataset Construction

Message Format

{
    "messages": [
        {"role": "system", "content": "You are an expert creative writer..."},
        {"role": "user", "content": "Write in the style of Author: Scene description..."},
        {"role": "assistant", "content": "The actual book text from chunk..."}
    ]
}

Multiple Variants Per Chunk

def build_examples(chunk, instruction, author, variants=2):
    examples = []
    for i in range(variants):
        system = SYSTEM_PROMPTS[i % len(SYSTEM_PROMPTS)]
        template = PROMPT_TEMPLATES[(chunk.id + i) % len(PROMPT_TEMPLATES)]
        user = template.format(author=author, desc=instruction)
        examples.append({"messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": user},
            {"role": "assistant", "content": chunk.text}
        ]})
    return examples

Phase 5: LoRA Training on Tinker

Configuration

CONFIG = {
    "model_name": "Qwen/Qwen3-8B-Base",  # Base, not instruct
    "lora_rank": 32,                      # 352MB adapter
    "learning_rate": 5e-4,                # Higher for LoRA
    "batch_size": 4,
    "epochs": 3,
}

Why Base Model?

Use base (pretrained) models, not instruction-tuned versions:

• Base models are more malleable for new styles
• Instruct models have patterns that resist overwriting
• Style is a low-level pattern that base models capture better

Training Loop

import tinker
from tinker import types

training_client = await service_client.create_lora_training_client_async(
    base_model="Qwen/Qwen3-8B-Base",
    rank=32
)

for epoch in range(3):
    for batch in batches:
        await training_client.forward_backward_async(batch, loss_fn="cross_entropy")
        await training_client.optim_step_async(types.AdamParams(learning_rate=5e-4))

result = await training_client.save_weights_for_sampler_async(name="final")

Phase 6: Validation

Modern Scenario Test

Test with scenarios that couldn't exist in the original book:

TEST_PROMPTS = [
    "Write about a barista making lattes",
    "Describe lovers communicating through text messages",
    "Write about someone anxious about climate change",
]

If the model applies style markers to modern scenarios, it learned style, not content.

Originality Verification

# Search training data for output phrases
grep "specific phrase from output" dataset.jsonl
# Should return: No matches

AI Detector Testing

Test outputs with GPTZero, Pangram, or ZeroGPT.

Known Issues and Solutions

Character Name Leakage

Symptom: Model uses original character names in new scenarios. Cause: Limited name diversity from one book. Solution: Train on multiple books or add synthetic examples.

Model Parrots Exact Phrases

Symptom: Outputs contain exact sentences from training data. Cause: Too few prompt variations or too many epochs. Solution: Use 15+ templates, limit to 3 epochs.

Fragmented Outputs

Symptom: Sentences feel incomplete. Cause: Poor segmentation breaking mid-thought. Solution: Always break at paragraph boundaries.

Guidelines

1. Always source ePub over PDF - OCR errors become learned patterns 2. Never break mid-sentence - Boundaries must be grammatically complete 3. Use diverse prompts - 15+ templates, 5+ system prompts 4. Use base models - Not instruct versions 5. Use smaller chunks - 150-400 words for more examples 6. Reserve test set - 50 examples minimum 7. Test on modern scenarios - Proves style transfer vs memorization 8. Verify originality - Grep training data for output phrases

Expected Results

Cost Estimate

Integration with Context Engineering Skills

This example applies several skills from the Agent Skills for Context Engineering collection:

project-development

The pipeline follows the staged, idempotent architecture pattern:

• Acquire: Extract text from ePub
• Prepare: Segment into training chunks
• Process: Generate synthetic instructions
• Parse: Build message format
• Render: Output Tinker-compatible JSONL
• Train: LoRA fine-tuning
• Validate: Modern scenario testing

Each phase is resumable and produces intermediate artifacts for debugging.

context-compression

Segmentation is a form of context compression for training. The core insight from context-compression applies: information density matters more than information quantity. Smaller, coherent chunks (150-400 words) produce better style transfer than larger, diluted chunks.

The two-tier strategy mirrors context compression evaluation:

• Tier 1: Fast, deterministic compression
• Tier 2: LLM-assisted for edge cases

multi-agent-patterns

The pipeline uses the supervisor/orchestrator pattern:

• Orchestrator coordinates phases and manages state
• Specialized agents (Extraction, Segmentation, Instruction, Builder) have isolated contexts
• Each agent receives only the information needed for its task

This matches the principle that sub-agents exist primarily to isolate context rather than simulate roles.

evaluation

Validation follows the end-state evaluation pattern:

• Functional testing: Does output match expected style markers?
• Originality verification: Is content genuinely generated?
• External validation: AI detector scores

The "modern scenario" test is a form of out-of-distribution evaluation that proves generalization.

context-fundamentals

Prompt diversity prevents attention collapse on single patterns. When training with identical prompt structures, the model memorizes the instruction-response mapping. Diverse templates force attention across the style patterns themselves.

References

Internal references:

• Segmentation Strategies - Text chunking patterns
• Tinker Format Specification - Datum structure
• Tinker API Documentation - Full API reference

Related skills from Agent Skills for Context Engineering:

• project-development - Pipeline architecture patterns
• context-compression - Compression strategies
• multi-agent-patterns - Agent coordination
• evaluation - Evaluation frameworks
• context-fundamentals - Attention and information density

External resources:

• Research Paper - Chakrabarty et al. 2025
• Dataset on Hugging Face
• Gertrude Stein Case Study - Complete working example

---

Skill Metadata

Created: 2025-12-26 Last Updated: 2025-12-28 Author: Muratcan Koylan Version: 2.0.0 Standalone: Yes (separate from main context-engineering collection)

AGENTS.md

Workspace memory for agents collaborating on this repository. Keep entries durable and broadly applicable; one-off task state belongs in chat or in a run thread, not here.

Learned User Preferences

• For autonomous research and repo-improvement work in this workspace, prefer proceeding through concrete research loops, subagents, validation, and edits when the scope is clear rather than asking broad process questions.
• Avoid stale regex or keyword-list heuristics in skills and scripts; prefer mechanism-level criteria, rubrics, and evidence-backed validation.
• Never push to GitHub or merge a PR without explicit user approval. Preparing branches, commits, and PRs is permitted only when the user has approved that specific action.
• Tone is technical CTO: direct, no marketing language, no exclamation marks, no emojis, no em dashes. State trade-offs and complexity upfront.
• When the scope spans multiple architectural decisions or irreversible changes, propose a plan first instead of executing.
• For benchmarks and evaluation work, hold to research-paper-grade methodology (statistical discipline, bias mitigation, ablations, reproducibility) over speed. Don't rush.

Learned Workspace Facts

• This repo is an autonomous research-to-skill organization. External AI research is curated through rubrics and distilled into context-engineering and harness-engineering skill updates.
• researcher/ is repo-native and file-based so agents can resume, audit, validate, and prepare PR-ready skill changes without a hosted scheduler.
• Per-run state lives in researcher/runs/<run-id>/run-state.json with explicit transitions (initialized -> retrieved -> evaluated -> proposed -> novelty_checked -> validated -> pr_ready -> closed). Use research_loop.py subcommands to advance state, never hand-edit run-state.json.
• Repo health (validate_repo.py) and per-run readiness (validate_run.py) are different questions. CI runs validate_platform_compat.py --require-reference-validator, validate_repo.py --strict, skill_health.py --strict --no-history, run_benchmarks.py, and check_activation_cases.py on every PR via .github/workflows/validate.yml.
• The mechanism registry (researcher/mechanisms/registry.jsonl) is the encyclopedia backbone. Promotion is gated by research_loop.py promote-mechanisms with a recorded reviewer; ledgers live under researcher/mechanisms/ledgers/.
• Claim provenance for numeric or volatile claims lives in researcher/claims/index.jsonl. Add an entry for any new benchmark or volatility-sensitive claim.
• The corpus index (researcher/corpus/index.json) is the machine-readable map of skills, activation scenarios, mechanisms, and claims. Update it when adding or restructuring skills.
• The continuous loop (researcher/scripts/loop_*.py) runs from launchd via researcher/orchestration/launchd/. It never invokes paid LLMs; HTTP retrieval is stdlib-only with a 1.5 MB cap and a 30-second timeout.
• Runtime state is not committed: researcher/queue/*.jsonl, researcher/queue/.locks/, researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}, and researcher/runs/*/ are gitignored. The seed run 20260515-035228-executable-autonomous-research-frameworks is the only committed run; it is closed as reference-only and serves as a worked example.
• The current prepared release version is 2.3.1 across .claude-plugin/marketplace.json, .plugin/plugin.json, and root SKILL.md. There are 15 skills (latent-briefing covers KV cache sharing between agents).
• Detailed lessons from building the researcher OS live in researcher/insights/auto-research-experiment.md (engineering rationale) and researcher/insights/how-we-built-this.md (project narrative and sharing templates); read both before extending the harness or writing release-facing prose.
• Benchmarks are staged in researcher/benchmarks/: Stage 0 deterministic harness (shipped), Stage 1 per-skill health via researcher/scripts/skill_health.py (shipped; output researcher/reports/skill-health.json is gitignored), Stage 2 router (shipped; results in researcher/benchmarks/router/results-published/), Stage 3 effectiveness (scaffolded, one task built), Stage 4 composition (future). researcher/benchmarks/PLAN.md is the methodology source of truth.
• Current corpus-hardening baseline: 16 accepted mechanisms, 12 provenance-tracked claims, 19 activation cases, and strict skill-health score 0.9117 with 0 flagged skills. Do not describe a skill improvement as complete unless the prose, mechanism registry, claim index, corpus index, activation fixtures, and validators all agree.
• Benchmark execution uses the Cursor SDK runner at researcher/benchmarks/sdk-runner/ (TypeScript, @cursor/sdk 1.0.13). The runner supports --concurrency N, --no-resume, per-run progress logging, format-failure retry, and worst-case retry-aware cost forecasting; default behavior is to resume by skipping plan items that already have result files. Result artifacts under researcher/benchmarks/{router,effectiveness}/results/ and history JSONLs (router-history.jsonl, effectiveness-history.jsonl) are gitignored.
• Published Stage 2 router-benchmark results: researcher/benchmarks/router/results-published/2026-05-15.md (baseline), researcher/benchmarks/router/results-published/2026-05-15-v2.md (post-rewrite with delta-vs-baseline table), and researcher/benchmarks/router/results-published/2026-05-19.md (post-corpus-hardening validation: 600/600 usable records, 0 format failures, top-1 Gemini 0.920 / Composer 0.913 / GPT-5.5 0.913 / Claude Opus 4.7 0.840). Headline finding: targeted description rewrites moved context-fundamentals top-1 by +23.4pp and project-development top-1 to 1.000; corpus-wide hardening did not cause broad routing collapse.

Repository Operating Defaults

• Deterministic checks before model judges. Always run validate_platform_compat.py --require-reference-validator and validate_repo.py --strict before claiming a skill-format or packaging change is complete.
• Adversarial benchmarks before declaring the harness safe. Add a scenario when a new failure mode is discovered.
• Append-only ledgers for accepted and rejected mechanisms so future agents do not rediscover failed paths.
• Atomic writes (tempfile + os.replace) and fcntl locks for any shared file the loop touches.
• Live execution is the highest-signal validation for orchestration code; smoke-test changes against the actual loop before declaring them safe.
• Cursor SDK is the only paid-API surface allowed for benchmarks. Privacy Mode required, apiKey passed explicitly per call, never settingSources: "all" in benchmarks (use [] for control, ["project"] with a curated .cursor/skills/ for ablation). Cost gates (--max-runs, --max-budget-usd, or --dry-run) must be set before any SDK call.
• Description quality is measurable. When changing skill activation descriptions, re-run the router benchmark with the same seed and fixture and publish the delta. Aggregate accuracy is a misleading unit; per-skill effect sizes and the confusion matrix are the right view.
• A skill is a multi-surface artifact. Changing the frontmatter description is not enough; the SKILL.md body When to Activate and Integration sections must be audited the same day so the body does not contradict the description that routed the agent to it. The router benchmark only sees descriptions (settingSources: []) and cannot catch body inconsistencies; only Stage 3 effectiveness benchmarks (which actually load skill bodies) measure body-alignment impact.
• Any runner that calls a paid API in a loop must have three features before execution: bounded parallelism via --concurrency, resume capability via results-folder scan, and per-run progress logging that surfaces stalls inside one call's duration.
• API keys provided in chat should be considered exposed; rotate immediately after use. Runner enforces this via apiKeyFingerprint() which only logs the last 4 characters.

Changelog

All notable changes to this project are documented here. Versions follow semantic versioning where practical, with skill content treated as data.

[2.3.1] - 2026-06-29

Fixed

• Cross-platform YAML frontmatter: 11 of 15 published skills used unquoted description values containing colons, which strict YAML parsers (Cursor, Claude Code, Codex, Agent Skills validators) reject. All skill descriptions now use YAML-safe quoting; memory-systems no longer uses a folded block scalar that repo validators misread as ">".
• Shared frontmatter parser: added researcher/scripts/skill_frontmatter.py and wired it into validate_repo.py, skill_health.py, check_activation_cases.py, and compare_skill_revisions.py. CI installs pyyaml for deterministic strict parsing. The parser handles LF/CRLF line endings, UTF-8 BOM, quoted scalars, and folded block scalars, and rejects empty, too-short, or indicator-only descriptions.
• Unit tests: added researcher/scripts/tests/test_skill_frontmatter.py (19 tests) covering parser edge cases, a strict-YAML regression guard for the unquoted-colon bug, format/parse round-trips, and a corpus integration test asserting every published skill parses clean. Wired into CI before the strict repo gate.
• Example skills: quoted the description fields in examples/digital-brain-skill/SKILL.md and examples/book-sft-pipeline/SKILL.md, which had the same unquoted-colon YAML hazard developers would copy.
• Manifest validation: validate_repo.py --strict now checks that .plugin/plugin.json and .claude-plugin/marketplace.json name the same bundled plugin and that Open Plugins skills discovery resolves to the same 15 published skills as the repository.
• Platform compatibility gate: added researcher/scripts/validate_platform_compat.py, which validates the published skills with the upstream agentskills CLI from skills-ref, checks Open Plugins and Claude marketplace discovery parity, and simulates directory-copy installs for .cursor/skills, .claude/skills, .codex/skills, and .agents/skills.
• Platform install docs: README now documents directory-based install paths for Cursor (.cursor/skills/), Claude Code (.claude/skills/), and Codex (.codex/skills/) instead of the broken flat-file .md pattern.
• Open Plugins discovery: .plugin/plugin.json now declares "skills": "./skills/". The repository does not commit .agents/skills or .cursor/skills symlinks because symlinks are fragile on Windows and in plugin packaging.

[2.3.0] - 2026-05-15

First release with measured benchmark results across four frontier models, closing the loop from "we wrote skill descriptions" to "we proved they route correctly."

Added

Stage 2 router benchmark, executed end-to-end

• 600 of 600 runs completed across composer-2, claude-opus-4-7, gpt-5.5, gemini-3.1-pro at 3 replications per (prompt, model). Initial v2.2.0 baseline at researcher/benchmarks/router/results-published/2026-05-15.md (566 of 600 due to the v1 runner dying); updated run after the description fixes at researcher/benchmarks/router/results-published/2026-05-15-v2.md with full delta-vs-baseline table.
• 50 ground-truth router prompts at researcher/benchmarks/router/prompts.jsonl covering positive controls, adversarial boundary pairs, combined-skill prompts, and negative controls.
• researcher/scripts/render_router_report.py with --baseline flag for delta reports.
• researcher/benchmarks/router/results-published/README.md explains the committed-summary vs gitignored-raw split.

Hardened SDK runner (researcher/benchmarks/sdk-runner/src/)

• Resume: scans the destination directory on startup and skips plan items that already have a per-run JSON. A killed sweep can be picked up exactly where it stopped; no wasted credits, no duplicate runs.
• Bounded parallelism: --concurrency N runs N agent calls simultaneously. Cuts the 600-run sweep from ~60 minutes (sequential) to ~15 minutes (concurrency=4) with identical correctness.
• Per-run progress logging: every completed run prints [N/total] model prompt rep=R status durationMs T1 ETA=duration. The v1 sweep silently stalled at 566 of 600 with no signal; the v2 sweep would have surfaced the cause immediately.
• Format-failure retry: transient empty or unparsable SDK responses are retried once before being recorded as format failures. This was added after the May 19 sweep produced transient blank outputs that succeeded on rerun.
• runConcurrently helper in common.ts, reusable by future runners.

Skill description rewrites (data-driven)

Targeted at the two routing failures the v2.2.0 baseline benchmark surfaced:

• context-fundamentals: rewrote to be unambiguously about conceptual foundations and explicitly route operational work to the specialized skills. Top-1 rate went from 0.255 to 0.489 (+23.4pp).
• project-development: tightened with explicit cross-references to tool-design. Top-1 rate went from 0.750 to 1.000 (now perfect routing).
• tool-design: tightened with explicit cross-references to project-development. Top-1 rate went from 0.729 to 0.807 (+7.8pp).

Skill body alignment with new descriptions

The router benchmark only sees frontmatter description because settingSources: [] excludes the SKILL.md body. The first description rewrite pass left the bodies (When to Activate, Practical Guidance, Integration) claiming the broader pre-rewrite scope, which would have steered the agent toward operational work the moment the skill actually activated in production. Aligned the bodies in a follow-up pass:

• context-fundamentals body: rewrote When to Activate to list conceptual triggers and explicit do-not-activate routing; removed the operational File-System-Based Access and Context Budgeting practical-guidance sections (owned by filesystem-context and context-optimization respectively); replaced with conceptual application advice plus a reading-order recommendation for new contributors; rewrote Integration as an explicit routing map across all 14 sibling skills. Internal version bump 2.0.0 -> 2.1.0.
• tool-design body: rewrote When to Activate to anchor on the unit of work (single tool or tool catalog) and listed adjacent decisions owned by project-development, multi-agent-patterns, context-optimization; rewrote Integration with explicit routing reasons. Internal version bump 2.0.0 -> 2.1.0.
• project-development body: rewrote When to Activate to anchor on project shape and pipeline; listed adjacent decisions owned by tool-design, context-optimization, multi-agent-patterns, harness-engineering; rewrote Integration with explicit routing reasons. Internal version bump 1.1.0 -> 1.2.0.

The body changes do not affect router-benchmark numbers (the router sees only descriptions) but they do affect what the agent loads when these skills activate. Stage 3 (effectiveness benchmark, which loads full bodies) is the right place to measure the impact of this alignment.

Corpus-wide skill hardening pass

After the targeted three-skill body alignment, every published skill was audited against the same standard: explicit ownership boundary, Do not activate routing, executable practical guidance, examples, gotchas, integration boundaries, mechanism coverage, claim provenance, and activation fixtures.

• Updated all 15 skill bodies with scoped improvements, including structural fixes for bdi-mental-states and hosted-agents, stronger negative routing across older skills, clearer examples for context failure modes, and claim-backed wording for volatile benchmark statements.
• Expanded researcher/mechanisms/registry.jsonl from 5 to 16 accepted mechanisms so every published skill owns at least one machine-readable behavior pattern.
• Expanded claim provenance from 6 to 12 records and replaced vague run-summary sources with concrete repo paths for BrowseComp, RULER/lost-in-middle, compression, d0, Latent Briefing, memory, and tool-output claims.
• Expanded activation regression coverage from 14 to 19 cases so every skill has deterministic routing coverage, including bdi-mental-states, context-degradation, hosted-agents, latent-briefing, and multi-agent-patterns.
• Tightened validate_repo.py --strict so Core Concepts, Practical Guidance, Examples, References, and explicit non-activation boundaries are now enforced rather than optional.
• Updated template/SKILL.md with the new corpus-wide standard: body/frontmatter alignment, mechanism registration, executable guidance, and claim-* provenance for volatile claims.
• Re-ran the no-API gates after the pass: validate_repo.py --strict passed with 0 errors / 0 warnings; skill_health.py --strict --no-history improved from corpus score 0.8111 / 2 flagged skills to 0.9117 / 0 flagged skills; check_activation_cases.py passed 19/19; run_benchmarks.py passed 3 checks and 7 adversarial scenarios.
• Re-ran the paid Stage 2 router benchmark after the corpus-wide pass: 600/600 usable records, 0 format failures after retrying transient format failures, published at researcher/benchmarks/router/results-published/2026-05-19.md. Per-model top-1: Gemini 0.920, Composer 0.913, GPT-5.5 0.913, Claude Opus 4.7 0.840. Remaining failures are concentrated in known ambiguous/negative-control prompts (p046, p048) and the context-fundamentals catch-all boundary.

Eleven new boundary regression cases

researcher/fixtures/activation-cases.jsonl grew from 8 to 19 cases. The first six new cases target specific confusions observed in the v2.2.0 baseline:

• activation-fundamentals-vs-degradation, activation-fundamentals-onboarding, activation-fundamentals-vs-optimization
• activation-tool-vs-project-structured-output, activation-tool-individual-tool, activation-tool-consolidation

These act as a tripwire so any future description change is held accountable.

The corpus-wide pass added five more cases for previously uncovered skills:

• activation-bdi-vs-memory
• activation-degradation-poisoning
• activation-hosted-vs-harness
• activation-latent-briefing-vs-memory
• activation-multi-agent-topology

Stage 1 skill health (still no API cost)

• researcher/scripts/skill_health.py: per-skill structural scoring. Initial corpus baseline: 0.8111 aggregate, 2 of 15 skills flagged (bdi-mental-states for missing required section, hosted-agents for multiple structural issues). After the corpus-wide hardening pass: 0.9117 aggregate, 0 flagged skills.
• Output at researcher/reports/skill-health.json (gitignored runtime artifact) + optional append to skill-health-history.jsonl.

Changed

• Version bumped 2.2.0 -> 2.3.0 across .claude-plugin/marketplace.json, .plugin/plugin.json, root SKILL.md.
• researcher/benchmarks/PLAN.md status table reflects Stage 0/1/2 shipped, Stage 3/4 still scaffolded.

Headline measured results

Per-model top-1 accuracy (baseline -> new descriptions, 600-run sweep at seed=1, fixture sha 8f974d9):

Per-skill top-1 rate change for the three skills targeted by description rewrites:

Format compliance: 99.5% (3 failures, all Gemini). Latency: Gemini ~9.1s median, others 3.3-4.2s. Total sweep cost approximately 7.20 USD against the 15 USD budget cap.

Honest scope caveats

• context-fundamentals improved a lot but is still the weakest skill (0.489 top-1). Remaining failures route to project-development for generic onboarding prompts. One more description pass may push it past 0.75.
• Two prompts remain at 0.00 across all models: p046 (Python reformatting, negative control) and p048 (evaluate KV compaction, genuinely ambiguous). Should be re-labeled or removed from positive-routing tests.
• advanced-evaluation looks regressed (-18.3pp) but is largely an artifact of the v2.2.0 baseline missing 11 attempts when the runner died at 566/600. Absolute correct count: 48 baseline -> 47 new.
• Stage 3 (real agent tasks with and without skills loaded) is still scaffolded but not executed; that is the next investment.
• No LLM-judge adapter for the run state machine. No automated source discovery beyond manual seed.

[2.2.0] - 2026-05-15

Added

Researcher operating system

• Mechanism registry (researcher/mechanisms/registry.jsonl) seeded with five accepted mechanisms (locked-editable-surfaces, durable-research-thread, deterministic-first-validation, structured-novelty-gate, pairwise-skill-revision).
• Mechanism ledgers (researcher/mechanisms/ledgers/accepted.jsonl, rejected.jsonl) for append-only promotion events.
• Claim provenance (researcher/claims/index.jsonl) for six volatile or benchmark-backed claims across evaluation, multi-agent-patterns, context-optimization, memory-systems, advanced-evaluation, and harness-engineering.
• Corpus index (researcher/corpus/index.json) mapping skills to activation scenarios, mechanism IDs, and claim IDs.
• Activation regression fixtures (researcher/fixtures/activation-cases.jsonl) covering high-risk skill-boundary pairs.
• Adversarial benchmark harness (researcher/benchmarks/scenarios/adversarial.jsonl + goldens) with seven scenarios that try to game the loop.
• Benchmark history (researcher/reports/benchmark-history.jsonl) for longitudinal trend tracking.
• Pairwise revision rubric and script (researcher/rubrics/pairwise-skill-revision.md, researcher/scripts/compare_skill_revisions.py).
• Run state machine in run-state.json with explicit transitions: initialized -> retrieved -> evaluated -> proposed -> novelty_checked -> validated -> pr_ready -> closed.

Continuous loop

• Queue infrastructure (researcher/queue/): inbox, parked, done, quarantine.
• Orchestration config (researcher/orchestration/config.json) with daily/active/parked/failure budgets.
• Discovery feeder (researcher/scripts/loop_discover.py) reading from researcher/discovery/manual-seed.jsonl.
• Loop step orchestrator (researcher/scripts/loop_step.py) that reaps closed runs, pulls from inbox, retrieves via stdlib urllib, and parks at human-review gates.
• Daily ops (researcher/scripts/loop_daily.py) running validators, benchmarks, activation cases, and writing dated snapshots.
• Status dashboard (researcher/scripts/loop_status.py) plus parked-review surface.
• launchd service definitions (researcher/orchestration/launchd/) with install/uninstall scripts and per-script wrappers.
• Continuous operation runbook (researcher/runbooks/continuous-operation.md).

Scripts

• researcher/scripts/validate_run.py: per-run publish readiness, skips closed runs.
• researcher/scripts/research_loop.py subcommands: retrieve, evaluate, propose, novelty, validate-run, pr-ready, close, promote-mechanisms.
• researcher/scripts/check_activation_cases.py: deterministic activation regression checker.
• researcher/scripts/run_benchmarks.py: runs deterministic gates, scenarios, optional history recording.
• researcher/scripts/loop_common.py: shared atomic-write helpers and fcntl locks.

CI

• .github/workflows/validate.yml runs validate_repo.py --strict, run_benchmarks.py, check_activation_cases.py, and Python compile checks on every push and PR.

Changed

• Skill activation surface refactored from keyword triggers to task-boundary descriptions in frontmatter and README. Affected: all 14 published skills plus the example skills in examples/.
• validate_repo.py hardened: duplicate JSON keys, exact doc sync, rubric IDs, run artifacts, registry schema, evidence paths, partial-retrieval approvals, root-level raw provenance, claims schema, corpus index consistency, activation cases, benchmark scenarios.
• novelty_check.py now compares mechanism registry overlap as the primary signal, with corpus overlap secondary.
• Mechanism registry evidence may now reference claim IDs from researcher/claims/index.jsonl in addition to URLs and repo paths.

Hardened

• All queue mutations use atomic temp-file os.replace and fcntl exclusive locks scoped per queue family.
• read_jsonl is tolerant: malformed lines quarantine to researcher/reports/jsonl-quarantine/ rather than crashing the loop.
• fetch_url allows only http(s):// and re-checks scheme after redirect.
• Closed runs are automatically reaped from parked.jsonl and recorded in done.jsonl.
• Inbox lock now held through init_run so concurrent loop_step invocations cannot exceed budgets.
• URL deduplication normalizes case before hashing.

Repository policy

• Active research runs under researcher/runs/*/ are runtime state and not committed. The seed run 20260515-035228-executable-autonomous-research-frameworks is kept as a worked-example fixture.
• Runtime queue and report files (researcher/queue/*.jsonl, researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}, researcher/queue/.locks/) are gitignored.

Out of scope for 2.2.0

• LLM-judge adapters for advancing retrieved -> evaluated automatically.
• Automated source discovery beyond the manual seed file (Parallel deep research and web search adapters are placeholders behind config toggles).
• Log rotation; benchmark history pruning.

[2.1.0] - 2026-05-14

Added

• harness-engineering skill: locked/editable surface model, durable threads, novelty gates, rollback, human approval boundaries.
• researcher/ directory v1: source registry, content/skill/harness rubrics, source-evaluation JSON template, skill-proposal template, autonomous research loop runbook, PR readiness runbook.

[2.0.0] - earlier

Baseline corpus of 13 skills distributed as a single Claude Code plugin.

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Agent Skills for Context Engineering: an open collection of 15 Agent Skills teaching context engineering and harness engineering principles for production AI agent systems. Skills are platform-agnostic (Claude Code, Cursor, Codex/OpenAI Agent Skills, GitHub Copilot, any Open Plugins-conformant tool). v2.3.1 ships a file-based researcher operating system with deterministic gates, cross-platform Agent Skills validation, and a continuous loop.

Context engineering is the discipline of curating everything that enters a model's context window (system prompts, tool definitions, retrieved documents, message history, tool outputs) to maximize signal within limited attention budget.

Repository Structure

• skills/ - 15 skill directories, each containing a SKILL.md with YAML frontmatter (name, description) and optional references/ and scripts/ subdirectories
• examples/ - 5 complete demonstration projects (digital-brain-skill, llm-as-judge-skills, book-sft-pipeline, x-to-book-system, interleaved-thinking)
• docs/ - Research materials and reference documentation
• researcher/ - File-based research-to-skill operating system: rubrics, mechanism registry, claim provenance, corpus index, run state machine, adversarial benchmarks, continuous loop, launchd service definitions
• template/SKILL.md - Canonical skill template (use when creating new skills)
• SKILL.md (root) - Collection-level metadata and skill map
• .claude-plugin/marketplace.json - Claude Code marketplace manifest (single bundled plugin, v2.3.1)
• .plugin/plugin.json - Open Plugins format manifest (v2.3.1)

Build & Test Commands

No top-level build system. Repo-level gates and per-project tooling below.

Top-level deterministic gates (run on every PR via CI)

python3 -m unittest researcher.scripts.tests.test_skill_frontmatter # parser and strict-YAML regression tests
python3 researcher/scripts/validate_platform_compat.py --require-reference-validator # Agent Skills reference validator + Cursor/Claude/Codex install-layout simulation
python3 researcher/scripts/validate_repo.py --strict       # corpus structure, manifests, rubric math, mechanism registry, claims, corpus index, activation cases, benchmark scenarios, run artifacts
python3 researcher/scripts/skill_health.py --strict --no-history  # deterministic skill-body quality gate
python3 researcher/scripts/run_benchmarks.py               # adversarial benchmark harness + repo + activation gates
python3 researcher/scripts/check_activation_cases.py       # skill-boundary regression fixtures

Per-run readiness (active runs only)

python3 researcher/scripts/validate_run.py --run-dir researcher/runs/<run-id>

Continuous loop (manual or launchd)

python3 researcher/scripts/loop_discover.py
python3 researcher/scripts/loop_step.py --allow-fetch
python3 researcher/scripts/loop_daily.py
python3 researcher/scripts/loop_status.py

researcher/orchestration/launchd/install.sh    # macOS daemon
researcher/orchestration/launchd/uninstall.sh

Example projects

examples/llm-as-judge-skills (TypeScript, Node >= 18)

cd examples/llm-as-judge-skills
npm install
npm run build        # tsc
npm test             # vitest (19 tests)
npm run lint         # eslint
npm run format       # prettier
npm run typecheck    # tsc --noEmit

examples/interleaved-thinking (Python >= 3.10)

cd examples/interleaved-thinking
pip install -e ".[dev]"
pytest               # pytest + pytest-asyncio
ruff check .         # linting (100 char line length)

examples/digital-brain-skill (Node.js)

cd examples/digital-brain-skill
npm run setup
npm run weekly-review
npm run content-ideas
npm run stale-contacts

Skill Authoring Rules

When creating or editing skills:

1. SKILL.md must stay under 500 lines: move detailed content to references/ directory 2. YAML frontmatter is required: must include name and description fields 3. Folder naming: lowercase with hyphens (e.g., context-fundamentals) 4. Write in third person: descriptions are injected into system prompts; inconsistent POV causes discovery issues 5. Platform-agnostic: no vendor-locked examples or platform-specific tool names without abstraction 6. Token-conscious: challenge each paragraph and assume an advanced audience 7. Body standard: include When to Activate, Core Concepts, Practical Guidance, Examples, Guidelines, Gotchas, Integration, and References 8. Explicit boundaries: every When to Activate section needs positive triggers plus a Do not activate block routing adjacent work to the right skill 9. Include a Gotchas section: experience-derived failure modes are the highest-signal content in any skill 10. Update root README.md when adding new skills 11. Update marketplace/plugin manifests when adding skills (.claude-plugin/marketplace.json, .plugin/plugin.json) 12. Update the corpus index (researcher/corpus/index.json) to map the new skill to activation scenarios, mechanism IDs, and claim IDs 13. Update mechanisms and claims: add registry entries for reusable behavior changes and claim-* provenance for numeric, benchmark, volatile, or vendor-performance claims 14. Run `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `check_activation_cases.py`, and `run_benchmarks.py` before committing skill changes

Researcher OS Rules

When working through the researcher operating system:

1. Initialize runs via `research_loop.py init`: it creates run-state.json, queue entry, thread log, source evaluation scaffold, and mechanism proposal template 2. Advance state explicitly: use retrieve, evaluate, propose, novelty, validate-run, pr-ready, close subcommands; do not edit run-state.json by hand 3. Promote mechanisms only after run readiness: research_loop.py promote-mechanisms requires --reviewed-by and a passing run-readiness check 4. Add claim provenance to researcher/claims/index.jsonl for any numeric, benchmark, or volatile claim added to a skill 5. Never invoke paid LLMs from the continuous loop: HTTP retrieval is stdlib-only, judge adapters are explicitly out of scope until budget-gated 6. Never commit runtime queue/report files: .gitignore covers researcher/queue/*.jsonl, researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}, and researcher/runs/*/ except the seed run

Plugin Architecture

All 15 skills are distributed as a single plugin (context-engineering) in the marketplace manifest. This avoids cache duplication: Claude Code caches each plugin's source directory separately, so multiple plugins pointing to source: "./" would each cache a full copy of the repo.

Progressive disclosure pattern: only skill names/descriptions load at startup; full content loads on activation.

Key Design Principles

• Context quality over quantity: attention scarcity and lost-in-middle behavior mean more context is not always better
• Sub-agents isolate context: they exist to manage attention budget, not simulate org roles
• Skills reference each other: use plain text skill names (not links) in Integration sections to avoid cross-directory reference issues
• Examples use Python pseudocode: conceptual demonstrations that work across environments, not production-ready implementations
• Deterministic first, model-judged second: structure, schema, rubric math, manifest sync, retrieval status, and registry shape must pass before any LLM judge is invoked
• Human-controlled merge: agents may prepare PRs and pass gates, but push and merge always require explicit human approval

Contributing to Agent Skills for Context Engineering

Thank you for your interest in contributing to this collection of Agent Skills for Context Engineering. This document provides guidelines and instructions for contributing.

How to Contribute

Reporting Issues

If you find errors, unclear explanations, or missing topics, please open an issue with:

• A clear description of the problem
• The skill and section where the issue was found
• Suggested improvements if you have them

Submitting Changes

For substantive changes, please:

1. Fork the repository 2. Create a feature branch for your changes 3. Make changes following the skill template structure 4. Ensure SKILL.md files remain under 500 lines 5. Add references or scripts as appropriate 6. Submit a pull request with a clear description of changes

Adding New Skills

When adding new skills:

1. Use the template in template/SKILL.md 2. Follow naming conventions (lowercase with hyphens) 3. Include both SKILL.md and appropriate references/scripts 4. Update the root README.md to include the new skill 5. Update root SKILL.md and manifests when publishing the skill. New published skills require an explicit .claude-plugin/marketplace.json skill path. .plugin/plugin.json normally only needs version or description changes because Open Plugins discovers ./skills/. 6. Update researcher/corpus/index.json with the new skill's name, activation scenarios, mechanism IDs, and claim IDs 7. Add at least one entry to researcher/fixtures/activation-cases.jsonl; include rejected or adjacent skills when the boundary is easy to confuse 8. Ensure content is platform-agnostic (works across Cursor, Claude Code, etc.) 9. Run the unit tests and deterministic gates before opening a PR:

• python3 -m pip install -r requirements-dev.txt
• python3 -m unittest researcher.scripts.tests.test_skill_frontmatter
• python3 researcher/scripts/validate_platform_compat.py --require-reference-validator
• python3 researcher/scripts/validate_repo.py --strict
• python3 researcher/scripts/skill_health.py --strict --no-history
• python3 researcher/scripts/check_activation_cases.py
• python3 researcher/scripts/run_benchmarks.py

Researcher Operating System Contributions

The repository ships with a file-based research-to-skill operating system in researcher/. Contributions that introduce skill changes derived from external research should flow through it.

Run lifecycle

initialized -> retrieved -> evaluated -> proposed -> novelty_checked -> validated -> pr_ready -> closed

Use researcher/scripts/research_loop.py subcommands rather than editing run-state.json by hand. Each transition appends to the run's thread log and updates the state machine atomically.

Mechanism promotion

New behavior changes proposed for the corpus go through researcher/mechanisms/registry.jsonl. The promotion path is gated:

1. Author the proposal in the run's proposals/mechanism-proposal.jsonl. 2. Pass validate_run.py --run-dir <run>. 3. Run research_loop.py promote-mechanisms --run-dir <run> --reviewed-by <handle>. This appends to the registry and to researcher/mechanisms/ledgers/accepted.jsonl.

Rejected mechanisms append to ledgers/rejected.jsonl so future agents do not rediscover them.

Claim provenance

Any numeric, benchmark, or volatile claim added to a published skill should also receive an entry in researcher/claims/index.jsonl with claim_id, owning_skill, section, source_url, retrieved_at, evidence_strength, volatility, and last_reviewed. The validator checks ownership and source paths.

Parked runs

Runs that hit human-review gates land in researcher/queue/parked.jsonl and the dashboard at researcher/reports/parked-review.md. Reviewers should:

1. Read researcher/runs/<run-id>/THREAD.md and sources/evidence/. 2. Complete the next required step (retrieve, evaluate, propose, novelty, validate-run, or pr-ready). 3. Close the run with research_loop.py close --status accepted|rejected|reference-only|abandoned --reason <text> --reviewed-by <handle>.

The continuous loop will reap closed runs into researcher/queue/done.jsonl on the next iteration.

Runtime state is not committed

researcher/runs/*/ (except the seed run), researcher/queue/*.jsonl, and researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md} are gitignored. PRs should not introduce new committed runs; bug fixtures belong in researcher/fixtures/ instead.

Skill Structure Requirements

Each skill must include:

• YAML frontmatter with name and description fields. Quote description values that contain colons (:) so strict YAML parsers used by Cursor, Claude Code, and Codex can load the skill. Run python3 researcher/scripts/validate_repo.py --strict before opening a PR.
• ## When to Activate with positive triggers and an explicit Do not activate boundary for adjacent skills
• ## Core Concepts focused on behavior-changing mechanisms, not generic background
• ## Practical Guidance with an executable workflow, checklist, decision table, or operating rule
• ## Examples with at least one worked artifact, before/after, or boundary example
• ## Guidelines, ## Gotchas, ## Integration, and ## References
• Integration notes that explain routing and composition boundaries, not only topical relationships

Any numeric, benchmark, volatile, or vendor-performance claim in a published skill must either reference a claim-* ID from researcher/claims/index.jsonl or be softened and moved to dated reference material. Any reusable behavior pattern should be represented in researcher/mechanisms/registry.jsonl and linked from researcher/corpus/index.json.

Optional additions:

• references/ directory with additional documentation
• scripts/ directory with executable examples
• Multiple markdown files for complex skills

Content Guidelines

Writing Style

• Be direct and precise
• Use technical terminology appropriately
• Include specific guidance, not vague recommendations
• Provide concrete examples
• Point out complexity and trade-offs

Avoiding Platform Specificity

Skills should work across agent platforms. Avoid:

• Platform-specific tool names without abstraction
• Vendor-locked examples
• Features specific to one agent product

Keeping Skills Focused

Each skill should have a single focus. If a topic grows too large, consider splitting into multiple skills with clear dependencies.

Code of Conduct

This project follows a professional, technical collaboration model. Be respectful of different perspectives and focus on improving the collective knowledge base.

Questions

For questions about contributing, please open an issue for discussion.

Engineering Production-Grade LLM Agents: A Technical Deep Dive The shift from prompt engineering to context engineering represents the most significant paradigm change in building LLM agents. As Anthropic's research articulates, the challenge isn't writing better prompts—it's curating "the smallest possible set of high-signal tokens that maximize the likelihood of desired outcomes." Inkeepanthropic This report synthesizes technical findings from major AI labs and framework developers on multi-agent architectures, context management, attention degradation, and agent reliability patterns. Multi-agent architectures: From orchestrators to swarms Production multi-agent systems have converged on three dominant patterns, each with distinct tradeoffs. Orchestrator-worker (supervisor) patterns place a central agent in control, delegating to specialists and synthesizing results. LangGraph's benchmarks found this architecture initially performed 50% worse than optimized versions due to the "telephone game" problem—supervisors paraphrasing sub-agent responses incorrectly. The fix: implementing a forward_message tool allowing sub-agents to pass responses directly to users. langchainLangChain Swarm architectures, pioneered by OpenAI's experimental Swarm framework, enable peer-to-peer handoffs where any agent transfers control to any other. LangGraph benchmarks show swarms slightly outperform supervisors because sub-agents respond directly to users, eliminating translation errors. langchainLangChain The core abstraction is elegantly simple: pythondef transfer_to_agent_b(): return agent_b # Handoff via function return

agent_a = Agent( name="Agent A", functions=[transfer_to_agent_b] ) Hierarchical patterns, implemented in CrewAI's Process.hierarchical mode, create management trees where managers decompose goals and delegate to subordinates. Activewizards This mirrors organizational structures and works well for complex, multi-stage tasks. The critical insight from Manus AI's production experience: sub-agents exist primarily to isolate context, not to anthropomorphize role division. Rlancemartin Context isolation prevents KV-cache penalties and avoids context confusion between specialized tasks. Context coordination and the file system as memory How agents share context determines both performance and cost. Manus AI identified KV-cache hit rate as the single most important production metric— Manusthe difference between $0.30/MTok (cached) and $3/MTok (uncached) for Claude Sonnet, a 10× cost differential. manus Three context-sharing patterns emerge from production systems: PatternMechanismUse CaseFull context delegationPlanner shares entire context with sub-agentComplex tasks requiring complete understandingInstruction passingPlanner creates instructions via function callSimple, well-defined subtasksFile system memoryAgents read/write to persistent storageUnlimited size, agent-operable context Claude Code exemplifies file-system-as-memory: rather than stuffing context windows, agents use grep, head, and tail to navigate codebases, storing query results and analyzing large databases without loading full data. AnthropicRlancemartin This "just-in-time" context loading maintains small active context while enabling access to arbitrarily large information. anthropic Manus AI's context engineering principles offer production-tested guidance: use append-only context (never modify previous actions), employ logit masking instead of tool removal to constrain actions, and keep errors in context for implicit belief updates rather than hiding failures. manusManus KV-cache optimization: From PagedAttention to prefix caching The KV-cache stores Key and Value tensors computed during inference, growing linearly with sequence length. Neptune.ai For LLaMA-2 13B, this means approximately 1MB per token per sequence—a 4K context consumes ~4GB, comparable to the model itself. Rohan-paul PagedAttention, introduced by vLLM, revolutionized memory efficiency by applying OS-inspired virtual memory concepts. Medium Instead of pre-allocating contiguous memory, it partitions KV cache into fixed-size blocks (typically 16 tokens), mapping logical blocks to non-contiguous physical memory via block tables. Results: 2-4× throughput improvement arXiv with up to 96% reduction in memory waste. Medium Prefix caching (Automatic Prefix Caching) reuses KV blocks across requests sharing identical prefixes, using hash-based block matching: hash(parent_hash, block_tokens, extra_hashes). Anthropic reports up to 90% cost savings and 85% latency reduction with prefix caching on Claude. Advanced quantization pushes efficiency further. SKVQ achieves 1M token context on 80GB GPUs using 2-bit keys and 1.5-bit values with only <5% accuracy drop. Emergent Mind Layer-Condensed KV caches only top layers for 26× throughput. Emergent Mind RazorAttention identifies "retrieval heads" that need full caches versus those that can use buffers, achieving 40-60% memory reduction. Emergent Mind Context rot: The hidden performance cliff Despite claims of 100K+ token context windows, empirical research reveals significant performance degradation—a phenomenon researchers call context rot. anthropic The "lost in the middle" effect, documented by Liu et al. (TACL 2024), shows a U-shaped performance curve: accuracy drops 10-40% when relevant information sits in the middle of context versus beginning or end. arXivACL Anthology The RULER benchmark delivers a sobering finding: only half of models claiming 32K+ context maintain satisfactory performance at 32K tokens. arXivOpenReview GPT-4 showed the least degradation (15.4 points from 4K to 128K), while most models dropped 30+ points. Medium Near-perfect scores on simple needle-in-haystack tests don't translate to real long-context understanding— trychromaRULER's multi-hop tracing, aggregation, and question-answering tasks expose the gap. arXivOpenReview Chroma's 2025 research across 18 LLMs identified critical patterns: trychroma

Distractor effect: Even a single irrelevant document reduces performance; multiple distractors compound degradation Needle-question similarity: Lower similarity pairs show faster degradation with context length trychroma Counterintuitive haystack structure: Shuffled (incoherent) haystacks produce better performance than logically coherent ones trychroma Model-specific behaviors: Claude shows lowest hallucination rates but high abstention under ambiguity; GPT shows highest hallucination rates with confident-but-incorrect responses trychroma

Four failure modes in production contexts Beyond simple degradation, long-running agents encounter distinct context failure patterns that require different mitigations: Context poisoning occurs when hallucinations or errors enter context and compound through repeated reference. Feluda As Drew Breunig documents, if an agent's "goals" section becomes poisoned, it develops nonsensical strategies that take "very long time to undo." Drew Breunig Symptoms include degraded output quality, tool misalignment, and hallucinations treated as facts. Context distraction emerges when context grows so long that models over-focus on context at the expense of training knowledge. The Gemini 2.5 technical report notes: "While Gemini 2.5 Pro supports 1M+ token context, making effective use of it for agents presents a new research frontier." Drew Breunig Context confusion arises when irrelevant information influences responses. As one practitioner observed: "If you put something in the context, the model has to pay attention to it. It may be irrelevant information or needless tool definitions, but the model will take it into account." Drew Breunig Context clash develops when accumulated information directly conflicts, documented by Microsoft and Salesforce research showing that sharding information across multiple prompts creates conflicting contexts that derail reasoning. Drew Breunig Mitigation strategies that work Effective context management employs four strategies, formalized by LangChain as the "four-bucket" approach: StrategyImplementationExampleWriteSave context outside windowScratchpads, memory stores, file systemSelectPull relevant context inRAG, memory retrieval, tool selectionCompressReduce tokens preserving infoSummarization, observation maskingIsolateSplit context across agentsSub-agents, sandboxes, state schemas Observation masking deserves special attention: replacing old tool outputs with fixed masks like "Previous X lines elided for brevity" often matches or exceeds LLM summarization performance while adding zero token overhead (versus 5-7% for summarization). Research shows observations comprise 83.9% of tokens in typical agent trajectories—masking offers significant efficiency gains. Architectural approaches include Core Context Aware (CCA) Attention, a plug-and-play module achieving 5.7× faster inference at 64K tokens, arXiv and Google's Chain of Agents (CoA), which breaks inputs into chunks processed by worker agents sequentially, reducing time complexity from n² to nk. Google Research Tool design for agent ergonomics Tools are contracts between deterministic systems and non-deterministic agents—design matters critically. anthropic Anthropic's guidance emphasizes minimizing functional overlap: "If a human can't definitively say which tool to use, an AI agent can't either." Anthropic The consolidation principle transforms API design: Instead ofImplementlist_users, list_events, create_eventschedule_event (finds availability + schedules)read_logssearch_logs (returns relevant lines with context)get_customer_by_id, list_transactions, list_notesget_customer_context (compiles all relevant info) Tool descriptions require engineering. Poor descriptions like "Search the database" with cryptic parameter names force agents to guess. Optimized descriptions include usage context ("Use this when the user asks about company policies"), examples ("Example: 'vacation policy remote employees'"), and defaults ("Start with 3-5 for most queries"). Response format options offer significant token savings: implementing a response_format parameter with DETAILED (full JSON, 206 tokens) versus CONCISE (essential info only, 72 tokens) cuts context consumption by 65% when full metadata isn't needed. Reasoning patterns and their measured impact ReAct (Reasoning + Acting) interleaves thinking with tool use: "Thought 1: [reasoning] → Action 1: [tool call] → Observation 1: [result]". Prompt Engineering Guide Performance gains are substantial: +34% absolute success rate on ALFWorld, +10% on WebShop versus imitation learning. React-lm However, 2024 research reveals brittleness—40-90% of generated thoughts lead to invalid actions depending on the model. arXiv Tree of Thoughts (ToT) explores multiple reasoning paths simultaneously. On Game of 24, performance jumps from 4% (Chain-of-Thought) to 74% with GPT-4 using ToT. KDnuggets The approach works by generating multiple candidates at each reasoning step, having the LLM self-evaluate progress, and using tree search (BFS/DFS) for exploration. Dynamic few-shot selection consistently outperforms static examples. LangChain benchmarks show Claude 3 Sonnet jumping from 16% to 52% accuracy with just 3 semantically similar examples—often matching or exceeding 13 static examples. The key is semantic similarity: retrieve examples similar to the current query rather than maintaining fixed lists. Hallucination prevention in agentic contexts Agentic settings amplify hallucination risk since errors compound across tool calls. A critical MIT survey finding: "No prior work demonstrates successful self-correction with feedback from prompted LLMs, except for tasks exceptionally suited for self-correction." What does work for self-correction:

External tool feedback: Code execution results, API verification, calculator outputs Retrieval grounding: Web search for fact verification Fine-tuned correction models: Models specifically trained for correction tasks

RAG-based grounding can decrease hallucination by 60-80% according to industry surveys. Implementation requires explicit constraints: "Answer based ONLY on the provided context. If the context doesn't contain relevant information, respond: 'I cannot find information about this in the provided documents.'" The Chain-of-Verification (CoVe) pattern generates verification questions about claims, answers them independently, compares answers with initial claims, and revises based on inconsistencies. ProCo framework achieves +6.8 EM on QA and +14.1% on arithmetic through systematic condition verification. Evaluation methods for production agents Anthropic's multi-agent evaluation approach uses a structured rubric: factual accuracy (claims match sources), citation accuracy (cited sources match claims), completeness (all aspects covered), source quality (primary versus secondary), and tool efficiency (reasonable usage). Anthropic Key benchmarks reveal capability gaps: BenchmarkFindingRULEROnly 50% of 32K+ models maintain performance at 32K tokens arXiv∞Bench"Existing long-context LLMs require significant advancements for 100K+"LongBench v2Best model achieves 50.1% accuracy; humans achieve 53.7% Longbench2τ-benchTests single/multi-agent cognitive architectures on real-world scenarios The methodology: start with small samples (~20 queries), use LLM-as-judge for scalable evaluation, supplement with human evaluation to catch automation misses, and focus on end-state evaluation for agents that mutate state. Anthropic Conclusion Building production LLM agents requires treating context as the central engineering concern rather than an afterthought. The research converges on several principles: Context quality trumps context length—despite 1M+ token windows, effective performance often degrades past 32K-256K tokens depending on task complexity. Use just-in-time context loading, observation masking, and sub-agent isolation to maintain signal quality. Multi-agent architecture selection depends on coordination needs: swarms for peer-to-peer handoffs with direct user interaction, supervisors for integrating diverse sub-agents with minimal assumptions, hierarchical patterns for complex decomposition tasks. Tool design directly impacts agent capability. Consolidate overlapping tools, return contextual information in error messages, implement response format options, and namespace clearly. anthropic Poor tool descriptions create failure modes no amount of prompt engineering can fix. Verification requires external grounding. Self-correction without external feedback doesn't work reliably. RAG, tool execution results, and multi-agent verification architectures provide the grounding necessary for production reliability. The field is rapidly evolving—KV-cache optimization, attention architectures, and evaluation methods continue advancing. Engineers building agents should monitor production metrics (especially KV-cache hit rates and token efficiency), implement compaction triggers at 80% of effective context limits, and design systems assuming context will degrade rather than hoping it won't.

Evaluating Context Compression for AI Agents By Factory Research - December 16, 2025 - 10 minute read -

Share

Engineering

Research

New

We built an evaluation framework to measure how much context different compression strategies preserve. After testing three approaches on real-world, long-running agent sessions spanning debugging, code review, and feature implementation, we found that structured summarization retains more useful information than alternatives from OpenAI and Anthropic.

Table of Contents

01 The problem

02 Measuring context quality

03 Three approaches to compression

04 A concrete example

05 How the LLM judge works

06 Results

07 What we learned

08 Methodology details

09 Appendix: LLM Judge Prompts and Rubrics

Tasteful abstract illustration evocative of memory and blurriness When an AI agent helps you work through a complex task across hundreds of messages, what happens when it runs out of memory? The answer determines whether your agent continues productively or starts asking "wait, what were we trying to do again?"

We built an evaluation framework to measure how much context different compression strategies preserve. After testing three approaches on real-world, long-running agent sessions (debugging, PR review, feature implementation, CI troubleshooting, data science, ML research), we found that structured summarization retains more useful information than alternative methods from OpenAI and Anthropic, without sacrificing compression efficiency.

Bar chart comparing quality scores by dimension across Factory, OpenAI, and Anthropic This post walks through the problem, our methodology, concrete examples of how different approaches perform, and what the results mean for building reliable AI agents.

The problem Long-running agent sessions can generate millions of tokens of conversation history. That far exceeds what any model can hold in working memory.

The naive solution is aggressive compression: squeeze everything into the smallest possible summary. But this increases the chance your agent forgets which files it modified or what approach it already tried. It is likely to waste tokens re-reading files and re-exploring dead ends.

The right optimization target is not tokens per request. It is tokens per task.

Measuring context quality Traditional metrics like ROUGE or embedding similarity do not tell you whether an agent can continue working effectively after compression. A summary might score high on lexical overlap while missing the one file path the agent needs to continue.

We designed a probe-based evaluation that directly measures functional quality. The idea is simple: after compression, ask the agent questions that require remembering specific details from the truncated history. If the compression preserved the right information, the agent answers correctly. If not, it guesses or hallucinates.

We use four probe types:

Probe type What it tests Example question Recall Factual retention "What was the original error message?" Artifact File tracking "Which files have we modified? Describe what changed in each." Continuation Task planning "What should we do next?" Decision Reasoning chain "We discussed options for the Redis issue. What did we decide?" Recall probes test whether specific facts survive compression. Artifact probes test whether the agent knows what files it touched. Continuation probes test whether the agent can pick up where it left off. Decision probes test whether the reasoning behind past choices is preserved.

We grade responses using an LLM judge (GPT-5.2) across six dimensions:

Dimension What it measures Accuracy Are technical details correct? File paths, function names, errors Context awareness Does the response reflect current conversation state? Artifact trail Does the agent know which files were read or modified? Completeness Does the response address all parts of the question? Continuity Can work continue without re-fetching information? Instruction following Does the response follow the probe format? Each dimension is scored 0-5 using detailed rubrics. The rubrics specify what constitutes a 0 ("Completely fails"), 3 ("Adequately meets with minor issues"), and 5 ("Excellently meets with no issues") for each criterion.

Why these dimensions matter for software development These dimensions were chosen specifically because they capture what goes wrong when coding agents lose context:

Artifact trail is critical because coding agents need to know which files they have touched. Without this, an agent might re-read files it already examined, make conflicting edits, or lose track of test results. A ChatGPT conversation can afford to forget earlier topics; a coding agent that forgets it modified auth.controller.ts will produce inconsistent work.

Continuity directly impacts token efficiency. When an agent cannot continue from where it left off, it re-fetches files and re-explores approaches it already tried. This wastes tokens and time, turning a single-pass task into an expensive multi-pass one.

Context awareness matters because coding sessions have state. The agent needs to know not just facts from the past, but the current state of the task: what has been tried, what failed, what is left to do. Generic summarization often captures "what happened" while losing "where we are."

Accuracy is non-negotiable for code. A wrong file path or misremembered function name leads to failed edits or hallucinated solutions. Unlike conversational AI where approximate recall is acceptable, coding agents need precise technical details.

Completeness ensures the agent addresses all parts of a multi-part request. When a user asks to "fix the bug and add tests," a complete response handles both. Incomplete responses force follow-up prompts and waste tokens on re-establishing context.

Instruction following verifies the agent respects constraints and formats. When asked to "only modify the auth module" or "output as JSON," the agent must comply. This dimension catches cases where compression preserved facts but lost the user's requirements.

Three approaches to compression We compared three production-ready compression strategies.

Factory maintains a structured, persistent summary with explicit sections for different information types: session intent, file modifications, decisions made, and next steps. When compression triggers, only the newly-truncated span is summarized and merged with the existing summary. We call this anchored iterative summarization.

The key insight is that structure forces preservation. By dedicating sections to specific information types, the summary cannot silently drop file paths or skip over decisions. Each section acts as a checklist: the summarizer must populate it or explicitly leave it empty. This prevents the gradual information loss that occurs with freeform summarization.

OpenAI offers the /responses/compact endpoint, which produces opaque, compressed representations optimized for reconstruction fidelity. It achieves the highest compression ratios (99.3%) but sacrifices interpretability. You cannot read the compressed output to verify what was preserved.

Anthropic provides built-in context compression through the Claude SDK. It produces detailed, structured summaries (typically 7-12k characters) with sections for analysis, files, pending tasks, and current state. The key difference from Factory is the update mechanism: Anthropic regenerates the full summary on each compression, while Factory's anchored approach incrementally merges new information into a persistent summary. This affects both consistency across compressions and which details survive repeated compression cycles.

A concrete example Consider a debugging session where a user reported a 401 error on the /api/auth/login endpoint despite valid credentials. Over 178 messages and 89,000 tokens, the assistant:

Read the authentication controller and identified JWT token generation Examined the middleware chain and found a misconfigured CORS policy Discovered the session store was using an expired Redis connection Fixed the Redis connection pooling configuration Added retry logic for transient connection failures Ran the test suite (14 tests passing, 2 failing) Fixed the remaining test failures related to mock setup Files examined: auth.controller.ts, middleware/cors.ts, config/redis.ts, services/session.service.ts, tests/auth.test.ts

After compression, we ask: "Before we continue, can you remind me what the original error was that started this debugging session? Please be specific about the error code and endpoint."

Here is how each method responded:

Factory response (scored 4.8/5):

"The original error was a 401 Unauthorized response from the /api/auth/login endpoint. Users were receiving this error even when providing valid username/password combinations. The root cause turned out to be a stale Redis connection in the session store - the JWT was being generated correctly, but the session could not be persisted, causing subsequent authenticated requests to fail."

Anthropic response (scored 3.9/5):

"The session started with debugging a login failure - specifically a 401 error on the authentication endpoint. We traced the issue through the auth flow and found it was related to session storage rather than the credential validation itself."

OpenAI response (scored 3.2/5):

"We were debugging an authentication issue. The login was failing for users. We looked at several files and found some configuration problems that needed to be fixed."

The Factory response names the exact endpoint (/api/auth/login), the error code (401), and the root cause (Redis session store). The Anthropic response gets the error code and general cause but loses the endpoint path. The OpenAI response loses almost all technical detail.

This pattern repeated across probe types. On artifact probes ("Which files have we modified?"), Factory scored 3.6 while OpenAI scored 2.8. Factory's summary explicitly lists files in a dedicated section. OpenAI's compression discards file paths as low-entropy content.

How the LLM judge works We use GPT-5.2 as an LLM judge, following the methodology established by Zheng et al. (2023) in their MT-Bench paper. Their work showed that GPT-4 achieves over 80% agreement with human preferences, matching the agreement level among humans themselves.

The judge receives the probe question, the model's response, the compacted conversation context, and (when available) ground truth. It then scores each rubric criterion with explicit reasoning.

Here is an abbreviated example of judge output for the Factory response above:

{ "criterionResults": [ { "criterionId": "accuracy_factual", "score": 5, "reasoning": "Response correctly identifies the 401 error, the specific endpoint (/api/auth/login), and the root cause (Redis connection issue)." }, { "criterionId": "accuracy_technical", "score": 5, "reasoning": "Technical details are accurate - JWT generation, session persistence, and the causal chain are correctly described." }, { "criterionId": "context_artifact_state", "score": 4, "reasoning": "Response demonstrates awareness of the debugging journey but does not enumerate all files examined." }, { "criterionId": "completeness_coverage", "score": 5, "reasoning": "Fully addresses the probe question with the error code, endpoint, symptom, and root cause." } ], "aggregateScore": 4.8 }

The judge does not know which compression method produced the response. It evaluates purely on response quality against the rubric.

Results We evaluated all three methods on over 36,000 messages from production sessions spanning PR review, testing, bug fixes, feature implementation, and refactoring. For each compression point, we generated four probe responses per method and graded them across six dimensions.

Method Overall Accuracy Context Artifact Complete Continuity Instruction Factory 3.70 4.04 4.01 2.45 4.44 3.80 4.99 Anthropic 3.44 3.74 3.56 2.33 4.37 3.67 4.95 OpenAI 3.35 3.43 3.64 2.19 4.37 3.77 4.92 Factory scores 0.35 points higher than OpenAI and 0.26 higher than Anthropic overall.

Radar chart showing quality profile comparison across all three methods Breaking down by dimension:

Accuracy shows the largest gap. Factory scores 4.04, Anthropic 3.74, OpenAI 3.43. The 0.61 point difference between Factory and OpenAI reflects how often technical details like file paths and error messages survive compression.

Context awareness favors Factory (4.01) over Anthropic (3.56), a 0.45 point gap. Both approaches include structured sections for current state. Factory's advantage comes from the anchored iterative approach: by merging new summaries into a persistent state rather than regenerating from scratch, key details are less likely to drift or disappear across multiple compression cycles.

Artifact trail is the weakest dimension for all methods, ranging from 2.19 to 2.45. Even Factory's structured approach struggles to maintain complete file tracking across long sessions. This suggests artifact preservation needs specialized handling beyond general summarization.

Completeness and instruction following show small differences. All methods produce responses that address the question and follow the format. The differentiation happens in the quality of the content, not its structure.

Horizontal bar chart showing Factory quality advantage by dimensionSide-by-side comparison of token reduction efficiency and summary quality Compression ratios tell an interesting story. OpenAI compresses to 99.3% (removing 99.3% of tokens), Anthropic to 98.7%, Factory to 98.6%. Factory retains about 0.7% more tokens than OpenAI, but gains 0.35 quality points. That tradeoff favors Factory for any task where re-fetching costs matter.

What we learned The biggest surprise was how much structure matters. Generic summarization treats all content as equally compressible. A file path might be "low entropy" from an information-theoretic perspective, but it is exactly what the agent needs to continue working. By forcing the summarizer to fill explicit sections for files, decisions, and next steps, Factory's format prevents the silent drift that happens when you regenerate summaries from scratch.

Compression ratio turned out to be the wrong metric entirely. OpenAI achieves 99.3% compression but scores 0.35 points lower on quality. Those lost details eventually require re-fetching, which can exceed the token savings. What matters is total tokens to complete a task, not tokens per request.

Artifact tracking remains an unsolved problem. All methods scored between 2.19 and 2.45 out of 5.0 on knowing which files were created, modified, or examined. Even with explicit file sections, Factory only reaches 2.45. This probably requires specialized handling beyond summarization: a separate artifact index, or explicit file-state tracking in the agent scaffolding.

Finally, probe-based evaluation captures something that traditional metrics miss. ROUGE measures lexical similarity between summaries. Our approach measures whether the summary actually enables task continuation. For agentic workflows, that distinction matters.

Methodology details Dataset: Hundreds of compression points over 36,611 messages. Sessions were collected from production software engineering sessions across real codebases from users who opted into a special research program.

Probe generation: For each compression point, we generated four probes (recall, artifact, continuation, decision) based on the truncated conversation history. Probes reference specific facts, files, and decisions from the pre-compression context.

Compression: We applied all three methods to identical conversation prefixes at each compression point. Factory summaries came from production. OpenAI and Anthropic summaries were generated by feeding the same prefix to their respective APIs.

Grading: GPT-5.2 scored each probe response against six rubric dimensions. Each dimension has 2-3 criteria with explicit scoring guides. We computed dimension scores as weighted averages of criteria, and overall scores as unweighted averages of dimensions.

Statistical note: The differences we report (0.26-0.35 points) are consistent across task types and session lengths. The pattern holds whether we look at short sessions or long ones, debugging tasks or feature implementation.

Appendix: LLM Judge Prompts and Rubrics Since the LLM judge is core to this evaluation, we provide the full prompts and rubrics here.

System Prompt The judge receives this system prompt:

You are an expert evaluator assessing AI assistant responses in software development conversations.

Your task is to grade responses against specific rubric criteria. For each criterion: 1. Read the criterion question carefully 2. Examine the response for evidence 3. Assign a score from 0-5 based on the scoring guide 4. Provide brief reasoning for your score

Be objective and consistent. Focus on what is present in the response, not what could have been included.

Rubric Criteria Each dimension contains 2-3 criteria. Here are the key criteria with their scoring guides:

Accuracy

Criterion Question 0 3 5 accuracy_factual Are facts, file paths, and technical details correct? Completely incorrect or fabricated Mostly accurate with minor errors Perfectly accurate accuracy_technical Are code references and technical concepts correct? Major technical errors Generally correct with minor issues Technically precise Context Awareness

Criterion Question 0 3 5 context_conversation_state Does the response reflect current conversation state? No awareness of prior context General awareness with gaps Full awareness of conversation history context_artifact_state Does the response reflect which files/artifacts were accessed? No awareness of artifacts Partial artifact awareness Complete artifact state awareness Artifact Trail Integrity

Criterion Question 0 3 5 artifact_files_created Does the agent know which files were created? No knowledge Knows most files Perfect knowledge artifact_files_modified Does the agent know which files were modified and what changed? No knowledge Good knowledge of most modifications Perfect knowledge of all modifications artifact_key_details Does the agent remember function names, variable names, error messages? No recall Recalls most key details Perfect recall Continuity Preservation

Criterion Question 0 3 5 continuity_work_state Can the agent continue without re-fetching previously accessed information? Cannot continue without re-fetching all context Can continue with minimal re-fetching Can continue seamlessly continuity_todo_state Does the agent maintain awareness of pending tasks? Lost track of all TODOs Good awareness with some gaps Perfect task awareness continuity_reasoning Does the agent retain rationale behind previous decisions? No memory of reasoning Generally remembers reasoning Excellent retention Completeness

Criterion Question 0 3 5 completeness_coverage Does the response address all parts of the question? Ignores most parts Addresses most parts Addresses all parts thoroughly completeness_depth Is sufficient detail provided? Superficial or missing detail Adequate detail Comprehensive detail Instruction Following

Criterion Question 0 3 5 instruction_format Does the response follow the requested format? Ignores format Generally follows format Perfectly follows format instruction_constraints Does the response respect stated constraints? Ignores constraints Mostly respects constraints Fully respects all constraints Grading Process For each probe response, the judge:

Receives the probe question, the model's response, and the compacted context Evaluates against each criterion in the rubric for that probe type Outputs structured JSON with scores and reasoning per criterion Computes dimension scores as weighted averages of criteria Computes overall score as unweighted average of dimensions The judge does not know which compression method produced the response being evaluated.

https://karpathy.bearblog.dev/auto-grade-hn/

• A lot more detail in my blog post https://karpathy.bearblog.dev/auto-grade-hn/
• GitHub repo of the project if you'd like to play https://github.com/karpathy/hn-time-capsule
• The actual results pages for your reading pleasure https://karpathy.ai/hncapsule/

karpathy Home Blog

Auto-grading decade-old Hacker News discussions with hindsight 10 Dec, 2025

hnhero

TLDR: https://karpathy.ai/hncapsule/

Yesterday I stumbled on this HN thread Show HN: Gemini Pro 3 hallucinates the HN front page 10 years from now, where Gemini 3 was hallucinating the frontpage of 10 years from now. One of the comments struck me a bit more though - Bjartr linked to the HN frontpage from exactly 10 years ago, i.e. December 2015. I was reading through the discussions of 10 years ago and mentally grading them for prescience when I realized that an LLM might actually be a lot better at this task. I copy pasted one of the article+comment threads manually into ChatGPT 5.1 Thinking and it gave me a beautiful analysis of what people thought + what actually happened in retrospect, even better and significantly more detailed than what I was doing manually. I realized that this task is actually a really good fit for LLMs and I was looking for excuses to vibe code something with the newly released Opus 4.5, so I got to work. I'm going to get all the front pages of December (31 days, 30 articles per day), get ChatGPT 5.1 Thinking to do the analysis, and present everything in a nice way for historical reading.

There are two macro reasons for why I think the exercise is interesting more generally:

I believe it is quite possible and desirable to train your forward future predictor given training and effort. I was reminded again of my tweets that said "Be good, future LLMs are watching". You can take that in many directions, but here I want to focus on the idea that future LLMs are watching. Everything we do today might be scrutinized in great detail in the future because doing so will be "free". A lot of the ways people behave currently I think make an implicit "security by obscurity" assumption. But if intelligence really does become too cheap to meter, it will become possible to do a perfect reconstruction and synthesis of everything. LLMs are watching (or humans using them might be). Best to be good. Vibe coding the actual project was relatively painless and took about 3 hours with Opus 4.5, with a few hickups but overall very impressive. The repository is on GitHub here: karpathy/hn-time-capsule. Here is the progression of what the code does:

Given a date, download the frontpage of 30 articles For each article, download/parse the article itself and the full comment thread using Algolia API. Package up everything into a markdown prompt asking for the analysis. Here is the prompt prefix I used: The following is an article that appeared on Hacker News 10 years ago, and the discussion thread.

Let's use our benefit of hindsight now in 6 sections:

1. Give a brief summary of the article and the discussion thread. 2. What ended up happening to this topic? (research the topic briefly and write a summary) 3. Give out awards for "Most prescient" and "Most wrong" comments, considering what happened. 4. Mention any other fun or notable aspects of the article or discussion. 5. Give out grades to specific people for their comments, considering what happened. 6. At the end, give a final score (from 0-10) for how interesting this article and its retrospect analysis was.

As for the format of Section 5, use the header "Final grades" and follow it with simply an unordered list of people and their grades in the format of "name: grade (optional comment)". Here is an example:

Final grades

• speckx: A+ (excellent predictions on ...)
• tosh: A (correctly predicted this or that ...)
• keepamovin: A
• bgwalter: D
• fsflover: F (completely wrong on ...)

Your list may contain more people of course than just this toy example. Please follow the format exactly because I will be parsing it programmatically. The idea is that I will accumulate the grades for each account to identify the accounts that were over long periods of time the most prescient or the most wrong.

As for the format of Section 6, use the prefix "Article hindsight analysis interestingness score:" and then the score (0-10) as a number. Give high scores to articles/discussions that are prominent, notable, or interesting in retrospect. Give low scores in cases where few predictions are made, or the topic is very niche or obscure, or the discussion is not very interesting in retrospect.

Here is an example: Article hindsight analysis interestingness score: 8 --- Submit prompt to GPT 5.1 Thinking via the OpenAI API Collect and parse the results Render the results into static HTML web pages for easy viewing Host the html result pages on my website: https://karpathy.ai/hncapsule/ Host all the intermediate results of the data directory if someone else would like to play. It's the file data.zip under the exact same url prefix (intentionally avoiding a direct link). I spent a few hours browsing around and found it to be very interesting. A few example threads just for fun:

December 3 2015 Swift went open source. December 6 2015 Launch of Figma December 11 2015 original announcement of OpenAI :'). December 16 2015 geohot is building Comma December 22 2015 SpaceX launch webcast: Orbcomm-2 Mission December 28 2015 Theranos struggles And then when you navigate over to the Hall of Fame, you can find the top commenters of Hacker News in December 2015, sorted by imdb-style score of their grade point average. In particular, congratulations to pcwalton, tptacek, paulmd, cstross, greglindahl, moxie, hannob, 0xcde4c3db, Manishearth, johncolanduoni - GPT 5.1 Thinking found your comments very insightful and prescient. You can also scroll all the way down to find the noise of HN, which I think we're all familiar with too :)

My code (wait, Opus' code?) on GitHub can be used to reproduce or tweak the results. Running 31 days of 30 articles through GPT 5.1 Thinking meant 31 * 30 = 930 LLM queries and cost about $58 and somewhere around ~1 hour. The LLM megaminds of the future might find this kind of a thing a lot easier, a lot faster and a lot cheaper.

-------

Quick new post: Auto-grading decade-old Hacker News discussions with hindsight

I took all the 930 frontpage Hacker News article+discussion of December 2015 and asked the GPT 5.1 Thinking API to do an in-hindsight analysis to identify the most/least prescient comments. This took ~3 hours to vibe code and ~1 hour and $60 to run. The idea was sparked by the HN article yesterday where Gemini 3 was asked to hallucinate the HN front page one decade forward.

More generally:

1. in-hindsight analysis has always fascinated me as a way to train your forward prediction model so reading the results is really interesting and 2. it's worth contemplating what it looks like when LLM megaminds of the future can do this kind of work a lot cheaper, faster and better. Every single bit of information you contribute to the internet can (and probably will be) scrutinized in great detail if it is "free". Hence also my earlier tweet from a while back - "be good, future LLMs are watching".

Congrats to the top 10 accounts pcwalton, tptacek, paulmd, cstross, greglindahl, moxie, hannob, 0xcde4c3db, Manishearth, and johncolanduoni - GPT 5.1 Thinking found your comments to be the most insightful and prescient of all comments of HN in December of 2015.