Llm Wiki

# {{VAULT_NAME}} — LLM Wiki

> **Topic:** {{TOPIC}}
> **Initialized:** {{DATE}}
> **Tool:** Any AGENTS.md-aware CLI (Codex, Cursor, Antigravity, OpenCode, Gemini CLI, etc.). Claude Code uses `CLAUDE.md`, which is identical.

You are the maintainer of this wiki. You read from `raw/`, you write to `wiki/`. You never edit `raw/`.

## The three layers

```
raw/     → sources (articles, papers, notes). IMMUTABLE. You only read.
wiki/    → the knowledge base. You own this. Create, update, cross-reference.
AGENTS.md → schema (this file). Co-evolved with the user.
```

## Vault structure

```
raw/
├── <sources>              # articles, papers, notes — IMMUTABLE
└── assets/                # downloaded images from clipped articles

wiki/
├── index.md               # content catalog — update every ingest
├── log.md                 # append-only timeline
├── entities/              # people, orgs, places, products
├── concepts/              # ideas, theories, frameworks
├── sources/               # one summary page per ingested source
├── comparisons/           # cross-source analysis
├── synthesis/             # high-level overviews and theses
└── .templates/            # page templates (reference only)
```

## Page frontmatter (required on every wiki page)

```yaml
---
title: <Title>
category: entity | concept | source | comparison | synthesis
summary: <one-line summary>
tags: [tag1, tag2]
sources: <count of sources referencing this page>
updated: YYYY-MM-DD
---
```

For `source` pages, also include:
```yaml
source_path: raw/<path>
source_date: YYYY-MM
authors: [author1, author2]
ingested: YYYY-MM-DD
```

## The three operations

### Ingest

When the user says "ingest this source" or points you at a file in `raw/`:

1. Read the source directly with your file-reading tool
2. **Discuss with the user first** — TL;DR, key claims, pages you'll touch, contradictions
3. Wait for confirmation
4. Create/merge the summary page at `wiki/sources/<slug>.md`
5. Update every relevant entity and concept page (typically 5-15 pages)
6. Flag contradictions with `> ⚠️ Contradiction:` callouts on both sides
7. Update `wiki/index.md`
8. Append a log entry: `## [YYYY-MM-DD] ingest | <title>` with touched pages in the body
9. Report back with a bulleted list of touched pages

If Python is available, use the helpers:
```bash
python <plugin-path>/scripts/ingest_source.py --vault . --source <path> --json
python <plugin-path>/scripts/append_log.py --vault . --op ingest --title "<title>"
python <plugin-path>/scripts/update_index.py --vault .
```

### Query

When the user asks a question:

1. Read `wiki/index.md` first
2. Pick 3-10 relevant pages across categories
3. Read them in full
4. Follow wikilinks opportunistically
5. Fall back to `wiki_search.py --query <terms>` if needed
6. Synthesize: direct answer → supporting detail → inline `[[sources/xxx]]` citations → "Related pages"
7. **Offer to file the answer back** as a new page

### Lint

When the user says "check the wiki" or periodically:

1. Run `lint_wiki.py` + `graph_analyzer.py`
2. Do semantic checks (contradictions, stale claims, concept gaps)
3. Present a report with suggested actions
4. Append a `lint` entry to `log.md`

## Iron rules

1. **`raw/` is immutable.** You read from it; you never write to it.
2. **All writes go to `wiki/`.** No exceptions.
3. **Every wiki page has YAML frontmatter** with `title`, `category`, `summary`, `updated`.
4. **Every ingest touches ≥5 files.**
5. **Every claim has a citation.**
6. **Contradictions get flagged inline.** Both pages.
7. **Good answers get filed back.** Explorations compound.

## Log format

```
## [YYYY-MM-DD] <op> | <title>
<optional detail>
```

Ops: `ingest`, `query`, `lint`, `create`, `update`, `delete`, `note`.

## Tools

Python scripts live wherever you installed the plugin. Standard library only.

- `init_vault.py`
- `ingest_source.py`
- `update_index.py`
- `append_log.py`
- `wiki_search.py`
- `lint_wiki.py`
- `graph_analyzer.py`
- `export_marp.py`

Run any of them with `--help`.

## Style

- Concise. Wiki pages are read, not generated.
- Short paragraphs. Bulleted lists where appropriate.
- Cite aggressively with `[[wikilinks]]`.
- Say "I don't know" when you don't. Don't invent content.
- Update `updated:` whenever you touch a page.

# {{VAULT_NAME}} — LLM Wiki

> **Topic:** {{TOPIC}}
> **Initialized:** {{DATE}}
> **Tool:** Claude Code (this file). A parallel `AGENTS.md` exists for Codex/Cursor/Antigravity.

You are the maintainer of this wiki. You read from `raw/`, you write to `wiki/`. You never edit `raw/`.

## The three layers

```
raw/     → sources (articles, papers, notes). IMMUTABLE. You only read.
wiki/    → the knowledge base. You own this. Create, update, cross-reference.
CLAUDE.md / AGENTS.md → schema (this file). Co-evolved with the user.
```

## Vault structure

```
raw/
├── <sources>              # articles, papers, notes — IMMUTABLE
└── assets/                # downloaded images from clipped articles

wiki/
├── index.md               # content catalog — update every ingest
├── log.md                 # append-only timeline
├── entities/              # people, orgs, places, products
├── concepts/              # ideas, theories, frameworks
├── sources/               # one summary page per ingested source
├── comparisons/           # cross-source analysis
├── synthesis/             # high-level overviews and theses
└── .templates/            # page templates (reference only)
```

## Page frontmatter (required on every wiki page)

```yaml
---
title: <Title>
category: entity | concept | source | comparison | synthesis
summary: <one-line summary>
tags: [tag1, tag2]
sources: <count of sources referencing this page>
updated: YYYY-MM-DD
---
```

For `source` pages, also include:
```yaml
source_path: raw/<path>
source_date: YYYY-MM (original publication)
authors: [author1, author2]
ingested: YYYY-MM-DD
```

## The three operations

### Ingest (`/wiki-ingest <path>`)

1. Run `python scripts/ingest_source.py --vault . --source <path> --json` to get the brief
2. Read the source directly
3. **Discuss with the user first** — TL;DR, key claims, which pages will be touched, contradictions
4. Wait for confirmation
5. Create or merge the summary page at `wiki/sources/<slug>.md`
6. Update every relevant entity and concept page (typically 5-15 pages)
7. Flag contradictions with `> ⚠️ Contradiction:` callouts on both sides
8. Update `wiki/index.md` (run `update_index.py` or edit inline)
9. Run `append_log.py --op ingest --title "<title>" --detail "<touched pages>"`
10. Report back with a bulleted list of touched pages

### Query (`/wiki-query <question>`)

1. Read `wiki/index.md` first
2. Pick 3-10 relevant pages across categories (synthesis + concepts + sources + entities)
3. Read them in full
4. Follow wikilinks opportunistically
5. Fall back to `wiki_search.py --query <terms>` if the index doesn't surface the answer
6. Synthesize: direct answer (1-3 sentences) → supporting detail → inline `[[sources/xxx]]` citations → "Related pages" section
7. **Offer to file the answer back** as a new page in `comparisons/` or `synthesis/`

### Lint (`/wiki-lint`)

1. Run `python scripts/lint_wiki.py --vault .` for mechanical checks
2. Run `python scripts/graph_analyzer.py --vault .` for structural stats
3. Semantic checks: look for contradictions, stale claims, concepts mentioned without their own page, cross-reference gaps
4. Present findings as a markdown report with suggested actions
5. Append a `lint` entry to `log.md`

## Iron rules

1. **`raw/` is immutable.** You read from it; you never write to it.
2. **All writes go to `wiki/`.** No exceptions.
3. **Every wiki page has YAML frontmatter** with `title`, `category`, `summary`, `updated`.
4. **Every ingest touches ≥5 files.** The source summary, 2-4 entity/concept pages, `index.md`, `log.md`.
5. **Every claim has a citation.** Link back to the `sources/<slug>` page.
6. **Contradictions get flagged inline.** Both pages get the callout.
7. **Good answers get filed back.** Explorations compound.

## Log format

```
## [YYYY-MM-DD] <op> | <title>
<optional detail — which pages touched, what changed>
```

Valid ops: `ingest`, `query`, `lint`, `create`, `update`, `delete`, `note`.

Grep the log: `grep "^## \[" wiki/log.md | tail -10`

## Tools

All scripts live at `~/.claude/skills/llm-wiki/scripts/` (or wherever you installed the plugin). Standard library only.

- `init_vault.py` — bootstrap a vault
- `ingest_source.py` — prep a source for ingest (metadata + preview)
- `update_index.py` — regenerate `wiki/index.md` from page frontmatter
- `append_log.py` — append a standardized log entry
- `wiki_search.py` — BM25 search fallback
- `lint_wiki.py` — mechanical health check
- `graph_analyzer.py` — link graph stats
- `export_marp.py` — render a page as a Marp slide deck

## Obsidian

The user opens this vault in Obsidian. They watch the graph view while you edit. Useful plugins: Graph view, Backlinks, Dataview, Marp, Templates, Git.

## Style

- Be concise. Wiki pages are read, not generated.
- Prefer short paragraphs. Bulleted lists where appropriate.
- Cite aggressively with `[[wikilinks]]`.
- When you're not sure, say so in the page. Don't invent content.
- Update `updated:` frontmatter whenever you touch a page.

# Cursor rules for {{VAULT_NAME}} LLM Wiki

You are the maintainer of this wiki. You read from `raw/` and write only to `wiki/`.
You never edit files in `raw/`. Full schema is in `AGENTS.md` — read it first.

Topic: {{TOPIC}}
Initialized: {{DATE}}

Core rules:
1. `raw/` is immutable. Read only.
2. All writes go to `wiki/`.
3. Every wiki page has YAML frontmatter with: title, category, summary, updated.
4. Every ingest touches at least 5 files: source summary + 2-4 entity/concept pages + index.md + log.md.
5. Every claim has a wikilink citation to its source page.
6. Contradictions get `> ⚠️ Contradiction:` callouts on both sides.
7. Good query answers get filed back into the wiki as new pages.

Operations: ingest / query / lint. Full workflows in AGENTS.md.

Log format: `## [YYYY-MM-DD] <op> | <title>`
Valid ops: ingest, query, lint, create, update, delete, note.

Scripts (Python stdlib) in `<plugin-path>/scripts/`:
- ingest_source.py, update_index.py, append_log.py
- lint_wiki.py, graph_analyzer.py, wiki_search.py, export_marp.py

Run any with `--help`. Use them when helpful — they're fast and deterministic.

Style: concise. Cite with `[[wikilinks]]`. Say "I don't know" when you don't.
Update `updated:` frontmatter on every touch.

# Index — {{VAULT_NAME}}

_Initialized {{DATE}} • 0 pages_

> Content-oriented catalog of every page in `wiki/`. Updated by
> `scripts/update_index.py` or during `/wiki-ingest`. Answer queries
> by reading this file first, then drilling into relevant pages.
>
> Topic: **{{TOPIC}}**

## Synthesis (0)
_No pages yet. Will appear here as you ingest sources and build high-level theses._

## Concept (0)
_No pages yet. Will populate with ideas, theories, methods as sources are ingested._

## Entity (0)
_No pages yet. People, organizations, places, products mentioned in your sources will show up here._

## Source (0)
_No pages yet. Each ingested source gets one summary page here._

## Comparison (0)
_No pages yet. Cross-source or cross-concept analyses will live here._

---

### First steps

1. Drop a source into `raw/`
2. Run `/wiki-ingest raw/<your-file>` in your LLM CLI
3. Watch this index populate

# Log — {{VAULT_NAME}}

> Append-only timeline. Every LLM operation leaves an entry here.
>
> Format: `## [YYYY-MM-DD] <op> | <title>` followed by an optional detail line.
> Valid ops: `ingest`, `query`, `lint`, `create`, `update`, `delete`, `note`.
>
> Grep the last 10 entries: `grep "^## \[" log.md | tail -10`

## [{{DATE}}] note | Vault initialized
Topic: **{{TOPIC}}**. Layers created: `raw/`, `wiki/{entities,concepts,sources,comparisons,synthesis}`.
Schema loader: `CLAUDE.md` + `AGENTS.md` + `.cursorrules`.

{
  "status": "ok",
  "log_path": "/tmp/test-vault/wiki/log.md",
  "date": "2026-04-11",
  "op": "ingest",
  "title": "Hello Monosemanticity",
  "header": "## [2026-04-11] ingest | Hello Monosemanticity",
  "detail": "touched 2 pages"
}

{
  "status": "ok",
  "vault": "/tmp/test-vault",
  "source": "wiki/concepts/sparse-autoencoder.md",
  "theme": "gaia",
  "output_dir": "slides",
  "rendered_count": 1,
  "rendered": ["slides/sparse-autoencoder.marp.md"]
}

{
  "total_pages": 2,
  "total_edges": 2,
  "top_outbound_hubs": [
    {"page": "sources/hello", "outbound": 1},
    {"page": "concepts/sparse-autoencoder", "outbound": 1}
  ],
  "top_inbound_hubs": [
    {"page": "sources/hello", "inbound": 1},
    {"page": "concepts/sparse-autoencoder", "inbound": 1}
  ],
  "orphans": [],
  "sinks": [],
  "components": [
    {"size": 2, "sample": ["concepts/sparse-autoencoder", "sources/hello"]}
  ],
  "component_count": 1
}

{
  "source_path": "/tmp/test-vault/raw/articles/hello.md",
  "relative": "raw/articles/hello.md",
  "bytes": 204,
  "sha256": "dcb7021b49882e26",
  "ext": ".md",
  "title_guess": "Hello Monosemanticity",
  "word_count": 28,
  "preview": "# Hello Monosemanticity\n\nAnthropic's Bricken et al. 2023 paper trained a sparse autoencoder on a one-layer transformer and found interpretable features.\nKey claim: the feature dictionary is overcomplete.\n",
  "existing_summary_page": null,
  "suggested_summary_path": "wiki/sources/hello-monosemanticity.md"
}

{
  "status": "ok",
  "vault_path": "/tmp/test-vault",
  "topic": "LLM interpretability",
  "tool": "all",
  "date": "2026-04-11",
  "installed_files": [
    "CLAUDE.md",
    "AGENTS.md",
    ".cursorrules",
    "wiki/index.md",
    "wiki/log.md"
  ],
  "page_templates_copied": 5,
  "layers": {
    "raw": "your sources — immutable",
    "wiki": "LLM-maintained knowledge base",
    "index": "wiki/index.md",
    "log": "wiki/log.md"
  },
  "next_steps": [
    "Open the vault in Obsidian",
    "Drop a source into raw/",
    "Run /wiki-ingest <path> in your LLM CLI"
  ]
}

{
  "vault": "/tmp/test-vault",
  "total_pages": 2,
  "orphans": [],
  "broken_links": [],
  "stale": [],
  "missing_frontmatter": [],
  "duplicate_titles": {},
  "log_gap": null
}

{
  "status": "ok",
  "vault": "/tmp/test-vault",
  "total_pages": 2,
  "by_category": {
    "concept": 1,
    "source": 1
  },
  "dry_run": false,
  "index_path": "/tmp/test-vault/wiki/index.md"
}

{
  "query": "sparse autoencoder",
  "hits": [
    {
      "path": "concepts/sparse-autoencoder.md",
      "score": 1.995,
      "snippet": "--- title: Sparse Autoencoder category: concept summary: Dictionary-learning method for interpretable features tags: [interpretability] sources: 1 updated: 2026-04-11 --- # Sparse Autoencoder See [[sources/hello]] for th…"
    }
  ]
}

#!/usr/bin/env python3
"""
append_log.py — Append a standardized entry to wiki/log.md.

The log is append-only and uses a consistent header so unix tools can parse it:
    ## [YYYY-MM-DD] <op> | <title>

A useful tip: if each entry starts with a consistent prefix, the log becomes
parseable with simple unix tools — `grep "^## \\[" log.md | tail -5` gives you
the last 5 entries.

Usage:
    python append_log.py --vault ~/vaults/research --op ingest --title "Anthropic Monosemanticity"
    python append_log.py --vault . --op query --title "interpretability vs mechinterp" --detail "3 pages touched"
    python append_log.py --vault . --op lint --title "weekly health check" --detail "2 contradictions" --json

Valid ops:
    ingest   — a source was read and integrated into the wiki
    query    — a question was answered (filed back as a page)
    lint     — a health-check pass ran
    create   — a new page was created outside of an ingest
    update   — an existing page was updated outside of an ingest
    delete   — a page was removed
    note     — freeform note (contradictions flagged, thesis revisions, etc.)

Exit codes:
    0   success
    1   invalid vault / missing log.md / invalid op / write failure
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import sys
from pathlib import Path

VALID_OPS = {"ingest", "query", "lint", "create", "update", "delete", "note"}


def _error(message, as_json=False):
    """Print an error and exit with code 1. Respects --json mode."""
    if as_json:
        print(json.dumps({"status": "error", "message": message}))
    else:
        print(f"[error] {message}", file=sys.stderr)
    sys.exit(1)


def validate_vault(vault):
    """Return the log.md path or raise if vault is invalid."""
    if not vault.exists():
        raise FileNotFoundError(f"vault does not exist: {vault}")
    log_path = vault / "wiki" / "log.md"
    if not log_path.exists():
        raise FileNotFoundError(f"{log_path} does not exist — is this a vault?")
    return log_path


def format_entry(op, title, detail):
    """Build the standardized log entry string."""
    today = dt.date.today().isoformat()
    header = f"## [{today}] {op} | {title}"
    body = f"\n{detail}\n" if detail else "\n"
    return today, header, f"\n{header}\n{body}"


def append_log(vault, op, title, detail, as_json=False):
    """Append a standardized entry to wiki/log.md inside the vault."""
    if op not in VALID_OPS:
        _error(f"unknown op '{op}'. Valid: {sorted(VALID_OPS)}", as_json)

    try:
        log_path = validate_vault(vault)
    except FileNotFoundError as e:
        _error(str(e), as_json)

    today, header, entry_text = format_entry(op, title, detail)

    try:
        with log_path.open("a", encoding="utf-8") as f:
            f.write(entry_text)
    except OSError as e:
        _error(f"failed to write {log_path}: {e}", as_json)

    result = {
        "status": "ok",
        "log_path": str(log_path),
        "date": today,
        "op": op,
        "title": title,
        "header": header,
        "detail": detail,
    }

    if as_json:
        print(json.dumps(result, indent=2))
    else:
        print(f"[ok] appended to {log_path}")
        print(f"     {header}")
        if detail:
            print(f"     detail: {detail}")

    return result


def main():
    p = argparse.ArgumentParser(
        description="Append a standardized entry to wiki/log.md",
        epilog="Format: ## [YYYY-MM-DD] <op> | <title>",
    )
    p.add_argument("--vault", required=True, help="Vault root directory")
    p.add_argument(
        "--op",
        required=True,
        choices=sorted(VALID_OPS),
        help="Operation type (ingest, query, lint, create, update, delete, note)",
    )
    p.add_argument("--title", required=True, help="Short title for the entry")
    p.add_argument("--detail", default=None, help="Optional detail text")
    p.add_argument(
        "--json", action="store_true", help="Emit result as JSON instead of human-readable"
    )
    args = p.parse_args()
    append_log(
        Path(args.vault).expanduser().resolve(),
        args.op,
        args.title,
        args.detail,
        as_json=args.json,
    )


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
export_marp.py — Render a wiki page (or subtree) as a Marp slide deck.

Marp is a Markdown-based slide format supported by an Obsidian plugin. This
script adds Marp frontmatter and converts `## H2` headings into slide breaks,
so any wiki page with H2 sections becomes a usable slide deck with zero
manual formatting.

Usage:
    python export_marp.py --vault . --page wiki/synthesis/interpretability-overview.md
    python export_marp.py --vault . --page wiki/concepts/ --theme gaia --out slides/
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path

FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)

MARP_HEADER = """---
marp: true
theme: {theme}
paginate: true
---

"""


def strip_frontmatter(text: str) -> str:
    return FRONTMATTER_RE.sub("", text, count=1)


def to_marp(text: str, theme: str) -> str:
    body = strip_frontmatter(text).strip()
    # Turn each "## " into a new slide separator.
    # First H1 → title slide. Subsequent H2 → slide breaks.
    lines = body.splitlines()
    out: list[str] = []
    seen_h1 = False
    for line in lines:
        if line.startswith("# ") and not seen_h1:
            out.append(line)
            out.append("")
            seen_h1 = True
            continue
        if line.startswith("## "):
            out.append("\n---\n")
            out.append(line)
            continue
        out.append(line)
    return MARP_HEADER.format(theme=theme) + "\n".join(out).strip() + "\n"


def render_one(src, out_path, theme, verbose=True):
    """Render a single markdown page as a Marp slide deck."""
    try:
        text = src.read_text(encoding="utf-8", errors="replace")
    except OSError as e:
        raise RuntimeError(f"failed to read {src}: {e}")
    out_path.parent.mkdir(parents=True, exist_ok=True)
    try:
        out_path.write_text(to_marp(text, theme), encoding="utf-8")
    except OSError as e:
        raise RuntimeError(f"failed to write {out_path}: {e}")
    if verbose:
        print(f"[ok] {src.name} -> {out_path}")
    return out_path


def _error(message, as_json=False):
    if as_json:
        print(json.dumps({"status": "error", "message": message}))
    else:
        print(f"[error] {message}", file=sys.stderr)
    sys.exit(1)


def main():
    p = argparse.ArgumentParser(
        description="Render a wiki page (or subtree) to a Marp slide deck.",
        epilog="Marp is a Markdown-based slide format supported by an Obsidian plugin.",
    )
    p.add_argument("--vault", required=True, help="Vault root directory")
    p.add_argument(
        "--page",
        required=True,
        help="Page or directory relative to the vault (e.g. wiki/synthesis/overview.md)",
    )
    p.add_argument(
        "--theme", default="default", choices=["default", "gaia", "uncover"], help="Marp theme"
    )
    p.add_argument(
        "--out", default="slides", help="Output directory relative to vault (default: slides)"
    )
    p.add_argument(
        "--json", action="store_true", help="Emit result as JSON instead of human-readable"
    )
    args = p.parse_args()

    vault = Path(args.vault).expanduser().resolve()
    if not vault.exists():
        _error(f"vault does not exist: {vault}", args.json)

    src = (vault / args.page).resolve()
    if not src.exists():
        _error(f"page not found: {src}", args.json)

    out_root = vault / args.out
    rendered = []

    try:
        if src.is_file():
            dest = out_root / src.name.replace(".md", ".marp.md")
            render_one(src, dest, args.theme, verbose=not args.json)
            rendered.append(str(dest.relative_to(vault)))
        else:
            for md in sorted(src.rglob("*.md")):
                rel = md.relative_to(src)
                dest = out_root / rel.with_suffix(".marp.md")
                render_one(md, dest, args.theme, verbose=not args.json)
                rendered.append(str(dest.relative_to(vault)))
    except RuntimeError as e:
        _error(str(e), args.json)

    if args.json:
        print(
            json.dumps(
                {
                    "status": "ok",
                    "vault": str(vault),
                    "source": str(src.relative_to(vault)),
                    "theme": args.theme,
                    "output_dir": args.out,
                    "rendered_count": len(rendered),
                    "rendered": rendered,
                },
                indent=2,
            )
        )


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
graph_analyzer.py — Analyze the wikilink graph of an LLM Wiki vault.

Reports hubs, orphans, bridges, and weakly-connected components so the LLM
knows where to focus cross-referencing work.

Usage:
    python graph_analyzer.py --vault ~/vaults/research
    python graph_analyzer.py --vault . --json
    python graph_analyzer.py --vault . --top 20
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import defaultdict
from pathlib import Path

WIKILINK_RE = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]*)?(?:\|[^\]]*)?\]\]")


def build_graph(vault: Path):
    wiki = vault / "wiki"
    if not wiki.exists():
        raise SystemExit(f"[error] {wiki} not found")
    nodes: set[str] = set()
    out: dict[str, set[str]] = defaultdict(set)
    inb: dict[str, set[str]] = defaultdict(set)
    stems: dict[str, str] = {}

    for md in wiki.rglob("*.md"):
        rel = md.relative_to(wiki)
        if rel.name in {"index.md", "log.md"}:
            continue
        if any(part.startswith(".") for part in rel.parts):
            continue
        key = str(rel).replace("\\", "/")[:-3]
        nodes.add(key)
        stems[Path(key).name] = key

    for md in wiki.rglob("*.md"):
        rel = md.relative_to(wiki)
        if rel.name in {"index.md", "log.md"} or any(p.startswith(".") for p in rel.parts):
            continue
        key = str(rel).replace("\\", "/")[:-3]
        text = md.read_text(encoding="utf-8", errors="replace")
        for m in WIKILINK_RE.finditer(text):
            target = m.group(1).strip()
            if target.endswith(".md"):
                target = target[:-3]
            if target in nodes:
                out[key].add(target)
                inb[target].add(key)
            elif Path(target).name in stems:
                resolved = stems[Path(target).name]
                out[key].add(resolved)
                inb[resolved].add(key)

    return nodes, out, inb


def connected_components(nodes: set[str], out: dict[str, set[str]], inb: dict[str, set[str]]):
    adj: dict[str, set[str]] = defaultdict(set)
    for n in nodes:
        adj[n] |= out.get(n, set())
        adj[n] |= inb.get(n, set())
    seen: set[str] = set()
    components: list[set[str]] = []
    for n in nodes:
        if n in seen:
            continue
        stack = [n]
        comp: set[str] = set()
        while stack:
            v = stack.pop()
            if v in seen:
                continue
            seen.add(v)
            comp.add(v)
            stack.extend(adj[v] - seen)
        components.append(comp)
    components.sort(key=len, reverse=True)
    return components


def analyze(vault: Path, top: int) -> dict:
    nodes, out, inb = build_graph(vault)
    hubs_out = sorted(nodes, key=lambda n: len(out.get(n, set())), reverse=True)[:top]
    hubs_in = sorted(nodes, key=lambda n: len(inb.get(n, set())), reverse=True)[:top]
    orphans = sorted(n for n in nodes if not inb.get(n))
    sinks = sorted(n for n in nodes if not out.get(n))
    comps = connected_components(nodes, out, inb)
    return {
        "total_pages": len(nodes),
        "total_edges": sum(len(v) for v in out.values()),
        "top_outbound_hubs": [{"page": h, "outbound": len(out.get(h, set()))} for h in hubs_out],
        "top_inbound_hubs": [{"page": h, "inbound": len(inb.get(h, set()))} for h in hubs_in],
        "orphans": orphans,
        "sinks": sinks,
        "components": [
            {"size": len(c), "sample": sorted(c)[:5]} for c in comps[:10]
        ],
        "component_count": len(comps),
    }


def main() -> None:
    p = argparse.ArgumentParser(description="Analyze the wikilink graph of an LLM Wiki vault")
    p.add_argument("--vault", required=True)
    p.add_argument("--top", type=int, default=10)
    p.add_argument("--json", action="store_true")
    args = p.parse_args()
    r = analyze(Path(args.vault).expanduser().resolve(), args.top)

    if args.json:
        print(json.dumps(r, indent=2, default=list))
        return

    print(f"LLM Wiki graph — {r['total_pages']} pages, {r['total_edges']} links")
    print(f"Connected components: {r['component_count']}")
    print()
    print("Top outbound hubs (pages that link to many others):")
    for h in r["top_outbound_hubs"]:
        print(f"  - {h['page']}  ({h['outbound']} out)")
    print()
    print("Top inbound hubs (pages many others link TO):")
    for h in r["top_inbound_hubs"]:
        print(f"  - {h['page']}  ({h['inbound']} in)")
    print()
    print(f"Orphans (no inbound): {len(r['orphans'])}")
    for o in r["orphans"][:10]:
        print(f"  - {o}")
    print()
    print(f"Sinks (no outbound): {len(r['sinks'])}")
    for s in r["sinks"][:10]:
        print(f"  - {s}")


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
ingest_source.py — Prepare a source for LLM ingestion.

This is a *helper* — it does not call an LLM. It extracts text and metadata from
a source file and emits a JSON brief the LLM (via the /wiki-ingest command or
the wiki-ingestor sub-agent) can read, discuss with the user, and use to update
the wiki.

Supported source types (stdlib only):
  .md  .txt  .html .htm  .json  .csv

For .pdf and binary formats, install optional readers yourself, or let the LLM
read the file directly via its Read tool.

Usage:
    python ingest_source.py --vault ~/vaults/research --source raw/paper.md
    python ingest_source.py --vault . --source raw/article.html --json

Output (JSON):
    {
      "source_path": "raw/paper.md",
      "relative": "raw/paper.md",
      "bytes": 12345,
      "sha256": "...",
      "ext": ".md",
      "title_guess": "Monosemanticity",
      "word_count": 8432,
      "preview": "First 1200 chars...",
      "existing_summary_page": "wiki/sources/monosemanticity.md" | null,
      "suggested_summary_path": "wiki/sources/monosemanticity.md"
    }
"""
from __future__ import annotations
import argparse
import hashlib
import html.parser
import json
import re
import sys
from pathlib import Path

PREVIEW_CHARS = 1200
SLUG_RE = re.compile(r"[^a-z0-9]+")


def slugify(text: str) -> str:
    text = text.lower().strip()
    text = SLUG_RE.sub("-", text).strip("-")
    return text[:60] or "untitled"


class _HTMLTextExtractor(html.parser.HTMLParser):
    def __init__(self) -> None:
        super().__init__()
        self.parts: list[str] = []
        self.title: str | None = None
        self._in_title = False
        self._skip = False

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        if tag in {"script", "style"}:
            self._skip = True
        if tag == "title":
            self._in_title = True

    def handle_endtag(self, tag: str) -> None:
        if tag in {"script", "style"}:
            self._skip = False
        if tag == "title":
            self._in_title = False

    def handle_data(self, data: str) -> None:
        if self._skip:
            return
        if self._in_title and self.title is None:
            self.title = data.strip() or None
        else:
            text = data.strip()
            if text:
                self.parts.append(text)

    def text(self) -> str:
        return "\n".join(self.parts)


def extract(path: Path) -> tuple[str, str | None]:
    ext = path.suffix.lower()
    data = path.read_bytes()
    if ext in {".md", ".txt"}:
        text = data.decode("utf-8", errors="replace")
        title = None
        for line in text.splitlines()[:20]:
            if line.startswith("# "):
                title = line[2:].strip()
                break
        return text, title
    if ext in {".html", ".htm"}:
        parser = _HTMLTextExtractor()
        try:
            parser.feed(data.decode("utf-8", errors="replace"))
        except Exception:
            pass
        return parser.text(), parser.title
    if ext == ".json":
        try:
            obj = json.loads(data.decode("utf-8", errors="replace"))
            return json.dumps(obj, indent=2)[:100000], None
        except Exception:
            return data.decode("utf-8", errors="replace"), None
    if ext == ".csv":
        text = data.decode("utf-8", errors="replace")
        head = "\n".join(text.splitlines()[:50])
        return head, None
    # Unknown: attempt utf-8 decode, let the LLM handle it
    try:
        return data.decode("utf-8", errors="replace"), None
    except Exception:
        return "", None


def main() -> None:
    p = argparse.ArgumentParser(description="Prepare a source for LLM ingestion.")
    p.add_argument("--vault", required=True)
    p.add_argument("--source", required=True, help="Path to the source file (inside raw/)")
    p.add_argument("--json", action="store_true", help="Emit JSON only")
    args = p.parse_args()

    vault = Path(args.vault).expanduser().resolve()
    src = Path(args.source).expanduser().resolve()
    if not src.exists():
        print(f"[error] source not found: {src}", file=sys.stderr)
        sys.exit(1)

    try:
        rel = src.relative_to(vault)
    except ValueError:
        rel = src

    text, title = extract(src)
    title_guess = title or src.stem.replace("-", " ").replace("_", " ").title()
    slug = slugify(title_guess)
    suggested = f"wiki/sources/{slug}.md"
    existing = vault / suggested
    existing_path = str(suggested) if existing.exists() else None

    brief = {
        "source_path": str(src),
        "relative": str(rel).replace("\\", "/"),
        "bytes": src.stat().st_size,
        "sha256": hashlib.sha256(src.read_bytes()).hexdigest()[:16],
        "ext": src.suffix.lower(),
        "title_guess": title_guess,
        "word_count": len(text.split()),
        "preview": text[:PREVIEW_CHARS],
        "existing_summary_page": existing_path,
        "suggested_summary_path": suggested,
    }

    if args.json:
        print(json.dumps(brief, indent=2, ensure_ascii=False))
    else:
        print(f"Source:         {brief['source_path']}")
        print(f"Title (guess):  {brief['title_guess']}")
        print(f"Size:           {brief['bytes']} bytes ({brief['word_count']} words)")
        print(f"SHA256 (short): {brief['sha256']}")
        print(f"Suggested page: {brief['suggested_summary_path']}")
        if existing_path:
            print(f"EXISTING PAGE:  {existing_path}  ← re-ingest / merge mode")
        print()
        print("--- preview ---")
        print(brief["preview"])
        print("--- /preview ---")


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
init_vault.py — Bootstrap an LLM Wiki vault.

Creates the three-layer structure (raw/, wiki/, schema files) and seeds it with
starter templates for CLAUDE.md, AGENTS.md, index.md, log.md, and page templates.

Usage:
    python init_vault.py --path ~/vaults/research --topic "LLM interpretability"
    python init_vault.py --path ./my-wiki --topic "Book: The Power Broker" --tool codex

The --tool flag controls which schema file(s) to install:
    claude-code  → CLAUDE.md (default)
    codex        → AGENTS.md
    cursor       → AGENTS.md + .cursorrules
    antigravity  → AGENTS.md
    all          → CLAUDE.md + AGENTS.md + .cursorrules (recommended for multi-tool)
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import sys
from pathlib import Path

SCRIPT_DIR = Path(__file__).resolve().parent
PLUGIN_DIR = SCRIPT_DIR.parent
ASSETS_DIR = PLUGIN_DIR / "assets"

VAULT_DIRS = [
    "raw",
    "raw/assets",
    "wiki",
    "wiki/entities",
    "wiki/concepts",
    "wiki/sources",
    "wiki/comparisons",
    "wiki/synthesis",
]

TOOL_FILES = {
    "claude-code": ["CLAUDE.md.template:CLAUDE.md"],
    "codex": ["AGENTS.md.template:AGENTS.md"],
    "cursor": ["AGENTS.md.template:AGENTS.md", "cursorrules.template:.cursorrules"],
    "antigravity": ["AGENTS.md.template:AGENTS.md"],
    "opencode": ["AGENTS.md.template:AGENTS.md"],
    "gemini-cli": ["AGENTS.md.template:AGENTS.md"],
    "all": [
        "CLAUDE.md.template:CLAUDE.md",
        "AGENTS.md.template:AGENTS.md",
        "cursorrules.template:.cursorrules",
    ],
}


def render_template(src, dest, variables):
    """Render a template file with {{VAR}} substitutions to dest."""
    if not src.exists():
        print(f"[warn] template missing: {src}", file=sys.stderr)
        return False
    try:
        text = src.read_text(encoding="utf-8")
    except OSError as e:
        print(f"[warn] could not read {src}: {e}", file=sys.stderr)
        return False
    for key, value in variables.items():
        text = text.replace("{{" + key + "}}", value)
    try:
        dest.write_text(text, encoding="utf-8")
    except OSError as e:
        print(f"[warn] could not write {dest}: {e}", file=sys.stderr)
        return False
    return True


def _error(message, as_json=False):
    if as_json:
        print(json.dumps({"status": "error", "message": message}))
    else:
        print(f"[error] {message}", file=sys.stderr)
    sys.exit(1)


def init_vault(vault_path, topic, tool, force, as_json=False):
    """Bootstrap a new LLM Wiki vault at vault_path."""
    if vault_path.exists() and any(vault_path.iterdir()) and not force:
        _error(f"{vault_path} is not empty. Use --force to overwrite.", as_json)

    try:
        vault_path.mkdir(parents=True, exist_ok=True)
        for d in VAULT_DIRS:
            (vault_path / d).mkdir(parents=True, exist_ok=True)
    except OSError as e:
        _error(f"failed to create vault structure: {e}", as_json)

    today = dt.date.today().isoformat()
    variables = {
        "TOPIC": topic,
        "DATE": today,
        "VAULT_NAME": vault_path.name,
    }

    installed_files = []

    # Schema files (CLAUDE.md / AGENTS.md / .cursorrules)
    for spec in TOOL_FILES.get(tool, TOOL_FILES["claude-code"]):
        src_name, dest_name = spec.split(":", 1)
        dest = vault_path / dest_name
        if render_template(ASSETS_DIR / src_name, dest, variables):
            installed_files.append(dest_name)

    # Index + log seeds
    for spec in [
        ("index.md.template", vault_path / "wiki" / "index.md"),
        ("log.md.template", vault_path / "wiki" / "log.md"),
    ]:
        if render_template(ASSETS_DIR / spec[0], spec[1], variables):
            installed_files.append(str(spec[1].relative_to(vault_path)))

    # Page templates (reference copies inside the vault)
    tmpl_dest = vault_path / "wiki" / ".templates"
    tmpl_dest.mkdir(exist_ok=True)
    src_tmpl = ASSETS_DIR / "page-templates"
    template_count = 0
    if src_tmpl.exists():
        for f in src_tmpl.iterdir():
            if f.is_file():
                try:
                    (tmpl_dest / f.name).write_text(
                        f.read_text(encoding="utf-8"), encoding="utf-8"
                    )
                    template_count += 1
                except OSError as e:
                    print(f"[warn] failed to copy template {f.name}: {e}", file=sys.stderr)

    # .gitignore — exclude Obsidian workspace files
    gitignore = vault_path / ".gitignore"
    gitignore.write_text(
        "\n".join([".obsidian/workspace*", ".obsidian/cache", ".DS_Store", ""]),
        encoding="utf-8",
    )

    result = {
        "status": "ok",
        "vault_path": str(vault_path),
        "topic": topic,
        "tool": tool,
        "date": today,
        "installed_files": installed_files,
        "page_templates_copied": template_count,
        "layers": {
            "raw": "your sources — immutable",
            "wiki": "LLM-maintained knowledge base",
            "index": "wiki/index.md",
            "log": "wiki/log.md",
        },
        "next_steps": [
            "Open the vault in Obsidian",
            "Drop a source into raw/",
            "Run /wiki-ingest <path> in your LLM CLI",
        ],
    }

    if as_json:
        print(json.dumps(result, indent=2))
        return result

    print(f"[ok] Initialized LLM Wiki vault at: {vault_path}")
    print(f"     Topic: {topic}")
    print(f"     Tool: {tool}")
    print(f"     Installed: {', '.join(installed_files)}")
    print(f"     Page templates copied: {template_count}")
    print("     Layers:")
    print("       raw/          (your sources — immutable)")
    print("       wiki/         (LLM-maintained knowledge base)")
    print("       wiki/index.md (catalog)")
    print("       wiki/log.md   (timeline)")
    print()
    print("Next steps:")
    print("  1. Open the vault in Obsidian")
    print("  2. Drop a source into raw/")
    print("  3. Run /wiki-ingest <path> in your LLM CLI")
    return result


def main():
    p = argparse.ArgumentParser(
        description="Initialize an LLM Wiki vault — the three-layer structure (raw/, wiki/, schema) Karpathy describes in the LLM Wiki gist.",
    )
    p.add_argument("--path", required=True, help="Vault directory to create/initialize")
    p.add_argument(
        "--topic",
        required=True,
        help="Short description of what this wiki is about (e.g. 'LLM interpretability')",
    )
    p.add_argument(
        "--tool",
        default="all",
        choices=sorted(TOOL_FILES.keys()),
        help="Which schema file(s) to install (default: all)",
    )
    p.add_argument(
        "--force", action="store_true", help="Overwrite non-empty target directory"
    )
    p.add_argument(
        "--json", action="store_true", help="Emit result as JSON instead of human-readable"
    )
    args = p.parse_args()
    init_vault(
        Path(args.path).expanduser().resolve(),
        args.topic,
        args.tool,
        args.force,
        as_json=args.json,
    )


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
lint_wiki.py — Health-check an LLM Wiki vault.

Surfaces structural problems the LLM-as-wiki-maintainer should fix:

  - orphans:        pages with zero inbound [[wikilinks]]
  - broken_links:   [[wikilinks]] pointing to non-existent pages
  - stale:          pages whose `updated:` frontmatter is older than --stale-days
  - missing_fm:     pages without a title/category/summary in frontmatter
  - duplicate_titles: two or more pages sharing the same title
  - log_gaps:       no log entry in the last --log-gap-days

Usage:
    python lint_wiki.py --vault ~/vaults/research
    python lint_wiki.py --vault . --stale-days 60 --json
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import re
import sys
from collections import defaultdict
from pathlib import Path

FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
WIKILINK_RE = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]*)?(?:\|[^\]]*)?\]\]")
LOG_ENTRY_RE = re.compile(r"^## \[(\d{4}-\d{2}-\d{2})\]", re.MULTILINE)


def parse_frontmatter(text: str) -> dict[str, str]:
    m = FRONTMATTER_RE.match(text)
    if not m:
        return {}
    fm: dict[str, str] = {}
    for line in m.group(1).splitlines():
        if ":" in line and not line.lstrip().startswith("#"):
            k, _, v = line.partition(":")
            fm[k.strip()] = v.strip().strip("'\"")
    return fm


def scan(vault: Path, stale_days: int, log_gap_days: int) -> dict:
    wiki = vault / "wiki"
    if not wiki.exists():
        raise SystemExit(f"[error] {wiki} not found")

    pages: dict[str, dict] = {}
    inbound: dict[str, set[str]] = defaultdict(set)
    outbound: dict[str, set[str]] = defaultdict(set)

    for md in wiki.rglob("*.md"):
        rel = md.relative_to(wiki)
        if rel.name in {"index.md", "log.md"}:
            continue
        if any(part.startswith(".") for part in rel.parts):
            continue
        key = str(rel).replace("\\", "/")[:-3]  # strip .md
        text = md.read_text(encoding="utf-8", errors="replace")
        fm = parse_frontmatter(text)
        pages[key] = {"path": key + ".md", "fm": fm, "text": text}

    # Build link graph
    stems = {Path(k).name: k for k in pages}
    for key, page in pages.items():
        for m in WIKILINK_RE.finditer(page["text"]):
            target = m.group(1).strip()
            # Normalize: strip .md, try full path match first, then stem
            if target.endswith(".md"):
                target = target[:-3]
            if target in pages:
                outbound[key].add(target)
                inbound[target].add(key)
            elif Path(target).name in stems:
                resolved = stems[Path(target).name]
                outbound[key].add(resolved)
                inbound[resolved].add(key)
            else:
                outbound[key].add(f"__BROKEN__:{target}")

    today = dt.date.today()
    stale_cutoff = today - dt.timedelta(days=stale_days)

    orphans = sorted(k for k in pages if not inbound.get(k))
    broken_links: list[tuple[str, str]] = []
    for src, targets in outbound.items():
        for t in targets:
            if t.startswith("__BROKEN__:"):
                broken_links.append((src, t.split(":", 1)[1]))
    broken_links.sort()

    stale: list[tuple[str, str]] = []
    missing_fm: list[str] = []
    titles: dict[str, list[str]] = defaultdict(list)
    for key, page in pages.items():
        fm = page["fm"]
        title = fm.get("title") or Path(key).name
        titles[title].append(key)
        required = {"title", "category", "summary"}
        if not required.issubset(fm.keys()):
            missing_fm.append(key)
        updated = fm.get("updated")
        if updated:
            try:
                d = dt.date.fromisoformat(updated)
                if d < stale_cutoff:
                    stale.append((key, updated))
            except ValueError:
                pass
    duplicate_titles = {t: ks for t, ks in titles.items() if len(ks) > 1}

    # Log gap check
    log_path = wiki / "log.md"
    log_gap = None
    if log_path.exists():
        log_text = log_path.read_text(encoding="utf-8", errors="replace")
        dates = [dt.date.fromisoformat(m) for m in LOG_ENTRY_RE.findall(log_text)]
        if dates:
            last = max(dates)
            gap = (today - last).days
            if gap > log_gap_days:
                log_gap = {"last_entry": last.isoformat(), "days_ago": gap}
        else:
            log_gap = {"last_entry": None, "days_ago": None}

    return {
        "vault": str(vault),
        "total_pages": len(pages),
        "orphans": orphans,
        "broken_links": broken_links,
        "stale": stale,
        "missing_frontmatter": sorted(missing_fm),
        "duplicate_titles": duplicate_titles,
        "log_gap": log_gap,
    }


def print_report(r: dict) -> None:
    print(f"LLM Wiki health check — {r['vault']}")
    print(f"Total pages: {r['total_pages']}")
    print()

    def header(label: str, count: int) -> None:
        sym = "OK" if count == 0 else "WARN"
        print(f"[{sym}] {label}: {count}")

    header("orphan pages", len(r["orphans"]))
    for p in r["orphans"][:20]:
        print(f"   - {p}")
    if len(r["orphans"]) > 20:
        print(f"   ... and {len(r['orphans']) - 20} more")
    print()

    header("broken wikilinks", len(r["broken_links"]))
    for src, tgt in r["broken_links"][:20]:
        print(f"   - {src} -> [[{tgt}]]")
    print()

    header("stale pages", len(r["stale"]))
    for p, d in r["stale"][:20]:
        print(f"   - {p} (updated {d})")
    print()

    header("pages missing frontmatter", len(r["missing_frontmatter"]))
    for p in r["missing_frontmatter"][:20]:
        print(f"   - {p}")
    print()

    header("duplicate titles", len(r["duplicate_titles"]))
    for title, keys in list(r["duplicate_titles"].items())[:10]:
        print(f"   - '{title}': {keys}")
    print()

    gap = r["log_gap"]
    if gap:
        print(f"[WARN] log gap: last entry {gap['last_entry']} ({gap['days_ago']} days ago)")
    else:
        print("[OK] log gap: recent")


def main() -> None:
    p = argparse.ArgumentParser(description="Lint an LLM Wiki vault")
    p.add_argument("--vault", required=True)
    p.add_argument("--stale-days", type=int, default=90)
    p.add_argument("--log-gap-days", type=int, default=14)
    p.add_argument("--json", action="store_true")
    args = p.parse_args()

    report = scan(
        Path(args.vault).expanduser().resolve(),
        stale_days=args.stale_days,
        log_gap_days=args.log_gap_days,
    )
    if args.json:
        print(json.dumps(report, indent=2, default=list))
    else:
        print_report(report)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
update_index.py — Regenerate wiki/index.md from the frontmatter of every wiki page.

The index is content-oriented: a catalog organized by category (entities, concepts,
sources, comparisons, synthesis), with one-line summaries read from each page's
YAML frontmatter.

Frontmatter convention (per page):
    ---
    title: Monosemanticity
    category: concept            # entity | concept | source | comparison | synthesis
    summary: Single-feature interpretability hypothesis from Anthropic's sparse autoencoder work
    tags: [interpretability, sparse-autoencoders]
    sources: 2                   # optional — count of sources referencing this page
    updated: 2026-04-10
    ---

Usage:
    python update_index.py --vault ~/vaults/research
    python update_index.py --vault . --dry-run
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import re
import sys
from collections import defaultdict
from pathlib import Path

FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
CATEGORY_ORDER = ["synthesis", "concept", "entity", "source", "comparison", "other"]
CATEGORY_DIRS = {
    "entities": "entity",
    "concepts": "concept",
    "sources": "source",
    "comparisons": "comparison",
    "synthesis": "synthesis",
}


def parse_frontmatter(text: str) -> dict[str, str]:
    m = FRONTMATTER_RE.match(text)
    if not m:
        return {}
    raw = m.group(1)
    fm: dict[str, str] = {}
    for line in raw.splitlines():
        if ":" in line and not line.lstrip().startswith("#"):
            key, _, value = line.partition(":")
            fm[key.strip()] = value.strip().strip("'\"")
    return fm


def infer_title(path: Path, fm: dict[str, str]) -> str:
    if "title" in fm:
        return fm["title"]
    return path.stem.replace("-", " ").replace("_", " ").title()


def scan_wiki(vault: Path) -> dict[str, list[dict]]:
    wiki = vault / "wiki"
    if not wiki.exists():
        print(f"[error] {wiki} not found", file=sys.stderr)
        sys.exit(1)

    pages: dict[str, list[dict]] = defaultdict(list)
    for md in sorted(wiki.rglob("*.md")):
        # Skip index, log, and template files
        rel = md.relative_to(wiki)
        if rel.name in {"index.md", "log.md"}:
            continue
        if any(part.startswith(".") for part in rel.parts):
            continue

        text = md.read_text(encoding="utf-8", errors="replace")
        fm = parse_frontmatter(text)

        # Category: prefer frontmatter, fall back to folder name
        category = fm.get("category")
        if not category and len(rel.parts) > 1:
            folder = rel.parts[0]
            category = CATEGORY_DIRS.get(folder, "other")
        category = category or "other"

        pages[category].append(
            {
                "path": str(rel).replace("\\", "/"),
                "title": infer_title(md, fm),
                "summary": fm.get("summary", ""),
                "tags": fm.get("tags", ""),
                "sources": fm.get("sources", ""),
                "updated": fm.get("updated", ""),
            }
        )

    # Sort each category by title
    for cat in pages:
        pages[cat].sort(key=lambda p: p["title"].lower())
    return pages


def render_index(pages: dict[str, list[dict]], vault_name: str) -> str:
    today = dt.date.today().isoformat()
    total = sum(len(v) for v in pages.values())

    lines = [
        f"# Index — {vault_name}",
        "",
        f"_Auto-generated {today} • {total} pages_",
        "",
        "> Content-oriented catalog of every page in `wiki/`. Updated by",
        "> `scripts/update_index.py` or during `/wiki-ingest`. Answer queries",
        "> by reading this file first, then drilling into relevant pages.",
        "",
    ]

    for cat in CATEGORY_ORDER:
        entries = pages.get(cat, [])
        if not entries:
            continue
        lines.append(f"## {cat.capitalize()} ({len(entries)})")
        lines.append("")
        for e in entries:
            summary = f" — {e['summary']}" if e["summary"] else ""
            link = f"[[{e['path'][:-3]}|{e['title']}]]"  # Obsidian wikilink, strip .md
            meta = []
            if e["sources"]:
                meta.append(f"{e['sources']} sources")
            if e["updated"]:
                meta.append(f"upd {e['updated']}")
            meta_str = f" _({' · '.join(meta)})_" if meta else ""
            lines.append(f"- {link}{summary}{meta_str}")
        lines.append("")

    return "\n".join(lines)


def main():
    p = argparse.ArgumentParser(
        description="Regenerate wiki/index.md from every wiki page's YAML frontmatter.",
        epilog="The index is organized by category (synthesis, concept, entity, source, comparison).",
    )
    p.add_argument("--vault", required=True, help="Vault root directory")
    p.add_argument(
        "--dry-run", action="store_true", help="Print to stdout instead of writing"
    )
    p.add_argument(
        "--json",
        action="store_true",
        help="Emit a JSON summary of the regeneration result",
    )
    args = p.parse_args()

    try:
        vault = Path(args.vault).expanduser().resolve()
        pages = scan_wiki(vault)
        content = render_index(pages, vault.name)
    except SystemExit:
        raise
    except Exception as e:
        if args.json:
            print(json.dumps({"status": "error", "message": str(e)}))
        else:
            print(f"[error] {e}", file=sys.stderr)
        sys.exit(1)

    total = sum(len(v) for v in pages.values())
    summary = {
        "status": "ok",
        "vault": str(vault),
        "total_pages": total,
        "by_category": {k: len(v) for k, v in pages.items()},
        "dry_run": args.dry_run,
    }

    if args.dry_run:
        if args.json:
            summary["content_preview"] = content[:500]
            print(json.dumps(summary, indent=2))
        else:
            print(content)
        return

    index_path = vault / "wiki" / "index.md"
    try:
        index_path.write_text(content, encoding="utf-8")
    except OSError as e:
        if args.json:
            print(json.dumps({"status": "error", "message": f"failed to write {index_path}: {e}"}))
        else:
            print(f"[error] failed to write {index_path}: {e}", file=sys.stderr)
        sys.exit(1)

    summary["index_path"] = str(index_path)
    if args.json:
        print(json.dumps(summary, indent=2))
    else:
        print(f"[ok] wrote {index_path} ({total} pages)")


if __name__ == "__main__":
    main()