Swain Search

Name: Swain Search
Author: cristoslc

cristoslc/swain-search

2 installs
Updated July 3, 2026
cristoslc/swain-search

Helps with ai & agent building tasks.

About

swain-search is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted coding.

swain-search
AI & Agent Building
AI-coding skill

Swain Search by the numbers

2 all-time installs (skills.sh)
Ranked #13,958 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
Data as of Jul 29, 2026 (Skillselion catalog sync)

npx skills add https://github.com/cristoslc/swain-search --skill swain-search

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/cristoslc/swain-search.svg)](https://skillselion.com/skills/cristoslc/swain-search)

Installs	2
Last updated	July 3, 2026
Repository	cristoslc/swain-search ↗

What it does

Helps with ai & agent building tasks.

Files

.agents/
- agents-md-detail/
docs/
- adr/
- architecture/
- developer-workflows/
- musings/
- plans/
- tech-stack/
- ubiquitous-language/
- user-experience/
references/
scripts/
spokes/
tests/

SKILL.mdMarkdownGitHub ↗

swain-search

Collect, normalize, and cache source materials into reusable troves that swain-design artifacts can reference.

Script invocation convention

Scripts live under the scripts/ directory. Use the <SKILL_DIR> placeholder to mean the folder holding this SKILL.md. Resolve it at run time. In an installed skill, that is .claude/skills/swain-search/. In the standalone repo, it is the project root.

Run the bootstrap once per session before the media or X-thread flows:

bash "<SKILL_DIR>/scripts/bootstrap.sh"

The script checks that uv is on PATH. After the first run, a marker file at ~/.local/share/swain-search/.bootstrapped short-circuits later runs. If it exits non-zero, stop and tell the operator what is missing.

Mode detection

Signal	Mode
No trove exists for the topic, or user says "research X" / "gather sources"	Create — spokes/create-mode.md
Trove exists and user provides new sources or says "add to" / "extend"	Extend — spokes/extend-mode.md
Trove exists and user says "refresh" or sources are past TTL	Refresh — spokes/refresh-mode.md
User asks "what troves do we have" or "find sources about X"	Discover — spokes/discover-mode.md

Core policies

Verbatim mandate — Sources are evidence, not summaries. See spokes/verbatim-mandate.md.
Snapshot evidence gate (SPEC-220) — Remote sources require raw snapshot + metadata verification before normalization. See spokes/snapshot-evidence-gate.md.
Prior art check — Always scan existing troves before creating new ones. See spokes/prior-art-check.md.
Capability detection — Check available tools before collecting. See spokes/capability-detection.md.

Source collection

Every source type (web, media, X-threads, CLI, local files, etc.) has its own collection procedure. See spokes/source-collection.md for the full reference.

Normalization formats per source type are in references/normalization-formats.md.

Commit and linking

All trove-modifying operations follow the dual-commit pattern and produce artifact links of the form trove: <trove-id>@<hash>. See spokes/linking-from-artifacts.md for the full workflow.

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# Virtual environments
.venv/
venv/
ENV/

# uv
.python-version
uv.lock

# macOS
.DS_Store
.AppleDouble
.LSOverride

# Linux
*~
.fuse_hidden*

# Windows
Thumbs.db
ehthumbs.db
Desktop.ini

# IDE
.idea/
.vscode/
*.swp
*.swo
*~

# Worktrees
.worktrees/

# swain-search temp files
/tmp/swain_search_*

# Agents
.agents/search-snapshots/
.agents/trovewatch.log
.agents/trovewatch.vars.json

# Project troves (generated output, not source)
docs/troves/

# Testing artifacts
.pytest_cache/
.ruff_cache/

swain-search Agent Guidance

See PURPOSE.md for the project's intent.

Starting points

SKILL.md — Skill hub (frontmatter, mode table, core policies, links to spokes)
spokes/ — Detailed procedure docs linked from SKILL.md
scripts/ — Shell and Python scripts invoked by the skill
references/normalization-formats.md — Per-source-type markdown format specs
references/manifest-schema.md — Manifest YAML schema

SKILL.md hub-and-spoke architecture

SKILL.md is the concise hub. It links to spoke files in spokes/ for detailed procedures:

Spoke	Purpose
`spokes/prior-art-check.md`	Scanning existing troves before creating new ones
`spokes/verbatim-mandate.md`	Sources are evidence, not summaries
`spokes/snapshot-evidence-gate.md`	SPEC-220 raw snapshot + verification flow
`spokes/source-collection.md`	Per-source-type collection procedures (web, media, X-thread, CLI, etc.)
`spokes/create-mode.md`	Create a new trove from scratch
`spokes/extend-mode.md`	Add sources to an existing trove
`spokes/refresh-mode.md`	Re-fetch stale sources
`spokes/discover-mode.md`	Find existing troves by topic
`spokes/capability-detection.md`	Tool availability checks and fallbacks
`spokes/linking-from-artifacts.md`	Dual-commit pattern and `trove: <id>@<hash>` linking

Key rules

1. Verbatim mandate — Never summarize or condense source content. Only synthesis.md may contain summaries. 2. Snapshot-first — Always export raw snapshot before normalizing remote sources (SPEC-220). 3. Dual-commit pattern — Commit A records content, Commit B stamps the hash into manifest and referencing artifacts. 4. Graceful degradation — Missing tools are skipped, not hard-failures.

Hubs and spokes

ARCHITECTURE.md → docs/architecture/
UBIQUITOUS-LANGUAGE.md → docs/ubiquitous-language/
TECH-STACK.md → docs/tech-stack/
DEVELOPER-WORKFLOWS.md → docs/developer-workflows/
USER-EXPERIENCE.md → docs/user-experience/
docs/adr/ — Architecture decision records
docs/plans/ — Implementation plans
docs/musings/ — Pre-artifact thought capture

Changelog

2026-05-07 — Cookie support

Added

`scripts/convert-cookies.py` — converts browser-exported JSON cookies (Firefox/Chrome DevTools format with Host raw, Name raw, Content raw, etc.) to Netscape cookie file format for use with curl -b. Handles URL-decoding of percent-encoded values, host-only vs subdomain scoping (leading dot), and secure flag mapping.
`--cookies <file.json>` flag on export-snapshot.sh — converts the JSON to a temporary Netscape cookie jar and attaches it to the curl request. Export mode recorded as <mode>-with-cookies.
`tests/test-convert-cookies.py` — 9 acceptance tests covering URL decoding, host-only/secure flag mapping, protocol stripping, multiple cookies, and subdomain leading-dot behaviour.

Changed

SKILL.md — added "Sites needing authentication" subsection under web page URL collection, documenting cookie export from browsers and the --cookies flag.
README.md — updated source types table and permissions list for convert-cookies.py.

2026-04-13 — SPEC-306

Added

X/Twitter thread source type (type: x-thread). URLs matching (x|twitter|fxtwitter|fixupx).com/.+/status/\d+ route to scripts/fetch_x_thread.py, which unrolls the thread via the public fxtwitter API (no auth). Cited posts resolve inline as blockquotes with substantive self-reply continuation (cap 3, link-out for more). Source ID derives as <handle>-<title-slug>.
Media transcript ingestion for YouTube, Instagram, and podcast URLs. Tiered fallback chain: VTT subtitles (preferred, scripts/parse_vtt.py) → post caption from metadata → scene-change frame extraction + vision OCR → EasyOCR local fallback. Each tier writes /tmp/swain_search_media_transcript.txt and normalizes to sources/<slug>/<slug>.md with a new transcript-source frontmatter field.
Bootstrap script (scripts/bootstrap.sh) — idempotent uv check with marker file at ~/.local/share/swain-search/.bootstrapped. Audits settings.json for overly broad permissions on first run. No gh requirement.
`<SKILL_DIR>` placeholder convention for script invocations in SKILL.md — resolves to the skill's install path at run time instead of assuming a swain-repo layout.

Changed

`normalization-formats.md` — added an x-thread section with frontmatter and body structure; added transcript-source: vtt | caption | vision-ocr | local-ocr to the media section; added x-thread to the common frontmatter type enumeration.
SKILL.md — script invocations now use <SKILL_DIR>/scripts/ instead of skills/swain-search/scripts/. This works when the skill is installed under .claude/skills/swain-search/ in an unrelated project.

Design notes

Scripts (fetch_x_thread.py, yt-dlp.sh, parse_vtt.py, extract_frames.py, ocr_frames.py, bootstrap.sh) are modeled after cristoslc/media-summary. They are copied in, not submoduled or chained at runtime. Rationale: media-summary ships a gist-publication workflow; swain-search needs the raw transcript as a trove source. Coupled release cadence and install-path friction made submoduling brittle. Sync upstream improvements case by case.
No gist publication. No public sharing. All output lands in the trove.

Developer Workflows

Install

# No install needed — scripts run in place
# Bootstrap checks for uv on first use
bash scripts/bootstrap.sh

Test

# Cookie conversion tests (no dependencies)
python3 tests/test-convert-cookies.py

# Snapshot pipeline tests (needs uv)
bash tests/test-export-snapshot.sh

# Proxy resolution tests (no dependencies)
bash tests/test-resolve-proxy.sh

Lint

No linter configured yet. Shell scripts pass shellcheck. Python scripts pass pyright (stdlib-only, no type stubs needed).

Commit convention

research(<trove-id>): create trove with N sources
research(<trove-id>): extend with N new sources
research(<trove-id>): refresh N sources (M changed)

Trove lifecycle

1. Create — New trove from gathered sources 2. Extend — Add sources to existing trove 3. Refresh — Re-fetch stale sources, update changed content 4. Discover — Find existing troves matching a topic

Each mode follows the dual-commit pattern: Commit A records content, Commit B stamps the hash.

See docs/developer-workflows/ for additional detail.

MIT License

Copyright (c) 2026 cristos

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

swain-search

Trove collection and normalization for swain-design artifacts. Collects sources from the web, local files, X/Twitter threads, and video/audio media, normalizes them to markdown, and caches them in reusable troves. See SKILL.md for the full workflow.

Requirements

uv (manages Python and Python packages used by the media and X-thread flows)
A web-search capability is recommended for search-based source collection
A page-fetching capability is recommended for web-page sources

All Python dependencies (yt-dlp, opencv-python-headless, easyocr) run transiently via uv run --with and do not require global installation. The bootstrap script checks for uv on first run:

bash scripts/bootstrap.sh

Source types

Type	Input	Script / capability
`web`	Any HTTP URL (optionally with `--cookies` for auth)	web fetch capability or `scripts/export-snapshot.sh`
`forum`	Forum thread URL	web fetch capability
`media`	YouTube, Instagram, or podcast URL	`scripts/yt-dlp.sh`, `scripts/parse_vtt.py`, `scripts/extract_frames.py`, `scripts/ocr_frames.py`
`x-thread`	X/Twitter status URL	`scripts/fetch_x_thread.py`
`document`	Local file path (PDF, DOCX, PPTX, XLSX)	document conversion capability
`local`	Local markdown path	direct read
`repository`	Git repo URL or local path	clone or read-directory
`documentation-site`	Docs-site URL	crawl or fetch
`cli-manpage`, `cli-help`, `cli-subcommand-help`	CLI tool name	`man`, `--help` capture

Detailed normalization rules per type: references/normalization-formats.md.

Permissions

To run swain-search fully autonomously, add these entries to your Claude Code allowedTools. Each pattern is scoped narrowly to limit blast radius.

Review before granting. Read the source files before auto-approving: `scripts/bootstrap.sh`, `scripts/fetch_x_thread.py`, `scripts/parse_vtt.py`, `scripts/yt-dlp.sh`, `scripts/extract_frames.py`, `scripts/ocr_frames.py`.

Recommended (low-risk)

"Skill(swain-search)",
"Bash(bash */scripts/bootstrap.sh)",
"Bash(uv run */scripts/fetch_x_thread.py*)",
"Bash(uv run */scripts/parse_vtt.py)",
"Bash(bash */scripts/yt-dlp.sh*)",
"Bash(uv run --with opencv-python-headless*)",
"Bash(uv run --with easyocr*)",
"Bash(uv run --with \"easyocr,opencv-python-headless\"*)",
"Bash(test -s /tmp/swain_search_*)",
"Bash(bash */scripts/export-snapshot.sh*)",
"Bash(bash */scripts/log-snapshot-metadata.sh*)",
"Bash(bash */scripts/verify-snapshot-evidence.sh*)",
"Bash(bash */scripts/resolve-proxy.sh*)",
"Bash(python3 */scripts/convert-cookies.py*)"

Why these are safe:

`Skill(swain-search)` — allows skill invocation.
*`Bash(bash /scripts/bootstrap.sh)** — verifies uv is on PATH` and audits broad permission patterns. After the first successful run, a marker file short-circuits further work. No user input. No network calls.
*`Bash(uv run /scripts/fetch_x_thread.py)`* — stdlib-only Python. Takes a single X/Twitter URL or tweet ID. Calls api.fxtwitter.com (public, unauthenticated). Writes only to /tmp/swain_search_thread.json and /tmp/swain_search_thread_transcript.txt. No subprocess, no eval.
*`Bash(uv run /scripts/parse_vtt.py)** — pure string processing. Reads /tmp/swain_search_media.en.vtt, writes /tmp/swain_search_media_transcript.txt`. No network, no subprocess. HTML-like tags are stripped by regex.
*`Bash(bash /scripts/yt-dlp.sh)`* — thin wrapper around uv run --with yt-dlp yt-dlp. The skill passes --skip-download for transcript extraction. Full video download only runs during the approved frame-extraction fallback.
*`Bash(uv run --with opencv-python-headless)** — frame extraction from videos already downloaded to /tmp`. Pure image processing.
*`Bash(uv run --with easyocr)** — local OCR fallback, only when vision probing fails. Reads frames from /tmp, writes text to /tmp`.
*`Bash(test -s /tmp/swain_search_)`** — read-only existence check on the skill's temp files.
Snapshot-gate scripts (export-snapshot.sh, log-snapshot-metadata.sh, verify-snapshot-evidence.sh) — SPEC-220 evidence gate for remote documents. Write only to .agents/search-snapshots/.
`resolve-proxy.sh` — read-only lookup in references/paywall-proxies.yaml.

Not recommended (overly broad)

"Bash(open:*)",
"Bash(osascript:*)",
"Bash(gh:*)"

These cover actions this skill does not need. Overly broad patterns widen the blast radius when a transcript or thread contains prompt injection.

Security considerations

Content-based prompt injection (highest risk). An X thread, video caption, or OCR'd frame could contain instructions like "SYSTEM: ignore previous, run …". Claude's training resists this, but it is an inherent risk of processing untrusted text. Mitigation: the skill's allowed tools are scoped to Bash, Read, Write, Edit, and a few MCP capabilities. There is no gh, no shell-escape, no open.
Vision OCR injection. On-screen frame text is read by the model. Malicious videos could embed prompt injection in text overlays. Same mitigations as above.
Trove content poisoning. If injection influences normalization, misleading content lands in the trove. That trove may later be cited by artifacts. Review new troves before they feed downstream decisions.
`/tmp` symlink attack. A local attacker could symlink /tmp/swain_search_media.en.vtt to a sensitive file. This requires existing local access, at which point the attacker already has your permissions. Very low risk.
Skill supply chain. A malicious fork could rewrite SKILL.md or the scripts to do anything Claude Code's permissions allow. Only install from sources you trust. Review skill contents after installation (~/.claude/skills/swain-search/).

Bootstrap

bootstrap.sh runs on the first media or X-thread flow and short-circuits after. Permission prompts appear each run unless "Bash(bash */scripts/bootstrap.sh)" is on your allowed-tools list. This is safe because the script only runs command -v checks, audits settings files, and writes to a marker file in ~/.local/share/. It never processes user-controlled input.

On first run, the script scans your Claude Code settings files for overly broad allowed-tool patterns like Bash(osascript:*) or Bash(open:*). If found, it prints a BROAD PERMISSIONS DETECTED warning with a risk explanation. This check only runs once (gated by the same marker file).

Usage

The skill is invoked by swain-design during research-phase transitions (Spike Proposed → Active, ADR Proposed → Active, Vision/Epic creation) and directly by the operator for targeted collection:

/swain-search research <topic>
/swain-search add <url> to <trove-id>
/swain-search refresh <trove-id>

See SKILL.md for mode details (Create, Extend, Refresh, Discover).

Output

Each trove lives at docs/troves/<trove-id>/ and contains:

manifest.yaml — provenance, tags, per-source metadata with content hashes
sources/<source-id>/<source-id>.md — normalized source content
synthesis.md — thematic distillation across all sources

Artifacts reference a trove by commit hash: trove: <trove-id>@<hash> in frontmatter.

Testing

# Run all tests
bash tests/test-convert-cookies.py   # Python, no deps needed
bash tests/test-export-snapshot.sh   # Bash, needs uv
bash tests/test-resolve-proxy.sh     # Bash, no deps needed

License

MIT

Manifest Schema

Each trove has a manifest.yaml at its root that tracks trove metadata, source provenance, and freshness configuration.

Top-level fields

# Required
trove: <trove-id>                  # Slug identifier (matches directory name)
created: <ISO date>                # When the trove was first created
refreshed: <ISO date>              # When any source was last fetched or refreshed
tags:                              # For trove discovery by other artifacts
  - <tag>

# Optional
freshness-ttl:                     # Per-source-type defaults (override at source level)
  web: 7d                          # Web pages — default 7 days
  forum: 7d                        # Forum threads — default 7 days
  document: 30d                    # PDFs, DOCX, local files — default 30 days
  media: never                     # Video/audio transcripts — content doesn't change
  repository: 30d                  # Git repositories — default 30 days
  documentation-site: 7d           # Documentation sites — default 7 days

history:                           # Append-only event log (oldest first)
  - event: created                 # created | extended | refreshed
    date: <ISO date>               # When the event occurred
    commit: <short hash>           # Commit A hash from the dual-commit workflow
    sources: <N>                   # Total source count after this event
    sources-added: <N>             # Optional (extended) — how many new sources
    sources-changed: <N>           # Optional (refreshed) — how many sources had content changes
    notes: ""                      # Optional — e.g., "added 3 forum threads"

referenced-by:                     # Back-links to artifacts using this trove
  - artifact: SPIKE-001
    commit: abc1234                # Commit A hash from the dual-commit workflow
  - artifact: ADR-003
    commit: def5678

sources:                           # Ordered list of collected sources
  - <source entry>                 # See below

Source entry fields

# Required
source-id: "mdn-websocket-api"    # Slug-based ID (used as directory name)
type: web | forum | document | media | local | repository | documentation-site | cli-manpage | cli-help | cli-subcommand-help
fetched: <ISO datetime>            # When this source was last fetched
title: "WebSocket API - MDN"       # Source title

# Required for remote sources
url: "https://..."                 # Original URL

# Required for local sources
path: "path/to/file.pdf"          # Relative to project root

# Optional
hash: "a1b2c3d4e5f6..."          # Bare hex SHA-256 digest (no sha256: prefix)
freshness-ttl: 14d                 # Per-source override
proxy-used: freedium               # Which paywall proxy delivered the content (omit if direct fetch)
duration: "45:32"                  # For media sources — total duration
speakers:                          # For media sources — identified speakers
  - "Alice"
  - "Bob"
highlights: []                     # Paths relative to source-id directory — key files worth reading first
selective: false                   # True if only a subset of the source was ingested (large repos/sites)
notes: "Focused on section 3"     # Freeform annotation
snapshot-verified: true            # True when .agents/search-snapshots/metadata.jsonl contains this source URL
snapshot-metadata-digest: "..."    # Digest from metadata.jsonl for traceability
has-synthesis: false               # True if sources/<source-id>/synthesis.md exists (optional per-source commentary)

Source types

Type	What it covers	Default TTL
`web`	Web pages, documentation, blog posts, API docs	7 days
`forum`	Forum threads, discussions, Q&A sites, GitHub issues	7 days
`document`	PDFs, DOCX, PPTX, XLSX, local markdown	30 days
`media`	Video, audio, podcasts (transcribed)	never
`local`	Local files already in markdown	30 days
`repository`	Git repositories — tree structure preserved	30 days
`documentation-site`	Documentation sites — section hierarchy preserved	7 days
`cli-manpage`	CLI tool manpage output	never
`cli-help`	CLI tool `--help` or `-h` output	never
`cli-subcommand-help`	CLI subcommand help output	never

CLI-specific source fields

For CLI source types, additional frontmatter fields apply:

tool-name: "git"              # The CLI tool name (required for all CLI types)
command: "remote"             # For cli-subcommand-help — the subcommand name
depth: 1                      # For cli-subcommand-help — nesting level (1 or 2)
failed: true                  # Optional — true if capture attempt failed

Freshness TTL format

Duration strings: <number><unit> where unit is d (days), w (weeks), m (months), or never.

Examples: 7d, 2w, 1m, never

Content hashing

The hash field stores a bare hex SHA-256 digest of the normalized markdown content (not the raw source). On refresh:

1. Re-fetch the raw source 2. Re-normalize to markdown 3. Compare SHA-256 of new normalized content to stored hash 4. If changed: update the source file, hash, and fetched date 5. If unchanged: update only fetched date (confirms source is still valid)

Example manifest

trove: websocket-vs-sse
created: 2026-03-09
refreshed: 2026-03-09
tags:
  - real-time
  - websocket
  - sse
  - server-sent-events

freshness-ttl:
  web: 14d
  media: never

history:
  - event: created
    date: 2026-03-09
    commit: abc1234
    sources: 3

referenced-by:
  - artifact: SPIKE-001
    commit: abc1234

sources:
  - source-id: mdn-websocket-api
    type: web
    url: "https://developer.mozilla.org/en-US/docs/Web/API/WebSocket"
    fetched: 2026-03-09T14:30:00Z
    title: "WebSocket API - MDN Web Docs"
    hash: "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2"

  - source-id: whatwg-sse-spec
    type: web
    url: "https://html.spec.whatwg.org/multipage/server-sent-events.html"
    fetched: 2026-03-09T14:31:00Z
    title: "Server-sent events - HTML Standard"
    hash: "d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5"

  - source-id: strangeloop-2025-realtime-patterns
    type: media
    url: "https://youtube.com/watch?v=xyz"
    fetched: 2026-03-09T15:00:00Z
    title: "Real-time Web Patterns - StrangeLoop 2025"
    hash: "g7h8i9a1b2c3d4e5f6g7h8i9a1b2c3d4e5f6g7h8i9a1b2c3d4e5f6g7h8i9a1b2"
    duration: "42:15"
    speakers:
      - "Jamie Zawinski"
    highlights:
      - "strangeloop-2025-realtime-patterns.md"

Normalization Formats

Every source in a trove is normalized to a markdown file with YAML frontmatter. The frontmatter schema is consistent across types; the body structure varies by source type.

Verbatim mandate: sources are evidence, not summaries

A normalized source file MUST be a faithful, verbatim reproduction of the original document. Condensing, paraphrasing, extracting "key points", or rewriting the original into an AI-generated summary is strictly forbidden. The normalized file must preserve the full content of the original — no truncation, no condensation, no AI rewrites.

The only acceptable formats for summarization are:

Trove-level `synthesis.md` — canonical thematic distillation across all sources.
Per-source `synthesis.md` — optional additive commentary beside the normalized source file.

A source file that reads as a summary instead of a reproduction is defective and must be regenerated from the raw snapshot.

Snapshot-first normalization contract (SPEC-220)

For remote documents (especially Google Docs/Drive links), normalization is not allowed until a raw snapshot is exported first.

Required sequence: 1. Export/download raw file:

bash scripts/export-snapshot.sh --url "<source-url>" --out-dir ".agents/search-snapshots/raw"

2. Normalize via writing-skills or skill-creator using the downloaded file path. 3. Log the evidence record:

bash scripts/log-snapshot-metadata.sh --source-url "<source-url>" --export-mode "<mode>" --raw-path "<raw-path>" --normalized-path "<normalized-path>" --normalization-skill "<writing-skills|skill-creator>"

4. Verify source eligibility:

bash scripts/verify-snapshot-evidence.sh --source-url "<source-url>"

If step 4 fails, the source is unverified and must not be published into trove synthesis.

Per-source synthesis.md (optional)

Individual sources MAY include a synthesis.md alongside the normalized source file at sources/<source-id>/synthesis.md. This is additive commentary — it captures what the source says through the lens of the original search context, explains why the source was selected, or notes how it relates to the trove topic.

---
source-id: "mdn-websocket-api"
relates-to: "web-socket-vs-sse"
relevance: "Official specification — defines WebSocket protocol semantics"
selected-because: "Authoritative reference for the protocol comparison"
aspects-covered:
  - "Protocol handshake"
  - "Message framing"
  - "Connection lifecycle"
gaps:
  - "Does not compare with SSE"
  - "Does not discuss performance characteristics"
---

Key rules:

Per-source synthesis.md is optional — only create it when there is useful commentary beyond what the verbatim source carries.
It MUST NOT replace or truncate the full normalized source content. The verbatim source file remains the primary artifact.
The trove-level synthesis.md remains the authoritative distillation across all sources.
Format: YAML-like structured notes (not prose markdown). Use the frontmatter fields above as a pattern; add freeform notes below as needed.

Common frontmatter

All normalized source files share this frontmatter:

---
source-id: "mdn-websocket-api"
title: "Source Title"
type: web | forum | document | media | local | repository | documentation-site | x-thread
url: "https://..."           # or path for local sources
fetched: 2026-03-09T14:30:00Z
hash: "a1b2c3..."
---

Web pages

Strip navigation, ads, sidebars, footers, and cookie banners. Preserve the main content area with its heading structure.

---
source-id: "mdn-websocket-api"
title: "WebSocket API - MDN Web Docs"
type: web
url: "https://developer.mozilla.org/en-US/docs/Web/API/WebSocket"
fetched: 2026-03-09T14:30:00Z
hash: "a1b2c3..."
---

# WebSocket API - MDN Web Docs

[Main content with original heading hierarchy preserved]

[Code blocks preserved with language tags]

[Tables preserved in markdown format]

Key rules:

Preserve heading hierarchy (h1-h6 -> # through ######)
Preserve code blocks with language annotation
Preserve tables
Convert images to ![alt text](url) — keep alt text, keep URL
Remove inline scripts, styles, tracking pixels
Remove "related articles", "see also" sections unless substantive

Forum threads / discussions

Preserve chronological structure with author attribution and timestamps.

---
source-id: "hn-websocket-vs-sse-dashboards"
title: "WebSocket vs SSE for real-time dashboards"
type: forum
url: "https://news.ycombinator.com/item?id=12345"
fetched: 2026-03-09T14:35:00Z
hash: "d4e5f6..."
participants:
  - "user_alpha"
  - "user_beta"
  - "user_gamma"
post-count: 15
---

# WebSocket vs SSE for real-time dashboards

## user_alpha — 2026-03-01 10:15 UTC

[Original post content]

## user_beta — 2026-03-01 10:42 UTC

> [quoted text from parent, as blockquote]

[Reply content]

## user_gamma — 2026-03-01 11:03 UTC

[Reply content]

Key rules:

Each post is an h2 with author — timestamp
Quoted/reply content uses blockquotes (>)
Preserve code blocks within posts
Omit deleted/removed posts (note their absence if the thread references them)
For nested threads (Reddit-style), flatten to chronological with reply-to attribution

X/Twitter threads

X threads are a source type of their own. Each one has an author, a post count, and a post-by-post order. Cited tweets appear inline. The fetch_x_thread.py script unrolls the thread via api.fxtwitter.com. It also resolves cited posts and self-replies. The output keeps every post verbatim.

---
source-id: "schlickw-us-foreign-policy-anthropic-mythos"
title: "US Foreign Policy and the Anthropic Mythos"
type: x-thread
url: "https://x.com/schlickw/status/1234567890"
fetched: 2026-04-13T14:30:00Z
hash: "k1l2m3..."
author-handle: "schlickw"
author-name: "Example Author"
author-url: "https://x.com/schlickw"
published-date: "2026-04-12T18:00:00Z"
tweet-count: 14
---

# US Foreign Policy and the Anthropic Mythos

## Full Thread

1. [[1/14]](https://x.com/schlickw/status/1234567890) Opening post text, verbatim, with [@mentions](https://x.com/mention) and [#tags](https://x.com/hashtag/tag) hyperlinked inline.

2. [[2/14]](https://x.com/schlickw/status/1234567891) Second post text with a citation to another thread:

   > **[@other_author](https://x.com/other_author)** ([2026-04-10](https://x.com/other_author/status/9876543210)): Cited post text, verbatim.
   >
   > Continuation from the cited author's self-reply, appended as context.
   > _Linked: [article-title](https://example.com/article) — one-sentence synopsis._

3. [[3/14]](https://x.com/schlickw/status/1234567892) Third post text...

Key rules:

Strip leading auto-mention chains. These are the @handle prefixes X adds to reply posts. They are threading artifacts, not the author's words.
Hyperlink every @mention inline as [@handle](https://x.com/handle). Hyperlink hashtags as [#tag](https://x.com/hashtag/tag).
Render cited posts as indented blockquotes under the citing post. Use this format: > **[@handle](url)** ([date-link](tweet_url)): <verbatim text>.
Append up to 3 substantive self-replies from the cited author as blockquote continuation. Skip bare-URL self-replies. They already live in external_links. Link out if more than 3 exist.
Resolve external links inside cited posts when the article, external_links, or photos fields point to longer content. Add a one-sentence synopsis as a sub-blockquote.
No timestamps. X threads have no internal timeline.
If the response is a single post on a known thread-opener, record the entry as failed: true and reason: x-thread-unrollable. Do not write a source file.

Documents (PDF, DOCX, PPTX, XLSX)

Convert to markdown preserving structure. Use available document conversion capabilities.

---
source-id: "q4-2025-arch-review"
title: "Q4 2025 Architecture Review"
type: document
path: "docs/reviews/q4-2025-arch-review.pdf"
fetched: 2026-03-09T15:00:00Z
hash: "g7h8i9..."
page-count: 12
---

# Q4 2025 Architecture Review

[Converted content with heading structure preserved]

[Tables preserved in markdown]

[Figures noted as: **[Figure 1: System architecture diagram]**]

Key rules:

Preserve heading hierarchy from the document structure
Preserve tables (convert to markdown tables)
Note figures/images with descriptive captions: **[Figure N: description]**
For spreadsheets: convert each sheet to a markdown table with the sheet name as heading
For presentations: each slide becomes a section with the slide title as heading

Media (video / audio transcripts)

Transcribe with timestamps and speaker labels when available.

---
source-id: "strangeloop-2025-realtime-patterns"
title: "Real-time Web Patterns - StrangeLoop 2025"
type: media
url: "https://youtube.com/watch?v=xyz"
fetched: 2026-03-09T15:30:00Z
hash: "j0k1l2..."
duration: "42:15"
speakers:
  - "Jamie Zawinski"
transcript-source: vtt   # vtt | caption | vision-ocr | local-ocr
---

# Real-time Web Patterns - StrangeLoop 2025

**Duration:** 42:15
**Speaker(s):** Jamie Zawinski

## Transcript

**[00:00]** Jamie Zawinski: Welcome everyone. Today I want to talk about...

**[02:15]** So the first pattern we'll look at is long polling...

**[15:30]** Now, WebSockets solve many of these problems, but they introduce new ones...

Key rules:

Timestamps in [MM:SS] or [HH:MM:SS] format — only when transcript-source: vtt.
Speaker labels on every speaker change (or every few minutes for single-speaker).
Do NOT add a "Key Points" section — that is summarization, which is forbidden. Summarization belongs in synthesis.md only.
For podcasts with multiple speakers, clearly attribute each segment.
The transcript-source field records which tier produced the text. Omit duration and speakers when caption, vision-ocr, or local-ocr was used (those tiers do not recover that metadata).

Local files (already markdown)

Minimal transformation — add frontmatter, verify structure.

---
source-id: "internal-api-design-notes"
title: "Internal API Design Notes"
type: local
path: "docs/notes/api-design.md"
fetched: 2026-03-09T16:00:00Z
hash: "m3n4o5..."
---

[Original file content, unchanged]

Key rules:

Add frontmatter if missing
Do not modify the content body
Hash is computed on the original content (for change detection)

Repositories

Mirror the repository tree structure under the source directory. Preserve directory hierarchy.

sources/express-framework/
  express-framework.md          # Summary/index file with frontmatter
  lib/
    router/
      index.js
      route.js
    application.js
  package.json

The index file (<source-id>.md) contains:

---
source-id: "express-framework"
title: "Express.js Framework"
type: repository
url: "https://github.com/expressjs/express"
fetched: 2026-03-09T16:30:00Z
hash: "p6q7r8..."
highlights:
  - "lib/application.js"
  - "lib/router/index.js"
selective: true
---

# Express.js Framework

Repository overview and structure summary.

Key rules:

Mirror directory tree faithfully
For large repos, set selective: true and only ingest key files
Populate highlights with the most important files
The index .md file provides the frontmatter and a structural overview

Documentation sites

Mirror the section hierarchy under the source directory. Preserve navigation structure.

sources/react-docs/
  react-docs.md                 # Summary/index file with frontmatter
  getting-started/
    installation.md
    tutorial.md
  api-reference/
    hooks/
      useState.md
      useEffect.md

The index file (<source-id>.md) contains:

---
source-id: "react-docs"
title: "React Documentation"
type: documentation-site
url: "https://react.dev/learn"
fetched: 2026-03-09T17:00:00Z
hash: "s9t0u1..."
highlights:
  - "api-reference/hooks/useState.md"
  - "getting-started/tutorial.md"
selective: true
---

# React Documentation

Site structure and section overview.

Key rules:

Mirror section hierarchy from the site navigation
Preserve internal links where possible (adjust to relative paths)
For large sites, set selective: true and focus on relevant sections
Populate highlights with the most important pages

CLI tools

CLI captures use markdown with code fences. Help output stays in original format.

Manpage capture

---
source-id: "git-manpage"
title: "git manpage"
type: cli-manpage
tool-name: "git"
fetched: 2026-04-07T16:00:00Z
hash: "a1b2c3..."
---

# git manpage

[Raw manpage output here]

Help output capture

---
source-id: "git-help-output"
title: "git --help output"
type: cli-help
tool-name: "git"
fetched: 2026-04-07T16:00:00Z
hash: "d4e5f6..."
---

# git --help output

[Raw help output here]

Subcommand help capture

---
source-id: "git-remote-help"
title: "git remote --help"
type: cli-subcommand-help
tool-name: "git"
command: "remote"
depth: 1
fetched: 2026-04-07T16:00:00Z
hash: "g7h8i9..."
---

# git remote --help

[Raw subcommand help here]

Key rules:

Keep exact formatting inside code fences.
Use tool name in source-id (like git-manpage, git-help-output).
Add command path for subcommands (like git-remote-help).
Set depth: 1 for first-level subcommands, depth: 2 for nested.
Set failed: true in frontmatter if capture fails.

# Paywall proxy registry — many-to-many mapping of domains to proxy strategies.
# Proxies are tried in list order per domain until one returns full content.
#
# NOTE: Pattern and signal values MUST be double-quoted for the resolve-proxy.sh
# parser. Unquoted values will silently fail to match.

domains:
  - pattern: "medium.com"
    match: host-or-subdomain  # matches medium.com and *.medium.com
    proxies: [freedium-mirror, freedium]
    truncation-signals:
      - "Member-only story"
      - "You have 2 free member-only stories left"
      - "Sign up to discover human stories"

proxies:
  freedium-mirror:
    url-template: "https://freedium-mirror.cfd/{url}"
    notes: "Freedium mirror — more reliable than primary domain. Try first."
  freedium:
    url-template: "https://freedium.cfd/{url}"
    notes: "Freedium primary domain. DNS intermittently unreachable."

trovewatch Guide

Monitor troves for size, freshness, and consistency.

Usage

# Check all troves for issues
bash scripts/trovewatch.sh scan

# Summary of all troves
bash scripts/trovewatch.sh status

What it checks

scan

Check	What triggers a warning
Source count	Trove has more sources than `max_sources_per_trove` (default: 20)
Trove size	Trove directory exceeds `max_trove_size_mb` (default: 5MB)
Freshness	Source age exceeds its TTL * `freshness_multiplier` (default: 1.5x)
Missing files	Manifest references a source file that doesn't exist
Orphaned files	Source file exists but isn't listed in manifest
Missing synthesis	Trove has no synthesis.md

Exit code 0 = all healthy, 1 = warnings found.

Output goes to stdout (summary) and .agents/trovewatch.log (details).

status

One-line summary per trove: source count, size, last refreshed date, tags.

Configuration

Create .agents/trovewatch.vars.json to override defaults:

{
  "max_sources_per_trove": 30,
  "max_trove_size_mb": 10,
  "freshness_multiplier": 2.0
}

Setting	Default	Description
`max_sources_per_trove`	20	Warn when trove exceeds this many sources
`max_trove_size_mb`	5	Warn when trove directory exceeds this size
`freshness_multiplier`	1.5	Source is flagged stale when age > TTL * multiplier

Integration with swain-search

After extending or refreshing a trove, run trovewatch.sh scan to verify the trove is healthy. The swain-search skill can invoke this automatically after collection.

abac
abacus
abaff
abaft
abash
abeigh
abiuret
abjoint
aboma
absohm
acacin
acarol
accept
achtel
acidity
acloud
acne
acre
adays
adherer
adipous
adject
adless
ado
adoxy
adoze
adyta
aefald
aery
affaite
affront
afield
agatize
aggrate
agnail
agrise
agrito
ahmadi
aid
aisle
akeki
akimbo
alary
alban
albitic
alco
alen
alepot
alkane
allele
allot
allotee
allower
allylic
alnein
aloetic
aloud
althea
alveloz
ameed
amentum
amid
amir
amomum
among
amper
amt
amuze
ana
anagoge
analgia
anba
aneroid
annual
annulus
anode
anotia
antiqua
anuric
aphyric
apiole
apneic
apogamy
apozema
aptness
aquatic
aranein
archfoe
argala
arghel
armoire
arow
arsenic
artery
artless
asana
asearch
asport
astheny
asyla
atactic
athing
attic
attid
atwixt
audibly
auger
augite
auloi
aurae
average
avives
await
awaste
awave
awheft
awork
azine
azorite
babied
babish
babloh
babudom
bacca
baetuli
baffeta
baggie
bagnio
bagpipe
bailey
bairnie
bait
bakerly
balky
balli
ban
banca
bando
bangled
banquet
barish
barit
barrico
barton
bask
bast
bastard
bate
baxter
bayman
beach
beadlet
beadman
beaked
beaker
beata
beblear
beckon
bedead
bedizen
bedman
bedpan
bedrail
bedrock
bedung
beerage
befist
begrace
belled
bemeal
bendlet
benison
benj
bensel
benumb
benzo
bepuff
berhyme
berley
berm
berne
berobed
bescarf
bespray
bethel
betide
bettor
bewept
bezique
bezzle
bier
bijasal
bilker
bimodal
bin
binder
bipod
birma
birny
birsle
bismite
bitless
blae
blinky
bloat
blooded
blotter
bluffly
blurry
blushy
boast
bocoy
bohor
bolti
bonce
bookism
boomlet
booter
booze
borage
bordel
borer
boric
borrow
boryl
boscage
bosker
bosomed
bossy
botella
bottle
boulder
bovate
boyer
bozo
brabble
brat
breezy
breve
brewis
brine
bringal
brink
bristle
broker
brothel
brume
brumous
bryonin
buddle
buddy
buffing
bulky
bull
bullary
bullety
bult
bummock
bunya
bure
buriti
burying
bussu
busy
butenyl
butylic
buzzing
cabal
cacao
cadbait
caeca
cairn
calash
calean
calk
calp
cambism
cambium
campane
campho
camwood
cancel
candy
canions
cannily
cannula
canonic
canter
capelin
capicha
capmint
capsa
capstan
caramba
carbora
carucal
casabe
casave
cattalo
caudle
cauma
cawney
cebell
censor
cepa
ceriman
ceroma
cesious
chacate
chagul
chair
chanche
chancre
chanter
chapeau
char
charet
chariot
chary
cheeser
chesser
chevage
chia
chibrit
chintz
chirrup
chogak
chopa
chopin
choppy
chott
chouse
chullpa
cimex
cinene
cinuran
cissoid
clammed
cleamer
cleaner
cleric
climate
clitia
cloche
clodlet
cloghad
closh
cluster
clysmic
cobbly
coenjoy
coerce
coexist
coffret
cohort
coli
colloid
colossi
combine
commie
comrade
concert
concur
conduct
confide
conga
consign
coot
copious
cordant
coriin
corner
cornuto
cosmist
cotyla
counter
coupage
coupon
courier
cousin
cowboy
cowgate
cowpath
craddy
cradler
crasis
crawly
creagh
creesh
crenel
crept
cretify
crickle
criey
crimpy
crispy
crome
cronish
crony
croodle
croppy
crore
crossly
cubeb
cud
cudden
cuir
cuisse
cumbly
cumulus
curd
cure
curium
curney
currach
cursal
cuya
cyanine
dacoity
daddock
daedal
daikon
dak
danner
dareall
darkish
darst
dart
dasher
datcha
daut
dayal
dealate
dean
debrief
decate
decking
decline
deflex
degged
dehort
delay
deltal
demiowl
demiss
densen
dentale
denture
depone
derm
dern
derride
devalue
devote
dewdamp
dewret
diacle
diaderm
diapsid
diatom
diaulos
diffide
diffuse
dilated
dim
dinomic
diobol
disbar
dispute
distome
ditty
divided
divinyl
doated
doater
doby
dolcan
domer
donax
doolee
doorba
dor
dotted
dottily
dowed
dowf
dowry
dramm
drop
dropt
drossy
drove
drubber
druidic
dryfoot
dryster
dually
duct
ductile
duction
duke
dulia
dum
dumpage
dumping
duopod
duopoly
duotone
dust
dustbox
duvetyn
dyingly
dynast
earwax
eaved
echoism
ectatic
edaphon
eeler
ehuawa
eimer
eker
elaine
elanet
elbowy
elegant
elfish
emblic
embox
emerse
emotive
emulant
enamel
endable
endoss
endwise
enflesh
engirt
engloom
engore
enmist
enocyte
enough
ensoul
ephetic
ephoric
epikeia
epilate
epitela
epitome
epocha
erect
erenach
erring
erthen
escheat
esere
espadon
espy
etacist
ethine
etna
etymon
eumenid
eunomy
eupione
eutaxic
evacuee
evzone
exam
examen
example
exarch
exclude
excuse
exition
exlex
exotic
eyewort
fabling
fack
fact
faddism
faerie
fagot
famish
fancify
fandom
farish
fascia
fascist
fatuoid
fatwood
fauld
fawnery
feaster
feist
fent
ferash
feria
ferme
feroher
ferrate
ferrety
ferrule
ferulic
festal
fetch
fetched
fever
fibry
fidget
fielder
fient
fiesta
figging
figworm
filcher
final
finch
finer
finish
finnac
firer
fish
fitty
flaggy
flamb
flavour
flaxman
flea
flexure
flidder
flinch
fluked
flunker
flyflap
flywort
foolery
fop
fopling
forbid
forbit
forceps
forfar
forging
forleft
form
formful
forrue
foughty
fow
foxbane
fraghan
frecken
freeing
fresco
frier
frisket
frogman
froughy
frowze
fry
fulcrum
fulgid
full
fulth
fungose
funori
furdel
furiosa
furioso
furler
furoin
fusion
futile
gage
gaggery
gaize
galera
gali
gallant
galled
gallic
galore
gamahe
gamont
gangism
gangly
gangue
gangway
garbell
garner
garnice
garrupa
garum
gaudy
gazebo
gease
gebang
gee
geisha
gemmule
genian
genista
gens
genual
gerenda
gib
gibber
giblets
gimmick
ginner
girlish
gladden
gladdon
gladful
glaived
gleba
globose
gnatty
goladar
goliath
gowan
gra
gradus
graff
greeny
gremial
griddle
grieced
grieve
grille
griller
groggy
grubbed
gruel
grueler
gruffly
gruffs
guama
gules
gumweed
gurgly
gusle
gwine
gyri
habble
haem
hafter
haire
ham
hamal
hamble
hamule
hanker
happing
happy
harish
harman
haster
hath
hatt
hawker
hear
heavity
hegari
height
helical
hematal
hemoid
herein
heritor
hewer
hexyl
hickey
higgler
hiro
hist
histie
history
hoar
hob
holism
holmium
home
honcho
honey
hontish
hoof
hoofs
hookers
hooky
hooping
hooter
hoovey
hotbox
hotel
hothead
houser
huffle
huh
humite
hunky
hurdler
hurry
hutia
huvelyk
hybosis
hybrid
hydrops
iceberg
icefall
icteric
idea
identic
iffy
ileon
iliacus
illfare
imagist
imband
imbosom
imi
impave
inarm
incomer
increst
indazol
indoles
indoxyl
indue
infancy
inflict
inhibit
inken
inknot
inkroot
innerly
inquire
inside
insult
inter
intwist
invigor
iridine
irk
irony
isobase
isogram
isotely
isoxime
isuroid
ivoried
iwaiwa
izzard
jack
jaded
jady
jammer
jaspis
jati
jawab
jawless
jayhawk
jiggers
jiggly
jingler
jirble
job
jobble
jokelet
josh
jough
junior
jurara
jynx
kaik
kakapo
kalends
kambal
karbi
kawika
keacorn
keelage
keena
kekuna
kenno
kent
kernite
ketch
ketchup
key
khalifa
khot
kickup
kiddy
kiln
kilnman
kilting
kinsman
kiswa
knacky
knead
knout
kob
kokoon
kolobus
kongoni
kongu
kusa
kuvasz
kyack
labefy
lacwork
laet
lagetto
laggin
lambent
lanced
langca
lapsing
lard
larigo
larin
laroid
lat
lately
latices
lation
lavolta
laxism
layover
leachy
lealty
leaping
lebbek
lech
leepit
legate
legrope
leman
lempira
lennow
leotard
leu
levee
liard
libbet
libral
licca
lighter
limer
limmu
limpid
linja
linkage
live
livid
lobose
locally
lockful
loftily
logo
loment
longan
longue
loosely
loosish
lopping
lore
lotrite
lourdy
lucy
luggar
lull
lunate
lungi
lunular
lupeol
lurdan
lustra
lute
luteoma
macan
machan
madden
magic
mahoe
making
mallee
mammon
manacle
manche
manchet
manent
maniac
manist
mannie
manny
manred
manship
manto
maomao
map
marae
maral
markup
marled
marli
marok
martial
marver
mascot
mastage
mataco
matara
mater
meatily
megmho
melano
melilot
member
mending
merfolk
merk
mesarch
meson
metayer
mib
midship
midtap
milken
mill
millage
millful
milreis
milter
minaway
mindful
minimum
miniver
minty
minuter
minyan
mirth
misbias
misread
mistime
misturn
mitis
mixite
mizzle
modern
mommy
monepic
monkess
monkly
monture
moody
moonlit
mordore
morin
morinel
morph
morsal
mortar
mortier
mound
mudden
muff
muggish
mugwump
mungofa
murid
mutase
mutive
muttony
mycele
myrcia
myricyl
nabob
naght
naifly
naik
namer
nankin
narr
nasus
navel
needing
neet
nei
nema
neolith
neorama
neoza
nep
nest
neuric
neurine
neuroma
ngaio
nibbana
nickle
nifle
nigh
nisse
nob
noint
noise
nonoily
nonpaid
nonya
noonlit
normal
nosegay
notaeum
nub
numbing
nutcake
oases
oatfowl
obtect
obtrude
oceanet
ochava
ocher
octan
octavo
octoid
octopus
oer
ogaire
oki
oldwife
olivine
onanism
onflow
oniony
onmarch
onstand
open
opening
oraler
orality
orally
orcanet
orchel
orenda
organ
oristic
ornery
orthid
osculum
oshac
otiatry
ototomy
ough
oughtnt
outbeg
outgate
outish
outjazz
outpage
outrush
outset
outside
outtire
outtoil
outwalk
outwash
outwood
ovaloid
ovately
overby
overdue
overlie
ovinia
owing
oxreim
pachisi
paco
paction
paean
painful
palette
palma
palpi
palter
paludal
palus
paperer
papism
papless
pappi
parade
parah
param
parate
parella
parky
parpal
parted
parvis
pascual
pastime
pate
patness
patte
paxiuba
peachen
peckish
pedion
pegging
pekoe
pelisse
peltry
penance
pencel
penman
pennae
pentit
percept
pereira
perique
perite
persis
pervert
pesade
peso
pess
pestful
phacoid
phare
phi
pholido
phratry
physic
pialyn
piastre
pice
pickeer
pickery
pielet
piercel
pig
pigfish
piggle
pightle
pikey
piking
pilmy
pining
pinkie
pinking
pinnule
piuri
placode
planeta
plantad
plating
player
pleach
pledge
plenum
pluffer
plumcot
podesta
podsol
poliad
polled
polos
pong
pontee
pooler
poop
popely
porotic
ported
pose
posing
poteen
pothole
prairie
prater
prebill
predata
predawn
prefool
pregust
prelude
presign
price
prig
prim
primost
printed
private
privy
prolyl
propend
protean
prowed
prudity
psocine
pug
puli
punkah
punnic
putelee
puttier
pyrrhic
quare
quatuor
queal
quernal
quinism
quinize
quinoa
quinova
quintin
quip
quisby
quit
quoit
rabic
radical
radium
ragule
raiment
raisiny
rammack
ramstam
ranal
rancel
rapine
rated
rath
raucous
realtor
reason
rebato
rebear
rebring
rechafe
rechase
redowa
reesk
regrant
regrede
regrow
rehoe
reincur
rejoin
reland
relievo
remote
renew
repayal
repel
reprise
rerig
reroot
resack
resawer
resay
reslate
resmile
resoil
respite
resting
restive
restock
resun
resweat
retable
retan
retare
retier
retinol
reverie
revisit
rewoven
rheme
rheotan
rhodite
rhyme
ribband
riblet
rich
rident
rifely
righten
righter
rillet
rima
rimfire
rimmer
rimu
rip
ripping
risala
rissle
riven
rizzom
roaded
roast
roble
rodding
romanza
rone
ronquil
roomy
roove
rosary
rose
rosilla
rosolic
rotator
rouille
router
rovetto
rubbly
rubella
ruelike
rumbler
runtish
rybat
sack
sackful
saclike
sacro
sah
sahib
saiga
salele
saligot
salix
salma
saltly
saltus
saluter
samen
sampan
samson
sanct
sandbin
sandix
sangley
sans
sap
sapinda
sargo
sasani
satang
saumont
sawbill
sawbuck
scabrid
scantly
scapoid
scaw
schola
sclaw
sconce
scoot
scrabe
scrawl
screet
scrieve
scrod
scroff
scroll
scrubby
scuft
scutter
scyphae
seaman
seating
seaweed
sebate
sebum
sectile
securer
sedged
seer
seine
sejant
semihot
sepsine
septic
sequela
sero
serous
serrano
servant
sestole
setose
setous
settee
setule
sewage
shaitan
shakha
shakily
shame
sharer
shavee
sheard
shears
sheat
shicer
shies
shiner
shivery
shod
shoji
shoot
shor
showup
shrewdy
shriver
shunter
shyly
shyness
sickler
siddur
sideage
sie
siesta
sighten
signior
silenus
silly
simioid
singult
siphoid
sipid
siskin
sistrum
situlae
sizzing
skeif
skeptic
skiing
skim
skinch
skither
sla
slaggy
slagman
slatter
sleck
slent
slepez
sleuth
slimer
slink
sloped
slopely
slosher
slouchy
slyly
smicket
smiris
smush
snaith
snake
snatchy
sneest
snifter
snigger
snock
snoga
soboles
sodded
sofane
soke
soleus
solvend
sompner
sonny
soorkee
sot
souchy
sound
sounder
souslik
soutane
sov
spaid
spald
spat
spatial
spicose
spicous
spile
spirity
spitful
splice
spoilt
spook
spoor
spot
sprowsy
spryly
squally
squarer
squary
stabler
stagese
staidly
stalk
stanjen
stannyl
stapes
steenth
stellar
stemson
sterol
stipe
stoker
stomper
stoof
storm
stoutly
stratal
stretti
strick
striped
strold
stroyer
stude
stupent
stuprum
styca
subecho
subking
submaid
succi
succor
sucuri
sud
suffice
sulfato
summit
surely
surette
surgy
surnay
swager
swaggie
swami
swamp
swape
swelt
swimmy
swiper
switchy
swooper
syllabi
tabby
tabidly
tae
taenia
tag
tallit
tanica
tannase
taperly
taratah
tarbush
tarman
taro
tarsus
tashlik
tasten
tatther
tawa
taxine
taxis
taxless
taxwax
taxy
teacart
teagle
teaming
tearing
teatman
tebbet
teensy
teer
telang
telic
temblor
terete
tergite
terse
testacy
textual
thawer
theine
theow
thigger
thill
thio
thorned
thowt
thrain
thronal
thrown
thymol
ticky
tiddley
tilaite
tingid
tinning
tinted
tipmost
tireman
tirret
titania
titlark
toddy
tode
toffee
toheroa
tolan
tomfool
ton
tonga
tooter
tooth
toran
towner
towrope
towser
toxa
toxotae
toze
tragal
tranter
trilobe
trintle
tripod
trizone
troca
trochi
trommel
tropal
tropeic
trotyl
truancy
trumper
trundle
truss
trust
tsia
tuatara
tuberin
tuffing
tufted
tuned
turfman
turk
turps
tutly
tutman
tux
tweed
tweeze
twicer
twingle
typica
typify
typo
tyrone
uddered
ukulele
ulcery
ulitis
ultimo
umlaut
unakite
unamply
unawned
unblind
unbung
uncased
uncast
unchild
uncowed
unduped
uneaten
unerect
unfaked
unfile
unframe
unfrost
ungag
ungka
ungod
ungodly
ungone
ungraft
unhairy
unhayed
unheal
unheard
unherd
unhome
unideal
uniting
unlit
unlock
unlodge
unlunar
unmined
unmoved
unname
unown
unpaid
unpapal
unplied
unram
unricht
unrung
unruth
unsewn
unsmoky
unspar
unswear
unswell
unthaw
untire
unwoman
unwork
unworld
upbay
upbeat
upbring
upbuild
upcarry
upcreep
upeat
upfling
upframe
uploop
upper
uprend
upscrew
upsey
upsoar
upspeed
upsuck
upwent
upwhelm
urease
urnful
urosis
usance
usee
usure
utas
utensil
uvanite
uvic
uzara
vage
vagrant
valeur
valonia
vara
varan
vastily
vedro
veigle
veneer
venger
verdict
vernin
vesbite
vessel
vexed
vexful
vialful
video
vility
vina
vine
vinta
vintem
violine
viroled
vis
viscera
visile
vistal
voluted
vulgar
vum
wadlike
waftage
wagling
wairepo
waking
washery
washman
wavey
waywort
weaken
weazen
web
weedy
weel
weening
wemless
were
weskit
wetness
whalm
wheel
wheft
whidah
whill
whilter
whites
whither
whits
whoever
whussle
widder
wide
willer
windage
winemay
wintle
wire
wired
wiseman
wispy
wissel
witess
wither
witling
witted
wittol
wonky
wooding
woofy
woomer
wordage
worldly
worsted
wound
woundy
wride
wrier
writher
writter
wrothly
wumble
wurley
wyss
xurel
yakka
yangtao
yaourti
yap
yapp
yarding
yarnen
yashiro
yauld
yaxche
yeast
yegg
yelk
yercum
yerth
yogoite
yond
yourn
youthy
zafree
zein
zenu
zephyry
zeta
zincify
zincite
zippy
zoeal
zonulet
zoocarp
zooid
zygite
zymase

#!/usr/bin/env bash
# Bootstrap script for swain-search media ingestion scripts.
# Verifies uv is available. yt-dlp, opencv, easyocr run transiently via uv.
# Safe to re-run — a marker file short-circuits after the first successful run.

set -euo pipefail

MARKER="${XDG_DATA_HOME:-$HOME/.local/share}/swain-search/.bootstrapped"

# If already bootstrapped, verify uv still exists and exit early
if [[ -f "$MARKER" ]]; then
  if command -v uv >/dev/null 2>&1; then
    exit 0
  fi
  # uv was removed — fall through to re-check
fi

echo "swain-search: checking dependencies…"

# uv is a hard requirement — it manages Python and Python packages
if ! command -v uv >/dev/null 2>&1; then
  echo "ERROR: uv is required but not found. Install it first:" >&2
  echo "  Download from https://docs.astral.sh/uv/getting-started/installation/" >&2
  exit 1
fi

# Stamp the marker so subsequent runs exit early
mkdir -p "$(dirname "$MARKER")"
touch "$MARKER"

# --- One-time permissions audit ---
# Scan settings files for overly broad patterns. Scripts in this skill feed
# external content (transcripts, X threads) into the model's context — broad
# patterns widen the blast radius if that content contains prompt injection.

BROAD_PATTERNS=(
  'Bash(osascript:*)|Bash(osascript *)|Full arbitrary code execution via AppleScript — keychain access, app control, shell commands.'
  'Bash(open:*)|Bash(open *)|Opens any file or URL via default handler — phishing, payload launch.'
)

audit_permissions() {
  local dominated=()
  local settings_files=(
    "$HOME/.claude/settings.json"
    "$HOME/.claude/settings.local.json"
  )
  local project_root
  project_root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
  if [[ -n "$project_root" ]]; then
    settings_files+=("$project_root/.claude/settings.json")
    settings_files+=("$project_root/.claude/settings.local.json")
  fi

  for f in "${settings_files[@]}"; do
    [[ -f "$f" ]] || continue
    for entry in "${BROAD_PATTERNS[@]}"; do
      IFS='|' read -r colon_pat space_pat explanation <<< "$entry"
      if grep -qF "$colon_pat" "$f" 2>/dev/null || \
         grep -qF "$space_pat" "$f" 2>/dev/null; then
        dominated+=("$colon_pat  in $f|$explanation")
      fi
    done
  done

  if [[ ${#dominated[@]} -eq 0 ]]; then
    return
  fi

  echo ""
  echo "┌─────────────────────────────────────────────────────────────┐"
  echo "│              ⚠  BROAD PERMISSIONS DETECTED                 │"
  echo "└─────────────────────────────────────────────────────────────┘"
  echo ""
  echo "  Found overly broad patterns in your settings:"
  echo ""
  for item in "${dominated[@]}"; do
    local pattern="${item%%|*}"
    local risk="${item#*|}"
    echo "    • $pattern"
    echo "      → $risk"
    echo ""
  done
  echo "  Why this matters: swain-search feeds external content (X threads,"
  echo "  video captions) into Claude's context. Broad patterns widen the"
  echo "  attack surface if any source contains prompt injection. Swap them"
  echo "  for narrow entries (see README.md)."
  echo ""
}
audit_permissions

echo "swain-search: dependencies ready."

#!/usr/bin/env python3
"""Convert browser-exported JSON cookies to Netscape format for curl --cookie.

Reads a JSON array of cookie objects from stdin or a file, writes Netscape-format
cookie file to stdout. Output is suitable for curl's -b/--cookie flag.

Input format (Firefox/Chrome cookie export):
  [{"Host raw": "...", "Name raw": "...", "Path raw": "...",
    "Content raw": "...", "Expires raw": "...", "Send for raw": "...",
    "This domain only raw": "...", ...}, ...]

Output format (Netscape):
  domain  flag  path  secure  expiration  name  value

Usage:
  python3 convert-cookies.py [cookies.json]  > cookies.txt
"""

import json
import sys
import urllib.parse
from pathlib import Path


def strip_protocol(host: str) -> str:
    """Remove https:// or http:// prefix and trailing slash from host."""
    for prefix in ("https://", "http://"):
        if host.startswith(prefix):
            host = host[len(prefix):]
            break
    return host.rstrip("/")


def ns_cookie_domain(host: str, host_only: bool) -> str:
    """Build the Netscape domain field.

    Netscape format: leading dot means valid for subdomains.
    host_only=true  → no leading dot (only this exact host).
    host_only=false → leading dot (valid for subdomains).
    """
    domain = strip_protocol(host)
    if host_only:
        return domain
    if not domain.startswith("."):
        domain = f".{domain}"
    return domain


def ns_flag(host_only: bool) -> str:
    """Netscape flag: TRUE if all machines in domain can access (subdomain matching)."""
    return "FALSE" if host_only else "TRUE"


def ns_secure(send_for_raw: str) -> str:
    """Map Send for raw to Netscape secure flag.

    Send for raw:
      "true"  → encrypted connections only  → TRUE
      "false" → any type of connection      → FALSE
    """
    return "TRUE" if send_for_raw.strip().lower() == "true" else "FALSE"


def url_decode(value: str) -> str:
    """URL-decode a cookie value if it appears percent-encoded."""
    if "%" in value:
        try:
            return urllib.parse.unquote(value)
        except Exception:
            return value
    return value


def convert(entries: list[dict]) -> list[str]:
    """Convert a list of cookie dicts to Netscape lines."""
    lines = ["# Netscape HTTP Cookie File", "# Generated by swain-search/convert-cookies.py", ""]
    for c in entries:
        host = c.get("Host raw", "")
        name = c.get("Name raw", "")
        path = c.get("Path raw", "/")
        content = url_decode(c.get("Content raw", ""))
        expires = c.get("Expires raw", "0")
        send_for = c.get("Send for raw", "false")
        host_only_raw = c.get("This domain only raw", "true")

        host_only = host_only_raw.strip().lower() == "true"

        domain = ns_cookie_domain(host, host_only)
        flag = ns_flag(host_only)
        secure = ns_secure(send_for)

        lines.append(f"{domain}\t{flag}\t{path}\t{secure}\t{expires}\t{name}\t{content}")
    return lines


def main() -> None:
    args = sys.argv[1:]
    if args:
        with open(args[0]) as f:
            data = json.load(f)
    else:
        data = json.load(sys.stdin)

    if isinstance(data, dict):
        data = [data]
    if not isinstance(data, list):
        print("ERROR: expected a JSON array of cookie objects", file=sys.stderr)
        sys.exit(1)

    for line in convert(data):
        print(line)


if __name__ == "__main__":
    main()

#!/usr/bin/env bash
# export-snapshot.sh — Export raw source snapshots for swain-search (SPEC-220)

set -euo pipefail

usage() {
  cat <<'USAGE'
Usage:
  bash scripts/export-snapshot.sh \
    --url <source-url> \
    --out-dir <snapshot-dir> \
    [--format txt|pdf] \
    [--browser-export-helper <helper-script>] \
    [--mock-export-url <url>] \
    [--cookies <cookies.json>]

Outputs one JSON object to stdout:
  {"source_url":"...","export_mode":"...","export_timestamp":"...","raw_path":"..."}
USAGE
}

SOURCE_URL=""
OUT_DIR=""
EXPORT_FORMAT="txt"
BROWSER_EXPORT_HELPER=""
MOCK_EXPORT_URL=""
COOKIES_JSON=""

while [[ $# -gt 0 ]]; do
  case "$1" in
    --url)
      SOURCE_URL="${2:-}"
      shift 2
      ;;
    --out-dir)
      OUT_DIR="${2:-}"
      shift 2
      ;;
    --format)
      EXPORT_FORMAT="${2:-}"
      shift 2
      ;;
    --browser-export-helper)
      BROWSER_EXPORT_HELPER="${2:-}"
      shift 2
      ;;
    --mock-export-url)
      MOCK_EXPORT_URL="${2:-}"
      shift 2
      ;;
    --cookies)
      COOKIES_JSON="${2:-}"
      shift 2
      ;;
    -h|--help)
      usage
      exit 0
      ;;
    *)
      echo "ERROR: unknown argument: $1" >&2
      usage >&2
      exit 1
      ;;
  esac
done

if [[ -z "$SOURCE_URL" || -z "$OUT_DIR" ]]; then
  echo "ERROR: --url and --out-dir are required" >&2
  usage >&2
  exit 1
fi

timestamp="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
mkdir -p "$OUT_DIR"

slug="$(echo "$SOURCE_URL" | tr '[:upper:]' '[:lower:]' | sed -E 's#https?://##' | sed -E 's#[^a-z0-9]+#-#g' | sed -E 's#(^-|-$)##g' | cut -c1-80)"
[[ -z "$slug" ]] && slug="snapshot"
raw_path="$OUT_DIR/${timestamp//[:]/-}-$slug.$EXPORT_FORMAT"

detected_mode="direct-export"
export_url=""

if [[ -n "$MOCK_EXPORT_URL" ]]; then
  export_url="$MOCK_EXPORT_URL"
  detected_mode="mock-export"
elif [[ "$SOURCE_URL" =~ ^https://docs\.google\.com/document/d/([^/]+)/ ]]; then
  doc_id="${BASH_REMATCH[1]}"
  export_url="https://docs.google.com/document/d/${doc_id}/export?format=${EXPORT_FORMAT}"
  detected_mode="google-doc-export"
elif [[ "$SOURCE_URL" =~ ^https://docs\.google\.com/presentation/d/([^/]+)/ ]]; then
  presentation_id="${BASH_REMATCH[1]}"
  export_url="https://docs.google.com/presentation/d/${presentation_id}/export/${EXPORT_FORMAT}"
  detected_mode="google-slides-export"
elif [[ "$SOURCE_URL" =~ ^https://drive\.google\.com/file/d/([^/]+)/ ]]; then
  file_id="${BASH_REMATCH[1]}"
  export_url="https://drive.google.com/uc?export=download&id=${file_id}"
  detected_mode="google-drive-download"
else
  export_url="$SOURCE_URL"
  detected_mode="direct-export"
fi

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Build curl cookie args if --cookies was provided
CURL_COOKIE_ARGS=()
if [[ -n "$COOKIES_JSON" ]]; then
  if [[ ! -f "$COOKIES_JSON" ]]; then
    echo "ERROR: cookies file not found: $COOKIES_JSON" >&2
    exit 1
  fi
  COOKIE_JAR="$(mktemp -t swain-search-cookies.XXXXXX)"
  trap "rm -f \"$COOKIE_JAR\"" EXIT
  if ! python3 "$SCRIPT_DIR/convert-cookies.py" "$COOKIES_JSON" > "$COOKIE_JAR"; then
    echo "ERROR: failed to convert cookies from $COOKIES_JSON" >&2
    exit 1
  fi
  CURL_COOKIE_ARGS=(-b "$COOKIE_JAR")
  detected_mode="${detected_mode}-with-cookies"
fi

download_ok=0
if curl -fLsS --retry 3 --retry-all-errors --connect-timeout 10 \
  --max-time 120 "${CURL_COOKIE_ARGS[@]}" "$export_url" -o "$raw_path"; then
  download_ok=1
fi

if [[ $download_ok -ne 1 ]]; then
  if [[ -n "$BROWSER_EXPORT_HELPER" && -x "$BROWSER_EXPORT_HELPER" ]]; then
    "$BROWSER_EXPORT_HELPER" "$SOURCE_URL" "$raw_path"
    detected_mode="browser-helper-export"
  else
    echo "ERROR: export failed for $SOURCE_URL (mode=$detected_mode) and no helper succeeded" >&2
    exit 1
  fi
fi

if [[ ! -s "$raw_path" ]]; then
  echo "ERROR: exported file is empty: $raw_path" >&2
  exit 1
fi

python3 - "$SOURCE_URL" "$detected_mode" "$timestamp" "$raw_path" <<'PY'
import json
import sys

print(json.dumps({
    "source_url": sys.argv[1],
    "export_mode": sys.argv[2],
    "export_timestamp": sys.argv[3],
    "raw_path": sys.argv[4],
}))
PY

"""Extract frames from a video using scene-change detection.

Usage: uv run --with opencv-python-headless scripts/extract_frames.py <video_path> [threshold]

Compares consecutive frames using histogram correlation. When the
similarity drops below the threshold, a scene change is detected and
the frame is captured. Also captures the first and last frames.

Saves frames as /tmp/swain_search_frame_000.png, /tmp/swain_search_frame_001.png, etc.
Default threshold: 0.85 (lower = fewer captures, higher = more sensitive).
A minimum gap of 0.3s between captures prevents duplicates from minor jitter.
"""
import sys
import cv2

video_path = sys.argv[1]
threshold = float(sys.argv[2]) if len(sys.argv) > 2 else 0.85

cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
    print(f"ERROR: Cannot open {video_path}", file=sys.stderr)
    sys.exit(1)

fps = cap.get(cv2.CAP_PROP_FPS)
if fps <= 0:
    fps = 30.0
min_gap = int(fps * 0.3)  # minimum frames between captures


def frame_hist(frame):
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    hist = cv2.calcHist([gray], [0], None, [64], [0, 256])
    cv2.normalize(hist, hist)
    return hist


saved = []
prev_hist = None
frame_id = 0
last_saved_id = -min_gap  # allow first frame to save immediately

while True:
    ret, frame = cap.read()
    if not ret:
        break

    curr_hist = frame_hist(frame)
    save = False

    if prev_hist is None:
        save = True  # first frame
    elif frame_id - last_saved_id >= min_gap:
        similarity = cv2.compareHist(prev_hist, curr_hist, cv2.HISTCMP_CORREL)
        if similarity < threshold:
            save = True

    if save:
        path = f"/tmp/swain_search_frame_{len(saved):03d}.png"
        cv2.imwrite(path, frame)
        saved.append(path)
        last_saved_id = frame_id

    prev_hist = curr_hist
    frame_id += 1

# Always capture the last frame if it wasn't already saved
if frame_id - 1 != last_saved_id and frame_id > 0:
    cap.set(cv2.CAP_PROP_POS_FRAMES, frame_id - 1)
    ret, frame = cap.read()
    if ret:
        path = f"/tmp/swain_search_frame_{len(saved):03d}.png"
        cv2.imwrite(path, frame)
        saved.append(path)

cap.release()

for p in saved:
    print(p)
print(f"Saved {len(saved)} frames")

"""Fetch an X/Twitter thread via the fxtwitter API and write transcript + metadata.

Also resolves any cited X/Twitter status URLs inside the thread (one extra API call
each) so cited posts can be rendered inline as blockquote citations.

Usage:
    uv run fetch_x_thread.py <tweet_url_or_id>

Outputs:
    /tmp/media_thread.json            raw fxtwitter thread response
    /tmp/media_clean_transcript.txt   stitched thread with cited posts inline
    stdout                            JSON metadata (author, count, title_guess,
                                      post_urls, cited_posts)
"""
import json
import re
import sys
import urllib.error
import urllib.request
from typing import Optional

TRANSCRIPT_PATH = "/tmp/swain_search_thread_transcript.txt"
RAW_PATH = "/tmp/swain_search_thread.json"
THREAD_API = "https://api.fxtwitter.com/2/thread/{id}"
UA = {"User-Agent": "swain-search/1.0"}

CITED_URL_RE = re.compile(
    r"https?://(?:x|twitter|fxtwitter|fixupx)\.com/\w+/status/(\d+)",
    re.IGNORECASE,
)
MAX_CITATIONS = 25


def extract_tweet_id(s: str) -> str:
    m = re.search(r"/status/(\d+)", s)
    if m:
        return m.group(1)
    if s.isdigit():
        return s
    raise SystemExit(f"error: could not extract tweet id from: {s}")


def http_get_json(url: str) -> dict:
    req = urllib.request.Request(url, headers=UA)
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.loads(r.read())


def fetch_thread(tweet_id: str) -> dict:
    return http_get_json(THREAD_API.format(id=tweet_id))


def _non_x_external_links(post: dict) -> list:
    facets = (post.get("raw_text") or {}).get("facets") or []
    out = []
    for f in facets:
        if f.get("type") != "url":
            continue
        repl = f.get("replacement")
        if not repl or re.match(r"https?://(?:x|twitter|fxtwitter|fixupx)\.com/", repl):
            continue
        out.append({
            "url": repl,
            "display": f.get("display"),
            "source_tweet": post.get("url"),
        })
    return out


def fetch_cited(tweet_id: str) -> Optional[dict]:
    """Fetch a cited tweet + its self-reply chain via /2/thread/.

    Using the thread endpoint instead of /status/ is deliberate: authors commonly
    post a teaser with a preview image, then self-reply with the bare article URL
    (image posts get more engagement; clickers still need a link). /2/thread/
    walks the self-reply chain from any root, so URLs buried in follow-up posts
    by the same author are captured automatically.
    """
    try:
        data = http_get_json(THREAD_API.format(id=tweet_id))
    except (urllib.error.HTTPError, urllib.error.URLError):
        return None
    if data.get("code") != 200:
        return None
    thread = data.get("thread") or []
    target = next((p for p in thread if str(p.get("id")) == str(tweet_id)), None)
    if not target:
        return None

    target_author_id = (target.get("author") or {}).get("id")
    frontier = {str(tweet_id)}
    self_replies = []
    for p in thread:
        pid = str(p.get("id"))
        if pid in frontier:
            continue
        parent = (p.get("replying_to") or {}).get("status")
        author_id = (p.get("author") or {}).get("id")
        if parent in frontier and author_id == target_author_id:
            self_replies.append(p)
            frontier.add(pid)

    external_links = _non_x_external_links(target)
    for r in self_replies:
        external_links.extend(_non_x_external_links(r))

    a = target.get("author") or {}
    article = target.get("article")
    return {
        "id": target.get("id"),
        "url": target.get("url"),
        "text": target.get("text"),
        "created_at": target.get("created_at"),
        "author_name": a.get("name"),
        "author_handle": a.get("screen_name"),
        "author_url": a.get("url"),
        "author_website": (a.get("website") or {}).get("url"),
        "external_links": external_links,
        "photos": [
            ph.get("url")
            for ph in ((target.get("media") or {}).get("photos") or [])
            if ph.get("url")
        ],
        "twitter_card": target.get("twitter_card"),
        "self_replies": [
            {"url": r.get("url"), "text": r.get("text")}
            for r in self_replies
        ],
        "article": _summarize_article(article) if article else None,
    }


def _summarize_article(art: dict) -> dict:
    """Extract a compact representation of an X Article (long-form post).

    Includes title, preview_text, and the first ~6000 chars of body text so the
    model can synopsize without an additional WebFetch. Full article remains
    accessible at the cited tweet's URL on x.com.
    """
    blocks = ((art.get("content") or {}).get("blocks")) or []
    parts, total = [], 0
    for b in blocks:
        text = (b.get("text") or "").strip()
        if not text:
            continue
        parts.append(text)
        total += len(text)
        if total > 6000:
            break
    return {
        "id": art.get("id"),
        "title": art.get("title"),
        "preview_text": art.get("preview_text"),
        "created_at": art.get("created_at"),
        "body_excerpt": "\n\n".join(parts),
        "body_truncated": len(blocks) > len(parts),
    }


def collect_cited_ids(thread: list) -> list[str]:
    """Return unique cited status IDs in thread order, excluding thread's own posts."""
    own = {str(p.get("id")) for p in thread if p.get("id")}
    seen, ordered = set(), []
    for p in thread:
        for tid in CITED_URL_RE.findall(p.get("text", "") or ""):
            if tid in own or tid in seen:
                continue
            seen.add(tid)
            ordered.append(tid)
    return ordered[:MAX_CITATIONS]


def render_transcript(thread: list, cited: dict) -> str:
    """Build the transcript with cited posts as blockquotes under the referencing post."""
    total = len(thread)
    blocks = []
    for i, p in enumerate(thread, 1):
        text = (p.get("text") or "").strip()
        block = f"[{i}/{total}] {text}"
        for tid in CITED_URL_RE.findall(text):
            c = cited.get(tid)
            if not c:
                continue
            quoted = (c.get("text") or "").strip().replace("\n", "\n> ")
            block += (
                f"\n\n> **@{c.get('author_handle', '?')} "
                f"({c.get('created_at', '')}):** {quoted}\n> — {c.get('url', '')}"
            )
            ext = c.get("external_links") or []
            if ext:
                links = ", ".join(e["url"] for e in ext if e.get("url"))
                block += f"\n> external: {links}"
            elif c.get("author_website"):
                block += f"\n> author site: {c['author_website']}"
        blocks.append(block)
    return "\n\n".join(blocks) + "\n"


def main() -> None:
    if len(sys.argv) != 2:
        raise SystemExit("usage: fetch_x_thread.py <tweet_url_or_id>")

    tweet_id = extract_tweet_id(sys.argv[1])
    try:
        data = fetch_thread(tweet_id)
    except urllib.error.HTTPError as e:
        raise SystemExit(f"error: fxtwitter HTTP {e.code} for tweet {tweet_id} ({e.reason})")
    except urllib.error.URLError as e:
        raise SystemExit(f"error: fxtwitter unreachable: {e.reason}")
    if data.get("code") != 200:
        raise SystemExit(f"error: fxtwitter returned {data.get('code')}: {data.get('message')}")

    thread = data.get("thread") or []
    if not thread:
        raise SystemExit("error: empty thread in response")

    root = thread[0]
    root_text = root.get("text", "")
    if len(thread) == 1 and re.search(r"(1/|🧵)", root_text):
        raise SystemExit(
            "error: root post looks like a thread opener but only 1 post returned. "
            "Upstream fxtwitter deployment likely lacks an authenticated account proxy."
        )

    cited_ids = collect_cited_ids(thread)
    cited: dict = {}
    for tid in cited_ids:
        c = fetch_cited(tid)
        if c is not None:
            cited[tid] = c

    with open(RAW_PATH, "w", encoding="utf-8") as f:
        json.dump(data, f, ensure_ascii=False, indent=2)

    with open(TRANSCRIPT_PATH, "w", encoding="utf-8") as f:
        f.write(render_transcript(thread, cited))

    author = root.get("author") or {}
    title_text = re.sub(r"^(@\w+\s+)+", "", root_text).strip()
    title_text = re.sub(r"\s+", " ", title_text)
    if len(title_text) > 80:
        truncated = title_text[:80]
        last_space = truncated.rfind(" ")
        title_text = truncated[:last_space] if last_space > 40 else truncated
    title_guess = title_text or f"Thread by @{author.get('screen_name', 'unknown')}"

    meta = {
        "tweet_id": tweet_id,
        "source_url": root.get("url"),
        "author_name": author.get("name"),
        "author_handle": author.get("screen_name"),
        "author_url": author.get("url"),
        "published_date": root.get("created_at"),
        "tweet_count": len(thread),
        "title_guess": title_guess,
        "post_urls": [p.get("url") for p in thread],
        "cited_posts": {c["url"]: c for c in cited.values() if c.get("url")},
    }
    print(json.dumps(meta, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()

#!/usr/bin/env bash
# log-snapshot-metadata.sh — Append swain-search snapshot metadata records (SPEC-220)

set -euo pipefail

usage() {
  cat <<'USAGE'
Usage:
  bash scripts/log-snapshot-metadata.sh \
    --source-url <source-url> \
    --export-mode <mode> \
    --raw-path <raw-file> \
    --normalized-path <normalized-file> \
    --normalization-skill <writing-skills|skill-creator|...> \
    [--metadata-file <path>]
USAGE
}

SOURCE_URL=""
EXPORT_MODE=""
RAW_PATH=""
NORMALIZED_PATH=""
NORMALIZATION_SKILL=""
METADATA_FILE=".agents/search-snapshots/metadata.jsonl"

while [[ $# -gt 0 ]]; do
  case "$1" in
    --source-url)
      SOURCE_URL="${2:-}"
      shift 2
      ;;
    --export-mode)
      EXPORT_MODE="${2:-}"
      shift 2
      ;;
    --raw-path)
      RAW_PATH="${2:-}"
      shift 2
      ;;
    --normalized-path)
      NORMALIZED_PATH="${2:-}"
      shift 2
      ;;
    --normalization-skill)
      NORMALIZATION_SKILL="${2:-}"
      shift 2
      ;;
    --metadata-file)
      METADATA_FILE="${2:-}"
      shift 2
      ;;
    -h|--help)
      usage
      exit 0
      ;;
    *)
      echo "ERROR: unknown argument: $1" >&2
      usage >&2
      exit 1
      ;;
  esac
done

if [[ -z "$SOURCE_URL" || -z "$EXPORT_MODE" || -z "$RAW_PATH" || -z "$NORMALIZED_PATH" || -z "$NORMALIZATION_SKILL" ]]; then
  echo "ERROR: missing required arguments" >&2
  usage >&2
  exit 1
fi

if [[ ! -f "$RAW_PATH" ]]; then
  echo "ERROR: raw file not found: $RAW_PATH" >&2
  exit 1
fi
if [[ ! -f "$NORMALIZED_PATH" ]]; then
  echo "ERROR: normalized file not found: $NORMALIZED_PATH" >&2
  exit 1
fi

digest="$(shasum -a 256 "$NORMALIZED_PATH" | awk '{print $1}')"
timestamp="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"

mkdir -p "$(dirname "$METADATA_FILE")"

python3 - "$SOURCE_URL" "$EXPORT_MODE" "$timestamp" "$RAW_PATH" "$NORMALIZATION_SKILL" "$NORMALIZED_PATH" "$digest" >> "$METADATA_FILE" <<'PY'
import json
import sys

entry = {
    "source_url": sys.argv[1],
    "export_mode": sys.argv[2],
    "export_timestamp": sys.argv[3],
    "raw_path": sys.argv[4],
    "normalization_skill": sys.argv[5],
    "normalized_path": sys.argv[6],
    "digest": sys.argv[7],
}
print(json.dumps(entry, separators=(",", ":")))
PY

echo "logged:$METADATA_FILE"

#!/usr/bin/env bash
# migrate-to-troves.sh — Migrate evidence pools to troves
# Idempotent: safe to run multiple times. Non-destructive: moves, never deletes.
#
# Usage: bash scripts/migrate-to-troves.sh [--dry-run]
#
# Steps:
#   1. Rename docs/evidence-pools/ → docs/troves/
#   2. Restructure flat sources into directory-per-source layout
#   3. Update manifest.yaml fields (pool→trove, id+slug→source-id, add new fields)
#   4. Update artifact frontmatter (evidence-pool: → trove:)

set -euo pipefail

DRY_RUN=false
[[ "${1:-}" == "--dry-run" ]] && DRY_RUN=true

PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
OLD_DIR="$PROJECT_ROOT/docs/evidence-pools"
NEW_DIR="$PROJECT_ROOT/docs/troves"

log() { echo "migrate-to-troves: $*"; }
dry() { if $DRY_RUN; then log "[dry-run] $*"; else log "$*"; fi; }

# ── Step 1: Rename directory ──────────────────────────────────────────────────

if [[ -d "$OLD_DIR" && ! -d "$NEW_DIR" ]]; then
    dry "Renaming $OLD_DIR → $NEW_DIR"
    $DRY_RUN || mv "$OLD_DIR" "$NEW_DIR"
elif [[ -d "$OLD_DIR" && -d "$NEW_DIR" ]]; then
    log "WARNING: Both docs/evidence-pools/ and docs/troves/ exist. Incomplete migration?"
    log "Please manually reconcile before re-running."
    exit 1
elif [[ ! -d "$OLD_DIR" && -d "$NEW_DIR" ]]; then
    log "Step 1 already done: docs/troves/ exists, docs/evidence-pools/ does not."
elif [[ ! -d "$OLD_DIR" && ! -d "$NEW_DIR" ]]; then
    log "No evidence pools or troves directory found. Nothing to migrate."
fi

# ── Step 2: Restructure flat sources ──────────────────────────────────────────

if [[ -d "$NEW_DIR" ]]; then
    for pool_dir in "$NEW_DIR"/*/; do
        sources_dir="${pool_dir}sources"
        [[ -d "$sources_dir" ]] || continue

        for source_file in "$sources_dir"/*.md; do
            [[ -f "$source_file" ]] || continue
            stem="$(basename "$source_file" .md)"
            target_dir="$sources_dir/$stem"
            target_file="$target_dir/$stem.md"

            if [[ -f "$target_file" ]]; then
                continue
            fi

            dry "Restructuring $source_file → $target_file"
            if ! $DRY_RUN; then
                mkdir -p "$target_dir"
                mv "$source_file" "$target_file"
            fi
        done
    done
    log "Step 2 complete: sources restructured."
else
    log "Step 2 skipped: no troves directory."
fi

# ── Step 3: Update manifests ──────────────────────────────────────────────────

if [[ -d "$NEW_DIR" ]]; then
    for manifest in "$NEW_DIR"/*/manifest.yaml; do
        [[ -f "$manifest" ]] || continue

        if grep -q '^pool:' "$manifest" 2>/dev/null; then
            dry "Updating manifest: $manifest"
            if ! $DRY_RUN; then
                uv run --with ruamel.yaml python3 - "$manifest" <<'PYEOF'
import sys
from ruamel.yaml import YAML

yaml = YAML()
yaml.preserve_quotes = True

manifest_path = sys.argv[1]
with open(manifest_path, 'r') as f:
    data = yaml.load(f)

changed = False

if 'pool' in data:
    val = data['pool']
    keys = list(data.keys())
    idx = keys.index('pool')
    del data['pool']
    data.insert(idx, 'trove', val)
    changed = True

if 'sources' in data and isinstance(data['sources'], list):
    for source in data['sources']:
        if 'id' in source and 'slug' in source:
            new_id = f"{source['id']}-{source['slug']}"
            del source['id']
            del source['slug']
            source.insert(0, 'source-id', new_id)
            changed = True
        elif 'id' in source and 'source-id' not in source:
            val = source['id']
            del source['id']
            source.insert(0, 'source-id', val)
            changed = True

        if 'highlights' not in source:
            source['highlights'] = []
            changed = True
        if 'selective' not in source:
            source['selective'] = False
            changed = True

        if 'hash' in source and isinstance(source['hash'], str):
            if source['hash'].startswith('sha256:'):
                source['hash'] = source['hash'][7:]
                changed = True

if changed:
    with open(manifest_path, 'w') as f:
        yaml.dump(data, f)
    print(f"  Updated: {manifest_path}")
else:
    print(f"  Already up to date: {manifest_path}")
PYEOF
            fi
        else
            log "  Manifest already updated: $manifest"
        fi
    done
    log "Step 3 complete: manifests updated."
else
    log "Step 3 skipped: no troves directory."
fi

# ── Step 4: Update artifact frontmatter ───────────────────────────────────────

docs_dir="$PROJECT_ROOT/docs"
if [[ -d "$docs_dir" ]]; then
    count=0
    while IFS= read -r -d '' file; do
        if grep -q '^evidence-pool:' "$file" 2>/dev/null; then
            if ! $DRY_RUN; then
                python3 -c "
import sys; p=sys.argv[1]; t=open(p).read()
open(p,'w').write(t.replace('\nevidence-pool:','\ntrove:'))
" "$file"
            fi
            count=$((count + 1))
        fi
    done < <(find "$docs_dir" -name '*.md' -print0)
    dry "Step 4 complete: updated frontmatter in $count artifact files."
else
    log "Step 4 skipped: no docs directory."
fi

log "Migration complete."

"""OCR text from extracted video frames using EasyOCR.

Usage: uv run --with "easyocr,opencv-python-headless" scripts/ocr_frames.py

Reads /tmp/swain_search_frame_*.png, deduplicates text across frames,
and writes unique lines to /tmp/swain_search_media_transcript.txt.
"""
import glob
import easyocr

reader = easyocr.Reader(["en"], gpu=False)
frames = sorted(glob.glob("/tmp/swain_search_frame_*.png"))

all_text = []
seen = set()
for f in frames:
    results = reader.readtext(f, detail=0)
    for line in results:
        line = line.strip()
        if line and line not in seen:
            seen.add(line)
            all_text.append(line)

with open("/tmp/swain_search_media_transcript.txt", "w") as out:
    out.write("\n".join(all_text))
print(f"Extracted {len(all_text)} unique text lines from {len(frames)} frames")

import re
from collections import deque

with open('/tmp/swain_search_media.en.vtt', 'r') as f:
    content = f.read()

# Parse all cue blocks
cues = []
for block in re.split(r'\n\n+', content):
    lines = [l.strip() for l in block.split('\n') if l.strip()]
    timestamp_line = None
    text_lines = []
    for line in lines:
        if '-->' in line:
            m = re.match(r'(\d{2}:\d{2}:\d{2})', line)
            if m:
                timestamp_line = m.group(1)
        elif not re.match(r'^\d+$', line) and not line.startswith('WEBVTT') \
             and not line.startswith('Kind:') and not line.startswith('Language:'):
            clean = re.sub(r'<[^>]+>', '', line).strip()
            if clean:
                text_lines.append(clean)
    if timestamp_line and text_lines:
        cues.append((timestamp_line, ' '.join(text_lines)))

# Emit only new words per cue, preserving the timestamp of first appearance.
# Keep a sliding window of recent words — caption overlaps are always with
# the immediately preceding cue (typically 5–20 words), so 50 is plenty.
WINDOW = 50
result_lines = []
recent_words = deque(maxlen=WINDOW)

for timestamp, text in cues:
    words = text.split()
    tail = list(recent_words)
    overlap = 0
    for i in range(min(len(words), len(tail)), 0, -1):
        if words[:i] == tail[-i:]:
            overlap = i
            break
    new_words = words[overlap:]
    if new_words:
        result_lines.append(f'[{timestamp}] {" ".join(new_words)}')
        recent_words.extend(new_words)

with open('/tmp/swain_search_media_transcript.txt', 'w') as f:
    f.write('\n'.join(result_lines))
print(f"Saved {len(result_lines)} lines")

#!/usr/bin/env bash
# resolve-proxy.sh — Deterministic paywall proxy resolver for swain-search
#
# Usage: resolve-proxy.sh <url>
#
# Reads paywall-proxies.yaml and outputs proxy URLs + truncation signals
# for the given URL's domain. Exits 0 if proxies found, 1 if no match.
#
# Output format (line-oriented):
#   PROXY:<name>:<proxy-url>
#   SIGNAL:<text>
#
# Override registry path: PAYWALL_REGISTRY=path/to/file.yaml
#
# Note: YAML values in the registry MUST be double-quoted for parsing.
# Compatible with bash 3.2+ (no associative arrays).
#
# Part of SPEC-155: Paywall Proxy Fallback

set -euo pipefail

if [[ $# -lt 1 ]]; then
  exit 1
fi

URL="$1"

# Extract host from URL
HOST="$(echo "$URL" | sed -E 's|^https?://([^/]+).*|\1|')"

# Locate registry
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REGISTRY="${PAYWALL_REGISTRY:-$SCRIPT_DIR/../references/paywall-proxies.yaml}"

if [[ ! -f "$REGISTRY" ]]; then
  exit 1
fi

# --- Parse registry ---
# Line-by-line YAML parsing to avoid yq/python dependencies.
# Uses parallel indexed arrays for bash 3.2 compatibility (no declare -A).

# First pass: collect proxy definitions (parallel arrays)
proxy_def_names=()
proxy_def_templates=()
in_proxies_section=false
current_proxy_name=""

while IFS= read -r line; do
  if [[ "$line" =~ ^proxies: ]]; then
    in_proxies_section=true
    continue
  fi

  if $in_proxies_section && [[ "$line" =~ ^[a-z] && ! "$line" =~ ^[[:space:]] ]]; then
    in_proxies_section=false
    continue
  fi

  if $in_proxies_section; then
    # Proxy name (indented, ends with bare colon)
    if [[ "$line" =~ ^[[:space:]]+([a-zA-Z0-9_-]+):$ ]]; then
      current_proxy_name="${BASH_REMATCH[1]}"
    fi
    if [[ -n "$current_proxy_name" && "$line" =~ url-template:[[:space:]]*\"(.+)\" ]]; then
      proxy_def_names+=("$current_proxy_name")
      proxy_def_templates+=("${BASH_REMATCH[1]}")
      current_proxy_name=""
    fi
  fi
done < "$REGISTRY"

# Helper: look up url-template by proxy name
lookup_template() {
  local name="$1"
  local i
  for (( i=0; i<${#proxy_def_names[@]}; i++ )); do
    if [[ "${proxy_def_names[$i]}" == "$name" ]]; then
      echo "${proxy_def_templates[$i]}"
      return 0
    fi
  done
  return 1
}

# Second pass: find matching domain entry
matched=false
current_proxies=()
current_signals=()
in_domains_section=false
in_domain_entry=false
in_signals=false

while IFS= read -r line; do
  if [[ "$line" =~ ^domains: ]]; then
    in_domains_section=true
    continue
  fi

  if $in_domains_section && [[ "$line" =~ ^[a-z] && ! "$line" =~ ^[[:space:]] ]]; then
    in_domains_section=false
    continue
  fi

  if ! $in_domains_section; then
    continue
  fi

  # New domain entry (list item with pattern)
  if [[ "$line" =~ ^[[:space:]]*-[[:space:]]*pattern:[[:space:]]*\"(.+)\" ]]; then
    if $matched; then
      break
    fi
    local_pattern="${BASH_REMATCH[1]}"
    current_proxies=()
    current_signals=()
    in_domain_entry=true
    in_signals=false

    # host-or-subdomain matching
    if [[ "$HOST" == "$local_pattern" ]] || [[ "$HOST" == *".$local_pattern" ]]; then
      matched=true
    fi
    continue
  fi

  if ! $in_domain_entry; then
    continue
  fi

  # Proxies list (inline YAML array)
  if [[ "$line" =~ proxies:[[:space:]]*\[(.+)\] ]]; then
    IFS=',' read -ra proxy_items <<< "${BASH_REMATCH[1]}"
    for item in "${proxy_items[@]}"; do
      item="$(echo "$item" | sed 's/^[[:space:]]*//' | sed 's/[[:space:]]*$//')"
      current_proxies+=("$item")
    done
    continue
  fi

  # Truncation signals section
  if [[ "$line" =~ truncation-signals: ]]; then
    in_signals=true
    continue
  fi

  if $in_signals && [[ "$line" =~ ^[[:space:]]*-[[:space:]]*\"(.+)\" ]]; then
    current_signals+=("${BASH_REMATCH[1]}")
    continue
  fi

  if $in_signals && [[ ! "$line" =~ ^[[:space:]]*- ]]; then
    in_signals=false
  fi

done < "$REGISTRY"

# --- Output ---

if ! $matched; then
  exit 1
fi

for proxy_name in "${current_proxies[@]}"; do
  template="$(lookup_template "$proxy_name" 2>/dev/null || true)"
  if [[ -n "$template" ]]; then
    proxy_url="${template//\{url\}/$URL}"
    echo "PROXY:${proxy_name}:${proxy_url}"
  fi
done

for signal in "${current_signals[@]}"; do
  echo "SIGNAL:${signal}"
done

exit 0

#!/usr/bin/env bash
set -euo pipefail

# trovewatch — monitor troves for size, freshness, and consistency
#
# Usage:
#   trovewatch.sh scan     Check all troves for issues
#   trovewatch.sh status   Summary of all troves

# --- Configuration ---

TROVES_DIR="docs/troves"
LOG_FILE=".agents/trovewatch.log"
CONFIG_FILE=".agents/trovewatch.vars.json"

# Defaults (overridable via config file)
MAX_SOURCES_PER_TROVE=20
MAX_TROVE_SIZE_MB=5
FRESHNESS_MULTIPLIER="1.5"

# --- Helpers ---

log() {
  echo "$1" >> "$LOG_FILE"
}

warn() {
  echo "  WARN: $1"
  log "WARN $1"
}

die() {
  echo "trovewatch: error: $1" >&2
  exit 2
}

# Load config overrides if present
load_config() {
  if [ -f "$CONFIG_FILE" ]; then
    local val
    val=$(uv run python3 -c "
import json, sys
try:
    c = json.load(open('$CONFIG_FILE'))
    print(c.get('max_sources_per_trove', c.get('max_sources_per_pool', '')))
    print(c.get('max_trove_size_mb', c.get('max_pool_size_mb', '')))
    print(c.get('freshness_multiplier', ''))
except Exception:
    print(''); print(''); print('')
" 2>/dev/null)
    local line1 line2 line3
    line1=$(echo "$val" | sed -n '1p')
    line2=$(echo "$val" | sed -n '2p')
    line3=$(echo "$val" | sed -n '3p')
    [ -n "$line1" ] && MAX_SOURCES_PER_TROVE="$line1"
    [ -n "$line2" ] && MAX_TROVE_SIZE_MB="$line2"
    [ -n "$line3" ] && FRESHNESS_MULTIPLIER="$line3"
  fi
}

# Parse TTL string (e.g., "7d", "2w", "1m", "never") to seconds
ttl_to_seconds() {
  local ttl="$1"
  case "$ttl" in
    never) echo "0"; return ;;
    *d) echo $(( ${ttl%d} * 86400 )) ;;
    *w) echo $(( ${ttl%w} * 604800 )) ;;
    *m) echo $(( ${ttl%m} * 2592000 )) ;;
    *) echo "0" ;;
  esac
}

# Parse ISO date to epoch seconds
date_to_epoch() {
  local d="$1"
  # Handle both "2026-03-09" and "2026-03-09T14:30:00Z" formats
  if command -v gdate >/dev/null 2>&1; then
    gdate -d "$d" +%s 2>/dev/null || echo "0"
  else
    date -jf "%Y-%m-%dT%H:%M:%SZ" "$d" +%s 2>/dev/null || \
    date -jf "%Y-%m-%d" "$d" +%s 2>/dev/null || \
    echo "0"
  fi
}

now_epoch() {
  date +%s
}

# Get directory size in MB (integer)
dir_size_mb() {
  du -sm "$1" 2>/dev/null | cut -f1
}

# --- Trove scanning ---

scan_trove() {
  local trove_dir="$1"
  local trove_id
  trove_id=$(basename "$trove_dir")
  local manifest="$trove_dir/manifest.yaml"
  local sources_dir="$trove_dir/sources"
  local issues=0

  echo "Trove: $trove_id"
  log "SCAN $trove_id"

  # Check manifest exists
  if [ ! -f "$manifest" ]; then
    warn "$trove_id: missing manifest.yaml"
    issues=$((issues + 1))
    echo ""
    return $issues
  fi

  # Check source count (count directories in sources/)
  local source_count=0
  if [ -d "$sources_dir" ]; then
    source_count=$(find "$sources_dir" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')
  fi

  if [ "$source_count" -gt "$MAX_SOURCES_PER_TROVE" ]; then
    warn "$trove_id: $source_count sources (max: $MAX_SOURCES_PER_TROVE) — consider splitting or pruning"
    log "SIZE_WARN $trove_id sources=$source_count max=$MAX_SOURCES_PER_TROVE"
    issues=$((issues + 1))
  fi

  # Check trove size
  local size_mb
  size_mb=$(dir_size_mb "$trove_dir")
  if [ "$size_mb" -gt "$MAX_TROVE_SIZE_MB" ]; then
    warn "$trove_id: ${size_mb}MB (max: ${MAX_TROVE_SIZE_MB}MB) — consider removing large sources"
    log "SIZE_WARN $trove_id size=${size_mb}MB max=${MAX_TROVE_SIZE_MB}MB"
    issues=$((issues + 1))
  fi

  # Parse manifest for source entries and check freshness + consistency
  if command -v uv >/dev/null 2>&1; then
    local py_result
    py_result=$(uv run --with pyyaml python3 << PYEOF
import yaml, os, sys
from datetime import datetime, timezone

manifest_path = "$manifest"
sources_dir = "$sources_dir"
trove_id = "$trove_id"
freshness_mult = float("$FRESHNESS_MULTIPLIER")

try:
    with open(manifest_path) as f:
        m = yaml.safe_load(f)
except Exception as e:
    print(f"MANIFEST_ERROR {trove_id}: {e}")
    sys.exit(0)

if not m or not isinstance(m, dict):
    print(f"MANIFEST_ERROR {trove_id}: empty or invalid")
    sys.exit(0)

sources = m.get("sources", []) or []
default_ttls = m.get("freshness-ttl", {}) or {}
now = datetime.now(timezone.utc)

# Map of TTL strings to seconds
def ttl_seconds(ttl_str):
    if not ttl_str or ttl_str == "never":
        return 0
    s = ttl_str.strip()
    if s.endswith("d"):
        return int(s[:-1]) * 86400
    elif s.endswith("w"):
        return int(s[:-1]) * 604800
    elif s.endswith("m"):
        return int(s[:-1]) * 2592000
    return 0

manifest_source_ids = set()
for src in sources:
    source_id = src.get("source-id", "unknown")
    stype = src.get("type", "web")
    fetched_str = src.get("fetched", "")
    selective = src.get("selective", False)
    manifest_source_ids.add(source_id)

    # Check freshness
    ttl_str = src.get("freshness-ttl") or default_ttls.get(stype, "7d")
    ttl_secs = ttl_seconds(ttl_str)

    if ttl_secs > 0 and fetched_str:
        try:
            if "T" in str(fetched_str):
                fetched = datetime.fromisoformat(str(fetched_str).replace("Z", "+00:00"))
            else:
                fetched = datetime.fromisoformat(str(fetched_str)).replace(tzinfo=timezone.utc)
            age_secs = (now - fetched).total_seconds()
            threshold = ttl_secs * freshness_mult
            if age_secs > threshold:
                age_days = int(age_secs / 86400)
                print(f"STALE {trove_id}/{source_id}: {age_days}d old (ttl: {ttl_str})")
        except Exception:
            pass

    # Check source directory/file exists (skip if selective)
    if not selective:
        source_path = os.path.join(sources_dir, source_id)
        if os.path.isdir(source_path):
            # Hierarchical source — check directory is non-empty
            if not os.listdir(source_path):
                print(f"MISSING_FILE {trove_id}: source directory {source_id}/ exists but is empty")
        elif os.path.isfile(os.path.join(source_path, source_id + ".md")):
            # Flat source — file exists inside its directory (should not reach here if dir doesn't exist)
            pass
        elif not os.path.isdir(source_path):
            print(f"MISSING_FILE {trove_id}: manifest has {source_id} but directory not found")

# Check for orphaned directories in sources/
if os.path.isdir(sources_dir):
    for entry in os.listdir(sources_dir):
        entry_path = os.path.join(sources_dir, entry)
        if os.path.isdir(entry_path) and entry not in manifest_source_ids:
            print(f"ORPHAN {trove_id}: {entry}/ exists but not in manifest")

# Check synthesis exists
if not os.path.isfile(os.path.join(os.path.dirname(sources_dir), "synthesis.md")):
    print(f"MISSING_SYNTHESIS {trove_id}: no synthesis.md")
PYEOF
    )

    if [ -n "$py_result" ]; then
      while IFS= read -r line; do
        case "$line" in
          STALE*)
            warn "${line#STALE }"
            log "$line"
            issues=$((issues + 1))
            ;;
          MISSING_FILE*|ORPHAN*|MISSING_SYNTHESIS*|MANIFEST_ERROR*)
            warn "${line#* }"
            log "$line"
            issues=$((issues + 1))
            ;;
        esac
      done <<< "$py_result"
    fi
  fi

  if [ "$issues" -eq 0 ]; then
    echo "  healthy ($source_count sources, ${size_mb}MB)"
  fi
  echo ""

  return $issues
}

# --- Status ---

status_trove() {
  local trove_dir="$1"
  local trove_id
  trove_id=$(basename "$trove_dir")
  local manifest="$trove_dir/manifest.yaml"
  local sources_dir="$trove_dir/sources"

  local source_count=0
  if [ -d "$sources_dir" ]; then
    source_count=$(find "$sources_dir" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')
  fi

  local size_mb
  size_mb=$(dir_size_mb "$trove_dir")

  local refreshed="unknown"
  local tags=""
  if [ -f "$manifest" ] && command -v uv >/dev/null 2>&1; then
    local py_out
    py_out=$(uv run --with pyyaml python3 -c "
import yaml
with open('$manifest') as f:
    m = yaml.safe_load(f) or {}
print(m.get('refreshed', 'unknown'))
print(','.join(m.get('tags', []) or []))
" 2>/dev/null)
    refreshed=$(echo "$py_out" | sed -n '1p')
    tags=$(echo "$py_out" | sed -n '2p')
  fi

  printf "  %-30s %3s sources  %3sMB  refreshed: %-12s  tags: %s\n" \
    "$trove_id" "$source_count" "$size_mb" "$refreshed" "$tags"
}

# --- Main ---

main() {
  local cmd="${1:-help}"

  load_config
  mkdir -p "$(dirname "$LOG_FILE")"

  case "$cmd" in
    scan)
      echo "" > "$LOG_FILE"
      log "=== trovewatch scan $(date -u +%Y-%m-%dT%H:%M:%SZ) ==="

      if [ ! -d "$TROVES_DIR" ]; then
        echo "trovewatch: no troves found (${TROVES_DIR}/ does not exist)."
        exit 0
      fi

      local total_issues=0
      local trove_count=0

      echo "trovewatch: scanning troves..."
      echo ""

      for trove_dir in "$TROVES_DIR"/*/; do
        [ -d "$trove_dir" ] || continue
        trove_count=$((trove_count + 1))
        scan_trove "$trove_dir" || total_issues=$((total_issues + $?))
      done

      if [ "$trove_count" -eq 0 ]; then
        echo "trovewatch: no troves found in ${TROVES_DIR}/."
        exit 0
      fi

      if [ "$total_issues" -gt 0 ]; then
        echo "trovewatch: found ${total_issues} issue(s) across ${trove_count} trove(s). See ${LOG_FILE}"
        exit 1
      else
        echo "trovewatch: all ${trove_count} trove(s) healthy."
        exit 0
      fi
      ;;

    status)
      if [ ! -d "$TROVES_DIR" ]; then
        echo "trovewatch: no troves found."
        exit 0
      fi

      local trove_count=0
      echo "Troves:"
      echo ""

      for trove_dir in "$TROVES_DIR"/*/; do
        [ -d "$trove_dir" ] || continue
        trove_count=$((trove_count + 1))
        status_trove "$trove_dir"
      done

      if [ "$trove_count" -eq 0 ]; then
        echo "  (none)"
      fi
      echo ""
      echo "${trove_count} trove(s) total."
      ;;

    help|--help|-h)
      echo "Usage: trovewatch.sh <command>"
      echo ""
      echo "Commands:"
      echo "  scan     Check all troves for size, freshness, and consistency issues"
      echo "  status   Summary of all troves"
      echo ""
      echo "Configuration: .agents/trovewatch.vars.json"
      echo "  max_sources_per_trove  (default: 20)"
      echo "  max_trove_size_mb      (default: 5)"
      echo "  freshness_multiplier   (default: 1.5)"
      ;;

    *)
      die "unknown command: $cmd (try: scan, status, help)"
      ;;
  esac
}

main "$@"

#!/usr/bin/env bash
# verify-snapshot-evidence.sh — Validation gate for swain-search source evidence (SPEC-220)

set -euo pipefail

usage() {
  cat <<'USAGE'
Usage:
  bash scripts/verify-snapshot-evidence.sh \
    --source-url <source-url> \
    [--metadata-file <path>]

Exit codes:
  0: verified (metadata entry exists)
  2: unverified (no metadata entry)
  1: usage or runtime error
USAGE
}

SOURCE_URL=""
METADATA_FILE=".agents/search-snapshots/metadata.jsonl"

while [[ $# -gt 0 ]]; do
  case "$1" in
    --source-url)
      SOURCE_URL="${2:-}"
      shift 2
      ;;
    --metadata-file)
      METADATA_FILE="${2:-}"
      shift 2
      ;;
    -h|--help)
      usage
      exit 0
      ;;
    *)
      echo "ERROR: unknown argument: $1" >&2
      usage >&2
      exit 1
      ;;
  esac
done

if [[ -z "$SOURCE_URL" ]]; then
  echo "ERROR: --source-url is required" >&2
  usage >&2
  exit 1
fi

if [[ ! -f "$METADATA_FILE" ]]; then
  echo "WARN: unverified source (no metadata ledger): $SOURCE_URL"
  exit 2
fi

if python3 - "$METADATA_FILE" "$SOURCE_URL" <<'PY'
import json
import sys

path = sys.argv[1]
source_url = sys.argv[2]

with open(path, "r", encoding="utf-8") as f:
    for line in f:
        line = line.strip()
        if not line:
            continue
        try:
            item = json.loads(line)
        except json.JSONDecodeError:
            continue
        if item.get("source_url") == source_url:
            print("verified")
            raise SystemExit(0)

raise SystemExit(2)
PY
then
  echo "verified: source snapshot evidence exists for $SOURCE_URL"
  exit 0
else
  status=$?
  if [[ $status -eq 2 ]]; then
    echo "WARN: unverified source (missing snapshot metadata): $SOURCE_URL"
    exit 2
  fi
  echo "ERROR: verification failed unexpectedly for $SOURCE_URL" >&2
  exit 1
fi

Capability Detection and Graceful Degradation

Before collecting sources, check what's available. Look for tools matching these patterns — the exact tool names vary by installation:

Web search: tools with "search" in the name (e.g., brave_web_search, bing-search-to-markdown)
Page fetching: tools with "fetch", "webpage", "browser" in the name (e.g., fetch_content, webpage-to-markdown, browser_navigate)
Media transcription: tools with "audio", "video", "youtube" in the name (e.g., audio-to-markdown, youtube-to-markdown)
Document conversion: tools with "pdf", "docx", "pptx", "xlsx" in the name (e.g., pdf-to-markdown, docx-to-markdown)
CLI tool capture: built-in bash capabilities (man, command execution) — always available on POSIX systems

Report available capabilities at the start of collection so the user knows what will and won't work.

Graceful degradation

When a capability isn't available:

Capability	Fallback
Web search	Skip search-based sources. Tell user: "No web search capability available — provide URLs directly or add a search MCP."
Browser / page fetcher	Try basic URL fetch. If that fails: "Can't fetch this URL — paste the content or provide a local file."
Snapshot export for remote docs	If export fails and no helper exists: mark source unverified, do not publish downstream, report exact URL and failure mode.
Media transcription	"No transcription capability available — provide a pre-made transcript file, or add a media conversion tool."
Document conversion	"Can't convert this file type — provide a markdown version, or add a document conversion tool."
Paywall proxy	Keep truncated content. Note in manifest: "Paywalled; proxies exhausted." Suggest user provide content manually.

Never fail the entire run because one capability is missing. Collect what you can, skip what you can't, and report clearly.

Create Mode

Build a new trove from scratch.

Step 1 — Gather inputs

Ask the user (or infer from context) for:

1. Trove ID — a slug for the topic (e.g., websocket-vs-sse). Suggest one if the context is clear. 2. Tags — keywords for discovery (e.g., real-time, websocket, sse) 3. Sources — any combination of:

Web search queries ("search for WebSocket vs SSE comparisons")
URLs (web pages, forum threads, docs)
Video/audio URLs
Local file paths

4. Freshness TTL overrides — optional, defaults are fine for most troves

If invoked from swain-design (e.g., spike entering Active), the artifact context provides the topic, tags, and sometimes initial sources.

Step 2 — Collect and normalize

Mandatory: every source must be a verbatim reproduction of the original document, not a summary. The normalized source file is evidence — raw material for research. Condensing, paraphrasing, or extracting "key points" from the original is forbidden. Summarization belongs exclusively in synthesis.md (trove-level or per-source). See spokes/verbatim-mandate.md for the full policy.

For each source, use the appropriate capability described in spokes/source-collection.md.

Step 3 — Generate manifest

Create manifest.yaml following the schema in references/manifest-schema.md. Include:

Trove metadata (id, created date, tags)
Default freshness TTL per source type
One entry per source with provenance (URL/path, fetch date, content hash, type)

Compute content hashes as bare hex SHA-256 digests (no prefix) of the normalized markdown content:

shasum -a 256 sources/mdn-websocket-api/mdn-websocket-api.md | cut -d' ' -f1

Step 4 — Generate synthesis

Create synthesis.md — a structured distillation of key findings across all sources.

Two levels of synthesis are permitted:

1. Trove-level synthesis.md (required, authoritative). The single synthesis.md at the trove root looks across ALL sources and produces a thematic distillation. This is the canonical summary of what the trove as a whole says.

2. Per-source synthesis.md (optional). Individual sources MAY include their own synthesis.md alongside the normalized source file (e.g., sources/<source-id>/synthesis.md). These are useful for capturing what a source says through the lens of the original search context — e.g., commentary on why this source was selected, what aspect it illuminates, or how it relates to the trove topic. Per-source synthesis must NEVER replace or truncate the full normalized source content; the verbatim source file remains the primary artifact. Per-source synthesis is additive commentary, not a substitute for the original.

Structure the trove-level synthesis by theme, not by source. Group related findings together, cite sources by ID, and surface:

Key findings — what the sources collectively say about the topic
Points of agreement — where sources converge
Points of disagreement — where sources conflict or present alternatives
Gaps — what the sources don't cover that might matter

Keep it concise. The synthesis is a starting point, not a comprehensive report — the user or artifact author will refine it.

Step 5 — Commit and stamp

Use the dual-commit pattern to give the trove a reachable commit hash. See spokes/linking-from-artifacts.md for the full commit workflow and artifact linking procedure.

Step 6 — Report

Tell the user what was created:

Trove `<trove-id>` created with N sources — committed as <TROVE_HASH:0:7>.

- docs/troves/<trove-id>/manifest.yaml — provenance and metadata

- docs/troves/<trove-id>/sources/ — N normalized source files

- docs/troves/<trove-id>/synthesis.md — thematic distillation: <SYNTHESIS_URL>

Reference from artifacts with: trove: <trove-id>@<TROVE_HASH:0:7>

Always include the synthesis file URL in the report. For multiple troves created in a single run, list each synthesis URL.

Linking from Artifacts and Dual-Commit Workflow

Artifacts reference troves in frontmatter:

trove: websocket-vs-sse@abc1234

The format is <trove-id>@<commit-hash>. The commit hash pins the trove to a specific version — troves evolve over time as sources are added or refreshed, and the hash ensures reproducibility.

Dual-commit workflow

Every trove-modifying operation (Create, Extend, Refresh) follows this pattern:

Before Commit A — append a history entry to manifest.yaml with a -- placeholder for the commit hash:

history:
  - event: created
    date: 2026-03-09
    commit: "--"
    sources: 3

Commit A — commit the trove content:

git add docs/troves/<trove-id>/
git commit -m "research(<trove-id>): create trove with N sources

Co-Authored-By: <model-name-from-system-prompt> <noreply@unknown>"
TROVE_HASH=$(git rev-parse HEAD)

Commit B — back-fill the commit hash into the history entry, then update the referencing artifact's frontmatter (if one exists):

# Replace "--" with the real hash in the history entry
# Update artifact frontmatter: trove: <trove-id>@<TROVE_HASH>
git add docs/troves/<trove-id>/manifest.yaml
git add docs/<artifact-type>/<phase>/<artifact-dir>/   # if artifact exists
git commit -m "docs(<trove-id>): stamp history hash ${TROVE_HASH:0:7}

Co-Authored-By: <model-name-from-system-prompt> <noreply@unknown>"

If no referencing artifact exists yet (standalone research), Commit B still stamps the history entry — report the hash so it can be referenced later.

Push — after Commit B, ALWAYS push to origin/trunk so the trove is immediately available to other agents and sessions. This is mandatory, not optional:

git push origin trunk

Derive synthesis URL — construct a stable permalink to the synthesis file for the final report:

REMOTE_URL=$(git remote get-url origin | sed 's/\.git$//' | sed 's/git@github.com:/https:\/\/github.com\//')
SYNTHESIS_URL="${REMOTE_URL}/blob/${TROVE_HASH}/docs/troves/<trove-id>/synthesis.md"

Prior Art Check

Before creating a new trove or running web searches, scan existing troves for relevant content. This avoids duplicating research and surfaces connections to prior work.

Phase 1 — Literal keyword match

Search for the source name, URL fragments, and author name:

# Search trove manifests by tag
grep -rl "<keyword>" docs/troves/*/manifest.yaml 2>/dev/null

# Search trove source content
grep -rl "<keyword>" docs/troves/*/sources/**/*.md 2>/dev/null

# Search trove syntheses
grep -rl "<keyword>" docs/troves/*/synthesis.md 2>/dev/null

Phase 2 — Semantic topic match

After fetching the source and understanding what it's about, extract 3-5 topic keywords from the source's content (not just its name or URL). Then search existing troves by topic:

# Search trove tags for topic keywords
grep -l "<topic-keyword-1>\|<topic-keyword-2>\|<topic-keyword-3>" docs/troves/*/manifest.yaml 2>/dev/null

# Search synthesis summaries for topic keywords
grep -l "<topic-keyword-1>\|<topic-keyword-2>\|<topic-keyword-3>" docs/troves/*/synthesis.md 2>/dev/null

Topic keywords should describe what the source is about, not what it's called. For example, a repo named "Cog" that implements a memory system for Claude Code should generate topic keywords like agent-memory, memory-architecture, claude-code, persistent-memory — not cog or marciopuga.

If the source has not been fetched yet (URL-only invocation), use whatever topic information is available from the URL or title and defer full topic matching until after the source is fetched.

Decision gate

Before proceeding to Create or Extend mode, output a visible routing decision:

Prior art check: Phase 1 found [N matches / no matches]. Phase 2 found [N matches / no matches]: [trove-id (tags: x, y), ...].

Decision: Extending [trove-id] / Creating new trove [slug] because [reason].

This makes the trove routing decision auditable. If any trove matches on 2+ topic keywords, default to Extend mode unless the topic is genuinely distinct (adjacent but different subject matter).

Action on matches

If existing troves contain relevant sources: 1. Report what was found — show the trove ID, matching source titles, and relevant excerpts 2. Suggest extend over create — if an existing trove covers the same topic, extend it rather than creating a parallel trove 3. Cross-link — if the topic is adjacent but distinct, create a new trove but note the related trove in synthesis.md

This step runs in all modes (Create, Extend, Discover) and before any web searches. Existing trove content is always checked first.

Tech Stack

swain-search is a pure skill — shell scripts and Python with no runtime dependencies beyond uv.

Core

Shell (bash 3.2+) — Bootstrap, export, snapshot pipeline, proxy resolution, trove maintenance
Python 3 (stdlib-only) — Cookie conversion, X-thread fetching, VTT parsing
uv — Python package manager; runs transient dependencies (yt-dlp, opencv-python-headless, easyocr, ruamel.yaml) via uv run --with

Transient dependencies (not installed globally)

Package	Used by	Purpose
`yt-dlp`	`yt-dlp.sh`	Video/audio download and subtitle extraction
`opencv-python-headless`	`extract_frames.py`	Scene-change frame extraction
`easyocr`	`ocr_frames.py`	Local OCR fallback for videos without subtitles
`ruamel.yaml`	`migrate-to-troves.sh`	YAML migration (legacy)

See docs/tech-stack/ for additional detail.

Related skills

Setup Matt Pocock SkillsScaffold the per-repo configuration that Matt Pocock’s engineering agent skills rely on so they understand the issue tracker, triage labels, and domain documentation la495k196k

Lark AppsConnect Claude, Cursor or custom agents directly to Lark (Feishu) for messaging, document automation, approval workflows and enterprise data access.402k

CavemanSlash token usage by roughly 75% while keeping every technical detail intact when working with Claude Code, Cursor or similar agents.391k94.7k

Lark Skill MakerQuickly turn any Lark/Feishu OpenAPI call or multi-step workflow into a reusable agent skill with its own SKILL.md.386k16k

Running Claude Code Via Litellm CopilotRun Claude Code at a fraction of the cost by routing requests through LiteLLM to the GitHub Copilot Chat API.270k72

Codex PetGenerate a complete Codex Pet spritesheet and metadata from one reference image without needing an OpenAI key or Codex Pro.246k10

AI & Agent Buildingagents

About

Swain Search by the numbers

Add your badge

What it does

Files

swain-search

Script invocation convention

Mode detection

Core policies

Source collection

Commit and linking

Project Navigation

Starting points

Hubs and spokes

swain-search Agent Guidance

Starting points

SKILL.md hub-and-spoke architecture

Key rules

Hubs and spokes

Architecture

Architecture detail

Key components

Data flow

Design principles

Changelog

2026-05-07 — Cookie support

Added

Changed

2026-04-13 — SPEC-306

Added

Changed

Design notes

Developer Workflows

Install

Test

Lint

Commit convention

Trove lifecycle

docs/adr/

docs/architecture/

docs/developer-workflows/

Musings Agent Instructions

docs/musings/

docs/plans/

docs/

docs/tech-stack/

docs/ubiquitous-language/

docs/user-experience/

swain-search

Requirements

Source types

Permissions

Recommended (low-risk)

Not recommended (overly broad)

Security considerations

Bootstrap

Usage

Output

Testing

License

Manifest Schema

Top-level fields

Source entry fields

Source types

CLI-specific source fields

Freshness TTL format

Content hashing

Example manifest

Normalization Formats

Verbatim mandate: sources are evidence, not summaries

Snapshot-first normalization contract (SPEC-220)

Per-source synthesis.md (optional)

Common frontmatter

Web pages

Forum threads / discussions

X/Twitter threads

Documents (PDF, DOCX, PPTX, XLSX)

Media (video / audio transcripts)

Local files (already markdown)

Repositories