Wow Digest

email:
  # GWS query to match newsletters (excludes marketing caught by email-rules)
  query: "newer_than:1d -category:promotions -category:social"
  max_results: 100

telegram:
  channels:
    - "macOS/iOS & AI"
    - "Записки злого терапевта"
    - "DEKSDEN (chat)"
    - "Точки над ИИ"
    - "@hegai AI Tools / Assistants / Agents"
    - "DEKSDEN notes"
    - "Екатерина Шульман"
    - "Поиск доказательной помощи."
    - "Запятые"
    - "СВП | Сообщество выпускников Психодемии"
    - "out of homeland"
    - "Freiberufler werden (самозанятость и бизнес в 🇩🇪)"
    - "CHAOS"
    - "CI in Berlin & around"
    - "Syntony Dialogue"
  # Excluded (own channels): КЛОДКОТ, AGENCY: Community, Опытным путём
  days: 1
  limit_per_channel: 50

You are a content curator filtering for EPISTEMIC FRICTION — items that destabilize current beliefs, connect previously unconnected domains, or introduce something unpredictable.

WOW ≠ excitement. WOW ≠ relevance. WOW = "this changes how I think about something."

## User Context

FOCUS AREAS:
{focus}

RECENT RESEARCH (last 30 days):
{research}

RECENT DAILY TOPICS (last 7 days):
{recent_topics}

## Candidates

{candidates}

## Instructions

For each candidate, produce a JSON object with these fields:

- title: the candidate's title
- source: source name
- source_type: email/telegram
- wow_score: integer 1-10, computed as min(relevance, 8) + surprise + bridge_value - predictability, normalized to 1-10
- relevance: 1-10 (capped at 8 in the formula — highly relevant but predictable should NOT win)
- surprise: 1-10 (does this contradict or complicate known beliefs?)
- bridge_value: 1-10 (does it connect two usually separate domains?)
- predictability: 1-10 (penalty — could you guess this from the subject line alone?)
- challenged_assumption: specific belief this destabilizes (if you can't name one, score is max 4)
- why_not_obvious: why you couldn't have written this hook before reading the content
- connected_recent_note: which focus area or recent research topic this touches
- hook: one-line hook for the daily note explaining WHY it's surprising (must be specific enough that it couldn't apply to any other article)

CRITICAL RULES:
1. If you cannot state a specific challenged_assumption, the item scores max 4.
2. If the hook could have been written without reading the article, reject and rewrite it.
3. Hype language does NOT equal surprise. "Groundbreaking new AI" is not WOW. "AI agents framed as liability containers rather than productivity tools" IS WOW.
4. Contrarian ≠ surprising. Provocative but low-quality takes score max 3.

HOOK QUALITY — STRICT ENFORCEMENT:
- BAD hooks (generic, could apply to anything): "Diverse topics can provide surprising insights that overlap in unexpected ways" / "Circadian rhythms could hold keys to mental health" / "State violence intersects with personal tragedy"
- GOOD hooks (specific, tells you the actual insight): "tDCS devices under $200 now replicate 80% of clinical brain stimulation results at home — the gap between lab and consumer is closing faster than regulation" / "Aakash built a full team standup-to-retro pipeline in Claude Code that replaces 4 separate SaaS tools — the OS layer is eating point solutions"
- TEST: If your hook works as a tweet that would make someone stop scrolling, it's specific enough. If it reads like an abstract, rewrite it.

Output a JSON array of scored candidates. Include ALL candidates, not just top ones — the selection logic runs separately.

pyyaml>=6.0

# External CLI tools required (not pip-installable):
# - gws: Google Workspace CLI (~/.claude/skills/gws/)
# - llm: Simon Willison's llm CLI (pip install llm)
# - telegram-telethon: tg.py script (~/.claude/skills/telegram-telethon/)

#!/usr/bin/env python3
"""Enrich candidates with thin snippets by fetching linked content via Firecrawl.

Runs between ingestion and salience filter. Detects candidates where the snippet
is too short (just a redirect link or empty) and fetches the actual article content.
"""
import argparse
import json
import os
import re
import sys
import time
from pathlib import Path

try:
    import requests
except ImportError:
    print("Error: requests not installed. Run: pip install requests", file=sys.stderr)
    sys.exit(1)

FIRECRAWL_API_KEY = os.environ.get("FIRECRAWL_API_KEY", "")
FIRECRAWL_URL = "https://api.firecrawl.dev/v1/scrape"
MIN_SNIPPET_LENGTH = 150
MAX_FETCHES_PER_RUN = 15
FETCH_DELAY = 1.0


def extract_url(text):
    """Extract the first HTTP(S) URL from text."""
    match = re.search(r'https?://[^\s<>"\')\]]+', text)
    return match.group(0) if match else None


def is_thin(candidate):
    """Check if a candidate has insufficient content for scoring."""
    snippet = candidate.get("snippet", "")
    # Strip common noise patterns
    clean = re.sub(r'https?://[^\s]+', '', snippet)
    clean = re.sub(r'Read on LinkedIn|Read this article|View this post on the web', '', clean)
    clean = re.sub(r'\s+', ' ', clean).strip()
    return len(clean) < MIN_SNIPPET_LENGTH


def fetch_content(url):
    """Fetch article content via Firecrawl scrape API."""
    if not FIRECRAWL_API_KEY:
        return None

    try:
        resp = requests.post(
            FIRECRAWL_URL,
            headers={
                "Authorization": f"Bearer {FIRECRAWL_API_KEY}",
                "Content-Type": "application/json",
            },
            json={
                "url": url,
                "formats": ["markdown"],
                "onlyMainContent": True,
            },
            timeout=30,
        )
        if resp.status_code == 200:
            data = resp.json()
            markdown = data.get("data", {}).get("markdown", "")
            return markdown[:1500] if markdown else None
        else:
            print(f"  Firecrawl {resp.status_code} for {url[:60]}", file=sys.stderr)
            return None
    except Exception as e:
        print(f"  Firecrawl error: {e}", file=sys.stderr)
        return None


def enrich_candidates(candidates):
    """Enrich thin candidates with fetched content."""
    enriched_count = 0
    fetch_count = 0

    for candidate in candidates:
        if not is_thin(candidate):
            continue

        if fetch_count >= MAX_FETCHES_PER_RUN:
            print(f"  Hit fetch limit ({MAX_FETCHES_PER_RUN}), stopping", file=sys.stderr)
            break

        # Find a URL to fetch
        url = extract_url(candidate.get("snippet", ""))
        if not url:
            url = candidate.get("url", "")
        if not url:
            continue

        # Skip non-article URLs
        if any(skip in url for skip in ["zoom.us", "meet.google", "cal.com", "luma.com/event/manage"]):
            continue

        # LinkedIn is behind auth — mark as needs-click, don't waste a fetch
        if "linkedin.com" in url:
            candidate["snippet"] = f"[LinkedIn article — content behind auth wall. Source is notable.] {candidate.get('snippet', '')}"
            candidate["needs_click"] = True
            enriched_count += 1
            continue

        print(f"  Fetching: {url[:70]}...", file=sys.stderr)
        content = fetch_content(url)
        fetch_count += 1

        if content and len(content) > MIN_SNIPPET_LENGTH:
            candidate["snippet"] = content[:1000]
            candidate["enriched"] = True
            enriched_count += 1

        time.sleep(FETCH_DELAY)

    return enriched_count


def main():
    parser = argparse.ArgumentParser(description="Enrich thin candidates with fetched content")
    parser.add_argument("--input", "-i", required=True, help="Input JSONL candidates")
    parser.add_argument("--output", "-o", required=True, help="Output JSONL (enriched)")
    args = parser.parse_args()

    candidates = []
    with open(args.input) as f:
        for line in f:
            if line.strip():
                candidates.append(json.loads(line))

    thin_count = sum(1 for c in candidates if is_thin(c))
    print(f"Enriching: {len(candidates)} candidates, {thin_count} have thin snippets", file=sys.stderr)

    enriched_count = enrich_candidates(candidates)
    print(f"  Enriched {enriched_count}/{thin_count} thin candidates", file=sys.stderr)

    with open(args.output, "w") as f:
        for c in candidates:
            f.write(json.dumps(c) + "\n")


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Collect human feedback on WOW digest items for eval gold set."""
import argparse
import json
import sys
from datetime import datetime
from pathlib import Path

EVAL_DIR = Path(__file__).parent.parent / ".wow-eval"


def record_feedback(selected_path, verdicts):
    """Append feedback entries to feedback.jsonl.

    verdicts: list of dicts with keys: title, verdict, reason (optional)
    verdict values: "wow", "meh", "noise", "already_knew"
    """
    EVAL_DIR.mkdir(parents=True, exist_ok=True)
    feedback_path = EVAL_DIR / "feedback.jsonl"

    with open(selected_path) as f:
        selected = json.load(f)

    with open(feedback_path, "a") as f:
        for item, verdict_info in zip(selected, verdicts):
            entry = {
                "timestamp": datetime.now().isoformat(),
                "title": item.get("title", ""),
                "source": item.get("source", ""),
                "source_type": item.get("source_type", ""),
                "wow_score_llm": item.get("wow_score", 0),
                "wow_score_human": verdict_info.get("score"),
                "verdict": verdict_info.get("verdict", ""),
                "reason": verdict_info.get("reason", ""),
                "challenged_assumption": item.get("challenged_assumption", ""),
                "hook": item.get("hook", ""),
            }
            f.write(json.dumps(entry) + "\n")

    count = sum(1 for _ in open(feedback_path))
    print(f"Feedback recorded. Total entries: {count}", file=sys.stderr)
    if count >= 50:
        print("Gold set ready for prompt sweep (50+ entries).", file=sys.stderr)


def show_stats():
    """Print feedback statistics."""
    feedback_path = EVAL_DIR / "feedback.jsonl"
    if not feedback_path.exists():
        print("No feedback collected yet.")
        return

    entries = [json.loads(line) for line in open(feedback_path)]
    verdicts = {}
    for e in entries:
        v = e.get("verdict", "unknown")
        verdicts[v] = verdicts.get(v, 0) + 1

    print(f"Total entries: {len(entries)}")
    for v, count in sorted(verdicts.items()):
        print(f"  {v}: {count}")

    if entries:
        llm_scores = [e["wow_score_llm"] for e in entries if e.get("wow_score_llm")]
        wow_items = [e for e in entries if e["verdict"] == "wow"]
        if llm_scores:
            print(f"  Avg LLM score (all): {sum(llm_scores)/len(llm_scores):.1f}")
        if wow_items:
            wow_llm = [e["wow_score_llm"] for e in wow_items if e.get("wow_score_llm")]
            if wow_llm:
                print(f"  Avg LLM score (wow only): {sum(wow_llm)/len(wow_llm):.1f}")


def main():
    parser = argparse.ArgumentParser(description="WOW digest feedback")
    sub = parser.add_subparsers(dest="command")

    record = sub.add_parser("record", help="Record feedback from JSON")
    record.add_argument("--selected", required=True, help="Path to selected items JSON")
    record.add_argument("--verdicts", required=True, help="Path to verdicts JSON")

    sub.add_parser("stats", help="Show feedback statistics")

    args = parser.parse_args()

    if args.command == "record":
        with open(args.verdicts) as f:
            verdicts = json.load(f)
        record_feedback(args.selected, verdicts)
    elif args.command == "stats":
        show_stats()
    else:
        parser.print_help()


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Multi-source content ingestion for WOW digest.

Pulls from email (GWS) and Telegram channels, normalizes to candidate JSONL.
"""
import argparse
import hashlib
import json
import subprocess
import sys
import base64
import re
from datetime import datetime, timedelta, timezone
from pathlib import Path

CONFIG_DIR = Path(__file__).parent.parent / "config"
EVAL_DIR = Path(__file__).parent.parent / ".wow-eval"


def load_config():
    import yaml
    config_path = CONFIG_DIR / "sources.yaml"
    with open(config_path) as f:
        return yaml.safe_load(f)


def extract_email_body(payload):
    """Extract plain text from Gmail message payload."""
    if "body" in payload and payload["body"].get("data"):
        if payload.get("mimeType", "") == "text/plain":
            return base64.urlsafe_b64decode(payload["body"]["data"]).decode("utf-8", errors="replace")
    if "parts" in payload:
        for part in payload["parts"]:
            if part["mimeType"] == "text/plain" and part["body"].get("data"):
                return base64.urlsafe_b64decode(part["body"]["data"]).decode("utf-8", errors="replace")
            result = extract_email_body(part)
            if result:
                return result
    # Fallback to HTML stripped
    if "body" in payload and payload["body"].get("data"):
        html = base64.urlsafe_b64decode(payload["body"]["data"]).decode("utf-8", errors="replace")
        text = re.sub(r"<style[^>]*>.*?</style>", "", html, flags=re.DOTALL)
        text = re.sub(r"<[^>]+>", " ", text)
        text = re.sub(r"&nbsp;", " ", text)
        text = re.sub(r"\s+", " ", text).strip()
        return text
    return ""


def ingest_email(config, days=1):
    """Pull newsletters from Gmail via GWS CLI."""
    query = config.get("email", {}).get("query", "newer_than:1d")
    max_results = config.get("email", {}).get("max_results", 100)
    query = query.replace("newer_than:1d", f"newer_than:{days}d")

    result = subprocess.run(
        ["gws", "gmail", "users", "messages", "list",
         "--params", json.dumps({"userId": "me", "q": query, "maxResults": max_results}),
         "--format", "json"],
        capture_output=True, text=True
    )
    if result.returncode != 0:
        print(f"GWS error: {result.stderr}", file=sys.stderr)
        return []

    data = json.loads(result.stdout)
    messages = data.get("messages", [])
    candidates = []

    for msg in messages:
        msg_result = subprocess.run(
            ["gws", "gmail", "users", "messages", "get",
             "--params", json.dumps({
                 "userId": "me",
                 "id": msg["id"],
                 "format": "full"
             }),
             "--format", "json"],
            capture_output=True, text=True
        )
        if msg_result.returncode != 0:
            continue

        msg_data = json.loads(msg_result.stdout)
        headers = {h["name"]: h["value"] for h in msg_data["payload"]["headers"]}

        body = extract_email_body(msg_data["payload"])
        snippet = body[:500] if body else msg_data.get("snippet", "")

        candidates.append({
            "source_type": "email",
            "source_name": headers.get("From", "Unknown"),
            "title": headers.get("Subject", "No subject"),
            "snippet": snippet,
            "url": "",
            "timestamp": headers.get("Date", ""),
            "message_id": msg["id"],
        })

    return candidates


def ingest_telegram(config, days=1):
    """Pull channel posts from Telegram via tg.py."""
    tg_config = config.get("telegram", {})
    channels = tg_config.get("channels", [])
    limit = tg_config.get("limit_per_channel", 50)
    candidates = []
    tg_script = Path.home() / ".claude/skills/telegram-telethon/scripts/tg.py"

    for channel in channels:
        result = subprocess.run(
            [sys.executable, str(tg_script), "recent",
             "--chat", channel,
             "--days", str(days),
             "--limit", str(limit),
             "--json"],
            capture_output=True, text=True
        )
        if result.returncode != 0:
            print(f"Telegram error for {channel}: {result.stderr}", file=sys.stderr)
            continue

        # tg.py outputs a JSON array, not NDJSON
        try:
            messages = json.loads(result.stdout)
        except json.JSONDecodeError:
            # Fallback: try line-by-line
            messages = []
            for line in result.stdout.strip().split("\n"):
                if not line:
                    continue
                try:
                    messages.append(json.loads(line))
                except json.JSONDecodeError:
                    continue

        if not isinstance(messages, list):
            messages = [messages]

        for msg in messages:
            text = msg.get("text", msg.get("message", ""))
            if not text or len(text) < 20:
                continue
            candidates.append({
                "source_type": "telegram",
                "source_name": channel,
                "title": text[:100] if text else "No text",
                "snippet": text[:500] if text else "",
                "url": msg.get("url", ""),
                "timestamp": msg.get("date", ""),
                "message_id": str(msg.get("id", "")),
            })

    return candidates


def candidate_hash(candidate):
    """Hash title+source for dedup across days."""
    key = f"{candidate.get('title', '').strip().lower()}|{candidate.get('source_name', '').strip().lower()}"
    return hashlib.sha256(key.encode()).hexdigest()[:16]


def load_recent_hashes(days=7):
    """Load candidate hashes from the last N days of .wow-eval/candidates/."""
    hashes = set()
    candidates_dir = EVAL_DIR / "candidates"
    if not candidates_dir.exists():
        return hashes
    cutoff = datetime.now() - timedelta(days=days)
    for f in candidates_dir.glob("*.jsonl"):
        try:
            date_str = f.stem
            file_date = datetime.strptime(date_str, "%Y%m%d")
            if file_date < cutoff:
                continue
        except ValueError:
            continue
        for line in f.read_text().strip().split("\n"):
            if not line:
                continue
            try:
                c = json.loads(line)
                hashes.add(candidate_hash(c))
            except json.JSONDecodeError:
                continue
    return hashes


def dedup_candidates(candidates, recent_hashes):
    """Remove candidates already seen in the last 7 days."""
    unique = []
    seen_this_run = set()
    dupes = 0
    for c in candidates:
        h = candidate_hash(c)
        if h in recent_hashes or h in seen_this_run:
            dupes += 1
            continue
        seen_this_run.add(h)
        unique.append(c)
    if dupes:
        print(f"  Dedup: removed {dupes} already-seen candidates", file=sys.stderr)
    return unique


def main():
    parser = argparse.ArgumentParser(description="Ingest content for WOW digest")
    parser.add_argument("--days", type=int, default=1, help="Look back N days")
    parser.add_argument("--output", "-o", required=True, help="Output JSONL path")
    parser.add_argument("--sources", nargs="+", default=["email", "telegram"],
                        choices=["email", "telegram"], help="Sources to pull from")
    parser.add_argument("--no-dedup", action="store_true", help="Skip dedup check")
    args = parser.parse_args()

    config = load_config()
    all_candidates = []

    if "email" in args.sources:
        print(f"Pulling email (last {args.days}d)...", file=sys.stderr)
        all_candidates.extend(ingest_email(config, args.days))
        print(f"  → {len(all_candidates)} email candidates", file=sys.stderr)

    if "telegram" in args.sources:
        print(f"Pulling Telegram (last {args.days}d)...", file=sys.stderr)
        tg_count_before = len(all_candidates)
        all_candidates.extend(ingest_telegram(config, args.days))
        print(f"  → {len(all_candidates) - tg_count_before} Telegram candidates", file=sys.stderr)

    if not args.no_dedup:
        recent_hashes = load_recent_hashes(days=7)
        all_candidates = dedup_candidates(all_candidates, recent_hashes)

    output_path = Path(args.output)
    output_path.parent.mkdir(parents=True, exist_ok=True)
    with open(output_path, "w") as f:
        for c in all_candidates:
            f.write(json.dumps(c) + "\n")

    print(f"Total: {len(all_candidates)} candidates → {args.output}", file=sys.stderr)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Cheap salience filter: drop obvious noise before expensive WOW scoring.

Removes marketing, payment confirmations, holiday greetings, and other
patterns from email-rules.md. Permissive — false positives OK.
"""
import argparse
import json
import re
import sys
from pathlib import Path

# Patterns that signal noise — matched case-insensitively against source_name + title + snippet
NOISE_SENDER_PATTERNS = [
    r"noreply@.*\.paypal\.",
    r"no-reply@revolut\.com",
    r"billing@zoom\.us",
    r"noreply@.*bybit\.com",
    r"no-reply@.*wise\.com",
    r"noreply@.*vodafone",
    r"@news\.paypal\.",
]

NOISE_TITLE_PATTERNS = [
    r"receipt for your payment",
    r"payment processed",
    r"withdrawal success",
    r"withdrawals have been sent",
    r"С Днем Победы",
    r"С Днём Победы",
    r"frohe weihnachten",
    r"happy holidays",
    r"your? (?:order|shipment|delivery)",
    r"bereit für ihren",
    r"mehr sicherheit mit passkeys",
    r"remember the tech you were dreaming",
]

NOISE_SOURCE_TYPES = {
    "email": NOISE_SENDER_PATTERNS,
}


def is_noise(candidate):
    """Return True if candidate matches known noise patterns."""
    source = candidate.get("source_name", "").lower()
    title = candidate.get("title", "").lower()
    snippet = candidate.get("snippet", "").lower()

    for pattern in NOISE_SENDER_PATTERNS:
        if re.search(pattern, source, re.IGNORECASE):
            return True

    for pattern in NOISE_TITLE_PATTERNS:
        if re.search(pattern, title, re.IGNORECASE):
            return True
        if re.search(pattern, snippet[:200], re.IGNORECASE):
            return True

    return False


def main():
    parser = argparse.ArgumentParser(description="Filter noise from candidates")
    parser.add_argument("--input", "-i", required=True, help="Input JSONL")
    parser.add_argument("--output", "-o", required=True, help="Output JSONL (filtered)")
    args = parser.parse_args()

    candidates = []
    with open(args.input) as f:
        for line in f:
            if line.strip():
                candidates.append(json.loads(line))

    filtered = [c for c in candidates if not is_noise(c)]
    dropped = len(candidates) - len(filtered)

    with open(args.output, "w") as f:
        for c in filtered:
            f.write(json.dumps(c) + "\n")

    print(f"Salience filter: {len(candidates)} → {len(filtered)} ({dropped} dropped)", file=sys.stderr)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""WOW scoring: score candidates for epistemic friction, select top items."""
import argparse
import json
import os
import subprocess
import sys
from datetime import datetime, timedelta
from pathlib import Path

VAULT = Path.home() / "Brains" / "brain"
CONFIG_DIR = Path(__file__).parent.parent / "config"
EVAL_DIR = Path(__file__).parent.parent / ".wow-eval"


def load_focus():
    """Load Base + Primary sections from My Focus.md."""
    focus_path = VAULT / "My Focus.md"
    if not focus_path.exists():
        return "No focus file found."
    text = focus_path.read_text()
    # Extract from ## Current & Urgent through ## Nice to have
    lines = text.split("\n")
    capture = False
    result = []
    for line in lines:
        if line.startswith("## Current") or line.startswith("## Base") or line.startswith("## Primary"):
            capture = True
        elif line.startswith("## Nice to have") or line.startswith("## My key"):
            capture = False
        if capture:
            result.append(line)
    return "\n".join(result) if result else text[:2000]


def load_recent_research(days=30):
    """Load titles + tags from recent ai-research files."""
    research_dir = VAULT / "ai-research"
    if not research_dir.exists():
        return "No ai-research directory."
    cutoff = datetime.now() - timedelta(days=days)
    entries = []
    for f in sorted(research_dir.glob("*.md"), reverse=True):
        # Parse date from filename: YYYYMMDD-topic.md
        try:
            date_str = f.stem[:8]
            file_date = datetime.strptime(date_str, "%Y%m%d")
            if file_date < cutoff:
                continue
        except ValueError:
            continue
        # Read frontmatter for tags
        text = f.read_text(errors="replace")
        tags = ""
        for line in text.split("\n")[:10]:
            if line.startswith("research_topic:"):
                tags = line.replace("research_topic:", "").strip()
                break
        entries.append(f"- {f.stem} [{tags}]")
    return "\n".join(entries[:30]) if entries else "No recent research files."


def load_recent_topics(days=7):
    """Extract headers from recent daily notes."""
    topics = []
    for i in range(days):
        date = datetime.now() - timedelta(days=i)
        daily_path = VAULT / "Daily" / f"{date.strftime('%Y%m%d')}.md"
        if not daily_path.exists():
            continue
        text = daily_path.read_text(errors="replace")
        for line in text.split("\n"):
            if line.startswith("## ") and line.strip() not in ("## do", "## log"):
                topics.append(f"- {date.strftime('%m/%d')}: {line.strip('# ').strip()}")
    return "\n".join(topics) if topics else "No recent daily note topics."


def score_candidates(candidates):
    """Send candidates through the WOW scoring prompt via llm CLI."""
    prompt_path = CONFIG_DIR / "wow_prompt.txt"
    prompt_template = prompt_path.read_text()

    focus = load_focus()
    research = load_recent_research()
    recent_topics = load_recent_topics()

    candidate_text = "\n\n".join(
        f"### Candidate {i+1}\nSource: {c['source_name']} ({c['source_type']})\n"
        f"Title: {c['title']}\nSnippet: {c['snippet'][:400]}"
        for i, c in enumerate(candidates)
    )

    prompt = prompt_template.format(
        focus=focus,
        research=research,
        recent_topics=recent_topics,
        candidates=candidate_text,
    )

    # Use llm CLI with default model
    result = subprocess.run(
        ["llm", "-s", "Respond with ONLY a JSON array. No markdown fences, no explanation.", prompt],
        capture_output=True, text=True
    )
    if result.returncode != 0:
        print(f"LLM error: {result.stderr}", file=sys.stderr)
        return []

    # Parse JSON from response
    response = result.stdout.strip()
    # Strip markdown fences if present
    if response.startswith("```"):
        response = "\n".join(response.split("\n")[1:])
    if response.endswith("```"):
        response = "\n".join(response.split("\n")[:-1])

    try:
        scored = json.loads(response)
    except json.JSONDecodeError as e:
        print(f"JSON parse error: {e}", file=sys.stderr)
        print(f"Raw response: {response[:500]}", file=sys.stderr)
        return []

    return scored


def select_top(scored, min_items=3, max_items=7):
    """Apply selection rules: diversity, source caps, sideways requirement."""
    scored.sort(key=lambda x: x.get("wow_score", 0), reverse=True)

    selected = []
    source_counts = {}
    topic_counts = {}

    for item in scored:
        if len(selected) >= max_items:
            break

        source = item.get("source", "")
        topic = item.get("connected_recent_note", "general")

        # Max 1 per source
        if source_counts.get(source, 0) >= 1:
            continue
        # Max 2 per topic cluster
        if topic_counts.get(topic, 0) >= 2:
            continue

        selected.append(item)
        source_counts[source] = source_counts.get(source, 0) + 1
        topic_counts[topic] = topic_counts.get(topic, 0) + 1

    # Ensure at least 1 sideways item (bridge_value > surprise)
    has_sideways = any(
        s.get("bridge_value", 0) > s.get("surprise", 0) for s in selected
    )
    if not has_sideways:
        for item in scored:
            if item not in selected and item.get("bridge_value", 0) > item.get("surprise", 0):
                if len(selected) >= max_items:
                    selected[-1] = item
                else:
                    selected.append(item)
                break

    return selected


def main():
    parser = argparse.ArgumentParser(description="Score candidates for WOW")
    parser.add_argument("--input", "-i", required=True, help="Input JSONL candidates")
    parser.add_argument("--output", "-o", required=True, help="Output JSON selected items")
    parser.add_argument("--all-scored", help="Also dump all scored items (for eval)")
    args = parser.parse_args()

    # Load candidates
    candidates = []
    with open(args.input) as f:
        for line in f:
            if line.strip():
                candidates.append(json.loads(line))

    if not candidates:
        print("No candidates to score.", file=sys.stderr)
        sys.exit(0)

    print(f"Scoring {len(candidates)} candidates...", file=sys.stderr)
    scored = score_candidates(candidates)

    if args.all_scored:
        Path(args.all_scored).parent.mkdir(parents=True, exist_ok=True)
        with open(args.all_scored, "w") as f:
            json.dump(scored, f, indent=2)
        print(f"All scored → {args.all_scored}", file=sys.stderr)

    # Merge original snippets back into scored items (LLM output loses them)
    candidate_by_title = {c.get("title", "").lower().strip(): c for c in candidates}
    for item in scored:
        title_key = item.get("title", "").lower().strip()
        if title_key in candidate_by_title:
            orig = candidate_by_title[title_key]
            item["snippet"] = orig.get("snippet", "")
            item["message_id"] = orig.get("message_id", "")
            item["source_type"] = orig.get("source_type", item.get("source_type", ""))

    selected = select_top(scored)
    print(f"Selected {len(selected)} WOW items", file=sys.stderr)

    with open(args.output, "w") as f:
        json.dump(selected, f, indent=2)

    # Print summary
    for item in selected:
        print(f"  [{item.get('wow_score', '?')}] {item.get('title', '?')[:60]}", file=sys.stderr)


if __name__ == "__main__":
    main()