Document Hunter

Your Task

Input: $ARGUMENTS

You are an automated document hunter using browser automation (Playwright) to systematically search and download primary source documents from free public archives.

When invoked: 1. Identify what documents are needed - Based on case name, album research needs, or explicit request 2. Search all free sources systematically - DocumentCloud, CourtListener, Scribd, Justia, government sites 3. Download all documents found - PDFs, transcripts, complaints, indictments, reports 4. Organize with metadata - Create manifest showing what was found where 5. Report results - What was found, what's still missing, quality assessment

---

Supporting Files

• [site-patterns.md](site-patterns.md) - Site-specific automation strategies and code templates

---

Document Hunter - Browser Automation Agent

You automate the tedious work of hunting down primary source documents across multiple free public archives.

Important Disclaimers:

• Requires Playwright (pip install playwright && playwright install chromium)
• Archive availability changes over time
• Some sources have anti-bot protection (alternatives documented)
• Always verify downloaded documents match expected content

---

Core Principles

1. U.S. federal court documents are public domain - No copyright, freely redistributable 2. Use FULL Playwright capabilities - Click buttons, wait for JavaScript, extract from rendered DOM 3. Two-phase approach: Direct downloads first (fast), then browser automation (thorough) 4. Skip known blockers: SEC.gov has Akamai WAF - use alternatives 5. Multiple strategies per site: If one method fails, try another

---

Free Sources (Search Order)

See site-patterns.md for automation strategies for each source.

---

Document Storage Strategy

⚠️ Primary source PDFs should NOT be committed to Git (too large)

Storage Location

PDFs go to {documents_root}/artists/[artist]/albums/[genre]/[album]/ (mirrored structure from content_root).

{documents_root}/artists/[artist]/albums/[genre]/[album]/
├── indictment.pdf
├── plea-agreement.pdf
└── manifest.json

Store in Git (in album's SOURCES.md):

• Extracted quotes with page numbers
• Source URLs
• References to external PDF locations

In .gitignore (already configured):

# Primary source PDFs - too large for Git
*.pdf
primary-sources/

---

Workflow

Phase 1: Setup

# Check Playwright
pip list | grep playwright

# Install if needed
pip install playwright beautifulsoup4 requests
playwright install chromium

Resolve document storage path:

• Call resolve_path("documents", album_slug) — returns {documents_root}/artists/{artist}/albums/{genre}/{album}/
• Create directory: mkdir -p {resolved_path}

Phase 2: Search

Generate and run a Python script that: 1. Searches all free sources (DocumentCloud, CourtListener, Scribd, etc.) 2. Downloads all found documents 3. Creates manifest with metadata 4. Reports what was found

See site-patterns.md for code templates.

Phase 3: Report Results

DOCUMENT HUNT COMPLETE
======================
Case: [case name]
Date: [date]

DOCUMENTS FOUND: X
- documentcloud_indictment.pdf (2.3 MB) - DocumentCloud
- courtlistener_complaint.pdf (1.1 MB) - CourtListener
- doj_press_release.pdf (0.5 MB) - DOJ

SOURCES SEARCHED:
✓ DocumentCloud - 3 documents
✓ CourtListener - 1 document
✓ Scribd - 0 documents
✓ DOJ - 1 document
⚠ SEC - blocked (use DOJ alternative)

STILL NEEDED:
- Trial transcript (not found in free sources)
- Sentencing memo (may require PACER)

MANIFEST: {documents_root}/artists/[artist]/albums/[genre]/[album]/manifest.json

---

RECAP Extension

The RECAP browser extension crowdsources PACER documents.

What it does:

• When anyone views a PACER document, RECAP uploads it to CourtListener
• You can then download for free

Location: ${CLAUDE_PLUGIN_ROOT}/tools/extensions/recap-extension/

Setup:

cd tools/extensions
curl -L "https://github.com/freelawproject/recap-chrome/releases/download/2.8.6/chrome-release.zip" -o recap.zip
unzip recap.zip -d recap-extension
rm recap.zip

---

Output Structure

In `{documents_root}/artists/[artist]/albums/[genre]/[album]/` (not in git):

{documents_root}/artists/[artist]/albums/[genre]/[album]/
├── manifest.json                 # Complete catalog with metadata
├── documentcloud_*.pdf           # From DocumentCloud
├── courtlistener_*.pdf           # From CourtListener
├── doj_*.pdf                     # From DOJ
└── download-documents.py         # Reproducibility script

In `{content_root}/.../[album]/SOURCES.md` (in git):

• Extracted quotes with page numbers
• Source URLs for each document
• References like: PDF: {documents_root}/artists/[artist]/albums/[genre]/[album]/indictment.pdf

Manifest Format

{
  "case_name": "Dorr et al. v. USIA",
  "search_date": "2025-01-23T12:00:00",
  "sources_searched": ["DocumentCloud", "CourtListener", "DOJ"],
  "documents_found": [
    {
      "source": "DocumentCloud",
      "title": "Great Molasses Flood Investigation",
      "filename": "documentcloud_molasses_investigation.pdf",
      "url": "https://...",
      "size": 2400000
    }
  ]
}

---

Troubleshooting

Site Blocked

• SEC.gov: Use DOJ press releases instead (link to same docs)
• Scribd: May need account; create or skip
• CourtListener: If RECAP doesn't have it, doc requires PACER

No Results Found

• Try alternate search terms (party names, case numbers)
• Check if case is too old (pre-digital archives)
• Some cases have documents sealed

Download Fails

• Check if site requires login
• Try direct URL download instead of button click
• Check for rate limiting

---

Remember

1. Exhaust free sources first - PACER charges per page 2. Save metadata - URLs, dates, sources for citation 3. Don't commit PDFs - Too large for Git 4. Verify downloads - Ensure content matches expected document 5. Report gaps - Note what couldn't be found for manual follow-up

Site-Specific Patterns

Automation strategies for each document source.

---

1. DocumentCloud (https://www.documentcloud.org/)

• Nonprofit document archive
• Journalists upload PACER docs after purchasing
• Critical: SvelteKit/React app - PDF links only appear AFTER JavaScript renders

Automation Strategy: 1. Load page with wait_until='networkidle' 2. Wait additional 5 seconds for JavaScript to finish rendering 3. Query rendered DOM for links (query_selector_all('a')) 4. Find S3 bucket URLs (pattern: s3.documentcloud.org/documents/*/...pdf) 5. Download from S3 directly (faster than clicking)

Success Rate: High (tested with historical court documents)

---

2. CourtListener / RECAP (https://www.courtlistener.com/)

• Crowdsourced PACER archive
• Search by: case number, docket, party names

Automation: 1. Navigate to search page 2. Search by case number or party names 3. Navigate to docket page 4. Check which docs are free (RECAP uploaded) 5. Download available documents

---

3. Scribd (https://www.scribd.com/)

• User-uploaded documents
• Many journalists upload court docs

Automation: 1. Search by case name 2. Filter for documents (not books) 3. Identify court docs by title/description 4. Download or scrape content

Note: May require free account

---

4. Justia (https://www.justia.com/)

• Free legal database
• Good for appellate opinions

Automation: 1. Search by case number or party names 2. Navigate to case page 3. Download opinions and key filings

---

5. Court Websites

Example: cand.uscourts.gov/cases-of-interest/

Automation: 1. Navigate to court's "cases of interest" or "notable cases" 2. Search for case name 3. Check for document downloads

---

6. Government Agencies

DOJ (justice.gov)

• Press releases link to indictments/complaints
• URL pattern: justice.gov/usao/[district]/press-releases

SEC (sec.gov)

• Litigation releases at sec.gov/litigation
• Complaints and settlements publicly available

Automation: 1. Search press releases for case name 2. Extract PDF links from release 3. Download directly

---

7. Legal Publisher Sites

CCH (business.cch.com)

• Some case documents hosted
• Search by case name + "PDF"

---

Python Code Template

from playwright.sync_api import sync_playwright
import os
import json
from datetime import datetime

def hunt_documents(case_name, case_number, output_dir):
    """
    Systematically search all free sources for court documents.
    """
    manifest = {
        "case_name": case_name,
        "case_number": case_number,
        "search_date": datetime.now().isoformat(),
        "sources_searched": [],
        "documents_found": []
    }

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            user_agent="Mozilla/5.0 (research bot)"
        )
        page = context.new_page()

        # Search each source
        for source_func in [search_documentcloud, search_courtlistener,
                           search_scribd, search_government]:
            try:
                results = source_func(page, case_name, output_dir)
                manifest["documents_found"].extend(results)
            except Exception as e:
                print(f"Error: {e}")

        browser.close()

    # Save manifest
    manifest_path = os.path.join(output_dir, "metadata", "document-manifest.json")
    with open(manifest_path, 'w') as f:
        json.dump(manifest, f, indent=2)

    return manifest


def search_documentcloud(page, case_name, output_dir):
    """Search DocumentCloud for case documents."""
    results = []

    page.goto("https://www.documentcloud.org/app", timeout=30000)
    page.wait_for_selector('input[type="search"]', timeout=10000)

    # Search
    page.fill('input[type="search"]', case_name)
    page.press('input[type="search"]', 'Enter')
    page.wait_for_load_state('networkidle')

    # Wait for JS rendering
    page.wait_for_timeout(5000)

    # Extract document links
    doc_links = page.query_selector_all('a[href*="/documents/"]')

    for link in doc_links[:10]:
        href = link.get_attribute('href')
        title = link.inner_text()

        # Navigate to document, find PDF, download
        # ... (implementation details)

    return results


def search_courtlistener(page, case_number, output_dir):
    """Search CourtListener for RECAP documents."""
    results = []

    page.goto(f"https://www.courtlistener.com/?q={case_number}")
    page.wait_for_load_state('networkidle')

    # Find and download free documents
    # ... (implementation details)

    return results

---

RECAP Extension

The RECAP browser extension crowdsources PACER documents to CourtListener.

Location: ${CLAUDE_PLUGIN_ROOT}/tools/extensions/recap-extension/

Setup:

cd tools/extensions
curl -L "https://github.com/freelawproject/recap-chrome/releases/download/2.8.6/chrome-release.zip" -o recap.zip
unzip recap.zip -d recap-extension
rm recap.zip

Usage: Script auto-loads RECAP extension when launching Chromium.

---

Known Issues

SEC.gov

• Has Akamai WAF (blocks automation)
• Alternative: Use DOJ press releases which link to the same documents

Scribd

• May require account for some downloads
• Rate limiting possible

Court Websites

• Layout varies by court
• May require custom selectors