Intel Agent

Intel Agent: Data Point Reconnaissance

When This Skill Activates

Activate when user requests:

• "recon [URL]"
• "intel report for [URL]"
• "find how to extract [data] from [URL]"
• "discover data from [URL]"
• "what's the best way to get [data] from [site]"
• "how would I scrape [data points] from [URL]"

This skill does NOT implement scrapers. It discovers extraction methods and outputs a structured intelligence report. For implementation, hand off to the poc-agent skill (builds + validates a minimal scraper from the report) or directly to web-scraper (productionization).

Terminology: Step N = strictly sequential — each step builds on the previous and cannot be skipped.

References (loaded on demand):

• reference/tool-reference.md — Tool signatures, known limitations, important rules
• reference/data-point-types.md — Type classification and search strategies per type
• reference/report-schema.md — Report output format
• strategies/cheerio-vs-browser-test.md — Cheerio vs Browser extraction test procedure

---

Input Parsing

Extract from the user's request: Target URL and Data points to extract.

Normalize each data point into a type (text / numeric / boolean / list / nested) with search terms. See reference/data-point-types.md for type classification and search strategies.

If data points are NOT specified, ask the user before proceeding.

---

The Workflow

Six steps, executed sequentially. Each step builds on the previous.

Step 1: Initialize & Capture Baseline Traffic

Start the MITM proxy with full-body persistence (required — the default ring buffer caps bodies at 4 KB, which truncates __NEXT_DATA__, JSON-LD, and most API responses). Then launch the stealth browser and capture the baseline.

proxy_start(persistence_enabled: true, capture_profile: "full", session_name: "recon-[domain]-[YYYYMMDD]", max_disk_mb: 2048)
interceptor_browser_launch(url: "[target URL]")
interceptor_browser_screenshot(target_id)

Note: proxy_start with persistence_enabled: true auto-starts a session. A subsequent proxy_session_start() will return the existing session (not error), but the cleaner pattern is to pass session_name directly to proxy_start. Capture the returned session_id — every body-search and HAR-export call needs it.

Hard-target fallback — camoufox (proxy-mcp ≥ 3.0.0): if cloakbrowser hits a Chromium-specific block (heavy Cloudflare Turnstile fingerprinting, Akamai bot manager rejecting Chrome JA4, Firefox-only feature checks), close the browser target and relaunch via interceptor_camoufox_launch({ headless: true, disable_coop: true }). This returns wsUrl + playwright_connect; you then drive Playwright outside MCP with firefox.connect(wsUrl) — interceptor_browser_*, humanizer_*, and the snapshot/screenshot/console steps below are bound to the cloakbrowser interceptor (id: "browser") and will not work against the camoufox target. Proxy-side tools (proxy_list_traffic, proxy_search_session_bodies, proxy_set_upstream, fingerprint capture, …) work identically. Use camoufox only when cloakbrowser is detected — most of this workflow assumes the cloakbrowser path.

Navigation tip: interceptor_browser_navigate(target_id, url, wait_for_proxy_capture: true) returns matchedHostExchangeIds — the exchange IDs for every target-host request fired during load. Pass those straight to proxy_get_session_exchange(session_id, exchange_id, include_body: true) in Step 2 instead of re-querying via proxy_list_traffic.

Console signal for site profiling: interceptor_browser_list_console(target_id) — React/hydration warnings → CSR SPA, network errors on /api/* → API-driven, clean console → static SSR.

Record: Session name/ID, page load status, loading behavior (SSR vs SPA), interstitials, framework indicators.

Dismiss interstitials: prefer locator-based clicks over CSS selectors — humanizer_click auto-waits for visible + enabled + stable + in-view. Try in this order:

humanizer_click(target_id, role: "button", name: "Accept all")
humanizer_click(target_id, text: "I agree")
humanizer_click(target_id, selector: ".cookie-accept")   # fallback

Multiple popups may appear sequentially (e.g., Cloudflare challenge then cookie consent) — repeat until the page is clean.

Step 2: Scan Response Bodies for Data Points

For each data point, search three locations to determine extraction methods.

2a. Raw HTML body (Cheerio method)

Use the session-backed full-body path — the ring buffer preview (proxy_get_exchange) truncates at 4 KB and will miss embedded JSON blobs.

proxy_search_session_bodies(session_id, query: "[data point search term]", hostname_contains: "[target domain]")
proxy_list_traffic(url_filter: "[target domain]", method_filter: "GET")   # locate main HTML exchange id
proxy_get_session_exchange(session_id, exchange_id: "[main HTML exchange id]", include_body: true)

If the search term is found in the raw HTML body → Cheerio works.

2b. JSON blobs in HTML (JSON-in-HTML method)

Search the raw HTML body (or use proxy_search_session_bodies directly) for: application/ld+json, __NEXT_DATA__, __INITIAL_STATE__, __NUXT__, __APOLLO_STATE__, __RELAY_STORE__. Parse and search for data points. If found → JSON-in-HTML works (preferred over Cheerio).

2c. Rendered DOM (Browser method)

interceptor_browser_snapshot(target_id)
# scoped (token-saver):
interceptor_browser_snapshot(target_id, selector: "main, article, [itemtype*='Product']")
# with refs (lets later humanizer_click reuse the locator):
interceptor_browser_snapshot(target_id, mode: "ai")

Output is YAML with role, name, text fields — match data-point search terms against those. If found in rendered DOM but NOT in raw HTML / JSON blobs / web storage → Browser extraction required.

2d. Web storage scan (often overlooked — SPAs hydrate from here)

interceptor_browser_list_storage_keys(target_id, storage_type: "local")
interceptor_browser_list_storage_keys(target_id, storage_type: "session")
interceptor_browser_get_storage_value(target_id, storage_type: "local", item_id: "[item_id]")

If a search term is found in a storage value → Browser extraction required, but the data is stable + parseable from storage keys (record the key name).

2e. Decision matrix

See strategies/cheerio-vs-browser-test.md for the detailed procedure.

2f. Trigger interactions for missing data points

For data points not found: proxy_clear_traffic(), interact (locator-based humanizer_click, humanizer_scroll), then re-check DOM + traffic. humanizer_click auto-waits for stability — no explicit idle needed before reading traffic.

2g. Full-page scroll capture (MANDATORY)

Scroll the entire page before API sniffing to capture lazy-loaded API calls:

1. proxy_clear_traffic() 2. Scroll 2-3x: humanizer_scroll(target_id, delta_y: 800) 3. Mid-scroll visual check: interceptor_browser_screenshot(target_id) + interceptor_browser_snapshot(target_id) — inspect for interstitials (CAPTCHA, cookie consent, modals). Dismiss via humanizer_click(role + name / text). If a hard block (CAPTCHA) is detected, stop scrolling and proceed to Step 3 with traffic captured so far. 4. Scroll 2-3x more: humanizer_scroll(target_id, delta_y: 800) 5. interceptor_browser_screenshot(target_id) + interceptor_browser_snapshot(target_id) — final state 6. proxy_list_traffic() — review newly triggered API calls

MANDATORY even if all data points already found — lazy APIs often provide better-structured data.

Step 3: Sniff APIs

3a. Filter existing traffic

The MITM proxy is the single source of truth — it sees strictly more than any in-browser network view (3rd-party domains, CONNECT tunnels, every TLS handshake):

proxy_list_traffic(url_filter: "/api/")
proxy_list_traffic(url_filter: "/graphql")
proxy_list_traffic(url_filter: "/_next/data/")
proxy_list_traffic(url_filter: "/wp-json/")
proxy_list_traffic(url_filter: ".json")
proxy_search_traffic(query: "application/json")
proxy_search_session_bodies(session_id, query: "__typename", content_type_contains: "application/json")

3b. Trigger more traffic via interactions

proxy_clear_traffic() → interact (pagination, search, filters, detail click, load more, sort) via locator-based humanizer_click → proxy_list_traffic(). No explicit idle — humanizer_click auto-waits for stability.

3c. Inspect each discovered endpoint

Pull full request + response body from the session (not the 4 KB preview): proxy_get_session_exchange(session_id, exchange_id, include_body: true). Record: URL, method, headers, auth, body, response structure, pagination, data points covered, rate limits. See reference/report-schema.md Section 4 for the full endpoint template.

Step 4: Assess Protection Under Active Config

Assume the active proxy/IP is the right one for this target — don't escalate through datacenter/residential tiers. Single-pass: characterize what the site returns under the current config, and surface a proxy/IP hypothesis only when blocking is observed.

Observe under the active config:

interceptor_browser_list_cookies(target_id)             # bot-management cookies (_abck, bm_sz, datadome, cf_clearance, ...)
proxy_search_traffic(query: "403")                      # blocked responses
proxy_search_traffic(query: "challenge")                # interstitials / JS challenges
proxy_search_traffic(query: "captcha")

For each access method tested in Steps 1–3 (main page, each API endpoint), record: status, blocked? (status ≥ 400 or interstitial served), cookies set, challenge type if any.

TLS verification: proxy_list_tls_fingerprints(hostname_filter: "[target domain]").

• JA3 varies + JA4 stable → browser ClientHello passthrough (cloakbrowser presents authentic Chrome fingerprint).
• JA3 identical across all connections → an upstream proxy re-terminates TLS (your real fingerprint is the proxy's, not Chrome's). For HTTP-only clients downstream, recommend proxy_set_fingerprint_spoof(chrome_*).

Cookie JA4-binding probe — record whether the WAF ties its session cookie to the originating TLS fingerprint. This decides whether the deployed actor's Tier 1 (impit) call needs to use the same impit browser profile as the cloak warmup, or whether any profile works.

Procedure: after the browser run mints session cookies (datadome / incap_ses / cf_clearance / etc.), have the consumer replay one representative GET against the same target with two distinct impit profiles (browser=firefox and browser=chrome) on the same warmed jar. Compare the responses:

• Both profiles return real-page bytes with the required data point

paths resolving → `tier1_cookie_replay: cross-engine`. JA4 is not part of the WAF's session-binding hash. The consumer can use any impit profile.

• Only the matched profile (the engine that minted the cookie)

returns real-page bytes; the mismatched profile returns a challenge / fresh cid / 200 with an interstitial body → `tier1_cookie_replay: matched-only`. The consumer's Tier 1 client must use the same impit profile as the cloak warmup, or the cascade can never ride Tier 1. DataDome is the canonical example.

• Could not test (Tier 1 unreachable for unrelated reasons; only

one profile ever attempted) → `tier1_cookie_replay: untested`.

Record the observation in §2 of the report and propagate the recommendation through to Phase 3 (PathPolicy.browser for the WAF-fronted host must match the warmup engine when matched-only). See the toolkit reference skills/_shared/references/treat-as-success.md for the underlying mechanism.

Proxy/IP hypothesis (record only if blocking was observed):

When the page loads but a specific endpoint 403s, or the page itself returns a challenge, do not re-test through a different proxy tier — flag the suspected cause for the human operator instead. Common patterns:

Out of scope: actually swapping proxies and re-running the recon. intel-agent reports under one configuration. Tier swapping is an operator decision based on the hypothesis above.

Step 5: Rank Extraction Methods

For each data point, rank available methods. Priority: API > JSON-in-HTML > Cheerio > Browser.

Per method include: source details (endpoint/selector/JSON path), required proxy level, confidence (High/Medium/Low).

Step 6: Generate Intelligence Report

Compile findings into the report format. See reference/report-schema.md for the complete structure.

Output contract — critical: the report is written to a file, not pasted into the chat. The report body is long (hundreds of lines); dumping it into the conversation wastes the caller's context window and makes the handoff harder (downstream skills read the file by path, not by scrolling back through chat).

• Write the full report to ./intel-<domain>-<YYYYMMDD>.md via the Write tool.
• The chat message at the end of the run is a short status summary only: report path, HAR path, session_id, data-point coverage count (e.g., "5 of 6 AVAILABLE"), any deviations or limitations, and the handoff line.
• Do NOT paste the report body into chat. If the user explicitly says "show me the report in chat", that's a separate follow-up request after the file exists.

6a. Export session evidence:

# Read analysis resources (free aggregation — no manual counting of proxy_list_traffic):
ReadMcpResourceTool("proxy://sessions/{session_id}/summary")    # §6 totals + §2 status breakdown
ReadMcpResourceTool("proxy://sessions/{session_id}/findings")   # §2 rate-limit endpoints, §4 per-endpoint errors
ReadMcpResourceTool("proxy://sessions/{session_id}/timeline")   # §2 block-onset minute (60s buckets)

proxy_get_session_handshakes(session_id)   # JA3/JA4 coverage for §2
proxy_session_stop()
proxy_export_har(session_id: "[session_id]", file_path: "recon-[domain]-[YYYYMMDD].har", include_bodies: true)

Include in Section 6 (all REQUIRED):

• Session name, ID, HAR path, handshake coverage
• From summary: total exchanges, avg duration, top hostnames, status code breakdown
• From findings: high-error endpoints, slowest exchanges, host error rates (filter to target domain before quoting — excludes 3rd-party analytics/CDN noise)
• From timeline: bucket where errorCount spikes → "protection engaged after ~N requests" signal for §2

---

What This Skill Does NOT Do

This skill is pure discovery. It does NOT write scraping code, create Actors, implement pagination, or deploy anything.

Handoff: After generating the report, pick one:

# Recommended — small validated POC first
Intelligence report complete. To build + validate a minimal POC scraper from
these findings, use the poc-agent skill:
  "build POC from intel report at /path/to/report.md"

# Or skip straight to production
To implement a production scraper, use the web-scraper skill:
  "scrape [data points] from [URL] using the intel report above"

Intel Agent

Reconnaissance skill for Claude Code that discovers how to extract data from any website — without writing a single line of scraping code.

Give it a URL and the data points you want. It launches a stealth browser through a MITM proxy, captures all traffic, sniffs APIs, tests protection levels, and returns a structured intelligence report with extraction strategies ranked by reliability.

Quick Start

recon https://www.alza.cz/some-product-d7752990.htm
  data points: name, price, description, availability

The skill handles everything: Cloudflare challenge solving, lazy-content discovery via full-page scroll, TLS fingerprint verification, and HAR evidence export.

What You Get

A structured report with:

• Per-data-point extraction methods ranked API > JSON-in-HTML > Cheerio > Browser, with exact JSON paths, CSS selectors, and API endpoint specs
• Protection assessment — Cloudflare/DataDome/Akamai detection, proxy tier requirements, TLS fingerprint analysis
• Discovered API endpoints — full specs (URL, method, headers, auth, pagination, rate limits)
• Entity identifier mapping — how the site's internal IDs relate to URLs and API parameters
• HAR evidence file — full request/response bodies for every exchange captured during recon

What It Does NOT Do

Write scraping code, create Apify Actors, or run extraction at scale. For implementation, hand off to the web-scraper skill.

Setup

1. Install proxy-mcp

The skill requires proxy-mcp ≥ 2.0.0 (cloakbrowser + Playwright) as an MCP server. The optional camoufox hard-target fallback (Step 1) needs proxy-mcp ≥ 3.0.0. One-liner:

claude mcp add proxy-mcp -- npx -y proxy-mcp@latest

Requires Node.js ≥ 20. First launch downloads a ~200 MB stealth Chromium binary (cached afterwards). To enable the camoufox path: pip install "camoufox[geoip]" && python3 -m camoufox fetch && sudo apt install libnss3-tools (or the macOS / Fedora equivalents).

2. Recommended permissions

The skill makes many MCP tool calls during a single recon session (traffic capture, browser control, humanizer interactions, session management). To avoid approving each one individually, add this to your project's .claude/settings.local.json:

{
  "permissions": {
    "allow": [
      "mcp__proxy-mcp"
    ]
  }
}

This auto-approves all proxy-mcp tools. All other tools still require manual approval.

3. Install the skill

Clone or copy this directory into your project:

git clone https://github.com/yfe404/intel-agent.git

Usage

# Full recon with specific data points
recon https://www.example.com/product/123
  data points: name, price, description, reviews, availability

# Let the skill ask what you need
intel report for https://news.example.com

# Direct question format
what's the best way to get product name, price, stock status from https://shop.example.com

How It Works

1. Initialize — Start MITM proxy with full-body persistence; launch cloakbrowser (stealth Chromium, source-level fingerprint patches, humanize on by default) via Playwright. For hard targets that block Chromium fingerprints (Cloudflare Turnstile, Akamai bot manager), fall back to camoufox (anti-detect Firefox via firefox.connect(wsUrl)) — proxy-mcp ≥ 3.0.0 2. Scan for data — Search raw HTML (full decompressed bodies), JSON blobs, web storage (local + session), and rendered ARIA snapshot for each data point 3. Full-page scroll — Mandatory scroll to trigger lazy-loaded APIs (descriptions, reviews, carousels) 4. Sniff APIs — Filter traffic for JSON endpoints, trigger interactions via locator-based clicks (role+name / text / label) to discover pagination/search/filter APIs 5. Test protection — Check Cloudflare/DataDome/Akamai indicators, verify TLS fingerprint passthrough, test proxy tiers (paired locale + timezone for geo-specific upstreams) 6. Export — Generate structured report + HAR file with full request/response bodies

File Structure

intel-agent/
├── SKILL.md                              # Main workflow (~200 lines)
├── README.md                             # This file
├── reference/
│   ├── report-schema.md                  # Report output format
│   ├── tool-reference.md                 # Tool signatures, caveats, rules
│   └── data-point-types.md              # Type classification & search strategies
└── strategies/
    └── cheerio-vs-browser-test.md        # Three-way extraction test procedure

Requires

• Claude Code with MCP support
• proxy-mcp ≥ 2.0.0 — MITM traffic interception with full-body on-disk persistence, cloakbrowser stealth browser, Playwright-driven locators, humanizer, session recording
• Node.js ≥ 20

(cloakbrowser ships its own stealth Chromium binary — no separate Chrome install needed.)

Data Point Types & Search Strategies

Reference for classifying data points and choosing search strategies during reconnaissance.

---

Type Classification

• text — Single string value (name, title, description)
• numeric — Number or currency (price, rating, count)
• boolean — True/false state (in stock, verified, active)
• list — Repeating items (reviews, variants, images)
• nested — Grouped sub-fields (address with street/city/zip, product with name/price/sku)

---

Search Strategies by Type

Text and Numeric (name, price, title, rating)

Direct value search. Look for the exact value (or close match) in:

• Raw HTML body text
• JSON blob values
• Rendered DOM text
• API response fields

Note on price: Price is often server-rendered only (no dedicated API) on e-commerce sites — this is common for SEO, anti-scraping, or personalization. If no price API is found after thorough traffic analysis, Cheerio extraction from SSR HTML is the expected method. Search with formatting variants (e.g., "299,-", "299,–", "299.00", "$299").

Boolean (in stock, verified, active)

Look for:

• Explicit boolean fields in API responses ("inStock": true)
• Text indicators in HTML ("In Stock", "Out of Stock", "Sold Out")
• CSS classes indicating state (.in-stock, .unavailable)
• Structured data properties (schema:availability)

List (reviews, variants, images, related items)

Look for:

• API responses: Array fields, pagination (indicates a list endpoint)
• HTML: Repeating DOM structures (.review-item, li.variant, .product-card)
• JSON blobs: Array values in __NEXT_DATA__ or ld+json

For lists, also determine:

• Total count (from API pagination metadata or DOM count)
• Whether all items load at once or require pagination/scroll
• Whether list items link to detail pages with more data

Nested (address, product details, author info)

Look for:

• API responses: Nested JSON objects ("address": {"street": "...", "city": "..."})
• JSON blobs: Nested structures in embedded JSON
• HTML: Grouped DOM elements (.address > .street, .address > .city)

Treat each sub-field as its own data point for extraction method ranking, but group them in the report.

Images and Media

Look for:

• src attributes on <img> tags
• CDN URL patterns (/images/, /media/, /cdn/)
• API response fields with image URLs ("imageUrl", "thumbnail", "photos")
• srcset attributes for responsive images
• Background images in CSS (background-image: url(...))
• Structured data image properties (schema:image)

Record the full CDN URL pattern — often images follow a template like https://cdn.site.com/images/{size}/{id}.jpg where size can be manipulated.

Intelligence Report Schema

Canonical output format for the intel-agent skill. Every report follows this structure.

---

Report Template

================================================================
INTEL REPORT: [domain.com]
================================================================
Target: [full URL]
Date: [YYYY-MM-DD HH:MM UTC]
Requested data points: [comma-separated list]
================================================================


## 1. SITE PROFILE

Framework: [Next.js / React SPA / WordPress / Static HTML / etc.]
Rendering: [SSR / CSR / SSG / Hybrid SSR+CSR]
Primary data source: [Internal REST API / GraphQL / HTML-embedded JSON / Static HTML]
Page type: [Product page / Listing page / Article / Search results / etc.]
URL routing:
  Pattern: [how entity IDs are embedded in URLs, e.g., "/product-slug-d{commodityId}.htm"]
  Redirect behavior: [e.g., "slug is decorative, numeric ID is authoritative, may redirect to canonical slug"]
  Canonical URL: [final URL after redirects, or from <link rel="canonical">]
  Authoritative identifier: [which part of the URL is the real key, e.g., "numeric ID in d{id}.htm"]
Entity identifiers:
  - [identifier name]: [value or pattern] — used by [list of endpoints]
  - [identifier name]: [value or pattern] — used by [list of endpoints]


## 2. PROTECTION ASSESSMENT

Active proxy/IP: [describe what was used — e.g., "Apify residential, country=US" or "Direct (no upstream)"]

| Access Method | Status | Blocked? | Challenge | Notes |
|--------------|--------|----------|-----------|-------|
| Main page    | [200/403/...] | [No/Yes] | [None / JS / CAPTCHA / Interstitial] | [cookies set, redirects, etc.] |
| [Endpoint 1] | [200/403/429] | [No/Yes] | [None/...] | [...] |
| [Endpoint 2] | [200/403/429] | [No/Yes] | [None/...] | [...] |

Protection system: [None / Cloudflare / DataDome / Akamai / Imperva / PerimeterX / Custom]
Rate limits: [observed limits per endpoint]
  - [Endpoint]: [N] req/min → recommended safe rate: [M] req/min
Stealth requirements: [cloakbrowser default sufficient / TLS spoofing needed for HTTP clients / camoufox (Firefox) needed for hard targets / advanced measures needed]
TLS fingerprint verification:
  - ClientHello passthrough: [Yes — browser ClientHello forwarded to target / No — proxy re-terminates TLS]
  - JA3 behavior: [varies per-connection (Chrome randomization) / identical across requests (proxy fingerprint)]
  - JA4 observed: [value, e.g., "t13d1517h2_8daaf6152771_b6f405a00624"]
  - Implication: [e.g., "HTTP-only clients need proxy_set_fingerprint_spoof(chrome_136)" or "Browser sessions are transparent"]
Protection cookies: [list of protection cookies observed]
Tier 1 cookie replay: [matched-only / cross-engine / untested]
  - Probe: replay one representative GET against the warmed jar with both impit `browser=firefox` and `browser=chrome` (see SKILL.md "Cookie JA4-binding probe").
  - matched-only → consumer's `PathPolicy.browser` MUST match the cloak warmup engine; mismatched profile mints fresh CID and serves challenge.
  - cross-engine  → any impit profile works; `PathPolicy.browser` only needs to match the cloak engine choice (Imperva/Linux-Chrome avoidance), not the impit profile.
  - untested      → could not run the probe (Tier 1 unreachable for other reasons, or only one profile attempted).
  - Evidence: [byte counts / fresh-CID header presence / status code from each profile]

Proxy/IP hypothesis (only if blocking observed under the active config):
  - Suspected cause: [datacenter IP on residential-only site / wrong-country IP on geo-locked site / API requires browser-warmed cookies / IP reputation / etc.]
  - Operator action: [re-run with residential exit / matching-country IP / browser-warmup → cookie replay / lower request rate / etc.]
  - Confidence: [High / Medium / Low — based on observed signal]


## 3. DATA POINT ANALYSIS

### 3.1 [Data Point Name]

Status: [AVAILABLE / PARTIALLY AVAILABLE / NOT FOUND]
Type: [text / numeric / boolean / list / nested]

Methods (ranked best → worst):

  1. [Best method] ← RECOMMENDED
     - Type: [API / JSON-in-HTML / Cheerio / Browser]
     - [For API]: Endpoint: [URL], Method: [GET/POST]
     - [For API]: Headers: [required headers]
     - [For API]: Auth: [none / cookie / token / API key]
     - [For API]: JSON path: [path to data point in response]
     - [For JSON-in-HTML]: Source: [__NEXT_DATA__ / ld+json / __INITIAL_STATE__]
     - [For JSON-in-HTML]: JSON path: [path to data point]
     - [For Cheerio]: Selector: [CSS selector]
     - [For Browser]: Snapshot location: [accessibility tree path]
     - [For Browser]: Required interaction: [none / scroll / click selector]
     - Confidence: [High / Medium / Low]

  2. [Fallback method]
     - [same fields as above]

### 3.2 [Data Point Name]

[same structure as 3.1]

[...repeat for each data point...]


## 4. DISCOVERED ENDPOINTS

### 4.1 [Endpoint Name]

URL: [full URL with parameter placeholders]
Method: [GET / POST]
Content-Type: [application/json / text/html / etc.]

Request:
  Headers:
    - [header]: [value or description]
  Parameters:
    - [param]: [type] — [description]
  Body (POST only):
    [body structure or example]

Response:
  Status: [200 / etc.]
  Structure:
    [JSON structure summary or example]
  Key fields:
    - [field]: [type] — [which data point this serves]

Authentication: [None / Cookie-based / Bearer token / API key]
Pagination:
  - Type: [offset / cursor / page-based / none]
  - Parameters: [page, limit, offset, cursor, etc.]
  - Total items indicator: [field name or "not provided"]
  - Has-more indicator: [field name or "not provided"]
Rate limit:
  - Observed: [N req/min or "not tested"]
  - Headers: [X-RateLimit-* values if present]

Data points covered: [list of data points this endpoint provides]

### 4.2 [Endpoint Name]

[same structure as 4.1]

[...repeat for each endpoint...]


## 5. RECOMMENDED STRATEGY

Summary:

| Data Point | Best Method | Endpoint/Source | Confidence |
|-----------|-------------|-----------------|------------|
| [name]   | [API/JSON/Cheerio/Browser] | [endpoint or source] | [H/M/L] |
| [name]   | [API/JSON/Cheerio/Browser] | [endpoint or source] | [H/M/L] |

Complexity estimate: [Low / Medium / High]
  - [Low]: All data from single API or JSON-in-HTML, no auth, no rate limits
  - [Medium]: Multiple sources, some auth or rate limits, but straightforward
  - [High]: Browser required, complex auth, strict rate limits, or multiple interaction steps

Estimated total items: [count or "unknown"]
Rate limit budget: [total requests needed] at [safe rate] = [estimated time]

Notes:
  - [any caveats, edge cases, or additional observations]


## 6. RAW EVIDENCE

Browser path: [cloakbrowser (default) / camoufox (Firefox fallback)]
Session ID: [REQUIRED — returned by proxy_start with persistence_enabled: true]
Session name: [intel-<target-domain>-<timestamp>]
HAR export: [REQUIRED — path written by proxy_export_har with include_bodies: true]
Capture profile: [must be "full" for intel-agent runs]
Screenshots: [list of screenshot file paths taken during reconnaissance — N/A on camoufox path]

Session summary (from `proxy://sessions/{id}/summary`):
  - Total exchanges: [N]
  - Avg duration: [N ms]
  - Top hostnames: [{hostname, count} × up to 10 — useful for CDN/analytics attribution]
  - Status code breakdown: [{200: N, 403: N, 429: N, ...}]
  - Method breakdown: [{GET: N, POST: N, ...}]

Session findings (from `proxy://sessions/{id}/findings`, filtered to target domain):
  - High-error endpoints: [{endpoint, errors} × up to 10]
  - Slowest exchanges: [{exchangeId, duration_ms, url} × up to 10]
  - Host error rates: [{hostname, total, errors, errorRate} — target domain row primary]

Session timeline (from `proxy://sessions/{id}/timeline`, 60s buckets):
  - Buckets: [{bucketStart, count, errorCount} series]
  - Block-onset observation: [e.g., "errorCount jumped from 0 to 7/12 at bucket 3 → ~N requests in ~3 minutes"]

Handshake metadata: [JA3/JA4 coverage — from proxy_get_session_handshakes]

================================================================
END OF REPORT
================================================================

---

Field Definitions

Status Values

• AVAILABLE: Data point found reliably in at least one source with high confidence
• PARTIALLY AVAILABLE: Data point found but with caveats (only on some pages, requires specific conditions, inconsistent presence)
• NOT FOUND: Data point not discovered in any source during reconnaissance
• INCONCLUSIVE: Data point search was limited (e.g. all sources checked but element only appears after a specific interaction we did not trigger). Cannot confirm presence or absence. The report notes what prevented a definitive check.

Confidence Levels

• High: Data point clearly present, tested and verified, consistent across multiple checks
• Medium: Data point found but structure may vary across pages, or only tested on limited samples
• Low: Data point sometimes present, or extraction method is uncertain (e.g., relies on specific page state)
• Inconclusive: Cannot determine — element only surfaces after an interaction we did not trigger, or a CAPTCHA stopped traversal. Re-run with the missing interaction explicitly performed (or with valid proxy credentials for protection-blocked endpoints) for definitive results.

Complexity Estimates

• Low: Single data source, no authentication, no rate limits, straightforward extraction. Implementation: < 1 hour.
• Medium: Multiple data sources or endpoints, some authentication or rate limits, but well-documented extraction paths. Implementation: 1-3 hours.
• High: Browser required for some data points, complex authentication flows, strict rate limits, multiple interaction steps, or data spread across many page types. Implementation: 3+ hours.

---

Guidelines for Report Generation

1. Be specific — Include exact URLs, exact JSON paths, exact selectors. The report should contain enough detail to implement without re-running reconnaissance.

2. Be honest about confidence — Mark things as "Low" confidence when uncertain rather than overstating findings.

3. Include all methods found — Even if an API method exists, also document the Cheerio/Browser fallback if available. Methods fail; fallbacks save time.

4. Note what wasn't found — If a requested data point wasn't discovered, say so clearly with notes on what was tried.

5. Keep endpoint specs complete — Every endpoint should have enough detail (URL, method, headers, auth, pagination) to make a working request without revisiting the site.

6. Record rate limits precisely — Include the actual header values or observed thresholds, not just "rate limited."

7. Separate facts from recommendations — Sections 1-4 are facts (what was observed). Section 5 is the recommendation (what to do). Keep them distinct.

---

Related

• Main workflow: See ../SKILL.md Step 6
• Cheerio vs Browser test: See ../strategies/cheerio-vs-browser-test.md

<!-- Canonical source. Copies exist in web-scraper/reference/tool-reference.md and poc-agent/reference/tool-reference.md — keep all three in sync when updating. -->

Tool Reference & Known Limitations

Reference material for the intel-agent skill. Consult when you need tool signatures, caveats, or usage guidance.

Targets proxy-mcp ≥ 2.0.0 (cloakbrowser + Playwright). For older proxy-mcp (1.x, interceptor_chrome_* + CDP sidecar), use the v1 branch of this skill. Camoufox tools (anti-detect Firefox, Step 1 hard-target fallback) require proxy-mcp ≥ 3.0.0.

---

Known Limitations & Tool Caveats

Body preview truncation (the single most important caveat)

• proxy_search_traffic(query) searches body previews only, not full response bodies
• proxy_get_exchange(exchange_id) returns bodyPreview capped at 4 KB
• bodySize field shows the true size — if it's larger than the preview, the preview is incomplete
• A search returning 0 results does NOT prove the content is absent from the full response
• Mitigation: always start the proxy with persistence_enabled: true, capture_profile: "full" (Step 1). Then use proxy_search_session_bodies(session_id, query: "search term") for decompressed full-body search with grep-like context snippets, and proxy_get_session_exchange(session_id, exchange_id, include_body: true) for one full body.
• If neither full body nor rendered DOM snapshot is available, mark the finding as INCONCLUSIVE — body truncated.

Live traffic search vs session search

• proxy_search_traffic() operates on the live in-memory traffic buffer (preview only, cleared by proxy_clear_traffic())
• proxy_query_session() searches session metadata (URL, hostname, path) — it does NOT search inside body content
• proxy_search_session_bodies() searches decompressed full bodies in the persisted session — this is the authoritative body search tool
• For finding text inside HTML/JSON response bodies, always use proxy_search_session_bodies()

Locator vs coordinate clicks

• humanizer_click accepts selector | role + name | text | label | x,y. Prefer the locator forms (role/name/text/label) — they auto-wait for visible + enabled + stable + in-view and handle iframes / shadow DOM / offscreen elements that coordinate-based clicks fail on.
• Fall back to selector for elements without good ARIA, and to x,y only when nothing else resolves.

No CDP surface

• proxy-mcp v2 drives the browser via Playwright. There is no CDP HTTP/WebSocket port, no DevTools sidecar, no attach/detach step. Tools take target_id from interceptor_browser_launch directly.
• If the v1 skill said "use interceptor_chrome_devtools_*" — translate to interceptor_browser_* and drop the attach step.

---

Tool Quick Reference

Initialization

Traffic Analysis (Live Buffer — header peek)

Traffic Analysis (Persisted Session — full bodies)

Browser Inspection

Human Interaction

(cloakbrowser already humanizes mouse/keyboard dispatch by default; the humanizer_* timing profile layers on top. Explicit idle waits are unnecessary — humanizer_click auto-waits for stability and interceptor_browser_navigate(..., wait_until: "networkidle") waits for XHR settle.)

Protection Testing

Session Management & Replay

MCP Resources (read via ReadMcpResourceTool, not tool calls)

---

Important Rules

• Tools take target_id from interceptor_browser_launch directly — no attach/detach step
• Always start the proxy with persistence_enabled: true, capture_profile: "full" — without this, bodies cap at 4 KB and most data-point evidence is truncated
• Prefer locator-based humanizer_click (role + name, text, label) over CSS selectors
• Always proxy_clear_traffic() before an interaction to isolate the traffic it generates
• For geographic testing, relaunch the browser with matching timezone + locale when switching upstream proxy countries — IP geo vs browser geo mismatch is a bot signal
• Use proxy_search_session_bodies() to search inside response/request bodies (not proxy_search_traffic() which only searches previews, and not proxy_query_session() which only searches metadata)
• TLS fingerprint spoofing (proxy_set_fingerprint_spoof) is for non-browser HTTP clients only — cloakbrowser already presents an authentic Chrome fingerprint at the source level
• Reach for camoufox only when cloakbrowser is detected. If you start with camoufox you lose interceptor_browser_* and humanizer_* for that target — drive Playwright yourself via firefox.connect(wsUrl) and skip the snapshot/click/scroll steps below. Host requirements (camoufox path only): pip install "camoufox[geoip]", python3 -m camoufox fetch, and NSS certutil (libnss3-tools/nss-tools/brew install nss)

Cheerio vs Browser Test

Determine whether each data point requires JavaScript rendering or can be extracted from raw HTML.

Overview

The three-way extraction test checks three locations for each data point: 1. Raw HTML — The server's initial HTML response (no JavaScript executed) 2. JSON blobs in HTML — Structured data embedded in <script> tags 3. Rendered DOM — The final page after JavaScript execution

The comparison tells you the simplest extraction method available.

---

Procedure

1. Get Raw HTML

Fetch the main document's response body from captured proxy traffic:

proxy_list_traffic(url_filter: "[target domain]")

Identify the main HTML document exchange — the first GET request returning text/html. Then:

proxy_get_exchange(exchange_id)

Save the response body for searching. This is what Cheerio would see — no JavaScript has run.

Body truncation check: proxy_get_exchange() returns a bodyPreview which may be truncated for large responses. Compare bodySize in the exchange metadata to the actual preview length.

• If bodySize significantly exceeds the preview length: the preview only covers a portion of the HTML (often just <head> and beginning of <body>). Searches of this preview are incomplete.
• To search the full body: use proxy_search_session_bodies(session_id, text: "search term", content_type_contains: "html") which decompresses and searches the complete body, returning context snippets around matches.
• To retrieve the full body of a specific exchange: use proxy_get_session_exchange(session_id, exchange_id: exchange_id, include_body: true).
• If the full body is unavailable: note that raw HTML search results are partial and defer authoritative presence/absence decisions to the rendered DOM snapshot (step 2). A search returning 0 results on a truncated body does NOT mean the data is absent.

2. Get Rendered DOM

Capture the YAML ARIA tree after JavaScript has executed. Scope with selector to save tokens on large pages; use mode: "ai" if you intend to click elements found in the snapshot in a later step:

interceptor_browser_snapshot(target_id)
# or scoped:
interceptor_browser_snapshot(target_id, selector: "main, article")
# or with refs for downstream interaction:
interceptor_browser_snapshot(target_id, mode: "ai")

This represents the fully rendered page — what a real user sees. Match data-point search terms against the role, name, and text fields in the YAML output.

3. Search for Each Data Point

For every data point, search both the raw HTML body and the rendered DOM snapshot for the value or its search terms.

Search strategy by data point type:

• Text (e.g., product name "Ultra Widget Pro"): Search for the exact string
• Numeric (e.g., price "$29.99"): Search for the number with and without formatting ("29.99", "$29.99", "2999")
• Boolean (e.g., in stock): Search for indicator text ("In Stock", "Available", "Sold Out")
• List (e.g., reviews): Search for at least one item's content to confirm presence
• Nested (e.g., address): Search for each sub-field independently

4. Check for JSON Blobs

Search the raw HTML response body for embedded JSON structures. These are script tags that contain structured data the frontend framework hydrates from.

Common patterns:

Schema.org / JSON-LD

<script type="application/ld+json">
{
  "@type": "Product",
  "name": "Ultra Widget Pro",
  "offers": { "price": "29.99" }
}
</script>

Search for: application/ld+json in the raw HTML.

Next.js

<script id="__NEXT_DATA__" type="application/json">
{
  "props": { "pageProps": { "product": { ... } } }
}
</script>

Search for: __NEXT_DATA__ in the raw HTML.

Redux / Vuex / Pinia Hydration

<script>
window.__INITIAL_STATE__ = { "products": { ... } }
</script>

Search for: __INITIAL_STATE__, __PRELOADED_STATE__, or __PINIA__ in the raw HTML.

Nuxt.js

<script>
window.__NUXT__ = { data: [{ product: { ... } }] }
</script>

Search for: __NUXT__ in the raw HTML.

Apollo / Relay GraphQL Cache

<script>
window.__APOLLO_STATE__ = { ... }
window.__RELAY_STORE__ = { ... }
</script>

Search for: __APOLLO_STATE__ or __RELAY_STORE__ in the raw HTML.

When a JSON blob is found, parse it and search for data point values within the JSON structure. Record the JSON path (e.g., props.pageProps.product.price).

---

Decision Matrix

For each data point, classify based on where it was found:

Found in Raw HTML → Cheerio Works

The data point is present in the server's initial HTML response. Cheerio (or any HTML parser) can extract it without a browser.

Advantages: Fast, lightweight, no browser overhead, highly parallelizable.

Record: The approximate location in the HTML (tag, class, ID) for selector construction.

Found in JSON Blob → JSON-in-HTML Works (Preferred)

The data point is present in an embedded JSON structure within the raw HTML.

Advantages: Structured data (no fragile selectors), easy to parse, often contains more fields than what's displayed, same performance as Cheerio.

Why preferred over Cheerio: JSON paths are more stable than CSS selectors. props.pageProps.product.price won't break when the site redesigns, but .product-detail .price-tag span.amount will.

Record: Which JSON blob (__NEXT_DATA__, ld+json, etc.) and the JSON path to the value.

Found in Rendered DOM Only → Browser Required

The data point appears only after JavaScript executes. It's not in the raw HTML or any JSON blob.

Implications: Must use a browser-based approach (Playwright, DevTools bridge). Slower and more resource-intensive.

Record: The accessibility tree location and any wait conditions needed.

Found in None → Requires Interaction

The data point isn't visible in any of the three locations. It may require user interaction to appear.

Next steps:

1. Scroll down — Content may lazy-load:

   proxy_clear_traffic()
   humanizer_scroll(target_id, delta_y: 2000)
   interceptor_browser_snapshot(target_id)
   proxy_list_traffic()

2. Click to reveal — Content behind tabs, accordions, "show more" buttons. Prefer locator-based clicks (role/text/label) over CSS selectors — proxy-mcp v2 auto-waits for visible + enabled + stable + in-view:

   proxy_clear_traffic()
   humanizer_click(target_id, role: "button", name: "Show more")
   # or: humanizer_click(target_id, text: "Read reviews")
   # or: humanizer_click(target_id, selector: ".reveal-btn")   # fallback
   interceptor_browser_snapshot(target_id)
   proxy_list_traffic()

3. Navigate to detail page — Data may only be on individual item pages, not list pages.

4. Search or filter — Some data only appears in specific views.

After each interaction, re-run the three-way test on the newly visible content and check proxy traffic for API calls that were triggered.

Record: The required interaction sequence and whether an API call was triggered (if so, the API method is preferred).

Truncated Body + Rendered DOM Available

When the body preview is truncated but the rendered DOM snapshot IS available:

• Use the rendered DOM as the authoritative check for "is the data point on the page"
• If found in rendered DOM: it exists on the page (method: Browser, but may also be extractable via Cheerio from the full HTML)
• If NOT in rendered DOM: genuinely absent from the page at load time (may still require interaction — go to step above)
• To determine if Cheerio works for DOM-found data points: use proxy_search_session_bodies(session_id, text: "[value]", content_type_contains: "html") to check if the value exists in the raw HTML body

---

Example Walkthrough

Target: https://shop.example.com/products/widget-pro Data points: product name, price, reviews, stock status

Raw HTML search results

JSON blob search results

Rendered DOM search results

Conclusions

For reviews: check proxy_list_traffic() for a reviews API endpoint. If found, API method is preferred over browser.

---

Related

• Main workflow: See ../SKILL.md Step 2
• Proxy tool reference: See ../reference/tool-reference.md