Browser Content Capture

Browser Content Capture

Capture web content that traditional scrapers cannot access using agent-browser CLI.

Overview

This skill enables content extraction from sources that require browser-level access:

• JavaScript-rendered SPAs (React, Vue, Angular apps)
• Login-protected documentation (private wikis, gated content)
• Dynamic content (infinite scroll, lazy loading, client-side routing)
• Multi-page site crawls (documentation trees, tutorial series)

When to Use

Use when:

• WebFetch returns empty or partial content
• Page requires JavaScript execution to render
• Content is behind authentication
• Need to navigate multi-page structures
• Extracting from client-side routed apps

Do NOT use when:

• Static HTML pages (use WebFetch - faster)
• Public API endpoints (use direct HTTP calls)
• Simple RSS/Atom feeds

---

Quick Start

Basic Capture Pattern

# 1. Navigate to URL
agent-browser open https://docs.example.com

# 2. Wait for content to render
agent-browser wait --load networkidle

# 3. Get interactive snapshot
agent-browser snapshot -i

# 4. Extract text content
agent-browser get text body

# 5. Take screenshot
agent-browser screenshot /tmp/capture.png

# 6. Close when done
agent-browser close

---

agent-browser Commands Reference

Quick reference: See references/agent-browser-commands.md or run agent-browser --help

---

Capture Patterns

Pattern 1: SPA Content Extraction

For React/Vue/Angular apps where content renders client-side:

# Navigate and wait for hydration
agent-browser open https://react-docs.example.com
agent-browser wait --load networkidle

# Get snapshot to identify content element
agent-browser snapshot -i

# Extract after framework mounts (use ref from snapshot)
agent-browser get text @e5  # Main content area

# Or use eval for custom extraction
agent-browser eval "document.querySelector('article').innerText"

Details: See references/spa-extraction.md

Pattern 2: Authentication Flow

For login-protected content:

# Navigate to login
agent-browser open https://docs.example.com/login
agent-browser snapshot -i

# Fill credentials (refs from snapshot)
agent-browser fill @e1 "user@example.com"  # Email field
agent-browser fill @e2 "password123"        # Password field

# Click submit and wait for redirect
agent-browser click @e3
agent-browser wait --url "**/dashboard"

# Save authenticated state for reuse
agent-browser state save /tmp/auth-state.json

# Now navigate to protected content
agent-browser open https://docs.example.com/private-docs

Details: See references/auth-handling.md

Pattern 3: Multi-Page Crawl

For documentation with navigation trees:

# Get all page links from sidebar
agent-browser open https://docs.example.com
agent-browser snapshot -i

# Extract links via eval
LINKS=$(agent-browser eval "JSON.stringify(Array.from(document.querySelectorAll('nav a')).map(a => a.href))")

# Iterate and capture each page
for link in $(echo "$LINKS" | jq -r '.[]'); do
    agent-browser open "$link"
    agent-browser wait --load networkidle
    agent-browser get text body > "/tmp/content-$(basename $link).txt"
done

Details: See references/multi-page-crawl.md

---

Session Management

Save and Reuse Authentication

# Login once and save state
agent-browser open https://app.example.com/login
agent-browser snapshot -i
agent-browser fill @e1 "$USERNAME"
agent-browser fill @e2 "$PASSWORD"
agent-browser click @e3
agent-browser wait --url "**/dashboard"
agent-browser state save /tmp/app-auth.json

# Later: restore state
agent-browser state load /tmp/app-auth.json
agent-browser open https://app.example.com/protected-content

Parallel Sessions

# Run isolated sessions for different tasks
agent-browser --session scrape1 open https://site1.com
agent-browser --session scrape2 open https://site2.com

# Extract from each
agent-browser --session scrape1 get text body > site1.txt
agent-browser --session scrape2 get text body > site2.txt

---

Fallback Strategy

Use this decision tree for content capture:

User requests content from URL
         │
         ▼
    ┌─────────────┐
    │ Try WebFetch│ ← Fast, no browser needed
    └─────────────┘
         │
    Content OK? ──Yes──► Done
         │
         No (empty/partial)
         │
         ▼
    ┌──────────────────┐
    │ Use agent-browser│
    └──────────────────┘
         │
    ├─ Known SPA (react, vue, angular) ──► wait --load networkidle
    ├─ Requires login ──► Authentication flow with state save
    └─ Dynamic content ──► wait @element or wait --text

---

Best Practices

1. Minimize Browser Usage

• Always try WebFetch first (10x faster, no browser overhead)
• Cache extracted content to avoid re-scraping
• Use get text @e# to extract only needed content

2. Handle Dynamic Content

• Always use wait after navigation
• Use wait --load networkidle for heavy SPAs
• Use wait --text "Expected" for specific content

3. Respect Rate Limits

• Add delays between page navigations
• Don't crawl faster than a human would browse
• Honor robots.txt and terms of service

4. Clean Extracted Content

• Use targeted refs from snapshot to extract main content
• Use eval to remove noise elements before extraction
• Convert to clean markdown for downstream processing

---

Troubleshooting

---

Related Skills

• browser-automation - agent-browser CLI quick start and integration
• webapp-testing - Playwright test automation patterns
• streaming-api-patterns - Handle SSE progress updates

---

Version: 2.0.0 (January ) Browser Tool: agent-browser CLI (replaces Playwright MCP)

Capability Details

spa-extraction

Keywords: react, vue, angular, spa, javascript, client-side, hydration, ssr Solves:

• WebFetch returns empty content
• Page requires JavaScript to render
• React/Vue app content extraction

auth-handling

Keywords: login, authentication, session, cookie, protected, private, gated Solves:

• Content behind login wall
• Need to authenticate first
• Private documentation access

multi-page-crawl

Keywords: crawl, sitemap, navigation, multiple pages, documentation, tutorial series Solves:

• Capture entire documentation site
• Extract multiple pages
• Follow navigation links

agent-browser-commands

Keywords: agent-browser, open, snapshot, click, fill, eval, get text Solves:

• Which command to use
• Browser automation reference
• agent-browser CLI guide

Browser Content Capture Checklist

Use this checklist when capturing content from web pages using agent-browser.

Pre-Capture

• [ ] Try WebFetch first - Only use browser if WebFetch fails
• [ ] Check robots.txt - Ensure scraping is allowed
• [ ] Verify ToS - Review site's terms of service
• [ ] Identify page type - Static, SPA, or auth-protected?

Page Analysis

• [ ] Find content selector - Identify main content container
• Common: article, main, .content, .markdown-body
• [ ] Find navigation selector - For multi-page crawls
• Common: nav a, .sidebar a, .toc a
• [ ] Check for dynamic loading - Lazy content, infinite scroll?
• [ ] Identify loading indicators - Spinners, skeletons, etc.
• [ ] Note framework - React, Vue, Angular, Next.js, Nuxt?

Capture Configuration

• [ ] Set appropriate wait - networkidle for SPAs
• [ ] Add hydration wait - wait --fn for React/Vue
• [ ] Configure rate limiting - 1-2 seconds between pages
• [ ] Plan error handling - Retry logic for failures

Single Page Capture

# 1. Navigate
agent-browser open "$TARGET_URL"

# 2. Wait for content
agent-browser wait --load networkidle

# 3. Get snapshot to identify elements
agent-browser snapshot -i

# 4. Extract content
agent-browser get text body
# Or use specific ref: agent-browser get text @e5

• [ ] Navigation successful
• [ ] Content visible in snapshot
• [ ] Extracted content is complete (not partial)
• [ ] No JavaScript errors (agent-browser errors)

Multi-Page Crawl

• [ ] Discover all pages - Extract navigation links first
• [ ] Deduplicate URLs - Remove duplicate/anchor links
• [ ] Order pages logically - Follow site structure
• [ ] Track visited pages - Prevent infinite loops
• [ ] Handle pagination - Next/Previous links

# Get all nav links
LINKS=$(agent-browser eval "JSON.stringify(Array.from(document.querySelectorAll('nav a')).map(a => a.href))")

# Crawl each
for link in $(echo "$LINKS" | jq -r '.[]'); do
    agent-browser open "$link"
    agent-browser wait --load networkidle
    agent-browser get text body > "/tmp/$(basename $link).txt"
    sleep 1
done

Authentication Handling

• [ ] Check if login required - Detect login redirects
• [ ] Choose auth method:
• [ ] Form-based login (fill @e1, click @e2)
• [ ] Headed mode for OAuth/SSO (AGENT_BROWSER_HEADED=1)
• [ ] Restore saved state (state load)
• [ ] Store no credentials in code - Use environment variables
• [ ] Verify login success - Check URL after redirect
• [ ] Save state for reuse - agent-browser state save

# Login flow
agent-browser open "$LOGIN_URL"
agent-browser snapshot -i
agent-browser fill @e1 "$USERNAME"
agent-browser fill @e2 "$PASSWORD"
agent-browser click @e3
agent-browser wait --url "**/dashboard"
agent-browser state save /tmp/auth.json

Content Extraction

• [ ] Remove noise elements - Nav, header, footer, ads
• [ ] Extract clean text - get text or eval innerText
• [ ] Preserve structure - Headings, lists, code blocks
• [ ] Extract code separately - Language detection, formatting
• [ ] Capture metadata - Title, URL, date

# Clean extraction
agent-browser eval "
['nav', 'header', 'footer', '.sidebar'].forEach(sel =>
    document.querySelectorAll(sel).forEach(el => el.remove()));
document.querySelector('main, article, .content').innerText;
"

Post-Capture

• [ ] Validate content - Not empty, not error page
• [ ] Clean whitespace - Remove excessive newlines
• [ ] Word count check - Reasonable length for page type
• [ ] Take screenshot - Visual verification

Troubleshooting

Quality Verification

• [ ] Random sample check - Review 3-5 captured pages manually
• [ ] Search test - Query for expected content
• [ ] Compare to source - Ensure no content lost
• [ ] Check code blocks - Properly formatted and complete

Documentation

• [ ] Record capture date - For freshness tracking
• [ ] Note refs used - For future re-crawls
• [ ] Document failures - Pages that couldn't be captured
• [ ] Save capture scripts - For reproducibility

agent-browser Quick Reference for Content Capture

Commands most relevant to browser content capture workflows. Run agent-browser --help for the full 60+ command reference.

Upstream docs: github.com/vercel-labs/agent-browser

Authentication Handling

Patterns for accessing login-protected content using agent-browser.

Authentication Methods

Method Comparison

Decision Tree

Protected content needed
         │
         ▼
    Have saved state?
         │
    ├─ Yes ──► Load state: agent-browser state load auth.json
    │
    └─ No ──► Check login type
                   │
         ├─ Simple form ──► Fill form with refs
         ├─ OAuth popup ──► Pause for user (--headed)
         └─ SSO ──► Pause for user (--headed)

---

Form-Based Login

Basic Login Flow

# 1. Navigate to login page
agent-browser open https://app.example.com/login

# 2. Wait for form to load
agent-browser wait --load networkidle

# 3. Get form structure
agent-browser snapshot -i
# Output shows: @e1 [input] "Email", @e2 [input] "Password", @e3 [button] "Sign In"

# 4. Fill credentials
agent-browser fill @e1 "$EMAIL"
agent-browser fill @e2 "$PASSWORD"

# 5. Submit form
agent-browser click @e3

# 6. Wait for redirect to dashboard
agent-browser wait --url "**/dashboard"

# 7. Save state for reuse
agent-browser state save /tmp/auth-state.json

# 8. Now navigate to protected content
agent-browser open https://app.example.com/private-docs

Multi-Step Login (Email then Password)

agent-browser open https://app.example.com/login
agent-browser snapshot -i

# Step 1: Email
agent-browser fill @e1 "$EMAIL"
agent-browser click @e2  # Next button

# Step 2: Password
agent-browser wait --fn "document.querySelector('[type=password]') !== null"
agent-browser snapshot -i
agent-browser fill @e1 "$PASSWORD"
agent-browser click @e2  # Sign in button

agent-browser wait --url "**/dashboard"

Handle Login Errors

# Check for error messages after login attempt
ERROR=$(agent-browser eval "
const err = document.querySelector('.error-message, .alert-error, [role=\"alert\"]');
err ? err.innerText : '';
")

if [[ -n "$ERROR" ]]; then
    echo "Login failed: $ERROR"
fi

---

OAuth/SSO Flows

For OAuth (Google, GitHub) and SSO, use headed mode for user interaction:

# 1. Start in headed mode
AGENT_BROWSER_HEADED=1 agent-browser open https://app.example.com/login

# 2. Click OAuth button
agent-browser snapshot -i
agent-browser click @e4  # "Sign in with Google"

# 3. PAUSE - User must complete OAuth flow
echo "Please complete sign-in in the browser window..."

# 4. Wait for redirect back to app
agent-browser wait --url "**/dashboard" --timeout 120000

# 5. Save state for future sessions
agent-browser state save /tmp/oauth-state.json

---

Session Management

Save and Restore State

# SAVE: After successful login
agent-browser state save /tmp/auth-state.json

# RESTORE: In new session
agent-browser state load /tmp/auth-state.json
agent-browser open https://app.example.com/dashboard

Check Login State

# Verify if already logged in
IS_LOGGED_IN=$(agent-browser eval "
const hasLogout = document.querySelector('[href*=\"logout\"], .logout-button');
const hasProfile = document.querySelector('.user-avatar, .profile-menu');
!!(hasLogout || hasProfile);
")

if [[ "$IS_LOGGED_IN" == "true" ]]; then
    echo "Already logged in"
else
    echo "Need to authenticate"
fi

Handle Session Expiry

# Check if redirected to login
CURRENT_URL=$(agent-browser get url)

if [[ "$CURRENT_URL" == *"/login"* ]]; then
    echo "Session expired, re-authenticating..."
    rm -f /tmp/auth-state.json
    # Trigger login flow again
fi

---

Persist Across Captures

When doing multiple captures, maintain login state:

# Login once
agent-browser open https://app.example.com/login
# ... fill credentials ...
agent-browser state save /tmp/auth.json

# Capture multiple pages (session persists)
PAGES=(
    "https://app.example.com/docs/intro"
    "https://app.example.com/docs/guide"
    "https://app.example.com/docs/api"
)

for page_url in "${PAGES[@]}"; do
    agent-browser open "$page_url"
    agent-browser wait --load networkidle
    agent-browser get text body > "/tmp/$(basename $page_url).txt"
done

---

Security Considerations

Never Store Credentials in Code

# BAD - Don't do this
PASSWORD="hardcoded-password"

# GOOD - Use environment variables
agent-browser fill @e2 "$APP_PASSWORD"

Secure State Files

# Set restrictive permissions
chmod 600 /tmp/auth-state.json

# Store in secure location
STATE_FILE="$HOME/.config/agent-browser/auth-state.json"
mkdir -p "$(dirname "$STATE_FILE")"

# Clean up after use
trap 'rm -f "$STATE_FILE"' EXIT

Handle Sensitive Sites with Headed Mode

For sites with:

• 2FA/MFA requirements
• CAPTCHA challenges
• Device verification

Use headed mode for manual completion:

AGENT_BROWSER_HEADED=1 agent-browser open https://secure-site.com/login
echo "Please complete authentication manually..."
agent-browser wait --url "**/authenticated"
agent-browser state save /tmp/secure-auth.json

---

Common Sites

GitHub Private Repos

# Use state from previous GitHub login
agent-browser state load /tmp/github-auth.json
agent-browser open https://github.com/org/private-repo
agent-browser wait --load networkidle

Confluence/Jira (SSO)

AGENT_BROWSER_HEADED=1 agent-browser open https://company.atlassian.net
echo "Complete SSO authentication..."
agent-browser wait --url "**/wiki" --timeout 120000
agent-browser state save /tmp/atlassian-auth.json

Notion

AGENT_BROWSER_HEADED=1 agent-browser open https://notion.so
echo "Complete Notion login..."
agent-browser wait --url "**/workspace"
agent-browser state save /tmp/notion-auth.json

Multi-Page Crawl

Patterns for extracting content from multiple pages using agent-browser.

Overview

Multi-page crawling is needed when:

• Documentation spans multiple pages
• Content is paginated
• Need to follow navigation links
• Building comprehensive content index

---

Basic Crawl Pattern

Extract Links, Then Visit

# 1. Navigate to starting page
agent-browser open https://docs.example.com

# 2. Wait for page to load
agent-browser wait --load networkidle

# 3. Extract all navigation links
LINKS=$(agent-browser eval "
JSON.stringify(
    Array.from(document.querySelectorAll('nav a, .sidebar a'))
        .map(a => a.href)
        .filter(href => href.startsWith('https://docs.example.com'))
)
")

# 4. Visit each link and extract
for link in $(echo "$LINKS" | jq -r '.[]'); do
    echo "Extracting: $link"
    agent-browser open "$link"
    agent-browser wait --load networkidle
    agent-browser get text body > "/tmp/$(basename "$link").txt"
done

# 5. Close browser
agent-browser close

---

Structured Crawl with Metadata

#!/bin/bash
# Crawl with metadata extraction

OUTPUT_DIR="/tmp/docs-crawl"
mkdir -p "$OUTPUT_DIR"

agent-browser open https://docs.example.com
agent-browser wait --load networkidle

# Get links with titles
PAGES=$(agent-browser eval "
JSON.stringify(
    Array.from(document.querySelectorAll('nav a'))
        .map(a => ({
            url: a.href,
            title: a.innerText.trim()
        }))
        .filter(p => p.url.startsWith(window.location.origin))
)
")

# Process each page
echo "$PAGES" | jq -c '.[]' | while read -r page; do
    URL=$(echo "$page" | jq -r '.url')
    TITLE=$(echo "$page" | jq -r '.title')
    FILENAME=$(echo "$TITLE" | tr ' ' '-' | tr '[:upper:]' '[:lower:]')

    echo "Crawling: $TITLE"
    agent-browser open "$URL"
    agent-browser wait --load networkidle

    # Save content with metadata
    {
        echo "---"
        echo "title: $TITLE"
        echo "url: $URL"
        echo "crawled_at: $(date -Iseconds)"
        echo "---"
        echo ""
        agent-browser get text body
    } > "$OUTPUT_DIR/$FILENAME.md"
done

agent-browser close
echo "Crawl complete: $(ls "$OUTPUT_DIR" | wc -l) pages"

---

Pagination Handling

Click-Based Pagination

#!/bin/bash
# Handle "Next" button pagination

PAGE=1
while true; do
    echo "Extracting page $PAGE..."

    # Extract current page content
    agent-browser get text body > "/tmp/page-$PAGE.txt"

    # Check for next button
    agent-browser snapshot -i
    NEXT_BUTTON=$(agent-browser eval "
        const next = document.querySelector('.next, [rel=\"next\"], a:has-text(\"Next\")');
        next ? 'found' : 'none';
    ")

    if [[ "$NEXT_BUTTON" == "none" ]]; then
        echo "No more pages"
        break
    fi

    # Click next
    agent-browser click @e1  # Next button ref from snapshot
    agent-browser wait --load networkidle
    ((PAGE++))
done

URL-Based Pagination

#!/bin/bash
# Handle URL parameter pagination

BASE_URL="https://api.example.com/docs"
PAGE=1

while true; do
    URL="${BASE_URL}?page=${PAGE}"
    echo "Fetching: $URL"

    agent-browser open "$URL"
    agent-browser wait --load networkidle

    # Check if page has content
    HAS_CONTENT=$(agent-browser eval "
        document.querySelector('.content').children.length > 0
    ")

    if [[ "$HAS_CONTENT" != "true" ]]; then
        echo "No more content at page $PAGE"
        break
    fi

    agent-browser get text body > "/tmp/page-$PAGE.txt"
    ((PAGE++))
done

---

Recursive Crawl

Follow All Links (Depth-Limited)

#!/bin/bash
# Recursive crawl with depth limit

MAX_DEPTH=3
VISITED_FILE="/tmp/visited-urls.txt"
touch "$VISITED_FILE"

crawl_page() {
    local url="$1"
    local depth="$2"

    # Skip if already visited
    grep -qF "$url" "$VISITED_FILE" && return

    # Skip if too deep
    [[ $depth -gt $MAX_DEPTH ]] && return

    echo "[$depth] Crawling: $url"
    echo "$url" >> "$VISITED_FILE"

    agent-browser open "$url"
    agent-browser wait --load networkidle

    # Save content
    local filename
    filename=$(echo "$url" | md5sum | cut -d' ' -f1)
    agent-browser get text body > "/tmp/crawl/$filename.txt"

    # Get child links
    local links
    links=$(agent-browser eval "
        JSON.stringify(
            Array.from(document.querySelectorAll('a'))
                .map(a => a.href)
                .filter(h => h.startsWith('$BASE_URL'))
        )
    ")

    # Recursively crawl children
    for link in $(echo "$links" | jq -r '.[]' | head -20); do
        crawl_page "$link" $((depth + 1))
    done
}

BASE_URL="https://docs.example.com"
mkdir -p /tmp/crawl
crawl_page "$BASE_URL" 0
agent-browser close

---

Parallel Crawling with Sessions

#!/bin/bash
# Use multiple sessions for parallel crawling

URLS=(
    "https://docs.example.com/page1"
    "https://docs.example.com/page2"
    "https://docs.example.com/page3"
    "https://docs.example.com/page4"
)

# Start parallel sessions
for i in "${!URLS[@]}"; do
    SESSION="crawler-$i"
    URL="${URLS[$i]}"

    (
        agent-browser --session "$SESSION" open "$URL"
        agent-browser --session "$SESSION" wait --load networkidle
        agent-browser --session "$SESSION" get text body > "/tmp/page-$i.txt"
        agent-browser --session "$SESSION" close
    ) &
done

# Wait for all to complete
wait
echo "All pages crawled"

---

Best Practices

1. Rate Limiting

# Add delay between requests
for url in "${URLS[@]}"; do
    agent-browser open "$url"
    agent-browser wait --load networkidle
    agent-browser get text body > "/tmp/$(basename "$url").txt"
    sleep 1  # 1 second delay between requests
done

2. Error Handling

# Handle failed pages gracefully
for url in "${URLS[@]}"; do
    if ! agent-browser open "$url" 2>/dev/null; then
        echo "Failed to load: $url" >> /tmp/failed-urls.txt
        continue
    fi
    agent-browser get text body > "/tmp/$(basename "$url").txt"
done

3. Resume Capability

# Skip already crawled pages
CRAWLED_DIR="/tmp/crawled"
mkdir -p "$CRAWLED_DIR"

for url in "${URLS[@]}"; do
    HASH=$(echo "$url" | md5sum | cut -d' ' -f1)
    OUTPUT="$CRAWLED_DIR/$HASH.txt"

    if [[ -f "$OUTPUT" ]]; then
        echo "Skipping (already crawled): $url"
        continue
    fi

    agent-browser open "$url"
    agent-browser wait --load networkidle
    agent-browser get text body > "$OUTPUT"
done

4. Respect robots.txt

# Check robots.txt before crawling
ROBOTS=$(curl -s "https://docs.example.com/robots.txt")
if echo "$ROBOTS" | grep -q "Disallow: /docs"; then
    echo "Crawling /docs is disallowed by robots.txt"
    exit 1
fi

SPA Content Extraction

Patterns for extracting content from JavaScript-rendered Single Page Applications using agent-browser.

Why SPAs Are Different

Traditional scrapers fail on SPAs because:

1. Initial HTML is empty - Content loads via JavaScript 2. Hydration timing - React/Vue must "hydrate" before content is interactive 3. Client-side routing - URLs change without page reloads 4. Lazy loading - Content loads as user scrolls 5. API-driven - Data fetched from backend after page load

Solution: Use agent-browser to wait for JavaScript execution.

---

Detection Patterns

Identify SPA Framework

# Check for React
agent-browser eval "window.__REACT_DEVTOOLS_GLOBAL_HOOK__ !== undefined"

# Check for Vue
agent-browser eval "window.__VUE__ !== undefined"

# Check for Angular
agent-browser eval "window.ng !== undefined"

# Check for Next.js
agent-browser eval "document.querySelector('#__next') !== null"

# Check for Nuxt
agent-browser eval "document.querySelector('#__nuxt') !== null"

---

React Extraction

Wait for React Hydration

# Navigate
agent-browser open https://react-docs.example.com

# Wait for React to render content
agent-browser wait --load networkidle

# Or wait for specific hydration marker
agent-browser wait --fn "document.querySelector('[data-hydrated]') !== null"

# Get snapshot to find content
agent-browser snapshot -i

# Extract content
agent-browser get text @e5

Next.js Specific

# Wait for Next.js
agent-browser open https://nextjs-site.com
agent-browser wait --fn "document.querySelector('#__next').children.length > 0"
agent-browser snapshot -i
agent-browser get text @e3

Docusaurus Sites

agent-browser open https://docusaurus-docs.com
agent-browser wait --load networkidle
agent-browser eval "document.querySelector('.theme-doc-markdown').innerText"

---

Vue Extraction

Wait for Vue Mount

# Navigate
agent-browser open https://vue-app.example.com

# Wait for Vue to mount
agent-browser wait --fn "document.querySelector('#app').children.length > 0"

# Or wait for Vue data attributes
agent-browser wait --fn "document.querySelector('[data-v-]') !== null"

# Extract
agent-browser snapshot -i
agent-browser get text @e4

Nuxt Specific

agent-browser open https://nuxt-site.com
agent-browser wait --fn "document.querySelector('#__nuxt').children.length > 0"
agent-browser snapshot -i
agent-browser get text @e2

VitePress/VuePress

# VitePress
agent-browser open https://vitepress-docs.com
agent-browser wait --fn "document.querySelector('.vp-doc') !== null"
agent-browser eval "document.querySelector('.vp-doc').innerText"

---

Angular Extraction

Wait for Angular Bootstrap

# Navigate
agent-browser open https://angular-app.example.com

# Wait for Angular
agent-browser wait --fn "document.querySelector('app-root').children.length > 0"

# Or check ng-version attribute
agent-browser wait --fn "document.querySelector('[ng-version]') !== null"

# Extract
agent-browser snapshot -i
agent-browser get text @e3

---

Generic SPA Patterns

Wait for Content, Not Framework

When framework is unknown, wait for visible content:

# Wait for meaningful content (page has substantial text)
agent-browser wait --fn "document.body.innerText.trim().length > 500"

# Or wait for specific text
agent-browser wait --text "Welcome"

Handle Infinite Scroll

# Scroll to load all content
agent-browser eval "
async function scrollToBottom() {
    let lastHeight = document.body.scrollHeight;
    while (true) {
        window.scrollTo(0, document.body.scrollHeight);
        await new Promise(r => setTimeout(r, 1000));
        if (document.body.scrollHeight === lastHeight) break;
        lastHeight = document.body.scrollHeight;
    }
    return document.body.innerText;
}
scrollToBottom();
"

Handle Lazy Images

# Trigger lazy image loading
agent-browser eval "
document.querySelectorAll('img[data-src]').forEach(img => img.src = img.dataset.src);
document.querySelectorAll('img[loading=\"lazy\"]').forEach(img => img.loading = 'eager');
"
agent-browser wait 2000

Extract Clean Content

# Remove noise elements before extraction
agent-browser eval "
['nav', 'header', 'footer', '.sidebar', '.ads', '.cookie-banner']
    .forEach(sel => document.querySelectorAll(sel).forEach(el => el.remove()));
const main = document.querySelector('main, article, .content, #content');
main ? main.innerText : document.body.innerText;
"

---

Troubleshooting

Crawl multi-page content from: $ARGUMENTS

Crawl Context (Auto-Detected)

• Base URL: $ARGUMENTS
• Agent-Browser Available: !which agent-browser >/dev/null 2>&1 && echo "✅ Yes" || echo "❌ Not found"
• Curl Available: !which curl >/dev/null 2>&1 && echo "✅ Yes" || echo "❌ Not found"
• Output Directory: !pwd/crawled
• Timestamp: !date +%Y%m%d-%H%M%S

Your Task

Crawl multiple pages from base URL: $ARGUMENTS

First check for a sitemap at $ARGUMENTS/sitemap.xml, then discover pages from navigation.

Crawl Workflow

1. Discover Pages

# Check for sitemap
curl -s "$ARGUMENTS/sitemap.xml" | grep -oP '<loc>\K[^<]+' || echo "No sitemap"

# Or discover from navigation
agent-browser open "$ARGUMENTS"
agent-browser eval "
  JSON.stringify(
    Array.from(document.querySelectorAll('nav a, .sidebar a'))
      .map(a => a.href)
      .filter(h => h.startsWith(window.location.origin))
  )
"

2. Crawl Script

#!/bin/bash
START_URL="$ARGUMENTS"
OUTPUT_DIR="./crawled"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

mkdir -p "$OUTPUT_DIR"

# Discover pages
agent-browser open "$START_URL"
agent-browser wait --load networkidle

# Extract links
LINKS=$(agent-browser eval "
  JSON.stringify(
    Array.from(document.querySelectorAll('nav a, .sidebar a'))
      .map(a => ({ href: a.href, title: a.innerText.trim() }))
      .filter(l => l.href && l.href.startsWith(window.location.origin))
  )
")

# Process each page
echo "$LINKS" | jq -c '.[]' | while read -r page; do
  URL=$(echo "$page" | jq -r '.href')
  TITLE=$(echo "$page" | jq -r '.title')
  
  agent-browser open "$URL"
  agent-browser wait --load networkidle
  agent-browser get text body > "$OUTPUT_DIR/$(echo "$TITLE" | tr ' /' '-').md"
  sleep 1
done

agent-browser close
echo "✅ Crawled to $OUTPUT_DIR"

Output

All pages saved to crawled/ directory with markdown format.