Search

Bright Data — Search

Find things on the web. Two commands live in this skill:

• `bdata search` — classic keyword SERP (Google/Bing/Yandex). Best when you want "what ranks for keyword X."
• `bdata discover` — AI intent-ranked discovery with optional page content. Best when you want "pages about topic Y that match intent Z."

For structured data from a known platform (Amazon, LinkedIn, TikTok, …), stop and use `data-feeds` instead.

Setup gate (run first)

if ! command -v bdata >/dev/null 2>&1; then
    echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md"
elif ! bdata zones >/dev/null 2>&1; then
    echo "bdata not authenticated — run: bdata login  (or: bdata login --device for SSH)"
fi

Halt and route to skills/bright-data-best-practices/references/cli-setup.md if either check fails.

Pick your path

Action

Core commands:

# Google SERP, structured JSON
bdata search "site:example.com privacy policy" --engine google --json --pretty

# Localized Bing (German results, German language)
bdata search "datenschutz" --engine bing --country de --language de --json

# Second page of results (0-indexed)
bdata search "machine learning papers" --page 1 --json

# Mobile SERP (rankings differ from desktop)
bdata search "best coffee shops" --device mobile --json

# News vertical
bdata search "openai" --type news --json --pretty

# Intent-ranked discovery
bdata discover "enterprise LLM platforms" \
    --intent "vendor pages with pricing" \
    --num-results 15 --json

# Discovery with page content in markdown
bdata discover "webhook best practices" \
    --include-content --num-results 10 -o results.json

# Date-filtered discovery
bdata discover "react server components" \
    --start-date 2025-01-01 --end-date 2025-12-31 --num-results 20

Full flag reference: `references/flags.md`.

search vs discover — pick the right one

Verification gate

1. JSON parses cleanly: jq . <output> returns 0. 2. Result array non-empty — if empty, the query is legitimately zero-result; relax the query and re-run. Don't claim success on empty results without telling the user. 3. Required fields present:

• search: results live at .organic[]; each has title + link
• discover: results live at .results[]; each has title + link; if --include-content, also content

4. For `discover --include-content`: no block-page signatures in the content field (same list as scrape, case-insensitive):

• Access Denied
• Just a moment
• Attention Required
• Checking your browser
• captcha
• cf-browser-verification
• cloudflare (with < 2KB total body)

5. Geo sanity: if the user expected country-specific results, inspect TLDs / languages of top results. If mis-localized, re-run with explicit --country and --language.

Red flags

• Using search to fetch content from Amazon, LinkedIn, TikTok, etc. when data-feeds returns clean structured data in one call.
• Scraping every SERP result blindly — filter first (domain allowlist, keyword in title, relevance heuristic).
• Confusing search (keyword) with discover (semantic). They answer different questions.
• Running multiple queries without deduping URLs across result sets before scraping.
• Assuming SERP order is universal — it's personalized by geo + device. Always set --country and --device explicitly for reproducibility.
• Using --page as a result count — it's a page index, not a limit. Each page returns ~10 results.
• Assuming SERP results are at .results[] — for bdata search they live at .organic[]. (Discover uses .results[].)
• Hardcoding --num-results 100 on discover without realizing the pipeline polls until that many are found; can be slow.

References

• `references/flags.md` — full flags for search and discover with when-to-use notes.
• `references/patterns.md` — multi-query dedup, SERP → filter → scrape pipeline, search vs discover decision, legacy curl fallback, shared verification checklist.
• `references/examples.md` — (1) single Google query, (2) localized Bing, (3) batch queries + dedup into URL list, (4) discover --include-content end-to-end.

Search — worked examples

Example 1 — single Google query

bdata search "postgres jsonb index performance" \
    --engine google --country us --json --pretty -o serp.json

# Verify
jq '.organic | length' serp.json        # > 0
jq -r '.organic[0] | "\(.title)\n\(.link)"' serp.json

Example 2 — localized Bing (German)

bdata search "datenschutz-grundverordnung leitfaden" \
    --engine bing --country de --language de --json -o serp-de.json

# Sanity-check locale: top results should have .de TLDs or German text
jq -r '.organic[0:5][].link' serp-de.json

Example 3 — batch queries + dedup into URL list

Given queries.txt with multiple related queries:

mkdir -p out
: > out/all-urls.txt

while IFS= read -r q; do
    hash=$(printf '%s' "$q" | md5sum | cut -c1-8)
    bdata search "$q" --engine google --country us --json \
        -o "out/serp-${hash}.json"
    jq -r '.organic[].link' "out/serp-${hash}.json" >> out/all-urls.txt
done < queries.txt

sort -u out/all-urls.txt > out/unique-urls.txt
echo "Total: $(wc -l < out/all-urls.txt)  Unique: $(wc -l < out/unique-urls.txt)"

Example 4 — discover --include-content end-to-end

Find recent articles on a topic, with body content in one call, ready to feed into an LLM:

bdata discover "post-quantum cryptography deployment" \
    --intent "practical deployment case studies from 2025" \
    --num-results 15 \
    --start-date 2025-01-01 \
    --include-content \
    --country us --language en \
    --timeout 900 \
    --json --pretty -o pqc.json

# Verify count
jq '.results | length' pqc.json              # should be ~15
jq -r '.results[0] | .title, .link' pqc.json
jq -r '.results[0].content' pqc.json | head  # body as markdown

# Block-page sanity (no content should match)
jq -r '.results[].content // empty' pqc.json \
    | grep -iE 'access denied|just a moment|captcha|cloudflare' \
    && echo "WARN: one or more results returned a block page"

bdata search / bdata discover — flag reference

Verified against @brightdata/cli v0.1.8 on 2026-04-19.

bdata search — classic keyword SERP

Usage: bdata search [options] <query>

SERP JSON output shape

When invoked with --json, bdata search returns an envelope with these top-level keys (from the Search_response type):

• general — query metadata (search engine, query, results count, language, device)
• organic — main result array. Each entry has rank, title, link, description (some also have global_rank)
• Vertical result arrays (present depending on the query and --type): news, images, shopping, videos, recipes, maps, paid, product_listing_ads
• Semantic blocks: knowledge_graph, people_also_ask, related_searches, perspectives

Each vertical has its own record shape:

• news → title, link, source, date, image
• images → title, link, source, original_image, image
• shopping → title, link, price, shop, rating, reviews_cnt

To extract just the links from the main organic results: jq -r '.organic[].link'. For news: jq -r '.news[].link'.

bdata discover — AI intent-ranked discovery

Usage: bdata discover [options] <query>

Discover JSON output shape

• results — array of result objects, each with title, link, description, relevance_score (and content when --include-content is set)
• status, timestamp, duration_seconds — job metadata

To extract links: jq -r '.results[].link'. To pull content bodies: jq -r '.results[].content'.

When to use search vs discover

Search — patterns

Verification checklist (shared across all Bright Data CLI skills)

Before claiming a search succeeded:

1. JSON parses cleanly (jq . <output> returns 0). 2. Result array non-empty — if empty, re-check; don't claim success on zero results silently. 3. Expected fields present: title + link at .organic[] (search) or .results[] (discover). If discover --include-content, results also have content. 4. No block-page signatures in discover --include-content bodies:

• Access Denied
• Just a moment
• Attention Required
• Checking your browser
• captcha
• cf-browser-verification
• cloudflare (with < 2KB total body)

5. Geo sanity: result TLDs / languages match the requested --country / --language.

Multi-query batch with dedup

Given queries.txt (one query per line), collect deduped result URLs:

mkdir -p out
: > out/all-urls.txt

while IFS= read -r q; do
    hash=$(printf '%s' "$q" | md5sum | cut -c1-8)
    bdata search "$q" --engine google --country us --json \
        -o "out/serp-${hash}.json"
    jq -r '.organic[].link' "out/serp-${hash}.json" >> out/all-urls.txt
done < queries.txt

sort -u out/all-urls.txt > out/urls.txt
wc -l out/urls.txt

bdata search's JSON envelope holds the main results at .organic[]. Confirm once with jq 'keys' on any SERP output if the shape ever surprises you.

SERP → filter → scrape pipeline

# 1. Search
bdata search "enterprise monitoring tools" --engine google --country us \
    --json -o serp.json

# 2. Filter (domain allowlist; relevance heuristic)
jq -r '.organic[]
    | select(.link | test("^https://(?!.*(reddit|pinterest))"))
    | select(.title | test("monitoring"; "i"))
    | .link' serp.json > urls.txt

# 3. Scrape (hands off to the scrape skill's patterns)
mkdir -p out
xargs -a urls.txt -n 1 -P 4 -I {} bash -c '
    url="$1"
    hash=$(printf "%s" "$url" | md5sum | cut -c1-8)
    bdata scrape "$url" -f markdown -o "out/${hash}.md" || echo "FAIL: $url" >&2
' _ {}

search vs discover decision

Default to search for keyword-exactness tasks (SEO research, "what ranks for X"). Default to discover when the user's description is a topic, intent, or concept rather than a keyword ("pages about how companies adopt LLMs", "recent articles on post-quantum crypto").

Rule of thumb: if the user's phrasing is a complete sentence or describes intent, discover. If it's a short keyword string, search.

Pagination with search

--page is 0-indexed. Loop pages until empty or duplicate:

prev_hash=""
for page in 0 1 2 3 4; do
    bdata search "long tail query" --page "$page" --json -o "p${page}.json"
    hash=$(sha1sum "p${page}.json" | awk '{print $1}')
    # Same hash twice → we're looping; break
    [[ "$hash" == "$prev_hash" ]] && { rm "p${page}.json"; break; }
    # Empty results → break
    count=$(jq '.organic | length' "p${page}.json")
    [[ "$count" == "0" ]] && { rm "p${page}.json"; break; }
    prev_hash=$hash
done

Discover: extracting links and content

# Just links (discover's array is at .results[])
bdata discover "enterprise LLM vendors" --num-results 20 --json -o disc.json
jq -r '.results[].link' disc.json > vendor-urls.txt

# Links + markdown bodies in one call
bdata discover "incident postmortems" \
    --intent "public post-mortem write-ups from 2025" \
    --num-results 15 --include-content \
    --json -o postmortems.json

# Pull just the bodies as separate markdown files
jq -r '.results[] | .link + "\n" + (.content // "")' postmortems.json
# Or write each to a file:
jq -c '.results[]' postmortems.json | while IFS= read -r row; do
    link=$(jq -r '.link' <<<"$row")
    hash=$(printf '%s' "$link" | md5sum | cut -c1-8)
    jq -r '.content // ""' <<<"$row" > "out/${hash}.md"
done

Legacy curl fallback (deprecated)

Only when CLI cannot be installed. SERP API endpoint via Web Unlocker. The CLI's bdata search prefers BRIGHTDATA_SERP_ZONE and falls back to BRIGHTDATA_UNLOCKER_ZONE; pick whichever is set in your environment:

curl -sS "https://api.brightdata.com/request" \
    -H "Authorization: Bearer $BRIGHTDATA_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{
        \"zone\": \"${BRIGHTDATA_SERP_ZONE:-$BRIGHTDATA_UNLOCKER_ZONE}\",
        \"url\": \"https://www.google.com/search?q=$(printf '%s' "$QUERY" | jq -sRr @uri)&brd_json=1\",
        \"format\": \"raw\"
    }"

Prefer the CLI path. This block exists only for environments without Node.js.