Competitor Intel

Competitor Intel

End-to-end competitor research and monitoring. Define the set, pull from every public surface that matters, diff against last run (or your own), and produce a brief that's actually useful — battle card, weekly digest, board update, or comparison-page input.

Out of scope — defer to other skills

competitor-intel is the integration layer — it pulls from every source and synthesizes. It uses HyperSEO for the rank/backlink/intersection slice but isn't the SEO research skill itself.

Requirements

• Hyper MCP installed and connected. https://app.hyperfx.ai/mcp
• At least one of these toolkits connected at https://app.hyperfx.ai/integrations:
• Firecrawl (highly recommended — backbone for any site/blog/pricing-page work)
• HyperSEO (needed for rank, backlink, domain-overlap analysis)
• Apify scrapers — Instagram, TikTok, LinkedIn, Twitter, Reddit, Google search, Google Trends
• Image generation (optional — only if the brief feeds a comparison-page or battle-card asset downstream)

If none of those tool prefixes appear in the agent's tool list (firecrawl_*, hyperseo_*, scrape_instagram*, scrape_tiktok*, search_tweets, scrape_reddit*, search_google_results, scrape_google_trends, web_scrape_page), stop and tell the user to enable the Hyper MCP and connect at least Firecrawl + one social scraper. The LinkedIn scraper (scrape_linkedin_profiles) is only present when that specific integration is enabled — gracefully skip the LinkedIn slice if it's missing rather than failing the whole brief.

Tool surface

Critical rules

1. Public-only data. Never scrape behind login walls or paywalls. If a target is gated, stop and surface that — don't try to bypass. 2. Pick the competitor set before scraping. 3–5 competitors is the sweet spot. 10+ produces an unreadable brief and burns scraper credits. If the user doesn't have a list, run hyperseo_competitors(domain=<their_domain>) first to surface the top organic competitors, then confirm the set with them. 3. Snapshot, then diff. First run is just a baseline — there's nothing to compare against. The value compounds on the second and third runs (what changed in pricing, what new posts went up, who lost rank). Make this expectation clear when the user runs it for the first time. 4. HyperSEO is credit-metered. Be intentional. Don't loop hyperseo_domain_overview over 12 competitors when you only need 4. Each call has cost — batch and cache. 5. Apify scrapers can be slow and rate-limited. Don't kick off 8 scrapes in parallel. Sequence them, and surface partial results to the user as they arrive instead of waiting for the full run. 6. Snapshot what you scraped, not just the analysis. Always include the source URL, scrape timestamp, and a short excerpt for any claim in the brief — otherwise next week's diff has nothing to compare against and the brief becomes unfalsifiable. 7. Don't over-interpret single data points. "Competitor X dropped a Reel that got 12K likes" is noise. "Competitor X has averaged 8K likes/post for the last 30 days, up from 2K" is signal. Build comparisons on aggregates, not anecdotes. 8. Stay clearly factual. Use neutral language ("Competitor X published Y on date Z, copy reads as…") not value judgments ("Competitor X's strategy is broken…"). The brief is intel, not opinion. 9. Disambiguate brand-name SERPs. A search for <competitor> alone often returns unrelated results that share the brand name (e.g. a search for "hyperfx" returns mostly HyperX headphones, not hyperfx.ai). Always pair the brand with a category modifier — <competitor> alternative, <competitor> reviews, <competitor> pricing, <competitor> vs <us> — to get clean SERPs. 10. Apify-backed scrapers fail intermittently. Expect occasional "fetch failed" or empty-result responses from scrape_instagram*, scrape_tiktok*, scrape_reddit*, scrape_google_trends, search_tweets, and search_google_results. Retry once after a short delay before reporting the source as missing — and surface partial results to the user rather than failing the whole brief if a scraper stays down.

Workflow

Phase 1 — Define the competitor set

Ask the user (or infer):

1. Their domain — needed to ground all relative comparisons. 2. Their competitor set — 3–5 names + domains. If unclear, surface the top organic competitors. hyperseo_competitors takes the keywords the user wants to win, not their domain — so first agree on 3–5 high-intent keywords for the user's category, then run:

hyperseo_competitors(
  keywords=["<category keyword 1>", "<category keyword 2>", "<category keyword 3>"],
  location_code=2840,  # 2840=US, 2826=UK, 2124=CA, 2036=AU
  limit=10
)

Then confirm the surfaced set with the user before continuing — never guess and proceed.

3. The job — what is this brief for? The shape changes by job:

• Battle card for sales → focus on positioning, pricing, "what to say when…" objection lines.
• Weekly digest for marketing/exec team → focus on what changed this week.
• Comparison-page input for marketing → focus on objective, comparable feature/pricing data.
• Board / board-prep update → focus on aggregate position (share of voice, rank deltas, growth).

4. The cadence — one-shot or recurring? Recurring scopes the source matrix tighter so each run completes in reasonable time.

If the user can't answer #3, stop and ask. Building intel without knowing the artifact is a guaranteed waste.

Phase 2 — Build the source matrix

Not every source matters for every competitor or every job. Per-competitor, decide which surfaces to pull from.

The full per-source playbook (when to use, what it reveals, quirks) lives in `references/source-by-source.md`.

Phase 3 — Pull the data

Sequence the pulls; don't fire everything in parallel. Order matters for cost and for letting partial results inform the next call.

Order to use:

1. Cheapest first — HyperSEO baselines (one call per competitor, reuse data across the brief):

hyperseo_domain_overview(domain="<competitor>.com", location_code=2840)
hyperseo_domain_keywords(domain="<competitor>.com", limit=50, location_code=2840)
hyperseo_domain_intersection(
  domain1="yourbrand.com",
  domain2="<competitor>.com",
  limit=50,
  location_code=2840
)

2. Site content via Firecrawl:

firecrawl_scrape_url(url="https://<competitor>.com/pricing")
firecrawl_scrape_url(url="https://<competitor>.com/")
firecrawl_extract_branding(url="https://<competitor>.com")  # logo, colors, voice

For a full-blog-archive deep dive use firecrawl_crawl_website once and check status with firecrawl_check_crawl_status. If Firecrawl returns near-empty for a JS-heavy SPA, fall back to web_scrape_page(url=..., use_proxy=true, ai_query="extract pricing tiers and prices").

3. Social — sequenced per platform (note: most scrapers take arrays of identifiers, not a single username):

scrape_instagram(direct_urls=["https://www.instagram.com/<competitor>/"], results_type="posts", results_limit=30)
scrape_instagram_followers_count(usernames=["<competitor>"])  # cheap, do it weekly
scrape_tiktok_videos(profiles=["<competitor>"], results_per_page=30)
search_tweets(from_user="<competitor_handle>", max_items=50)

# Only if scrape_linkedin_profiles is in your MCP tool list:
scrape_linkedin_profiles(urls=["https://www.linkedin.com/company/<competitor>"])

4. Sentiment & demand:

scrape_reddit(searches=["<competitor>"], max_items=50, sort="new", time="month")
scrape_google_trends(
  search_terms=["<competitor>", "yourbrand"],
  time_range="today 3-m",  # options: now 7-d, today 1-m, today 3-m, today 5-y, all
  geo="US"
)
search_google_results(query="<competitor> reviews", num_results=20, country="us")

5. AI search visibility (only for SaaS/B2B usually):

# AI Overview takes ONE keyword at a time + integer location_code
hyperseo_ai_overview(keyword="<your category keyword>", location_code=2840)

# track_mentions runs the query against OpenAI / Claude / Perplexity and returns citations
hyperseo_track_mentions(
  query="best <your category> tools",
  brands=["<competitor>", "yourbrand"]
)

Surface partial results as each phase completes — don't wait for the whole pull to finish before showing the user something.

Phase 4 — Diff

The point of the skill is delta, not snapshot. For each competitor, compare:

For a first run, there's no "last" — the diff section in the brief becomes "baseline established, will diff against this on next run." Tell the user this explicitly so they don't expect insight on day 1.

Phase 5 — Brief

Synthesize into the artifact. Brief shapes by job (full templates in `references/brief-templates.md`):

• Battle card — 1 page, per-competitor: positioning summary, 3 strongest competitor claims, 3 counter-claims for our side, common objections + responses, win-loss patterns.
• Weekly digest — 1 page total, all competitors: bullet list of what changed this week per competitor, ranked by significance.
• Comparison-page input — feature matrix table, pricing table, plan limits table — all sourced with URLs and timestamps.
• Board-prep update — share-of-voice, rank deltas, content velocity, sentiment trend over the period.

Output standards (apply to every brief):

• Always cite source + timestamp for every claim. "Competitor X raised the Pro tier from $79 → $99 (pricing page, 2026-04-29)" beats "X raised prices."
• Use neutral language. Brief is for the reader to draw their own conclusion.
• Distinguish observation from interpretation. Mark interpretation explicitly: "Observation: rank dropped 12 positions on 'invoice automation' over 30d. Possible interpretation: …".
• Keep one ranked list of "things that matter" at the top of the brief. Bury everything else below it. Most readers stop after the first 3 bullets.

Phase 6 — Monitor on cadence (optional)

If the user wants ongoing monitoring rather than a one-shot:

1. Save the source matrix and competitor set somewhere durable (bigquery_*, Notion, a Sheets tab, or just inline in the user's prompt template). 2. Schedule the agent to re-run the pull on a cadence — weekly is the sweet spot. Daily produces too much noise; monthly misses fast moves. 3. The brief shape becomes the weekly digest by default, with the option to escalate to a deep brief when something significant changes.

Reference workflows

Brief Templates

Four brief shapes, four jobs. Pick the shape that matches the job decided in Phase 1 of `SKILL.md`, then fill it from the data pulled in Phase 3 with diffs from Phase 4. Source playbook for what to pull lives in `source-by-source.md`.

Output rules apply to every brief:

• Every claim has a source URL + scrape timestamp. Unsourced claims get cut.
• Use neutral language. The reader interprets; the brief reports.
• Mark interpretation explicitly when present: prefix with Observation: or Interpretation:.
• Lead with the 3 things that matter most. Bury the rest below.
• Keep aggregates over anecdotes — "averaged 8K likes/post over 30d" beats "one post got 12K."

Template 1 — Battle card (sales)

Goal: Equip a salesperson to handle a competitive deal in real time. 1 page per competitor.

Length: Strict 1-page-per-competitor cap. If it doesn't fit, the salesperson won't read it.

# Battle Card — [Competitor Name]
**As of:** YYYY-MM-DD
**Owner:** [internal owner of this card]

## At a glance
- **Category position:** [1-line summary, e.g. "Mid-market SaaS, US-focused, $15M ARR estimate"]
- **Last meaningful change:** [e.g. "Raised Series B Apr 2026, expanded to EU"]
- **Where they win:** [1-line]
- **Where we win:** [1-line]

## Their pitch (verbatim from their site)
- **Headline:** "[homepage h1]" — [pricing-page or homepage URL]
- **Top 3 value props:** [from features page or homepage subheads]
- **Pricing entry point:** $[X]/mo (Pro: $[Y]/mo) — [pricing URL]
- **Free trial / freemium:** [Yes — N days, with constraints / No]

## What they say about us (if anything)
- [Direct quotes from their comparison pages, blog posts, or sales calls if you have them]
- "" if none found — say so explicitly

## Common objection lines + counter
| They'll say | You say |
| --- | --- |
| "But [competitor] is cheaper" | "[concrete answer with a number/feature]" |
| "[competitor] integrates with X natively" | "[concrete answer]" |
| "[competitor] has been around longer" | "[concrete answer]" |

## Win patterns (if you have win-loss data, otherwise mark TODO)
- We win when: [scenario] — [evidence]
- We lose when: [scenario] — [evidence]

## Recent moves (90 days)
- [date] — [what changed, source URL]
- [date] — [what changed, source URL]
- [date] — [what changed, source URL]

## Don't say
- [Anything that's a lie]
- [Anything that's punching down]
- [Anything that triggers their lawyers]

Worked example (excerpt):

# Battle Card — Convertly
**As of:** 2026-04-30

## At a glance
- **Category position:** Mid-market marketing automation, US/EU, ~$22M ARR (LinkedIn headcount × ARR-per-head benchmark)
- **Last meaningful change:** Pricing raised on Pro tier $79 → $99 (apr 2026 — pricing page diff vs Mar 2026 archive)
- **Where they win:** Native HubSpot sync, mature email editor
- **Where we win:** AI-driven segmentation, $30/mo cheaper at the entry tier

## Common objection lines + counter
| They'll say | You say |
| --- | --- |
| "Convertly is the established player" | "Established with a UI from 2019 — show them our editor side-by-side" |
| "Our HubSpot is already wired up" | "We do HubSpot natively too — here's the 5-min migration video" |

Template 2 — Weekly digest

Goal: One scrollable summary the marketing/exec team reads every Monday. 1 page total, all competitors.

Length: ~1 page (<300 words). Read time under 90 seconds.

# Competitor Digest — Week of [Mon date]

## Top 3 things that matter
1. **[Competitor A] [thing they did]** — [why it matters in <20 words]. [source URL]
2. **[Competitor B] [thing they did]** — [why it matters]. [source URL]
3. **[Competitor C] [thing they did]** — [why it matters]. [source URL]

## Per-competitor changes (this week vs last week)

### Competitor A
- Site: [diff or "no change"]
- Pricing: [diff or "no change"]
- Social: [follower/post deltas, biggest post]
- Search: [rank changes, new content surfacing]
- Mentions: [reddit/twitter activity delta]

### Competitor B
- Site: [...]
- Pricing: [...]
- Social: [...]
- Search: [...]
- Mentions: [...]

### Competitor C
[same shape]

## Watch list (next week)
- [Competitor X is rumored to be launching Y by [date] — verify]
- [Competitor Y's pricing-page A/B test is still running — capture before it ends]

Critical: sections with no change get an explicit "no change this week" line — don't omit them, or readers can't tell whether you forgot or there's nothing.

Template 3 — Comparison-page input

Goal: Feed the marketing team objective, comparable data they can render into a comparison page (yourbrand.com/vs/competitor) without lying.

Length: As long as needed for the matrix to be complete. Optimized for accuracy + comparability, not brevity.

# Comparison Data — [Competitor Name] vs Us
**Pulled:** YYYY-MM-DD
**Sources:** [list of URLs scraped]

## Pricing matrix

| Plan | Price (USD/mo, paid annually) | Price (USD/mo, paid monthly) | Seats | [Key limit 1] | [Key limit 2] | Source |
| --- | --- | --- | --- | --- | --- | --- |
| Their Free | $0 | $0 | 1 | [N] | [N] | [URL] |
| Their Pro | $[X] | $[X+] | [N] | [N] | [N] | [URL] |
| Their Business | $[X] | $[X+] | [N] | [N] | [N] | [URL] |
| Our Free | $0 | $0 | [N] | [N] | [N] | yourbrand.com/pricing |
| Our Pro | $[X] | $[X+] | [N] | [N] | [N] | yourbrand.com/pricing |
| Our Business | $[X] | $[X+] | [N] | [N] | [N] | yourbrand.com/pricing |

## Feature matrix

| Feature | Them | Us | Notes |
| --- | --- | --- | --- |
| [Feature 1] | ✓ | ✓ | Both have it |
| [Feature 2] | ✓ | — | Their differentiation |
| [Feature 3] | — | ✓ | Our differentiation |
| [Feature 4] | Beta | ✓ GA | Note maturity |
| [Feature 5] | "Coming soon" (no date) | ✓ | Note vaporware risk |

## Integration coverage

- **Their integrations:** [list, with link to their integrations page]
- **Our integrations:** [list]
- **Overlap:** [N]
- **Their unique:** [N]
- **Our unique:** [N]

## Trial / freemium policy

| | Them | Us |
| --- | --- | --- |
| Free plan? | [Yes / No] | [Yes / No] |
| Free trial days | [N] | [N] |
| Credit card required? | [Yes / No] | [Yes / No] |
| Auto-converts to paid? | [Yes / No] | [Yes / No] |

## Verbatim positioning (their words, captured from site)

| Page | Quoted copy | URL |
| --- | --- | --- |
| Homepage hero | "[exact h1]" | [URL] |
| Pro tier subhead | "[exact text]" | [URL] |
| About page | "[exact mission text]" | [URL] |

## Disclosure
All data above pulled on [date]. Pricing and features can change — re-pull before publishing the comparison page if more than 7 days have passed.

Why so structured: comparison pages are legal-adjacent. The matrix forces apples-to-apples comparisons that are defensible. The "verbatim positioning" section is what lets you quote them on the comparison page without inventing.

Template 4 — Board-prep update

Goal: Quarterly or board-meeting-prep slide content. Executives, not operators. Aggregates, not anecdotes.

Length: ~1 page across all competitors, with one chart per metric.

# Competitive Position — [QN YYYY]

## Headline
[1 sentence the board chair could repeat from memory.] e.g. "We've closed the rank gap with [Top Competitor] from -47 to -12 over the last 90 days while sustaining 2x organic traffic growth."

## Share of voice (where this matters)
| Channel | Us | [Comp A] | [Comp B] | [Comp C] | Source / metric |
| --- | --- | --- | --- | --- | --- |
| Organic search (est. monthly traffic) | [N] | [N] | [N] | [N] | hyperseo_domain_overview |
| Backlinks (90d delta) | +[N] | +[N] | +[N] | +[N] | hyperseo_backlinks_history |
| AI Overview citations (count over [keywords]) | [N] | [N] | [N] | [N] | hyperseo_ai_overview |
| Instagram followers (current / 90d delta) | [N] / +[N] | [N] / +[N] | [N] / +[N] | [N] / +[N] | scrape_instagram |
| TikTok median views/post (90d) | [N] | [N] | [N] | [N] | scrape_tiktok_videos |

## Rank position on the 10 keywords that matter most
| Keyword | Us | [Comp A] | [Comp B] | [Comp C] | 90d delta (us) |
| --- | --- | --- | --- | --- | --- |
| [keyword 1] | [N] | [N] | [N] | [N] | +/- [N] |
| [keyword 2] | [N] | [N] | [N] | [N] | +/- [N] |
| ... | | | | | |

## What changed this quarter
- **Our wins:** [3 bullets, sourced]
- **Their wins:** [3 bullets, sourced]
- **Misses (ours):** [2 bullets, sourced]

## Strategic read
[1 paragraph max — interpret the data above. What does the trend imply for next quarter's focus.]

## Risks to watch
- [Competitor X is doing [Y]. If it lands, it pressures our [Z].]
- [Category headwind: e.g., "Google AI Overview adoption shifts demand away from organic clicks."]

Critical for board format: every data point has a source attribution column or footnote. Boards ask "where does this number come from?" — pre-empt it.

Worked example — diff section of a weekly digest

The hardest section of any brief is the diff. What does a useful diff look like?

Bad (snapshot, not diff):

Convertly's homepage says "Marketing automation that grows with you."

Good (diff with context):

Convertly homepage diff vs last week:

H1 changed from "The marketing automation platform" → "Marketing automation that grows with you."

CTA button changed from "Start free trial" → "See pricing."

Interpretation: moving from acquisition-led ("trial") to consideration-led ("pricing") messaging. Consistent with their pricing-page raise — likely shifting upmarket.

Source: convertly.com/, scraped 2026-04-29 vs 2026-04-22 archive.

The interpretation is short, marked, and non-essential — a reader can ignore it and still get the observation. That's the right shape.

Source-by-Source Playbook

Per-source operator guide for competitor intel. For each source: what it reveals, when to use it, the right tool call, common pitfalls. Read the section that matches the source you're pulling from. The cross-source workflow lives in `SKILL.md`; brief output structure lives in `brief-templates.md`.

Firecrawl — site, blog, pricing, landing pages

Firecrawl is the backbone for any web-content slice. Use it whenever the question is "what does the competitor's site itself say."

What it reveals

• Pricing tiers, plan limits, add-ons, free-trial terms
• Hero copy, positioning, value props, primary CTAs
• Customer logos, testimonials, social proof
• Feature pages — what they emphasize and what they hide
• Brand assets — logo, color palette, fonts (firecrawl_extract_branding)
• Visual snapshot for evidence (firecrawl_screenshot)
• Full blog archive for content-strategy analysis (firecrawl_crawl_website)

Tool calls

Pitfalls

• JS-heavy SPAs may not render fully. If the page comes back near-empty, fall back to web_scrape_page (stealth-proxy + JS render). Bonus: web_scrape_page accepts an ai_query argument that extracts targeted info from the page in one call (e.g. ai_query="extract the pricing tier names and monthly prices").
• `firecrawl_crawl_website` can be slow + credit-heavy. Cap with max_pages for first run. A 200-post blog archive easily becomes a 10-minute job.
• Pricing pages with toggles (monthly / annual). A single scrape captures the default state. Run twice — once for monthly, once for annual — by including the URL parameter or hash in the URL.
• Geo-fenced pages. Firecrawl scrapes from a default region; pricing in EUR vs USD vs GBP varies. Note the apparent locale of the result in the brief.

HyperSEO — rankings, backlinks, domain overlap, AI search

The credit-metered SEO surface (DataForSEO under the hood). Use intentionally — every call has cost. For full HyperSEO depth on keyword research, see the `seo-research` skill.

What's most useful for competitor intel (vs general SEO)

Pitfalls

• Looping over competitors burns credits fast. Always batch — pull hyperseo_domain_overview for all 5 competitors in succession, then move to next call. Don't interleave.
• `hyperseo_competitors` returns 100s — cap your set early. Take the top 5 organic competitors, then validate with the user before going deeper. A 12-competitor brief is unreadable.
• Backlink and rank data lags by days/weeks depending on the underlying provider. Don't use this for "what happened yesterday" — use it for trends.
• AI Overview citations are volatile. A single AI Overview pull is noise. Track over time; the trend is what matters.
• Country/locale matters. A US-default hyperseo_serp_results is meaningless for a competitor whose primary market is the UK. Always set location_code explicitly (integer — e.g. 2826 for UK, 2840 for US, 2124 for CA, 2036 for AU).

Apify scrapers — organic social

The Apify-backed scrapers each handle one platform. Use the right one for each platform; don't try to make Firecrawl scrape Instagram (it can't handle the auth/render).

Instagram

Useful for: organic engagement trend, content cadence, hashtag patterns, what creative is working for them. Don't mistake total followers for engagement health — a 200K-follower account averaging 800 likes/post is in trouble; a 30K averaging 4K is winning.

Pitfalls: private accounts are unscrapable (don't try). Some posts return without all metadata if Instagram has changed its DOM — re-run later usually fixes it.

TikTok

Useful for: posting cadence, viral moments, content format trends (which videos hit 100K vs 5K). TikTok's algorithm is hit-driven, so look at the distribution of view counts across the last 30 videos, not the average.

Pitfalls: TikTok aggressively rate-limits. Don't run more than ~3 username scrapes per minute — pace.

LinkedIn (conditional — integration must be enabled)

Useful for: company-page positioning shifts, exec hires (a new VP of Marketing usually signals strategy change), thought-leadership content from execs. Pulled per-URL, so for a competitor org grab the company URL plus 2–3 key exec URLs.

Important: this tool is only present in the agent's tool list when the LinkedIn-scraper integration is enabled in the workspace. If you don't see scrape_linkedin_profiles in the tool inventory, skip the LinkedIn slice entirely — don't fail the whole brief.

Pitfalls: LinkedIn aggressively detects + blocks scraping. Even with the integration enabled, expect occasional empty results. Don't pull more than a handful at a time.

Twitter / X

Useful for: real-time signals, launch announcements, exec / founder voice, where in the funnel a customer is when they tweet about the competitor.

Pitfalls: Twitter / X search is increasingly limited; some tweets are missing. Don't expect comprehensive coverage. Use for signal, not census.

Reddit

Useful for: unfiltered customer sentiment, competitive comparisons users do themselves, complaints / praise. The single best source for "what do real people say about this competitor."

Pitfalls: Reddit threads can be old — sort by new and filter by date. A post from 2021 doesn't reflect today's product. Self-promotion is rampant — discount any thread that looks astroturfed.

Google search & trends

Useful for:

• search_google_results(query="<competitor> reviews", num_results=20) — what review sites surface, what's on page 1 (positive / negative).
• search_google_results(query="<competitor> alternative", num_results=20) — who Google considers their competition.
• search_google_results(query="<competitor> vs <us>", num_results=20) — existing comparison content (gold for understanding the conversation already happening).
• scrape_google_trends(search_terms=["<competitor>", "<us>"], time_range="today 3-m", geo="US") — relative interest delta. The single chart that always lands in a board update.

Pitfalls: SERPs are personalized. Results vary by location and history. Always set geo= explicitly when using trends, and assume search results are roughly directional, not exact.

Web scraping fallbacks

For sites Firecrawl can't render (heavy SPA, JS-locked, anti-bot):

web_scrape_page(url=..., use_proxy=true, stealth_proxy=true, ai_query="extract the pricing tier names and monthly prices")

The ai_query argument turns one scrape call into a one-shot extraction — no follow-up parsing needed for well-defined fields. Use it whenever you know in advance what slice of the page matters.

Lighter-weight alternatives:

• web_fetch_page(url=...) — straight HTTP fetch, no JS render
• web_loader(url=...) — quick text extraction, less overhead than web_scrape_page

Order of fallback when Firecrawl is empty: web_scrape_page (with ai_query) → web_fetch_page → web_loader.

Ecommerce competitor specifics

When the competitor is a DTC / ecommerce brand:

These are stronger than firecrawl_scrape_url on a PDP because they normalize the product fields across platforms (Shopify / WooCommerce / etc.) instead of returning raw HTML.

Choosing the right source for the job

Cost & rate-limit discipline

Every source has cost (HyperSEO credits, Apify compute, Firecrawl quota). Operating principles:

1. Cheapest call first. A hyperseo_domain_overview is cheaper than a firecrawl_crawl_website of their whole blog. Pull the cheap stuff to scope the expensive stuff. 2. Per-competitor budget. Decide before pulling: "for each competitor I'll spend ~5 calls." Stick to it. Drift kills credit budgets. 3. Sequence, don't parallelize. 8 parallel scrapes = 8 timeouts. 8 sequential scrapes = 8 results, with the option to bail early if the early ones don't reveal anything. 4. Cache + re-use within the session. A hyperseo_domain_overview result is stable for a day or two — don't re-pull within the same brief. 5. Tell the user the budget upfront. "This will hit ~25 HyperSEO calls and ~12 Firecrawl scrapes for the 4-competitor brief. OK to proceed?" Avoids a $50 surprise on the bill.