Apify Ultimate Scraper

Universal web scraper

AI-driven data extraction from ~100 Actors across 15+ platforms via the Apify CLI.

Rule: Pass `--json` and redirect stderr with `2>/dev/null` on data-returning commands (actors call, actors start, actors info, actors search, datasets get-items, runs info). JSON output is stable across CLI versions. stderr contains progress messages and version warnings that break JSON parsers if not redirected.

This rule does not apply to status/auth commands (apify info, apify --version, apify login). For those, use 2>&1 so authentication and version errors are visible.

Exception: if --input returns no data, re-run with 2>&1 to confirm whether the cause is a missing schema vs. a network/auth error.

Prerequisites

• Apify CLI v1.4.0+ (npm install -g apify-cli)
• Authenticated session (see below)

Authentication

If a CLI command fails with an auth error, authenticate using one of these methods:

1. OAuth (interactive): apify login (opens browser) 2. Environment variable: export APIFY_TOKEN=your_token_here 3. From .env file: source .env (if the file contains APIFY_TOKEN=...)

Generate token: https://console.apify.com/settings/integrations

Workflow

Step 0: Verify CLI readiness before doing anything else

Before using the Apify CLI, always verify the local environment:

1. Check that the CLI is installed:

    apify --help

If this fails, install the CLI first:

       npm install -g apify-cli

2. Check that the CLI is authenticated:

    # Auth check — do NOT pipe to /dev/null, you need to see errors
    apify info 2>&1

If this shows the user is not logged in, instruct them to authenticate with a token:

    apify login --token TOKEN

3. Run Apify CLI commands with all permissions when needed by the agent sandbox.

4. Assume many Apify commands block with zero output until completion. For blocking runs, set block_until_ms to at least 60000.

5. For long or unknown-duration runs, prefer the async pattern:

    apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null

Then poll the run status:

    apify info actor-runs/RUN_ID --json

Check .status for SUCCEEDED or FAILED.

Step 1: Understand goal and select Actor

Identify the target platform and use case. Read references/actor-index.md to find the right Actor. Prefer apify-tier actors; use community-tier only when no apify actor covers the task. For input schemas, fetch dynamically: apify actors info "ACTOR_ID" --input --json 2>/dev/null If the output is empty, re-run without the redirect (2>&1) to surface auth or network errors before proceeding.

If the task involves a multi-step pipeline, also read the matching workflow guide:

If no Actor matches in the index, search dynamically:

apify actors search "KEYWORDS" --json --limit 10 2>/dev/null

From results: items[].username/items[].name (Actor ID), items[].title, items[].stats.totalUsers30Days, items[].currentPricingInfo.pricingModel.

Step 2: Fetch Actor schema and check gotchas

Some Actors don't register an input schema with the platform (their schema lives in code). Try schema sources in this order — fall through on empty/error:

1. Input schema (human-readable):

    apify actors info "ACTOR_ID" --input 2>/dev/null

If output is Error: No input schema found for this Actor, skip to source 2.

2. Input schema (JSON keys only):

    apify actors info "ACTOR_ID" --input --json 2>/dev/null | jq '.input.schema.properties // empty | keys'

Empty result means no registered schema — fall through to source 3. To drill into a specific field:

    apify actors info "ACTOR_ID" --input --json 2>/dev/null | jq '.input.schema.properties.FIELD_NAME'

3. README fallback (always works, contains usage examples):

    apify actors info "ACTOR_ID" --readme 2>/dev/null

Grep the README for an "Input" / "Example input" section to copy the JSON shape.

4. Last resort — call with minimal known input (e.g. {"startUrls":[{"url":"..."}]} for crawlers) and let the Actor surface validation errors that reveal required fields. See references/gotchas.md for known-good minimal inputs for common Actors.

Also read references/gotchas.md to check for common pitfalls and cost guardrails for the selected Actor.

Step 3: Configure and run

Skip user preferences for simple lookups (e.g., "Nike's follower count"). Go straight to running with quick answer mode.

For larger tasks, confirm output format (quick answer / CSV / JSON) and result count.

Before starting the run, double-check whether the task is short enough for a blocking call or should use the async pattern from Step 0.

Standard run (blocking):

    apify actors call "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null

From output: .id (run ID), .status, .defaultDatasetId, .stats.durationMillis

Fetch results:

    apify datasets get-items DATASET_ID --format json

For CSV: apify datasets get-items DATASET_ID --format csv

Quick answer mode: Fetch results as JSON, pick top 5, present formatted in chat.

Save to file: Fetch results, use Write tool to save as YYYY-MM-DD_descriptive-name.csv or .json.

Large/long-running scrapes:

    apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null

Poll: apify info actor-runs/RUN_ID --json (check .status for SUCCEEDED or FAILED).

Step 4: Deliver results

Report: result count, file location (if saved), key data fields, and links:

• Dataset: https://console.apify.com/storage/datasets/DATASET_ID
• Run: https://console.apify.com/actors/runs/RUN_ID

For multi-step workflows: suggest the next pipeline step from the workflow guide.

Troubleshooting

Common errors and pitfalls are documented in references/gotchas.md. Read it before running PPE (pay-per-event) Actors.

Instagram

Facebook

TikTok

YouTube

X/Twitter

LinkedIn

Google Maps

Google Search and Trends

Reviews (cross-platform)

Real estate and hospitality

SEO tools

Content and web crawling

Other platforms

Enrichment and contacts

Gotchas and cost guardrails

Pricing models

To check an Actor's pricing:

apify actors info "ACTOR_ID" --json

Read .currentPricingInfo.pricingModel and .currentPricingInfo.pricePerEvent.

Cost estimation protocol

Before running any PPE Actor:

1. Get the per-event price from Actor info (.currentPricingInfo.pricePerEvent) 2. Multiply by the requested result count 3. Present the estimate to the user with this disclaimer:

Estimated cost: ~$X for Y results. This is a rough estimate only - actual costs can vary significantly depending on the Actor, data complexity, retries, and platform changes. Always check your Apify billing dashboard for actual charges.

4. If estimate > $5: warn explicitly 5. If estimate > $20: require explicit user confirmation before proceeding

Important: Cost estimates in the workflow guides are approximate and may be inaccurate. Always present them as rough guidance with the disclaimer above, never as exact amounts.

Common pitfalls

CLI version warning on stderr Outdated Apify CLI installations print a "You are using an old version of Apify CLI" banner to stderr on every invocation. This breaks naive JSON parsing when stderr is not redirected. Always redirect stderr explicitly on data-returning commands (2>/dev/null). For status/auth commands (apify info, apify login), use 2>&1 so errors are visible rather than swallowed.

jq exit-code 5 on null paths jq exits with code 5 when a filter selects null on a strict path (e.g. .input.schema.properties when .input is missing). This causes the shell to treat the command as failed, masking the real issue (no registered schema). Use the // empty alternative operator in jq filters — e.g. jq '.input.schema.properties // empty | keys' — so the output is simply empty on a missing path and the agent can branch on empty output instead of a non-zero exit code.

Cookie-dependent Actors Some social media scrapers require cookies or login sessions. If an Actor returns auth errors or empty results unexpectedly, check its README:

apify actors info "ACTOR_ID" --readme

Look for mentions of "cookies", "login", "session", or "proxy".

Rate limiting on large scrapes Platforms throttle or block large-volume scraping. Mitigations:

• Use proxy configuration when available: "proxyConfiguration": {"useApifyProxy": true}
• Set reasonable concurrency limits (check the Actor's maxConcurrency input)
• For 1,000+ results, suggest splitting into smaller batches

Empty results Common causes:

• Too-narrow search query or geo-restriction (try broader terms)
• Platform blocking without proxy (enable Apify Proxy)
• Actor requires cookies/login but none provided
• Wrong input field name (always verify with --input --json)

maxResults vs maxCrawledPages Different Actors use different limit field names. Common variants:

• maxResults, resultsLimit, maxItems - limit output items
• maxCrawledPages, maxRequestsPerCrawl - limit pages visited

Always fetch the input schema to find the correct field for the specific Actor.

Deprecated Actors Check .isDeprecated in apify actors info --json. If true: 1. Search for alternatives: apify actors search "SIMILAR_KEYWORDS" --json 2. Prefer apify tier replacements over community alternatives

LinkedIn pricing LinkedIn Actors are all PPE and vary significantly:

• harvestapi/ Actors: generally cheaper ($0.001-0.01/result)
• apimaestro/ Actors: generally more expensive ($0.005-0.02/result)
• dev_fusion/ Actors: mid-range, useful for mass scraping with email enrichment

Always compare pricing before selecting a LinkedIn Actor.

SEO tool pricing radeance/ SEO scrapers (SimilarWeb, Ahrefs, SEMrush, Moz) have the highest per-result costs ($0.005-0.0275/result). For large-scale SEO analysis, estimate costs carefully and suggest batching.

Error recovery

Why Apify Actors vs raw HTTP scraping

Many n8n and automation workflows use raw HTTP Request nodes or self-hosted Puppeteer for web scraping. These hit common walls that Apify Actors handle transparently:

Cloudflare and WAF bypass Raw HTTP requests fail on sites with Cloudflare Turnstile, DataDome, or other WAFs. Apify Actors use residential proxies and browser fingerprint rotation automatically. For the toughest sites, use apify/camoufox-scraper.

JavaScript-rendered pages (SPAs) React, Vue, and Angular sites return empty HTML to plain HTTP requests. Apify's apify/playwright-scraper and apify/camoufox-scraper fully render JavaScript before extracting data.

Anti-bot fingerprinting Even headless browsers get detected via TLS fingerprints (JA3 hashes). Apify's browser pool rotates fingerprints across requests automatically.

Session and cookie management Social media platforms (LinkedIn, Instagram) require persistent sessions. Social media Actors handle cookie management and session rotation internally.

Scaling without infrastructure Self-hosted Puppeteer at scale requires 4-8 GB RAM per browser instance. Apify Actors run on serverless infrastructure - no browser pool management, no RAM provisioning, no Docker orchestration.

Platform-specific rate limits

Instagram: Aggressive rate limiting. Keep maxResults under 200 per run for profile/post scrapers. Use delays between runs. Instagram API scrapers (apify/instagram-api-scraper) have higher limits than browser-based ones.

LinkedIn: All LinkedIn Actors are community-maintained and PPE. LinkedIn actively blocks scraping at scale. Keep batch sizes under 100 profiles. Space runs at least 5 minutes apart. Expect occasional empty results.

TikTok: Anti-bot measures increasing. clockworks/tiktok-scraper handles most cases. For blocked regions, enable Apify Proxy with residential IPs.

Google Maps: Generally stable. Set language: "en" explicitly for consistent results. Large-area searches may return different results depending on zoom level - use specific location queries over broad city names.

Amazon/E-commerce: Heavy anti-bot. The apify/e-commerce-scraping-tool handles this via built-in proxy rotation. Raw HTTP requests will fail.

Brand monitoring workflows

Cross-platform brand mention tracking

When: User wants to monitor brand mentions, hashtags, or sentiment across social platforms.

Pipeline (run each independently, combine results)

1. Instagram mentions -> apify/instagram-tagged-scraper

• Key input: username (brand handle)

2. Instagram hashtags -> apify/instagram-hashtag-scraper

• Key input: hashtags (branded hashtags)

3. X/Twitter mentions -> apidojo/tweet-scraper

• Key input: searchTerms (brand name, handle, hashtags)

4. Reddit mentions -> trudax/reddit-scraper-lite

• Key input: searchQuery (brand name)

Output fields

Instagram: caption, likesCount, commentsCount, timestamp, ownerUsername X/Twitter: text, retweetCount, likeCount, replyCount, createdAt, author Reddit: title, body, score, numComments, subreddit, createdAt

Gotcha

This is a parallel workflow, not sequential. Run each Actor independently. Combine results by date for a timeline view.

Twitter/X real-time mention routing

When: User wants to route brand mentions on X to the right team channel - negative to support, positive to wins - with sentiment scoring.

Pipeline

1. Collect tweets -> apidojo/tweet-scraper

• Key input: searchTerms (brand name + variants), maxItems, since (ISO date for incremental runs)

2. Score sentiment -> tri_angle/social-media-sentiment-analysis-tool

• Pipe: results[].url -> urls
• Key input: urls, platforms

Output fields

Step 1: text, author.userName, createdAt, likeCount, retweetCount, url Step 2: sentiment (positive/negative/neutral), score, text, platform

Gotcha

Use since on each run (store last tweet createdAt in Sheets) to avoid reprocessing the same mentions. Without dedup, alerts fire on the same tweet repeatedly.

Reddit brand and topic monitoring

When: User wants weekly surfacing of brand mentions, product feedback, and competitor comparisons from Reddit.

Pipeline

1. Scrape Reddit -> trudax/reddit-scraper-lite

• Key input: subreddits (target subreddit array), searchTerms (brand + competitor names), maxItems, sort (hot/new/top)

Output fields

Step 1: title, body, subreddit, url, score, numberOfComments, createdAt

Gotcha

Set sort: "new" for monitoring runs; use sort: "top" for periodic digest reports. Mixing both in one run returns inconsistent result sets.

Multi-platform social listening with sentiment

When: User wants a unified brand health view across Instagram, Facebook, TikTok, and Twitter simultaneously.

Pipeline (run in parallel)

1. Instagram -> apify/instagram-search-scraper

• Key input: searchTerms (brand variants), maxItems

2. Facebook -> apify/facebook-search-scraper

• Key input: searchTerms, maxItems

3. TikTok -> clockworks/tiktok-user-search-scraper

• Key input: searchTerms, maxItems

4. Twitter -> apidojo/tweet-scraper

• Key input: searchTerms, maxItems

5. Sentiment scoring -> tri_angle/social-media-sentiment-analysis-tool

• Pipe: merged post URLs from steps 1-4 -> urls
• Key input: urls, platforms

Output fields

Steps 1-4 (normalized): text, platform, author, timestamp, engagementCount Step 5: sentiment, score, text, platform

Sentiment analysis

When: User wants sentiment scoring on collected mentions.

Pipeline

1. Collect mentions (use any step from above) 2. Analyze sentiment -> tri_angle/social-media-sentiment-analysis-tool

• Pipe: collected post URLs -> urls
• Key input: urls, platforms

Output fields

Step 2: sentiment (positive/negative/neutral), score, text, platform

Company research workflows

Company intelligence profiling for sales or ABM

When: User has a list of target accounts and wants structured firmographic data, ICP signals, and key personnel for outreach or account-based marketing.

Pipeline

1. Crawl company website -> apify/website-content-crawler

• Key input: startUrls (company domains), maxCrawlDepth (2), includeUrlGlobs (about, pricing, team, careers, blog)

2. Enrich with LinkedIn company data -> harvestapi/linkedin-company

• Pipe: company name or LinkedIn URL extracted from WCC text -> Actor input
• Key input: company identifier, includeEmployees: false

3. AI extract structured signals (n8n: OpenAI node outputs JSON schema with companySize, industry, techStack, keyPersonnel, painSignals) 4. Store in Supabase, Airtable, or HubSpot with AI-extracted fields as custom properties

Output fields

WCC: text (per page), url LinkedIn: employeeCount, industry, headquarters, description, specialties AI-extracted: companySize, industry, techStack, keyPersonnel, painSignals

Gotcha

WCC crawl at depth 2 can return 20-50 pages per company. For large batches, set maxCrawlPages: 5 focused on the About and Pricing pages via includeUrlGlobs. This keeps cost and latency manageable without sacrificing signal quality.

---

Startup scouting from Product Hunt

When: User wants weekly discovery of recently launched or funded startups in a target category for investor outreach, partnership, or competitive tracking.

Pipeline

1. Scrape Product Hunt launches -> apify/web-scraper

• Key input: startUrls (Product Hunt today/weekly/topic pages), maxCrawlPages (3-5)
• Note: no dedicated Actor exists - search apify actors search "product hunt" for community options

2. Filter by category + upvote threshold (n8n: Filter node on extracted upvotes, category) 3. Crawl company sites -> apify/website-content-crawler

• Pipe: results[].website -> startUrls
• Key input: maxCrawlPages (3), includeUrlGlobs (about, team)

4. LinkedIn founder lookup -> harvestapi/linkedin-profile-search (by name + company)

• Pipe: extracted founder names from step 3 -> search input

5. AI ICP scoring (n8n: OpenAI node scores each startup against defined criteria) 6. Output to Airtable pipeline + Slack alert for top matches

Output fields

Step 1: title, tagline, upvotes, website, makers[].name, makers[].profileUrl WCC: text, url LinkedIn: fullName, headline, profileUrl, currentCompany

Gotcha

Product Hunt ranking changes throughout the day. Schedule the scrape for end-of-day (11 pm UTC) to capture final vote counts. For AngelList/Wellfound, no maintained public Actor exists - use apify/website-content-crawler on search result pages as a fallback.

---

Sales meeting prep from LinkedIn and news

When: A calendar event is detected and the user needs a briefing on meeting attendees - their recent activity, company context, and conversation talking points - delivered before the meeting.

Pipeline

1. Trigger on calendar event (n8n: Google Calendar Trigger, filter for events starting in 30 min) 2. Extract attendee LinkedIn activity -> harvestapi/linkedin-profile-scraper + harvestapi/linkedin-profile-posts

• Key input: profiles (attendee LinkedIn URLs), maxPosts (5 recent posts)

3. Extract company context -> harvestapi/linkedin-company

• Pipe: results[].currentCompany -> company name

4. Search recent news -> apify/google-search-scraper

• Pipe: company name + "news" -> queries
• Key input: queries, maxResultsPerPage (5)

5. AI synthesize brief (n8n: OpenAI node produces: key topics, recent signals, suggested talking points) 6. Deliver via Gmail or WhatsApp node 30 minutes before meeting start

Output fields

Profiles: fullName, headline, recentPosts[], experience[0] Posts: postText, publishedAt, likes, comments Company: description, employeeCount, recentNews Search: organicResults[].title, organicResults[].snippet, organicResults[].url

Cost estimate

All HarvestAPI Actors are PPE. Per meeting with 2 attendees: profile scraper ~$0.02, posts ~$0.01, company ~$0.005, Google search ~$0.01. Total: ~$0.05 per meeting prep.

Gotcha

LinkedIn profile URLs must be in the calendar event description or a linked CRM record - they won't auto-resolve from email addresses. Set up a step in your CRM or calendar template to include LinkedIn URLs for attendees. Without a valid profileUrl, the HarvestAPI Actors return empty results.

Competitive intelligence workflows

Competitor ad monitoring

When: User wants to see competitor advertising creatives, targeting, or ad spend signals.

Pipeline

1. Scrape ad library -> apify/facebook-ads-scraper

• Key input: searchQuery (competitor name), country, adType, maxItems

Output fields

Step 1: adTitle, adBody, adCreativeUrl, startDate, pageInfo.name, platform

Gotcha

Facebook Ad Library is public data, no auth needed. But results are limited to currently active or recently inactive ads.

---

Competitor web presence analysis

When: User wants traffic, rankings, and SEO data for competitor domains.

Pipeline

1. Get traffic data -> radeance/similarweb-scraper

• Key input: urls (competitor domains)

2. Get backlink profile -> radeance/ahrefs-scraper

• Key input: urls (same domains)

Output fields

Step 1: globalRank, monthlyVisits, bounceRate, avgVisitDuration, trafficSources Step 2: domainRating, backlinks, referringDomains, organicKeywords

Cost estimate

radeance/ Actors cost $0.005-0.0275/result. A single domain audit across both steps costs ~$0.04-0.06.

---

Competitor website change detection

When: User wants to monitor competitor pricing pages, feature announcements, or product pages and get alerted when meaningful changes occur.

Pipeline

1. Detect changes -> tri_angle/website-changes-detector

• Key input: startUrls (competitor page URLs), notificationEmail, checkIntervalHours

Output fields

Step 1: url, changedAt, diff (text diff), screenshotUrl

Gotcha

tri_angle/website-changes-detector handles baseline storage internally - do not attempt to manage baselines externally or you will lose the diff history between runs.

---

Competitor SERP position monitoring

When: User wants to track where competitor domains rank for target keywords and get alerted on significant position shifts.

Pipeline

1. Scrape SERP rankings -> apify/google-search-scraper

• Key input: queries (target keywords array), countryCode, maxResultsPerPage

2. Track traffic estimates -> radeance/similarweb-scraper

• Key input: urls (competitor domains)
• Pipe: run separately per competitor domain after extracting domains from step 1 results

Output fields

Step 1: organicResults[].url, organicResults[].position, organicResults[].title Step 2: globalRank, monthlyVisits, trafficSources

Cost estimate

radeance/similarweb-scraper costs ~$0.02-0.03 per domain. For 5 competitors, budget ~$0.10-0.15 per weekly run.

---

Competitor feature and pricing benchmarking

When: User wants a structured comparison of competitor pricing tiers, feature lists, and positioning across 5-10 competitor sites.

Pipeline

1. Crawl pricing and feature pages -> apify/website-content-crawler

• Key input: startUrls (competitor pricing page URLs), maxCrawlDepth (set to 1), includeUrlGlobs

2. Extract structured data -> AI node (GPT-4o or Claude)

• Pipe: results[].text -> extraction prompt per competitor
• Key input: extraction schema (tiers, prices, key features, positioning statement)

Output fields

Step 1: text (clean markdown with pricing tables), url, metadata.title Step 2: AI-extracted structured JSON with tiers, prices, feature flags per competitor

Gotcha

Set maxCrawlDepth: 1 and use includeUrlGlobs to restrict crawl to pricing and features paths only. Without this, WCC will crawl the full site and inflate cost significantly.

Contact enrichment workflows

Website contact extraction from a URL list

When: User has a list of company websites and wants emails, phone numbers, and social links for outreach.

Pipeline

1. Extract contact info -> vdrmota/contact-info-scraper or compass/contact-details-scraper-standby

• Key input: startUrls (company website URLs), maxDepth (crawl depth, 1-2 is usually enough)

2. Supplement with social signals -> apify/social-media-leads-analyzer

• Pipe: results[].domain -> input domain list
• Key input: domains (company domains), extractLinkedIn, extractTwitter

3. Dedup and verify (n8n: dedup by email domain, optional ZeroBounce/Hunter node for verification)

Output fields

Step 1: emails[], phones[], linkedinUrl, twitterUrl, domain Step 2: socialProfiles, linkedinCompanyUrl, facebookUrl

Gotcha

contact-details-scraper-standby is a Standby Actor - it stays warm and responds in < 1s, making it ideal for real-time enrichment in webhook flows. Use it when latency matters. For batch jobs, vdrmota/contact-info-scraper is more cost-effective.

---

LinkedIn warm lead identification from post comments

When: User wants to find engaged prospects who commented on relevant LinkedIn posts (competitor content, thought leader posts, industry discussions).

Pipeline

1. Scrape post comments -> harvestapi/linkedin-post-comments

• Key input: postUrl (LinkedIn post URL), maxComments

2. Enrich commenter profiles -> harvestapi/linkedin-profile-scraper

• Pipe: results[].commenter.profileUrl -> urls
• Key input: urls, includeEmail: true

3. Filter by ICP criteria (n8n: Filter node on headline or companyName) 4. AI draft outreach (n8n: OpenAI node generates personalized message using commentText + headline)

Output fields

Step 1: commenter.name, commenter.headline, commenter.profileUrl, commentText, timestamp Step 2: experience[], education[], email, phone, skills[]

Cost estimate

Both Actors are PPE. Step 1 ~ $0.005/comment. Step 2 with includeEmail: true ~ $0.01/profile. For 100 commenters enriched: ~$1.50 total.

Gotcha

Not all LinkedIn posts are publicly accessible. Test postUrl manually before building a workflow around it. Private or restricted posts return empty results - no error, just zero items.

---

Real-time lead enrichment on form submission

When: A prospect fills a form and their company needs to be enriched automatically for ICP scoring and sales routing.

Pipeline

1. Receive form webhook (n8n: Webhook trigger from HubSpot/Typeform/custom form) 2. Extract company domain (n8n: Function node parses email domain) 3. Crawl company site -> apify/website-content-crawler

• Key input: startUrls (company domain), maxCrawlPages (3-5), includeUrlGlobs (about, pricing, careers, team)

4. Enrich with LinkedIn firmographics -> harvestapi/linkedin-company (optional, for headcount + industry)

• Pipe: company name or LinkedIn URL derived from WCC output

5. AI extract signals (n8n: OpenAI node extracts companySize, industry, techStack, ICPFit score from crawl text) 6. Route (n8n: Switch node sends high-ICP leads to Slack sales channel, others to nurture sequence)

Output fields

WCC: text, title, url, metadata.description LinkedIn: employeeCount, industry, headquarters, description, website

Gotcha

WCC crawl on a small startup site can take 30-60 seconds. For synchronous form flows, set maxCrawlPages: 3 and use a timeout. If latency is critical, use apify/cheerio-scraper for the About page only and skip LinkedIn enrichment for first response.

Content and SEO workflows

Website content extraction for RAG

When: User wants to crawl a website and extract clean text for AI/LLM pipelines or knowledge bases.

Pipeline

1. Crawl website -> apify/website-content-crawler

• Key input: startUrls, maxCrawlPages, crawlerType ("cheerio" for speed, "playwright" for JS sites)

Output fields

Step 1: url, title, text, markdown, metadata, links[]

Gotcha

For JS-heavy sites (SPAs), set crawlerType: "playwright". For static sites, use "cheerio" (10x faster). For anti-bot sites, use apify/camoufox-scraper instead.

SERP-based SEO content brief generation

When: User wants to generate a content brief for a target keyword by analyzing competitor SERP results and heading structures.

Pipeline

1. SERP scrape -> apify/google-search-scraper

• Key input: queries, countryCode, maxResultsPerPage

2. Extract heading structure -> apify/cheerio-scraper

• Pipe: results[].organicResults[].url -> startUrls (filter non-article URLs first)
• Key input: startUrls, pageFunction (extract h1/h2/h3 nodes)

Output fields

Step 1: organicResults[].url, organicResults[].title, organicResults[].snippet Step 2: heading structure (h1/h2/h3 text), word count per page

Gotcha

Use apify/cheerio-scraper for heading extraction - it's 10x faster than website-content-crawler for HTML-only pages. Only escalate to website-content-crawler when you need full body text for AI synthesis.

Sitemap content audit

When: User wants to crawl all URLs on a competitor's site from the sitemap and build a keyword and topic inventory.

Pipeline

1. Extract sitemap URLs -> apify/sitemap-extractor

• Key input: startUrls (sitemap.xml URL)

2. Crawl each URL -> apify/website-content-crawler

• Pipe: results[].urls[] -> startUrls
• Key input: startUrls, maxCrawlPages, htmlTransformer (readableText)

Output fields

Step 1: urls[] (all discovered page URLs) Step 2: text, metadata, url

Gotcha

Large sitemaps (1,000+ URLs) can be expensive. Filter step 1 results to a specific path prefix (e.g., /blog/) before passing to step 2 to avoid crawling low-value pages like tag archives and pagination.

Keyword rank tracking with alerts

When: User wants weekly tracking of keyword positions for own domain and competitors, with Slack alerts on drops over 5 positions.

Pipeline

1. SERP scrape -> apify/google-search-scraper

• Key input: queries (tracked keywords array), countryCode, device (desktop/mobile), maxResultsPerPage (set to 100)

Output fields

Step 1: organicResults[].position, organicResults[].url, organicResults[].domain

Cost estimate

apify/google-search-scraper is a fixed-cost Actor. 100 keywords weekly ≈ $1-3/month depending on query volume.

Gotcha

Set maxResultsPerPage: 100 to capture positions 11-100. Default is 10 results, which misses any keyword ranking outside page 1 - making it impossible to detect rank drops from position 12 to 18.

SERP analysis

When: User wants to analyze search engine results for specific keywords.

Pipeline

1. Google SERP -> apify/google-search-scraper

• Key input: queries, maxPagesPerQuery, countryCode, languageCode

Output fields

Step 1: organicResults[] (title, url, description, position), paidResults[], peopleAlsoAsk[], relatedSearches[]

Deep research agent

When: User wants an AI agent that takes a research question, generates search queries, crawls top results, and synthesizes findings into a structured report.

Pipeline

1. Generate search queries (LLM node - 3 queries per research question) 2. SERP scrape -> apify/google-search-scraper

• Pipe: generated queries -> queries
• Key input: queries, maxResultsPerPage

3. Extract content -> apify/rag-web-browser

• Pipe: results[].organicResults[].url (AI-selected relevant URLs) -> query
• Key input: query, maxResults, requestTimeoutSecs

4. Synthesize (LLM node - final report generation)

Output fields

Step 2: organicResults[].url, organicResults[].snippet Step 3: text, url, metadata.title

Gotcha

Use apify/rag-web-browser at step 3 rather than website-content-crawler - it's optimized for agent-based single-URL retrieval and returns clean markdown with lower latency. Reserve website-content-crawler for bulk batch crawls.

Domain authority and backlink analysis

When: User wants SEO metrics for specific domains.

Pipeline

1. Traffic overview -> radeance/similarweb-scraper

• Key input: urls

2. Backlink profile -> radeance/ahrefs-scraper

• Key input: urls

3. Domain authority -> radeance/semrush-scraper

• Key input: urls

Output fields

Step 1: globalRank, monthlyVisits, bounceRate, trafficSources Step 2: domainRating, backlinks, referringDomains, organicKeywords Step 3: authorityScore, organicSearchTraffic, paidSearchTraffic

Cost estimate

All radeance/ SEO Actors are PPE at $0.005-0.0275/result. Running all 3 for one domain costs ~$0.05-0.08. For 50 domains, estimate $2.50-$4.00.

E-commerce price monitoring workflows

Competitor product price monitoring with alerts

When: User wants to track competitor prices across product pages and get notified when prices change.

Pipeline

1. Scrape product pages -> apify/e-commerce-scraping-tool

• Key input: startUrls (competitor product page URLs), proxyConfiguration

2. Match products across sites -> tri_angle/e-commerce-product-matching-tool

• Pipe: results[].url + results[].name -> matching input
• Key input: source dataset ID from step 1, target product list

3. Compare vs. baseline (n8n logic: read Google Sheets last_price, compute % change, filter if changed) 4. Alert via Telegram/Slack node with price delta

Output fields

Step 1: price, currency, name, sku, availability, url Step 2: matched pairs with similarityScore, sourceProduct, targetProduct

Cost estimate

apify/e-commerce-scraping-tool is pay-per-result. For 200 product URLs daily, expect ~$0.50-$1/run depending on site complexity.

Gotcha

Many e-commerce sites use bot protection. If e-commerce-scraping-tool returns empty prices, fall back to apify/camoufox-scraper with residential proxy. Set sessionPoolName to reuse sessions and reduce blocks.

---

Amazon product and review tracking

When: User wants to monitor own or competitor Amazon listings for price drops or review score changes.

Pipeline

1. Extract Amazon data -> apify/e-commerce-scraping-tool

• Key input: startUrls (Amazon product URLs), extractReviews (bool)

2. Compare vs. stored baseline (n8n: read last values from Sheets or DB) 3. Alert on new low price or rating drop (n8n: If node + Telegram/Slack send)

Output fields

price, currency, rating, reviewsCount, title, asin, availability

Cost estimate

Flat per-result pricing. 50 ASINs daily ~ $0.10-$0.25/run.

Gotcha

Amazon aggressively rotates prices and sometimes shows regional prices. Always store currency alongside price. For review text (not just counts), search apify actors search "amazon reviews" for a dedicated Actor.

---

Supplier catalog extraction to draft products

When: User wants to pull new products from a supplier portal and create draft listings with AI-enriched descriptions.

Pipeline

1. Crawl supplier catalog -> apify/playwright-scraper (JS-heavy portals) or apify/cheerio-scraper (static HTML)

• Key input: startUrls (supplier category pages), pseudoUrls (product URL patterns), maxCrawlPages

2. Extract product content -> apify/website-content-crawler (optional second pass for detail pages)

• Pipe: results[].url -> startUrls
• Key input: maxCrawlPages (1 per product), htmlTransformer: "readableText"

3. AI rewrite (n8n: OpenAI node generates SEO title + bullets from raw specs) 4. Create draft product (n8n: Shopify node POST /products.json with status: "draft")

Output fields

Step 1/2: text, url, metadata.title, inline image URLs

Cost estimate

Depends on catalog size. playwright-scraper is PPE; 500 product pages ~ $1-3.

Gotcha

Supplier portals often require login. Use apify/playwright-scraper with initialCookies or a pre-login script in preNavigationHooks. Never hardcode credentials - pass via Actor input from n8n credentials store.

---

Multi-site deal and coupon monitoring

When: User wants to detect when competitors run promotions or publish coupon codes so marketing can respond.

Pipeline

1. Scrape deals pages -> apify/e-commerce-scraping-tool

• Key input: startUrls (competitor deal/sale page URLs), proxyConfiguration

2. Dynamic JS deal pages (fallback) -> apify/camoufox-scraper

• Pipe: failed URLs from step 1 -> startUrls

3. AI extract promotion details (n8n: OpenAI node extracts discount %, promo code, expiry from raw text) 4. Dedup and alert (n8n: compare against stored deals DB, Slack notify on new deals)

Output fields

Raw: price, discountText, url; AI-extracted: promoCode, validUntil, discountPercent, category

Cost estimate

Light scraping - deals pages are few. Expect < $0.20/run for 20 competitor pages.

Gotcha

Promo codes and flash deals may only be visible after login or in geofenced regions. Test each target URL manually first. AI extraction of expiry dates is unreliable - treat as best-effort signal, not exact data.

Influencer vetting workflows

Instagram creator vetting

When: User wants to evaluate an influencer's profile, audience, and engagement quality.

Pipeline

1. Get profile data -> apify/instagram-profile-scraper

• Key input: usernames (list of handles)

2. Analyze engagement -> apify/instagram-comment-scraper

• Pipe: results[].latestPosts[].url -> directUrls (pick 3-5 recent posts)
• Key input: directUrls, resultsLimit

Output fields

Step 1: username, fullName, followersCount, followsCount, postsCount, biography, isVerified, latestPosts[] Step 2: text, ownerUsername, timestamp (scan for bot patterns: generic praise, emoji-only, irrelevant content)

Gotcha

High follower count with low comment quality suggests fake followers. Compare comment sentiment to post content.

---

Cross-platform influencer discovery

When: User wants to find an influencer's presence across multiple platforms.

Pipeline

1. Search across platforms -> tri_angle/social-media-finder

• Key input: query (influencer name or handle), platforms

Output fields

Step 1: platform, profileUrl, username, followers, isVerified

---

TikTok creator vetting

When: User wants to vet TikTok creators by niche or handle for partnership fit based on engagement rate and content quality.

Pipeline

1. Get profile metrics -> clockworks/tiktok-profile-scraper

• Key input: profiles (username array), resultsPerPage

2. Pull recent videos -> clockworks/tiktok-video-scraper

• Pipe: results[].authorMeta.name -> profiles
• Key input: profiles, maxItems

Output fields

Step 1: authorMeta.name, authorMeta.fans, authorMeta.heart, authorMeta.video Step 2: diggCount, playCount, commentCount, shareCount, createTimeISO, hashtags, text

Gotcha

TikTok engagement rate must be calculated manually: (diggCount + commentCount + shareCount) / playCount. The Actor does not return a pre-calculated ER field.

---

YouTube channel audit

When: User wants to audit YouTube channels for subscriber growth, average views, and topic consistency before sponsorship.

Pipeline

1. Get channel overview -> streamers/youtube-channel-scraper

• Key input: startUrls (channel URLs), maxResults

2. Pull video metrics -> streamers/youtube-scraper

• Pipe: results[].channelUrl -> startUrls
• Key input: startUrls, maxResults

3. Analyze content themes -> curious_coder/youtube-transcript-scraper

• Pipe: results[].url -> video URLs (pick 5-10 recent videos)
• Key input: video URLs

Output fields

Step 1: channelName, numberOfSubscribers, channelTotalViews, channelUrl Step 2: videos[].viewCount, videos[].likeCount, videos[].title, videos[].publishedAt Step 3: transcript (raw text for AI topic classification)

---

Cross-platform hashtag discovery

When: User wants to discover new influencer candidates across Instagram, TikTok, and YouTube using niche hashtags for a unified shortlist.

Pipeline

1. Instagram hashtag scrape -> apify/instagram-hashtag-scraper

• Key input: hashtags (array), resultsLimit

2. TikTok hashtag scrape -> clockworks/tiktok-hashtag-scraper

• Key input: hashtags (same array), maxItems

3. YouTube hashtag scrape -> streamers/youtube-video-scraper-by-hashtag

• Key input: hashtags (same array), resultsLimit

Output fields

Step 1: ownerUsername, followersCount, profileUrl, likesCount, commentsCount Step 2: authorMeta.name, authorMeta.fans, playCount, diggCount, shareCount Step 3: channelName, numberOfSubscribers, viewCount, channelUrl

Gotcha

Each platform returns platform-specific field names. Normalize to a common schema (username, platform, followersCount, avgEngagement, profileUrl) in a downstream merge step before scoring.

Job market and recruitment workflows

Job listing research

When: User wants to find and analyze job postings by role, company, or location.

Pipeline

1. Search jobs -> harvestapi/linkedin-job-search

• Key input: keyword, location, datePosted, limit

2. Get job details -> apimaestro/linkedin-job-detail

• Pipe: results[].jobUrl -> urls
• Key input: urls

Output fields

Step 1: title, company, location, jobUrl, postedDate, applicantsCount Step 2: description, requirements, seniority, employmentType, salary

Gotcha

Both Actors are PPE. Step 1: ~$0.001/job. Step 2: ~$0.005/job. For 200 jobs, total ~$1.20. Estimate and confirm with user.

Candidate sourcing

When: User wants to find potential candidates matching specific criteria.

Pipeline

1. Search profiles -> harvestapi/linkedin-profile-search

• Key input: keyword, title, location, industry, limit

2. Enrich with details -> apimaestro/linkedin-profile-full-sections-scraper

• Pipe: results[].profileUrl -> urls
• Key input: urls

Output fields

Step 1: fullName, headline, location, profileUrl, currentCompany Step 2: experience[], education[], skills[], certifications[], languages[]

Gotcha

Step 2 (apimaestro/linkedin-profile-full-sections-scraper) costs ~$0.01/profile - the most expensive LinkedIn scraper. Use sparingly for shortlisted candidates only.

Sales signal outreach - job posting as buying signal

When: User wants to monitor company job postings as a signal to identify sales opportunities - e.g., a "Head of Data Engineering" hire suggests budget for data tooling.

Pipeline

1. Monitor target postings -> harvestapi/linkedin-job-search

• Key input: searchUrl (LinkedIn Jobs URL with company or role filters), keywords

2. Get company context -> harvestapi/linkedin-company

• Pipe: results[].companyUrl -> companyUrls

Output fields

Step 1: title, companyName, description, employmentType, seniorityLevel, jobUrl Step 2: name, industry, employeeCount, description, specialties[]

Gotcha

Job descriptions contain implicit buying signals - tech stack mentions, pain points, and headcount growth. Pass description to an LLM to extract inferred tech stack and budget tier before prioritizing outreach. Contact finding (Hunter.io) uses the native n8n node, not an Apify Actor.

Upwork job monitoring for freelancers

When: User wants to continuously monitor Upwork for new jobs matching their skills.

Pipeline

1. Scrape Upwork search -> apify/playwright-scraper

• Key input: startUrls (Upwork search URL with skill filters), pseudoUrls, maxCrawledPages

Output fields

Step 1: title, description, budget, clientJobsPosted, clientHireRate, postedAt, url

Gotcha

No dedicated Upwork Actor exists in Apify Store - verify with apify actors search "upwork" for community options before defaulting to apify/playwright-scraper. Upwork pages are JS-heavy so Playwright is required over basic HTTP scraping. For high-frequency monitoring (every 15 min), store seen job URLs to avoid re-processing duplicates.

GitHub contributor discovery

When: User wants to find developers who contribute to specific open-source projects.

Pipeline

1. Get contributors -> janbuchar/github-contributors-scraper

• Key input: repoUrls

Output fields

Step 1: username, contributions, profileUrl, avatarUrl

Knowledge base and RAG pipeline workflows

Website to RAG knowledge base via sitemap crawl

When: User wants to ingest an entire website or documentation site into a vector database for AI retrieval (chatbots, search, AI agents).

Pipeline

1. Extract sitemap -> apify/sitemap-extractor

• Key input: sitemapUrl or domain

2. Crawl and convert to markdown -> apify/website-content-crawler

• Pipe: results[].url -> startUrls (or pass dataset ID)
• Key input: startUrls, maxCrawlPages, htmlTransformer: "readableText", outputFormats: ["markdown"]

3. Chunk and embed (n8n: Recursive Character Text Splitter -> OpenAI Embeddings node) 4. Upsert to vector DB (n8n: Supabase / Qdrant node with document + metadata)

Output fields

text (clean markdown), url, metadata.title, metadata.description, crawledAt

Gotcha

apify/rag-web-browser is purpose-built for RAG use cases and returns pre-chunked, clean text without boilerplate - use it when you want simpler output and don't need full site coverage. For comprehensive crawls (full docs sites, 100+ pages), website-content-crawler gives more control over depth and URL filtering.

---

Deep research agent with web crawling

When: User or an AI agent submits a research question and wants a synthesized report drawn from live web sources.

Pipeline

1. Generate search queries (n8n: AI node expands research question into 3-5 distinct queries) 2. Search -> apify/google-search-scraper

• Pipe: generated queries -> queries (array)
• Key input: queries, maxResultsPerPage (5-10)

3. Retrieve content -> apify/rag-web-browser

• Pipe: results[].organicResults[].url -> query (RAG browser takes query + crawls most relevant result)
• Key input: query, maxResults, requestTimeoutSecs

4. Synthesize (n8n: OpenAI node assembles final report from per-source summaries) 5. Output to n8n Data Table, Notion, or Google Docs

Output fields

Search: organicResults[].url, organicResults[].title, organicResults[].snippet RAG browser: text, url, metadata.title

Gotcha

apify/rag-web-browser fetches and summarizes a single URL per call. To process multiple search results in parallel, use n8n's Split In Batches node with a concurrency of 3-5 rather than running them sequentially. This cuts total runtime significantly for 10+ URLs.

---

Scheduled news monitoring to AI knowledge feed

When: User wants to track industry news sources daily, filter new articles, summarize them, and store in a searchable knowledge base (Notion, NocoDB, Supabase).

Pipeline

1. Extract articles -> lukaskrivka/article-extractor-smart

• Key input: startUrls (news site listing pages), maxCrawlPages, articleSelector (optional CSS hint)

2. Filter new articles only (n8n: compare publishedAt or URL against stored records in DB) 3. Full article content (optional) -> apify/website-content-crawler

• Pipe: new article url values -> startUrls
• Use when listing-page extract is too short for quality summarization

4. AI summarize + tag (n8n: OpenAI node generates 3-sentence summary + keyword tags) 5. Upsert to knowledge base (n8n: Notion / NocoDB / Supabase node)

Output fields

Step 1: title, text, publishedAt, author, url, tags Step 3 (WCC): full text, metadata.title, metadata.description

Gotcha

lukaskrivka/article-extractor-smart handles most news formats well, but paywalled sites return truncated content. Check text length - if consistently under 200 characters for a given source, that site is paywalled and should be removed from the list. Deduplicate by URL before summarizing to avoid re-processing old articles on re-runs.

Lead generation workflows

Local business leads with email enrichment

When: User wants business contacts, emails, or phone numbers for businesses in a specific location.

Pipeline

1. Find businesses -> compass/crawler-google-places

• Key input: searchStringsArray, locationQuery, maxCrawledPlaces

2. Enrich with contacts -> compass/enrich-google-maps-dataset-with-contacts

• Pipe: results[].url -> startUrls (or pass the dataset ID directly)
• Key input: datasetId (from step 1), maxRequestsPerCrawl

Output fields

Step 1: title, address, phone, website, categoryName, totalScore, reviewsCount, url Step 2: emails[], phones[], socialLinks, linkedInUrl, twitterUrl

Gotcha

Google Maps results vary by language and location. Set language: "en" explicitly. Also set locationQuery to a specific city/region, not just a country.

---

B2B prospect discovery via LinkedIn

When: User wants to find professionals by role, company, or industry.

Pipeline

1. Search profiles -> harvestapi/linkedin-profile-search

• Key input: keyword, location, title, limit

2. Enrich with details -> harvestapi/linkedin-profile-scraper

• Pipe: results[].profileUrl -> urls
• Key input: urls, includeEmail (set to true for email discovery)

Output fields

Step 1: fullName, headline, location, profileUrl, currentCompany Step 2: experience[], education[], skills[], email, phone

Cost estimate

Step 2 with includeEmail: true costs ~$0.01/profile. For 500 profiles, budget ~$5.

Gotcha

LinkedIn Actors are all PPE. Estimate and confirm with user before running at scale.

---

Sales Navigator bulk lead extraction

When: User wants daily 100-1,000 lead extraction from a Sales Navigator search for outbound sequences.

Pipeline

1. Extract leads -> harvestapi/linkedin-profile-search

• Key input: searchUrl (Sales Navigator search URL), maxResults, proxy settings

2. Verify emails -> native n8n Hunter.io node or HTTP Request to ZeroBounce API

• Pipe: results[].email -> email verification input

Output fields

Step 1: fullName, email, companyName, jobTitle, connectionDegree, profileUrl Step 2: result (valid/risky/invalid), score

Cost estimate

harvestapi/linkedin-profile-search is PPE. 1,000 leads at typical rates runs ~$5-10. Confirm before scheduling daily runs.

Gotcha

Sales Navigator URL must be a saved search URL, not a one-time results URL. The URL changes each session unless saved.

---

SERP-based B2B prospect discovery

When: User wants to find companies matching niche keywords via Google, AI-qualify them against ICP criteria, and push qualified leads to CRM.

Pipeline

1. Find companies -> apify/google-search-scraper

• Key input: queries (search terms array), maxResultsPerPage, countryCode

2. Crawl company sites -> apify/website-content-crawler

• Pipe: results[].organicResults[].url -> startUrls
• Key input: startUrls, maxCrawlDepth (set to 2), maxCrawlPages (set to 5)

Output fields

Step 1: organicResults[].url, organicResults[].title, organicResults[].snippet Step 2: text (clean markdown), url, metadata.title, metadata.description

Gotcha

Pass only company root domains from SERP results into WCC - not individual blog post URLs. Filter organicResults[].url for root domains before piping.

---

Apollo leads + AI website icebreakers

When: User has an Apollo lead list with company websites and wants personalized cold email icebreakers generated from each company's web presence.

Pipeline

1. Scrape company sites -> apify/website-content-crawler

• Key input: startUrls (homepage URLs from Apollo export), maxCrawlDepth (set to 2), maxCrawlPages (set to 5)

2. Generate icebreakers -> AI node (GPT-4o or Claude)

• Pipe: results[].text -> prompt context per lead
• Key input: company summary + lead name + role

Output fields

Step 1: text (clean markdown), metadata.title, metadata.description, url Step 2: AI-generated icebreaker string per lead

Gotcha

Some Apollo exports include LinkedIn URLs instead of company websites. Filter the list for http URLs before passing to WCC - LinkedIn blocks crawlers.

---

Reddit community lead mining

When: User wants to find prospects actively posting problems that their product or service solves in relevant subreddits.

Pipeline

1. Mine subreddit posts -> trudax/reddit-scraper-lite

• Key input: startUrls (subreddit URLs), searchTerms (problem keywords), maxItems, sort (hot/new/top)

2. Qualify leads -> AI node

• Pipe: results[].title, results[].body -> qualification prompt
• Key input: ICP criteria, pain point keywords

Output fields

Step 1: title, body, subreddit, url, score, numberOfComments, createdAt, author Step 2: AI qualification score, extracted contact intent, suggested outreach angle

Gotcha

Reddit usernames are pseudonymous - there is no direct email enrichment path. The output is intent signals and post URLs for manual outreach via Reddit DM or to cross-reference against other platforms.

Real estate and hospitality workflows

Property search and analysis

When: User wants to find and compare property listings in a specific area.

Pipeline

1. Search properties -> tri_angle/redfin-search

• Key input: location, propertyType, minPrice, maxPrice

2. Get details -> tri_angle/redfin-detail

• Pipe: results[].url -> startUrls
• Key input: startUrls

Output fields

Step 1: address, price, beds, baths, sqft, url, status Step 2: description, yearBuilt, lotSize, priceHistory[], taxHistory[], schools[]

Airbnb market analysis

When: User wants to analyze Airbnb listings, pricing, and reviews in a destination.

Pipeline

1. Search listings -> tri_angle/new-fast-airbnb-scraper

• Key input: location, checkIn, checkOut, maxItems

2. Get reviews -> tri_angle/airbnb-reviews-scraper

• Pipe: results[].url -> startUrls
• Key input: startUrls, maxReviews

Output fields

Step 1: name, price, rating, reviews, type, amenities[], url, images[] Step 2: text, rating, date, reviewerName

Gotcha

Airbnb pricing varies by date. Always set checkIn and checkOut for accurate pricing. For market analysis, run multiple date ranges to capture seasonal variation.

Real estate lead scoring and agent routing

When: User wants to qualify inbound leads from listing portals by budget signals and urgency, then route them to the right agent.

Pipeline

1. Search matching properties -> tri_angle/redfin-search

• Key input: location, minPrice, maxPrice (from lead payload)

2. Enrich lead with LinkedIn signals -> harvestapi/linkedin-profile-scraper

• Key input: profileUrls (optional - use only when lead email resolves to a LinkedIn profile)

Output fields

Step 1: address, price, beds, baths, status, url Step 2: headline, currentCompany, experience[] (income/seniority signals)

Gotcha

The LinkedIn enrichment step is optional - only run it when the lead's identity is known and a LinkedIn profile URL is available. The core routing logic (hot/warm/cold tier + agent assignment) runs on the MLS webhook payload itself, with scraping used as enrichment. Lead scoring and routing output fields are AI-generated: leadTier, assignedAgent, routingReason.

Construction and pre-market property discovery

When: User wants to find new-construction projects or pre-market inventory before they appear on major listing portals.

Pipeline

1. Scrape construction portals -> apify/playwright-scraper

• Key input: startUrls (local MLS or construction project portal URLs), proxyConfiguration

2. Extract clean text -> lukaskrivka/article-extractor-smart

• Pipe: results[].url -> urls

Output fields

Step 1: raw HTML / structured page data Step 2: projectName, price, location, possessionDate, constructionStatus

Gotcha

No market-specific Actor exists for most construction portals (e.g., 99acres). Run apify actors search "real estate" to check for community-built options before using apify/playwright-scraper. For JS-heavy portals, playwright-scraper is required. Step 2 cleans raw output into structured fields - pipe all Step 1 URLs through it.

Multi-source property comparison

When: User wants to compare listings across Zillow, Realtor, Zumper, and other US/UK sources.

Pipeline

1. Aggregate listings -> tri_angle/real-estate-aggregator

• Key input: location, propertyType, sources (Zillow, Realtor, Zumper, Apartments.com, Rightmove)

Output fields

Step 1: address, price, beds, baths, sqft, source, url, listingDate

Review analysis workflows

Google Maps review extraction

When: User wants to collect and analyze business reviews from Google Maps.

Pipeline

1. Find businesses -> compass/crawler-google-places

• Key input: searchStringsArray, locationQuery, maxCrawledPlaces

2. Extract reviews -> compass/Google-Maps-Reviews-Scraper

• Pipe: results[].url -> startUrls
• Key input: startUrls, maxReviews

Output fields

Step 1: title, totalScore, reviewsCount, url, categoryName Step 2: text, stars, publishedAtDate, reviewerName, ownerResponse

Competitor review intelligence

When: User wants to extract competitor reviews to surface customer pain points and compare against own product strengths for positioning.

Pipeline

1. Scrape competitor reviews -> compass/Google-Maps-Reviews-Scraper

• Key input: startUrls (competitor Google Maps URLs), maxReviews, sort (newest or most relevant)

2. Yelp competitor reviews -> tri_angle/yelp-review-scraper

• Key input: startUrls (competitor Yelp URLs), maxReviews

Output fields

Step 1: stars, text, name, publishedAtDate, reviewId Step 2: text, rating, date, userName

Gotcha

Run steps 1 and 2 in parallel for the same competitor, then merge by date. AI analysis works best when you label each review with the competitor name before passing to LLM for theme extraction.

Google Play app review monitoring

When: User wants daily low-rating alerts from Google Play to route urgent negative feedback to the support team.

Pipeline

1. Scrape app reviews -> apify/playwright-scraper

• Key input: startUrls (Google Play app URL), maxRequestsPerCrawl

2. Filter and alert (n8n native - IF node)

• Pipe: results[].stars -> filter where stars < 4

Output fields

Step 1: stars, text, date, appVersion, thumbsUpCount

Gotcha

Google Play uses heavy client-side rendering. Use apify/playwright-scraper rather than cheerio. If results are thin, search Apify Store for a dedicated Google Play reviews Actor - the ecosystem updates frequently.

Cross-platform hotel/restaurant reviews

When: User wants reviews aggregated from multiple platforms for the same business.

Pipeline (hotels)

1. Aggregate reviews -> tri_angle/hotel-review-aggregator

• Key input: urls (hotel URLs from TripAdvisor, Yelp, Google Maps, Booking.com, etc.)

Pipeline (restaurants)

1. Aggregate reviews -> tri_angle/restaurant-review-aggregator

• Key input: urls (restaurant URLs from Yelp, Google Maps, DoorDash, UberEats, etc.)

Output fields

Both: text, rating, date, platform, reviewerName, title

Multi-platform review aggregation for hospitality

When: User wants a weekly sentiment digest across TripAdvisor, Booking.com, Google, and Yelp for a property - including theme extraction by category (service, rooms, location, price).

Pipeline

1. Aggregate all platforms -> tri_angle/hotel-review-aggregator

• Key input: startUrls (property page URLs per platform), maxReviews, includeReviews

2. Airbnb reviews (if applicable) -> tri_angle/airbnb-reviews-scraper

• Key input: startUrls (Airbnb listing URLs), maxReviews

Output fields

Step 1: stars, text, title, reviewDate, source, userProfile.name Step 2: stars, text, reviewDate, reviewerName

Gotcha

Review aggregators pull from multiple platforms in one run - cheaper than running separate scrapers per platform. Use the aggregators when covering 3+ platforms. For Airbnb specifically, run the dedicated tri_angle/airbnb-reviews-scraper separately and merge by date.

Yelp review pipeline

When: User wants Yelp reviews for businesses in a specific area.

Pipeline

1. Find businesses -> tri_angle/get-yelp-urls

• Key input: location, category

2. Extract reviews -> tri_angle/yelp-review-scraper

• Pipe: results[].url -> startUrls
• Key input: startUrls, maxReviews

Output fields

Step 1: name, url, rating, reviewCount, address Step 2: text, rating, date, userName

Social media analytics workflows

Instagram account performance analysis

When: User wants engagement metrics and content performance for an Instagram account.

Pipeline

1. Get profile -> apify/instagram-profile-scraper

• Key input: usernames

2. Get recent posts -> apify/instagram-post-scraper

• Key input: directUrls (from profile's latestPosts[].url) or usernames

Output fields

Step 1: followersCount, followsCount, postsCount, biography, isVerified Step 2: caption, likesCount, commentsCount, timestamp, type (photo/video/reel), url

TikTok creator analytics

When: User wants performance data for a TikTok creator.

Pipeline

1. Get profile -> clockworks/tiktok-profile-scraper

• Key input: profiles (handles or URLs)

Output fields

Step 1: nickname, followers, following, likes, videos, verified, recentVideos[] (with views, likes, shares per video)

Instagram competitor content analysis

When: User wants to identify top-performing content formats and engagement patterns from competitor Instagram accounts.

Pipeline

1. Get competitor posts -> apify/instagram-post-scraper

• Key input: usernames (competitor handles), resultsLimit (100), scrapePostsUntilDate

2. Get reels separately -> apify/instagram-reel-scraper

• Key input: usernames (same handles)

Output fields

Step 1: likesCount, commentsCount, timestamp, type (post/reel/story), caption, displayUrl, url Step 2: likesCount, commentsCount, playsCount, duration, caption, url

Gotcha

Run both Actors to capture full content mix - the post scraper may under-count reels. Calculate engagement rate per post (likes + comments / follower count) and sort to surface top performers.

LinkedIn company page analytics

When: User wants to track LinkedIn post performance for a company page or benchmark against competitors.

Pipeline

1. Get company posts -> harvestapi/linkedin-company-posts

• Key input: companyUrl, maxPosts, publishedAfter

2. Enrich post details -> apimaestro/linkedin-post-detail

• Pipe: results[].url -> urls

Output fields

Step 1: likesCount, commentsCount, repostsCount, text, publishedAt, url Step 2: reactions{} (breakdown by type), topComments[], impressions

Gotcha

Both Actors are PPE. Step 1: ~$0.002/post, Step 2: ~$0.005/post. For 100 posts across 3 companies, estimate ~$2.10. Confirm with user before running.

Multi-platform engagement comparison

When: User wants to compare an account's performance across platforms.

Pipeline (run independently, combine)

1. Instagram -> apify/instagram-profile-scraper with usernames 2. TikTok -> clockworks/tiktok-profile-scraper with profiles 3. YouTube -> streamers/youtube-channel-scraper with channelUrls 4. X/Twitter -> apidojo/twitter-user-scraper with handles

Output fields

Instagram: followersCount, postsCount, biography TikTok: followers, likes, videos YouTube: subscriberCount, videoCount, viewCount X/Twitter: followers, tweets, likes

Gotcha

Parallel workflow - run each Actor independently. Normalize metric names for comparison (followers/subscribers, posts/videos/tweets).

Trend and keyword research workflows

Google Trends analysis

When: User wants to analyze search demand trends for keywords or topics.

Pipeline

1. Get trend data -> apify/google-trends-scraper

• Key input: searchTerms, timeRange, geo (country code)

Output fields

Step 1: term, timelineData[] (date, value), relatedQueries[], relatedTopics[]

Cross-platform hashtag research

When: User wants to evaluate a hashtag's reach and usage across platforms.

Pipeline

1. Cross-platform overview -> apify/social-media-hashtag-research

• Key input: hashtags, platforms (instagram, youtube, tiktok, facebook)

Output fields

Step 1: hashtag, platform, postsCount, topPosts[], relatedHashtags[]

TikTok trend discovery

When: User wants to find trending content, sounds, or hashtags on TikTok.

Pipeline

1. Trending content -> clockworks/tiktok-trends-scraper

• Key input: channel (trending category)

2. Explore categories -> clockworks/tiktok-explore-scraper

• Key input: exploreCategories

Output fields

Step 1: videoUrl, description, likes, shares, views, author, music Step 2: category, posts[], authors[], music[]

Reddit trend and community insight mining

When: User wants to surface emerging trends, product feedback themes, or competitor mentions from Reddit communities.

Pipeline

1. Scrape subreddits -> trudax/reddit-scraper-lite

• Key input: startUrls (subreddit URLs), maxItems, sort (hot/rising/new)

Output fields

Step 1: title, body, subreddit, score, numberOfComments, url, createdAt, comments[]

Gotcha

Use sort: rising for early trend signals, sort: hot for confirmed trending topics. Filter by score threshold (e.g., >50) to reduce noise. Comments array provides qualitative context for AI sentiment analysis.

YouTube outlier video discovery

When: User wants to identify breakout videos in a niche with disproportionate views vs. channel subscriber count - a signal for content strategy pivots.

Pipeline

1. Search niche videos -> streamers/youtube-scraper

• Key input: searchTerms, maxResults, sort (viewCount), uploadDate (filter range)

2. Get channel context -> streamers/youtube-channel-scraper

• Pipe: results[].channelUrl -> channelUrls

Output fields

Step 1: title, viewCount, likeCount, commentCount, channelName, publishedAt, url Step 2: subscriberCount, videoCount, viewCount (channel totals)

Gotcha

Outlier score = video viewCount / channel subscriberCount. Ratios > 10x indicate breakout potential. Run Step 2 on a filtered shortlist only - no need to fetch channel data for every result.

Content topic validation

When: User wants to validate whether a topic has demand before creating content.

Pipeline

1. Search demand -> apify/google-trends-scraper

• Key input: searchTerms (topic keywords)

2. Social reach -> apify/social-media-hashtag-research

• Key input: hashtags (topic hashtags)

Output fields

Step 1: timelineData[] (trending up/down), relatedQueries[] Step 2: postsCount per platform, topPosts[]

Gotcha

Google Trends shows relative interest (0-100 scale), not absolute volume. Combine with hashtag post counts for a fuller picture.