Apify Scrapers

Apify Scrapers

Overview

Scrape content from major social platforms using Apify actors. Each platform has optimized settings for cost and quality.

Quick Decision Tree

What do you want to scrape?
│
├── Social Media Posts
│   ├── Twitter/X → references/twitter.md
│   │   └── Script: scripts/scrape_twitter_ai_trends.py
│   │
│   ├── Reddit → references/reddit.md
│   │   └── Script: scripts/scrape_reddit_ai_tech.py
│   │
│   ├── LinkedIn → references/linkedin.md
│   │   └── Script: scripts/scrape_linkedin_posts.py
│   │
│   ├── Instagram → references/instagram.md
│   │   └── Script: scripts/scrape_instagram.py
│   │   └── Modes: profile, posts, hashtag, reels, comments
│   │
│   ├── Facebook → references/facebook.md
│   │   └── Script: scripts/scrape_facebook.py
│   │   └── Modes: page, posts, reviews, groups, marketplace
│   │
│   ├── TikTok → references/multi-platform.md
│   │   └── Script: scripts/scrape_multi_platform.py
│   │
│   └── YouTube → references/multi-platform.md
│       └── Script: scripts/scrape_multi_platform.py
│
├── Business/Places
│   ├── Google Maps businesses → references/google-maps.md
│   │   └── Script: scripts/scrape_google_maps.py
│   │   └── Modes: search, place, reviews
│   │
│   └── Contact info from websites → references/contact-enrichment.md
│       └── Script: scripts/scrape_contact_info.py
│       └── Extract: emails, phone numbers, social profiles
│
├── Auto-detect URL type → references/url-detect.md
│   └── Script: scripts/scrape_content_by_url.py
│
├── Trend Analysis (NEW)
│   └── Enriched trend analysis → workflows/trend-analysis.md
│       └── Script: scripts/analyze_trends.py
│       └── Features: velocity scoring, lifecycle staging, opportunity scoring
│
└── Workflows (multi-step)
    ├── Lead generation → workflows/lead-generation.md
    ├── Influencer discovery → workflows/influencer-discovery.md
    ├── Competitor analysis → workflows/competitor-intel.md
    ├── Trend analysis → workflows/trend-analysis.md
    └── Competitor Ads Intelligence (NEW) → workflows/competitor-ads.md
        └── Script: scripts/scrape_competitor_ads.py
        └── Platforms: Facebook Ads Library, Google Ads Transparency
        └── Features: Spend estimates, creative analysis, benchmarking

Environment Setup

# Required in .env
APIFY_TOKEN=apify_api_xxxxx

Get your API key: https://console.apify.com/account/integrations

Common Usage Patterns

Scrape Twitter Trends

python scripts/scrape_twitter_ai_trends.py --query "AI agents" --max-tweets 50

Scrape Reddit Discussions

python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning,LocalLLaMA" --max-posts 100

Scrape LinkedIn Author

python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username" --max-posts 30

Auto-detect and Scrape URL

python scripts/scrape_content_by_url.py "https://x.com/user/status/123456"

Scrape Instagram Profile

python scripts/scrape_instagram.py profile "https://instagram.com/username" --max-posts 20

Scrape Instagram Hashtag

python scripts/scrape_instagram.py hashtag "#artificialintelligence" --max-posts 50

Scrape Instagram Reels

python scripts/scrape_instagram.py reels "https://instagram.com/username" --max-reels 30

Scrape Facebook Page

python scripts/scrape_facebook.py page "https://facebook.com/pagename" --max-posts 50

Scrape Facebook Reviews

python scripts/scrape_facebook.py reviews "https://facebook.com/pagename" --max-reviews 100

Scrape Facebook Marketplace

python scripts/scrape_facebook.py marketplace "laptops in san francisco" --max-items 30

Scrape Google Maps Businesses

python scripts/scrape_google_maps.py search "AI consulting firms in New York" --max-results 50

Scrape Google Maps Reviews

python scripts/scrape_google_maps.py reviews "ChIJN1t_tDeuEmsRUsoyG83frY4" --max-reviews 100

Extract Contact Info from Websites

python scripts/scrape_contact_info.py "https://example.com" --depth 2

Bulk Contact Enrichment

python scripts/scrape_contact_info.py --urls-file companies.txt --output contacts.json

Scrape Competitor Ads (Single Competitor)

python scripts/scrape_competitor_ads.py "Nike" --platforms facebook google --country US --days 30

Compare Multiple Competitors' Ads

python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --compare --output comparison.json

Discover Advertisers by Keyword

python scripts/scrape_competitor_ads.py --search "running shoes" --country US --max-ads 200

Filter Competitor Ads by Media Type

python scripts/scrape_competitor_ads.py "Netflix" "Disney+" --platforms facebook --media-types video --days 7

Analyze Trends (NEW)

# Analyze specific topic with enrichments
python scripts/analyze_trends.py "artificial intelligence" --sources google instagram tiktok --days 90

# Discover trending topics in category
python scripts/analyze_trends.py --category technology --discover --top 50

# Compare multiple trends
python scripts/analyze_trends.py "AI" "blockchain" "metaverse" --compare

# Export HTML trend report
python scripts/analyze_trends.py "sustainable fashion" --format html --output trend_report.html

Cost Estimates

Output Location

All scraped data saves to .tmp/ with timestamped filenames:

• .tmp/twitter_ai_trends_YYYYMMDD.json
• .tmp/reddit_ai_tech_YYYYMMDD.json
• .tmp/linkedin_posts_YYYYMMDD_HHMMSS.json

Security Notes

Credential Handling

• Store APIFY_TOKEN in .env file (never commit to git)
• Rotate API tokens periodically via Apify Console
• Never log or print API tokens in script output
• Use environment variables, not hardcoded values

Data Privacy

• Scraped data contains only publicly available content
• Social media posts may include PII (names, handles, profile info)
• Data is stored locally in .tmp/ directory
• No data is retained by Apify after actor run completes
• Consider data minimization - only scrape what you need

Access Scopes

• Apify tokens have full account access (no granular scopes)
• Use separate Apify accounts for different projects if needed
• Monitor usage via Apify Console dashboard

Compliance Considerations

• Terms of Service: Respect each platform's ToS (Twitter, Reddit, LinkedIn)
• Rate Limiting: Actors have built-in rate limiting to avoid bans
• Robots.txt: Some actors may bypass robots.txt - use responsibly
• GDPR: Scraped PII may be subject to GDPR if EU residents
• Ethical Use: Only scrape public data; never bypass authentication
• Proxy Ethics: Residential proxies should be used ethically

Troubleshooting

Common Issues

Issue: Actor run failed

Symptoms: Script terminates with "Actor run failed" or timeout error Cause: Invalid actor ID, insufficient proxy credits, or actor configuration issue Solution:

• Verify the actor ID is correct in the script
• Check Apify Console for actor run logs
• Ensure proxy settings match actor requirements
• Try running with default proxy settings first

Issue: Empty results returned

Symptoms: Script completes but returns 0 items Cause: Content blocked by platform, invalid query, or proxy being detected Solution:

• Try a different proxy type (residential vs datacenter)
• Simplify the search query
• Reduce the number of results requested
• Check if the platform is blocking scraping attempts

Issue: Rate limited by platform

Symptoms: Script fails with 429 errors or "rate limited" messages Cause: Too many requests in a short time period Solution:

• Add delays between requests (actor settings)
• Reduce concurrent requests
• Use proxy rotation
• Wait and retry after a cooldown period

Issue: Invalid API token

Symptoms: Authentication error or "invalid token" message Cause: Token expired, revoked, or incorrectly set Solution:

• Regenerate API token in Apify Console
• Verify token is correctly set in .env file
• Check for leading/trailing whitespace in token
• Ensure APIFY_TOKEN environment variable is loaded

Issue: Proxy connection errors

Symptoms: Connection timeout or proxy errors Cause: Proxy pool exhausted or geo-restriction issues Solution:

• Switch proxy type (basic, residential, or datacenter)
• Verify proxy credit balance in Apify Console
• Try a different proxy country/region
• Disable proxy to test if that's the root cause

Resources

Platform References

• references/twitter.md - Twitter/X scraping details
• references/reddit.md - Reddit scraping with subreddit targeting
• references/linkedin.md - LinkedIn post scraping (author or search mode)
• references/instagram.md - Instagram profile, posts, hashtag, reels, and comments scraping
• references/facebook.md - Facebook page, posts, reviews, groups, and marketplace scraping
• references/multi-platform.md - TikTok and YouTube scraping
• references/url-detect.md - Auto-detect URL type and scrape

Business/Places References

• references/google-maps.md - Google Maps business search, place details, and reviews
• references/contact-enrichment.md - Extract emails, phone numbers, and social profiles from websites

Workflow References

• workflows/lead-generation.md - Multi-step lead generation workflow
• workflows/influencer-discovery.md - Find and analyze influencers across platforms
• workflows/competitor-intel.md - Competitive intelligence gathering workflow
• workflows/trend-analysis.md - Enriched multi-platform trend analysis with scoring

Integration Patterns

Scrape and Enrich

Skills: apify-scrapers → parallel-research Use case: Scrape social media posts, then enrich with deep research Flow: 1. Scrape Twitter/Reddit for mentions of a topic 2. Extract company names or URLs from posts 3. Use parallel-research to get detailed info on each company

Scrape and Summarize

Skills: apify-scrapers → content-generation Use case: Create newsletter content from social media trends Flow: 1. Scrape trending AI posts from Twitter 2. Pass scraped data to content-generation summarize 3. Generate a formatted newsletter section

Scrape and Archive

Skills: apify-scrapers → google-workspace Use case: Save scraped data to Google Drive for team access Flow: 1. Scrape LinkedIn posts from target accounts 2. Format data as CSV or JSON 3. Upload to Google Drive client folder via google-workspace

Trend Analysis + Content Strategy

Skills: apify-scrapers (trend-analysis) → content-generation Use case: Identify trending topics and create content strategy Flow: 1. Run trend analysis: python scripts/analyze_trends.py "AI productivity" --sources all 2. Review lifecycle stage and opportunity score 3. Use content-generation to create content for high-opportunity trends 4. Focus on emerging trends with high velocity scores

Competitive Trend Monitoring

Skills: apify-scrapers (trend-analysis) → parallel-research Use case: Monitor competitor visibility in trending topics Flow: 1. Analyze industry trends: python scripts/analyze_trends.py --category "your-industry" --discover 2. Compare your brand vs competitors in those trends 3. Use parallel-research for deep dive on gaps 4. Generate competitive intelligence report

Contact Enrichment

Overview

Extract emails, phone numbers, and social media links from websites using Apify's contact info scraper.

Actor Used

vdrmota/contact-info-scraper

Inputs

CLI Usage

# Single website
python scripts/enrich_contacts.py "https://example.com"

# Multiple websites
python scripts/enrich_contacts.py "https://site1.com" "https://site2.com"

# From file (one URL per line)
python scripts/enrich_contacts.py --from-file company_websites.txt

# With options
python scripts/enrich_contacts.py "https://example.com" --max-pages 10 --output contacts.json

# Without social links
python scripts/enrich_contacts.py "https://example.com" --no-social

Output Structure

{
  "scraped_at": "2025-01-30T...",
  "platform": "contact_enrichment",
  "total_sites": 5,
  "total_emails_found": 12,
  "total_phones_found": 8,
  "data": [
    {
      "url": "https://example.com",
      "emails": ["info@example.com", "sales@example.com"],
      "phones": ["+1-555-123-4567"],
      "social": {
        "linkedin": "...",
        "twitter": "...",
        "facebook": "..."
      }
    }
  ]
}

Cost Estimates

• ~$0.01-0.02 per website
• 100 websites ≈ $1-2

Error Handling

Testing Checklist

• Pre-flight: APIFY_TOKEN set, URLs are valid
• Smoke test: Run on known site with contact page
• Validate: Check emails are valid format

Use Cases

• Lead generation: Enrich Google Maps results
• Sales prospecting: Get contacts from company websites
• CRM enrichment: Fill in missing contact data

Related Skills

• google-maps - Find businesses, then enrich
• attio-crm - Save enriched contacts
• parallel-research - Company research

Facebook Scraping

Overview

Facebook scraping provides access to public pages, posts, reviews, groups, and marketplace listings. Each data type uses a specialized Apify actor optimized for that content category.

Capabilities:

• Scrape public Facebook page information and metadata
• Extract posts from pages, profiles, and groups
• Collect reviews from business pages
• Monitor group discussions and activity
• Search marketplace listings by location and category

Limitations:

• Private profiles and closed groups are not accessible
• Login-protected content requires authentication (not recommended)
• Rate limits apply per actor run

Actors Used

Inputs

Page Scraper

Posts Scraper

Reviews Scraper

Groups Scraper

Marketplace Scraper

CLI Usage

# Scrape page metadata
python scripts/scrape_facebook.py page "https://facebook.com/pagename"

# Get recent posts from a page
python scripts/scrape_facebook.py posts "https://facebook.com/pagename" --max-posts 50

# Collect business reviews
python scripts/scrape_facebook.py reviews "https://facebook.com/pagename" --max-reviews 100

# Monitor group activity
python scripts/scrape_facebook.py groups "https://facebook.com/groups/groupname" --max-posts 50

# Search marketplace listings
python scripts/scrape_facebook.py marketplace "laptops" --location "New York" --max-items 50

# Marketplace with price filters
python scripts/scrape_facebook.py marketplace "iphone" --location "Los Angeles" --min-price 200 --max-price 800 --max-items 25

Output Structure

Page Output

{
  "page": {
    "id": "page_id",
    "name": "Page Name",
    "url": "https://facebook.com/pagename",
    "category": "Technology Company",
    "about": "Company description...",
    "followers": 150000,
    "likes": 145000,
    "website": "https://example.com",
    "phone": "+1-555-0100",
    "email": "contact@example.com",
    "address": "123 Main St, City, State",
    "hours": {
      "monday": "9:00 AM - 5:00 PM",
      "tuesday": "9:00 AM - 5:00 PM"
    },
    "founded": "2015",
    "verified": true
  },
  "scraped_at": "2024-01-15T10:30:00Z"
}

Posts Output

{
  "posts": [
    {
      "id": "post_id",
      "text": "Post content here...",
      "url": "https://facebook.com/pagename/posts/12345",
      "created_at": "2024-01-14T15:30:00Z",
      "likes": 1250,
      "comments": 45,
      "shares": 120,
      "media": [
        {
          "type": "image",
          "url": "https://..."
        }
      ],
      "author": {
        "name": "Page Name",
        "id": "page_id"
      }
    }
  ],
  "scraped_at": "2024-01-15T10:30:00Z",
  "total_count": 50
}

Reviews Output

{
  "reviews": [
    {
      "id": "review_id",
      "rating": 5,
      "text": "Great service! Highly recommended...",
      "author": {
        "name": "John Doe",
        "id": "user_id",
        "profile_url": "https://facebook.com/johndoe"
      },
      "created_at": "2024-01-10T08:15:00Z",
      "helpful_count": 12,
      "reply": {
        "text": "Thank you for your feedback!",
        "created_at": "2024-01-10T14:00:00Z"
      }
    }
  ],
  "average_rating": 4.5,
  "total_reviews": 523,
  "scraped_at": "2024-01-15T10:30:00Z"
}

Groups Output

{
  "group": {
    "id": "group_id",
    "name": "Group Name",
    "url": "https://facebook.com/groups/groupname",
    "members": 25000,
    "privacy": "public",
    "description": "Group description..."
  },
  "posts": [
    {
      "id": "post_id",
      "text": "Post content...",
      "author": {
        "name": "Member Name",
        "id": "user_id"
      },
      "created_at": "2024-01-14T12:00:00Z",
      "likes": 45,
      "comments": [
        {
          "id": "comment_id",
          "text": "Comment text...",
          "author": "Commenter Name",
          "created_at": "2024-01-14T12:30:00Z"
        }
      ]
    }
  ],
  "scraped_at": "2024-01-15T10:30:00Z",
  "total_count": 50
}

Marketplace Output

{
  "listings": [
    {
      "id": "listing_id",
      "title": "MacBook Pro 2023",
      "price": 1200,
      "currency": "USD",
      "condition": "Used - Like New",
      "description": "Selling my MacBook Pro...",
      "location": {
        "city": "New York",
        "state": "NY",
        "distance": "5 miles away"
      },
      "seller": {
        "name": "Jane Smith",
        "id": "user_id",
        "joined": "2019",
        "response_rate": "Usually responds within an hour"
      },
      "images": [
        "https://..."
      ],
      "url": "https://facebook.com/marketplace/item/12345",
      "posted_at": "2024-01-13T09:00:00Z"
    }
  ],
  "search_query": "laptops",
  "location": "New York",
  "scraped_at": "2024-01-15T10:30:00Z",
  "total_count": 50
}

Cost Estimates

Cost Protection:

• Set maxCostPerRun in actor options
• Start with small maxItems for testing
• Monitor Apify dashboard for usage

Error Handling

Recovery Strategies

1. Automatic retry: Exponential backoff (30s, 60s, 120s) for transient failures 2. Graceful degradation: Reduce maxItems on timeout (50 -> 25 -> 10) 3. Proxy rotation: Built into actors; automatic IP switching 4. Cost limits: Set maxCostPerRun to prevent runaway costs 5. Alerting: Log failures with URL and error details

Testing Checklist

Pre-flight

• [ ] APIFY_API_TOKEN set in .env
• [ ] Dependencies installed (pip install apify-client python-dotenv)
• [ ] Network connectivity to api.apify.com

Smoke Tests

# Test page scraping (use a known public page)
python scripts/scrape_facebook.py page "https://facebook.com/meta" --dry-run

# Test posts with minimal results
python scripts/scrape_facebook.py posts "https://facebook.com/meta" --max-posts 5

# Test marketplace with location
python scripts/scrape_facebook.py marketplace "phone" --location "Chicago" --max-items 5

Validation

• [ ] Response contains expected top-level keys
• [ ] scraped_at timestamp is present and valid
• [ ] Item counts match requested limits
• [ ] No error messages in output
• [ ] URLs in results are valid Facebook links
• [ ] Engagement metrics are integers (not strings)
• [ ] Dates parse correctly

Edge Cases

• [ ] Page with no posts returns empty array (not error)
• [ ] Business page without reviews handles gracefully
• [ ] Marketplace with no results returns empty listings
• [ ] Very long post text is preserved (not truncated)

Performance Tips

Batch Processing

• Combine multiple page URLs in single actor run for page and posts modes
• Use parallel runs for different data types (pages + reviews simultaneously)
• Process results as they stream for large datasets

Rate Limit Handling

• Implement 30-second delays between runs targeting same page
• Distribute requests across time windows
• Use built-in proxy rotation (enabled by default)

Cost Optimization

• Start with maxItems: 10 for testing
• Use date filters (maxPostDate) to limit historical data
• Cache results locally to avoid re-scraping same content
• Disable comments (maxComments: 0) if not needed

Memory Usage

• Stream large result sets to file instead of memory
• Process marketplace results in chunks of 100
• Clear actor storage after successful extraction

Data Freshness

• Posts/reviews: Daily scraping for active monitoring
• Page metadata: Weekly updates sufficient
• Marketplace: Real-time or hourly for time-sensitive listings

Related Skills

Example Workflows

Brand Monitoring

# 1. Get latest posts from brand page
python scripts/scrape_facebook.py posts "https://facebook.com/brandname" --max-posts 20

# 2. Check recent reviews
python scripts/scrape_facebook.py reviews "https://facebook.com/brandname" --max-reviews 50

# 3. Export to Google Sheets for analysis
python execution/export_to_sheets.py .tmp/facebook_results.json "Brand Monitoring"

Competitive Intelligence

# Scrape competitor pages
for page in "competitor1" "competitor2" "competitor3"; do
  python scripts/scrape_facebook.py posts "https://facebook.com/$page" --max-posts 30
done

Market Research

# Search marketplace for product category
python scripts/scrape_facebook.py marketplace "electric bikes" \
  --location "San Francisco" \
  --min-price 500 \
  --max-price 3000 \
  --max-items 100

Google Maps/Places Scraping

Overview

Scrape Google Maps for business listings, place details, reviews, and contact information. Ideal for lead generation, competitor analysis, and market research.

Actors

Inputs

Search Mode

Place Mode

Reviews Mode

CLI Usage

# Search for businesses in a location
python scripts/scrape_google_maps.py search "coffee shops" --location "San Francisco" --max-results 50

# Search with rating filter
python scripts/scrape_google_maps.py search "restaurants" --location "NYC" --min-rating 4.0 --max-results 100

# Search with zoom level for larger area
python scripts/scrape_google_maps.py search "gyms" --location "Los Angeles" --zoom 12 --max-results 200

# Extract details from a specific place
python scripts/scrape_google_maps.py place "https://maps.google.com/maps/place/..."

# Batch extract multiple places
python scripts/scrape_google_maps.py place "url1" "url2" "url3"

# Scrape reviews for a place
python scripts/scrape_google_maps.py reviews "https://maps.google.com/maps/place/..." --max-reviews 100 --sort-by newest

# Get highest rated reviews
python scripts/scrape_google_maps.py reviews "https://maps.google.com/..." --max-reviews 50 --sort-by highest

Output Structure

Search/Place Output

{
  "places": [
    {
      "place_id": "ChIJN1t_tDeuEmsRUsoyG83frY4",
      "name": "Business Name",
      "address": "123 Main St, City, State 12345",
      "phone": "+1-555-123-4567",
      "website": "https://example.com",
      "rating": 4.5,
      "reviews_count": 234,
      "price_level": "$$",
      "categories": ["Coffee Shop", "Cafe"],
      "coordinates": {
        "lat": 37.7749,
        "lng": -122.4194
      },
      "hours": {
        "monday": "7:00 AM - 8:00 PM",
        "tuesday": "7:00 AM - 8:00 PM"
      },
      "url": "https://maps.google.com/maps/place/..."
    }
  ],
  "scraped_at": "2024-01-15T10:30:00Z",
  "total_count": 50,
  "query": "coffee shops",
  "location": "San Francisco"
}

Reviews Output

{
  "place": {
    "name": "Business Name",
    "place_id": "ChIJN1t_tDeuEmsRUsoyG83frY4"
  },
  "reviews": [
    {
      "author": "John D.",
      "rating": 5,
      "text": "Great coffee and friendly staff!",
      "date": "2024-01-10",
      "likes": 12,
      "owner_response": "Thank you for visiting!"
    }
  ],
  "scraped_at": "2024-01-15T10:30:00Z",
  "total_count": 100
}

Cost Estimates

Examples:

• 50 places search: ~$0.50-1.00
• 100 reviews: ~$0.50-1.00
• Batch 20 place URLs: ~$0.20-0.60

Testing Checklist

Pre-flight

• [ ] APIFY_API_TOKEN set in .env
• [ ] Dependencies installed (pip install apify-client python-dotenv)
• [ ] Network connectivity to api.apify.com

Smoke Test

# Quick search test (minimal results)
python scripts/scrape_google_maps.py search "coffee" --location "San Francisco" --max-results 5

# Single place extraction
python scripts/scrape_google_maps.py place "https://maps.google.com/maps/place/Starbucks/..."

# Reviews test (small batch)
python scripts/scrape_google_maps.py reviews "https://maps.google.com/..." --max-reviews 10

Validation

• [ ] Response contains places array with expected fields
• [ ] place_id is present for each result
• [ ] coordinates contains valid lat/lng
• [ ] phone and website populated when available
• [ ] rating is between 0-5
• [ ] reviews_count is a positive integer
• [ ] scraped_at timestamp is present and valid
• [ ] No error messages in output

Error Handling

Recovery Strategies

1. Automatic retry: Implement exponential backoff (30s, 60s, 120s) for transient failures 2. Graceful degradation: Reduce max_results if actor times out 3. URL validation: Pre-validate Google Maps URLs before submitting 4. Cost protection: Set maxCostPerRun to prevent runaway costs 5. Batch chunking: Split large place URL lists into batches of 20-50

Performance Tips

Search Optimization

• Use specific zoom levels: 10 for city-wide, 14 for neighborhoods, 17 for specific streets
• Lower zoom = larger area = more results but slower
• Add min_rating filter to reduce low-quality results

Batch Processing

• Combine multiple place URLs in single actor run (up to 100)
• Use max_results to limit during testing
• Process results incrementally for large datasets

Cost Optimization

• Start with max_results: 10 for testing queries
• Use min_rating to filter out irrelevant results early
• Cache place_ids to avoid re-scraping known places
• Batch place detail requests instead of individual calls

Memory Usage

• Stream large result sets to file for >500 places
• Process reviews in batches of 100
• Use pagination for very large datasets

Use Cases

Lead Generation

# Find potential clients in target area
python scripts/scrape_google_maps.py search "dental clinics" --location "Austin, TX" --max-results 200 --min-rating 3.5

Competitor Analysis

# Analyze competitor reviews
python scripts/scrape_google_maps.py reviews "https://maps.google.com/competitor-url" --max-reviews 500 --sort-by newest

Market Research

# Map business density in area
python scripts/scrape_google_maps.py search "restaurants" --location "Downtown Seattle" --zoom 15 --max-results 500

Contact Enrichment

# Get phone/website for known businesses
python scripts/scrape_google_maps.py place "url1" "url2" "url3" "url4" "url5"

Related Skills

Example Pipeline

# 1. Scrape leads from Google Maps
python scripts/scrape_google_maps.py search "marketing agencies" --location "Chicago" --max-results 100

# 2. Enrich with company research
python ~/.claude/skills/parallel-research/scripts/research.py company "Company Name"

# 3. Save to CRM
python ~/.claude/skills/attio-crm/scripts/create_company.py --name "Company" --website "url" --phone "phone"

# 4. Notify team
python ~/.claude/skills/slack-automation/scripts/send_message.py "#leads" "New leads added from Chicago search"

Notes

• Google Maps data updates frequently; re-scrape periodically for accuracy
• Some businesses may have incomplete data (missing phone/website)
• Review scraping respects Google's display limits
• Place IDs are stable and can be used for tracking/deduplication
• Coordinates are WGS84 format (standard GPS)

Instagram Scraping

Overview

Comprehensive Instagram scraping capabilities supporting multiple modes: profile data, posts, hashtag discovery, reels, and comments. Each mode uses a specialized Apify actor optimized for that content type.

Actors Used

Inputs

Profile Mode

Posts Mode

Hashtag Mode

Reels Mode

Comments Mode

CLI Usage

Profile Scraping

# Single profile
python scripts/scrape_instagram.py profile cristiano

# Multiple profiles
python scripts/scrape_instagram.py profile cristiano messi neymarjr

# With custom output
python scripts/scrape_instagram.py profile natgeo --output natgeo_profile.json

# Batch competitor analysis
python scripts/scrape_instagram.py profile competitor1 competitor2 competitor3 --output competitors.json

Posts Scraping

# Default 50 posts from profile
python scripts/scrape_instagram.py posts username

# Multiple profiles with custom limit
python scripts/scrape_instagram.py posts username1 username2 --max-posts 100

# Quick scan with minimal posts
python scripts/scrape_instagram.py posts natgeo --max-posts 10 --output natgeo_posts.json

# Comprehensive profile posts
python scripts/scrape_instagram.py posts redbull --max-posts 200

Hashtag Scraping

# Single hashtag
python scripts/scrape_instagram.py hashtag artificialintelligence

# Multiple hashtags
python scripts/scrape_instagram.py hashtag ai ml machinelearning --max-posts 100

# Trending topic research
python scripts/scrape_instagram.py hashtag tech startup entrepreneur --max-posts 50

# Niche discovery
python scripts/scrape_instagram.py hashtag smallbusiness shoplocal --max-posts 200 --output niche_posts.json

Reels Scraping

# Default 20 reels from profile
python scripts/scrape_instagram.py reels username

# Multiple profiles
python scripts/scrape_instagram.py reels redbull gopro --max-reels 30

# Viral content research
python scripts/scrape_instagram.py reels tiktok_migrants --max-reels 50 --output viral_reels.json

Comments Scraping

# Single post
python scripts/scrape_instagram.py comments "https://www.instagram.com/p/ABC123/"

# Multiple posts
python scripts/scrape_instagram.py comments "https://www.instagram.com/p/ABC123/" "https://www.instagram.com/p/DEF456/" --max-comments 100

# Reel comments
python scripts/scrape_instagram.py comments "https://www.instagram.com/reel/XYZ789/" --max-comments 200

# Sentiment analysis prep
python scripts/scrape_instagram.py comments "https://www.instagram.com/p/ABC123/" --max-comments 500 --output sentiment_data.json

Output Structure

Profile Output

{
  "scraped_at": "2025-01-15T10:30:00.000Z",
  "platform": "instagram",
  "mode": "profile",
  "total_count": 1,
  "query": ["cristiano"],
  "run_id": "abc123",
  "data": [
    {
      "username": "cristiano",
      "full_name": "Cristiano Ronaldo",
      "biography": "Football player...",
      "external_url": "https://example.com",
      "followers_count": 500000000,
      "following_count": 500,
      "posts_count": 3500,
      "is_verified": true,
      "is_private": false,
      "is_business": true,
      "business_category": "Athlete",
      "profile_pic_url": "https://...",
      "profile_pic_url_hd": "https://...",
      "profile_url": "https://www.instagram.com/cristiano/",
      "id": "12345678"
    }
  ]
}

Posts Output

{
  "scraped_at": "2025-01-15T10:30:00.000Z",
  "platform": "instagram",
  "mode": "posts",
  "total_count": 50,
  "query": ["natgeo"],
  "run_id": "abc123",
  "data": [
    {
      "id": "post_id",
      "shortcode": "ABC123xyz",
      "caption": "Beautiful sunset captured in...",
      "owner_username": "natgeo",
      "timestamp": "2025-01-14T18:00:00.000Z",
      "likes_count": 150000,
      "comments_count": 2500,
      "video_view_count": 0,
      "video_play_count": 0,
      "is_video": false,
      "type": "Image",
      "display_url": "https://...",
      "video_url": "",
      "post_url": "https://www.instagram.com/p/ABC123xyz/",
      "location": "Yellowstone National Park",
      "hashtags": ["nature", "photography", "wildlife"],
      "mentions": ["photographer_name"],
      "engagement_score": 155000
    }
  ]
}

Hashtag Output

{
  "scraped_at": "2025-01-15T10:30:00.000Z",
  "platform": "instagram",
  "mode": "hashtag",
  "total_count": 100,
  "query": ["artificialintelligence"],
  "run_id": "abc123",
  "data": [
    {
      "id": "post_id",
      "shortcode": "XYZ789abc",
      "caption": "The future of AI is here...",
      "owner_username": "tech_influencer",
      "timestamp": "2025-01-14T12:00:00.000Z",
      "likes_count": 5000,
      "comments_count": 150,
      "video_view_count": 0,
      "is_video": false,
      "type": "Sidecar",
      "display_url": "https://...",
      "video_url": "",
      "post_url": "https://www.instagram.com/p/XYZ789abc/",
      "hashtags": ["ai", "artificialintelligence", "tech"],
      "source_hashtag": "artificialintelligence",
      "engagement_score": 5300
    }
  ]
}

Reels Output

{
  "scraped_at": "2025-01-15T10:30:00.000Z",
  "platform": "instagram",
  "mode": "reels",
  "total_count": 20,
  "query": ["redbull"],
  "run_id": "abc123",
  "data": [
    {
      "id": "reel_id",
      "shortcode": "REE123lzz",
      "caption": "Extreme sports at its finest...",
      "owner_username": "redbull",
      "timestamp": "2025-01-13T09:00:00.000Z",
      "likes_count": 250000,
      "comments_count": 5000,
      "play_count": 5000000,
      "view_count": 4500000,
      "duration": 30,
      "video_url": "https://...",
      "thumbnail_url": "https://...",
      "reel_url": "https://www.instagram.com/reel/REE123lzz/",
      "audio_title": "Original Audio",
      "audio_artist": "redbull",
      "hashtags": ["redbull", "extremesports"],
      "engagement_score": 305000
    }
  ]
}

Comments Output

{
  "scraped_at": "2025-01-15T10:30:00.000Z",
  "platform": "instagram",
  "mode": "comments",
  "total_count": 100,
  "query": ["https://www.instagram.com/p/ABC123/"],
  "run_id": "abc123",
  "data": [
    {
      "id": "comment_id",
      "text": "This is amazing! Love this content",
      "owner_username": "commenter_name",
      "owner_profile_pic": "https://...",
      "timestamp": "2025-01-14T19:30:00.000Z",
      "likes_count": 150,
      "replies_count": 5,
      "post_shortcode": "ABC123",
      "post_url": "https://www.instagram.com/p/ABC123/",
      "is_reply": false,
      "parent_comment_id": ""
    }
  ]
}

Cost Estimates

Notes:

• Pay-per-result pricing varies by actor and data complexity
• Profile scraping is most expensive due to detailed metadata
• Costs may vary based on Apify pricing changes
• Set maxCostPerRun in actor options to prevent runaway costs

Error Handling

Recovery Strategies

1. Automatic retry: Implement exponential backoff (30s, 60s, 120s) for transient failures 2. Graceful degradation: If profile fails, continue with remaining profiles 3. Batch splitting: For large requests, split into chunks of 10-20 items 4. Cost protection: Set maxCostPerRun in actor options 5. Fallback actors: Some actors have alternatives with different pricing/capabilities

Testing Checklist

Pre-flight

• [ ] APIFY_TOKEN set in .env (note: uses APIFY_TOKEN, not APIFY_API_TOKEN)
• [ ] Dependencies installed (pip install apify-client python-dotenv)
• [ ] Network connectivity to api.apify.com
• [ ] Sufficient Apify credits available
• [ ] .tmp/ directory exists or can be created

Smoke Tests

# Profile: Quick test with known public profile
python scripts/scrape_instagram.py profile natgeo --output test_profile.json

# Posts: Minimal posts from public account
python scripts/scrape_instagram.py posts natgeo --max-posts 5 --output test_posts.json

# Hashtag: Popular hashtag with few results
python scripts/scrape_instagram.py hashtag photography --max-posts 5 --output test_hashtag.json

# Reels: Minimal reels from active account
python scripts/scrape_instagram.py reels redbull --max-reels 5 --output test_reels.json

# Comments: Popular post with many comments
python scripts/scrape_instagram.py comments "https://www.instagram.com/p/KNOWN_POST_ID/" --max-comments 10 --output test_comments.json

Validation Checklist

Profile Mode

• [ ] Response contains data array with profile objects
• [ ] Each profile has username, followers_count, posts_count
• [ ] is_verified and is_private flags present
• [ ] profile_url is valid Instagram URL
• [ ] scraped_at timestamp is present and valid

Posts Mode

• [ ] Response contains data array sorted by engagement_score (descending)
• [ ] Each post has shortcode, caption, likes_count, comments_count
• [ ] post_url is valid and matches shortcode
• [ ] owner_username matches requested profile
• [ ] hashtags and mentions arrays present

Hashtag Mode

• [ ] Response contains data array sorted by engagement_score
• [ ] source_hashtag field matches requested hashtag
• [ ] Posts are from various users (not just one account)
• [ ] total_count respects --max-posts limit

Reels Mode

• [ ] Response contains data array with reel objects
• [ ] Each reel has play_count, view_count, duration
• [ ] video_url and thumbnail_url present
• [ ] audio_title and audio_artist populated when available

Comments Mode

• [ ] Response contains data array sorted by likes_count
• [ ] Each comment has text, owner_username, likes_count
• [ ] is_reply flag distinguishes top-level vs reply comments
• [ ] post_shortcode matches input URL

Performance Tips

Batch Processing

• Batch usernames: Combine multiple usernames in single run

  python scripts/scrape_instagram.py profile user1 user2 user3 user4 user5

• Parallel hashtags: Search multiple related hashtags together

  python scripts/scrape_instagram.py hashtag ai ml deeplearning neuralnetworks

• Limit for testing: Use small limits during development

  python scripts/scrape_instagram.py posts username --max-posts 10

Cost Optimization

• Start with minimal items (--max-posts 10) for testing
• Use profile mode first to verify account exists before scraping posts
• Cache results locally to avoid re-scraping unchanged profiles
• Use hashtag mode for discovery, then target specific profiles

Discovery Strategy

1. Find influencers: Use hashtag mode to find active accounts in a niche 2. Analyze profiles: Scrape profiles to compare follower counts and engagement 3. Deep dive: Scrape posts from top performers to analyze content strategy 4. Engagement analysis: Scrape comments to understand audience sentiment

Rate Limit Handling

• Space out large requests (wait 60s between batches)
• Use smaller batches for sensitive accounts
• Monitor Apify run status for rate limit warnings
• The actors handle proxy rotation automatically

Memory Usage

• For >500 items, consider splitting into multiple runs
• Process results in chunks when analyzing large datasets
• Stream results to file instead of holding in memory

Related Skills

Example Workflow

# 1. Discover accounts via hashtag
python scripts/scrape_instagram.py hashtag sustainablefashion --max-posts 100

# 2. Identify top accounts from results
# (manually or via script analysis)

# 3. Get detailed profile data
python scripts/scrape_instagram.py profile account1 account2 account3

# 4. Analyze their best content
python scripts/scrape_instagram.py posts account1 --max-posts 50

# 5. Understand audience sentiment
python scripts/scrape_instagram.py comments "https://instagram.com/p/top_post/" --max-comments 200

# 6. Generate report (using content-generation skill)
# 7. Save to Google Drive (using google-workspace skill)

LinkedIn Scraping

Actor

harvestapi/linkedin-post-search

Modes

Author Mode

Scrape posts from a specific LinkedIn profile.

Search Mode

Scrape posts matching search keywords.

Inputs

CLI Usage

# Scrape from author profile
python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username"

# Multiple profiles
python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/user1" "https://linkedin.com/in/user2"

# Search by keyword
python scripts/scrape_linkedin_posts.py search "AI agents" "automation tools"

# With comments
python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username" --scrape-comments

Output Structure

{
  "posts": [
    {
      "id": "post_id",
      "text": "post content",
      "author_name": "Full Name",
      "author_url": "profile_url",
      "posted_at": "timestamp",
      "likes": 0,
      "comments": 0,
      "reposts": 0,
      "post_url": "url_to_post",
      "media_urls": [],
      "hashtags": []
    }
  ],
  "scraped_at": "timestamp",
  "mode": "author|search",
  "query": "input used"
}

Cost

Higher than other platforms. Disable comments/reactions unless needed.

Testing Checklist

Pre-flight

• [ ] APIFY_API_TOKEN set in .env
• [ ] Dependencies installed (pip install apify-client python-dotenv)
• [ ] Network connectivity to api.apify.com
• [ ] Test LinkedIn profile URL is valid and public

Smoke Test

# Test author mode with a known public profile (minimal posts)
python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/satlovsolutions" --max-posts 5

# Test search mode with a simple keyword
python scripts/scrape_linkedin_posts.py search "AI agents" --max-posts 5

Validation

• [ ] Response contains posts array with expected fields (id, text, author_name, likes)
• [ ] scraped_at timestamp is present and valid
• [ ] mode matches requested mode (author or search)
• [ ] query field contains the input used
• [ ] Post URLs are valid LinkedIn URLs
• [ ] Author URLs are valid LinkedIn profile URLs
• [ ] Comments included only when --scrape-comments flag used
• [ ] No error messages related to profile access
• [ ] Cost is higher than Twitter/Reddit - monitor usage

Error Handling

Recovery Strategies

1. Automatic retry: Wait 2-5 minutes between retries for blocked requests 2. Graceful degradation: Disable scrape_comments and scrape_reactions to reduce detection risk 3. Batch limits: Process max 5 profiles per run to avoid rate limits 4. Cost protection: Set budget limits as LinkedIn scraping is expensive 5. Alerting: Log blocked URLs for manual verification

Performance Tips

Batch Processing

• Combine multiple URLs in single actor run
• Use maxItems to limit results when testing
• Process results as they stream (if supported)

Rate Limit Handling

• Implement exponential backoff between requests
• Use proxies to distribute load
• Respect platform rate limits (Twitter: 300/15min)

Cost Optimization

• Start with maxItems: 10 for testing
• Use date filters to reduce volume
• Cache results locally to avoid re-scraping

Memory Usage

• Process large result sets in chunks
• Stream to file instead of memory for >1000 items

Multi-Platform Scraping (TikTok & YouTube)

Script

scripts/scrape_multi_platform.py

TikTok

Actor

clockworks/tiktok-scraper

Inputs

Usage

python scripts/scrape_multi_platform.py tiktok --hashtags "AI,tech" --max-videos 50

YouTube

Actor

streamers/youtube-scraper

Inputs

Usage

python scripts/scrape_multi_platform.py youtube --query "AI tutorial" --max-results 20

# With subtitles
python scripts/scrape_multi_platform.py youtube --query "AI" --subtitles

Output Structure

TikTok

{
  "videos": [
    {
      "id": "video_id",
      "description": "caption",
      "author": "username",
      "likes": 0,
      "comments": 0,
      "shares": 0,
      "video_url": "url"
    }
  ]
}

YouTube

{
  "videos": [
    {
      "id": "video_id",
      "title": "Video Title",
      "channelName": "Channel",
      "viewCount": 0,
      "likeCount": 0,
      "description": "...",
      "subtitles": "transcript text"
    }
  ]
}

Cost Estimates

• TikTok: ~$0.005 per video
• YouTube: ~$0.01-0.05 per video (more with subtitles)

Testing Checklist

Pre-flight

• [ ] APIFY_API_TOKEN set in .env
• [ ] Dependencies installed (pip install apify-client python-dotenv)
• [ ] Network connectivity to api.apify.com

Smoke Test

# Test TikTok with a simple hashtag
python scripts/scrape_multi_platform.py tiktok --hashtags "tech" --max-videos 5

# Test YouTube with a simple search
python scripts/scrape_multi_platform.py youtube --query "AI tutorial" --max-results 5

# Test YouTube with subtitles
python scripts/scrape_multi_platform.py youtube --query "Python basics" --max-results 3 --subtitles

Validation

TikTok

• [ ] Response contains videos array with expected fields (id, description, author, likes)
• [ ] Video URLs are accessible
• [ ] Engagement metrics present (likes, comments, shares)
• [ ] Cost estimate: ~$0.025 for 5 videos

YouTube

• [ ] Response contains videos array with expected fields (id, title, channelName, viewCount)
• [ ] Video IDs are valid YouTube video IDs
• [ ] subtitles field populated when --subtitles flag used
• [ ] Channel names and descriptions present
• [ ] Cost estimate: ~$0.05-0.25 for 5 videos (more with subtitles)

Error Handling

Recovery Strategies

1. Automatic retry: Implement exponential backoff (60s, 120s, 240s) for rate limits 2. Graceful degradation: If subtitles fail, return video metadata without transcript 3. Platform fallback: If one platform fails entirely, continue with successful platforms 4. Batch processing: Limit to 20 videos per run to avoid timeouts 5. Cost monitoring: Track per-platform costs and set alerts at thresholds

Performance Tips

Batch Processing

• Combine multiple URLs in single actor run
• Use maxItems to limit results when testing
• Process results as they stream (if supported)

Rate Limit Handling

• Implement exponential backoff between requests
• Use proxies to distribute load
• Respect platform rate limits (Twitter: 300/15min)

Cost Optimization

• Start with maxItems: 10 for testing
• Use date filters to reduce volume
• Cache results locally to avoid re-scraping

Memory Usage

• Process large result sets in chunks
• Stream to file instead of memory for >1000 items

Reddit Scraping

Actor

trudax/reddit-scraper

Inputs

CLI Usage

# Default AI subreddits
python scripts/scrape_reddit_ai_tech.py

# Specific subreddits
python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning,LocalLLaMA,ClaudeAI"

# With time filter
python scripts/scrape_reddit_ai_tech.py --time week --sort top

Output Structure

{
  "posts": [
    {
      "id": "post_id",
      "title": "post title",
      "subreddit": "MachineLearning",
      "author": "username",
      "score": 1234,
      "num_comments": 56,
      "url": "post_url",
      "created_utc": "timestamp",
      "selftext": "post content",
      "top_comments": []
    }
  ],
  "scraped_at": "timestamp",
  "total_count": 0
}

Best Subreddits for AI/Tech

• r/artificial - AI news
• r/MachineLearning - ML research
• r/LocalLLaMA - Local AI models
• r/ChatGPT, r/OpenAI, r/ClaudeAI - Specific tools
• r/technology, r/Futurology - General tech

Cost

~$0.001-0.005 per post

Testing Checklist

Pre-flight

• [ ] APIFY_API_TOKEN set in .env
• [ ] Dependencies installed (pip install apify-client python-dotenv)
• [ ] Network connectivity to api.apify.com

Smoke Test

# Quick test with default AI subreddits (minimal posts)
python scripts/scrape_reddit_ai_tech.py --max-posts 5

# Test specific subreddit
python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning" --max-posts 5 --sort hot

Validation

• [ ] Response contains posts array with expected fields (id, title, subreddit, score)
• [ ] scraped_at timestamp is present and valid
• [ ] total_count matches actual posts returned
• [ ] Posts are from correct subreddit(s)
• [ ] Sort order matches requested (hot, top, new)
• [ ] Time filter applied correctly (day, week, etc.)
• [ ] top_comments included when requested
• [ ] Cost estimate: ~$0.005-0.025 for 5 posts

Error Handling

Recovery Strategies

1. Automatic retry: Implement exponential backoff (30s, 60s, 120s) for rate limits 2. Graceful degradation: If a subreddit fails, continue with remaining subreddits 3. Fallback subreddits: Maintain list of alternative subs for each topic 4. Batch processing: Split large requests into smaller chunks (10 posts at a time)

Performance Tips

Batch Processing

• Combine multiple URLs in single actor run
• Use maxItems to limit results when testing
• Process results as they stream (if supported)

Rate Limit Handling

• Implement exponential backoff between requests
• Use proxies to distribute load
• Respect platform rate limits (Twitter: 300/15min)

Cost Optimization

• Start with maxItems: 10 for testing
• Use date filters to reduce volume
• Cache results locally to avoid re-scraping

Memory Usage

• Process large result sets in chunks
• Stream to file instead of memory for >1000 items

Twitter/X Scraping

Actor

kaitoeasyapi/twitter-x-data-tweet-scraper-pay-per-result-cheapest

Inputs

CLI Usage

# Basic search
python scripts/scrape_twitter_ai_trends.py --query "OpenAI"

# With engagement filter
python scripts/scrape_twitter_ai_trends.py --query "ChatGPT" --max-tweets 100 --min-likes 10

Output Structure

{
  "tweets": [
    {
      "id": "tweet_id",
      "text": "tweet content",
      "author": "username",
      "created_at": "timestamp",
      "likes": 0,
      "retweets": 0,
      "replies": 0,
      "url": "tweet_url"
    }
  ],
  "scraped_at": "timestamp",
  "total_count": 0
}

Cost

~$0.00025 per tweet ($0.10-0.50 per 100 tweets)

Notes

• Pay-per-result pricing
• Filters: Excludes retweets, excludes replies
• Sorted by engagement (likes + retweets) descending

Testing Checklist

Pre-flight

• [ ] APIFY_API_TOKEN set in .env
• [ ] Dependencies installed (pip install apify-client python-dotenv)
• [ ] Network connectivity to api.apify.com

Smoke Test

# Quick test with a simple query (minimal results)
python scripts/scrape_twitter_ai_trends.py --query "test" --max-tweets 5

# Verify with a known popular term
python scripts/scrape_twitter_ai_trends.py --query "OpenAI" --max-tweets 10

Validation

• [ ] Response contains tweets array with expected fields (id, text, author, likes)
• [ ] scraped_at timestamp is present and valid
• [ ] total_count matches actual tweets returned
• [ ] No error messages in output
• [ ] Tweets are sorted by engagement (descending)
• [ ] No retweets or replies in results (filter working)
• [ ] Cost estimate: ~$0.0025 for 10 tweets

Error Handling

Recovery Strategies

1. Automatic retry: Implement exponential backoff (30s, 60s, 120s) for transient failures 2. Graceful degradation: If actor fails, fall back to reduced max_tweets (25 instead of 100) 3. Cost protection: Set maxCostPerRun in actor options to prevent runaway costs 4. Alerting: Log failures with query details for investigation

Performance Tips

Batch Processing

• Combine multiple URLs in single actor run
• Use maxItems to limit results when testing
• Process results as they stream (if supported)

Rate Limit Handling

• Implement exponential backoff between requests
• Use proxies to distribute load
• Respect platform rate limits (Twitter: 300/15min)

Cost Optimization

• Start with maxItems: 10 for testing
• Use date filters to reduce volume
• Cache results locally to avoid re-scraping

Memory Usage

• Process large result sets in chunks
• Stream to file instead of memory for >1000 items

URL Auto-Detection & Scraping

Script

scripts/scrape_content_by_url.py

Overview

Automatically detects URL type and uses the appropriate scraper.

Supported URL Types

Usage

# Auto-detect and scrape
python scripts/scrape_content_by_url.py "https://x.com/user/status/123456"

# Force type
python scripts/scrape_content_by_url.py "https://example.com/article" --type website

URL Type Detection Logic

def detect_url_type(url):
    if "twitter.com" in url or "x.com" in url:
        return "twitter"
    elif "youtube.com" in url or "youtu.be" in url:
        return "youtube"
    elif "reddit.com" in url:
        return "reddit"
    else:
        return "website"

Output Structure

{
  "url": "original_url",
  "type": "twitter|youtube|reddit|website",
  "scraped_at": "timestamp",
  "content": {
    // Type-specific content
  }
}

Twitter Output Fields

• text, author, likeCount, retweetCount, createdAt

YouTube Output Fields

• title, description, channelName, viewCount, subtitles

Reddit Output Fields

• title, body, author, score, comments

Website Output Fields (Firecrawl)

• title, text (markdown), url, description, metadata

Cost Estimates

• Twitter: ~$0.00025 per tweet
• YouTube: ~$0.01-0.05 per video
• Reddit: ~$0.001-0.005 per post
• Website: ~1 Firecrawl credit per page

Testing Checklist

Pre-flight

• [ ] APIFY_API_TOKEN set in .env
• [ ] FIRECRAWL_API_KEY set in .env (for website fallback)
• [ ] Dependencies installed (pip install apify-client firecrawl-py python-dotenv)
• [ ] Network connectivity to api.apify.com and api.firecrawl.dev

Smoke Test

# Test Twitter/X URL detection
python scripts/scrape_content_by_url.py "https://x.com/OpenAI/status/1234567890"

# Test YouTube URL detection
python scripts/scrape_content_by_url.py "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

# Test Reddit URL detection
python scripts/scrape_content_by_url.py "https://www.reddit.com/r/MachineLearning/comments/abc123"

# Test generic website (Firecrawl fallback)
python scripts/scrape_content_by_url.py "https://example.com/article"

# Force website type
python scripts/scrape_content_by_url.py "https://x.com/somepage" --type website

Validation

• [ ] Response contains url (original input), type (detected), scraped_at
• [ ] type correctly detected for each URL pattern:
• twitter.com/* or x.com/* -> twitter
• youtube.com/* or youtu.be/* -> youtube
• reddit.com/* -> reddit
• Other URLs -> website
• [ ] content object contains type-specific fields (see Output Fields)
• [ ] --type flag overrides auto-detection when specified
• [ ] No errors when URL is unreachable (graceful failure)
• [ ] Cost varies by detected type (see Cost Estimates above)

Error Handling

Recovery Strategies

1. Automatic retry: 3 attempts with exponential backoff for transient failures 2. Graceful degradation: If specialized scraper fails, try generic website scraper 3. URL validation: Pre-validate URL accessibility before scraping 4. Type override: Allow --type flag to force specific scraper if auto-detection fails 5. Caching: Cache successful scrapes to avoid redundant requests for same URL

Performance Tips

Batch Processing

• Combine multiple URLs in single actor run
• Use maxItems to limit results when testing
• Process results as they stream (if supported)

Rate Limit Handling

• Implement exponential backoff between requests
• Use proxies to distribute load
• Respect platform rate limits (Twitter: 300/15min)

Cost Optimization

• Start with maxItems: 10 for testing
• Use date filters to reduce volume
• Cache results locally to avoid re-scraping

Memory Usage

• Process large result sets in chunks
• Stream to file instead of memory for >1000 items

Audience Analysis Workflow

Overview

Comprehensive cross-platform audience analysis tool that scrapes follower demographics, engagement patterns, and content performance across Instagram, Facebook, YouTube, and TikTok. Goes beyond basic metrics to provide actionable insights including quality scores, optimal posting times, and growth trajectories.

Architecture

Input: @username or brand handle
         │
         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    PLATFORM SCRAPERS                            │
├─────────────────────────────────────────────────────────────────┤
│  Instagram     │  Facebook      │  YouTube       │  TikTok      │
│  - Profile     │  - Page        │  - Channel     │  - Profile   │
│  - Posts       │  - Posts       │  - Videos      │  - Videos    │
│  - Reels       │  - Reviews     │  - Comments    │  - Comments  │
│  - Comments    │  - Engagement  │  - Subscribers │  - Followers │
└─────────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    ANALYSIS ENGINE                              │
├─────────────────────────────────────────────────────────────────┤
│  Engagement Rate Calculation      │  Audience Quality Score    │
│  Best Posting Times Analysis      │  Content Performance Matrix│
│  Industry Benchmarking            │  Growth Trajectory         │
│  Cross-Platform Overlap           │  Sentiment Analysis        │
└─────────────────────────────────────────────────────────────────┘
         │
         ▼
Output: Comprehensive Audience Report (JSON/HTML/CSV)

Apify Actors Used

Key Metrics Calculated

1. Engagement Rate

Platform-specific engagement rate formulas:

• Instagram: (likes + comments) / followers * 100
• Facebook: (reactions + comments + shares) / followers * 100
• YouTube: (likes + comments) / views * 100
• TikTok: (likes + comments + shares) / views * 100

2. Audience Quality Score (A-F)

Based on engagement-to-follower ratio compared to industry benchmarks:

3. Industry Benchmarks

Pre-loaded benchmarks for common industries:

4. Best Posting Times

Analyzes post timestamps and engagement to determine optimal posting windows:

{
  "weekday_best": "Wednesday",
  "weekend_best": "Saturday",
  "best_hours": ["9:00 AM", "12:00 PM", "7:00 PM"],
  "timezone": "UTC",
  "confidence": 0.85
}

5. Content Performance Matrix

Categorizes content types by performance:

6. Growth Trajectory

Analyzes follower growth patterns:

• Accelerating: Growth rate increasing month-over-month
• Stable: Consistent growth rate
• Decelerating: Growth rate slowing
• Declining: Losing followers
• Viral: Sudden spike in growth

CLI Usage

Single Account Analysis

# Basic analysis on all platforms
python scripts/analyze_audience.py @nike

# Specify platforms
python scripts/analyze_audience.py @nike --platforms instagram facebook youtube

# With industry benchmark
python scripts/analyze_audience.py @nike --benchmark sportswear

# Custom output format
python scripts/analyze_audience.py @nike --format html --output nike_report.html

Multi-Account Comparison

# Compare competitors
python scripts/analyze_audience.py @nike @adidas @puma --compare

# Compare with CSV output
python scripts/analyze_audience.py @nike @adidas --compare --format csv --output comparison.csv

Advanced Options

# Full analysis with all enrichments
python scripts/analyze_audience.py @brand \
  --platforms instagram facebook youtube tiktok \
  --benchmark fashion \
  --max-posts 100 \
  --include-comments \
  --format html \
  --output full_report.html

# Quick summary (profile only, no posts)
python scripts/analyze_audience.py @brand --quick

# JSON for programmatic use
python scripts/analyze_audience.py @brand --format json --output data.json

Output Formats

JSON (Default)

{
  "analysis_timestamp": "2025-01-31T10:30:00Z",
  "account": "@nike",
  "platforms": {
    "instagram": {
      "username": "nike",
      "followers": 250000000,
      "following": 150,
      "posts_count": 2500,
      "engagement_rate": 1.2,
      "quality_score": "B",
      "follower_following_ratio": 1666666.7,
      "avg_likes_per_post": 2500000,
      "avg_comments_per_post": 15000,
      "best_posting_times": {
        "weekday": "Wednesday",
        "hours": ["9:00 AM", "6:00 PM"]
      },
      "top_content_types": [
        {"type": "Reels", "engagement": 1.8},
        {"type": "Carousel", "engagement": 1.3},
        {"type": "Single Image", "engagement": 0.9}
      ],
      "recent_growth": {
        "monthly_rate": 0.5,
        "trajectory": "stable"
      },
      "top_hashtags": ["#nike", "#justdoit", "#sports"],
      "sentiment_score": 0.72
    },
    "facebook": { ... },
    "youtube": { ... },
    "tiktok": { ... }
  },
  "cross_platform_summary": {
    "total_reach": 520000000,
    "avg_engagement_rate": 1.35,
    "strongest_platform": "instagram",
    "weakest_platform": "facebook",
    "overall_quality_score": "B+",
    "growth_trend": "stable",
    "estimated_audience_overlap": 0.25
  },
  "benchmark_comparison": {
    "industry": "sportswear",
    "engagement_vs_benchmark": "+15%",
    "growth_vs_benchmark": "-5%",
    "position": "above average"
  },
  "recommendations": [
    "Increase TikTok posting frequency to match Instagram engagement",
    "YouTube engagement below benchmark - consider more community posts",
    "Reels performing 50% better than static posts - prioritize video content",
    "Best posting time is Wednesday 9AM - consider scheduling around this"
  ]
}

HTML Report

Generates a styled HTML report with:

• Executive summary dashboard
• Platform-by-platform breakdown with charts
• Engagement timeline visualization
• Content performance heatmap
• Competitive comparison (if multiple accounts)
• Actionable recommendations

CSV Export

For spreadsheet analysis:

account,platform,followers,engagement_rate,quality_score,growth_rate
nike,instagram,250000000,1.2,B,0.5
nike,facebook,35000000,0.8,C,0.2
nike,youtube,1500000,2.1,B+,0.8

Cost Estimates

Integration with Other Skills

With parallel-research

# Deep research on brand + audience analysis
python scripts/analyze_audience.py @brand --output .tmp/audience.json
python parallel-research/scripts/research.py "Analyze {brand} marketing strategy" --context .tmp/audience.json

With content-generation

# Generate social media strategy based on analysis
python scripts/analyze_audience.py @client --output .tmp/analysis.json
python content-generation/scripts/generate.py "social media strategy" --data .tmp/analysis.json

With google-workspace

# Save report to Google Drive
python scripts/analyze_audience.py @brand --format html --output .tmp/report.html
python google-workspace/scripts/upload.py .tmp/report.html --folder "Client Reports"

Error Handling

Environment Variables

# Required
APIFY_TOKEN=your_apify_token

# Optional
AUDIENCE_ANALYSIS_MAX_POSTS=50
AUDIENCE_ANALYSIS_TIMEOUT=300

Related Workflows

• competitor-intel.md - Full competitor analysis
• influencer-discovery.md - Find influencers by criteria
• lead-generation.md - Extract leads from social

Changelog

• v1.0 - Initial release with 4-platform support
• v1.1 - Added industry benchmarks and quality scores
• v1.2 - Added HTML report generation
• v1.3 - Added multi-account comparison mode

Competitor Ads Intelligence Workflow

Overview

Comprehensive competitor advertising intelligence across Facebook/Meta Ads Library and Google Ads Transparency Center. This workflow extracts, analyzes, and benchmarks competitor advertising strategies with rich analytics and comparison capabilities.

What This Does:

• Scrape competitor ads from Meta Ads Library (Facebook, Instagram, Messenger, Audience Network)
• Extract Google Ads Transparency Center data (Search, YouTube, Display)
• Generate spend estimates and performance insights
• Compare multiple competitors side-by-side
• Extract messaging themes and creative strategies

Why This is Enriched (Beyond Basic Apify Skills):

• Aggregated summary analytics across all scraped ads
• Multi-competitor benchmarking with relative rankings
• Creative type distribution analysis
• Messaging theme extraction and categorization
• Platform focus comparison
• Historical trend insights

Workflow Architecture

                    +-----------------+
                    |   CLI Input     |
                    | (competitors,   |
                    |  platforms,     |
                    |  options)       |
                    +--------+--------+
                             |
            +----------------+----------------+
            |                                 |
    +-------v-------+               +---------v---------+
    |  Facebook Ads |               |  Google Ads       |
    |  Library      |               |  Transparency     |
    |  Scraper      |               |  Center Scraper   |
    +-------+-------+               +---------+---------+
            |                                 |
            +----------------+----------------+
                             |
                    +--------v--------+
                    |   Data Fusion   |
                    |   & Enrichment  |
                    +--------+--------+
                             |
                    +--------v--------+
                    |   Analytics     |
                    |   Engine        |
                    +--------+--------+
                             |
            +----------------+----------------+
            |                |                |
    +-------v-------+ +------v------+ +-------v-------+
    |  Summary      | | Comparison  | |   Trends      |
    |  Statistics   | | Matrix      | |   Analysis    |
    +---------------+ +-------------+ +---------------+
                             |
                    +--------v--------+
                    |   JSON Output   |
                    |   with Rich     |
                    |   Analytics     |
                    +--------+--------+

Apify Actors Used

Input Parameters

Competitor Targeting

Filtering Options

Output Options

CLI Usage Examples

Basic Usage

# Single competitor analysis
python scripts/scrape_competitor_ads.py "Nike" --platforms facebook google --country US --days 30

# Multiple competitor names
python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --platforms facebook --max-ads 200

# Using domain names
python scripts/scrape_competitor_ads.py "nike.com" "adidas.com" --platforms google

Comparison Mode

# Side-by-side comparison of competitors
python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --compare --output comparison.json

# Full benchmark with all metrics
python scripts/scrape_competitor_ads.py "Shopify" "BigCommerce" "WooCommerce" \
  --compare \
  --platforms facebook google \
  --days 60 \
  --output ecommerce_comparison.json

Keyword Discovery

# Find advertisers by keyword/topic
python scripts/scrape_competitor_ads.py --search "running shoes" --country US --max-ads 200

# Discover competitors in a niche
python scripts/scrape_competitor_ads.py --search "AI writing assistant" --platforms facebook google --days 14

# Industry-specific discovery
python scripts/scrape_competitor_ads.py --search "CRM software small business" --country GB

Advanced Filtering

# Video ads only from active campaigns
python scripts/scrape_competitor_ads.py "Netflix" "Disney+" "HBO Max" \
  --media-types video \
  --status active \
  --days 7

# High-spend advertisers only
python scripts/scrape_competitor_ads.py "Salesforce" "HubSpot" "Zoho" \
  --min-spend 10000 \
  --platforms facebook

# Non-English markets
python scripts/scrape_competitor_ads.py "Mercado Libre" "Amazon" \
  --country MX \
  --languages es

Output Structure

Full Output Schema

{
  "metadata": {
    "scraped_at": "2024-01-15T10:30:00Z",
    "competitors_analyzed": ["Nike", "Adidas", "Puma"],
    "platforms": ["facebook", "google"],
    "country": "US",
    "date_range": {
      "start": "2023-12-15",
      "end": "2024-01-15"
    },
    "filters_applied": {
      "status": "active",
      "media_types": "all",
      "exclude_political": true
    }
  },

  "competitors": [
    {
      "name": "Nike",
      "identifier": "nike",
      "facebook_page_id": "123456789",
      "google_advertiser_id": "AR123456789",
      "platforms": {
        "facebook": {
          "total_ads": 145,
          "active_ads": 98,
          "inactive_ads": 47,
          "spend_estimate": {
            "lower_bound_usd": 50000,
            "upper_bound_usd": 250000,
            "currency": "USD"
          },
          "reach_estimate": {
            "lower_bound": 5000000,
            "upper_bound": 25000000
          },
          "platform_distribution": {
            "facebook": 60,
            "instagram": 45,
            "messenger": 15,
            "audience_network": 25
          },
          "media_breakdown": {
            "image": 45,
            "video": 55,
            "carousel": 30,
            "collection": 15
          },
          "ads": [
            {
              "id": "ad_123456",
              "status": "active",
              "start_date": "2024-01-01",
              "end_date": null,
              "creative": {
                "type": "video",
                "headline": "Just Do It",
                "body_text": "Explore the new Nike Air Max collection...",
                "call_to_action": "Shop Now",
                "landing_url": "https://nike.com/air-max",
                "media_urls": ["https://..."]
              },
              "targeting": {
                "age_range": "18-54",
                "gender": "all",
                "locations": ["United States"],
                "interests": ["Running", "Fitness", "Sports"]
              },
              "impressions_estimate": {
                "lower_bound": 100000,
                "upper_bound": 500000
              },
              "spend_estimate": {
                "lower_bound_usd": 5000,
                "upper_bound_usd": 25000
              }
            }
          ]
        },
        "google": {
          "total_ads": 78,
          "formats": {
            "text": 45,
            "image": 20,
            "video": 13
          },
          "regions_targeted": ["United States", "Canada"],
          "ads": [...]
        }
      }
    }
  ],

  "summary": {
    "total_ads_analyzed": 523,
    "total_competitors": 3,
    "total_spend_estimate": {
      "lower_bound_usd": 150000,
      "upper_bound_usd": 750000
    },
    "platform_distribution": {
      "facebook": 245,
      "instagram": 120,
      "google_search": 80,
      "google_display": 45,
      "youtube": 33
    },
    "creative_breakdown": {
      "video": 180,
      "image": 220,
      "carousel": 75,
      "text": 48
    },
    "status_breakdown": {
      "active": 380,
      "inactive": 143
    },
    "top_messaging_themes": [
      {
        "theme": "Performance/Speed",
        "frequency": 45,
        "keywords": ["fast", "performance", "speed", "run"]
      },
      {
        "theme": "Innovation/Technology",
        "frequency": 38,
        "keywords": ["new", "technology", "innovation", "advanced"]
      },
      {
        "theme": "Lifestyle/Culture",
        "frequency": 32,
        "keywords": ["style", "culture", "life", "everyday"]
      }
    ],
    "top_call_to_actions": [
      {"cta": "Shop Now", "count": 180},
      {"cta": "Learn More", "count": 95},
      {"cta": "Sign Up", "count": 45}
    ],
    "avg_ad_duration_days": 21,
    "ads_per_competitor_avg": 174.3
  },

  "comparison": {
    "spend_ranking": [
      {"competitor": "Nike", "spend_estimate_mid": 150000, "rank": 1},
      {"competitor": "Adidas", "spend_estimate_mid": 120000, "rank": 2},
      {"competitor": "Puma", "spend_estimate_mid": 45000, "rank": 3}
    ],
    "activity_ranking": [
      {"competitor": "Nike", "total_ads": 223, "active_rate": 0.68, "rank": 1},
      {"competitor": "Adidas", "total_ads": 185, "active_rate": 0.72, "rank": 2},
      {"competitor": "Puma", "total_ads": 115, "active_rate": 0.61, "rank": 3}
    ],
    "platform_focus": {
      "Nike": {"primary": "facebook", "secondary": "instagram", "facebook_pct": 0.45},
      "Adidas": {"primary": "instagram", "secondary": "facebook", "instagram_pct": 0.52},
      "Puma": {"primary": "facebook", "secondary": "google", "facebook_pct": 0.55}
    },
    "creative_strategy": {
      "Nike": {"primary_format": "video", "video_pct": 0.55},
      "Adidas": {"primary_format": "carousel", "carousel_pct": 0.48},
      "Puma": {"primary_format": "image", "image_pct": 0.62}
    },
    "messaging_overlap": {
      "shared_themes": ["performance", "style", "innovation"],
      "unique_to_nike": ["athlete endorsement", "competition"],
      "unique_to_adidas": ["sustainability", "collaboration"],
      "unique_to_puma": ["affordability", "streetwear"]
    }
  },

  "trends": {
    "ad_volume_by_week": [
      {"week": "2023-W51", "ads_launched": 45},
      {"week": "2024-W01", "ads_launched": 78},
      {"week": "2024-W02", "ads_launched": 62}
    ],
    "spend_trend": "increasing",
    "format_trend": "video_growing",
    "platform_shift": "instagram_focus_increasing"
  }
}

Workflow Steps

Step 1: Configure Analysis Parameters

# Define your competitive set
COMPETITORS=("Nike" "Adidas" "Puma" "Under Armour" "New Balance")

# Set analysis parameters
COUNTRY="US"
PLATFORMS="facebook google"
DAYS=30
MAX_ADS=200

Step 2: Run Competitor Ads Scraper

# Execute the scraper
python scripts/scrape_competitor_ads.py \
  "${COMPETITORS[@]}" \
  --platforms $PLATFORMS \
  --country $COUNTRY \
  --days $DAYS \
  --max-ads $MAX_ADS \
  --compare \
  --output competitor_ads_analysis.json

Step 3: Review Summary Statistics

The script outputs key metrics to console:

=== Competitor Ads Analysis Complete ===

Total Ads Analyzed: 523
Competitors: 5
Platforms: Facebook, Google

Spend Ranking:
  1. Nike - $100K-$250K
  2. Adidas - $75K-$200K
  3. Under Armour - $50K-$150K
  4. New Balance - $40K-$120K
  5. Puma - $30K-$100K

Top Creative Formats:
  - Video: 38%
  - Image: 35%
  - Carousel: 18%
  - Text: 9%

Results saved to: .tmp/competitor_ads_analysis.json

Step 4: Deep Dive Analysis (Optional)

# Focus on specific competitor with more details
python scripts/scrape_competitor_ads.py "Nike" \
  --platforms facebook \
  --max-ads 500 \
  --include-creatives \
  --output nike_deep_dive.json

# Keyword-based discovery to find new competitors
python scripts/scrape_competitor_ads.py \
  --search "athletic footwear" \
  --country US \
  --max-ads 300 \
  --output athletic_footwear_landscape.json

Step 5: Export and Share

# Upload results to Google Drive
python ~/.claude/skills/google-workspace/scripts/upload_to_drive.py \
  .tmp/competitor_ads_analysis.json \
  --folder "Competitive Intelligence"

# Generate report with parallel-research
python ~/.claude/skills/parallel-research/scripts/parallel_research.py \
  chat "Analyze this competitor advertising data and provide strategic recommendations: $(cat .tmp/competitor_ads_analysis.json | head -1000)"

Analytics Features (Enriched)

1. Spend Estimation

The script aggregates spend estimates from Meta Ads Library (which provides lower/upper bounds) and normalizes across competitors:

• Lower Bound: Conservative minimum spend estimate
• Upper Bound: Maximum possible spend estimate
• Midpoint: Calculated average for ranking
• Relative Spend: Percentage of total competitive spend

2. Creative Strategy Analysis

Automatic categorization of creative approaches:

• Format Distribution: Video vs. image vs. carousel vs. text
• Primary Format: Most-used creative type per competitor
• Format Trend: Whether video/carousel usage is increasing
• A/B Testing Signals: Multiple creatives with similar messaging

3. Messaging Theme Extraction

Natural language analysis of ad copy to identify:

• Top Themes: Most common messaging categories
• Theme Keywords: Words associated with each theme
• Unique Positioning: Themes used by only one competitor
• Shared Territory: Common messaging across all competitors

4. Platform Strategy

Cross-platform analysis including:

• Primary Platform: Where competitor focuses most spend
• Platform Mix: Distribution across FB/IG/Google/YouTube
• Platform Shift: Changes in platform focus over time
• Cross-Platform Consistency: Same ads across platforms

5. Competitive Benchmarking

Comparison metrics when --compare flag is used:

• Spend Ranking: Competitors ranked by estimated spend
• Activity Ranking: By total number of ads
• Active Rate: Percentage of ads currently running
• Creative Innovation Score: Variety of formats/messaging

Cost Estimates

Cost Optimization Tips:

• Use --max-ads to limit per-competitor extraction
• Focus on --status active to skip inactive ads
• Use single platform when cross-platform isn't needed
• Cache results and only refresh weekly

Error Handling

Integration Patterns

Pattern 1: Weekly Competitive Report

#!/bin/bash
# weekly_competitor_report.sh

# Scrape competitor ads
python scripts/scrape_competitor_ads.py \
  "Competitor1" "Competitor2" "Competitor3" \
  --compare \
  --platforms facebook google \
  --days 7 \
  --output weekly_ads.json

# Generate report with AI
python ~/.claude/skills/parallel-research/scripts/parallel_research.py \
  chat "Create a weekly competitive advertising report from: $(cat .tmp/weekly_ads.json)"

# Upload to Drive
python ~/.claude/skills/google-workspace/scripts/upload_to_drive.py \
  .tmp/weekly_ads.json \
  --folder "Weekly Reports"

Pattern 2: New Competitor Discovery

# Find who's advertising in your space
python scripts/scrape_competitor_ads.py \
  --search "your product category" \
  --country US \
  --max-ads 500 \
  --output landscape.json

# Extract unique advertisers
cat .tmp/landscape.json | jq '.summary.unique_advertisers'

Pattern 3: Campaign Launch Monitoring

# Before your campaign launch, benchmark competitors
python scripts/scrape_competitor_ads.py \
  "MainCompetitor" \
  --platforms facebook \
  --days 1 \
  --status active \
  --output pre_launch_baseline.json

# After launch, compare activity changes
python scripts/scrape_competitor_ads.py \
  "MainCompetitor" \
  --platforms facebook \
  --days 1 \
  --status active \
  --output post_launch_snapshot.json

Related Workflows

Security Notes

• Ad data is public (from official transparency libraries)
• No authentication required for Meta Ads Library
• No authentication required for Google Ads Transparency
• Respect rate limits to avoid IP blocks
• Store results locally (not transmitted to third parties)

Changelog

• v1.0.0 - Initial release with Facebook and Google support
• Includes summary analytics, comparison mode, and theme extraction
• Enriched beyond basic Apify skill with comprehensive benchmarking

Competitor Intelligence Workflow

Overview

Gather comprehensive intelligence on competitors across social media, reviews, and web presence.

Workflow Steps

Step 1: Map Competitors
├── Google Maps search for industry
├── Identify top players
└── Get website URLs

Step 2: Social Media Analysis
├── Facebook page metrics and posts
├── Instagram presence and engagement
├── LinkedIn company posts
├── Twitter/X activity

Step 3: Review Analysis
├── Google Maps reviews (sentiment, themes)
├── Facebook reviews
└── Identify strengths/weaknesses

Step 4: Content Analysis
├── Scrape competitor websites
├── Extract key messaging
└── Compare offerings

Step 5: Synthesize Report
├── Use parallel-research for analysis
├── Generate competitive report
└── Save to Google Drive

Example Commands

# Step 1: Find competitors in your space
python scripts/scrape_google_maps.py search "digital marketing agency" --location "Austin, TX" --max-results 20 --output competitors.json

# Step 2a: Facebook presence
python scripts/scrape_facebook.py page "https://facebook.com/competitor1" --output fb_competitor1.json
python scripts/scrape_facebook.py posts "https://facebook.com/competitor1" --max-posts 50 --output fb_posts.json

# Step 2b: Instagram presence
python scripts/scrape_instagram.py profile competitor1 competitor2 --output ig_profiles.json
python scripts/scrape_instagram.py posts competitor1 --max-posts 50 --output ig_posts.json

# Step 3: Reviews analysis
python scripts/scrape_google_maps.py reviews "https://maps.google.com/place/competitor1" --max-reviews 100 --output gmaps_reviews.json
python scripts/scrape_facebook.py reviews "https://facebook.com/competitor1" --max-reviews 50 --output fb_reviews.json

# Step 4: Website content
python scripts/scrape_multi_platform.py website --urls "https://competitor1.com" --max-pages 20 --output website_content.json

Metrics to Compare

Cost Estimate

• 5 competitors full analysis: ~$10-15
• 20 competitors basic scan: ~$5-10

Output: Competitive Matrix

Related Skills

• parallel-research - Deep company research
• content-generation - Generate competitive report
• google-workspace - Save analysis to Drive

Influencer Discovery Workflow

Overview

Comprehensive multi-platform influencer discovery system with scoring algorithms, tier classification, fake follower detection, brand safety analysis, and contact extraction. Find and evaluate influencers across Instagram, TikTok, YouTube, and Twitter/X for partnerships, collaborations, or outreach campaigns.

Key Features

• Multi-Platform Discovery: Instagram, TikTok, YouTube, Twitter/X
• Influencer Scoring Algorithm: Weighted scoring based on 5 factors
• Tier Classification: Nano to Mega influencer categorization
• Fake Follower Detection: Heuristic-based authenticity scoring
• Brand Safety Analysis: Content sentiment and risk flagging
• Contact Extraction: Email, website, and link-in-bio extraction
• Cost Estimation: Estimated post rates and CPM by tier

---

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                      INFLUENCER DISCOVERY WORKFLOW                   │
└─────────────────────────────────────────────────────────────────────┘
                                    │
     ┌──────────────────────────────┼──────────────────────────────┐
     ▼                              ▼                              ▼
┌─────────────┐            ┌─────────────┐            ┌─────────────┐
│  DISCOVERY  │            │  ENRICHMENT │            │   OUTPUT    │
│   SOURCES   │            │   ENGINES   │            │   FORMATS   │
├─────────────┤            ├─────────────┤            ├─────────────┤
│ Hashtags    │───────────▶│ Scoring     │───────────▶│ JSON        │
│ Keywords    │            │ Authenticity│            │ CSV         │
│ Competitors │            │ Brand Safety│            │ Sheets      │
│ Similar Acc │            │ Contacts    │            │ CRM         │
│ Location    │            │ Categories  │            │             │
└─────────────┘            └─────────────┘            └─────────────┘

---

Workflow Steps

Step 1: Configure Discovery Parameters
├── Choose platform(s): instagram, tiktok, youtube, twitter
├── Set discovery method: hashtag, keyword, competitor, similar
├── Define filters: tier, engagement, followers, location
└── Set result limits and output format

Step 2: Scrape Platform Data
├── Instagram: hashtag posts → extract creator handles
├── TikTok: hashtag videos → extract author profiles
├── YouTube: search results → extract channel data
└── Twitter: search tweets → extract user profiles

Step 3: Extract Unique Influencers
├── Deduplicate creators from posts
├── Aggregate engagement metrics
├── Calculate averages per creator
└── Build initial influencer profiles

Step 4: Enrich & Score Influencers
├── Calculate engagement rate
├── Estimate authenticity score
├── Analyze brand safety
├── Extract contact information
├── Categorize content types
├── Calculate overall influencer score
└── Classify tier and estimate rates

Step 5: Filter & Rank Results
├── Apply tier filters
├── Apply engagement thresholds
├── Apply follower range filters
├── Sort by influencer score
└── Generate distribution analytics

Step 6: Export for Outreach
├── JSON: Full data for processing
├── CSV: Spreadsheet-ready format
├── Google Sheets: Direct import
└── CRM: Attio/HubSpot ready

---

Influencer Scoring Algorithm

Weighted Scoring Model

Score Calculation

# Engagement normalized (5% = perfect 100)
engagement_score = min(100, (engagement_rate / 5) * 100)

# Posting normalized (7 posts/week = 100)
posting_score = min(100, (posts_per_week / 7) * 100)

# Final weighted score
influencer_score = (
    engagement_score * 0.30 +
    authenticity_score * 0.25 +
    relevance_score * 0.20 +
    posting_score * 0.15 +
    growth_score * 0.10
)

Score Interpretation

---

Tier Classification

CPM Estimates by Platform

---

Fake Follower Detection

Authenticity Heuristics

HIGH AUTHENTICITY (80-100%):
├── Engagement rate matches tier expectations
├── Diverse commenters (many unique users)
├── Thoughtful comments (avg length > 50 chars)
└── Consistent engagement across posts

MEDIUM AUTHENTICITY (50-79%):
├── Slightly below-expected engagement
├── Some comment diversity
└── Mixed comment quality

LOW AUTHENTICITY (0-49%):
├── Very low engagement for follower count
├── Same users commenting repeatedly
├── Short, generic comments ("Nice!", "Amazing!")
├── Sudden follower spikes
└── Engagement/follower ratio anomalies

Detection Algorithm

# Expected engagement by tier
expected = {"nano": 5.0, "micro": 3.0, "mid": 2.0, "macro": 1.5, "mega": 1.0}

# Penalize low engagement
if engagement < expected * 0.3:
    score -= 40  # Major red flag
elif engagement < expected * 0.5:
    score -= 25

# Penalize suspiciously high engagement
if engagement > expected * 5:
    score -= 30  # Possible fake engagement

# Bonus for quality comments
if avg_comment_length > 50:
    score += 5

# Bonus for diverse commenters
if unique_commenters / total_comments > 0.8:
    score += 10

---

Brand Safety Analysis

Risk Levels

Flagged Categories

Red Flags (High Risk):

• Controversial topics
• Political content
• NSFW/Adult content
• Gambling
• Violence
• Drug/Alcohol promotion
• Tobacco
• Weapons

Yellow Flags (Moderate Risk):

• Strong opinions
• Rant content
• Debate/Drama
• Call-out content
• "Beef" between creators

Analysis Output

{
  "status": "yellow",
  "description": "Some potentially risky content",
  "red_flags": 1,
  "yellow_flags": 2,
  "flagged_terms": [
    {"term": "controversial", "severity": "red"},
    {"term": "opinion", "severity": "yellow"}
  ]
}

---

Contact Extraction

Extracted Data Points

Email Extraction Patterns

# Standard email
[\w.+-]+@[\w-]+\.[\w.-]+

# Business email patterns
business[\s:]+([^\s]+@[\w-]+\.[\w.-]+)
collab[\s:]+([^\s]+@[\w-]+\.[\w.-]+)
inquiry[\s:]+([^\s]+@[\w-]+\.[\w.-]+)
contact[\s:]+([^\s]+@[\w-]+\.[\w.-]+)

---

CLI Commands

Basic Discovery

# Discover by hashtag on Instagram
python scripts/discover_influencers.py --hashtags fitness health --platform instagram

# Discover by topic (auto-converts to hashtags)
python scripts/discover_influencers.py --topic "sustainable fashion" --platform instagram

# Multi-platform discovery
python scripts/discover_influencers.py --topic "AI" --platforms instagram tiktok youtube

Filtered Discovery

# Filter by tier
python scripts/discover_influencers.py --hashtags tech --tier micro mid

# Filter by engagement rate
python scripts/discover_influencers.py --hashtags beauty --min-engagement 3.0

# Filter by follower count
python scripts/discover_influencers.py --hashtags travel --min-followers 10000 --max-followers 100000

# Combined filters
python scripts/discover_influencers.py \
  --hashtags fitness health nutrition \
  --platform instagram \
  --tier micro \
  --min-engagement 3.0 \
  --min-followers 10000 \
  --max-results 200

Enriched Discovery

# Fetch detailed profile data (slower, more accurate)
python scripts/discover_influencers.py --hashtags beauty --platform instagram --fetch-profiles

# Extract contact information
python scripts/discover_influencers.py --hashtags tech --extract-contacts --output contacts.json

Export Options

# Export as JSON (default)
python scripts/discover_influencers.py --topic "AI" --output influencers.json

# Export as CSV for spreadsheets
python scripts/discover_influencers.py --topic "AI" --format csv --output outreach_list.csv

# Full outreach list with contacts
python scripts/discover_influencers.py \
  --hashtags tech AI startup \
  --tier micro mid \
  --min-engagement 2.0 \
  --extract-contacts \
  --format csv \
  --output tech_influencers_outreach.csv

---

Output Format

JSON Output Structure

{
  "query": {
    "hashtags": ["fitness", "health"],
    "platforms": ["instagram"],
    "tiers": ["micro"],
    "min_engagement": 3.0,
    "min_followers": 10000
  },
  "discovered_at": "2024-01-15T10:30:00",
  "total_found": 150,
  "influencers": [
    {
      "username": "@fitnessguru",
      "platform": "instagram",
      "full_name": "John Fitness",
      "followers": 85000,
      "engagement_rate": 4.2,
      "influencer_score": 87,
      "tier": "micro",
      "tier_label": "Micro (10K-100K)",
      "authenticity_score": 92,
      "verified": false,
      "estimated_post_rate": "$250-1.5K",
      "cpm_estimate": "$5-15",
      "contact": {
        "email": "business@fitnessguru.com",
        "website": "https://fitnessguru.com",
        "linktree": "https://linktr.ee/fitnessguru"
      },
      "top_hashtags": ["fitness", "workout", "gym", "health", "gains"],
      "content_categories": ["fitness", "health", "lifestyle"],
      "brand_safety": "green",
      "brand_safety_detail": {
        "status": "green",
        "description": "No significant brand safety concerns",
        "red_flags": 0,
        "yellow_flags": 0
      },
      "profile_url": "https://www.instagram.com/fitnessguru/",
      "bio": "Certified PT | Transform your body | Business: business@fitnessguru.com",
      "sample_posts_analyzed": 25,
      "avg_likes": 3500,
      "avg_comments": 125
    }
  ],
  "tier_distribution": {
    "nano": 45,
    "micro": 78,
    "mid": 20,
    "macro": 5,
    "mega": 2
  },
  "platform_breakdown": {
    "instagram": 150
  },
  "scoring_weights": {
    "engagement_rate": 0.30,
    "follower_authenticity": 0.25,
    "content_relevance": 0.20,
    "posting_consistency": 0.15,
    "growth_rate": 0.10
  }
}

CSV Output Columns

---

Platform-Specific Actors

Instagram

TikTok

YouTube

Twitter/X

---

Cost Estimation

Per Discovery Run

Optimization Tips

1. Start narrow: Use specific hashtags, not broad topics 2. Limit results: Start with 50-100, increase if needed 3. Skip profiles initially: Only fetch if you need bio data 4. Use tier filters: Narrow by tier before enriching 5. Batch similar searches: Combine related hashtags

---

Integration Workflows

Outreach Campaign Pipeline

1. Discover influencers
   python scripts/discover_influencers.py --topic "AI" --tier micro mid --format csv

2. Import to Google Sheets (use google-workspace skill)
   # Creates filterable spreadsheet

3. Review and shortlist manually
   # Add "Status" column, filter top candidates

4. Enrich with Parallel Research (use parallel-research skill)
   # Get additional company/personal intel

5. Add to CRM (use attio-crm skill)
   # Create contacts with tags

6. Launch outreach sequence
   # Use email automation or manual outreach

Competitor Follower Mining

1. Find competitor's top posts
   python scripts/scrape_instagram.py posts @competitor --max-posts 50

2. Extract commenters (power users)
   # Parse comments for engaged followers

3. Filter for influencer-level accounts
   python scripts/discover_influencers.py --profiles [list] --min-followers 10000

4. Analyze and score
   # Full enrichment on filtered list

Brand Partnership Qualification

1. Initial discovery with broad filters
   python scripts/discover_influencers.py --hashtags [industry] --max-results 500

2. Apply strict brand safety filter
   # Keep only "green" status

3. Filter by authenticity
   # Keep only 70%+ authenticity

4. Export qualified leads
   python scripts/discover_influencers.py [...] --min-authenticity 70 --format csv

---

Troubleshooting

Common Issues

Rate Limits

• Instagram: ~100 profiles/hour without proxy
• TikTok: ~200 videos/hour
• YouTube: ~100 videos/hour
• Twitter: ~150 tweets/hour

Data Quality

• Engagement rate accuracy: Based on sample posts, may vary
• Authenticity score: Heuristic estimate, not definitive
• Contact extraction: Depends on bio content quality
• Brand safety: Keyword-based, not sentiment analysis

---

Best Practices

For Discovery

1. Use niche hashtags: #veganrecipes > #food 2. Combine related terms: 3-5 hashtags per search 3. Start with micro tier: Best engagement/cost ratio 4. Check recent activity: Filter for active creators 5. Verify content fit: Review actual posts before outreach

For Outreach

1. Personalize messages: Reference specific content 2. Start with email: More professional than DMs 3. Clear value proposition: What's in it for them? 4. Respect rates: Don't lowball based on estimates 5. Build relationships: Long-term > one-off

For Campaigns

1. Mix tiers: Nano for authenticity, Macro for reach 2. Platform match: Choose platform for audience fit 3. Content guidelines: Clear but not restrictive 4. Track performance: Unique codes/links per influencer 5. Measure ROI: CPM, CPA, engagement lift

---

Related Skills

• google-workspace: Export to Google Sheets
• attio-crm: Add influencers as CRM contacts
• parallel-research: Deep research on top candidates
• content-generation: Create outreach templates
• slack-automation: Alert on new discoveries

---

References

• Apify Instagram Scraper
• Apify TikTok Scraper
• Apify YouTube Scraper
• Influencer Marketing Hub - Rate Calculator

Lead Generation Workflow

Overview

Multi-step workflow to discover businesses, extract contact info, and prepare leads for CRM.

Workflow Steps

Step 1: Discover Businesses
├── Google Maps search for target industry
└── Output: List of businesses with basic info

Step 2: Extract Contact Info
├── Run contact enrichment on business websites
└── Output: Emails, phones, social links

Step 3: Enrich with Research (Optional)
├── Use parallel-research to get company info
└── Output: Company size, funding, news

Step 4: Save to CRM or Sheets
├── Use attio-crm or google-workspace
└── Output: Leads in your system

Example Commands

# Step 1: Find coffee shops in SF
python scripts/scrape_google_maps.py search "coffee shops" --location "San Francisco" --max-results 100 --output leads_raw.json

# Step 2: Extract contact info from websites
cat leads_raw.json | jq -r '.data[].website' > websites.txt
python scripts/enrich_contacts.py --from-file websites.txt --output leads_enriched.json

# Step 3: (Optional) Research each company
# Use parallel-research skill

# Step 4: Save to CRM
# Use attio-crm or google-workspace skill

Cost Estimate

Tips

• Start with specific location + industry queries
• Filter by rating to focus on established businesses
• Use contact enrichment only on sites that have websites
• Dedupe emails before importing to CRM

Related Skills

• google-workspace - Save leads to Google Sheets
• attio-crm - Add companies and contacts
• parallel-research - Enrich with company intel