Research Brightdata

Bright Data Research Skill

Advanced web research powered by Bright Data MCP - perform market analysis, competitive intelligence, data extraction, and comprehensive web research with anti-bot protection.

Overview

This skill provides complete workflows for automated web research using Bright Data MCP. Handle search discovery, content collection, structured data extraction, and comprehensive analysis with browser automation support.

When This Skill Applies

Activate this skill when the user's request involves:

• Web scraping and data collection
• Market research and competitive analysis
• Price monitoring and comparison
• Product information extraction
• Search engine result analysis
• Large-scale web data gathering
• Research requiring anti-bot protection

Core Capabilities

Search and Discovery

Use search_engine tool to find relevant sources:

{
  "tool": "search_engine",
  "parameters": {
    "query": "site:etsy.com nba merchandise",
    "engine": "google",
    "cursor": "0"
  }
}

Search strategies:

• Use site operators: "site:etsy.com keywords"
• Use exact phrases: "machine learning in healthcare"
• Exclude terms: "iphone -case -cover"
• Paginate with cursor: "0", "1", "2" for more results

Content Collection

Three collection modes based on research depth:

Quick Mode (3-5 URLs, serial processing):

• Use scrape_as_markdown for each URL
• Best for: Fast overviews, fact-checking

Standard Mode (10-20 URLs, parallel batch):

• Use scrape_batch for up to 10 URLs concurrently
• Best for: Market research, competitive analysis

Deep Mode (20-50 URLs, browser automation):

• Use scraping_browser_navigate for JavaScript-rendered pages
• Use scraping_browser_links to discover page links
• Use scraping_browser_click for interactions
• Best for: Dynamic content, multi-page extraction

Data Extraction

Use extract tool for AI-powered structured data extraction:

{
  "tool": "extract",
  "parameters": {
    "url": "https://example.com/product",
    "extraction_prompt": "Extract: product name, price as number, rating (0-5), number of reviews, seller name, availability status"
  }
}

Common extraction schemas:

• E-commerce: name, price, rating, reviews, seller, availability
• Articles: title, author, date, summary, key points
• Companies: name, industry, founded, headquarters, employee count

Output Formats

Three report formats for different use cases:

Report Format (default):

• Executive summary
• Key findings with evidence
• Detailed analysis
• Methodology and recommendations
• Source references

JSON Format:

• Structured data for API integration
• All raw and processed data
• Metadata and provenance
• Statistical analysis

Markdown Format:

• Clean, readable content
• Tables and lists
• Source links
• Minimal formatting

Research Workflow

Phase 1: Query Analysis

Understand the research intent:

• Scope: How broad/deep should research be?
• Key entities: Products, companies, topics
• Target sources: Which sites/platforms?
• Data needed: What fields to extract?

Phase 2: Source Discovery

Use search_engine to find URLs: 1. Execute initial search 2. Extract URLs from SERP 3. Filter irrelevant domains 4. Paginate if needed 5. Prioritize by relevance

Phase 3: Content Collection

Choose appropriate mode:

• Quick: scrape_as_markdown per URL
• Standard: scrape_batch 10 URLs at once
• Deep: scraping_browser_navigate + browser tools

Handle errors gracefully:

• Retry failed URLs with alternative methods
• Log errors for transparency
• Continue with available data

Phase 4: Data Extraction

Apply extraction schema:

• Use extract with custom prompts
• Validate extracted data
• Handle missing/malformed data
• Ensure data quality

Phase 5: Analysis & Synthesis

Process and analyze:

• Clean and normalize data
• Perform statistical analysis
• Identify patterns and trends
• Cross-reference sources
• Validate findings

Phase 6: Report Generation

Generate output:

• Report: Comprehensive document with all sections
• JSON: Structured data for processing
• Markdown: Clean, readable content

Best Practices

Search Strategy

• Start broad, then narrow down
• Use site operators for targeted searches
• Try multiple search engines if needed
• Set realistic limits (10-20 URLs usually sufficient)

Performance

• Use scrape_batch for parallel processing (10x faster)
• Only use deep mode when necessary (much slower)
• Set appropriate timeouts
• Monitor success rates
• Avoid token limits: Batch 1-2 URLs at a time for large pages (Etsy, Amazon, etc.)

Data Quality

• Always validate extracted data
• Cross-reference multiple sources
• Check for outliers and anomalies
• Normalize formats (dates, currencies, units)

Error Handling

• Implement retry logic
• Have fallback strategies
• Log errors for debugging
• Don't fail on individual URL errors

Ethical Considerations

• Respect robots.txt
• Don't overwhelm servers
• Rate limit requests
• Cite sources properly
• Don't misuse personal data

Common Research Scenarios

E-commerce Market Research

Query: "site:etsy.com nba merchandise"
Mode: standard
Extract: product name, price, rating, reviews, seller
Output: report

Expected: Price analysis, popular products, top sellers

Price Comparison

Query: "iphone 15 pro max 256GB price comparison"
Mode: standard
Extract: retailer, price, availability, shipping
Output: json

Expected: Structured comparison with best deal identified

Academic Research

Query: "machine learning in healthcare 2024 papers"
Mode: standard
Extract: title, authors, date, key findings, methodology
Output: report

Expected: Literature review with trends and insights

Competitive Intelligence

Query: "competitor.com features pricing"
Mode: deep
Extract: feature name, description, pricing tier, availability
Output: report

Expected: Feature comparison, pricing analysis, recommendations

Tool Reference

search_engine

Purpose: Find relevant web pages Parameters: query (required), engine (google/bing/yandex), cursor (page number) Returns: SERP results in markdown

scrape_as_markdown

Purpose: Get clean, AI-ready markdown Parameters: url (required) Returns: Formatted markdown without ads/clutter

scrape_as_html

Purpose: Get raw HTML Parameters: url (required) Returns: Complete HTML document

extract

Purpose: AI-powered structured data extraction Parameters: url (required), extraction_prompt (optional) Returns: JSON object with extracted data

scrape_batch

Purpose: Process multiple URLs in parallel Parameters: urls (array, max 10) Returns: Array of page contents

scraping_browser_navigate

Purpose: Navigate JavaScript-rendered pages Parameters: url (required) Returns: Page info (title, URL, status)

scraping_browser_click

Purpose: Click elements on page Parameters: selector (CSS selector) Returns: Action result

scraping_browser_links

Purpose: Get all links on current page Parameters: None Returns: Array of links with text, href, selector

Troubleshooting

No search results

• Try different search engine (bing, yandex)
• Simplify the query
• Check for typos
• Use broader search terms

Scraping fails

• URL might be JavaScript-rendered → use mode=deep
• URL might be blocked → try alternative URL
• Check if URL is accessible in browser

Extraction incomplete

• Provide more specific extraction prompt
• Check if data exists on page
• Try scraping as markdown first to see content

Slow performance

• Reduce max_results
• Use mode=standard instead of deep
• Check network connectivity
• Close unnecessary browser sessions

Token limit exceeded

• Symptom: "Output exceeds maximum allowed tokens" error
• Cause: Batch scraping too many large pages at once OR reading large files
• Why this limit exists:
• Memory protection: Prevents memory overflow from loading too much content
• Performance optimization: Ensures fast response times
• Context management: Preserves space for other content in the conversation
• System stability: Prevents crashes or errors
• Can this limit be increased?:
• No - This is a hard system limit in Claude Code
• Cannot be changed via configuration files
• Purpose: Protect system stability and performance
• Workarounds:
• For scraping: Reduce batch size to 1-2 URLs for large pages
• For reading files: Use Read with offset and limit to read in chunks
• For specific content: Use Grep to search for specific patterns
• For finding files: Use Glob to find files by pattern

Additional Resources

Reference Files

For detailed workflows and techniques:

• `references/search-discovery.md` - Search strategies and URL discovery
• `references/content-scraping.md` - Content collection methods
• `references/data-extraction.md` - Extraction schemas and validation
• `references/deep-scraping.md` - Browser automation techniques
• `references/analysis-report.md` - Analysis and report generation

Example Files

Complete research examples:

• `examples/market-research-etsy-nba.md` - E-commerce market research
• `examples/competitive-analysis-pricing.md` - Price comparison workflow
• `examples/academic-research-ml-healthcare.md` - Academic literature review

Limitations

• Requires Bright Data MCP server configuration
• Needs valid Bright Data API token
• Subject to API rate limits
• Browser automation is slower than direct scraping
• Some sites may still block access
• Quality depends on source content

Progressive Disclosure

This SKILL.md provides core workflows and quick reference (approximately 2,000 words).

For detailed implementation patterns, advanced techniques, and comprehensive examples, consult the references/ files which load as needed during research tasks.

Example: Academic Research on Machine Learning

Conduct academic research on a specific topic.

Goal

Research recent developments in machine learning for healthcare in 2024.

Command

/research-brightdata \
  query="machine learning in healthcare 2024 research papers medical AI applications" \
  mode=standard \
  max_results=25 \
  sources=["google", "bing"] \
  extract_schema="Extract: paper title, authors, publication date, journal or conference name, research focus area, key findings, methodology, medical application area (diagnosis/treatment/discovery/other), DOI or URL" \
  output_format=report

Expected Workflow

Phase 1: Discovery

Search for academic sources:

Query: machine learning in healthcare 2024
Sources: Google Scholar, research websites, academic institutions
Expected results: 25 relevant papers and articles

Phase 2: Collection

Batch scrape academic pages:

URLs collected: 25
Batch processing: Parallel batches of 10
Time: ~40 seconds

Phase 3: Extraction

Extract structured research data:

{
  "title": "Deep Learning for Early Cancer Detection from Medical Imaging",
  "authors": ["Dr. Sarah Chen", "Dr. Michael Ross"],
  "publication_date": "2024-03-15",
  "journal": "Nature Medicine",
  "focus_area": "Medical Imaging",
  "key_findings": [
    "Achieved 94% accuracy in early-stage detection",
    "Reduced false positives by 40% compared to previous methods",
    "Validated across 3 hospital systems"
  ],
  "methodology": "Convolutional neural networks trained on 50,000 images",
  "application": "diagnosis",
  "url": "https://nature.com/articles/..."
}

Phase 4: Analysis

Synthesize research findings:

• Group by application area
• Identify trends and patterns
• Note leading institutions and researchers
• Find common methodologies

Phase 5: Report

Generate academic-style research report.

Expected Output

# Academic Research Report: Machine Learning in Healthcare (2024)

## Executive Summary

This report synthesizes findings from 25 recent papers on machine learning applications in healthcare published in 2024. Key trends include significant advances in medical imaging diagnostics, drug discovery acceleration, and personalized treatment recommendations. The field shows strong growth in AI-assisted diagnosis, with several systems achieving clinical validation.

## Key Findings

### 1. Medical Imaging Dominates Research (40% of papers)

**Trend**: Deep learning for medical imaging remains the most researched area.

**Notable Developments**:
- Early cancer detection accuracy improved to 94% (Chen et al., Nature Medicine)
- Reduced diagnostic errors in radiology by 35% (AI-RAD collaboration)
- Real-time image analysis during surgeries (Stanford Medicine)

**Impact**: Faster, more accurate diagnoses leading to earlier interventions.

### 2. Drug Discovery Acceleration (25% of papers)

**Trend**: ML dramatically speeding up drug discovery pipelines.

**Key Developments**:
- Protein structure prediction reaching 95% accuracy (DeepMind-MIT)
- AI-generated molecules entering clinical trials
- Reduced drug discovery timeline from 5 years to 18 months

**Impact**: New treatments reaching patients faster, lower R&D costs.

### 3. Personalized Medicine (20% of papers)

**Trend**: Treatment recommendations based on individual patient data.

**Advances**:
- AI predicting patient responses to chemotherapy with 89% accuracy
- Customized treatment plans reducing side effects by 50%
- Integration of genomic data with clinical decision systems

### 4. Administrative Applications (15% of papers)

**Trend**: AI optimizing healthcare operations and resource allocation.

**Applications**:
- Patient no-show prediction (85% accuracy)
- Resource optimization reducing wait times by 30%
- Automated medical coding and billing

## Leading Research Institutions

1. **Stanford Medicine** - 4 papers (medical imaging, personalized medicine)
2. **MIT CSAIL** - 3 papers (drug discovery, diagnostics)
3. **Google Health** - 3 papers (imaging, screening)
4. **Mayo Clinic** - 2 papers (clinical applications)
5. **Johns Hopkins** - 2 papers (surgery assistance, diagnostics)

## Common Methodologies

1. **Convolutional Neural Networks** - 60% of imaging papers
2. **Transformer Models** - 40% of text/EMR analysis papers
3. **Reinforcement Learning** - 25% of treatment optimization papers
4. **Ensemble Methods** - 30% of papers combine multiple approaches

## Validation and Deployment

### Clinical Validation Status
- **Fully Validated**: 35% of systems (tested in clinical settings)
- **Retrospective Validation**: 45% (tested on historical data)
- **Pre-clinical**: 20% (still in development)

### Deployment Status
- **In Clinical Use**: 8 systems
- **In Clinical Trials**: 12 systems
- **Research Prototype**: 15 systems

## Challenges and Limitations

### Technical Challenges
- Data quality and standardization issues
- Model interpretability ("black box" problem)
- Generalization across different patient populations

### Regulatory and Ethical Challenges
- FDA approval processes lagging behind technology
- Patient data privacy concerns
- Liability and accountability questions
- Bias in training data leading to health disparities

### Resource Challenges
- High computational requirements
- Need for large, labeled datasets
- Integration with existing healthcare systems
- Training for medical professionals

## Future Directions

Based on current research trends:

### Near Term (2024-2025)
- Expanded clinical validation studies
- Improved model interpretability
- Better integration with electronic health records
- Standardization of evaluation metrics

### Medium Term (2025-2027)
- Regulatory frameworks catching up
- Multi-modal AI systems (imaging + genomics + clinical data)
- Edge deployment for real-time applications
- Increased focus on fairness and bias reduction

### Long Term (2027+)
- AI as standard of care in diagnostics
- Personalized treatment plans for all patients
- Drug discovery timelines reduced to <1 year
- Autonomous surgical systems

## Recommendations for Researchers

1. **Focus on Interpretability**: Make models explainable to clinicians
2. **Real-World Validation**: Test in diverse clinical settings
3. **Address Bias**: Actively work to reduce health disparities
4. **Collaborate**: Partner with healthcare providers early
5. **Open Data**: Share datasets and models when possible

## Notable Papers of 2024

1. **"Deep Learning for Early Cancer Detection"** - Chen et al., Nature Medicine
   - 94% accuracy in early-stage detection
   - Validated across 3 hospital systems

2. **"AI-Driven Drug Discovery: A 40% Speed Improvement"** - MIT CSAIL
   - New molecular discovery pipeline
   - 2 drug candidates in clinical trials

3. **"Personalized Chemotherapy Response Prediction"** - Stanford Medicine
   - 89% accuracy in predicting patient responses
   - 50% reduction in adverse effects

4. **"Real-Time Surgical Assistance System"** - Johns Hopkins
   - Live guidance during surgeries
   - 30% reduction in surgical complications

## Methodology

This research was conducted using:
- **Search Engines**: Google, Google Scholar, Bing
- **Sources**: 25 papers from peer-reviewed journals and conferences
- **Timeframe**: January - December 2024
- **Quality Filter**: Only peer-reviewed or pre-print with institutional affiliation
- **Data Extraction**: Structured extraction of title, authors, findings, methods
- **Analysis**: Thematic analysis and trend identification

**Limitations**:
- May not capture all relevant research
- Publication bias toward positive results
- Fast-moving field, some recent work may be missed
- Non-English publications underrepresented

## Sources

[Complete list of 25 papers with URLs and DOIs]

---

**Report Generated**: 2024-01-22
**Research Period**: January 2024 - present
**Total Papers Analyzed**: 25
**Confidence Level**: High

Variations

Specific Application Focus

/research-brightdata \
  query="AI in radiology deep learning medical imaging 2024" \
  mode=standard \
  max_results=20

Literature Review

/research-brightdata \
  query="systematic review machine learning healthcare applications" \
  extract_schema="Extract: review scope, number of papers reviewed, main conclusions, gaps identified, future work" \
  mode=standard \
  max_results=15

Researcher-Specific

/research-brightdata \
  query="Andrew Ng machine learning healthcare papers 2024" \
  mode=quick \
  max_results=10

Tips for Academic Research

1. Use Scholar Sources: Include Google Scholar in sources 2. Specific Queries: Use academic terminology and exact phrases 3. Check Publication Dates: Verify currency of information 4. Cross-Reference: Validate findings across multiple sources 5. Track Citations: Look for highly cited papers 6. Institutional Sources: Prioritize university and research institution websites 7. Peer-Reviewed: Focus on peer-reviewed journals and conferences

Quality Indicators

High Quality:

• Peer-reviewed journals (Nature, Science, NEJM, JAMA)
• Top conferences (NeurIPS, ICML, ACL)
• Reputable institutions (MIT, Stanford, Google Health)

Medium Quality:

• Pre-print servers (arXiv, bioRxiv) - not yet peer-reviewed
• Conference proceedings
• Institutional technical reports

Lower Quality:

• Blog posts and news articles
• Non-peer-reviewed content
• Content without clear authorship or affiliation

Example: Competitive Price Comparison

Compare product prices across multiple retailers.

Goal

Compare iPhone 15 Pro Max prices across major retailers to find the best deal.

Command

/research-brightdata \
  query="iphone 15 pro max 256GB price amazon bestbuy walmart target" \
  mode=standard \
  max_results=15 \
  extract_schema="Extract: retailer name, product name including model and storage, price as number (USD), currency, availability (in stock/out of stock), shipping cost, delivery estimate, any promotions or discounts" \
  output_format=json

Expected Workflow

Phase 1: Discovery

Search for product pages across retailers:

Query: iphone 15 pro max 256GB price
Target retailers: Amazon, Best Buy, Walmart, Target
Expected results: 3-4 pages per retailer

Phase 2: Collection

Batch scrape retailer product pages:

URLs collected: 12-15
Batch processing: 10 URLs at a time
Time: ~20 seconds

Phase 3: Extraction

Extract pricing and availability data:

{
  "retailer": "Walmart",
  "product": "Apple iPhone 15 Pro Max 256GB",
  "price": 999.00,
  "currency": "USD",
  "availability": "in_stock",
  "shipping": 0,
  "delivery": "2-3 business days",
  "promotions": "Free shipping, no tax in some states"
}

Phase 4: Analysis

Compare prices and find best deal:

• Calculate total price (including shipping and tax)
• Check availability
• Factor in delivery time
• Identify promotions

Phase 5: Report

Generate comparison with recommendation.

Expected Output

{
  "query": "iphone 15 pro max 256GB price comparison",
  "timestamp": "2024-01-22T10:30:00Z",
  "summary": {
    "total_retailers": 4,
    "lowest_price": 999.00,
    "highest_price": 1199.00,
    "average_price": 1089.50,
    "savings_range": "$0 - $200"
  },
  "comparison": [
    {
      "rank": 1,
      "retailer": "Walmart",
      "product": "iPhone 15 Pro Max 256GB",
      "price": 999.00,
      "shipping": 0,
      "total": 999.00,
      "availability": "in_stock",
      "delivery": "2-3 business days",
      "promotions": "Free shipping",
      "notes": "Best price, free shipping"
    },
    {
      "rank": 2,
      "retailer": "Amazon",
      "price": 1049.00,
      "shipping": 0,
      "total": 1049.00,
      "availability": "in_stock",
      "delivery": "Next-day delivery for Prime members",
      "promotions": "None",
      "notes": "Faster delivery available"
    },
    {
      "rank": 3,
      "retailer": "Best Buy",
      "price": 1099.00,
      "shipping": 0,
      "total": 1099.00,
      "availability": "in_stock",
      "delivery": "Same-day pickup available",
      "promotions": "Student discount available",
      "notes": "Good for immediate pickup"
    },
    {
      "rank": 4,
      "retailer": "Target",
      "price": 1199.00,
      "shipping": 0,
      "total": 1199.00,
      "availability": "in_stock",
      "delivery": "2-4 business days",
      "promotions": "RedCard discount 5%",
      "notes": "RedCard brings price to $1139.05"
    }
  ],
  "recommendation": {
    "best_price": "Walmart at $999.00 (save $200 vs MSRP)",
    "fastest_delivery": "Amazon with next-day for Prime members",
    "best_pickup": "Best Buy with same-day pickup",
    "overall_best": "Walmart - best price with free shipping"
  },
  "sources": [
    "https://www.walmart.com/ip/...",
    "https://www.amazon.com/dp/...",
    "https://www.bestbuy.com/site/...",
    "https://www.target.com/p/..."
  ]
}

Variations

Quick Price Check

/research-brightdata \
  query="iphone 15 price" \
  mode=quick \
  max_results=5 \
  output_format=markdown

Multiple Product Comparison

/research-brightdata \
  query="Galaxy S24 Ultra vs iPhone 15 Pro Max vs Pixel 8 Pro price comparison" \
  mode=standard \
  extract_schema="Extract: product name, price, screen size, storage, key features" \
  max_results=10

With Trade-in Values

/research-brightdata \
  query="iphone 15 pro max trade in value amazon bestbuy" \
  extract_schema="Extract: retailer name, new price, trade-in offer for iPhone 14 Pro Max, final price after trade-in" \
  mode=standard

Tips for This Use Case

1. Be Specific: Include exact model and storage capacity 2. Check Availability: Stock levels vary by retailer 3. Consider Total Cost: Include shipping, tax, and fees 4. Factor in Timing: Delivery speed may matter more than small price differences 5. Look for Promotions: Student discounts, credit card offers, etc.

Analysis Considerations

Price Matching

• Some retailers price match (Best Buy, Target)
• Bring proof of lower price
• Check price match policies before purchasing

Additional Costs

• Sales tax (varies by location)
• Shipping (usually free for expensive items)
• Accessories (cases, chargers not included)

Timing

• New product releases may lower prices
• Holiday sales (Black Friday, Cyber Monday)
• Back-to-school promotions

Trade-in Programs

• Apple Trade In
• Carrier trade-in deals
• Retailer trade-in promotions

Example: Etsy NBA Merchandise Research

Complete workflow for researching NBA merchandise on Etsy.

Goal

Research NBA merchandise on Etsy to understand:

• Product variety and pricing
• Popular sellers and ratings
• Price ranges by category
• Customer satisfaction levels

Command

/research-brightdata \
  query="site:etsy.com nba merchandise jerseys shirts" \
  mode=standard \
  max_results=20 \
  extract_schema="Extract: product name, price as number, currency, rating (0-5), number of reviews, seller name, product category, availability status" \
  output_format=report

Expected Workflow

Phase 1: Discovery

Search engine finds Etsy pages:

Query: site:etsy.com nba merchandise jerseys shirts
Engine: google
Results: ~20 Etsy product and category pages

Phase 2: Collection

Batch scrape up to 10 URLs concurrently:

URLs collected: 20
Batch 1: 10 URLs scraped in parallel
Batch 2: 10 URLs scraped in parallel
Time: ~30 seconds total

Phase 3: Extraction

Extract structured data from each page:

{
  "product_name": "NBA Vintage Basketball Jersey",
  "price": 45.99,
  "currency": "USD",
  "rating": 4.7,
  "review_count": 234,
  "seller": "RetroSports",
  "category": "Jerseys",
  "availability": "in_stock"
}

Phase 4: Analysis

Analyze extracted data:

• Price range: $15 - $120
• Average rating: 4.6/5
• Top categories: Jerseys (40%), T-shirts (30%), Accessories (20%), Other (10%)
• Top 5 sellers by rating and volume

Phase 5: Report

Generate comprehensive report with findings and recommendations.

Expected Output

# Research Report: NBA Merchandise on Etsy

## Executive Summary

Analyzed 20 NBA merchandise listings on Etsy. Prices range from $15 to $120 with an average of $52. Customer satisfaction is high (4.6/5 average rating). Jerseys are the most popular category (40% of listings), followed by t-shirts (30%).

## Key Findings

### Pricing Analysis
- **Price Range**: $15 - $120
- **Average Price**: $52
- **Median Price**: $48
- **Best Value**: "Custom NBA T-Shirt" at $22 with 4.8/5 rating

### Top Categories
1. **Jerseys** (40% of listings)
   - Price range: $45 - $120
   - Average rating: 4.5/5

2. **T-Shirts** (30% of listings)
   - Price range: $15 - $45
   - Average rating: 4.7/5

3. **Accessories** (20% of listings)
   - Price range: $18 - $65
   - Average rating: 4.6/5

### Top Sellers by Rating
1. RetroSports - 4.8/5 (234 reviews)
2. HoopsFanatic - 4.7/5 (189 reviews)
3. CourtSideGear - 4.7/5 (156 reviews)
4. BasketballBasics - 4.6/5 (98 reviews)
5. TeamSpiritShop - 4.5/5 (87 reviews)

### Recommendations
- **Best Value**: Custom t-shirts under $25 with high ratings
- **Premium Market**: Jerseys priced $80+ have fewer sales but higher margins
- **Opportunity**: Limited accessory selection, room for growth

## Methodology
- Sources: Google search with site:etsy.com filter
- URLs Analyzed: 20
- Data Points Extracted: 140 (7 per listing)
- Confidence: High

## Sources
[Complete list of 20 Etsy URLs analyzed]

Variations

Quick Overview

/research-brightdata \
  query="site:etsy.com nba" \
  mode=quick \
  max_results=5

Deep Analysis with Browser Automation

/research-brightdata \
  query="site:etsy.com nba jerseys" \
  mode=deep \
  max_results=30 \
  output_format=report

Price Comparison Only

/research-brightdata \
  query="site:etsy.com nba jersey price" \
  extract_schema="Extract: product name, price, seller name" \
  output_format=json

Tips for This Use Case

1. Use Site Filter: site:etsy.com ensures only Etsy results 2. Specific Categories: Add specific terms (jerseys, shirts, hats) for focused results 3. Standard Mode: Best balance of speed and depth for e-commerce 4. Custom Extraction: Specify exactly which fields to extract

Common Issues

Issue: Some Etsy pages require JavaScript Solution: Use mode=deep for browser automation

Issue: Duplicate listings from same seller Solution: Report should include deduplication

Issue: Variations (sizes, colors) listed separately Solution: Group by base product in analysis

Issue: Token limit exceeded when batch scraping Solution:

• Reduce batch size to 1-2 URLs at a time
• Use scrape_as_markdown for individual URLs
• Process URLs sequentially instead of in parallel
• If output is saved, use Read with offset and limit to read in chunks

Analysis and Report Template

This template handles the final analysis of extracted data and generates comprehensive reports.

Process

Step 1: Data Cleaning and Normalization

Deduplication:

• Remove exact duplicates
• Remove near-duplicates (90%+ similarity)
• Keep the most complete/authoritative version

Normalization:

• Dates: Convert to ISO 8601 format (YYYY-MM-DD)
• Currencies: Convert to numbers, store currency separately
• Numbers: Remove commas, formatting
• Text: Trim whitespace, normalize line breaks
• URLs: Ensure well-formed, remove tracking parameters

Standardization:

• Units: Convert to standard units (USD, km, kg, etc.)
• Categories: Map to standard taxonomy
• Ratings: Convert to common scale (0-5 or 0-10)
• Locations: Standardize country/city names

Step 2: Statistical Analysis

Descriptive Statistics:

{
  "count": total_number_of_items,
  "min": minimum_value,
  "max": maximum_value,
  "mean": average_value,
  "median": middle_value,
  "mode": most_common_value,
  "stddev": standard_deviation,
  "quartiles": {
    "q1": 25th_percentile,
    "q2": 50th_percentile,
    "q3": 75th_percentile
  }
}

Distribution Analysis:

• Identify clusters and patterns
• Detect outliers (values > 2 std deviations from mean)
• Analyze frequency distributions
• Visualize with histograms (if applicable)

Trend Analysis:

• Compare across time periods
• Identify growth/decline patterns
• Calculate rates of change
• Detect seasonality

Step 3: Comparative Analysis

Cross-Source Comparison:

• Compare data from different sources
• Identify agreements and discrepancies
• Validate findings across sources
• Calculate inter-source reliability

Benchmarking:

• Compare against industry averages
• Identify above/below average performers
• Calculate percentiles and rankings
• Highlight leaders and laggards

Correlation Analysis:

• Find relationships between variables
• Calculate correlation coefficients
• Identify causal factors (where appropriate)
• Spot unexpected correlations

Step 4: Insight Generation

Key Findings: 1. Most Important: What are the top 3-5 most significant insights? 2. Surprising: What's unexpected or counterintuitive? 3. Actionable: What can the user do with this information? 4. Trends: What patterns or changes are emerging? 5. Outliers: What stands out from the norm?

Data-Backed Insights: Every insight should include:

• The finding (what)
• The evidence (data)
• The significance (so what)
• The implication (now what)

Step 5: Report Generation

Choose the appropriate format based on output_format:

Markdown Format

Clean, readable markdown with:

• Clear headings hierarchy
• Bullet points for lists
• Tables for structured data
• Code blocks for JSON/data
• Links to sources

JSON Format

Structured JSON with:

• Metadata (query, timestamp, method)
• Summary statistics
• Raw and processed data
• Sources and provenance
• Validation information

Report Format

Comprehensive document with:

• Executive summary (1-2 pages)
• Detailed findings (5-10 pages)
• Methodology section (1 page)
• Data visualizations (tables, charts)
• Recommendations section (1-2 pages)
• Appendices (raw data, sources)

Report Structure

Executive Summary

Purpose: Give key insights in 2-3 minutes

Content:

• Brief overview of research scope
• Top 3-5 key findings
• Most important recommendations
• Confidence level in findings

Length: 2-3 paragraphs

Key Findings

Purpose: Detail the main discoveries

Structure:

## Finding 1: [Descriptive Title]

**Summary**: One-sentence summary

**Details**: Elaborate on the finding with supporting data

**Evidence**:
- Data point 1
- Data point 2
- Data point 3

**Impact**: Why this matters

**Source**: [Source URL](https://example.com)

Detailed Analysis

Purpose: Provide in-depth exploration

Content:

• Statistical breakdown
• Comparisons and benchmarks
• Trends over time
• Subgroup analyses
• Correlations and relationships

Methodology

Purpose: Document research process

Content:

• Sources searched
• Number of URLs analyzed
• Tools and methods used
• Data quality assessment
• Limitations and caveats

Recommendations

Purpose: Actionable next steps

Format:

## Recommendation 1: [Action Title]

**Priority**: High/Medium/Low

**Action**: What to do

**Rationale**: Why to do it

**Expected Impact**: What outcome to expect

**Effort**: How difficult (Easy/Medium/Hard)

Sources

Purpose: Provide traceability

Format: Numbered list with URLs and titles

Data Visualization

Tables

For structured data:

| Product | Price | Rating | Reviews |
|---------|-------|--------|---------|
| Item 1  | $29.99| 4.5    | 1,234   |
| Item 2  | $49.99| 4.7    | 567     |

Statistics

For numerical data:

## Price Analysis

- **Count**: 45 products
- **Range**: $15.00 - $299.99
- **Average**: $87.50
- **Median**: $79.99
- **Std Dev**: $45.30

**Distribution**:
- Under $50: 12 products (27%)
- $50-$100: 20 products (44%)
- $100-$150: 8 products (18%)
- Over $150: 5 products (11%)

Rankings

For comparative data:

## Top 5 by Rating

1. **Product A** - 4.9/5 (2,345 reviews)
2. **Product B** - 4.8/5 (1,234 reviews)
3. **Product C** - 4.7/5 (987 reviews)
4. **Product D** - 4.6/5 (765 reviews)
5. **Product E** - 4.5/5 (654 reviews)

Quality Assurance

Data Quality Checks

• [ ] All data points have sources
• [ ] Statistics are calculated correctly
• [ ] Findings are supported by data
• [ ] No contradictions in the report
• [ ] All URLs are accessible
• [ ] Spelling and grammar checked
• [ ] Formatting is consistent

Validation

• Cross-check findings with raw data
• Verify statistical calculations
• Ensure logical consistency
• Check for bias or misinterpretation

Output Examples

Quick Research Output

# Quick Research: iPhone 15 Price

**Date**: 2024-01-22
**Sources**: Google, Amazon, Best Buy
**URLs Analyzed**: 5

## Summary

Found 5 retailers selling iPhone 15 Pro Max 256GB.
Price range: $999 - $1,099
Best deal: Walmart at $999 (save $100)

## Prices

1. Walmart: $999 ✓ Best Price
2. Amazon: $1,049
3. Best Buy: $1,069
4. Target: $1,079
5. Apple: $1,099

## Recommendation

Buy from Walmart to save $100 compared to Apple.

Comprehensive Report Output

See the main template for full report structure.

Best Practices

1. Start with Summary: Most users only read the executive summary 2. Use Visuals: Tables, lists, and formatting improve readability 3. Be Specific: Use exact numbers, not vague descriptions 4. Provide Context: Explain why findings matter 5. Cite Sources: Every data point should have a source 6. Be Honest: Admit limitations and uncertainties 7. Make it Actionable: Tie findings to recommendations 8. Keep it Concise: Respect the user's time

Content Scraping Template

This template handles the collection of content from discovered URLs.

Parameters

• urls: Array of URLs to scrape
• mode: scraping mode (serial/batch/browser)
• format: output format (markdown/html)

Process

Quick Mode (Serial Processing)

Process URLs one at a time:

{
  "tool": "scrape_as_markdown",
  "parameters": {
    "url": "{{url}}"
  }
}

Use when:

• Testing new sources
• Small number of URLs (< 5)
• When order matters (sequential dependencies)

Standard Mode (Parallel Batch)

Process up to 10 URLs concurrently:

{
  "tool": "scrape_batch",
  "parameters": {
    "urls": [
      "{{url1}}",
      "{{url2}}",
      "...up to 10 URLs..."
    ]
  }
}

Use when:

• Processing 5-20 URLs
• Speed is important
• URLs are independent of each other

Batching Strategy:

• Split large URL lists into batches of 10
• Process batches sequentially
• Track success/failure per URL

Deep Mode (Browser Automation)

For JavaScript-rendered pages:

{
  "tool": "scraping_browser_navigate",
  "parameters": {
    "url": "{{url}}"
  }
}

Use when:

• Pages require JavaScript execution
• Content loads dynamically (AJAX, React, etc.)
• Need to interact with the page
• Sites have strong anti-bot measures

Error Handling

Retry Strategy

If scrape fails: 1. First retry: Use alternative method

• scrape_as_markdown → scrape_as_html
• scrape_as_markdown → scraping_browser_navigate

2. Second retry: After 5 second delay

3. Give up: After 2 failed attempts

Error Logging

Track errors with context:

{
  "url": "https://example.com",
  "error": "Connection timeout",
  "method": "scrape_as_markdown",
  "timestamp": "2024-01-22T10:30:00Z"
}

Content Validation

After scraping, validate:

• [ ] Content is not empty
• [ ] Content is not error page (404, 500, etc.)
• [ ] Content is not CAPTCHA page
• [ ] Content has meaningful text (> 100 characters)
• [ ] Content is in expected language

Output Format

{
  "total_urls": {{total}},
  "successful": {{count}},
  "failed": {{count}},
  "contents": [
    {
      "url": "https://example.com/page1",
      "content": "# Page Title\n\nContent here...",
      "word_count": 542,
      "language": "en",
      "scrape_method": "scrape_as_markdown",
      "timestamp": "2024-01-22T10:30:00Z"
    }
  ],
  "errors": [
    {
      "url": "https://example.com/page2",
      "error": "Connection timeout",
      "retries": 2
    }
  ]
}

Performance Tips

1. Use scrape_batch: 10x faster than serial processing 2. Set Timeouts: Don't wait indefinitely for slow pages 3. Parallelize: Process multiple batches concurrently if supported 4. Cache Results: Store scraped content to avoid re-scraping 5. Monitor Success Rate: If >30% fail, adjust strategy 6. Avoid Token Limits:

• What is the limit?: 25,000 tokens maximum per Read operation
• Why this limit exists:
• Memory protection: Prevents memory overflow from loading too much content
• Performance optimization: Ensures fast response times
• Context management: Preserves space for other content in conversation
• System stability: Prevents crashes or errors
• Can this be increased?:
• No - This is a hard system limit in Claude Code
• Cannot be changed via configuration files
• Purpose: Protect system stability and performance
• Workarounds for large files:
• Use Read with offset and limit to read in chunks (e.g., 100 lines at a time)
• Use Grep to search for specific patterns instead of reading entire file
• Use Glob to find files by pattern
• Workarounds for scraping:
• Large pages (Etsy, Amazon, etc.) can exceed 25,000 tokens
• Batch 1-2 URLs at a time for large pages
• Use scrape_as_markdown for individual URLs if needed

Content Cleaning

After scraping, clean the content:

• Remove navigation menus
• Remove footer and sidebar content
• Remove advertisements
• Remove popups and modals
• Normalize whitespace
• Fix encoding issues

Bright Data's scrape_as_markdown does most of this automatically, but additional cleaning may be needed for some sites.

Data Extraction Template

This template handles the extraction of structured data from scraped content.

Parameters

• content: Scraped content (markdown or HTML)
• extraction_prompt: Instructions for what data to extract
• schema: Expected data structure (optional)

Process

Option 1: AI-Powered Extraction

Use Bright Data's extract tool for intelligent extraction:

{
  "tool": "extract",
  "parameters": {
    "url": "{{url}}",
    "extraction_prompt": "{{extraction_prompt}}"
  }
}

Extraction Prompt Examples:

E-commerce Products:

Extract: product name, price (as number, exclude currency symbol),
availability (in stock/out of stock), rating (0-5 scale),
number of reviews, seller name, main features (as array of strings),
product image URL

Articles:

Extract: article title, author name, publication date (YYYY-MM-DD),
article category, main topics (array of 3-5 keywords),
summary (max 200 words), key insights (array of bullet points)

General:

Extract the main heading, all subheadings with their levels,
the main content summary, all links with their anchor text,
and metadata like date, author, tags

Option 2: Manual Extraction

Parse the scraped content directly:

# From Markdown Content

1. Extract heading (first # heading)
2. Extract all subheadings (##, ###)
3. Extract lists and tables
4. Extract links: [text](url) format
5. Extract metadata (dates, authors, etc.)

Data Validation

After extraction, validate:

Type Validation

• Numbers: Ensure numeric fields are actual numbers
• Dates: Ensure dates are valid (YYYY-MM-DD format)
• URLs: Ensure URLs are well-formed
• Emails: Ensure email format is correct

Completeness Check

• All required fields are present
• No null/empty values in required fields
• Arrays have at least one element
• Strings are not empty strings

Range Validation

• Ratings: 0-5 or 0-10
• Prices: Greater than 0
• Percentages: 0-100
• Dates: Within reasonable range

Consistency Check

• Related fields make sense together
• No contradictory information
• Units are consistent (currency, measurements)

Common Extraction Schemas

E-commerce Product

{
  "product_name": "string",
  "price": "number",
  "currency": "string (USD, EUR, etc.)",
  "availability": "string (in_stock, out_of_stock, preorder)",
  "rating": "number (0-5)",
  "review_count": "number",
  "seller": {
    "name": "string",
    "rating": "number",
    "url": "string"
  },
  "features": ["string"],
  "images": ["string (URLs)"],
  "url": "string",
  "scraped_at": "ISO 8601 timestamp"
}

Article/Blog Post

{
  "title": "string",
  "author": "string",
  "publication_date": "YYYY-MM-DD",
  "category": "string",
  "tags": ["string"],
  "summary": "string",
  "key_points": ["string"],
  "word_count": "number",
  "reading_time_minutes": "number",
  "url": "string",
  "scraped_at": "ISO 8601 timestamp"
}

Company/Organization

{
  "name": "string",
  "industry": "string",
  "founded_year": "number",
  "headquarters": "string",
  "employee_count": "string (e.g., '10,000+')",
  "website": "string (URL)",
  "description": "string",
  "key_people": [{"name": "string", "title": "string"}],
  "products_services": ["string"],
  "url": "string",
  "scraped_at": "ISO 8601 timestamp"
}

Handling Missing Data

When a field is missing: 1. Use default values: null, 0, "", [] 2. Mark as missing: Add field like "field_missing": true 3. Log the issue: Track which fields are frequently missing 4. Don't fail: Continue with available data

Handling Malformed Data

When data doesn't match expected format: 1. Try to fix: Common transformations

• Currency: "$1,234.56" → 1234.56
• Dates: "Jan 1, 2024" → "2024-01-01"
• Ratings: "4.5 out of 5" → 4.5

2. Flag for review: Add "validation_errors": ["..."] 3. Use best effort: Extract what you can 4. Document the issue: Explain what was malformed

Output Format

{
  "extraction_method": "ai|manual",
  "total_items": {{count}},
  "successful": {{count}},
  "failed": {{count}},
  "data": [
    {
      "item_id": "unique_id",
      "source_url": "https://example.com",
      "extracted_data": { /* extracted fields */ },
      "validation": {
        "is_valid": true,
        "errors": [],
        "warnings": []
      },
      "extracted_at": "ISO 8601 timestamp"
    }
  ],
  "statistics": {
    "fields_extracted": {{count}},
    "avg_completeness": "0.85",
    "missing_fields": ["field1", "field2"]
  }
}

Best Practices

1. Be Specific: The more specific your extraction prompt, the better the results 2. Provide Examples: Include examples of expected output in the prompt 3. Iterate: Test and refine extraction prompts based on results 4. Validate: Always validate extracted data 5. Handle Edge Cases: Plan for missing, malformed, or unexpected data 6. Document: Keep track of what works and what doesn't

Deep Scraping Template

This template handles browser automation for JavaScript-rendered or interactive pages.

When to Use Deep Scraping

Use browser automation when:

• Content requires JavaScript to render (React, Vue, Angular apps)
• Pages load data dynamically via AJAX
• Need to interact with the page (click, scroll, fill forms)
• Sites have strong anti-bot measures
• Need to handle pagination dynamically
• Content is behind login walls

Process

Step 1: Navigate to Page

{
  "tool": "scraping_browser_navigate",
  "parameters": {
    "url": "{{url}}"
  }
}

Expected Response:

Successfully navigated to https://example.com
Title: Page Title
URL: https://example.com

Wait Strategy:

• Browser automatically waits for page load (up to 120 seconds)
• For dynamic content, add additional wait time
• Look for specific elements to appear

Step 2: Discover Page Elements

Get all clickable elements and links:

{
  "tool": "scraping_browser_links",
  "parameters": {}
}

Expected Response:

[
  {
    "text": "Next Page",
    "href": "/page/2",
    "selector": "a.pagination-next"
  },
  {
    "text": "Load More",
    "href": "#",
    "selector": "button#load-more"
  }
]

Step 3: Extract Content

After navigation, you can:

• Use scrape_as_markdown on the current page
• Extract content manually from page HTML
• Use extract with the current URL

Step 4: Interact with Page

Click elements using CSS selectors:

{
  "tool": "scraping_browser_click",
  "parameters": {
    "selector": "{{css_selector}}"
  }
}

Common Interactions:

Pagination:

{
  "tool": "scraping_browser_click",
  "parameters": {
    "selector": "a.next-page"
  }
}

Load More Buttons:

{
  "tool": "scraping_browser_click",
  "parameters": {
    "selector": "button.load-more"
  }
}

Dropdowns/Filters:

{
  "tool": "scraping_browser_click",
  "parameters": {
    "selector": "select#sort-by option[value='price-desc']"
  }
}

Step 5: Repeat for Multi-Page Content

For content spread across multiple pages:

1. Navigate to initial URL 2. Extract content from page 1 3. Click "Next" button 4. Wait for page load 5. Extract content from page 2 6. Repeat until no more pages

Advanced Techniques

Scrolling for Lazy Loading

Some pages load content as you scroll:

1. Navigate to page
2. Scroll down (execute JavaScript)
3. Wait for new content to load
4. Repeat until no new content appears
5. Extract all content

Handling Infinite Scroll

1. Navigate to page
2. While true:
   - Get current height
   - Scroll to bottom
   - Wait 2 seconds
   - Check if height increased
   - If no increase, break
3. Extract all loaded content

Waiting for Specific Elements

1. Navigate to page
2. Wait for element with selector ".data-loaded" to appear
3. Extract content

Handling Modals and Popups

1. Navigate to page
2. Check for modal: ".modal" or ".popup"
3. If present, click close button: ".close-modal"
4. Wait for modal to disappear
5. Proceed with extraction

CSS Selector Tips

Basic Selectors

• #id - Element by ID
• .class - Element by class
• tag - Element by tag name
• tag.class - Tag with class

Combinations

• div.container > ul > li - Direct children
• div.container ul li - Any descendants
• button[type="submit"] - Attribute selector

Useful Patterns

• a.next-page - Next/prev buttons
• button.load-more - Load more buttons
• .pagination a - Pagination links
• .item:nth-child(n) - Specific item
• [data-page="2"] - Data attributes

Testing Selectors

Before using in automation: 1. Open page in browser 2. Open DevTools (F12) 3. Use document.querySelector('selector') in console 4. Verify it returns the correct element

Error Handling

Navigation Failures

• Timeout: Page takes too long to load
• Solution: Increase wait time or check if page is accessible
• Blocked: Site blocks automated browsers
• Solution: Use different user agent or try scrape_as_markdown

Click Failures

• Element not found: Selector doesn't match any element
• Solution: Verify selector, check if element exists
• Element not clickable: Element is obscured or disabled
• Solution: Wait for page to stabilize, check if element is enabled
• Navigation doesn't occur: Click doesn't navigate
• Solution: Element might trigger JavaScript, handle accordingly

Session Issues

• Session lost: Browser session disconnected
• Solution: Navigate again to restore session
• Cookie/State lost: Page lost authentication or state
• Solution: Re-navigate and re-authenticate if needed

Best Practices

1. Use Specific Selectors: More specific = more reliable 2. Wait for Stability: Don't click immediately after page load 3. Handle Errors Gracefully: Have fallback strategies 4. Log Everything: Track navigation, clicks, and extractions 5. Respect the Site: Don't overwhelm with rapid actions 6. Test First: Manually test the workflow before automating 7. Use Session Persistence: Browser keeps cookies/state across calls

Example Workflow: Multi-Page Product Listing

Goal: Extract all products from a multi-page category page

1. Navigate to category page:
   scraping_browser_navigate("https://example.com/category")

2. Initialize: products = [], page = 1

3. While true:
   a. Extract product data from current page
   b. Add to products array
   c. Look for "Next" button
   d. If no "Next" button, break
   e. Click "Next" button
   f. Wait for page load
   g. page += 1

4. Return all products

Output Format

{
  "workflow": "multi_page_extraction",
  "starting_url": "https://example.com",
  "total_pages": {{number}},
  "total_items": {{number}},
  "data": [
    {
      "page": 1,
      "items": [ /* extracted items */ ]
    },
    {
      "page": 2,
      "items": [ /* extracted items */ ]
    }
  ],
  "actions_taken": [
    "navigate",
    "extract",
    "click .next-page",
    "extract"
  ],
  "duration_seconds": {{time}}
}

Performance Considerations

• Browser automation is slower: 10-50x slower than direct scraping
• Use only when necessary: Prefer scrape_as_markdown when possible
• Parallel sessions: Can maintain multiple browser sessions for different domains
• Timeout management: Set appropriate timeouts to avoid hanging
• Resource intensive: Uses more memory and CPU than direct scraping

Search Discovery Template

This template handles the discovery phase of research, finding relevant URLs through search engines.

Parameters

• query: Search query
• engine: Search engine to use (google/bing/yandex)
• max_results: Maximum number of URLs to discover
• site_filter: Optional site restriction (e.g., "site:etsy.com")

Process

Step 1: Initial Search

{
  "tool": "search_engine",
  "parameters": {
    "query": "{{query}}",
    "engine": "{{engine}}",
    "cursor": "0"
  }
}

Step 2: Extract URLs

Parse the SERP results and extract all URLs. Look for:

• Organic search results
• Exclude: ads, sponsored content, social media links

Step 3: Filter and Validate

For each URL:

• Check domain reputation
• Verify URL is accessible
• Filter out irrelevant domains
• Deduplicate URLs

Step 4: Paginate if Needed

If URL count < max_results:

{
  "tool": "search_engine",
  "parameters": {
    "query": "{{query}}",
    "engine": "{{engine}}",
    "cursor": "1"
  }
}

Continue until max_results reached or no more results available.

Step 5: Prioritize and Rank

Sort URLs by: 1. Domain authority (established sites first) 2. Title relevance (keyword matching) 3. URL structure (clean, descriptive URLs) 4. Expected content quality

Output Format

{
  "query": "{{query}}",
  "total_found": {{total}},
  "urls": [
    {
      "url": "https://example.com/page1",
      "title": "Page Title",
      "relevance_score": 0.95,
      "domain_authority": "high"
    },
    {
      "url": "https://example.com/page2",
      "title": "Another Page",
      "relevance_score": 0.87,
      "domain_authority": "medium"
    }
  ]
}

Best Practices

1. Use Specific Queries: "site:etsy.com nba jerseys" instead of "nba" 2. Check Multiple Engines: If Google fails, try Bing or Yandex 3. Validate URLs: Remove malformed or suspicious URLs 4. Set Realistic Limits: 10-20 URLs is usually sufficient for standard mode 5. Log Search Metadata: Record search engine, cursor, total results for transparency