Scrapling

scrapling - routing-first adaptive web scraping

Keyword: scrapling · adaptive scraping · stealthy fetch · scrapling spider

>

Respect each target site's terms, robots, rate limits, authorization boundaries, and anti-abuse policies.

Scrapling is most useful when you choose the smallest scraping mode that fits the job:

• parse known HTML before reaching for a browser
• start with plain HTTP before rendering JavaScript
• escalate to stealth only when simpler paths fail
• switch to spiders only when crawl state and multi-page orchestration become the real problem

When to use this skill

• The user needs one page or a few pages extracted and you must decide between parser-only, HTTP fetch, browser rendering, or stealth
• The user wants to prototype with scrapling extract, scrapling shell, or wrapper scripts instead of writing Python first
• The user wants adaptive selector recovery because the target DOM changes over time
• The user wants Scrapling exposed to an agent via MCP
• The user is about to turn repeated fetches into a site crawl and needs to know when Scrapling spiders are justified
• The user keeps drifting toward Playwright/Crawlee-style orchestration, direct API interception, or managed unblockers and needs an honest route-out

Instructions

Step 1: Capture one intake packet before choosing tools

Collect the minimum packet:

• Target class: raw HTML, normal web page, JS-rendered app, protected site, or many-page crawl
• Desired output: CSS/XPath extraction, Markdown/text file, structured records, or long-running crawl output
• Operational need: quick probe, repeated session reuse, agent/MCP access, or resumable crawling
• Constraint: install budget, browser availability, auth/proxy limits, and whether direct API/JSON access may be better than DOM scraping

Use this rule set:

1. If HTML is already available, start with Selector 2. If a live page is needed, start with Fetcher 3. If the content is rendered client-side, move to DynamicFetcher 4. If the rendered path is still blocked or empty, move to StealthyFetcher 5. If the job is now queueing, link following, retries, or checkpoints, move to spiders 6. If the real need is browser automation, pre-navigation request interception, or a managed unblocker, route out instead of over-promising Scrapling

Detailed packet examples and route-outs live in references/intake-packets-and-route-outs.md.

Step 2: Install only the profile you need

Use a virtual environment unless the user explicitly wants a system install.

bash scripts/install.sh --profile parser
bash scripts/install.sh --profile fetchers
bash scripts/install.sh --profile shell
bash scripts/install.sh --profile ai
bash scripts/install.sh --profile all

Installation guidance:

• parser — local HTML parsing only
• fetchers — HTTP + browser-backed fetchers
• shell — interactive CLI shell workflows
• ai — MCP server workflows
• all — only when the user truly needs the full surface

Browser-backed flows require scrapling install. Parser-only workflows do not.

Step 3: Choose the lightest extraction path that fits

A. Parser-only path

Use this when HTML is already available or the user mainly needs selector logic.

from scrapling import Selector

page = Selector(html_doc, url="https://example.com")
items = page.css("article h2::text").getall()

Use this path for:

• local HTML fixtures
• saved responses
• quick selector authoring
• low-cost extraction without network/browser concerns

B. HTTP fetch path

Use Fetcher or FetcherSession when plain HTTP is likely enough.

from scrapling.fetchers import Fetcher

page = Fetcher.get("https://example.com", impersonate="chrome")
title = page.css("title::text").get()

Prefer this path for:

• docs, blogs, many marketing pages, search results, simple product pages
• cheap repeated requests with session reuse
• cases where the user only needs rendered HTML if evidence proves HTTP is insufficient

C. Dynamic browser path

Use DynamicFetcher or DynamicSession when the target is JS-rendered.

from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.fetch(
    "https://example.com/app",
    network_idle=True,
    wait_selector=".content"
)

Reach for this when:

• the page is a SPA/dashboard
• content appears only after client-side rendering
• small interaction or wait controls are enough

D. Stealth path

Use StealthyFetcher or StealthySession only when the lighter paths fail because of protection.

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://example.com/protected",
    headless=True,
    solve_cloudflare=True
)

Important:

• Treat anti-bot handling as a documented capability, not a guarantee
• If a target still fails or requires stronger infra, route honestly to proxies or managed unblockers instead of pretending one more flag will always fix it

Fetcher/session details live in references/fetchers-and-sessions.md.

Step 4: Use adaptive scraping only when selector drift is the problem

Adaptive scraping is valuable when the DOM changes but the semantic target is still the same.

from scrapling.fetchers import Fetcher

Fetcher.configure(adaptive=True)
page = Fetcher.get("https://example.com")
saved = page.css(".product", auto_save=True)
relocated = page.css(".product", adaptive=True)

Remember:

• adaptive=True is opt-in
• auto_save=True stores fingerprints for later reuse
• manual save(), retrieve(), and relocate() flows are available
• this helps brittle selectors, not auth failures or full site redesign strategy

Parser/adaptive details live in references/parser-and-adaptive.md.

Step 5: Choose the right operator surface

CLI / shell

Use CLI when the user wants quick output files or a terminal-first probe.

bash scripts/run-extract.sh get "https://example.com" article.md --css-selector "article"
bash scripts/run-extract.sh fetch "https://app.example.com" content.md --network-idle
bash scripts/run-extract.sh stealth "https://protected.example.com" content.md --solve-cloudflare

Use scrapling shell when the user wants an interactive REPL rather than a standalone script.

MCP

Use MCP when the goal is agent-facing access instead of a bespoke scraper script.

bash scripts/run-mcp.sh
bash scripts/run-mcp.sh --http --host 127.0.0.1 --port 8000

Start with stdio for local agent integration; use HTTP only when the environment already expects streamable HTTP.

CLI/MCP details live in references/cli-and-mcp.md.

Step 6: Switch to spiders only when crawl state matters

Use spiders when the task needs:

• multiple pages and link following
• retries and blocked-response handling
• concurrency and session routing
• pause/resume checkpoints via crawldir

For one or two isolated pages, fetchers remain simpler.

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    async def parse(self, response: Response):
        for quote in response.css(".quote"):
            yield {"text": quote.css(".text::text").get()}

Spider details live in references/spiders.md.

Examples

Example 1: I already have HTML and just need selectors

Use Selector, not a browser.

Example 2: A marketing page loads over plain HTTP

Start with Fetcher or scrapling extract get, then escalate only if evidence says the content is missing.

Example 3: A React dashboard is empty until JS runs

Move to DynamicFetcher and use network_idle or wait_selector.

Example 4: A protected target returns a challenge page

Try StealthyFetcher, but make it explicit that stronger proxy/unblocker infrastructure may still be required.

Example 5: An agent needs bounded web extraction

Expose Scrapling through MCP instead of forcing a handwritten scraper first.

Example 6: The task has become a real crawl

Route from repeated fetches into spiders with checkpointing and session control.

Best practices

1. Start with the smallest viable mode and escalate only on evidence. 2. Prefer direct API/JSON access when that is the real source of truth; do not force DOM scraping for browser-heavy apps. 3. Reuse sessions for repeated requests instead of relaunching browsers on every page. 4. Use adaptive scraping for selector drift, not as a blanket fix for every failure mode. 5. Prefer CSS selectors plus .md or .txt outputs to keep model context smaller. 6. Treat CLI, MCP, and spiders as operator surfaces layered on top of the same routing logic, not three unrelated features. 7. Route honestly to Playwright/Crawlee-style automation, external unblockers, or another extraction service when Scrapling is no longer the best fit.

References

• references/intake-packets-and-route-outs.md
• references/fetchers-and-sessions.md
• references/parser-and-adaptive.md
• references/cli-and-mcp.md
• references/spiders.md
• scripts/install.sh
• scripts/run-extract.sh
• scripts/run-mcp.sh
• Scrapling GitHub Repository
• Scrapling Documentation

CLI and MCP

Scrapling's CLI covers install, shell, extraction, and MCP server startup.

Install and shell

pip install "scrapling[shell]"
scrapling install
scrapling shell
scrapling shell -c 'Fetcher.get("https://example.com").css("title::text").get()'

scrapling shell is useful when the user wants an interactive REPL backed by IPython instead of a standalone script.

Extract commands

The extract group supports:

• get
• post
• put
• delete
• fetch
• stealthy-fetch

Output format follows the target file extension:

• .md for Markdown conversion
• .html for raw HTML
• .txt for cleaner text content

Examples:

scrapling extract get "https://example.com" article.md
scrapling extract post "https://api.example.com/search" result.html --json '{"query":"shoes"}'
scrapling extract fetch "https://app.example.com" dashboard.md --network-idle
scrapling extract stealthy-fetch "https://protected.example.com" data.txt --solve-cloudflare

Useful shared flags:

• -s, --css-selector
• --proxy
• --timeout
• -H, --headers or --extra-headers

Useful browser flags:

• --headless or --no-headless
• --network-idle
• --wait
• --wait-selector
• --real-chrome

Stealth-only flags:

• --solve-cloudflare
• --block-webrtc
• --allow-webgl or --block-webgl
• --hide-canvas

Wrapper script

This skill includes:

bash scripts/run-extract.sh <mode> <url> <output_file> [extra args...]

Mode aliases:

• get
• post
• put
• delete
• fetch
• stealthy-fetch
• dynamic -> fetch
• stealth -> stealthy-fetch
• auto -> get

MCP server

Scrapling's MCP server exposes six main tools:

• get
• bulk_get
• fetch
• bulk_fetch
• stealthy_fetch
• bulk_stealthy_fetch

Local stdio mode:

pip install "scrapling[ai]"
scrapling install
scrapling mcp

Streamable HTTP mode:

scrapling mcp --http --host 0.0.0.0 --port 8000

Wrapper:

bash scripts/run-mcp.sh --http --host 127.0.0.1 --port 8000

Minimal Claude-style config:

{
  "mcpServers": {
    "ScraplingServer": {
      "command": "scrapling",
      "args": ["mcp"]
    }
  }
}

Operational guidance

• Start with stdio MCP for local agent use
• Use HTTP transport only when the environment already expects streamable HTTP
• Prefer CSS selectors in extraction commands to avoid returning giant documents
• Use .md output for model-friendly inspection unless raw HTML is necessary

Fetchers and Sessions

Scrapling has three main fetching tiers. Treat them as an escalation ladder, not interchangeable defaults.

1. Fetcher

Use Fetcher or FetcherSession when plain HTTP is enough.

• Engine: curl_cffi
• Strengths: speed, low memory, browser fingerprint impersonation, stealthy headers, HTTP/3 support
• Best for: blogs, docs, APIs, mostly static HTML, lightweight paginated sites

Example:

from scrapling.fetchers import FetcherSession

with FetcherSession(impersonate="chrome") as session:
    page = session.get("https://quotes.toscrape.com/", stealthy_headers=True)
    quotes = page.css(".quote .text::text").getall()

Useful knobs:

• impersonate
• stealthy_headers
• proxy
• timeout
• selector_config

2. DynamicFetcher

Use DynamicFetcher or DynamicSession when a browser is required but the target is not especially hostile.

• Engine: Playwright-backed Chromium or local Chrome
• Strengths: JavaScript rendering, waiting controls, page_action, real browser option
• Best for: SPAs, dashboards, lazy-loaded lists, small automation steps

Common options:

• headless
• disable_resources
• network_idle
• wait
• wait_selector
• real_chrome
• page_action
• proxy

page_action assumes some familiarity with Playwright's page API. Do not present it as a no-knowledge shortcut.

Example:

from scrapling.fetchers import DynamicSession

with DynamicSession(headless=True, network_idle=True) as session:
    page = session.fetch("https://example.com/app")
    data = page.css(".row .name::text").getall()

3. StealthyFetcher

Use StealthyFetcher or StealthySession when anti-bot systems interfere with normal rendering.

• Engine: patchright-backed browser stack
• Strengths: harder fingerprint evasion, Cloudflare Turnstile or interstitial solving, browser hardening toggles
• Best for: protected commerce, travel, ticketing, and sites that return blocks or empty states to simpler fetchers

Common options beyond DynamicFetcher:

• solve_cloudflare
• block_webrtc
• allow_webgl
• hide_canvas
• max_pages

Treat anti-bot handling as a documented capability, not a guarantee.

Example:

from scrapling.fetchers import StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:
    page = session.fetch("https://nopecha.com/demo/cloudflare")
    links = page.css("#padded_content a::attr(href)").getall()

Session guidance

Prefer session classes when:

• You need repeated requests with the same config
• You want connection, cookie, or browser reuse
• You want a rotating pool of browser tabs
• You are building a spider or multi-step workflow

Use one-off class methods only for short probes or tiny scripts.

Quick decision rule

1. Start with Fetcher 2. If the page needs JavaScript, move to DynamicFetcher 3. If the rendered result is still blocked, move to StealthyFetcher

Do not jump straight to stealth unless there is evidence that simpler paths fail.

Intake Packets and Route-outs

Use this note when the front door needs to stay small.

1. Intake packet

Ask for the smallest packet that decides the mode:

2. Quick routing table

3. Honest route-outs

Route out instead of stretching Scrapling when the user actually needs:

• Direct API interception or request hooks before navigation → consider Playwright/Crawlee-style browser automation or direct HTTP requests if the real data source is JSON/API, not the rendered DOM.
• Reliable managed anti-bot infrastructure → consider external proxy/unblocker providers rather than implying StealthyFetcher will always succeed.
• Large-scale distributed crawling and queue orchestration → consider a crawler framework or existing crawl stack if Scrapling spiders are no longer the smallest practical abstraction.
• Fast LLM-ready page ingestion without custom extraction logic → consider an agent-facing extraction service when deterministic custom scraping is not required.

4. Failure-language guidance

Say this explicitly when needed:

• "Start with the lightest mode that proves the data is reachable."
• "Stealth is a later escalation, not the default."
• "Adaptive scraping helps with selector drift, not auth or anti-bot guarantees."
• "If the real source of truth is an API response, route to direct request interception instead of forcing DOM scraping."
• "If the site still fails after honest stealth attempts, escalate to a managed unblocker or different stack rather than promising more flags will fix it."

Parser and Adaptive Scraping

Every fetcher returns a Response object, and Response inherits Scrapling's Selector parsing engine.

Selection primitives

Supported selection patterns:

• CSS selectors: page.css(".product")
• XPath selectors: page.xpath("//article")
• Text nodes: page.css("h1::text").get()
• Attribute values: page.css("a::attr(href)").getall()
• Text search: find_by_text(...)
• Regex search: find_by_regex(...)
• Similar elements: element.find_similar(...)

Scrapling's docs are centered on HTML parsing. XML feeds are not the current target surface.

Useful reminders:

• css() and xpath() return selector collections
• .get() and .getall() are the most convenient for text or attribute extraction
• find_similar() is useful after locating one representative item by text or regex

Adaptive scraping

Adaptive scraping lets Scrapling relocate changed elements without using an LLM.

Two phases:

1. Save an element fingerprint 2. Match a future element with similar properties

CSS or XPath flow

from scrapling.fetchers import Fetcher

Fetcher.configure(adaptive=True)
page = Fetcher.get("https://example.com")

original = page.css("#primary-cta", auto_save=True)
recovered = page.css("#primary-cta", adaptive=True)

Key points:

• adaptive=True must be enabled globally or in selector config
• auto_save=True stores the current matched element
• later, adaptive=True tells Scrapling to relocate the element if the original selector stops matching

Manual save and relocate flow

Use this when the original element was found by text, regex, or another indirect method.

element = page.find_by_text("Tipping the Velvet", first_match=True)
page.save(element, "featured-book")
saved = page.retrieve("featured-book")
matches = page.relocate(saved, selector_type=True)

adaptive_domain

Use adaptive_domain when equivalent pages live on different domains or archival hosts and should share the same adaptive fingerprint store.

This is especially useful for:

• archived vs live versions of the same site
• domain migrations
• CDN or locale host changes

Adaptive scraping is not automatic magic:

• you must enable adaptive=True
• you must save an element first through auto_save=True or manual save()
• relocation quality still depends on the surviving similarity of the target element

Similar-element discovery

find_similar() is useful when you have one known good element and need its peers.

Typical pattern:

1. find a product link by text 2. call .find_similar() on that element 3. iterate the sibling product cards

This is often easier than writing brittle container selectors from scratch.

Practical recommendations

• Use CSS first; drop to XPath when CSS becomes awkward
• Save adaptive fingerprints only for selectors that repeatedly break
• Keep identifiers meaningful when using manual adaptive storage
• Prefer text or attribute extraction over raw HTML blobs unless structure inspection is required

Spiders

Scrapling spiders are a Scrapy-inspired async crawling framework built on top of Scrapling's fetchers and parser.

When to switch from fetchers to spiders

Use a spider when the job needs:

• multiple pages and link following
• concurrent request scheduling
• retry and blocked-response handling
• checkpoint pause or resume
• different session types in one crawl

For one or two isolated pages, plain fetchers are simpler.

Minimal spider

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10

    async def parse(self, response: Response):
        for quote in response.css(".quote"):
            yield {
                "text": quote.css(".text::text").get(),
                "author": quote.css(".author::text").get(),
            }

        next_page = response.css(".next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page)

result = QuotesSpider().start()
result.items.to_json("quotes.json")

Mental model

The spider system is composed of:

• Spider: your crawl definition
• Scheduler: queued request storage with deduplication
• Crawler Engine: concurrency and callback orchestration
• Session Manager: routes requests to the correct fetcher session
• checkpoint storage: saves pending requests and seen fingerprints to disk

Sessions inside spiders

Spiders can use multiple named sessions, including:

• FetcherSession
• AsyncDynamicSession
• AsyncStealthySession

Requests choose the session by sid, which lets the same spider mix cheap HTTP paths with heavier browser paths only where needed.

Pause and resume

Use crawldir when the crawl must resume after interruption.

• Pending requests and seen fingerprints are checkpointed
• Graceful shutdown writes a final checkpoint
• Restarting with the same crawldir resumes instead of rebuilding start_requests()

This is the Scrapling equivalent of job-directory style crawling.

Hooks and control points

Useful spider customization areas include:

• start_requests()
• parse() and additional callbacks
• blocked-response detection or retry logic
• item export through result.items.to_json() or .to_jsonl()
• streaming flows via async for item in spider.stream()

Practical recommendations

• Keep concurrency modest until the target site proves stable
• Route only the hard pages through stealth sessions
• Use allowed_domains and explicit follow logic to keep scope clean
• Turn on checkpointing for long-running crawls or unstable networks

Experiment 0 - baseline

Score: 4/6 Change: Initial draft only. Reasoning: Establish the baseline before mutating the skill. Result: Covered the high-level surface area but was too light on adaptive relocation and CLI operational specifics. Remaining failures: Adaptive details and concrete CLI guidance were not explicit enough.

Experiment 1 - keep

Score: 6/6 Change: Added the fetcher escalation ladder, adaptive examples, wrapper scripts, MCP tool framing, and clearer spider-switch guidance. Reasoning: The baseline risk was that the skill named features without making them operationally selectable. Result: All six binary evals passed. Remaining failures: None in the current eval suite.