ShadowCrawl
Connect your agent to a hosted ShadowCrawl MCP endpoint to bypass Cloudflare, DataDome, and LinkedIn-style blocks without running Rust locally.
Overview
ShadowCrawl (shadow-crawl remote) is an MCP server for the Idea phase that offers hosted stealth search and scraping with cyborg HITL against major anti-bot stacks.
What is this MCP server?
- Streamable HTTP remote at shadowcrawl--devshero.run.tools for zero-local-server setups
- Cyborg HITL approach when automation cannot complete on hardened targets
- Bypass positioning for Cloudflare, DataDome, and LinkedIn-class protections
- Same ShadowCrawl product family as the stdio/Rust registry entry (version 2.0.0-rc remote)
- MCP remotes block in server schema for drop-in agent configuration
- Remote URL: https://shadowcrawl--devshero.run.tools (streamable-http)
- Catalog version 2.0.0-rc for this remote registry name
Community signal: 66 GitHub stars.
What problem does it solve?
You do not want to operate a local Rust scraper but still need your agent to read bot-gated pages during research.
Who is it for?
Builders who prefer remote MCP endpoints and need LinkedIn or enterprise-gated pages in agent-driven research loops.
Skip if: Air-gapped environments, strict data-residency requirements, or scraping that must run entirely inside your own VPC without third-party remotes.
What do I get? / Deliverables
Registering the streamable-http remote gives your agent ShadowCrawl tools immediately, with HITL when a target refuses unattended access.
- Agent-callable stealth search and scrape via hosted MCP
- HITL escalation path for non-robot or high-friction pages
Recommended MCP Servers
Journey fit
Remote MCP fits builders who want research tooling on day one of an idea without maintaining scraper infrastructure. Research is the default shelf because streamable-http ShadowCrawl is positioned for search and scrape during discovery, not for production app runtime.
How it compares
Hosted MCP remote for stealth browse/scrape, not the self-hosted io.github.DevsHero/ShadowCrawl stdio binary entry.
Common Questions / FAQ
Who is shadow-crawl for?
shadow-crawl is for developers using MCP remotes who want ShadowCrawl’s bypass and HITL scraping without managing the Rust server locally.
When should I use shadow-crawl?
Use the remote when you are researching markets or integrations and hits on Cloudflare, DataDome, or Linkedin block your agent’s normal fetch path.
How do I add shadow-crawl to my agent?
Add an MCP remote entry pointing to https://shadowcrawl--devshero.run.tools with type streamable-http in your client config, then enable the ShadowCrawl tools per provider auth instructions.