AI Crawler Policy MCP
- Updated May 17, 2026
- lazymac2x/ai-crawler-policy-api
io.github.lazymac2x/ai-crawler-policy is a MCP server that helps agents craft web crawler and AI-bot access policies via a Cloudflare Workers remote endpoint.
About
io.github.lazymac2x/ai-crawler-policy is a hosted MCP server that lets coding agents help you define how search engines and AI crawlers may access your public pages. As developers move from Build to Launch, misconfigured robots rules either leak staging paths to indexers or block legitimate discovery—including emerging AI search crawlers. This remote integration exposes policy-oriented tools over streamable HTTP so Claude Code, Cursor, or Codex can propose robots.txt snippets, discuss vendor-specific user agents, and align policies with your sitemap and marketing pages. It complements static hosting on Cloudflare or elsewhere: you remain responsible for deploying the final files. The server is particularly relevant when you ship content-heavy SaaS, docs sites, or blogs where GEO and SEO stakes are high. It is policy guidance via MCP, not a WAF replacement or automated crawler firewall.
- Streamable-http MCP remote on Cloudflare Workers (v1.0.0)
- Agent-assisted crawler and AI-bot policy wording
- Targets solo sites and docs that need clear allow/disallow rules
- No local server—connect via api.lazy-mac.com endpoint
- Repository published on GitHub for the API implementation
AI Crawler Policy MCP by the numbers
- Data as of Aug 10, 2026 (Skillselion catalog sync)
claude mcp add --transport http ai-crawler-policy https://api.lazy-mac.com/ai-crawler-policy/mcpAdd your badge
Show developers this MCP server is listed on Skillselion. Paste this into your README.
| Transport | HTTP |
|---|---|
| Auth | None |
| Last updated | May 17, 2026 |
| Repository | lazymac2x/ai-crawler-policy-api ↗ |
What it does
Help your agent draft and refine robots.txt, crawler rules, and AI-bot policies so your site is visible to humans without oversharing to scrapers.
Who is it for?
Best when you're launching marketing sites or docs and want agent-guided SEO and AI-crawler rules without hiring an SEO agency.
Skip if: Apps with no public web surface, or enterprises that manage crawl policy entirely in a centralized security edge platform.
What you get
After adding the MCP remote, your agent can produce deployable crawler policy text tuned to your routes and launch goals.
- Draft robots.txt or policy snippets ready to deploy
- Agent-explained tradeoffs between indexation and bot blocking
- Updated rules you can revisit when new crawlers emerge
By the numbers
- 1 remote MCP URL (streamable-http)
- Manifest version 1.0.0
- Cloudflare Workers MCP server: ai-crawler-policy
README.md
ai-crawler-policy-api
Cloudflare Workers MCP server: ai-crawler-policy. REST + MCP (JSON-RPC 2.0). Live: api.lazy-mac.com/ai-crawler-policy
Recommended MCP Servers
1stDibs MCPMCP server for browsing and searching items on 1stDibs marketplace.
2Captcha MCPFull 2Captcha API (43 tools) for Claude Code: 31 captcha solvers + management + webhook.1
4fetch MCPFetch a URL and get clean Markdown with metadata.
Acrawl MCPAutonomous web crawler.11
Agentfetch MCPToken-budgeted web fetch for AI agents — auto-routes Jina, FireCrawl, Trafilatura, PDF.3AgenticTotem Web Extractor MCPAI web extraction: send URLs + a JSON Schema, get clean structured data.How it compares
Launch-time MCP policy assistant, not a runtime bot-detection appliance or generic web-scraping skill.
FAQ
Who is ai-crawler-policy for?
Developers shipping public websites who need clear robots and AI-bot rules and already use MCP-enabled coding agents.
When should I use ai-crawler-policy?
Use it at Launch when you publish or update a site and must balance SEO visibility with blocking unwanted AI crawlers.
How do I add ai-crawler-policy to my agent?
Register https://api.lazy-mac.com/ai-crawler-policy/mcp as a streamable-http MCP server in your agent client.