Ai Crawler Policy
Help your agent draft and refine robots.txt, crawler rules, and AI-bot policies so your site is visible to humans without oversharing to scrapers.
Overview
io.github.lazymac2x/ai-crawler-policy is a MCP server for the Launch phase that helps agents craft web crawler and AI-bot access policies via a Cloudflare Workers remote endpoint.
What is this MCP server?
- Streamable-http MCP remote on Cloudflare Workers (v1.0.0)
- Agent-assisted crawler and AI-bot policy wording
- Targets solo sites and docs that need clear allow/disallow rules
- No local server—connect via api.lazy-mac.com endpoint
- Repository published on GitHub for the API implementation
- 1 remote MCP URL (streamable-http)
- Manifest version 1.0.0
- Cloudflare Workers MCP server: ai-crawler-policy
What problem does it solve?
Small teams struggle to keep robots.txt and AI crawler rules accurate as new bots appear, risking SEO loss or unwanted scraping.
Who is it for?
Indie founders launching marketing sites or docs who want agent-guided SEO and AI-crawler rules without hiring an SEO agency.
Skip if: Apps with no public web surface, or enterprises that manage crawl policy entirely in a centralized security edge platform.
What do I get? / Deliverables
After adding the MCP remote, your agent can produce deployable crawler policy text tuned to your routes and launch goals.
- Draft robots.txt or policy snippets ready to deploy
- Agent-explained tradeoffs between indexation and bot blocking
- Updated rules you can revisit when new crawlers emerge
Recommended MCP Servers
Journey fit
Crawler policy is a Launch concern because it directly affects indexing, AI search visibility, and how bots interact with a live site. SEO subphase includes crawl allowances, meta robots, and bot-specific directives that shape discovery and GEO outcomes.
How it compares
Launch-time MCP policy assistant, not a runtime bot-detection appliance or generic web-scraping skill.
Common Questions / FAQ
Who is ai-crawler-policy for?
Solo builders shipping public websites who need clear robots and AI-bot rules and already use MCP-enabled coding agents.
When should I use ai-crawler-policy?
Use it at Launch when you publish or update a site and must balance SEO visibility with blocking unwanted AI crawlers.
How do I add ai-crawler-policy to my agent?
Register https://api.lazy-mac.com/ai-crawler-policy/mcp as a streamable-http MCP server in your agent client.