Robots Txt
- 906 installs
- 787 repo stars
- Updated June 9, 2026
- kostja94/marketing-skills
robots-txt is a marketing-skills agent skill that helps developers generate, audit, and optimize robots.txt files controlling how search engines and AI crawlers access and index site paths.
About
robots-txt is a skill from the kostja94/marketing-skills repository focused on robots.txt creation, validation, and optimization for developer-owned websites. It guides agents through User-agent rules, Allow and Disallow path directives, crawl-delay settings, and sitemap references so search engines and AI crawlers respect intended access boundaries. Developers reach for robots-txt when launching a new domain, blocking staging environments from indexing, allowing AI crawlers selectively, or fixing accidental noindex caused by overly broad Disallow rules. The skill fits technical SEO workflows where a correct robots.txt at the domain root prevents wasted crawl budget and unintended exposure of private routes.
- Generates production-ready robots.txt with proper Allow/Disallow, Sitemap, and Clean-param directives
- Audits existing robots.txt for accidental blocks of search engines or AI crawlers like GPTBot
- Provides AI-crawler-specific blocking strategies while preserving essential search visibility
- Differentiates crawl control (robots.txt) from index control (noindex)
- Reads project-context.md to align rules with your site's indexing goals
Robots Txt by the numbers
- 906 all-time installs (skills.sh)
- +6 installs in the week ending Jul 29, 2026 (Skillselion tracking)
- Ranked #476 of 1,879 Marketing & SEO skills by installs in the Skillselion catalog
- Security screen: LOW risk (skills.sh audit)
- Data as of Jul 31, 2026 (Skillselion catalog sync)
npx skills add https://github.com/kostja94/marketing-skills --skill robots-txtAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 906 |
|---|---|
| repo stars | ★ 787 |
| Security audit | 3 / 3 scanners passed |
| Last updated | June 9, 2026 |
| Repository | kostja94/marketing-skills ↗ |
How do you write a correct robots.txt file?
Generate, audit, and optimize robots.txt files that control how search engines and AI crawlers access their site.
Who is it for?
Developers launching or maintaining websites who need technically correct robots.txt crawl control for SEO and AI bots.
Skip if: Teams relying solely on meta robots tags without root-level crawl policy or apps with no public web surface.
When should I use this skill?
A site needs robots.txt created, crawlers index blocked paths, or AI bot access rules must be updated.
What you get
Validated robots.txt with User-agent rules, path directives, and optional Sitemap URL references.
- robots.txt file with crawl directives
- Audit report of existing robots.txt issues
Files
SEO: Local
Guides local SEO: Google Business Profile, NAP consistency, and citation building. Businesses with accurate NAP across 40+ authoritative sites see ~19% higher visibility in Google Maps. Use this skill when optimizing for local search, setting up GBP, or auditing citations.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Initial Assessment
Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read Sections 4 (Audience), 5 (Website).
Identify: 1. Business type: Storefront vs service-area 2. Location: Single or multiple 3. Current listings: Existing GBP, directories
NAP Consistency
NAP = Name, Address, Phone. Critical for local rankings.
| Rule | Guideline |
|---|---|
| Exact match | "Street" vs "St." or "LLC" inconsistency = Google may treat as different entities |
| Fix first | Audit and fix inconsistencies before adding new citations |
| Tools | BrightLocal, Whitespark, Moz Local for audit |
Google Business Profile
| Element | Guideline |
|---|---|
| Address | Physical address; no P.O. boxes |
| Description | 750 chars; primary keywords in first 100 |
| Hours | Accurate; seasonal availability |
| Category | Primary category matches business type |
| Service-area | Hide address if no storefront; define service areas |
Citation Building
Targeted precision over submitting to every directory.
Priority order: 1. Google Business Profile 2. Apple Maps 3. Yelp, Bing Places, Facebook 4. Better Business Bureau, Foursquare, Nextdoor 5. Niche directories (Healthgrades, Angi, etc.)
Citation Audit
- Incorrect or outdated data
- Duplicate entries
- Missing listings on key directories
Fix before adding; compounding errors harm rankings.
Output Format
- NAP (exact format for consistency)
- GBP optimization checklist
- Citation priority list
- Audit findings (if applicable)
Related Skills
- geo: GEO for AI search; local + AI overlap
- localization-strategy: Multilingual; local + i18n
- directory-submission: Directory listings; different from local citations
Related skills
Seo AuditRun structured SEO audits on their SaaS site or content hub and receive a prioritized action plan.173k42.2kCopywritingGenerate, rewrite, or strengthen persuasive website and landing-page copy that converts visitors into users.164k42.2k
Viral Short FormQuickly generate high-retention hooks, scripts, and outlines for TikTok, Reels, YouTube Shorts, and carousels.132k67Viral HooksWrite and critique viral hooks for short-form video opening sequences.123k67Viral Captions And CtasOptimize social media captions and CTAs for viral short-form video reach and saves.123k67Viral Youtube ShortsWrite and diagnose YouTube Shorts for Shorts Feed and long-form funnel.123k67FAQ
What does the robots-txt skill help developers create?
The robots-txt skill helps developers create and optimize robots.txt files with User-agent blocks, Allow and Disallow path rules, and Sitemap references that control how search engines and AI crawlers access a site.
When should developers use the robots-txt skill?
Developers should use the robots-txt skill when launching a site, blocking staging from indexing, allowing or blocking AI crawlers, or fixing crawl issues caused by incorrect Disallow rules in an existing robots.txt.
Is Robots Txt safe to install?
skills.sh reports 3 of 3 security scanners passed. Review the Security Audits panel on this page before installing in production.