Site Crawlability
- 812 installs
- 787 repo stars
- Updated June 9, 2026
- kostja94/marketing-skills
site-crawlability is an SEO diagnostic skill that finds and fixes crawlability issues blocking search engines and AI crawlers from indexing a site for developers who need pages discoverable by Google and AI search bots.
About
site-crawlability is a kostja94/marketing-skills guide for diagnosing and fixing crawlability problems that prevent search engines and AI crawlers from properly indexing a website. The skill addresses technical blockers such as robots.txt misconfigurations, missing or malformed sitemaps, redirect chains, and server responses that stop bot access. Developers reach for site-crawlability when pages fail to appear in search or AI-generated answers despite published content, when GSC shows crawl errors, or when launching a site that must be accessible to both traditional crawlers and AI indexing bots.
- Diagnoses redirect chains, broken links, orphan pages and crawl budget waste
- Optimizes site architecture so every page is reachable within 3–4 clicks
- Provides AI crawler specific recommendations including SSR and GPTBot/ClaudeBot handling
- Recommends pagination over infinite scroll for better indexation
- Delivers prioritized fix list with internal linking opportunities
Site Crawlability by the numbers
- 812 all-time installs (skills.sh)
- +7 installs in the week ending Jul 28, 2026 (Skillselion tracking)
- Ranked #546 of 1,879 Marketing & SEO skills by installs in the Skillselion catalog
- Security screen: LOW risk (skills.sh audit)
- Data as of Jul 31, 2026 (Skillselion catalog sync)
npx skills add https://github.com/kostja94/marketing-skills --skill site-crawlabilityAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 812 |
|---|---|
| repo stars | ★ 787 |
| Security audit | 3 / 3 scanners passed |
| Last updated | June 9, 2026 |
| Repository | kostja94/marketing-skills ↗ |
How do you fix site crawlability for AI crawlers?
Diagnose and fix crawlability issues that prevent search engines and AI crawlers from properly indexing their site.
Who is it for?
Developers launching or auditing websites who need search engines and AI crawlers to fully index published pages without technical blockers.
Skip if: Developers focused solely on content writing or paid ads without any technical crawl or indexation requirements.
When should I use this skill?
The user reports crawl errors, indexing failures, robots.txt issues, or asks to make a site crawlable for search engines and AI crawlers.
What you get
Crawl error fixes, updated robots.txt and sitemap configuration, and verified bot-accessible page indexation
- Fixed robots.txt and sitemap
- Crawl error remediation checklist
Files
SEO: Local
Guides local SEO: Google Business Profile, NAP consistency, and citation building. Businesses with accurate NAP across 40+ authoritative sites see ~19% higher visibility in Google Maps. Use this skill when optimizing for local search, setting up GBP, or auditing citations.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Initial Assessment
Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read Sections 4 (Audience), 5 (Website).
Identify: 1. Business type: Storefront vs service-area 2. Location: Single or multiple 3. Current listings: Existing GBP, directories
NAP Consistency
NAP = Name, Address, Phone. Critical for local rankings.
| Rule | Guideline |
|---|---|
| Exact match | "Street" vs "St." or "LLC" inconsistency = Google may treat as different entities |
| Fix first | Audit and fix inconsistencies before adding new citations |
| Tools | BrightLocal, Whitespark, Moz Local for audit |
Google Business Profile
| Element | Guideline |
|---|---|
| Address | Physical address; no P.O. boxes |
| Description | 750 chars; primary keywords in first 100 |
| Hours | Accurate; seasonal availability |
| Category | Primary category matches business type |
| Service-area | Hide address if no storefront; define service areas |
Citation Building
Targeted precision over submitting to every directory.
Priority order: 1. Google Business Profile 2. Apple Maps 3. Yelp, Bing Places, Facebook 4. Better Business Bureau, Foursquare, Nextdoor 5. Niche directories (Healthgrades, Angi, etc.)
Citation Audit
- Incorrect or outdated data
- Duplicate entries
- Missing listings on key directories
Fix before adding; compounding errors harm rankings.
Output Format
- NAP (exact format for consistency)
- GBP optimization checklist
- Citation priority list
- Audit findings (if applicable)
Related Skills
- geo: GEO for AI search; local + AI overlap
- localization-strategy: Multilingual; local + i18n
- directory-submission: Directory listings; different from local citations
Related skills
Seo AuditRun structured SEO audits on their SaaS site or content hub and receive a prioritized action plan.173k42.2kCopywritingGenerate, rewrite, or strengthen persuasive website and landing-page copy that converts visitors into users.164k42.2k
Viral Short FormQuickly generate high-retention hooks, scripts, and outlines for TikTok, Reels, YouTube Shorts, and carousels.132k67Viral HooksWrite and critique viral hooks for short-form video opening sequences.123k67Viral Captions And CtasOptimize social media captions and CTAs for viral short-form video reach and saves.123k67Viral Youtube ShortsWrite and diagnose YouTube Shorts for Shorts Feed and long-form funnel.123k67FAQ
What crawlability issues does site-crawlability address?
The site-crawlability skill diagnoses robots.txt misconfigurations, sitemap gaps, redirect chains, and server responses that block search engines and AI crawlers from indexing site pages.
When should developers use site-crawlability?
Developers use site-crawlability when published pages fail to index, Google Search Console shows crawl errors, or AI search bots cannot access content due to technical blockers.
Is Site Crawlability safe to install?
skills.sh reports 3 of 3 security scanners passed. Review the Security Audits panel on this page before installing in production.