Now liveThe Skillselion MCP - thousands of ranked skills, loaded into your agent mid-task. No install.Get it →
infrasity-labs avatar

Llms Txt Checker

infrasity-labs/dev-gtm-claude-skills

  • 80 installs
  • 93 repo stars
  • Updated June 28, 2026
  • infrasity-labs/dev-gtm-claude-skills

LLMs.txt Checker is an agent skill that audits robots.txt, llms.txt, and llms-full.txt on a domain and reports AI-readiness findings with fixes.

About

LLMs.txt Checker is an agent skill that treats AI discoverability as something you can measure, not assume. You supply a domain or URL; the skill normalizes it, fetches robots.txt, llms.txt, and llms-full.txt, and audits each response against a structured checklist spanning file structure, content completeness, and signals that matter for tools like Claude Code, Cursor, and Copilot. The output is a readable report with pass, warn, and fail rows and concrete remediation guidance—useful when developer marketing or GEO teams need evidence before a docs launch, after a redesign, or when competitors already ship llms-full content. It is a checker, not a generator: it does not rewrite your site for you, but it tells you exactly what is missing or misconfigured so you can fix files and cross-references quickly.

  • Normalizes a domain or URL and probes robots.txt, llms.txt, and llms-full.txt via curl
  • Scores each file against a structured checklist covering structure, completeness, and AI-readiness signals
  • Returns a formatted report with pass, warn, and fail findings plus actionable fixes
  • Targets DevRel and technical writers doing pre-launch or post-redesign AI discoverability validation
  • Activates from /dev-gtm llms-txt-check or natural-language requests about llms.txt and AI-readiness

Llms Txt Checker by the numbers

  • 80 all-time installs (skills.sh)
  • +4 installs in the week ending Jul 25, 2026 (Skillselion tracking)
  • Ranked #1,179 of 1,881 Marketing & SEO skills by installs in the Skillselion catalog
  • Data as of Jul 26, 2026 (Skillselion catalog sync)
npx skills add https://github.com/infrasity-labs/dev-gtm-claude-skills --skill llms-txt-checker

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

Listed on Skillselion
Installs80
repo stars★ 93
Last updatedJune 28, 2026
Repositoryinfrasity-labs/dev-gtm-claude-skills ↗

What it does

Audit a site’s robots.txt, llms.txt, and llms-full.txt so AI coding tools and answer engines can discover your docs correctly.

Who is it for?

Best when you're validating docs.example.com-style properties before launch, GEO experiments, or agent-tooling announcements.

Skip if: Skip if you only need on-page keyword SEO without llms.txt/robots governance, or sites where you cannot allow outbound HTTP fetches to the target domain.

When should I use this skill?

You provide a domain or URL and ask about llms.txt, llms-full.txt, robots.txt, or AI-readiness for docs.

What you get

You receive a scored audit across three probed files with pass/warn/fail detail and specific fixes so you can publish compliant llms.txt assets before distribution and GEO work.

  • Formatted AI-readiness audit report
  • Per-file pass/warn/fail findings
  • Actionable remediation list

By the numbers

  • Probes 3 AI-readiness files: robots.txt, llms.txt, and llms-full.txt
  • Structured checklist with pass/warn/fail report formatting

Files

SKILL.mdMarkdownGitHub ↗

LLMs.txt Checker Skill

Audits any domain's AI-readiness by using curl to directly probe robots.txt, llms.txt, and llms-full.txt, then scores each file against a structured checklist and delivers a formatted report with pass/warn/fail findings and actionable fixes.

The user provides only a domain (e.g. anthropic.com or docs.example.com). Claude uses bash_tool with curl commands to directly probe the domain — no guessing, no page-scraping required.

---

How it works

Instead of relying on web_fetch and hoping links surface organically, this skill uses curl via bash_tool to directly request the well-known paths for robots.txt, llms.txt, and llms-full.txt. This is reliable, fast, and works regardless of how the site is built.

The curl commands follow HTTP redirects, capture response codes, and save content to temp files for auditing.

---

Step-by-Step Workflow

Step 1: Normalise the domain

Take the user-provided input and strip any trailing slashes, http://, https://, or path segments to get a clean base domain (e.g. docs.anthropic.com). If the user provides a full URL like https://docs.anthropic.com/en/home, extract just docs.anthropic.com.

---

Step 2: Fetch all three files via curl

Run the following curl commands using bash_tool. Use -L to follow redirects, -s for silent mode, -o to save content, -w to capture HTTP status codes, and a reasonable timeout (--max-time 10).

DOMAIN="<normalised-domain>"
# Ensure files exist to prevent "No such file or directory" errors if curl fails
touch /tmp/robots.txt /tmp/llms.txt /tmp/llms-full.txt

# Fetch robots.txt
curl -L -s -o /tmp/robots.txt -w "%{http_code}" --max-time 10 "https://$DOMAIN/robots.txt" > /tmp/robots_status.txt

# Fetch llms.txt
curl -L -s -o /tmp/llms.txt -w "%{http_code}" --max-time 10 "https://$DOMAIN/llms.txt" > /tmp/llms_status.txt

# Fetch llms-full.txt
curl -L -s -o /tmp/llms-full.txt -w "%{http_code}" --max-time 10 "https://$DOMAIN/llms-full.txt" > /tmp/llms_full_status.txt

# Print status codes and file sizes for inspection
echo "robots.txt: $(cat /tmp/robots_status.txt) | $(wc -c < /tmp/robots.txt) bytes"
echo "llms.txt:   $(cat /tmp/llms_status.txt) | $(wc -c < /tmp/llms.txt) bytes"
echo "llms-full.txt: $(cat /tmp/llms_full_status.txt) | $(wc -c < /tmp/llms-full.txt) bytes"

Interpret the HTTP status codes:

  • 200 → file exists, read and audit the content
  • 301/302 → followed automatically by -L; final destination counts
  • 404 → file does not exist at this path
  • 403/429/5xx → server-side block or error; note it explicitly
  • 000 → connection failed (domain unreachable or timeout)

---

Step 3: Read and classify results

After the curl commands complete, read the saved files:

if [ "$(cat /tmp/robots_status.txt)" = "200" ]; then
  cat /tmp/robots.txt
fi
if [ "$(cat /tmp/llms_status.txt)" = "200" ]; then
  cat /tmp/llms.txt
fi
if [ "$(cat /tmp/llms_full_status.txt)" = "200" ]; then
  head -200 /tmp/llms-full.txt
  wc -l /tmp/llms-full.txt   # get total line count
  wc -c /tmp/llms-full.txt   # get total byte size
fi

Case A — Both `llms.txt` (200) AND `llms-full.txt` (200)

  • Both files fetched successfully; proceed to the Audit Checklist (Step 4)

Case B — Only `llms.txt` (200), `llms-full.txt` returned 404

  • Audit llms.txt
  • Scan its content for any internal reference to llms-full.txt (it may be hosted at a non-standard path)
  • If a custom path is found → curl that path and audit it
  • If not found → report llms-full.txt as absent and not referenced

Case C — `llms.txt` returned 404

  • Report that neither file is present at the standard paths
  • Note whether robots.txt gave any hints (some sites reference llms.txt inside robots.txt)
  • Report clearly to the user (see Response Templates section below)

robots.txt (always check regardless of Case)

  • Even if llms.txt is missing, always read and audit robots.txt for AI-access signals

---

Step 4: Audit the files

llms.txt Audit

Check for the following. Mark each ✅ or ❌:

Structure

  • [ ] Starts with a single # H1 title (site/product name)
  • [ ] Has a > blockquote summary immediately below H1 (1–2 sentence description)
  • [ ] Uses ## H2 sections to group links (e.g. Docs, API Reference, Guides, OpenAPI Specs)
  • [ ] Each link follows format: - [Page Title](https://absolute-url): brief description
  • [ ] Has an ## Optional section for secondary/non-essential content (not required but best practice)
  • [ ] No nested headings inside H2 link sections
  • [ ] No images, HTML, or tables (plain markdown only)

Content completeness

  • [ ] Core product/feature pages are listed
  • [ ] API reference pages are included (if applicable)
  • [ ] Getting started / quickstart pages included
  • [ ] SDK/integration guides included (if applicable)
  • [ ] Link descriptions are meaningful (not just page titles repeated)
  • [ ] All links use absolute URLs (not relative paths)
  • [ ] No broken or 404 links visible

AI-readiness signals

  • [ ] References llms-full.txt (either directly or in a Documentation Sets section)
  • [ ] Segmented sets for different use cases (advanced but excellent — e.g. Scalekit's topic-specific .txt files)
llms-full.txt Audit (if available)
  • [ ] File exists and is non-empty
  • [ ] Contains full page content (not just links)
  • [ ] Has clear document boundary markers between pages (e.g. --- or # DOCUMENT BOUNDARY)
  • [ ] Each section has a Source: URL reference
  • [ ] Content is clean markdown (no raw HTML, no JS artifacts)
  • [ ] Reasonably sized (warn if extremely large — may exceed LLM context windows)
robots.txt Signal (check opportunistically)

If robots.txt was surfaced during the process:

  • [ ] User-agent: * with Allow: / — all bots permitted
  • [ ] ai-input=yes — explicitly permits AI agents to use content
  • [ ] ai-train=no — training blocked (common and acceptable)
  • [ ] Any Disallow rules that would block AI crawlers

---

Step 5: Deliver the report

Structure the output as:

## LLMs.txt Audit: [domain]

### Discovery
[What was found and how it was surfaced]

### llms.txt — ✅ Found / ❌ Not Found
[Audit results with ✅/❌ per checklist item]
[Notable strengths]
[Issues found]

### llms-full.txt — ✅ Found / ❌ Not Found / ⚠️ Not Referenced
[Audit results or explanation]

### robots.txt Signal
[If available — what it says about AI access]

### Summary & Recommendations
[3–5 actionable bullets]

---

Response Templates

Neither llms.txt nor llms-full.txt surfaced

Neither llms.txt nor llms-full.txt was discoverable from the provided URL.

>

This means AI agents and LLMs browsing your docs will have no structured index to work from — they'll need to crawl individual pages or guess at your content structure.

>

To fix this, surface the llms.txt URL somewhere Claude (and other AI tools) can see it when fetching your page. Good options:
- Add it to your page footer (e.g. LLM usage: /llms.txt)
- Include it in a blockquote at the top of your docs homepage or .md page version (e.g. > Documentation index available at: https://yourdomain.com/llms.txt)
- Reference it in your robots.txt or a <meta> tag

>

Once it's linked from a page that AI agents naturally land on, it becomes discoverable automatically.

llms.txt found but llms-full.txt not referenced

llms.txt was found and audited. However, llms-full.txt was not referenced anywhere in the file.

>

llms-full.txt is the companion file containing the full content of all documentation pages in a single file — useful for AI coding assistants (Cursor, Claude Code, Copilot) that need deep context without fetching dozens of individual pages.

>

To add it: Reference it in your llms.txt under a ## Documentation Sets section or similar, like:
```
- Complete documentation: full content of all pages
```
If you're on Mintlify, it's auto-generated — just make sure it's linked.

---

Key facts to keep in mind

  • Mintlify auto-generates both llms.txt and llms-full.txt for all projects, and adds HTTP headers (Link: </llms.txt>; rel="llms-txt") for discovery
  • Fern also auto-generates both files
  • Starlight (Astro) does not auto-generate — must be added manually
  • GitBook auto-generates llms.txt
  • The llms.txt standard was proposed by Jeremy Howard (fast.ai) in September 2024
  • llms-full.txt is not part of the original spec but has become widely adopted as the companion file
  • No major AI crawler has officially committed to following these files, but Cursor, Claude Code, and similar tools actively use them

Related skills

coreyhaines31Seo AuditRun structured SEO audits on their SaaS site or content hub and receive a prioritized action plan.167k41.1kcoreyhaines31CopywritingGenerate, rewrite, or strengthen persuasive website and landing-page copy that converts visitors into users.158k41.1kvyralcontentViral Short FormQuickly generate high-retention hooks, scripts, and outlines for TikTok, Reels, YouTube Shorts, and carousels.132k64vyralcontentViral HooksWrite and critique viral hooks for short-form video opening sequences.123k64vyralcontentViral Captions And CtasOptimize social media captions and CTAs for viral short-form video reach and saves.123k64vyralcontentViral Youtube ShortsWrite and diagnose YouTube Shorts for Shorts Feed and long-form funnel.123k64

How it compares

Use this checker for live llms.txt/robots probes and structured scoring—not as a replacement for writing llms-full content or a full technical SEO crawl suite.

FAQ

Who is llms-txt-checker for?

Developer marketers, GEO/AEO practitioners, and technical writers who need a fast, repeatable audit that AI coding tools can actually discover their documentation.

When should I use llms-txt-checker?

At Launch/GEO before campaigns; during Build/docs when adding llms.txt to a repo; at Grow/content when refreshing docs discoverability; anytime you have a domain and ask about llms.txt or AI-readiness.

Is llms-txt-checker safe to install?

Review the Security Audits panel on this Prism page and treat the skill as network-active—it fetches public URLs on domains you specify via curl-like requests.

Marketing & SEOseocontentdistribution
Browse by category
AI & AgentsFrontend & DesignBackend & DataDev ToolsAutomationTesting & ReviewSecurityMonitoring & CloudDocs & Planning
Design & frontend
Design systemsAnimation & motionAccessibility (a11y)CSS & TailwindUI designUXDesign to codeshadcn/uiLanding pagesBest design skills
Most searched skills
find-skillscavemanfrontend-designbrainstormingsystematic-debuggingstop-slopemil-design-engui-ux-pro-maxkarpathy-guidelinesweb-design-guidelinesreact-best-practicesgrill-meimprovemicrosoft-foundryagent-browser
Browse all listings
All agent skillsAll MCP serversAll marketplacesAll plugins

This week in AI coding

Five minutes, every Monday - the tools, releases and tactics for developers.

unsubscribe anytime.