Llms Txt

llms.txt

Overview

AEO (Answer Engine Optimization) covers how your site is understood and cited by AI assistants (ChatGPT, Perplexity, Claude, etc.). `llms.txt` is the primary convention: a Markdown file at /llms.txt that gives models a curated map of your most important pages.

It complements (does not replace) robots.txt and sitemap.xml. It is not a W3C/IETF standard; treat it as durable, machine-readable site documentation you control.

Reference: llmstxt.org · GitHub spec repo

---

When to use llms.txt

Not a substitute for: SEO (use sitemap + structured data), crawl policy (use robots.txt), or full-text search indexing.

---

File location and delivery

• Canonical path: https://example.com/llms.txt (site root).
• Optional: subpath (e.g. /docs/llms.txt) or /.well-known/llms.txt (root is preferred).
• Content-Type: text/plain; charset=utf-8 (Markdown body; plain text MIME is conventional).
• Multilingual: one file per language root (e.g. /en/llms.txt, /es/llms.txt).

Static file

Place public/llms.txt in Next.js/Vite/static hosts. Served as-is at /llms.txt.

Dynamic route (Next.js App Router)

When content comes from a CMS or database, use a route handler:

// app/llms.txt/route.ts
export async function GET() {
  const content = buildLlmsTxt(); // assemble Markdown string
  return new Response(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, s-maxage=3600, stale-while-revalidate=86400',
    },
  });
}

Build the Markdown string in code; keep sections in spec order. Fetch dynamic slugs server-side (blog posts, docs) and append link lines.

---

Spec format (required order)

Per llmstxt.org, sections appear in this order:

1. H1 — project or site name (required; only mandatory section). 2. Blockquote — one short summary (> ...) with key context for interpreting the rest. 3. Body (optional) — paragraphs or lists with extra guidance; no headings in this block. 4. H2 sections — each section is a “file list” of markdown links. 5. `## Optional` (special) — secondary links agents may skip when context is limited.

Link line syntax

Each entry is a markdown list item:

- [Link title](https://absolute-url): Optional one-line description.

• [name](url) is required.
• : description is optional but strongly recommended.
• Use absolute URLs.
• Prefer links to `.md` versions of pages when available (same path + .md, or index.html.md for extensionless URLs).

Minimal example

# Acme Docs

> Acme is a task API for teams. REST + webhooks; auth via API keys.

Use the Quick Start first, then the API reference for endpoints.

## Docs

- [Quick start](https://docs.acme.com/quickstart.md): Install SDK and send your first request
- [API reference](https://docs.acme.com/api.md): All endpoints and error codes

## Examples

- [Node webhook handler](https://github.com/acme/examples/blob/main/webhook.ts): Verify signatures and process events

## Optional

- [Changelog](https://docs.acme.com/changelog.md): Release history

---

Companion conventions

Markdown mirrors (.md URLs)

Pages useful to LLMs should expose a clean Markdown copy at the same URL + `.md` (or index.html.md). Example: https://docs.fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md.

llms-full.txt / llms-ctx

• llms-full.txt — expanded file with page content inlined (single fetch for agents).
• llms-ctx.txt / llms-ctx-full.txt — generated via `llms_txt2ctx` CLI from llms.txt (XML-ish bundle for tools like Claude).

Use when documentation is large; most marketing sites only need llms.txt.

---

vs robots.txt and sitemap.xml

llms.txt is aimed at inference (on-demand agent context), not training policy or full-site enumeration.

---

Authoring guidelines

• Concise blockquote and descriptions; avoid marketing fluff agents cannot act on.
• Curate — link to 10–30 high-value pages, not every URL in the sitemap.
• Stable URLs — prefer canonical paths; update llms.txt when slugs change.
• No ambiguous jargon without a one-line explanation.
• Test — feed the file (or llms-ctx output) to an agent and verify it answers common questions.
• Put secondary links (changelog, legal, old posts) under ## Optional.

---

What to include (typical sections)

Adapt section names to the site; only ## Optional has special semantics in the spec.

---

Framework integrations

Common generators (see llmstxt.org integrations):

• VitePress: vitepress-plugin-llms
• Docusaurus: docusaurus-plugin-llms
• Drupal: LLM Support recipe
• CLI: llms_txt2ctx (Python) — parse and expand to context files
• PHP: llms-txt-php — read/write spec-compliant files

For custom stacks (Next.js + Sanity, etc.), a route.ts handler that builds Markdown from queries is the usual pattern.

---

Common mistakes

• Missing H1 — only required element; file is invalid without it.
• Wrong section order — blockquote before H2s; body before first H2.
• Relative URLs — use absolute https:// links.
• Dumping sitemap — too many low-value links defeats the purpose.
• Using llms.txt for robots rules — use robots.txt instead.
• Broken link format — must be - [Title](URL): description; parsers expect this shape.
• Skipping descriptions — weak for agents choosing which link to fetch next.

---

Related skills

• sitemap — exhaustive URL list for search crawlers (different purpose than llms.txt).
• json-ld — schema.org structured data for Google rich results.

---

Additional resources

• reference.md — Official spec, tools, directories, examples, comparisons.
• Spec: https://llmstxt.org/
• GitHub: https://github.com/AnswerDotAI/llms-txt

License

Skill License - Free Use

This skill (the documentation, structure, and implementation) was created by Ender Puentes <Endev/> and is provided for free and open use. You are free to:

• Use this skill in any project, personal or commercial
• Modify the skill to fit your needs
• Distribute the skill to others
• Share modified versions of the skill
• Include this skill in your own skill collections

No restrictions apply – this skill is available for unrestricted use. Attribution is appreciated but not required.

Note: This skill (llms-txt) documents the llms.txt convention (llmstxt.org, Answer.AI / Jeremy Howard); the skill itself is an independent work.

llms.txt — Reference & Official Documentation

This file complements the llms-txt skill with official documentation links and extra context for indexing.

Official specification

• llmstxt.org (primary spec): https://llmstxt.org/
• GitHub repository: https://github.com/AnswerDotAI/llms-txt
• Proposer: Jeremy Howard (Answer.AI), September 2024
• Status: Community convention; not W3C/IETF; syntax stable across adopters as of 2026

Spec structure (normative summary)

From https://llmstxt.org/#format — sections in order:

1. # Title (H1) — required 2. > Blockquote — short summary (recommended) 3. Free Markdown body — paragraphs/lists, no headings (optional) 4. ## Section — H2 delimited file lists (zero or more) 5. ## Optional — special section; links may be skipped for shorter context

Link item format:

- [name](url): optional notes

Markdown page mirrors

Spec proposal: LLM-readable Markdown at same URL + .md:

• https://example.com/page.html → https://example.com/page.html.md
• Extensionless: append index.html.md

Reference: https://llmstxt.org/#proposal

Expanded context files

Tool: llms_txt2ctx — https://llmstxt.org/ (Integrations section)

Comparison with other standards

llms.txt does not embed robots.txt rules.

HTTP delivery

• Path: /llms.txt (root canonical)
• Media type: text/plain; charset=utf-8 (common practice)
• Cache: public, s-maxage=3600, stale-while-revalidate=86400 or similar for dynamic routes
• Optional mirror: /.well-known/llms.txt

Next.js patterns

Static

public/llms.txt — zero config; served at /llms.txt.

Dynamic route handler

app/llms.txt/route.ts   → GET returns Markdown string

Use server actions or CMS queries to populate blog/docs slugs. Return Response with text/plain charset.

Route segment note

Folder name llms.txt is valid in App Router; exports GET handler.

Authoring best practices (from spec)

Source: https://llmstxt.org/#example

• Concise, clear language
• Informative descriptions on every link
• Avoid ambiguous terms or unexplained jargon
• Expand with llms_txt2ctx and test with real LLM queries

Integrations and tooling

From https://llmstxt.org/ (Integrations):

Directories and discovery

• llmstxt.org lists directories of sites publishing llms.txt (see site footer / directories section)
• Validator/generator tools: e.g. https://llmtxt.info/ (community reference, validator, generator)

Real-world examples

• FastHTML: https://fastht.ml/llms.txt (reference implementation cited on llmstxt.org)
• Anthropic, Cloudflare, Stripe, Mintlify, Vercel docs (widely cited adopters)
• Zod: https://zod.dev/llms.txt — API index for agents

Multilingual sites

Convention: separate files per locale root, e.g.:

• https://example.com/en/llms.txt
• https://example.com/es/llms.txt

Cross-link locale files in the body section if helpful.

SEO / GEO notes

• No major search engine has publicly committed to ranking based on llms.txt alone (2026).
• Still valuable as agent infrastructure and documentation contract.
• Pair with sitemap, robots.txt, and JSON-LD for complete discoverability stack.

Related skills

• sitemap: sitemaps.org protocol for search engine crawling.
• json-ld: schema.org structured data for rich results.