Llms Txt

llms.txt Generator

Generate and maintain llms.txt files that help AI answer engines surface your project accurately.

Install

git clone https://github.com/thatrebeccarae/claude-marketing.git && cp -r claude-marketing/skills/llms-txt ~/.claude/skills/

When to Use

• Launching a new open-source project or documentation site
• Major documentation restructure or content overhaul
• Improving your project's visibility in AI search (ChatGPT, Perplexity, Google AI Overviews)
• Onboarding a project to AI-friendly discoverability standards
• Periodic refresh after significant repo changes

What Is llms.txt

llms.txt is a plain-text markdown file placed in a project's root that gives LLMs a curated map of the project's most important content. Think of it as robots.txt for AI comprehension — instead of telling crawlers where they can go, it tells them what matters and how the project is organized.

The specification was proposed by Answer.AI and is documented at llmstxt.org. Adoption is growing across developer tools, documentation sites, and open-source projects. Projects with an llms.txt are easier for AI to understand, cite, and recommend accurately.

Usage

/llms-txt generate [repo-path]

Scan a repository and generate a new llms.txt file. If no path is provided, uses the current working directory.

/llms-txt audit [repo-path]

Check an existing llms.txt for completeness, broken links, stale descriptions, and missing high-priority content. Produces a report with specific recommendations.

/llms-txt update [repo-path]

Refresh an existing llms.txt based on current repo state. Preserves manually curated descriptions while adding new content and removing references to deleted files.

Procedure

Step 1: Scan Repo Structure

Identify all documentation-relevant files in the repository:

• README.md (root and significant subdirectories)
• docs/ directory and its contents
• API documentation (OpenAPI specs, API reference pages)
• Tutorials, guides, and getting-started content
• CHANGELOG.md, CONTRIBUTING.md, FAQ.md
• Architecture and design decision docs
• Configuration and deployment guides
• Example directories with their own READMEs

Step 2: Prioritize Content

Rank discovered content by importance to an LLM trying to understand the project:

Step 3: Extract Metadata

For each content page, extract or write:

• Title: Clear, descriptive page title
• URL or path: Where to find the content (full URL for hosted docs, relative path for repo files)
• Description: One-line action-oriented summary of what the page covers

Step 4: Generate llms.txt

Assemble the file following the specification format:

1. Title line: # Project Name 2. Description block: 2-3 sentence summary of what the project does, who it is for, and its primary use case 3. Sections: Group content logically (e.g., "Getting Started", "API", "Guides", "Community") 4. Content items: Each item is a markdown link with a colon-separated description

See REFERENCE.md for the exact format specification.

Step 5: Optionally Generate llms-full.txt

For projects that benefit from it, generate an expanded version that inlines the actual content of key pages. This is useful for smaller projects where the full documentation fits in a single context window.

Step 6: User Review

Present the generated llms.txt for review. Flag any decisions made during curation:

• Content that was excluded and why
• Descriptions that were inferred vs extracted from existing metadata
• Sections where additional documentation would improve AI discoverability

Step 7: Write to Repo Root

Save llms.txt (and optionally llms-full.txt) to the repository root.

For projects with hosted documentation sites, also note the recommended placement for the hosted version (site root, e.g., https://docs.example.com/llms.txt).

Hosted Docs and GitHub Pages

If the project has a documentation site (GitHub Pages, ReadTheDocs, Docusaurus, etc.):

• Generate llms.txt with full URLs pointing to the hosted docs, not repo file paths
• Place the file where it will be served at https://yourdomain.com/llms.txt
• For GitHub Pages: add llms.txt to the docs source directory so it deploys automatically
• Consider adding llms.txt to your sitemap or linking it from robots.txt

Key Principles

1. Curate, don't dump. An llms.txt that lists every file in the repo is worse than useless. Select the 10-30 most important pages that give an LLM the clearest picture of the project. 2. Prioritize entry points. The first few items should answer: what is this, who is it for, and how do I start? 3. Keep descriptions action-oriented. "How to configure authentication for SSO providers" beats "Authentication configuration page." 4. Match the reader's mental model. Organize sections the way a newcomer would learn the project, not the way the repo is structured. 5. Maintain freshness. Stale llms.txt with broken links or outdated descriptions erodes trust. Run /llms-txt audit after major documentation changes.

Integration with Other Skills

• aeo-geo-optimizer — llms.txt complements broader AI search optimization; use both for maximum AI discoverability
• technical-seo-audit — Ensure AI crawlers can access your docs before generating llms.txt
• github-readme — A strong README is the foundation of a good llms.txt; optimize it first

For the full specification format, priority ranking criteria, and placement guidance, see REFERENCE.md.

llms.txt Generator — Examples

Example 1: Generate llms.txt for a CLI Tool

Prompt

Generate an llms.txt for this CLI tool repo. It has a README, docs/ folder with a getting-started guide, API reference, and a few tutorials.

What the skill does

1. Scans and ranks: P0 (README, getting-started, api-reference), P1 (tutorials), P2 (CHANGELOG, CONTRIBUTING).

2. Generates llms.txt:

# fastcli

> A fast, ergonomic command-line framework for building CLI tools in Node.js.
> Supports argument parsing, subcommands, interactive prompts, and plugin
> extensions. Used by teams shipping internal tools and developer CLIs.

## Getting Started

- [README](https://github.com/example/fastcli#readme): What fastcli is, installation, and a 30-second usage example
- [Getting Started Guide](docs/getting-started.md): Install fastcli and build your first CLI command in under 5 minutes

## API

- [API Reference](docs/api-reference.md): Complete API with all commands, options, and configuration parameters

## Tutorials

- [Authentication Setup](docs/tutorials/auth-setup.md): Add token-based auth to your CLI with built-in keychain storage
- [CI Integration](docs/tutorials/ci-integration.md): Run fastcli commands in GitHub Actions and other CI environments

## Community

- [Changelog](CHANGELOG.md): Release history with breaking changes and new features
- [Contributing](CONTRIBUTING.md): Development setup, PR process, and code standards

3. Flags decisions: Notes that tutorials/plugins.md was excluded (marked as draft).

---

Example 2: Generate llms.txt for a Skill Collection

Prompt

Generate an llms.txt for the claude-marketing repo — 50+ skills across SEO, ads, content, design, and developer tools.

What the skill does

1. Curates rather than dumps: Groups by category, links to catalog as primary index, highlights 5-8 most distinctive skills.

2. Generates llms.txt:

# claude-marketing

> A collection of 50+ Claude Code skills for marketing, SEO, content creation,
> paid ads, and developer tools. Install individual skills or browse the full
> catalog. Built for marketers and developers using Claude Code.

## Overview

- [README](https://github.com/thatrebeccarae/claude-marketing#readme): What claude-marketing is, how to install skills, and the full category list
- [Skill Catalog](CATALOG.md): Browsable catalog of all 50+ skills with descriptions and categories

## Featured Skills

- [AEO/GEO Optimizer](skills/aeo-geo-optimizer): Optimize content for AI answer engines
- [Frontend Design](skills/frontend-design): Design-system-aware UI development with modern CSS
- [Brand DNA](skills/brand-dna): Extract brand voice, values, and positioning from existing content

## Categories

- [SEO Skills](CATALOG.md#seo): Technical SEO, AEO/GEO, schema markup
- [Content Skills](CATALOG.md#content): Content creation, brand voice, copywriting
- [Ads Skills](CATALOG.md#ads): Google, Facebook, LinkedIn, TikTok Ads
- [Developer Tools](CATALOG.md#developer-tools): MCP builder, GitHub README, llms.txt

3. Flags decisions: Individual SKILL.md files not linked — catalog is the canonical index.

---

Example 3: Audit an Existing llms.txt

Prompt

Audit our llms.txt — we added API v2 docs and restructured last month but haven't updated it.

What the skill does

1. Cross-references existing llms.txt against current repo state.

2. Produces audit report:

• Broken: docs/api-v1-reference.md (deleted), docs/tutorials/legacy-auth.md (moved)
• Missing: docs/api-v2-reference.md (P0), docs/guides/migration-v1-to-v2.md (P1)
• Stale: Getting Started description says "v1 API" — now covers v2

3. Recommends fixes: Remove broken links, add v2 reference and migration guide, update stale description. Asks whether to regenerate or apply targeted updates.

llms.txt Generator — Reference

llms.txt Specification Format

The specification (from llmstxt.org) defines a simple markdown structure:

# Project Name

> Brief project description in 2-3 sentences. What it does, who it is for,
> and its primary value proposition.

## Section Name

- [Page Title](url-or-path): One-line description of what this page covers
- [Page Title](url-or-path): One-line description of what this page covers

Format Rules

1. Title: Single H1 with the project name 2. Description: Blockquote immediately after the title. 2-3 sentences. Answers: what is this, who uses it, what problem does it solve. 3. Sections: H2 headings grouping related content. Flat structure — no H3 nesting. 4. Items: Unordered list with markdown link, colon, one-line description 5. Plain markdown: No HTML, no frontmatter, no metadata beyond the content itself

llms-full.txt Format

The expanded companion inlines actual page content so an LLM can consume the entire project in one pass. Use H3 headings for each inlined page under H2 section headings.

• Generate when: Small/mid-size projects (~50K tokens total docs), or when primary goal is LLM comprehension
• Skip when: Large doc sites (>100 pages), frequently changing docs, or docs already well-structured individually

Priority Ranking Criteria

Exclude: Auto-generated API docs with no narrative, internal dev notes, draft RFCs, test fixtures, duplicate content.

Writing Effective Descriptions

Description Patterns by Content Type

File Placement

Repository: Place llms.txt in the repo root alongside README.md.

Hosted docs: Serve at the site root (https://docs.example.com/llms.txt).

Optionally reference in robots.txt: LLMs-Txt: https://example.com/llms.txt

How AI Answer Engines Use llms.txt

1. RAG retrieval: Curated index helps AI identify relevant pages, reducing hallucination 2. Content mapping: Section structure builds a mental model without reading every page 3. Citation sourcing: Pre-written descriptions enable accurate citations 4. Context efficiency: Full project understanding in a fraction of the tokens vs crawling

Auto-Regeneration with GitHub Actions

The demodrive-ai/llms-txt-action GitHub Action auto-regenerates llms.txt on push or schedule:

name: Update llms.txt
on:
  push:
    branches: [main]
    paths: ['docs/**', 'README.md']
  schedule:
    - cron: '0 6 * * 1'  # weekly Monday 6am

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: demodrive-ai/llms-txt-action@v1
        with:
          output: llms.txt
      - uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "chore: regenerate llms.txt"

Auto-generated files benefit from manual curation. The action provides a starting point, but hand-tuned descriptions and section organization will outperform fully automated output.