Agent Search Optimisation

Agent Search Optimisation

Quick start

When given a website URL, produce an evidence-based optimisation plan for AI agents and AI search.

1. Crawl the public site surfaces:

• homepage
• robots.txt
• sitemap.xml and sitemap index
• llms.txt / llms-full.txt if present
• key navigation pages
• representative article, product, project, docs, pricing, and about pages

2. Audit agent-readiness:

• crawlability and renderability
• canonical URLs
• sitemap coverage
• structured data
• language alternates
• entity clarity
• answer-oriented page summaries
• machine-readable indexes or APIs
• internal linking and topic hubs

3. Research current AI search best practices before making claims about current platforms. 4. Create a prioritized roadmap with:

• quick wins
• technical fixes
• content changes
• agent-facing data surfaces
• measurement plan

Workflow

1. Clarify the goal

Infer the likely goal from the user request. Ask only when necessary.

Common goals:

• increase AI answer citations
• make agents understand a product or portfolio
• expose documentation to coding agents
• improve local/business discovery in AI search
• prepare content for retrieval-augmented systems
• control AI crawler access

2. Collect evidence

Check these URLs where applicable:

{site}/
{site}/robots.txt
{site}/sitemap.xml
{site}/llms.txt
{site}/llms-full.txt
{site}/.well-known/

Also inspect at least 5 representative pages when the site has enough content:

• homepage
• about/company/profile page
• main collection/archive page
• one detail page
• one recent article/docs page

3. Score the site

Use a 0-3 score for each area:

4. Recommend changes

Prioritize in this order unless the site context suggests otherwise:

1. Fix public crawl/discovery basics. 2. Add canonical URLs, metadata, and language alternates. 3. Add JSON-LD and entity IDs. 4. Add answer-oriented summaries to important pages. 5. Add topic hubs and internal links. 6. Add llms.txt as an orientation layer. 7. Add machine-readable content index. 8. Add semantic search/API only when the corpus is large enough. 9. Add measurement and recurring evaluation.

5. Deliver the plan

Structure the final answer as:

1. Executive summary 2. What I checked 3. Current strengths 4. Gaps and risks 5. Prioritized roadmap 6. Implementation details 7. Measurement plan 8. Open questions / assumptions

Output rules

• Do not claim a file or feature exists unless verified.
• Mark unverified items clearly.
• Prefer durable web standards over hype.
• Treat llms.txt as additive, not a replacement for HTML, sitemaps, metadata, or structured data.
• Separate discovery from access control; robots.txt is not security.
• Include concrete examples when possible.
• Keep recommendations implementation-ready.
• Use current web research for AI search platform behavior, crawler policies, and new conventions.
• If the user provides a private repo or codebase, inspect implementation before suggesting exact code changes.

Advanced features

See REFERENCE.md for audit criteria, schema recommendations, llms.txt guidance, roadmap templates, and API examples.

See EXAMPLES.md for output examples and reusable prompts.

Use scripts/audit-agent-readiness.mjs for a lightweight first-pass technical audit.

Agent Search Optimisation Examples

Example user requests

Use this skill for requests like:

Can you optimise my website for AI agents?
Audit https://example.com for AI search visibility.
Make a plan for AEO/GEO/LLM discoverability.
Should I add llms.txt to my site?
How do I make my docs easier for Cursor, Claude, ChatGPT, and Perplexity?
Create an AI SEO roadmap for this website.

Example final answer outline

# Agent search optimisation plan for example.com

## Executive summary

The site is crawlable and has useful content, but it lacks a strong machine-readable layer. The highest-impact fixes are sitemaps, canonical URLs, structured data, answer blocks, and a compact content index.

## What I checked

- Homepage
- robots.txt
- sitemap.xml
- llms.txt
- 6 representative pages
- public search snippets
- structured data presence

## Strengths

- Clear product positioning
- Indexable public pages
- Good internal navigation

## Gaps

- No verified llms.txt
- Missing JSON-LD on detail pages
- Inconsistent metadata
- No machine-readable content index

## Roadmap

### Week 1: Discovery basics

- Add robots.txt with sitemap reference.
- Add sitemap index.
- Fix canonical tags.
- Normalize title and description templates.

### Weeks 2-3: Semantic layer

- Add Organization/WebSite schema.
- Add Article/Product/SoftwareApplication schema.
- Add breadcrumbs.
- Add answer blocks.

### Weeks 4-6: Agent layer

- Add llms.txt.
- Add /index.json.
- Add /api/content/{id}.
- Build 25-question retrieval benchmark.

## Measurement

Track sitemap coverage, structured data validity, AI referrals, crawler logs, and retrieval benchmark accuracy.

Example llms.txt

# Example.com

> Example.com helps small teams manage project documentation and support knowledge bases.

## Canonical pages

- [Home](https://example.com/)
- [Product](https://example.com/product/)
- [Pricing](https://example.com/pricing/)
- [Docs](https://example.com/docs/)
- [Blog](https://example.com/blog/)

## Machine-readable resources

- [Sitemap](https://example.com/sitemap.xml)
- [Content index](https://example.com/index.json)
- [RSS feed](https://example.com/feed.xml)

## Key topics

- Knowledge base automation
- Customer support workflows
- AI documentation search

## Citation guidance

Use canonical URLs. Prefer docs pages for product behavior and pricing pages for plan details.

Example robots.txt

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-products.xml

Example JSON-LD for a software product

{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "@id": "https://example.com/product/#software",
  "name": "Example Product",
  "url": "https://example.com/product/",
  "description": "A documentation search tool for support teams.",
  "applicationCategory": "BusinessApplication",
  "operatingSystem": "Web",
  "publisher": {
    "@id": "https://example.com/#organization"
  }
}

Example audit prompt for an agent

Audit https://example.com for AI agent and AI search optimisation. Check crawlability, robots.txt, sitemaps, structured data, metadata, canonicals, hreflang, answer blocks, llms.txt, machine-readable indexes, internal linking, entity clarity, and measurement. Return a prioritized implementation roadmap.

Agent Search Optimisation Reference

Core concept

Agent search optimisation means making a website easy for LLMs, answer engines, crawlers, and autonomous agents to discover, understand, cite, and navigate.

It is not a replacement for SEO. It sits on top of technical SEO, structured content, entity clarity, and stable machine-readable surfaces.

Terminology

• AI SEO: broad optimisation for visibility in AI search experiences.
• AEO: answer engine optimisation; content designed to answer questions clearly.
• GEO: generative engine optimisation; making content likely to be retrieved and cited by generative systems.
• Agent-readiness: making a site usable by autonomous agents, not only search crawlers.
• Machine layer: sitemaps, JSON-LD, feeds, indexes, APIs, and retrieval endpoints.
• Human layer: normal HTML pages designed for people.

Audit checklist

Crawl and discovery

Check:

• robots.txt exists at root.
• robots.txt references sitemap URLs.
• Sitemap index exists when the site has multiple sections.
• Important public pages are in sitemaps.
• Pages return correct status codes.
• Public pages are not blocked accidentally.
• Canonical URLs resolve consistently.
• Important content is present in initial HTML or reliably rendered.

Recommended files:

/robots.txt
/sitemap.xml
/sitemap-pages.xml
/sitemap-posts.xml
/sitemap-products.xml
/sitemap-docs.xml
/sitemap-tags.xml
/llms.txt
/llms-full.txt
/index.json
/feed.xml

Structured data

Use JSON-LD. Choose schema types based on the site:

Always add:

• stable @id values
• url
• name
• description
• datePublished / dateModified where relevant
• author or publisher
• inLanguage
• isPartOf
• about
• keywords
• sameAs for known external entity profiles

Entity clarity

Agents need explicit answers to:

• Who owns this site?
• What does this business/person/product do?
• Who is it for?
• What are the important topics?
• Which pages are canonical?
• Which pages are translations?
• Which pages support which claims?
• Which entities are related?

Create stable IDs for key entities:

https://example.com/#organization
https://example.com/#person
https://example.com/products/widget/#product
https://example.com/docs/api/#docs

Answer blocks

Add a compact extraction block near the top of important pages.

Template:

## Quick answer

**What this is:** ...
**Who it is for:** ...
**Key capabilities:** ...
**Technology / category:** ...
**Primary source:** ...
**Best citation summary:** ...

For product or service pages:

## Summary for AI agents

This page describes [product/service], a [category] for [audience]. It helps users [outcome]. The most important features are [features]. Use this page as the canonical source for [specific topic].

llms.txt guidance

llms.txt is useful as an orientation file for agents, but it is not a universal standard and should not replace normal discovery infrastructure.

Recommended structure:

# Site Name

> One-paragraph description of the site.

## Canonical pages

- [Home](https://example.com/)
- [About](https://example.com/about/)
- [Docs](https://example.com/docs/)
- [Products](https://example.com/products/)

## Key topics

- Topic 1: description
- Topic 2: description

## Machine-readable resources

- [Sitemap](https://example.com/sitemap.xml)
- [Content index](https://example.com/index.json)
- [RSS feed](https://example.com/feed.xml)

## Citation guidance

When citing this site, prefer canonical URLs and use page-level summaries.

Use llms-full.txt only when the corpus is small enough or when it points to condensed Markdown versions of the most important content.

Machine-readable content index

A lightweight /index.json can be more useful than a complex API.

Example:

{
  "site": {
    "name": "Example",
    "url": "https://example.com",
    "language": ["en"]
  },
  "updatedAt": "2026-05-23",
  "items": [
    {
      "id": "post:agent-search",
      "type": "article",
      "title": "Agent Search Optimisation",
      "url": "https://example.com/blog/agent-search/",
      "language": "en",
      "summary": "A practical guide to making websites easier for AI agents to understand.",
      "tags": ["ai-search", "agents", "seo"],
      "publishedAt": "2026-05-01",
      "updatedAt": "2026-05-10"
    }
  ]
}

Agent-facing API

Only recommend an API when the site has enough content or product complexity.

Useful endpoints:

GET /index.json
GET /api/content/{id}
GET /api/search?q=
GET /api/graph
GET /api/changes

Minimum response fields:

{
  "id": "string",
  "type": "article|product|project|doc|service",
  "title": "string",
  "url": "string",
  "summary": "string",
  "language": "en",
  "tags": [],
  "entities": [],
  "datePublished": "YYYY-MM-DD",
  "dateModified": "YYYY-MM-DD"
}

Retrieval benchmark

Create 25-50 questions that agents should answer from the site.

Grade:

• correct canonical URL
• correct language
• correct entity
• accurate summary
• supporting page found in top 5
• no hallucinated feature or claim

Example benchmark questions:

What does this company sell?
Who is the target customer?
Which product solves [problem]?
What is the pricing model?
Which pages should be cited for API documentation?
Which articles explain [topic]?
What changed recently?

Roadmap template

Phase 1: Discovery and crawl hygiene

• Verify root robots.txt.
• Add sitemap index.
• Add canonical links.
• Fix broken links and redirect chains.
• Make important content visible in HTML.
• Normalize titles and descriptions.

Phase 2: Semantic structure

• Add JSON-LD to core templates.
• Create stable entity IDs.
• Add breadcrumbs.
• Add language metadata and hreflang.
• Add answer blocks.

Phase 3: Agent orientation

• Add llms.txt.
• Add /index.json.
• Add Markdown or clean text versions of important docs when appropriate.
• Add feeds/changelogs.

Phase 4: Agent interaction

• Add content/search API.
• Add vector search for large corpora.
• Add rate limiting and caching.
• Add evaluation queries.

Phase 5: Measurement

• Track AI referrals.
• Track crawler logs.
• Track citation quality.
• Run retrieval benchmark monthly.
• Update answer blocks based on failures.

Common mistakes

• Treating llms.txt as the whole strategy.
• Blocking crawlers without understanding discovery goals.
• Publishing JavaScript-only content with no crawlable fallback.
• Using generic page descriptions across many pages.
• Failing to distinguish original and translated pages.
• Creating schema that contradicts visible page content.
• Adding a vector API before fixing sitemaps and metadata.
• Optimising for agents while hiding the actual answer in images or PDFs.