Nextjs Seo

Next.js SEO Optimization

Comprehensive SEO guide for Next.js App Router applications.

Quick SEO Audit

Run this checklist for any Next.js project:

1. Check robots.txt: curl https://your-site.com/robots.txt 2. Check sitemap: curl https://your-site.com/sitemap.xml 3. Check metadata: View page source, search for <title> and <meta name="description"> 4. Check JSON-LD: View page source, search for application/ld+json 5. Check Core Web Vitals: Use PageSpeed Insights (pagespeed.web.dev) and the Search Console CWV report for field data — Lighthouse is lab-only and can't measure INP

Essential Files

app/layout.tsx - Root Metadata

import type { Metadata, Viewport } from 'next';

// Viewport must be a separate export — `themeColor`, `colorScheme`, and
// `viewport` inside the `metadata` object are not supported.
export const viewport: Viewport = {
  width: 'device-width',
  initialScale: 1,
  maximumScale: 5,
  userScalable: true,
  themeColor: [
    { media: '(prefers-color-scheme: light)', color: '#ffffff' },
    { media: '(prefers-color-scheme: dark)', color: '#0a0a0a' },
  ],
};

export const metadata: Metadata = {
  metadataBase: new URL('https://your-site.com'),
  title: {
    default: 'Site Title - Main Keyword',
    template: '%s | Site Name',
  },
  description: 'Compelling description with keywords (150-160 chars; Google typically displays this range)',
  keywords: ['keyword1', 'keyword2', 'keyword3'],
  openGraph: {
    type: 'website',
    locale: 'en_US',
    url: 'https://your-site.com',
    siteName: 'Site Name',
    title: 'Site Title',
    description: 'Description for social sharing',
    images: [{ url: '/og-image.png', width: 1200, height: 630, alt: 'Site preview' }],
  },
  twitter: {
    card: 'summary_large_image',
    title: 'Site Title',
    description: 'Description for Twitter',
    images: ['/og-image.png'],
  },
  alternates: {
    canonical: '/',
  },
  robots: {
    index: true,
    follow: true,
  },
};

app/sitemap.ts - Dynamic Sitemap

import type { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  const baseUrl = 'https://your-site.com';

  return [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 1,
      images: [`${baseUrl}/og-image.png`], // Image Sitemap entry
    },
    {
      url: `${baseUrl}/about`,
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
  ];
}

app/robots.ts - Robots Configuration

import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = 'https://your-site.com';

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/'],
        // Do NOT disallow /_next/ — crawlers need render-critical CSS/JS
        // Do NOT add bot-specific rules (Googlebot, Bingbot) unless overriding wildcard
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

host was omitted intentionally — it's a non-standard directive Google ignores. Use canonical URLs / 301s to declare the preferred host instead. See references/sitemap-robots.md.

app/manifest.ts - Web App Manifest

import type { MetadataRoute } from 'next';

export default function manifest(): MetadataRoute.Manifest {
  return {
    name: 'Site Name',
    short_name: 'Site',
    description: 'Site description',
    start_url: '/',
    display: 'standalone',
    background_color: '#ffffff',
    theme_color: '#0a0a0a',
    icons: [
      { src: '/icon-192.png', sizes: '192x192', type: 'image/png' },
      { src: '/icon-512.png', sizes: '512x512', type: 'image/png' },
    ],
  };
}

Same MetadataRoute family as sitemap/robots; place at the root of app/. Minor for ranking, but expected for PWA completeness. (A static app/manifest.json works too.)

OG / Twitter Images

Three ways to set social images — prefer the file conventions over hand-syncing URLs in the metadata object:

1. External URL in metadata (the openGraph.images / twitter.images examples above) — fine for externally hosted images. 2. Static file convention (recommended default): drop opengraph-image.(png|jpg|gif) and/or twitter-image.* into a route segment (app/opengraph-image.png for the root, app/blog/opengraph-image.png for /blog). Next.js auto-emits og:image/twitter:image + :type/:width/:height. A deeper, more specific image overrides one above it. Add alt text with a sibling opengraph-image.alt.txt. Build fails if the file exceeds 8 MB (OG) / 5 MB (Twitter). 3. Dynamic generation with `ImageResponse` (per-page/per-post images):

// app/blog/[slug]/opengraph-image.tsx
import { ImageResponse } from 'next/og';

export const alt = 'Post preview';
export const size = { width: 1200, height: 630 };
export const contentType = 'image/png';

export default async function Image({ params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params;            // params is a Promise in v16
  const post = await getPost(slug);
  return new ImageResponse(
    <div style={{ display: 'flex', fontSize: 64, width: '100%', height: '100%' }}>{post.title}</div>,
    { ...size },
  );
}

ImageResponse renders via Satori — flexbox only, no `display: grid`. These files are statically optimized at build time unless they read request-time data. See references/metadata-api.md for fonts, generateImageMetadata, and the favicon/icon.tsx/apple-icon conventions.

Key Principles

Cache Components & SEO

With cacheComponents: true in next.config.ts (the v16 top-level flag that unifies the old experimental.dynamicIO/ppr/useCache), use the "use cache" directive for SEO-critical server components:

// app/(home)/sections/hero-section.tsx
import { cacheLife, cacheTag } from "next/cache";

export async function HeroSection() {
  "use cache";
  cacheLife("hours");   // SEO content that changes a few times/day; see profiles below
  cacheTag("hero");     // Invalidate via updateTag("hero") in a Server Action

  const data = await fetchData();
  return <div>{/* SEO-visible content */}</div>;
}

Built-in `cacheLife` profiles (stale / revalidate / expire): seconds (30s/1s/1m), minutes (5m/1m/1h), hours (5m/1h/1d), days (5m/1d/1w), weeks (5m/1w/30d), max (5m/30d/1y), and the implicit default (5m/15m/never). For SEO pages pick by how often content changes — days for blog/docs, max for legal/marketing. (minutes revalidates every 1 min — too aggressive for most SEO content.)

Key rules:

• "use cache" must be the first statement in the function body (or at the top of the file for file-level caching)
• No cookies()/headers()/searchParams inside a plain "use cache" scope — good for SEO, since indexable content should be request-agnostic. ("use cache: private" does allow them, but is never prerendered, so it never lands in the static SEO shell.)
• Invalidate with updateTag("hero") inside a Server Action (read-your-writes), or revalidateTag("hero") from a Route Handler / webhook — prefer these over export const revalidate
• Short-lived caches (seconds, or revalidate < 5 min) are excluded from the prerender and become dynamic holes that need a <Suspense> boundary — keep SEO-critical content on a longer profile so it stays in the static shell
• Sitemaps and metadata are static by default — only add "use cache" (+ cacheTag) if they fetch CMS/dynamic data you want to invalidate on publish

Rendering Strategy for SEO

Core Web Vitals Targets

• Measured on field data, not lab. Google ranks on the 75th percentile of real users (Chrome UX Report, 28-day rolling window, mobile/desktop separate). A URL group passes only when ≥75% of visits hit "Good" on all three. Use PageSpeed Insights and the Search Console CWV report for the real signal — Lighthouse is lab-only and cannot measure INP.
• INP replaced FID as a Core Web Vital on 2024-03-12; FID is deprecated. INP is the most commonly failed metric — prioritize it.
• Page experience is a tiebreaker, not a standalone ranking system (Google de-emphasized it). Good CWV won't rescue thin content; content relevance and quality come first. Treat CWV as baseline UX hygiene.
• Myths to ignore: 2026 SEO blogs falsely claim "LCP was lowered to 2.0s" and invent an "Engagement Reliability" metric. Neither exists in any Google/web.dev source — the thresholds above are current and unchanged since 2021.

Ranking Signals Beyond Technical SEO

Metadata + CWV alone don't drive rankings. Keep these in mind (out of scope for this skill, but pointers):

• Helpful content is part of core ranking (since 2024-03), evaluated continuously — not an episodic penalty.
• E-E-A-T (Experience, Expertise, Authoritativeness, Trust): cite real authors/credentials and first-hand experience, especially on YMYL pages.
• Mobile-first indexing is complete (since 2024-07): Google indexes the mobile rendering only. Ensure the mobile view has the same content, metadata, and structured data as desktop; never block mobile resources. (Mostly automatic with Next.js responsive design.)

References

• Metadata API: See references/metadata-api.md — generateMetadata, OG/icon file conventions, ImageResponse, manifest
• Sitemap & Robots: See references/sitemap-robots.md
• JSON-LD Structured Data: See references/json-ld.md
• AI Search (GEO/AEO) & AI Crawlers: See references/ai-search.md
• SEO Audit Checklist: See references/checklist.md
• Troubleshooting: See references/troubleshooting.md

Common Mistakes to Avoid

1. Mixing next-seo with Metadata API - Use only Metadata API in App Router 2. Missing canonical URLs - Always set alternates.canonical 3. Using CSR for SEO pages - Use SSG/SSR for indexable content 4. Blocking `/_next/` in robots.txt - Crawlers need render-critical CSS/JS; never disallow /_next/ 5. Missing metadataBase - Required for relative URLs in metadata 6. Viewport in metadata - Must be a separate export 7. Mixing metadata object and generateMetadata - Use one or the other in the same route segment 8. Duplicating icons in metadata + file conventions - Prefer favicon.ico/icon.*/opengraph-image.* file conventions; they auto-emit tags and override the metadata object 9. Blanket-blocking AI crawlers - GPTBot disallow: / blocks training but leaves you in AI search; don't accidentally block citation bots (OAI-SearchBot, PerplexityBot). See references/ai-search.md

Quick Fixes

Add noindex to a page

export const metadata: Metadata = {
  robots: {
    index: false,
    follow: false,
  },
};

Dynamic metadata per page

type Props = { params: Promise<{ id: string }> };

export async function generateMetadata({ params }: Props): Promise<Metadata> {
  const { id } = await params;            // params is a Promise in current Next.js
  const product = await getProduct(id);
  return {
    title: product.name,
    description: product.description,
  };
}

Canonical for dynamic routes

type Props = { params: Promise<{ slug: string }> };

export async function generateMetadata({ params }: Props): Promise<Metadata> {
  const { slug } = await params;
  return {
    alternates: {
      canonical: `/products/${slug}`,
    },
  };
}

AI Search Optimization (GEO / AEO) & AI Crawlers

How to make a Next.js site visible in AI answer engines (Google AI Overviews / AI Mode, ChatGPT Search, Perplexity, Gemini, Claude) — grounded in what's actually established, not hype.

Contents

• Core principle (layer on top of SEO, not a replacement)
• Google AI Overviews / AI Mode — official eligibility
• llms.txt — honest status
• AI crawlers: training vs search/citation bots
• Recommended robots.ts pattern
• robots.txt is advisory (WAF/edge for real blocking)
• Content structure that drives AI citation
• Structured data for AI search
• Measuring AI search visibility

Core principle

GEO (Generative Engine Optimization) / AEO (Answer Engine Optimization) = optimizing so your content is cited in AI-generated answers. It is a layer on top of classic SEO, not a replacement. Most tactics that help AI citation also help users and traditional search — favor those low-risk, high-overlap moves over AI-specific "tricks."

Google AI Overviews / AI Mode — official eligibility

Per Google's official guidance: there are no special requirements to appear in AI Overviews or AI Mode.

"You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add."

Eligibility = pages that are indexed and eligible to be shown in Google Search with a snippet. So the baseline is normal technical SEO: crawlable, indexable, server-rendered content. (Source: developers.google.com/search/docs/appearance/ai-features)

Do not promise ranking/citation gains from any "AI-specific file or schema." Google says none are needed.

llms.txt — honest status

llms.txt is a community proposal, not an adopted standard:

• Not supported by Google. John Mueller publicly compared it to the discredited keywords meta tag.
• Negligible real usage by AI crawlers, and no demonstrated correlation between having an llms.txt and being cited in AI answers.
• The one legitimate, working use case: documentation / developer-tool sites whose users paste docs into IDE/coding agents (Cursor, Claude Code, Copilot, Cline). Those agents do look for /llms.txt and /llms-full.txt. Next.js itself ships https://nextjs.org/docs/llms.txt.

So: recommend llms.txt only for docs/dev-tool sites as an AI-assistant ergonomics nicety — not as an SEO/citation ranking tactic. There is no MetadataRoute helper; implement as a Route Handler:

// app/llms.txt/route.ts
export const dynamic = 'force-static';

export function GET() {
  const body = `# Site Name\n\n> One-line summary.\n\n## Docs\n- [Getting started](https://your-site.com/docs)\n`;
  return new Response(body, { headers: { 'Content-Type': 'text/plain' } });
}

AI crawlers: training vs search/citation bots

The key 2026 concept: separate training crawlers from search/citation crawlers.

• Training crawlers collect content to train models. Blocking them opts you out of training and costs you no referral traffic.
• Search / citation crawlers fetch content in real time to power AI answers. Blocking them removes you from that engine's AI answers — now a high-value channel.
• User-initiated fetchers retrieve a page a user explicitly asked the assistant about. Blocking them means the AI can't read a page the user pointed it to.
• Training opt-out tokens (Google-Extended, Applebot-Extended) are robots.txt tokens, not crawlers — they opt you out of Gemini/Apple-Intelligence training with no effect on Search ranking or AI Overviews (AI Overviews use Googlebot, so you can't leave AI Overviews via robots.txt without leaving Search).

anthropic-ai and Claude-Web are deprecated legacy tokens — current Anthropic bots are ClaudeBot, Claude-SearchBot, Claude-User. Don't copy-paste old block lists that only target the deprecated names. Verify current user-agents against vendor docs (they change).

Recommended robots.ts pattern

A defensible 2026 default: allow search/citation bots (stay in AI answers), optionally opt out of training, and block the worst-behaved bot (Bytespider).

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = 'https://your-site.com';
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/api/', '/admin/'] },

      // Stay in AI search/answers (recommended):
      { userAgent: 'OAI-SearchBot', allow: '/' },
      { userAgent: 'ChatGPT-User', allow: '/' },
      { userAgent: 'Claude-SearchBot', allow: '/' },
      { userAgent: 'Claude-User', allow: '/' },
      { userAgent: 'PerplexityBot', allow: '/' },

      // Optional: opt out of model TRAINING (policy decision; no traffic cost):
      { userAgent: 'GPTBot', disallow: '/' },
      { userAgent: 'ClaudeBot', disallow: '/' },
      { userAgent: 'CCBot', disallow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },   // token, not a crawler
      { userAgent: 'Applebot-Extended', disallow: '/' },  // token, not a crawler

      // Block the worst-behaved scraper:
      { userAgent: 'Bytespider', disallow: '/' },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

This is a policy decision, not a forced default. The "maximize AI visibility" variant simply allows everything except known bad actors. Blocking training is the conservative choice; blocking search/citation bots is usually a mistake.

robots.txt is advisory (use a WAF for real blocking)

robots.txt is honored by compliant bots (OpenAI, Anthropic, Google, Apple, Amazon, Meta, Common Crawl) but is voluntary:

• Bytespider widely ignores disallow rules.
• Perplexity was caught by Cloudflare (Aug 2025) using stealth undeclared crawlers to evade no-crawl directives, and was delisted from Cloudflare's Verified Bots program.
• User-initiated fetchers often ignore robots.txt because the fetch was user-requested.

For true enforcement use a WAF / edge layer (Cloudflare AI bot blocking, Vercel firewall / bot management) — not robots.txt alone. Conversely, audit for accidental blocks: old "block all AI" templates and WAF rules can silently disallow OAI-SearchBot/Claude-SearchBot/PerplexityBot and drop you from AI answers. Verify with server logs which bots actually visit.

Content structure that drives AI citation

The low-controversy, established part of GEO — also good for users and classic SEO:

• Answer the primary question directly in the first ~200 words.
• Make sections self-contained (AI retrieval is passage-level); lead each section with a direct answer, then expand.
• Clear H2/H3 hierarchy; add a TL;DR and Q&A/FAQ blocks (AI relies heavily on question→answer pairs).
• Publish original data / first-hand experience (strong E-E-A-T and citation signal).
• Keep content fresh with a visible "Last updated" date.
• SSR/SSG, textual content. AI retrieval bots render JS poorly — worse than Googlebot — so client-only content is often invisible to them. Reuse Next.js strengths (SSG / "use cache" / SSR) as the AI-friendly default.

Authority for AI citation skews toward entity authority and earned media (consistent brand/entity mentions, authoritative author bios, third-party coverage) over link-volume tactics like generic directory listings.

Structured data for AI search

Schema is not required for AI Overviews (per Google), but well-formed JSON-LD that matches visible content plausibly helps AI systems parse, ground, and cite content — useful even for types that no longer yield SERP rich results (e.g. FAQPage, HowTo). Treat this as correlation / trust signal, not a confirmed ranking factor. Prioritize Organization, Article, Product, Breadcrumb. See json-ld.md.

Measuring AI search visibility

• Segment AI-referral traffic in GA4 by referrer (chatgpt.com, perplexity.ai, gemini.google.com, etc.).
• Track citation share-of-voice with emerging third-party AI-visibility tools (verify independently; the space is young — avoid over-relying on any single paid tool).

Next.js SEO Audit Checklist

Contents

Critical | Important | Nice to Have | Audit Tools | Red Flags

Critical (Must Have)

Technical Foundation

• [ ] metadataBase set in root layout
• [ ] Unique <title> on every page (50-60 chars)
• [ ] Unique meta description on every page (150-160 chars)
• [ ] robots.txt exists and allows crawling
• [ ] sitemap.xml exists and is valid
• [ ] Sitemap submitted to Google Search Console
• [ ] No noindex on pages you want indexed
• [ ] Canonical URLs set for all pages
• [ ] viewport exported separately from metadata
• [ ] favicon.ico (or app/icon) present — appears in Google SERPs and browser tabs
• [ ] app/manifest.ts present (name, short_name, theme_color, icons) — PWA completeness

Rendering

• [ ] SEO pages use SSG, SSR, or "use cache" Cache Components (not CSR)
• [ ] Content visible without JavaScript (test with JS disabled)
• [ ] No client-side only content for SEO-critical text

Core Web Vitals

• [ ] LCP (Largest Contentful Paint) < 2.5s
• [ ] INP (Interaction to Next Paint) < 200ms
• [ ] INP optimized (INP replaced FID in March 2024)
• [ ] CLS (Cumulative Layout Shift) < 0.1
• [ ] CWV checked on FIELD data (PageSpeed Insights / Search Console CrUX, 75th percentile) — not just Lighthouse (Lighthouse can't measure INP)
• [ ] Mobile parity — same content/metadata/structured-data on mobile (mobile-first indexing complete since July 2024)

Important (Should Have)

Structured Data

• [ ] WebSite schema on homepage
• [ ] Organization schema
• [ ] Relevant page-specific schemas (Article, Product) for rich results
• [ ] FAQPage = AI-search/LLM signal only (rich results removed 2026-05-07)
• [ ] JSON-LD matches visible content
• [ ] Validated with Rich Results Test

Open Graph & Social

• [ ] Open Graph title and description
• [ ] OG image (1200x630 recommended)
• [ ] OG image set via opengraph-image file convention or ImageResponse (not just a hardcoded URL)
• [ ] Twitter Card configured
• [ ] Images tested with Facebook Debugger

Links & Navigation

• [ ] Internal links use <Link> component
• [ ] No broken internal links
• [ ] Logical URL structure
• [ ] Breadcrumbs implemented (if applicable)

Images

• [ ] All images have alt text
• [ ] Images use next/image component
• [ ] Images in sitemap
• [ ] Appropriate image sizes (no oversized images)

Nice to Have (Optimization)

Performance

• [ ] JavaScript bundle optimized
• [ ] Fonts use next/font
• [ ] Critical CSS inlined
• [ ] Third-party scripts deferred

International (if applicable)

• [ ] hreflang tags for language versions
• [ ] Localized sitemaps
• [ ] Language-specific metadata

Advanced

• [ ] Video sitemap (if video content)
• [ ] News sitemap (if news site)
• [ ] App links configured (if mobile app)

Audit Tools

Quick Commands

# Check robots.txt
curl https://your-site.com/robots.txt

# Check sitemap
curl https://your-site.com/sitemap.xml

# Check if indexed
# Search in Google: site:your-site.com

# Test mobile rendering
# Use Chrome DevTools device emulation

Red Flags to Watch

1. "Discovered - currently not indexed" in GSC 2. Duplicate title tags across pages 3. Missing canonical URLs 4. Blocked resources in robots.txt 5. Slow LCP (> 4s) 6. High CLS (> 0.25) 7. No structured data 8. Missing alt text on images

JSON-LD Structured Data in Next.js

Structured data helps search engines understand your content and enables rich results.

Contents

• Implementation Pattern
• Common Schemas — WebSite, Organization, WebApplication, FAQPage, Product, Article, BreadcrumbList
• Deprecated / no longer rich-result-eligible
• Which schema types still drive rich results (2026)
• @graph multi-entity pattern
• Structured data for AI search
• Usage in Next.js
• Testing Tools
• Best Practices

Implementation Pattern

// components/seo/json-ld.tsx
type JsonLdProps = {
  data: Record<string, unknown>;
};

export function JsonLd({ data }: JsonLdProps) {
  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{
        __html: JSON.stringify(data).replace(/</g, '\\u003c'), // XSS protection
      }}
    />
  );
}

Common Schemas

WebSite Schema

const websiteSchema = {
  '@context': 'https://schema.org',
  '@type': 'WebSite',
  name: 'Site Name',
  url: 'https://your-site.com',
  description: 'Site description',
  inLanguage: 'en',
  publisher: {
    '@type': 'Organization',
    name: 'Organization Name',
  },
};

Organization Schema

const organizationSchema = {
  '@context': 'https://schema.org',
  '@type': 'Organization',
  name: 'Company Name',
  url: 'https://your-site.com',
  logo: {
    '@type': 'ImageObject',
    url: 'https://your-site.com/logo.png',
    width: 512,
    height: 512,
  },
  sameAs: [
    'https://twitter.com/company',
    'https://linkedin.com/company/company',
    'https://github.com/company',
  ],
  contactPoint: {
    '@type': 'ContactPoint',
    email: 'contact@company.com',
    contactType: 'customer service',
  },
  foundingDate: 'YYYY', // your real founding year
  areaServed: {
    '@type': 'Country',
    name: 'Finland',
  },
};

WebApplication Schema

const webAppSchema = {
  '@context': 'https://schema.org',
  '@type': 'WebApplication',
  name: 'App Name',
  url: 'https://your-site.com',
  description: 'App description',
  applicationCategory: 'UtilityApplication',
  operatingSystem: 'Any',
  browserRequirements: 'Requires JavaScript',
  offers: {
    '@type': 'Offer',
    price: '0',
    priceCurrency: 'EUR',
  },
  featureList: [
    'Feature 1',
    'Feature 2',
    'Feature 3',
  ],
};

FAQPage Schema

⚠️ FAQ rich results are deprecated. Google restricted them to authoritative gov/health sites in Aug 2023 and fully removed them for all sites as of 2026-05-07 (Rich Results Test support drops June 2026, Search Console API August 2026). FAQPage no longer produces any rich result in Google Search. Keep this markup only as an optional AI-search / LLM-extraction signal (machine-readable Q&A) — not for SERP enhancement. Existing markup is harmless but has no visible SERP effect.

const faqSchema = {
  '@context': 'https://schema.org',
  '@type': 'FAQPage',
  mainEntity: [
    {
      '@type': 'Question',
      name: 'What is your product?',
      acceptedAnswer: {
        '@type': 'Answer',
        text: 'Our product is a tool that helps you...',
      },
    },
    {
      '@type': 'Question',
      name: 'How much does it cost?',
      acceptedAnswer: {
        '@type': 'Answer',
        text: 'Our service is completely free to use.',
      },
    },
  ],
};

Important: FAQPage schema must match visible FAQ content on the page. Google rejects rich results if JSON-LD doesn't match visible content.

Product Schema

const productSchema = {
  '@context': 'https://schema.org',
  '@type': 'Product',
  name: 'Product Name',
  image: ['https://your-site.com/product.jpg'],
  description: 'Product description',
  sku: 'SKU123',
  brand: {
    '@type': 'Brand',
    name: 'Brand Name',
  },
  offers: {
    '@type': 'Offer',
    url: 'https://your-site.com/product',
    priceCurrency: 'EUR',
    price: '99.99',
    priceValidUntil: '2026-12-31', // use a real future date
    availability: 'https://schema.org/InStock',
    itemCondition: 'https://schema.org/NewCondition',
  },
  aggregateRating: {
    '@type': 'AggregateRating',
    ratingValue: '4.5',
    reviewCount: '89',
  },
};

Product snippet vs merchant listing experience

Google treats Product markup as two distinct experiences:

• (a) Product snippet — for editorial / non-purchase pages (reviews, roundups, comparisons). Supports review features (aggregateRating / review) and pros & cons via positiveNotes / negativeNotes. No price required.
• (b) Merchant listing experience — for pages where the product is purchasable. Needs offers with price + priceCurrency + availability, and benefits from shippingDetails and hasMerchantReturnPolicy for richer shopping results.

For products with variants, use ProductGroup with hasVariant, variesBy, and a stable productGroupID:

const productGroupSchema = {
  '@context': 'https://schema.org',
  '@type': 'ProductGroup',
  name: 'T-Shirt',
  productGroupID: 'TSHIRT-001',
  variesBy: ['https://schema.org/color', 'https://schema.org/size'],
  hasVariant: [
    { '@type': 'Product', sku: 'TSHIRT-001-RED-M', color: 'Red', size: 'M' },
    { '@type': 'Product', sku: 'TSHIRT-001-BLU-L', color: 'Blue', size: 'L' },
  ],
};

Article Schema

const articleSchema = {
  '@context': 'https://schema.org',
  '@type': 'Article',
  headline: 'Article Title',
  description: 'Article description',
  image: 'https://your-site.com/article-image.jpg',
  datePublished: 'YYYY-MM-DDT08:00:00+00:00', // set dynamically from the CMS, not hardcoded
  dateModified: 'YYYY-MM-DDT10:00:00+00:00', // set dynamically from the CMS, not hardcoded
  author: {
    '@type': 'Person',
    name: 'Author Name',
    url: 'https://author-website.com',
  },
  publisher: {
    '@type': 'Organization',
    name: 'Publisher Name',
    logo: {
      '@type': 'ImageObject',
      url: 'https://your-site.com/logo.png',
    },
  },
};

BreadcrumbList Schema

const breadcrumbSchema = {
  '@context': 'https://schema.org',
  '@type': 'BreadcrumbList',
  itemListElement: [
    {
      '@type': 'ListItem',
      position: 1,
      name: 'Home',
      item: 'https://your-site.com',
    },
    {
      '@type': 'ListItem',
      position: 2,
      name: 'Products',
      item: 'https://your-site.com/products',
    },
    {
      '@type': 'ListItem',
      position: 3,
      name: 'Product Name',
      item: 'https://your-site.com/products/product-slug',
    },
  ],
};

Deprecated / no longer rich-result-eligible

Do not implement these for SERP rich results — Google no longer renders them:

• FAQ — removed for all sites as of 2026-05-07.
• HowTo — deprecated September 2023.
• The 6 features Google retired in 2025 (Book Actions was initially on this list but was un-deprecated in June 2025 — it remains limited to large book providers):
• Course Info
• Claim Review / Fact Check
• Estimated Salary
• Learning Video
• Special Announcement
• Vehicle Listing
• Practice Problems — deprecated June 2025; support fully removed January 2026.
• Dataset markup is only used by Dataset Search, not Google Search results (clarified November 2025).

You may still emit some of these as machine-readable signals (e.g. for AI / LLM extraction), but expect zero visible SERP enhancement from Google.

Which schema types still drive rich results (2026)

Google's Search Gallery is the source of truth for which structured-data types are currently eligible for rich results — check it before investing in any schema. High-value types for typical Next.js sites:

• Product / merchant listing — product snippets and shopping results
• Review snippet — star ratings
• Breadcrumb — breadcrumb trail in SERP
• Article — news/blog/article enhancements
• Recipe
• Event
• Video
• Organization — logo / knowledge panel signals
• LocalBusiness
• Job posting
• Software app

@graph multi-entity pattern

Use a single <script type="application/ld+json"> with an @graph array to wire multiple entities together via @id cross-references. This avoids duplicating the Organization on every page and lets Google connect the dots:

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://your-site.com/#organization",
      "name": "Company Name",
      "url": "https://your-site.com",
      "logo": "https://your-site.com/logo.png"
    },
    {
      "@type": "WebSite",
      "@id": "https://your-site.com/#website",
      "url": "https://your-site.com",
      "name": "Site Name",
      "publisher": { "@id": "https://your-site.com/#organization" }
    },
    {
      "@type": "WebPage",
      "@id": "https://your-site.com/products/product-slug/#webpage",
      "url": "https://your-site.com/products/product-slug",
      "name": "Product Name",
      "isPartOf": { "@id": "https://your-site.com/#website" },
      "breadcrumb": { "@id": "https://your-site.com/products/product-slug/#breadcrumb" }
    },
    {
      "@type": "BreadcrumbList",
      "@id": "https://your-site.com/products/product-slug/#breadcrumb",
      "itemListElement": [
        { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://your-site.com" },
        { "@type": "ListItem", "position": 2, "name": "Products", "item": "https://your-site.com/products" },
        { "@type": "ListItem", "position": 3, "name": "Product Name", "item": "https://your-site.com/products/product-slug" }
      ]
    }
  ]
}

Structured data for AI search

Schema is not required for AI Overviews — Google has stated structured data is not needed to appear in AI Overviews. Still, well-formed JSON-LD that matches the visible page plausibly helps AI systems parse, ground, and cite your content. Frame this as a correlation / trust signal, not a confirmed ranking factor. See ai-search.md for AI-search and GEO guidance.

Caveat: the Rich Results Test only validates currently-supported types, so valid FAQ/HowTo markup will correctly show "no eligible rich results" — that is expected, not an error.

Usage in Next.js

// app/layout.tsx
import { JsonLd } from '@/components/seo/json-ld';

export default function RootLayout({ children }) {
  return (
    <html lang="en">
      <body>
        <JsonLd data={websiteSchema} />
        <JsonLd data={organizationSchema} />
        {children}
      </body>
    </html>
  );
}

Testing Tools

1. Google Rich Results Test: https://search.google.com/test/rich-results 2. Schema.org Validator: https://validator.schema.org/ 3. JSON-LD Playground: https://json-ld.org/playground/

Best Practices

1. Match visible content - JSON-LD must reflect what users see 2. Use XSS protection - Always escape < characters 3. Don't duplicate - One schema type per page (except @graph) 4. Keep updated - Update dateModified when content changes 5. Test regularly - Validate after changes

Next.js Metadata API

Complete guide for implementing SEO metadata in Next.js App Router.

Contents

• Static vs Dynamic Metadata (generateMetadata full signature, parent, memoization)
• Complete Metadata Object (incl. facebook, pinterest, appleWebApp, other)
• Viewport Configuration
• Metadata Merging (shallow-merge gotcha)
• File-based metadata & priority
• OG / Twitter images: file conventions + ImageResponse + generateImageMetadata
• Web App Manifest & icon file conventions
• generateMetadata with Cache Components
• Open Graph image sizes / Twitter card types
• Streaming Metadata
• Best Practices

Static vs Dynamic Metadata

Static Metadata (metadata object)

Use when metadata is known at build time:

// app/layout.tsx or app/page.tsx
import type { Metadata } from 'next';

export const metadata: Metadata = {
  title: 'Page Title',
  description: 'Page description',
};

Dynamic Metadata (generateMetadata)

Use when metadata depends on route params or external data:

// app/products/[id]/page.tsx
import type { Metadata, ResolvingMetadata } from 'next';

type Props = {
  params: Promise<{ id: string }>;
  searchParams: Promise<{ [key: string]: string | string[] | undefined }>; // page.js only
};

export async function generateMetadata(
  { params }: Props,
  parent: ResolvingMetadata, // optional 2nd arg: read/extend parent metadata
): Promise<Metadata> {
  const { id } = await params; // params & searchParams are Promises in current Next.js
  const product = await getProduct(id);

  // Extend rather than replace parent OG images:
  const previousImages = (await parent).openGraph?.images || [];

  return {
    title: product.name,
    description: product.description,
    openGraph: {
      images: [product.image, ...previousImages],
    },
  };
}

Notes:

• searchParams is only available in page.js segments (not layout.js).
• redirect() and notFound() can be called inside generateMetadata (useful when a fetched entity doesn't exist).
• v16 typed helpers: type the first arg with PageProps<'/products/[id]'> or LayoutProps<'/...'> instead of a hand-rolled Props type.
• Avoid duplicate fetches: fetch() is auto-memoized between generateMetadata and the page. For non-fetch data (DB/ORM), wrap the loader in React's cache() so it runs once.

Complete Metadata Object

import type { Metadata } from 'next';

export const metadata: Metadata = {
  // Base URL for relative paths
  metadataBase: new URL('https://your-site.com'),

  // Title configuration
  title: {
    default: 'Default Title',        // Used when no page title
    template: '%s | Site Name',      // Template for child pages
    absolute: 'Override All',        // Ignores template
  },

  // Description (150-160 characters recommended)
  description: 'Compelling meta description with target keywords',

  // Keywords (less important now, but still used)
  keywords: ['keyword1', 'keyword2', 'long-tail keyword'],

  // Author information
  authors: [{ name: 'Author Name', url: 'https://author.com' }],
  creator: 'Creator Name',
  publisher: 'Publisher Name',

  // Robots directives
  robots: {
    index: true,
    follow: true,
    nocache: false,
    googleBot: {
      index: true,
      follow: true,
      noimageindex: false,
      'max-video-preview': -1,
      'max-image-preview': 'large',
      'max-snippet': -1,
    },
  },

  // Canonical and alternates
  alternates: {
    canonical: '/',
    languages: {
      'en-US': '/en-US',
      'fi-FI': '/fi-FI',
    },
    media: { 'only screen and (max-width: 600px)': 'https://m.your-site.com' },
    types: { 'application/rss+xml': 'https://your-site.com/rss' }, // advertise feeds
  },

  // Open Graph (Facebook, LinkedIn)
  openGraph: {
    type: 'website',
    locale: 'en_US',
    url: 'https://your-site.com',
    siteName: 'Site Name',
    title: 'Open Graph Title',
    description: 'Open Graph description',
    images: [
      {
        url: '/og-image.png',
        width: 1200,
        height: 630,
        alt: 'Image alt text',
        type: 'image/png',
      },
    ],
  },

  // Twitter Cards
  twitter: {
    card: 'summary_large_image',  // or 'summary' for square images
    site: '@username',
    creator: '@creator',
    title: 'Twitter Title',
    description: 'Twitter description',
    images: ['/twitter-image.png'],
  },

  // Icons
  icons: {
    icon: '/favicon.ico',
    shortcut: '/favicon-16x16.png',
    apple: '/apple-touch-icon.png',
  },

  // Verification tags
  verification: {
    google: 'google-verification-code',
    yandex: 'yandex-verification-code',
  },

  // App links
  appLinks: {
    ios: {
      url: 'https://app.example.com/ios',
      app_store_id: 'app_store_id',
    },
    android: {
      package: 'com.example.app',
      app_name: 'App Name',
    },
  },

  // Format detection (disable auto-linking)
  formatDetection: {
    email: false,
    address: false,
    telephone: false,
  },

  // Category
  category: 'technology',

  // PWA manifest link (or use app/manifest.ts — see below)
  manifest: '/manifest.webmanifest',

  // Social platform extras
  facebook: { appId: '1234567890' },     // Facebook Social Plugins
  pinterest: { richPin: true },          // Pinterest Rich Pins
  appleWebApp: { capable: true, title: 'Site', statusBarStyle: 'default' },

  // Escape hatch for custom / newly-released meta tags not yet typed
  other: { 'custom-tag': 'value' },
};

Viewport Configuration

Important: viewport must be a separate export, not a field in metadata:

import type { Viewport } from 'next';

export const viewport: Viewport = {
  width: 'device-width',
  initialScale: 1,
  maximumScale: 5,
  userScalable: true,
  viewportFit: 'cover',
  themeColor: [
    { media: '(prefers-color-scheme: light)', color: '#ffffff' },
    { media: '(prefers-color-scheme: dark)', color: '#0a0a0a' },
  ],
  colorScheme: 'light dark',
};

Metadata Merging

Metadata merges from root to leaf. Child metadata overrides parent:

app/layout.tsx (base metadata)
  └── app/blog/layout.tsx (adds/overrides)
        └── app/blog/[slug]/page.tsx (final metadata)

Shallow-merge gotcha: merging is shallow. Redefining a nested object like openGraph or robots in a child segment replaces the entire parent object — sibling keys are lost. Setting only openGraph.title in a child drops the parent's openGraph.description/images. Fix by exporting shared nested fields and spreading them:

// app/shared-metadata.ts
export const sharedOpenGraph = { images: ['/og-image.png'], siteName: 'Site Name' };

// child segment
export const metadata = {
  openGraph: { ...sharedOpenGraph, title: 'Child title' },
};

File-based Metadata & Priority

File conventions (favicon.ico, icon.*, apple-icon.*, opengraph-image.*, twitter-image.*, manifest.*, sitemap.*, robots.*) take priority over and override the metadata object / generateMetadata. Next.js recommends file conventions for icons and OG images over hand-syncing the icons/openGraph.images config — and avoid using both for the same asset to prevent duplicate <head> tags.

OG / Twitter Images (file conventions + ImageResponse)

Three approaches (SKILL.md has the quick version; details here):

1. External URL — set openGraph.images / twitter.images in metadata (shown above). Use for externally hosted images.

2. Static file convention (recommended default): place opengraph-image.(jpg|jpeg|png|gif) / twitter-image.* in a route segment. Next.js emits og:image/twitter:image + :type/:width/:height. A deeper segment's image overrides one above. Alt text via a sibling opengraph-image.alt.txt (→ og:image:alt). Build fails if a static file exceeds 8 MB (OG) / 5 MB (Twitter).

3. Code-generated with `ImageResponse`:

// app/blog/[slug]/opengraph-image.tsx
import { ImageResponse } from 'next/og';
import { readFile } from 'node:fs/promises';
import { join } from 'node:path';

export const alt = 'Post preview';            // → og:image:alt
export const size = { width: 1200, height: 630 }; // → og:image:width/height
export const contentType = 'image/png';       // → og:image:type

export default async function Image({ params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params;              // params is a Promise in v16
  const post = await getPost(slug);
  const font = await readFile(join(process.cwd(), 'assets/Inter-SemiBold.ttf'));

  return new ImageResponse(
    (
      <div style={{ display: 'flex', width: '100%', height: '100%', fontSize: 64 }}>
        {post.title}
      </div>
    ),
    { ...size, fonts: [{ name: 'Inter', data: font, style: 'normal', weight: 600 }] },
  );
}

• Default export must return one of Blob | ArrayBuffer | TypedArray | DataView | ReadableStream | Response — ImageResponse satisfies this.
• Satori rendering: flexbox + a subset of CSS only; display: grid is unsupported. Load local images via readFile (base64 data URI) under the Node.js runtime.
• Caching: these are special Route Handlers, statically optimized (built once, cached) unless they read request-time APIs or uncached data; they accept the same route segment config as pages.
• Multiple images per route: export generateImageMetadata() returning an array of { id (required), alt?, size?, contentType? }; the default Image({ id, params }) receives both as Promises (v16).

Web App Manifest & Icon File Conventions

Manifest: app/manifest.ts returning MetadataRoute.Manifest (see SKILL.md for the full example) — or a static app/manifest.(json|webmanifest).

Icons (prefer over the metadata `icons` field):

• favicon.ico — root app/ only; appears in browser tabs and Google SERPs.
• app/icon.(ico|jpg|jpeg|png|svg) and app/apple-icon.(jpg|jpeg|png) — auto-emit <link rel="icon"> / apple-touch-icon with correct type/sizes.
• Code-generated: app/icon.tsx / app/apple-icon.tsx with ImageResponse (export size, contentType). Note: favicon.ico cannot be code-generated — use icon.* or a static .ico.
• Multiple icons via numeric suffixes (icon1.png, icon2.png); .svg icons get sizes="any".

generateMetadata with Cache Components

When cacheComponents is enabled, a generateMetadata that reads runtime data (cookies/headers/searchParams or uncached fetches) while the rest of the page is prerenderable raises an error requiring an explicit choice:

• External (non-runtime) data: add "use cache" inside generateMetadata (with cacheTag for invalidation).
• Genuine runtime data: signal intent with a DynamicMarker component (await connection()) inside a <Suspense> boundary so the page can still prerender a static shell.

Open Graph Image Sizes

Twitter Card Types

Streaming Metadata

For dynamically rendered pages, generateMetadata resolves as part of rendering, and the resulting tags are appended to the `<body>` once it resolves — without blocking the initial UI. This improves TTFB/LCP. (Prerendered/static pages resolve metadata at build time and put it in <head> normally — no streaming.)

• JS-capable bots (Googlebot): read the streamed tags after executing JS and inspecting the full DOM.
• HTML-limited bots: metadata keeps blocking and is placed in <head>. Next.js detects these by User-Agent; the built-in list includes Twitterbot, Slackbot, Bingbot, facebookexternalhit, and more.

htmlLimitedBots overrides (replaces) the entire built-in list — it does NOT append to it:

// next.config.ts
import type { NextConfig } from 'next';

const config: NextConfig = {
  // Fully DISABLE streaming (all bots get blocking metadata):
  htmlLimitedBots: /.*/,

  // ⚠️ A narrow regex like /facebookexternalhit|linkedinbot/ is DANGEROUS:
  // it REPLACES the default list, so Bingbot/Twitterbot/Slackbot would lose
  // their blocking metadata and get broken previews. Only override if you
  // fully understand you're replacing the whole list.
};

export default config;

Streaming metadata is an advanced feature — the default is correct for almost all cases, so usually you should not set htmlLimitedBots at all.

Best Practices

1. Always set metadataBase - Required for relative URLs. URL composition: a missing metadataBase + a relative URL = build error; an absolute URL in any field ignores metadataBase. OG/Twitter image URLs must resolve to absolute URLs. 2. Use title templates - Consistent branding across pages 3. Write unique descriptions - Each page needs unique description 4. Include canonical URLs - Prevent duplicate content issues 5. Test with validators - Use the Facebook Sharing Debugger; for X, preview in the post composer or use a third-party OG preview tool (e.g. opengraph.xyz) 6. Don't mix static and dynamic - Use either metadata object or generateMetadata in the same route segment (a layout can use static metadata while its child page uses generateMetadata) 7. `themeColor`/`colorScheme`/`viewport` are deprecated inside `metadata` - use the separate export const viewport (see above)

Sitemap & Robots.txt in Next.js

Contents

• Sitemap Configuration — basic, dynamic, image, video, multiple, localized sitemaps
• Robots.txt Configuration
• Static file conventions
• Sitemap Best Practices
• Robots.txt Best Practices

Sitemap Configuration

Basic Static Sitemap

// app/sitemap.ts
import type { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://your-site.com',
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 1,
    },
    {
      url: 'https://your-site.com/about',
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
  ];
}

Dynamic Sitemap with Database

// app/sitemap.ts
import type { MetadataRoute } from 'next';
import { getAllPosts } from '@/lib/posts';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://your-site.com';
  const posts = await getAllPosts();

  const postUrls = posts.map((post) => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly' as const,
    priority: 0.7,
  }));

  return [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    ...postUrls,
  ];
}

Image Sitemap

// app/sitemap.ts
import type { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  const baseUrl = 'https://your-site.com';

  return [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 1,
      images: [
        `${baseUrl}/og-image.png`,
        `${baseUrl}/hero-image.jpg`,
      ],
    },
  ];
}

Video Sitemap

// app/sitemap.ts
import type { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://your-site.com/video-page',
      lastModified: new Date(),
      videos: [
        {
          title: 'Video Title',
          thumbnail_loc: 'https://your-site.com/thumbnail.jpg',
          description: 'Video description',
        },
      ],
    },
  ];
}

Multiple Sitemaps (Large Sites)

// app/sitemap.ts
import type { MetadataRoute } from 'next';

export async function generateSitemaps() {
  // Return array of sitemap IDs
  return [{ id: 0 }, { id: 1 }, { id: 2 }];
}

export default async function sitemap(props: {
  id: Promise<string>;
}): Promise<MetadataRoute.Sitemap> {
  const id = await props.id;
  const start = Number(id) * 50000;
  const end = start + 50000;

  const products = await getProducts(start, end);

  return products.map((product) => ({
    url: `https://your-site.com/products/${product.id}`,
    lastModified: product.updatedAt,
  }));
}
// Generates: /sitemap/0.xml, /sitemap/1.xml, /sitemap/2.xml

Note: Sitemaps can ALSO be split by nesting sitemap.(xml|ts|js) under

route segments (e.g. app/products/sitemap.ts). Generated multi-sitemaps are

served at /.../sitemap/[id].xml relative to the file's route segment — so a

root app/sitemap.ts with generateSitemaps yields /sitemap/0.xml, while

app/products/sitemap.ts yields /products/sitemap/0.xml.

Localized Sitemap

// app/sitemap.ts
import type { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://your-site.com',
      lastModified: new Date(),
      alternates: {
        languages: {
          en: 'https://your-site.com/en',
          fi: 'https://your-site.com/fi',
          sv: 'https://your-site.com/sv',
        },
      },
    },
  ];
}

Robots.txt Configuration

Basic Robots.txt

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/api/', '/admin/'],
      // Never disallow /_next/ — crawlers need render-critical CSS/JS
    },
    sitemap: 'https://your-site.com/sitemap.xml',
  };
}

Multiple User Agents

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/'],
      },
      {
        userAgent: 'Googlebot',
        allow: '/',
        disallow: '/admin/',
        crawlDelay: 2, // optional; Googlebot ignores crawl-delay, Bing/Yandex honor it
      },
      {
        userAgent: 'GPTBot',
        disallow: '/', // Opts out of OpenAI model TRAINING only (not citation/search)
      },
    ],
    sitemap: 'https://your-site.com/sitemap.xml',
    host: 'https://your-site.com',
  };
}

`host` caveat: host is type-valid but a **non-standard directive Google

ignores** (originally Yandex-only). Prefer canonical URLs / 301 redirects to

declare the preferred host.

AI crawlers: Blanket-blocking GPTBot only opts out of training — it

does not block citation/search bots. Citation bots (OAI-SearchBot,

PerplexityBot) should usually stay allowed so your content can be cited.

AI crawler control (training vs search/citation bots, the full 2026 user-agent

list, and a recommended pattern) lives in ai-search.md.

Environment-Based Robots

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_BASE_URL || 'https://your-site.com';

  // Block indexing on non-production
  if (process.env.NODE_ENV !== 'production') {
    return {
      rules: {
        userAgent: '*',
        disallow: '/',
      },
    };
  }

  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/api/', '/admin/'],
    },
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

Static file conventions

Hand-authored app/sitemap.xml and app/robots.txt files are also valid first-class conventions — good alternatives to the programmatic .ts forms for small or simple sites that don't need dynamic generation.

# app/robots.txt
User-Agent: *
Allow: /
Disallow: /private/

Sitemap: https://your-site.com/sitemap.xml

<!-- app/sitemap.xml -->
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://your-site.com</loc>
    <lastmod>2026-01-01</lastmod>
  </url>
</urlset>

Sitemap Best Practices

Google ignores `priority` and `changeFrequency` — only lastModified

(lastmod) is used, and only when accurate. Set lastmod from real

content-update timestamps; don't over-invest in priority tuning.

Robots.txt Best Practices

1. Don't block CSS/JS - Google needs them for rendering 2. Don't block sitemap - Never disallow /sitemap.xml 3. Use specific paths - /admin/ instead of broad blocks 4. Test before deploy - Use the Search Console robots.txt report (Settings → robots.txt) and the URL Inspection tool

MetadataRoute.Robots fields

Per-rule fields: userAgent, allow, disallow, crawlDelay?: number. Top-level fields: sitemap, host.

• crawlDelay?: number — seconds between requests. **Googlebot ignores

crawl-delay; Bing/Yandex honor it.**

• host — non-standard, ignored by Google (see host caveat above).

SEO Troubleshooting Guide

Contents

• Google Indexing Issues
• Google Search Console Usage
• Common Technical Issues
• Building Authority
• Timeline Expectations
• Debug Checklist
• Tools

Google Indexing Issues

"Discovered - currently not indexed"

Meaning: Google found the URL but hasn't crawled it yet.

Causes:

• New website (low crawl priority)
• Low-quality signals
• Crawl budget exhaustion

Solutions: 1. Request indexing via URL Inspection tool 2. Build quality backlinks 3. Improve internal linking 4. Wait (can take weeks for new sites)

"Crawled - currently not indexed"

Meaning: Google crawled but chose not to index.

Causes:

• Thin content
• Duplicate content
• Low-quality content
• Technical issues

Solutions: 1. Add more unique, valuable content 2. Check for duplicate content issues 3. Ensure canonical URLs are correct 4. Improve E-E-A-T signals (Experience, Expertise, Authoritativeness, Trust)

"URL is not on Google"

Meaning: Page is not in Google's index.

Steps: 1. Check robots.txt isn't blocking 2. Check for noindex meta tag 3. Check canonical URL points to correct page 4. Request indexing in GSC

"Blocked by robots.txt"

Solution: Update app/robots.ts:

// Remove the blocking rule
export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      // Remove or fix disallow rules
    },
  };
}

Google Search Console Usage

URL Inspection Tool

1. Go to Google Search Console 2. Enter URL in search bar at top 3. Check:

• "URL is on Google" status
• "Page fetch" success
• "Indexing allowed" status
• "User-declared canonical"
• "Google-selected canonical"

Request Indexing

1. Use URL Inspection tool 2. Click "Request Indexing" 3. Wait (don't spam - once is enough) 4. Check back in 1-2 weeks

Pages Report

Navigate to: Indexing > Pages

Common Technical Issues

JavaScript Rendering Problems

Symptom: Content missing in URL Inspection → View Crawled Page rendered HTML.

Solutions: 1. Use SSR/SSG instead of CSR for SEO content 2. Check with URL Inspection "View Crawled Page" 3. Ensure critical content is in initial HTML

Duplicate Content

Symptom: Multiple URLs with same content.

Solutions:

// Set canonical URL
export const metadata: Metadata = {
  alternates: {
    canonical: '/correct-url',
  },
};

Redirect Chains

Symptom: Multiple redirects (A → B → C).

Solution: Redirect directly to final URL:

// next.config.ts
export default {
  async redirects() {
    return [
      {
        source: '/old-url',
        destination: '/final-url', // Direct to final
        permanent: true,
      },
    ];
  },
};

Slow Page Speed

Symptom: High LCP, poor Core Web Vitals.

Solutions: 1. Use next/image for images 2. Use next/font for fonts 3. Implement lazy loading 4. Reduce JavaScript bundle size 5. Use SSG where possible

Accidentally blocked from AI search

Symptom: Site not appearing or cited in AI answers (ChatGPT, Perplexity, Google AI Overviews).

Causes:

• Old "block all AI" robots.txt templates
• WAF/CDN rules disallowing citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
• Only deprecated tokens (anthropic-ai/Claude-Web) targeted, leaving real bots blocked or unhandled

Fix: 1. Allow citation bots in robots.txt 2. Audit WAF/CDN rules together with robots.txt 3. Verify access via server logs

See ai-search.md for AI crawler rules and GEO guidance.

Building Authority

For New Sites

1. Submit to GSC - Add sitemap 2. Build backlinks - Quality over quantity 3. Social signals - Share content 4. Directory listings - Relevant directories 5. Guest posts - Industry blogs

Backlink Sources

Timeline Expectations

Debug Checklist

When a page isn't indexed:

1. [ ] Check robots.txt allows crawling 2. [ ] Check no noindex tag 3. [ ] Check canonical URL is correct 4. [ ] Check page returns 200 status 5. [ ] Check content is valuable and unique 6. [ ] Check page is linked from other pages 7. [ ] Use URL Inspection tool 8. [ ] Request indexing (once) 9. [ ] Wait and monitor

Tools