Seo Technical

Technical SEO Audit

Categories

1. Crawlability

• robots.txt: exists, valid, not blocking important resources
• XML sitemap: exists, referenced in robots.txt, valid format
• Noindex tags: intentional vs accidental
• Crawl depth: important pages within 3 clicks of homepage
• JavaScript rendering: check if critical content requires JS execution
• Crawl budget: for large sites (>10k pages), efficiency matters

AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

Known AI crawlers:

Key distinctions:

• Blocking Google-Extended prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use Googlebot)
• Blocking GPTBot prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (ChatGPT-User)
• ~3-5% of websites now use AI-specific robots.txt rules

Example, selective AI crawler blocking:

# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /

Recommendation: Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the seo-geo skill for full AI visibility optimization.

2. Indexability

• Canonical tags: self-referencing, no conflicts with noindex
• Duplicate content: near-duplicates, parameter URLs, www vs non-www
• Thin content: pages below minimum word counts per type
• Pagination: rel=next/prev or load-more pattern
• Hreflang: correct for multi-language/multi-region sites
• Index bloat: unnecessary pages consuming crawl budget

3. Security

• HTTPS: enforced, valid SSL certificate, no mixed content
• Security headers:
• Content-Security-Policy (CSP)
• Strict-Transport-Security (HSTS)
• X-Frame-Options
• X-Content-Type-Options
• Referrer-Policy
• HSTS preload: check preload list inclusion for high-security sites

4. URL Structure

• Clean URLs: descriptive, hyphenated, no query parameters for content
• Hierarchy: logical folder structure reflecting site architecture
• Redirects: no chains (max 1 hop), 301 for permanent moves
• URL length: flag >100 characters
• Trailing slashes: consistent usage

5. Mobile Optimization

• Responsive design: viewport meta tag, responsive CSS
• Touch targets: minimum 48x48px with 8px spacing
• Font size: minimum 16px base
• No horizontal scroll
• Mobile-first indexing: Google indexes mobile version. Mobile-first indexing is 100% complete as of July 5, 2024. Google now crawls and indexes ALL websites exclusively with the mobile Googlebot user-agent.

6. Core Web Vitals

• LCP (Largest Contentful Paint): target <2.5s
• INP (Interaction to Next Paint): target <200ms
• INP replaced FID on March 12, 2024. FID was fully removed from all Chrome tools (CrUX API, PageSpeed Insights, Lighthouse) on September 9, 2024. Do NOT reference FID anywhere.
• CLS (Cumulative Layout Shift): target <0.1
• Evaluation uses 75th percentile of real user data
• Use PageSpeed Insights API or CrUX data if MCP available

7. Structured Data

• Detection: JSON-LD (preferred), Microdata, RDFa
• Validation against Google's supported types
• See seo-schema skill for full analysis

8. JavaScript Rendering

• Check if content visible in initial HTML vs requires JS
• Identify client-side rendered (CSR) vs server-side rendered (SSR)
• Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues
• Verify dynamic rendering setup if applicable

JavaScript SEO: Canonical & Indexing Guidance (December 2025)

Google updated its JavaScript SEO documentation in December 2025 with critical clarifications:

1. Canonical conflicts: If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server-rendered HTML and JS-rendered output. 2. noindex with JavaScript: If raw HTML contains <meta name="robots" content="noindex"> but JavaScript removes it, Google MAY still honor the noindex from raw HTML. Serve correct robots directives in the initial HTML response. 3. Non-200 status codes: Google does NOT render JavaScript on pages returning non-200 HTTP status codes. Any content or meta tags injected via JS on error pages will be invisible to Googlebot. 4. Structured data in JavaScript: Product, Article, and other structured data injected via JS may face delayed processing. For time-sensitive structured data (especially e-commerce Product markup), include it in the initial server-rendered HTML.

Best practice: Serve critical SEO elements (canonical, meta robots, structured data, title, meta description) in the initial server-rendered HTML rather than relying on JavaScript injection.

9. IndexNow Protocol

• Check if site supports IndexNow for Bing, Yandex, Naver
• Supported by search engines other than Google
• Recommend implementation for faster indexing on non-Google engines

Agent-Friendly Pages (forward-looking)

AI agents (not just AI summarizers) increasingly read sites through three channels: vision models on screenshots, raw HTML/DOM, and the accessibility tree (the cleanest signal). Audit criteria — semantic HTML (real <button> and <a>, not <div onclick>), label associations, interactive target sizing, layout stability across templates, cursor: pointer correctness — live in references/agent-friendly-pages.md.

Audit command

# Render with Playwright + capture accessibility tree, then score
python3 scripts/agent_ux_check.py https://example.com --json

The scanner outputs an Agent-UX score (0-100) plus itemized issues:

• HTML findings: real buttons / anchors, <div onclick> widgets, semantic

landmarks, inputs without <label for>, inputs without ARIA labels

• Accessibility tree findings: total nodes, interactive nodes, unnamed

interactive elements, role="generic" ratio

The accessibility-tree snapshot uses Playwright's page.accessibility.snapshot(interesting_only=False). To capture the tree without scoring, use python3 scripts/render_page.py <url> --a11y-tree --json.

Surface findings as opportunities, not failures. The standards (WebMCP, agent UX heuristics) are early — don't gate audits on a sub-100 score.

Output

Technical Score: XX/100

Category Breakdown

Critical Issues (fix immediately)

High Priority (fix within 1 week)

Medium Priority (fix within 1 month)

Low Priority (backlog)

DataForSEO Integration (Optional)

If DataForSEO MCP tools are available, use on_page_instant_pages for real page analysis (status codes, page timing, broken links, on-page checks), on_page_lighthouse for Lighthouse audits (performance, accessibility, SEO scores), and domain_analytics_technologies_domain_technologies for technology stack detection.

Google API Integration (Optional)

If Google API credentials are configured, use python3 scripts/pagespeed_check.py <url> --json for real PSI + CrUX field data (replaces lab-only CWV estimates), python3 scripts/crux_history.py <url> --json for 25-week CWV trends, and python3 scripts/gsc_inspect.py <url> --json for real indexation status per URL.

Error Handling

Agent-friendly pages — audit reference (May 2026)

The next wave of AI search is not summarization — it's agents acting on the user's behalf (search, compare, buy, book). Google's AI optimization guide and the linked web.dev article describe three channels through which agents interpret your site:

1. Screenshots + vision model — interprets visual hierarchy, button prominence, layout. Slow and token-expensive. 2. Raw HTML / DOM — nesting, IDs, classes, data attributes. 3. The accessibility tree — the browser-native semantic distillation (roles, names, states). The cleanest signal of the three.

Modern agents combine all three. Optimizing for the accessibility tree is the single highest-leverage move; if your accessibility tree is broken, no amount of visual polish saves you.

Primary sources:

• Google AI optimization guide:

https://developers.google.com/search/docs/fundamentals/ai-optimization-guide

• web.dev (referenced from the guide above): article on building

agent-friendly websites

Audit checklist

1. Use real interactive elements

If you cannot use a real interactive tag, supply ARIA: role="button", role="link", tabindex="0", plus key handlers for Enter and Space.

Why it matters: the accessibility tree exposes real interactive elements with their roles. Custom div widgets often appear in the tree with no role at all — agents skip them.

2. Label associations

Every form input must have an associated label:

<label for="email">Email</label>
<input id="email" type="email" name="email">

Or use aria-label / aria-labelledby where a visible label isn't possible. Agents that read the accessibility tree get the field purpose directly from the associated label — without it, the input is a void.

3. Interactive target size

Visual-analysis pipelines filter out interactive elements smaller than ~8 square pixels of unobscured area. Tap-target accessibility minimums (24×24 WCAG AA, 44×44 Apple HIG) are stricter and pass the agent gate by default.

Audit: any clickable element below 24×24px is a candidate for agent invisibility, in addition to the WCAG failure.

4. Don't cover interactive nodes with transparent overlays

Vision models discard covered nodes when computing what's "interactive at this position". Common offenders:

• Full-card click handlers that overlay every child link.
• Transparent cookie-consent layers persisting beyond consent.
• Modal portals with pointer-events: auto left on after dismiss.
• "Ghost" tracking pixels with position: absolute; inset: 0.

5. Layout stability

If "Add to cart" lives in different positions on /category/shoes vs /category/bags, screenshot-based agents have to relearn the page each visit. Keep functionally identical actions in the same screen quadrant across templates.

Cross-reference: this overlaps with CLS (Cumulative Layout Shift) in Core Web Vitals, but the agent-UX concern is broader — it covers page-to-page stability, not just within-page shift.

6. cursor: pointer as a legitimate signal

Vision models read cursor: pointer (set by default on <a> / <button>) as a hint that an element is actionable. Do not override it to cursor: default on truly interactive elements just for visual minimalism.

Inverse: do not apply cursor: pointer to non-interactive elements — that makes agents click things that do nothing.

7. Stable, meaningful selectors

Agents that fall back to DOM parsing rely on:

• Real semantic tags (<nav>, <main>, <article>, <section>, <aside>)
• Stable id attributes on top-level layout containers
• data-* attributes that describe purpose, not implementation

Avoid auto-generated class names like __sc_a4b7d9e2 as the only handle on a critical interactive element — agents can target them but cannot tell what they mean.

Forward-looking: WebMCP

Google's AI optimization guide name-drops WebMCP, a proposed standard for direct site-to-agent interaction (analogous to MCP in the Claude / Anthropic ecosystem, but operating at the page level). There is an early preview program; broad adoption is not expected before 2027.

Audit posture: mention WebMCP in reports as a forward-looking signal worth tracking. Do not flag the absence of WebMCP support as a finding — the standard is not yet stable.

Quick-audit one-liner

For a fast smoke check, capture the accessibility tree via Lighthouse or Chrome DevTools and look for:

• Any interactive element with role="generic" → broken semantics.
• Any input without an accessible name → missing label.
• Any <div> with onclick and no role / tabindex → custom widget that

agents won't see.

scripts/render_page.py --mode auto already loads pages headlessly; extending it with an accessibility-tree dump (page.accessibility.snapshot() in Playwright) is the natural place to land an automated agent-UX check in a future iteration.

Last verified

2026-05-18. Update when:

• WebMCP graduates from preview to stable.
• Google publishes a separate agent-UX scoring framework.
• web.dev publishes a follow-up article with revised criteria.