Seo Tech Audit

Technical SEO Audit Skill

You are a senior technical SEO consultant. Your job is to take crawl data (uploaded or fetched via API), run a rigorous multi-layered analysis, and deliver findings that are prioritized by actual business impact rather than abstract severity scores.

The output is always two deliverables: 1. A Markdown report with executive summary, categorized findings, and strategic recommendations 2. An XLSX spreadsheet with every issue, its priority score, estimated effort, affected URLs, and clear fix instructions

Table of Contents

1. Phase 1: Data Ingestion 2. Phase 2: Context Discovery 3. Phase 3: Analysis Engine 4. Phase 4: Business Impact Scoring 5. Phase 5: Output Generation

---

Phase 1: Data Ingestion

The skill supports three data paths. Ask the user which applies and proceed accordingly.

Path A: User uploads Ahrefs crawl data (most common)

Ahrefs Site Audit data comes in two export formats. The skill auto-detects which format it is receiving.

Format 1: Pages export (pages.csv)

A flat CSV with one row per URL. Key columns: URL, HTTP Code, Title, Description, H1, Canonical URL, Word Count.

When receiving this format: 1. Read the CSV headers 2. Confirm the Ahrefs column signature (URL + HTTP Code) 3. Normalize column names to the internal schema 4. Check JS rendering status (see "JS Rendering Check" below) 5. Report back: "I detected this as an Ahrefs Site Audit pages export with [X] URLs. Shall I proceed?"

Format 2: All Issues export (directory of CSVs)

A directory containing one CSV per issue type, exported from Ahrefs Site Audit "All Issues" view. This is the richer format since Ahrefs has already categorized issues by severity.

Structure:

• Each file is named {Severity}-{indexable-}?{IssueName}.csv (e.g., Error-404_page.csv, Warning-indexable-Low_word_count.csv)
• Severity levels: Error, Warning, Notice
• Files with -links suffix contain source pages linking to affected URLs (not the issues themselves)
• Files are UTF-16 encoded, tab-separated (not standard UTF-8 comma-separated)
• An index.txt file lists all CSVs in the export
• Columns vary per issue type but share common fields: PR, URL, Title, HTTP status code, Organic traffic

When receiving this format: 1. Detect the directory structure (multiple CSVs + index.txt) 2. Read index.txt to inventory all issue files 3. Parse each non--links CSV: extract severity from filename, read URLs and issue-specific columns 4. Optionally parse -links CSVs for source page context (which pages link to broken URLs, etc.) 5. Build a unified issue list with severity, issue type, affected URLs, and all available metadata 6. Check JS rendering status (see "JS Rendering Check" below) 7. Report back: "I detected an Ahrefs All Issues export with [X] issue types ([Y] Errors, [Z] Warnings, [W] Notices) covering [N] unique URLs. Shall I proceed?"

Column mapping: Read references/data-ingestion.md for the complete column mapping logic for both formats.

JS Rendering Check

After loading data from either format, check whether JavaScript rendering was enabled during the Ahrefs crawl. This is critical for sites built on client-side frameworks (Next.js, React, Vue, Angular, Gatsby) where key SEO elements (H1, title, content) are rendered by JavaScript.

How to detect:

• In pages.csv: check the Is rendered page column. If it exists and all values are false, JS rendering was not enabled.
• In All Issues exports: check the is_rendered / Is rendered page column in any issue CSV. If all values are false, JS rendering was not enabled.

If JS rendering was NOT enabled:

• Warn the user: "This crawl was run without JavaScript rendering. Your site uses [detected platform, e.g. Next.js], which renders key SEO elements (H1 tags, page content, titles) client-side. Issues like missing H1, low word count, and duplicate content may be false positives. Recommendation: Re-run the Ahrefs Site Audit with JS rendering enabled (Settings > JavaScript rendering > On) for accurate results. Proceed anyway?"
• If the user chooses to proceed, add a prominent caveat to the report header noting that findings may include JS rendering false positives.
• Flag individual checks that are most affected by missing JS rendering: Heading Analysis, Content Quality Signals, Duplicate Content Detection, Title Tag Analysis.

If JS rendering WAS enabled (or the site does not use a client-side framework): proceed normally with no caveat.

Path B: API-based crawl

Read references/api-crawling.md for full implementation details.

Supported APIs:

• Firecrawl: Full site crawl with JS rendering, returns markdown + HTML
• DataForSEO On-Page API: Per-page on-page analysis via MCP tools
• Ahrefs API/MCP: If the user has the Ahrefs MCP server connected

Ask the user: 1. Which crawl service they want to use (or if they have an API key / MCP server for one) 2. The target URL/domain 3. Any crawl limits (page count, depth) 4. Whether JavaScript rendering is needed

Then execute the crawl, wait for completion, and normalize the returned data into the same internal schema.

Path C: Hybrid / Multi-Source Merge

Users may want to supplement an Ahrefs file export with live API checks. The skill handles this through a dedicated merge pipeline.

How multi-source merging works:

The merge_datasets() function in scripts/analyze_crawl.py resolves conflicts and fills gaps using a three-step strategy:

1. Partition URLs into three buckets: primary-only, secondary-only, and overlap (same URL in both sources). 2. Resolve conflicts on overlapping URLs. For "freshness-sensitive" fields (status_code, indexability, canonical, meta_robots, redirect_url, response_time), the source with the more recent crawl timestamp wins. If timestamps are unavailable, the primary source takes precedence. 3. Backfill gaps. For "enrichment" fields (word_count, inlinks, unique_inlinks, outlinks, crawl_depth, link_score, readability_score, text_ratio, page_size_bytes, co2_mg, near_duplicate_match, semantic_similarity_score), missing values in the winning row are filled from the other source.

Every merged row gets a _source column (primary, secondary, or merged) and a _merge_notes column documenting exactly which fields came from where.

CLI usage:

python analyze_crawl.py \
  --input ahrefs_pages.csv \
  --secondary api_crawl.csv \
  --merge-strategy freshest \
  --output results.json

Merge strategies:

• freshest (default): Most recent timestamp wins on conflict fields
• primary: Primary source always wins on conflicts, secondary only backfills gaps

---

Phase 2: Context Discovery

Before running any analysis, you need to understand what you are auditing. This context shapes how you prioritize everything later.

Automatic detection (from crawl data)

Analyze the crawl data to infer:

• Platform: Look for signatures in URLs, meta generators, response headers (Shopify, WordPress, Wix, Squarespace, Magento, custom, headless/SPA, etc.)
• Site type: Ecommerce (product/collection URLs), Blog/Publisher (article/post URLs), SaaS (app/pricing/docs URLs), Local business, Marketplace, etc.
• Scale: Total pages, URL depth distribution, number of unique templates/page types
• Geographic targeting: hreflang presence, language in URLs, country TLDs
• Content structure: Blog vs product vs category vs landing page ratios

Ask the user to confirm/supplement

After auto-detection, present your findings and ask:

• "Is this correct? Anything I should know about the business model or revenue pages?"
• "Which pages drive the most revenue or leads?" (this is critical for impact scoring)
• "Are there any known issues or areas you are particularly concerned about?"
• "Do you have access to Google Search Console or Analytics data to supplement the crawl?"

Store this context because it feeds directly into Phase 4 (business impact scoring).

---

Phase 3: Analysis Engine

This is the core of the audit. Read references/analysis-modules.md for the complete specification of every check.

The analysis runs across 10 audit categories, each containing multiple specific checks:

Category 1: Crawlability & Accessibility

• Robots.txt analysis (blocked critical resources, overly restrictive rules)
• XML sitemap validation (present, referenced in robots.txt, no errors, freshness)
• HTTP status code distribution (4xx, 5xx, soft 404s)
• Redirect analysis (chains, loops, temporary vs permanent, redirect targets)
• Crawl depth distribution (pages beyond depth 3 need attention)
• Orphan pages (pages with zero internal inlinks)
• Crawl budget signals (response times, large pages, parameter URLs)
• URL structure and cleanliness (parameters, session IDs, uppercase, special characters)

Category 2: Indexability & Index Management

• Indexability status distribution (indexable vs non-indexable and why)
• Canonical tag audit (missing, self-referencing, conflicting, cross-domain)
• Meta robots and X-Robots-Tag directives (noindex, nofollow patterns)
• Pagination handling (rel=next/prev, parameter-based, load-more/infinite scroll)
• Duplicate content detection (near-duplicates via hash comparison, thin content clusters)
• Parameter handling (URL parameters creating duplicate content)

Category 3: On-Page SEO Elements

• Title tag analysis (missing, duplicate, too long/short, keyword presence, brand format)
• Meta description analysis (missing, duplicate, too long/short, compelling copy signals)
• Heading hierarchy (missing H1, multiple H1s, H1 matching title, heading structure)
• Content quality signals (word count distribution, thin pages, text-to-HTML ratio)
• Internal linking patterns (link equity distribution, hub pages, isolated clusters)
• Keyword cannibalization detection (multiple pages targeting same terms based on titles/H1s)
• Image optimization (missing alt text, oversized images, modern format usage)

Category 4: Site Architecture & Internal Linking

• Site depth analysis and visualization
• Click depth from homepage to key pages
• Internal link distribution (pages with too few or too many links)
• Navigation structure assessment
• Breadcrumb implementation
• Faceted navigation and filter handling (for ecommerce)
• Content silos and topical clustering

Category 5: Performance & Core Web Vitals

• Page size distribution (HTML, total transferred bytes)
• Response time analysis (slow pages, server performance)
• CO2 and sustainability metrics (if available in crawl data)
• Core Web Vitals guidance (LCP, INP, CLS best practices by platform)
• Resource optimization recommendations (based on page weight data)

Category 6: Mobile & Rendering

• Mobile alternate links and responsive signals
• Viewport and mobile-friendliness indicators
• JavaScript rendering concerns (if SPA/framework detected)
• AMP implementation (if present)

Category 7: Structured Data & Schema

• Schema markup presence and types detected
• Missing schema opportunities by page type (Product, Article, FAQ, LocalBusiness, etc.)
• Platform-specific schema recommendations (e.g. Shopify product schema gaps)

Category 8: Security & Protocol

• HTTPS implementation (mixed content, HTTP pages remaining)
• HSTS headers
• Security headers assessment

Category 9: International SEO

• Hreflang implementation audit (if present)
• Language targeting consistency
• Regional URL structure

Category 10: AI & Future Readiness

• llms.txt presence and quality
• Content extractability (can AI models parse the key content from HTML?)
• Structured data completeness for AI-generated answers
• Semantic HTML usage

---

Phase 4: Business Impact Scoring

This is what separates a useful audit from a generic checklist dump. Read references/impact-scoring.md for the full methodology.

Every issue gets scored on three dimensions:

1. SEO Impact (1-10): How much does this issue affect search visibility?

• Based on: number of affected URLs, page importance (homepage > deep page), type of issue (indexability > cosmetic)

2. Business Impact (1-10): How much revenue or leads are at risk?

• Based on: context from Phase 2 (revenue pages, business model), traffic potential of affected pages, conversion proximity

3. Fix Effort (1-10, where 1 = easiest): How hard is this to fix?

• Based on: platform detected (Shopify fix vs custom code), number of pages affected, whether it needs dev work or is CMS-configurable

Priority Score = (SEO Impact × 0.4) + (Business Impact × 0.4) + ((10 - Fix Effort) × 0.2)

This means high-impact, easy-to-fix issues rise to the top automatically.

Platform-Aware Recommendations

The fix instructions adapt based on the detected platform:

• Shopify: Reference specific Shopify admin paths, theme liquid files, app recommendations
• WordPress: Reference specific plugins (Yoast, RankMath), theme functions, .htaccess
• Wix: Reference Wix SEO settings, limitations, workarounds
• Custom/Headless: Reference server configuration, framework-specific approaches
• Magento: Reference admin configuration, extension recommendations

---

Phase 5: Output Generation

Markdown Report Structure

Generate the report following this exact structure:

# Technical SEO Audit Report: [Domain]
**Audit Date**: [Date]
**Audited By**: AI Technical SEO Audit (powered by [crawl tool used])
**Total URLs Analyzed**: [count]
**Platform Detected**: [platform]
**Site Type**: [type]

## Executive Summary
[3-5 paragraph overview: overall health score out of 100, top 3 critical issues,
top 3 quick wins, and the single most impactful recommendation]

## Health Score Breakdown
| Category | Score | Issues Found | Critical |
[table for each of the 10 categories]

## Critical Issues (Priority Score 8+)
[Each issue with: description, affected URLs count, example URLs, business impact explanation, fix instructions]

## High Priority Issues (Priority Score 6-7.9)
[Same format]

## Medium Priority Issues (Priority Score 4-5.9)
[Same format]

## Low Priority Issues (Priority Score <4)
[Same format]

## Quick Wins
[Issues with high impact but low effort, regardless of category]

## Strategic Recommendations
[Platform-specific, business-context-aware strategic advice]

## Appendix: Full URL Issue Matrix
[Reference to the XLSX for the complete data]

XLSX Spreadsheet Structure

Generate the XLSX spreadsheet using openpyxl (via pandas to_excel). The workbook contains these sheets:

1. Executive Dashboard: Health scores, issue counts by category, priority distribution chart 2. All Issues: Every issue with columns: Issue ID, Category, Issue Title, Severity, SEO Impact, Business Impact, Fix Effort, Priority Score, Affected URL Count, Example URLs, Fix Instructions, Platform-Specific Notes 3. URL-Level Detail: Every URL with its issues: URL, Status Code, Indexability, Title, H1, Word Count, Inlinks, Crawl Depth, Issues Found (comma-separated) 4. Quick Wins: Filtered view of high-impact, low-effort items 5. Redirect Map: All redirects with chains mapped out 6. Duplicate Content: Near-duplicate page clusters 7. Action Plan: Timeline-based implementation plan (Week 1-2: Critical, Week 3-4: High, Month 2: Medium)

---

Execution Flow

When this skill triggers, follow this sequence:

Step 0: Intake Questionnaire

Before touching any data, ask the user these questions. Present them as a single numbered list and wait for answers before proceeding.

1. What data do you have? "Are you uploading an Ahrefs export (Pages CSV or All Issues directory), or would you like me to crawl the site via API (Firecrawl, DataForSEO, Ahrefs MCP)?" 2. Was JavaScript rendering enabled? "Did you enable JavaScript rendering in the Ahrefs crawl settings? (Settings > JavaScript rendering > On). This matters for sites built on React, Next.js, Vue, Angular, or Gatsby — without it, many issues will be false positives." 3. What does this site do? "What's the business model? (ecommerce, SaaS, lead gen, publisher, etc.) Which pages drive the most revenue or leads?" 4. What platform is the site on? "Do you know the CMS or framework? (Shopify, WordPress, Wix, custom, headless, etc.) I'll auto-detect from the data too, but knowing upfront helps." 5. Any known concerns? "Are there specific issues you're already aware of or areas you want me to focus on?" 6. Supplementary data? "Do you have Google Search Console or Analytics data to layer in? This helps me weight issues by actual traffic impact."

If the user answers inline with their initial message (e.g. "here's my Ahrefs export, it's a Shopify store"), skip questions they've already answered. Only ask what's still unknown.

Cowork vs Claude Code

Claude Code can run the Python analysis script (scripts/analyze_crawl.py) directly, generate XLSX files, and handle large crawl datasets (thousands of URLs). This is the full-featured experience.

Cowork cannot execute scripts or generate files. In Cowork, perform the analysis manually by reading the uploaded CSV data and applying the audit checks from references/analysis-modules.md directly. This works well for small-to-medium sites (under ~200 URLs). For larger sites, recommend the user switch to Claude Code for the automated pipeline.

Steps 1-6: Core Audit

1. Ingest data: Use Path A, B, or C from Phase 1 2. Discover context: Run auto-detection, confirm with user (Phase 2). Cross-reference against intake answers. 3. Run analysis: Execute all 10 categories from Phase 3

• Read references/analysis-modules.md for detailed check specifications
• Use scripts/analyze_crawl.py for automated data processing (Claude Code only)

4. Score and prioritize: Apply Phase 4 scoring to every issue found

• Read references/impact-scoring.md for scoring calibration

5. Generate outputs: Create both deliverables per Phase 5

• Use openpyxl (via pandas) to generate the XLSX spreadsheet (Claude Code only)
• In Cowork, output the full Markdown report directly in the conversation
• If the user requests a Word document, use python-docx to generate it (Claude Code only)

6. Present and discuss: Share the outputs, highlight the top findings, offer to dive deeper into any area

---

Important Principles

• Never produce a generic checklist. Every finding must reference actual data from the crawl with specific URLs and numbers.
• Context is everything. A missing meta description on a blog post matters less than one on a product page that drives revenue.
• Platform awareness saves time. Do not recommend .htaccess changes to a Shopify user.
• Explain the "so what". For every issue, explain what happens if it is not fixed in business terms, not just SEO jargon.
• Be honest about severity. Not everything is critical. Over-escalating destroys trust.
• Adapt to scale. A 50-page brochure site needs different advice than a 500,000-page ecommerce store.

Analysis Modules Reference

This document specifies every individual check within each of the 10 audit categories. Each check includes what to look for, severity thresholds, and how to report findings.

---

Category 1: Crawlability & Accessibility

1.1 HTTP Status Code Distribution

What to check: Group all URLs by status code range. Thresholds:

• 4xx errors on internal pages = Critical if > 0 (especially on important pages)
• 5xx errors = Critical (server problems)
• High 3xx ratio (> 15% of pages are redirects) = High priority
• Soft 404s (200 status but thin/error content, word count < 50) = High priority

Report format: Table with status code breakdown + list of affected URLs for non-2xx.

1.2 Redirect Analysis

What to check: All pages with status 3xx or redirect_url populated.

• Redirect chains: Follow redirects to find chains (A -> B -> C). Any chain > 2 hops is an issue.
• Redirect loops: Detect circular redirects (A -> B -> A).
• Temporary redirects: 302/307 redirects that should be 301s (especially if > 6 months old).
• Redirect targets: Where do redirects land? Check for redirects to 4xx, external domains, or non-canonical URLs.
• HTTP to HTTPS redirects: Should exist for all HTTP URLs.

Thresholds:

• Redirect chains (3+ hops) = Critical
• Redirect loops = Critical
• 302 on permanent moves = High
• Redirects to 4xx = Critical

1.3 Crawl Depth Analysis

What to check: Distribution of crawl_depth values across all URLs.

• Pages at depth 0 = Homepage
• Pages at depth 1-2 = Good, easily crawlable
• Pages at depth 3 = Acceptable for large sites
• Pages at depth 4+ = Potential crawl budget issue

Thresholds:

• Revenue/key pages at depth 3+ = Critical
• > 20% of indexable pages at depth 4+ = High
• Any page at depth 6+ = Medium (likely orphaned or poorly linked)

Special handling by site type:

• Ecommerce: Product pages should be depth 2-3 max (home > category > product)
• Blog: Articles should be depth 1-3 (home > blog > article)
• SaaS: Core pages (pricing, features) should be depth 1

1.4 Orphan Pages

What to check: Pages with zero or very few internal inlinks (unique_inlinks <= 1, excluding self-links). Context matters: Navigation links count. A page linked only from the sitemap but not from any navigation or content is effectively orphaned for users.

Thresholds:

• Indexable page with 0 inlinks = Critical
• Indexable page with only 1 inlink = High (fragile)

1.5 URL Structure Quality

What to check:

• URLs with parameters (contains ?) that create duplicate content
• URLs with uppercase characters
• URLs with special characters or encoded spaces
• URLs exceeding 200 characters
• URLs with session IDs or tracking parameters
• URLs with double slashes (excluding protocol)
• Non-descriptive URLs (numeric IDs only, no keywords)

Thresholds:

• Parameter URLs creating duplicates = High
• Session IDs in URLs = Critical
• Others = Low to Medium depending on scale

1.6 Response Time Analysis

What to check: Distribution of response_time values.

• Mean and median response time
• Pages with response time > 1 second
• Pages with response time > 3 seconds
• Patterns (e.g. all product pages slow, all blog pages fast)

Thresholds:

• Average response time > 1s = High
• Any page > 3s = Critical for that page
• > 20% of pages > 1s = High (server performance issue)

1.7 Robots.txt Analysis

What to check (if robots.txt content available):

• Disallow rules blocking important pages or resources
• Sitemap reference present
• Crawl-delay directive (can slow indexing)
• Wildcard rules that may be too broad
• Different rules for different user agents

Thresholds:

• Blocking CSS/JS resources = High (rendering issues)
• Blocking important page sections = Critical
• No sitemap reference = Low

1.8 XML Sitemap Validation

What to check (if sitemap data available):

• Sitemap exists and is accessible
• Sitemap is referenced in robots.txt
• All indexable pages are in the sitemap
• No non-indexable pages in the sitemap (noindex, 4xx, redirects)
• Sitemap is not stale (lastmod dates)
• Sitemap size within limits (< 50MB, < 50,000 URLs per file)

Thresholds:

• No sitemap = High
• Sitemap contains noindex/4xx URLs = Medium
• Key pages missing from sitemap = High

---

Category 2: Indexability & Index Management

2.1 Indexability Distribution

What to check: Count of Indexable vs Non-Indexable pages and reasons.

• Group non-indexable by reason: noindex, canonicalized, blocked by robots, 3xx, 4xx/5xx
• Check if any pages are unintentionally non-indexable

Report format: Pie chart data + breakdown table.

2.2 Canonical Tag Audit

What to check:

• Missing canonicals: Indexable pages without a canonical tag
• Self-referencing canonicals: Present and correct (good practice)
• Non-self-referencing canonicals: Page points canonical to a different URL (intentional? or error?)
• Canonical to non-indexable: Canonical target is noindex, 4xx, or redirects (broken chain)
• Canonical mismatch: Canonical URL differs from the actual URL only by trailing slash, www, or protocol
• Cross-domain canonicals: Pointing to external domains (rare, verify intentional)
• Canonical chains: Page A canonicals to B, B canonicals to C

Thresholds:

• Missing canonical on indexable page = Medium
• Canonical to 4xx/noindex = Critical
• Canonical chains = High

2.3 Meta Robots & Directives

What to check:

• Pages with noindex that should be indexed (assess by page type and importance)
• Pages with nofollow that block link equity flow
• Pages with both noindex and a canonical to another page (conflicting signals)
• X-Robots-Tag overriding meta robots

Thresholds:

• Revenue pages with noindex = Critical
• Noindex + canonical conflict = High

2.4 Pagination

What to check:

• Pages with rel=next/prev attributes
• Paginated series that are not self-canonicalized (each page should canonical to itself)
• Paginated pages that are noindex (Google still recommends indexing paginated pages)
• Missing pagination for large content sets
• "View All" page availability

Thresholds:

• Paginated pages canonicalized to page 1 = High (loses deep content)
• Noindex on paginated pages = Medium

2.5 Duplicate Content Detection

What to check:

• Exact duplicates: Pages with identical content hash
• Near duplicates: Pages with near_duplicate_count > 0 or semantic_similarity_score > 0.85
• Thin content clusters: Groups of pages with very similar content (common in ecommerce with color/size variants)
• Parameter duplicates: Same page accessible via different URL parameters

Thresholds:

• Exact duplicates without canonical handling = Critical
• Near duplicates on competing keywords = High
• Thin content clusters = Medium (consolidation opportunity)

---

Category 3: On-Page SEO Elements

3.1 Title Tag Analysis

What to check:

• Missing titles: No title tag
• Empty titles: Title tag exists but empty
• Duplicate titles: Multiple pages with identical titles
• Title too short: < 30 characters (under-optimized)
• Title too long: > 60 characters (may be truncated in SERPs)
• Title pixel width: > 580 pixels (will be truncated in SERPs)
• Title matches H1: Good signal (or flag if they are identical for every page — might be templated)
• Brand in title: Consistent brand format (e.g. "Page Title | Brand")

Thresholds:

• Missing title on indexable page = Critical
• Duplicate titles = High
• Too long/short = Medium

3.2 Meta Description Analysis

What to check:

• Missing meta descriptions: Indexable pages without one
• Empty meta descriptions: Tag exists but empty
• Duplicate meta descriptions: Multiple pages with identical descriptions
• Too short: < 70 characters
• Too long: > 160 characters (may be truncated)
• Pixel width: > 920 pixels

Thresholds:

• Missing on key/revenue pages = High
• Missing on other pages = Medium
• Duplicate descriptions = Medium

3.3 Heading Analysis

What to check:

• Missing H1: Indexable page has no H1
• Multiple H1s: Page has more than one H1
• Empty H1: H1 tag exists but is empty
• H1 length: Very short (< 10 chars) or very long (> 70 chars)
• H1 matches title: Good practice check
• Heading hierarchy: H2s present, logical structure

Thresholds:

• Missing H1 on indexable page = High
• Multiple H1s = Medium (less of an issue in HTML5 but still a signal)

3.4 Content Quality Signals

What to check:

• Thin content: Pages with word_count < 300 (threshold varies by page type)
• Product pages: < 100 words is thin
• Blog posts: < 300 words is thin
• Category pages: < 50 words is thin (expected to be lighter)
• Word count distribution: Histogram showing content depth across site
• Readability scores: Average and outliers
• Text-to-HTML ratio: Pages below 10% may be too template-heavy

Thresholds:

• Blog post < 300 words = High
• Product page < 100 words = Medium
• Very low text ratio (< 5%) = Medium

3.5 Keyword Cannibalization Detection

What to check: Multiple indexable pages with very similar titles or H1s that could compete for the same search queries.

• Group pages by similar title patterns (fuzzy matching)
• Group pages by identical or near-identical H1s
• Flag pages in the same subfolder targeting the same apparent topic

This is a heuristic analysis from crawl data alone. Note: for definitive cannibalization analysis, GSC data (which pages rank for the same queries) is needed.

Thresholds:

• 2+ indexable pages with near-identical titles = High
• Multiple collection/category pages with overlapping terms = Medium

3.6 Image Optimization (if data available)

What to check:

• Images missing alt text
• Oversized images (> 200KB)
• Non-modern formats (no WebP/AVIF alternatives)
• Broken image links

---

Category 4: Site Architecture & Internal Linking

4.1 Internal Link Distribution

What to check:

• Pages with highest inlinks (navigation hubs)
• Pages with lowest inlinks (under-linked, may be orphaned)
• Average inlinks per page
• Inlink distribution by page type (products vs blog vs categories)

4.2 Link Equity Flow

What to check:

• Link score distribution (if available)
• Pages with high link score but low inlinks (efficiently linked)
• Pages with low link score despite many inlinks (diluted)
• Nofollow on internal links (leaking equity)

4.3 Content Silo Analysis

What to check:

• Group pages by URL folder structure
• Identify topical clusters
• Check cross-linking between silos
• Identify pages that break the silo structure

4.4 Navigation & Breadcrumbs

What to check:

• Consistent navigation link pattern (inferred from inlink counts)
• Breadcrumb presence (from structured data if available)
• Faceted navigation creating parameter URLs (ecommerce)

---

Category 5: Performance & Core Web Vitals

5.1 Page Weight Analysis

What to check:

• Page size distribution (page_size_bytes)
• Total transferred bytes distribution
• Pages exceeding 3MB total transferred
• Heaviest pages and their types

Thresholds:

• Page > 3MB = High
• Page > 5MB = Critical
• Average page > 2MB = High (site-wide issue)

5.2 Response Time Performance

What to check:

• Already covered in 1.6, but here focus on patterns:
• Slow page types (which templates are slowest?)
• Server vs content issues (large pages vs slow server)

5.3 Core Web Vitals Guidance

Note: Crawl data alone cannot measure CWV (needs real user data or lab data). However, provide:

• LCP risks: Large pages, slow response times, heavy above-the-fold content
• INP risks: Heavy JavaScript (detected from SPA/framework signatures)
• CLS risks: Missing image dimensions (if detectable), dynamic content injection
• Platform-specific CWV advice (e.g. Shopify theme optimization, WordPress plugin bloat)

5.4 Sustainability Metrics

What to check (if CO2 data available):

• CO2 per page distribution
• Carbon rating distribution
• Worst offending pages
• Estimated total site carbon footprint

---

Category 6: Mobile & Rendering

6.1 Mobile Signals

What to check:

• Mobile alternate links present
• Responsive design indicators
• AMP implementation (if any)

6.2 JavaScript Rendering Concerns

What to check:

• Platform is SPA/headless (Next.js, React, Angular, Vue)
• Content dependent on JavaScript rendering
• If Firecrawl was used with JS rendering, compare rendered vs raw content

---

Category 7: Structured Data & Schema

7.1 Schema Presence

What to check (if structured data columns available or parsed from HTML):

• Which pages have structured data
• Which schema types are present (Product, Article, FAQ, Organization, BreadcrumbList, etc.)
• Schema validation issues

7.2 Missing Schema Opportunities

Platform and page-type specific recommendations:

• Ecommerce product pages: Product, Offer, AggregateRating, Review, BreadcrumbList
• Blog/article pages: Article, Author, BreadcrumbList, FAQ, HowTo
• Homepage: Organization, WebSite with SearchAction
• Local business: LocalBusiness, OpeningHours
• FAQ pages: FAQ schema
• Category pages: CollectionPage, ItemList

7.3 Deprecated Schema Warnings

Flag use of schema types Google no longer supports for rich results (as of 2025-2026):

• HowTo (reduced rich result support)
• FAQ (limited to authoritative sources)

---

Category 8: Security & Protocol

8.1 HTTPS Implementation

What to check:

• Any HTTP (non-HTTPS) pages in the crawl
• Mixed content (HTTPS pages linking to HTTP resources)
• HTTP to HTTPS redirect implementation

8.2 Security Headers

What to check (if header data available):

• HSTS present
• Content-Security-Policy
• X-Frame-Options
• X-Content-Type-Options

---

Category 9: International SEO

9.1 Hreflang Audit

What to check (if hreflang data present):

• Hreflang return links (bidirectional linking)
• Hreflang pointing to non-indexable pages
• Missing x-default
• Language/region code validity
• Hreflang conflicts with canonical

9.2 Language Consistency

What to check:

• HTML lang attribute matches content language
• Consistent language across page elements

---

Category 10: AI & Future Readiness

10.1 llms.txt

What to check:

• Presence of llms.txt in root directory
• Quality and completeness of llms.txt content

10.2 Content Extractability

What to check:

• Is key content in semantic HTML (article, main, section) vs generic divs?
• Is content available in initial HTML or only after JS execution?
• Are heading hierarchies logical and descriptive?

10.3 Structured Data Completeness

What to check:

• Are all entity types properly marked up?
• Is there enough structured data for AI to build knowledge graph entries?

---

Running the Analysis

For each category, the analysis script (scripts/analyze_crawl.py) processes the normalized data and returns a structured findings object:

{
    "category": "Crawlability & Accessibility",
    "category_id": 1,
    "health_score": 72,  # 0-100
    "checks": [
        {
            "check_id": "1.1",
            "check_name": "HTTP Status Code Distribution",
            "status": "warning",  # pass / warning / critical / info
            "summary": "3 pages returning 4xx errors",
            "affected_urls_count": 3,
            "affected_urls": ["https://example.com/old-page", ...],
            "details": { ... },  # Check-specific data
            "seo_impact": 6,
            "business_impact": 4,
            "fix_effort": 2,
            "priority_score": 5.6,
            "fix_instructions": "Set up 301 redirects for these broken URLs to their closest equivalent pages.",
            "platform_notes": "In Shopify, go to Settings > Navigation > URL Redirects to add these."
        }
    ]
}

API-Based Crawling Reference

This reference covers how to use external crawl APIs to gather site data when the user does not have a pre-existing crawl file.

---

Firecrawl Integration

Firecrawl is the recommended API for live crawling. It handles JavaScript rendering, respects robots.txt, and returns clean data.

Prerequisites

• Firecrawl API key (user must provide or have in environment as FIRECRAWL_API_KEY)
• Python with firecrawl-py package (pip install firecrawl-py --break-system-packages)

Basic Crawl Implementation

from firecrawl import FirecrawlApp
import json
import time

def crawl_site(api_key, target_url, max_pages=100, max_depth=3):
    """
    Crawl a website using Firecrawl API.

    Args:
        api_key: Firecrawl API key
        target_url: Starting URL to crawl
        max_pages: Maximum number of pages to crawl (each page = 1 credit)
        max_depth: Maximum crawl depth from starting URL

    Returns:
        list of page data dicts normalized to internal schema
    """
    app = FirecrawlApp(api_key=api_key)

    # Start the crawl
    crawl_result = app.crawl_url(
        target_url,
        params={
            'limit': max_pages,
            'maxDepth': max_depth,
            'scrapeOptions': {
                'formats': ['markdown', 'html'],
                'includeTags': ['title', 'meta', 'h1', 'h2', 'h3', 'link', 'a'],
            }
        },
        poll_interval=5  # Check every 5 seconds
    )

    return crawl_result

def normalize_firecrawl_data(crawl_result):
    """
    Convert Firecrawl response to internal schema format.
    """
    from bs4 import BeautifulSoup

    pages = []
    for page in crawl_result.get('data', []):
        metadata = page.get('metadata', {})
        html = page.get('html', '')
        markdown = page.get('markdown', '')

        # Parse HTML for additional fields
        soup = BeautifulSoup(html, 'html.parser') if html else None

        row = {
            'url': metadata.get('sourceURL', ''),
            'status_code': metadata.get('statusCode', 200),
            'title': metadata.get('title', ''),
            'meta_description': metadata.get('description', ''),
            'language': metadata.get('language', ''),
            'content_type': 'text/html',
        }

        if soup:
            # Extract H1
            h1_tag = soup.find('h1')
            row['h1'] = h1_tag.get_text(strip=True) if h1_tag else ''

            # Extract H2
            h2_tag = soup.find('h2')
            row['h2'] = h2_tag.get_text(strip=True) if h2_tag else ''

            # Extract canonical
            canonical_tag = soup.find('link', rel='canonical')
            row['canonical'] = canonical_tag.get('href', '') if canonical_tag else ''

            # Extract meta robots
            robots_tag = soup.find('meta', attrs={'name': 'robots'})
            row['meta_robots'] = robots_tag.get('content', '') if robots_tag else ''

            # Page size
            row['page_size_bytes'] = len(html.encode('utf-8'))

            # Word count from markdown (more accurate than HTML)
            row['word_count'] = len(markdown.split()) if markdown else 0

            # Text ratio
            text_content = soup.get_text()
            text_bytes = len(text_content.encode('utf-8'))
            html_bytes = len(html.encode('utf-8'))
            row['text_ratio'] = round((text_bytes / html_bytes) * 100, 3) if html_bytes > 0 else 0

            # Count internal links
            links = page.get('links', [])
            domain = row['url'].split('/')[2] if '/' in row['url'] else ''
            row['inlinks'] = 0  # Will be calculated in post-processing
            row['outlinks'] = len(links)
            row['external_outlinks'] = sum(1 for l in links if domain not in l)

            # Extract hreflang
            hreflang_tags = soup.find_all('link', rel='alternate', hreflang=True)
            row['hreflang_count'] = len(hreflang_tags)

            # Extract structured data
            json_ld_scripts = soup.find_all('script', type='application/ld+json')
            row['structured_data_count'] = len(json_ld_scripts)
            if json_ld_scripts:
                try:
                    schemas = []
                    for script in json_ld_scripts:
                        data = json.loads(script.string)
                        if isinstance(data, dict):
                            schemas.append(data.get('@type', 'Unknown'))
                        elif isinstance(data, list):
                            for item in data:
                                schemas.append(item.get('@type', 'Unknown'))
                    row['structured_data_types'] = ', '.join(schemas)
                except (json.JSONDecodeError, AttributeError):
                    row['structured_data_types'] = ''

        pages.append(row)

    # Post-process: calculate inlinks
    url_set = {p['url'] for p in pages}
    inlink_counts = {url: 0 for url in url_set}
    for page in crawl_result.get('data', []):
        for link in page.get('links', []):
            if link in inlink_counts:
                inlink_counts[link] += 1

    for page_row in pages:
        page_row['inlinks'] = inlink_counts.get(page_row['url'], 0)

    return pages

Crawl Configuration Guide

Always confirm with the user before starting large crawls due to credit consumption.

Rate Limiting and Error Handling

Firecrawl handles rate limiting internally, but if you encounter issues:

• 429 errors: Wait and retry with exponential backoff
• Timeout: Increase poll_interval for large sites
• Partial results: Firecrawl returns partial data if crawl is interrupted; use what is available

---

DataForSEO On-Page API

If the user has DataForSEO tools available in the environment, use the instant_pages tool:

# Using the DataForSEO MCP tool
result = instant_pages(url="https://example.com")

This provides on-page metrics for a single URL. For full site audits, iterate over a URL list (obtained from sitemap or another source).

---

Post-Crawl Enrichment

After the initial crawl (regardless of method), optionally enrich the data:

1. Robots.txt fetch: GET {domain}/robots.txt to check directives 2. Sitemap fetch: GET {domain}/sitemap.xml (and any referenced sitemaps) to cross-reference crawled vs listed URLs 3. llms.txt check: GET {domain}/llms.txt for AI readiness assessment 4. Search Console data: If the user provides GSC access, merge impression/click data for traffic-weighted prioritization

Data Ingestion Reference

Column Mapping and Normalization

All crawl data is normalized into a standard internal schema. This reference defines how to map columns from Ahrefs exports and API sources.

Internal Schema

These are the normalized column names used throughout the analysis:

---

Ahrefs Column Mapping

Ahrefs Site Audit (pages.csv)

Detection signature: Headers contain "URL" AND "HTTP Code"

URL -> url
HTTP Code -> status_code
Title -> title
Description -> meta_description
H1 -> h1
Canonical URL -> canonical
Word Count -> word_count
Internal Links In -> inlinks
Depth -> crawl_depth

Ahrefs All Issues Export (directory of CSVs)

Detection signature: A directory containing an index.txt file and multiple CSVs with filenames matching {Error|Warning|Notice}-*.csv.

Encoding: UTF-16 with BOM, tab-separated. Must be decoded before parsing:

# Python
df = pd.read_csv(filepath, sep='\t', encoding='utf-16')

# CLI
iconv -f UTF-16 -t UTF-8 file.csv

Filename parsing:

The filename encodes severity, indexability scope, and issue type:

Error-404_page.csv                              -> severity: error, issue: 404 page
Warning-indexable-Low_word_count.csv             -> severity: warning, scope: indexable, issue: low word count
Notice-indexable-Title_tag_changed.csv           -> severity: notice, scope: indexable, issue: title tag changed
Error-indexable-Orphan_page_(has_no_incoming_internal_links).csv -> severity: error, scope: indexable, issue: orphan page

Files ending in -links.csv contain the source pages that link to affected URLs (e.g., which pages link to a 404). These are not issues themselves but provide context for fix instructions. Parse them separately.

Common columns across most issue CSVs:

PR -> priority_rank (Ahrefs internal priority)
URL -> url
Title -> title
Content type -> content_type
Is rendered page -> is_rendered
HTTP status code -> status_code
Organic traffic -> organic_traffic
Depth -> crawl_depth
Is indexable page -> indexability (convert true/false to Indexable/Non-Indexable)
No. of all inlinks -> inlinks
First found at -> first_found_at

Issue-specific columns (vary by issue type):

Link context files (`-links` suffix) column mapping:

Link type -> link_type
Is nofollow -> is_nofollow
Source URL -> source_url
Source HTTP status code -> source_status_code
Target URL -> target_url (the affected URL)
Target HTTP status code -> target_status_code
Anchor -> anchor_text
Is source canonical -> is_source_canonical
Is source noindex -> is_source_noindex
Is link internal -> is_internal

Building the unified issue list:

1. Parse each non--links CSV in the directory 2. Extract severity from filename prefix (Error = critical/high, Warning = medium/high, Notice = low/medium) 3. Extract issue name from filename (replace underscores with spaces, strip severity prefix and -indexable- scope) 4. For each row, create an issue record: {url, severity, issue_type, scope, status_code, organic_traffic, ...issue-specific fields} 5. Deduplicate URLs that appear across multiple issue files (a single URL can have multiple issues) 6. Optionally enrich with -links data to show which pages link to broken/redirected URLs

Mapping Ahrefs severities to audit priority:

The indexable- scope prefix in filenames indicates the issue specifically affects indexable pages, which typically warrants a higher business impact score.

---

API-Based Crawl Data Normalization

Firecrawl Response Mapping

Firecrawl returns page data in this structure per page:

markdown: string (full page content as markdown)
html: string (raw HTML)
metadata:
  title: string
  description: string
  language: string
  sourceURL: string
  statusCode: number
links: array of URLs found on page

Map to internal schema:

metadata.sourceURL -> url
metadata.statusCode -> status_code
metadata.title -> title
metadata.description -> meta_description
metadata.language -> language

For fields not provided by Firecrawl (H1, canonical, word count, etc.), parse the HTML content:

• Extract H1 from first <h1> tag in HTML
• Extract canonical from <link rel="canonical"> in HTML
• Calculate word count from the markdown content
• Extract meta robots from <meta name="robots"> in HTML
• Calculate page_size_bytes from len(html.encode('utf-8'))

DataForSEO On-Page API Mapping

If the user has DataForSEO tools available, use the instant_pages tool:

meta.title -> title
meta.description -> meta_description
meta.htags.h1[0] -> h1
page_timing.duration -> response_time
onpage_score -> (store as additional metric)

---

Platform Detection Signatures

After loading data, scan URLs and metadata to detect the platform:

---

Data Validation

After ingestion and normalization, run these validation checks:

1. URL count sanity: Report total URLs loaded. If < 10, warn the user the crawl may be incomplete. 2. Status code distribution: Summarize counts by status code range (2xx, 3xx, 4xx, 5xx). 3. Missing critical fields: Report what percentage of rows have empty title, meta description, H1, canonical. 4. Data freshness: If crawl timestamp is available, report when the crawl was performed. 5. Encoding check: Ensure no garbled characters from encoding mismatches.

Present a quick summary table to the user before proceeding to analysis.

For pages.csv format:

Data Source: Ahrefs Site Audit (pages.csv)
Total URLs: 197
Status 2xx: 185 (93.9%)
Status 3xx: 8 (4.1%)
Status 4xx: 3 (1.5%)
Status 5xx: 1 (0.5%)
Platform Detected: Shopify
Crawl Date: 2026-03-04

For All Issues format:

Data Source: Ahrefs All Issues Export (91 CSVs)
Issue Breakdown: 17 Errors, 28 Warnings, 46 Notices
Unique URLs Affected: 2,847
Top Errors: 404 pages (12), Orphan pages (8), Duplicate without canonical (287)
Top Warnings: 3XX redirects (1,077), Slow pages (15), Missing meta descriptions (13)
Platform Detected: Next.js

Business Impact Scoring Reference

This document defines how every issue found during the audit is scored and prioritized based on actual business impact rather than abstract technical severity.

---

The Three Scoring Dimensions

1. SEO Impact (1-10)

How much does this issue affect search engine visibility?

2. Business Impact (1-10)

How much revenue, leads, or business value is at risk?

This dimension requires the context gathered in Phase 2 (site type, revenue pages, business model).

Page importance hierarchy (default, adjust based on user context):

For ecommerce: 1. Homepage, checkout, cart 2. Product pages 3. Collection/category pages 4. Landing pages 5. Blog posts 6. Information pages (about, contact, policies)

For SaaS: 1. Homepage, pricing, sign-up flow 2. Feature/solution pages 3. Comparison/alternative pages 4. Blog posts (especially bottom-funnel) 5. Documentation 6. Information pages

For publishers/blogs: 1. Homepage 2. Top-traffic articles 3. Category/topic pages 4. Recent articles 5. Older content 6. Utility pages

3. Fix Effort (1-10, where 1 = easiest)

How much work is needed to resolve this issue?

Platform-adjusted effort scoring:

The same fix can have very different effort levels depending on the platform:

---

Priority Score Calculation

Priority Score = (SEO Impact × 0.4) + (Business Impact × 0.4) + ((10 - Fix Effort) × 0.2)

The formula weights SEO and business impact equally (40% each), with a 20% bonus for ease of fixing. This naturally surfaces "quick wins" (high impact, low effort) while still prioritizing the most impactful issues overall.

Priority Bands

---

Quick Wins Identification

A "Quick Win" is any issue where:

• Priority Score >= 5.0 AND Fix Effort <= 3

These are the items that should be highlighted prominently because they offer the best return on time invested. Present them as a separate section in both the report and spreadsheet.

---

Overall Health Score Calculation

The site-wide health score (0-100) is calculated per category and overall:

Per-Category Score

Category Score = 100 - (sum of issue penalties in that category)

Where each issue penalty is:

• Critical issue: -15 points (minimum 5, maximum 15, scaled by affected URL percentage)
• High issue: -8 points
• Medium issue: -4 points
• Low issue: -1 point

Category score is clamped between 0 and 100.

Overall Score

Overall Score = Weighted average of category scores

Category weights (reflecting relative importance to overall SEO health):

These weights shift based on context:

• For an international site, bump International SEO to 10% and reduce others proportionally
• For a headless/SPA site, bump Mobile & Rendering to 10%
• For ecommerce, bump Structured Data to 8% (product schema is critical)

---

Score Interpretation Guide

Include this in the report so stakeholders understand what the numbers mean:

---

Contextual Adjustments

The scoring system adjusts based on Phase 2 context:

Scale Adjustment

For very large sites (> 10,000 pages), issues affecting < 1% of pages get their SEO Impact reduced by 2 points (minimum 1). The logic: 50 broken links on a 100-page site is 50% of the site (critical), but 50 broken links on a 50,000-page site is 0.1% (much less critical).

Revenue Page Boost

Any issue affecting pages the user identified as revenue-critical gets a +2 boost to Business Impact (capped at 10).

Seasonal Adjustment

If the user mentions upcoming peak periods (Black Friday, product launch, etc.), time-sensitive fixes get a +1 boost to Business Impact.