Diagnose Seo

# Diagnose SEO

Structured diagnostic framework for crawl issues, canonicalization errors,
indexation problems, and rendering failures.

## Diagnostic Approach

Technical SEO problems fall into four categories. Diagnose in this order — each
layer depends on the previous one working correctly:

1. **Crawlability** — Can search engines find and access the pages?
2. **Indexability** — Are the pages allowed to be indexed?
3. **Renderability** — Can search engines see the full content?
4. **Signals** — Are the right signals (titles, structured data, links) in place?

## Layer 1: Crawlability

Check these in order:

### robots.txt
- Fetch `[domain]/robots.txt` and review the rules
- Look for overly broad `Disallow` rules blocking important paths
- Verify `Sitemap:` directive points to the correct sitemap URL
- Check for different rules per user-agent (Googlebot vs others)

**Common mistakes:**
- `Disallow: /` blocking the entire site (often left from staging)
- Blocking CSS/JS files that Googlebot needs for rendering
- Blocking API or AJAX endpoints that load dynamic content
- Staging robots.txt accidentally deployed to production

### XML Sitemap
- Fetch the sitemap URL(s) and check:
  - Does it return 200? Is it valid XML?
  - Does it list all important pages?
  - Does it exclude pages that shouldn't be indexed (404s, redirects, noindex pages)?
  - Are `<lastmod>` dates accurate and recent?
  - For large sites: is there a sitemap index?

### Site Architecture
- Pages should be reachable within 3 clicks from the homepage
- Check for orphan pages (no internal links pointing to them)
- Check for redirect chains (page A → B → C — should be A → C)
- Check for redirect loops

### Server Response
- Do all important pages return HTTP 200?
- Check for unexpected 301/302 redirects
- Check for soft 404s (page returns 200 but shows "not found" content)
- Verify HTTPS is enforced (HTTP should 301 to HTTPS)

## Layer 2: Indexability

### Meta Robots / X-Robots-Tag
- Check for `<meta name="robots" content="noindex">` on pages that should be indexed
- Check HTTP headers for `X-Robots-Tag: noindex`
- Common cause: CMS accidentally applying noindex to pagination, tag pages, or new pages

### Canonical Tags
- Every page should have a `<link rel="canonical">` pointing to itself (self-referencing canonical)
- Check for canonical tags pointing to wrong pages (common in paginated content, filtered URLs)
- Check for conflicting signals: canonical says page A, but noindex is set, or the page redirects

**Canonical diagnosis checklist:**
- [ ] Does the canonical URL match the actual URL?
- [ ] Is the canonical URL accessible (returns 200)?
- [ ] Does the canonical URL have the same content?
- [ ] Is there only one canonical tag on the page?

### Duplicate Content
- Check for the same content accessible at multiple URLs:
  - With and without trailing slash (`/page` vs `/page/`)
  - With and without `www` (`example.com` vs `www.example.com`)
  - HTTP vs HTTPS
  - URL parameters creating duplicate pages (`?sort=price`, `?page=1`)
- Each duplicate set needs one canonical URL; all others should redirect or use canonical tags

## Layer 3: Renderability

### JavaScript Rendering
- Does the page content appear in the raw HTML source? Or is it loaded via JavaScript?
- If JS-rendered: does Googlebot see the full content? (Use URL Inspection tool in Search Console)
- Check for content hidden behind click events, tabs, or accordions
- Check for lazy-loaded content that only appears on scroll

### Core Content Visibility
- Is the main content in the initial HTML? Or loaded async