Sdk Docs Auditor

SDK Docs Auditor

Produces a comprehensive, cross-referenced audit of any SDK documentation site with a fully styled downloadable HTML report.

What this skill does

1. Discovers all SDK pages via llms.txt first, falling back to sitemap.xml (using curl), then homepage nav crawl 2. Fetches and reads every relevant SDK page 3. Audits six fixed sections: Installation, Quick Start, Error Handling, Troubleshooting, Examples, Best Practices 4. Cross-references every gap across ALL other SDK pages — never flag something as missing if it exists elsewhere 5. Scores each section 0–100 and assigns a rating tier 6. Generates a beautiful, self-contained, downloadable HTML report

---

Step 1 — Discover all SDK pages

Use a three-tier discovery strategy, trying each method in order until one succeeds.

1a. Try llms.txt first (preferred)

Run a bash curl command to fetch llms.txt:

curl -s <docs_url>/llms.txt

If found:

• Extract every URL from lines matching the pattern - [Page Title](URL): description
• Filter to SDK-relevant pages only — keep URLs whose path contains any of:

sdk, installation, quickstart, quick-start, error, troubleshoot, example, best-practice, getting-started, reference, api-reference, overview, client, service, memory, search, worker, job, policy, agent, team

• Exclude: marketing pages, changelog, blog, legal, community/forum pages
• Store as SDK_PAGES[] — list of {title, url, description}
• Note in the report: "Discovery method: llms.txt"

1b. Fallback — Try sitemap.xml

If llms.txt is unavailable or returns no useful URLs, run a bash curl command to fetch the sitemap:

curl -s <docs_url>/sitemap.xml | python3 -c "import sys, re; print('\n'.join(re.findall(r'<loc>(.*?)</loc>', sys.stdin.read())))"

If the sitemap returns URLs:

• Parse every <loc> entry to get the full URL list
• Filter using the same keyword list above
• Store as SDK_PAGES[] — list of {title, url}
• Note in the report: "Discovery method: sitemap.xml — N total URLs found, M SDK-relevant kept"

Also check for a sitemap index (multiple sitemaps) by looking for <sitemapindex> in the response. If found, curl each child sitemap and aggregate all URLs before filtering.

1c. Final fallback — Homepage nav crawl

If both llms.txt and sitemap.xml fail, fetch the docs homepage (<docs_url>) and extract all links from the nav sidebar or sitemap structure, filtering using the same keyword list.

• Note in the report: "Discovery method: homepage nav crawl (llms.txt and sitemap.xml unavailable)"

1d. Identify section mapping

From the discovered pages, identify which pages map to the six audit targets:

If a dedicated page is not found for a section, note it — the absence itself is a finding.

Build the corpus

All discovered pages form the page corpus used for both section auditing and cross-referencing.

Report discovery summary:

Found N pages via [sitemap.xml / llms.txt / nav crawl]. Kept M SDK-relevant pages. Fetching now...

---

Step 2 — Fetch every page in the corpus

Fetch each page using bash_tool with curl, stripping HTML tags via Python to extract plain text. Do NOT use web_fetch for corpus pages — curl is more reliable and avoids permission errors.

Use this pattern for each page URL:

curl -s "https://docs.example.com/sdk/some-page" -L | python3 -c "
import sys, re, html
content = sys.stdin.read()
content = re.sub(r'<script[^>]*>.*?</script>', '', content, flags=re.DOTALL)
content = re.sub(r'<style[^>]*>.*?</style>', '', content, flags=re.DOTALL)
text = re.sub(r'<[^>]+>', ' ', content)
text = html.unescape(text)
text = re.sub(r'\s+', ' ', text).strip()
print(text[:6000])
"

Batch multiple pages in a single bash call using a loop to minimise round-trips:

for page in installation quick-start error-handling troubleshooting examples best-practices overview api-reference; do echo "=== PAGE: $page ===" curl -s "https://docs.example.com/sdk/$page" -L | python3 -c " import sys, re, html content = sys.stdin.read() content = re.sub(r'<script[^>]>.?</script>', '', content, flags=re.DOTALL) content = re.sub(r'<style[^>]>.?</style>', '', content, flags=re.DOTALL) text = re.sub(r'<[^>]+>', ' ', content) text = html.unescape(text) text = re.sub(r'\s+', ' ', text).strip() print(text[:6000]) " echo "" done

For large sites (>30 pages), prioritise in this order:
1. The six target section pages (installation, quick-start, error-handling, troubleshooting, examples, best-practices)
2. Overview / client overview pages
3. API reference
4. Service-specific pages (agents, jobs, workers, policies, memory, search, etc.)

---

## Step 3 — Audit the six target sections

For each of the six sections below, read its page carefully and evaluate against the criteria. Then check every other fetched page to see if any gap is actually addressed elsewhere.

### 3a. Installation

**Must have (deduct heavily if missing):**
- Prerequisites with exact versions (Python/Node/language version, package manager)
- At least one complete install command (pip/npm/etc.)
- Authentication setup with exact steps to obtain and configure credentials
- A working verification snippet that proves the install succeeded
- At least one expected output or confirmation of success

**Should have (deduct moderately):**
- Multiple install methods (package manager, source, Docker)
- IDE/editor setup
- Environment variable configuration with `.env` examples
- Version compatibility table
- Optional dependency explanations (what each extra includes)
- Constructor parameter reference (all params, types, defaults)

**Common gaps to check across other pages:**
- Env-var auto-detection (does the client read env vars when called with no args?) — check api-reference, overview
- Advanced constructor params (timeout, retries, org) — check api-reference
- Version pinning guidance — check any getting-started pages

### 3b. Quick Start

**Must have:**
- A minimal, numbered step-by-step path a first-time user can follow in under 5 minutes
- Complete working code from zero to first successful call
- Expected output for every code snippet
- Explanation of any prerequisite service/resource (queues, workers, etc.)

**Should have:**
- Both env-var and inline credential patterns
- Multiple install options (pip + poetry, etc.)
- Links to deeper docs for each concept introduced

**Common gaps to check:**
- Worker/queue prerequisites — check workers/environments pages
- Valid model/runtime IDs — check models/runtimes service pages
- Execute call signature accuracy — check api-reference and agents service pages

### 3c. Error Handling

**Must have:**
- Complete exception hierarchy (all exception classes, inheritance tree)
- Every exception class documented with: description, import path, code example
- At minimum: authentication, connection, timeout, rate-limit, API/HTTP errors
- HTTP status code branching (400/401/403/404/500 at minimum)
- At least one resilience pattern (retry with backoff)

**Should have:**
- Resource-specific exceptions (one per major service)
- Streaming error handling
- Context manager cleanup pattern
- Testing/pytest examples for error scenarios
- Do/don't anti-pattern examples

**Cross-reference check — critical:**
- List every exception class that appears in the api-reference or service pages
- Flag any that are missing from this page
- Check execute/method call signatures match across all pages — flag any discrepancies

### 3d. Troubleshooting

**Must have:**
- Authentication failures with diagnostic steps
- Connection / timeout issues with solutions
- At least one service-specific troubleshooting section

**Should have:**
- Troubleshooting for every major service (workers, jobs, policies, agents, etc.)
- Self-hosted / custom base_url issues
- Import errors with correct import paths
- A summary error-message lookup table
- Rate limiting with backoff references
- Actionable "getting help" checklist with actual links/commands

**Cross-reference check:**
- For every service page that has its own error handling section, check if its scenarios appear here
- Check if dataset/resource creation methods are linked when "not found" errors are described
- Check if health check API is referenced in the "verify status" step

### 3e. Examples

**Must have:**
- At least one complete, runnable end-to-end example per major service
- Expected output shown for every example
- Error handling within workflow examples

**Should have:**
- Coverage across: agent lifecycle, jobs/scheduling, memory/recall, semantic search, policy management, worker management, streaming
- No "workflow" examples that only survey resources without executing anything
- Multi-turn / session management examples where the SDK supports sessions

**Cross-reference check — this is where most gaps accumulate:**
- For every service that has its own dedicated page with examples, check if those examples are represented here
- Flag each missing service category with the source page that has the examples

### 3f. Best Practices

**Must have:**
- Authentication / credential security section
- Error handling / retry logic
- Performance (batching, pagination, caching)

**Should have:**
- One best-practices section per major service (workers, jobs, policies, agents)
- Resource cleanup / context managers
- Code organisation (type hints, reusable functions)
- Testing patterns
- Concurrency / thread-safety guidance
- Secret rotation / credential lifecycle

**Cross-reference check:**
- For every service page that defines its own best practices, check if those patterns appear here
- Note if concurrency/thread-safety is addressed anywhere in the entire corpus

---

## Step 4 — Score each section

Score 0–100 using this rubric:

| Score | Meaning |
|-------|---------|
| 85–100 | Strong — all must-haves present, most should-haves, few gaps |
| 70–84 | Good — all must-haves, some should-haves missing |
| 50–69 | Adequate — most must-haves but notable gaps |
| 30–49 | Weak — multiple must-haves missing |
| 0–29 | Poor — section is superficial or largely absent |

Assign a rating label: **Strong / Good / Adequate / Weak / Poor**

---

## Step 5 — Build the gap list for each section

For every gap:
- Write a clear one-sentence description of what is missing
- Tag it with where the content EXISTS in the corpus (if it does): `[covered in: page-name]`
- Or tag it: `[not covered anywhere in corpus]`

This distinction is critical — a gap that exists nowhere needs a new page; a gap that exists elsewhere just needs a cross-reference or consolidation.

---

## Step 6 — Derive top priority recommendations

Rank the top 6–8 actions by impact × effort. Prioritise:
1. Factual errors / signature mismatches (highest priority — causes user failures)
2. Must-have gaps not covered anywhere
3. Must-have gaps covered elsewhere but not linked
4. High-value should-have gaps
5. Cross-referencing and consolidation opportunities

---

## Step 7 — Generate the HTML report

Read the template at `assets/report-template.html` and inject all audit data into it.

Fill these placeholders:
- `__AUDIT_URL__` → the audited URL
- `__PAGES_FETCHED__` → total pages fetched
- `__AUDIT_DATE__` → today's date
- `__SECTION_DATA_JSON__` → JSON blob (see schema below)
- `__PRIORITIES_JSON__` → JSON blob (see schema below)
- `__OVERALL_SCORE__` → integer 0-100 (weighted average of six scores)

### Data schema

{ "sections": [ { "num": "01", "name": "Installation Guide", "rating": "Good", "score": 74, "strengths": ["string", ...], "gaps": [ { "text": "Description of the gap", "covered_in": ["page-name-1", "page-name-2"], "nowhere": false } ] } ] }

Set `"nowhere": true` and `"covered_in": []` when the gap exists nowhere in the corpus.

{ "priorities": [ { "rank": 1, "text": "Fix X in Y page. The correct form per api-reference is Z.", "type": "error" } ] }

Priority types: `"error"` (factual mistake), `"missing"` (not covered anywhere), `"xref"` (covered elsewhere, needs linking), `"improvement"` (should-have gap).

---

## Output

Save the final report to `/mnt/user-data/outputs/sdk-audit-report.html` and present it with `present_files`.

Tell the user:
- How many pages were fetched
- The overall score and what it means
- The single most critical finding

Do NOT reproduce the full report text in the chat — just present the file and give the brief summary.

SDK Docs Auditor

Crawls any SDK documentation site, audits six core sections against a structured checklist, cross-references every gap across the full page corpus, and produces a scored downloadable HTML report.

---

What this skill does

Most SDK doc quality checks are manual and inconsistent. This skill automates a full audit by discovering every relevant SDK page via llms.txt or sitemap.xml, fetching each one, and evaluating six sections (Installation, Quick Start, Error Handling, Troubleshooting, Examples, and Best Practices) against detailed must-have and should-have criteria.

The key differentiator: every gap is cross-referenced across the entire page corpus before being flagged. If the content exists elsewhere in the docs but is not linked, the audit says so, distinguishing "missing content" from "content that exists but isn't where it should be."

Built for:

• Developer marketing and DevRel teams doing a structured quality review of their own SDK docs before a launch or release
• Technical writers running a gap analysis to prioritise what to write next
• Agencies and consultants producing a scored deliverable for a client SDK review

---

Installation

Claude Code (Recommended)

Clone the repo. The skill activates automatically when you open it in Claude Code:

git clone https://github.com/Infrasity-Labs/dev-gtm-claude-skills.git
cd dev-gtm-claude-skills
claude

Then trigger it with:

/dev-gtm sdk-audit https://docs.example.com/sdk

Or just describe what you want. Claude activates the skill when you provide a docs URL and ask for an audit, review, or quality check.

Claude Web (Free / Pro)

1. Go to [Settings → Capabilities](https://claude.ai/settings/capabilities) and enable Code execution and file creation 2. Go to [Customize → Skills](https://claude.ai/customize/skills) 3. Click + → Create skill → Upload a skill 4. Zip this skill folder and upload it:

cd dev-gtm-claude-skills/skills/sdk-docs-auditor
zip -r ../sdk-docs-auditor.zip .

Upload sdk-docs-auditor.zip and toggle it on.

Egress network required. This skill uses curl to fetch pages from the target docs site. In Claude Code, you will be prompted to allow network access when the skill runs. In Claude Web, code execution (enabled above) covers this automatically.

---

How to use

Audit the SDK docs at https://docs.example.com/sdk

How good are the docs at https://docs.example.com?

Review the SDK documentation at docs.example.com

/dev-gtm sdk-audit https://docs.example.com

Claude picks up any message that includes a docs URL alongside words like: audit, review, check, analyse, quality, completeness, or gaps.

---

Inputs

No other inputs are required. Page discovery, fetching, and section mapping all happen automatically.

---

Output

A self-contained HTML report file saved to /mnt/user-data/outputs/sdk-audit-report.html and presented with a download link.

The report contains:

• Overall score (0–100): weighted average of the six section scores
• Per-section scores and rating tiers: Strong / Good / Adequate / Weak / Poor
• Strengths: what is done well in each section
• Gap list: every gap tagged as either [covered in: page-name] or [not covered anywhere in corpus]
• Top priority recommendations: 6–8 ranked actions by impact, distinguishing factual errors, missing content, and cross-referencing opportunities

After presenting the file, Claude gives a one-line summary: pages fetched, overall score, and the single most critical finding.

---

Things to know

Cross-referencing is the core value. The audit never flags a gap without first checking whether the content exists somewhere else in the docs. "Gap covered elsewhere but not linked" is treated differently from "gap that does not exist anywhere."

Discovery falls back gracefully. The skill tries llms.txt first, then sitemap.xml, then the homepage nav. It always reports which method was used.

Large sites are prioritised intelligently. For sites with more than 30 pages, the six target section pages are fetched first, followed by overview and API reference pages, then service-specific pages.

The report file does not appear in chat. The full report HTML is presented as a download, not reproduced as chat text. Claude gives a brief summary inline.

---

How it works

1. Discover all SDK pages: tries llms.txt first (extracts URLs from link lines), falls back to sitemap.xml (parses <loc> entries), then the homepage nav; filters to SDK-relevant paths only 2. Map sections: identifies which discovered pages correspond to the six audit targets (Installation, Quick Start, Error Handling, Troubleshooting, Examples, Best Practices) 3. Fetch every page: runs curl on each URL, strips HTML with Python to extract plain text; batches multiple pages in one bash call to minimise round-trips 4. Audit each section: evaluates must-haves (scored heavily) and should-haves (scored moderately) against the fetched content 5. Cross-reference gaps: for every gap identified, searches all other fetched pages to determine whether the content exists elsewhere in the corpus 6. Score sections: assigns 0–100 per section and a rating tier based on how many must-haves and should-haves are met 7. Rank recommendations: orders the top 6–8 actions by impact × effort: factual errors first, then unmet must-haves, then cross-referencing gaps, then should-haves 8. Generate the HTML report: injects all audit data into assets/report-template.html and saves the file

---

Config schema reference

The JSON Claude assembles and injects into the report template:

{
  "audit_url": "https://docs.example.com/sdk",
  "pages_fetched": 24,
  "audit_date": "2026-05-29",
  "overall_score": 71,
  "sections": [
    {
      "num": "01",
      "name": "Installation Guide",
      "rating": "Good",
      "score": 74,
      "strengths": [
        "Prerequisites with exact Python version listed",
        "pip install command present with package name"
      ],
      "gaps": [
        {
          "text": "No verification snippet to confirm the install succeeded",
          "covered_in": [],
          "nowhere": true
        },
        {
          "text": "Constructor parameter reference (timeout, retries) missing from this page",
          "covered_in": ["api-reference"],
          "nowhere": false
        }
      ]
    }
  ],
  "priorities": [
    {
      "rank": 1,
      "text": "Fix execute() call signature on quick-start page: api-reference shows it takes a session_id parameter that is absent from the quick-start example",
      "type": "error"
    },
    {
      "rank": 2,
      "text": "Add a verification snippet to the Installation page, not covered anywhere in the corpus",
      "type": "missing"
    }
  ]
}

Priority types: "error" (factual mistake), "missing" (not covered anywhere), "xref" (covered elsewhere, needs linking), "improvement" (should-have gap).

---

File structure

sdk-docs-auditor/
├── SKILL.md                          # Skill instructions Claude follows
├── README.md                         # This file
├── assets/
│   └── report-template.html          # HTML report shell with injected placeholders
└── references/
    └── cross-reference-guide.md      # Rules for cross-referencing gaps across the corpus

Cross-Reference Guide

When auditing any gap, always check if the content exists in another page before flagging it as "not covered anywhere". This reference describes the most common cross-reference patterns.

Pages to always fetch and cross-check

1. `/sdk/installation` or /getting-started 2. `/sdk/quick-start` or /quickstart 3. `/sdk/error-handling` or /errors 4. `/sdk/troubleshooting` 5. `/sdk/examples` 6. `/sdk/best-practices` 7. `/sdk/api-reference` or /sdk/reference — CRITICAL for exception class names 8. `/sdk/overview` or /sdk/control-plane-client-overview — CRITICAL for constructor params 9. Service pages: agents, teams, jobs, workers, policies, environments, models, runtimes, skills, secrets, integrations, memory, semantic-search, context-graph

Common cross-reference patterns by gap type

Constructor / client setup gaps

• Env-var auto-detection → check: api-reference, overview, control-plane-client-overview
• Advanced params (timeout, max_retries, org_name) → check: api-reference
• Custom base_url → check: api-reference, overview

Exception class gaps

• List all exception classes → check: api-reference exceptions section
• Service-specific exceptions (TeamError, JobError, etc.) → check: respective service page error handling section
• Import paths → check: api-reference

Method signature discrepancies

• Execute call signature → check: api-reference, control-plane-agents
• Any method params → check: api-reference first, then the service-specific page

Missing examples

• Memory examples → check: context-graph-memory
• Semantic search → check: context-graph-semantic-search
• Job scheduling → check: control-plane-jobs
• Policy management → check: control-plane-policies
• Worker management → check: control-plane-workers
• Agent lifecycle → check: control-plane-agents

Missing best practices

• Worker best practices → check: control-plane-workers best practices section
• Job best practices → check: control-plane-jobs best practices section
• Policy best practices → check: control-plane-policies best practices section
• Agent best practices → check: control-plane-agents best practices section

Missing troubleshooting

• Worker issues → check: control-plane-workers error handling section
• Job errors → check: control-plane-jobs error handling section
• Policy errors → check: control-plane-policies error handling section
• Dataset not found → check: control-plane-client-overview or dataset service page
• Health check URL → check: api-reference health service section, examples

Gap classification rules

Priority type classification

Scoring deductions guide

Installation (start at 100)

• Missing env-var auto-detection: -8
• Missing optional extras explanation: -6
• Missing version pinning guidance: -5
• Missing advanced constructor params: -7
• No verification snippet with expected output: -10
• No prerequisites section: -15
• No install command: -20

Quick Start (start at 100)

• No minimal numbered step path: -15
• No expected output on any snippet: -10
• Worker/queue prerequisite unexplained: -12
• Model IDs not discoverable: -8
• Signature inconsistency with api-reference: -10
• No link to deeper docs: -5

Error Handling (start at 100)

• Missing exception class from api-reference: -8 each
• No exception hierarchy diagram: -10
• Missing HTTP 401/403: -6
• No streaming error guidance: -7
• Wrong method signature in an example: -12
• Missing ControlPlaneError base usage: -5

Troubleshooting (start at 100)

• No worker troubleshooting: -15
• No self-hosted/base_url guidance: -10
• Circular resolution for a common error: -8
• Missing service-specific section: -8 each
• No health check reference in "getting help": -5
• Rate limiting has no section: -8

Examples (start at 100)

• Missing entire service category: -10 each
• No expected output on examples: -8
• Workflow example doesn't actually execute: -8
• No error handling in workflows: -7
• Missing multi-turn/session example: -6

Best Practices (start at 100)

• Missing worker best practices: -8
• Missing job best practices: -7
• Missing policy best practices: -7
• Missing agent best practices: -7
• No concurrency/thread-safety: -10 (not covered anywhere penalty)
• No secret rotation guidance: -8
• Metrics example connects to nothing: -3