
A11y Audit
- 174 installs
- 2 repo stars
- Updated July 22, 2026
- snapsynapse/skill-a11y-audit
A11y Audit is an agent skill that runs WCAG-oriented accessibility scans and structured issue planning with safe tracker dry-runs.
About
A11y Audit is a SnapSynapse agent skill bundle that helps solo builders and small teams find, classify, and report web accessibility problems using scripted discovery and scanning plus WCAG-aware issue planning. Instead of ad-hoc Lighthouse copy-paste, the skill packages versioned references, planning helpers, and eval fixtures so agents can dry-run markdown reports and issue payloads before touching a live tracker. Recent releases strengthened context-aware severity thresholds, duplicate keys against existing issues, and additional standards labels on planned findings. Discovery improvements keep published sitemap URLs honest across robots.txt, indexes, and redirects, with runnable regression fixtures for sampling and cross-origin preservation. Use it when you are hardening a marketing site, SaaS dashboard, or extension UI before launch, or when you need repeatable audit output for clients. Intermediate complexity reflects configuring thresholds, running local scripts, and optionally connecting issue modes. It complements manual keyboard and screen-reader checks rather than replacing human verification for every WCAG success criterion.
- a11y-audit bundle v15 with changelog-driven issue planning and markdown+issues dry-run validation
- plan-issues.js dry-run reads project context, severity thresholds, priority, WCAG labels, and duplicate detection
- Discovery path preserves cross-origin sitemap URLs with eval-9 and eval-10 runnable fixtures in-repo
- Issue tracker integration docs include a safe dry-run path aligned to references/issue-trackers.md
- Local scan.js workflow with auto-installed deps/ and standards-aware issue metadata
A11y Audit by the numbers
- 174 all-time installs (skills.sh)
- +1 installs in the week ending Aug 5, 2026 (Skillselion tracking)
- Ranked #840 of 2,153 Testing & QA skills by installs in the Skillselion catalog
- Security screen: MEDIUM risk (skills.sh audit)
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/snapsynapse/skill-a11y-audit --skill a11y-auditAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 174 |
|---|---|
| repo stars | ★ 2 |
| Security audit | 1 / 3 scanners passed |
| Last updated | July 22, 2026 |
| Repository | snapsynapse/skill-a11y-audit ↗ |
What it does
Run structured accessibility audits with WCAG-oriented issue planning, discovery fixtures, and optional tracker dry-runs before you ship web UI.
Who is it for?
Best when you're shipping customer-facing web apps and want agent-driven a11y workflows with local scripts and eval-backed discovery behavior.
Skip if: Skip if you only need a one-click browser extension audit with no scripting, or non-web products with no DOM surface to scan.
When should I use this skill?
You need to audit web accessibility, plan WCAG-prioritized issues, or validate discovery and issue dry-runs before tracker writes.
What you get
You get discovery-correct URL coverage, scored findings, and dry-run issue plans you can promote to markdown reports or trackers after review.
- Accessibility scan results with WCAG-oriented labels and priorities
- Dry-run issue plan (markdown + issues) suitable for tracker import after human review
- Discovery output that preserves sitemap URLs for comprehensive page coverage
By the numbers
- Skill bundle version 15 (2026-04-23) with v14 discovery fixtures eval-9 and eval-10
- Runnable eval-2 fixture covers strongest local mitigation for issue mode short of live tracker writes
- plan-issues.js dry-run adds priority, WCAG labels, additional standards, and duplicate marking from existing-key index
Files
Accessibility Audit
Architecture
This skill operates as a single layer. It reads the project environment, runs automated accessibility tools, maps findings to compliance standards, and produces output in a configurable format. No external skill dependency is required.
Store project-specific audit state in the target workspace, not in the installed skill directory. Default path: .a11y-audit/PROJECT_CONTEXT.md at the workspace root. When that file exists, use it for project-specific configuration: output mode, additional compliance standards, issue tracker settings, route lists, color palettes, and cross-references to existing documentation. When absent, use WCAG 2.1 AA as the sole standard, markdown as the output mode, and generic defaults for everything else.
Use references/project-context-template.md as the canonical schema for that file, including minimal and markdown+issues examples.
Prefer bundled helpers over ad hoc generation when they fit:
scripts/bootstrap-context.jscreates a workspace-local
.a11y-audit/PROJECT_CONTEXT.md from simple inputs.
scripts/scan.jsruns reusable axe-based scans and records optional
Lighthouse execution intent. Use --summary to reduce output size (keeps full violation detail, strips node data from passes/inapplicable).
scripts/report.jsgenerates the markdown report and JSON data file
from scan.js output. Handles WCAG compliance matrix, violation aggregation, and color-contrast detail extraction deterministically.
scripts/discover.jsidentifies template groups on large sites and
selects representative pages for scanning. Reads sitemap.xml first, falls back to HTML navigation crawl. Outputs a scan plan with template groups and a ready-to-use URL list for scan.js. Discovery is same-origin by default; use --allow-cross-origin-sitemaps only after the user approves sitemap-declared cross-origin targets.
Dependencies
scan.js requires axe-core and puppeteer. It resolves these in order:
1. Skill-local deps/ directory (sibling to scripts/) 2. Target project node_modules/ (and common workspace subdirs) 3. Global npm modules
If not found anywhere, scan.js auto-installs both packages to the skill-local deps/ directory. This means the skill works against any project without requiring the target to have accessibility tooling installed. The deps/ directory is gitignored.
Because scan.js may auto-install missing dependencies, agents should ask before invoking scan.js when the target workspace does not already provide axe-core and the chosen browser automation package.
The bundled scanner supports Puppeteer only. Treat --browser as a fixed option, not a user-controlled package installer.
Platform-Specific References
- If running in Claude Code, read
references/claude-code.mdfor
.claude/launch.json handling and Preview tool usage.
- If running in Codex, read
references/codex.mdfor workspace-local
state handling and execution assumptions.
references/output-contract.mdandreferences/output-schema.json
are encoded in scripts/report.js. Read them only when modifying the report script.
- Read
references/issue-trackers.mdonly whenoutput_modeis
markdown+issues.
- If the user wants to operationalize recurring audits in CI, start from
assets/ci/github-actions/accessibility-audit.yml.
- Prefer
scripts/plan-issues.jsbefore live ticket creation when you
need a safe review and deduplication pass.
The skill does not modify source code. It is an auditor, not a fixer. Findings are reported with remediation guidance; the user decides what to act on.
Output Modes
The skill supports three output modes, configured via the output_mode field in .a11y-audit/PROJECT_CONTEXT.md:
| Mode | Output | Use Case |
|---|---|---|
markdown | Markdown report only | Human review, documentation |
markdown+json | Markdown report + JSON data file | CI integration, dashboards, trend tracking |
markdown+issues | Markdown report + issue tracker tickets | Active remediation workflow |
On first run, if no output_mode is set in .a11y-audit/PROJECT_CONTEXT.md, ask the user which mode they prefer and persist their choice by appending an ## Output Configuration section to that file. If no context file exists, create it at the default path following references/project-context-template.md. Prefer scripts/bootstrap-context.js for first-run context creation when a simple generated file is sufficient.
The markdown+json mode writes a companion file alongside the report: audit-YYYY-MM-DD.json containing the raw axe-core results, Lighthouse scores, and the compliance matrix as structured data. This file is machine-readable and can be consumed by CI pipelines, dashboards, or trend-tracking tools.
The markdown+issues mode requires additional configuration in the context file (see Phase 6).
---
Pipeline
An accessibility audit moves through six phases. Each phase produces data the next phase consumes. Phases 1-4 always run. Phase 5 produces output based on the configured output mode. Phase 6 runs only in markdown+issues mode and requires explicit user confirmation.
The user can request a partial run. Common patterns:
- "Quick scan": Phases 1-2 only, results summarized in conversation
- "Full audit": Phases 1-5, output per configured mode
- "Audit with issues": Phases 1-6, report plus tracker tickets
Phase 1 -- Environment Discovery
Purpose: Understand the project before scanning.
1. Read .a11y-audit/PROJECT_CONTEXT.md if it exists (standards, routes, output mode, labels). If absent, create it via scripts/bootstrap-context.js or from references/project-context-template.md. 2. Read package.json for tech stack, existing a11y tooling, and available browser automation (puppeteer / playwright). 3. Build a scannable URL list from router config or HTML file glob. For sites with many pages (>15 routes), prefer scripts/discover.js to classify pages into template groups and select representatives. Review the scan plan with the user before proceeding to Phase 2. 4. Confirm a dev server is reachable. Check .claude/launch.json and platform-specific references for launch hints. If the app starts on a different URL than expected, switch to the live URL, record the mismatch, and update the context file. 5. Ask before installing any missing dependencies.
Output: Structured summary reported to the user before proceeding.
Phase 2 -- Automated Scanning
Purpose: Run automated accessibility checks against live pages.
Prerequisite: A running dev server (or production URL provided by the user).
axe-core Scanning
Prefer the bundled scripts/scan.js before writing a throwaway scan script. Use an ad hoc script only when the workspace needs behavior that the bundled script does not yet support.
The reusable scanner should:
1. Imports puppeteer and axe-core 2. Launches a headless browser 3. For each target URL: a. Navigates to the page b. Waits for network idle (waitUntil: 'networkidle0') c. Injects axe-core: reads the axe-core source file from node_modules/axe-core/axe.min.js and injects it via page.evaluate() d. Runs the audit: page.evaluate(() => axe.run()) e. Collects the results JSON 4. Closes the browser 5. Writes raw results to JSON
Example invocation:
node a11y-audit/scripts/scan.js \
--root . \
--urls http://127.0.0.1:3000/,http://127.0.0.1:3000/about \
--output /tmp/a11y-scan.json \
--summaryAdapt to the project. If dependencies live in a frontend subdirectory, point --root at the workspace root; the bundled script already checks common frontend paths. If that still fails, fall back to an ad hoc script or run from the frontend directory directly.
Common mistake: Do not use require() in an ES module project. Check package.json for "type": "module". If present, use import syntax and the __dirname workaround in any ad hoc script. If absent, require() is fine.
Playwright alternative: If the project uses Playwright instead of Puppeteer, adapt the script: replace puppeteer.launch() with chromium.launch(), use page.goto() the same way, and inject axe-core via page.evaluate(). The axe-core injection pattern is identical. Use whichever browser automation library the project already has installed.
Lighthouse Scanning
Run Lighthouse CLI against each target URL:
npx lighthouse <url> \
--output=json \
--output-path=stdout \
--only-categories=accessibility \
--chrome-flags="--headless --no-sandbox" \
--quietParse the JSON output. Extract:
categories.accessibility.score(0-1, multiply by 100)auditswherescore !== 1(failed or partial audits)- Each audit's
descriptionanddetails.items
If Lighthouse is unavailable or fails (common in CI environments), skip it and note the gap in the report. In the executive summary and methodology, explicitly say whether Lighthouse was skipped because the CLI was missing, Chrome launch failed, or another runtime error occurred. axe-core results alone are sufficient for a valid audit.
Scope Control
- Default: scan routes discovered in Phase 1
- If a discover.js scan plan exists, use its
scanListfor--urls.
The report methodology will record the sampling strategy.
- If more than 10 routes exist and no discover plan is available, ask
the user which to scan or whether to scan all
- The user can provide a specific URL list to override discovery
- For SPAs: navigate via the router, not by reloading the page (some
routes may not work as direct URLs)
Result Structure
For each page, collect:
url: the scanned URLviolations: array of axe violations, each withid,impact
(critical/serious/moderate/minor), description, help, helpUrl, tags (WCAG criteria), nodes (affected elements with selectors)
passes: count of passing rulesincomplete: rules that could not be fully evaluatedlighthouseScore: 0-100 (if available)lighthouseAudits: failed audit details (if available)
Phase 3 -- Compliance Mapping
Purpose: Map automated findings to WCAG 2.1 AA success criteria and any project-specific standards.
scripts/report.js handles the compliance matrix deterministically. It hardcodes all 50 WCAG 2.1 Level A and AA criteria, maps axe tags to success criteria, and produces the matrix as part of its markdown and JSON output. You do not need to build the matrix manually.
If .a11y-audit/PROJECT_CONTEXT.md specifies additional standards (e.g., CAN-ASC-6.2), build a secondary mapping. Cross-reference automated findings where the standard maps to WCAG criteria. For requirements that go beyond WCAG (equity, organizational processes, transparency), note them as manual review items referencing the project's existing conformance documentation.
Phase 4 -- Manual Check Guidance
Purpose: Generate targeted checklists for what automation cannot verify, prioritized by the automated findings.
For each WCAG criterion marked "Manual" in the Phase 3 matrix, generate a testing item. Organize by testing method: Keyboard Navigation, Screen Reader, Visual Inspection, Cognitive, and Timing/Motion.
Dynamic prioritization: Do not produce a static checklist. Use the Phase 2 results to focus manual effort:
- If axe found color-contrast violations, prioritize visual inspection
items (SC 1.4.1, 1.4.11, 1.4.10, 1.4.12, 1.4.13)
- If axe found ARIA or landmark violations, prioritize screen reader
items (SC 1.3.1, 4.1.3, 3.3.1, 3.3.2)
- If axe found heading or structure violations, prioritize keyboard
navigation items (SC 2.4.3, 2.4.7, 2.1.1)
- If no form-related violations were found, deprioritize form testing
(SC 3.3.3, 3.3.4) with a note that automated checks passed
- Always include timing items (SC 2.2.1, 2.2.2, 2.3.1) since these
cannot be automated at all
Each checklist item specifies: the WCAG criterion, what to test, how to test it, and which pages to focus on (pages where automated issues were found get priority).
If .a11y-audit/PROJECT_CONTEXT.md references an existing testing guide, cross-link to it rather than duplicating procedures.
Phase 5 -- Output Generation
Purpose: Produce output based on the configured output mode.
Run scripts/report.js to generate the markdown report and JSON data file from the Phase 2 scan output:
node a11y-audit/scripts/report.js \
--input /tmp/a11y-scan.json \
--output-dir docs/accessibility/audits \
--project-name "Project Name" \
--runtime-url http://127.0.0.1:3000 \
--expected-url http://localhost:3000 \
--discover /tmp/a11y-discover.jsonPass --discover when a discover.js scan plan was used. This adds a Sampling Strategy subsection to the report methodology documenting template groups and coverage ratio.
Pass --previous <prior-audit.json> to generate a Delta from Previous Audit section showing fixed rules, new rules, changed instance counts, and net progress.
The script produces audit-YYYY-MM-DD.md and audit-YYYY-MM-DD.json following the contracts in references/output-contract.md and references/output-schema.json. You do not need to read those reference files unless modifying the report script itself.
After running report.js, review its output and fill in the Manual Testing Recommendations section with the Phase 4 guidance (report.js leaves a placeholder for this since it requires reasoning about the specific findings pattern).
.a11y-audit/PROJECT_CONTEXT.md can override the output path.
If the user wants a recurring or on-demand CI job, adapt assets/ci/github-actions/accessibility-audit.yml to the target workspace instead of inventing a workflow from scratch.
Phase 6 -- Issue Creation (conditional)
Purpose: Create issue tracker tickets for findings. Runs only when the output mode is markdown+issues.
This phase requires explicit user confirmation. Before creating any tickets, show the user how many will be created, at what priority levels, and ask for approval.
Read references/issue-trackers.md for tracker configuration, deduplication, priority mapping, and ticket structure. Use the tracker settings from .a11y-audit/PROJECT_CONTEXT.md.
---
Verification
After completing an audit, verify these quality checks:
1. axe results valid: Compare violation count against a manual axe DevTools browser extension run on the same page. Counts should match within tolerance (axe versions may differ slightly).
2. Lighthouse score consistent: Compare against a manual Chrome DevTools Lighthouse run when Lighthouse was actually executed. Should be within 5 points.
3. WCAG matrix complete: All 50 AA criteria appear in the compliance matrix. No criterion is missing.
Treat the matrix as evidence-oriented status reporting. Do not frame it as proof of full conformance, because many WCAG criteria remain manual even in a strong automated run.
4. Report structure: All required sections present. Tables render correctly in a markdown viewer.
5. JSON validity (markdown+json mode): JSON file parses without error. Violation counts match the markdown report.
6. Issue deduplication (markdown+issues mode): Run the skill twice. The second run should create zero duplicate tickets.
7. Output mode persistence: After first run, verify the output mode is saved to .a11y-audit/PROJECT_CONTEXT.md and used automatically on next run.
8. Runtime URL reconciliation: If the app started on a different local URL than expected, verify the report records the mismatch and the context file reflects the actual working base_url.
---
What This Skill Does NOT Do
- Visual regression testing: does not compare screenshots between
runs. Use Percy, Chromatic, or BackstopJS for that.
- PDF accessibility: does not audit PDF documents for tagged
structure, reading order, or alternative text.
- Real device/AT testing: runs in headless Chromium only. Cannot
test on real iOS/Android or with real screen readers. Phase 4 generates manual checklists for this.
- Code fixes: reports findings but does not modify source code.
- VPAT generation: does not produce Voluntary Product Accessibility
Templates (specific legal format).
- Continuous monitoring: runs on demand, not as a CI pipeline.
The markdown+json output mode provides structured data for building CI integrations, but the skill itself does not run in CI.
- Third-party auditing: only audits the project's own frontend,
not embedded third-party services.
# Auto-installed dependencies for scan.js
deps/
interface:
display_name: "Accessibility Audit"
short_description: "Audit web apps for WCAG issues"
default_prompt: "Use $a11y-audit to audit this web project for accessibility issues and produce a WCAG-mapped report."
name: Accessibility Audit
on:
workflow_dispatch:
schedule:
- cron: "0 14 * * 1"
jobs:
audit:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: "20"
- name: Install dependencies
run: npm ci
- name: Start app
run: npm run dev -- --host 127.0.0.1 &
- name: Wait for app
run: npx wait-on http://127.0.0.1:3000
- name: Run accessibility scan
run: >
node a11y-audit/scripts/scan.js
--root .
--urls http://127.0.0.1:3000/,http://127.0.0.1:3000/about
--output artifacts/a11y-scan.json
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: accessibility-audit
path: artifacts/
{
"date": "2026-03-26",
"tool": "a11y-audit report.js v1",
"pages": [
"https://sam-rogers.com/about/",
"https://sam-rogers.com/archive/",
"https://sam-rogers.com/blog/",
"https://sam-rogers.com/contact/",
"https://sam-rogers.com/now/",
"https://sam-rogers.com/privacy/",
"https://sam-rogers.com/projects/",
"https://sam-rogers.com/tags/",
"https://sam-rogers.com/blog/substack/",
"https://sam-rogers.com/blog/fringer/",
"https://sam-rogers.com/now/2022-11/",
"https://sam-rogers.com/now/2021-11/",
"https://sam-rogers.com/about/bio/",
"https://sam-rogers.com/about/colophon/",
"https://sam-rogers.com/"
],
"lighthouse": {
"status": "skipped",
"reason": "Lighthouse disabled for this run"
},
"summary": {
"critical": 0,
"serious": 5,
"moderate": 28,
"minor": 0
},
"violations": [
{
"rule": "landmark-unique",
"impact": "moderate",
"wcag": [],
"pages": [
"https://sam-rogers.com/about/",
"https://sam-rogers.com/archive/",
"https://sam-rogers.com/blog/",
"https://sam-rogers.com/contact/",
"https://sam-rogers.com/now/",
"https://sam-rogers.com/privacy/",
"https://sam-rogers.com/projects/",
"https://sam-rogers.com/tags/",
"https://sam-rogers.com/blog/substack/",
"https://sam-rogers.com/blog/fringer/",
"https://sam-rogers.com/now/2022-11/",
"https://sam-rogers.com/now/2021-11/",
"https://sam-rogers.com/about/bio/",
"https://sam-rogers.com/about/colophon/",
"https://sam-rogers.com/"
],
"instances": 15
},
{
"rule": "region",
"impact": "moderate",
"wcag": [],
"pages": [
"https://sam-rogers.com/about/",
"https://sam-rogers.com/archive/",
"https://sam-rogers.com/blog/",
"https://sam-rogers.com/contact/",
"https://sam-rogers.com/now/",
"https://sam-rogers.com/privacy/",
"https://sam-rogers.com/about/bio/",
"https://sam-rogers.com/about/colophon/"
],
"instances": 8
},
{
"rule": "landmark-one-main",
"impact": "moderate",
"wcag": [],
"pages": [
"https://sam-rogers.com/archive/",
"https://sam-rogers.com/blog/"
],
"instances": 2
},
{
"rule": "heading-order",
"impact": "moderate",
"wcag": [],
"pages": [
"https://sam-rogers.com/contact/",
"https://sam-rogers.com/privacy/"
],
"instances": 2
},
{
"rule": "page-has-heading-one",
"impact": "moderate",
"wcag": [],
"pages": [
"https://sam-rogers.com/projects/"
],
"instances": 1
},
{
"rule": "frame-title",
"impact": "serious",
"wcag": [
"4.1.2"
],
"pages": [
"https://sam-rogers.com/blog/substack/"
],
"instances": 1
},
{
"rule": "list",
"impact": "serious",
"wcag": [
"1.3.1"
],
"pages": [
"https://sam-rogers.com/blog/substack/",
"https://sam-rogers.com/blog/fringer/",
"https://sam-rogers.com/now/2022-11/",
"https://sam-rogers.com/now/2021-11/"
],
"instances": 4
}
],
"matrix": {
"1.1.1": "pass",
"1.2.1": "manual",
"1.2.2": "not-applicable",
"1.2.3": "manual",
"1.2.4": "manual",
"1.2.5": "manual",
"1.3.1": "fail",
"1.3.2": "manual",
"1.3.3": "manual",
"1.3.4": "manual",
"1.3.5": "not-applicable",
"1.4.1": "pass",
"1.4.2": "not-applicable",
"1.4.3": "pass",
"1.4.4": "pass",
"1.4.5": "manual",
"1.4.10": "manual",
"1.4.11": "manual",
"1.4.12": "pass",
"1.4.13": "manual",
"2.1.1": "not-applicable",
"2.1.2": "manual",
"2.1.4": "manual",
"2.2.1": "not-applicable",
"2.2.2": "not-applicable",
"2.3.1": "manual",
"2.4.1": "pass",
"2.4.2": "pass",
"2.4.3": "manual",
"2.4.4": "pass",
"2.4.5": "manual",
"2.4.6": "manual",
"2.4.7": "manual",
"2.5.1": "manual",
"2.5.2": "manual",
"2.5.3": "manual",
"2.5.4": "manual",
"3.1.1": "pass",
"3.1.2": "not-applicable",
"3.2.1": "manual",
"3.2.2": "manual",
"3.2.3": "manual",
"3.2.4": "manual",
"3.3.1": "manual",
"3.3.2": "pass",
"3.3.3": "manual",
"3.3.4": "manual",
"4.1.1": "manual",
"4.1.2": "fail",
"4.1.3": "manual"
},
"expected_url": "https://sam-rogers.com",
"runtime_url": "https://sam-rogers.com",
"sampling": {
"source": "sitemap (/sitemap.xml)",
"totalPages": 206,
"selectedPages": 15,
"groups": [
{
"pattern": "about",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "archive",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "blog",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "contact",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "now",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "privacy",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "projects",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "tags",
"count": 1,
"scanned": 1,
"entity": null
},
{
"pattern": "blog/*",
"count": 143,
"scanned": 2,
"entity": null
},
{
"pattern": "now/*",
"count": 52,
"scanned": 2,
"entity": null
},
{
"pattern": "about/*",
"count": 2,
"scanned": 2,
"entity": null
},
{
"pattern": "/",
"count": 1,
"scanned": 1,
"entity": null
}
]
},
"sharedTemplates": [
{
"patterns": [
"about",
"now",
"about/*"
],
"rules": [
"landmark-unique",
"region"
],
"pageCount": 4
},
{
"patterns": [
"archive",
"blog"
],
"rules": [
"landmark-one-main",
"landmark-unique",
"region"
],
"pageCount": 2
},
{
"patterns": [
"contact",
"privacy"
],
"rules": [
"heading-order",
"landmark-unique",
"region"
],
"pageCount": 2
},
{
"patterns": [
"tags",
"/"
],
"rules": [
"landmark-unique"
],
"pageCount": 2
},
{
"patterns": [
"blog/*",
"now/*"
],
"rules": [
"landmark-unique",
"list"
],
"pageCount": 3
}
]
}Accessibility Audit Report
Header
| Field | Value |
|---|---|
| Project | sam-rogers.com |
| Date | 2026-03-26 |
| Standards | WCAG 2.1 AA |
| Tool Version | axe-core (via scan.js); report.js v1 |
| URL | https://sam-rogers.com |
Executive Summary
sam-rogers.com was audited across 15 page(s). Automated scanning found 33 issue instance(s) across 7 rule(s).
| Impact | Instances |
|---|---|
| serious | 5 |
| moderate | 28 |
Lighthouse was skipped: Lighthouse disabled for this run.
Automated Scan Results
Pages Scanned
- https://sam-rogers.com/about/
- https://sam-rogers.com/archive/
- https://sam-rogers.com/blog/
- https://sam-rogers.com/contact/
- https://sam-rogers.com/now/
- https://sam-rogers.com/privacy/
- https://sam-rogers.com/projects/
- https://sam-rogers.com/tags/
- https://sam-rogers.com/blog/substack/
- https://sam-rogers.com/blog/fringer/
- https://sam-rogers.com/now/2022-11/
- https://sam-rogers.com/now/2021-11/
- https://sam-rogers.com/about/bio/
- https://sam-rogers.com/about/colophon/
- https://sam-rogers.com/
Findings by Rule
| Rule | Impact | Instances | Pages | WCAG |
|---|---|---|---|---|
| landmark-unique | moderate | 15 | 15 | - |
| region | moderate | 8 | 8 | - |
| landmark-one-main | moderate | 2 | 2 | - |
| heading-order | moderate | 2 | 2 | - |
| page-has-heading-one | moderate | 1 | 1 | - |
| frame-title | serious | 1 | 1 | SC 4.1.2 |
| list | serious | 4 | 4 | SC 1.3.1 |
Quick Fixes
- list (serious, 4 instances): Ensure
<li>elements are direct children of<ul>or<ol>. Do not place non-list content directly inside list containers. - region (moderate, 8 instances): Ensure all page content is inside a landmark region (
<main>,<nav>,<header>,<footer>, orrole="..."). - landmark-one-main (moderate, 2 instances): Wrap the primary content area in a
<main>element. This also resolves mostregionviolations. - heading-order (moderate, 2 instances): Heading levels should increase by one (h1 → h2 → h3). Do not skip levels.
WCAG 2.1 AA Compliance Matrix
This is an automation-assisted status view, not a conformance certification.
| SC | Name | Level | Status |
|---|---|---|---|
| Perceivable | |||
| SC 1.1.1 | Non-text Content | A | Pass |
| SC 1.2.1 | Audio-only and Video-only (Prerecorded) | A | Manual |
| SC 1.2.2 | Captions (Prerecorded) | A | N/A |
| SC 1.2.3 | Audio Description or Media Alternative (Prerecorded) | A | Manual |
| SC 1.2.4 | Captions (Live) | AA | Manual |
| SC 1.2.5 | Audio Description (Prerecorded) | AA | Manual |
| SC 1.3.1 | Info and Relationships | A | Fail |
| SC 1.3.2 | Meaningful Sequence | A | Manual |
| SC 1.3.3 | Sensory Characteristics | A | Manual |
| SC 1.3.4 | Orientation | AA | Manual |
| SC 1.3.5 | Identify Input Purpose | AA | N/A |
| SC 1.4.1 | Use of Color | A | Pass |
| SC 1.4.2 | Audio Control | A | N/A |
| SC 1.4.3 | Contrast (Minimum) | AA | Pass |
| SC 1.4.4 | Resize Text | AA | Pass |
| SC 1.4.5 | Images of Text | AA | Manual |
| SC 1.4.10 | Reflow | AA | Manual |
| SC 1.4.11 | Non-text Contrast | AA | Manual |
| SC 1.4.12 | Text Spacing | AA | Pass |
| SC 1.4.13 | Content on Hover or Focus | AA | Manual |
| Operable | |||
| SC 2.1.1 | Keyboard | A | N/A |
| SC 2.1.2 | No Keyboard Trap | A | Manual |
| SC 2.1.4 | Character Key Shortcuts | A | Manual |
| SC 2.2.1 | Timing Adjustable | A | N/A |
| SC 2.2.2 | Pause, Stop, Hide | A | N/A |
| SC 2.3.1 | Three Flashes or Below Threshold | A | Manual |
| SC 2.4.1 | Bypass Blocks | A | Pass |
| SC 2.4.2 | Page Titled | A | Pass |
| SC 2.4.3 | Focus Order | A | Manual |
| SC 2.4.4 | Link Purpose (In Context) | A | Pass |
| SC 2.4.5 | Multiple Ways | AA | Manual |
| SC 2.4.6 | Headings and Labels | AA | Manual |
| SC 2.4.7 | Focus Visible | AA | Manual |
| SC 2.5.1 | Pointer Gestures | A | Manual |
| SC 2.5.2 | Pointer Cancellation | A | Manual |
| SC 2.5.3 | Label in Name | A | Manual |
| SC 2.5.4 | Motion Actuation | A | Manual |
| Understandable | |||
| SC 3.1.1 | Language of Page | A | Pass |
| SC 3.1.2 | Language of Parts | AA | N/A |
| SC 3.2.1 | On Focus | A | Manual |
| SC 3.2.2 | On Input | A | Manual |
| SC 3.2.3 | Consistent Navigation | AA | Manual |
| SC 3.2.4 | Consistent Identification | AA | Manual |
| SC 3.3.1 | Error Identification | A | Manual |
| SC 3.3.2 | Labels or Instructions | A | Pass |
| SC 3.3.3 | Error Suggestion | AA | Manual |
| SC 3.3.4 | Error Prevention (Legal, Financial, Data) | AA | Manual |
| Robust | |||
| SC 4.1.1 | Parsing | A | Manual |
| SC 4.1.2 | Name, Role, Value | A | Fail |
| SC 4.1.3 | Status Messages | AA | Manual |
Manual Testing Recommendations
<!-- report.js: This section should be populated by the auditing agent --> <!-- based on Phase 4 manual check guidance, which requires reasoning --> <!-- about the specific findings pattern. -->
Remediation Priority
| Priority | Rule | Impact | Instances | WCAG |
|---|---|---|---|---|
| P0 | frame-title | serious | 1 | SC 4.1.2 |
| P1 | list | serious | 4 | SC 1.3.1 |
| P2 | landmark-unique | moderate | 15 | - |
| P3 | region | moderate | 8 | - |
| P3 | landmark-one-main | moderate | 2 | - |
| P3 | heading-order | moderate | 2 | - |
| P3 | page-has-heading-one | moderate | 1 | - |
Issues Created
Issue creation was not executed by the report generator.
Methodology
| Field | Value |
|---|---|
| Scan Date | 2026-03-26 |
| Pages Scanned | 15 |
| Viewport | 1280 x 800 |
| Browser | Headless Chromium (Puppeteer) |
| Scanner | axe-core via scan.js |
| Lighthouse | Skipped: Lighthouse disabled for this run |
| Runtime URL | https://sam-rogers.com |
| Expected URL | https://sam-rogers.com |
Sampling Strategy
Pages were selected via template-aware sampling (discover.js). 206 total pages were classified into 12 template groups; 15 representative pages were scanned.
| Template Group | Total Pages | Scanned | Selection |
|---|---|---|---|
about | 1 | 1 | top-level page — always included |
archive | 1 | 1 | top-level page — always included |
blog | 1 | 1 | top-level page — always included |
contact | 1 | 1 | top-level page — always included |
now | 1 | 1 | top-level page — always included |
privacy | 1 | 1 | top-level page — always included |
projects | 1 | 1 | top-level page — always included |
tags | 1 | 1 | top-level page — always included |
blog/* | 143 | 2 | 2 of 143 — by DOM complexity (scores: 7→2) |
now/* | 52 | 2 | 2 of 52 — by DOM complexity (scores: 5→3) |
about/* | 2 | 2 | all 2 — small group |
/ | 1 | 1 | singleton — always included |
Shared Template Patterns
Template groups with identical violation fingerprints share the same underlying issues. Fixing the shared template resolves the issue across all pages in those groups.
- *Shared issues on `about`, `now`, `about/`:** landmark-unique, region
- Shared issues on `archive`, `blog`: landmark-one-main, landmark-unique, region
- Shared issues on `contact`, `privacy`: heading-order, landmark-unique, region
- Shared issues on `tags`, `/`: landmark-unique
- *Shared issues on `blog/
,now/`:* landmark-unique, list
Accessibility Issue Plan
Input: audit-sample.json Threshold: P1 Planned tickets: 1
| Priority | Rule | Route | Instances | Dedup Key | Summary |
|---|---|---|---|---|---|
| P1 | color-contrast | / | 3 | <!-- a11y-audit-key: color-contrast::/ --> | Elements must meet minimum color contrast ratio thresholds |
Use this plan for user review and deduplication checks before live ticket creation.
Changelog
v2.1.1 -- 2026-06-04
scripts/scan.js(v5):--sitemapnow transparently recurses into
<sitemapindex> documents, following each child <sitemap><loc> to the leaf URL set. Cycle-protected (50-doc cap), and find/replace runs before the child fetch so the rewritten host applies recursively. Caught while wiring publedge.org to the composite action — its sitemap.xml is a 7-section index, which the previous loader couldn't walk.
v2.1.0 -- 2026-06-03
- Added CI-ready CLI flags to
scripts/scan.js: --sitemap <url>discovers targets from a sitemap.xml at scan time,
with optional --sitemap-find / --sitemap-replace host rewriting and --sitemap-exclude <regex> URL filtering.
--fail-on errorsmakes the script exit non-zero (status 2) when
axe reports any violation, so the scan's exit code carries the gate semantic to CI without callers parsing JSON.
- Added a reusable GitHub composite action at
.github/actions/scan/action.yml that wraps the scanner (optional http-server, scan, artifact upload). Consumer repos that previously hand-rolled pa11y-ci now collapse to a few lines invoking this action.
v2.0.1 -- 2026-05-31
- Hardened
scan.jsdependency installation by validating browser
selection before dependency lookup and invoking npm install with argv rather than shell interpolation.
- Added bounded discovery fetches, redirect limits, same-origin discovery
defaults, explicit cross-origin sitemap opt-in, and origin disclosure in discovery/report output.
- Escaped target-derived Markdown fields in generated audit reports and
issue plans to reduce agent-facing instruction injection and table breakage from untrusted scan data.
- Documented the repository release, bundle inventory, and assistant
guide version model.
Assistant guide v0.3.0 -- 2026-05-24
- Added Level 4 provenance metadata for the GuideCheck
assistant-guide.txt artifact, including a text sidecar manifest URL.
- Prepared the public repository copy at
/assistant-guide.txtas the
cross-channel hash anchor for the published guide.
- Tightened action block metadata to satisfy the GuideCheck hosted
verifier's guide-file checks.
v16 -- 2026-05-19
- Added
evals/run-evals.js, a deterministic offline eval runner that
validates discovery fixtures, issue planning, quick-scan behavior, skipped-Lighthouse report contracts, page-aware deltas, dependency policy documentation, syntax checks, JSON parsing, YAML parsing, and context bootstrap smoke behavior.
- Added fixture coverage for
eval-3quick-scan behavior andeval-4
skipped-Lighthouse report contract behavior.
- Updated
package.json,.github/workflows/validate-skill.yml, and
CONTRIBUTING.md so npm run validate is the single local and CI validation entrypoint.
- Updated
SKILL.mdto make the dependency auto-install consent
boundary explicit before agents invoke scan.js in workspaces without existing scan dependencies.
- Updated
evals/evals.jsonso the human acceptance ledger records the
deterministic eval-3 fixture and the new eval-4 contract fixture.
v15 -- 2026-04-23
- Expanded
scripts/plan-issues.jsso the dry-run path now reads
project context, applies configured severity thresholds, adds priority and WCAG labels, records additional standards, and marks duplicates from a supplied existing-key index.
- Added runnable fixture coverage for
eval-2, covering the strongest
local mitigation path for issue mode short of authenticated live tracker writes.
- Updated
references/issue-trackers.mdso the documented safe dry-run
path matches the richer helper behavior.
v14 -- 2026-04-23
- Added
evals/run-discover-fixture.js, a reusable mocked-network
fixture runner for discover.js.
- Added runnable fixture assets for
eval-9(cross-origin sitemap
preservation) and eval-10 (deterministic discovery sampling).
- Validated both discovery fixtures locally so the two discovery-related
regressions now have in-repo reproduction paths, not just notes in evals/evals.json.
v13 -- 2026-04-23
- Discovery now preserves published sitemap URLs instead of rewriting
them onto the runtime origin. This includes robots.txt sitemap entries, sitemap indexes, nested sitemap URLs, and redirect targets.
- Discovery sampling is now deterministic. Candidate selection uses
stable spread indexes and fingerprint ranking now breaks ties by URL, eliminating noisy CI diffs across repeated runs.
discover.jsis importable without changing CLI behavior, which makes
mocked-network regression validation possible inside the repo.
report.jsdelta comparison is now page-aware. A rule that moves
between pages or template groups with the same instance count is reported as changed rather than unchanged.
- Added and validated eval coverage for the resolved issues:
eval-9 (cross-origin sitemap preservation), eval-10 (deterministic discovery), and eval-11 (page-aware delta changes).
v12 -- 2026-03-26
- Self-contained dependencies: scan.js now resolves axe-core and
puppeteer from skill-local deps/ → project → global → auto-install. The skill works against any project without requiring accessibility tooling to be pre-installed.
- Quick Fixes: report.js includes actionable one-liner remediation
hints for ~17 common axe rules, sorted by impact severity.
- Delta comparison:
report.js --previous <prior-audit.json>shows
fixed rules (strikethrough), new rules, changed instance counts with direction arrows, and net totals.
- Null-label fix: API enrichment labels no longer show "null" when
the API manifest lacks count data for an entity type.
v11 -- 2026-03-26
- discover.js: Template-aware page discovery with sitemap-first
approach. Falls back to HTML navigation crawl if no sitemap exists.
- DOM fingerprinting: Loads candidate pages and scores structural
complexity (tables, details, forms, interactive attrs). Picks the most and least complex pages per group instead of alphabetic spread.
- API entity enrichment: Reads
/api/v1/index.json(or similar)
and annotates groups with entity names/counts (e.g., "25 regulations").
- Shared template detection: report.js cross-references per-page
violation fingerprints with discover groups. Surfaces which template groups share identical issues so developers fix the shared template once.
- No-sitemap fallback validated: HTML crawl (depth 2) found 132
pages and all key template groups on AI Regulation Reference.
- Validated on AI Regulation Reference: 746 pages → 16 groups →
22 scanned → 12 serious violations found on templates the previous top-level-only scan missed entirely.
v10 -- 2026-03-26
- report.js: New deterministic report generator (
scripts/report.js)
handles Phases 3 and 5: WCAG compliance matrix (hardcoded 50 criteria), violation aggregation across pages, color-contrast detail extraction, markdown report per output-contract.md, and JSON per output-schema.json. The LLM no longer builds these manually (~3000 tokens saved).
- scan.js --summary: Added
--summaryflag toscripts/scan.jsthat
keeps full violation detail but strips node data from passes and inapplicable arrays, reducing output size (~500 tokens saved).
- Phase 1 condensed: Replaced ~30-line framework-by-framework
enumeration with ~10 focused lines. The agent already knows how to discover project structure (~500 tokens saved).
- Reference reads removed: Phase 5 now invokes report.js directly.
The agent no longer reads output-contract.md or output-schema.json during normal runs (~800 tokens saved).
- Phase 4 stays LLM-generated: Manual check guidance requires
reasoning about the specific findings pattern and remains the agent's responsibility.
v9 -- 2026-03-03
- First-run context validation: Recorded a passing result for eval-6
by creating a workspace-local context file in /tmp with the bundled bootstrap helper.
- Missing-browser-automation validation: Recorded a passing result
for eval-7 by exercising the scanner against a workspace with axe-core present but no Puppeteer dependency, confirming a clear blocker message.
- Issue planning mitigation: Added
scripts/plan-issues.jsand a
sample issue plan artifact so markdown+issues mode has a safe dry-run path before live tracker writes.
- Validation stance: Reduced the remaining publish-time runtime gap
to the live authenticated tracker path rather than the whole issue-mode workflow.
v8 -- 2026-03-03
- Direct degraded-path validation: Ran the bundled
scripts/scan.js
helper against PAICE2 and recorded passing results for eval-4 (Lighthouse unavailable but report still generated) and eval-5 (expected URL wrong but runtime URL reconciled and persisted).
- Eval results updated: Added concrete
resultsentries for eval-4
and eval-5 in evals/evals.json.
- CI discoverability: Updated
SKILL.mdso the GitHub Actions
starter in assets/ci/github-actions/accessibility-audit.yml is part of the visible operating guidance rather than a hidden asset.
- Execution stance: Kept the bundled scanner Puppeteer-first for
now. Playwright remains a documented fallback path but is not yet a first-class helper implementation.
v7 -- 2026-03-03
- Reusable scripts: Added
scripts/scan.jsfor reusable axe-based
scanning and scripts/bootstrap-context.js for first-run workspace context creation.
- Reference decomposition: Split detailed output rules into
references/output-contract.md, issue creation rules into references/issue-trackers.md, and the JSON contract into references/output-schema.json.
- Eval expansion: Added explicit eval coverage for missing
Lighthouse, runtime URL reconciliation, first-run context creation, missing browser automation, and issue deduplication.
- Operational assets: Added sample markdown/JSON output artifacts and
a GitHub Actions workflow template for scheduled or on-demand audits.
- Core skill cleanup: Updated
SKILL.mdto prefer bundled helpers
and focused references over repeated inline detail.
v6 -- 2026-03-03
- Runtime URL handling: Updated
SKILL.mdso a local port mismatch
is treated as a normal adaptation path. The skill now switches to the live URL, records the mismatch in methodology, and updates the workspace-local base_url.
- Lighthouse degraded mode: Clarified that missing or failing
Lighthouse is a documented partial-audit path, not a failure. The report must now state the skip reason explicitly in the executive summary and methodology.
- Eval alignment: Updated
evals/evals.jsonso eval-1 allows URL
reconciliation and Lighthouse-optional execution instead of assuming a fixed port and guaranteed Lighthouse score.
- Handoff update: Recorded the Codex eval-1 findings from PAICE2 so
the next session starts from observed runtime behavior rather than inferred gaps.
v5 -- 2026-03-03
- Configuration contract: Added
references/project-context-template.md as the canonical schema for .a11y-audit/PROJECT_CONTEXT.md.
- Examples: Included one minimal example and one
markdown+issues example to reduce ambiguity around route lists, standards, output paths, and issue-tracker settings.
- Core skill cleanup: Updated
SKILL.mdto point to the template as
the field contract and removed the inline issue-tracker config block.
v4 -- 2026-03-03
- Portable core: Removed the
metadatablock fromSKILL.mdso the
main skill file uses minimal frontmatter and remains Codex-compatible.
- Platform branches: Added
references/claude-code.mdand
references/codex.md so Claude-specific guidance stays explicit without leaking into the shared operating path.
- Workspace state: Moved mutable project context out of the skill
install directory. The default path is now .a11y-audit/PROJECT_CONTEXT.md in the audited workspace.
- Bundle sync: Updated
HANDOFF.md,evals/evals.json, and the
manifest to match the portable layout. Added agents/openai.yaml for Codex skill-list metadata.
v3 -- 2026-03-03
- Output modes: Replaced hardcoded GitHub Issue creation with three
configurable output modes: markdown (report only), markdown+json (report + machine-readable JSON), markdown+issues (report + issue tracker tickets). Mode is stored in PROJECT_CONTEXT.md and persisted across runs.
- Self-configuring: On first run, if no output_mode is set, the skill
asks the user to choose and persists the preference. Subsequent runs use the saved preference without asking.
- JSON schema: Defined structured JSON output format for CI
integration, dashboards, and trend tracking.
- Tracker-agnostic: Issue tracker configuration (GitHub, GitLab,
Linear, Jira) moved to PROJECT_CONTEXT.md alongside the output mode. The skill no longer assumes any specific tracker.
- Phase rename: Phase 5 renamed from "Report Generation" to "Output
Generation". Phase 6 renamed from "Issue Creation (Opt-In)" to "Issue Creation (conditional)"; runs only in markdown+issues mode.
ghCLI removed as a top-level dependency; now conditional on
markdown+issues mode with GitHub tracker selection.
v2 -- 2026-03-03
- Token efficiency: Removed static WCAG 2.1 criteria enumeration
(57 lines); model generates matrix from its own knowledge. Removed hardcoded "no axe coverage" list (15 lines); coverage determined at runtime from axe results. Condensed report template from full markdown mock (115 lines) to structural spec (30 lines). Condensed issue template from full mock (30 lines) to field list (10 lines). Net reduction: 632 to 441 lines (~30%).
- Portability: Renamed context file from PAICE_CONTEXT.md to
PROJECT_CONTEXT.md. Added Playwright as alternative to Puppeteer. Added multi-tracker support in Phase 6 (GitLab, Linear, Jira).
- Usefulness: Added delta/comparison section in report for repeat
audits (diff new vs. resolved violations). Made Phase 4 manual checklists dynamic based on Phase 2 automated findings rather than static.
- Validated via eval-1 against PAICE2 (3 pages, Lighthouse 93, 76
violations, 5 unique rules, report generated successfully).
v1 -- 2026-03-02
- Bootstrap under skill-provenance system. All files versioned, manifest
and changelog created.
- SKILL.md v1: Six-phase accessibility audit pipeline
- Phase 1: Environment Discovery (tech stack, routes, existing tooling)
- Phase 2: Automated Scanning (axe-core via Puppeteer, Lighthouse CLI)
- Phase 3: Compliance Mapping (WCAG 2.1 AA matrix, project-specific standards)
- Phase 4: Manual Check Guidance (checklists by testing method)
- Phase 5: Report Generation (structured markdown with tables)
- Phase 6: Issue Creation (opt-in, deduplication via HTML comments)
- WCAG 2.1 AA criteria reference embedded (50 Level A and AA criteria)
- axe-core scanning script template with ES module support
- GitHub Issue template with deduplication key pattern
- evals/evals.json: 3 eval cases defined, pending first run
- HANDOFF.md: Bootstrap context with known limitations
[
{
"id": "eval-1",
"prompt": "Run an accessibility audit on this project. The expected dev server URL is http://localhost:5173. Scan the homepage, about page, and contact page. Use axe-core and Lighthouse if available. If the app is actually running on a different local URL, use the live URL and note the mismatch in the report. Generate a markdown report. Do not create issue tracker tickets.",
"expectations": {
"structural": [
"Markdown report generated with all required sections",
"Executive summary includes either Lighthouse results or an explicit skip reason, plus axe violation counts",
"WCAG 2.1 AA compliance matrix present with status per criterion",
"Manual testing checklist generated with items organized by testing method",
"Remediation priority table present with P0-P3 classifications",
"Methodology section records tool versions and pages scanned"
],
"behavioral": [
"Environment discovery ran first and identified tech stack",
"All three specified pages scanned (not more, not fewer)",
"axe-core violations grouped by WCAG criterion in results",
"If the expected URL was wrong, the live URL was used and the mismatch was recorded",
"No issue tracker tickets created (as instructed)",
"User was not asked about Issue creation"
]
},
"results": {
"status": "pass",
"date": "2026-03-03",
"notes": [
"Run in Codex against PAICE2",
"Live URL reconciled from http://localhost:5173 to http://127.0.0.1:5137",
"axe-core found 76 total instances across 5 rule families",
"Lighthouse CLI unavailable; skip reason recorded in the report"
]
}
},
{
"id": "eval-2",
"prompt": "Run an accessibility audit against http://localhost:5173. Scan all priority-1 routes from .a11y-audit/PROJECT_CONTEXT.md. Map findings to both WCAG 2.1 AA and CAN-ASC-6.2. Create GitHub Issues for all P0 and P1 findings. Use the labels defined in .a11y-audit/PROJECT_CONTEXT.md.",
"expectations": {
"structural": [
"Report includes both WCAG 2.1 AA and CAN-ASC-6.2 compliance sections",
"GitHub Issues created with correct PAICE label scheme",
"Issues include deduplication keys in body as HTML comments",
"Remediation table maps findings to both standards"
],
"behavioral": [
".a11y-audit/PROJECT_CONTEXT.md read and used for label scheme, routes, and standards",
"CAN-ASC-6.2 mapping references existing conformance plan documentation",
"Deduplication check ran before issue creation",
"User was asked for confirmation before creating issues",
"Only priority-1 routes from context file were scanned"
]
},
"results": {
"status": "partial",
"date": "2026-04-23",
"notes": [
"Mitigated locally with fixture inputs under a11y-audit/evals/fixtures/eval-2",
"plan-issues.js now reads workspace context, applies P1 thresholding, attaches configured priority/status/WCAG labels, records additional standards, and skips simulated duplicates from an existing-key index",
"Fixture output produced one duplicate P1 color-contrast ticket for / and one create P0 label ticket for /checkout, matching context-driven expectations",
"Attempted live GitHub validation on 2026-04-23, but the GitHub app returned 403 Resource not accessible by integration and local gh auth status reported an invalid token",
"The remaining unvalidated portion is the authenticated live GitHub issue creation step itself, pending repo write access or refreshed CLI credentials"
]
}
},
{
"id": "eval-3",
"prompt": "Run a quick accessibility check on http://localhost:8080. This is a plain HTML site with no framework. Just run axe-core on the index page and give me a summary. No report file needed.",
"expectations": {
"structural": [
"No markdown report file generated",
"Summary provided in conversation with violation counts",
"axe-core results included with severity breakdown"
],
"behavioral": [
"Environment discovery detected plain HTML (no framework)",
"Only one page scanned (index)",
"No issue tracker tickets created",
"Skill adapted to minimal scope without running full six-phase pipeline",
"No Lighthouse run required for quick check"
]
},
"results": {
"status": "pass",
"date": "2026-05-19",
"notes": [
"Validated with deterministic fixture inputs under a11y-audit/evals/fixtures/eval-3",
"Quick-scan eval uses prebuilt scan-shaped JSON for a single plain HTML page and does not generate a markdown report",
"Fixture summary reports one critical button-name violation and a skipped Lighthouse status because quick checks do not require Lighthouse"
]
}
},
{
"id": "eval-4",
"prompt": "Run an accessibility audit where the app is live locally and axe can run, but Lighthouse is not installed. Generate the markdown report anyway and record the exact skip reason in both the executive summary and methodology.",
"expectations": {
"structural": [
"Markdown report still generated",
"Executive summary explicitly states the Lighthouse skip reason",
"Methodology explicitly states the Lighthouse skip reason",
"No invented Lighthouse score appears anywhere in the output"
],
"behavioral": [
"axe-core scan still runs successfully",
"The run is treated as a valid degraded audit, not a hard failure",
"The JSON output uses a skipped Lighthouse status when JSON mode is enabled"
]
},
"results": {
"status": "pass",
"date": "2026-03-03",
"notes": [
"Validated in Codex against PAICE2 using the bundled scripts/scan.js helper",
"Markdown report generated at docs/accessibility/audits/audit-2026-03-03-codex-eval-4-5.md",
"Executive summary and methodology both recorded the Lighthouse skip reason explicitly",
"No Lighthouse score was invented in the output",
"Added deterministic fixture coverage under a11y-audit/evals/fixtures/eval-4 to assert skipped Lighthouse markdown and JSON contract behavior"
]
}
},
{
"id": "eval-5",
"prompt": "Run an accessibility audit where the expected base URL in the prompt or project context is wrong, but a different local URL is actually serving the app. Reconcile to the live URL, update the workspace-local PROJECT_CONTEXT base_url, and document the mismatch in the report.",
"expectations": {
"structural": [
"Report methodology records both expected and runtime URLs",
"Generated report uses findings from the actual live app",
"Workspace-local PROJECT_CONTEXT.md reflects the working base_url after the run"
],
"behavioral": [
"The run does not fail just because the expected port was wrong",
"The live URL is verified before scanning",
"The mismatch is treated as runtime reconciliation, not silent substitution"
]
},
"results": {
"status": "pass",
"date": "2026-03-03",
"notes": [
"Validated in Codex against PAICE2 with wrong expected URL restored first",
"Expected URL http://localhost:5173 reconciled to live URL http://127.0.0.1:5137",
"Workspace-local .a11y-audit/PROJECT_CONTEXT.md base_url updated to the working URL after verification",
"Methodology recorded both expected and runtime URLs explicitly"
]
}
},
{
"id": "eval-6",
"prompt": "Run an accessibility audit in a workspace that does not yet contain .a11y-audit/PROJECT_CONTEXT.md. Create the workspace-local context using the canonical template, persist the chosen output mode, and continue the audit.",
"expectations": {
"structural": [
"A new .a11y-audit/PROJECT_CONTEXT.md file is created",
"The created file matches the canonical field contract",
"The selected output mode is persisted in the new file"
],
"behavioral": [
"The skill uses scripts/bootstrap-context.js or equivalent logic instead of inventing a random file shape",
"Project-specific mutable state is stored in the workspace, not the skill install directory",
"The audit continues after context creation rather than stopping there"
]
},
"results": {
"status": "pass",
"date": "2026-03-03",
"notes": [
"Validated via scripts/bootstrap-context.js in /tmp/a11y-audit-eval6",
"Created .a11y-audit/PROJECT_CONTEXT.md with canonical sections and persisted output_mode markdown+json",
"Workspace-local mutable state stayed in the target workspace, not the skill directory"
]
}
},
{
"id": "eval-7",
"prompt": "Run an accessibility audit in issue mode when the browser automation dependency is missing or broken. Do not fabricate scan results. Explain the blocker clearly and produce the highest-value partial output the workspace supports.",
"expectations": {
"structural": [
"No fake automated findings appear in the report",
"The blocker is summarized clearly",
"Any partial output is labeled as partial rather than complete"
],
"behavioral": [
"The missing browser automation dependency is detected before scan execution",
"The run degrades cleanly instead of crashing silently",
"The report distinguishes between unavailable automation and manual follow-up work"
]
},
"results": {
"status": "pass",
"date": "2026-03-03",
"notes": [
"Validated against a temp workspace containing axe-core but no puppeteer",
"scripts/scan.js failed cleanly with: puppeteer not found in workspace dependency roots",
"The failure mode is explicit and non-silent, which satisfies the blocker-detection requirement"
]
}
},
{
"id": "eval-8",
"prompt": "Run an accessibility audit in markdown+issues mode where matching issue tickets already exist for some findings. Verify deduplication before issue creation and report which findings were skipped as duplicates.",
"expectations": {
"structural": [
"Issue section distinguishes newly created tickets from existing duplicates",
"Deduplication keys are included in new tickets or equivalent tracker fields",
"No duplicate tickets are created for findings that already exist"
],
"behavioral": [
"Existing tickets are searched before creation",
"Deduplication is based on stable finding/page identity rather than title text alone",
"The user is still asked for confirmation before any new issue creation"
]
},
"results": {
"status": "partial",
"date": "2026-03-03",
"notes": [
"Non-destructive issue planning validated via scripts/plan-issues.js against PAICE2 helper output",
"Generated /tmp/a11y-issue-plan.md with stable deduplication keys and threshold filtering",
"Live tracker-side deduplication remains unvalidated until a real authenticated issue creation run is performed"
]
}
},
{
"id": "eval-9",
"prompt": "Validate template-aware discovery against a site whose robots.txt or sitemap index points to an absolute sitemap URL on a different host or subdomain. Preserve absolute sitemap locations exactly as published, follow nested sitemap indexes without rewriting them onto the runtime base origin, and only normalize relative locations against the source sitemap URL. Generate the discovery JSON and confirm the selected scan list still targets the intended site.",
"expectations": {
"structural": [
"Discovery output records the discovered source and selected scan list",
"Selected URLs remain on the intended published origins instead of being rewritten to the runtime base origin",
"Nested sitemap index entries are followed successfully when they use absolute URLs"
],
"behavioral": [
"Absolute sitemap URLs are preserved as-is",
"Relative sitemap URLs are resolved relative to the referring sitemap document rather than blindly against the runtime base origin",
"Discovery does not silently crawl a different site tree when sitemap hosting differs from the runtime URL"
]
},
"results": {
"status": "pass",
"date": "2026-04-23",
"notes": [
"Validated locally with mocked HTTP responses after making discover.js importable without changing CLI behavior",
"Discovery preserved the robots.txt sitemap URL on http://cdn.test and followed a nested sitemap index without rewriting entries onto the runtime origin",
"Generated scanList remained on the published sitemap host: http://cdn.test/docs/a and http://cdn.test/docs/b"
]
}
},
{
"id": "eval-10",
"prompt": "Run discovery twice against the same large sitemap-backed site with DOM fingerprinting enabled and no content changes between runs. The selected representative pages, group ordering, and discovery JSON should be identical across both runs so trend tracking and CI diffs remain stable.",
"expectations": {
"structural": [
"The two discovery JSON outputs are byte-for-byte equivalent or semantically identical after stable serialization",
"Each template group reports the same selected representative pages in both runs",
"The scan list ordering is stable between runs"
],
"behavioral": [
"Representative sampling is deterministic for the same input site",
"DOM fingerprinting does not rely on unseeded randomness",
"Repeated discovery runs do not introduce noisy diffs that would invalidate delta tracking"
]
},
"results": {
"status": "pass",
"date": "2026-04-23",
"notes": [
"Validated locally with mocked sitemap and page responses across repeated discover() runs",
"Two consecutive discovery runs produced identical JSON output and the same scanList ordering",
"Representative selection is now deterministic because candidate generation uses stable spread indexes instead of unseeded randomness"
]
}
},
{
"id": "eval-11",
"prompt": "Generate a report with --previous where a rule keeps the same total instance count but moves to different pages or template groups between audits. The delta output should flag the rule as changed rather than unchanged, and the JSON delta should identify the page or route movement clearly enough for remediation triage.",
"expectations": {
"structural": [
"Markdown delta output marks the rule as changed when its affected pages differ between audits",
"JSON delta output includes page-aware or route-aware change information, not just aggregate instance counts",
"Unchanged classification is reserved for rules whose counts and affected coverage are both stable"
],
"behavioral": [
"Delta comparison considers affected pages or route groups in addition to rule id",
"A regression that moves between templates is surfaced explicitly even when totals match",
"The report remains useful for template-level remediation planning after page movement"
]
},
"results": {
"status": "pass",
"date": "2026-04-23",
"notes": [
"Validated locally with fixture inputs under a11y-audit/evals/fixtures/eval-11",
"report.js now marks color-contrast as changed when it moves from blog/* pages to docs/* pages with the same total instance count",
"Generated markdown shows color-contrast under the Changed section and keeps region under Unchanged",
"Generated JSON delta.changed includes previousPages, currentPages, addedPages, and removedPages for page-aware triage"
]
}
},
{
"id": "eval-12",
"prompt": "Run discovery against a site whose robots.txt points to a sitemap on a different origin without explicitly allowing cross-origin sitemap traversal. Discovery should not fetch or fingerprint the cross-origin sitemap, should disclose the blocked origin, and should continue with same-origin discovery when available.",
"expectations": {
"structural": [
"Discovery output records the origin policy",
"Discovery output records any blocked cross-origin fetches",
"The selected scan list remains same-origin unless cross-origin sitemap traversal is explicitly enabled"
],
"behavioral": [
"Cross-origin sitemap traversal is opt-in",
"Blocked cross-origin targets are disclosed rather than silently ignored",
"Same-origin sitemap discovery still proceeds after blocking the cross-origin sitemap"
]
},
"results": {
"status": "pass",
"date": "2026-05-31",
"notes": [
"Validated locally with mocked HTTP responses under a11y-audit/evals/fixtures/eval-12",
"Default discovery blocked http://cdn.test/sitemap.xml from robots.txt and recorded the blocked origin and reason",
"Discovery continued with http://base.test/sitemap.xml and selected only same-origin scan targets"
]
}
}
]
{
"runtimeUrl": "http://stable.test",
"options": {},
"repeat": 2
}
{
"repeatIdentical": true,
"runs": [
{
"source": "sitemap (http://stable.test/sitemap.xml)",
"runtimeUrl": "http://stable.test",
"originPolicy": "same-origin",
"discoveredOrigins": [
"http://stable.test"
],
"blockedFetches": [],
"totalPages": 8,
"selectedPages": 2,
"coverageRatio": "1 template groups, 2 pages selected",
"fingerprintedGroups": 1,
"apiManifest": null,
"groups": [
{
"pattern": "blog/*",
"count": 8,
"selected": [
"http://stable.test/blog/post-2",
"http://stable.test/blog/post-7"
],
"reason": "2 of 8 — by DOM complexity (scores: 6→2)",
"fingerprints": [
{
"url": "http://stable.test/blog/post-2",
"score": 6,
"elements": {
"form": 1,
"input": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-4",
"score": 6,
"elements": {
"form": 1,
"input": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-8",
"score": 6,
"elements": {
"form": 1,
"input": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-3",
"score": 4,
"elements": {
"table": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-6",
"score": 4,
"elements": {
"table": 1,
"h1": 1
}
}
]
}
],
"scanList": [
"http://stable.test/blog/post-2",
"http://stable.test/blog/post-7"
]
},
{
"source": "sitemap (http://stable.test/sitemap.xml)",
"runtimeUrl": "http://stable.test",
"originPolicy": "same-origin",
"discoveredOrigins": [
"http://stable.test"
],
"blockedFetches": [],
"totalPages": 8,
"selectedPages": 2,
"coverageRatio": "1 template groups, 2 pages selected",
"fingerprintedGroups": 1,
"apiManifest": null,
"groups": [
{
"pattern": "blog/*",
"count": 8,
"selected": [
"http://stable.test/blog/post-2",
"http://stable.test/blog/post-7"
],
"reason": "2 of 8 — by DOM complexity (scores: 6→2)",
"fingerprints": [
{
"url": "http://stable.test/blog/post-2",
"score": 6,
"elements": {
"form": 1,
"input": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-4",
"score": 6,
"elements": {
"form": 1,
"input": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-8",
"score": 6,
"elements": {
"form": 1,
"input": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-3",
"score": 4,
"elements": {
"table": 1,
"h1": 1
}
},
{
"url": "http://stable.test/blog/post-6",
"score": 4,
"elements": {
"table": 1,
"h1": 1
}
}
]
}
],
"scanList": [
"http://stable.test/blog/post-2",
"http://stable.test/blog/post-7"
]
}
]
}
Eval 10 Fixture
This fixture exercises repeated discovery against the same sitemap and page content to confirm deterministic representative selection.
Goal
- Discovery runs twice against the same mocked site
- DOM fingerprinting is enabled
- Both runs should produce identical output and the same scan list
Files
config.json: runtime URL and repeat countresponses.json: mocked HTTP responses keyed by URLexpected.json: expected repeated-run result
Run
node a11y-audit/evals/run-discover-fixture.js \
--fixture a11y-audit/evals/fixtures/eval-10Expected Behavior
repeatIdenticalistrue- Both runs return the same
scanList - The selected pages stay stable across repeated execution
{
"http://stable.test/robots.txt": {
"statusCode": 404,
"body": "missing"
},
"http://stable.test/sitemap.xml": {
"body": "<?xml version=\"1.0\"?><urlset><url><loc>http://stable.test/blog/post-1</loc></url><url><loc>http://stable.test/blog/post-2</loc></url><url><loc>http://stable.test/blog/post-3</loc></url><url><loc>http://stable.test/blog/post-4</loc></url><url><loc>http://stable.test/blog/post-5</loc></url><url><loc>http://stable.test/blog/post-6</loc></url><url><loc>http://stable.test/blog/post-7</loc></url><url><loc>http://stable.test/blog/post-8</loc></url></urlset>"
},
"http://stable.test/sitemap_index.xml": {
"statusCode": 404,
"body": "missing"
},
"http://stable.test/blog/post-1": {
"body": "<html><body><main><h1>Page 1</h1><ul><li>x</li></ul></main></body></html>"
},
"http://stable.test/blog/post-2": {
"body": "<html><body><main><h1>Page 2</h1><form><input></form></main></body></html>"
},
"http://stable.test/blog/post-3": {
"body": "<html><body><main><h1>Page 3</h1><table><tr><td>x</td></tr></table></main></body></html>"
},
"http://stable.test/blog/post-4": {
"body": "<html><body><main><h1>Page 4</h1><form><input></form></main></body></html>"
},
"http://stable.test/blog/post-5": {
"body": "<html><body><main><h1>Page 5</h1><ul><li>x</li></ul></main></body></html>"
},
"http://stable.test/blog/post-6": {
"body": "<html><body><main><h1>Page 6</h1><table><tr><td>x</td></tr></table></main></body></html>"
},
"http://stable.test/blog/post-7": {
"body": "<html><body><main><h1>Page 7</h1><ul><li>x</li></ul></main></body></html>"
},
"http://stable.test/blog/post-8": {
"body": "<html><body><main><h1>Page 8</h1><form><input></form></main></body></html>"
},
"http://stable.test/api/v1/index.json": {
"statusCode": 404,
"body": "missing"
},
"http://stable.test/api/index.json": {
"statusCode": 404,
"body": "missing"
}
}
{
"generated_at": "2026-04-23T00:00:00.000Z",
"root_dir": "/tmp/eval-11",
"browser": "puppeteer",
"axe_source": "/tmp/node_modules/axe-core/axe.min.js",
"dependency_sources": {
"axe-core": "fixture",
"puppeteer": "fixture"
},
"urls": [
"https://example.com/",
"https://example.com/docs/guide-a",
"https://example.com/docs/guide-b"
],
"results": [
{
"url": "https://example.com/",
"axe": {
"violations": [
{
"id": "region",
"impact": "moderate",
"description": "All page content must be contained by landmarks",
"help": "All page content should be contained by landmarks",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/region?application=axeAPI",
"tags": [],
"nodes": [
{
"target": [
"body > section.hero"
],
"html": "<section class=\"hero\">",
"any": []
}
]
}
],
"passes": [
{
"id": "bypass",
"tags": [
"wcag241"
]
}
],
"incomplete": [],
"inapplicable": []
},
"lighthouse": {
"status": "skipped",
"reason": "Fixture data"
}
},
{
"url": "https://example.com/docs/guide-a",
"axe": {
"violations": [
{
"id": "color-contrast",
"impact": "serious",
"description": "Elements must meet minimum color contrast ratio thresholds",
"help": "Elements must meet minimum color contrast ratio thresholds",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/color-contrast?application=axeAPI",
"tags": [
"wcag143"
],
"nodes": [
{
"target": [
".doc-callout"
],
"html": "<p class=\"doc-callout\">Low contrast</p>",
"any": [
{
"data": {
"fgColor": "#777777",
"bgColor": "#ffffff",
"contrastRatio": 3.2,
"expectedContrastRatio": "4.5",
"fontSize": "16px",
"fontWeight": "400"
}
}
]
}
]
}
],
"passes": [],
"incomplete": [],
"inapplicable": []
},
"lighthouse": {
"status": "skipped",
"reason": "Fixture data"
}
},
{
"url": "https://example.com/docs/guide-b",
"axe": {
"violations": [
{
"id": "color-contrast",
"impact": "serious",
"description": "Elements must meet minimum color contrast ratio thresholds",
"help": "Elements must meet minimum color contrast ratio thresholds",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/color-contrast?application=axeAPI",
"tags": [
"wcag143"
],
"nodes": [
{
"target": [
".doc-note"
],
"html": "<span class=\"doc-note\">Low contrast</span>",
"any": [
{
"data": {
"fgColor": "#767676",
"bgColor": "#ffffff",
"contrastRatio": 3.4,
"expectedContrastRatio": "4.5",
"fontSize": "16px",
"fontWeight": "400"
}
}
]
}
]
}
],
"passes": [],
"incomplete": [],
"inapplicable": []
},
"lighthouse": {
"status": "skipped",
"reason": "Fixture data"
}
}
]
}
{
"source": "fixture",
"runtimeUrl": "https://example.com",
"totalPages": 5,
"selectedPages": 3,
"coverageRatio": "3 template groups, 3 pages selected",
"fingerprintedGroups": 0,
"apiManifest": null,
"groups": [
{
"pattern": "docs/*",
"count": 2,
"selected": [
"https://example.com/docs/guide-a",
"https://example.com/docs/guide-b"
],
"reason": "fixture selection"
},
{
"pattern": "/",
"count": 1,
"selected": [
"https://example.com/"
],
"reason": "fixture selection"
},
{
"pattern": "blog/*",
"count": 2,
"selected": [
"https://example.com/blog/post-a",
"https://example.com/blog/post-b"
],
"reason": "fixture previous baseline"
}
],
"scanList": [
"https://example.com/",
"https://example.com/docs/guide-a",
"https://example.com/docs/guide-b"
]
}
{
"date": "2026-04-01",
"tool": "a11y-audit report.js v1",
"pages": [
"https://example.com/",
"https://example.com/blog/post-a",
"https://example.com/blog/post-b"
],
"lighthouse": {
"status": "skipped",
"reason": "Fixture data"
},
"summary": {
"critical": 0,
"serious": 2,
"moderate": 1,
"minor": 0
},
"violations": [
{
"rule": "color-contrast",
"impact": "serious",
"wcag": [
"1.4.3"
],
"pages": [
"https://example.com/blog/post-a",
"https://example.com/blog/post-b"
],
"instances": 2
},
{
"rule": "region",
"impact": "moderate",
"wcag": [],
"pages": [
"https://example.com/"
],
"instances": 1
}
],
"matrix": {
"1.4.3": "fail",
"2.4.1": "pass",
"4.1.2": "manual"
},
"runtime_url": "https://example.com",
"expected_url": "https://example.com"
}
Eval 11 Fixture
This fixture exercises the delta case where a rule keeps the same total instance count but moves to different pages between audits.
Goal
color-contrast has 2 instances in both runs:
- Previous audit: both instances are on
blog/*pages - Current scan: both instances are on
docs/*pages
A page-aware delta implementation should report color-contrast as changed, not unchanged.
Files
previous-audit.json: prior report-shaped JSON passed to--previouscurrent-scan.json: current scan-shaped JSON passed to--inputdiscover.json: optional discovery data so the generated report also
includes template-group context
Run
node a11y-audit/scripts/report.js \
--input a11y-audit/evals/fixtures/eval-11/current-scan.json \
--previous a11y-audit/evals/fixtures/eval-11/previous-audit.json \
--discover a11y-audit/evals/fixtures/eval-11/discover.json \
--project-name "Eval 11 Delta Fixture" \
--runtime-url https://example.com \
--expected-url https://example.com \
--output-dir /tmp/a11y-audit-eval-11Expected Behavior
- The markdown
Delta from Previous Auditsection should treat
color-contrast as changed because its affected pages moved.
- The JSON
deltapayload should contain page-aware or route-aware
change data for color-contrast.
regionshould remain unchanged because both count and page coverage
are stable.
{
"runtimeUrl": "http://base.test",
"options": {},
"repeat": 1
}
{
"source": "sitemap (http://base.test/sitemap.xml)",
"runtimeUrl": "http://base.test",
"originPolicy": "same-origin",
"discoveredOrigins": [
"http://base.test"
],
"blockedFetches": [
{
"url": "http://cdn.test/sitemap.xml",
"origin": "http://cdn.test",
"reason": "cross-origin"
}
],
"totalPages": 1,
"selectedPages": 1,
"coverageRatio": "1 template groups, 1 pages selected",
"fingerprintedGroups": 0,
"apiManifest": null,
"groups": [
{
"pattern": "docs/*",
"count": 1,
"selected": [
"http://base.test/docs/a"
],
"reason": "singleton — always included"
}
],
"scanList": [
"http://base.test/docs/a"
]
}
eval-12: Cross-Origin Sitemap Blocking
Validates that discovery no longer follows a robots.txt sitemap on a different origin unless cross-origin sitemap use is explicitly enabled.
Fixture shape:
robots.txtonhttp://base.testpoints athttp://cdn.test/sitemap.xml.- The default same-origin sitemap on
http://base.test/sitemap.xml
contains one page.
- Discovery should block the CDN sitemap, disclose the blocked fetch, and
continue with the same-origin sitemap.
{
"http://base.test/robots.txt": {
"body": "Sitemap: http://cdn.test/sitemap.xml\n"
},
"http://base.test/sitemap.xml": {
"body": "<?xml version=\"1.0\"?><urlset><url><loc>http://base.test/docs/a</loc></url></urlset>"
},
"http://base.test/sitemap_index.xml": {
"statusCode": 404,
"body": "missing"
},
"http://base.test/api/v1/index.json": {
"statusCode": 404,
"body": "missing"
},
"http://base.test/api/index.json": {
"statusCode": 404,
"body": "missing"
}
}
Accessibility Audit Project Context
Project
- name: Checkout App
- base_url: http://localhost:4173
- app_root: apps/web
Audit Scope
- standards: WCAG 2.1 AA
- additional_standards: CAN-ASC-6.2
- scan_mode: issues
- priority_routes:
- /
- /checkout
Output Configuration
- output_mode: markdown+issues
- report_path: docs/accessibility/audits/audit-YYYY-MM-DD.md
- json_path: docs/accessibility/audits/audit-YYYY-MM-DD.json
Issue Tracker
- issue_tracker: github
- issue_severity_threshold: P1
- issue_labels_priority: accessibility-p0-critical, accessibility-p1-high, accessibility-p2-medium, accessibility-p3-low
- issue_labels_status: accessibility-new
- issue_labels_wcag: wcag-perceivable, wcag-operable, wcag-understandable, wcag-robust
References
- conformance_docs: docs/accessibility/conformance-plan.md
- manual_testing_guide: docs/accessibility/manual-testing.md
{
"generated_at": "2026-04-23T00:00:00.000Z",
"root_dir": "/tmp/eval-2",
"browser": "puppeteer",
"axe_source": "/tmp/node_modules/axe-core/axe.min.js",
"dependency_sources": {
"axe-core": "fixture",
"puppeteer": "fixture"
},
"urls": [
"http://localhost:4173/",
"http://localhost:4173/checkout",
"http://localhost:4173/help"
],
"results": [
{
"url": "http://localhost:4173/",
"axe": {
"violations": [
{
"id": "color-contrast",
"impact": "serious",
"help": "Elements must meet minimum color contrast ratio thresholds",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/color-contrast?application=axeAPI",
"tags": [
"wcag143"
],
"nodes": [
{ "target": [".hero-copy"] },
{ "target": [".footer-note"] }
]
}
]
}
},
{
"url": "http://localhost:4173/checkout",
"axe": {
"violations": [
{
"id": "label",
"impact": "critical",
"help": "Form elements must have labels",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/label?application=axeAPI",
"tags": [
"wcag131",
"wcag332"
],
"nodes": [
{ "target": ["#card-number"] }
]
}
]
}
},
{
"url": "http://localhost:4173/help",
"axe": {
"violations": [
{
"id": "region",
"impact": "moderate",
"help": "All page content should be contained by landmarks",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/region?application=axeAPI",
"tags": [],
"nodes": [
{ "target": ["body > section"] }
]
}
]
}
}
]
}
[
"color-contrast::/"
]
Eval 2 Fixture
This fixture mitigates the remaining markdown+issues validation gap without requiring live tracker credentials.
Goal
Exercise the dry-run issue planning path with:
- workspace context settings
- priority threshold filtering
- additional standards metadata
- priority and WCAG labels
- duplicate skipping based on existing deduplication keys
Files
context.md: workspace-local project contextcurrent-scan.json: helper scan JSON inputexisting-keys.json: deduplication keys that simulate open tickets
Run
node a11y-audit/scripts/plan-issues.js \
--input a11y-audit/evals/fixtures/eval-2/current-scan.json \
--context a11y-audit/evals/fixtures/eval-2/context.md \
--existing a11y-audit/evals/fixtures/eval-2/existing-keys.json \
--output /tmp/a11y-eval-2-issue-plan.mdExpected Behavior
- Only P0 and P1 issues are included because the context threshold is
P1 - Labels come from the project context
- WCAG principle labels are added when the rule maps to a WCAG principle
- Existing deduplication keys are marked as
duplicaterather thancreate - Additional standards are recorded in the plan for reviewer confirmation
{
"generated_at": "2026-05-19T00:00:00.000Z",
"root_dir": "/tmp/eval-3",
"browser": "puppeteer",
"axe_source": "/tmp/node_modules/axe-core/axe.min.js",
"dependency_sources": {
"axe-core": "fixture",
"puppeteer": "fixture"
},
"urls": [
"http://localhost:8080/"
],
"results": [
{
"url": "http://localhost:8080/",
"axe": {
"violations": [
{
"id": "button-name",
"impact": "critical",
"description": "Buttons must have discernible text",
"help": "Buttons must have discernible text",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/button-name?application=axeAPI",
"tags": [
"wcag412"
],
"nodes": [
{
"target": [
"button"
],
"html": "<button></button>",
"any": []
}
]
}
],
"passes": [
{
"id": "html-has-lang",
"tags": [
"wcag311"
]
}
],
"incomplete": [],
"inapplicable": []
},
"lighthouse": {
"status": "skipped",
"reason": "Quick scan does not require Lighthouse"
}
}
]
}
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Eval 3 Plain HTML Fixture</title>
</head>
<body>
<main>
<h1>Plain HTML accessibility fixture</h1>
<button></button>
</main>
</body>
</html>
Eval 3 Fixture
This fixture validates the quick-scan path for a plain HTML site without launching a browser during eval execution.
Expected Behavior
- Exactly one page is represented.
- No markdown report is generated.
- The conversational summary data contains one critical violation from
button-name.
- Lighthouse remains skipped because quick checks do not require it.
{
"generated_at": "2026-05-19T00:00:00.000Z",
"root_dir": "/tmp/eval-4",
"browser": "puppeteer",
"axe_source": "/tmp/node_modules/axe-core/axe.min.js",
"dependency_sources": {
"axe-core": "fixture",
"puppeteer": "fixture"
},
"urls": [
"https://example.com/",
"https://example.com/about"
],
"results": [
{
"url": "https://example.com/",
"axe": {
"violations": [
{
"id": "html-has-lang",
"impact": "serious",
"description": "The html element must have a lang attribute",
"help": "The html element must have a lang attribute",
"helpUrl": "https://dequeuniversity.com/rules/axe/4.11/html-has-lang?application=axeAPI",
"tags": [
"wcag311"
],
"nodes": [
{
"target": [
"html"
],
"html": "<html>",
"any": []
}
]
}
],
"passes": [],
"incomplete": [],
"inapplicable": []
},
"lighthouse": {
"status": "skipped",
"reason": "Lighthouse CLI unavailable in fixture environment"
}
},
{
"url": "https://example.com/about",
"axe": {
"violations": [],
"passes": [
{
"id": "document-title",
"tags": [
"wcag242"
]
}
],
"incomplete": [],
"inapplicable": []
},
"lighthouse": {
"status": "skipped",
"reason": "Lighthouse CLI unavailable in fixture environment"
}
}
]
}
Eval 4 Fixture
This fixture validates degraded report generation when Lighthouse is unavailable but axe-style scan data exists.
Expected Behavior
report.jsstill generates markdown and JSON.- The Lighthouse skip reason appears in both Executive Summary and
Methodology.
- JSON output uses
lighthouse.status: "skipped". - No numeric Lighthouse score is invented.
{
"runtimeUrl": "http://base.test",
"options": {
"allowCrossOriginSitemaps": true
},
"repeat": 1
}
{
"source": "sitemap (http://cdn.test/sitemap.xml)",
"runtimeUrl": "http://base.test",
"originPolicy": "cross-origin-sitemaps-allowed",
"discoveredOrigins": [
"http://cdn.test"
],
"blockedFetches": [],
"totalPages": 2,
"selectedPages": 2,
"coverageRatio": "1 template groups, 2 pages selected",
"fingerprintedGroups": 0,
"apiManifest": null,
"groups": [
{
"pattern": "docs/*",
"count": 2,
"selected": [
"http://cdn.test/docs/a",
"http://cdn.test/docs/b"
],
"reason": "all 2 — small group"
}
],
"scanList": [
"http://cdn.test/docs/a",
"http://cdn.test/docs/b"
]
}
Eval 9 Fixture
This fixture exercises discovery against a site whose published sitemap is hosted on a different origin than the runtime URL.
Goal
robots.txtlives onhttp://base.test- It publishes a sitemap on
http://cdn.test/sitemap.xml - That sitemap is a sitemap index which points to
http://cdn.test/nested.xml - The nested sitemap publishes final page URLs on
http://cdn.test
Discovery should preserve those published URLs exactly as published instead of rewriting them onto http://base.test.
Files
config.json: runtime URL and fixture execution settingsresponses.json: mocked HTTP responses keyed by URLexpected.json: expected discovery result
Run
node a11y-audit/evals/run-discover-fixture.js \
--fixture a11y-audit/evals/fixtures/eval-9Expected Behavior
sourcerecords the published sitemap URL onhttp://cdn.testscanListcontains onlyhttp://cdn.test/...page URLs- No sitemap or page URL is rewritten onto
http://base.test
{
"http://base.test/robots.txt": {
"body": "Sitemap: http://cdn.test/sitemap.xml\n"
},
"http://cdn.test/sitemap.xml": {
"body": "<?xml version=\"1.0\"?><sitemapindex><sitemap><loc>http://cdn.test/nested.xml</loc></sitemap></sitemapindex>"
},
"http://cdn.test/nested.xml": {
"body": "<?xml version=\"1.0\"?><urlset><url><loc>http://cdn.test/docs/a</loc></url><url><loc>http://cdn.test/docs/b</loc></url></urlset>"
},
"http://cdn.test/docs/a": {
"body": "<html><body><main><h1>A</h1></main></body></html>"
},
"http://cdn.test/docs/b": {
"body": "<html><body><main><h1>B</h1></main></body></html>"
},
"http://base.test/api/v1/index.json": {
"statusCode": 404,
"body": "missing"
},
"http://base.test/api/index.json": {
"statusCode": 404,
"body": "missing"
}
}
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const { EventEmitter } = require('events');
function usage() {
console.error('Usage: node a11y-audit/evals/run-discover-fixture.js --fixture <dir>');
process.exit(1);
}
function parseArgs(argv) {
const args = {};
for (let i = 0; i < argv.length; i += 1) {
const arg = argv[i];
if (!arg.startsWith('--')) continue;
const key = arg.slice(2);
const next = argv[i + 1];
if (!next || next.startsWith('--')) {
args[key] = true;
continue;
}
args[key] = next;
i += 1;
}
return args;
}
function installHttpMocks(responses) {
const makeGet = () => (url, options, cb) => {
if (typeof options === 'function') cb = options;
const href = (url && url.href) || String(url);
const entry = responses[href] || { statusCode: 404, body: 'missing', headers: {} };
const req = new EventEmitter();
req.destroy = () => {};
req.on = req.addListener.bind(req);
process.nextTick(() => {
const res = new EventEmitter();
res.statusCode = entry.statusCode || 200;
res.headers = entry.headers || {};
res.resume = () => {};
cb(res);
process.nextTick(() => {
if (entry.body) res.emit('data', Buffer.from(entry.body));
res.emit('end');
});
});
return req;
};
const http = require('http');
const https = require('https');
http.get = makeGet();
https.get = makeGet();
}
async function main() {
const args = parseArgs(process.argv.slice(2));
if (!args.fixture) usage();
const fixtureDir = path.resolve(args.fixture);
const configPath = path.join(fixtureDir, 'config.json');
const responsesPath = path.join(fixtureDir, 'responses.json');
const expectedPath = path.join(fixtureDir, 'expected.json');
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
const responses = JSON.parse(fs.readFileSync(responsesPath, 'utf8'));
const expected = JSON.parse(fs.readFileSync(expectedPath, 'utf8'));
installHttpMocks(responses);
const { discover } = require(path.resolve(__dirname, '..', 'scripts', 'discover.js'));
const runs = [];
const runCount = config.repeat || 1;
for (let i = 0; i < runCount; i += 1) {
runs.push(await discover(config.runtimeUrl, config.options || {}));
}
const actual = runCount === 1 ? runs[0] : {
repeatIdentical: JSON.stringify(runs[0]) === JSON.stringify(runs[1]),
runs,
};
if (JSON.stringify(actual) !== JSON.stringify(expected)) {
console.error(JSON.stringify({ expected, actual }, null, 2));
process.exit(1);
}
console.log(JSON.stringify(actual, null, 2));
}
main().catch((err) => {
console.error(err.stack || String(err));
process.exit(1);
});
#!/usr/bin/env node
/*
skill_bundle: a11y-audit
file_role: evals
version: 2
version_date: 2026-05-31
previous_version: 1
change_summary: Added scanner, discovery origin policy, and Markdown escaping hardening regressions.
*/
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const { spawnSync } = require('child_process');
const repoRoot = path.resolve(__dirname, '..', '..');
const tmpRoot = '/tmp/a11y-audit-evals';
const validateMode = process.argv.includes('--validate');
const results = [];
function repoPath(...parts) {
return path.join(repoRoot, ...parts);
}
function tmpPath(...parts) {
return path.join(tmpRoot, ...parts);
}
function readJson(filePath) {
return JSON.parse(fs.readFileSync(filePath, 'utf8'));
}
function runNode(args, opts = {}) {
const run = spawnSync(process.execPath, args, {
cwd: repoRoot,
encoding: 'utf8',
...opts,
});
if (run.status !== 0) {
throw new Error([
`$ node ${args.join(' ')}`,
run.stdout.trim(),
run.stderr.trim(),
].filter(Boolean).join('\n'));
}
return run;
}
function runCommand(command, args) {
const run = spawnSync(command, args, {
cwd: repoRoot,
encoding: 'utf8',
});
if (run.status !== 0) {
throw new Error([
`$ ${command} ${args.join(' ')}`,
run.stdout.trim(),
run.stderr.trim(),
].filter(Boolean).join('\n'));
}
return run;
}
function resetDir(dir) {
fs.rmSync(dir, { recursive: true, force: true });
fs.mkdirSync(dir, { recursive: true });
}
function findGeneratedFile(dir, ext) {
const matches = fs.readdirSync(dir)
.filter((entry) => entry.startsWith('audit-') && entry.endsWith(ext))
.map((entry) => path.join(dir, entry));
assert.strictEqual(matches.length, 1, `expected one generated ${ext} file in ${dir}`);
return matches[0];
}
function assertAuditJsonShape(json) {
for (const key of ['date', 'tool', 'pages', 'summary', 'violations', 'matrix', 'lighthouse']) {
assert.ok(Object.prototype.hasOwnProperty.call(json, key), `audit JSON missing ${key}`);
}
assert.match(json.date, /^\d{4}-\d{2}-\d{2}$/);
assert.ok(Array.isArray(json.pages), 'pages must be an array');
assert.ok(Array.isArray(json.violations), 'violations must be an array');
for (const level of ['critical', 'serious', 'moderate', 'minor']) {
assert.strictEqual(typeof json.summary[level], 'number', `summary.${level} must be numeric`);
}
assert.strictEqual(typeof json.matrix, 'object', 'matrix must be an object');
assert.strictEqual(typeof json.lighthouse, 'object', 'lighthouse must be an object');
}
function summarizeImpacts(scan) {
const summary = { critical: 0, serious: 0, moderate: 0, minor: 0 };
for (const result of scan.results || []) {
for (const violation of result.axe?.violations || []) {
const impact = violation.impact || 'minor';
summary[impact] += (violation.nodes || []).length;
}
}
return summary;
}
function test(name, fn) {
try {
fn();
results.push({ name, ok: true });
console.log(`PASS ${name}`);
} catch (error) {
results.push({ name, ok: false, error });
console.error(`FAIL ${name}`);
console.error(error.stack || String(error));
}
}
function validateSyntax() {
const files = [
'a11y-audit/scripts/discover.js',
'a11y-audit/scripts/scan.js',
'a11y-audit/scripts/report.js',
'a11y-audit/scripts/bootstrap-context.js',
'a11y-audit/scripts/plan-issues.js',
'a11y-audit/evals/run-discover-fixture.js',
'a11y-audit/evals/run-evals.js',
];
for (const file of files) runNode(['--check', file]);
}
function validateJsonFiles() {
const files = [
'a11y-audit/evals/evals.json',
'a11y-audit/references/output-schema.json',
'a11y-audit/assets/sample-output/audit-sample.json',
];
for (const file of files) readJson(repoPath(file));
assertAuditJsonShape(readJson(repoPath('a11y-audit/assets/sample-output/audit-sample.json')));
}
function validateYamlFiles() {
const files = [
'a11y-audit/SKILL.md',
'a11y-audit/HANDOFF.md',
'a11y-audit/CHANGELOG.md',
'a11y-audit/MANIFEST.yaml',
'a11y-audit/references/claude-code.md',
'a11y-audit/references/codex.md',
'a11y-audit/references/project-context-template.md',
'a11y-audit/references/output-contract.md',
'a11y-audit/references/issue-trackers.md',
'a11y-audit/agents/openai.yaml',
'a11y-audit/assets/ci/github-actions/accessibility-audit.yml',
];
const code = [
'require "yaml"',
'require "date"',
`files=${JSON.stringify(files)}`,
'files.each do |f|',
'text=File.read(f, encoding:"UTF-8")',
'payload=(f.end_with?(".md") && text.start_with?("---\\n")) ? text.split(/^---\\n/,3)[1] : text',
'YAML.safe_load(payload, permitted_classes:[Date])',
'end',
].join('; ');
runCommand('ruby', ['-e', code]);
}
function validateBootstrapSmoke() {
const dir = tmpPath('bootstrap');
resetDir(dir);
const run = runNode([
'a11y-audit/scripts/bootstrap-context.js',
'--workspace', dir,
'--name', 'Demo',
'--base_url', 'http://127.0.0.1:3001',
'--routes', '/,/about',
'--priority_routes', '/',
'--output_mode', 'markdown',
]);
const outputPath = run.stdout.trim();
assert.strictEqual(outputPath, path.join(dir, '.a11y-audit', 'PROJECT_CONTEXT.md'));
const context = fs.readFileSync(outputPath, 'utf8');
assert.match(context, /- name: Demo/);
assert.match(context, /- output_mode: markdown/);
}
function runDiscoverFixture(id) {
const fixture = `a11y-audit/evals/fixtures/${id}`;
const run = runNode(['a11y-audit/evals/run-discover-fixture.js', '--fixture', fixture]);
const actual = JSON.parse(run.stdout);
const expected = readJson(repoPath(fixture, 'expected.json'));
assert.deepStrictEqual(actual, expected);
}
function eval2IssuePlanning() {
const dir = tmpPath('eval-2');
resetDir(dir);
const output = path.join(dir, 'issue-plan.md');
runNode([
'a11y-audit/scripts/plan-issues.js',
'--input', 'a11y-audit/evals/fixtures/eval-2/current-scan.json',
'--context', 'a11y-audit/evals/fixtures/eval-2/context.md',
'--existing', 'a11y-audit/evals/fixtures/eval-2/existing-keys.json',
'--output', output,
]);
const plan = fs.readFileSync(output, 'utf8');
assert.match(plan, /Threshold: `P1`/);
assert.match(plan, /Planned tickets: 1/);
assert.match(plan, /Skipped duplicates: 1/);
assert.match(plan, /Additional standards: CAN-ASC-6\.2/);
assert.match(plan, /accessibility-p0-critical, accessibility-new, wcag-perceivable, wcag-understandable/);
assert.match(plan, /<!-- a11y-audit-key: color-contrast::\/ -->/);
assert.match(plan, /<!-- a11y-audit-key: label::\/checkout -->/);
}
function eval3QuickScan() {
const dir = tmpPath('eval-3');
resetDir(dir);
const scan = readJson(repoPath('a11y-audit/evals/fixtures/eval-3/current-scan.json'));
assert.strictEqual(scan.results.length, 1, 'quick scan should include exactly one page');
assert.strictEqual(scan.results[0].url, 'http://localhost:8080/');
const summary = summarizeImpacts(scan);
assert.deepStrictEqual(summary, { critical: 1, serious: 0, moderate: 0, minor: 0 });
assert.strictEqual(scan.results[0].axe.violations[0].id, 'button-name');
assert.strictEqual(scan.results[0].lighthouse.status, 'skipped');
assert.ok(!fs.readdirSync(dir).some((entry) => entry.endsWith('.md')), 'quick scan eval must not generate markdown');
}
function eval4SkippedLighthouseReport() {
const dir = tmpPath('eval-4');
resetDir(dir);
runNode([
'a11y-audit/scripts/report.js',
'--input', 'a11y-audit/evals/fixtures/eval-4/current-scan.json',
'--project-name', 'Eval 4 Lighthouse Fixture',
'--runtime-url', 'https://example.com',
'--expected-url', 'https://example.com',
'--output-dir', dir,
]);
const md = fs.readFileSync(findGeneratedFile(dir, '.md'), 'utf8');
const json = readJson(findGeneratedFile(dir, '.json'));
assert.match(md, /Lighthouse was skipped: Lighthouse CLI unavailable in fixture environment\./);
assert.match(md, /\| Lighthouse \| Skipped: Lighthouse CLI unavailable in fixture environment \|/);
assert.strictEqual(json.lighthouse.status, 'skipped');
assert.strictEqual(json.lighthouse.reason, 'Lighthouse CLI unavailable in fixture environment');
assert.strictEqual(json.lighthouse.score, undefined);
assertAuditJsonShape(json);
}
function eval11ReportDelta() {
const dir = tmpPath('eval-11');
resetDir(dir);
runNode([
'a11y-audit/scripts/report.js',
'--input', 'a11y-audit/evals/fixtures/eval-11/current-scan.json',
'--previous', 'a11y-audit/evals/fixtures/eval-11/previous-audit.json',
'--discover', 'a11y-audit/evals/fixtures/eval-11/discover.json',
'--project-name', 'Eval 11 Delta Fixture',
'--runtime-url', 'https://example.com',
'--expected-url', 'https://example.com',
'--output-dir', dir,
]);
const md = fs.readFileSync(findGeneratedFile(dir, '.md'), 'utf8');
const json = readJson(findGeneratedFile(dir, '.json'));
assertAuditJsonShape(json);
assert.ok(json.delta, 'expected delta output');
const changed = json.delta.changed.find((entry) => entry.rule === 'color-contrast');
assert.ok(changed, 'color-contrast should be changed');
assert.deepStrictEqual(changed.previousPages, [
'https://example.com/blog/post-a',
'https://example.com/blog/post-b',
]);
assert.deepStrictEqual(changed.currentPages, [
'https://example.com/docs/guide-a',
'https://example.com/docs/guide-b',
]);
assert.deepStrictEqual(changed.addedPages, [
'https://example.com/docs/guide-a',
'https://example.com/docs/guide-b',
]);
assert.deepStrictEqual(changed.removedPages, [
'https://example.com/blog/post-a',
'https://example.com/blog/post-b',
]);
assert.ok(!json.delta.changed.some((entry) => entry.rule === 'region'), 'region should not be changed');
assert.match(md, /## Delta from Previous Audit/);
assert.match(md, /\*\*Changed\*\*:/);
assert.match(md, /color-contrast:/);
assert.match(md, /\*\*Unchanged\*\*: region/);
}
function scannerBrowserValidation() {
const scan = require(repoPath('a11y-audit/scripts/scan.js'));
assert.strictEqual(scan.validateBrowserLib('puppeteer'), 'puppeteer');
assert.throws(
() => scan.validateBrowserLib('puppeteer; echo injected'),
/Unsupported browser library/
);
const scanSource = fs.readFileSync(repoPath('a11y-audit/scripts/scan.js'), 'utf8');
assert.match(scanSource, /spawnSync\('npm', \['install', '--prefix', SKILL_DEPS_DIR, packageName\]/);
assert.doesNotMatch(scanSource, /execSync\(`npm install/);
}
function markdownEscapingRegression() {
const dir = tmpPath('markdown-escaping');
resetDir(dir);
const inputPath = path.join(dir, 'scan.json');
fs.writeFileSync(inputPath, JSON.stringify({
generated_at: '2026-05-31T00:00:00Z',
urls: ['https://example.com/a|b'],
results: [
{
url: 'https://example.com/a|b',
axe: {
violations: [
{
id: 'color-contrast',
impact: 'serious',
description: 'desc',
help: 'Fix | this\nnow',
helpUrl: 'javascript:alert(1)',
tags: ['wcag111'],
nodes: [
{
target: ['main` | td'],
html: '<main></main>',
any: [{ data: { fgColor: '#000', bgColor: '#fff', contrastRatio: 1.2, expectedContrastRatio: 4.5 } }],
},
],
},
],
passes: [],
incomplete: [],
inapplicable: [],
},
lighthouse: { status: 'skipped', reason: 'not | run\nnow' },
},
],
}, null, 2));
runNode([
'a11y-audit/scripts/report.js',
'--input', inputPath,
'--project-name', 'Demo | Project',
'--runtime-url', 'https://example.com/a|b',
'--output-dir', dir,
]);
const md = fs.readFileSync(findGeneratedFile(dir, '.md'), 'utf8');
assert.match(md, /Demo \\| Project/);
assert.match(md, /not \\| run now/);
assert.match(md, /color-contrast/);
assert.doesNotMatch(md, /\]\(javascript:alert/);
assert.match(md, /main\\` \\| td/);
}
function issuePlanEscapingRegression() {
const dir = tmpPath('issue-plan-escaping');
resetDir(dir);
const inputPath = path.join(dir, 'scan.json');
const outputPath = path.join(dir, 'issue-plan.md');
fs.writeFileSync(inputPath, JSON.stringify({
results: [
{
url: 'https://example.com/route--><script>|x',
axe: {
violations: [
{
id: 'label|bad',
impact: 'critical',
help: 'Do | not\nexecute',
tags: ['wcag131'],
nodes: [{ target: ['input'] }],
},
],
},
},
],
}, null, 2));
runNode([
'a11y-audit/scripts/plan-issues.js',
'--input', inputPath,
'--output', outputPath,
]);
const plan = fs.readFileSync(outputPath, 'utf8');
assert.match(plan, /label\\|bad/);
assert.match(plan, /Do \\| not execute/);
assert.doesNotMatch(plan, /--><script>/);
}
function dependencyPolicyCheck() {
const scanSource = fs.readFileSync(repoPath('a11y-audit/scripts/scan.js'), 'utf8');
const skill = fs.readFileSync(repoPath('a11y-audit/SKILL.md'), 'utf8');
assert.match(scanSource, /spawnSync\('npm'/);
assert.match(scanSource, /skill-deps \(auto-installed\)/);
assert.match(skill, /`scan\.js` may auto-install missing dependencies/);
assert.match(skill, /ask before invoking scan\.js/);
}
resetDir(tmpRoot);
if (validateMode) {
test('syntax checks cover bundled scripts and eval harnesses', validateSyntax);
test('JSON files parse and sample output matches audit shape', validateJsonFiles);
test('YAML and frontmatter files parse', validateYamlFiles);
test('bootstrap-context smoke test creates workspace context', validateBootstrapSmoke);
}
test('eval-9 preserves cross-origin sitemap URLs', () => runDiscoverFixture('eval-9'));
test('eval-10 keeps discovery deterministic', () => runDiscoverFixture('eval-10'));
test('eval-12 blocks cross-origin sitemaps unless explicitly allowed', () => runDiscoverFixture('eval-12'));
test('eval-2 plans issues with labels and deduplication', eval2IssuePlanning);
test('eval-3 quick scan summarizes one plain HTML page', eval3QuickScan);
test('eval-4 reports skipped Lighthouse without inventing scores', eval4SkippedLighthouseReport);
test('eval-11 reports page-aware delta movement', eval11ReportDelta);
test('scan.js rejects unsupported browser package names before install', scannerBrowserValidation);
test('report.js escapes target-derived markdown fields', markdownEscapingRegression);
test('plan-issues.js escapes target-derived markdown fields', issuePlanEscapingRegression);
test('scan.js dependency auto-install policy is documented', dependencyPolicyCheck);
const failed = results.filter((result) => !result.ok);
console.log(`\n${results.length - failed.length}/${results.length} checks passed`);
if (failed.length > 0) process.exit(1);
Accessibility Audit Skill -- Handoff Document
What This Is
A portable accessibility-audit skill bundle for Claude Code and Codex. The core workflow lives in SKILL.md; platform-specific notes live in references/claude-code.md and references/codex.md.
Current State: v16, self-contained and executable-eval validated
The workflow has been run successfully in Claude Code for eval-1. Codex eval-1 has been exercised against PAICE2. The bundle now includes reusable scripts, focused reference files, expanded eval coverage, sample output artifacts, and a CI template. The direct degraded paths for Lighthouse-unavailable, runtime URL reconciliation, first-run context creation, and missing-browser-automation handling have been validated. The repo now has a deterministic offline eval runner wired to npm run validate, which is also the GitHub Actions validation command.
The three most recent correctness issues are now resolved:
discover.jspreserves published sitemap URLs acrossrobots.txt,
sitemap indexes, nested sitemap files, and redirects instead of rewriting them onto the runtime origin.
- Representative page selection is deterministic across repeated runs.
report.jsdelta comparison is page-aware and reports rule movement
even when total instance counts stay flat.
Those fixes now have runnable local regression fixtures:
eval-2covers context-aware issue planning, standards carry-through,
configured labels, thresholding, and duplicate skipping in dry-run mode
eval-3covers quick-scan summary behavior without report generationeval-4covers skipped-Lighthouse markdown and JSON report contractseval-9covers cross-origin sitemap preservationeval-10covers deterministic discovery across repeated runseval-11covers page-aware delta reporting
A full audit was run 2026-03-26 against the AI Regulation Reference (10-page static HTML site, http://127.0.0.1:8081). The audit found 3 rules / 69 instances (color-contrast, landmark-one-main, region), all of which were remediated to zero violations. This run revealed four token-efficiency improvements, all now implemented in v10.
Files in this directory
| File | Purpose |
|---|---|
| SKILL.md | Portable six-phase audit pipeline (main skill) |
| MANIFEST.yaml | Bundle metadata, dependencies, file inventory |
| CHANGELOG.md | Append-only change history |
| HANDOFF.md | This file -- current state and next steps |
| evals/evals.json | 11 eval cases with passing results recorded for eval-1 through eval-11 where deterministic or prior runtime validation exists |
| evals/run-evals.js | Offline executable eval and validation runner |
| references/claude-code.md | Claude-specific launch and Preview notes |
| references/codex.md | Codex-specific execution notes |
| references/output-contract.md | Markdown/JSON output rules |
| references/issue-trackers.md | Issue creation and deduplication rules |
| references/output-schema.json | Stable JSON output schema |
| references/project-context-template.md | Canonical context-file contract |
| scripts/scan.js | Reusable axe-based scanning helper (--summary flag) |
| scripts/bootstrap-context.js | First-run context bootstrap helper |
| scripts/discover.js | Template-aware page discovery and sampling |
| scripts/report.js | Deterministic report generator for Phases 3+5 |
| scripts/plan-issues.js | Non-destructive issue planning helper |
| assets/sample-output/ | Sample markdown and JSON artifacts |
| assets/ci/github-actions/accessibility-audit.yml | CI workflow starter |
| agents/openai.yaml | Codex UI metadata |
Where to Put the Skill
- Upstream (generic):
/Users/snap/Git/skill-a11y-audit/ - Claude install:
.claude/skills/a11y-audit/in the target project - Codex install:
$CODEX_HOME/skills/a11y-audit/or equivalent skill import path - Project-specific mutable state:
.a11y-audit/PROJECT_CONTEXT.mdin the target workspace
Dependencies
| Dependency | Required? | Check |
|---|---|---|
axe-core (npm) | Yes | ls node_modules/axe-core |
puppeteer or playwright (npm) | Yes | ls node_modules/puppeteer or ls node_modules/playwright |
lighthouse (npm/CLI) | Recommended | npx lighthouse --version |
| issue tracker CLI | Phase 6 only | gh --version, glab --version, or tracker equivalent |
Known Limitations
1. No real AT testing. The skill runs headless Chromium only. Screen reader, voice control, and mobile AT testing require manual procedures. Phase 4 generates checklists for this.
2. SPA navigation. For single-page applications, the scanning script navigates via direct URL. Pages that require client-side routing state (e.g., post-login pages, multi-step flows) may not render correctly in headless mode. The user may need to provide authenticated session cookies or skip those routes.
3. axe-core version coupling. Results depend on the installed axe-core version. Different versions may report different violations. The report records the version used.
4. Lighthouse variance. Lighthouse scores vary between runs due to rendering timing. The skill runs once per page and reports the result; it does not average multiple runs.
5. Lighthouse optionality is real. Some projects will have axe-core and browser automation installed but no runnable Lighthouse CLI. The skill now treats this as a normal degraded mode and requires the report to state the skip reason explicitly.
6. No CI integration. The skill runs on demand via an interactive agent. A separate GitHub Actions workflow would be needed for continuous accessibility monitoring.
7. Label creation. Phase 6 assumes GitHub labels already exist. It does not create labels. If a label does not exist, gh issue create will create it automatically, but the label will lack a description and color.
8. Expected URL drift. Local dev servers may bind to a different port than the prompt or context file expects. The skill now updates the workspace-local context to the working URL and records the mismatch in the report methodology.
9. Puppeteer-first scanner. The bundled scanner currently supports Puppeteer directly. Playwright remains a documented fallback path in the skill, but the helper script has not been expanded to first-class Playwright support yet.
10. Live issue tracker path still pending. The skill now has explicit issue-tracker reference guidance, a non-destructive issue planner, and deduplication keys, but the end-to-end authenticated ticket creation path has not yet been re-run after the refactor.
Completed: Token-Efficiency Improvements (v10)
All four improvements from the 2026-03-26 audit are now implemented:
1. `scripts/report.js` (~3000 tokens saved): Deterministic report generator with hardcoded 50 WCAG 2.1 AA criteria, axe tag mapping, violation aggregation, color-contrast detail extraction, and output generation per output-contract.md and output-schema.json. 2. `--summary` flag on scan.js (~500 tokens saved): Strips node detail from passes/inapplicable arrays, adds per-page counts. 3. Phase 1 condensed (~500 tokens saved): Replaced ~30-line enumeration with ~10 focused lines. 4. Reference reads removed (~800 tokens saved): Phase 5 invokes report.js directly; agent no longer reads output-contract.md or output-schema.json during normal runs.
Suggested Next Steps
1. Run eval-2 against a real authenticated tracker for full live issue-mode validation 2. Keep scripts/scan.js Puppeteer-first unless a Playwright-native project forces expansion 3. Use scripts/plan-issues.js as the default dry-run step before any live ticket creation 4. Copy updated skill back to .claude/skills/a11y-audit/ in target projects 5. Keep new regression fixes covered by npm run validate before updating bundle metadata
Claude Code Notes
Use this file only when running the skill in Claude Code.
Dev Server Discovery
- Check
.claude/launch.jsonfor project-specific launch targets, URLs,
and environment hints before inventing a new startup command.
- If Claude Preview MCP tools are available and a local server needs to
be started, prefer preview_start over ad hoc shell commands.
- If
.claude/launch.json, package scripts, and repo docs disagree,
report the mismatch before scanning.
Workspace-Local Context
- Store mutable audit preferences in the target workspace, not in the
installed skill directory.
- Default path:
.a11y-audit/PROJECT_CONTEXT.mdat the workspace root. - If the project already keeps accessibility planning docs elsewhere,
record those paths in the context file instead of duplicating content.
Practical Guidance
- Prefer the Preview-provided URL when it differs from a guessed
localhost port.
- Treat
.claude/launch.jsonas a project hint, not as proof that the
app is healthy. Confirm the selected URL responds before running scans.
Codex Notes
Use this file only when running the skill in Codex.
Dev Server Discovery
- Check
package.json, repo docs, and existing running processes first. - Treat
.claude/launch.jsonas an optional repo artifact if it exists;
do not assume Claude Preview tooling is available.
- If the workspace contains multiple frontend apps, identify the target
app before scanning or state the assumption explicitly.
Workspace-Local Context
- Store mutable audit preferences in the target workspace, not in the
installed skill directory.
- Default path:
.a11y-audit/PROJECT_CONTEXT.mdat the workspace root. - Reuse an existing project-local context file when present; otherwise,
create the default file only after confirming the audit scope.
Practical Guidance
- Ask before installing missing packages such as
axe-core,
puppeteer, playwright, or lighthouse.
- Prefer project-local commands and dependencies over global tooling.
- If browser automation is blocked by missing dependencies or a missing
running app, summarize the blocker and continue with the highest-value partial audit the workspace supports.
Issue Trackers
Read this file only when output_mode is markdown+issues.
Preconditions
- Require explicit user confirmation before creating any tickets.
- Load tracker settings from
.a11y-audit/PROJECT_CONTEXT.md. - Create tickets only at or above the configured severity threshold.
- Prefer
scripts/plan-issues.jsfirst when you need a non-destructive
dry run for user review or deduplication planning.
Supported Trackers
| Tracker | CLI / API | Create Pattern |
|---|---|---|
| GitHub | gh | gh issue create --title "..." --label "..." --body "..." |
| GitLab | glab | glab issue create --title "..." --label "..." --description "..." |
| Linear | API or CLI | Create via API payload |
| Jira | jira or API | jira issue create --project "..." --type Bug --summary "..." |
Ticket Structure
Each ticket should contain:
- WCAG criterion
- axe rule ID
- severity / priority
- affected pages
- plain-language description
- impact statement
- affected selectors or representative nodes
- suggested remediation
- references to axe help and WCAG understanding docs
- audit metadata
Title format:
[A11y] [Severity] [Page]: [Brief description]
Deduplication
- Search for existing open tickets before creating new ones.
- Default deduplication key:
<!-- a11y-audit-key: [rule-id]::[page-path] -->
- If the tracker strips HTML comments, use a custom field or label.
- If a match already exists, skip creation and record the existing ticket
in the report.
Priority Mapping
| axe Impact | Priority |
|---|---|
| critical | P0 |
| serious | P1 |
| moderate | P2 |
| minor | P3 |
Reporting Rule
Even when tickets are created, the markdown report remains the primary audit artifact. The issue section should summarize what was created, what was skipped as duplicate, and what remained below threshold.
Safe Dry-Run Path
Before live issue creation, you can generate a markdown issue plan from helper scan JSON plus project context:
node a11y-audit/scripts/plan-issues.js \
--input docs/accessibility/audits/audit-YYYY-MM-DD.json \
--context .a11y-audit/PROJECT_CONTEXT.md \
--existing /tmp/a11y-existing-keys.json \
--output /tmp/a11y-issue-plan.md \
--threshold P1Use that plan to confirm scope, configured labels, standards mapping, deduplication keys, duplicate skipping, and priority threshold before invoking a tracker CLI.
Output Contract
Read this file when generating audit deliverables.
Markdown Report
Write a markdown report with these sections in order:
1. Header 2. Executive Summary 3. Automated Scan Results 4. WCAG 2.1 AA Compliance Matrix 5. Delta from Previous Audit 6. Project-Specific Standard 7. Manual Testing Recommendations 8. Remediation Priority 9. Issues Created 10. Methodology
Rules:
- Use valid GFM tables.
- Reference WCAG criteria as
SC X.X.X. - Use axe
helpUrllinks when available. - Omit sections that are truly empty.
- If Lighthouse was skipped, say why in both Executive Summary and Methodology.
- If the runtime URL differs from the expected URL, record both.
- Phrase compliance status conservatively: this is an automation-assisted audit view, not a conformance certification.
JSON Output
When output_mode is markdown+json or markdown+issues, write a JSON file alongside the markdown report. Use the schema in references/output-schema.json.
Delta Section
- Include this section only if a prior audit output exists.
- Show new violations, resolved violations, and score changes when you
have enough evidence to compare runs.
- If comparison data is missing or non-comparable, omit the section
rather than inventing a weak delta.
Degraded Modes
- If Lighthouse is missing, write
lighthouse.status: "skipped"in the
JSON output and explain the reason.
- If browser automation is blocked entirely, do not generate a fake
report. Summarize the blocker and produce the highest-value partial output the workspace supports.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://agentskills.local/a11y-audit/output-schema.json",
"title": "Accessibility Audit Output",
"type": "object",
"required": ["date", "tool", "pages", "summary", "violations", "matrix", "lighthouse"],
"properties": {
"date": {
"type": "string",
"pattern": "^\\d{4}-\\d{2}-\\d{2}$"
},
"tool": {
"type": "string"
},
"pages": {
"type": "array",
"items": { "type": "string" }
},
"runtime_url": {
"type": "string"
},
"expected_url": {
"type": "string"
},
"lighthouse": {
"oneOf": [
{
"type": "object",
"additionalProperties": {
"type": "integer",
"minimum": 0,
"maximum": 100
}
},
{
"type": "object",
"required": ["status", "reason"],
"properties": {
"status": { "const": "skipped" },
"reason": { "type": "string" }
},
"additionalProperties": true
}
]
},
"summary": {
"type": "object",
"required": ["critical", "serious", "moderate", "minor"],
"properties": {
"critical": { "type": "integer", "minimum": 0 },
"serious": { "type": "integer", "minimum": 0 },
"moderate": { "type": "integer", "minimum": 0 },
"minor": { "type": "integer", "minimum": 0 }
},
"additionalProperties": false
},
"violations": {
"type": "array",
"items": {
"type": "object",
"required": ["rule", "impact", "pages", "instances"],
"properties": {
"rule": { "type": "string" },
"impact": {
"type": "string",
"enum": ["critical", "serious", "moderate", "minor"]
},
"wcag": {
"type": "array",
"items": { "type": "string" }
},
"pages": {
"type": "array",
"items": { "type": "string" }
},
"instances": { "type": "integer", "minimum": 0 }
},
"additionalProperties": true
}
},
"matrix": {
"type": "object",
"additionalProperties": {
"type": "string",
"enum": ["pass", "fail", "manual", "not-applicable"]
}
}
},
"additionalProperties": true
}
Related skills
How it compares
Executable audit + issue-planning skill bundle—not a design-system component library or a legal compliance certification service.
FAQ
Who is a11y-audit for?
Developers and small teams shipping web products who want scripted accessibility discovery, scanning, and WCAG-labeled issue planning from their coding agent.
When should I use a11y-audit?
In Build frontend while fixing components, in Ship review before release, and in Launch geo when public pages must meet accessibility expectations for broader discoverability and trust.
Is a11y-audit safe to install?
Scripts may read your site and filesystem; review the Security Audits panel on this Prism page and prefer dry-run issue modes until you trust tracker credentials and network behavior.