Automated Assessment

Skill Name

One paragraph description of the skill's purpose.

When to Use This Skill

Describe the scenarios when this skill should be activated.

Core Workflow

Describe the main workflow steps.

Using Reference Documentation

When doing X, consult references/x-guide.md for details.

When doing Y, consult references/y-patterns.md for examples.

Running Scripts

To validate something:

scripts/validate.sh /path/to/project

Using Asset Templates

To set up something, copy assets/template.yaml to your project.

Quick Reference

Pre-Commit Checklist

• [ ] Check 1
• [ ] Check 2
• [ ] Check 3

---

Contributing: https://github.com/netresearch/my-skill-name

LLM Rubric: dependency-compatibility Domain

This file contains detailed rubrics for LLM-based reviews in the dependency-compatibility domain. Reference specific sections using markdown anchors (e.g., assets/llm-rubric-dependency-compatibility.md#api-compatibility).

---

api-compatibility

Checkpoint: Verify API Compatibility Across Declared Versions

Requirement: All method calls, constructor invocations, and class references must be compatible with every major version declared in composer.json constraints.

Verification Steps:

1. Parse composer.json to find dependencies with multi-major-version constraints (e.g., ^2.0 || ^3.0) 2. For each such dependency, identify all usages in Classes/ or src/ 3. Check whether called methods exist on the relevant interfaces/classes across all declared versions 4. Flag any usage that is version-specific without conditional logic

Evaluation Criteria:

Evidence Required:

• List the multi-version constraints found
• For each version-specific API call, quote the file and line
• Note whether version-conditional logic exists

---

mock-validity

Checkpoint: Verify Test Mocks Match Real Interfaces

Requirement: Test mocks and stubs must reference methods that actually exist on the interfaces or classes being mocked.

Verification Steps:

1. Find all test files using getMockBuilder(), createMock(), prophesize(), or similar 2. For each mock, identify the target class/interface 3. Verify that each ->method('X') call references a method that exists on the target 4. Verify that ->willReturn() values match expected return types 5. Cross-reference with all declared major versions of the dependency

Evaluation Criteria:

Evidence Required:

• List each mock with its target class and mocked methods
• Flag methods that don't exist on the target in specific versions
• Quote the test file and line number

---

phpstan-ignore-validity

Checkpoint: Verify PHPStan Ignore Tags Are Valid

Requirement: All @phpstan-ignore tags must suppress real errors and work correctly across all supported versions.

Verification Steps:

1. Find all @phpstan-ignore-* annotations in source and test files 2. Verify each tag suppresses an error that actually occurs 3. Check that tags use specific error identifiers (not blanket ignores) 4. Verify tags are necessary across all supported PHP and dependency versions 5. Check for explanatory comments on each ignore tag

Evaluation Criteria:

Evidence Required:

• List each ignore tag with its location and suppressed error
• Flag tags that appear unnecessary (no corresponding error)
• Flag tags using generic suppression without identifiers

---

test-assertion-quality

Checkpoint: Verify Test Assertions Are Specific

Requirement: Tests must use specific assertions that would catch subtle implementation changes.

Verification Steps:

1. Scan test files for assertion patterns 2. Flag assertTrue($a == $b) patterns (should be assertEquals) 3. Flag tests with no assertions (only testing "no exception") 4. Flag overly generic assertions that would pass despite behavior changes 5. Check that error paths have dedicated test methods

Evaluation Criteria:

Evidence Required:

• Count of specific vs generic assertions
• List any tests with no assertions
• List any tests that would pass despite implementation changes
• Note coverage of error/edge case paths

---

pre-push-validation

Checkpoint: Verify Local CI Checks Were Run

Requirement: All local CI checks (PHPStan, PHPUnit, PHP-CS-Fixer, Rector) must have been run and passing before code is pushed.

Verification Steps:

1. Check for the presence of CI tool configurations (phpstan.neon, phpunit.xml, .php-cs-fixer.php, rector.php) 2. For each configured tool, verify it can run without errors 3. Check for a pre-push hook or CI script that enforces these checks 4. Verify that all tools pass on the current codebase state

Evaluation Criteria:

Evidence Required:

• List configured CI tools and their results
• Note whether a pre-push hook exists
• Quote any CI tool errors

LLM Rubric: repo-health Domain

This file contains detailed rubrics for LLM-based reviews in the repo-health domain. Reference specific sections using markdown anchors (e.g., references/llm-rubric.md#badge-order).

---

badge-order

Checkpoint: Verify README Badge Ordering

Requirement: Badges in README.md must follow the standard order for TYPO3 extension projects.

Expected Order:

<!-- Row 1: CI/Quality badges -->
[![CI](...)][ci]
[![codecov](...)][codecov]
[![Documentation](...)][docs]

<!-- Row 2: Security badges -->
[![OpenSSF Scorecard](...)][scorecard]
[![OpenSSF Best Practices](...)][bestpractices]
[![SLSA 3](...)][slsa]

<!-- Row 3: Standards badges -->
[![PHPStan](...)][phpstan]
[![PHP 8.x+](...)][php]
[![TYPO3 vXX](...)][typo3]
[![License](...)][license]
[![Latest Release](...)][release]

<!-- Row 4: TYPO3 TER badges (if published to TER) -->
![Composer](https://typo3-badges.dev/badge/EXT_KEY/composer/shields.svg)
![Downloads](https://typo3-badges.dev/badge/EXT_KEY/downloads/shields.svg)

Evaluation Criteria:

Evidence Required:

• Quote the actual badge order from README.md
• Identify which badges are misplaced if failing

---

readme-structure

Checkpoint: Verify README Has Standard Sections

Requirement: README.md should have clear sections covering essential topics.

Required Sections:

1. Header - Extension name, badges, brief description 2. Installation - Composer command, TER link, or manual installation 3. Configuration - TypoScript setup, Extension Manager settings, or "no configuration needed" 4. Usage - How to use the extension (examples, screenshots) 5. Development - How to contribute, run tests, lint code 6. License - License type and link 7. Credits (optional) - Contributors, sponsors

Evaluation Criteria:

Evidence Required:

• List the section headers found in README.md
• Note which required sections are missing

---

branding-compliance

Checkpoint: Verify Netresearch Branding

Requirement: Project follows Netresearch branding guidelines.

Checklist:

1. Logo: Extension icon uses Netresearch brand color #2F99A4 as primary 2. Description: Repository description ends with - by Netresearch 3. Credits: README credits Netresearch appropriately 4. Colors: Any custom UI uses brand colors:

• Primary: #2F99A4 (teal)
• Accent: #FF4D00 (orange)
• Neutral: #585961 (gray)

Evaluation Criteria:

Evidence Required:

• Quote the repository description
• Quote the credits/attribution section from README

---

agents-md-accuracy

Checkpoint: Verify AGENTS.md Accuracy

Requirement: AGENTS.md content must match actual codebase state.

Verification Steps:

1. File listings - Do documented files exist? 2. Command listings - Do Makefile targets exist? 3. Module counts - Do numbers match actual file counts? 4. Descriptions - Do module descriptions match docstrings?

Evaluation Criteria:

Evidence Required:

• List any files documented but not found
• List any commands documented but not in Makefile
• Compare documented counts vs actual counts

---

topics-compliance

Checkpoint: Verify GitHub Repository Topics

Requirement: TYPO3 extension repositories must have required topics.

Required Topics:

• typo3 (always required)
• typo3-extension (always required)
• php (always required)
• 2-5 domain-specific topics (e.g., ckeditor, llm, ai)

Evaluation Criteria:

Evidence Required:

• List current repository topics
• Note which required topics are missing
• Suggest domain topics if fewer than 2

---

description-format

Checkpoint: Verify Repository Description Format

Requirement: GitHub repository description must follow the standard format.

Format: <What the extension does> - by Netresearch

Examples:

• Good: TYPO3 extension for context-based content rendering - by Netresearch
• Good: AI-powered content writer for TYPO3 - by Netresearch
• Bad: Contexts extension (too vague, missing branding)
• Bad: TYPO3 extension for contexts (missing branding)

Evaluation Criteria:

Evidence Required:

• Quote the current repository description
• If failing, suggest improved description

Calibration: keeping checkpoints honest over time

Checkpoints are predictions, not verdicts. A checkpoint that fires error / warning / info is claiming the project is more or less likely to ship a defect. That claim has to be checked against reality periodically, or it decays into ritual.

This reference covers:

• Calibration Debt — when checkpoints stop predicting anything true
• Calibration Anchor — what every generated checkpoint must record so the audit has something to test against
• Anti-pattern: the checkpoint ratchet

Calibration Debt

A checkpoint accumulates calibration debt when any of these is true:

• It has never flagged a real defect in N consecutive /assess runs (likely redundant — the rule is already enforced upstream, or the failure mode no longer occurs).
• A real defect shipped that the checkpoint should have caught but didn't (false negative — the rule is too narrow).
• The checkpoint repeatedly flags issues the team dismisses as noise (false positive — the rule is miscalibrated or context-blind).

Periodic calibration

Once per release cycle (or quarterly for low-velocity repos), audit checkpoint output against:

• Bugs reported in production since the last audit (would any checkpoint have caught this?)
• PR review rejections (did a checkpoint that passed correlate with a human reviewer flagging something?)
• Checkpoint dismissal rate (how often is the user overriding a finding?)

Use --review to surface candidates; demote to info, retire, or tighten the rule based on findings. The point is not to maximize checkpoint count — it is to keep the ones that still predict something true.

Calibration Anchor

This rule applies at checkpoint generation time, via add-checkpoints. For each generated checkpoint, record (in a comment or accompanying note) the defect class it predicts:

- id: TYPO3-CGL-01
  # Predicts: PHP-CS-Fixer violations that block CI green.
  # Calibration: retire if no real CI break in 6 months of /assess runs.
  type: command
  # (other fields omitted)

Why: every checkpoint claims to predict something. If you cannot name the defect class in one line, the checkpoint is testing the rule, not the outcome — and will decay into ritual. The periodic calibration above has nothing to anchor against without this.

A checkpoint without a predicted defect class is a candidate for info severity at most, never error.

Anti-pattern: the checkpoint ratchet

Adding checkpoints to plug every past defect, never retiring any, until the suite is slow and noisy enough that the team stops reading it. A checkpoint that nobody acts on is worse than no checkpoint — it costs CI time AND trains the team to ignore the report.

Inspiration

The framing comes from Spotify Engineering — Better Experiments with LLM Evals: A Funnel, Not a Fork. Their core observation: "without offline-online signal calibration, our evals are opinions, not evidence." The same logic transfers to mechanical checkpoints — without periodic outcome correlation, a green checkpoint is an opinion, not evidence.

Checkpoint Coverage Requirements

This document defines the minimum checkpoint coverage requirements for skills that enforce code quality, testing, or dependency management standards.

Purpose

When a skill SHOULD have caught an issue but did not, it usually means the skill's checkpoints.yaml is missing coverage for that failure class. This reference defines what categories of checkpoints each skill domain must include.

Required Coverage Categories

API Compatibility

Skills dealing with dependency management or version upgrades must include checkpoints that verify:

Checkpoint template:

- id: XX-API-01
  domain: dependency-compatibility
  prompt: |
    Verify that all method calls on dependency interfaces/classes exist
    across ALL major versions declared in composer.json constraints.
    Check for:
    1. Methods called that don't exist in some versions
    2. Constructor signatures that changed between versions
    3. Return types that changed between versions
    4. Classes/interfaces that were renamed or removed
  severity: error
  desc: "API calls must be compatible with all declared dependency versions"

Test Mock Validity

Skills dealing with testing must include checkpoints that verify:

Checkpoint template:

- id: XX-MOCK-01
  domain: code-quality
  prompt: |
    Verify that all test mocks and stubs reference methods that actually
    exist on the interfaces/classes being mocked. Check:
    1. ->method('X') calls reference methods that exist on the mock target
    2. ->willReturn() values match the actual method return types
    3. Mock constructors match the real class constructors
    Report any mocks that reference non-existent methods.
  severity: error
  desc: "Test mocks must reference methods that exist on mocked interfaces"

PHPStan Ignore Tag Validity

Skills dealing with static analysis must include checkpoints that verify:

Checkpoint template:

- id: XX-STAN-01
  domain: code-quality
  prompt: |
    Verify that all @phpstan-ignore tags in the codebase:
    1. Suppress errors that actually exist (not stale/outdated tags)
    2. Use specific error identifiers, not blanket ignores
    3. Are necessary across ALL supported PHP and dependency versions
    4. Have comments explaining WHY the ignore is needed
    Report any ignore tags that appear unnecessary or overly broad.
  severity: warning
  desc: "PHPStan ignore tags must be valid and specific"

Test Assertion Specificity

Skills dealing with testing must include checkpoints that verify:

Checkpoint template:

- id: XX-TEST-01
  domain: code-quality
  prompt: |
    Verify test assertion quality:
    1. Tests use specific assertions (assertEquals, assertSame, assertInstanceOf)
       not generic ones (assertTrue with comparison)
    2. Tests assert meaningful output values, not just absence of exceptions
    3. Tests would fail if the implementation behavior changed subtly
    4. Error/edge case paths have dedicated test methods
    Report any tests that could pass despite broken implementation.
  severity: warning
  desc: "Test assertions must be specific enough to catch regressions"

Validating Checkpoint Coverage

The /assess --review command can identify skills with missing checkpoint coverage. When a real-world issue would not have been caught by existing checkpoints, it is classified as a skill-gap.

Coverage Audit Workflow

1. Identify the failure class -- what kind of issue was missed? 2. Map to coverage category -- which category above does it fall into? 3. Check existing checkpoints -- does the skill's checkpoints.yaml have a checkpoint for this category? 4. If missing, add checkpoint -- use the templates above as starting points

Minimum Coverage Matrix

Required = must have at least one checkpoint in this category Recommended = should have unless skill explicitly doesn't cover this domain -- = not applicable to this skill's scope

Pre-Push Validation Gate

Every project should run local CI checks before pushing. The assessment framework can verify this via the pre-push domain.

PP-01: Local CI Checks Passing

Before any git push, the following must have been run and passed:

Checkpoint template:

- id: PP-01
  type: command
  pattern: "test ! -f vendor/bin/phpstan || vendor/bin/phpstan analyse --no-progress --error-format=raw 2>&1 | tail -1 | grep -q 'No errors'"
  severity: error
  desc: "PHPStan analysis must pass locally"

- id: PP-02
  type: command
  pattern: "test ! -f vendor/bin/phpunit || vendor/bin/phpunit --no-coverage 2>&1 | tail -1 | grep -qE '(OK|No tests)'"
  severity: error
  desc: "Unit tests must pass locally"

- id: PP-03
  type: command
  pattern: "test ! -f vendor/bin/php-cs-fixer || vendor/bin/php-cs-fixer fix --dry-run --diff"
  severity: warning
  desc: "Code style must be clean (PHP-CS-Fixer)"

- id: PP-04
  type: command
  pattern: "test ! -f vendor/bin/rector || vendor/bin/rector process --dry-run"
  severity: warning
  desc: "Rector should report no pending changes"

Checkpoint Workflow

Detailed assessment workflow, checkpoint format, agent prompts, and validation rules.

Assessment Workflow Steps

Step 1: Discover Checkpoints

# Find all skills with checkpoints
for skill_dir in ~/.claude/plugins/cache/*/skills/*/; do
  skill_md="$skill_dir/SKILL.md"
  checkpoints_yaml="$skill_dir/checkpoints.yaml"

  # Check override in front matter
  override=$(grep -E "^checkpoints:" "$skill_md" 2>/dev/null | cut -d: -f2 | tr -d ' ')

  if [[ -n "$override" ]]; then
    checkpoint_file="$skill_dir/$override"
  elif [[ -f "$checkpoints_yaml" ]]; then
    checkpoint_file="$checkpoints_yaml"
  else
    continue  # No checkpoints for this skill
  fi

  echo "Found: $checkpoint_file"
done

Checkpoint Discovery (Convention-with-Override)

For each skill, checkpoints are discovered using this logic:

1. Parse SKILL.md front matter
2. If `checkpoints:` key exists -> use that explicit path (override)
3. Else if checkpoints.yaml exists in skill root -> use it (convention)
4. Else -> no checkpoints, skip this skill

Skill Structure with Checkpoints

my-skill/
├── SKILL.md              # Skill content, optional checkpoints: key
├── checkpoints.yaml      # Auto-discovered by convention
└── references/
    └── llm-rubric.md     # LLM review prompts (optional)

Why Convention-with-Override?

Step 2: Evaluate Preconditions

For each discovered skill, evaluate its preconditions: block. All preconditions must pass (AND logic). If any precondition fails, the entire skill is silently skipped -- this is not an error.

# For each discovered checkpoint file:
# 1. Parse the preconditions block
# 2. Run each precondition (same types as mechanical checks)
# 3. If ANY fails -> skip this skill entirely
# 4. If ALL pass -> proceed to mechanical checks

This prevents irrelevant skills from producing false negatives (e.g., TYPO3 checks on a Go project).

Step 3: Run Scripted Checks (Tier 1)

For each mechanical checkpoint (in skills that passed preconditions):

scripts/run-checkpoints.sh <checkpoint-file.yaml> <project-root>

This runs all file_exists, contains, regex, etc. checks without any LLM involvement.

Step 4: Run Domain Agents (Tier 2)

Group llm_review checkpoints by domain, spawn one agent per domain:

Agent: repo-health
Checkpoints: GH-15, GH-16, NB-01, AG-01
Prompt: "You are auditing repo health. Verify these checkpoints..."
Output: JSON with pass/fail per checkpoint

Step 5: Aggregate Results

Collect all results into compliance report:

{
  "project": "netresearch/contexts",
  "timestamp": "2026-01-30T19:00:00Z",
  "overall_status": "FAIL",
  "summary": {
    "total": 45,
    "pass": 38,
    "fail": 5,
    "skip": 2
  },
  "checkpoints": [
    {"id": "GH-01", "skill": "github-project", "status": "pass", "evidence": "README.md exists"},
    {"id": "GH-03", "skill": "github-project", "status": "fail", "evidence": "Missing codecov badge"}
  ]
}

Checkpoints YAML Format

Create checkpoints.yaml in your skill root:

version: 1
skill_id: github-project

mechanical:
  - id: GH-01
    type: file_exists
    target: README.md
    severity: error
    desc: "README.md must exist"

  - id: GH-02
    type: contains
    target: README.md
    pattern: "codecov.io"
    severity: warning
    desc: "README should have Codecov badge"

llm_reviews:
  - id: GH-15
    domain: repo-health
    rubric: references/llm-rubric.md#badge-order
    severity: warning
    desc: "Verify badge ordering follows standard"

  - id: GH-16
    domain: repo-health
    prompt: |
      Check README structure for standard sections:
      - Installation/Setup
      - Configuration
      - Development
      - License
    severity: info
    desc: "README should have standard sections"

For full schema documentation, see references/checkpoints-schema.md.

Agent Prompt Template

Each domain agent receives this prompt:

You are an automated compliance auditor for projects.

## Your Task
Verify the project against ONLY the checkpoints listed below.
You must NOT fix issues - only report compliance status.

## Output Format
Return ONLY a JSON object with this exact structure:
{
  "domain": "repo-health",
  "checkpoints": [
    {
      "id": "GH-15",
      "status": "pass" | "fail" | "skip",
      "evidence": "Quote the specific line/file or explain why it fails/passes"
    }
  ]
}

## Checkpoints to Verify
[CHECKPOINTS INJECTED HERE]

## Rules
- Every checkpoint MUST have a status (no nulls)
- Evidence MUST be specific (line numbers, quotes)
- "skip" only if checkpoint doesn't apply to this project type
- Be strict - when in doubt, mark as "fail"

Validation Rules

The assessment is NOT complete until:

• [ ] All skills were scanned for checkpoints
• [ ] All scripted checks returned exit code
• [ ] All domain agents returned valid JSON
• [ ] All checkpoints have non-null status
• [ ] Evidence field is non-empty for all fail/pass

If ANY validation fails, retry that component.

Implementation Notes

Why Domain Batching?

• Not 20 agents (one per skill) - too expensive, rate limits
• Not 1 agent (all skills) - context overload, satisficing
• 3-4 domain agents - balanced context, related checks grouped

Why Scripted Checks First?

• Zero LLM cost for mechanical checks
• 100% accuracy (no hallucination)
• Faster than LLM
• Catches 60-70% of issues without any LLM calls

Checkpoint ID Convention

{SKILL_PREFIX}-{NUMBER}

GH-01  = github-project checkpoint 1
ER-01  = enterprise-readiness checkpoint 1
TC-01  = typo3-conformance checkpoint 1
DC-01  = dependency-compatibility checkpoint 1
PP-01  = pre-push checkpoint 1

Migration Path

1. Phase 1: Add checkpoints to pilot skills (github-project, enterprise-readiness, agents) 2. Phase 2: Test assessment on contexts project 3. Phase 3: Add checkpoints to remaining skills 4. Phase 4: Integrate with CI (automated assessment on PR)

Review & Auto-improve Workflow

The --review and --autoimprove flags close the feedback loop from assessment results back into skill definitions. Instead of just reporting failures, they analyze why checkpoints fail and propose changes to the skills themselves.

Feedback Loop

assessment → failures → categorization → improvement proposals → skill updates

1. Run normal assessment (mechanical + LLM checks) 2. Categorize each failure by root cause 3. Generate improvement proposals 4. Optionally create GitHub issues in skill repos

--review Output

--review produces a categorized failure report. Each failure is classified:

Output format:

{
  "review": {
    "fixable": [
      {"id": "GH-03", "skill": "github-project", "fix_command": "/github-project"}
    ],
    "skill_gaps": [
      {"id": "TC-12", "skill": "typo3-conformance", "gap": "No checkpoint for PSR-14 event usage"}
    ],
    "checkpoint_issues": [
      {"id": "ER-05", "skill": "enterprise-readiness", "issue": "severity:error but only applies to public packages"}
    ]
  }
}

--autoimprove Workflow

--autoimprove extends --review with concrete fix proposals:

1. Autofix phase: Run --autofix for all fixable failures 2. Analysis phase: For remaining failures (skill-gap and checkpoint-issue), analyze root cause 3. Proposal phase: Generate structured improvement proposals 4. Issue phase (with --create-issues): File GitHub issues in the relevant skill repos

Improvement Proposal Format

Each proposal targets a specific file in a skill repo:

{
  "improvements": [
    {
      "skill": "typo3-conformance",
      "category": "skill-gap",
      "checkpoint_id": "TC-12",
      "proposed_action": "add_checkpoint",
      "reason": "No checkpoint verifies PSR-14 event listener registration",
      "target_file": "checkpoints.yaml",
      "suggestion": {
        "id": "TC-15",
        "type": "contains",
        "target": "Configuration/Services.yaml",
        "pattern": "listener",
        "severity": "warning",
        "desc": "Extensions should register event listeners via Services.yaml"
      }
    },
    {
      "skill": "enterprise-readiness",
      "category": "checkpoint-issue",
      "checkpoint_id": "ER-05",
      "proposed_action": "modify_checkpoint",
      "reason": "Severity error is too strict for private extensions",
      "target_file": "checkpoints.yaml",
      "suggestion": {
        "change": "severity",
        "from": "error",
        "to": "warning"
      }
    },
    {
      "skill": "github-project",
      "category": "checkpoint-issue",
      "checkpoint_id": "GH-08",
      "proposed_action": "add_precondition",
      "reason": "Check fails on non-TYPO3 projects that don't use Codecov",
      "target_file": "checkpoints.yaml",
      "suggestion": {
        "add_precondition": {
          "type": "file_exists",
          "target": "composer.json"
        }
      }
    }
  ]
}

Proposed Actions

--create-issues Integration

When --autoimprove --create-issues is used, each improvement proposal becomes a GitHub issue in the target skill's repository:

• Title: [assessment] {proposed_action}: {checkpoint_id} — {short reason}
• Body: Full proposal JSON, evidence from the assessment, and suggested fix
• Labels: assessment, improvement
• Repo: Determined from skill metadata (e.g., netresearch/typo3-conformance-skill)

Issues are only created for skill-gap and checkpoint-issue categories — fixable items are handled by --autofix.

Troubleshooting

"Checkpoint X has null status"

Agent failed to evaluate that checkpoint. Re-run with verbose mode.

"Domain agent returned invalid JSON"

Prompt may need adjustment. Check agent output for parsing errors.

"Scripted check failed unexpectedly"

Verify target path is correct. Check if file exists.

"No checkpoints found for skill X"

Skill doesn't have checkpoints.yaml and no override in front matter. Add checkpoints.yaml following the schema in `references/checkpoints-schema.md`.

Checkpoints YAML Schema

This document defines the schema for checkpoints.yaml files used by the automated-assessment skill.

File Location

Checkpoints files should be placed in skill repositories following the convention-with-override pattern:

1. Convention: If checkpoints.yaml exists in the skill root, it will be auto-discovered 2. Override: If SKILL.md front matter contains checkpoints: path/to/file.yaml, that path is used instead

my-skill/
├── SKILL.md              # Skill content
├── checkpoints.yaml      # Auto-discovered by convention
└── references/
    └── llm-rubric.md     # LLM review prompts (optional)

Schema Version

Current schema version: 1

Full Schema

# checkpoints.yaml
version: 1
skill_id: github-project  # Must match skill name

# Preconditions - evaluated BEFORE any checks; if any fail, skill is skipped
preconditions:
  - type: file_exists
    target: ext_emconf.php
  - type: json_path
    target: composer.json
    pattern: '.type == "typo3-cms-extension"'

# Mechanical checks - run by scripted runner (no LLM needed)
mechanical:
  - id: GH-01                          # Unique ID: SKILL_PREFIX-NUMBER
    type: file_exists                  # Check type (see types below)
    target: README.md                  # File/path to check
    severity: error                    # error | warning | info
    desc: "README.md must exist"       # Human-readable description
    fix_skill: agent-rules             # Optional: skill that can fix this (overrides skill_id)

  - id: GH-02
    type: contains
    target: README.md
    pattern: "codecov.io"              # Pattern to search for
    severity: warning
    desc: "README should have Codecov badge"

  - id: GH-03
    type: regex
    target: .github/workflows/*.yml    # Supports glob patterns
    pattern: "uses: [^@]+@[a-f0-9]{40}"
    severity: error
    desc: "Actions must be pinned to SHA"

# LLM-based reviews - require agent judgment
llm_reviews:
  - id: GH-15
    domain: repo-health                # Groups related reviews
    rubric: references/llm-rubric.md#badge-order  # Markdown anchor
    severity: warning
    desc: "Verify badge ordering follows standard"

  - id: GH-16
    domain: repo-health
    prompt: |                          # Inline prompt (alternative to rubric)
      Verify the README has these sections:
      - Installation/Setup
      - Usage/Configuration
      - Development
      - License
    severity: info
    desc: "README should have standard structure"

Preconditions

Preconditions are evaluated before any mechanical or LLM checks run. They act as guards that determine whether a skill is applicable to the target repository.

Behavior

• All preconditions must pass (AND logic)
• If any precondition fails, the entire skill is skipped (this is not an error)
• Preconditions are not reported as findings -- they silently gate the skill

Fields

Preconditions reuse the same types as mechanical checks but require fewer fields:

No id, severity, or desc fields are needed.

Examples

Skip the skill if the repo is not a TYPO3 extension:

preconditions:
  - type: file_exists
    target: ext_emconf.php
  - type: json_path
    target: composer.json
    pattern: '.type == "typo3-cms-extension"'

Skip the skill if no Go source files exist:

preconditions:
  - type: command
    pattern: "ls *.go 2>/dev/null"

Skip the skill if the repo has no CI configuration:

preconditions:
  - type: file_exists
    target: .github/workflows

Checkpoint ID Convention

{SKILL_PREFIX}-{NUMBER}

GH-01  = github-project checkpoint 1
ER-01  = enterprise-readiness checkpoint 1
TC-01  = typo3-conformance checkpoint 1
TT-01  = typo3-testing checkpoint 1
AG-01  = agents checkpoint 1

Mechanical Check Types

Type Details

file_exists / file_not_exists

- id: GH-01
  type: file_exists
  target: README.md
  severity: error

contains / not_contains

Literal string search (not regex):

- id: GH-02
  type: contains
  target: README.md
  pattern: "codecov.io"
  severity: warning

regex

Extended regex pattern. Target supports glob patterns:

- id: GH-03
  type: regex
  target: .github/workflows/*.yml
  pattern: "uses: [^@]+@[a-f0-9]{40}"
  severity: error

regex_not

Inverse of regex. Passes if the pattern is NOT found in any matching file. Target supports glob patterns:

- id: ER-25
  type: regex_not
  target: .github/workflows/*.yml
  pattern: "uses: [^@]+@v[0-9]"
  severity: error
  desc: "Actions must not use mutable tag references"

json_path

Uses jq to evaluate path. Passes if result is truthy:

- id: PM-01
  type: json_path
  target: composer.json
  pattern: '.require["php"]'
  severity: error

gh_api

GitHub API check (requires gh CLI authentication):

- id: GH-13
  type: gh_api
  endpoint: repos/{owner}/{repo}/topics
  expect_contains: ["typo3", "typo3-extension", "php"]
  severity: error

The {owner} and {repo} placeholders are replaced at runtime.

command

Run arbitrary command, check exit code:

- id: TC-10
  type: command
  pattern: "composer validate --strict"
  severity: error

LLM Review Fields

*Either rubric or prompt is required.

Domain Groups

Related checkpoints are grouped into domains for efficient LLM batching.

These are the currently supported LLM review domains used for automated assessment. Other domain labels mentioned in SKILL.md or the main README (such as git-workflow, docker, ddev, or upgrade) refer to broader skill concepts and are not LLM review domains.

Severity Levels

Fix Skill Override

Mechanical checkpoints support an optional fix_skill field:

fix_skill: agent-rules  # Optional: skill that can fix this checkpoint's failures

When present, fix_skill overrides the default skill_id to fix_command mapping for autofix. This is useful when a checkpoint defined in one skill (e.g., agents) is best fixed by a different skill (e.g., agent-rules which generates AGENTS.md). If not specified, the checkpoint's fix_skill defaults to the file's skill_id.

Resolution Logic

The assessment skill uses this logic to find checkpoints:

def find_checkpoints(skill_path):
    skill_md = read_yaml_front_matter(f"{skill_path}/SKILL.md")

    # Override: explicit path in front matter
    if "checkpoints" in skill_md:
        return f"{skill_path}/{skill_md['checkpoints']}"

    # Convention: checkpoints.yaml in skill root
    convention_path = f"{skill_path}/checkpoints.yaml"
    if file_exists(convention_path):
        return convention_path

    # No checkpoints for this skill
    return None

Example: github-project Checkpoints

version: 1
skill_id: github-project

preconditions:
  - type: file_exists
    target: .github

mechanical:
  - id: GH-01
    type: file_exists
    target: README.md
    severity: error
    desc: "README.md must exist"

  - id: GH-02
    type: file_exists
    target: LICENSE
    severity: error
    desc: "LICENSE file must exist"

  - id: GH-03
    type: file_exists
    target: SECURITY.md
    severity: warning
    desc: "SECURITY.md should exist"

  - id: GH-04
    type: file_exists
    target: .github/CODEOWNERS
    severity: warning
    desc: "CODEOWNERS should exist"

  - id: GH-05
    type: contains
    target: README.md
    pattern: "codecov.io"
    severity: warning
    desc: "README should have Codecov badge"

  - id: GH-06
    type: regex
    target: README.md
    pattern: "img.shields.io.*license"
    severity: warning
    desc: "README should have license badge"

llm_reviews:
  - id: GH-15
    domain: repo-health
    rubric: references/llm-rubric.md#badge-order
    severity: warning
    desc: "Verify badge ordering follows standard"

  - id: GH-16
    domain: repo-health
    prompt: |
      Check README structure for:
      - Installation section
      - Configuration section
      - Development section
      - License section
    severity: info
    desc: "README should have standard sections"

Validation

Run the validator to check your checkpoints.yaml:

~/.claude/skills/automated-assessment/scripts/validate-checkpoints.sh checkpoints.yaml

The validator checks:

• YAML syntax
• Required fields present
• Valid checkpoint types
• Unique IDs
• Severity values

Dependency Compatibility Assessment

When a project's composer.json declares constraints spanning multiple major versions (e.g., ^2.0 || ^3.0), the automated-assessment framework can trigger a dependency compatibility assessment to verify that the codebase actually works with each supported major version.

Trigger Conditions

The assessment activates when ANY of these conditions are met in composer.json:

1. `require` or `require-dev` contains a constraint with || separating major versions 2. `require` uses a range like >=2.0 <5.0 spanning multiple major versions 3. `require` uses ^ or ~ with a base version that is not the latest major

Examples that trigger:

{
  "require": {
    "intervention/image": "^2.0 || ^3.0 || ^4.0",
    "typo3/cms-core": "^12.4 || ^13.4"
  }
}

Assessment Workflow

For each dependency with multi-major-version constraints:

Step 1: Determine Supported Major Versions

Parse the constraint to extract each major version range. For ^2.0 || ^3.0 || ^4.0, the versions are 2.*, 3.*, 4.*.

Step 2: Install Each Major Version

For each major version, run:

# Create a temporary composer.json override or use --with flag
composer require "vendor/package:^MAJOR.0" --no-interaction --dry-run 2>&1

If --dry-run succeeds, proceed with actual install in a temporary directory:

cp -r . /tmp/compat-test-vMAJOR
cd /tmp/compat-test-vMAJOR
composer require "vendor/package:^MAJOR.0" --no-interaction 2>&1

Step 3: Run PHPStan Against Each Version

vendor/bin/phpstan analyse --no-progress --error-format=json 2>&1

Record error count and specific errors per version.

Step 4: Run Unit Tests Against Each Version

vendor/bin/phpunit --no-coverage 2>&1

Record test results (pass/fail/error counts) per version.

Step 5: Compare Results

Flag version-specific failures:

• PHPStan errors that appear only with certain major versions
• Test failures that appear only with certain major versions
• Installation failures for declared-but-incompatible versions

Key Checkpoints

DC-01: Multi-Version Constraint Detection

• Type: command
• What: Detect composer.json constraints spanning multiple major versions
• Severity: info (detection only, not a failure)

DC-02: All Declared Versions Installable

• Type: command
• What: Each major version in the constraint can be installed without conflicts
• Severity: error

DC-03: PHPStan Clean Across All Versions

• Type: llm_review
• What: No version-specific PHPStan errors (methods missing, type mismatches)
• Severity: error

DC-04: Tests Pass Across All Versions

• Type: llm_review
• What: No version-specific test failures
• Severity: error

DC-05: API Compatibility Verified

• Type: llm_review
• What: Code does not call methods that exist in one major version but not another
• Severity: error

Common Failure Patterns

Method Existence Across Versions

// intervention/image v2: $image->encode('webp')
// intervention/image v3: $image->toWebp()
// intervention/image v4: $image->encodeByMediaType('image/webp')

Assessment should flag when code uses methods specific to only one major version without version-conditional logic.

Constructor Signature Changes

// v2: new ImageManager(['driver' => 'gd'])
// v3: new ImageManager(new Driver())
// v4: new ImageManager(new Driver())

PHPStan Ignore Tag Validity

PHPStan @phpstan-ignore tags that suppress errors on one version may mask real issues on another. Assessment should verify that ignore tags are necessary across all supported versions.

Mock Validity in Tests

Test mocks must mock methods that exist on the interface/class across all supported versions. A mock for ->encode() is invalid if the interface no longer has that method in a newer version.

Report Format

{
  "dependency": "intervention/image",
  "constraint": "^2.0 || ^3.0 || ^4.0",
  "versions_tested": [
    {
      "version": "2.*",
      "install": "pass",
      "phpstan_errors": 0,
      "test_result": "pass",
      "test_count": 42
    },
    {
      "version": "3.*",
      "install": "pass",
      "phpstan_errors": 3,
      "test_result": "fail",
      "test_count": 42,
      "failures": ["testImageOptimize: Call to undefined method encode()"]
    },
    {
      "version": "4.*",
      "install": "pass",
      "phpstan_errors": 0,
      "test_result": "pass",
      "test_count": 42
    }
  ],
  "version_specific_issues": [
    "Method encode() does not exist on Intervention\\Image\\Interfaces\\ImageInterface in v3"
  ]
}

Integration with Assessment

This assessment is triggered automatically when the /assess command detects multi-major-version constraints. It can also be invoked directly:

/assess dependency-compatibility

The assessment adds its results to the standard compliance report under the dependency-compatibility domain.

Learning-derived Checkpoints

How `checkpoint` destination materializations from retro-skill translate into entries in a target skill's checkpoints.yaml. This document is the contract between retro-skill (which proposes checkpoints) and automated-assessment-skill (which defines the YAML schema and the verifier runtime).

Authoritative schema source: references/checkpoints-schema.md. This document only describes the routing-from-retro contract; field definitions live in the schema reference.

When this applies

A friction finding routes to the checkpoint destination when all of the following hold:

• The rule is mechanically detectable (regex, file presence, command exit code, JSON/YAML path query)
• It can be checked without LLM reasoning
• It enforces a stable, project-scoped or skill-scoped invariant

If the rule needs context understanding → route to `llm_reviews` (see schema §LLM Review Fields). The retro-skill destination is skill-update only when the change is to the skill's prose, templates, or scripts. LLM-judgable but mechanically expressed rules belong in llm_reviews: inside checkpoints.yaml, not in the mechanical list and not as a skill-update.

If the rule needs to fire pre-action (e.g. pre-commit) → harness-artefact destination instead.

If the rule depends on environment (PHP project, Node project) → consider preconditions: block (schema §Preconditions) rather than a brittle conditional in the check itself.

YAML schema (canonical reference)

The authoritative schema is in references/checkpoints-schema.md. A checkpoint entry has these fields:

- id: <PREFIX>-<NN>                   # e.g. AH-22, SR-15, RT-10
  type: <one of 10 types — see below>
  target: <path or glob>              # Required for most types; not for `command`
  pattern: <regex|jq path|yq path|shell command>
  severity: error | warning | info
  desc: "<what the check enforces>"
  fix_skill: <skill-id>               # Optional; overrides default fix routing

Field names by type

The historical value: field appears in a few older agent-harness AH-* entries; the canonical schema uses pattern: everywhere. New learning-derived checkpoints MUST use pattern: and the schema-canonical names. Don't propagate the value: legacy.

ID convention

<PREFIX> is the target skill's existing checkpoint prefix (e.g. AH- for agent-harness, SR- for skill-repo, RT- for retro-skill). <NN> is the next free number.

There is no universal severity→ID-range mapping. Each skill chooses its own numbering policy:

• Some skills (e.g. agent-harness) band IDs by maturity level: 01-09 = Level 1, 10-19 = Level 2, 20-29 = Level 3.
• Other skills (e.g. skill-repo) run a flat sequence (SR-01 through SR-NN) with no level grouping.

retro-skill MUST:

1. Inspect the target skill's existing checkpoints.yaml. 2. Identify the numbering convention in use. 3. Pick the next free ID matching that convention. 4. Document the choice in the PR body so reviewers see the reasoning.

Format the ID as <PREFIX>-<NN> (single hyphen between prefix and number). Don't write <PREFIX>NN (no hyphen) or <PREFIX>--<NN> (double hyphen).

Severity guidance

Learning-derived checkpoints default to `warning` unless:

• The friction caused upstream failure (CI break, push rejection) → error
• The check is purely informational or optional → info

Severity is independent of ID range. Setting severity wrong is a more common failure than picking the wrong ID.

fix_skill field (routing autofix)

fix_skill: retro

When present, fix_skill overrides the default skill_id for autofix routing. Set it when the checkpoint lives in skill A but the fix is owned by skill B.

For retro-skill-proposed checkpoints, fix_skill is usually appropriate when:

• A friction in skill A pointed to a missing convention enforced by skill B
• E.g. agent-harness/checkpoints.yaml has AH-22: PR template includes retro question with fix_skill: retro (because retro-skill knows how to add the template content)

llm_reviews: routing

A checkpoint that requires LLM judgment lives in llm_reviews: (top-level list in checkpoints.yaml, not in mechanical:). See schema §LLM Review Fields.

Example:

llm_reviews:
  - id: SR-LLM-01
    target: skills/skill-repo/SKILL.md
    prompt: "Does this SKILL.md trigger description include enough specific verbs for an agent to recognize when to invoke it?"
    severity: warning

retro-skill should propose to llm_reviews: (not mechanical:) when the friction is "the file says X but should clearly imply Y" — judgment, not pattern matching.

File structure (complete picture)

A checkpoints.yaml is a complete YAML file with this top-level structure (per schema §Full Schema):

version: 1
skill_id: <skill-name>
preconditions:                       # Optional gate: skill only applies if these hold
  - type: file_exists
    target: ext_emconf.php
mechanical:                          # List of mechanical checks
  - id: <PREFIX>-<NN>
    type: ...
    ...
llm_reviews:                         # Optional list of LLM-judged checks
  - id: <PREFIX>-LLM-<NN>
    ...

retro-skill MUST APPEND to the existing mechanical: (or llm_reviews:) list inside the existing file. Don't emit a fragment of YAML with no version/skill_id — that won't parse as a checkpoints file.

Three common check patterns (with examples)

file_exists — artefact must be present

Use when: friction caused by missing file (template, hook, doc).

- id: AH-23
  type: file_exists
  target: "{.claude/hooks/session-end.json,hooks/session-end.json}"
  severity: info
  desc: "SessionEnd hook configured (optional)"
  fix_skill: retro

regex — file must contain a pattern

Use when: file must contain specific content.

- id: SR-NN
  type: regex
  target: skills/skill-repo/SKILL.md
  pattern: "^## When to use this skill"
  severity: warning
  desc: "SKILL.md has a 'When to use' section"

command — complex multi-file or scripted check

Use when: the check is conditional or spans multiple files.

- id: AH-22
  type: command
  pattern: "grep -liE '(retro|reusable.*pattern)' .github/pull_request_template.md .github/PULL_REQUEST_TEMPLATE/*.md .gitlab/merge_request_templates/*.md 2>/dev/null | head -1 | grep -q ."
  severity: warning
  desc: "PR/MR template includes retro question for agent-authored work"
  fix_skill: retro

The command MUST exit 0 on pass, non-zero on fail, run fast (<2s), have no side effects, and be deterministic (no time/random state).

Workflow for retro-skill

When /retro proposes a checkpoint destination:

1. Locate target skill's checkpoints.yaml (via discovery → repo URL → clone/worktree). 2. Read the existing file: identify skill_id, existing IDs, numbering convention. 3. Choose check type from the 10 canonical types (prefer file_exists/contains/regex over command when possible). 4. Assign next free ID matching the existing convention. 5. Set severity per guidance above. 6. Set fix_skill if the fix is owned by a different skill than the one hosting the checkpoint. 7. Draft the YAML block aligned with references/checkpoints-schema.md. 8. Append to mechanical: (or llm_reviews: if LLM-judged) — do NOT emit a YAML fragment standalone. 9. Document in the PR body: friction signal, why this is mechanically checkable, why this severity, why this type. 10. Run the assessment verifier locally:

    # In automated-assessment-skill or via skill's local validator
    bash skills/automated-assessment/scripts/validate-checkpoints.sh <path-to-checkpoints.yaml>

Confirm the new entry parses and the new checkpoint actually fires for the friction case. 11. Eval stub: if the target skill supports evals, include a regression eval. The eval format is skill-specific — read the target's existing evals/ to match its convention.

Anti-patterns

• Field name `value:` for regex/contains — schema uses pattern:. Don't perpetuate the legacy.
• Severity-to-ID-range mapping — each skill chooses its own ID policy; don't impose error → 10-19, warning → 20-29 universally.
• Standalone YAML fragments — checkpoints live inside a mechanical: or llm_reviews: list within a complete file; retro must append, not emit standalone.
• `mechanical:` for LLM-judgable rules — route to llm_reviews: instead.
• Too narrow — checkpoint that matches one historical case (e.g. matches a unique string). Generalize.
• Too broad — checkpoint that fires on unrelated content. Tighten.
• Missing `fix_skill` when the fix lives elsewhere — autofix will route to the wrong skill.
• No verification step — adding a checkpoint without running the verifier locally to prove it fires.

See also

• references/checkpoints-schema.md — Full YAML schema reference (authoritative; this doc only describes routing)
• references/checkpoint-workflow.md — How the verifier runtime processes checkpoints
• references/verification-patterns.md — Common verification patterns
• retro-skill destination-taxonomy — Where this fits in retro's 6 destinations
• retro-skill classification-heuristic — Friction → checkpoint mapping

Migration Guide: Adding Checkpoints to Skills

This guide explains how to add checkpoints to existing skills for automated assessment.

Overview

The automated-assessment skill uses checkpoints to systematically verify projects against all skills. Checkpoints are either:

1. Mechanical - Script-runnable checks (file exists, contains pattern, etc.) 2. LLM Reviews - Subjective checks requiring agent judgment

Migration Steps

Step 1: Create checkpoints.yaml

In your skill's root directory, create checkpoints.yaml:

version: 1
skill_id: your-skill-name

mechanical:
  - id: YS-01
    type: file_exists
    target: README.md
    severity: error
    desc: "README.md must exist"

llm_reviews:
  - id: YS-10
    domain: repo-health
    prompt: "Verify README follows skill guidelines"
    severity: warning
    desc: "README should follow standards"

Step 2: Define Preconditions

If your skill only applies to certain project types, add a preconditions: block. All preconditions must pass (AND logic) for the skill's checks to run. If any precondition fails, the entire skill is silently skipped.

version: 1
skill_id: your-skill-name

preconditions:
  - type: file_exists
    target: ext_emconf.php          # Only TYPO3 extensions
  - type: json_path
    target: composer.json
    pattern: '.type == "typo3-cms-extension"'

mechanical:
  # ...checks only run if preconditions pass

Preconditions reuse the same check types as mechanical checks (file_exists, contains, regex, json_path, command, etc.) but do not require id, severity, or desc fields.

Common patterns:

If your skill applies universally (e.g., github-project), you can omit preconditions entirely.

Step 3: Choose Checkpoint IDs

Use a consistent prefix based on your skill name:

Step 4: Convert Requirements to Checkpoints

For each requirement in your skill, determine the checkpoint type:

Step 5: Assign Severity Levels

Step 6: Create LLM Rubric (Optional)

For complex LLM reviews, create a rubric file in references/:

# references/llm-rubric.md

## badge-order

### Checkpoint: Verify Badge Ordering

**Requirement:** Badges must appear in standard order.

**Expected Order:**
1. CI badges
2. Security badges
3. Standards badges
4. TER badges

**Evaluation:**
| Status | Condition |
|--------|-----------|
| pass | Badges follow expected order |
| fail | Badges are out of order |
| skip | No badges present |

Reference it in checkpoints.yaml:

llm_reviews:
  - id: GH-15
    domain: repo-health
    rubric: references/llm-rubric.md#badge-order
    severity: warning

Step 7: Test Your Checkpoints

Run the checkpoint runner on a test project:

~/.claude/skills/automated-assessment/scripts/run-checkpoints.sh \
  /path/to/your/skill/checkpoints.yaml \
  /path/to/test/project

Step 8: Verify Checkpoint Coverage

Ensure all key requirements from your skill have corresponding checkpoints:

1. Read through your SKILL.md 2. List all explicit requirements 3. Map each to a checkpoint 4. Check for gaps

Example Migration: github-project Skill

Before (Requirements in SKILL.md)

## Required Files
- README.md (required)
- LICENSE (required)
- SECURITY.md (recommended)
- CODEOWNERS (recommended)

## Badge Requirements
- CI status badge
- Codecov badge
- License badge

After (checkpoints.yaml)

version: 1
skill_id: github-project

mechanical:
  - id: GH-01
    type: file_exists
    target: README.md
    severity: error
    desc: "README.md must exist"

  - id: GH-02
    type: file_exists
    target: LICENSE
    severity: error
    desc: "LICENSE file must exist"

  - id: GH-03
    type: file_exists
    target: SECURITY.md
    severity: warning
    desc: "SECURITY.md should exist"

  - id: GH-04
    type: file_exists
    target: .github/CODEOWNERS
    severity: warning
    desc: "CODEOWNERS should exist"

  - id: GH-05
    type: regex
    target: README.md
    pattern: "github.com/.*/actions/workflows"
    severity: error
    desc: "README should have CI badge"

  - id: GH-06
    type: contains
    target: README.md
    pattern: "codecov.io"
    severity: warning
    desc: "README should have Codecov badge"

Rollout Plan

Phase 1: Pilot Skills

Add checkpoints to 3 skills:

• github-project
• enterprise-readiness
• agents

Phase 2: Test Assessment

Run /assess on the contexts project to validate the system.

Phase 3: Remaining Skills

Add checkpoints to all other skills:

• typo3-conformance
• typo3-testing
• php-modernization
• security-audit
• typo3-docs
• netresearch-branding

Phase 4: CI Integration

Add assessment to CI pipeline to run on every PR.

Troubleshooting

"Checkpoint runner can't find my checkpoints.yaml"

Ensure the file is in the skill root directory, or specify the path in SKILL.md front matter:

---
name: my-skill
checkpoints: custom/path/checkpoints.yaml
---

"My regex pattern isn't matching"

The script uses grep -E (extended regex). Test your pattern:

grep -qE "your-pattern" /path/to/file && echo "Match" || echo "No match"

"JSON path check fails"

The script uses jq. Test your path:

jq -e '.your.path' /path/to/file.json

"LLM review domain not recognized"

Valid LLM review domains currently supported for automated assessment: repo-health, security, code-quality, documentation, dependency-compatibility, pre-push. Other domain labels mentioned in SKILL.md or the main README (such as git-workflow, docker, ddev, or upgrade) refer to broader skill concepts and are not LLM review domains.

Verification Patterns for Checkpoints

Reusable checkpoint patterns that enforce "evidence before assertion" — i.e. no claim of pass/tested/verified without an artifact backing it.

Why this exists

Sessions where an agent declares work "tested and verified" without actually running the tests are the single most expensive failure mode. The cost is a full extra PR cycle, plus lost trust. Cheap to prevent: require a checkable artifact — a CI run URL, a test output file, a lint report — and make the checkpoint fail if that artifact is missing.

Runner compatibility

All patterns below use type: command. The current scripted runner (scripts/run-checkpoints.sh) skips type: gh_api in batch mode, so GitHub-API-backed checks must be written as command with an inline gh api invocation. gh api natively resolves {owner} and {repo} placeholders when executed inside a git repository, so no additional templating is needed.

Portable shell is required — the same checkpoints run on Linux CI and developer macOS machines. GNU-only flags like stat -c and date -d are avoided in favor of find -mtime and gh api-supplied timestamps.

Pattern 1: CI Run Recency

Assert that the repository's default branch has a recent successful CI run.

- id: AA-01
  type: command
  pattern: |
    DEFAULT=$(gh repo view --json defaultBranchRef --jq .defaultBranchRef.name)
    AGE=$(gh api "repos/{owner}/{repo}/actions/runs?branch=$DEFAULT&status=success&per_page=1" \
      --jq '.workflow_runs[0].updated_at | fromdateiso8601 | (now - .)')
    [ -n "$AGE" ] && [ "${AGE%.*}" -lt 86400 ]
  severity: error
  desc: "Default branch must have a successful CI run within the last 24 hours"

Uses the repo's actual default branch (not hardcoded main), and lets jq compute the age in seconds.

Pattern 2: Test-Report Artifact Present

Assert a machine-readable test report exists and was produced in the last day. Uses find -mtime for macOS/Linux portability.

- id: AA-02
  type: command
  pattern: 'find build/logs/junit.xml -maxdepth 0 -mtime -1 2>/dev/null | grep -q .'
  severity: warning
  desc: "Test report must exist and be <24h old"

find -mtime -1 matches files modified within the last day on both GNU and BSD find. The -maxdepth 0 addresses a single named file without recursion.

Pattern 3: Release-Workflow Green Before Tag

For skill and plugin repos: the Release workflow run for the most recent tag must have completed successfully. Prevents the "tag before version-bump PR merged" class of bug from reaching users.

- id: AA-03
  type: command
  pattern: |
    LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null) || exit 0  # no tags yet
    STATUS=$(gh api "repos/{owner}/{repo}/actions/runs?head_sha=$(git rev-list -n1 "$LATEST_TAG")&per_page=5" \
      --jq '[.workflow_runs[] | select(.name == "Release")] | .[0].conclusion')
    [ "$STATUS" = "success" ]
  severity: error
  desc: "Latest tag's Release workflow must be green"

Uses the tag's commit SHA (via git rev-list -n1) plus gh api with {owner}/{repo} placeholders, which gh resolves from the git remote. Exits 0 (pass) if the repo has no tags yet.

Pattern 4: Batch-Operation Dry-Run Artifact

When an operation is scoped to >3 repos, a plan.md or .batch-plan.yml artifact should exist at the orchestrating repo's root, produced before execution. This is the audit trail for "we knew what we were going to do before we did it."

- id: AA-04
  type: file_exists
  target: .batch-plan.yml
  severity: warning
  desc: "Multi-repo batch operations must leave a dry-run plan artifact"

This checkpoint is only meaningful on orchestrator repos. Non-orchestrator repos should either not ship it, or guard with a precondition (e.g. a marker file).

Pattern 5: LLM-Review Rubric — Claim Substantiation

For LLM checkpoints that audit PRs, commits, or release notes, invoke this rubric:

- id: AA-05
  domain: repo-health
  rubric: references/verification-patterns.md#claim-substantiation-rubric
  severity: error
  desc: "Claims of tested/verified must cite an artifact"

The Markdown anchor claim-substantiation-rubric matches the heading below after GitHub-style slugification (lowercase, hyphen-joined).

Claim Substantiation Rubric

Rubric body (loaded by the LLM checkpoint):

For any PR comment, commit message, changelog entry, or release-notes passage containing the words "tested", "verified", "working", "confirmed", or "passes":

>

- PASS if the same PR, commit, or release has a linked CI run URL, test-output gist, artifact upload, or pasted command output that supports the claim.

- FAIL if the claim is bare — no artifact, no run URL, no output, no cited SHA.

>

Record each failure with: location (file:line or PR comment URL), the unsubstantiated claim verbatim, and what would have satisfied the check (e.g. "link to phpunit output").

When to adopt these

• All skill/plugin repos: Pattern 3 (guards releases)
• Skill repos with automated tests: Patterns 1, 2
• Repos that drive multi-repo ops: Pattern 4
• Every skill with an `llm_reviews` section: Pattern 5 rubric

Anti-patterns to avoid in checkpoints