Skill Graph Audit

Skill Graph Audit

Overview

Build a directed graph of Skill(plugin:name) invocations across the marketplace and surface composition patterns: which skills are heavily referenced (hubs), which orchestrate many others (orchestrators), which have no incoming or outgoing references (isolates), and which point at non-existent skills (dangling references).

The federation graph is now derivable from source rather than hand-curated.

When To Use

• Before a documentation pass on skill composition
• After a renaming or retirement to catch broken Skill() references
• During quarterly audits to spot orphaned skills
• When evaluating consolidation candidates (hubs are higher-risk to merge)
• When a new skill's outbound references should be sanity-checked

When NOT To Use

• For per-skill quality scoring, use Skill(abstract:skills-eval) instead
• For frontmatter/structure validation, use Skill(abstract:plugin-review)
• For hook-specific audits, use Skill(abstract:hooks-eval)

Quick Start

python3 plugins/abstract/scripts/skill_graph.py \
  --plugins-root plugins --top-n 10

For machine-readable output:

python3 plugins/abstract/scripts/skill_graph.py \
  --plugins-root plugins --format json --output reports/skill-graph.json

See modules/usage.md for full CLI reference and example workflows.

Core Outputs

See modules/interpretation.md for false-positive guidance and isolation taxonomy.

Dogfood Evidence

This skill itself was scaffolded TDD-first; on first run against plugins/, it caught two genuine dangling refs that the manual audit (2026-04-25) had missed:

• attune:makefile-generation -> abstract:makefile-dogfooder

(script name confused with skill name)

• imbue:karpathy-principles -> spec-kit:speckit-clarify

(command referenced as skill)

Both were converted to correct command-style references in the same session.

Verification

Two ways to validate the audit output is trustworthy:

1. Test-suite correctness check: Run pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.py to confirm extraction, graph construction, ranking, isolate detection, and dangling-ref classification all pass on the current code. The -o addopts= flag bypasses the package-wide coverage gate, which would otherwise fail on a single-file run. 2. Round-trip smoke check: Note the dangling-ref count from a baseline run, fix one or more flagged references, then rerun and verify the count drops by at least the number fixed. If the count does not move, the report is stale or the regex missed a syntax variant.

Exit Criteria

• [ ] The graph builds: skill_graph.py runs against plugins/

without error and emits a node/edge count.

• [ ] Dangling references are classified into bugs, external, and

placeholders (the three Core Outputs rows resolve).

• [ ] Every Dangling: bugs entry is either fixed in the same

session or filed as a tracked issue.

• [ ] pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.py

passes.

• [ ] The round-trip smoke check shows the dangling-ref count drops

by at least the number of references fixed.

Related Skills

• Skill(abstract:skills-eval): per-skill quality scoring
• Skill(abstract:plugin-review): plugin manifest and structure
• Skill(abstract:hooks-eval): hook-specific validation
• Skill(abstract:rules-eval): rules directory validation

References

• Implementation: plugins/abstract/scripts/skill_graph.py
• Tests: plugins/abstract/tests/scripts/test_skill_graph.py
• Composition documentation:

docs/quality-gates.md#skill-level-quality-gate-composition

• Skill role taxonomy: docs/skill-integration-guide.md#skill-role-taxonomy

Interpreting Graph Metrics

Isolate Taxonomy

A skill flagged as "isolate" (zero inbound, zero outbound) is not necessarily broken. Per docs/skill-integration-guide.md#skill-role-taxonomy, three legitimate roles produce zero edges:

1. Library skills

Skills consumed via dependencies: frontmatter from other skills or via Python imports rather than Skill() calls. Example: abstract:shared-patterns. Action: confirm dependencies: field in callers.

2. Entrypoint skills

Skills invoked directly by users via slash commands or by an external orchestrator (e.g. egregore:summon). Example: abstract:plugin-review. Action: confirm a corresponding command file exists in plugins/<plugin>/commands/.

3. Hook-target skills

Skills that hooks redirect to. Example: imbue:proof-of-work. Action: confirm a PreToolUse/PostToolUse hook in plugins/<plugin>/hooks.json references the skill.

A skill that fits none of the three is a true orphan and a candidate for retirement.

Hub Sensitivity

Skills with high inbound count are load-bearing. Before retiring or splitting one:

• Run rg "Skill\\(<plugin>:<name>\\)" plugins/ to enumerate callers
• Open a deprecation issue with at least 30-day notice
• Provide a migration target in the deprecation note

The current top-5 hubs (as of 2026-04-25) are:

1. scribe:slop-detector 2. attune:project-brainstorming 3. sanctum:git-workspace-review 4. attune:project-planning 5. attune:project-specification

Dangling Reference Triage

Cross-Plugin Coupling

A high count of cross-plugin edges (src plugin != dst plugin) is healthy ecosystem behaviour, not a problem. A high count of intra-plugin edges (src plugin == dst plugin) suggests a plugin-internal federation worth documenting in the plugin's README.

Common False Positives

• Skill names in code blocks demonstrating example usage are still

parsed. If documenting a hypothetical skill, use <plugin>:<name> without backticks or surround with <!-- example -->.

• Skill names mentioned in docs/decisions/ outside SKILL.md files

are not parsed (only SKILL.md is the source of truth).

Usage Reference

CLI Flags

python3 plugins/abstract/scripts/skill_graph.py [OPTIONS]

  --plugins-root PATH    Root containing <plugin>/skills/<name>/ tree
                         (default: plugins)
  --top-n INT            Top N hubs/orchestrators to show (default: 10)
  --format {text,json}   Output format (default: text)
  --output PATH          Write to file instead of stdout

Common Workflows

Pre-release dangling-ref check

python3 plugins/abstract/scripts/skill_graph.py \
  --plugins-root plugins --format json --output /tmp/graph.json

python3 -c "
import json
report = json.load(open('/tmp/graph.json'))
bugs = report['dangling_refs']['bugs']
if bugs:
    print(f'BLOCKING: {len(bugs)} dangling refs')
    for b in bugs:
        print(f'  {b[\"source\"]} -> {b[\"target\"]}')
    raise SystemExit(1)
print('OK: 0 internal dangling references')
"

Find consolidation candidates

Hubs with >5 inbound references are core API; orchestrators with >5 outbound references are coordination points. The intersection (hub AND orchestrator) is the federation backbone.

python3 plugins/abstract/scripts/skill_graph.py --top-n 20 \
  | tee /tmp/graph.txt

Update composition documentation

Generate the federation table for docs/quality-gates.md from report JSON instead of curating manually.

Updating External Plugin Allowlist

If a new external plugin is referenced (one not yet in KNOWN_EXTERNAL_PLUGINS), update the constant in plugins/abstract/scripts/skill_graph.py so refs to it are classified as external rather than bugs.

Limitations

• Detects only Skill(plugin:name) invocations. Free-text mentions

in prose are not parsed.

• Self-references (a skill referencing itself) are skipped to avoid

cycles in counts.

• Module-level dependencies: and modules: frontmatter are not

yet treated as edges; see backlog item for planned extension.