Skill Evolution

Skill Evolution Manager

Enables skills to automatically improve based on usage patterns, user edits, and success rates. Provides version control with safe rollback capability.

Overview

• Reviewing how skills are performing across sessions
• Identifying patterns in user edits to skill outputs
• Applying learned improvements to skill templates
• Rolling back problematic skill changes
• Tracking skill version history and success rates

Quick Reference

---

How It Works

The skill evolution system operates in three phases:

COLLECT                    ANALYZE                    ACT
───────                    ───────                    ───
┌─────────────┐           ┌─────────────┐           ┌─────────────┐
│ PostTool    │──────────▶│ Evolution   │──────────▶│ /ork:skill- │
│ Edit        │  patterns │ Analyzer    │ suggest   │ evolution   │
│ Tracker     │           │ Engine      │           │ command     │
└─────────────┘           └─────────────┘           └─────────────┘
     │                          │                          │
     ▼                          ▼                          ▼
┌─────────────┐           ┌─────────────┐           ┌─────────────┐
│ edit-       │           │ evolution-  │           │ versions/   │
│ patterns.   │           │ registry.   │           │ snapshots   │
│ jsonl       │           │ json        │           │             │
└─────────────┘           └─────────────┘           └─────────────┘

Load details: Read("${CLAUDE_SKILL_DIR}/rules/pattern-detection-heuristics.md") for tracked edit patterns and detection regexes. Load details: Read("${CLAUDE_SKILL_DIR}/rules/confidence-scoring.md") for suggestion thresholds.

---

Subcommands

Each subcommand is documented with implementation details, shell commands, and sample output. Load details: Read("${CLAUDE_SKILL_DIR}/references/evolution-commands.md")

Report (Default)

/ork:skill-evolution — Shows evolution report for all tracked skills with usage counts, success rates, and pending suggestions.

Analyze

/ork:skill-evolution analyze <skill-id> — Deep-dives into edit patterns for a specific skill, showing frequency, sample counts, and confidence scores.

Evolve

/ork:skill-evolution evolve <skill-id> — Interactive review of improvement suggestions. Uses AskUserQuestion for each suggestion (Apply / Skip / Reject). Creates version snapshot before applying.

History

/ork:skill-evolution history <skill-id> — Shows version history with performance metrics per version.

Rollback

/ork:skill-evolution rollback <skill-id> <version> — Restores a previous version after confirmation. Current version is backed up automatically.

---

Data Files

---

Auto-Evolution Safety

Load details: Read("${CLAUDE_SKILL_DIR}/rules/auto-evolution-triggers.md") for full safety mechanisms, health monitoring, and trigger criteria.

Key safeguards: version snapshots before changes, auto-alert on >20% success rate drop, human review required, rejected suggestions never re-suggested.

---

References

Load on demand with Read("${CLAUDE_SKILL_DIR}/references/<file>"):

Rules

Load on demand with Read("${CLAUDE_SKILL_DIR}/rules/<file>"):

---

Related Skills

• ork:configure - Configure OrchestKit settings
• ork:doctor - Diagnose OrchestKit issues
• feedback-dashboard - View comprehensive feedback metrics

Evolution Analysis Methodology

Reference guide for understanding how the skill evolution system analyzes patterns and generates suggestions.

Pattern Detection Algorithm

1. Data Collection (PostTool Hook)

When a Write or Edit tool is used after a skill was recently loaded:

IF skill_loaded_within(5_minutes) AND tool IN (Write, Edit):
    content = get_edit_content()
    patterns = detect_patterns(content)
    IF patterns.length > 0:
        log_to_edit_patterns_jsonl(skill_id, patterns)

2. Pattern Matching

The system uses regex patterns to categorize edits:

PATTERN_DETECTORS=(
    ["add_pagination"]="limit.*offset|page.*size|cursor.*pagination|Paginated"
    ["add_rate_limiting"]="rate.?limit|throttl|RateLimiter|requests.?per"
    ["add_caching"]="@cache|cache_key|TTL|redis|memcache|@cached"
    ["add_retry_logic"]="retry|backoff|max_attempts|tenacity|Retry"
    ["add_error_handling"]="try.*catch|except|raise.*Exception|throw.*Error"
    ["add_validation"]="validate|Validator|@validate|Pydantic|Zod|yup"
    ["add_logging"]="logger\.|logging\.|console\.log|winston|pino"
    ["add_types"]=": *(str|int|bool|List|Dict|Optional)|interface\s|type\s.*="
    ["add_auth_check"]="@auth|@require_auth|isAuthenticated|requiresAuth"
    ["add_test_case"]="def test_|it\(|describe\(|expect\(|@pytest"
)

3. Frequency Calculation

For each skill with sufficient usage:

frequency = pattern_count / total_skill_uses

4. Confidence Scoring

Confidence combines frequency with sample size:

confidence = frequency × min(samples / 20, 1.0)

This means:

• 100% frequency with 5 samples = 0.25 confidence (needs more data)
• 100% frequency with 20+ samples = 1.0 confidence (high certainty)
• 70% frequency with 15 samples = 0.53 confidence (moderate)

Suggestion Thresholds

Suggestion Types

Add Suggestions

Generated when users frequently add similar content:

{
  "type": "add",
  "target": "template",
  "pattern": "add_pagination",
  "reason": "85% of users add pagination after using this skill"
}

Remove Suggestions

Generated when users frequently remove generated content:

{
  "type": "remove",
  "target": "template",
  "pattern": "remove_comments",
  "reason": "72% of users remove docstrings from generated code"
}

Analysis Best Practices

1. Wait for sufficient data: Don't act on suggestions until MIN_SAMPLES reached 2. Review high-confidence first: Focus on suggestions with confidence > 0.80 3. Consider context: A pattern may be added for specific use cases only 4. Monitor after changes: Track success rate changes after evolution

Interpreting Results

High-Value Improvements

• Frequency > 80%, Confidence > 0.70
• Pattern is universally applicable
• Easy to add to skill template

Conditional Improvements

• Frequency 50-80%
• May be context-dependent
• Consider adding as optional reference

Skip/Investigate

• Frequency < 50%
• Might be edge case or user preference
• Review individual edit patterns for context

Evolution Subcommand Reference

Detailed implementation and sample output for each subcommand.

---

Subcommand: Report (Default)

Usage: /ork:skill-evolution

Shows evolution report for all tracked skills.

Implementation

# Run the evolution engine report
"${CLAUDE_PROJECT_DIR}/.claude/scripts/evolution-engine.sh" report

Sample Output

Skill Evolution Report
══════════════════════════════════════════════════════════════

Skills Summary:
┌────────────────────────────┬─────────┬─────────┬───────────┬────────────┐
│ Skill                      │ Uses    │ Success │ Avg Edits │ Suggestions│
├────────────────────────────┼─────────┼─────────┼───────────┼────────────┤
│ api-design-framework       │     156 │     94% │       1.8 │          2 │
│ database-schema-designer   │      89 │     91% │       2.1 │          1 │
│ fastapi-patterns           │      67 │     88% │       2.4 │          3 │
└────────────────────────────┴─────────┴─────────┴───────────┴────────────┘

Summary:
  Skills tracked: 3
  Total uses: 312
  Overall success rate: 91%

Top Pending Suggestions:
1. 93% | api-design-framework | add add_pagination
2. 88% | api-design-framework | add add_rate_limiting
3. 85% | fastapi-patterns | add add_error_handling

---

Subcommand: Analyze

Usage: /ork:skill-evolution analyze <skill-id>

Analyzes edit patterns for a specific skill.

Implementation

# Run analysis for specific skill
"${CLAUDE_PROJECT_DIR}/.claude/scripts/evolution-engine.sh" analyze "$SKILL_ID"

Sample Output

Skill Analysis: api-design-framework
────────────────────────────────────
Uses: 156 | Success: 94% | Avg Edits: 1.8

Edit Patterns Detected:
┌──────────────────────────┬─────────┬──────────┬────────────┐
│ Pattern                  │ Freq    │ Samples  │ Confidence │
├──────────────────────────┼─────────┼──────────┼────────────┤
│ add_pagination           │    85%  │ 132/156  │       0.93 │
│ add_rate_limiting        │    72%  │ 112/156  │       0.88 │
│ add_error_handling       │    45%  │  70/156  │       0.56 │
└──────────────────────────┴─────────┴──────────┴────────────┘

Pending Suggestions:
1. 93% conf: ADD add_pagination to template
2. 88% conf: ADD add_rate_limiting to template

Run `/ork:skill-evolution evolve api-design-framework` to review

---

Subcommand: Evolve

Usage: /ork:skill-evolution evolve <skill-id>

Interactive review and application of improvement suggestions.

Implementation

1. Get Suggestions:

SUGGESTIONS=$("${CLAUDE_PROJECT_DIR}/.claude/scripts/evolution-engine.sh" suggest "$SKILL_ID")

2. For Each Suggestion, Present Interactive Options:

Use AskUserQuestion to let the user decide on each suggestion:

{
  "questions": [{
    "question": "Apply suggestion: ADD add_pagination to template? (93% confidence, 132/156 users add this)",
    "header": "Evolution",
    "options": [
      {"label": "Apply", "description": "Add this pattern to the skill template"},
      {"label": "Skip", "description": "Skip for now, ask again later"},
      {"label": "Reject", "description": "Never suggest this again"}
    ],
    "multiSelect": false
  }]
}

3. On Apply:

• Create version snapshot first
• Apply the suggestion to skill files
• Update evolution registry

4. On Reject:

• Mark suggestion as rejected in registry
• Will not be suggested again

Applying Suggestions

When a user accepts a suggestion, the implementation depends on the suggestion type:

For `add` suggestions to templates:

• Add the pattern to the skill's template files
• Update SKILL.md with new guidance

For `add` suggestions to references:

• Create new reference file in references/ directory

For `remove` suggestions:

• Remove the identified content
• Archive in version snapshot first

---

Subcommand: History

Usage: /ork:skill-evolution history <skill-id>

Shows version history with performance metrics.

Implementation

# Run version manager list
"${CLAUDE_PROJECT_DIR}/.claude/scripts/version-manager.sh" list "$SKILL_ID"

Sample Output

Version History: api-design-framework
══════════════════════════════════════════════════════════════

Current Version: 1.2.0

┌─────────┬────────────┬─────────┬───────┬───────────┬────────────────────────────┐
│ Version │ Date       │ Success │ Uses  │ Avg Edits │ Changelog                  │
├─────────┼────────────┼─────────┼───────┼───────────┼────────────────────────────┤
│ 1.2.0   │ 2026-01-14 │    94%  │   156 │       1.8 │ Added pagination pattern   │
│ 1.1.0   │ 2026-01-05 │    89%  │    80 │       2.3 │ Added error handling ref   │
│ 1.0.0   │ 2025-11-01 │    78%  │    45 │       3.2 │ Initial release            │
└─────────┴────────────┴─────────┴───────┴───────────┴────────────────────────────┘

---

Subcommand: Rollback

Usage: /ork:skill-evolution rollback <skill-id> <version>

Restores a skill to a previous version.

Implementation

1. Confirm with User:

Use AskUserQuestion for confirmation:

{
  "questions": [{
    "question": "Rollback api-design-framework from 1.2.0 to 1.0.0? Current version will be backed up.",
    "header": "Rollback",
    "options": [
      {"label": "Confirm Rollback", "description": "Restore version 1.0.0"},
      {"label": "Cancel", "description": "Keep current version"}
    ],
    "multiSelect": false
  }]
}

2. On Confirm:

"${CLAUDE_PROJECT_DIR}/.claude/scripts/version-manager.sh" restore "$SKILL_ID" "$VERSION"

3. Report Result:

Restored api-design-framework to version 1.0.0
Previous version backed up to: versions/.backup-1.2.0-1736867234

Storage Patterns: Rolling Logbook vs Index-Per-Entry

When a skill (or a project's .claude/rules/*.md files) needs to accumulate state across sessions — decisions, observations, patterns, knowledge — there are two storage patterns. Pick wrong and Claude Code's 40k-char auto-load threshold ambushes you 3-6 months later.

TL;DR

If unsure, default to index-per-entry — bounded by construction.

Pattern 1 — Rolling Logbook

Single Markdown file appended forever.

.claude/rules/recent-decisions.md
  # Recent Project Decisions
  ## 2026-04-15 — Use Postgres not MongoDB
  ## 2026-04-22 — Brainstorm: dual-write
  ## ... (one entry every week, no upper bound)

Strengths

• Trivial to write: >> file.md and you're done
• Operator-readable as chronological narrative
• One file means one place to check in / out of cache
• Diffs cleanly in PRs (additions only)

Weaknesses

• Grows unbounded. No mechanism stops you at any size.
• CC auto-loads everything in .claude/rules/*.md into every <system-reminder>. At 40,000 chars CC emits a yellow warning, but by then you've burned that context on every prompt for weeks.
• Stale entries pollute the loaded context (Q1 2024 decisions for a Q4 2026 session — irrelevant but billed).
• Recursive search/replace breaks (one wrong-line edit corrupts a 30k file).

Concrete failure case: yonatan-hq/platform/.claude/rules/recent-decisions.md ballooned to 53.8k chars in seven months. CC flagged it as "Large file will impact performance" — every session paid ~14k extra context tokens before the operator noticed.

Pattern 2 — Index-Per-Entry

One small index file with one bullet per entry; individual entries live in sibling files loaded on-demand via Read.

.claude/rules/                            ←  scanned by CC's auto-loader
├── index.md                              ←  ≤ 200 lines, only file auto-loaded
│   - [Use Postgres not MongoDB](decisions/2026-04-15-postgres.md) — chose Postgres for jsonb support
│   - [Dual-write analytics](decisions/2026-04-22-dual-write.md) — HTTP sink alongside local JSONL
│   ...
└── decisions/                            ←  NOT auto-loaded (subdirectory)
    ├── 2026-04-15-postgres.md            ←  loaded only when relevant via Read
    ├── 2026-04-22-dual-write.md
    └── ...

Strengths

• Bounded. Index grows by one line per entry — even 500 entries is ~50 lines.
• On-demand load. Operator (or Claude) reads the specific entry that matters; the other 499 stay on disk.
• CC's auto-loader stops at the subdirectory boundary — by convention CC globs *.md at the rules root, not recursive.
• Each entry is independently meaningful, addressable, and editable.
• Old entries don't pollute current context.

Weaknesses

• Two-step writes: append to index + write new file. Five extra seconds per entry. Bash function or skill can hide this.
• Two-step reads: scan index, then Read the relevant file. Costs one extra tool call.
• More PR-diff noise (one new file per entry vs append to one).
• Filenames must be unique and well-chosen — bad naming kills the on-demand pattern.

Migration Path

When a rolling logbook crosses the 30k-char mark, migrate proactively:

1. Move the rolling file aside: git mv recent-decisions.md decisions/_legacy-rolling.md 2. Create decisions/ subdirectory. 3. Split entries one-per-file. A small script can split on ## headers:

   csplit -k recent-decisions.md '/^## /' '{*}'

4. Create index.md with one bullet per file. 5. Update any skill/hook references to point at the index.

The _legacy-rolling.md stays accessible via Read but won't auto-load (not at .claude/rules/*.md root).

Pattern selection by skill

Detection

The lifecycle/rules-size-check hook (#1815) warns at 35k chars (WARN) and 38k chars (CRITICAL) when a .claude/rules/*.md file would auto-load above CC's 40k threshold. Operator-facing stderr signal on every SessionStart — gives you ~5k chars of runway before CC starts complaining.

When the rolling pattern is still right

Don't blanket-reject rolling logbooks. They're correct when:

• The file has a natural upper bound (e.g., "this lists the 12 active milestones — milestones don't accumulate, they close")
• Total bytes are known to stay under 20k chars even at 5× growth
• Readability as a single narrative is the primary read mode

The 40k-char cliff isn't a hard rule — it's a heuristic for "auto-loaded into every prompt." If your file isn't at .claude/rules/*.md root, it isn't auto-loaded and the trade-off shifts.

Related

• lifecycle/rules-size-check (hook, #1815) — pre-flight warning when a file approaches the cliff
• src/skills/CONTRIBUTING-SKILLS.md#storage-patterns — short pointer to this reference
• src/skills/memory/ — index-per-entry pattern done well (canonical reference implementation)

Version Management Guide

Reference guide for managing skill versions with safe rollback capability.

Version Structure

Each skill can have versioned snapshots stored in:

skills/<category>/<skill-name>/
├── SKILL.md                 # Current version
├── SKILL.md        # Current metadata
├── references/              # Current references
├── scripts/               # Current templates
└── versions/
    ├── manifest.json        # Version history metadata
    ├── 1.0.0/
    │   ├── SKILL.md
    │   ├── SKILL.md
    │   ├── references/
    │   └── CHANGELOG.md
    └── 1.1.0/
        ├── SKILL.md
        ├── SKILL.md
        ├── references/
        └── CHANGELOG.md

Manifest Schema

The manifest.json tracks version history:

{
  "$schema": "../../../../../../.claude/schemas/skill-evolution.schema.json",
  "skillId": "api-design-framework",
  "currentVersion": "1.2.0",
  "versions": [
    {
      "version": "1.0.0",
      "date": "2025-11-01",
      "successRate": 0.78,
      "uses": 45,
      "avgEdits": 3.2,
      "changelog": "Initial release"
    },
    {
      "version": "1.1.0",
      "date": "2026-01-05",
      "successRate": 0.89,
      "uses": 80,
      "avgEdits": 1.8,
      "changelog": "Added pagination pattern (85% users added manually)"
    }
  ],
  "suggestions": [],
  "editPatterns": {},
  "lastAnalyzed": "2026-01-14T10:30:00Z"
}

Versioning Workflow

Creating a Version

1. Before making changes, create a version snapshot:

   version-manager.sh create <skill-id> "Description of changes"

2. The system:

• Bumps version number (patch by default)
• Copies current files to versions/<new-version>/
• Records current metrics in manifest
• Creates CHANGELOG.md

Comparing Versions

Compare two versions to see what changed:

version-manager.sh diff <skill-id> 1.0.0 1.1.0

Shows:

• File differences (unified diff)
• Metrics comparison (success rate, uses, avg edits)

Restoring a Version

If a change causes problems, rollback:

version-manager.sh restore <skill-id> <version>

The system: 1. Backs up current version to .backup-<version>-<timestamp> 2. Copies snapshot files to skill root 3. Updates manifest with rollback entry

Automatic Safety Checks

Rollback Triggers

The system monitors for:

Health Check Integration

The posttool hooks monitor skill health:

check_skill_health() {
    local skill_id="$1"
    local current_rate=$(get_recent_success_rate "$skill_id" 10)
    local baseline_rate=$(get_version_baseline "$skill_id")

    if (( $(echo "$baseline_rate - $current_rate > 0.20" | bc -l) )); then
        echo "WARNING: $skill_id dropped from ${baseline_rate} to ${current_rate}"
    fi
}

Best Practices

When to Create Versions

• Before applying evolution suggestions
• Before major skill modifications
• After validating improvements work well
• At regular intervals (weekly/monthly) for active skills

Version Naming

Use semantic versioning:

• Major (2.0.0): Breaking changes to skill behavior
• Minor (1.1.0): New features/patterns added
• Patch (1.0.1): Bug fixes, minor improvements

Cleanup Policy

• Keep last 5 versions minimum
• Archive versions older than 90 days
• Never delete versions with good metrics (baseline references)

Metrics Interpretation

Success Rate Trends

Average Edits Trends

Recovery Scenarios

Accidental Breaking Change

# 1. Check history
version-manager.sh list <skill-id>

# 2. Find last good version
version-manager.sh metrics <skill-id>

# 3. Restore
version-manager.sh restore <skill-id> 1.1.0

Gradual Degradation

# 1. Compare versions
version-manager.sh diff <skill-id> 1.0.0 1.2.0

# 2. Identify problematic changes
# 3. Create new version fixing issues

Skill Evolution Rules

Index

Auto-Evolution Safety & Trigger Criteria

Safety Mechanisms

1. Version Snapshots: Always created before changes 2. Rollback Triggers: Auto-alert if success rate drops >20% 3. Human Review: High-confidence suggestions require approval 4. Rejection Memory: Rejected suggestions are never re-suggested

Health Monitoring

The system monitors skill health and can trigger warnings:

WARNING: api-design-framework success rate dropped from 94% to 71%
Consider: /ork:skill-evolution rollback api-design-framework 1.1.0

Incorrect:

# Auto-apply pattern after 2 uses, no rollback tracking
confidence: 60%, samples: 2 → APPLY

Correct:

# Require minimum samples and high confidence before suggesting
confidence: 85%, samples: 8 → SUGGEST (requires human approval)
confidence: 60%, samples: 2 → TRACK ONLY (below threshold)

When Auto-Evolution Activates

• Pattern frequency exceeds the Add Threshold (70%)
• At least Minimum Samples (5) uses recorded
• No prior rejection for the same pattern on the same skill
• Current skill version success rate is stable (no recent drops)

When Rollback Is Triggered

• Success rate drops more than 20% after an evolution
• Alert is surfaced in the next report or analyze invocation
• User is prompted to rollback via AskUserQuestion

Confidence Scoring & Suggestion Thresholds

Thresholds

Confidence Calculation

Confidence is calculated as the ratio of users who apply a pattern to total uses:

confidence = pattern_frequency / total_uses

• Below 70%: Pattern tracked but no suggestion generated
• 70%-84%: Suggestion generated, requires human approval via evolve subcommand
• 85%+: Auto-apply eligible (still requires human confirmation via AskUserQuestion)

Incorrect:

# Apply pattern with only 2 data points
pattern_frequency: 2/3 (67%) → auto-apply  # Too few samples, unreliable

Correct:

# Wait for minimum samples before generating suggestions
pattern_frequency: 6/8 (75%) → suggest (requires approval)
pattern_frequency: 2/3 (67%) → track only (below 5 minimum samples)

Suggestion States

Suggestions progress through: pending → applied | rejected

• Applied: Pattern added to skill template, version bumped
• Rejected: Marked in registry, never re-suggested for this skill

Edit Pattern Detection Heuristics

The system tracks these common edit patterns users apply after skill output:

Incorrect:

# Generic pattern — matches too broadly
{"pattern": "add_.*", "regex": ".*"}  # Matches everything, useless signal

Correct:

# Specific pattern with focused regex
{"pattern": "add_pagination", "regex": r"limit.*offset|cursor.*pagination"}

How Detection Works

The PostTool Edit Tracker hook monitors file edits after skill invocations. When a user edits skill output, the edit is classified against the patterns above using regex matching. Results are appended to .claude/feedback/edit-patterns.jsonl.