Evaluation Framework

Table of Contents

• Overview
• When to Use
• Core Pattern
• 1. Define Criteria
• 2. Score Each Criterion
• 3. Calculate Weighted Total
• 4. Apply Decision Thresholds
• Quick Start
• Define Your Evaluation
• Example: Code Review Evaluation
• Evaluation Workflow
• Common Use Cases
• Integration Pattern
• Detailed Resources
• Exit Criteria

Evaluation Framework

Overview

A generic framework for weighted scoring and threshold-based decision making. Provides reusable patterns for evaluating any artifact against configurable criteria with consistent scoring methodology.

This framework abstracts the common pattern of: define criteria → assign weights → score against criteria → apply thresholds → make decisions.

When To Use

• Implementing quality gates or evaluation rubrics
• Building scoring systems for artifacts, proposals, or submissions
• Need consistent evaluation methodology across different domains
• Want threshold-based automated decision making
• Creating assessment tools with weighted criteria

When NOT To Use

• Simple pass/fail without scoring needs

Core Pattern

1. Define Criteria

criteria:
  - name: criterion_name
    weight: 0.30          # 30% of total score
    description: What this measures
    scoring_guide:
      90-100: Exceptional
      70-89: Strong
      50-69: Acceptable
      30-49: Weak
      0-29: Poor

Verification: Run the command with --help flag to verify availability.

2. Score Each Criterion

scores = {
    "criterion_1": 85,  # Out of 100
    "criterion_2": 92,
    "criterion_3": 78,
}

Verification: Run the command with --help flag to verify availability.

3. Calculate Weighted Total

total = sum(score * weights[criterion] for criterion, score in scores.items())
# Example: (85 × 0.30) + (92 × 0.40) + (78 × 0.30) = 85.5

Verification: Run the command with --help flag to verify availability.

4. Apply Decision Thresholds

thresholds:
  80-100: Accept with priority
  60-79: Accept with conditions
  40-59: Review required
  20-39: Reject with feedback
  0-19: Reject

Verification: Run the command with --help flag to verify availability.

Quick Start

Define Your Evaluation

1. Identify criteria: What aspects matter for your domain? 2. Assign weights: Which criteria are most important? (sum to 1.0) 3. Create scoring guides: What does each score range mean? 4. Set thresholds: What total scores trigger which decisions?

Example: Code Review Evaluation

criteria:
  correctness: {weight: 0.40, description: Does code work as intended?}
  maintainability: {weight: 0.25, description: Is it readable?}
  performance: {weight: 0.20, description: Meets performance needs?}
  testing: {weight: 0.15, description: Tests detailed?}

thresholds:
  85-100: Approve immediately
  70-84: Approve with minor feedback
  50-69: Request changes
  0-49: Reject, major issues

Verification: Run pytest -v to verify tests pass.

Evaluation Workflow

**Verification:** Run the command with `--help` flag to verify availability.
1. Review artifact against each criterion
2. Assign 0-100 score for each criterion
3. Calculate: total = Σ(score × weight)
4. Compare total to thresholds
5. Take action based on threshold range

Verification: Run the command with --help flag to verify availability.

Common Use Cases

Quality Gates: Code review, PR approval, release readiness Content Evaluation: Document quality, knowledge intake, skill assessment Resource Allocation: Backlog prioritization, investment decisions, triage

Integration Pattern

# In your skill's frontmatter
dependencies: [leyline:evaluation-framework]

Verification: Run the command with --help flag to verify availability.

Then customize the framework for your domain:

• Define domain-specific criteria
• Set appropriate weights for your context
• Establish meaningful thresholds
• Document what each score range means

Detailed Resources

• Scoring Patterns: See modules/scoring-patterns.md for detailed methodology
• Decision Thresholds: See modules/decision-thresholds.md for threshold design

Exit Criteria

• [ ] Criteria defined with clear descriptions
• [ ] Weights assigned and sum to 1.0
• [ ] Scoring guides documented for each criterion
• [ ] Thresholds mapped to specific actions
• [ ] Evaluation process documented and reproducible

Integration Guide

How to integrate the evaluation-framework skill into your plugin.

For memory-palace (Knowledge Intake)

The memory-palace evaluation rubric can now depend on this shared framework:

# In knowledge-intake/modules/evaluation-rubric.md
---
name: evaluation-rubric
dependencies: [leyline:evaluation-framework]
---

# Knowledge Evaluation Rubric

Based on the [evaluation-framework](leyline:evaluation-framework) with domain-specific criteria for knowledge intake.

## Criteria (following evaluation-framework pattern)

### 1. Novelty (25%)
See [scoring-patterns](leyline:evaluation-framework/modules/scoring-patterns.md) for methodology.

[Rest of domain-specific details...]

For abstract (Quality Metrics)

The abstract quality-metrics module can reference this framework:

# In skills-eval/modules/quality-metrics.md
---
name: quality-metrics
dependencies: [leyline:evaluation-framework]
---

# Quality Metrics Framework

Based on [evaluation-framework](leyline:evaluation-framework) for skill quality assessment.

## Scoring Categories (following evaluation-framework pattern)

### Structure Compliance (0-100)
Uses weighted scoring from [evaluation-framework](leyline:evaluation-framework).

[Rest of domain-specific details...]

Benefits of Integration

Reduced Duplication

• Common scoring methodology in one place
• Shared threshold patterns
• Single source of truth for evaluation concepts

Consistency

• Same terminology across plugins
• Consistent scoring scales
• Unified decision-making patterns

Maintainability

• Update evaluation patterns once
• All consumers benefit from improvements
• Clear dependency chain

Migration Path

1. Add Dependency: Update frontmatter to include leyline:evaluation-framework 2. Reference Core Patterns: Link to framework for common concepts 3. Focus on Domain: Keep only domain-specific details in your skill 4. Remove Duplication: Delete explanations now in framework

Example: Before and After

Before (Duplicated)

# My Evaluation

## Weighted Scoring

We use a weighted scoring system where each criterion has a weight...
[300 lines of generic explanation]

## Domain-Specific Criteria
[50 lines of actual domain logic]

After (DRY with Framework)

# My Evaluation

Uses [evaluation-framework](leyline:evaluation-framework) for weighted scoring.

## Domain-Specific Criteria
[50 lines of actual domain logic with references to framework patterns]

Result: 350 lines → 60 lines, clearer focus on domain logic.

Decision Thresholds

Patterns and best practices for designing effective threshold-based decision frameworks.

Core Concepts

What Are Thresholds?

Thresholds are score ranges that map to specific decisions or actions. They transform continuous scores into discrete decision points.

Score Range → Decision → Action
80-100      → Accept   → Deploy immediately
60-79       → Review   → Manual approval needed
0-59        → Reject   → Send back for revision

Why Use Thresholds?

• Consistency: Same score always gets same decision
• Automation: Enable automated decision-making
• Clarity: Clear criteria for each outcome
• Accountability: Documented decision logic

Threshold Design Patterns

Binary Thresholds

Simplest pattern - pass or fail:

thresholds:
  70-100: Pass
  0-69:   Fail

Use when:

• Decision is truly binary (deploy/don't deploy)
• No middle ground exists
• Automation is critical

Multi-Tier Thresholds

Multiple decision levels with different actions:

thresholds:
  90-100: Excellent - Fast track
  75-89:  Good - Standard process
  60-74:  Fair - Additional review
  40-59:  Poor - Major revisions needed
  0-39:   Fail - Reject

Use when:

• Different quality levels warrant different treatments
• Resources should be allocated differently
• Multiple stakeholders with different concerns

Conditional Thresholds

Thresholds that depend on context:

thresholds:
  production_deployment:
    90-100: Auto-deploy
    80-89:  Deploy with manual verification
    0-79:   Block deployment

  development_deployment:
    60-100: Auto-deploy
    0-59:   Block deployment

Use when:

• Risk tolerance varies by context
• Different environments have different requirements
• Stakeholder needs differ

Compound Thresholds

Multiple criteria must meet thresholds:

decision_rules:
  approve_if:
    - overall_score >= 80
    - all_critical_criteria >= 70
    - no_blocking_issues

  reject_if:
    - overall_score < 60
    - OR any_critical_criterion < 50
    - OR has_security_vulnerability

Use when:

• Single criteria can veto a decision
• Minimum bars exist across dimensions
• Safety-critical decisions

Setting Threshold Levels

Data-Driven Approach

Use historical data to inform thresholds:

# Analyze past evaluations
historical_scores = [...]
percentiles = {
    "p90": 87,  # 90th percentile
    "p75": 78,  # 75th percentile
    "p50": 65,  # median
    "p25": 52,  # 25th percentile
}

# Set thresholds based on desired selectivity
thresholds = {
    "excellent": percentiles["p90"],  # Top 10%
    "good": percentiles["p75"],       # Top 25%
    "acceptable": percentiles["p50"],  # Top 50%
}

Risk-Based Approach

Set thresholds based on acceptable risk levels:

# High-risk decision (production deployment)
thresholds:
  90-100: Proceed - minimal risk
  80-89:  Caution - some risk, monitor closely
  0-79:   Block - unacceptable risk

# Low-risk decision (internal tool)
thresholds:
  70-100: Proceed - acceptable risk
  50-69:  Proceed with warning
  0-49:   Review - may still proceed

Stakeholder-Driven Approach

Align thresholds with stakeholder expectations:

# Engineering standards
technical_thresholds:
  maintain: 85
  acceptable: 70

# Business requirements
business_thresholds:
  launch_ready: 75
  beta_ready: 60

Action Mapping

Explicit Actions

Map each threshold range to specific, actionable steps:

80-100:
  decision: Approve
  actions:
    - Auto-merge PR
    - Deploy to production
    - Notify stakeholders
    - Update metrics

60-79:
  decision: Conditional Approve
  actions:
    - Request senior review
    - Deploy to staging
    - Schedule follow-up
    - Document concerns

40-59:
  decision: Request Changes
  actions:
    - Block merge
    - Create detailed feedback
    - Assign back to author
    - Set expected timeline

0-39:
  decision: Reject
  actions:
    - Close PR
    - Document reasons
    - Suggest alternatives
    - Offer guidance

Graduated Responses

Scale response intensity with score:

critical_issues_by_threshold:
  0-39:   Block completely
  40-59:  Block with waiver option
  60-74:  Warning, proceed if acknowledged
  75-89:  Info only, no action required
  90-100: No issues detected

Escalation Paths

Define who decides at each threshold:

approval_authority:
  90-100: Automated approval
  80-89:  Team lead approval
  70-79:  Manager approval
  60-69:  Director approval required
  0-59:   Automatic rejection

Threshold Validation

Testing Threshold Effectiveness

# Evaluate threshold performance
def validate_thresholds(historical_data, thresholds):
    metrics = {
        "false_positives": 0,  # Approved but failed
        "false_negatives": 0,  # Rejected but would succeed
        "true_positives": 0,   # Approved and succeeded
        "true_negatives": 0,   # Rejected correctly
    }

    for case in historical_data:
        decision = apply_threshold(case.score, thresholds)
        actual = case.actual_outcome

        if decision == "approve" and actual == "success":
            metrics["true_positives"] += 1
        elif decision == "approve" and actual == "failure":
            metrics["false_positives"] += 1
        # ... etc

    return metrics

Calibration Over Time

Thresholds should evolve:

threshold_evolution:
  initial: 70    # Start conservative
  after_3_months: 75   # Tighten as quality improves
  after_6_months: 80   # Continue raising bar
  target: 85     # Long-term goal

Special Cases

Veto Criteria

Some criteria can override total score:

def apply_decision(scores, weights):
    total = calculate_weighted_score(scores, weights)

    # Veto conditions
    if scores["security"] < 60:
        return "REJECT - Security threshold not met"

    if scores["legal_compliance"] < 80:
        return "REJECT - Compliance requirement not met"

    # Standard threshold logic
    if total >= 80:
        return "APPROVE"
    elif total >= 60:
        return "REVIEW"
    else:
        return "REJECT"

Confidence Intervals

Account for scoring uncertainty:

score_with_confidence:
  point_estimate: 75
  confidence_interval: [70, 80]
  confidence_level: 0.95

threshold_application:
  pessimistic: use_lower_bound(70)  # Conservative
  expected: use_point_estimate(75)   # Balanced
  optimistic: use_upper_bound(80)    # Aggressive

Grace Periods

Allow time for improvement:

initial_evaluation:
  score: 65
  threshold: 70
  decision: Provisional acceptance with 30-day improvement plan

follow_up_evaluation:
  score: 72
  decision: Full acceptance

Common Patterns

Quality Gates

gate_sequence:
  gate_1_unit_tests:
    threshold: 80
    blocker: true

  gate_2_integration_tests:
    threshold: 75
    blocker: true

  gate_3_performance:
    threshold: 70
    blocker: false  # Warning only

Progressive Disclosure

# Initial quick check
rapid_assessment:
  threshold: 50
  action: If pass, proceed to detailed evaluation

# Detailed evaluation
full_assessment:
  threshold: 75
  action: If pass, approve

Hysteresis

Different thresholds for entering vs. exiting a state:

status_transitions:
  promote_to_production:
    threshold: 85

  remain_in_production:
    threshold: 70  # Lower bar to stay than to enter

  demote_from_production:
    threshold: 69

Best Practices

Clear Boundaries

Avoid overlapping ranges:

• Good: 80-100, 60-79, 0-59
• Bad: 80-100, 60-80, 0-60

Document Rationale

threshold: 75
rationale: |
  Historical data shows 75+ correlates with 95% success rate
  in production. This balances velocity with quality.
  Reviewed quarterly based on actual outcomes.

Make Thresholds Visible

# Include in reports
evaluation_result = {
    "score": 78,
    "threshold_range": "60-79",
    "decision": "Conditional Approve",
    "distance_to_next_tier": 2,  # Points to reach 80
    "improvements_needed": [
        "Increase test coverage by 5%",
        "Resolve 2 remaining linting issues"
    ]
}

Plan for Edge Cases

edge_case_handling:
  score_exactly_on_boundary:
    score: 80
    rule: "Round up - belongs to higher tier"

  missing_criterion_data:
    rule: "Use conservative estimate or require completion"

  partial_evaluations:
    rule: "Scale thresholds proportionally to evaluated criteria"

Validation Checklist

Before deploying threshold-based decisions:

• [ ] Thresholds cover full 0-100 range
• [ ] No gaps or overlaps between ranges
• [ ] Each range maps to specific action
• [ ] Actions are documented and achievable
• [ ] Veto criteria clearly defined
• [ ] Edge cases handled explicitly
• [ ] Rationale documented for each threshold
• [ ] Review process established
• [ ] Metrics tracked for validation
• [ ] Escalation paths defined

Evaluation Rubric

Concrete rubric templates you can copy and adapt. A rubric fixes four things before any artifact is scored: the dimensions (what you measure), the scale (how you score each dimension), the weights (how dimensions combine), and the aggregation rule (how a final number is produced).

Anatomy of a Rubric

Every reusable rubric has the same five parts. Skip any of them and reviewers drift.

The first four are below; decision mapping is in modules/decision-thresholds.md.

Picking a Scale

Three scales cover most cases. Pick one per rubric and do not mix.

0-2 Ordinal (Pass / Partial / Fail)

Use when the dimension is binary in spirit but you want to allow partial credit. Cheap to score, hard to game.

Best for: gates, checklists, presence-of-evidence dimensions ("has tests", "has version pin").

1-5 Likert

Use when reviewers must distinguish "good" from "great". Five points let raters separate adequate from excellent without false precision. Anchor every odd score (1, 3, 5) with a worked example.

Best for: skill quality, doc quality, code review.

0-100 Continuous

Use when you have measurable inputs (coverage percent, benchmark numbers) or need fine differentiation across many items. More expensive to calibrate; risks false precision if anchors are vague.

Best for: large rankings, ML evals, automated scoring.

Rule of thumb: if you cannot tell the difference between score N and N+1 without a reference example, your scale is too granular.

Template 1: Skill Quality Rubric

Drop-in rubric for evaluating Claude Code skills. Scale: 1-5.

rubric: skill-quality
scale: 1-5
dimensions:
  activation_clarity:
    weight: 0.20
    question: "Does the skill activate at the right moment?"
    anchors:
      5: "Trigger conditions list 3+ phrases; tested in subagent"
      3: "Trigger described prose-only; activates most cases"
      1: "Activation unclear; depends on user remembering name"
  evidence_grounding:
    weight: 0.20
    question: "Are claims backed by tests, sources, or runs?"
    anchors:
      5: "Every recommendation cites a working example or test"
      3: "Claims are plausible; partial evidence"
      1: "Assertions with no proof; pattern-matched advice"
  scope_discipline:
    weight: 0.15
    question: "Does it stay in its lane?"
    anchors:
      5: "Explicit when-not-to-use; defers to siblings"
      3: "Stays in scope but no deferral guidance"
      1: "Overlaps with other skills; no boundaries"
  token_economy:
    weight: 0.15
    question: "Is the skill body small and modules paged?"
    anchors:
      5: "SKILL.md under 500 lines; modules loaded on demand"
      3: "Single file 500-1500 lines; no progressive loading"
      1: "Monolithic 2000+ lines; loaded eagerly"
  testability:
    weight: 0.15
    question: "Can the skill be exercised by a subagent test?"
    anchors:
      5: "RED/GREEN test exists; documented invocation"
      3: "Manual test plan exists; not automated"
      1: "No test path documented"
  composition:
    weight: 0.15
    question: "Does it play with sibling skills cleanly?"
    anchors:
      5: "Declares dependencies; cited by 2+ peer skills"
      3: "Standalone; no broken refs"
      1: "Dangling Skill() refs or duplicated logic"
aggregation: weighted_sum
range: [1.0, 5.0]

Template 2: Plugin Quality Rubric

For plugin-level review. Mixes 0-2 gate dimensions with 1-5 quality dimensions. Plugins fail if any gate is 0.

rubric: plugin-quality
gates:                       # 0-2 each; any 0 fails
  manifest_valid:
    description: "plugin.json parses and schema validates"
  no_dangling_refs:
    description: "Every Skill() and command ref resolves"
  declared_deps_match_imports:
    description: "Frontmatter dependencies match real imports"
quality:                     # 1-5 each; weighted
  skill_quality_avg:
    weight: 0.30
    source: "mean of skill-quality rubric scores"
  doc_quality:
    weight: 0.20
    anchors:
      5: "README has thesis, examples, anti-goals"
      3: "README describes purpose; no examples"
      1: "Stub README"
  test_coverage:
    weight: 0.20
    anchors:
      5: ">= 85% lines, branch coverage tracked"
      3: "60-85% lines"
      1: "< 60% or no test target"
  release_hygiene:
    weight: 0.15
    anchors:
      5: "CHANGELOG, semver, signed tags"
      3: "Versions bumped but no changelog entries"
      1: "No version bumps; tags missing"
  cross_plugin_fit:
    weight: 0.15
    anchors:
      5: "Used by 2+ peer plugins; clear API"
      3: "Self-contained; no peers consume it"
      1: "Duplicates work done by another plugin"
aggregation: |
  if any gate == 0: fail
  else: quality_score = sum(weight_i * score_i)
range: [1.0, 5.0]

Template 3: Feature Backlog Rubric

For prioritizing backlog items. Compact wrapper around RICE+WSJF; details are in the scoring-framework module of the feature-review skill (under plugins/imbue/).

rubric: feature-backlog
scale: fibonacci [1, 2, 3, 5, 8, 13]
value_dimensions:
  reach: {weight: 0.25}
  impact: {weight: 0.30}
  business_value: {weight: 0.25}
  time_criticality: {weight: 0.20}
cost_dimensions:
  effort: {weight: 0.40}
  risk: {weight: 0.30}
  complexity: {weight: 0.30}
aggregation: |
  value = sum(w_v * score_v)
  cost = sum(w_c * score_c)
  priority = (value / cost) * confidence
range: [0.0, 13.0]

Aggregation Rules

Pick one explicitly. Each has trade-offs.

Weighted sum is the default. Pick anything else only if you can name the dimension that should be allowed to veto.

Worked Example: Scoring a Skill

A reviewer applies the skill-quality rubric to a fictional tome:research skill.

activation_clarity:  4   (3+ trigger phrases; no subagent test)
evidence_grounding:  5   (every claim has a worked search)
scope_discipline:    4   (defers to dig and synthesize)
token_economy:       3   (SKILL.md is 1100 lines, no paging)
testability:         3   (manual plan only)
composition:         5   (used by feature-review and minister)

weighted_sum =
   4 * 0.20 + 5 * 0.20 + 4 * 0.15 +
   3 * 0.15 + 3 * 0.15 + 5 * 0.15
 = 0.80 + 1.00 + 0.60 + 0.45 + 0.45 + 0.75
 = 4.05

Decision mapping (per decision-thresholds.md):

A 4.05 ships with the token-economy gap tracked.

Pitfalls

Inventing dimensions per artifact. If reviewers add their own dimensions, the rubric is dead. Lock the dimension list and version the rubric.

Anchors only at the extremes. "5 = best, 1 = worst" guarantees central tendency bias. Anchor odd scores at minimum, with a real example.

Weights summing to 1.0 by accident. State the sum and assert it in the file. A weight of 0.30 next to 0.40 next to 0.40 sums to 1.10 and silently overweights all dimensions.

Mixing scales within one rubric. A 0-100 dimension combined with a 1-5 dimension under weighted sum hands the 0-100 dimension 20x the influence. Normalize first or keep scales matched.

No decision mapping. A score that triggers nothing trains reviewers to score for vibes.

Rubric never revisited. If 90% of artifacts score 4-5, the rubric stopped discriminating. Recalibrate anchors against the current population yearly.

Cross-Reference

See modules/scoring-patterns.md for calibration and modules/decision-thresholds.md for mapping scores to actions.

Multi-Metric Evaluation Methodology

When a decision depends on more than one metric, you need a rule for combining them. The rule you pick determines which trade-offs are visible and which are silently smoothed away. This module catalogs five families of combination rules from Multi-Criteria Decision Analysis (MCDA), shows the math, and gives a decision tree for picking one.

The Combination Problem

Given n metrics m_1...m_n, a candidate scores a vector s = (s_1, ..., s_n). To rank candidates you need a rule f(s) -> R or a partial-order operator. The rules differ in what they assume about substitutability between metrics and about the decision-maker's preferences.

Normalization First

Every method below assumes metrics are on comparable scales. Skip normalization and units silently dominate weights.

Document which method was used and why. The choice affects final ranks more than reviewers usually expect.

Method 1: Weighted Sum (SAW)

Simple Additive Weighting. The default. Every other method should justify why weighted sum was rejected.

score(s) = sum_i (w_i * s_i)
subject to: sum_i (w_i) = 1.0,  w_i >= 0

Assumes: metrics are mutually preferentially independent and freely substitutable. Trading one unit of m_1 for w_1/w_2 units of m_2 is acceptable everywhere on the scale.

Use when: dimensions are independent, similarly scaled, and a rich basket of moderate scores is preferable to a spiky basket with one zero.

Avoid when: any metric can be a deal-breaker (security, correctness). A high score on six dimensions cannot compensate for a zero on the seventh.

Worked example:

weights = {quality: 0.4, latency: 0.3, cost: 0.3}
scores  = {quality: 4,   latency: 5,   cost: 2}
total   = 0.4*4 + 0.3*5 + 0.3*2 = 1.6 + 1.5 + 0.6 = 3.7

Method 2: Weighted Product / Geometric Mean

Compensatory but punishes low scores more than weighted sum.

score(s) = product_i (s_i ** w_i)

Assumes: a one-step drop in a low score hurts more than a one-step drop in a high score. Multiplicative rather than additive.

Use when: you want imbalance to be visible but not absolute. A score of (5, 5, 1) should rank below (3, 3, 3) even though their sums are equal.

Worked example (same inputs as above):

score = 4^0.4 * 5^0.3 * 2^0.3
      = 1.741 * 1.621 * 1.231
      = 3.475

A single low score costs more than under SAW.

Method 3: Lexicographic Ordering

Strict priority: rank by m_1, break ties by m_2, then m_3.

sort candidates by m_1 desc;
  within ties, sort by m_2 desc;
    within ties, sort by m_3 desc;
      ...

Assumes: there is a true priority order, and no amount of m_2 can compensate for any deficit in m_1.

Use when: regulatory or safety dimensions exist with hard primacy. Example: "Pick the safest option; among equally safe, pick the cheapest."

Avoid when: top-priority metric has fine-grained differences. Lexicographic ignores all secondary data once m_1 differs by any amount.

Worked example:

candidates: [(safety=5, cost=2), (safety=4, cost=5)]
m_1 = safety: 5 > 4, so (5,2) wins, full stop.
The cost gap of 3 is invisible.

Method 4: Pareto Front (No Aggregation)

Refuse to aggregate. Report the non-dominated set.

A dominates B iff:
  for all i: s_i(A) >= s_i(B)  AND
  exists j:  s_j(A) >  s_j(B)

Pareto front = { x : no y dominates x }

Assumes: nothing about substitutability. Hands the trade-off back to the decision-maker.

Use when: weights are contested or unknown, or you want to surface the trade-off space rather than collapse it.

Worked example:

candidates = [
  A: (quality=5, cost=4),
  B: (quality=4, cost=2),
  C: (quality=3, cost=3),
  D: (quality=2, cost=5),
]
A dominates D (5>=2, 4<=5? cost lower is better, so flip)
After flipping cost: minimize cost, maximize quality.
Front: A (best quality), B (best balance), D (cheapest).
C is dominated by B (B beats C on both axes).

The front size grows roughly with the square root of n candidates; useful as a shortlist filter, not a final pick.

Method 5: TOPSIS

Technique for Order Preference by Similarity to Ideal Solution. Rank by distance to a synthetic best and worst.

for each metric i, compute:
  ideal_i      = max(s_i across candidates)  if benefit
                 min(s_i across candidates)  if cost
  anti_ideal_i = the opposite

distance_to_ideal(x)      = sqrt(sum_i (w_i * (x_i - ideal_i))^2)
distance_to_anti_ideal(x) = sqrt(sum_i (w_i * (x_i - anti_ideal_i))^2)

closeness(x) = d_anti(x) / (d_ideal(x) + d_anti(x))
rank by closeness desc.

Assumes: the best candidate minimizes Euclidean distance to the ideal in normalized space.

Use when: candidates differ along many axes and you want a single defensible ranking that respects both positive and negative reference points.

Avoid when: dimensions are not commensurable even after normalization, or when reviewers will not accept a geometric distance interpretation.

Picking a Method

Decision tree:

1. Is any metric veto-class (security, correctness)?
   -> Yes: weighted sum + gate, or lexicographic
   -> No: continue
2. Are weights contested or unknown?
   -> Yes: Pareto front (no aggregation)
   -> No: continue
3. Should imbalance be punished superlinearly?
   -> Yes: weighted product
   -> No: continue
4. Do you need defensible single ranking with reference points?
   -> Yes: TOPSIS
   -> No: weighted sum (default)

Sensitivity Analysis (Required)

No matter the method, vary weights by +/- 20% and check whether top-3 ranks change. If they do, the result is weight-driven and needs more evidence before action.

def rank_stability(scores, weights, variation=0.20, top_k=3):
    base = rank(scores, weights)[:top_k]
    flips = 0
    for w_name in weights:
        for delta in (+variation, -variation):
            perturbed = adjust(weights, w_name, delta)
            new = rank(scores, perturbed)[:top_k]
            if set(new) != set(base):
                flips += 1
    return flips / (2 * len(weights))    # 0.0 = stable, 1.0 = chaos

A flip rate above 0.3 means the ranking is fragile. Either gather more data per metric or switch to Pareto front.

Worked Example: Combining Methods

A team scores three skill candidates on (quality, cost, adoption) for inclusion in a release.

Candidates:
  alpha: q=5, c=2, a=3
  beta:  q=4, c=4, a=5
  gamma: q=3, c=5, a=2

Step 1 (gate): require q >= 3. All pass.

Step 2 (weighted sum), weights {q: 0.5, c: 0.2, a: 0.3}:
  alpha: 2.5 + 0.4 + 0.9 = 3.8
  beta:  2.0 + 0.8 + 1.5 = 4.3
  gamma: 1.5 + 1.0 + 0.6 = 3.1

Step 3 (sensitivity): vary q weight by +/- 0.1
  q=0.6: alpha=4.0, beta=4.2, gamma=3.4 -> beta still wins
  q=0.4: alpha=3.6, beta=4.4, gamma=2.8 -> beta still wins

Decision: ship beta. Rank stable; gates clear.

Pitfalls

Combining unnormalized metrics. A latency in milliseconds and a cost in dollars cannot share weights until both are scaled to the same range. Normalize first.

Equal weights as a default. Equal weights are a strong prior, not a neutral one. State the choice and justify it or derive weights from AHP / expert elicitation.

Aggregating away the trade-off. A single number hides the shape of the candidate. Always report the per-metric vector alongside the aggregate.

Skipping sensitivity. A ranking that flips when one weight moves 10% is not a finding; it is a coin flip dressed up in arithmetic.

Lexicographic on noisy metrics. If m_1 is measured with +/- 5% noise, lexicographic will treat noise as a tiebreaker. Bucket m_1 into bands first.

Cross-Reference

See modules/scoring-patterns.md for calibration of inputs, modules/quality-metrics.md for concrete metric thresholds, and modules/decision-thresholds.md for mapping aggregate scores to actions.

Quality Metrics

Concrete numeric thresholds for evaluating code, docs, and skill artifacts. Each metric below has a name, a default threshold, a measurement command, and a citation. Defaults are starting points: tighten or loosen them per repository based on historical data, then document the deviation in your repo's evaluation config.

Code Quality Metrics

Test Coverage

Line and branch coverage from a coverage tool.

Why these numbers: Google's testing guide reports diminishing returns above 90%; below 60% bug regression risk doubles in tracked studies. See "How Google Tests Software" (Whittaker, Arbon, Carollo).

Measure: pytest --cov --cov-branch --cov-report=term (Python), cargo tarpaulin (Rust), go test -cover (Go).

Cyclomatic Complexity (per function)

McCabe complexity counts independent paths through a function.

Source: McCabe 1976; SEI guidance. Functions above 10 have a measurably higher defect rate.

Measure: radon cc -s -a src/ (Python), gocyclo -over 10 . (Go), clippy::cognitive_complexity (Rust).

Function Length

Long functions correlate with high cyclomatic complexity; flag both, fix the one with worse trend.

Cognitive Complexity (Sonar)

Variant of cyclomatic complexity that penalizes nesting. More predictive of human readability.

Source: SonarSource white paper "Cognitive Complexity" (2017).

Test-to-Code Ratio

Ratio of test code lines to non-test code lines.

Measure:

test_lines=$(find tests/ -name "*.py" | xargs wc -l | tail -1 | awk '{print $1}')
code_lines=$(find src/  -name "*.py" | xargs wc -l | tail -1 | awk '{print $1}')
echo "ratio = $(bc -l <<< "$test_lines / $code_lines")"

Mutation Score

Percent of injected mutations killed by the test suite. Catches the case where coverage is high but assertions are weak.

Measure: mutmut run (Python), cargo mutants (Rust), pitest (Java).

Duplication

Percent of lines duplicated across the repo (token-level, not byte).

Measure: jscpd . or pmd cpd.

Documentation Quality Metrics

Coverage of Public API

Fraction of public symbols with at least one docstring or doc comment.

Measure: interrogate -v src/ (Python), cargo doc --no-deps checked for missing-docs lint.

Doc-to-Code Ratio

Lines of prose docs per 100 lines of code.

Slop Indicators

Concrete prose markers that flag low-quality docs. Each indicator has a per-1000-words target.

The full slop checklist is in Skill(scribe:slop-detector). Apply to any markdown longer than 100 words before merge.

Reading Level

Flesch reading ease for non-reference docs.

Measure: pip install textstat; python -c "import textstat,sys; print(textstat.flesch_reading_ease(open(sys.argv[1]).read()))" file.md.

Link Health

Measure: lychee --no-progress docs/.

Skill Quality Metrics

For Claude Code skill files specifically.

SKILL.md Length

Source: token economy guidance in Skill(abstract:modular-skills).

Module Count and Depth

Trigger Phrase Count

Number of activation phrases in the skill description.

Frontmatter Fields Required

Missing fields fail validation in Skill(abstract:skills-eval).

Worked Example: Scoring a Skill File

A reviewer measures a fictional tome:research skill against the metrics above.

Code-side metrics (its scripts/ directory):
  line coverage:        82%   -> meets standard tier
  branch coverage:      71%   -> meets standard tier
  cyclomatic max:       14    -> review candidate (one func)
  test/code ratio:      0.7   -> healthy
  mutation score:       63%   -> acceptable, weak edge
  duplication:          2.1%  -> healthy

Doc-side metrics (its SKILL.md and modules):
  SKILL.md lines:       420   -> acceptable
  module count:         6     -> hub-and-spoke ok
  trigger phrases:      4     -> ok
  em dashes / 1k words: 1.2   -> within target
  banned words:         0     -> clean
  Flesch:               55    -> senior engineer
  broken links:         0     -> clean

Aggregate verdict (using weighted_sum from
multi-metric-evaluation-methodology.md):
  weights: {coverage: 0.3, complexity: 0.2, doc: 0.3,
            slop: 0.2}
  score (1-5): (4 * 0.3) + (3 * 0.2) + (4 * 0.3) +
               (5 * 0.2) = 4.0
  Action: ship; track the one high-complexity function
  and the mutation gap.

Calibration

Metrics drift. Re-measure quarterly.

1. Compute the distribution of each metric across the current population. 2. Set thresholds at the 25th / 50th / 75th percentile of the current healthy-set, not at vendor defaults. 3. Track the trend: if median complexity rose by 20% in a quarter, the threshold may already be too lenient.

Pitfalls

Coverage as a single number. 90% line coverage with 0% branch coverage hides every conditional. Track both.

Complexity averaged over a file. A file with one 80-cyclo function and ten 2-cyclo functions has an average of 9 and looks fine. Track the maximum, not the mean.

Ratios without absolute floors. A 1.5 test-to-code ratio on a 200-line codebase says nothing. Pair every ratio with an absolute minimum (for example: at least 50 test cases for any ratio claim to count).

Doc-coverage gaming. Single-line docstrings on every public symbol pass the gate and teach nothing. Spot-check samples for content, not just presence.

Slop indicators applied to generated content. Stack traces and machine-generated tables will trip prose metrics. Exclude generated paths from the slop scan.

Thresholds copied without context. A safety-critical control system needs 100% MC/DC coverage; a marketing site does not. Tier your thresholds by criticality before applying.

Cross-Reference

See modules/scoring-patterns.md for combining these metrics into a calibrated rubric and modules/multi-metric-evaluation-methodology.md for aggregation rules.

Scoring Patterns

Detailed patterns and best practices for consistent, calibrated scoring across evaluations using Multi-Criteria Decision Analysis (MCDA) principles.

Mathematical Foundation

This scoring methodology follows research-validated MCDA practices:

• Normalization: Vector normalization (scale-invariant)
• Weighting: Validated through AHP or expert elicitation
• Sensitivity: All weights tested for robustness
• Reproducibility: Same inputs → same outputs

Related: Multi-Metric Evaluation Methodology

Scoring Methodology

The 0-100 Scale

Use a consistent 0-100 scale for all criteria scoring:

90-100: Exceptional - Exceeds all expectations
70-89:  Strong - Meets expectations with notable strengths
50-69:  Acceptable - Meets minimum requirements
30-49:  Weak - Below standards, needs improvement
0-29:   Poor - Does not meet basic requirements

Calibration Guidelines

Avoid score inflation: Not everything can be 90+. Reserve high scores for truly exceptional work.

Use the full range: Don't cluster scores in 70-85 range. Differentiate clearly.

Anchor to examples: Document reference examples at each score level for consistency.

Inter-rater reliability: Multiple evaluators should score similarly when using the same rubric.

Validate normalization: Use scale-invariant normalization (vector normalization) to ensure rankings don't change with unit conversions.

Test sensitivity: Verify rankings are stable to reasonable weight variations (±20%).

Criterion Design Patterns

Quantitative Criteria

For measurable attributes, use objective thresholds:

performance:
  90-100: Response time < 100ms
  70-89:  Response time 100-200ms
  50-69:  Response time 200-500ms
  30-49:  Response time 500-1000ms
  0-29:   Response time > 1000ms

Qualitative Criteria

For subjective attributes, provide clear descriptors:

clarity:
  90-100: Crystal clear, zero ambiguity, exemplary explanations
  70-89:  Clear with minor gaps, mostly well-explained
  50-69:  Generally understandable, some confusion possible
  30-49:  Confusing in places, requires clarification
  0-29:   Unclear or incomprehensible

Composite Criteria

Break complex criteria into sub-components:

code_quality:
  components:
    - readability (40%)
    - efficiency (30%)
    - error_handling (30%)

  calculate: weighted_avg(sub_scores)

Weight Assignment Patterns

Critical: Validate Your Weights

Don't use arbitrary weights. Derive them systematically:

Option 1: Analytic Hierarchy Process (AHP)

• Pairwise comparison of criteria
• Calculates weights with consistency checks
• Requires consistency ratio < 0.1

Option 2: Expert Judgment Elicitation

• Structured process with 5-15 experts
• Calibration questions to assess accuracy
• Performance-based weighting of contributions

Option 3: Empirical Validation

• Test weights against historical outcomes
• Adjust based on predictive validity
• Document validation results

Priority-Based Weighting

Assign weights based on importance to outcome:

# Critical success factors get highest weight
critical_criteria = 0.50-0.70  # Must-have qualities
important_criteria = 0.20-0.30  # Nice-to-have qualities
supplemental_criteria = 0.05-0.15  # Additional considerations

# Requirement: Document how weights were derived
weights_derivation:
  method: "AHP"  # or "expert_judgment" or "empirical"
  experts: 5
  consistency_ratio: 0.04
  date: "2025-01-07"

Equal Weighting

When criteria are equally important:

num_criteria = 5
weight_per_criterion = 1.0 / num_criteria  # 0.20 each

Stakeholder-Driven Weighting

Different stakeholders may weight criteria differently:

engineering_perspective:
  technical_quality: 0.50
  maintainability: 0.30
  performance: 0.20

business_perspective:
  time_to_market: 0.40
  feature_completeness: 0.35
  technical_quality: 0.25

Scoring Consistency Patterns

Reference Anchors

Document specific examples at key score levels:

criterion: documentation_quality

anchors:
  95: "See: project-alpha/docs - detailed, clear, examples"
  80: "See: project-beta/docs - good coverage, minor gaps"
  65: "See: project-gamma/docs - basic, functional"
  40: "See: project-delta/docs - incomplete, unclear"

Comparative Scoring

Score relative to known benchmarks:

Score = (artifact_performance / benchmark_performance) × 100

Example:
  Test coverage: 85%
  Benchmark: 90%
  Score: (85/90) × 100 = 94.4

Rubric Matrices

Use decision matrices for complex evaluations:

Dimension 1: Completeness (columns)
Dimension 2: Quality (rows)

           Partial  Complete  detailed
Excellent    70       85         95
Good         55       70         85
Fair         40       55         70
Poor         25       40         55

Advanced Patterns

Penalty Systems

Apply deductions for specific issues:

base_score = 85
penalties = {
    "security_vulnerability": -20,
    "breaking_change": -15,
    "missing_tests": -10,
    "documentation_gap": -5
}

final_score = max(0, base_score - sum(applicable_penalties))

Bonus Systems

Award extra points for exceptional qualities:

base_score = 75
bonuses = {
    "innovation": +10,
    "exceptional_performance": +10,
    "comprehensive_testing": +5
}

final_score = min(100, base_score + sum(applicable_bonuses))

Non-Linear Scoring

Some criteria may warrant non-linear scales:

# Exponential for critical metrics
security_score = 100 * (1 - e^(-vulnerabilities))

# Logarithmic for diminishing returns
feature_score = 100 * log(features_implemented + 1) / log(total_features + 1)

Conditional Scoring

Some criteria only apply in certain contexts:

scoring_rules:
  - if: artifact_type == "public_api"
    then: apply_criteria(backward_compatibility, weight=0.30)

  - if: artifact_type == "internal_tool"
    then: skip_criteria(backward_compatibility)

Scoring Workflow

1. Pre-Evaluation Preparation

- Review scoring rubric
- Understand each criterion
- Check reference anchors
- Calibrate expectations

2. Initial Assessment

- Quick pass through all criteria
- Note obvious strengths/weaknesses
- Identify areas needing deeper analysis

3. Detailed Scoring

- Score each criterion independently
- Document reasoning for each score
- Note specific evidence supporting score
- Flag edge cases or uncertainties

4. Review and Calibration

- Check scores against anchors
- Verify consistency across criteria
- Adjust outliers if needed
- Document final rationale

5. Calculate and Decide

- Apply weights to compute total
- Compare to decision thresholds
- Document recommended action
- Provide specific feedback

Common Pitfalls

Halo Effect

Don't let overall impression influence individual criterion scores. Score each independently.

Central Tendency Bias

Don't cluster all scores around 50-70. Use the full range when appropriate.

Recency Bias

Don't overweight recent observations. Consider the full artifact.

Confirmation Bias

Don't score to match a predetermined conclusion. Let the rubric guide you.

Inconsistent Rigor

Apply the same level of scrutiny to all artifacts, regardless of source or context.

Validation Checks

Before finalizing scores:

• [ ] All criteria scored on 0-100 scale
• [ ] Scores match rubric descriptions
• [ ] Evidence documented for each score
• [ ] Weights sum to 1.0
• [ ] Weights validated through AHP or expert elicitation
• [ ] Normalization method documented (vector/minmax/log)
• [ ] Scale invariance tested (unit changes don't affect rankings)
• [ ] Sensitivity analysis completed (±20% weight variation)
• [ ] Calculations verified
• [ ] Threshold determination clear
• [ ] Feedback actionable and specific

Required Documentation

Every evaluation must document:

evaluation_metadata:
  normalization:
    method: "vector"  # or "minmax" or "log"
    scale_invariant: true
    rationale: "Vector normalization preserves rankings under unit changes"

  weighting:
    method: "AHP"  # or "expert_judgment" or "empirical"
    derivation_date: "2025-01-07"
    experts: 5
    consistency_ratio: 0.04  # < 0.1 required

  sensitivity:
    variation_tested: 0.20  # ±20%
    critical_weights: ["content_quality"]  # Rankings sensitive to these
    stable_weights: ["documentation"]  # Rankings robust
    spearman_correlation: 0.92  # Overall stability

  aggregation:
    method: "weighted_sum"  # or "TOPSIS" or "Pareto"
    independence_assumption: "Preferential independence assumed"
    trade_offs: "Documented in multi-dimensional report"

Evaluation Framework

A generic weighted scoring and threshold-based decision framework for evaluating artifacts against configurable criteria.

Purpose

This skill provides reusable evaluation patterns that can be customized for different domains. It abstracts the common pattern of:

1. Define criteria with weights 2. Score against criteria 3. Calculate weighted total 4. Apply decision thresholds 5. Take appropriate actions

Structure

evaluation-framework/
├── SKILL.md                          # Hub - core patterns (149 lines)
└── modules/
    ├── scoring-patterns.md           # Detailed scoring methodology
    └── decision-thresholds.md        # Threshold design patterns

Usage

As a Dependency

# In your skill's frontmatter
dependencies: [leyline:evaluation-framework]

Common Use Cases

• Quality Gates: Code review decisions, PR approval, release readiness
• Content Evaluation: Document quality, knowledge intake, skill assessment
• Resource Allocation: Backlog prioritization, investment decisions, triage

Design Principles

• Generic and Reusable: Works across different evaluation domains
• Configurable: Users define their own criteria and weights
• Consistent: Same methodology applies everywhere
• Actionable: Clear mapping from scores to decisions

Consumers

This framework is designed to be consumed by:

• memory-palace/skills/knowledge-intake/modules/evaluation-rubric.md
• abstract/skills/skills-eval/modules/quality-metrics.md
• Any plugin needing systematic evaluation with weighted criteria

Token Budget

• SKILL.md: ~550 tokens (estimated)
• scoring-patterns.md: ~800 tokens (estimated)
• decision-thresholds.md: ~700 tokens (estimated)
• Total: ~2050 tokens (progressive loading)