Now liveThe Skillselion MCP - thousands of ranked skills, loaded into your agent mid-task. No install.Get it →
terrylica avatar

Evolutionary Metric Ranking

terrylica/cc-skills

  • 111 installs
  • 62 repo stars
  • Updated August 3, 2026
  • terrylica/cc-skills

Use evolutionary-metric-ranking for development tasks

About

evolutionary-metric-ranking: A skill for development. This provides functionality for development workflows.

  • evolutionary-metric-ranking

Evolutionary Metric Ranking by the numbers

  • 111 all-time installs (skills.sh)
  • Ranked #2,929 of 4,347 Backend & APIs skills by installs in the Skillselion catalog
  • Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/terrylica/cc-skills --skill evolutionary-metric-ranking

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

Listed on Skillselion
Installs111
repo stars★ 62
Last updatedAugust 3, 2026
Repositoryterrylica/cc-skills ↗

What it does

Use evolutionary-metric-ranking for development tasks

Files

SKILL.mdMarkdownGitHub ↗

Evolutionary Metric Ranking

Methodology for systematically zooming into high-quality configurations across multiple evaluation metrics using per-metric percentile cutoffs, intersection-based filtering, and evolutionary optimization. Domain-agnostic principles with quantitative trading case studies.

Companion skills: rangebar-eval-metrics (metric definitions) | adaptive-wfo-epoch (WFO integration) | backtesting-py-oracle (SQL validation)

---

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

When to Use This Skill

Use this skill when:

  • Ranking and filtering configs/strategies/models across multiple quality metrics
  • Searching for optimal per-metric thresholds that select the best subset
  • Identifying which metrics are binding constraints vs inert dimensions
  • Running multi-objective optimization (Optuna TPE / NSGA-II) over filter parameters
  • Performing forensic analysis on optimization results (universal champions, feature themes)
  • Designing a metric registry for pluggable evaluation systems

---

Core Principles

P1 - Percentile Ranks, Not Raw Values

Raw metric values live on incompatible scales (Kelly in [-1,1], trade count in [50, 5000], Omega in [0.8, 2.0]). Percentile ranking normalizes every metric to [0, 100], making cross-metric comparison meaningful.

Rule: scipy.stats.rankdata(method='average') scaled to [0, 100]
      None/NaN/Inf -> percentile 0 (worst)
      "Lower is better" metrics -> negate before ranking (100 = best)

Why average ties: Tied values receive the mean of the ranks they would span. This prevents artificial discrimination between genuinely identical values.

P2 - Independent Per-Metric Cutoffs

Each metric gets its own independently-tunable cutoff. cutoff=20 means "only configs in the top 20% survive this filter." This creates a 12-dimensional (or N-dimensional) search space where each axis controls one quality dimension.

cutoff=100 -> no filter (everything passes)
cutoff=50  -> top 50% survives
cutoff=10  -> top 10% survives (stringent)
cutoff=0   -> nothing passes

Why independent, not uniform: Different metrics have different discrimination power. Uniform tightening (all metrics at the same cutoff) wastes filtering budget on inert dimensions while under-filtering on binding constraints.

P3 - Intersection = Multi-Metric Excellence

A config survives the final filter only if it passes ALL per-metric cutoffs simultaneously. This intersection logic ensures no single-metric champion sneaks through with terrible performance elsewhere.

survivors = metric_1_pass AND metric_2_pass AND ... AND metric_N_pass

Why intersection, not scoring: Weighted-sum scoring hides metric failures. A config with 99th percentile Sharpe but 1st percentile regularity would score well in a weighted sum but is clearly deficient. Intersection enforces minimum quality across every dimension.

P4 - Start Wide Open, Tighten Evolutionarily

All cutoffs default to 100% (no filter). The optimizer progressively tightens cutoffs to find the combination that best satisfies the chosen objective. This is the opposite of starting strict and relaxing.

Initial state:  All cutoffs = 100 (1008 configs survive)
After search:   Each cutoff independently tuned (11 configs survive)

Why start wide: Starting strict risks missing the global optimum by immediately excluding configs that would survive under a different cutoff combination. Wide-to-narrow exploration is characteristic of global optimization.

P5 - Multiple Objectives Reveal Different Truths

No single objective function captures "quality." Run multiple objectives and compare survivor sets. Configs that survive all objectives are the most robust.

ObjectiveAsksReveals
max_survivors_min_cutoffMost configs at tightest cutoffs?Efficient frontier of quantity vs stringency
quality_at_target_nBest quality in top N?Optimal cutoffs for a target portfolio size
tightest_nonemptyAbsolute tightest with >= 1 survivor?Universal champion (sole survivor)
pareto_efficiencySurvivors vs tightness trade-off?Full Pareto front (NSGA-II)
diversity_rewardAre cutoffs non-redundant?Which metrics provide independent information

Cross-objective consistency: A config that appears in ALL objective survivor sets is the most defensible selection. One that appears in only one is likely an artifact of that objective's bias.

P6 - Binding Metrics Identification

After optimization, identify binding metrics - those that would increase the intersection if relaxed to 100%. Non-binding metrics are either already loose or perfectly correlated with a binding metric.

For each metric with cutoff < 100:
    Relax this metric to 100, keep others fixed
    If intersection grows: this metric IS binding
    If intersection unchanged: this metric is redundant at current cutoffs

Why this matters: Binding metrics are the actual constraints on your quality frontier. Effort to improve configs should focus on binding dimensions.

P7 - Inert Dimension Detection

A metric is inert if it provides zero discrimination across the population. Detect this before optimization to reduce dimensionality.

If max(metric) == min(metric) across all configs: INERT
If percentile spread < 5 points: NEAR-INERT

Action: Remove inert metrics from the search space or permanently set their cutoff to 100. Including them wastes optimization budget.

P8 - Forensic Post-Analysis

After optimization, perform forensic analysis to extract actionable insights:

1. Universal champions - configs surviving ALL objectives 2. Feature frequency - which features appear most in survivors 3. Metric binding sequence - order in which metrics become binding as cutoffs tighten 4. Tightening curve - intersection size vs uniform cutoff (100% -> 5%) 5. Metric discrimination power - which metric kills the most configs at each tightening step

---

Architecture Pattern

Metric JSONL files (pre-computed)
        |
        v
MetricSpec Registry  <-- Defines name, direction, source, cutoff var
        |
        v
Percentile Ranker    <-- scipy.stats.rankdata, None->0, flip lower-is-better
        |
        v
Per-Metric Cutoff    <-- Each metric independently filtered
        |
        v
Intersection         <-- Configs passing ALL cutoffs
        |
        v
Evolutionary Search  <-- Optuna TPE/NSGA-II tunes cutoffs
        |
        v
Forensic Analysis    <-- Cross-objective consistency, binding metrics

MetricSpec Registry

The registry is the single source of truth for metric definitions. Each entry is a frozen dataclass:

@dataclass(frozen=True)
class MetricSpec:
    name: str              # Internal key (e.g., "tamrs")
    label: str             # Display label (e.g., "TAMRS")
    higher_is_better: bool # Direction for percentile ranking
    default_cutoff: int    # Default percentile cutoff (100 = no filter)
    source_file: str       # JSONL filename containing raw values
    source_field: str      # Field name in JSONL records

Design principle: Adding a new metric = adding one MetricSpec entry. No other code changes required. The ranking, cutoff, intersection, and optimization machinery is fully generic.

Env Var Convention

Each metric's cutoff is controlled by a namespaced environment variable:

RBP_RANK_CUT_{METRIC_NAME_UPPER} = integer [0, 100]

This enables:

  • Shell-level override without code changes
  • Copy-paste of optimizer output directly into next run
  • CI/CD integration via environment configuration
  • Mise task integration via [env] blocks

---

Evolutionary Optimizer Design

Sampler Selection

ScenarioSamplerWhy
Single-objectiveTPE (Tree-Parzen Estimator)Bayesian, handles integer/categorical, good for 10-20 dimensions
Multi-objective (2+)NSGA-IIPareto-frontier discovery, population-based

Determinism: Always seed the sampler (seed=42). Optimization results must be reproducible.

Search Space Design

def suggest_cutoffs(trial):
    cutoffs = {}
    for spec in metric_registry:
        cutoffs[spec.name] = trial.suggest_int(spec.name, 5, 100, step=5)
    return cutoffs

Why step=5: Reduces the search space by 20x (20 values per metric vs 100) while maintaining sufficient granularity. For 12 metrics, this is 20^12 = 4 x 10^15 vs 100^12 = 10^24.

Why lower bound = 5: cutoff=0 always produces empty intersection. Values below 5 are too stringent to be useful in practice.

Data Pre-Loading (Critical Performance Pattern)

# Load metric data ONCE, share across all trials
metric_data = load_metric_data(results_dir, metric_registry)

def objective(trial):
    cutoffs = suggest_cutoffs(trial)
    # Pass pre-loaded data - avoids disk I/O per trial
    result = run_ranking_with_cutoffs(cutoffs, metric_data=metric_data)
    return obj_fn(result, cutoffs)

Why: Each trial evaluates in ~6ms when data is pre-loaded (pure NumPy/set operations). Without pre-loading, each trial incurs ~50ms of disk I/O. At 10,000 trials, this is 60 seconds vs 500 seconds.

Objective Function Patterns

Pattern 1 - Ratio Optimization
def obj_max_survivors_min_cutoff(result, cutoffs):
    n = result["n_intersection"]
    if n == 0:
        return 0.0
    mean_cutoff = sum(cutoffs.values()) / len(cutoffs)
    return n / mean_cutoff  # More survivors per unit of looseness

Use when: Exploring the efficiency frontier - how much quality can you get for how much filtering?

Pattern 2 - Constrained Quality
def obj_quality_at_target_n(result, cutoffs, target_n=10):
    n = result["n_intersection"]
    avg_pct = result["avg_percentile"]
    if n < target_n:
        return avg_pct * (n / target_n)  # Partial credit
    return avg_pct  # Full credit: maximize quality

Use when: You have a target portfolio size and want the highest quality subset.

Pattern 3 - Minimum Budget
def obj_tightest_nonempty(result, cutoffs):
    n = result["n_intersection"]
    if n == 0:
        return 0.0
    total_budget = sum(cutoffs.values())
    return max_possible_budget - total_budget  # Lower budget = better

Use when: Finding the single most universally excellent config.

Pattern 4 - Diversity Reward
def obj_diversity_reward(result, cutoffs):
    n = result["n_intersection"]
    if n == 0:
        return 0.0
    n_binding = result["n_binding_metrics"]
    n_active = sum(1 for v in cutoffs.values() if v < 100)
    if n_active == 0:
        return 0.0
    efficiency = n_binding / n_active
    return n * efficiency

Use when: Ensuring that tightened cutoffs provide independent information, not redundant filtering.

Pattern 5 - Pareto (Multi-Objective)
study = optuna.create_study(
    directions=["maximize", "minimize"],  # max survivors, min cutoff
    sampler=optuna.samplers.NSGAIISampler(seed=42),
)
def objective(trial):
    cutoffs = suggest_cutoffs(trial)
    result = run_ranking_with_cutoffs(cutoffs, metric_data=metric_data)
    return result["n_intersection"], sum(cutoffs.values()) / len(cutoffs)

Use when: You want to see the full trade-off landscape between two competing objectives.

---

Forensic Analysis Protocol

After running all objectives, perform this analysis:

Step 1 - Cross-Objective Survivor Sets

For each objective:
    survivors_{objective} = set of configs in final intersection

universal_champions = survivors_1 AND survivors_2 AND ... AND survivors_K

If a config survives all K objective functions, it is robust to objective choice.

Step 2 - Feature Theme Extraction

Count feature appearances across all survivors:

feature_counts = Counter()
for config_id in quality_survivors:
    for feature in config_id.split("__"):
        feature_counts[feature.split("_")[0]] += 1

Dominant features reveal the underlying market microstructure that the ranking system is selecting for.

Step 3 - Uniform Tightening Curve

Apply the same cutoff to ALL metrics and plot intersection size:

@100%: 1008 survivors (no filter)
@80%:  502 survivors
@60%:  210 survivors
@40%:   68 survivors
@20%:   12 survivors
@10%:    3 survivors
@5%:     0 survivors

The shape of this curve reveals whether the metric space has natural clusters or is uniformly distributed.

Step 4 - Binding Sequence

Tighten uniformly and at each step identify which metric was the "tightest killer" - the metric that eliminated the most configs:

@90%: 410 survivors | tightest killer: rachev (-57)
@80%: 132 survivors | tightest killer: headroom (-27)
@70%:  29 survivors | tightest killer: n_trades (-12)
@60%:   6 survivors | tightest killer: dsr (-6)

This reveals the binding constraint hierarchy.

---

Implementation Checklist

When implementing this methodology in a new domain:

1. [ ] Define MetricSpec registry (name, direction, source, default cutoff) 2. [ ] Implement percentile ranking (scipy.stats.rankdata) 3. [ ] Implement per-metric cutoff application 4. [ ] Implement set intersection across all metrics 5. [ ] Add env var override for each cutoff 6. [ ] Create run_ranking_with_cutoffs() API function 7. [ ] Add binding metric detection 8. [ ] Create tightening analysis function 9. [ ] Write markdown report generator 10. [ ] Add Optuna optimizer with at least 3 objective functions 11. [ ] Pre-load metric data for optimizer performance 12. [ ] Run 5-objective forensic analysis (10K+ trials per objective) 13. [ ] Extract universal champions (cross-objective consistency) 14. [ ] Identify inert dimensions (remove from search space) 15. [ ] Document binding constraint sequence 16. [ ] Record feature themes in survivors

---

Anti-Patterns

Anti-PatternSymptomFixSeverity
Weighted-sum scoringSingle metric dominates, others ignoredUse intersection (P3)CRITICAL
Starting strictMiss global optimum, premature convergenceStart at 100%, tighten (P4)HIGH
Uniform cutoffs onlyOver-filters inert metrics, under-filters binding onesPer-metric independent cutoffs (P2)HIGH
Single objectiveArtifact of objective biasRun 5+ objectives, check consistency (P5)HIGH
Raw value comparisonScale-dependent, misleadingAlways use percentile ranks (P1)HIGH
Including inert metricsWastes optimization budgetDetect and remove inert dimensions (P7)MEDIUM
No data pre-loadingOptimizer 10x slowerPre-load once, share across trialsMEDIUM
Unseeded optimizerNon-reproducible resultsAlways seed sampler (seed=42)MEDIUM
Missing forensic analysisRaw numbers without insightRun full forensic protocol (P8)MEDIUM

---

References

TopicReference File
Range Bar Case Studycase-study-rangebar-ranking.md
Objective Functionsobjective-functions.md
Metric Design Guidemetric-design-guide.md

Related Skills

SkillRelationship
rangebar-eval-metricsMetric definitions (TAMRS, Omega, DSR, etc.) fed into ranking
adaptive-wfo-epochWalk-Forward metrics that could be ranked
backtesting-py-oracleValidates trade outcomes used in metric computation
sharpe-ratio-non-iid-correctionsDSR computation with non-IID corrections

Dependencies

pip install scipy numpy optuna>=4.7

---

TodoWrite Task Templates

Template A - Implement Ranking System (New Project)

1. [Preflight] Identify all evaluation metrics and their JSONL sources
2. [Preflight] Define MetricSpec registry (name, direction, source_file, source_field)
3. [Execute] Implement percentile_ranks() with scipy.stats.rankdata
4. [Execute] Implement apply_cutoff() and intersection()
5. [Execute] Add env var override for each metric cutoff (RANK_CUT_{NAME})
6. [Execute] Create run_ranking_with_cutoffs() API for optimizer
7. [Execute] Add binding metric detection and tightening analysis
8. [Execute] Write markdown report generator
9. [Verify] Unit tests for all pure functions (14+ tests)
10. [Verify] Run with default cutoffs (100%) - all configs should survive

Template B - Add Evolutionary Optimizer

1. [Preflight] Verify ranking module has run_ranking_with_cutoffs() API
2. [Preflight] Add optuna>=4.7 dependency
3. [Execute] Implement 5 objective functions
4. [Execute] Create suggest_cutoffs() with step=5 search space
5. [Execute] Pre-load metric data once, share across trials
6. [Execute] Handle pareto_efficiency (NSGA-II) as special case
7. [Execute] Write JSONL output with provenance (git commit, timestamp)
8. [Verify] POC with 10 trials - verify non-trivial cutoffs found
9. [Verify] Full run with 10K trials per objective

Template C - Forensic Analysis

1. [Preflight] Collect optimization results from all 5 objectives
2. [Execute] Extract survivor sets per objective
3. [Execute] Compute cross-objective intersection (universal champions)
4. [Execute] Run uniform tightening analysis (100% -> 5%)
5. [Execute] Identify binding metrics at each tightening step
6. [Execute] Extract feature themes from quality survivors
7. [Execute] Detect inert dimensions (zero discrimination)
8. [Verify] Document findings in structured summary table

---

Post-Change Checklist (Self-Maintenance)

After modifying this skill:

1. [ ] Principles P1-P8 remain internally consistent 2. [ ] Anti-patterns table covers new patterns discovered 3. [ ] References in references/ are up to date 4. [ ] Case study reflects latest production results 5. [ ] Implementation checklist is complete and ordered 6. [ ] Plugin README updated if description changed

---

Troubleshooting

IssueCauseSolution
All cutoffs converge to 100%Metrics are all correlatedCheck for metric redundancy (Spearman r > 0.95)
Zero intersection at mild cutoffsOne metric has near-zero varianceDetect inert dimensions (P7)
Optimizer takes too longDisk I/O per trialPre-load metric data (see Performance section)
Different objectives give same answerObjectives poorly differentiatedVerify objective formulas test different trade-offs
Universal champion is mediocreSurvival != excellenceCheck raw values, not just survival
Binding sequence changes across runsUnseeded optimizerAlways use seed=42
Too many survivorsCutoffs too looseIncrease n_trials, lower step size
Zero survivorsCutoffs too tightCheck for inert metrics inflating dimensionality

Post-Execution Reflection

After this skill completes, reflect before closing the task:

0. Locate yourself. — Find this SKILL.md's canonical path (Glob for this skill's name) before editing. All corrections target THIS file and its sibling references/ — never other documentation. 1. What failed? — Fix the instruction that caused it. If it could recur, add it as an anti-pattern. 2. What worked better than expected? — Promote it to recommended practice. Document why. 3. What drifted? — Any script, reference, or external dependency that no longer matches reality gets fixed now. 4. Log it. — Every change gets an evolution-log entry with trigger, fix, and evidence.

Do NOT defer. The next invocation inherits whatever you leave behind

Related skills

open.feishu.cnLark Openapi ExplorerInstantly explore, test, and generate calls against the full Lark (Feishu) OpenAPI surface without leaving their agent workflow.493klarksuiteLark EventConsume real-time events from Lark/Feishu as structured NDJSON streams inside AI agent workflows.387k16klarksuiteLark Openapi ExplorerWhen an existing Lark/Feishu skill or CLI command cannot fulfill a specific requirement and they need to discover and invoke the exact native OpenAPI endpoint.386k16kscrapegraphaiJust ScrapeQuickly search, crawl, extract structured JSON, or monitor web pages without writing custom scraping code.245k41larksuiteLark AppsQuery the current visibility and permission scope of a Lark (Feishu) app without writing HTTP client code.235k16kmattpocockDesign An InterfaceGenerate multiple radically different interface or API designs in parallel before committing to one.188k198k
Backend & APIsbackendintegrations
Browse by category
AI & AgentsFrontend & DesignBackend & DataDev ToolsAutomationTesting & ReviewSecurityMonitoring & CloudDocs & Planning
Design & frontend
Design systemsAnimation & motionAccessibility (a11y)CSS & TailwindUI designUXDesign to codeshadcn/uiLanding pagesBest design skills
Most searched skills
find-skillscavemanfrontend-designbrainstormingsystematic-debuggingstop-slopemil-design-engui-ux-pro-maxkarpathy-guidelinesweb-design-guidelinesreact-best-practicesgrill-meimprovemicrosoft-foundryagent-browser
Browse all listings
All agent skillsAll MCP serversAll marketplacesAll plugins

This week in AI coding

Five minutes, every Monday - the tools, releases and tactics for developers.

unsubscribe anytime.