Can Jev pick agent skills faster and cheaper than an LLM?

Setup

The setup in one paragraph

The Skillselion MCP has a load_skill tool. A coding agent sends it a task, such as "playwright e2e tests", and gets back one skill from our catalog of agent skills to load. Behind it sits a search ranker: keyword search, a domain match and a semantic (vector) search, merged and scored. We wanted to know whether a model could choose better from the ranker's own shortlist, fast enough to sit in front of every skill load. So for every logged request we took the ranker's top 10 candidates and asked three pickers to choose one, or none: the ranker itself, Jev, and Claude Haiku 5.5. Two judges from different model families then marked which candidates were a good fit, without knowing who picked what.

Finding 01

Jev answers in 212 ms, 8.5 times faster than Haiku

Speed is where Jev stands out most.

1. 1Jev 1.14212 ms
2. 2Claude Haiku 5.51,793 ms

At the median Jev was 8.5 times faster, and at the 95th percentile 9.9 times faster. Each pick saved about 1.6 seconds, which adds up to 26 minutes of waiting across a thousand skill loads.

Jev is fast because it does not write text. It reads the task and the options once and returns a typed answer with a probability for each option. Haiku has to generate its answer token by token.

We measured from one client, six requests at a time, with each request making its four model calls in parallel. We called Jev directly at api.typesafe.ai and Haiku through the OpenRouter proxy, so part of Haiku's time is the proxy hop.

Finding 02

Five cents per thousand picks

A Jev pick averaged 1,116 input tokens. At Jev's list price of $0.042 per million input tokens, with output free (TypeSafe models page), a thousand picks cost $0.047.

Jev's input price is 2.4 times lower than Haiku's list price on OpenRouter on 2026-11-08 ($0.10 per million input tokens), and Haiku also pays $0.50 per million output tokens, including any reasoning. Jev pays nothing for output. We did not log Haiku's token counts, and the two models count tokens differently, so we compare list prices rather than publish a per-pick Haiku cost.

At that rate a million skill picks cost about $47.

Finding 03

25 to 28 points more good picks than our search ranker

Choosing from the same ten candidates our ranker returned, Jev's pick was a good fit 89.0% of the time by the Sonnet judge and 89.5% by the GPT judge. The ranker's own pick was good 61.4% and 64.0% of the time.

1. 1Jev 1.1489.0%
2. 2Claude Haiku 5.588.2%
3. 3Search ranker61.4%

Good picks by the GPT judge: Jev 89.5%, Haiku 84.2%, search ranker 64.0%. Match with the judge's single best candidate (Sonnet / GPT): Jev 65.8% / 67.1%, Haiku 71.5% / 64.0%, search ranker 27.2% / 22.8%. 228 real requests.

Jev also matched the judge's single best candidate 2.4 to 2.9 times as often as the ranker. By the Sonnet judge, Jev was right and the ranker wrong on 73 requests; the reverse happened on 10. By the GPT judge the split was 65 to 7. Both gaps are significant by an exact McNemar test, and they stay significant after correcting for every test on this page.

The ranker's retrieval works: the right skill is usually in its top 10. Jev fixes the weak step, the final choice. One caveat applies to this finding in particular: the candidate list was not shuffled as we intended (see the method).

Finding 04

Haiku-level accuracy without generating text

We added Claude Haiku 5.5 as a third picker to answer the obvious question: would any small language model do as well? On accuracy, Jev and Haiku are level.

124 of the 228 scored requests carry the burst flag described in the method, and the mix matters. By the Sonnet judge, Jev and Haiku were within one point of each other (89.0% and 88.2%). By the GPT judge, Jev led by 5.3 points (89.5% to 84.2%). On the 90 cleanest requests, Haiku led by both judges, by up to 5.6 points (96.7% and 93.3% against 91.1% and 92.2%). None of these Jev-against-Haiku gaps is significant after correcting for the six tests on this page, so we call it a tie.

That tie is the good news. Jev reaches the accuracy of a capable language model while answering 8.5 times faster, at a lower list price, and with a confidence score that separates its good picks from its weaker ones, which the LLM does not give you.

Finding 05

Jev knows when no skill fits

Some requests have no good answer in the shortlist. On 10 requests both judges agreed that none of the ten candidates fit. Jev answered "none" on all 10. So did Haiku. The ranker caught 6.

Jev was also the picker least likely to give up on a good shortlist. We counted the requests where a picker said "none" although both judges saw at least one good candidate:

1. 1Jev 1.147
2. 2Claude Haiku 5.524
3. 3Search ranker24

A wrong "none" sends the agent away with nothing, and the user never learns there was a good skill. Jev made that mistake 7 times against Haiku's 24. Most of the gap came from the August burst of accessibility queries (5 against 16); on the 90 clean requests it was 1 against 2.

Finding 06

A confidence score you can route on

Every Jev Choice answer comes with a confidence between 0 and 1, computed from how the probability spreads across the options. We grouped Jev's 228 answers by that number and checked them against the Sonnet judge:

1. 1Confidence 0.9 and above (36 requests)97.2%
2. 2Confidence 0.6 to 0.9 (73 requests)97.3%
3. 3Confidence below 0.6 (119 requests)81.5%

At 0.6 or more, 97% of Jev's picks were good. On the 90 clean requests the bands were closer: 93.2% at 0.6 or more against 89.1% below. That gives us a simple rule with no extra calls: act on Jev's pick when confidence is 0.6 or more, and show the agent the top few candidates when it is lower. A language model's text answer gives you no such number unless you build one.

Finding 07

The same answer on a second run

Before the three-picker run, we ran a simpler version: the ranker against Jev only, Sonnet as the only judge. It scored 556 of the 567 requests; the other 11 failed on judge errors. It gave the same picture. The ranker's pick was good 68.0% of the time and Jev's 92.1%. On the 258 clean requests it was 77.9% against 94.2%.

227 requests appeared in both runs with the same candidates in the same order. Jev gave the same answer on 211 of them, 93%. Our Sonnet judge, at temperature 0, repeated its own best pick 204 times and its full list of good candidates only 148 times. The judge's task was harder (list every fit and the best one), and we did not re-run Haiku, so this is not a like-for-like comparison. It does show that Jev's answers hold steady from run to run.

Practical

What we are changing in the MCP

1. Shadow mode first. The next version of skillselion-mcp will ask Jev in parallel with the ranker and log both picks, without changing what the agent gets. We switch only if the logged picks from real usage agree with this test.
2. A confidence gate. If we switch, Jev's pick goes through only at confidence 0.6 or more. Below that, the agent gets the top three candidates to choose from.
3. A separate "does anything fit" question. TypeSafe documents a Companion Noul pattern for this: pair the Choice with a yes/no question about whether any option fits. Jev already handled "none" well inside the Choice; we will test whether the extra question cuts the 7 false "none" answers.

If you are building something similar, TypeSafe's own skill suggestion cookbook ranks a 194-skill catalog with two requests per turn. Our test is narrower: one request, picking from a shortlist a search engine already built.

Method

How we ran the test

Where the requests come from. The Skillselion MCP (npm skillselion-mcp) logs each load_skill query to our API after removing email addresses, common API token formats, file paths and backtick-quoted code, cut to 200 characters. The agent's context, code and files are never logged. Between 2026-07-01 and 2026-11-06 it logged 1,018 distinct queries. We removed our own 228 evaluation queries, exact skill-id lookups and short test strings, which left 568. One had no candidates, which left 567.

Two flags, kept in the data. 287 requests were first and last seen inside a two-day window, 2026-08-24 and 2026-08-25, when a wave of accessibility queries arrived. They carry a burst flag. 18 were attack probes rather than skill requests: script injection, shell injection, path traversal, server-side request forgery URLs, fake tool definitions and "ignore previous instructions" prompts. They carry a junk flag. The "clean" numbers in this report exclude both flags.

The shortlist. For every request we re-ran the live matcher with no context and kept its top 10 candidates: name, category, a summary cut to 280 characters, and install count. Jev, Haiku and both judges saw exactly this text, as a list lettered A to J.

A flaw in the candidate order. We meant to shuffle the list so that a letter carried no information about the ranker's order. The shuffle we wrote, seeded from each query's text, did not do that. In the 228 scored requests the ranker's first choice always sat at letter H, I or J (H in 134). Every picker and judge saw the same order, so a shared preference for a position could raise their agreement with each other. We checked what we could from the stored answers: Jev chose the ranker's pick in 62 of 228 requests and Haiku in 68, and both spread their picks across all ten letters, so neither copied the ranker's slot. The judges marked letter H as a good fit 113 and 125 times, against an average of about 96 and 107 per letter; H held the ranker's first choice, which is often a real fit, so we cannot separate content from position there. A rerun with a proper shuffle would settle it.

The pickers. - Search ranker: its own top candidate, or "none" if that candidate fell below the relevance floor the MCP uses in production. - Jev: model jev-latest, which resolved to jev-1.14.0 on every request. One Choice question through POST /v1/systemone: the state is the task, the options are the ten candidates plus "none". - Claude Haiku 5.5 (anthropic/claude-haiku-5.5 on OpenRouter), temperature 0, the same text, asked for a JSON answer. We did not set a reasoning option, so OpenRouter's default for the model applied.

A setup that did not favor Jev. TypeSafe's Choice documentation recommends giving the full option list rather than a shortlist, and cautions that a "none of the above" option may not work well. We did both, because that is how our MCP works. If anything, this handicapped Jev.

The judges. Claude Sonnet 5.5 and GPT-5.6 Terra Pro, both through OpenRouter at temperature 0. Each judge saw the task and the ten candidates and listed every candidate that was a good fit plus the single best one, or said none fit. Judges never saw any picker's answer. A pick counts as good when the judge listed it, or when the picker said "none" and the judge found nothing. The judges agreed on the single best candidate in 162 of 228 requests and on whether Jev's pick was good in 90.8% of them.

Statistics. Pickers are compared on the same requests with an exact two-sided McNemar test on the requests where one was right and the other wrong.

Why 228 and not 567. The three-picker run attempted all 567 requests. Partway through, the prepaid OpenRouter credit used for Haiku and both judges ran out, and every request whose four calls did not all succeed was dropped: 335 on payment errors, 4 on malformed judge replies. The list is sorted by how often a query was sent, and early requests had the best chance of finishing, so the 228 lean toward repeated queries: 1.96 calls per query on average, against 1.39 across all 567. We chose not to buy more credit for this edition. The first run (Finding 07) covers 556.

What this test cannot tell you. - The judges are language models. They read the same summaries the pickers read, so a judge and a model picker may share blind spots. A judge cannot tell whether the skill helped the user. - Every request is query-only. Real MCP calls also carry the agent's context, which we never log, so the ranker and Jev both had less to go on than in production. - Latency was measured from one machine; your network will differ. - The test covers one catalog and one task over two days. Read it as a result for skill picking, not as a general benchmark of Jev or Haiku.

Data

Download the dataset

Two CSV files, one row per request. They carry no query text: the queries are real people's requests and we do not publish them. Both are free to use with attribution under CC BY 4.0: cite "Skillselion Jev Skill-Picking Experiment, November 2026" and link this page.

Download the main run CSV · 228 rows · three pickers, two judges

Download the first run CSV · 556 rows · ranker against Jev, one judge

Before publishing, a verification script recomputed every result figure on this page from these two files and failed on any difference. Some method figures come from our private query logs and run records and cannot be recomputed from the public files, because that would expose the queries: 1,018 logged queries, 568 after filtering, 567 with candidates, 287 burst requests, 18 attack probes, the 1.96 against 1.39 calls per query, the 2026-08-24/25 burst window, and the failure counts (335 payment errors and 4 malformed replies in the main run, 11 judge errors in the first run).