Cost Benchmark
- 590 installs
- 67k repo stars
- Updated August 4, 2026
- ruvnet/ruflo
cost-benchmark is a ruflo skill that runs scripts/bench.mjs on a structural plus adversarial corpus and writes measured-versus-claimed benchmark tables to docs/benchmarks/runs/.
About
cost-benchmark is a ruflo verification skill that runs scripts/bench.mjs against the structural and adversarial cases in bench/booster-corpus.json, optionally comparing the local booster to Gemini, Sonnet, or Opus baselines via --llm or --anthropic flags. Results land as per-case and summary tables under docs/benchmarks/runs/, backing measurable claims in cost-booster-edit and cost-booster-route skills. Developers reach for cost-benchmark before publishing a release to confirm booster win rate has not regressed, after expanding the corpus, or when auditing claimed cost-reduction percentages. The skill allows Bash execution only and acts as the measurable gate between routing edits and public performance statements.
- cost-benchmark
Cost Benchmark by the numbers
- 590 all-time installs (skills.sh)
- +6 installs in the week ending Aug 5, 2026 (Skillselion tracking)
- Ranked #683 of 4,347 Backend & APIs skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/ruvnet/ruflo --skill cost-benchmarkAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 590 |
|---|---|
| repo stars | ★ 67k |
| Last updated | August 4, 2026 |
| Repository | ruvnet/ruflo ↗ |
How do you verify LLM booster benchmark claims?
Use cost-benchmark for development tasks
Who is it for?
Maintainers of ruflo cost-booster routing who must regression-test win rate before releases or corpus expansions.
Skip if: Teams not using ruflo booster routing or developers needing production LLM cost dashboards should use observability tooling instead.
When should I use this skill?
Before a ruflo release, after editing bench/booster-corpus.json, or when auditing claimed booster savings against the structural plus adversarial corpus.
What you get
Per-case benchmark rows, summary measured-versus-claimed table, and persisted run artifacts under docs/benchmarks/runs/.
- Benchmark run tables
- Measured-versus-claimed summary
- docs/benchmarks/runs/ artifacts
By the numbers
- Executes scripts/bench.mjs against bench/booster-corpus.json
- Persists per-case and summary results under docs/benchmarks/runs/
Files
Cost Benchmark
Runs scripts/bench.mjs against the structural+adversarial corpus and writes per-case + summary results to docs/benchmarks/runs/. This is the verification gate that backs every measurable claim in cost-booster-edit / cost-booster-route.
When to use
- Before publishing a release — verify booster win rate didn't regress.
- After expanding
bench/booster-corpus.json— confirm new cases route correctly. - When auditing a "claimed upstream" tag — flip it to "verified" once the bench supports it.
- On a cost question ("is Sonnet 4.6 cheaper than Opus 4.7 for these tasks?") — re-run with
BENCH_ANTHROPIC=1.
Steps
1. Run the bench from `v3/` (where agent-booster resolves):
( cd v3 && node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # booster only — free, ~85 ms
( cd v3 && BENCH_LLM_BASELINE=1 node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Gemini 2.0 Flash (cheap)
( cd v3 && BENCH_LLM_BASELINE=1 BENCH_ANTHROPIC=1 \
node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Sonnet 4.6 + Opus 4.72. Inspect the markdown summary printed to stdout. The gate metric is winRate (Tier 1 cases). Adversarial cases are tracked separately as escalationRate.
3. Persisted output lands at:
docs/benchmarks/runs/latest.json— pointer to the most recent rundocs/benchmarks/runs/<ISO-timestamp>.json— historical record
4. Read it back in subsequent skills (e.g. cost-report step 2 reads latest.json for live tier-spend numbers).
Smoke gates
winRate ≥ 0.80on Tier 1 cases (smoke step 23). Lower the threshold by editingscripts/smoke.sh.escalationRateis reported but ungated — adversarial cases are diagnostic.
Env overrides
| Env var | Default | Purpose |
|---|---|---|
BENCH_LLM_BASELINE | unset | =1 runs the OpenAI-compat baseline |
BENCH_LLM_MODEL | models/gemini-2.0-flash | Override the OpenAI-compat model |
BENCH_LLM_BASE_URL | Gemini OpenAI shim | Override endpoint |
BENCH_ANTHROPIC | unset | =1 runs Anthropic baseline (Sonnet 4.6 + Opus 4.7) |
BENCH_ANTHROPIC_MODELS | claude-sonnet-4-6,claude-opus-4-7 | Comma-separated Claude IDs |
BENCH_OUT | timestamped file | Override output path |
BENCH_QUIET=1 | unset | Suppress markdown summary |
API keys auto-pulled from gcloud secrets (GOOGLE_AI_API_KEY, ANTHROPIC_API_KEY); override with BENCH_LLM_API_KEY / BENCH_ANTHROPIC_API_KEY.
Cross-references
ADR-0002 §"Decision 1" / §"Riskiest assumption" · cost-booster-edit/SKILL.md (verification table consumes this skill's output) · cost-report/SKILL.md step 2 (reads runs/latest.json).
Related skills
Lark Openapi ExplorerInstantly explore, test, and generate calls against the full Lark (Feishu) OpenAPI surface without leaving their agent workflow.493k
Lark EventConsume real-time events from Lark/Feishu as structured NDJSON streams inside AI agent workflows.387k16kLark Openapi ExplorerWhen an existing Lark/Feishu skill or CLI command cannot fulfill a specific requirement and they need to discover and invoke the exact native OpenAPI endpoint.386k16k
Just ScrapeQuickly search, crawl, extract structured JSON, or monitor web pages without writing custom scraping code.245k41Lark AppsQuery the current visibility and permission scope of a Lark (Feishu) app without writing HTTP client code.235k16k
Design An InterfaceGenerate multiple radically different interface or API designs in parallel before committing to one.188k198kHow it compares
Pick cost-benchmark to measure routing performance; use cost-booster-edit or cost-booster-route to change the routing logic being tested.
FAQ
What script does cost-benchmark execute?
cost-benchmark runs scripts/bench.mjs against bench/booster-corpus.json. Output includes per-case results and a summary measured-versus-claimed table saved under docs/benchmarks/runs/.
When should cost-benchmark run in ruflo?
cost-benchmark should run before publishing a ruflo release to verify booster win rate did not regress, after expanding the adversarial corpus, or when auditing routing claims from cost-booster skills.