Llm Evals
- 10 repo stars
- Updated January 30, 2026
- vanman2024/ai-dev-marketplace
Test and evaluate LLMs with promptfoo, DeepEval, golden datasets, and Supabase-backed eval tracking, including CI regression testing.
About
Skill providing an LLM testing and evaluation framework with promptfoo, DeepEval, golden datasets, and Supabase-backed eval tracking. A developer uses it to score prompt and model quality, catch regressions in CI, and gate LLM changes before shipping.
- promptfoo & DeepEval
- Golden datasets
- Regression testing in CI
- Faithfulness & relevance metrics
Llm Evals by the numbers
- Data as of Jul 7, 2026 (Skillselion catalog sync)
/plugin marketplace add vanman2024/ai-dev-marketplace/plugin install llm-evals@ai-dev-marketplaceAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| repo stars | ★ 10 |
|---|---|
| Last updated | January 30, 2026 |
| Repository | vanman2024/ai-dev-marketplace ↗ |
What it does
Test and evaluate LLMs with promptfoo, DeepEval, golden datasets, and Supabase-backed eval tracking, including CI regression testing.
README.md
LLM Evals Plugin
Comprehensive LLM testing and evaluation framework for AI applications.
Features
- promptfoo - Prompt regression testing and comparison
- DeepEval - pytest-style LLM evaluation
- Golden Datasets - Curated test cases with expected outputs
- Eval Tracking - Supabase-backed runs, cases, and scores
- CI/CD Integration - GitHub Actions for continuous eval
Quick Start
# Build complete eval system
/llm-evals:build my-evals
# Add specific features
/llm-evals:add promptfoo # Prompt regression testing
/llm-evals:add deepeval # pytest-style LLM tests
/llm-evals:add golden-dataset # Golden test datasets
/llm-evals:add supabase-tracking # Eval run tracking
Components
promptfoo Integration
- YAML-based prompt testing
- Multi-provider comparison
- Assertion-based validation
- HTML report generation
DeepEval Integration
- pytest-style test syntax
- Built-in metrics (faithfulness, relevance, toxicity)
- Custom metric creation
- Async test execution
Golden Datasets
- Structured test cases
- Expected output validation
- Category-based organization
- Version tracking
Supabase Tracking
eval_runs- Track evaluation runseval_cases- Individual test caseseval_scores- Metric scores per case- Dashboard queries
Agents
eval-orchestrator- Coordinates evaluation workflowspromptfoo-specialist- promptfoo configuration and executiondeepeval-specialist- DeepEval test creation and runningdataset-curator- Golden dataset management
Documentation
Related skills
TddFollow test-driven development with a strict red-green-refactor loop when creating reliable features or fixing bugs.544k196k
Test Driven DevelopmentEnforce writing failing tests before any production implementation code.181k264kQaRun conversational QA sessions that turn user-reported bugs into well-written, domain-aware GitHub issues without manual ticket writing.179k196kMigrate To ShoehornAutomatically update TypeScript test files that rely on unsafe `as` type assertions by replacing them with type-safe partial objects from @total-typescript/shoehorn.164k196k
Webapp TestingVerify frontend behavior, debug UI issues, capture screenshots, and inspect logs of a running local web application using Playwright.123k165k
Playwright CliRun browser automation, generate element snapshots, inspect DOM attributes, and execute Playwright tests from the terminal.102k12.3k