Ai Evals
Design rubrics, test cases, and measurement loops so LLM features have evals that act like a product requirements doc for model quality.
Overview
AI Evals is an agent skill most often used in Build (also Ship) that helps solo builders create rubrics, test cases, and systematic measurements so LLM product quality is defined and trackable like a PRD.
Install
npx skills add https://github.com/refoundai/lenny-skills --skill ai-evalsWhat is this skill?
- Lenny-guest framing: evals as a distinct skill from traditional software testing (error analysis, open coding workflow r
- Guides rubrics, benchmarks, and systematic tests to measure AI output quality
- Starts from what you are evaluating and what “good” looks like before implementation details
- Positions evals as the PRD when “the model is the product”
- Sourced from refoundai/lenny-skills ai-evals pack aggregating practitioner insights
- Lenny-skills pack cites 2 guests and 2 mentions on AI evals themes
Adoption & trust: 1.4k installs on skills.sh; 1k GitHub stars; 3/3 security scanners passed (skills.sh audits).
What problem does it solve?
Your AI feature ships prompt changes on gut feel and you have no rubric or test suite to know if quality improved or regressed.
Who is it for?
Indie builders adding a chat, codegen, or classification agent who need a first eval harness before scaling usage.
Skip if: Traditional non-LLM CRUD apps with deterministic tests only, or teams wanting fully automated CI eval infrastructure without any product definition work.
When should I use this skill?
User is building evals for LLM products, measuring model quality, creating test cases, designing rubrics, or systematically measuring AI output quality.
What do I get? / Deliverables
You end with a concrete eval approach—what to measure, example cases, and rubric shape—ready to run repeatedly as you change models or prompts.
- Eval approach doc: rubric dimensions, test case list, and measurement method
- Actionable next steps to implement runs (manual or scripted) against golden cases
Recommended Skills
Journey fit
Spans multiple journey phases - primary shelf plus alternate fits below.
Build/agent-tooling is canonical because eval design is part of shipping reliable AI features—defining what “good” means before and while you wire prompts, tools, and retrieval. Agent-tooling covers systematic quality measurement for LLM products, distinct from generic unit tests or one-off prompt tweaks.
Where it fits
Define success criteria and 15–20 golden prompts before wiring a new RAG assistant into your app.
Run a structured eval pass before swapping model versions so you can compare scores on the same rubric.
Refresh test cases after user-reported failure modes so the next prompt patch has a measurable target.
How it compares
Product-quality eval design for LLM features—not the same as linting, unit tests alone, or generic brainstorming.
Common Questions / FAQ
Who is ai-evals for?
Product-minded solo builders and small teams building LLM features who need systematic quality measurement beyond ad-hoc prompt tryouts.
When should I use ai-evals?
In Build while defining agent behavior and test cases; in Ship when preparing release checks or regression suites after prompt, tool, or model changes.
Is ai-evals safe to install?
It is advisory workflow content from a skills pack; review the Security Audits panel on this Prism page and avoid piping production secrets into eval fixtures.
SKILL.md
READMESKILL.md - Ai Evals
# AI Evaluation (Evals) - All Guest Insights *2 guests, 2 mentions* --- ## Hamel Husain & Shreya Shankar *Hamel Husain & Shreya Shankar* > "Both the chief product officers of Anthropic and OpenAI shared that evals are becoming the most important new skill for product builders." **Insight:** The guests explicitly define this as a 'new skill' that is distinct from traditional software testing or general AI strategy. It involves a specific multi-step workflow (Error Analysis, Open Coding, A ## Brendan Foody *Brendan Foody* > "If the model is the product, then the eval is the product requirement document." **Insight:** The guest explicitly states we are entering the 'era of evals' and describes it as a core bottleneck for AI labs. It involves creating rubrics, benchmarks, and systematic tests to measure model capabi --- name: ai-evals description: Help users create and run AI evaluations. Use when someone is building evals for LLM products, measuring model quality, creating test cases, designing rubrics, or trying to systematically measure AI output quality. --- # AI Evals Help the user create systematic evaluations for AI products using insights from AI practitioners. ## How to Help When the user asks for help with AI evals: 1. **Understand what they're evaluating** - Ask what AI feature or model they're testing and what "good" looks like 2. **Help design the eval approach** - Suggest rubrics, test cases, and measurement methods 3. **Guide implementation** - Help them think through edge cases, scoring criteria, and iteration cycles 4. **Connect to product requirements** - Ensure evals align with actual user needs, not just technical metrics ## Core Principles ### Evals are the new PRD Brendan Foody: "If the model is the product, then the eval is the product requirement document." Evals define what success looks like in AI products—they're not optional quality checks, they're core specifications. ### Evals are a core product skill Hamel Husain & Shreya Shankar: "Both the chief product officers of Anthropic and OpenAI shared that evals are becoming the most important new skill for product builders." This isn't just for ML engineers—product people need to master this. ### The workflow matters Building good evals involves error analysis, open coding (writing down what's wrong), clustering failure patterns, and creating rubrics. It's a systematic process, not a one-time test. ## Questions to Help Users - "What does 'good' look like for this AI output?" - "What are the most common failure modes you've seen?" - "How will you know if the model got better or worse?" - "Are you measuring what users actually care about?" - "Have you manually reviewed enough outputs to understand failure patterns?" ## Common Mistakes to Flag - **Skipping manual review** - You can't write good evals without first understanding failure patterns through manual trace analysis - **Using vague criteria** - "The output should be good" isn't an eval; you need specific, measurable criteria - **LLM-as-judge without validation** - If using an LLM to judge, you must validate that judge against human experts - **Likert scales over binary** - Force Pass/Fail decisions; 1-5 scales produce meaningless averages ## Deep Dive For all 2 insights from 2 guests, see `references/guest-insights.md` ## Related Skills - Building with LLMs - AI Product Strategy - Evaluating New Technology