Agent Evaluation

Agent Evaluation

Overview

LLM-as-judge evaluation framework that scores AI-generated content on 5 dimensions using a 1-5 rubric. Agents evaluate outputs, compute a weighted composite score, and emit a structured verdict with evidence citations.

Core principle: Systematic quality verification before claiming completion. Agent-studio currently has no way to verify agent output quality — this skill fills that gap.

When to Use

Always:

• Before marking a task complete (pair with verification-before-completion)
• After a plan is generated (evaluate plan quality)
• After code review outputs (evaluate review quality)
• During reflection cycles (evaluate agent responses)
• When comparing multiple agent outputs

Don't Use:

• For binary pass/fail checks (use verification-before-completion instead)
• For security audits (use security-architect skill)
• For syntax/lint checking (use pnpm lint:fix)

The 5-Dimension Rubric

Every evaluation scores all 5 dimensions on a 1-5 scale:

Scoring Scale (1-5)

Execution Process

Step 1: Load the Output to Evaluate

Identify what is being evaluated:

- Agent response (text)
- Plan document (file path)
- Code review output (text/file)
- Skill invocation result (text)
- Task completion claim (TaskGet metadata)

Step 2: Score Each Dimension

For each of the 5 dimensions, provide:

1. Score (1-5): The numeric score 2. Evidence: Direct quote or file reference from the evaluated output 3. Rationale: Why this score was given (1-2 sentences)

Dimension 1: Accuracy

Checklist:
- [ ] Claims are factually correct (verify against codebase if possible)
- [ ] No hallucinated file paths, function names, or API calls
- [ ] Numbers and counts are accurate
- [ ] No contradictions with existing documentation

Dimension 2: Groundedness

Checklist:
- [ ] Claims cite specific files, line numbers, or task IDs
- [ ] Recommendations reference observable evidence
- [ ] No unsupported assertions ("this is probably X")
- [ ] Code examples use actual project patterns

Dimension 3: Coherence

Checklist:
- [ ] Logical flow from problem → analysis → recommendation
- [ ] No internal contradictions
- [ ] Terminology is consistent throughout
- [ ] Steps are in a rational order

Dimension 4: Completeness

Checklist:
- [ ] All required aspects of the task are addressed
- [ ] Edge cases are mentioned (if relevant)
- [ ] No critical gaps that would block action
- [ ] Follow-up steps are included

Dimension 5: Helpfulness

Checklist:
- [ ] Provides actionable next steps (not just observations)
- [ ] Concrete enough to act on without further clarification
- [ ] Reduces ambiguity rather than adding it
- [ ] Appropriate for the intended audience

Step 3: Compute Weighted Composite Score

composite = (accuracy × 0.30) + (groundedness × 0.25) + (completeness × 0.20) + (coherence × 0.15) + (helpfulness × 0.10)

Step 4: Determine Verdict

Step 5: Emit Structured Verdict

Output the verdict in this format:

## Evaluation Verdict

**Output Evaluated**: [Brief description of what was evaluated]
**Evaluator**: [Agent name / task ID]
**Date**: [ISO 8601 date]

### Dimension Scores

| Dimension     | Score | Weight | Weighted Score |
| ------------- | ----- | ------ | -------------- |
| Accuracy      | X/5   | 30%    | X.X            |
| Groundedness  | X/5   | 25%    | X.X            |
| Completeness  | X/5   | 20%    | X.X            |
| Coherence     | X/5   | 15%    | X.X            |
| Helpfulness   | X/5   | 10%    | X.X            |
| **Composite** |       |        | **X.X / 5.0**  |

### Evidence Citations

**Accuracy (X/5)**:

> [Direct quote or file:line reference]
> Rationale: [Why this score]

**Groundedness (X/5)**:

> [Direct quote or file:line reference]
> Rationale: [Why this score]

**Completeness (X/5)**:

> [Direct quote or file:line reference]
> Rationale: [Why this score]

**Coherence (X/5)**:

> [Direct quote or file:line reference]
> Rationale: [Why this score]

**Helpfulness (X/5)**:

> [Direct quote or file:line reference]
> Rationale: [Why this score]

### Verdict: [EXCELLENT | GOOD | ADEQUATE | POOR | FAILING]

**Summary**: [1-2 sentence overall assessment]

**Required Actions** (if verdict is ADEQUATE or worse):

1. [Specific improvement needed]
2. [Specific improvement needed]

Usage Examples

Evaluate a Plan Document

// Load plan document
Read({ file_path: '.claude/context/plans/auth-design-plan-2026-02-21.md' });

// Evaluate against 5-dimension rubric
Skill({ skill: 'agent-evaluation' });
// Provide the plan content as the output to evaluate

Evaluate Agent Response Before Completion

// Agent generates implementation summary
// Before marking task complete, evaluate the summary quality
Skill({ skill: 'agent-evaluation' });
// If composite < 3.5, request improvements before TaskUpdate(completed)

Evaluate Code Review Output

// After code-reviewer runs, evaluate the review quality
Skill({ skill: 'agent-evaluation' });
// Ensures review is grounded in actual code evidence, not assertions

Batch Evaluation (comparing two outputs)

// Evaluate output A
// Save verdict A
// Evaluate output B
// Save verdict B
// Compare composites → choose higher scoring output

Integration with Verification-Before-Completion

The recommended quality gate pattern:

// Step 1: Do the work
// Step 2: Evaluate with agent-evaluation
Skill({ skill: 'agent-evaluation' });
// If verdict is POOR or FAILING → rework before proceeding
// If verdict is ADEQUATE or better → proceed to verification
// Step 3: Final gate
Skill({ skill: 'verification-before-completion' });
// Step 4: Mark complete
TaskUpdate({ taskId: 'X', status: 'completed' });

Iron Laws

1. NO COMPLETION CLAIM WITHOUT EVALUATION EVIDENCE — If composite score < 2.5 (POOR or FAILING), rework the output before marking any task complete. 2. ALWAYS score all 5 dimensions — never skip dimensions to save time; each dimension catches different failure modes (accuracy ≠ completeness ≠ groundedness). 3. ALWAYS cite specific evidence for every dimension score — "Evidence: [file:line or direct quote]" is mandatory, not optional. Assertions without grounding are invalid. 4. ALWAYS use the weighted composite — accuracy×0.30 + groundedness×0.25 + completeness×0.20 + coherence×0.15 + helpfulness×0.10. Never use simple average. 5. NEVER evaluate before the work is complete — evaluating incomplete outputs produces falsely low scores and wastes context budget.

Anti-Patterns

Assigned Agents

This skill is used by:

• qa — Primary: validates test outputs and QA reports before completion
• code-reviewer — Supporting: evaluates code review quality
• reflection-agent — Supporting: evaluates agent responses during reflection cycles

Memory Protocol (MANDATORY)

Before starting:

cat .claude/context/memory/learnings.md

Check for:

• Previous evaluation scores for similar outputs
• Known quality patterns in this codebase
• Common failure modes for this task type

After completing:

• Evaluation pattern found -> .claude/context/memory/learnings.md
• Quality issue identified -> .claude/context/memory/issues.md
• Decision about rubric weights -> .claude/context/memory/decisions.md

ASSUME INTERRUPTION: Your context may reset. If it's not in memory, it didn't happen.

Invoke the agent-evaluation skill and follow it exactly as presented to you

<!-- Agent: developer | Task: #2 | Session: 2026-02-21 -->

Research Requirements: agent-evaluation Skill

Date: 2026-02-21 Query Intent: LLM-as-judge evaluation frameworks, multi-dimension rubrics, AI output quality measurement

Research Methodology

Key Findings

Design Constraints (mapped to skill design)

1. 5-dimension rubric (accuracy, groundedness, coherence, completeness, helpfulness) — mirrors established LLM evaluation literature (G-Eval, MT-Bench). Weights: accuracy 30%, groundedness 25%, completeness 20%, coherence 15%, helpfulness 10%.

2. Evidence citation required — Every dimension score must cite a direct quote or file:line reference. Prevents grade inflation from vague scoring.

3. Verdict threshold gates — Composite <2.5 = POOR/FAILING -> must rework before completion. Enables integration with verification-before-completion as a pre-gate.

Non-Goals

• NOT a performance benchmark (latency, cost) — use separate tooling
• NOT a security audit — use security-architect
• NOT a code linter — use pnpm lint:fix

Codebase Patterns Examined

• .claude/skills/verification-before-completion/SKILL.md — Pairs with this skill as downstream gate
• .claude/skills/checklist-generator/SKILL.md — Similar quality gate pattern
• .claude/skills/tdd/SKILL.md — Reference skill for SKILL.md structure

Sources

• Researcher-provided context (task-2 batch A)
• LMSYS Chatbot Arena evaluation methodology (established rubric patterns)
• G-Eval paper patterns (multi-dimension LLM evaluation)
• Exa MCP unavailable — used researcher agent context as primary source
• Fallback: WebFetch + arXiv for LLM evaluation methodology validation

agent-evaluation Rules

Purpose

LLM-as-judge evaluation framework with 5-dimension rubric (accuracy, groundedness, coherence, completeness, helpfulness) for scoring AI-generated content quality with weighted composite scores and evidence citations

Best Practices

• Always evaluate all 5 dimensions before computing composite score
• Cite specific evidence from the output being evaluated for each dimension score
• Use the weighted composite (not simple average) for final verdict
• Pair with verification-before-completion for pre-completion quality gates
• Document evaluation verdicts in task metadata for traceability

Integration Points

See SKILL.md for complete documentation.

agent-evaluation Implementation Template

Goal

• Define target outcome and acceptance criteria.

TDD

1. Red 2. Green 3. Refactor

Verification

• lint
• format
• targeted tests