Llm Judge

LLM Judge

Compare code implementations across multiple repositories using structured evaluation.

Usage

llm-judge <spec> <repo1> <repo2> [repo3...] [--labels=...] [--weights=...] [--branch=...]

Arguments

Workflow

1. Parse $ARGUMENTS into spec_path, repo_paths, labels, weights, and branch. 2. Validate the spec file, each repo path, and the minimum repo count. 3. Read the spec document into memory. 4. Load this skill and the supporting reference files. 5. Gather facts per repository (one Phase 1 unit per repo) — facts only, no scoring. 6. Validate the repo-agent JSON results before proceeding. 7. Score each dimension (one Phase 2 unit per dimension). 8. Aggregate scores, compute weighted totals, rank repos, and write the report. 9. Display the markdown summary and verify the JSON report.

Hard gates

Sequenced workflow: do not start the next phase until the current gate passes. Each pass condition must be checkable (file on disk, non-empty content, or json.load succeeds)—not “I reviewed internally.”

Parallelism is allowed within a phase (all Phase 1 tasks together; all Phase 2 tasks together), but Phase 2 must not start until Gate B passes, and the user-visible summary must not precede Gate D.

Command Workflow

Step 1: Parse Arguments

Parse $ARGUMENTS to extract:

• spec_path: first positional argument
• repo_paths: remaining positional arguments (must be 2+)
• labels: from --labels or derived from directory names
• weights: from --weights or defaults
• branch: from --branch or main

Default Weights:

{
  "functionality": 30,
  "security": 25,
  "tests": 20,
  "overengineering": 15,
  "dead_code": 10
}

Step 2: Validate Inputs

[ -f "$SPEC_PATH" ] || { echo "Error: Spec file not found: $SPEC_PATH"; exit 1; }

for repo in "${REPO_PATHS[@]}"; do
  [ -d "$repo/.git" ] || { echo "Error: Not a git repository: $repo"; exit 1; }
done

[ ${#REPO_PATHS[@]} -ge 2 ] || { echo "Error: Need at least 2 repositories to compare"; exit 1; }

Step 3: Read Spec Document

SPEC_CONTENT=$(cat "$SPEC_PATH") || { echo "Error: Failed to read spec file: $SPEC_PATH"; exit 1; }
[ -z "$SPEC_CONTENT" ] && { echo "Error: Spec file is empty: $SPEC_PATH"; exit 1; }

Step 4: Load the Skill

Load this llm-judge skill and its reference files into context.

Step 5: Phase 1 - Gather Facts Per Repo

If the agent supports subagents, dispatch one Phase 1 repo agent per repository in parallel; otherwise run the same fact-gathering steps sequentially, one repo at a time — the output is identical either way. Give each unit this brief:

You are a Phase 1 Repo Agent for the LLM Judge evaluation.

**Your Repo:** $LABEL at $REPO_PATH

**Spec Document:**
$SPEC_CONTENT

**Instructions:**
1. Load the **llm-judge** skill's references/repo-agent.md for detailed instructions
2. Follow references/fact-schema.md for the output format
3. Load the **llm-artifacts-detection** skill ([../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../beagle-core/skills/llm-artifacts-detection/SKILL.md), if available) for dead-code/overengineering analysis

Explore the repository and gather facts. Return ONLY valid JSON following the fact schema.

Do NOT score or judge. Only gather facts.

Collect all repo outputs into ALL_FACTS.

Step 6: Validate Phase 1 Results

echo "$FACTS" | python3 -c "import json,sys; json.load(sys.stdin)" 2>/dev/null || { echo "Error: Invalid JSON from $LABEL"; exit 1; }

Step 7: Phase 2 - Score Per Dimension

If the agent supports subagents, dispatch one judge agent per dimension (five total) in parallel; otherwise score each dimension sequentially — identical output. Give each unit this brief:

You are the $DIMENSION Judge for the LLM Judge evaluation.

**Spec Document:**
$SPEC_CONTENT

**Facts from all repos:**
$ALL_FACTS_JSON

**Instructions:**
1. Load the **llm-judge** skill's references/judge-agents.md for detailed instructions
2. Follow references/scoring-rubrics.md for the $DIMENSION rubric

Score each repo on $DIMENSION. Return ONLY valid JSON with scores and justifications.

Step 8: Aggregate Scores

for repo_label in labels:
    scores[repo_label] = {}
    for dimension in dimensions:
        scores[repo_label][dimension] = judge_outputs[dimension]['scores'][repo_label]

    weighted_total = sum(
        scores[repo_label][dim]['score'] * weights[dim] / 100
        for dim in dimensions
    )
    scores[repo_label]['weighted_total'] = round(weighted_total, 2)

ranking = sorted(labels, key=lambda l: scores[l]['weighted_total'], reverse=True)

Step 9: Generate Verdict

Name the winner, explain why they won, and note any close calls or trade-offs.

Step 10: Write JSON Report

mkdir -p .beagle

Write .beagle/llm-judge-report.json with version, timestamp, repo metadata, weights, scores, ranking, and verdict.

Step 11: Display Summary

Render a markdown summary with the scores table, ranking, verdict, and detailed justifications.

Step 12: Verification

python3 -c "import json; json.load(open('.beagle/llm-judge-report.json'))" && echo "Valid report"

Output Shape

The generated report should include:

• repo labels and paths
• per-dimension scores and justifications
• weighted totals and ranking
• a verdict explaining the winner

Reference Files

Scoring Model

Scoring Scale

Phase 1: Gathering Facts Per Repo

For each repository (in parallel via subagents if supported, otherwise sequentially), run a fact-gathering unit with:

You are a Phase 1 Repo Agent for the LLM Judge evaluation.

**Your Repo:** $REPO_LABEL at $REPO_PATH
**Spec Document:**
$SPEC_CONTENT

**Instructions:** Follow the **llm-judge** skill's references/repo-agent.md

Gather facts and return a JSON object following the schema in references/fact-schema.md.

Load the **llm-artifacts-detection** skill ([../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../beagle-core/skills/llm-artifacts-detection/SKILL.md), if available) for dead code and overengineering analysis.

Return ONLY valid JSON, no markdown or explanations.

Collect all repo-agent outputs into ALL_FACTS.

Phase 2: Scoring Per Dimension

After all Phase 1 facts are collected, score the five dimensions (in parallel via subagents if supported, otherwise sequentially), one unit per dimension:

You are the $DIMENSION Judge for the LLM Judge evaluation.

**Spec Document:**
$SPEC_CONTENT

**Facts from all repos:**
$ALL_FACTS_JSON

**Instructions:** Follow the **llm-judge** skill's references/judge-agents.md

Score each repo on $DIMENSION using the rubric in references/scoring-rubrics.md.

Return ONLY valid JSON following the judge output schema.

Aggregation

1. Collect the five judge outputs. 2. Compute each repo's weighted total with the configured weights. 3. Rank repos by weighted total in descending order. 4. Generate a verdict that explains the result and any close calls. 5. Write .beagle/llm-judge-report.json.

Output

Display a markdown summary with scores, ranking, verdict, and detailed justifications.

Verification

Before completing (maps to Hard gates D and E):

1. Gate D: .beagle/llm-judge-report.json exists and json.load succeeds. 2. Gate E / completeness: Every repo label has scores for every dimension; each weighted_total equals the sum over dimensions of (score × weight / 100) using the configured weights; markdown summary matches the JSON report.

Rules

• Always validate inputs before proceeding
• Complete all Phase 1 fact-gathering before any Phase 2 scoring (parallel within a phase if subagents are supported, otherwise sequential)
• Run one Phase 2 unit per dimension
• Every score must have a justification
• Write the JSON report before displaying the summary

Fact Schema

JSON schema for structured facts gathered by Phase 1 Repo Agents.

Full Schema

{
  "repo_label": "string - Display name for this repo",
  "repo_path": "string - Absolute path to repo",
  "git_info": {
    "branch": "string - Current branch name",
    "base": "string - Base branch (usually main)",
    "files_changed": "number - Count of changed files",
    "additions": "number - Lines added",
    "deletions": "number - Lines deleted",
    "diff_summary": "string - Brief description of changes"
  },
  "functionality": {
    "spec_requirements": ["array of requirement strings extracted from spec"],
    "implemented": ["array of requirements found implemented"],
    "missing": ["array of requirements not found"],
    "partially_implemented": ["array of requirements with incomplete implementation"],
    "test_results": {
      "ran": "boolean - Whether tests were executed",
      "framework": "string - pytest, jest, go test, etc.",
      "passed": "number",
      "failed": "number",
      "skipped": "number",
      "error_summary": "string - Brief description of failures if any"
    }
  },
  "security": {
    "findings": [
      {
        "file": "string - File path",
        "line": "number - Line number",
        "issue": "string - Description of security issue",
        "severity": "high | medium | low",
        "category": "string - OWASP category if applicable"
      }
    ],
    "patterns_observed": ["array of positive security patterns found"]
  },
  "tests": {
    "test_count": "number - Total test count",
    "coverage_estimate": "none | low | moderate | high",
    "dry_violations": [
      {
        "file": "string",
        "line": "number",
        "description": "string"
      }
    ],
    "mocking_approach": "string - Description of mocking strategy",
    "test_quality_notes": "string - General observations"
  },
  "overengineering": {
    "abstractions": [
      {
        "file": "string",
        "line": "number",
        "issue": "string - Description of over-abstraction"
      }
    ],
    "defensive_code": [
      {
        "file": "string",
        "line": "number",
        "issue": "string"
      }
    ],
    "config_complexity": "low | medium | high"
  },
  "dead_code": {
    "unused_imports": ["array of file:line references"],
    "unused_functions": ["array of file:line references"],
    "unused_variables": ["array of file:line references"],
    "todo_comments": "number - Count of TODO/FIXME",
    "commented_code_blocks": "number - Count of commented code"
  }
}

Example

{
  "repo_label": "Claude",
  "repo_path": "/path/to/repo-a",
  "git_info": {
    "branch": "main",
    "base": "main",
    "files_changed": 42,
    "additions": 1250,
    "deletions": 380,
    "diff_summary": "Adds auth flow and data export features"
  },
  "functionality": {
    "spec_requirements": ["auth flow", "data export", "rate limiting"],
    "implemented": ["auth flow", "data export"],
    "missing": ["rate limiting"],
    "partially_implemented": [],
    "test_results": {
      "ran": true,
      "framework": "pytest",
      "passed": 45,
      "failed": 2,
      "skipped": 1,
      "error_summary": "2 tests fail on edge case validation"
    }
  },
  "security": {
    "findings": [
      {
        "file": "src/api.py",
        "line": 42,
        "issue": "SQL string concatenation instead of parameterized query",
        "severity": "high",
        "category": "Injection"
      }
    ],
    "patterns_observed": ["input validation present", "no secrets in code", "HTTPS enforced"]
  },
  "tests": {
    "test_count": 48,
    "coverage_estimate": "moderate",
    "dry_violations": [
      {
        "file": "tests/test_api.py",
        "line": 15,
        "description": "Setup code repeated in 5 test functions"
      }
    ],
    "mocking_approach": "Mocks at adapter boundary, uses pytest fixtures",
    "test_quality_notes": "Good isolation, some DRY issues"
  },
  "overengineering": {
    "abstractions": [
      {
        "file": "src/factory.py",
        "line": 1,
        "issue": "Factory pattern for single implementation"
      }
    ],
    "defensive_code": [],
    "config_complexity": "low"
  },
  "dead_code": {
    "unused_imports": ["src/utils.py:3"],
    "unused_functions": [],
    "unused_variables": [],
    "todo_comments": 2,
    "commented_code_blocks": 1
  }
}

Judge Agent Instructions

Instructions for Phase 2 agents that score implementations on a single dimension.

Role

You are a scoring judge. You receive facts gathered from ALL repositories and score each one on YOUR specific dimension using the rubrics in scoring-rubrics.md.

Inputs You Receive

1. Spec Document: The original requirements 2. Facts Array: JSON facts from all repos (output of Phase 1) 3. Your Dimension: One of: functionality, security, tests, overengineering, dead_code

Your Task

Produce a JSON object with scores and justifications for each repo.

Output Schema

{
  "dimension": "functionality",
  "scores": {
    "RepoLabel1": {
      "score": 4,
      "justification": "Clear explanation of why this score was assigned",
      "evidence": ["Specific facts that support this score"]
    },
    "RepoLabel2": {
      "score": 5,
      "justification": "...",
      "evidence": ["..."]
    }
  },
  "ranking": ["RepoLabel2", "RepoLabel1"],
  "notes": "Optional comparative notes"
}

Scoring Process

1. Read the rubric for your dimension from scoring-rubrics.md 2. For each repo's facts:

• Extract the relevant section (e.g., facts.functionality for functionality judge)
• Apply the rubric criteria
• Assign a 1-5 score
• Write a clear justification citing specific evidence

3. Rank the repos by score (highest first)

Dimension-Specific Instructions

Functionality Judge

Focus on facts.functionality:

• Compare spec_requirements to implemented and missing
• Weight test results heavily (test_results.passed vs failed)
• Consider partially_implemented as half credit

Security Judge

Focus on facts.security:

• Count and weight findings by severity
• High severity = major deduction
• Positive patterns_observed can offset minor issues

Tests Judge

Focus on facts.tests:

• Evaluate coverage_estimate
• Count dry_violations (more = worse)
• Consider mocking_approach quality
• Raw test_count relative to codebase size

Overengineering Judge

Focus on facts.overengineering:

• Count abstractions issues
• Count defensive_code issues
• Consider config_complexity
• FEWER issues = HIGHER score (inverse)

Dead Code Judge

Focus on facts.dead_code:

• Sum all unused items
• Weight unused_functions > unused_imports
• Count todo_comments and commented_code_blocks
• FEWER issues = HIGHER score (inverse)

Important Rules

1. Use the rubric - Don't invent criteria 2. Be consistent - Apply the same standards to all repos 3. Cite evidence - Every score needs justification from facts 4. Be comparative - Rankings should reflect relative quality 5. Valid JSON only - Output must be parseable

Repo Agent Instructions

Instructions for Phase 1 agents that gather facts from a single repository.

Role

You are a fact-gathering agent. Your job is to explore a repository and extract structured facts WITHOUT making judgments or assigning scores. Scoring happens in Phase 2 by separate judge agents.

Inputs You Receive

1. Spec Document: The requirements/plan that was given to the LLM to implement 2. Repo Path: Absolute path to the repository you're analyzing 3. Repo Label: Display name for this repo (e.g., "Claude", "GPT-4") 4. Branch Info: Which branch to compare (default: current vs main)

Your Task

Produce a JSON object following the schema in fact-schema.md.

Step-by-Step Process

1. Gather Git Info

# Get branch name
git -C $REPO_PATH rev-parse --abbrev-ref HEAD

# Get diff stats
git -C $REPO_PATH diff --stat main...HEAD

# Count files changed
git -C $REPO_PATH diff --name-only main...HEAD | wc -l

2. Analyze Functionality

1. Read the spec document carefully 2. Extract discrete requirements as a list 3. Explore the codebase to determine which requirements are implemented 4. Run tests if available:

# Detect and run tests
cd $REPO_PATH

# Python
if [ -f pytest.ini ] || [ -f pyproject.toml ] || [ -d tests ]; then
  pytest --tb=short 2>&1
fi

# JavaScript/TypeScript
if [ -f package.json ]; then
  npm test 2>&1 || yarn test 2>&1
fi

# Go
if [ -f go.mod ]; then
  go test ./... 2>&1
fi

3. Analyze Security

Look for common vulnerabilities:

• SQL injection (string concatenation in queries)
• Command injection (unsanitized shell commands)
• XSS (unsanitized user input in HTML)
• Hardcoded secrets (API keys, passwords)
• Missing input validation
• Insecure deserialization

Also note positive patterns:

• Input validation present
• Parameterized queries
• Authentication checks
• Rate limiting

4. Analyze Tests

• Count test files and test functions
• Look for DRY violations (repeated setup code)
• Assess mocking strategy
• Estimate coverage (file count ratio, critical paths tested)

5. Analyze Overengineering

Use patterns from the llm-artifacts-detection skill (../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md):

• Unnecessary abstractions (interfaces with single impl)
• Factory patterns for simple objects
• Excessive defensive coding
• Over-configuration

6. Analyze Dead Code

• Unused imports (grep for imports, check usage)
• TODO/FIXME comments
• Commented-out code blocks
• Unused functions/variables

Output Format

Return ONLY the JSON object. No markdown, no explanations. The JSON must be valid and follow fact-schema.md.

Important Rules

1. Do not score - Only gather facts 2. Be thorough - Check all changed files 3. Be specific - Include file:line references 4. Be objective - Report what you find, not opinions 5. Use the skill - Load the llm-artifacts-detection skill (../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md) for dead code/overengineering

Scoring Rubrics

Detailed rubrics for each of the 5 judging dimensions. Judges use these to assign consistent 1-5 scores.

General Scoring Scale

---

Functionality (30% weight)

Evaluates whether the implementation meets the spec requirements and works correctly.

Key Evidence:

• functionality.implemented vs functionality.spec_requirements
• functionality.test_results.passed vs functionality.test_results.failed
• functionality.missing and functionality.partially_implemented

---

Security (25% weight)

Evaluates security posture and absence of vulnerabilities.

Severity Weights:

• High: SQL injection, command injection, auth bypass, secrets in code
• Medium: XSS, CSRF, insecure deserialization, missing input validation
• Low: Information disclosure, verbose errors, missing security headers

Key Evidence:

• security.findings (count and severity)
• security.patterns_observed

---

Test Quality (20% weight)

Evaluates test coverage, DRY adherence, and testing practices.

Key Evidence:

• tests.coverage_estimate
• tests.dry_violations (count)
• tests.mocking_approach
• tests.test_count relative to codebase size

---

Overengineering (15% weight)

Evaluates simplicity and absence of unnecessary complexity.

Key Evidence:

• overengineering.abstractions (count)
• overengineering.defensive_code (count)
• overengineering.config_complexity

---

Dead Code (10% weight)

Evaluates cleanliness and absence of unused/obsolete code.

Key Evidence:

• dead_code.unused_imports (count)
• dead_code.unused_functions (count)
• dead_code.todo_comments
• dead_code.commented_code_blocks