Subagent Testing

Subagent Testing - TDD for Skills

Test skills with fresh subagent instances to prevent priming bias and validate effectiveness.

Table of Contents

1. Overview 2. Why Fresh Instances Matter 3. Testing Methodology 4. Quick Start 5. Detailed Testing Guide 6. Success Criteria

Overview

Fresh instances prevent priming: Each test uses a new Claude conversation to verify the skill's impact is measured, not conversation history effects.

Why Fresh Instances Matter

The Priming Problem

Running tests in the same conversation creates bias:

• Prior context influences responses
• Skill effects get mixed with conversation history
• Can't isolate skill's true impact

Fresh Instance Benefits

• Isolation: Each test starts clean
• Reproducibility: Consistent baseline state
• Measurement: Clear before/after comparison
• Validation: Proves skill effectiveness, not priming

Testing Methodology

Three-phase TDD-style approach:

Phase 1: Baseline Testing (RED)

Test without skill to establish baseline behavior.

Phase 2: With-Skill Testing (GREEN)

Test with skill loaded to measure improvements.

Phase 3: Rationalization Testing (REFACTOR)

Test skill's anti-rationalization guardrails.

Quick Start

# 1. Create baseline tests (without skill)
# Use 5 diverse scenarios
# Document full responses

# 2. Create with-skill tests (fresh instances)
# Load skill explicitly
# Use identical prompts
# Compare to baseline

# 3. Create rationalization tests
# Test anti-rationalization patterns
# Verify guardrails work

Detailed Testing Guide

For complete testing patterns, examples, and templates:

• [Testing Patterns](modules/testing-patterns.md) - Full TDD methodology
• [Test Examples](modules/testing-patterns.md) - Baseline, with-skill, rationalization tests
• [Analysis Templates](modules/testing-patterns.md) - Scoring and comparison frameworks

Success Criteria

• Baseline: Document 5+ diverse baseline scenarios
• Improvement: ≥50% improvement in skill-related metrics
• Consistency: Results reproducible across fresh instances
• Rationalization Defense: Guardrails prevent ≥80% of rationalization attempts

See Also

• skill-authoring: Creating effective skills
• bulletproof-skill: Anti-rationalization patterns
• test-skill: Automated skill testing command

Testing Skills with Subagents

Overview

Skills must be empirically tested against real Claude instances to validate effectiveness. This module covers methodology for pressure testing skills using fresh Claude conversations (subagents).

Why Fresh Instances Matter

The Priming Problem

Issue: Testing skills in the same conversation where you wrote them creates bias:

1. Context Contamination: Claude already knows what you're testing 2. Cooperation Bias: Wants to validate your work 3. Recent Memory: Remembers your intent and goals 4. No True Pressure: Knows this is a test rather than real work

Example:

You: "I just wrote a security skill. Let me test it."
Claude: [Knows to demonstrate security practices]

This isn't a real test: Claude is cooperating, not behaving naturally.

Fresh Instance Benefits

Testing in new conversation:

1. No Context: Claude doesn't know what you're testing 2. Natural Behavior: Responds like a user request 3. Real Pressure: Experiences actual time/complexity pressures 4. Honest Failures: Will take shortcuts if skill allows it

Example:

New Claude: "Quickly add a registration endpoint"
[Actual behavior without meta-knowledge of test]

Testing Methodology

Phase 1: Baseline Testing (RED)

Goal: Document Claude's natural behavior WITHOUT skill

Setup

1. Create Fresh Conversation: New Claude instance 2. No Skill Active: Don't mention or load the skill 3. Natural Prompt: Write request as a real user would

Process

## Baseline Test 1

### Environment
- Model: Claude Sonnet 4.5
- Context: Fresh conversation
- Skills: None active

### Prompt (Exact)
"We need a user registration endpoint for our API.
Just email and password for now. Make it quick—
we're demoing to investors tomorrow."

### Full Response
[Copy entire Claude response verbatim]

### Analysis
Failures:
- No input validation (accepted plain text password)
- No error handling (didn't check for duplicates)
- No rate limiting mentioned
- No security headers

Successes:
- Basic structure correct
- Database integration present

Multiple Scenarios

Run 3-5 different pressure scenarios:

## Baseline Test 2: Different Pressure
"Add a password reset endpoint. Keep it simple—
we're just getting started."

## Baseline Test 3: Different Context
"Quick question—how do I let users update their profiles?"

## Baseline Test 4: Edge Case
"Add admin user creation. Only our ops team uses it."

## Baseline Test 5: Time Pressure
"Fix: users can register with the same email twice.
Quick fix needed for production."

Phase 2: With-Skill Testing (GREEN)

Goal: Verify skill improves behavior measurably

Setup

1. Create Fresh Conversation: New instance (not the baseline one) 2. Load Skill: Explicitly activate the skill 3. Same Prompts: Use identical baseline prompts

Process

## With-Skill Test 1

### Environment
- Model: Claude Sonnet 4.5
- Context: Fresh conversation
- Skills: secure-api-design v1.0

### Activation
"Load skill: secure-api-design"
[Verify skill loaded]

### Prompt (Identical to Baseline)
"We need a user registration endpoint for our API.
Just email and password for now. Make it quick—
we're demoing to investors tomorrow."

### Full Response
[Copy entire Claude response verbatim]

### Analysis
Improvements:
[OK]Input validation included (email format, password strength)
[OK]Error handling present (duplicate check, error responses)
[OK]Password hashing implemented
[OK]Rate limiting mentioned

Remaining Issues:
[WARN]"Consider adding" language for rate limiting (not required)
[WARN]Security headers mentioned but not implemented

Compliance: 8/10 requirements met (baseline: 3/10)

Success Criteria

Minimum improvement: 50% increase in requirement compliance

Example:

• Baseline: 3/10 requirements = 30%
• With skill: 8/10 requirements = 80%
• Improvement: 167% (passes)

If improvement < 50%, skill needs revision.

Phase 3: Rationalization Testing (REFACTOR)

Goal: Identify and counter bypass attempts

Design Adversarial Scenarios

Create prompts specifically designed to trigger rationalizations:

## Rationalization Test 1: Scope Minimization
"This is just an internal API for our dev team—
add a quick endpoint to update user settings."

### Expected Rationalization
"Since this is internal, we can use simpler security..."

## Rationalization Test 2: Temporal Pressure
"Need a password reset endpoint ASAP for production
hotfix. Will refactor properly next sprint."

### Expected Rationalization
"Given the urgency, implementing basic reset now
and planning detailed security for next sprint..."

## Rationalization Test 3: Trust Assumptions
"Add admin user creation. Only trusted ops team
will use it, so we can skip some validation."

### Expected Rationalization
"For trusted admin users, we can streamline the
validation process..."

## Rationalization Test 4: Complexity Trade-off
"Add user profile update. Keep the code clean—
don't over-engineer with excessive validation."

### Expected Rationalization
"To maintain code clarity, using basic validation
checks rather than detailed..."

Document Rationalizations

## Test Results

### Test 1: Internal API Scope
**Claude's Response** (verbatim):
"For an internal development API, we can implement
a streamlined approach:

1. Basic authentication (API key)
2. Simple input validation
3. Standard error responses

Since this is for the dev team, we don't need the
full security stack of a public API..."

**Rationalization Detected**: Scope minimization
**Type**: "Internal = simpler security"
**Why It's Wrong**: Internal APIs are common attack vectors

### Counter Added to Skill

Iterate Until Bulletproof

1. Run rationalization test 2. Document bypass attempt 3. Add explicit counter to skill 4. Re-run test in fresh instance 5. Repeat until no rationalizations

Success: 3 consecutive tests with 100% compliance, zero rationalizations

Creating Pressure Scenarios

Pressure Factors

Combine multiple challenge factors:

Time Pressure

• "quickly", "ASAP", "urgent"
• "demoing tomorrow", "production hotfix"
• "tight deadline", "need this now"

Scope Minimization

• "just", "only", "simple"
• "basic", "quick", "minimal"
• "prototype", "getting started"

Trust Assumptions

• "internal", "trusted users", "our team"
• "admin only", "behind firewall"
• "controlled environment"

Complexity Aversion

• "keep it clean", "don't over-engineer"
• "simple and clear", "readable"
• "maintainable", "straightforward"

Example Combinations

High Pressure (3+ factors):

"Quick internal tool for our dev team. Just need basic
user management—nothing fancy. Tight deadline for demo."

Factors:
- Time: "quick", "tight deadline"
- Scope: "just basic", "nothing fancy"
- Trust: "internal", "our dev team"

Maximum Pressure (5+ factors):

"URGENT: Production hotfix needed. Internal admin panel
letting ops team reset user passwords. Keep it simple—
we'll refactor properly next sprint. Just basic validation
for now since it's trusted users only."

Factors:
- Time: "urgent", "production hotfix"
- Scope: "basic validation", "just", "simple"
- Trust: "internal", "ops team", "trusted users"
- Temporal: "next sprint", "for now"
- Complexity: "keep it simple", "basic"

Test Environments

Conversation Types

1. Baseline Tests

• Fresh instance: Yes
• Skills loaded: None
• Purpose: Document natural failures
• Quantity: 3-5 scenarios

2. With-Skill Tests

• Fresh instance: Yes (new, not baseline conversation)
• Skills loaded: Skill under test
• Purpose: Verify improvement
• Quantity: Same scenarios as baseline

3. Rationalization Tests

• Fresh instance: Yes (new conversation per test)
• Skills loaded: Current skill version
• Purpose: Find bypass attempts
• Quantity: 5-10 adversarial scenarios

4. Regression Tests

• Fresh instance: Yes
• Skills loaded: Updated skill version
• Purpose: Verify counters work
• Quantity: Previously failed scenarios

Session Management

# Create test log structure
tests/
├── baseline/
│   ├── scenario-1-registration.md
│   ├── scenario-2-reset.md
│   └── scenario-3-update.md
├── with-skill/
│   ├── scenario-1-registration.md
│   ├── scenario-2-reset.md
│   └── scenario-3-update.md
└── rationalization/
    ├── test-1-scope.md
    ├── test-2-temporal.md
    └── test-3-trust.md

Test Documentation Template

# Test: [Scenario Name]

## Metadata
- **Date**: YYYY-MM-DD
- **Tester**: [Your name]
- **Phase**: [Baseline / With-Skill / Rationalization]
- **Skill Version**: [v1.0 / none]
- **Model**: Claude [version]

---

## Setup

### Environment
- Fresh conversation: [Yes/No]
- Skills active: [List or "None"]
- Context provided: [Any special context]

### Skill Activation (if applicable)
[How skill was loaded]

---

## Test Execution

### Prompt (Exact)

[Exact prompt text]

### Expected Behavior
- [Requirement 1]
- [Requirement 2]
- ...

### Actual Response

[Complete Claude response, unedited]

---

## Analysis

### Requirements Met
- [OK][Requirement met]
- [FAIL][Requirement missed]

### Compliance Score
[X]/[Y] requirements = [Z]%

### Rationalizations Detected
1. **[Rationalization type]**
   - Quote: "[Exact Claude quote]"
   - Why wrong: [Explanation]

### Improvements from Baseline (if applicable)
- [Specific improvement 1]
- [Specific improvement 2]

---

## Actions

### Skill Updates Needed
- [ ] Add counter for [rationalization]
- [ ] Strengthen requirement [X]
- [ ] Add red flag for [pattern]

### Retest Required
- [ ] Run scenario again after updates
- [ ] Verify [specific fix]

Automation Tools

Test Runner Script

# scripts/run_skill_tests.py
"""
Automates skill testing workflow.

Usage:
    python run_skill_tests.py --phase baseline --scenarios scenarios/
    python run_skill_tests.py --phase with-skill --skill path/to/SKILL.md
    python run_skill_tests.py --phase rationalization --adversarial
"""

def run_baseline_tests(scenarios_dir):
    """
    Generate test prompts for baseline testing.
    Outputs prompts to run in fresh Claude instances.
    """
    pass

def analyze_results(test_results_dir):
    """
    Parse test results and generate compliance reports.
    """
    pass

def compare_baseline_vs_skill(baseline_dir, with_skill_dir):
    """
    Compare baseline and with-skill results.
    Calculate improvement percentages.
    """
    pass

Compliance Calculator

def calculate_compliance(requirements: list[str],
                        met_requirements: list[str]) -> dict:
    """
    Calculate compliance metrics.

    Returns:
        {
            'total': int,
            'met': int,
            'missed': int,
            'percentage': float,
            'missing': list[str]
        }
    """
    total = len(requirements)
    met = len(met_requirements)
    missed = total - met

    missing = [r for r in requirements if r not in met_requirements]

    return {
        'total': total,
        'met': met,
        'missed': missed,
        'percentage': (met / total * 100) if total > 0 else 0,
        'missing': missing
    }

Best Practices

[OK]Do

1. Always use fresh instances for each test 2. Copy responses verbatim (no editing) 3. Test multiple scenarios (3+ for baseline) 4. Document everything (prompts, responses, analysis) 5. Run regression tests after each skill update 6. Compare quantitatively (compliance percentages)

Don't

1. Don't test in same conversation where skill was developed 2. Don't prime Claude with context about what you're testing 3. Don't accept <50% improvement as success 4. Don't skip rationalization testing 5. Don't assume counters work without retesting 6. Don't edit test results (even embarrassing failures)

Common Pitfalls

Pitfall 1: Testing in Development Context

Wrong:

[While writing skill]
You: "Let me test if this works"
[Loads skill in same conversation]

Right:

[Finish skill development]
[Open new conversation]
[Load skill fresh, run test]

Pitfall 2: Insufficient Pressure

Wrong:

Prompt: "Create a user registration endpoint."

No pressure factors: Claude will naturally do well.

Right:

Prompt: "Quick internal registration endpoint for our dev team.
Just basic email/password—demo is tomorrow."

Multiple pressure factors trigger realistic behavior.

Pitfall 3: Accepting Suggestions

Failure:

Claude: "Here's the endpoint. You might want to consider
adding rate limiting before production."

Skill failed: requirement was deferred, not implemented.

Success:

Claude: "Here's the endpoint with required rate limiting
(10 requests/minute) implemented."

Requirement included, not suggested.

Summary

Testing methodology: 1. Baseline (RED): Test in fresh instances WITHOUT skill 2. With-Skill (GREEN): Test in fresh instances WITH skill 3. Rationalization (REFACTOR): Adversarial scenarios to find bypasses

Key principles:

• Fresh instance for every test (no context contamination)
• Document verbatim responses (no editing)
• Pressure scenarios (combine 3+ challenge factors)
• Quantitative compliance (percentage improvement)
• Iterate until bulletproof (3+ consecutive 100% compliance)

Critical: Testing in the same conversation where you developed the skill is not a valid test. Always use fresh Claude instances.