Tdd

Test-Driven Development (TDD)

Overview

This skill implements Canon TDD with AI-specific guardrails:

1. Build or update a scenario list. 2. Execute exactly one scenario as a runnable test. 3. Prove RED. 4. Implement minimum change for GREEN. 5. Optionally refactor. 6. Repeat until scenario list is empty.

When to Use

Use for:

• New features
• Bug fixes
• Behavior changes
• Repository-scale patching driven by tests
• AI-assisted code generation where tests are executable specifications

Ask human approval before bypassing only for:

• Throwaway prototypes
• Purely declarative config edits with no execution path
• One-off migration scripts that will not be maintained

The Iron Law

NO PRODUCTION CODE WITHOUT A FAILING TEST FIRST

If code was written first, discard and restart from RED.

Canon Loop

Step 0: Create/refresh scenario backlog

Before building the backlog, query memory for past failure signatures and reusable test templates:

Skill({ skill: 'memory-search' }); // query: "<feature-name> test failure signatures"

Read .claude/context/memory/learnings.md for recurring anti-patterns relevant to this task.

Then:

• Keep a short ordered list of test scenarios for this task.
• Prioritize by design signal and risk, not by implementation convenience.
• Add discovered scenarios during execution.
• Reuse templates from memory — do not repeat failure patterns already documented.

Step 1: Pick exactly one scenario and write one runnable test

• One behavior per cycle.
• Use clear behavior names.
• Favor real collaborators; mock only external boundaries.

Step 2: Prove RED

• Run the narrowest test command.
• Failure must be due to missing behavior, not syntax or setup errors.
• Record red evidence (test file and failing assertion message).

Step 3: Implement minimum GREEN patch

• Implement only what current red test requires.
• No speculative APIs or unrelated cleanup.
• Keep patch bounded to current scenario.

Step 4: Prove GREEN

• Re-run narrow test command.
• Run impacted suite (or package-level test set).
• Confirm no regressions.

Flakiness Gate (mandatory for async, hook, or nondeterministic tests):

For tests that involve async I/O, stop hooks, timers, or file system operations, a single pass is insufficient. Require 3 consecutive passes before declaring GREEN:

# Run 3 times — all 3 must pass
node --test tests/hooks/routing-guard.test.cjs && \
node --test tests/hooks/routing-guard.test.cjs && \
node --test tests/hooks/routing-guard.test.cjs

A test that passes once and fails on the second run is RED, not GREEN. Do not advance to Step 5 until 3 consecutive passes are confirmed.

Mutation Testing Gate (security-critical code only):

For security hooks, routing validators, auth logic, and any code path that controls access or trust decisions, run Stryker mutation testing after achieving GREEN to verify that tests genuinely catch faults and are not vacuously passing.

# Run Stryker mutation testing (threshold: 85%)
npx stryker run
# Require mutationScore >= 85 in stryker.config.json

For fast-check-based property tests on security hooks, the fail-closed property is the mutation-equivalent gate:

// fast-check fail-closed property — must hold for any input
fc.assert(
  fc.property(fc.anything(), input => {
    const result = securityHook(input);
    // Hook must NEVER return allow=true for malformed/unexpected input
    expect(result.allow).not.toBe(true);
  })
);

Skip this gate for non-security application code (Step 4 → Step 5 directly).

Step 5: Optional refactor

• Refactor only with green tests.
• Re-run the same test set after refactor.

Step 5.5: Property-Based Testing (recommended for utility functions and security hooks)

After refactor (or after Step 4 for security-critical code), consider supplementing example-based tests with property-based tests. PBT achieves 23.1–37.3% pass@1 improvement over example-based TDD alone for LLM code generation (arXiv:2506.18315) by breaking the self-deception cycle.

When to invoke:

• Utility functions (encode/decode, parsers, serializers, calculators)
• Security hooks (input validators, sanitizers, access control logic)
• Any function where invariants, round-trip properties, or mathematical properties can be stated

Invocation:

Skill({ skill: 'property-based-testing' });

Key property patterns to identify:

PBT is a supplement to Canon TDD, not a replacement. Canon RED/GREEN/REFACTOR completes first; PBT runs after GREEN is confirmed.

Step 6: Repeat until backlog empty

AI-Assisted Guardrails

• Use tests as executable prompt context; keep prompts short and test-focused.
• Prefer deterministic tests (stable fixtures, no nondeterministic ordering).
• Use bounded repair loops: max 3 repair attempts per scenario before redesign.
• Run anti-test-hacking checks:
• Verify changed assertions still express original requirement.
• Add at least one negative test for bug-fix tasks.
• Ensure code does not branch on test-only artifacts.

Memory Acceleration Layer

Use lightweight memory only to reduce repeated setup and triage:

• preferred repo-local test/lint/format commands
• recurring failure signatures and short fix summaries
• recurring anti-pattern reminders
• reusable scenario templates

Reference: references/tdd-memory-profile.md

Hard rules:

• memory never bypasses RED proof
• memory never changes Canon sequence
• keep profile bounded and low-noise

Test-Driven Prompting (TDP) — 2026 Standard Pattern

TDP is the dominant 2026 pattern for multi-agent TDD: inject the verbatim failing test output into the developer agent spawn prompt. This eliminates interpretation errors — the developer sees exactly what the test runner sees.

Pattern

Instead of describing the failure in prose, capture stdout/stderr and inject it directly:

// Step 1: Run test and capture raw output
const { execSync } = require('child_process');
let testOutput = '';
try {
  execSync('node --test tests/hooks/routing-guard.test.cjs', { encoding: 'utf-8' });
} catch (e) {
  testOutput = e.stdout + e.stderr; // Verbatim failure output
}

// Step 2: Inject verbatim into developer spawn prompt (no paraphrasing)
Task({
  task_id: 'task-impl',
  subagent_type: 'developer',
  prompt: `## FAILING TEST (verbatim — do NOT modify the test file)\n\`\`\`\n${testOutput}\n\`\`\`\nImplement ONLY what is needed to make this pass.`,
});

Why TDP Works

• Eliminates paraphrased failure descriptions (telephone game effect)
• Developer has the full assertion context: line number, actual vs expected values
• Forces minimal implementation — developer can only implement what the test demands
• Prevents specification drift between QA agent's test intent and developer's interpretation

TDP + Multi-Agent TDD Decomposition

Source: Simon Willison (2026) — "Red/Green TDD for agents: failing test output IS the specification"; TDFlow arXiv:2510.23761.

Autonomous TDD with ralph-loop (Session-Persistent Iteration)

For repository-scale TDD where sessions may be interrupted, wire ralph-loop (Mode 2 — router-managed) to maintain the TDD scenario backlog across interruptions:

TDD State Schema

Maintain a TDD-specific state file at .claude/context/runtime/tdd-state.json:

{
  "scenarios": [
    {
      "id": "sc-001",
      "description": "routing-guard blocks Write on creator paths",
      "status": "pending"
    },
    { "id": "sc-002", "description": "spawn-token-guard warns at 80K tokens", "status": "green" }
  ],
  "completedScenarios": [
    {
      "id": "sc-002",
      "evidenceCommand": "node --test tests/hooks/spawn-token-guard.test.cjs",
      "passedAt": "2026-03-12T10:00:00Z"
    }
  ],
  "currentScenario": "sc-001",
  "evidenceLog": [
    {
      "scenarioId": "sc-001",
      "phase": "red",
      "output": "AssertionError: expected exit code 2, got 0",
      "timestamp": "..."
    }
  ]
}

Resume Pattern

At the start of each iteration, read the TDD state file:

// Step 0 — before building/refreshing backlog
const state = JSON.parse(
  fs.readFileSync('.claude/context/runtime/tdd-state.json', 'utf-8') || '{}'
);
const completedIds = (state.completedScenarios || []).map(s => s.id);
const remaining = (state.scenarios || []).filter(s => !completedIds.includes(s.id));
// Pick next scenario from remaining — never re-run completed ones

Integration with ralph-loop Mode 2

1. Router spawns qa agent with { task_id, subagent_type: 'qa', prompt: TDP_PROMPT + verbatim state } 2. qa writes test → runs → captures output → updates tdd-state.json (phase: red) 3. Router spawns developer with TDP prompt (verbatim test output injected) 4. developer implements → updates tdd-state.json (phase: green) 5. Router checks remaining.length === 0 → emit RALPH_AUDIT_COMPLETE_NO_FINDINGS 6. If remaining > 0 → loop back to step 1 with next scenario

Anti-pattern: Never re-run scenarios already marked green in state — this wastes iterations and may corrupt evidence logs.

Repository-Scale and Class-Level Guidance

• For repository-scale work, decompose by failing test cluster and assign one cluster per loop.
• For class-level synthesis, derive a method dependency order and implement one method at a time with method-level public tests.
• Keep long-context pressure low by limiting each loop to one scenario and one patch objective.

Verification Checklist

• [ ] Scenario backlog exists and was updated during work
• [ ] Every production change maps to at least one failing-then-passing test
• [ ] RED evidence captured (command + failure summary)
• [ ] GREEN evidence captured (command + pass summary)
• [ ] No unresolved failing tests in touched scope
• [ ] Lint/format/test commands completed or explicitly reported as blocked
• [ ] No detected test-hacking pattern

Pre-Completion Commands (Project-Scoped)

Use the project's actual commands. Typical sequence:

# 1) targeted test
pnpm test <target>
# 2) impacted suite
pnpm test
# 3) lint
pnpm lint
# 4) format check
pnpm format:check

If the repo uses different scripts, replace these with local equivalents and report exactly what ran.

Rationalization Countermeasures

• "I will add tests later" -> stop and write current red test.
• "This is too small to test" -> write one minimal behavior test.
• "I already manually tested" -> manual runs do not replace executable regression tests.
• "I spent too long to delete pre-test code" -> sunk cost; restart from RED.

Related Files

• references/research-requirements.md
• references/tdd-memory-profile.md
• testing-anti-patterns.md
• rules/tdd.md
• templates/implementation-template.md

Research Basis

This skill is aligned with:

• Martin Fowler TDD (Dec 11, 2023)
• Kent Beck Canon TDD (Dec 11, 2023)
• Rafique & Misic meta-analysis, IEEE TSE DOI:10.1109/TSE.2012.28
• LLM4TDD (arXiv:2312.04687)
• Test-Driven Development for Code Generation (arXiv:2402.13521)
• Tests as Prompt (arXiv:2505.09027)
• SWE-Flow (arXiv:2506.09003)
• TDFlow (arXiv:2510.23761)
• Scaling TDD from Functions to Classes (arXiv:2602.03557)

Memory Protocol

Before starting: Read .claude/context/memory/learnings.md

After completing:

• New pattern -> .claude/context/memory/learnings.md
• Issue found -> .claude/context/memory/issues.md
• Decision made -> .claude/context/memory/decisions.md

Assume interruption: if it is not in memory, it did not happen.

Agent-Studio TDD Extensions (2026)

Hook Testing Pattern

Hooks use stdin/stdout JSON protocol:

const proc = require('child_process').spawn('node', ['.claude/hooks/routing/routing-guard.cjs'], {
  shell: false,
});
proc.stdin.write(JSON.stringify({ tool_name: 'Write', tool_input: {} }));
proc.stdin.end();
// Exit 0=allow, 2=block

Memory TDD

Mock MemoryRecord. Test confidence gate (threshold 0.7). Use atomic writes.

Property-Based Testing

Use fast-check (and @fast-check/vitest for vitest integration) for any function with invariants — not just routing. fast-check 3.x (2025) adds improved unicode, date, and bigint arbitraries.

Routing invariant (existing):

import fc from 'fast-check';
fc.assert(
  fc.property(fc.string(), intent => {
    return typeof routeIntent(intent) === 'string';
  })
);

Memory serialization roundtrip (new):

// Property: serialize(deserialize(x)) === x for all JSON-serializable values
fc.assert(
  fc.property(fc.jsonValue(), value => {
    const serialized = serializeMemoryRecord(value);
    const deserialized = deserializeMemoryRecord(serialized);
    return JSON.stringify(deserialized) === JSON.stringify(value);
  })
);

Hook validation invariant (new):

// Property: for any tool input, isValidInput(x) === !isBlocked(x)
// (validation and blocking must be inverses)
fc.assert(
  fc.property(fc.record({ tool_name: fc.string(), tool_input: fc.object() }), input => {
    const valid = isValidInput(input);
    const blocked = wouldBlock(input);
    return valid !== blocked || (!valid && blocked); // blocked implies invalid
  })
);

Path normalization idempotency (new):

// Property: normalize(normalize(path)) === normalize(path) (idempotent)
fc.assert(
  fc.property(fc.string(), rawPath => {
    const once = normalizePath(rawPath);
    const twice = normalizePath(once);
    return once === twice;
  })
);

Schema validation stability (new):

// Property: validate(schema, x) never throws uncaught exception for any input
fc.assert(
  fc.property(fc.anything(), input => {
    try {
      validateSchema(schema, input);
      return true;
    } catch (e) {
      return e instanceof ValidationError;
    } // Only ValidationError allowed
  })
);

Contract Testing

Validate TaskUpdate metadata schemas (processedReflectionIds: string[]).

Multi-Agent TDD Decomposition (2026 Standard)

Based on TDFlow (arXiv:2510.23761, 94.3% SWE-Bench Verified), monolithic TDD agents score 60–70%. Split into specialized sub-agents:

Pattern:

1. QA agent writes test → commits test file alone (no implementation) 2. Developer agent implements → runs tests → commits implementation 3. Reflection agent reviews diff: if test assertions changed → FAIL (test-hacking)

Test-hacking detection: reflection-agent checks git diff HEAD~1 HEAD -- '*.test.*' — any assertion changes after implementation commit = REJECT.

When to use: repository-scale TDD, complex features with multiple behaviors, any task where a single agent might rationalize test changes.

TDAID Phase Mapping (Test-Driven AI-Assisted Development, 2025-2026)

TDAID extends classic TDD with explicit Planning and Validation gates:

Key TDAID anti-patterns to detect in Validate phase:

• Deleting test assertions to make tests pass
• Hardcoding expected values
• Mocking away the behavior being tested
• Making implementation superficially compliant without satisfying the specification intent

Research basis: TDAID (awesome-testing.com, 2025), TDAD agent-to-agent variant (arXiv:2603.08806, 2026), TDFlow (arXiv:2510.23761, 2025)

LSP Pre-RED Type Verification

Before writing a failing test, verify the API contract exists to prevent "fails due to wrong API" rather than "fails due to missing behavior":

# Step 1: Find the target function's file + line
pnpm search:code "functionName"

# Step 2: Verify signature with LSP hover
lsp_hover({ filePath: "/abs/path/to/file.ts", line: 42, character: 10 })
# Returns: function signature, parameter types, return type

# Step 3: Write test using VERIFIED signature
# Now RED is guaranteed to fail due to missing behavior, not API mismatch

Rule: If lsp_hover returns empty (CJS file or LSP not active) → fall back to ripgrep rg -n "functionName" --type ts to read the actual signature.

When NOT needed: trivially new functions that don't exist yet (LSP has nothing to return).

Contract Testing (Hook Boundaries — Expanded)

Hook contracts define the stdin/stdout JSON protocol. Test at the boundary:

// Hook contract test pattern
const proc = spawn('node', ['.claude/hooks/routing/routing-guard.cjs'], { shell: false });
const input = JSON.stringify({
  tool_name: 'Edit',
  tool_input: { file_path: '.claude/agents/core/developer.md' },
});
proc.stdin.write(input);
proc.stdin.end();

// Assert: exit code 2 (block) for protected paths
// Assert: stdout JSON contains { allow: false, message: /Gate 4/ }

TaskUpdate metadata contract:

// Validate processedReflectionIds schema
const schema = {
  type: 'object',
  required: ['processedReflectionIds'],
  properties: { processedReflectionIds: { type: 'array', items: { type: 'string' } } },
  additionalProperties: false,
};

Agent-Studio hook contracts to test:

• routing-guard.cjs: blocks Task without task_id (exit 2)
• unified-creator-guard.cjs: blocks Write to .claude/skills/**/SKILL.md (exit 2)
• spawn-token-guard.cjs: warns at 80K tokens (exit 0 + message)

Test Runner Selection (node --test vs Vitest 4)

Agent Studio uses node --test (built-in Node.js test runner) as the default for all .cjs CommonJS files (hooks, lib, scripts). Vitest 4 is the recommended runner for ESM/TypeScript files.

Why `node --test` for `.cjs`: Vitest requires Vite configuration and ESM-compatible modules. Agent Studio hooks use require() and CommonJS — node --test works without transpilation.

Why Vitest 4 for `.ts`/ESM: Boot time drops from ~8s (Jest) to ~1.2s (Vitest). First-class TypeScript + ESM support, Browser Mode (stable v4), and jest-compatible describe/it/expect API (migration = config change only).

Anti-pattern: Do NOT use Jest for new files. Vitest is the 2025-2026 standard for ESM/TypeScript.

# Current Agent Studio pattern (CJS hooks and lib)
node --test tests/lib/routing/routing-table.test.cjs

# Future ESM/TypeScript pattern
pnpm vitest run tests/lib/routing/routing-table.test.ts

AI Output Evaluation Testing (Non-Deterministic Agents)

LLM/agent outputs are non-deterministic — binary pass/fail assertions are insufficient. Use score-based evaluation and tool-call sequence validation instead.

Score-Based Assertion Pattern

// Agent output evaluation — score dimensions 0.0-1.0
function evaluateAgentOutput(output, expectations) {
  const scores = {
    relevance: scoreRelevance(output, expectations.topic), // 0.0-1.0
    safety: scoreSafety(output), // 0.0-1.0
    faithfulness: scoreFaithfulness(output, expectations.facts), // 0.0-1.0
    format: scoreFormat(output, expectations.schema), // 0.0-1.0
  };
  const overall = Object.values(scores).reduce((a, b) => a + b) / Object.keys(scores).length;
  return { scores, overall, pass: overall >= 0.75 };
}

// Test: agent output meets quality threshold
test('researcher agent output is relevant and safe', () => {
  const result = evaluateAgentOutput(agentOutput, { topic: 'TDD patterns', facts: knownFacts });
  expect(result.scores.safety).toBeGreaterThanOrEqual(0.9); // Hard floor for safety
  expect(result.overall).toBeGreaterThanOrEqual(0.75); // 75% overall threshold
});

Tool-Call Sequence Validation

For agent tests, validate the sequence and count of tool calls, not just the final output:

// Spy on tool calls and assert ordering
const toolCallLog = [];
const mockTaskUpdate = jest.fn(args => {
  toolCallLog.push({ tool: 'TaskUpdate', args });
});
const mockBash = jest.fn(args => {
  toolCallLog.push({ tool: 'Bash', args });
});

// Run agent under test with mocked tools
await runAgent({ TaskUpdate: mockTaskUpdate, Bash: mockBash });

// Assert: TaskUpdate(in_progress) called BEFORE TaskUpdate(completed)
const inProgressIdx = toolCallLog.findIndex(
  c => c.tool === 'TaskUpdate' && c.args.status === 'in_progress'
);
const completedIdx = toolCallLog.findIndex(
  c => c.tool === 'TaskUpdate' && c.args.status === 'completed'
);
expect(inProgressIdx).toBeLessThan(completedIdx); // Ordering enforced
expect(inProgressIdx).toBeGreaterThanOrEqual(0); // Must have been called
expect(completedIdx).toBeGreaterThanOrEqual(0); // Must have been called

Rule: Never test the text content of LLM-generated prose. Test structure, schema validity, tool-call sequences, and score thresholds.

Reference: Simon Willison (2025) — "Red/Green TDD for agents: write assertions on tool-call sequences and structured outputs."

MSW v2 HTTP Mocking (API Boundary Testing)

Use MSW (Mock Service Worker) v2 to test skills and agents that make external HTTP calls. MSW intercepts at the network level — no monkey-patching of fetch, no code changes in production.

pnpm add -D msw@2

Setup Pattern (Node.js / Vitest)

import { setupServer } from 'msw/node';
import { http, HttpResponse } from 'msw';

// Define handlers — these describe the expected API contract
const handlers = [
  http.get('https://api.example.com/search', ({ request }) => {
    const url = new URL(request.url);
    return HttpResponse.json({
      results: [{ id: 1, title: `Result for: ${url.searchParams.get('q')}` }],
    });
  }),
];

const server = setupServer(...handlers);
beforeAll(() => server.listen({ onUnhandledRequest: 'error' }));
afterEach(() => server.resetHandlers());
afterAll(() => server.close());

// Test: researcher skill makes HTTP call and processes response
test('researcher skill fetches and parses search results', async () => {
  const results = await researcherSkill.search('TDD patterns 2026');
  expect(results).toHaveLength(1);
  expect(results[0].title).toContain('TDD patterns');
});

Override Per-Test for Error Cases

test('researcher skill handles 503 gracefully', async () => {
  server.use(
    http.get('https://api.example.com/search', () => HttpResponse.json({}, { status: 503 }))
  );
  const results = await researcherSkill.search('TDD patterns');
  expect(results).toEqual([]); // Graceful empty fallback
});

Key benefits over manual mocking:

• Tests exercise real HTTP client code paths (not mocked abstractions)
• onUnhandledRequest: 'error' catches unintentional external calls during tests
• Handlers define request/response contracts — doubles as documentation

Agent-Studio targets for MSW boundary tests:

• researcher skill → WebSearch/WebFetch HTTP calls
• github-ops skill → GitHub API calls
• Any agent using mcp__Exa__web_search_exa or WebFetch

Mutation Testing (Stryker JS)

Mutation testing validates test QUALITY, not just coverage. Run after achieving 100% line coverage:

Stryker + Vitest (2026 Standard — ESM/TypeScript projects)

# Install (once per project) — use vitest-runner for ESM/TypeScript
pnpm add -D @stryker-mutator/core @stryker-mutator/vitest-runner vitest

// stryker.config.mjs — working configuration for Vitest projects
/** @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */
export default {
  testRunner: 'vitest',
  vitest: {
    configFile: 'vitest.config.ts', // optional: path to your vitest config
    related: true, // default: run only tests related to mutated file
  },
  thresholds: { high: 80, low: 60, break: 50 },
  reporters: ['html', 'progress'],
};

# Run mutation tests (use incremental to speed up local loops)
pnpm stryker run --incremental

# Target threshold: >80% mutation score
# Score = (killed mutations / total mutations) × 100

Vitest runner limitations (StrykerJS 7.x):

• Browser Mode not supported — threads mode only
• Always uses perTest coverage analysis (ignores coverageAnalysis config)
• For .cjs files using node --test, use @stryker-mutator/jest-runner as fallback

Stryker + node:test (CommonJS/.cjs projects)

pnpm add -D @stryker-mutator/core @stryker-mutator/jest-runner

Interpret results:

• Killed — test suite caught the mutation ✓
• Survived — test suite MISSED this code path (add assertion)
• No coverage — no test exercises this line at all (add test)

When to run: after completing a TDD cycle for security-critical code (hooks, validators, routing logic). Not required for all code — prioritize by risk.

Agent-Studio priority targets for mutation testing:

• .claude/hooks/routing/routing-guard.cjs
• .claude/hooks/safety/unified-creator-guard.cjs
• .claude/lib/routing/routing-table.cjs

Validation Phase: TDAD Dependency Map (P0)

Before committing, agents MUST identify which test files cover the changed source files. Use compiler-assisted reference discovery or targeted grep:

# Find tests that import the changed file
grep -r "import.*changedFile\|require.*changedFile" tests/

# Or use LSP to find all references
lsp_findReferences({ filePath: "/path/to/changed/file.ts", line: 1, character: 1 })

Build a dependency map and run ONLY those tests first (70% faster regression detection per arXiv:2603.17973):

# 1. Run targeted tests (impacted tests only)
pnpm test tests/hooks/routing-guard.test.cjs

# 2. Verify no regressions in targeted scope
# 3. Only then run full suite
pnpm test

Rationale: Full test suites can exceed 5 minutes on large repos. Targeted testing catches regressions in 15-30 seconds, freeing context for next scenarios in a long TDD loop.

Validation Phase: Spec-Gaming Detection (P0)

In the Validate phase, verify the implementation hasn't gamed test assertions:

Checklist:

• [ ] Tests assert behavior, not implementation details (no testing private variables or class internals)
• [ ] No hardcoded expected values were copied from test to implementation
• [ ] Mutation score ≥80% indicates test quality is sufficient for detecting regressions
• [ ] Review: could the code pass tests while being fundamentally wrong?

Run mutation testing if available:

# Test suite strength validation — mutations should be caught
pnpm stryker run

# If mutation score < 80%, tests are too weak:
# - Add negative tests
# - Add boundary condition tests
# - Verify assertions are on behavior, not mocks

Spec-gaming examples to catch:

• ✗ Implementation hardcodes return 42 to pass test expecting 42 → mutation testing catches this
• ✗ Test mocks behavior instead of asserting it → mutation testing shows 0% mutation killed
• ✗ Test checks log message instead of behavior → flip the assertion, implementation still passes

Agent-Studio targets: After completing security-critical hook or routing changes, run mutation testing before marking task complete.

Invoke the tdd skill and follow it exactly as presented to you

TDD Skill Observations

This directory contains observations and learnings from applying the Test-Driven Development skill in agent-studio.

Purpose

The observations/ directory serves as a feedback loop for continuous improvement of the TDD skill. Agents should record:

1. Patterns that worked well - Successful test-driven patterns and approaches 2. Recurring obstacles - Common blockers when applying TDD with AI 3. Failure cases - Tests that were hard to write or scenarios that exposed design issues 4. Edge cases discovered - New scenarios not covered by original skill documentation 5. Model-specific insights - Observations about how different LLM models handle TDD cycles

Structure

Observations are recorded in JSONL format:

{
  "timestamp": "2026-03-03T10:00:00Z",
  "type": "pattern|obstacle|failure|edge_case|model_insight",
  "description": "Human-readable description of the observation",
  "context": "Where this observation applies (implementation, testing, refactoring, etc.)",
  "severity": "critical|high|medium|low",
  "suggestedImprovement": "Potential skill update or documentation fix"
}

When to Write Observations

• After a TDD cycle completes (Red-Green-Refactor)
• When RED phase is particularly clear or particularly difficult
• When GREEN phase reveals design issues
• When REFACTOR exposes duplication patterns
• When a test cannot be written without violating TDD principles
• When a model struggles with the canonical sequence

Example Observations

Pattern (what worked)

{
  "timestamp": "2026-03-03T10:30:00Z",
  "type": "pattern",
  "description": "Writing test names as behavior statements (test_user_can_login_with_valid_credentials) made test purpose crystal clear during implementation",
  "context": "RED phase",
  "severity": "medium",
  "suggestedImprovement": "Emphasize behavior-driven test naming in TDD skill documentation"
}

Obstacle (what blocked progress)

{
  "timestamp": "2026-03-03T11:00:00Z",
  "type": "obstacle",
  "description": "Testing async database calls required complex mocking setup, violating 'real code' principle",
  "context": "RED and GREEN phases",
  "severity": "high",
  "suggestedImprovement": "Add guidance for testing async I/O boundaries without excessive mocking"
}

Integration with Skill Evolution

Observations are automatically analyzed quarterly to:

1. Identify patterns requiring documentation updates 2. Discover new guidance needed in SKILL.md 3. Assess model-specific coaching requirements 4. Refine the canonical TDD sequence

References

• TDD Skill Documentation
• Testing Anti-Patterns - Common patterns to avoid
• Agent Studio TDD Rules

TDD Research Requirements (Updated 2026-02-15)

Intent

Update the tdd skill to reflect current canonical TDD and AI-assisted TDD evidence while avoiding workflow overengineering.

Sources (Exa-first, canonical + arXiv)

1. Martin Fowler, "Test Driven Development" (2023-12-11): https://martinfowler.com/bliki/TestDrivenDevelopment.html 2. Kent Beck, "Canon TDD" (2023-12-11): https://tidyfirst.substack.com/p/canon-tdd 3. Rafique & Misic, IEEE TSE meta-analysis DOI:10.1109/TSE.2012.28 4. LLM4TDD (arXiv:2312.04687): https://arxiv.org/abs/2312.04687 5. Test-Driven Development for Code Generation (arXiv:2402.13521): https://arxiv.org/abs/2402.13521 6. Tests as Prompt (arXiv:2505.09027): https://arxiv.org/abs/2505.09027 7. SWE-Flow (arXiv:2506.09003): https://arxiv.org/abs/2506.09003 8. TDFlow (arXiv:2510.23761): https://arxiv.org/abs/2510.23761 9. Class-level TDD generation (arXiv:2602.03557): https://arxiv.org/abs/2602.03557

Actionable Constraints

1. Canon sequence is mandatory: scenario list -> one runnable test -> RED -> GREEN -> optional refactor -> repeat. 2. AI workflows must include anti-test-hacking checks and bounded repair loops. 3. Tests should stay short, deterministic, and requirement-facing (tests as executable specification). 4. Repository-scale work should decompose by failing test clusters; class-level work should sequence by method dependencies. 5. Preserve simplicity: no new orchestration subsystem, no mandatory multi-agent decomposition for all tasks.

Mapping to Skill Artifacts

• SKILL.md: Canon loop, AI guardrails, repo/class guidance.
• hooks/pre-execute.cjs: input validation for TDD contract shape.
• hooks/post-execute.cjs: completion warnings when RED/GREEN evidence is missing.
• schemas/input.schema.json: defines invocation contract.
• schemas/output.schema.json: defines evidence-based output contract.
• rules/tdd.md: concise enforceable operating rules.
• templates/implementation-template.md: scenario backlog + evidence-first template.
• references/tdd-memory-profile.md: bounded runtime memory acceleration guidance.
• .claude/workflows/tdd-skill-workflow.md: concise TDD execution sequence.

Non-Goals (Simplicity Guard)

• No autonomous test generation engine inside this skill.
• No mandatory mutation testing gate for all repos.
• No forced repository-wide refactor phase.
• No new runtime hook registration in .claude/settings.json for this update.

TDD Memory Profile

Purpose

Use a small runtime profile to reduce repeated setup and triage work during TDD loops.

Path: .claude/context/runtime/tdd-memory-profile.json

Allowed Data

1. Preferred local commands:

• testCommand
• lintCommand
• formatCommand

2. Recurrent failure signatures with concise fix summaries. 3. Recurrent anti-pattern reminders. 4. Reusable scenario templates.

Limits

• Max profile size: 16 KB.
• Max entries per bucket: 20.
• Max string value length: 180 chars.
• Keep data recent and actionable; drop stale entries first.

Non-Goals

• Do not store raw test logs or full stack traces.
• Do not store sensitive content.
• Do not use memory to skip RED verification or alter Canon TDD sequence.

tdd Rules

Purpose

Canon TDD for humans and AI agents. Use for production code changes by writing tests first, proving RED, implementing minimal GREEN, and refactoring safely.

Best Practices

• Keep a visible test scenario backlog and execute one scenario at a time
• Prove RED before code changes and keep evidence in command output
• Implement smallest GREEN patch that satisfies current failing test only
• Use bounded repair loops and anti-test-hacking checks before completion

Integration Points

See SKILL.md for complete documentation.

TDD Implementation Template

Change Goal

• Problem:
• Acceptance criteria:
• Out of scope:

Scenario Backlog (Required)

Cycle Evidence (One scenario per loop)

Scenario ID

RED

• Test added/updated:
• Command:
• Failure summary:

GREEN

• Production files changed:
• Command:
• Pass summary:

REFACTOR (Optional)

• Refactor performed:
• Verification command:
• Result:

Anti-Test-Hacking Check

• [ ] Assertions still express the original requirement
• [ ] No test-only branch or feature flag added
• [ ] Negative test included for bug-fix scenarios

Completion Checks

• [ ] All scenario backlog items resolved or deferred with reason
• [ ] All touched tests pass
• [ ] Lint command run
• [ ] Format check command run

Testing Anti-Patterns

Load this reference when: writing or changing tests, adding mocks, or tempted to add test-only methods to production code.

Overview

Tests must verify real behavior, not mock behavior. Mocks are a means to isolate, not the thing being tested.

Core principle: Test what the code does, not what the mocks do.

Following strict TDD prevents these anti-patterns.

The Iron Laws

1. NEVER test mock behavior
2. NEVER add test-only methods to production classes
3. NEVER mock without understanding dependencies

Anti-Pattern 1: Testing Mock Behavior

The violation:

// BAD: Testing that the mock exists
test('renders sidebar', () => {
  render(<Page />);
  expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument();
});

Why this is wrong:

• You're verifying the mock works, not that the component works
• Test passes when mock is present, fails when it's not
• Tells you nothing about real behavior

Your human partner's correction: "Are we testing the behavior of a mock?"

The fix:

// GOOD: Test real component or don't mock it
test('renders sidebar', () => {
  render(<Page />);  // Don't mock sidebar
  expect(screen.getByRole('navigation')).toBeInTheDocument();
});

// OR if sidebar must be mocked for isolation:
// Don't assert on the mock - test Page's behavior with sidebar present

Gate Function

BEFORE asserting on any mock element:
  Ask: "Am I testing real component behavior or just mock existence?"

  IF testing mock existence:
    STOP - Delete the assertion or unmock the component

  Test real behavior instead

Anti-Pattern 2: Test-Only Methods in Production

The violation:

// BAD: destroy() only used in tests
class Session {
  async destroy() {
    // Looks like production API!
    await this._workspaceManager?.destroyWorkspace(this.id);
    // ... cleanup
  }
}

// In tests
afterEach(() => session.destroy());

Why this is wrong:

• Production class polluted with test-only code
• Dangerous if accidentally called in production
• Violates YAGNI and separation of concerns
• Confuses object lifecycle with entity lifecycle

The fix:

// GOOD: Test utilities handle test cleanup
// Session has no destroy() - it's stateless in production

// In test-utils/
export async function cleanupSession(session: Session) {
  const workspace = session.getWorkspaceInfo();
  if (workspace) {
    await workspaceManager.destroyWorkspace(workspace.id);
  }
}

// In tests
afterEach(() => cleanupSession(session));

Gate Function

BEFORE adding any method to production class:
  Ask: "Is this only used by tests?"

  IF yes:
    STOP - Don't add it
    Put it in test utilities instead

  Ask: "Does this class own this resource's lifecycle?"

  IF no:
    STOP - Wrong class for this method

Anti-Pattern 3: Mocking Without Understanding

The violation:

// BAD: Mock breaks test logic
test('detects duplicate server', () => {
  // Mock prevents config write that test depends on!
  vi.mock('ToolCatalog', () => ({
    discoverAndCacheTools: vi.fn().mockResolvedValue(undefined),
  }));

  await addServer(config);
  await addServer(config); // Should throw - but won't!
});

Why this is wrong:

• Mocked method had side effect test depended on (writing config)
• Over-mocking to "be safe" breaks actual behavior
• Test passes for wrong reason or fails mysteriously

The fix:

// GOOD: Mock at correct level
test('detects duplicate server', () => {
  // Mock the slow part, preserve behavior test needs
  vi.mock('MCPServerManager'); // Just mock slow server startup

  await addServer(config); // Config written
  await addServer(config); // Duplicate detected
});

Gate Function

BEFORE mocking any method:
  STOP - Don't mock yet

  1. Ask: "What side effects does the real method have?"
  2. Ask: "Does this test depend on any of those side effects?"
  3. Ask: "Do I fully understand what this test needs?"

  IF depends on side effects:
    Mock at lower level (the actual slow/external operation)
    OR use test doubles that preserve necessary behavior
    NOT the high-level method the test depends on

  IF unsure what test depends on:
    Run test with real implementation FIRST
    Observe what actually needs to happen
    THEN add minimal mocking at the right level

  Red flags:
    - "I'll mock this to be safe"
    - "This might be slow, better mock it"
    - Mocking without understanding the dependency chain

Anti-Pattern 4: Incomplete Mocks

The violation:

// BAD: Partial mock - only fields you think you need
const mockResponse = {
  status: 'success',
  data: { userId: '123', name: 'Alice' },
  // Missing: metadata that downstream code uses
};

// Later: breaks when code accesses response.metadata.requestId

Why this is wrong:

• Partial mocks hide structural assumptions - You only mocked fields you know about
• Downstream code may depend on fields you didn't include - Silent failures
• Tests pass but integration fails - Mock incomplete, real API complete
• False confidence - Test proves nothing about real behavior

The Iron Rule: Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses.

The fix:

// GOOD: Mirror real API completeness
const mockResponse = {
  status: 'success',
  data: { userId: '123', name: 'Alice' },
  metadata: { requestId: 'req-789', timestamp: 1234567890 },
  // All fields real API returns
};

Gate Function

BEFORE creating mock responses:
  Check: "What fields does the real API response contain?"

  Actions:
    1. Examine actual API response from docs/examples
    2. Include ALL fields system might consume downstream
    3. Verify mock matches real response schema completely

  Critical:
    If you're creating a mock, you must understand the ENTIRE structure
    Partial mocks fail silently when code depends on omitted fields

  If uncertain: Include all documented fields

Anti-Pattern 5: Integration Tests as Afterthought

The violation:

Implementation complete
No tests written
"Ready for testing"

Why this is wrong:

• Testing is part of implementation, not optional follow-up
• TDD would have caught this
• Can't claim complete without tests

The fix:

TDD cycle:
1. Write failing test
2. Implement to pass
3. Refactor
4. THEN claim complete

When Mocks Become Too Complex

Warning signs:

• Mock setup longer than test logic
• Mocking everything to make test pass
• Mocks missing methods real components have
• Test breaks when mock changes

Your human partner's question: "Do we need to be using a mock here?"

Consider: Integration tests with real components often simpler than complex mocks

TDD Prevents These Anti-Patterns

Why TDD helps:

1. Write test first - Forces you to think about what you're actually testing 2. Watch it fail - Confirms test tests real behavior, not mocks 3. Minimal implementation - No test-only methods creep in 4. Real dependencies - You see what the test actually needs before mocking

If you're testing mock behavior, you violated TDD - you added mocks without watching test fail against real code first.

Quick Reference

Red Flags

• Assertion checks for *-mock test IDs
• Methods only called in test files
• Mock setup is >50% of test
• Test fails when you remove mock
• Can't explain why mock is needed
• Mocking "just to be safe"

The Bottom Line

Mocks are tools to isolate, not things to test.

If TDD reveals you're testing mock behavior, you've gone wrong.

Fix: Test real behavior or question why you're mocking at all.