Test Updates

Table of Contents

• Overview
• Core Philosophy
• What It Is
• Quick Start
• Quick Checklist for First Time Use
• detailed Test Update
• Targeted Test Updates
• TDD for New Features
• Using the Scripts Directly
• When to Use It
• Workflow Integration
• Phase 1: Discovery
• Phase 2: Strategy
• Phase 3: Implementation
• Phase 4: Validation
• Quality Assurance
• Examples
• BDD-Style Test Generation
• Test Enhancement
• Integration with Existing Skills
• Success Metrics
• Troubleshooting FAQ
• Common Issues
• Performance Tips
• Getting Help

Test Updates and Maintenance

Overview

detailed test management system that applies TDD/BDD principles to maintain, generate, and enhance tests across codebases. This skill practices what it preaches - it uses TDD principles for its own development and serves as a living example of best practices.

Core Philosophy

• RED-GREEN-REFACTOR: Strict adherence to TDD cycle
• Behavior-First: BDD patterns that describe what code should do
• Invariant-Encoding: Tests guard design decisions, not just behavior
• Meta Dogfooding: The skill's own tests demonstrate the principles it teaches
• Quality Gates: detailed validation before considering tests complete

What It Is

A modular test management system that:

• Discovers what needs testing or updating
• Generates tests following TDD principles
• Enhances existing tests with BDD patterns
• Validate test quality through multiple lenses

Quick Start

Quick Checklist for First Time Use

• [ ] validate pytest is installed (pip install pytest)
• [ ] Have your source code in src/ or similar directory
• [ ] Create a tests/ directory if it doesn't exist
• [ ] Run Skill(sanctum:git-workspace-review) first to understand changes
• [ ] Start with Skill(test-updates) --target <specific-module> for focused updates

detailed Test Update

# Run full test update workflow
Skill(test-updates)

Verification: Run pytest -v to verify tests pass.

Targeted Test Updates

# Update tests for specific paths
Skill(test-updates) --target src/sanctum/agents
Skill(test-updates) --target tests/test_commit_messages.py

Verification: Run pytest -v to verify tests pass.

TDD for New Features

# Apply TDD to new code
Skill(test-updates) --tdd-only --target new_feature.py

Verification: Run pytest -v to verify tests pass.

Using the Scripts Directly

Human-Readable Output:

# Analyze test coverage gaps
python plugins/sanctum/scripts/test_analyzer.py --scan src/

# Generate test scaffolding
python plugins/sanctum/scripts/test_generator.py \
    --source src/my_module.py --style pytest_bdd

# Check test quality
python plugins/sanctum/scripts/quality_checker.py \
    --validate tests/test_my_module.py

Verification: Run pytest -v to verify tests pass.

Programmatic Output (for Claude Code):

# Get JSON output for programmatic parsing - test_analyzer
python plugins/sanctum/scripts/test_analyzer.py \
    --scan src/ --output-json

# Returns:
# {
#   "success": true,
#   "data": {
#     "source_files": ["src/module.py", ...],
#     "test_files": ["tests/test_module.py", ...],
#     "uncovered_files": ["module_without_tests", ...],
#     "coverage_gaps": [{"file": "...", "reason": "..."}]
#   }
# }

# Get JSON output - test_generator
python plugins/sanctum/scripts/test_generator.py \
    --source src/my_module.py --output-json

# Returns:
# {
#   "success": true,
#   "data": {
#     "test_file": "path/to/test_my_module.py",
#     "source_file": "src/my_module.py",
#     "style": "pytest_bdd",
#     "fixtures_included": true,
#     "edge_cases_included": true,
#     "error_cases_included": true
#   }
# }

# Get JSON output - quality_checker
python plugins/sanctum/scripts/quality_checker.py \
    --validate tests/test_my_module.py --output-json

# Returns:
# {
#   "success": true,
#   "data": {
#     "static_analysis": {...},
#     "dynamic_validation": {...},
#     "metrics": {...},
#     "quality_score": 85,
#     "quality_level": "QualityLevel.GOOD",
#     "recommendations": [...]
#   }
# }

Verification: Run pytest -v to verify tests pass.

When To Use It

Use this skill when you need to:

• Update tests after code changes
• Generate tests for new features
• Improve existing test quality
• validate detailed test coverage

Perfect for:

• Pre-commit test validation
• CI/CD pipeline integration
• Refactoring with test safety
• Onboarding new developers

When NOT To Use

• Auditing

test suites - use pensive:test-review

• Writing production code
• focus on implementation first
• Auditing

test suites - use pensive:test-review

• Writing production code
• focus on implementation first

Workflow Integration

Phase 1: Discovery

1. Scan codebase for test gaps 2. Analyze recent changes 3. Identify broken or outdated tests

See modules/test-discovery.md for detection patterns.

Phase 2: Strategy

1. Choose appropriate BDD style (see modules/bdd-patterns.md) 2. Plan test structure 3. Define quality criteria 4. Identify design invariants to encode as tests

Phase 2.5: Invariant-Encoding Tests

Before writing behavioral tests, identify the design invariants that the code relies on and write tests that would break if those invariants were violated.

What to encode:

• Module boundary constraints (A never imports from B)
• Data flow direction (events flow publisher-to-subscriber,

never the reverse)

• API contract shapes (public interfaces don't change

without versioning)

• Data structure choices (if a map was chosen over a list,

test the properties that justify that choice)

• Error handling strategies (fail-fast boundaries, recovery

zones)

Example:

def test_plugins_never_import_from_other_plugins():
    """Encode the invariant: plugins are independent modules.

    If this test breaks, someone is coupling plugins
    directly. Present the 3 options to a human:
    1. Preserve: revert the import, keep plugins independent
    2. Layer: add a shared interface in leyline instead
    3. Revise: merge the plugins (requires ADR)
    """
    for plugin_dir in plugin_dirs:
        imports = extract_imports(plugin_dir)
        for imp in imports:
            assert not imp.startswith("plugins."), (
                f"{plugin_dir} imports {imp} — "
                f"violates plugin independence invariant"
            )

Why this matters: Tests that encode invariants are load-bearing. When an agent later encounters a feature that clashes with the invariant, the test failure forces a conscious decision rather than a silent drift. Without these tests, bad invariant decisions compound until the codebase is unsalvageable.

When updating existing tests:

If an invariant-encoding test needs to change, do NOT silently update the assertion. Flag it for human review with the three options: preserve the invariant, layer on top, or revise the invariant. This is a judgment call that requires human wisdom: models default to the "average" of training data and get these wrong far too often.

Phase 3: Implementation

1. Write failing tests (RED) - see modules/tdd-workflow.md 2. Implement minimal passing code (GREEN) 3. Refactor for clarity (REFACTOR)

See modules/test-generation.md for generation templates.

Phase 4: Validation

1. Static analysis and linting 2. Dynamic test execution 3. Coverage and quality metrics

See modules/quality-validation.md for validation criteria.

Quality Assurance

The skill applies multiple quality checks:

• Static: Linting, type checking, pattern validation
• Dynamic: Test execution in sandboxed environments
• Metrics: Coverage, mutation score, complexity analysis
• Invariant: Verify design-decision tests are not weakened
• Review: Structured checklists for peer validation

Examples

BDD-Style Test Generation

See modules/bdd-patterns.md for additional patterns.

class TestGitWorkflow:
    """BDD-style tests for Git workflow operations."""

    def test_commit_workflow_with_staged_changes(self):
        """
        GIVEN a Git repository with staged changes
        WHEN the user runs the commit workflow
        THEN it should create a commit with proper message format
        AND all tests should pass
        """
        # Test implementation following TDD principles
        pass

Verification: Run pytest -v to verify tests pass.

Test Enhancement

• Add edge cases and error scenarios
• Include performance benchmarks
• Add mutation testing for robustness

See modules/test-enhancement.md for enhancement strategies.

Integration with Existing Skills

1. git-workspace-review: Get context of changes 2. file-analysis: Understand code structure 3. test-driven-development: Apply strict TDD discipline 4. skills-eval: Validate quality and compliance

Success Metrics

• Test coverage > 85%
• All tests follow BDD patterns
• Zero broken tests in CI
• Mutation score > 80%

Troubleshooting FAQ

Common Issues

Q: Tests are failing after generation A: This is expected! The skill follows TDD principles - generated tests are designed to fail first. Follow the RED-GREEN-REFACTOR cycle: 1. Run the test and confirm it fails for the right reason 2. Implement minimal code to make it pass 3. Refactor for clarity

Q: Quality score is low despite having tests A: Check for these common issues:

• Missing BDD patterns (Given/When/Then)
• Vague assertions like assert result is not None
• Tests without documentation
• Long, complex tests (>50 lines)

Q: Generated tests don't match my code structure A: The scripts analyze AST patterns and may need guidance:

• Use --style flag to match your preferred BDD style
• Check that source files have proper function/class definitions
• Review the generated scaffolding and customize as needed

Q: Mutation testing takes too long A: Mutation testing is resource-intensive:

• Use --quick-mutation flag for subset testing
• Focus on critical modules first
• Run overnight for detailed analysis

Q: Can't find tests for my file A: The analyzer uses naming conventions:

• Source: my_module.py → Test: test_my_module.py
• Check that test files follow pytest naming patterns
• validate test directory structure is standard

Performance Tips

• Large codebases: Use --target to focus on specific directories
• CI integration: Run validation in parallel with other checks
• Memory usage: Process files in batches for very large projects

Getting Help

1. Check script outputs for detailed error messages 2. Use --verbose flag for more information 3. Review the validation report for specific recommendations 4. Start with small modules to understand patterns before scaling

BDD Patterns Module

Overview

Provides multiple Behavior-Driven Development styles and patterns for creating expressive, behavior-focused tests.

Available Styles

Choosing the Right Style

Decision Guide

Mixing Styles

• Use Gherkin for critical user journeys
• Use BDD-pytest for unit and API tests
• Use Docstring BDD for simple utilities
• Maintain consistency within modules

Best Practices

Naming Conventions

• Tests: test_[behavior]_[when]_[expected]
• Given/When/Then: Clear separation of concerns
• Scenarios: Describe business value, not technical details

Test Organization

Group related BDD scenarios in test classes with clear setup and teardown.

---

Gherkin Style

Feature files with Given/When/Then scenarios for complex user workflows and cross-team collaboration.

Feature File Structure

Feature: Git Workflow Management
  As a developer
  I want to automate git workflows
  So that I can maintain clean commit history

  Scenario: Commit with staged changes
    Given a git repository with staged changes
    When I run the commit workflow
    Then a commit should be created with proper message
    And all tests should pass

  Scenario Outline: Multiple file types
    Given a git repository with staged <file_type> files
    When I run the commit workflow
    Then the commit should reference <file_type>
    And the commit type should be <commit_type>

    Examples:
      | file_type | commit_type |
      | source    | feat       |
      | test      | test       |
      | docs      | docs       |

Step Definitions

@given('a git repository with staged changes')
def step_given_git_repo_with_changes(context):
    context.repo = create_test_repo()
    context.repo.stage_changes(['file1.py', 'file2.py'])

@when('I run the commit workflow')
def step_when_run_commit_workflow(context):
    context.result = run_commit_workflow(context.repo)

@then('a commit should be created with proper message')
def step_then_commit_created(context):
    assert context.repo.has_commit()
    assert context.repo.last_commit_message().startswith('feat:')

When to Use

• Complex user workflows
• Acceptance criteria documentation
• Cross-team collaboration
• Living documentation requirements

---

Pytest Style

BDD-style pytest tests with descriptive names and docstrings for unit and API testing.

Structure Example

class TestGitWorkflow:
    """BDD-style tests for Git workflow operations."""

    @pytest.mark.bdd
    def test_commit_workflow_with_staged_changes(self):
        """
        GIVEN a Git repository with staged changes
        WHEN the user runs the commit workflow
        THEN it should create a commit with proper message format
        AND all tests should pass
        """
        # Given
        repo = create_git_repo()
        repo.stage_changes(['feature.py'])

        # When
        result = run_commit_workflow(repo)

        # Then
        assert result.success is True
        assert repo.has_commit()
        assert repo.last_commit_message().startswith('feat:')

    @pytest.mark.bdd
    def test_commit_workflow_rejects_empty_changes(self):
        """
        GIVEN a Git repository with no staged changes
        WHEN the user runs the commit workflow
        THEN it should reject with appropriate error message
        """
        # Given
        repo = create_git_repo()  # No changes staged

        # When
        result = run_commit_workflow(repo)

        # Then
        assert result.success is False
        assert "no staged changes" in result.error.lower()

Best Practices

• Descriptive names: Describe behavior, not implementation
• Clear sections: Use Given/When/Then in docstrings
• Single responsibility: One behavior per test
• Meaningful assertions: Test specific outcomes

When to Use

• Unit tests with behavior focus
• API testing
• Service layer testing
• Developer-facing documentation

---

Docstring Style

Simple BDD pattern using docstrings for quick behavior documentation and simple unit tests.

Structure Example

def test_git_status_parsing():
    """Test parsing git status output.

    GIVEN git status output with modified and untracked files
    WHEN parsing the status
    THEN it should return structured file information
    AND correctly identify file states
    """
    status_output = """
    M modified_file.py
    A  added_file.py
    ?? untracked_file.py
    """

    result = parse_git_status(status_output)

    assert 'modified_file.py' in result.modified
    assert 'added_file.py' in result.added
    assert 'untracked_file.py' in result.untracked

Best Practices

• Clear docstrings: Include Given/When/Then
• Simple structure: Ideal for utilities and helpers
• Quick documentation: Minimal overhead for behavior specs
• Focused tests: One clear behavior per test

When to Use

• Simple unit tests
• Internal module testing
• Quick behavior documentation
• Utility function testing

Content Test Discovery

Detects when modified markdown files are "execution markdown" requiring content assertions, and identifies test gaps.

Execution Markdown Detection

Files matching ALL of these criteria are execution markdown:

1. File extension is .md 2. Path contains skills/, agents/, modules/, or commands/ 3. File is NOT named README.md, CHANGELOG.md, or located under docs/ directories

def is_execution_markdown(file_path: str) -> bool:
    """Markdown that Claude interprets as behavioral instructions."""
    path = Path(file_path)
    exec_dirs = {"skills", "agents", "modules", "commands"}
    skip_names = {"README.md", "CHANGELOG.md"}
    return (
        path.suffix == ".md"
        and any(d in path.parts for d in exec_dirs)
        and path.name not in skip_names
        and "docs" not in path.parts
    )

Priority Reclassification

Override the default test-discovery priority scoring for execution markdown:

Test Gap Detection

When execution markdown is modified, check for a corresponding content test class.

Naming Convention

Detection Heuristic

Look for existing content test classes by checking:

1. Test file exists at the expected path 2. File contains a class ending in Content (e.g., TestClearContextSkillContent) 3. File contains fixtures that read .md files (e.g., skill_content, module_content)

If no content test class exists, flag as a content test gap.

When to Generate vs. Skip

Not every markdown change needs new content tests.

Generate Content Tests When

• A new skill or module is created (no existing tests)
• Code examples (JSON, YAML, Python) are added or modified (L2 needed)
• Version references are added or changed (L3 cross-reference needed)
• Decision frameworks or behavioral guidance is modified (L3 contract needed)
• Forbidden behavior patterns are specified (L3 anti-pattern detection needed)

Skip Content Tests When

• Typo or grammar fix only (no behavioral change)
• Whitespace or formatting changes
• Changes to prose that don't affect decision logic
• Changes already covered by scribe:slop-detector (style, not behavior)

Integration

This module is loaded during Phase 1 (Discovery) of the test-updates workflow. It extends git-based change detection to recognize execution markdown as high-priority test targets.

Reference: leyline:testing-quality-standards/modules/content-assertion-levels.md for the L1/L2/L3 taxonomy that determines which level of tests to generate.

Quality Validation Module

Overview

detailed test quality assurance through static analysis, dynamic validation, metrics tracking, and structured peer review.

Validation Categories

1. Static Analysis

Validate test code without execution (details below).

2. Dynamic Validation

Execute tests to verify they actually work (details below).

3. Metrics Validation

Track quantitative quality measures (details below).

4. Peer Review Checklist

Structured validation for human review.

Quality Gates Checklist

QUALITY_GATES = {
    "structure": [
        "Test follows BDD pattern with Given/When/Then",
        "Test has descriptive name explaining behavior",
        "Test is independent and isolated",
        "Test uses appropriate fixtures or setup",
    ],
    "assertions": [
        "Assertions are specific and meaningful",
        "Error messages are descriptive",
        "Both positive and negative cases tested",
        "Edge cases are covered",
    ],
    "maintenance": [
        "Test is readable and understandable",
        "Test data is clearly defined",
        "External dependencies are mocked",
        "Test documentation is adequate",
    ],
    "performance": [
        "Test runs quickly (< 1 second)",
        "No unnecessary I/O operations",
        "Memory usage is reasonable",
        "Tests are parallelizable",
    ],
}

Validation Workflow

def run_validation_pipeline(test_path, source_path=None):
    """Run complete validation pipeline."""

    report = ValidationReport()

    # Phase 1: Static Analysis
    static_issues = validate_static_quality(test_path)
    report.add_section("Static Analysis", static_issues)

    # Phase 2: Dynamic Validation
    execution_results = validate_test_execution(test_path)
    report.add_section("Dynamic Validation", execution_results)

    # Phase 3: Mutation Testing (if source provided)
    if source_path:
        mutation_score = run_mutation_tests(test_path, source_path)
        report.add_section("Mutation Testing", {"score": mutation_score})

    # Phase 4: Metrics Validation
    coverage_violations = validate_coverage_metrics(execution_results["coverage"])
    report.add_section("Coverage Metrics", coverage_violations)

    # Phase 5: Complexity Analysis
    complexity = calculate_test_complexity(test_path)
    report.add_section("Complexity Metrics", complexity)

    return report

Quality Standards

Minimum Requirements

• Coverage: 85% line, 80% branch, 90% function
• Mutation Score: 80% or higher
• Test Speed: < 1 second per test
• Independence: No test dependencies
• BDD Compliance: All tests follow BDD patterns

Excellence Criteria

• Coverage: 95% line, 90% branch, 100% function
• Mutation Score: 90% or higher
• Test Speed: < 0.5 seconds per test
• Documentation: detailed behavior description
• Maintainability: Clear, readable, well-structured

Failure Modes

Tests failing validation should: 1. Generate detailed issue reports 2. Suggest specific improvements 3. Provide examples of fixes 4. Block merging until resolved

---

Static Analysis

Validate test code without execution using pattern matching and AST analysis.

Code Quality Checks

def validate_static_quality(test_file):
    """Perform static quality validation."""

    issues = []

    # Check test naming
    if not test_has_descriptive_name(test_file):
        issues.append("Test name should describe behavior")

    # Check BDD structure
    if not has_bdd_structure(test_file):
        issues.append("Test should follow BDD pattern")

    # Check assertion quality
    if has_vague_assertions(test_file):
        issues.append("Use specific, meaningful assertions")

    # Check test independence
    if tests_have_dependencies(test_file):
        issues.append("Tests should be independent")

    return issues

Pattern Validation

BDD_PATTERNS = {
    "given_pattern": r"GIVEN\s+.+",
    "when_pattern": r"WHEN\s+.+",
    "then_pattern": r"THEN\s+.+",
    "and_pattern": r"AND\s+.+",
}

def validate_bdd_patterns(test_content):
    """Validate BDD pattern usage."""
    missing_patterns = []

    for pattern_name, pattern_regex in BDD_PATTERNS.items():
        if not re.search(pattern_regex, test_content, re.IGNORECASE):
            missing_patterns.append(pattern_name)

    return missing_patterns

Validation Categories

• Naming: Descriptive, behavior-focused test names
• Structure: Proper BDD patterns and organization
• Assertions: Specific, meaningful checks
• Independence: No test dependencies
• Documentation: Clear docstrings and comments

---

Dynamic Validation

Executes tests to verify they actually work and measure their quality.

Test Execution Validation

def validate_test_execution(test_path):
    """Validate test executes correctly."""

    results = {
        "passes": False,
        "failures": [],
        "errors": [],
        "warnings": [],
        "coverage": 0,
    }

    # Run tests in isolated environment
    test_result = pytest.main([
        test_path,
        "-v",
        "--tb=short",
        "--cov=src",
        "--cov-report=json",
    ])

    # Analyze results
    if test_result == 0:
        results["passes"] = True
    else:
        # Parse failures and errors
        results["failures"] = parse_test_failures()
        results["errors"] = parse_test_errors()

    # Load coverage data
    results["coverage"] = load_coverage_data()

    return results

Mutation Testing

def run_mutation_tests(test_path, source_path):
    """Run mutation testing to verify test quality."""

    mutations = generate_mutations(source_path)
    killed_mutants = 0
    total_mutants = len(mutations)

    for mutation in mutations:
        # Apply mutation
        apply_mutation(source_path, mutation)

        # Run tests
        if pytest.main([test_path, "-q"]) != 0:
            killed_mutants += 1  # Test caught the mutation

        # Restore original code
        restore_original(source_path)

    mutation_score = killed_mutants / total_mutants
    return mutation_score

Performance Testing

• Execution time: Tests should run quickly (< 1 second)
• Memory usage: No memory leaks or excessive consumption
• Parallel execution: Tests should run independently
• Resource cleanup: Proper teardown after each test

---

Quality Metrics

Tracks quantitative quality measures for test suites.

Coverage Metrics

def validate_coverage_metrics(coverage_data):
    """Validate test coverage meets standards."""

    metrics = {
        "line_coverage": coverage_data["lines_covered"] / coverage_data["lines_valid"],
        "branch_coverage": coverage_data["branches_covered"] / coverage_data["branches_valid"],
        "function_coverage": coverage_data["functions_covered"] / coverage_data["functions_valid"],
    }

    standards = {
        "line_coverage": 0.85,  # 85% minimum
        "branch_coverage": 0.80,  # 80% minimum
        "function_coverage": 0.90,  # 90% minimum
    }

    violations = []
    for metric, value in metrics.items():
        if value < standards[metric]:
            violations.append(f"{metric}: {value:.1%} < {standards[metric]:.1%}")

    return violations

Test Complexity Metrics

def calculate_test_complexity(test_file):
    """Calculate cyclomatic complexity of tests."""

    complexity_metrics = {
        "average_assertions_per_test": 0,
        "test_length_violations": 0,
        "setup_complexity": 0,
        "mock_count": 0,
    }

    # Analyze each test
    for test in extract_tests(test_file):
        assertions = count_assertions(test)
        if assertions > 5:
            complexity_metrics["test_length_violations"] += 1

        complexity_metrics["average_assertions_per_test"] += assertions
        complexity_metrics["mock_count"] += count_mocks(test)

    complexity_metrics["average_assertions_per_test"] /= len(extract_tests(test_file))

    return complexity_metrics

Quality Score Calculation

Combine multiple metrics into an overall score:

• Static analysis (20%)
• Dynamic validation (30%)
• Coverage metrics (20%)
• Mutation testing (20%)
• Complexity (10%)

TDD Workflow Module

Table of Contents

• Overview
• The TDD Cycle
• RED Phase: Write Failing Test
• GREEN Phase: Minimal Implementation
• REFACTOR Phase: Clean Up
• TDD Discipline Rules
• Error Handling in TDD
• Advanced TDD Patterns

Overview

Implements strict Test-Driven Development workflow with RED-GREEN-REFACTOR cycle. This module validates all test creation follows proper TDD discipline.

The TDD Cycle

RED Phase: Write Failing Test

Principles:

• Write ONE test at a time
• Test must FAIL for the right reason
• No production code exists yet
• Test describes desired behavior

Implementation Pattern:

def test_new_feature_behavior():
    """
    GIVEN a specific context
    WHEN an action is performed
    THEN expected outcome occurs
    """
    # Arrange - Set up test context
    context = create_test_context()

    # Act - Execute the behavior
    result = perform_action(context)

    # Assert - Verify the outcome
    assert result == expected_value

# Run and verify it fails: pytest -xvs test_file.py::test_new_feature_behavior

Verification Steps

1. Run the test: Must fail 2. Check failure reason: Should be "feature not implemented" 3. Confirm test quality: Clear, focused, one behavior

GREEN Phase: Minimal Implementation

Principles:

• Write simplest code to pass
• No extra features
• Don't fix other tests
• Keep it ugly if it works

Implementation Pattern:

# Minimal implementation - just enough to pass
def perform_action(context):
    if context.should_succeed:
        return expected_value
    raise NotImplementedError("Feature not yet implemented")

Verification Steps

1. Run the test: Must pass 2. Check other tests: All still passing 3. No warnings/errors: Clean execution

REFACTOR Phase: Clean Up

Principles:

• Tests must stay green
• Remove duplication
• Improve names and structure
• Add necessary abstractions

Refactoring Checklist:

• [ ] Extract magic numbers to constants
• [ ] Improve variable names
• [ ] Remove code duplication
• [ ] Add helpful comments
• [ ] validate single responsibility

TDD Discipline Rules

Iron Rules

1. NO production code without a failing test first 2. Watch it fail - Don't skip this step 3. Write minimal code - No extra features 4. Refactor only when green - Clean up with safety net

Common Violations to Avoid

• Writing code before tests
• "I'll test it after" mentality
• Keeping implementation as "reference"
• Skipping the failure verification
• Adding extra features in GREEN phase

Error Handling in TDD

Test Errors vs Failures

• Error: Syntax, imports, setup issues - Fix immediately
• Failure: Assertion fails - Good! This is expected

Debugging Process

1. Test fails unexpectedly → Check test logic 2. Implementation doesn't work → Simplify further 3. Other tests break → Check for side effects

Advanced TDD Patterns

Outside-In TDD

• Start with acceptance/feature tests
• Work inward to unit tests
• Maintain failing test chain

Mocking Strategies

• Mock external dependencies
• Use dependency injection
• Test behavior, not implementation

Parameterized Tests

@pytest.mark.parametrize("input,expected", [
    ("valid_input", "expected_output"),
    ("edge_case", "edge_output"),
])
def test_multiple_scenarios(input, expected):
    assert process(input) == expected

Test Discovery Module

Overview

Identifies what needs testing or updating by analyzing code structure, git changes, and existing test coverage.

Discovery Strategies

1. detailed Codebase Scan

• Analyze all Python files for test coverage
• Identify functions, classes, and modules without tests
• Check for public API without corresponding tests

2. Git-Based Change Detection

• Parse git diff to find modified files
• Identify new functions or changed signatures
• Detect breaking changes requiring test updates

3. Targeted Analysis

• Accept specific paths or patterns
• Deep dive into particular modules
• Custom filters based on user criteria

Analysis Patterns

Code Structure Analysis

# Example patterns for identifying test needs
def discover_test_targets(codebase_path):
    """Discover what needs testing."""

    # Find Python modules
    modules = find_python_modules(codebase_path)

    # Analyze each module for test coverage
    for module in modules:
        public_functions = extract_public_functions(module)
        test_coverage = analyze_existing_tests(module)

        if test_coverage < 1.0:  # 100% coverage target
            report_missing_tests(module, public_functions, test_coverage)

Change Impact Analysis

def analyze_git_changes():
    """Analyze git changes for test impact."""

    # Get changed files
    changed_files = git_diff --name-only HEAD~1

    # Categorize changes
    for file in changed_files:
        if file.endswith('.py'):
            if is_test_file(file):
                mark_for_review(file)  # May need updates
            else:
                mark_for_test_update(file)  # Code changed

Discovery Outputs

Test Gap Report

• Missing test files
• Uncovered functions/methods
• Modules with low coverage
• Edge cases not tested

Change Impact Report

• Files modified since last test run
• Functions with changed signatures
• Breaking changes detected
• Integration points affected

Priority Scoring

• High: Public API changes, execution markdown changes (SKILL.md files, agent definitions)
• Medium: Internal refactoring, module markdown changes (files under modules/ directories)
• Low: README, CHANGELOG, non-execution documentation, test-only changes

Execution Markdown Detection

Files under skills/, agents/, modules/, or commands/ with .md extension are execution markdown: Claude interprets them as behavioral instructions. These are NOT low-priority documentation changes.

When execution markdown is modified, check for corresponding content tests using the L1/L2/L3 taxonomy. See modules/content-test-discovery.md for detection heuristics and gap analysis, and modules/generation/content-test-templates.md for BDD test scaffolding.

Test Enhancement Module

Overview

Improves existing tests by applying BDD patterns, adding edge cases, and increasing test quality. Transforms basic tests into detailed behavior specifications.

Enhancement Strategies

1. BDD Pattern Application

Transforms traditional tests into BDD-style tests (details below).

2. Edge Case Expansion

Adds detailed edge case testing (details below).

3. Test Organization

Improves test structure and maintainability (details below).

Quality Enhancement Rules

The Rule of Three

For every assertion, add: 1. Positive case: Expected behavior 2. Negative case: Error handling 3. Edge case: Boundary condition

AAA Pattern (Arrange-Act-Assert)

def test_workflow():
    # Arrange - Setup everything needed
    context = create_test_context()
    expected = prepare_expected_result()

    # Act - Perform the action
    result = perform_action(context)

    # Assert - Verify outcomes
    assert result == expected

Test Data Factory Pattern

Create reusable test data factories for consistent test setup.

Enhancement Checklist

For each existing test:

• [ ] Add BDD-style docstring with Given/When/Then
• [ ] Include edge cases and error scenarios
• [ ] Use descriptive test names
• [ ] Add appropriate fixtures
• [ ] Verify test independence
• [ ] Add performance assertions if relevant
• [ ] Include behavior documentation
• [ ] Mock external dependencies appropriately

---

BDD Transformation

Transforms traditional tests into BDD-style tests with clear behavior specifications.

Before: Traditional Test

def test_commit():
    repo = GitRepo()
    repo.add('file.txt')
    result = repo.commit('message')
    assert result is True

After: BDD-Style Test

@pytest.mark.bdd
def test_commit_workflow_with_staged_file():
    """
    GIVEN a Git repository with a staged file
    WHEN the user commits with a message
    THEN the commit should be created successfully
    AND the commit message should match
    """
    # Given
    repo = GitRepo()
    repo.add('file.txt')

    # When
    result = repo.commit('Add new feature')

    # Then
    assert result is True
    assert repo.get_last_commit_message() == 'Add new feature'

Transformation Steps

1. Add descriptive test name: Describe behavior, not implementation 2. Add BDD docstring: Include Given/When/Then clauses 3. Structure test with AAA: Arrange-Act-Assert 4. Add specific assertions: Test behavior, not just truthiness

---

Edge Cases

Systematically adds detailed edge case testing to existing tests.

The Rule of Three

For every assertion, add: 1. Positive case: Expected behavior 2. Negative case: Error handling 3. Edge case: Boundary condition

Example Expansion

Original:

def test_parse_number():
    assert parse_number("123") == 123

Enhanced:

@pytest.mark.parametrize("input_str,expected,description", [
    ("123", 123, "valid positive integer"),
    ("-456", -456, "valid negative integer"),
    ("0", 0, "zero value"),
    ("3.14", 3.14, "valid float"),
    ("1e5", 100000, "scientific notation"),
])
def test_parse_number_valid_inputs(input_str, expected, description):
    """
    GIVEN various valid number strings
    WHEN parsing the string
    THEN it should return the correct number
    """
    assert parse_number(input_str) == expected

@pytest.mark.parametrize("invalid_input", [
    "abc",
    "",
    "12.34.56",
    "1,234",
    None,
])
def test_parse_number_invalid_inputs(invalid_input):
    """
    GIVEN invalid number inputs
    WHEN parsing the string
    THEN it should raise a ValueError
    """
    with pytest.raises(ValueError):
        parse_number(invalid_input)

Common Edge Cases

• Strings: Empty, whitespace, special characters, unicode
• Numbers: Zero, negative, maximum/minimum values, infinity
• Collections: Empty, single item, maximum capacity
• Dates: Leap years, timezone changes, daylight saving
• Files: Missing, permissions, full disk, network errors

---

Organization Patterns

Restructures tests for better maintainability and clarity.

Test Organization Example

class TestGitRepository:
    """BDD-style test suite for GitRepository operations."""

    @pytest.fixture(autouse=True)
    def setup_repo(self, tmp_path):
        """Setup a test repository for each test."""
        self.repo_path = tmp_path / "test_repo"
        self.repo = GitRepository(self.repo_path)
        self.repo.init()

    @pytest.mark.bdd
    def test_init_creates_git_directory(self):
        """
        GIVEN a directory path
        WHEN initializing a git repository
        THEN it should create a .git directory
        """
        assert (self.repo_path / ".git").exists()

    @pytest.mark.bdd
    def test_init_with_existing_repo_raises_error(self):
        """
        GIVEN an existing git repository
        WHEN initializing again
        THEN it should raise RepositoryError
        """
        with pytest.raises(RepositoryError):
            GitRepository(self.repo_path).init()

AAA Pattern (Arrange-Act-Assert)

def test_workflow():
    # Arrange - Setup everything needed
    context = create_test_context()
    expected = prepare_expected_result()

    # Act - Perform the action
    result = perform_action(context)

    # Assert - Verify outcomes
    assert result == expected

Test Data Factory Pattern

class TestDataFactory:
    """Factory for creating test data."""

    @staticmethod
    def create_git_repo(branch="main", with_commits=False):
        repo = GitRepository()
        repo.init(branch)

        if with_commits:
            repo.add("README.md")
            repo.commit("Initial commit")

        return repo

    @staticmethod
    def create_user(role="user", **overrides):
        default_user = {
            "name": "Test User",
            "email": "test@example.com",
            "role": role,
        }
        default_user.update(overrides)
        return User(**default_user)

Test Generation Module

Overview

Automated test scaffolding and generation following TDD/BDD principles. Creates test templates that developers complete using proper TDD workflow.

Capabilities

• Generation strategies: Code analysis, git change detection, API-based
• Test templates: Function, class, and API scaffolding
• Smart features: Parameter discovery, error scenarios, context-aware patterns
• Content tests: BDD templates for skill content assertions

Workflow

1. Analyze: Parse code structure and dependencies 2. Discover: Identify test scenarios and edge cases 3. Generate: Create test scaffolding with BDD patterns 4. Review: Validate generated tests 5. Complete: Developer finishes with TDD cycle

Best Practices

Do Generate

• Test scaffolding with TODO comments
• BDD-style structure templates
• Parameterized test skeletons
• Mock/stub setup patterns

Don't Generate

• Actual test implementations
• Complex assertions
• Business logic
• Mock behavior (too specific)

---

Generation Strategies

Different approaches for discovering what needs testing and generating appropriate test scaffolding.

From Code Analysis

Analyzes existing code to generate appropriate test scaffolding.

def generate_tests_from_code(code_path):
    """Generate test scaffolding from code analysis."""

    # Parse the code
    ast_tree = ast.parse(open(code_path).read())

    # Extract testable elements
    functions = extract_functions(ast_tree)
    classes = extract_classes(ast_tree)

    # Generate test templates
    for func in functions:
        generate_function_test_template(func)

    for cls in classes:
        generate_class_test_template(cls)

From Git Changes

Generates tests for new or modified code.

def generate_tests_for_changes(git_diff):
    """Generate tests based on git changes."""

    changes = parse_git_diff(git_diff)

    for change in changes:
        if change.type == 'new_function':
            generate_new_function_test(change)
        elif change.type == 'modified_signature':
            generate_updated_test(change)
        elif change.type == 'new_class':
            generate_class_test_suite(change)

From API Definitions

Generates integration tests from API contracts or OpenAPI specs.

def generate_api_tests(openapi_spec):
    """Generate BDD-style API tests from OpenAPI spec."""

    for endpoint in openapi_spec.paths:
        for method in endpoint.methods:
            generate_endpoint_test(endpoint, method)

Strategy Selection

Choose based on your needs:

• Code Analysis: For existing code without tests
• Git Changes: For recent modifications
• API Definitions: For contract-first development

---

Test Templates

Standard templates for different types of tests following BDD patterns.

Function Test Template

def test_{function_name}_{scenario}():
    """
    GIVEN {given_context}
    WHEN {when_action}
    THEN {then_expected}
    """
    # TODO: Arrange - Set up test context
    # TODO: Act - Execute the function
    # TODO: Assert - Verify the outcome
    pass

Class Test Template

class Test{ClassName}:
    """BDD-style tests for {ClassName} behavior."""

    def setup_method(self):
        """Setup test instance."""
        self.instance = {ClassName}()

    @pytest.mark.bdd
    def test_{method_name}_{scenario}(self):
        """
        GIVEN {given_context}
        WHEN {when_action}
        THEN {then_expected}
        """
        # TODO: Implement test following BDD pattern
        pass

    def teardown_method(self):
        """Cleanup after each test."""
        pass

API Test Template

@pytest.mark.bdd
def test_{endpoint}_{method}_{scenario}(client):
    """
    GIVEN {given_context}
    WHEN making {method} request to {endpoint}
    THEN response should be {expected_status}
    AND response should contain {expected_content}
    """
    # TODO: Setup request data
    # TODO: Make API call
    # TODO: Verify response
    pass

Using Templates

Templates provide scaffolding that developers complete using TDD: 1. Write the failing test (RED) 2. Implement minimal code to pass (GREEN) 3. Refactor for clarity (REFACTOR)

---

Smart Features

Advanced features that make test generation more intelligent and context-aware.

Parameter Discovery

def discover_test_parameters(func):
    """Discover parameters for test generation."""

    params = inspect.signature(func).parameters

    test_cases = []

    # Happy path
    test_cases.append(generate_happy_path_test(params))

    # Edge cases
    for param in params:
        if param.annotation == str:
            test_cases.append(generate_string_edge_cases(param.name))
        elif param.annotation == int:
            test_cases.append(generate_numeric_edge_cases(param.name))

    return test_cases

Error Scenario Generation

def generate_error_scenarios(func):
    """Generate error handling test scenarios."""

    scenarios = []

    # Type errors
    scenarios.extend(generate_type_error_tests(func))

    # Value errors
    scenarios.extend(generate_value_error_tests(func))

    # Dependency errors
    scenarios.extend(generate_dependency_error_tests(func))

    return scenarios

Context-Aware Patterns

Recognizes common patterns to generate specialized tests:

• Repository pattern
• Service pattern
• Command pattern
• Factory pattern

Quality-Aware Generation

Includes:

• Smart assertion generation
• Parameterized test skeletons
• Mock/stub setup patterns

---

Content Test Templates

BDD test templates for each content assertion level. Use these as scaffolding when generating content tests for execution markdown files.

Reference: leyline:testing-quality-standards/modules/content-assertion-levels.md

Level 1: Keyword Presence

Minimum viable content test. Validates structural completeness.

from pathlib import Path
import pytest


class TestExampleSkillContent:  # Rename to match your skill
    """Feature: example skill has required structural elements.

    As a skill interpreted by Claude Code
    I want all required sections to be present
    So that Claude has complete instructions to follow.

    Level 1: Structural presence checks.
    """

    @pytest.fixture
    def skill_path(self) -> Path:
        # Adjust "example-skill" to your actual skill directory name
        return Path(__file__).parents[3] / "skills" / "example-skill" / "SKILL.md"

    @pytest.fixture
    def skill_content(self, skill_path: Path) -> str:
        return skill_path.read_text()

    @pytest.mark.bdd
    @pytest.mark.unit
    def test_skill_has_required_sections(self, skill_content: str) -> None:
        """Given the skill content
        When Claude loads it for execution
        Then all required sections must be present."""
        required = [
            "## When To Use",
            "## When NOT To Use",
            # Add skill-specific required sections
        ]
        for section in required:
            assert section in skill_content, f"Missing '{section}'"

    @pytest.mark.bdd
    @pytest.mark.unit
    @pytest.mark.parametrize("module_name", [
        # List modules referenced in SKILL.md
    ])
    def test_referenced_modules_exist(
        self, skill_path: Path, module_name: str
    ) -> None:
        """Given modules referenced in the skill
        Then each must exist on disk with content."""
        module_path = skill_path.parent / "modules" / module_name
        assert module_path.exists(), f"Referenced module {module_name} not found"
        content = module_path.read_text()
        min_lines = 10
        assert len(content.splitlines()) >= min_lines, (
            f"Module {module_name} has fewer than {min_lines} lines"
        )

Level 2: Code Example Validity

Validates embedded code examples parse correctly and have required schema.

import json
import re

# --- Level 2: Code example validity ---
# Add these methods inside your Test*Content class

@pytest.fixture
def json_code_blocks(self, skill_content: str):
    """Extract all JSON code blocks from the skill."""
    return re.findall(r"```json\n(.*?)```", skill_content, re.DOTALL)

@pytest.mark.bdd
@pytest.mark.unit
def test_all_json_examples_parse(self, json_code_blocks) -> None:
    """Given JSON code blocks in the skill
    When Claude copies them as configuration templates
    Then every block must be valid JSON."""
    assert len(json_code_blocks) > 0, "Skill should contain JSON examples"
    for i, block in enumerate(json_code_blocks):
        try:
            json.loads(block)
        except json.JSONDecodeError as exc:
            pytest.fail(f"JSON block #{i + 1} is invalid: {exc}")

@pytest.mark.bdd
@pytest.mark.unit
def test_version_references_exist(self, skill_content: str) -> None:
    """Given version references in the skill
    Then each must follow semantic versioning format."""
    versions = re.findall(r"\d+\.\d+\.\d+", skill_content)
    # Verify at least one version reference exists
    # (adjust based on whether skill uses version gates)
    assert len(versions) >= 1, "Expected at least one version reference"

Level 3: Behavioral Contracts

Validates semantic correctness, cross-references, and anti-patterns.

# --- Level 3: Behavioral contracts ---
# Add these methods inside your Test*Content class
# Requires: re (imported in Level 2), Path (imported in Level 1)

@pytest.mark.bdd
@pytest.mark.unit
def test_no_forbidden_language(self, skill_content: str) -> None:
    """Given the skill instructs Claude's behavior
    When Claude reads the instructions
    Then it must NOT find manipulative imperatives.

    Imperative language causes Claude to ignore user intent
    and force actions without consent.
    """
    forbidden = [
        "YOU MUST EXECUTE THIS NOW",
        "MANDATORY AUTO-CONTINUATION",
        # Add context-specific forbidden phrases
    ]
    for phrase in forbidden:
        assert phrase not in skill_content, (
            f"Contains manipulative language: '{phrase}'. "
            "Instructions should be informational, not imperative."
        )

@pytest.mark.bdd
@pytest.mark.unit
def test_offers_multiple_strategies(self, skill_content: str) -> None:
    """Given the skill guides Claude's decisions
    Then it must offer multiple approaches, not force one path.

    Single-path guidance removes user agency.
    """
    strategies = [
        # List expected alternative strategies
    ]
    found = [s for s in strategies if s.lower() in skill_content.lower()]
    min_strategies = 3
    assert len(found) >= min_strategies, (
        f"Too few strategies: {found}, need at least {min_strategies}"
    )

@pytest.mark.bdd
@pytest.mark.unit
def test_version_refs_cross_reference_docs(
    self, skill_content: str
) -> None:
    """Given version references in the skill
    Then each must exist in compatibility documentation.

    Prevents Claude from citing nonexistent versions.
    """
    versions = set(re.findall(r"2\.1\.(\d+)", skill_content))
    compat_dir = (
        Path(__file__).parents[4]  # Adjust depth for your test location
        / "abstract"
        / "docs"
        / "compatibility"
    )
    compat_content = ""
    for compat_file in compat_dir.glob("compatibility-features*.md"):
        compat_content += compat_file.read_text()

    for minor in versions:
        version_str = f"2.1.{minor}"
        assert version_str in compat_content, (
            f"References {version_str} but it's missing from "
            "compatibility-features*.md"
        )

Choosing the Right Level

Common Fixtures

These fixtures appear across all three exemplar test classes:

@pytest.fixture
def skill_path(self) -> Path:
    """Resolve path to the skill file under test."""
    depth = 3  # Adjust based on test file location relative to plugin root
    return Path(__file__).parents[depth] / "skills" / "skill-name" / "SKILL.md"

@pytest.fixture
def skill_content(self, skill_path: Path) -> str:
    """Read the full skill content for assertion."""
    return skill_path.read_text()

@pytest.fixture
def module_path(self) -> Path:
    """Resolve path to a specific module file."""
    depth = 3  # Adjust based on test file location relative to plugin root
    return Path(__file__).parents[depth] / "skills" / "skill-name" / "modules" / "module.md"

Adjust parents[N] based on your test file's depth relative to the plugin root.