Testing Quality Standards

Testing Quality Standards

Shared quality standards and metrics for testing across all plugins in the Claude Night Market ecosystem.

When To Use

• Establishing test quality gates and coverage targets
• Validating test suite against quality standards

When NOT To Use

• Exploratory testing or spike work
• Projects with established quality gates that meet requirements

Table of Contents

1. Coverage Thresholds 2. Quality Metrics 3. Detailed Topics

Coverage Thresholds

Quality Metrics

Structure

• [ ] Clear test organization
• [ ] Meaningful test names
• [ ] Proper setup/teardown
• [ ] Isolated test cases

Coverage

• [ ] Critical paths covered
• [ ] Edge cases tested
• [ ] Error conditions handled
• [ ] Integration points verified

Maintainability

• [ ] DRY test code
• [ ] Reusable fixtures
• [ ] Clear assertions
• [ ] Minimal mocking

Reliability

• [ ] No flaky tests
• [ ] Deterministic execution
• [ ] No order dependencies
• [ ] Fast feedback loop

Detailed Topics

For implementation patterns and examples:

• [Anti-Patterns](modules/anti-patterns.md) - Common testing mistakes with before/after examples
• [Best Practices](modules/best-practices.md) - Core testing principles and exit criteria
• [Content Assertion Levels](modules/content-assertion-levels.md) - L1/L2/L3 taxonomy for testing LLM-interpreted markdown files

Integration with Plugin Testing

This skill provides foundational standards referenced by:

• pensive:test-review - Uses coverage thresholds and quality metrics
• parseltongue:python-testing - Uses anti-patterns and best practices
• sanctum:test-* - Uses quality checklist and content assertion levels for test validation
• imbue:proof-of-work - Uses content assertion levels to enforce Iron Law on execution markdown

Reference in your skill's frontmatter:

dependencies: [leyline:testing-quality-standards]

Verification: Run pytest -v to verify tests pass.

Troubleshooting

Common Issues

Tests not discovered Ensure test files match pattern test_*.py or *_test.py. Run pytest --collect-only to verify.

Import errors Check that the module being tested is in PYTHONPATH or install with pip install -e .

Async tests failing Install pytest-asyncio and decorate test functions with @pytest.mark.asyncio

Testing Anti-Patterns

Testing Implementation Details

# Bad: Testing private methods
def test_internal_method():
    service = UserService()
    result = service._validate_email("test@example.com")  # Testing private method
    assert result is True

# Good: Testing public behavior
def test_user_creation_validates_email():
    service = UserService()
    with pytest.raises(ValidationError):
        service.create_user("Alice", "invalid-email")

Over-Mocking

# Bad: Mocking simple calculations
@patch("calculator.add")
def test_total(mock_add):
    mock_add.return_value = 5
    assert calculate_total(2, 3) == 5

# Good: Test the actual logic
def test_total():
    assert calculate_total(2, 3) == 5

Missing Assertions

# Bad: No assertions
def test_user_creation():
    user = create_user("Alice", "alice@example.com")
    print(f"Created user: {user}")  # No assertions

# Good: Clear assertions
def test_user_creation():
    user = create_user("Alice", "alice@example.com")
    assert user.name == "Alice"
    assert user.email == "alice@example.com"
    assert user.is_active is True

Shared Mutable State

# Bad: Shared mutable state between tests
shared_data = []

def test_append():
    shared_data.append(1)
    assert len(shared_data) == 1

def test_another():
    shared_data.append(2)
    assert len(shared_data) == 1  # Fails due to shared state

# Good: Use fixtures for isolation
@pytest.fixture
def data():
    return []

def test_append(data):
    data.append(1)
    assert len(data) == 1

Test Order Dependencies

# Bad: Tests depend on execution order
def test_step_1():
    global user
    user = create_user("Alice")

def test_step_2():
    # Depends on test_step_1 running first
    assert user.name == "Alice"

# Good: Independent tests with fixtures
@pytest.fixture
def user():
    return create_user("Alice")

def test_user_creation(user):
    assert user.name == "Alice"

Dead Waits

# Bad: Arbitrary sleeps
def test_async_operation():
    start_operation()
    time.sleep(5)  # Arbitrary wait
    assert operation_complete()

# Good: Wait for condition
def test_async_operation():
    start_operation()
    wait_for_condition(lambda: operation_complete(), timeout=5)
    assert operation_complete()

Testing Best Practices

Core Principles

1. Test behavior, not implementation - Focus on what, not how 2. One concept per test - Keep tests focused 3. Arrange-Act-Assert - Consistent structure 4. BDD language - Given/When/Then for clarity 5. Fast feedback - Tests should run quickly

Naming and Structure

6. Descriptive names - test_user_creation_with_invalid_email_raises_error 7. Independent tests - No shared state between tests 8. Use fixtures - Avoid setup duplication

Boundaries and Coverage

9. Mock at boundaries - Only mock external dependencies 10. Measure coverage - Aim for meaningful, not just high

Exit Criteria

• Coverage thresholds documented and understood
• Quality metrics defined and measurable
• Anti-patterns identified and avoided
• Best practices applied consistently

Content Assertion Levels

Why Content Assertions Exist

Markdown files under skills/, agents/, modules/, and commands/ are execution markdown: Claude Code interprets them as behavioral instructions, not static documentation. When these files contain broken JSON schemas, stale version references, or manipulative language, Claude produces broken outputs or harmful behaviors.

Content assertions catch these problems before users encounter them. The guiding question: "If this content were wrong, what would Claude do incorrectly?"

The Three Levels

Level 1: Keyword Presence

Validates that required sections, terminology, and module references exist.

assert "## When To Use" in skill_content
assert "pre-invocation" in module_content.lower()

Cheapest to write. Catches structural regressions (missing sections, dropped references) but not semantic errors (wrong JSON schema, stale version).

Level 2: Code Example Validity

Parses embedded code examples and validates structure. Catches broken examples that Claude would copy verbatim into user configurations.

# Extract and validate all JSON code blocks
blocks = re.findall(r"```json\n(.*?)```", content, re.DOTALL)
for block in blocks:
    parsed = json.loads(block)  # Must parse without error

# Validate schema structure
assert "matcher" in hook_def
assert hook_def["type"] in ("command", "http")

Also covers: YAML frontmatter parsing, version string extraction with regex, section extraction for targeted validation.

Level 3: Behavioral Contracts

Validates semantic correctness across documents and ensures the content teaches Claude correct behavior.

Cross-reference validation: Version strings referenced in one document must exist in compatibility docs.

versions = re.findall(r"2\.1\.(\d+)", module_content)
compat_texts = [p.read_text() for p in compat_dir.glob("compatibility-features*.md")]
compat_content = "\n".join(compat_texts)
for v in versions:
    assert f"2.1.{v}" in compat_content

Anti-pattern detection: Forbidden language that causes Claude to ignore user intent.

forbidden = ["MANDATORY AUTO-CONTINUATION", "YOU MUST EXECUTE THIS NOW"]
for phrase in forbidden:
    assert phrase not in skill_content

Decision framework completeness: Skills must offer multiple strategies, not force one path.

strategies = ["/clear", "/catchup", "auto-compact", "continuation"]
found = [s for s in strategies if s.lower() in content.lower()]
assert len(found) >= 3

When to Apply Each Level

Test Class Conventions

• Name content assertion classes with a Content suffix: TestSubagentCoordinationModuleContent
• State the level in the class docstring: Level 2: Version references are internally consistent.
• Use @pytest.mark.bdd and @pytest.mark.unit markers
• Use Path(__file__).parents[N] for relative path resolution to skill files
• Fixtures: skill_path returns the Path, skill_content calls .read_text()

What NOT to Test with Content Assertions

• Prose style or word choice (use scribe:slop-detector instead)
• Exact wording that makes tests brittle to rewording
• Line counts or file sizes (not behavioral)
• Formatting or whitespace

Exemplars

Three test classes established the taxonomy in practice:

File locations:

• plugins/abstract/tests/test_skill_structure.py
• plugins/conserve/tests/unit/skills/test_clear_context.py
• plugins/conserve/tests/unit/skills/test_context_optimization.py