Test Quality Inspector

Example Test Inspection Report

Scenario: User Registration Feature

Engineer's Test Suite

# test_user_registration.py

def test_user_creation():
    """Test user creation"""
    user = create_user("test@example.com", "password123")
    assert user

def test_login():
    """Test login"""
    user = create_user("test@example.com", "password123")
    result = login("test@example.com", "password123")
    assert result

def test_duplicate_email():
    """Test duplicate email"""
    create_user("test@example.com", "password123")
    user2 = create_user("test@example.com", "password456")
    assert user2

---

QA Inspection Report

Test 1: test_user_creation()

Claimed Intent: Test user creation Actually Tests: Object instantiation only

Issues Found

🔴 CRITICAL: Worthless Assertion

assert user  # Line 5

• Would pass with corrupt data
• Would pass with wrong email
• Would pass with invalid password hash
• Would pass if database save failed

Mental Debug: User object with garbage data would pass this test.

🟠 HIGH: Missing Validation Tests

• No test for invalid email format
• No test for weak password
• No test for duplicate email (separate test exists but broken)
• No test for SQL injection in email

🟠 HIGH: No Persistence Verification

• Test doesn't verify user is saved to database
• Test doesn't verify user can be retrieved
• Transaction might be rolled back

🟡 MEDIUM: Vague Test Name

• "test_user_creation" doesn't describe what should happen
• Better: "test_user_creation_with_valid_data_persists_to_database"

Suggested Improvements

def test_user_creation_with_valid_data_persists_to_database():
    """Creating a user with valid email and password should:
    1. Create user object with correct attributes
    2. Save to database
    3. Hash password (not store plaintext)
    4. Set default role and active status
    """
    email = "test@example.com"
    password = "SecurePass123!"

    user = create_user(email, password)

    # Verify user object
    assert user.id is not None, "User should have an ID after creation"
    assert user.email == email, "Email should match input"
    assert user.is_active is True, "New users should be active by default"
    assert user.role == "user", "New users should have 'user' role"
    assert user.created_at is not None, "Created timestamp should be set"

    # Verify password is hashed, not plaintext
    assert user.password_hash != password, "Password should be hashed"
    assert len(user.password_hash) > 50, "Password hash should be substantial"

    # Verify persistence
    retrieved_user = User.get_by_email(email)
    assert retrieved_user is not None, "User should be retrievable from database"
    assert retrieved_user.id == user.id, "Retrieved user should match created user"

def test_user_creation_with_invalid_email_format_raises_validation_error():
    """Creating a user with malformed email should raise ValidationError"""
    invalid_emails = [
        "not-an-email",
        "@example.com",
        "test@",
        "test space@example.com",
        "",
    ]

    for invalid_email in invalid_emails:
        with pytest.raises(ValidationError) as exc:
            create_user(invalid_email, "password123")

        assert "email" in str(exc.value).lower()
        assert "invalid" in str(exc.value).lower()

def test_user_creation_with_weak_password_raises_validation_error():
    """Creating a user with weak password should raise ValidationError"""
    weak_passwords = [
        "123",          # Too short
        "password",     # No numbers
        "12345678",     # No letters
        "",             # Empty
    ]

    for weak_password in weak_passwords:
        with pytest.raises(ValidationError) as exc:
            create_user("test@example.com", weak_password)

        assert "password" in str(exc.value).lower()

Risk Level: 🔴 CRITICAL Action: ❌ BLOCK - Core functionality not tested Estimated Fix Time: 30 minutes

---

Test 2: test_login()

Claimed Intent: Test login Actually Tests: Function call completes

Issues Found

🔴 CRITICAL: Worthless Assertion

assert result  # Line 11

• Passes with any truthy value
• Doesn't verify session/token
• Doesn't verify user authentication state

🔴 CRITICAL: Missing Negative Tests

• No test for wrong password
• No test for non-existent user
• No test for locked account
• No test for expired credentials

🟠 HIGH: No Session Verification

• Doesn't verify authentication token
• Doesn't verify session expiry
• Doesn't verify user context in session

🟡 MEDIUM: Test Depends on Previous Test

• Creates user in this test
• Should use fixture or setup
• Tests should be independent

Suggested Improvements

@pytest.fixture
def registered_user():
    """Fixture providing a registered user for login tests"""
    user = create_user("test@example.com", "SecurePass123!")
    yield user
    # Cleanup if needed
    User.delete(user.id)

def test_login_with_valid_credentials_returns_authenticated_session(registered_user):
    """Logging in with correct email and password should:
    1. Return authentication token/session
    2. Set authenticated state
    3. Include user context
    4. Set appropriate expiry
    """
    session = login(registered_user.email, "SecurePass123!")

    assert session is not None, "Login should return session"
    assert session.is_authenticated is True, "Session should be authenticated"
    assert session.user_id == registered_user.id, "Session should contain user ID"
    assert session.token is not None, "Session should have authentication token"
    assert session.expires_at > datetime.now(), "Session should have future expiry"
    assert (session.expires_at - datetime.now()).seconds >= 3600, "Session should last at least 1 hour"

def test_login_with_wrong_password_raises_authentication_error(registered_user):
    """Logging in with incorrect password should raise AuthenticationError"""
    with pytest.raises(AuthenticationError) as exc:
        login(registered_user.email, "WrongPassword")

    assert "Invalid credentials" in str(exc.value)
    assert "password" in str(exc.value).lower()

def test_login_with_nonexistent_email_raises_authentication_error():
    """Logging in with non-existent email should raise AuthenticationError"""
    with pytest.raises(AuthenticationError) as exc:
        login("doesnotexist@example.com", "password")

    assert "Invalid credentials" in str(exc.value)
    # Note: Don't reveal if email exists (security)

def test_login_with_locked_account_raises_account_locked_error(registered_user):
    """Logging in to locked account should raise AccountLockedError"""
    lock_account(registered_user.id)

    with pytest.raises(AccountLockedError) as exc:
        login(registered_user.email, "SecurePass123!")

    assert registered_user.email in str(exc.value)

def test_login_with_empty_password_raises_validation_error(registered_user):
    """Logging in with empty password should raise ValidationError"""
    with pytest.raises(ValidationError) as exc:
        login(registered_user.email, "")

    assert "password" in str(exc.value).lower()
    assert "required" in str(exc.value).lower()

Risk Level: 🔴 CRITICAL Action: ❌ BLOCK - Authentication not actually tested Estimated Fix Time: 45 minutes

---

Test 3: test_duplicate_email()

Claimed Intent: Test duplicate email handling Actually Tests: Second user creation succeeds (WRONG!)

Issues Found

🔴 CRITICAL: Test is Backwards

user2 = create_user("test@example.com", "password456")
assert user2  # Line 17

• This test expects duplicate creation to SUCCEED
• It should expect it to FAIL with an error
• Test passes when it should fail
• This is testing the opposite of what's needed

🔴 CRITICAL: False Confidence

• Production bug: duplicate emails are allowed
• Test claims to verify duplicate prevention
• Test actually verifies duplicates work
• QA might approve thinking it's covered

🟡 MEDIUM: Same Email Issue as Other Tests

• If this fixed to expect error, needs all improvements from Test 1

Suggested Fix

def test_create_user_with_duplicate_email_raises_integrity_error():
    """Creating a user with an email that already exists should:
    1. Raise IntegrityError or ValidationError
    2. Not create duplicate user in database
    3. Preserve existing user data
    """
    email = "test@example.com"

    # Create first user
    user1 = create_user(email, "FirstPassword123!")
    initial_count = User.count()

    # Attempt to create duplicate
    with pytest.raises((IntegrityError, ValidationError)) as exc:
        create_user(email, "SecondPassword456!")

    assert "email" in str(exc.value).lower()
    assert "duplicate" in str(exc.value).lower() or "exists" in str(exc.value).lower()

    # Verify no new user created
    assert User.count() == initial_count, "User count should not increase"

    # Verify original user unchanged
    original_user = User.get_by_email(email)
    assert original_user.id == user1.id, "Original user should be intact"
    assert original_user.verify_password("FirstPassword123!"), "Original password should work"
    assert not original_user.verify_password("SecondPassword456!"), "New password should not work"

Risk Level: 🔴 CRITICAL Action: ❌ BLOCK - Test verifies opposite of requirement Estimated Fix Time: 20 minutes

---

Summary Report

Overall Assessment

Test Suite Quality: 🔴 FAILING

Critical Issues: 3

• Test 1: Doesn't actually test user creation
• Test 2: Doesn't actually test authentication
• Test 3: Tests opposite of requirement

Total Tests: 3 Effective Tests: 0 Coverage: High (claims) Protection: None (reality)

Risk Assessment

Production Risk: 🔴 EXTREME

Current test suite provides zero protection against:

• Data corruption in user creation
• Authentication bypass
• Duplicate email registration
• Password security issues
• Database integrity issues

Confidence Level: 0% - Tests passing means nothing

Required Actions

Immediate (Block Merge)

1. Rewrite all three tests with proper assertions 2. Add negative test cases (12+ tests needed) 3. Verify tests catch intentional bugs 4. Add fixture for test user management

Follow-up (Required for completion)

1. Add edge case tests (15+ additional tests) 2. Add integration tests for full registration flow 3. Add security tests (SQL injection, XSS, etc.) 4. Add performance tests for registration endpoint

Estimated Timeline

• Fix critical issues: 2-3 hours
• Complete test suite: 1 day
• Review and iteration: 0.5 days

Total: 1.5-2 days for proper test coverage

Recommendation

❌ BLOCK MERGE

Do not approve this PR. Tests provide false confidence and mask critical bugs.

Evidence:

• All tests would pass with completely broken functionality
• Duplicate email test verifies the opposite of requirements
• No actual behavior is verified

Next Steps: 1. Engineer rewrites tests following examples above 2. QA re-inspects rewritten tests 3. QA verifies tests catch intentional bugs 4. Only then approve merge

---

Lessons for Engineer

What Went Wrong

1. Wrote tests after code - Led to tests that just confirm code runs 2. Weak assertions - "assert x" proves nothing 3. No mental debugging - Didn't verify tests catch bugs 4. No negative testing - Only tested happy path 5. Misunderstood duplicate test - Test verified opposite

How to Improve

1. Write tests first (TDD) - Prevents these issues 2. Specific assertions - Verify exact values 3. Mental debugging - Break code, ensure test fails 4. Test failures explicitly - Every success needs failure test 5. Read test name carefully - Test what you claim to test

TDD Would Have Prevented This

If tests were written first:

# Write this FIRST (it will fail):
def test_user_creation_with_valid_data_persists_to_database():
    user = create_user("test@example.com", "password")
    assert user.email == "test@example.com"  # Will fail until create_user works
    ...

# Then implement create_user to make it pass

See the Test-Driven Development skill for complete TDD workflow (available in the skill library for comprehensive TDD guidance).

---

Sign-off

QA Inspector: [Your name] Date: [Date] Status: ❌ REJECTED Reason: Tests provide zero protection, must be rewritten Re-inspection Required: Yes

---

This is what thorough test inspection looks like. Better to catch these issues now than in production.

Assertion Quality Guide

Assertion Strength Spectrum

Level 1: Worthless (Don't use)

assert result              # Always passes unless None/False
assert x                   # Meaningless
assert True                # Always passes

Problem: These pass even when functionality is completely broken.

Level 2: Existence Only (Very weak)

assert result is not None
assert len(items) > 0
assert user

Problem: Verifies something exists, not that it's correct.

When acceptable: As a precondition before stronger assertions.

Level 3: Type Checking (Weak)

assert isinstance(result, dict)
assert type(user) == User

Problem: Right type, but could be garbage data.

When acceptable: Combined with value assertions.

Level 4: Property Existence (Moderate)

assert 'email' in user
assert hasattr(response, 'status_code')

Problem: Property exists but could have wrong value.

When acceptable: When structure matters more than values.

Level 5: Value Verification (Strong)

assert user.email == "test@example.com"
assert response.status_code == 200
assert len(items) == 5

Good: Verifies specific correctness.

Still missing: Failure cases, edge conditions.

Level 6: Complete Verification (Strongest)

# Success case
assert user.email == "test@example.com"
assert user.is_active is True
assert user.created_at <= datetime.now()

# Failure case
with pytest.raises(ValidationError) as exc:
    create_user("invalid-email")
assert "Invalid email format" in str(exc.value)

# Side effects
assert User.count() == initial_count + 1

Best: Verifies correctness AND failure modes AND side effects.

Assertion Patterns

Pattern 1: Value Assertions

Bad: Existence Only

def test_user_creation():
    user = create_user("test@example.com", "password")
    assert user  # Weak!

Good: Specific Values

def test_user_creation_sets_correct_attributes():
    user = create_user("test@example.com", "password123")

    assert user.email == "test@example.com"
    assert user.is_active is True
    assert user.role == "user"
    assert user.created_at is not None

Pattern 2: Collection Assertions

Bad: Count Only

def test_get_users():
    users = get_all_users()
    assert len(users) > 0  # Weak!

Good: Content Verification

def test_get_users_returns_all_active_users():
    create_user("user1@example.com")
    create_user("user2@example.com")
    create_inactive_user("inactive@example.com")

    users = get_all_users()

    assert len(users) == 2
    emails = [u.email for u in users]
    assert "user1@example.com" in emails
    assert "user2@example.com" in emails
    assert "inactive@example.com" not in emails

Pattern 3: Error Assertions

Bad: Generic Exception

def test_invalid_input():
    try:
        process_data(None)
        assert False, "Should have raised"
    except Exception:
        pass  # Too generic!

Good: Specific Error with Message

def test_process_data_with_none_raises_validation_error():
    with pytest.raises(ValidationError) as exc:
        process_data(None)

    assert "Data cannot be None" in str(exc.value)
    assert exc.value.field == "data"

Pattern 4: State Change Assertions

Bad: No Before/After Check

def test_update_user():
    user = get_user(1)
    update_user(1, email="new@example.com")
    # No verification of actual change!

Good: Verify State Transition

def test_update_user_email_changes_email_field():
    user = create_user("old@example.com")
    original_email = user.email

    update_user(user.id, email="new@example.com")
    updated_user = get_user(user.id)

    assert updated_user.email == "new@example.com"
    assert updated_user.email != original_email
    assert updated_user.id == user.id  # Same user

Pattern 5: API Response Assertions

Bad: Status Code Only

def test_api_endpoint():
    response = client.get("/api/users")
    assert response.status_code == 200  # Weak!

Good: Complete Response Verification

def test_get_users_endpoint_returns_user_list():
    create_user("test1@example.com")
    create_user("test2@example.com")

    response = client.get("/api/users")

    assert response.status_code == 200
    assert response.headers["Content-Type"] == "application/json"

    data = response.json()
    assert "users" in data
    assert len(data["users"]) == 2
    assert data["users"][0]["email"] == "test1@example.com"

Assertion Anti-Patterns

Anti-Pattern 1: Testing Mock Behavior

# BAD
def test_send_email():
    mock_mailer = Mock()
    send_email(mock_mailer, "test@example.com", "Hello")
    assert mock_mailer.send.called  # Testing mock!

Fix: Test real behavior or use test doubles that verify contracts.

Anti-Pattern 2: Asserting Implementation Details

# BAD
def test_password_hashing():
    user = User("test", "password")
    assert user._hash_algorithm == "bcrypt"  # Implementation detail!

Fix: Test behavior (password verification works) not implementation.

Anti-Pattern 3: Multiple Concepts in One Assert

# BAD
def test_user_and_profile():
    result = create_user_with_profile(...)
    assert result.user and result.profile  # Too vague!

Fix: Separate assertions for each concept.

Anti-Pattern 4: Overly Lenient Assertions

# BAD
def test_calculate_total():
    total = calculate_total([1, 2, 3])
    assert total > 0  # Way too lenient!

Fix: Assert exact expected value.

Assertion Best Practices

1. One Concept Per Assertion

# Good: Clear what's being verified
assert user.email == expected_email
assert user.is_active is True
assert user.created_at is not None

2. Meaningful Failure Messages

# Good: Helpful when test fails
assert user.age >= 18, f"User age {user.age} is below minimum 18"

3. Assert Actual vs Expected

# Good: Clear which is which
assert actual == expected  # Convention: actual first

4. Test Both Paths

# Good: Success and failure
def test_valid_login_succeeds():
    assert login("user", "pass").success is True

def test_invalid_login_fails():
    assert login("user", "wrong").success is False

5. Verify Side Effects

# Good: Check state changes
def test_delete_user_removes_from_database():
    user = create_user("test@example.com")
    initial_count = User.count()

    delete_user(user.id)

    assert User.count() == initial_count - 1
    assert User.get(user.id) is None

Assertion Checklist

For each assertion, ask:

• [ ] Specificity: Is this assertion specific enough?
• [ ] Meaningfulness: Would this fail if functionality breaks?
• [ ] Completeness: Are all aspects verified?
• [ ] Clarity: Is it obvious what's being tested?
• [ ] Failure Messages: Will I know why it failed?

Quick Reference

Strong Assertions ✅

assert actual == expected_value
assert result.field == specific_value
assert len(collection) == expected_count
assert "expected text" in result.message
with pytest.raises(SpecificError):
    dangerous_operation()

Weak Assertions ❌

assert result
assert result is not None
assert len(collection) > 0
assert True
assert mock.called

Remember

"An assertion that would pass with broken code is not an assertion."

"Weak assertions create false confidence."

"Test behavior, not implementation."

"Verify correctness, not just existence."

Test Inspection Checklist

Quick Inspection (2 minutes per test)

Intent Check

• [ ] Test name describes behavior, not method
• [ ] Expected behavior is stated
• [ ] Single clear purpose

Assertion Check

• [ ] Assertions match intent
• [ ] Specific values, not just "not None"
• [ ] Would catch regressions

Failure Check

• [ ] Error cases tested
• [ ] Would fail if feature removed
• [ ] Meaningful failure messages

Deep Inspection (5-10 minutes per test)

1. Intent Analysis

Question: What is this test supposed to verify?

Checklist:
[ ] Test name clearly states behavior
[ ] Docstring explains expected outcome
[ ] Setup reflects realistic scenario
[ ] Test has single responsibility

Red Flags:

• "test_method_name" (tests method, not behavior)
• "test_user" (too vague)
• "test_works" (what does "works" mean?)

2. Setup Quality

Question: Is the setup realistic and complete?

Checklist:
[ ] Test data matches production patterns
[ ] All dependencies are initialized
[ ] State is valid and achievable
[ ] Mocks are justified and complete

Red Flags:

• Mock data that would never occur
• Missing required fields
• Bypassing normal constraints
• Over-simplified scenarios

3. Execution Verification

Question: Does execution match intent?

Checklist:
[ ] Real code paths are exercised
[ ] System under test is not mocked
[ ] Integration points are tested
[ ] Side effects are verifiable

Red Flags:

• Mocking the system under test
• Testing mock behavior
• Skipping critical paths
• No integration verification

4. Assertion Strength

Question: Do assertions prove correctness?

Checklist:
[ ] Assertions verify specific values
[ ] Success criteria are explicit
[ ] Failure cases are tested
[ ] Error messages are meaningful

Red Flags:

• assert result (too weak)
• assert x is not None (existence only)
• assert mock.called (testing mock)
• No negative assertions

5. Regression Prevention

Question: Would this catch real bugs?

Checklist:
[ ] Test would fail if feature removed
[ ] Boundary conditions tested
[ ] Edge cases covered
[ ] Known bug patterns caught

Red Flags:

• Test passes with broken code
• Only happy path tested
• No boundary testing
• Missing error scenarios

Inspection Report Template

### Test: [test_name]

**Intent:** [What test claims to verify]
**Actually Tests:** [What it really tests]

**Strengths:**
- [Good aspects]

**Issues:**
1. [Issue] - [Impact]
2. [Issue] - [Impact]

**Suggestions:**
1. [Specific improvement]
2. [Additional test case]
3. [Assertion strengthening]

**Risk Level:** [LOW/MEDIUM/HIGH]
**Action:** [APPROVE/REQUEST_CHANGES/BLOCK]

Risk Assessment

HIGH Risk (Block merge)

• Weak assertions that would miss bugs
• Testing mock behavior only
• Missing critical failure cases
• Test passes with broken functionality

MEDIUM Risk (Request changes)

• Incomplete coverage of scenarios
• Weak but not broken assertions
• Missing some error cases
• Could be stronger

LOW Risk (Approve with notes)

• Minor naming improvements
• Additional nice-to-have tests
• Documentation enhancements
• Style consistency

Common Patterns to Inspect

Pattern 1: CRUD Operations

# Check:
[ ] Create: Valid data, duplicate prevention, validation
[ ] Read: Exists, doesn't exist, multiple results
[ ] Update: Valid changes, invalid changes, concurrency
[ ] Delete: Exists, doesn't exist, cascade effects

Pattern 2: Authentication/Authorization

# Check:
[ ] Valid credentials succeed
[ ] Invalid credentials fail
[ ] Locked accounts rejected
[ ] Expired tokens rejected
[ ] Insufficient permissions denied

Pattern 3: Data Validation

# Check:
[ ] Valid data accepted
[ ] Invalid format rejected
[ ] Missing required fields rejected
[ ] Boundary values tested
[ ] Type coercion tested

Pattern 4: API Endpoints

# Check:
[ ] Success response structure
[ ] Error response structure
[ ] Status codes correct
[ ] Request validation
[ ] Response validation

Mental Debugging Technique

For each test, mentally introduce bugs:

Bug 1: Remove Core Logic

If I comment out the main functionality,
would this test fail?

If NO: Test is not testing the right thing

Bug 2: Return Wrong Data

If I return incorrect values,
would assertions catch it?

If NO: Assertions are too weak

Bug 3: Skip Validation

If I remove input validation,
would test catch invalid data?

If NO: Missing negative test cases

Bug 4: Break Error Handling

If I make errors silently fail,
would test detect it?

If NO: Not testing failure paths

Inspection Efficiency Tips

Quick Wins

1. Check test names first (30 seconds) 2. Scan assertions (30 seconds) 3. Look for negative tests (30 seconds) 4. Check mock usage (30 seconds)

Deep Dive Triggers

• Test name is vague
• Only one or two assertions
• No error cases visible
• Heavy mock usage
• Test recently added

Batch Inspection

For test suite review:
1. Group tests by feature
2. Check coverage gaps between tests
3. Look for redundant tests
4. Identify missing scenarios
5. Verify integration tests exist

Remember

✅ Good inspection prevents:

• False confidence from weak tests
• Production bugs slipping through
• Wasted time on bad tests
• Technical debt accumulation

❌ Don't just check:

• That tests exist
• That tests pass
• That coverage is high
• That names follow convention

✓ Always verify:

• Tests test the right thing
• Assertions are meaningful
• Failures are caught
• Regressions are prevented

Test Quality Red Flags

Immediate Red Flags (Block Merge)

🚩 Category 1: Worthless Assertions

Red Flag: "assert result"

def test_get_user():
    result = get_user(1)
    assert result  # Passes unless None/False!

Why Bad: Passes with ANY truthy value, including garbage data.

Impact: Won't catch most bugs.

Action: Require specific value assertions.

Red Flag: "assert not None"

def test_create_user():
    user = create_user("test@example.com")
    assert user is not None  # Only checks existence!

Why Bad: Verifies object exists, not that it's correct.

Impact: Corrupt data still passes.

Action: Verify actual properties.

🚩 Category 2: Testing Mocks

Red Flag: Mocking System Under Test

def test_user_service():
    mock_service = Mock(UserService)
    mock_service.create.return_value = User()
    result = mock_service.create("test")  # Testing mock!
    assert result

Why Bad: Tests mock behavior, not real service.

Impact: Real service could be broken.

Action: Test real service with test dependencies.

Red Flag: Asserting on Mocks

def test_send_notification():
    mock_mailer = Mock()
    send_notification(mock_mailer, "Hello")
    assert mock_mailer.send.called  # Asserting mock!

Why Bad: Mock always does what you tell it.

Impact: Real mailer might not work.

Action: Use test SMTP or capture real emails.

🚩 Category 3: Missing Negative Tests

Red Flag: Only Happy Path

# Only this test exists:
def test_login_succeeds():
    result = login("user", "pass")
    assert result.success

Missing:

• Wrong password test
• Locked account test
• Non-existent user test
• Empty password test

Impact: Failures not caught until production.

Action: Require failure test for each success test.

🚩 Category 4: Vague Test Names

Red Flag: Method Names as Test Names

def test_create_user():  # Too vague!
def test_login():        # What about login?
def test_validate():     # Validate what?

Why Bad: Doesn't describe expected behavior.

Impact: Hard to know what failed.

Action: Names must describe behavior:

• test_create_user_with_valid_email_persists_to_database
• test_login_with_wrong_password_raises_authentication_error

🚩 Category 5: False Positives

Red Flag: Test Passes with Broken Code

def test_data_processing():
    process_data([1, 2, 3])
    assert True  # Always passes!

Mental Debug: Comment out process_data - test still passes!

Why Bad: Provides zero protection.

Impact: Bugs slip through.

Action: Remove test or fix assertions.

Warning Red Flags (Request Changes)

⚠️ Missing Edge Cases

def test_divide():
    assert divide(10, 2) == 5
    # Missing: divide by zero, negative numbers, floats

Action: Add boundary condition tests.

⚠️ Incomplete Mocks

mock_db = Mock()
mock_db.query.return_value = [User()]
# Missing: error cases, empty results, timeouts

Action: Mock all realistic scenarios.

⚠️ Weak Error Assertions

try:
    dangerous_operation()
except Exception:  # Too broad!
    pass

Action: Assert specific exception type and message.

⚠️ No Teardown/Cleanup

def test_file_creation():
    create_file("/tmp/test.txt")
    # File left behind!

Action: Add cleanup or use fixtures.

⚠️ Test Interdependence

def test_a():
    global state
    state = "configured"

def test_b():
    # Depends on test_a running first!
    assert state == "configured"

Action: Make tests independent.

Red Flag Detection Checklist

Quick Scan (30 seconds)

[ ] Assertions are specific (not just "assert x")
[ ] Test name describes behavior
[ ] No mocking of system under test
[ ] Negative test cases exist
[ ] Would fail if feature removed

Deep Inspection (2 minutes)

[ ] Edge cases covered
[ ] Error messages meaningful
[ ] Setup is realistic
[ ] Teardown present
[ ] Tests are independent
[ ] Mocks are justified
[ ] All paths tested

Red Flag Severity Matrix

🔴 CRITICAL (Block immediately)

• Testing mock behavior
• Always-passing tests
• No assertions at all
• Mocking system under test

🟠 HIGH (Strong recommend blocking)

• Only "assert not None"
• No negative tests
• Vague test names
• Would pass with broken code

🟡 MEDIUM (Request improvements)

• Missing edge cases
• Weak error handling tests
• Incomplete scenarios
• Poor test organization

🟢 LOW (Notes for improvement)

• Could be more descriptive
• Additional tests would help
• Minor refactoring suggestions
• Documentation improvements

Common Patterns of Problematic Tests

Pattern 1: The Optimist

# Only tests that things work
def test_happy_path_1(): ...
def test_happy_path_2(): ...
# No sad path tests!

Red Flag: No tests for failures, errors, or edge cases.

Pattern 2: The Mock Enthusiast

# Everything is mocked
mock_service = Mock()
mock_repo = Mock()
mock_validator = Mock()
# Nothing real is tested!

Red Flag: Over-mocking hides real integration issues.

Pattern 3: The Existence Checker

# Just checks things exist
assert user
assert response
assert result is not None

Red Flag: Verifies existence, not correctness.

Pattern 4: The False Friend

# Test that can't fail
def test_always_passes():
    do_something()
    assert True  # Will never fail!

Red Flag: Gives false confidence.

Pattern 5: The Mysterious Failure

# Unhelpful when it fails
assert process_data(input) == process_data(expected)
# Which step failed? No idea!

Red Flag: Can't debug from failure message.

Detection Techniques

Technique 1: Mental Debugging

For each test, mentally:
1. Comment out core functionality
2. Would test fail?
3. If NO: Test is broken

Example:
def test_save_user():
    user = User("test")
    # save_user(user)  # Commented out
    assert user  # Still passes! RED FLAG

Technique 2: Garbage Data Test

For each test, mentally:
1. Return garbage data
2. Would assertions catch it?
3. If NO: Assertions are weak

Example:
def test_get_user():
    user = get_user(1)  # Returns {"garbage": true}
    assert user  # Passes with garbage! RED FLAG

Technique 3: Wrong Type Test

For each test, mentally:
1. Return wrong type
2. Would test fail?
3. If NO: Type checking missing

Example:
def test_calculate():
    result = calculate(5, 3)  # Returns "8" (string!)
    assert result  # Passes with wrong type! RED FLAG

Technique 4: Empty Result Test

For each test, mentally:
1. Return empty/zero/None
2. Should this be valid?
3. Is there a test for it?

Example:
def test_get_users():
    users = get_all_users()  # Returns []
    assert users  # Fails! But is empty valid? UNCLEAR

Automated Red Flag Checkers

Check 1: Weak Assertion Detector

# Scan test file for:
grep -r "assert result$" tests/
grep -r "assert.*is not None" tests/
grep -r "assert True" tests/

Check 2: Mock Overuse Detector

# Count mocks per test:
if mock_count > real_object_count:
    FLAG as "Over-mocking"

Check 3: Missing Negative Test Detector

# For each "test_*_succeeds":
if not exists("test_*_fails"):
    FLAG as "Missing negative test"

Check 4: Vague Name Detector

# Flag patterns:
- test_method_name
- test_class
- test_works
- test_user (without verb)

Response Templates

Template 1: Weak Assertions

**Issue:** Weak Assertions

This test uses weak assertions that would pass even with broken functionality:
- Line X: `assert user` - Only checks if truthy
- Line Y: `assert result is not None` - Only checks existence

**Impact:** High - Won't catch data corruption or logic errors

**Recommendation:**
Replace with specific assertions:

assert user.email == "expected@example.com" assert user.is_active is True assert user.role == "user"

**Action Required:** Block until improved

Template 2: Testing Mocks

**Issue:** Testing Mock Behavior

This test verifies mock behavior instead of real functionality:
- Line X: Mocking the system under test
- Line Y: Asserting that mock was called

**Impact:** Critical - Real code could be completely broken

**Recommendation:**
Test real objects with test dependencies:

Use test database, not mocks

with test_database(): user = service.create_user("test@example.com") assert user.id is not None

**Action Required:** Block merge - Must be rewritten

Template 3: Missing Negative Tests

**Issue:** Missing Failure Cases

Only happy path is tested. Missing tests for:
- Invalid input
- Error conditions
- Edge cases
- Boundary conditions

**Impact:** Medium-High - Failures won't be caught

**Recommendation:**
Add negative tests:
- `test_create_user_with_invalid_email_raises_error`
- `test_create_user_with_duplicate_email_raises_error`
- `test_create_user_with_missing_password_raises_error`

**Action Required:** Request changes before merge

Remember

🚩 If you see a red flag, stop and inspect closely. 🚨 Multiple red flags = immediate block. ✅ No red flags ≠ good test (but it's a start). 🔍 When in doubt, perform mental debugging.

"A red flag ignored today is a production bug tomorrow."