Code Permutation Testing

Code Permutation Testing

Overview

This skill enables systematic testing of code variations, edge cases, and alternative implementations to ensure robustness and comprehensive test coverage. It provides methodologies for input permutation, code path analysis, mutation testing, and implementation exploration.

When to Use

Invoke this skill when:

• Testing functions with complex input domains requiring boundary testing
• Exploring edge cases and corner conditions systematically
• Analyzing code path coverage and branch conditions
• Running mutation testing to verify test suite effectiveness
• Comparing alternative implementations for correctness
• Generating comprehensive test suites for critical code
• Validating error handling and recovery paths

Core Testing Modes

1. Input Permutation Testing

Generate comprehensive test cases covering:

• Boundary values: Min, max, just inside, just outside boundaries
• Edge cases: Empty inputs, null values, extreme sizes
• Type variations: Different numeric types, string encodings, data structures
• Combinatorial testing: Pairwise and n-wise testing of input combinations

Load references/boundary_patterns.md for common boundary testing patterns and strategies.

Use scripts/generate_boundaries.py to automatically generate boundary test cases from function signatures.

2. Code Path Analysis

Analyze all possible execution paths through code:

• Branch coverage: Test all conditional branches
• Loop coverage: Zero, one, many, maximum iterations
• Exception paths: Force error conditions systematically
• State transitions: Test all valid and invalid state changes

Use scripts/analyze_paths.py to identify uncovered code paths and generate test suggestions.

3. Mutation Testing

Verify test suite effectiveness by introducing controlled mutations:

For Rust projects:

# Install cargo-mutants
cargo install cargo-mutants

# Run mutation testing
cargo mutants --no-shuffle --test-timeout 30

# Generate detailed report
cargo mutants --json > mutations.json

For Python projects:

# Install mutmut
pip install mutmut

# Run mutation testing
mutmut run --paths-to-mutate src/

# Show results
mutmut results

Load references/mutation_testing.md for detailed mutation testing guidance and interpretation.

4. Implementation Alternatives

Test different algorithmic approaches:

• Iterative vs Recursive: Compare implementations for stack safety
• Mutable vs Immutable: Test functional and imperative variations
• Synchronous vs Asynchronous: Verify concurrency behavior
• Naive vs Optimized: Validate optimizations maintain correctness

Workflow

Step 1: Analyze Target Code

First, understand the code structure:

• Identify function signatures and parameter types
• Map control flow and decision points
• Note error handling and edge cases
• Document assumptions and invariants

Step 2: Generate Test Matrix

Create comprehensive test coverage:

# Example for a function: divide(a: i32, b: i32) -> Result<i32, Error>

# Boundary tests
test_cases = [
    # Normal cases
    (10, 2, Ok(5)),
    (10, 3, Ok(3)),

    # Boundary values
    (i32::MAX, 1, Ok(i32::MAX)),
    (i32::MIN, 1, Ok(i32::MIN)),
    (i32::MIN, -1, Err(Overflow)),  # Special overflow case

    # Edge cases
    (0, 5, Ok(0)),
    (5, 0, Err(DivisionByZero)),
    (0, 0, Err(DivisionByZero)),

    # Sign variations
    (-10, 2, Ok(-5)),
    (10, -2, Ok(-5)),
    (-10, -2, Ok(5)),
]

Step 3: Execute Permutation Tests

Run the generated test matrix:

• Execute each test case
• Verify expected outcomes
• Check for unexpected behaviors
• Monitor performance characteristics

Step 4: Analyze Coverage

Evaluate test completeness:

• Generate coverage reports
• Identify uncovered paths
• Run mutation testing
• Add tests for gaps

Step 5: Document Results

Create comprehensive test documentation:

• List all tested scenarios
• Document discovered edge cases
• Note performance characteristics
• Highlight critical paths

Language-Specific Guidance

Rust

#[cfg(test)]
mod permutation_tests {
    use super::*;
    use proptest::prelude::*;

    // Property-based testing for automatic permutation
    proptest! {
        #[test]
        fn test_function_properties(
            a in any::<i32>(),
            b in any::<i32>().prop_filter("non-zero", |x| *x != 0)
        ) {
            let result = divide(a, b);
            prop_assert!(result.is_ok());
            prop_assert_eq!(result.unwrap(), a / b);
        }
    }

    // Boundary value testing
    #[test]
    fn test_boundaries() {
        let boundaries = vec![
            i32::MIN, i32::MIN + 1, -1, 0, 1, i32::MAX - 1, i32::MAX
        ];

        for a in &boundaries {
            for b in &boundaries {
                if *b != 0 {
                    let result = divide(*a, *b);
                    // Verify result or expected error
                }
            }
        }
    }
}

Python

import hypothesis.strategies as st
from hypothesis import given, assume
import pytest

# Property-based testing
@given(st.integers(), st.integers())
def test_division_properties(a, b):
    assume(b != 0)  # Precondition
    result = divide(a, b)
    assert result == a // b

# Parametrized boundary testing
@pytest.mark.parametrize("a,b,expected", [
    (10, 2, 5),
    (sys.maxsize, 1, sys.maxsize),
    (-sys.maxsize-1, 1, -sys.maxsize-1),
    (0, 5, 0),
    # Add more boundary cases
])
def test_boundaries(a, b, expected):
    assert divide(a, b) == expected

Resources

Scripts

• scripts/generate_boundaries.py - Analyzes function signatures and generates boundary test cases
• scripts/analyze_paths.py - Identifies code paths and suggests tests for uncovered branches

References

• references/boundary_patterns.md - Common patterns for boundary value analysis and edge case identification
• references/mutation_testing.md - Comprehensive guide to mutation testing tools and result interpretation

Best Practices

1. Start with boundaries: Test limits before normal cases 2. Consider combinations: Use pairwise testing for multiple parameters 3. Test error paths: Ensure error handling is robust 4. Use property-based testing: Let tools generate test cases automatically 5. Verify with mutations: Ensure tests actually catch bugs 6. Document discoveries: Record edge cases for future reference 7. Automate generation: Use scripts to create test matrices 8. Monitor performance: Track execution time across permutations

Reference Documentation for Code Permutation Testing

This is a placeholder for detailed reference documentation. Replace with actual reference content or delete if not needed.

Example real reference docs from other skills:

• product-management/references/communication.md - Comprehensive guide for status updates
• product-management/references/context_building.md - Deep-dive on gathering context
• bigquery/references/ - API references and query examples

When Reference Docs Are Useful

Reference docs are ideal for:

• Comprehensive API documentation
• Detailed workflow guides
• Complex multi-step processes
• Information too lengthy for main SKILL.md
• Content that's only needed for specific use cases

Structure Suggestions

API Reference Example

• Overview
• Authentication
• Endpoints with examples
• Error codes
• Rate limits

Workflow Guide Example

• Prerequisites
• Step-by-step instructions
• Common patterns
• Troubleshooting
• Best practices

Boundary Testing Patterns

Overview

Boundary value analysis (BVA) is a testing technique that focuses on values at the edges of input domains. Most errors occur at boundaries rather than in the middle of valid ranges.

Core Boundary Categories

1. Numeric Boundaries

Integer Types

For any integer type with range [MIN, MAX]:

Test Values:
- MIN - 1 (underflow)
- MIN (minimum valid)
- MIN + 1 (just inside minimum)
- -1 (if signed)
- 0 (zero boundary)
- 1 (smallest positive)
- MAX - 1 (just inside maximum)
- MAX (maximum valid)
- MAX + 1 (overflow)

Language-specific limits:

// Rust
i8:  [-128, 127]
i16: [-32768, 32767]
i32: [-2147483648, 2147483647]
i64: [-9223372036854775808, 9223372036854775807]
u8:  [0, 255]
u16: [0, 65535]
u32: [0, 4294967295]
u64: [0, 18446744073709551615]

# Python
import sys
sys.maxsize  # 9223372036854775807 on 64-bit
-sys.maxsize - 1  # -9223372036854775808

Floating Point

Special values to test:

• 0.0 and -0.0
• Smallest positive: f32::MIN_POSITIVE, f64::MIN_POSITIVE
• Largest finite: f32::MAX, f64::MAX
• Infinity: f32::INFINITY, f32::NEG_INFINITY
• Not-a-Number: f32::NAN
• Subnormal numbers (very small)
• Epsilon boundaries for precision

2. Collection Boundaries

Arrays/Vectors/Lists

Test Sizes:
- Empty collection (size 0)
- Single element (size 1)
- Two elements (size 2) - tests pair logic
- Typical size (3-10 elements)
- Large size (1000+ elements)
- Maximum allowed size
- Size that triggers reallocation

Strings

Test Cases:
- Empty string ""
- Single character "a"
- Single space " "
- Only whitespace "   \t\n"
- Maximum length string
- Unicode boundaries (ASCII vs UTF-8)
- Special characters (\0, \n, \r, \t)
- Mixed scripts (Latin + Cyrillic + Emoji)

3. Date/Time Boundaries

Critical Dates:
- Epoch: 1970-01-01 00:00:00 UTC
- Unix timestamp limits: 2038-01-19 (32-bit)
- Leap years: Feb 28/29
- DST transitions
- Time zone boundaries (+14:00 to -12:00)
- Month boundaries (28, 29, 30, 31 days)
- Year boundaries (Dec 31 -> Jan 1)

4. Loop Boundaries

Iteration Counts:
- 0 iterations (never enters loop)
- 1 iteration (single pass)
- 2 iterations (tests loop continuation)
- N-1 iterations (off-by-one check)
- N iterations (exact expected)
- N+1 iterations (off-by-one check)
- Maximum iterations before timeout

Boundary Testing Strategies

Strategy 1: Two-Value Boundary Testing

Test exactly on the boundary and just outside:

• For upper bound N: test N and N+1
• For lower bound M: test M and M-1

Strategy 2: Three-Value Boundary Testing

Test on, just inside, and just outside:

• For upper bound N: test N-1, N, N+1
• For lower bound M: test M-1, M, M+1

Strategy 3: Domain Matrix Testing

For functions with multiple parameters, create a matrix:

def test_rectangle_area(width, height):
    boundaries_width = [0, 1, 100, 1000]
    boundaries_height = [0, 1, 100, 1000]

    for w in boundaries_width:
        for h in boundaries_height:
            result = calculate_area(w, h)
            # Verify result

Strategy 4: Equivalence Partitioning + Boundaries

Divide input into equivalence classes, then test boundaries:

Age Groups Example:
- Invalid: age < 0 → test -1, -100
- Child: 0 ≤ age < 13 → test 0, 1, 12, 13
- Teen: 13 ≤ age < 20 → test 13, 14, 19, 20
- Adult: 20 ≤ age < 65 → test 20, 21, 64, 65
- Senior: age ≥ 65 → test 65, 66, 100, 150

Common Boundary Bugs

Off-by-One Errors

// Wrong: excludes last element
for i in 0..array.len() - 1 {  // Should be 0..array.len()
    process(array[i]);
}

// Wrong: includes invalid index
for i in 0..=array.len() {  // Should be 0..array.len()
    process(array[i]);  // Panic on last iteration
}

Integer Overflow

// Dangerous
let result = a + b;  // Can overflow

// Safe
let result = a.checked_add(b).ok_or(Error::Overflow)?;

Floating Point Comparison

// Wrong
if float_a == float_b {  // Exact comparison fails

// Right
if (float_a - float_b).abs() < EPSILON {

Empty Collection Handling

# Wrong
def average(numbers):
    return sum(numbers) / len(numbers)  # Division by zero

# Right
def average(numbers):
    if not numbers:
        return None
    return sum(numbers) / len(numbers)

Boundary Test Checklist

For Any Function

• [ ] Identify all input parameters
• [ ] Determine valid ranges for each parameter
• [ ] List boundary values for each range
• [ ] Test combinations of boundaries (if multiple params)
• [ ] Verify error handling at invalid boundaries
• [ ] Check for overflow/underflow conditions
• [ ] Test special values (null, empty, zero)

For Numeric Functions

• [ ] Test with 0, 1, -1
• [ ] Test MIN and MAX values for type
• [ ] Test just inside and outside valid range
• [ ] Test overflow scenarios
• [ ] Test precision boundaries (for floats)
• [ ] Test NaN, Infinity (for floats)

For String Functions

• [ ] Empty string
• [ ] Single character
• [ ] Maximum length
• [ ] Special characters (\0, \n, etc.)
• [ ] Unicode edge cases
• [ ] Whitespace-only strings

For Collection Functions

• [ ] Empty collection
• [ ] Single element
• [ ] Two elements (pair operations)
• [ ] Maximum size
• [ ] Duplicate elements
• [ ] Sorted vs unsorted

For Time/Date Functions

• [ ] Epoch boundaries
• [ ] Leap years
• [ ] DST transitions
• [ ] Time zone boundaries
• [ ] Month/year transitions
• [ ] Invalid dates (Feb 30, etc.)

Automated Boundary Generation

Python Example

def generate_integer_boundaries(min_val, max_val):
    """Generate boundary test values for integer range."""
    boundaries = set()

    # Add boundary values
    boundaries.add(min_val - 1)  # Underflow
    boundaries.add(min_val)      # Minimum
    boundaries.add(min_val + 1)  # Just inside min

    # Add zero if in range
    if min_val <= 0 <= max_val:
        boundaries.add(-1)
        boundaries.add(0)
        boundaries.add(1)

    boundaries.add(max_val - 1)  # Just inside max
    boundaries.add(max_val)      # Maximum
    boundaries.add(max_val + 1)  # Overflow

    return sorted(boundaries)

# Usage
test_values = generate_integer_boundaries(-100, 100)

Rust Example

fn generate_boundaries<T>(min: T, max: T) -> Vec<T>
where
    T: Copy + PartialOrd + std::ops::Add<Output = T> + std::ops::Sub<Output = T> + From<i8>,
{
    let mut boundaries = Vec::new();
    let one = T::from(1);

    // Boundary values
    boundaries.push(min);
    boundaries.push(max);

    // Just inside boundaries (if possible)
    if min < max {
        boundaries.push(min + one);
        boundaries.push(max - one);
    }

    boundaries
}

Best Practices

1. Document boundary assumptions: Make valid ranges explicit 2. Test boundaries first: They're most likely to fail 3. Combine boundaries: Test multiple parameters at boundaries simultaneously 4. Use property-based testing: Tools like QuickCheck/Hypothesis can generate boundaries 5. Consider domain knowledge: Business rules may create additional boundaries 6. Test boundary interactions: How do boundaries in one field affect another? 7. Automate generation: Use scripts to generate boundary test cases 8. Review historical bugs: Past boundary bugs indicate testing gaps

Mutation Testing Guide

What is Mutation Testing?

Mutation testing is a technique to evaluate the quality of your test suite by introducing small changes (mutations) to your code and checking if tests catch these changes. If a mutation survives (tests still pass), it indicates a gap in test coverage.

Core Concepts

Mutants

A mutant is a version of your code with one small, deliberate change. Common mutations include:

• Arithmetic: + → -, * → /
• Relational: < → <=, == → !=
• Logical: && → ||, ! → (removed)
• Constant: 0 → 1, true → false
• Statement: Remove statements or function calls
• Return: Change or remove return values

Mutation Score

Mutation Score = (Killed Mutants / Total Mutants) × 100%

• Killed: Test suite detected the mutation (tests failed)
• Survived: Tests passed despite mutation (coverage gap)
• Timeout: Mutation caused infinite loop
• Unviable: Mutation created invalid code

Target: 80%+ mutation score for critical code

Tool-Specific Guides

Rust: cargo-mutants

Installation

cargo install cargo-mutants

Basic Usage

# Run mutation testing on entire project
cargo mutants

# Test specific package
cargo mutants -p my_package

# Test specific file
cargo mutants --file src/lib.rs

# Generate detailed JSON report
cargo mutants --json > mutants.json

# Run with timeout per test
cargo mutants --test-timeout 30

# Run in parallel (faster)
cargo mutants --jobs 4

Configuration

Create .cargo/mutants.toml:

# Exclude files from mutation
exclude_dirs = ["tests", "benches"]
exclude_globs = ["**/generated.rs"]

# Customize timeout
timeout_multiplier = 1.5

# Error on surviving mutants (CI)
error_on_survived = true

Interpreting Results

SURVIVED src/calculator.rs:45: replace + with - in add()
    → Your tests don't verify the addition operation properly

KILLED src/validator.rs:23: replace >= with > in check_age()
    → Good! Tests caught this boundary change

TIMEOUT src/loop.rs:67: replace < with <= in while loop
    → Mutation likely caused infinite loop

Python: mutmut

Installation

pip install mutmut

Basic Usage

# Initialize mutmut
mutmut run --paths-to-mutate src/ --tests-dir tests/

# Show results summary
mutmut results

# Show surviving mutants
mutmut show

# Show specific mutant
mutmut show 47

# Apply mutant to see the change
mutmut apply 47

# Generate HTML report
mutmut html

Configuration

Create .mutmut.yml:

paths_to_mutate:
  - src/

tests_dir: tests/

runner: python -m pytest

dict_synonyms:
  - dict
  - OrderedDict

exclude_patterns:
  - test_*
  - */migrations/*

Common Python Mutations

# Original
def calculate_discount(price, rate):
    if rate > 0.5:
        rate = 0.5
    return price * (1 - rate)

# Mutations:
# 1. rate > 0.5 → rate >= 0.5
# 2. rate > 0.5 → rate < 0.5
# 3. rate = 0.5 → rate = 0.4
# 4. return price * (1 - rate) → return price * (1 + rate)

JavaScript/TypeScript: Stryker

Installation

npm install --save-dev @stryker-mutator/core
npx stryker init  # Interactive setup

Configuration

stryker.config.js:

module.exports = {
  mutate: ['src/**/*.js', '!src/**/*.test.js'],
  testRunner: 'jest',
  coverageAnalysis: 'perTest',
  thresholds: { high: 80, low: 60, break: 50 },
  mutator: {
    excludedMutations: ['StringLiteral'],
  },
};

Running Stryker

# Run mutation testing
npx stryker run

# Generate HTML report
npx stryker run --reporters html

# Run specific mutators
npx stryker run --mutate 'src/utils.js'

Go: go-mutesting

Installation

go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest

Usage

# Run on package
go-mutesting ./...

# With specific test timeout
go-mutesting --timeout 10s ./...

# Generate report
go-mutesting --report mutations.json ./...

Mutation Patterns and Strategies

High-Value Mutations

Focus on mutations that reveal important test gaps:

1. Boundary Mutations: < → <=, > → >= 2. Off-by-One: i → i+1, len-1 → len 3. Null/Empty Checks: Remove null checks 4. Error Handling: Change error returns 5. Business Logic: Invert conditions in domain rules

Mutation Testing Workflow

graph TD
    A[Run Mutation Testing] --> B{Mutants Survived?}
    B -->|Yes| C[Analyze Survivors]
    B -->|No| D[Good Coverage!]
    C --> E[Write Missing Tests]
    E --> F[Re-run Mutations]
    F --> B

Analyzing Surviving Mutants

For each surviving mutant:

1. Is it testable? Some mutations might not affect observable behavior 2. Is it important? Focus on business logic over implementation details 3. What test is missing? Write a test that would kill this mutant 4. Is the code necessary? Surviving mutants might indicate dead code

Common Mutation Categories

Arithmetic Operator Mutations

// Original
let result = a + b;

// Mutations
let result = a - b;  // AOR (Arithmetic Operator Replacement)
let result = a * b;  // AOR
let result = a / b;  // AOR
let result = a % b;  // AOR

Conditional Boundary Mutations

// Original
if x > 10 {

// Mutations
if x >= 10 {  // CBM (Conditional Boundary Mutation)
if x < 10 {   // ROR (Relational Operator Replacement)
if x == 10 {  // ROR
if true {     // COR (Conditional Operator Replacement)
if false {    // COR

Logical Operator Mutations

// Original
if a && b {

// Mutations
if a || b {   // LOR (Logical Operator Replacement)
if !a && b {  // LCR (Logical Connector Replacement)
if a {        // Statement Deletion

Return Value Mutations

// Original
return Some(value);

// Mutations
return None;           // RVM (Return Value Mutation)
return Some(value + 1); // RVM
// (return deleted)     // Statement Deletion

Best Practices

1. Start Small

Begin with critical modules:

• Core business logic
• Security-sensitive code
• Frequently modified code
• Code with history of bugs

2. Set Realistic Goals

• 100% mutation coverage is rarely achievable or necessary
• Aim for 80%+ on critical code
• 60%+ on general application code
• Lower thresholds for UI/presentation code

3. Handle Equivalent Mutants

Some mutations don't change behavior:

// Original
let x = a * 0;

// Equivalent mutant (same result)
let x = a & 0;

Mark these as ignored in your tool configuration.

4. Optimize Performance

Mutation testing is slow. Speed it up:

• Run incrementally (only changed files)
• Use parallel execution
• Set reasonable timeouts
• Cache unchanged results
• Run in CI on pull requests only

5. Integrate with CI/CD

# GitHub Actions example
- name: Run Mutation Tests
  run: |
    cargo mutants --error-on-survived --json > mutations.json
  continue-on-error: true

- name: Comment PR
  uses: actions/github-script@v6
  with:
    script: |
      const mutations = require('./mutations.json');
      // Post summary to PR

Interpreting Results

Good Mutations to Kill

// This mutation should be killed
fn calculate_tax(amount: f64) -> f64 {
    amount * 0.15  // → amount * 0.14
}
// Test should verify exact tax calculation

Acceptable Survivors

// This mutation might survive acceptably
fn log_error(msg: &str) {
    eprintln!("Error: {}", msg);  // → println!
}
// If logging isn't tested, that might be OK

Critical Survivors

// This MUST be killed
fn check_authorization(user: &User) -> bool {
    user.is_admin  // → !user.is_admin
}
// Security logic must be thoroughly tested

Mutation Testing Anti-Patterns

1. Testing Implementation, Not Behavior

// Bad: Tests internal details
#[test]
fn test_uses_hashmap() {
    let cache = Cache::new();
    assert!(cache.internal_map.is_empty());  // Too specific
}

// Good: Tests behavior
#[test]
fn test_cache_starts_empty() {
    let cache = Cache::new();
    assert_eq!(cache.get("key"), None);
}

2. Mutation Score Gaming

Don't write useless tests just to kill mutants:

// Bad: Pointless test to kill mutant
#[test]
fn test_constant() {
    assert_eq!(BUFFER_SIZE, 1024);  // Tests a constant
}

3. Ignoring Timeout Mutants

Timeouts often indicate missing infinite loop tests:

// If this mutation causes timeout, add a test
while i < len {  // → while i <= len
    // ... process
    i += 1;
}

Quick Reference

Mutation Testing Checklist

• [ ] Install mutation testing tool for your language
• [ ] Configure exclusions (tests, generated code)
• [ ] Run on critical modules first
• [ ] Analyze surviving mutants
• [ ] Write tests to kill important survivors
• [ ] Set mutation score thresholds
• [ ] Integrate into CI pipeline
• [ ] Document equivalent mutants
• [ ] Review results regularly

Recommended Thresholds by Code Type