Silent Degradation Audit

Silent Degradation Audit Skill

Overview

Production-ready skill for detecting silent degradation across codebases. Uses multi-wave audit system with 6 specialized category agents, multi-agent validation panel, and convergence detection. Battle-tested on CyberGym codebase (~250 bugs found).

When to Use This Skill

Use this skill when:

• Code has reliability issues but unclear where
• Systems fail silently without operator visibility
• Error handling exists but effectiveness unknown
• Need comprehensive audit across multiple failure modes
• Preparing for production deployment
• Post-mortem analysis after silent failures

Don't use for:

• Code style or formatting issues (use linters)
• Performance optimization (use profilers)
• Security vulnerabilities (use security scanners)
• Simple one-off code reviews (use /analyze)

Key Features

Multi-Wave Progressive Audit

• Wave 1: Broad scan, finds obvious issues (40-50% of total)
• Wave 2-3: Deeper analysis, finds hidden issues (30-40%)
• Wave 4-6: Edge cases and subtleties (10-20%)
• Convergence: Stops when < 10 new findings or < 5% of Wave 1

6 Category Agents

1. Dependency Failures (Category A): "What happens when X is down?" 2. Config Errors (Category B): "What happens when config is wrong?" 3. Background Work (Category C): "What happens when background work fails?" 4. Test Effectiveness (Category D): "Do tests actually detect failures?" 5. Operator Visibility (Category E): "Is the error visible to operators?" 6. Functional Stubs (Category F): "Does this code actually do what its name says?"

Multi-Agent Validation Panel

• 3 agents review findings: Security, Architect, Builder
• 2/3 consensus required to validate finding
• Prevents false positives and unnecessary changes
• Tracks strong vs weak consensus

Language-Agnostic

Supports 9 languages with language-specific patterns:

• Python, JavaScript, TypeScript
• Rust, Go, Java, C#
• Ruby, PHP

Integration Modes

Standalone Invocation

Direct skill invocation for focused audit:

/silent-degradation-audit path/to/codebase

Sub-Loop in Quality Audit Workflow

Integrated as Phase 2 of quality-audit-workflow:

quality-audit-workflow calls silent-degradation-audit
→ Returns findings to quality workflow
→ Quality workflow applies fixes
→ Continues to next phase

Usage

Basic Usage

# Audit entire codebase
/silent-degradation-audit .

# Audit specific directory
/silent-degradation-audit ./src

# With custom exclusions
/silent-degradation-audit . --exclusions .my-exclusions.json

Configuration

Create .silent-degradation-config.json in codebase root:

{
  "convergence": {
    "absolute_threshold": 10,
    "relative_threshold": 0.05
  },
  "max_waves": 6,
  "exclusions": {
    "patterns": ["*.test.js", "test_*.py", "**/__tests__/**"]
  },
  "categories": {
    "enabled": [
      "dependency-failures",
      "config-errors",
      "background-work",
      "test-effectiveness",
      "operator-visibility",
      "functional-stubs"
    ]
  }
}

Exclusion Lists

Global Exclusions

Edit ~/.amplihack/.claude/skills/silent-degradation-audit/exclusions-global.json:

[
  {
    "pattern": "*.test.*",
    "reason": "Test files excluded from production audits",
    "category": "*"
  },
  {
    "pattern": "**/vendor/**",
    "reason": "Third-party code",
    "category": "*"
  }
]

Repository-Specific Exclusions

Create .silent-degradation-exclusions.json in repository root:

[
  {
    "pattern": "src/legacy/*.py",
    "reason": "Legacy code being replaced",
    "category": "*",
    "wave": 1
  },
  {
    "pattern": "api/endpoints.py:42",
    "reason": "Empty dict is valid API response",
    "category": "functional-stubs",
    "type": "exact"
  }
]

Output

Report Format

Generates .silent-degradation-report.md:

# Silent Degradation Audit Report

## Summary

- **Total Waves**: 4
- **Total Findings**: 137
- **Converged**: Yes
- **Convergence Ratio**: 4.2%

## Convergence Progress

Wave 1: ██████████████████████████████████████████████████ 120
Wave 2: ███████████████████████████ 65 (54.2% of Wave 1)
Wave 3: ████████ 18 (15.0% of Wave 1)
Wave 4: ██ 5 (4.2% of Wave 1)

Status: ✓ CONVERGED
Reason: Relative threshold met: 4.2% < 5.0%

## Findings by Category

### dependency-failures (42 findings)

- High: 15
- Medium: 20
- Low: 7

[... continues for all 6 categories ...]

Findings Format

Generates .silent-degradation-findings.json:

[
  {
    "id": "dep-001",
    "category": "dependency-failures",
    "severity": "high",
    "file": "src/payments.py",
    "line": 89,
    "description": "Payment API failure silently falls back to mock",
    "impact": "Production system using mock payments, no real charges",
    "visibility": "None - no logs or metrics",
    "recommendation": "Add explicit failure logging and metric, or fail fast",
    "wave": 1,
    "validation": {
      "result": "VALIDATED",
      "consensus": "strong",
      "votes": {
        "security": "APPROVE",
        "architect": "APPROVE",
        "builder": "APPROVE"
      }
    }
  },
  ...
]

Workflow Details

Phase 1: Initialization

1. Create convergence tracker with thresholds 2. Initialize exclusion manager 3. Set up audit state

Phase 2: Language Detection

1. Scan codebase for file extensions 2. Identify languages (> 5 files or > 5% threshold) 3. Load language-specific patterns

Phase 3: Load Exclusions

1. Load global exclusions from skill directory 2. Load repository-specific exclusions 3. Merge into single exclusion list

Phase 4: Wave Loop

For each wave (until convergence):

1. Category Analysis (6 agents in parallel)

• Each agent scans for category-specific issues
• Uses language-specific patterns
• Excludes previous findings

2. Validation Panel (3 agents in parallel)

• Security agent reviews security implications
• Architect agent reviews design impact
• Builder agent reviews implementation feasibility

3. Vote Tallying

• Require 2/3 consensus (APPROVE)
• Track strong vs weak consensus
• Flag inconclusive for human review

4. Exclusion Filtering

• Apply global and repo-specific exclusions
• Filter out duplicates

5. State Update

• Add new findings to total
• Record wave metrics

6. Convergence Check

• Absolute: < 10 new findings
• Relative: < 5% of Wave 1 findings
• Break if converged

Phase 5: Report Generation

1. Generate convergence plot 2. Calculate metrics summary 3. Categorize findings by type and severity 4. Write markdown report 5. Write JSON findings

Architecture

Directory Structure

.claude/skills/silent-degradation-audit/
├── SKILL.md                    # This file
├── reference.md                # Detailed patterns and examples
├── examples.md                 # Usage examples
├── patterns.md                 # Language-specific patterns
├── README.md                   # Quick start
├── category_agents/            # 6 category agent definitions
│   ├── dependency-failures.md
│   ├── config-errors.md
│   ├── background-work.md
│   ├── test-effectiveness.md
│   ├── operator-visibility.md
│   └── functional-stubs.md
├── validation_panel/           # Validation panel specs
│   ├── panel-spec.md
│   └── voting-rules.md
├── recipe/                     # Recipe-based workflow
│   └── audit-workflow.yaml
└── tools/                      # Python utilities
    ├── exclusion_manager.py
    ├── language_detector.py
    ├── convergence_tracker.py
    └── __init__.py

Component Responsibilities

Category Agents:

• Scan codebase for category-specific issues
• Use language-specific patterns
• Produce findings with severity, impact, recommendation

Validation Panel:

• Review findings from multiple perspectives
• Vote APPROVE/REJECT/ABSTAIN
• Require 2/3 consensus

Convergence Tracker:

• Track findings per wave
• Calculate convergence metrics
• Determine when to stop

Exclusion Manager:

• Load and merge exclusion lists
• Filter findings against patterns
• Add new exclusions

Language Detector:

• Identify languages in codebase
• Load language-specific patterns
• Support 9 languages

Best Practices

Running First Audit

1. Start with small scope: Audit single service/module first 2. Review Wave 1 carefully: Establishes baseline 3. Tune exclusions: Add false positives to exclusion list 4. Verify fixes: Test fixes before applying broadly

Exclusion Management

When to add exclusions:

• False positives (finding not actually an issue)
• Intentional design (behavior is correct as-is)
• Legacy code (not worth fixing right now)
• Third-party code (can't modify)

When NOT to add exclusions:

• Real issues you don't want to fix
• Issues without time to fix now
• Issues that seem hard

Better approach: Fix real issues, prioritize by severity.

Validation Tuning

If too many false positives:

• Review validation panel prompts
• Increase consensus threshold (require unanimous)
• Add category-specific validation rules

If missing real issues:

• Review category agent patterns
• Add language-specific patterns
• Decrease consensus threshold (1/3 approval)

Wave Management

Typical wave characteristics:

• Wave 1: 40-50% of findings (obvious issues)
• Wave 2: 25-30% (deeper issues)
• Wave 3: 15-20% (subtle issues)
• Wave 4+: < 10% each (edge cases)

If waves not converging:

• Check for duplicate findings (exclusion not working)
• Review category agent overlap (agents finding same things)
• Consider lowering convergence threshold

Metrics and Monitoring

Success Metrics

Track these over time:

Audit Success:
- Convergence reached: Yes/No
- Waves to convergence: 4 (target: 3-5)
- Total findings: 137 (varies by codebase)
- Validation rate: 75% (target: 60-80%)

Finding Distribution:
- High severity: 15% (target: < 20%)
- Medium severity: 45% (target: 40-60%)
- Low severity: 40% (target: 30-50%)

Panel Effectiveness:
- Strong consensus: 60% (target: > 50%)
- Weak consensus: 30% (target: 20-40%)
- Inconclusive: 10% (target: < 10%)
- Abstention rate: 5% (target: < 10%)

Quality Indicators

Healthy audit:

• Converges in 3-5 waves
• Validation rate 60-80%
• Strong consensus > 50%
• Abstention rate < 10%

Warning signs:

• Doesn't converge after 6 waves (agents finding same things)
• Validation rate > 95% (rubber stamping)
• Validation rate < 40% (too strict)
• Inconclusive rate > 20% (poor context)

Troubleshooting

"Audit not converging"

Symptoms: Reaches max waves without convergence

Causes:

• Category agents finding duplicate issues
• Exclusion filtering not working
• Convergence threshold too tight

Solutions:

1. Review findings for duplicates 2. Check exclusion patterns are matching 3. Increase relative threshold to 10% 4. Reduce max waves to 5

"Too many false positives"

Symptoms: Validation rate > 95%, many non-issues

Causes:

• Category agents too aggressive
• Validation panel too permissive
• Patterns not tuned for codebase

Solutions:

1. Review category agent patterns 2. Add exclusions for false positive patterns 3. Require unanimous validation (3/3) 4. Tune language-specific patterns

"Missing real issues"

Symptoms: Known issues not in findings

Causes:

• Category agent gaps
• Exclusion too broad
• Validation panel too strict

Solutions:

1. Check if issue matches any category 2. Review exclusion list for overly broad patterns 3. Lower consensus threshold to 1/3 4. Add specific patterns for missed issues

"Validation panel abstaining"

Symptoms: High abstention rate (> 20%)

Causes:

• Insufficient context in findings
• Agent prompts unclear
• Findings outside agent expertise

Solutions:

1. Include more code context in findings 2. Review and improve agent prompts 3. Add fourth "generalist" agent 4. Improve finding descriptions

Advanced Configuration

Custom Category Agents

Create custom category agent in category_agents/custom.md:

# Category Custom: My Special Cases

## Core Question

"What happens when [specific scenario]?"

## Detection Focus

[Patterns to detect...]

## Language-Specific Patterns

[Language examples...]

Then enable in config:

{
  "categories": {
    "enabled": [
      "dependency-failures",
      "config-errors",
      "background-work",
      "test-effectiveness",
      "operator-visibility",
      "functional-stubs",
      "custom"
    ]
  }
}

Custom Validation Panel

Override validation panel with different agents:

# In recipe/audit-workflow.yaml
validation_panel:
  agents:
    - security
    - architect
    - builder
    - domain-expert # Add domain-specific agent

  consensus:
    required: 0.75 # Require 3/4 approval

Staged Rollout

Audit codebase incrementally:

# Phase 1: Critical services only
/silent-degradation-audit ./services/payments ./services/auth

# Phase 2: All services
/silent-degradation-audit ./services

# Phase 3: Full codebase
/silent-degradation-audit .

See Also

• reference.md - Detailed technical reference
• examples.md - Real-world usage examples
• patterns.md - Language-specific degradation patterns
• README.md - Quick start guide
• category_agents/ - Individual category agent documentation
• validation_panel/ - Validation panel specifications

Changelog

Version 1.0.0 (2025-02-24)

• Initial release
• 6 category agents (A-F)
• Multi-agent validation panel (2/3 consensus)
• Convergence detection (dual thresholds)
• Language-agnostic (9 languages)
• Battle-tested on CyberGym (~250 bugs)
• Integration modes: standalone + sub-loop

Category C: Background Work Agent

Role

Specialized agent for detecting silent degradation in asynchronous, background, and scheduled work. Asks "What happens when background work fails?"

Core Question

"What happens when background work fails?"

Where "background work" includes:

• Async tasks and futures
• Message queue consumers
• Cron jobs and scheduled tasks
• Background threads and workers
• Event handlers and callbacks
• Webhook receivers

Detection Focus

Async Task Failures

1. Fire-and-Forget

• Tasks launched without awaiting result
• Exceptions in async context not caught
• No retry or error handling

2. Promise/Future Abandonment

• Promises created but never awaited
• Futures dropped without checking result
• Async operations assumed to succeed

3. Callback Failures

• Exception in callback handler ignored
• Callback registration failures silent
• No timeout on callback execution

Queue Processing Failures

1. Message Loss

• Message acknowledged before processing
• Processing failure doesn't requeue
• Dead letter queue silently accumulating

2. Consumer Failures

• Consumer crashes without alerting
• Consumer stalls (no messages processed)
• Poison messages block queue

3. Batch Processing

• Partial batch success treated as full success
• Individual item failures not tracked
• No visibility into batch progress

Scheduled Work Failures

1. Cron Job Failures

• Job fails but cron continues
• Job never runs (bad schedule)
• Job runs but takes no action

2. Timer-Based Work

• Timer fires but handler fails
• Timer stops firing (uncaught exception)
• Timer drift (expected hourly, actually every 90 minutes)

3. Event Polling

• Polling loop stops but system continues
• Events processed but handlers fail
• Event backlog growing without visibility

Language-Specific Patterns

Python

# Anti-pattern: Fire-and-forget async
async def process_data():
    asyncio.create_task(expensive_operation())  # No await, no error handling

# Anti-pattern: Thread exception ignored
def background_worker():
    try:
        while True:
            process_item()
    except Exception:
        pass  # Thread dies silently

# Anti-pattern: Celery task failure silent
@app.task
def process_order(order_id):
    # If this fails, no one knows unless explicitly checking
    process_payment(order_id)

JavaScript/TypeScript

// Anti-pattern: Promise not awaited
async function handler() {
  processInBackground(); // Returns promise, not awaited
}

// Anti-pattern: Catch without logging
queue.on("message", async (msg) => {
  try {
    await process(msg);
  } catch {
    // Silent failure, message lost
  }
});

// Anti-pattern: Event handler failure ignored
emitter.on("event", (data) => {
  dangerousOperation(data); // Throws, event emitter continues
});

Rust

// Anti-pattern: Task spawned without join
tokio::spawn(async {
    dangerous_operation().await  // Panic not visible
});

// Anti-pattern: Background task result ignored
let handle = thread::spawn(|| {
    process_forever()  // Error not checked
});
// handle never joined

Go

// Anti-pattern: Goroutine panic not recovered
go func() {
    processMessages()  // Panic kills goroutine, silent
}()

// Anti-pattern: Error channel not read
errCh := make(chan error)
go func() {
    errCh <- processData()  // If no reader, goroutine blocks
}()

Java

// Anti-pattern: ExecutorService exception ignored
executor.submit(() -> {
    processItem();  // Exception caught by Future, never checked
});

// Anti-pattern: @Async method failure silent
@Async
public void processOrder(Order order) {
    // Exception here is logged but not surfaced
    paymentService.charge(order);
}

C#

```c# // Anti-pattern: Fire-and-forget Task Task.Run(() => ProcessData()); // Exception not observed

// Anti-pattern: Background service failure hidden protected override async Task ExecuteAsync(CancellationToken token) { try { await ProcessForever(token); } catch { // Service stops, no one notified } }

## Detection Strategy

### Phase 1: Async Pattern Analysis

- Find async functions that return unawaited tasks
- Check for fire-and-forget patterns
- Identify callback registrations without error handlers

### Phase 2: Queue Integration Analysis

- Locate message queue consumers
- Check acknowledgment vs. processing order
- Verify dead letter queue monitoring

### Phase 3: Scheduled Work Analysis

- Find cron job definitions
- Check timer and polling implementations
- Verify health checks for scheduled work

### Phase 4: Error Propagation Analysis

- Check if background errors are logged
- Verify metrics for background task success/failure
- Identify alerting for background work issues

## Validation Criteria

A finding is valid if:

1. **Failure is silent**: Background work fails with no immediate visibility
2. **No retry or recovery**: Failure is permanent with no remediation
3. **No monitoring**: No metrics, logs, or alerts for the failure
4. **Impact unclear**: Can't determine if background work is healthy

## Output Format

{ "category": "background-work", "severity": "high|medium|low", "file": "path/to/worker.py", "line": 67, "description": "Celery task failure not monitored or retried", "impact": "Order processing silently fails, customer never charged", "visibility": "Task shows failed in Celery but no alert", "recommendation": "Add metrics for task success/failure rate and alert on threshold" }

## Integration Points

- **With operator-visibility**: Background failures must be visible
- **With test-effectiveness**: Tests should verify background work failures
- **With dependency-failures**: Background work often depends on external services

## Common Exclusions

- Best-effort background work (explicitly documented as optional)
- Background work with explicit monitoring (metrics + alerts)
- Fire-and-forget patterns with clear documentation

## Battle-Tested Insights (from CyberGym ~250 bug audit)

1. **Most common**: Fire-and-forget async without error handling (45%)
2. **Most dangerous**: Queue consumers dying silently (30%)
3. **Most overlooked**: Cron jobs failing without alerting (15%)
4. **Most fixable**: Add try/catch with logging in background handlers (85% quick wins)

## Red Flags

- `asyncio.create_task()` without await or exception handler
- Thread/goroutine spawned without join/panic recovery
- Message acknowledged before processing complete
- Cron job with no health check or monitoring
- Executor service submit() without future check
- Background task with no success/failure metrics

Category B: Config Errors Agent

Role

Specialized agent for detecting silent degradation when configuration is wrong, missing, or ignored. Asks "What happens when config is wrong?"

Core Question

"What happens when config is wrong?"

Where "wrong" includes:

• Missing required configuration values
• Invalid/malformed configuration values
• Configuration values that silently fall back to defaults
• Environment-specific config applied incorrectly

Detection Focus

Missing Configuration

1. Environment Variables

• os.getenv("API_KEY", "default_key") - Silent default dangerous
• Missing required env vars using fallback values
• No validation of env var presence

2. Configuration Files

• Missing config file silently uses defaults
• Partial config file with missing sections
• Invalid config format (JSON/YAML parse errors) caught but ignored

3. Runtime Configuration

• Feature flags defaulting to enabled/disabled without visibility
• A/B test assignments falling back silently
• Regional settings using wrong defaults

Invalid Configuration

1. Type Mismatches

• String where integer expected, converted silently
• Boolean flags parsed incorrectly ("false" string vs false boolean)
• Array vs. single value confusion

2. Value Range Violations

• Port numbers outside valid range
• Percentages > 100 or < 0
• Negative timeouts silently clamped

3. Format Violations

• Invalid URLs parsed with defaults
• Malformed connection strings
• Bad regex patterns that fail to compile

Silent Defaults

1. Dangerous Defaults

• Production system using development defaults
• Security settings defaulting to permissive
• Resource limits defaulting to unbounded

2. Environment Confusion

• Production config applied in staging
• Staging secrets used in production
• Local development config in CI/CD

3. No Validation on Load

• Config loaded but never validated
• Validation errors caught but ignored
• Invalid config causes failures later, not at load time

Language-Specific Patterns

Python

# Anti-pattern: Dangerous default
API_KEY = os.getenv("API_KEY", "default_insecure_key")

# Anti-pattern: Silent config file failure
try:
    config = json.load(open("config.json"))
except FileNotFoundError:
    config = {}  # Empty config, no error

# Anti-pattern: Type coercion hiding errors
timeout = int(os.getenv("TIMEOUT", 30))  # "invalid" becomes error, but "30.5" becomes 30

JavaScript/TypeScript

// Anti-pattern: Missing env var silent default
const apiUrl = process.env.API_URL || "http://localhost:3000";

// Anti-pattern: Config parse error ignored
let config;
try {
  config = JSON.parse(fs.readFileSync("config.json"));
} catch {
  config = {}; // Silent fallback
}

// Anti-pattern: No type validation
const port = parseInt(process.env.PORT) || 3000; // "abc" becomes NaN, then 3000

Rust

// Anti-pattern: Config error silently defaulted
let config = Config::from_file("app.toml")
    .unwrap_or_default();  // No indication config file failed

// Anti-pattern: Environment variable parsing ignores errors
let timeout = env::var("TIMEOUT")
    .ok()
    .and_then(|s| s.parse().ok())
    .unwrap_or(30);  // Multiple failure points, all silent

Go

// Anti-pattern: Missing env var with dangerous default
apiKey := os.Getenv("API_KEY")
if apiKey == "" {
    apiKey = "default"  // Silent insecure default  // pragma: allowlist secret
}

// Anti-pattern: Config file error ignored
config, err := LoadConfig("app.yaml")
if err != nil {
    config = &Config{}  // Empty config, no visibility
}

Java

// Anti-pattern: Property missing uses default
String apiKey = System.getProperty("api.key", "default");

// Anti-pattern: Config exception caught and ignored
Properties config = new Properties();
try {
    config.load(new FileInputStream("app.properties"));
} catch (IOException e) {
    // Empty config, silent failure
}

C#

```c# // Anti-pattern: Config section missing returns null, used anyway var apiKey = Configuration["ApiKey"] ?? "default";

// Anti-pattern: Config binding errors ignored services.Configure<AppSettings>(Configuration.GetSection("AppSettings")); // No validation if section missing or malformed

## Detection Strategy

### Phase 1: Environment Variable Analysis

- Find all `getenv()` calls with defaults
- Check for required env vars without defaults
- Identify security-sensitive config with defaults

### Phase 2: Config File Analysis

- Locate config file loading code
- Check error handling for missing/invalid files
- Verify config validation after load

### Phase 3: Default Value Analysis

- Find dangerous defaults (credentials, URLs, ports)
- Check for environment-specific defaults
- Identify resource limit defaults

### Phase 4: Validation Gap Analysis

- Check if config is validated after load
- Verify type checking and range validation
- Identify config used before validation

## Validation Criteria

A finding is valid if:

1. **Silent failure**: Config error occurs but system continues
2. **Dangerous behavior**: System runs with insecure/incorrect config
3. **No visibility**: No log/alert when config is wrong
4. **Operator impact**: Operators can't tell config is wrong without deep inspection

## Output Format

{ "category": "config-errors", "severity": "high|medium|low", "file": "path/to/config.py", "line": 23, "description": "API_KEY environment variable defaults to insecure value", "impact": "Production system using development credentials", "visibility": "None - no warning when env var missing", "recommendation": "Fail fast if API_KEY not set, or log warning with metric" }

## Integration Points

- **With dependency-failures**: Config errors often look like dependency failures
- **With operator-visibility**: Config problems must be visible to operators
- **With test-effectiveness**: Tests should verify behavior with bad config

## Common Exclusions

- Development-only config with explicit dev defaults
- Optional features with documented fallback behavior
- Config that fails fast on startup (not silent)

## Battle-Tested Insights (from CyberGym ~250 bug audit)

1. **Most common**: Missing env vars with silent defaults (35% of findings)
2. **Most dangerous**: Security config defaulting to permissive (30%)
3. **Most overlooked**: Type coercion hiding validation errors (20%)
4. **Most fixable**: Add explicit validation on config load (75% quick wins)

## Red Flags

- Any config with "default" in the variable name
- Credentials or secrets with fallback values
- Port numbers or URLs with hardcoded defaults
- Config loaded in try/except with empty fallback
- No validation after config loading

Category A: Dependency Failures Agent

Role

Specialized agent for detecting silent degradation when dependencies fail or are unavailable. Asks "What happens when X is down?"

Core Question

"What happens when X is down?"

Where X is:

• External services (APIs, databases, message queues)
• Internal modules or packages
• System dependencies (network, filesystem, OS features)
• Third-party libraries

Detection Focus

Import-Time Failures

1. Missing Module Handling

• try: import optional_lib except ImportError: optional_lib = None
• Conditional feature availability based on imports
• Missing type checking when library absent

2. Version Conflicts

• Incompatible API usage with fallback to older version
• Feature detection vs. version checking
• Silent feature degradation with version mismatch

3. Transitive Dependencies

• Deep dependency failures masked by multiple layers
• Optional sub-dependencies silently missing

Runtime Failures

1. Connection Failures

• API endpoints unreachable
• Database connection pools exhausted
• Timeout handling that swallows errors

2. Fallback Chains

• Primary → Secondary → Tertiary fallbacks without visibility
• Each fallback silently accepting degraded functionality
• No indication which fallback is active

3. Silent Substitutions

• Mock/stub objects substituted for real implementations
• Default values replacing missing external data
• Cached data used when fresh data unavailable

Language-Specific Patterns

Python

# Anti-pattern: Silent import failure
try:
    import redis
    REDIS_AVAILABLE = True
except ImportError:
    REDIS_AVAILABLE = False
    redis = None  # No indication to users

# Anti-pattern: Silent API degradation
try:
    result = expensive_api_call()
except RequestException:
    result = {}  # Empty result indistinguishable from "no data"

JavaScript/TypeScript

// Anti-pattern: Optional dependency silently missing
let logger;
try {
  logger = require("winston");
} catch {
  logger = { info: () => {}, error: () => {} }; // Silent no-op
}

// Anti-pattern: API fallback without visibility
async function fetchData() {
  try {
    return await primaryAPI.get();
  } catch {
    return await fallbackAPI.get(); // No indication which source
  }
}

Rust

// Anti-pattern: Optional feature silently disabled
#[cfg(feature = "redis")]
use redis::Client;

#[cfg(not(feature = "redis"))]
type Client = ();  // No-op type, silent degradation

Go

// Anti-pattern: Error ignored
db, err := sql.Open("postgres", connStr)
if err != nil {
    db = nil  // Silent fallback to no database
}

Java

// Anti-pattern: Exception swallowing
try {
    client = new RedisClient(config);
} catch (ConnectionException e) {
    client = new NoOpClient();  // Silent substitution
}

C#

```c# // Anti-pattern: Optional service silently unavailable try { _cache = new RedisCache(config); } catch { _cache = new NullCache(); // Silent degradation }

## Detection Strategy

### Phase 1: Import Analysis

- Scan for try/except around imports
- Check for conditional feature flags
- Identify optional dependencies in requirements

### Phase 2: Connection Point Analysis

- Find database connection initialization
- Locate API client creation
- Identify message queue connections

### Phase 3: Fallback Chain Analysis

- Map fallback hierarchies
- Identify silent substitutions
- Check for visibility at each level

### Phase 4: Error Handling Review

- Review exception handlers for dependency failures
- Check timeout handling
- Verify retry logic has visibility

## Validation Criteria

A finding is valid if:

1. **Failure is invisible**: No log, metric, or alert when dependency fails
2. **Degradation is silent**: System continues with reduced functionality
3. **No operator visibility**: No way for operators to know degradation occurred
4. **Potential impact**: Degradation affects user-visible functionality or data quality

## Output Format

{ "category": "dependency-failures", "severity": "high|medium|low", "file": "path/to/file.py", "line": 42, "description": "Redis import failure silently disables caching", "impact": "System runs without cache, 10x slower", "visibility": "None - no logs or metrics", "recommendation": "Add explicit cache unavailability warning and metrics" }

## Integration Points

- **With config-errors**: Dependency failures often stem from config issues
- **With operator-visibility**: Dependency failures must be visible to operators
- **With test-effectiveness**: Tests should verify behavior when dependencies fail

## Common Exclusions

- Development-only optional dependencies (linters, formatters)
- Explicitly documented optional features
- Features with clear "requires X" documentation

## Battle-Tested Insights (from CyberGym ~250 bug audit)

1. **Most common**: Import-time failures with no visibility (40% of findings)
2. **Most dangerous**: Silent API fallbacks that return stale/wrong data (25%)
3. **Most overlooked**: Transitive dependency failures (20%)
4. **Most fixable**: Add logging at fallback points (80% quick wins)

Category F: Functional Stubs Agent

Role

NEW category agent for detecting code that looks functional but is actually a stub or placeholder. Asks "Does this code actually do what its name says?"

Core Question

"Does this code actually do what its name says?"

Focus areas:

• Empty or trivial implementations
• Parameters that are ignored
• Methods that always return the same value
• Incomplete implementations masquerading as complete
• Interface stubs in production code

Detection Focus

Empty Returns

1. Empty Collections

• Methods returning {}, [], Array.Empty<>(), Vec::new()
• Method name suggests data retrieval but returns empty
• No indication if empty means "no data" or "not implemented"

2. Constant Returns

• Method always returns same value regardless of input
• return true, return null, return 0
• Method name suggests computation but returns constant

3. No-Op Methods

• Method body is pass, {}, empty block
• Interface method with trivial implementation
• Method that doesn't modify any state

Ignored Parameters

1. Unused Parameters

• Parameters in signature but never referenced in body
• _ = param pattern (explicitly discarded)
• Complex parameter objects that are ignored

2. Partial Parameter Usage

• Method accepts 5 parameters, uses 1
• Key parameters like ID or credentials ignored
• Parameters that should affect behavior but don't

3. Parameter Shadowing

• Parameter replaced with hardcoded value
• Parameter passed to method that ignores it
• Parameter in chain where later methods ignore it

Disproportionate Simplicity

1. Complex Name, Simple Body

• validateComplexBusinessRules() returns true
• calculateRiskScore() returns 0
• processPayment() does nothing

2. Missing Business Logic

• Method name suggests complex logic
• Implementation is trivial or missing
• Comments like "TODO: Implement actual logic"

3. Stub Markers

• Contains NotImplementedError, todo!(), unimplemented!()
• Comments saying "stub" or "placeholder"
• throw new UnsupportedOperationException()

Production Stubs

1. Interface Implementations

• Required interface method implemented as stub
• Abstract method override that does nothing
• Contract requires implementation but it's empty

2. Conditional Implementations

• if (FEATURE_FLAG) { real_logic } else { stub }
• Feature flag always false in production
• "Coming soon" implementations

3. Test Artifacts in Production

• Mock objects in production code
• Fake implementations not behind test flag
• Test-only methods called from production

Language-Specific Patterns

Python

# Anti-pattern: Empty dict return
def get_user_preferences(user_id):
    """Retrieve user preferences from database."""
    return {}  # Stub - no database query

# Anti-pattern: Parameters ignored
def validate_payment(amount, currency, card_number, cvv):
    """Validate payment details."""
    return True  # Ignores all parameters

# Anti-pattern: Trivial implementation
def calculate_shipping_cost(origin, destination, weight):
    """Calculate shipping cost based on distance and weight."""
    return 0  # No actual calculation

# Anti-pattern: Not implemented marker
def process_refund(order_id):
    raise NotImplementedError("Refunds not yet supported")

JavaScript/TypeScript

// Anti-pattern: Empty object return
function getUserPermissions(userId) {
  return {}; // No permissions loaded
}

// Anti-pattern: Unused parameters
function processOrder(orderId, userId, items, payment) {
  return { success: true }; // Parameters not used
}

// Anti-pattern: Stub interface
class CacheService {
  get(key) {
    return null;
  } // Stub
  set(key, value) {} // No-op
  delete(key) {} // No-op
}

TypeScript

// Anti-pattern: Interface stub with type assertion
interface PaymentProcessor {
  charge(amount: number): Promise<PaymentResult>;
}

class StubProcessor implements PaymentProcessor {
  async charge(amount: number): Promise<PaymentResult> {
    return { success: true } as PaymentResult; // Stub
  }
}

Rust

// Anti-pattern: Todo marker
fn process_transaction(tx: Transaction) -> Result<(), Error> {
    todo!("Implement transaction processing")
}

// Anti-pattern: Empty vec return
fn get_active_users() -> Vec<User> {
    Vec::new()  // No users loaded
}

// Anti-pattern: Unused parameters
fn validate_input(_data: &str, _schema: &Schema) -> bool {
    true  // Parameters explicitly ignored
}

Go

// Anti-pattern: Nil return
func GetUserProfile(userID string) *UserProfile {
    return nil  // No profile loaded
}

// Anti-pattern: Parameters ignored
func ValidateRequest(req *http.Request, rules []Rule) error {
    return nil  // Rules never checked
}

// Anti-pattern: Empty slice
func FetchOrders(customerID string) []Order {
    return []Order{}  // No orders fetched
}

Java

// Anti-pattern: Interface stub
@Override
public List<Order> getOrders(String customerId) {
    return Collections.emptyList();  // Stub
}

// Anti-pattern: UnsupportedOperation
@Override
public void processPayment(Payment payment) {
    throw new UnsupportedOperationException("Not implemented yet");
}

// Anti-pattern: Parameters ignored
public boolean validate(String data, Schema schema, Context ctx) {
    return true;  // All parameters ignored
}

C#

// Anti-pattern: Task.CompletedTask stub
public async Task<Result> ProcessAsync(Request request) {
    return await Task.FromResult(new Result { Success = true });
    // Stub - no actual processing
}

// Anti-pattern: Default return
public Order GetOrder(int orderId) {
    return default;  // Stub returning null
}

// Anti-pattern: NotImplementedException
public void SaveData(Data data) {
    throw new NotImplementedException();
}

Ruby

# Anti-pattern: Empty hash return
def user_settings(user_id)
  {}  # No settings loaded
end

# Anti-pattern: Ignored parameters
def validate_order(order, rules, context)
  true  # Nothing validated
end

# Anti-pattern: Placeholder
def process_payment(payment)
  raise NotImplementedError, "Payment processing coming soon"
end

PHP

// Anti-pattern: Empty array return
function getProducts($categoryId) {
    return [];  // No products fetched
}

// Anti-pattern: Parameters unused
function validateInput($data, $rules, $options) {
    return true;  // Nothing validated
}

// Anti-pattern: Stub method
function processOrder($order) {
    // TODO: Implement order processing
    return true;
}

Detection Strategy

Phase 1: Return Value Analysis

• Find methods returning empty collections
• Identify constant return values
• Check if return matches method name expectations

Phase 2: Parameter Usage Analysis

• Check if all parameters are used in method body
• Identify explicitly discarded parameters (_)
• Verify complex parameters are accessed

Phase 3: Complexity Analysis

• Compare method name complexity to body complexity
• Find methods with only return statements
• Identify trivial implementations

Phase 4: Stub Marker Detection

• Search for NotImplementedError, todo!(), etc.
• Find TODO/FIXME comments about implementation
• Check for test-only code in production paths

Validation Criteria

A finding is valid if:

1. Name suggests functionality: Method/function name implies it does something 2. Implementation is stub: Body is empty, trivial, or placeholder 3. Called from production: Not test-only or debug code 4. No clear indicator: Not obviously marked as stub/placeholder

Output Format

{
  "category": "functional-stubs",
  "severity": "high|medium|low",
  "file": "src/payments.py",
  "line": 45,
  "function": "calculate_tax",
  "description": "Method returns constant 0 instead of calculating tax",
  "signature": "calculate_tax(amount, state, county)",
  "implementation": "return 0",
  "parameters_used": 0,
  "parameters_total": 3,
  "impact": "Tax never calculated, orders undercharged",
  "recommendation": "Implement actual tax calculation or mark as stub"
}

Integration Points

• With test-effectiveness: Stubs may have tests that pass trivially
• With operator-visibility: Stubs may fail silently
• With config-errors: Stub behavior may seem like config issue

Common Exclusions

• Explicitly documented stubs (with clear comments)
• Abstract base class methods (meant to be overridden)
• Methods with clear names like stub_* or mock_*
• Interface default implementations (intentionally minimal)

Battle-Tested Insights (from CyberGym ~250 bug audit)

1. Most common: Empty collection returns (40% of Category F findings) 2. Most dangerous: Payment/auth stubs in production (30%) 3. Most overlooked: Parameters silently ignored (20%) 4. Most fixable: Add NotImplementedError to make stub explicit (60% quick wins)

Detection Heuristics

High Confidence (Likely Stub)

• Method returns empty collection AND has "get", "fetch", "load" in name
• Method has 3+ parameters AND uses none of them
• Method body is single line: return true/false/0/null/{}/[]
• Method has TODO/FIXME comment about implementation
• Method throws NotImplementedError or similar

Medium Confidence (Possibly Stub)

• Method returns constant AND name suggests computation
• Method has complex name AND simple body (< 5 lines)
• Parameters explicitly discarded with _ pattern
• Interface implementation that does nothing

Low Confidence (Needs Context)

• Private method returning empty (may be helper)
• Method with "default" in name returning default value
• Builder pattern method returning this
• Early return optimization

Red Flags

• "calculate" in name but returns constant
• "validate" in name but returns true
• "process" in name but has no side effects
• "get" in name but returns empty
• Method accepts data but doesn't use it
• Method has 10+ parameters, uses 1
• Comment says "TODO", "FIXME", "stub", "placeholder"
• Throws NotImplementedError in production code
• Interface implementation is one-line no-op

Category E: Operator Visibility Agent

Role

Specialized agent for detecting silent degradation where errors occur but operators have no way to know. Asks "Is the error visible to operators?"

Core Question

"Is the error visible to operators?"

Focus areas:

• Logging gaps
• Metrics blind spots
• Alert coverage
• Dashboard visibility
• Diagnostic capabilities

Detection Focus

Logging Gaps

1. Silent Exceptions

• Exception caught but not logged
• Generic "error occurred" with no details
• Debug-level logs for production issues

2. Missing Context

• Log without request ID or correlation ID
• No user/session information
• Missing critical business context (order ID, user ID)

3. Log Level Misuse

• Errors logged at INFO level
• Critical issues at DEBUG level
• Warning fatigue (too many warnings, real issues hidden)

Metrics Blind Spots

1. No Error Metrics

• Successful operations counted, failures not
• Latency tracked, but not error rate
• Request count without success/failure breakdown

2. Aggregation Hides Issues

• Average hides outliers
• Total count hides percentage
• Per-minute misses sub-minute spikes

3. Missing Business Metrics

• Technical metrics (CPU, memory) but no business metrics
• Infrastructure healthy but business logic failing
• No SLI/SLO tracking

Alert Gaps

1. No Alerts Defined

• Metrics collected but not alerted on
• Logs written but no log-based alerts
• Manual checking required to find issues

2. Alert Fatigue

• Too many false positives
• Alerts ignored or muted
• No escalation path

3. Threshold Problems

• Thresholds too loose (miss issues)
• Thresholds too tight (constant noise)
• Static thresholds (should be dynamic)

Dashboard Gaps

1. No Visibility

• No dashboard for service health
• Metrics exist but not visualized
• Dashboard shows infrastructure not business

2. Wrong Granularity

• Dashboard shows hourly, issues happen in seconds
• Dashboard aggregates, hiding specific failures
• No drill-down capability

3. No Historical Context

• Can't compare current to baseline
• No trend visualization
• Can't see if degradation is worsening

Language-Specific Patterns

Python

# Anti-pattern: Exception not logged
try:
    process_payment(order)
except PaymentException:
    return {"error": "payment failed"}  # No log, no metric

# Anti-pattern: Generic error log
except Exception as e:
    logger.error("Error occurred")  # No context, no exception details

# Anti-pattern: Success tracked, failure not
metrics.increment("orders.processed")
# Missing: metrics.increment("orders.failed") on error

JavaScript/TypeScript

// Anti-pattern: Catch without logging
try {
  await processOrder(order);
} catch (err) {
  return { error: 'Failed' };  // Silent failure
}

// Anti-pattern: Console.log in production
catch (err) {
  console.log('Error:', err);  // Not sent to logging service
}

// Anti-pattern: No correlation ID
logger.info('Processing order');  // Can't trace request through system

Rust

// Anti-pattern: Error converted to None
let result = process().ok();  // Error discarded, no visibility

// Anti-pattern: Error logged at wrong level
if let Err(e) = critical_operation() {
    debug!("Operation failed: {}", e);  // Should be error!
}

Go

// Anti-pattern: Error ignored
if err := processOrder(order); err != nil {
    return err  // Propagated but never logged
}

// Anti-pattern: No structured logging
log.Println("Error processing order")  // No context, hard to query

Java

// Anti-pattern: Exception caught and hidden
catch (PaymentException e) {
    return Response.status(500).build();  // No log
}

// Anti-pattern: Stack trace logged but not error details
catch (Exception e) {
    e.printStackTrace();  // Not in structured logs
}

C#

// Anti-pattern: Exception swallowed
catch (Exception ex) {
    // TODO: Add logging
    return BadRequest();
}

// Anti-pattern: No metrics on error path
try {
    ProcessOrder(order);
    _metrics.Increment("orders.success");
} catch {
    // Missing: _metrics.Increment("orders.failure")
}

Detection Strategy

Phase 1: Exception Handling Analysis

• Find all exception handlers
• Check if exceptions are logged
• Verify log level appropriate for severity

Phase 2: Metrics Coverage Analysis

• Identify success metrics
• Check for corresponding failure metrics
• Verify SLI/SLO metrics exist

Phase 3: Alert Configuration Analysis

• Check if critical paths have alerts
• Verify alert thresholds make sense
• Check for alert coverage gaps

Phase 4: Observability Stack Analysis

• Verify logging infrastructure integrated
• Check metrics collection configured
• Verify dashboards exist and used

Validation Criteria

A finding is valid if:

1. Error can occur: Code path exists where error happens 2. No operator visibility: Error not logged, metriced, or alerted 3. Operator needs to know: Error impacts users or system health 4. No current monitoring: Not covered by existing observability

Output Format

{
  "category": "operator-visibility",
  "severity": "high|medium|low",
  "file": "src/payments.py",
  "line": 89,
  "description": "Payment failure not logged or metriced",
  "error_path": "PaymentException caught but not recorded",
  "impact": "Operators can't see payment failure rate",
  "current_visibility": "None",
  "recommendation": "Add logger.error() with order context and metrics.increment('payment.failures')"
}

Integration Points

• With all other categories: Every failure type needs visibility
• With dependency-failures: Dependency failures must be visible
• With background-work: Background failures especially need visibility

Common Exclusions

• Errors already logged and metriced (check implementation carefully)
• Expected errors with documented visibility (e.g., user input validation)
• Errors with automatic alerting (verify alerts actually fire)

Battle-Tested Insights (from CyberGym ~250 bug audit)

1. Most common: Exceptions caught but not logged (55%) 2. Most dangerous: Critical path failures with no metrics (30%) 3. Most overlooked: Async failures not visible to operators (10%) 4. Most fixable: Add logging to existing exception handlers (90% quick wins)

Observability Checklist

For each error path, verify:

• [ ] Logging: Error logged with context (user ID, request ID, details)
• [ ] Log Level: Appropriate level (ERROR for errors, not DEBUG/INFO)
• [ ] Metrics: Error rate metric exists and incremented
• [ ] Metrics Granularity: Can drill down by error type, service, endpoint
• [ ] Alerting: Alert defined for error rate threshold
• [ ] Alert Tuning: Alert threshold tested and not too noisy
• [ ] Dashboard: Error rate visible on service dashboard
• [ ] Tracing: Distributed trace includes error information
• [ ] Correlation: Can trace error from log to metric to trace
• [ ] Runbook: Operator knows what to do when alert fires

Red Flags

• except Exception: without logger.error()
• catch (Exception e) { } with empty block
• Success metric incremented, no corresponding failure metric
• Error returned to caller but not logged locally
• console.log() or System.out.println() in production code
• Debug-level logging for user-impacting errors
• No dashboard for critical service
• Alerts commented out or disabled
• "TODO: Add logging" comments in exception handlers

Category D: Test Effectiveness Agent

Role

Specialized agent for detecting gaps in test coverage where tests pass but don't actually verify error conditions. Asks "Do tests actually detect failures?"

Core Question

"Do tests actually detect failures?"

Focus areas:

• Error case coverage
• Failure mode testing
• Test assertions vs. silent passes
• Mock behavior vs. real behavior
• Integration test gaps

Detection Focus

Missing Error Cases

1. Happy Path Only

• Tests verify success case only
• No tests for exception paths
• No tests for timeout scenarios

2. Shallow Mocking

• Mocks always return success
• Mocks never raise exceptions
• Real dependencies behave differently than mocks

3. Assertion Gaps

• Test runs but doesn't assert critical outcomes
• Assertions check wrong thing (200 status, but not body)
• Silent pass when test should fail

False Positives

1. Tests That Can't Fail

• Test mocks everything, nothing can break
• Test assertions always true
• Test setup guarantees success

2. Flaky Tests Ignored

• Tests marked skip or xfail
• Tests with "sometimes fails" comments
• Retry logic hiding real failures

3. Coverage Theater

• High coverage percentage, low error coverage
• Tests exist but don't verify behavior
• Tests added to hit coverage targets

Integration Gaps

1. Unit vs. Integration Mismatch

• Unit tests mock external dependencies
• Integration tests never run
• Real behavior differs from mocked behavior

2. Environment-Specific Failures

• Tests pass locally, fail in CI
• Tests pass in CI, fail in production
• Tests don't cover production configuration

3. Timing and Concurrency

• Tests run synchronously
• Production runs concurrently
• Race conditions not tested

Language-Specific Patterns

Python

# Anti-pattern: Happy path only
def test_process_order():
    result = process_order(valid_order)
    assert result.success
    # Missing: What if order invalid? Payment fails? Network error?

# Anti-pattern: Mock never fails
@patch('payment_service.charge')
def test_checkout(mock_charge):
    mock_charge.return_value = True  # Always succeeds
    checkout(order)
    # Never tests charge failure case

# Anti-pattern: No assertion
def test_background_task():
    process_in_background(data)  # Launches task
    # Test passes even if task fails

JavaScript/TypeScript

// Anti-pattern: Happy path only
test("fetches data", async () => {
  const data = await fetchData();
  expect(data).toBeDefined();
  // Missing: Network error? Timeout? Invalid response?
});

// Anti-pattern: Mock hides real behavior
jest.mock("./api");
test("processes order", () => {
  api.charge.mockResolvedValue({ success: true });
  // Real API has retry logic, rate limits, errors
});

// Anti-pattern: Async test doesn't await
test("saves data", () => {
  saveToDatabase(data); // Returns promise, not awaited
  // Test finishes before save completes
});

Rust

// Anti-pattern: Error path not tested
#[test]
fn test_parse_config() {
    let config = parse_config("valid.toml").unwrap();
    assert_eq!(config.port, 8080);
    // Missing: Invalid TOML? Missing file? Bad values?
}

// Anti-pattern: Mock hides real complexity
#[test]
fn test_fetch() {
    let mock = MockClient::new();
    let result = fetch(&mock);  // Mock always succeeds
    // Real client has timeouts, retries, errors
}

Go

// Anti-pattern: Error not checked
func TestProcess(t *testing.T) {
    result := Process(validInput)
    // Missing: err := Process(invalidInput)
    if result.Success {
        // Test passes
    }
}

// Anti-pattern: Mock doesn't match interface
type MockDB struct{}
func (m *MockDB) Query() Result {
    return Result{Data: "test"}  // Never returns error
}
// Real DB returns errors

Java

// Anti-pattern: Exception path not tested
@Test
public void testProcessOrder() {
    Order order = new Order(validData);
    Result result = orderService.process(order);
    assertTrue(result.isSuccess());
    // Missing: @Test(expected = PaymentException.class)
}

// Anti-pattern: Mock hides timing issues
@Mock
PaymentService paymentService;

@Test
public void testCheckout() {
    when(paymentService.charge()).thenReturn(success);
    // Real service can timeout, have retries
}

C#

// Anti-pattern: Happy path only
[Fact]
public void ProcessOrder_ValidOrder_Succeeds() {
    var result = _service.ProcessOrder(validOrder);
    Assert.True(result.Success);
    // Missing: Invalid order? Payment failure? Timeout?
}

// Anti-pattern: Async void not awaited
[Fact]
public async Task SaveData() {
    _service.SaveAsync(data);  // Not awaited
    // Test completes before save finishes
}

Detection Strategy

Phase 1: Test Coverage Analysis

• Identify functions with only happy path tests
• Find exception handlers not covered by tests
• Check for timeout/retry code without tests

Phase 2: Mock Analysis

• Review mocks that never return errors
• Check if mocks match real interface behavior
• Identify integration test gaps

Phase 3: Assertion Analysis

• Find tests without assertions
• Check for weak assertions (just checks not null)
• Identify tests that can't fail

Phase 4: Error Path Coverage

• Map error paths in production code
• Check which error paths have tests
• Identify untested exception handling

Validation Criteria

A finding is valid if:

1. Real failure path exists: Production code has error handling 2. No test coverage: Error path not tested 3. Silent pass possible: Test could pass even if error handling broken 4. Production impact: Untested code runs in production

Output Format

{
  "category": "test-effectiveness",
  "severity": "high|medium|low",
  "file": "tests/test_orders.py",
  "line": 45,
  "function": "process_order",
  "description": "Payment failure path not tested",
  "production_code": "src/orders.py:123",
  "missing_test": "Test for PaymentException handling",
  "impact": "Payment failure code could be broken, tests still pass",
  "recommendation": "Add test_payment_failure() with mock that raises exception"
}

Integration Points

• With dependency-failures: Tests should verify behavior when dependencies fail
• With config-errors: Tests should verify behavior with bad config
• With background-work: Tests should verify async failure handling

Common Exclusions

• Defensive error handling that's truly unreachable
• Third-party library error paths (test integration, not library internals)
• Error paths explicitly marked as unreachable with comments

Battle-Tested Insights (from CyberGym ~250 bug audit)

1. Most common: Happy path tests only (60% of findings) 2. Most dangerous: Exception handlers with no tests (25%) 3. Most overlooked: Async error handling not tested (10%) 4. Most fixable: Add error case test for each happy path test (70% quick wins)

Red Flags

• Test file has 10+ tests, all pass, zero use exception mock
• Function has try/except, test file has no exception testing
• Mock service always returns success
• Test has no assertions (or only assert True)
• Test marked as @skip or @xfail with "flaky" comment
• Integration tests commented out or never run in CI
• Test coverage report shows 90%+ but error handlers not covered

How to Use the Silent Degradation Audit Skill

Step-by-step guide to detecting silent failures in your codebase.

Quick Start

/silent-degradation-audit /path/to/your/project

Runs multi-wave audit detecting 6 categories of silent failures.

What You'll Find

The audit detects bugs where systems silently degrade instead of failing visibly:

• API errors returned as empty data
• Config errors using unsafe defaults
• Background jobs failing without alerts
• Tests that pass on both success AND failure
• Errors invisible to monitoring
• Functions that don't do what their names say

Step 1: Run Your First Audit

cd ~/projects/my-api
/silent-degradation-audit .

Output: .silent-degradation-report.md with findings

Step 2: Review Results

cat .silent-degradation-report.md

Example findings:

Wave 1: 27 findings

Critical (3):
- auth.py:45 - Exception returns None, caller treats as authenticated
- processor.py:123 - Background task fails, no alert
- config.py:67 - Missing API_KEY, uses "default"

Step 3: Fix Issues

Before:

def authenticate(token):
    try:
        return verify_token(token)
    except TokenExpired:
        return None  # Silent failure!

After:

def authenticate(token):
    try:
        return verify_token(token)
    except TokenExpired:
        raise AuthenticationError("Token expired")

Using Exclusion Lists

Create .silent-degradation-exclusions.json:

[
  {
    "pattern": "tests/**/*.py",
    "reason": "Test fixtures intentionally silent",
    "type": "glob"
  }
]

Multi-Language Projects

Automatically detects Python, JavaScript, TypeScript, Rust, Go, Java, C#, Ruby, PHP.

Each language has specific patterns:

• Python: except: pass
• JavaScript: .catch(err => {})
• Rust: .unwrap()
• Go: _, _ = ...

See patterns.md for complete list.

Advanced Configuration

# Custom thresholds
/silent-degradation-audit . --convergence-absolute 5

# Specific categories only
/silent-degradation-audit . --categories dependency,config

# Debug mode
export AUDIT_DEBUG=1
/silent-degradation-audit .

Integration with CI

# .github/workflows/audit.yml
- name: Weekly audit
  run: /silent-degradation-audit .

Troubleshooting

No findings? Check language detection in report header.

Too many false positives? Add to exclusion list.

Takes too long? Audit specific directories or reduce max waves.

Next Steps

1. Fix Critical/High findings first 2. Re-run to verify convergence 3. Add tests for error paths 4. Add monitoring/alerts for silent failures

See examples.md for complete walkthroughs and reference.md for full API.

Silent Degradation Audit

Multi-wave audit system for detecting code that fails silently in production.

Quick Start

# Audit your codebase
/silent-degradation-audit ./src

# Review findings
cat .silent-degradation-report.md
cat .silent-degradation-findings.json

What It Detects

6 categories of silent degradation:

1. Dependency Failures: Import errors, missing modules, API fallbacks 2. Config Errors: Missing env vars, bad defaults, silent config failures 3. Background Work: Async task failures, queue processing errors 4. Test Effectiveness: Happy path tests that miss error cases 5. Operator Visibility: Missing logs, metrics, alerts 6. Functional Stubs: Empty implementations, ignored parameters

How It Works

Multi-Wave Progressive Audit:

• Wave 1: Finds obvious issues (40-50% of total)
• Wave 2-3: Deeper analysis (30-40%)
• Wave 4+: Edge cases (10-20%)
• Stops when < 10 new findings or < 5% of Wave 1

Multi-Agent Validation:

• 3 agents review each finding (Security, Architect, Builder)
• 2/3 consensus required to validate
• Prevents false positives

Output

Report (.silent-degradation-report.md):

• Summary statistics
• Convergence progress plot
• Findings by category and severity

Findings (.silent-degradation-findings.json):

• Detailed findings with location, description, impact
• Validation results and votes
• Fix recommendations

Configuration

Create .silent-degradation-config.json:

{
  "convergence": {
    "absolute_threshold": 10,
    "relative_threshold": 0.05
  },
  "max_waves": 6
}

Exclusions

Add to .silent-degradation-exclusions.json:

[
  {
    "pattern": "*.test.*",
    "reason": "Test files",
    "category": "*"
  }
]

Integration Modes

Standalone:

/silent-degradation-audit path/to/code

Sub-loop in quality-audit-workflow:

quality-audit-workflow → Phase 2 → silent-degradation-audit

Supported Languages

Python, JavaScript, TypeScript, Rust, Go, Java, C#, Ruby, PHP

Battle-Tested

Used on CyberGym codebase, found ~250 bugs across all 6 categories.

Documentation

• SKILL.md - Complete documentation
• reference.md - Technical reference
• examples.md - Usage examples
• patterns.md - Language-specific patterns
• category_agents/ - Category agent specs
• validation_panel/ - Validation panel docs

Requirements

• Claude Code with agent support
• Recipe Runner enabled
• Python 3.8+ (for utility tools)

Example Output

Wave 1: ██████████████████████████████████████████████████ 120
Wave 2: ███████████████████████████ 65 (54.2% of Wave 1)
Wave 3: ████████ 18 (15.0% of Wave 1)
Wave 4: ██ 5 (4.2% of Wave 1)

Status: ✓ CONVERGED
Reason: Relative threshold met: 4.2% < 5.0%

Findings:
- dependency-failures: 42 (High: 15, Medium: 20, Low: 7)
- config-errors: 28 (High: 8, Medium: 12, Low: 8)
- background-work: 19 (High: 6, Medium: 9, Low: 4)
- test-effectiveness: 23 (High: 2, Medium: 15, Low: 6)
- operator-visibility: 18 (High: 9, Medium: 7, Low: 2)
- functional-stubs: 7 (High: 1, Medium: 4, Low: 2)

Total: 137 findings validated by panel

License

Part of the amplihack agentic coding framework

Validation Panel Specification

Purpose

Multi-agent validation panel ensures findings are legitimate before applying fixes. Uses 2/3 consensus mechanism to prevent false positives and unnecessary changes.

Panel Composition

Three specialized agents vote on each finding:

1. Security Agent - Assesses security implications 2. Architect Agent - Evaluates design and architectural impact 3. Builder Agent - Considers implementation complexity and cost

Voting Process

Step 1: Present Finding

Each finding is presented to all three agents simultaneously with:

• Finding details (file, line, description, category)
• Code context (surrounding lines)
• Impact assessment from category agent
• Proposed fix recommendation

Step 2: Independent Evaluation

Each agent evaluates the finding independently based on their specialty:

Security Agent Focus:

• Does this finding represent a real security risk?
• Could this lead to data exposure or unauthorized access?
• Is the impact correctly assessed?

Architect Agent Focus:

• Is this finding consistent with system architecture?
• Does fixing it improve or harm design quality?
• Are there broader architectural implications?

Builder Agent Focus:

• Is this finding actionable (can be fixed)?
• What is the implementation complexity?
• Are there dependencies or side effects?

Step 3: Vote Casting

Each agent casts one vote:

• APPROVE: Finding is valid, should be fixed
• REJECT: Finding is false positive or not worth fixing
• ABSTAIN: Agent lacks context to decide (rarely used)

Step 4: Consensus Determination

2/3 Consensus Required:

• Need 2 out of 3 APPROVE votes to validate finding
• 2 or more REJECT votes invalidates finding
• ABSTAIN votes don't count toward consensus

Consensus Outcomes:

• AAA (3 approve) → VALIDATED (strong consensus)
• AAR (2 approve, 1 reject) → VALIDATED (weak consensus)
• ARR (1 approve, 2 reject) → REJECTED
• RRR (3 reject) → REJECTED (strong rejection)
• AAB, ARB → VALIDATED (abstain doesn't block)
• RRB, RBB → REJECTED (abstain doesn't block)
• ABB → INCONCLUSIVE (requeue for human review)

Validation Criteria

Criteria for APPROVE Vote

Agent should vote APPROVE if finding meets ALL:

1. Real Issue: Not a false positive, issue actually exists 2. Actionable: Can be fixed with reasonable effort 3. Impactful: Fixing improves code quality, security, or reliability 4. Scope-Appropriate: Within scope of current audit (not general refactoring)

Criteria for REJECT Vote

Agent should vote REJECT if finding meets ANY:

1. False Positive: Issue doesn't actually exist 2. Not Fixable: Fixing would require unreasonable effort or break things 3. No Impact: Fixing provides no meaningful benefit 4. Out of Scope: Not a silent degradation issue (different category) 5. Already Addressed: Issue fixed elsewhere or mitigated

Criteria for ABSTAIN Vote

Agent should vote ABSTAIN if:

1. Insufficient Context: Not enough information to decide 2. Outside Expertise: Issue outside agent's specialty area 3. Conflicting Information: Finding contains contradictory data

Vote Justification

Each vote must include brief justification:

{
  "agent": "security",
  "vote": "APPROVE",
  "justification": "Missing exception logging creates security blind spot - attackers can probe without detection"
}

Batch Processing

For efficiency, validation panel can process findings in batches:

1. Group findings by category or file 2. Present batch to all three agents 3. Collect votes for entire batch 4. Apply consensus logic to each finding

Conflict Resolution

When 3-agent panel can't reach consensus (rare):

1. Inconclusive Result (e.g., ABB):

• Flag for human review
• Include all agent justifications
• Don't apply automatic fix

2. Split Decision (AAR):

• Proceed with caution
• Apply fix but mark for review
• Monitor for unintended consequences

Output Format

{
  "finding_id": "dep-001",
  "validation_result": "VALIDATED|REJECTED|INCONCLUSIVE",
  "consensus_type": "strong|weak|none",
  "votes": [
    {
      "agent": "security",
      "vote": "APPROVE",
      "justification": "Real security risk"
    },
    {
      "agent": "architect",
      "vote": "APPROVE",
      "justification": "Improves error handling design"
    },
    {
      "agent": "builder",
      "vote": "REJECT",
      "justification": "Fix would require refactoring 10+ files"
    }
  ],
  "recommendation": "Proceed with fix (2/3 approval)",
  "review_required": false
}

Integration with Audit Workflow

Before Fix Application

1. Category agents complete wave and generate findings 2. Validation panel reviews all findings 3. Only VALIDATED findings proceed to fix phase 4. REJECTED findings added to exclusion list 5. INCONCLUSIVE findings flagged for human review

During Waves

Each wave:

1. Category agents find new issues (6 agents in parallel) 2. Validation panel reviews findings (3 agents in parallel) 3. Validated findings counted toward convergence 4. Process repeats until convergence

Metrics

Track validation panel effectiveness:

• Validation Rate: % of findings validated
• Rejection Rate: % of findings rejected
• Consensus Strength: % strong consensus (AAA or RRR)
• Agent Agreement: How often each pair agrees
• False Positive Rate: % of validated findings that were actually false (requires manual audit)

Tuning Thresholds

If validation panel is:

Too Strict (rejecting valid findings):

• Review REJECT justifications
• Adjust agent prompts to be more permissive
• Consider 1/3 threshold for low-severity findings

Too Permissive (approving false positives):

• Review APPROVE justifications
• Require stronger evidence in prompts
• Consider requiring unanimous approval for high-severity

Examples

Example 1: Strong Approval (AAA)

Finding: Exception caught but not logged

Votes:

• Security: APPROVE - "Log gap creates security blind spot"
• Architect: APPROVE - "Violates observability principles"
• Builder: APPROVE - "One-line fix, add logger.error()"

Result: VALIDATED (strong consensus) → Apply fix

Example 2: Weak Approval (AAR)

Finding: Optional dependency missing silent fallback

Votes:

• Security: APPROVE - "Dependency failure should be visible"
• Architect: APPROVE - "Fallback violates fail-fast principle"
• Builder: REJECT - "Optional dependency intended to be optional"

Result: VALIDATED (weak consensus) → Apply fix but mark for review

Example 3: Rejection (ARR)

Finding: Empty list returned from get_items()

Votes:

• Security: APPROVE - "Stub in production code"
• Architect: REJECT - "Empty list is valid response for 'no items'"
• Builder: REJECT - "Method works correctly, not a stub"

Result: REJECTED → Add to exclusion list

Example 4: Inconclusive (ABB)

Finding: Complex validation method returns true

Votes:

• Security: APPROVE - "Looks like stub, validation not implemented"
• Architect: ABSTAIN - "Need business context to determine if valid"
• Builder: ABSTAIN - "Can't tell if intentional or stub without docs"

Result: INCONCLUSIVE → Flag for human review

Anti-Patterns to Avoid

1. Rubber Stamping: Agents always voting APPROVE 2. Over-Rejection: Agents being overly conservative 3. Insufficient Justification: Votes without reasoning 4. Agent Bias: One agent's opinion dominating 5. Context Blindness: Not considering full code context

Quality Assurance

Periodically audit validation panel by:

1. Manually reviewing sample of VALIDATED findings 2. Manually reviewing sample of REJECTED findings 3. Checking if INCONCLUSIVE findings need process improvement 4. Comparing validation outcomes across different codebases 5. Measuring fix success rate (do validated fixes actually improve code?)

Voting Rules and Implementation

Vote Tallying Logic

Python implementation of 2/3 consensus voting mechanism.

from enum import Enum
from typing import Callable, Dict, List, Any, Tuple


class Vote(Enum):
    """Vote options for validation panel."""
    APPROVE = "APPROVE"
    REJECT = "REJECT"
    ABSTAIN = "ABSTAIN"


class ValidationResult(Enum):
    """Validation outcomes."""
    VALIDATED = "VALIDATED"
    REJECTED = "REJECTED"
    INCONCLUSIVE = "INCONCLUSIVE"


class ConsensusType(Enum):
    """Strength of consensus."""
    STRONG = "strong"  # AAA or RRR
    WEAK = "weak"      # AAR or ARR with 2/3
    NONE = "none"      # Inconclusive


def tally_votes(votes: List[Dict[str, Any]]) -> Tuple[ValidationResult, ConsensusType, str]:
    """Tally votes from validation panel and determine outcome.

    Args:
        votes: List of vote dictionaries with keys:
            - agent: str (agent name)
            - vote: str (APPROVE, REJECT, or ABSTAIN)
            - justification: str (reason for vote)

    Returns:
        Tuple of (validation_result, consensus_type, recommendation)

    Consensus Rules:
        - 3 APPROVE → VALIDATED (strong)
        - 2 APPROVE, 1 REJECT → VALIDATED (weak)
        - 2 APPROVE, 1 ABSTAIN → VALIDATED (weak)
        - 1 APPROVE, 2 REJECT → REJECTED
        - 0 APPROVE, 3 REJECT → REJECTED (strong)
        - 1 REJECT, 2 ABSTAIN → REJECTED (weak)
        - 1 APPROVE, 2 ABSTAIN → INCONCLUSIVE
        - 0 APPROVE, 0 REJECT, 3 ABSTAIN → INCONCLUSIVE
    """
    if not votes:
        return ValidationResult.INCONCLUSIVE, ConsensusType.NONE, "No votes received"

    approve_count = sum(1 for v in votes if v.get("vote") == Vote.APPROVE.value)
    reject_count = sum(1 for v in votes if v.get("vote") == Vote.REJECT.value)
    abstain_count = sum(1 for v in votes if v.get("vote") == Vote.ABSTAIN.value)

    total_votes = len(votes)

    # Strong consensus: all agree
    if approve_count == total_votes:
        return (
            ValidationResult.VALIDATED,
            ConsensusType.STRONG,
            "Strong consensus: all agents approve"
        )

    if reject_count == total_votes:
        return (
            ValidationResult.REJECTED,
            ConsensusType.STRONG,
            "Strong consensus: all agents reject"
        )

    # 2/3 threshold (excluding abstentions)
    active_votes = total_votes - abstain_count

    if active_votes == 0:
        return (
            ValidationResult.INCONCLUSIVE,
            ConsensusType.NONE,
            "All agents abstained"
        )

    # Need at least 2 votes to reach consensus
    if active_votes < 2:
        return (
            ValidationResult.INCONCLUSIVE,
            ConsensusType.NONE,
            "Insufficient active votes for consensus"
        )

    # Check for 2/3 approval
    if approve_count >= 2:
        if abstain_count > 0:
            return (
                ValidationResult.VALIDATED,
                ConsensusType.WEAK,
                f"Weak consensus: {approve_count} approve, {abstain_count} abstain"
            )
        else:
            return (
                ValidationResult.VALIDATED,
                ConsensusType.WEAK,
                f"Weak consensus: {approve_count} approve, {reject_count} reject"
            )

    # Check for 2/3 rejection
    if reject_count >= 2:
        return (
            ValidationResult.REJECTED,
            ConsensusType.WEAK if abstain_count > 0 else ConsensusType.STRONG,
            f"Rejected: {reject_count} reject, {approve_count} approve"
        )

    # Couldn't reach consensus
    return (
        ValidationResult.INCONCLUSIVE,
        ConsensusType.NONE,
        f"No consensus: {approve_count} approve, {reject_count} reject, {abstain_count} abstain"
    )


def validate_finding(finding: Dict[str, Any], agent_votes: List[Dict[str, Any]]) -> Dict[str, Any]:
    """Validate a finding using validation panel votes.

    Args:
        finding: The finding to validate
        agent_votes: List of votes from validation panel agents

    Returns:
        Dictionary with validation result and details
    """
    result, consensus, recommendation = tally_votes(agent_votes)

    return {
        "finding_id": finding.get("id", "unknown"),
        "validation_result": result.value,
        "consensus_type": consensus.value,
        "votes": agent_votes,
        "recommendation": recommendation,
        "review_required": result == ValidationResult.INCONCLUSIVE,
        "should_fix": result == ValidationResult.VALIDATED,
    }


def batch_validate_findings(
    findings: List[Dict[str, Any]],
    vote_collector: Callable[[Dict[str, Any]], List[Dict[str, Any]]]
) -> List[Dict[str, Any]]:
    """Validate a batch of findings.

    Args:
        findings: List of findings to validate
        vote_collector: Function that collects votes for a finding
            Signature: vote_collector(finding) -> List[Dict[str, Any]]

    Returns:
        List of validation results
    """
    results = []

    for finding in findings:
        try:
            votes = vote_collector(finding)
            result = validate_finding(finding, votes)
            results.append(result)
        except Exception as e:
            results.append({
                "finding_id": finding.get("id", "unknown"),
                "validation_result": ValidationResult.INCONCLUSIVE.value,
                "error": str(e),
                "review_required": True,
            })

    return results


def format_validation_summary(results: List[Dict[str, Any]]) -> str:
    """Format validation results as human-readable summary.

    Args:
        results: List of validation results

    Returns:
        Formatted summary string
    """
    total = len(results)
    validated = sum(1 for r in results if r.get("validation_result") == ValidationResult.VALIDATED.value)
    rejected = sum(1 for r in results if r.get("validation_result") == ValidationResult.REJECTED.value)
    inconclusive = sum(1 for r in results if r.get("validation_result") == ValidationResult.INCONCLUSIVE.value)

    strong_consensus = sum(
        1 for r in results
        if r.get("consensus_type") == ConsensusType.STRONG.value
    )

    lines = [
        "Validation Panel Results",
        "=" * 50,
        f"Total Findings: {total}",
        f"Validated: {validated} ({validated/total*100:.1f}%)",
        f"Rejected: {rejected} ({rejected/total*100:.1f}%)",
        f"Inconclusive: {inconclusive} ({inconclusive/total*100:.1f}%)",
        "",
        f"Strong Consensus: {strong_consensus} ({strong_consensus/total*100:.1f}%)",
        "",
        "Findings requiring fix: " + ", ".join(
            r.get("finding_id", "?")
            for r in results
            if r.get("should_fix", False)
        ) if validated > 0 else "None",
        "",
        "Findings requiring human review: " + ", ".join(
            r.get("finding_id", "?")
            for r in results
            if r.get("review_required", False)
        ) if inconclusive > 0 else "None",
    ]

    return "\n".join(lines)


# Example usage
if __name__ == "__main__":
    # Example finding
    finding = {
        "id": "dep-001",
        "category": "dependency-failures",
        "description": "Exception caught but not logged",
    }

    # Example votes
    votes = [
        {
            "agent": "security",
            "vote": Vote.APPROVE.value,
            "justification": "Creates security blind spot"
        },
        {
            "agent": "architect",
            "vote": Vote.APPROVE.value,
            "justification": "Violates observability principles"
        },
        {
            "agent": "builder",
            "vote": Vote.APPROVE.value,
            "justification": "Simple one-line fix"
        }
    ]

    result = validate_finding(finding, votes)
    print(f"Result: {result['validation_result']}")
    print(f"Consensus: {result['consensus_type']}")
    print(f"Recommendation: {result['recommendation']}")

Vote Pattern Examples

Pattern Matrix

Decision Tree

Start
├─ All agents agree?
│  ├─ Yes (AAA) → VALIDATED (strong) ✓
│  └─ Yes (RRR) → REJECTED (strong) ✗
│
├─ 2 or more APPROVE?
│  ├─ Yes (AAR or AAB) → VALIDATED (weak) ✓
│  └─ No → Continue
│
├─ 2 or more REJECT?
│  ├─ Yes (ARR or RRB) → REJECTED (weak) ✗
│  └─ No → INCONCLUSIVE ⚠
│
└─ Otherwise → INCONCLUSIVE ⚠

Tie-Breaking Rules

Note: With 3 agents, true ties (1-1-1) are rare but possible.

When votes are tied (1 approve, 1 reject, 1 abstain):

1. Result: INCONCLUSIVE 2. Action: Flag for human review 3. Reasoning: Need more information or fourth opinion

Alternative approach (not recommended):

• Use severity as tie-breaker (high severity → approve, low severity → reject)
• Risk: Could override agent judgment

Abstention Guidelines

Agents should abstain when:

1. Insufficient Context

• Code snippet too small
• Missing business logic context
• External dependencies not documented

2. Outside Expertise

• Security agent: Non-security issue
• Architect agent: Low-level implementation detail
• Builder agent: High-level architecture decision

3. Conflicting Information

• Finding description contradicts code
• Multiple interpretations possible
• Unclear what "correct" behavior should be

Important: Abstentions should be rare (< 10% of votes). High abstention rate indicates:

• Poor finding descriptions
• Insufficient context provided
• Agent prompts need refinement

Metrics and Monitoring

Track these metrics over time:

def calculate_panel_metrics(results: List[Dict[str, Any]]) -> Dict[str, float]:
    """Calculate validation panel effectiveness metrics."""
    total = len(results)
    if total == 0:
        return {}

    return {
        "validation_rate": sum(
            1 for r in results
            if r["validation_result"] == "VALIDATED"
        ) / total,

        "rejection_rate": sum(
            1 for r in results
            if r["validation_result"] == "REJECTED"
        ) / total,

        "inconclusive_rate": sum(
            1 for r in results
            if r["validation_result"] == "INCONCLUSIVE"
        ) / total,

        "strong_consensus_rate": sum(
            1 for r in results
            if r["consensus_type"] == "strong"
        ) / total,

        "abstention_rate": sum(
            sum(1 for v in r["votes"] if v["vote"] == "ABSTAIN")
            for r in results
        ) / (total * 3),  # 3 agents per finding
    }

Quality Thresholds

Healthy validation panel:

• Validation rate: 60-80%
• Rejection rate: 15-30%
• Inconclusive rate: < 10%
• Strong consensus rate: > 50%
• Abstention rate: < 10%

Warning signs:

• Validation rate > 95% (rubber stamping)
• Validation rate < 40% (too strict)
• Inconclusive rate > 20% (poor context)
• Abstention rate > 20% (agent tuning needed)