
Code Hardcode Audit
- 144 installs
- 62 repo stars
- Updated August 3, 2026
- terrylica/cc-skills
Use code-hardcode-audit for development tasks
About
code-hardcode-audit: A skill for development. This provides functionality for development workflows.
- code-hardcode-audit
Code Hardcode Audit by the numbers
- 144 all-time installs (skills.sh)
- +1 installs in the week ending Aug 5, 2026 (Skillselion tracking)
- Ranked #2,604 of 4,347 Backend & APIs skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/terrylica/cc-skills --skill code-hardcode-auditAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 144 |
|---|---|
| repo stars | ★ 62 |
| Last updated | August 3, 2026 |
| Repository | terrylica/cc-skills ↗ |
What it does
Use code-hardcode-audit for development tasks
Files
Code Hardcode Audit
Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.
When to Use This Skill
Use this skill when the user mentions:
- "hardcoded values", "hardcodes", "magic numbers"
- "constant detection", "find constants"
- "duplicate constants", "DRY violations"
- "code audit", "hardcode audit"
- "PLR2004", "semgrep", "jscpd", "gitleaks", "ast-grep", "SSoT violations"
- "secret scanning", "leaked secrets", "API keys", "bandit", "trufflehog", "whispers"
- "passwords in code", "credential leaks", "entropy detection"
- "config file secrets", "hardcoded credentials"
Quick Start
# Preflight — verify all tools installed and configured
uv run --python 3.14 --script scripts/preflight.py -- .
# Full audit (all 9 tools, preflight + both outputs)
uv run --python 3.14 --script scripts/audit_hardcodes.py -- src/
# Individual tools (all respect .gitignore):
# Python credential detection (passwords, tokens, API keys in variable names)
uv run --python 3.14 --script scripts/run_bandit.py -- src/
# Entropy-based secret detection (catches secrets regex can't)
uv run --python 3.14 --script scripts/run_trufflehog.py -- src/
# Config file secrets (YAML, JSON, Dockerfile, .env, .properties)
uv run --python 3.14 --script scripts/run_whispers.py -- src/
# AST-based hardcode detection (numeric args, URLs, paths, sleep)
uv run --python 3.14 --script scripts/run_ast_grep.py -- src/
# Python magic numbers only (fastest)
uv run --python 3.14 --script scripts/run_ruff_plr.py -- src/
# Pattern-based detection (URLs, ports, paths, sleep, circuit breaker)
uv run --python 3.14 --script scripts/run_semgrep.py -- src/
# Env-var coverage audit (BaseSettings cross-reference)
uv run --python 3.14 --script scripts/audit_env_coverage.py -- src/
# Copy-paste detection
uv run --python 3.14 --script scripts/run_jscpd.py -- src/
# Regex-based secret scanning (API keys, tokens, passwords)
uv run --python 3.14 --script scripts/run_gitleaks.py -- src/Tool Overview
| Tool | Detection Focus | Language Support | Speed |
|---|---|---|---|
| Preflight | Tool availability + config validation | N/A | Instant |
| Bandit | Hardcoded passwords, tokens in Python (B105-7) | Python | Fast |
| TruffleHog | Entropy-based secret + API verification | Any (file-based) | Medium |
| Whispers | Config file secrets (YAML, JSON, Docker, .env) | Config files | Medium |
| ast-grep | Hardcoded literals in args, sleep, URLs, paths | Multi-language | Fast |
| Ruff PLR2004 | Magic value comparisons | Python | Fast |
| Semgrep | URLs, ports, paths, credentials, retry config | Multi-language | Medium |
| Env-coverage | BaseSettings cross-reference, coverage gaps | Python | Fast |
| jscpd | Duplicate code blocks | Multi-language | Slow |
| gitleaks | Regex-based secrets, API keys, passwords | Any (file-based) | Fast |
Output Formats
JSON (--output json)
{
"summary": {
"total_findings": 42,
"by_tool": { "ruff": 15, "semgrep": 20, "jscpd": 7 },
"by_severity": { "high": 5, "medium": 25, "low": 12 }
},
"findings": [
{
"id": "MAGIC-001",
"tool": "ruff",
"rule": "PLR2004",
"file": "src/config.py",
"line": 42,
"column": 8,
"message": "Magic value used in comparison: 8123",
"severity": "medium",
"suggested_fix": "Extract to named constant"
}
],
"refactoring_plan": [
{
"priority": 1,
"action": "Create constants/ports.py",
"finding_ids": ["MAGIC-001", "MAGIC-003"]
}
]
}Compiler-like Text (--output text)
src/config.py:42:8: PLR2004 Magic value used in comparison: 8123 [ruff]
src/probe.py:15:1: hardcoded-url Hardcoded URL detected [semgrep]
src/client.py:20-35: Clone detected (16 lines, 95% similarity) [jscpd]
Summary: 42 findings (ruff: 15, semgrep: 20, jscpd: 7)CLI Options
--output {json,text,both} Output format (default: both)
--tools {all,ast-grep,ruff,semgrep,jscpd,gitleaks,env-coverage,bandit,trufflehog,whispers} Tools to run
--severity {all,high,medium,low} Filter by severity (default: all)
--exclude PATTERN Glob pattern to exclude (repeatable)
--no-parallel Disable parallel execution
--skip-preflight Skip tool availability checkReferences
- Tool Comparison - Detailed tool capabilities
- Output Schema - JSON schema specification
- Troubleshooting - Common issues and fixes
Related
- ADR-0046: Semantic Constants Abstraction
- ADR-0047: Code Hardcode Audit Skill
code-clone-assistant- PMD CPD-based clone detection (DRY focus)
---
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| Ruff PLR2004 zero output | PLR2004 globally suppressed | Run preflight: uv run --python 3.14 --script scripts/preflight.py -- . |
| Ruff PLR2004 not found | Ruff not installed or old | uv tool install ruff or upgrade |
| ast-grep not found | Binary not installed | cargo install ast-grep or brew install ast-grep |
| Semgrep timeout | Large codebase scan | Use --exclude to limit scope |
| jscpd memory error | Too many files | Increase Node heap: NODE_OPTIONS=--max-old-space-size=4096 |
| gitleaks false positives | Test data flagged | Add patterns to .gitleaks.toml allowlist |
| Env-coverage misses | Not using BaseSettings | Only detects pydantic BaseSettings; other config patterns skipped |
| No findings in output | Wrong directory specified | Verify path exists and contains source files |
| JSON parse error | Tool output malformed | Run tool individually with --output text |
| Missing tool in PATH | Tool not installed globally | Run preflight first, then install missing tools |
| Bandit false positives | password = '' in init | Filter B105 by confidence: --confidence HIGH |
| TruffleHog timeout | Scanning .venv/node_modules | All tools respect .gitignore; ensure large dirs are gitignored |
| TruffleHog regex error | Glob patterns in .gitignore | Complex globs (**/*.rs.bk) are auto-skipped; only simple names used |
| Whispers slow scan | Large directories | Exclude via .gitignore; whispers config auto-generated from it |
| Whispers zero findings | No config files in scope | Whispers targets YAML/JSON/Docker/INI; use on project root, not src/ |
| Severity filter empty | No findings at that level | Use --severity all to see all findings |
Post-Execution Reflection
After this skill completes, check before closing:
1. Did the command succeed? — If not, fix the instruction or error table that caused the failure. 2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match. 3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.
Only update if the issue is real and reproducible — not speculative.
id: hardcoded-numeric-arg
language: python
severity: warning
message: >-
Bare numeric literal in function call argument — extract to named constant.
note: >-
Numeric literals in function calls (timeout=30, retries=3) are magic numbers
that should be named constants or config values for maintainability.
rule:
kind: keyword_argument
has:
kind: integer
field: value
id: hardcoded-path-string
language: python
severity: warning
message: >-
Hardcoded filesystem path — use pathlib or configuration.
note: >-
Absolute paths break across machines and deployments. Use Path objects,
environment variables, or platformdirs for portable paths.
rule:
kind: string
regex: "^[\"']/(tmp|var|etc|home|Users|opt)/"
id: hardcoded-sleep
language: python
severity: warning
message: >-
Hardcoded sleep duration — extract to named constant for tuning.
note: >-
Sleep durations are operational parameters that often need tuning per environment.
Create SLEEP_* or DELAY_* constants backed by env vars.
rule:
any:
- pattern: time.sleep($N)
- pattern: asyncio.sleep($N)
- pattern: await asyncio.sleep($N)
constraints:
N:
kind: integer
id: hardcoded-url-string
language: python
severity: warning
message: >-
Hardcoded URL string — extract to constant or configuration.
note: >-
URLs change across environments (dev/staging/prod). Move to a constants
module or BaseSettings class with env-var override.
rule:
kind: string
regex: "^[\"']https?://"
ruleDirs:
- rules
rules:
# Rule 1: Hardcoded URLs
- id: hardcoded-url
pattern-either:
- pattern: '"http://$..."'
- pattern: '"https://$..."'
- pattern: "'http://$...'"
- pattern: "'https://$...'"
message: "Hardcoded URL detected. Consider extracting to a constant or configuration."
severity: WARNING
languages: [python, javascript, typescript, go, java]
metadata:
category: hardcode
subcategory: url
suggested_fix: "Move to constants module or environment variable"
# Rule 2: Hardcoded Ports (simplified)
- id: hardcoded-port
pattern-either:
- pattern: "port=8123"
- pattern: "port=8443"
- pattern: "port=9000"
- pattern: "port=9440"
- pattern: "port=3000"
- pattern: "port=5000"
- pattern: "port=5432"
- pattern: "port=3306"
- pattern: "port=6379"
- pattern: "port=27017"
message: "Hardcoded port number. Consider extracting to a named constant."
severity: WARNING
languages: [python, javascript, typescript, go, java]
metadata:
category: hardcode
subcategory: port
suggested_fix: "Create PORT_* constant in constants module"
# Rule 3: Hardcoded Timeframe Strings
- id: hardcoded-timeframe
pattern-either:
- pattern: '"1h"'
- pattern: '"4h"'
- pattern: '"1d"'
- pattern: '"1m"'
- pattern: '"15m"'
- pattern: '"30m"'
- pattern: '"1w"'
- pattern: "'1h'"
- pattern: "'4h'"
- pattern: "'1d'"
- pattern: "'1m'"
- pattern: "'15m'"
- pattern: "'30m'"
- pattern: "'1w'"
message: "Hardcoded timeframe string. Consider using a Timeframe type or constant."
severity: INFO
languages: [python, javascript, typescript]
metadata:
category: hardcode
subcategory: timeframe
suggested_fix: "Use TIMEFRAME_* constant or Timeframe enum"
# Rule 4: Hardcoded File Paths
- id: hardcoded-path
pattern-either:
- pattern: '"/tmp/$..."'
- pattern: '"/var/$..."'
- pattern: '"/etc/$..."'
- pattern: '"/home/$..."'
- pattern: '"/Users/$..."'
- pattern: "'/tmp/$...'"
- pattern: "'/var/$...'"
- pattern: "'/etc/$...'"
- pattern: "'/home/$...'"
- pattern: "'/Users/$...'"
message: "Hardcoded file path detected. Consider using pathlib or configuration."
severity: WARNING
languages: [python, javascript, typescript, go]
metadata:
category: hardcode
subcategory: path
suggested_fix: "Use Path objects or environment-based configuration"
# Rule 5: Potential Hardcoded Credentials
- id: hardcoded-credential
patterns:
- pattern-either:
- pattern: 'password="$VALUE"'
- pattern: "password='$VALUE'"
- pattern: 'api_key="$VALUE"'
- pattern: "api_key='$VALUE'"
- pattern: 'secret="$VALUE"'
- pattern: "secret='$VALUE'"
- pattern: 'token="$VALUE"'
- pattern: "token='$VALUE'"
- pattern-not: '$VAR=""'
- pattern-not: "$VAR=''"
message: "Potential hardcoded credential. Use environment variables or secrets manager."
severity: ERROR
languages: [python, javascript, typescript]
metadata:
category: security
subcategory: credential
suggested_fix: "Use os.environ, Doppler, or secrets manager"
# Rule 6: Hardcoded Retry Configuration (simplified)
- id: hardcoded-retry-config
pattern-either:
- pattern: "max_retries=3"
- pattern: "max_retries=5"
- pattern: "retry_count=3"
- pattern: "max_attempts=3"
- pattern: "max_attempts=5"
- pattern: "timeout=30"
- pattern: "timeout=60"
- pattern: "timeout=120"
- pattern: "delay=1"
- pattern: "delay=5"
message: "Hardcoded retry/timeout configuration. Consider extracting to constants."
severity: INFO
languages: [python, javascript, typescript]
metadata:
category: hardcode
subcategory: retry
suggested_fix: "Create RETRY_* or TIMEOUT_* constants"
# Rule 7: Hardcoded API Limits (simplified)
- id: hardcoded-api-limit
pattern-either:
- pattern: "limit=100"
- pattern: "limit=1000"
- pattern: "page_size=100"
- pattern: "batch_size=100"
- pattern: "batch_size=1000"
- pattern: "chunk_size=1000"
- pattern: "max_results=100"
- pattern: "max_results=1000"
message: "Hardcoded API limit. Consider extracting to a named constant."
severity: INFO
languages: [python, javascript, typescript]
metadata:
category: hardcode
subcategory: api_limit
suggested_fix: "Create API_CHUNK_SIZE or similar constant"
# Rule 8: Hardcoded Sleep Durations
- id: hardcoded-sleep-duration
pattern-either:
- pattern: time.sleep($N)
- pattern: asyncio.sleep($N)
- pattern: await asyncio.sleep($N)
message: "Hardcoded sleep duration. Extract to a named SLEEP_* or DELAY_* constant."
severity: INFO
languages: [python]
metadata:
category: hardcode
subcategory: sleep
suggested_fix: "Create SLEEP_* or DELAY_* constant backed by env var"
# Rule 9: Hardcoded Circuit Breaker Config
- id: hardcoded-circuit-breaker
pattern-either:
- pattern: CircuitBreaker(failure_threshold=$N, ...)
- pattern: CircuitBreaker(..., recovery_timeout=$N, ...)
message: "Hardcoded circuit breaker config. Extract to constants or BaseSettings."
severity: WARNING
languages: [python]
metadata:
category: hardcode
subcategory: circuit_breaker
suggested_fix: "Create CB_FAILURE_THRESHOLD, CB_RECOVERY_TIMEOUT constants"
# Rule 10: Hardcoded Stamina Retry
- id: hardcoded-stamina-retry
pattern-either:
- pattern: stamina.retry(attempts=$N, ...)
- pattern: "@stamina.retry(attempts=$N, ...)"
message: "Hardcoded stamina retry attempts. Extract to configuration."
severity: INFO
languages: [python]
metadata:
category: hardcode
subcategory: retry
suggested_fix: "Create RETRY_ATTEMPTS constant or add to BaseSettings"
# Rule 11: Hardcoded Lock/Redis Timeout
- id: hardcoded-lock-timeout
pattern-either:
- pattern: blocking_timeout=$N
- pattern: lock_timeout=$N
- pattern: "redis.lock($KEY, timeout=$N, ...)"
- pattern: "redis.lock($KEY, ..., blocking_timeout=$N, ...)"
message: "Hardcoded lock/Redis timeout. Extract to configuration."
severity: WARNING
languages: [python]
metadata:
category: hardcode
subcategory: lock_timeout
suggested_fix: "Create LOCK_TIMEOUT constant or add to BaseSettings"
# Rule 12: Hardcoded Pydantic Field Default
- id: hardcoded-pydantic-field-default
pattern-either:
- pattern: "Field(default=$N, ...)"
- pattern: "Field($N, ...)"
message: "Hardcoded Pydantic Field default. Consider BaseSettings with env-var override."
severity: INFO
languages: [python]
metadata:
category: hardcode
subcategory: pydantic_default
suggested_fix: "Use BaseSettings with env-var override or Field(default_factory=...)"
Evolution Log
Convention: Reverse chronological order (newest on top, oldest at bottom). Prepend new entries.
---
2026-02-26: Initial Evolution Log
Status: Skill is in use and maintained. Track improvements here.
Purpose
This evolution log tracks updates to the skill. Each entry should note:
- What changed (content, structure, tooling)
- Why it changed (bug fix, feature request, best practice)
- Files affected
How to Use
1. When updating SKILL.md or references, add an entry here with the date 2. Keep entries reverse-chronological (newest first) 3. Link to ADRs or GitHub issues when relevant 4. Reference specific line changes when helpful
---
Skill: Code Hardcode Audit
Output Schema
JSON Schema
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"summary": {
"type": "object",
"properties": {
"total_findings": { "type": "integer" },
"by_tool": {
"type": "object",
"additionalProperties": { "type": "integer" }
},
"by_severity": {
"type": "object",
"additionalProperties": { "type": "integer" }
}
}
},
"findings": {
"type": "array",
"items": { "$ref": "#/definitions/Finding" }
},
"errors": {
"type": "array",
"items": { "type": "string" }
}
},
"definitions": {
"Finding": {
"type": "object",
"required": ["id", "tool", "rule", "file", "line"],
"properties": {
"id": { "type": "string", "pattern": "^(RUFF|SGRP|JSCPD)-[0-9]{3}$" },
"tool": { "enum": ["ruff", "semgrep", "jscpd"] },
"rule": { "type": "string" },
"file": { "type": "string" },
"line": { "type": "integer" },
"column": { "type": "integer" },
"end_line": { "type": ["integer", "null"] },
"message": { "type": "string" },
"severity": { "enum": ["high", "medium", "low"] },
"suggested_fix": { "type": "string" }
}
}
}
}Example Output
Full Audit (JSON)
{
"summary": {
"total_findings": 5,
"by_tool": {
"ruff": 2,
"semgrep": 2,
"jscpd": 1
},
"by_severity": {
"high": 1,
"medium": 3,
"low": 1
}
},
"findings": [
{
"id": "RUFF-001",
"tool": "ruff",
"rule": "PLR2004",
"file": "src/config.py",
"line": 42,
"column": 8,
"message": "Magic value used in comparison: 8123",
"severity": "medium",
"suggested_fix": "Extract to named constant"
},
{
"id": "SGRP-001",
"tool": "semgrep",
"rule": "hardcoded-credential",
"file": "src/client.py",
"line": 15,
"column": 1,
"message": "Potential hardcoded credential. Use environment variables.",
"severity": "high",
"suggested_fix": "Use os.environ, Doppler, or secrets manager"
},
{
"id": "JSCPD-001",
"tool": "jscpd",
"rule": "duplicate-code",
"file": "src/handlers/a.py",
"line": 20,
"end_line": 45,
"message": "Clone detected with src/handlers/b.py (25 lines)",
"severity": "low",
"suggested_fix": "Extract to shared function or module"
}
],
"errors": []
}Text Output
src/config.py:42:8: PLR2004 Magic value used in comparison: 8123 [ruff]
src/client.py:15:1: hardcoded-credential Potential hardcoded credential [semgrep]
src/handlers/a.py:20-45: duplicate-code Clone detected (25 lines) [jscpd]
Summary: 5 findings (ruff: 2, semgrep: 2, jscpd: 1)Finding ID Format
| Prefix | Tool | Example |
|---|---|---|
RUFF- | Ruff PLR2004 | RUFF-001 |
SGRP- | Semgrep | SGRP-001 |
JSCPD- | jscpd | JSCPD-001 |
Severity Levels
| Level | Meaning | Action |
|---|---|---|
high | Security risk or critical issue | Fix immediately |
medium | Code quality issue | Fix in current sprint |
low | Minor improvement | Track for later |
Tool-Specific Severity Mapping
| Tool | Default Severity | Notes |
|---|---|---|
| Ruff | medium | Magic numbers are quality issues |
| Semgrep | Varies by rule | Credentials = high, timeframes = low |
| jscpd | low | Duplicates are refactoring candidates |
Skill: Code Hardcode Audit
Tool Comparison
Overview
| Tool | Detection Focus | Language Support | Speed | Install |
|---|---|---|---|---|
| Ruff PLR2004 | Magic value comparisons | Python only | Fast | uv tool install ruff |
| Semgrep | Pattern-based (URLs, ports, credentials) | Multi-language | Medium | brew install semgrep |
| jscpd | Duplicate code blocks | Multi-language | Slow | npx jscpd (on-demand) |
Detection Capabilities
Ruff PLR2004
Detects: Magic numbers in comparisons
# DETECTED
if timeout > 30: # PLR2004: Magic value 30
if port == 8123: # PLR2004: Magic value 8123
# NOT DETECTED (by design)
DEFAULT_TIMEOUT = 30 # Assignment, not comparisonLimitations:
- Python only
- Only comparisons, not assignments
- Doesn't detect string literals
Semgrep (Custom Rules)
Detects: 7 pattern categories
| Rule ID | Detects | Severity |
|---|---|---|
hardcoded-url | HTTP/HTTPS URLs | WARNING |
hardcoded-port | Port numbers | WARNING |
hardcoded-timeframe | "1h", "4h", "1d" strings | INFO |
hardcoded-path | /tmp, /var, /home paths | WARNING |
hardcoded-credential | password=, api_key=, token= | ERROR |
hardcoded-retry-config | max_retries=, timeout= | INFO |
hardcoded-api-limit | limit=, batch_size= | INFO |
Limitations:
- Requires rule tuning to reduce false positives
- Pattern matching may miss obfuscated values
jscpd
Detects: Copy-paste code blocks (DRY violations)
# DETECTED: Identical blocks across files
def process_a():
data = fetch()
validate(data)
transform(data)
return data
def process_b(): # Clone of process_a
data = fetch()
validate(data)
transform(data)
return dataLimitations:
- Slower than other tools (full AST parsing)
- Requires Node.js (available via mise)
- High threshold to avoid false positives
Complementary Coverage
┌─────────────────────────────────────────────────────────────┐
│ Hardcode Detection │
├─────────────────┬─────────────────┬─────────────────────────┤
│ Ruff PLR2004 │ Semgrep │ jscpd │
│ │ │ │
│ Magic numbers │ URLs, ports │ Duplicate blocks │
│ in comparisons │ paths, creds │ (any language) │
│ │ timeframes │ │
│ Python only │ Multi-language │ Multi-language │
└─────────────────┴─────────────────┴─────────────────────────┘When to Use Each Tool
| Scenario | Recommended Tool |
|---|---|
| Quick Python magic number check | Ruff alone |
| Security audit (credentials, URLs) | Semgrep alone |
| DRY violation detection | jscpd alone |
| Comprehensive audit | All three (orchestrator) |
| CI/CD integration | Ruff + Semgrep (faster) |
Skill: Code Hardcode Audit
Troubleshooting
Tool Not Found Errors
ruff not found
Error: ruff not foundFix: Install Ruff globally with uv:
uv tool install ruffsemgrep not found
Error: semgrep not foundFix: Install Semgrep with Homebrew:
brew install semgrepnpx not found
Error: npx not foundFix: Install Node.js via mise:
mise install node
mise use --global nodeSemgrep Issues
Rules file not found
Error: Semgrep rules not found: /path/to/assets/semgrep-hardcode-rules.yamlCause: Running script from wrong location or rules file missing.
Fix: Verify rules file exists:
/usr/bin/env bash << 'TROUBLESHOOTING_SCRIPT_EOF'
# Environment-agnostic path
PLUGIN_DIR="${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/cc-skills/plugins/itp}"
ls "$PLUGIN_DIR/skills/code-hardcode-audit/assets/semgrep-hardcode-rules.yaml"
TROUBLESHOOTING_SCRIPT_EOFToo many false positives
Cause: Default rules are broad for maximum detection.
Fix: Customize rules by editing assets/semgrep-hardcode-rules.yaml:
# Add exclusions
patterns:
- pattern: '"http://$..."'
# Exclude test files
- pattern-not-inside: |
def test_$...():
...Semgrep timeout
Error: Semgrep timed outCause: Large codebase or complex rules.
Fix: Use --exclude to skip large directories:
uv run --script audit_hardcodes.py -- src/ --exclude "node_modules" --exclude ".venv"jscpd Issues
jscpd timeout
Error: jscpd timed out after 5 minutesCause: Very large codebase.
Fix:
1. Exclude non-essential directories 2. Run jscpd separately on smaller directories
uv run --script run_jscpd.py -- src/core/No duplicates found (false negative)
Cause: Default threshold too high.
Fix: Lower detection threshold in jscpd config. Create .jscpd.json:
{
"threshold": 5,
"minLines": 3,
"minTokens": 25
}Node.js version mismatch
Error: jscpd requires Node.js >= 16Fix: Update Node.js via mise:
mise install node
mise use --global nodeRuff Issues
No findings (false negative)
Cause: PLR2004 only detects magic numbers in comparisons.
# NOT detected (assignments)
TIMEOUT = 30
port = 8123
# DETECTED (comparisons)
if timeout > 30:
if port == 8123:Fix: Use Semgrep for broader detection of hardcoded values in assignments.
Ruff version compatibility
Error: Unknown rule: PLR2004Cause: Old Ruff version.
Fix: Update Ruff:
uv tool install --upgrade ruffGeneral Issues
Permission denied
Error: Permission denied: /path/to/fileFix: Check file permissions or run with appropriate user:
chmod -R u+r /path/to/directoryOut of memory
Cause: Very large codebase causing memory exhaustion.
Fix:
1. Use --no-parallel to run tools sequentially 2. Process directories individually
uv run --script audit_hardcodes.py -- src/ --no-parallelJSON parse error
Error: Error parsing outputCause: Tool produced invalid JSON (often mixed with warnings).
Fix:
1. Check tool stderr for warnings 2. Run tool individually to isolate the issue
ruff check --select PLR2004 --output-format json src/ 2>&1Getting Help
1. Check tool-specific documentation:
2. Report skill issues:
- Check ADR-0047 for design decisions
- Review the code-hardcode-audit SKILL.md
# /// script
# requires-python = ">=3.14"
# dependencies = []
# ///
"""Env-var coverage audit: cross-reference pydantic BaseSettings vs bare constants.
Usage:
uv run --python 3.14 --script audit_env_coverage.py -- <path> [--output {json,text}]
Finds module-level constants and inline literals that lack env-var backing
via pydantic BaseSettings. Reports coverage gaps.
"""
import argparse
import ast
import json
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
# Patterns for inline keyword args that should be configurable
CONFIGURABLE_KWARGS = frozenset({
"timeout", "max_retries", "retries", "attempts", "max_attempts",
"delay", "backoff", "interval", "port", "batch_size", "chunk_size",
"limit", "max_results", "page_size", "threshold", "max_failures",
"workers", "max_workers", "ttl", "budget", "cooldown",
"blocking_timeout", "lock_timeout", "recovery_timeout",
"failure_threshold",
})
ALL_CAPS_RE = re.compile(r"^[A-Z][A-Z0-9_]{2,}$")
@dataclass
class SettingsField:
"""A field from a pydantic BaseSettings class."""
class_name: str
field_name: str
file: str
line: int
@dataclass
class BareConstant:
"""A module-level ALL_CAPS constant with a literal value."""
name: str
value: str
file: str
line: int
@dataclass
class InlineLiteral:
"""A numeric literal in a configurable keyword argument."""
kwarg: str
value: str
file: str
line: int
func_name: str = ""
def _extract_settings_fields(tree: ast.Module, filepath: str) -> list[SettingsField]:
"""Find all fields in BaseSettings subclasses."""
fields = []
for node in ast.walk(tree):
if not isinstance(node, ast.ClassDef):
continue
# Check if any base is BaseSettings (simple name check)
is_settings = any(
(isinstance(b, ast.Name) and b.id == "BaseSettings")
or (isinstance(b, ast.Attribute) and b.attr == "BaseSettings")
for b in node.bases
)
if not is_settings:
continue
for item in node.body:
if isinstance(item, ast.AnnAssign) and isinstance(item.target, ast.Name):
fields.append(SettingsField(
class_name=node.name,
field_name=item.target.id,
file=filepath,
line=item.lineno,
))
elif isinstance(item, ast.Assign):
for target in item.targets:
if isinstance(target, ast.Name):
fields.append(SettingsField(
class_name=node.name,
field_name=target.id,
file=filepath,
line=item.lineno,
))
return fields
def _extract_bare_constants(tree: ast.Module, filepath: str) -> list[BareConstant]:
"""Find module-level ALL_CAPS = literal assignments."""
constants = []
for node in tree.body:
if not isinstance(node, ast.Assign):
continue
for target in node.targets:
if not isinstance(target, ast.Name):
continue
if not ALL_CAPS_RE.match(target.id):
continue
if isinstance(node.value, ast.Constant):
constants.append(BareConstant(
name=target.id,
value=repr(node.value.value),
file=filepath,
line=node.lineno,
))
return constants
def _extract_inline_literals(tree: ast.Module, filepath: str) -> list[InlineLiteral]:
"""Find numeric literals in configurable keyword arguments."""
literals = []
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
func_name = ""
if isinstance(node.func, ast.Name):
func_name = node.func.id
elif isinstance(node.func, ast.Attribute):
func_name = node.func.attr
for kw in node.keywords:
if kw.arg and kw.arg.lower() in CONFIGURABLE_KWARGS:
if isinstance(kw.value, ast.Constant) and isinstance(kw.value.value, (int, float)):
literals.append(InlineLiteral(
kwarg=kw.arg,
value=repr(kw.value.value),
file=filepath,
line=kw.value.lineno,
func_name=func_name,
))
return literals
def audit_env_coverage(target: Path, output_format: str = "text") -> int:
"""Scan Python files and report env-var coverage gaps."""
all_settings: list[SettingsField] = []
all_constants: list[BareConstant] = []
all_literals: list[InlineLiteral] = []
# Collect all .py files
if target.is_file():
py_files = [target] if target.suffix == ".py" else []
else:
py_files = sorted(target.rglob("*.py"))
for py_file in py_files:
try:
source = py_file.read_text(encoding="utf-8")
tree = ast.parse(source, filename=str(py_file))
except (SyntaxError, UnicodeDecodeError):
continue
rel = str(py_file)
all_settings.extend(_extract_settings_fields(tree, rel))
all_constants.extend(_extract_bare_constants(tree, rel))
all_literals.extend(_extract_inline_literals(tree, rel))
# Build set of env-backed field names (case-insensitive)
env_backed = {f.field_name.lower() for f in all_settings}
# Classify constants
findings = []
finding_id = 0
for const in all_constants:
name_lower = const.name.lower()
# Strip common prefixes for matching (_DEFAULT_, _MAX_, etc.)
stripped = re.sub(r"^_?(DEFAULT_|MAX_|MIN_)?", "", name_lower)
has_backing = name_lower in env_backed or stripped in env_backed
if not has_backing:
finding_id += 1
findings.append({
"id": f"ENVCOV-{finding_id:03d}",
"tool": "env-coverage",
"rule": "constant-no-env-backing",
"file": const.file,
"line": const.line,
"column": 0,
"end_line": None,
"message": f"Constant {const.name}={const.value} has no BaseSettings env-var override",
"severity": "medium",
"suggested_fix": f"Add {const.name.lower()} field to a BaseSettings class",
})
for lit in all_literals:
finding_id += 1
ctx = f" in {lit.func_name}()" if lit.func_name else ""
findings.append({
"id": f"ENVCOV-{finding_id:03d}",
"tool": "env-coverage",
"rule": "inline-literal",
"file": lit.file,
"line": lit.line,
"column": 0,
"end_line": None,
"message": f"Inline literal {lit.kwarg}={lit.value}{ctx} — extract to named constant",
"severity": "high",
"suggested_fix": f"Create {lit.kwarg.upper()} constant or add to BaseSettings",
})
# Sort by file, line
findings.sort(key=lambda f: (f["file"], f["line"]))
if output_format == "json":
by_rule: dict[str, int] = {}
for f in findings:
by_rule[f["rule"]] = by_rule.get(f["rule"], 0) + 1
output = {
"tool": "env-coverage",
"summary": {
"total_settings_classes": len({f.class_name for f in all_settings}),
"total_settings_fields": len(all_settings),
"total_bare_constants": len(all_constants),
"total_inline_literals": len(all_literals),
"total_findings": len(findings),
"by_rule": by_rule,
},
"findings": findings,
}
print(json.dumps(output, indent=2))
else:
if not findings:
print("No env-var coverage gaps detected")
print(f" BaseSettings classes: {len({f.class_name for f in all_settings})}")
print(f" Env-backed fields: {len(all_settings)}")
print(f" Module constants: {len(all_constants)}")
else:
for f in findings:
print(f"{f['file']}:{f['line']}: {f['rule']} {f['message']} [env-coverage]")
print(f"\nSummary: {len(findings)} coverage gap(s)")
print(f" BaseSettings fields: {len(all_settings)}")
print(f" Bare constants without env backing: {sum(1 for f in findings if f['rule'] == 'constant-no-env-backing')}")
print(f" Inline literals: {sum(1 for f in findings if f['rule'] == 'inline-literal')}")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Audit env-var coverage for Python constants")
parser.add_argument("path", type=Path, help="Path to audit")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return audit_env_coverage(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""Orchestrator for code hardcode audit combining 9 tools.
Usage:
uv run --script audit_hardcodes.py -- <path> [options]
Options:
--output {json,text,both} Output format (default: both)
--tools {all,ruff,semgrep,jscpd,gitleaks,ast-grep,env-coverage} Tools to run
--severity {all,high,medium,low} Filter by severity (default: all)
--exclude PATTERN Glob pattern to exclude (repeatable)
--no-parallel Disable parallel execution
--skip-preflight Skip tool availability check
"""
import argparse
import json
import os
import subprocess
import sys
import tempfile
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
# ADR: 2025-12-08-mise-env-centralized-config
# Configuration via environment variables with defaults for backward compatibility
AUDIT_PARALLEL_WORKERS = int(os.environ.get("AUDIT_PARALLEL_WORKERS", "4"))
AUDIT_JSCPD_TIMEOUT = int(os.environ.get("AUDIT_JSCPD_TIMEOUT", "300"))
AUDIT_GITLEAKS_TIMEOUT = int(os.environ.get("AUDIT_GITLEAKS_TIMEOUT", "120"))
AUDIT_ASTGREP_TIMEOUT = int(os.environ.get("AUDIT_ASTGREP_TIMEOUT", "60"))
AUDIT_BANDIT_TIMEOUT = int(os.environ.get("AUDIT_BANDIT_TIMEOUT", "120"))
AUDIT_TRUFFLEHOG_TIMEOUT = int(os.environ.get("AUDIT_TRUFFLEHOG_TIMEOUT", "300"))
AUDIT_WHISPERS_TIMEOUT = int(os.environ.get("AUDIT_WHISPERS_TIMEOUT", "120"))
@dataclass
class Finding:
"""A single finding from any tool."""
id: str
tool: str
rule: str
file: str
line: int
column: int = 0
end_line: int | None = None
message: str = ""
severity: str = "medium"
suggested_fix: str = ""
def to_dict(self) -> dict[str, Any]:
return {
"id": self.id,
"tool": self.tool,
"rule": self.rule,
"file": self.file,
"line": self.line,
"column": self.column,
"end_line": self.end_line,
"message": self.message,
"severity": self.severity,
"suggested_fix": self.suggested_fix,
}
def to_text(self) -> str:
loc = f"{self.file}:{self.line}"
if self.column:
loc += f":{self.column}"
return f"{loc}: {self.rule} {self.message} [{self.tool}]"
@dataclass
class AuditResult:
"""Aggregated results from all tools."""
findings: list[Finding] = field(default_factory=list)
errors: list[str] = field(default_factory=list)
def add_finding(self, finding: Finding) -> None:
self.findings.append(finding)
def add_error(self, error: str) -> None:
self.errors.append(error)
def summary(self) -> dict[str, Any]:
by_tool: dict[str, int] = {}
by_severity: dict[str, int] = {}
for f in self.findings:
by_tool[f.tool] = by_tool.get(f.tool, 0) + 1
by_severity[f.severity] = by_severity.get(f.severity, 0) + 1
return {
"total_findings": len(self.findings),
"by_tool": by_tool,
"by_severity": by_severity,
}
def to_json(self) -> str:
return json.dumps(
{
"summary": self.summary(),
"findings": [f.to_dict() for f in self.findings],
"errors": self.errors,
},
indent=2,
)
def to_text(self) -> str:
lines = [f.to_text() for f in self.findings]
summary = self.summary()
tool_counts = ", ".join(f"{k}: {v}" for k, v in summary["by_tool"].items())
lines.append("")
lines.append(f"Summary: {summary['total_findings']} findings ({tool_counts})")
if self.errors:
lines.append(f"Errors: {len(self.errors)}")
for e in self.errors:
lines.append(f" - {e}")
return "\n".join(lines)
def run_ruff(target: Path, excludes: list[str]) -> list[Finding]:
"""Run Ruff PLR2004 check and return findings."""
cmd = ["ruff", "check", "--select", "PLR2004", "--output-format", "json", str(target)]
for pattern in excludes:
cmd.extend(["--exclude", pattern])
try:
result = subprocess.run(cmd, capture_output=True, text=True)
# Ruff returns exit code 1 when findings exist
if result.stdout:
data = json.loads(result.stdout)
findings = []
for i, item in enumerate(data):
findings.append(
Finding(
id=f"RUFF-{i + 1:03d}",
tool="ruff",
rule=item.get("code", "PLR2004"),
file=item.get("filename", ""),
line=item.get("location", {}).get("row", 0),
column=item.get("location", {}).get("column", 0),
message=item.get("message", ""),
severity="medium",
suggested_fix="Extract to named constant",
)
)
return findings
except (json.JSONDecodeError, FileNotFoundError) as e:
print(f"ruff error: {e}", file=sys.stderr)
return []
def _extract_rule_id(check_id: str) -> str:
"""Extract clean rule ID from Semgrep's path-based check_id.
Semgrep generates check_ids like:
'Users.terryli..claude.skills.code-hardcode-audit.assets.hardcoded-timeframe'
This extracts just 'hardcoded-timeframe'.
"""
# Split by dots and find the actual rule name (after 'assets' or last segment)
parts = check_id.split(".")
# Look for 'assets' marker, take everything after it
if "assets" in parts:
idx = parts.index("assets")
return ".".join(parts[idx + 1 :]) if idx + 1 < len(parts) else check_id
# Fallback: return last segment
return parts[-1] if parts else check_id
def run_semgrep(target: Path, excludes: list[str]) -> list[Finding]:
"""Run Semgrep with custom rules and return findings."""
rules_path = Path(__file__).parent.parent / "assets" / "semgrep-hardcode-rules.yaml"
if not rules_path.exists():
print(f"semgrep rules not found: {rules_path}", file=sys.stderr)
return []
cmd = ["semgrep", "--config", str(rules_path), "--json", str(target)]
for pattern in excludes:
cmd.extend(["--exclude", pattern])
try:
result = subprocess.run(cmd, capture_output=True, text=True)
if result.stdout:
data = json.loads(result.stdout)
findings = []
for i, item in enumerate(data.get("results", [])):
severity_map = {"ERROR": "high", "WARNING": "medium", "INFO": "low"}
raw_check_id = item.get("check_id", "")
findings.append(
Finding(
id=f"SGRP-{i + 1:03d}",
tool="semgrep",
rule=_extract_rule_id(raw_check_id),
file=item.get("path", ""),
line=item.get("start", {}).get("line", 0),
column=item.get("start", {}).get("col", 0),
end_line=item.get("end", {}).get("line"),
message=item.get("extra", {}).get("message", ""),
severity=severity_map.get(
item.get("extra", {}).get("severity", "WARNING"), "medium"
),
suggested_fix=item.get("extra", {})
.get("metadata", {})
.get("suggested_fix", ""),
)
)
return findings
except (json.JSONDecodeError, FileNotFoundError) as e:
print(f"semgrep error: {e}", file=sys.stderr)
return []
def run_jscpd(target: Path, excludes: list[str]) -> list[Finding]:
"""Run jscpd via npx and return findings."""
with tempfile.TemporaryDirectory() as tmpdir:
output_dir = Path(tmpdir)
cmd = ["npx", "jscpd", "--reporters", "json", "--output", str(output_dir), str(target)]
for pattern in excludes:
cmd.extend(["--ignore", pattern])
try:
subprocess.run(cmd, capture_output=True, text=True, timeout=AUDIT_JSCPD_TIMEOUT)
report_path = output_dir / "jscpd-report.json"
if report_path.exists():
data = json.loads(report_path.read_text())
findings = []
for i, dup in enumerate(data.get("duplicates", [])):
first = dup.get("firstFile", {})
second = dup.get("secondFile", {})
findings.append(
Finding(
id=f"JSCPD-{i + 1:03d}",
tool="jscpd",
rule="duplicate-code",
file=first.get("name", ""),
line=first.get("startLoc", {}).get("line", 0),
end_line=first.get("endLoc", {}).get("line"),
message=f"Clone detected with {second.get('name', '')} "
f"({dup.get('lines', 0)} lines, "
f"{dup.get('fragment', '')})",
severity="low",
suggested_fix="Extract to shared function or module",
)
)
return findings
except (json.JSONDecodeError, FileNotFoundError, subprocess.TimeoutExpired) as e:
print(f"jscpd error: {e}", file=sys.stderr)
return []
def run_gitleaks(target: Path, excludes: list[str]) -> list[Finding]:
"""Run gitleaks for secret detection and return findings.
Uses modern 'dir' command for directory scanning (v8.19.0+).
Exit codes: 0 = clean, 1 = secrets found (expected), 2+ = error.
"""
cmd = [
"gitleaks",
"dir",
str(target),
"--report-format",
"json",
"--report-path",
"/dev/stdout",
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=AUDIT_GITLEAKS_TIMEOUT)
# Exit code 1 = secrets found (expected, not an error)
if result.returncode in (0, 1) and result.stdout.strip():
try:
data = json.loads(result.stdout)
findings = []
for i, item in enumerate(data):
findings.append(
Finding(
id=f"GITLEAKS-{i + 1:03d}",
tool="gitleaks",
rule=item.get("RuleID", "secret"),
file=item.get("File", ""),
line=item.get("StartLine", 0),
end_line=item.get("EndLine"),
message=f"Secret detected: {item.get('Match', '')[:30]}...",
severity="high", # Secrets are always high severity
suggested_fix="Remove secret and rotate credentials",
)
)
return findings
except json.JSONDecodeError:
pass
return []
except FileNotFoundError:
print("gitleaks not found. Install with: mise use --global gitleaks", file=sys.stderr)
return []
except subprocess.TimeoutExpired:
print(f"gitleaks timed out after {AUDIT_GITLEAKS_TIMEOUT} seconds", file=sys.stderr)
return []
def run_ast_grep(target: Path, excludes: list[str]) -> list[Finding]:
"""Run ast-grep with hardcode detection rules and return findings."""
import shutil
rules_dir = Path(__file__).parent.parent / "assets" / "ast-grep-hardcode"
if not rules_dir.exists():
print(f"ast-grep rules not found: {rules_dir}", file=sys.stderr)
return []
sg = None
for name in ("sg", "ast-grep"):
if shutil.which(name):
sg = name
break
if not sg:
print("ast-grep not found. Install with: cargo install ast-grep", file=sys.stderr)
return []
cmd = [sg, "scan", str(target.resolve()), "--json=stream"]
try:
result = subprocess.run(
cmd, capture_output=True, text=True,
timeout=AUDIT_ASTGREP_TIMEOUT, cwd=str(rules_dir),
)
findings = []
severity_map = {"error": "high", "warning": "medium", "hint": "low", "info": "low"}
if result.stdout.strip():
for line in result.stdout.strip().splitlines():
if not line.strip():
continue
try:
item = json.loads(line)
except json.JSONDecodeError:
continue
findings.append(Finding(
id=f"ASTGREP-{len(findings) + 1:03d}",
tool="ast-grep",
rule=item.get("ruleId", "unknown"),
file=item.get("file", ""),
line=item.get("range", {}).get("start", {}).get("line", 0),
column=item.get("range", {}).get("start", {}).get("column", 0),
end_line=item.get("range", {}).get("end", {}).get("line"),
message=item.get("message", ""),
severity=severity_map.get(item.get("severity", "warning"), "medium"),
suggested_fix=item.get("note", ""),
))
return findings
except (FileNotFoundError, subprocess.TimeoutExpired) as e:
print(f"ast-grep error: {e}", file=sys.stderr)
return []
def run_env_coverage(target: Path, excludes: list[str]) -> list[Finding]:
"""Run env-coverage audit via subprocess and return findings."""
script = Path(__file__).parent / "audit_env_coverage.py"
if not script.exists():
print(f"env-coverage script not found: {script}", file=sys.stderr)
return []
cmd = ["uv", "run", "--python", "3.14", "--script", str(script), "--", str(target), "--output", "json"]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
if result.stdout.strip():
data = json.loads(result.stdout)
findings = []
for item in data.get("findings", []):
findings.append(Finding(
id=item.get("id", f"ENVCOV-{len(findings) + 1:03d}"),
tool="env-coverage",
rule=item.get("rule", ""),
file=item.get("file", ""),
line=item.get("line", 0),
column=item.get("column", 0),
end_line=item.get("end_line"),
message=item.get("message", ""),
severity=item.get("severity", "medium"),
suggested_fix=item.get("suggested_fix", ""),
))
return findings
except (json.JSONDecodeError, FileNotFoundError, subprocess.TimeoutExpired) as e:
print(f"env-coverage error: {e}", file=sys.stderr)
return []
def _gitignore_excludes(target: Path, regex_safe: bool = False) -> list[str]:
"""Read .gitignore from target and return directory names to exclude.
Args:
regex_safe: If True, skip glob patterns with *, ?, [, { (for trufflehog regex mode).
"""
excludes = []
gitignore = target / ".gitignore"
if gitignore.exists():
for line in gitignore.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#") or line.startswith("!"):
continue
clean = line.rstrip("/")
if regex_safe and any(c in clean for c in ("*", "?", "[", "{")):
continue
if clean:
excludes.append(clean)
# Always exclude .venv even if not in .gitignore
if ".venv" not in excludes:
excludes.append(".venv")
return excludes
def run_bandit(target: Path, excludes: list[str]) -> list[Finding]:
"""Run Bandit B105/B106/B107 for hardcoded credential detection."""
gi_excludes = _gitignore_excludes(target)
bandit_excludes = [str(target / e) for e in gi_excludes]
cmd = [
"bandit", "-r", str(target),
"-t", "B105,B106,B107",
"-f", "json",
"--exclude", ",".join(bandit_excludes),
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=AUDIT_BANDIT_TIMEOUT)
raw = result.stdout or ""
brace_index = raw.find("{")
if brace_index == -1:
return []
data = json.loads(raw[brace_index:])
findings = []
for i, item in enumerate(data.get("results", [])):
findings.append(Finding(
id=f"BANDIT-{i + 1:03d}",
tool="bandit",
rule=item.get("test_id", "B105"),
file=item.get("filename", ""),
line=item.get("line_number", 0),
column=item.get("col_offset", 0),
message=item.get("issue_text", ""),
severity="high", # Credential findings are always high
suggested_fix="Move to environment variable or secret manager",
))
return findings
except (json.JSONDecodeError, FileNotFoundError, subprocess.TimeoutExpired) as e:
print(f"bandit error: {e}", file=sys.stderr)
return []
def run_trufflehog(target: Path, excludes: list[str]) -> list[Finding]:
"""Run TruffleHog for entropy-based secret detection with .gitignore awareness."""
gi_excludes = _gitignore_excludes(target, regex_safe=True)
gi_excludes.append(".git")
# trufflehog --exclude-paths expects a file with patterns
exclude_file = tempfile.NamedTemporaryFile(
mode="w", suffix=".txt", delete=False, prefix="trufflehog-exclude-"
)
try:
exclude_file.write("\n".join(gi_excludes) + "\n")
exclude_file.close()
cmd = [
"trufflehog", "filesystem", str(target),
"--json", "--no-update",
"--exclude-paths", exclude_file.name,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=AUDIT_TRUFFLEHOG_TIMEOUT)
findings = []
if result.returncode == 0 and result.stdout.strip():
for line in result.stdout.strip().splitlines():
if not line.strip():
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
fs = obj.get("SourceMetadata", {}).get("Data", {}).get("Filesystem", {})
verified = obj.get("Verified", False)
findings.append(Finding(
id=f"TRUFFLEHOG-{len(findings) + 1:03d}",
tool="trufflehog",
rule=obj.get("DetectorName", "unknown"),
file=fs.get("file", ""),
line=fs.get("line", 0),
message=f"{obj.get('DetectorName', '')} ({'verified' if verified else 'unverified'}): {obj.get('Raw', '')[:30]}...",
severity="high" if verified else "medium",
suggested_fix="Remove secret and rotate credentials",
))
return findings
except (FileNotFoundError, subprocess.TimeoutExpired) as e:
print(f"trufflehog error: {e}", file=sys.stderr)
return []
finally:
Path(exclude_file.name).unlink(missing_ok=True)
def run_whispers(target: Path, excludes: list[str]) -> list[Finding]:
"""Run Whispers for config-file secret detection with .gitignore awareness."""
gi_excludes = _gitignore_excludes(target)
gi_excludes.append(".git")
# Build whispers config YAML
config_lines = ["exclude:", " files:"]
for p in gi_excludes:
config_lines.append(f' - "{p}/**"')
config_yaml = "\n".join(config_lines) + "\n"
cfg_file = tempfile.NamedTemporaryFile(
mode="w", suffix=".yml", delete=False, prefix="whispers-config-"
)
try:
cfg_file.write(config_yaml)
cfg_file.close()
cmd = ["whispers", str(target), "-j", "-c", cfg_file.name]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=AUDIT_WHISPERS_TIMEOUT)
findings = []
if result.stdout and result.stdout.strip():
data = json.loads(result.stdout)
for i, item in enumerate(data):
findings.append(Finding(
id=f"WHISPERS-{i + 1:03d}",
tool="whispers",
rule=item.get("rule_id", "unknown"),
file=item.get("file", ""),
line=item.get("line", 0),
message=f"{item.get('key', '')}: {item.get('message', '')}",
severity={"Critical": "high", "High": "high", "Medium": "medium", "Low": "low"}.get(
item.get("severity", "Low"), "low"
),
suggested_fix="Move to environment variable or secret manager",
))
return findings
except (json.JSONDecodeError, FileNotFoundError, subprocess.TimeoutExpired) as e:
print(f"whispers error: {e}", file=sys.stderr)
return []
finally:
Path(cfg_file.name).unlink(missing_ok=True)
def run_preflight(target: Path) -> bool:
"""Run preflight check and return True if all tools available."""
script = Path(__file__).parent / "preflight.py"
if not script.exists():
return True # Skip if preflight script not available
cmd = ["uv", "run", "--python", "3.14", "--script", str(script), "--", str(target), "--output", "text"]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
print(result.stdout, end="")
if result.stderr:
print(result.stderr, end="", file=sys.stderr)
return result.returncode == 0
except (FileNotFoundError, subprocess.TimeoutExpired):
return True # Don't block on preflight failures
def run_audit(
target: Path,
tools: list[str],
excludes: list[str],
parallel: bool = True,
) -> AuditResult:
"""Run selected tools and aggregate results."""
result = AuditResult()
tool_funcs = {
"ruff": run_ruff,
"semgrep": run_semgrep,
"jscpd": run_jscpd,
"gitleaks": run_gitleaks,
"ast-grep": run_ast_grep,
"env-coverage": run_env_coverage,
"bandit": run_bandit,
"trufflehog": run_trufflehog,
"whispers": run_whispers,
}
selected = tools if tools != ["all"] else list(tool_funcs.keys())
if parallel:
with ThreadPoolExecutor(max_workers=AUDIT_PARALLEL_WORKERS) as executor:
futures = {
executor.submit(tool_funcs[name], target, excludes): name
for name in selected
if name in tool_funcs
}
for future in as_completed(futures):
name = futures[future]
try:
for finding in future.result():
result.add_finding(finding)
except Exception as e:
result.add_error(f"{name}: {e}")
else:
for name in selected:
if name in tool_funcs:
try:
for finding in tool_funcs[name](target, excludes):
result.add_finding(finding)
except Exception as e:
result.add_error(f"{name}: {e}")
# Sort by file, then line
result.findings.sort(key=lambda f: (f.file, f.line))
return result
def filter_by_severity(result: AuditResult, severity: str) -> AuditResult:
"""Filter findings by severity level."""
if severity == "all":
return result
severity_order = {"high": 0, "medium": 1, "low": 2}
threshold = severity_order.get(severity, 2)
filtered = AuditResult(errors=result.errors)
for f in result.findings:
if severity_order.get(f.severity, 2) <= threshold:
filtered.add_finding(f)
return filtered
def main() -> int:
parser = argparse.ArgumentParser(
description="Audit code for hardcoded values using 9 detection tools"
)
parser.add_argument("path", type=Path, help="Path to audit")
parser.add_argument(
"--output",
choices=["json", "text", "both"],
default="both",
help="Output format",
)
parser.add_argument(
"--tools",
choices=["all", "ruff", "semgrep", "jscpd", "gitleaks", "ast-grep", "env-coverage",
"bandit", "trufflehog", "whispers"],
nargs="+",
default=["all"],
help="Tools to run",
)
parser.add_argument(
"--severity",
choices=["all", "high", "medium", "low"],
default="all",
help="Filter by minimum severity",
)
parser.add_argument(
"--exclude",
action="append",
default=[],
help="Glob pattern to exclude",
)
parser.add_argument(
"--no-parallel",
action="store_true",
help="Disable parallel execution",
)
parser.add_argument(
"--skip-preflight",
action="store_true",
help="Skip tool availability preflight check",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
if not args.skip_preflight:
run_preflight(args.path)
print() # Separator between preflight and audit output
result = run_audit(
target=args.path,
tools=args.tools,
excludes=args.exclude,
parallel=not args.no_parallel,
)
result = filter_by_severity(result, args.severity)
if args.output == "json":
print(result.to_json())
elif args.output == "text":
print(result.to_text())
else: # both
print(result.to_text())
print("\n--- JSON Output ---")
print(result.to_json())
return 0 if not result.errors else 1
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.14"
# dependencies = []
# ///
"""Preflight check for hardcode audit tools.
Usage:
uv run --python 3.14 --script preflight.py -- <path> [--output {json,text}]
Verifies all audit tools are installed and properly configured.
Detects silent misconfigurations like globally suppressed Ruff PLR2004.
"""
import argparse
import json
import shutil
import subprocess
import sys
import tomllib
from dataclasses import dataclass, field
from pathlib import Path
@dataclass
class ToolCheck:
"""Result of checking a single tool."""
name: str
installed: bool
version: str | None = None
issues: list[str] = field(default_factory=list)
install_cmd: str = ""
def to_dict(self) -> dict:
d = {"name": self.name, "installed": self.installed, "version": self.version, "issues": self.issues}
if not self.installed:
d["install_cmd"] = self.install_cmd
return d
def _get_version(cmd: list[str]) -> str | None:
"""Run a version command and extract the version string."""
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=5)
if result.returncode == 0:
# Take first line, strip common prefixes
line = result.stdout.strip().splitlines()[0]
# Extract version-like substring
for part in line.split():
if part[0].isdigit():
return part
return line
except (FileNotFoundError, subprocess.TimeoutExpired, IndexError):
pass
return None
def _find_ruff_config(target: Path) -> Path | None:
"""Walk up from target to find ruff config file."""
search = target.resolve()
if search.is_file():
search = search.parent
while True:
for name in ("ruff.toml", ".ruff.toml", "pyproject.toml"):
candidate = search / name
if candidate.is_file():
return candidate
parent = search.parent
if parent == search:
break
search = parent
return None
def _check_plr2004_suppressed(config_path: Path) -> list[str]:
"""Check if PLR2004 is suppressed in a ruff config file."""
issues = []
try:
data = tomllib.loads(config_path.read_text())
except Exception:
return issues
# pyproject.toml nests under [tool.ruff]
ruff_cfg = data
if config_path.name == "pyproject.toml":
ruff_cfg = data.get("tool", {}).get("ruff", {})
lint = ruff_cfg.get("lint", {})
# Check global ignore list
ignore = lint.get("ignore", [])
for rule in ignore:
if rule in ("PLR2004", "PLR"):
issues.append(f"PLR2004 globally suppressed in {config_path.name} [tool.ruff.lint.ignore]")
break
# Check per-file-ignores
per_file = lint.get("per-file-ignores", {})
for pattern, rules in per_file.items():
if pattern in ("*", "*.py", "**/*.py"):
for rule in rules:
if rule in ("PLR2004", "PLR"):
issues.append(f"PLR2004 suppressed for '{pattern}' in {config_path.name} [tool.ruff.lint.per-file-ignores]")
break
# Check select — if select is specified but doesn't include PLR/PLR2004
select = lint.get("select", None)
if select is not None and "ALL" not in select:
has_plr = any(r in select for r in ("PLR", "PLR2004", "PL"))
if not has_plr:
issues.append(f"PLR2004 not in [tool.ruff.lint.select] in {config_path.name} — rule won't run")
return issues
def check_ruff(target: Path) -> ToolCheck:
"""Check ruff installation and PLR2004 configuration."""
check = ToolCheck(name="ruff", installed=False, install_cmd="uv tool install ruff")
path = shutil.which("ruff")
if not path:
return check
check.installed = True
check.version = _get_version(["ruff", "--version"])
# Check PLR2004 suppression in project config
config = _find_ruff_config(target)
if config:
check.issues.extend(_check_plr2004_suppressed(config))
return check
def check_semgrep() -> ToolCheck:
"""Check semgrep installation."""
check = ToolCheck(name="semgrep", installed=False, install_cmd="brew install semgrep")
if shutil.which("semgrep"):
check.installed = True
check.version = _get_version(["semgrep", "--version"])
return check
def check_jscpd() -> ToolCheck:
"""Check jscpd availability via npx."""
check = ToolCheck(name="jscpd", installed=False, install_cmd="npm install -g jscpd")
if shutil.which("npx"):
check.installed = True
check.version = "via npx"
return check
def check_gitleaks() -> ToolCheck:
"""Check gitleaks installation."""
check = ToolCheck(name="gitleaks", installed=False, install_cmd="mise use --global gitleaks")
if shutil.which("gitleaks"):
check.installed = True
check.version = _get_version(["gitleaks", "version"])
return check
def check_ast_grep() -> ToolCheck:
"""Check ast-grep installation (binary name: sg or ast-grep)."""
check = ToolCheck(name="ast-grep", installed=False, install_cmd="cargo install ast-grep")
for binary in ("sg", "ast-grep"):
if shutil.which(binary):
check.installed = True
check.version = _get_version([binary, "--version"])
break
return check
def check_bandit() -> ToolCheck:
"""Check bandit installation."""
check = ToolCheck(name="bandit", installed=False, install_cmd="uv tool install bandit")
if shutil.which("bandit"):
check.installed = True
check.version = _get_version(["bandit", "--version"])
return check
def check_trufflehog() -> ToolCheck:
"""Check trufflehog installation."""
check = ToolCheck(name="trufflehog", installed=False, install_cmd="brew install trufflehog")
if shutil.which("trufflehog"):
check.installed = True
check.version = _get_version(["trufflehog", "--version"])
return check
def check_whispers() -> ToolCheck:
"""Check whispers installation."""
check = ToolCheck(name="whispers", installed=False, install_cmd="uv tool install whispers")
if shutil.which("whispers"):
check.installed = True
check.version = _get_version(["whispers", "--version"])
return check
def run_preflight(target: Path, output_format: str = "text") -> int:
"""Run all preflight checks and report results."""
checks = [
check_ruff(target),
check_semgrep(),
check_jscpd(),
check_gitleaks(),
check_ast_grep(),
check_bandit(),
check_trufflehog(),
check_whispers(),
]
has_missing = any(not c.installed for c in checks)
has_issues = any(c.issues for c in checks)
if has_missing:
status = "fail"
elif has_issues:
status = "warn"
else:
status = "pass"
if output_format == "json":
output = {
"status": status,
"tools": [c.to_dict() for c in checks],
}
print(json.dumps(output, indent=2))
else:
print("=== Hardcode Audit Preflight ===\n")
for c in checks:
if c.installed:
icon = "✓" if not c.issues else "⚠"
ver = f" ({c.version})" if c.version else ""
print(f" {icon} {c.name}{ver}")
for issue in c.issues:
print(f" ⚠ {issue}")
else:
print(f" ✗ {c.name} — not found")
print(f" Install: {c.install_cmd}")
print()
if status == "pass":
print("Status: PASS — all tools available and configured")
elif status == "warn":
print("Status: WARN — tools available but configuration issues detected")
else:
print("Status: FAIL — missing required tools")
return 0 if status in ("pass", "warn") else 1
def main() -> int:
parser = argparse.ArgumentParser(description="Preflight check for hardcode audit tools")
parser.add_argument("path", type=Path, help="Target path (used to find ruff config)")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_preflight(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.14"
# dependencies = []
# ///
"""ast-grep wrapper for AST-based hardcode detection.
Usage:
uv run --python 3.14 --script run_ast_grep.py -- <path> [--output {json,text}]
Detects hardcoded literals in assignments, function arguments, return values,
URLs, and file paths using AST pattern matching (not regex).
"""
import argparse
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
RULES_DIR = Path(__file__).parent.parent / "assets" / "ast-grep-hardcode"
TIMEOUT = int(os.environ.get("AUDIT_ASTGREP_TIMEOUT", "60"))
def _find_sg_binary() -> str | None:
"""Find ast-grep binary (sg or ast-grep)."""
for name in ("sg", "ast-grep"):
if shutil.which(name):
return name
return None
def run_ast_grep(target: Path, output_format: str = "text") -> int:
"""Run ast-grep with hardcode detection rules."""
sg = _find_sg_binary()
if not sg:
print("Error: ast-grep not found. Install with: cargo install ast-grep", file=sys.stderr)
return 1
if not RULES_DIR.exists():
print(f"Error: ast-grep rules not found: {RULES_DIR}", file=sys.stderr)
return 1
cmd = [sg, "scan", str(target.resolve()), "--json=stream"]
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=TIMEOUT,
cwd=str(RULES_DIR),
)
# Parse NDJSON (one JSON object per line)
findings = []
if result.stdout.strip():
for line in result.stdout.strip().splitlines():
line = line.strip()
if not line:
continue
try:
item = json.loads(line)
findings.append(item)
except json.JSONDecodeError:
continue
if output_format == "json":
severity_map = {"error": "high", "warning": "medium", "hint": "low", "info": "low"}
output_findings = []
for i, item in enumerate(findings):
output_findings.append({
"id": f"ASTGREP-{i + 1:03d}",
"tool": "ast-grep",
"rule": item.get("ruleId", "unknown"),
"file": item.get("file", ""),
"line": item.get("range", {}).get("start", {}).get("line", 0),
"column": item.get("range", {}).get("start", {}).get("column", 0),
"end_line": item.get("range", {}).get("end", {}).get("line"),
"message": item.get("message", ""),
"severity": severity_map.get(item.get("severity", "warning"), "medium"),
"suggested_fix": item.get("note", ""),
})
output = {
"tool": "ast-grep",
"rules_dir": str(RULES_DIR),
"total_findings": len(output_findings),
"findings": output_findings,
}
print(json.dumps(output, indent=2))
else:
if not findings:
print("No hardcode patterns detected (ast-grep)")
else:
for item in findings:
file = item.get("file", "?")
line = item.get("range", {}).get("start", {}).get("line", 0)
rule = item.get("ruleId", "unknown")
msg = item.get("message", "")
print(f"{file}:{line}: {rule} {msg} [ast-grep]")
print(f"\nTotal: {len(findings)} finding(s)")
return 0
except FileNotFoundError:
print("Error: ast-grep not found. Install with: cargo install ast-grep", file=sys.stderr)
return 1
except subprocess.TimeoutExpired:
print(f"Error: ast-grep timed out after {TIMEOUT} seconds", file=sys.stderr)
return 1
except json.JSONDecodeError as e:
print(f"Error parsing ast-grep output: {e}", file=sys.stderr)
return 1
def main() -> int:
parser = argparse.ArgumentParser(description="Run ast-grep AST-based hardcode detection")
parser.add_argument("path", type=Path, help="Path to check")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_ast_grep(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""Bandit hardcoded password/secret detection wrapper.
Usage:
uv run --script run_bandit.py -- <path> [--output {json,text}]
Detects hardcoded passwords and secrets (B105, B106, B107) in Python code.
"""
import argparse
import json
import subprocess
import sys
from pathlib import Path
def _gitignore_excludes(target: Path) -> list[str]:
"""Read .gitignore and return exclude paths for bandit."""
gitignore = target / ".gitignore"
excludes = [str(target / ".venv")] # Always exclude .venv
if gitignore.exists():
for line in gitignore.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
# Convert gitignore patterns to absolute paths for bandit
clean = line.rstrip("/")
if clean:
excludes.append(str(target / clean))
return excludes
def run_bandit(target: Path, output_format: str = "text") -> int:
"""Run Bandit B105/B106/B107 check."""
excludes = _gitignore_excludes(target)
cmd = [
"bandit",
"-r",
str(target),
"-t",
"B105,B106,B107",
"-f",
"json",
"--exclude",
",".join(excludes),
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
# Bandit returns exit code 1 when findings exist — not an error
raw = result.stdout or ""
# Bandit JSON output may have a non-JSON prefix; find the first `{`
brace_index = raw.find("{")
if brace_index == -1:
if output_format == "json":
output = {
"tool": "bandit",
"rule": "B105-B107",
"total_findings": 0,
"findings": [],
}
print(json.dumps(output, indent=2))
if result.stderr:
print(result.stderr, file=sys.stderr)
return 0
json_text = raw[brace_index:]
data = json.loads(json_text)
results = data.get("results", [])
if output_format == "json":
output = {
"tool": "bandit",
"rule": "B105-B107",
"total_findings": len(results),
"findings": results,
}
print(json.dumps(output, indent=2))
else:
for finding in results:
filename = finding.get("filename", "")
line_number = finding.get("line_number", 0)
test_id = finding.get("test_id", "")
issue_text = finding.get("issue_text", "")
print(f"{filename}:{line_number}: {test_id} - {issue_text} [bandit]")
if result.stderr:
print(result.stderr, file=sys.stderr)
return 0 if not results else result.returncode
except FileNotFoundError:
print("Error: bandit not found. Install with: uv tool install bandit", file=sys.stderr)
return 1
except subprocess.TimeoutExpired:
print("Error: bandit timed out after 120 seconds", file=sys.stderr)
return 1
except json.JSONDecodeError as e:
print(f"Error parsing bandit output: {e}", file=sys.stderr)
return 1
def main() -> int:
parser = argparse.ArgumentParser(description="Run Bandit hardcoded secret detection")
parser.add_argument("path", type=Path, help="Path to check")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_bandit(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""Gitleaks wrapper for secret detection.
Usage:
uv run --script run_gitleaks.py -- <path> [--output {json,text}]
Examples:
uv run --script run_gitleaks.py -- src/
uv run --script run_gitleaks.py -- . --output json
"""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
from pathlib import Path
def run_gitleaks(target: Path, output_format: str = "text") -> int:
"""Run gitleaks on target directory.
Args:
target: Directory to scan
output_format: Output format (json or text)
Returns:
Exit code (0 = no secrets, 1 = secrets found or error)
"""
# Use modern 'dir' command for directory scanning (v8.19.0+)
cmd = [
"gitleaks",
"dir",
str(target),
"--report-format",
"json",
"--report-path",
"/dev/stdout",
"--verbose",
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
# Exit code 0 = clean, 1 = secrets found, 2+ = error
if result.returncode == 0:
if output_format == "json":
output = {
"tool": "gitleaks",
"rule": "secret-detection",
"total_findings": 0,
"findings": [],
}
print(json.dumps(output, indent=2))
else:
print("No secrets detected")
return 0
elif result.returncode == 1:
# Secrets found - parse JSON output
findings = []
if result.stdout.strip():
try:
findings = json.loads(result.stdout)
except json.JSONDecodeError:
pass
if output_format == "json":
output = {
"tool": "gitleaks",
"rule": "secret-detection",
"total_findings": len(findings),
"findings": findings,
}
print(json.dumps(output, indent=2))
else:
# Text format - compiler-style output
for f in findings:
file_path = f.get("File", "unknown")
line = f.get("StartLine", 0)
rule_id = f.get("RuleID", "secret")
match_text = f.get("Match", "")[:50] # Truncate for safety
print(f"{file_path}:{line}: {rule_id} - {match_text}... [gitleaks]")
print(f"\nTotal: {len(findings)} secret(s) detected")
return 1
else:
# Tool error
print(
f"gitleaks error (exit {result.returncode}): {result.stderr}",
file=sys.stderr,
)
return result.returncode
except FileNotFoundError:
print(
"Error: gitleaks not found. Install with: mise use --global gitleaks",
file=sys.stderr,
)
return 1
except subprocess.TimeoutExpired:
print("Error: gitleaks timed out after 120 seconds", file=sys.stderr)
return 1
def main() -> int:
"""CLI entry point."""
parser = argparse.ArgumentParser(description="Run gitleaks for secret detection")
parser.add_argument("path", type=Path, help="Directory to scan")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format (default: text)",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_gitleaks(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""jscpd wrapper for copy-paste detection.
Usage:
uv run --script run_jscpd.py -- <path> [--output {json,text}]
Detects duplicate code blocks using jscpd via npx.
Requires Node.js (available via mise).
"""
import argparse
import json
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
def run_jscpd(target: Path, output_format: str = "text") -> int:
"""Run jscpd via npx for duplicate detection."""
# Check npx availability
if not shutil.which("npx"):
print(
"Error: npx not found. Install Node.js via mise: mise install node",
file=sys.stderr,
)
return 1
with tempfile.TemporaryDirectory() as tmpdir:
output_dir = Path(tmpdir)
cmd = [
"npx",
"jscpd",
"--reporters",
"json",
"--output",
str(output_dir),
str(target),
]
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=300,
)
report_path = output_dir / "jscpd-report.json"
if report_path.exists():
data = json.loads(report_path.read_text())
if output_format == "json":
output = {
"tool": "jscpd",
"total_findings": len(data.get("duplicates", [])),
"statistics": data.get("statistics", {}),
"findings": data.get("duplicates", []),
}
print(json.dumps(output, indent=2))
else:
# Text format
duplicates = data.get("duplicates", [])
if duplicates:
print(f"Found {len(duplicates)} duplicate(s):\n")
for i, dup in enumerate(duplicates, 1):
first = dup.get("firstFile", {})
second = dup.get("secondFile", {})
lines = dup.get("lines", 0)
print(f"{i}. {first.get('name', '')}:{first.get('startLoc', {}).get('line', 0)}-{first.get('endLoc', {}).get('line', 0)}")
print(f" Clone of: {second.get('name', '')}:{second.get('startLoc', {}).get('line', 0)}-{second.get('endLoc', {}).get('line', 0)}")
print(f" Lines: {lines}\n")
else:
print("No duplicates found.")
# Summary
stats = data.get("statistics", {})
total = stats.get("total", {})
print("\nSummary:")
print(f" Files analyzed: {total.get('sources', 0)}")
print(f" Lines analyzed: {total.get('lines', 0)}")
print(f" Duplicates: {total.get('clones', 0)}")
print(f" Duplicate lines: {total.get('duplicatedLines', 0)}")
percentage = total.get("percentage", 0)
print(f" Duplication: {percentage:.2f}%")
return 0
else:
if result.stderr:
print(result.stderr, file=sys.stderr)
print("No jscpd report generated.", file=sys.stderr)
return 1
except subprocess.TimeoutExpired:
print("Error: jscpd timed out after 5 minutes", file=sys.stderr)
return 1
except json.JSONDecodeError as e:
print(f"Error parsing jscpd output: {e}", file=sys.stderr)
return 1
def main() -> int:
parser = argparse.ArgumentParser(description="Run jscpd copy-paste detection")
parser.add_argument("path", type=Path, help="Path to check")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_jscpd(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""Ruff PLR2004 wrapper for magic number detection.
Usage:
uv run --script run_ruff_plr.py -- <path> [--output {json,text}]
Detects magic value comparisons in Python code.
"""
import argparse
import json
import subprocess
import sys
from pathlib import Path
def run_ruff_plr(target: Path, output_format: str = "text") -> int:
"""Run Ruff PLR2004 check."""
ruff_format = "json" if output_format == "json" else "concise"
cmd = [
"ruff",
"check",
"--select",
"PLR2004",
"--output-format",
ruff_format,
str(target),
]
try:
result = subprocess.run(cmd, capture_output=True, text=True)
if output_format == "json" and result.stdout:
# Wrap in standard schema
data = json.loads(result.stdout)
output = {
"tool": "ruff",
"rule": "PLR2004",
"total_findings": len(data),
"findings": data,
}
print(json.dumps(output, indent=2))
else:
if result.stdout:
print(result.stdout)
if result.stderr:
print(result.stderr, file=sys.stderr)
return result.returncode
except FileNotFoundError:
print("Error: ruff not found. Install with: uv tool install ruff", file=sys.stderr)
return 1
except json.JSONDecodeError as e:
print(f"Error parsing ruff output: {e}", file=sys.stderr)
return 1
def main() -> int:
parser = argparse.ArgumentParser(description="Run Ruff PLR2004 magic number detection")
parser.add_argument("path", type=Path, help="Path to check")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_ruff_plr(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""Semgrep wrapper for pattern-based hardcode detection.
Usage:
uv run --script run_semgrep.py -- <path> [--output {json,text}]
Detects hardcoded URLs, ports, paths, credentials, and API limits.
"""
import argparse
import json
import subprocess
import sys
from pathlib import Path
def run_semgrep(target: Path, output_format: str = "text") -> int:
"""Run Semgrep with custom hardcode rules."""
rules_path = Path(__file__).parent.parent / "assets" / "semgrep-hardcode-rules.yaml"
if not rules_path.exists():
print(f"Error: Semgrep rules not found: {rules_path}", file=sys.stderr)
return 1
cmd = [
"semgrep",
"--config",
str(rules_path),
str(target),
]
if output_format == "json":
cmd.append("--json")
try:
result = subprocess.run(cmd, capture_output=True, text=True)
if output_format == "json" and result.stdout:
# Wrap in standard schema
data = json.loads(result.stdout)
output = {
"tool": "semgrep",
"rules_file": str(rules_path),
"total_findings": len(data.get("results", [])),
"findings": data.get("results", []),
"errors": data.get("errors", []),
}
print(json.dumps(output, indent=2))
else:
if result.stdout:
print(result.stdout)
if result.stderr:
# Filter out semgrep info messages
for line in result.stderr.splitlines():
if not line.startswith("Scanning"):
print(line, file=sys.stderr)
return 0 if result.returncode in (0, 1) else result.returncode
except FileNotFoundError:
print(
"Error: semgrep not found. Install with: brew install semgrep",
file=sys.stderr,
)
return 1
except json.JSONDecodeError as e:
print(f"Error parsing semgrep output: {e}", file=sys.stderr)
return 1
def main() -> int:
parser = argparse.ArgumentParser(
description="Run Semgrep pattern-based hardcode detection"
)
parser.add_argument("path", type=Path, help="Path to check")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_semgrep(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""Trufflehog wrapper for secret detection.
Usage:
uv run --script run_trufflehog.py -- <path> [--output {json,text}]
Examples:
uv run --script run_trufflehog.py -- src/
uv run --script run_trufflehog.py -- . --output json
"""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import tempfile
from pathlib import Path
# Always exclude these (even if not in .gitignore)
_ALWAYS_EXCLUDE = [".git"]
def run_trufflehog(target: Path, output_format: str = "text") -> int:
"""Run trufflehog on target directory.
Args:
target: Directory to scan
output_format: Output format (json or text)
Returns:
Exit code (0 = no secrets or findings present, 1 = error)
"""
# Build exclude patterns from .gitignore + always-exclude
# NOTE: trufflehog --exclude-paths expects REGEX, not globs
patterns = list(_ALWAYS_EXCLUDE)
gitignore = target / ".gitignore"
if gitignore.exists():
for line in gitignore.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#") or line.startswith("!"):
continue
clean = line.rstrip("/")
# Skip complex globs that can't be trivially converted to regex
if any(c in clean for c in ("*", "?", "[", "{")):
continue
# Simple directory/file names become regex patterns
if clean:
patterns.append(clean)
# Write exclude patterns to a temp file (trufflehog --exclude-paths expects a file)
exclude_file = tempfile.NamedTemporaryFile(
mode="w", suffix=".txt", delete=False, prefix="trufflehog-exclude-"
)
try:
exclude_file.write("\n".join(patterns) + "\n")
exclude_file.close()
cmd = [
"trufflehog",
"filesystem",
str(target),
"--json",
"--no-update",
"--exclude-paths",
exclude_file.name,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
# trufflehog exit code 0 = success (findings may or may not exist);
# check stdout for NDJSON lines to determine if secrets were found.
if result.returncode != 0:
print(
f"trufflehog error (exit {result.returncode}): {result.stderr}",
file=sys.stderr,
)
return result.returncode
# Parse NDJSON output (one JSON object per line)
findings = []
for line in result.stdout.splitlines():
line = line.strip()
if not line:
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
# Extract fields from trufflehog NDJSON structure
source_meta = obj.get("SourceMetadata", {})
fs_data = source_meta.get("Data", {}).get("Filesystem", {})
file_path = fs_data.get("file", "unknown")
line_num = fs_data.get("line", 0)
detector_name = obj.get("DetectorName", "unknown")
verified = obj.get("Verified", False)
raw = obj.get("Raw", "")[:30] # Truncate to 30 chars for safety
description = obj.get("DetectorDescription", "")
findings.append(
{
"file": file_path,
"line": line_num,
"detector": detector_name,
"verified": verified,
"raw_truncated": raw,
"description": description,
}
)
if output_format == "json":
output = {
"tool": "trufflehog",
"rule": "secret-detection-entropy",
"total_findings": len(findings),
"findings": findings,
}
print(json.dumps(output, indent=2))
else:
if not findings:
print("No secrets detected")
else:
for f in findings:
verified_label = "verified" if f["verified"] else "unverified"
description = f["description"]
desc_part = f" - {description}" if description else ""
print(
f"{f['file']}:{f['line']}: {f['detector']}"
f" ({verified_label}){desc_part} [trufflehog]"
)
print(f"\nTotal: {len(findings)} secret(s) detected")
return 1 if findings else 0
return 1 if findings else 0
except FileNotFoundError:
print(
"Error: trufflehog not found. Install with: brew install trufflehog",
file=sys.stderr,
)
return 1
except subprocess.TimeoutExpired:
print("Error: trufflehog timed out after 300 seconds", file=sys.stderr)
return 1
finally:
Path(exclude_file.name).unlink(missing_ok=True)
def main() -> int:
"""CLI entry point."""
parser = argparse.ArgumentParser(description="Run trufflehog for secret detection")
parser.add_argument("path", type=Path, help="Directory to scan")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format (default: text)",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_trufflehog(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""Whispers wrapper for secret/credential detection.
Usage:
uv run --script run_whispers.py -- <path> [--output {json,text}]
Detects hardcoded secrets, credentials, API keys, and sensitive values.
"""
import argparse
import json
import subprocess
import sys
import tempfile
from pathlib import Path
def _build_whispers_config(target: Path) -> str:
"""Build whispers config YAML with .gitignore-aware exclusions."""
patterns = [".git/**"] # Always exclude .git
gitignore = target / ".gitignore"
if gitignore.exists():
for line in gitignore.read_text().splitlines():
line = line.strip()
if line and not line.startswith("#") and not line.startswith("!"):
clean = line.rstrip("/")
if clean:
patterns.append(f"{clean}/**")
lines = ["exclude:", " files:"]
for p in patterns:
lines.append(f' - "{p}"')
return "\n".join(lines) + "\n"
def run_whispers(target: Path, output_format: str = "text") -> int:
"""Run Whispers secret detection with .gitignore-aware exclusions."""
config_yaml = _build_whispers_config(target)
with tempfile.NamedTemporaryFile(
mode="w",
suffix=".yml",
prefix="whispers_config_",
delete=False,
) as cfg_file:
cfg_file.write(config_yaml)
cfg_path = cfg_file.name
cmd = [
"whispers",
str(target),
"-j",
"-c",
cfg_path,
]
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=120,
)
if output_format == "json":
if result.stdout and result.stdout.strip():
findings = json.loads(result.stdout)
else:
findings = []
output = {
"tool": "whispers",
"rule": "config-secret-detection",
"total_findings": len(findings),
"findings": findings,
}
print(json.dumps(output, indent=2))
else:
if result.stdout and result.stdout.strip():
findings = json.loads(result.stdout)
for finding in findings:
file_path = finding.get("file", "")
line = finding.get("line", "")
rule_id = finding.get("rule_id", "")
key = finding.get("key", "")
value = str(finding.get("value", ""))
truncated_value = value[:30] if len(value) > 30 else value
print(f"{file_path}:{line}: {rule_id} - {key}={truncated_value} [whispers]")
if result.stderr:
for line in result.stderr.splitlines():
print(line, file=sys.stderr)
return 0 if result.returncode in (0, 1) else result.returncode
except FileNotFoundError:
print(
"Error: whispers not found. Install with: uv tool install whispers",
file=sys.stderr,
)
return 1
except subprocess.TimeoutExpired:
print(
"Error: whispers timed out after 120 seconds.",
file=sys.stderr,
)
return 1
except json.JSONDecodeError as e:
print(f"Error parsing whispers output: {e}", file=sys.stderr)
return 1
finally:
Path(cfg_path).unlink(missing_ok=True)
def main() -> int:
parser = argparse.ArgumentParser(
description="Run Whispers secret/credential detection"
)
parser.add_argument("path", type=Path, help="Path to check")
parser.add_argument(
"--output",
choices=["json", "text"],
default="text",
help="Output format",
)
# uv run --script <file> -- <args> passes literal '--' to the script
argv = [a for a in sys.argv[1:] if a != "--"]
args = parser.parse_args(argv)
if not args.path.exists():
print(f"Error: Path does not exist: {args.path}", file=sys.stderr)
return 1
return run_whispers(args.path, args.output)
if __name__ == "__main__":
sys.exit(main())