Deep Dive Analysis

Deep Dive Analysis Skill

Overview

This skill combines mechanical structure extraction with Claude's semantic understanding to produce comprehensive codebase documentation. Unlike simple AST parsing, this skill captures:

• WHAT the code does (structure, functions, classes)
• WHY it exists (business purpose, design decisions)
• HOW it integrates (dependencies, contracts, flows)
• CONSEQUENCES of changes (side effects, failure modes)

Language Support

.sql files are disambiguated against PL/SQL by inspecting content for Oracle-specific markers (CREATE OR REPLACE PACKAGE, DBMS_OUTPUT, %TYPE, %ROWTYPE, UTL_FILE, PRAGMA AUTONOMOUS, etc.). PostgreSQL plpgsql is correctly classified as SQL.

Prerequisites

The scripts are designed to work out of the box with just the Python stdlib. Tree-sitter is optional and improves accuracy for Java / JavaScript / TypeScript:

# Optional: install for higher-fidelity parsing
pip install -r scripts/requirements.txt
# or
uv pip install -r scripts/requirements.txt

What changes when tree-sitter is installed:

• Java: nested classes, generic type parameters, annotations, multi-line declarations parsed correctly. Without it, the regex fallback still finds top-level classes, methods, imports, and constants.
• JavaScript / TypeScript: arrow functions in object/class properties, decorators, template literals, JSX elements parsed correctly. Without it, the regex fallback handles top-level declarations, ES6 import/export, and CommonJS require.
• Rust: lifetimes, generic bounds (where clauses), impl blocks with trait bounds, attribute macros parsed correctly. Without it, the regex fallback still finds top-level fns, structs/enums/traits/impls/mods, use declarations, and UPPER_CASE constants.
• Python / SQL / PL-SQL: no change. Python always uses stdlib ast; SQL/PL-SQL always use the regex DDL extractor.

The active parser is reported in ParseResult.notes and in the CLI output: parser=stdlib-ast, parser=tree-sitter, or parser=regex-fallback.

Capabilities

Mechanical Analysis (Scripts):

• Extract code structure (classes, functions, imports)
• Map dependencies (internal/external)
• Find symbol usages across the codebase
• Track analysis progress
• Classify files by criticality

Semantic Analysis (Claude AI):

• Recognize architectural and design patterns
• Identify red flags and anti-patterns
• Trace data and control flows
• Document contracts and invariants
• Assess quality and maintainability

Documentation Maintenance:

• Review and maintain documentation (Phase 8)
• Fix broken links and update navigation indexes
• Analyze and rewrite code comments (antirez standards)

Use this skill when:

• Analyzing a codebase you're unfamiliar with
• Generating documentation that explains WHY, not just WHAT
• Identifying architectural patterns and anti-patterns
• Performing code review with semantic understanding
• Onboarding to a new project

Prerequisites

This skill is invoked by the /deep-dive-analysis command. The command creates and manages state automatically in .deep-dive/ under the target directory:

1. `.deep-dive/state.json` -- phase tracking (auto-created by the command) 2. `.deep-dive/<phase-number>-<name>.md` -- per-phase output documents

The legacy standalone flow using analysis_progress.json and DEEP_DIVE_PLAN.md at project root is no longer the primary path -- prefer invoking /deep-dive-analysis <target>.

CRITICAL PRINCIPLE: ABSOLUTE SOURCE OF TRUTH

THE DOCUMENTATION GENERATED BY THIS SKILL IS THE ABSOLUTE AND UNQUESTIONABLE SOURCE OF TRUTH FOR YOUR PROJECT.

>

ANY INFORMATION NOT VERIFIED WITH IRREFUTABLE EVIDENCE FROM SOURCE CODE IS FALSE, UNRELIABLE, AND UNACCEPTABLE.

Mandatory Rules (VIOLATION = FAILURE)

1. NEVER document anything without reading the actual source code first 2. NEVER assume any existing documentation, comment, or docstring is accurate 3. NEVER write documentation based on memory, inference, or "what should be" 4. ALWAYS derive truth EXCLUSIVELY from reading and tracing actual code 5. ALWAYS provide source file + qualified symbol name for every technical claim 6. ALWAYS verify state machines, enums, constants against actual definitions 7. TREAT all pre-existing docs as unverified claims requiring validation 8. MARK any unverifiable statement as [UNVERIFIED - REQUIRES CODE CHECK] 9. USE qualified symbol names in markers (file.py::Class.method), never line numbers -- line numbers break on any edit

See references/analysis-templates.md for the full verification trust model, temporal purity principle, and documentation status markers.

Output Usage Guide

After analysis completes, consult the right file for your task:

Forbidden Files

The analysis NEVER reads or includes contents from sensitive files: .env, .env.*, credentials.*, secrets.*, *.pem, *.key, *.p12, *.pfx, id_rsa*, id_ed25519*, .npmrc, .pypirc, .netrc, or any file containing API keys, passwords, or tokens. If encountered, note file existence only - never quote contents.

Available Commands

1. Analyze Single File

# Python
python .claude/skills/deep-dive-analysis/scripts/analyze_file.py \
  --file src/utils/circuit_breaker.py \
  --output-format markdown

# Java
python .claude/skills/deep-dive-analysis/scripts/analyze_file.py \
  --file src/main/java/com/example/UserService.java

# TypeScript
python .claude/skills/deep-dive-analysis/scripts/analyze_file.py \
  --file src/services/auth.ts

# SQL / PL-SQL
python .claude/skills/deep-dive-analysis/scripts/analyze_file.py \
  --file migrations/0042_users.sql

python .claude/skills/deep-dive-analysis/scripts/analyze_file.py \
  --file packages/user_pkg.pkb

Parameters:

• --file / -f: Relative path to file - REQUIRED. Any supported extension (see Language Support table).
• --output-format / -o: Output format (json, markdown, summary) - default: summary
• --find-usages / -u: Find all usages of exported symbols - default: false
• --update-progress / -p: Update analysis_progress.json - default: false

2. Check Progress

python .claude/skills/deep-dive-analysis/scripts/check_progress.py \
  --phase 1 --status pending

3. Find Usages

python .claude/skills/deep-dive-analysis/scripts/analyze_file.py \
  --symbol CircuitBreaker --file src/utils/circuit_breaker.py

4. Generate Phase Report

python .claude/skills/deep-dive-analysis/scripts/analyze_file.py \
  --phase 1 --output-format markdown --output-file docs/01_domains/COMMON_LIBRARY.md

---

Phase 8: Documentation Review Commands

5. Scan Documentation Health

python .claude/skills/deep-dive-analysis/scripts/doc_review.py scan \
  --path docs/ --output doc_health_report.json

6. Validate Links

python .claude/skills/deep-dive-analysis/scripts/doc_review.py validate-links \
  --path docs/ --fix

7. Verify Against Source Code

python .claude/skills/deep-dive-analysis/scripts/doc_review.py verify \
  --doc docs/agents/lifecycle.md --source src/agents/lifecycle.py

8. Update Navigation Indexes

python .claude/skills/deep-dive-analysis/scripts/doc_review.py update-indexes \
  --search-index docs/00_navigation/SEARCH_INDEX.md \
  --by-domain docs/00_navigation/BY_DOMAIN.md

9. Full Documentation Maintenance

python .claude/skills/deep-dive-analysis/scripts/doc_review.py full-maintenance \
  --path docs/ --auto-fix --output doc_health_report.json

Executes: scan health, validate/fix links, identify obsolete files, update indexes, generate report.

---

Comment Quality Commands (Antirez Standards)

10. Analyze Comment Quality

python .claude/skills/deep-dive-analysis/scripts/rewrite_comments.py analyze \
  src/main.py --report

11. Scan Directory for Comment Issues

python .claude/skills/deep-dive-analysis/scripts/rewrite_comments.py scan \
  src/ --recursive --issues-only

12. Generate Comment Health Report

python .claude/skills/deep-dive-analysis/scripts/rewrite_comments.py report \
  src/ --output comment_health.md

13. Rewrite Comments

python .claude/skills/deep-dive-analysis/scripts/rewrite_comments.py rewrite \
  src/main.py --apply --backup

14. View Standards Reference

python .claude/skills/deep-dive-analysis/scripts/rewrite_comments.py standards

---

File Classification Criteria

---

AI-Powered Semantic Analysis

Five Layers of Understanding

Pattern Recognition

Red Flags to Identify

ARCHITECTURE:
- GOD CLASS: >10 public methods or >500 LOC
- CIRCULAR DEPENDENCY: A -> B -> C -> A
- LEAKY ABSTRACTION: Implementation details in interface

RELIABILITY:
- SWALLOWED EXCEPTION: Empty catch blocks
- MISSING TIMEOUT: Network calls without timeout
- RACE CONDITION: Shared mutable state without sync

SECURITY:
- HARDCODED SECRET: Passwords, API keys in code
- SQL INJECTION: String concatenation in queries
- MISSING VALIDATION: Unsanitized user input

AI Analysis Workflow

1. SCRIPTS RUN FIRST -> classifier.py, ast_parser.py, usage_finder.py
2. CLAUDE ANALYZES -> Read source, apply semantic questions, recognize patterns, identify red flags
3. CLAUDE DOCUMENTS -> Use template, explain WHY not just WHAT, document contracts
4. VERIFY -> Check against runtime behavior, validate with code traces

Analysis Loop Workflow

1. CLASSIFY -> LOC, dependencies, critical patterns, assign classification
2. READ & MAP -> AST structure, classes, functions, constants, state mutations
3. DEPENDENCY CHECK -> Internal imports, external imports, external calls
4. CONTEXT ANALYSIS -> Symbol usages, importing modules, message flows
5. RUNTIME VERIFICATION (Critical/High-Complexity) -> Log analysis, flow verification
6. DOCUMENTATION -> Update progress, generate report, cross-reference

Best Practices

Source Code Analysis (Phases 1-7)

1. Start with Phase 1 - foundation modules inform everything else 2. Track progress with --update-progress 3. Never skip runtime verification for critical/high-complexity files 4. Cross-reference with CONTEXT.md after analysis

Documentation Maintenance (Phase 8)

1. Run scan first to understand current state 2. Fix links before content - broken links indicate structural issues 3. Verify against code before updating documentation 4. Update indexes last to reflect final state

Team Mode Integration

The classic /deep-dive-analysis command runs three subagents in two waves on a single target. For monorepos, multi-language repos, or large codebases where a single deep-dive's context window grows uncomfortable, switch to the team variant:

/agent-teams:team-deep-dive <target>

The team command: 1. Auto-detects partitions (workspaces, top-level dirs, or language clusters) and asks you to confirm at a checkpoint. 2. Spawns three deep-dive workers per partition in two waves (Wave 1 = Structure; Wave 2 = Behavior + Quality). Wave 2 workers read every partition's Wave 1 output, so cross-partition contracts and flows can be cited directly. 3. Synthesizes a backward-compatible .deep-dive/01..07.md set that any downstream consumer (/agent-teams:team-review, /codebase-mapper:map-codebase, /project-setup:create-claude-md) can pick up without changes. 4. Adds .deep-dive/08-interconnect-map.md produced by senior-review:semantic-interconnect-mapper on top of the consolidated set, giving a global Call Graph, Contracts, Invariants, and Integration Hot-Spots view.

Choosing between classic and team

Output layout (team mode)

.deep-dive/
├── state.json
├── partitions/
│   └── <name>/{01..06}.md          ← per-partition reports
├── 01-structure.md .. 07-final-report.md   ← consolidated (compat with classic)
└── 08-interconnect-map.md          ← new, global cross-partition map

See docs/plans/2026-05-20-deep-dive-team-mode-design.md for the full architecture.

References

• references/analysis-templates.md - Verification trust model, temporal purity principle, documentation status markers, comment classification, maintenance workflows
• references/AI_ANALYSIS_METHODOLOGY.md - Complete analysis methodology
• references/SEMANTIC_PATTERNS.md - Pattern recognition guide
• references/ANTIREZ_COMMENTING_STANDARDS.md - Comment taxonomy
• references/DEEP_DIVE_PLAN.md - Master analysis plan with all phase definitions
• templates/semantic_analysis.md - AI-powered per-file analysis template
• templates/analysis_report.md - Module-level report template

Resources

• Scripts: scripts/ - analysis tools (Python runtime, multi-language targets)
• ast_parser.py - structural extraction dispatcher (Phases 1-7)
• analyze_file.py - per-file CLI (classification + structure + usages)
• classifier.py - language-aware criticality classifier
• usage_finder.py - cross-file symbol usage finder (multi-language extensions)
• comment_rewriter.py - multi-language comment analysis engine
• rewrite_comments.py - comment quality CLI (scan / analyze / rewrite / report)
• doc_review.py - documentation maintenance (Phase 8)
• check_progress.py / progress_tracker.py - phase progress tracking
• languages/ - per-language adapters (Python ast, Java/JS/TS/Rust via tree-sitter or regex, SQL/PL-SQL regex)
• base.py - shared dataclasses + LanguageAdapter Protocol
• __init__.py - extension dispatch (detect_language, get_adapter)
• comments.py - per-language comment lexer (includes rustdoc post-processor)
• _treesitter.py - optional tree-sitter loader with fallbacks
• python.py, java.py, javascript.py, typescript.py, sql.py, plsql.py, rust.py
• requirements.txt - optional dependencies (tree-sitter + language-pack, click)

AI-Powered Code Analysis Methodology

This document defines how Claude should semantically analyze source code beyond mechanical AST extraction.

---

Core Principle: Understanding Over Extraction

╔══════════════════════════════════════════════════════════════════════════════╗
║                         THE SEMANTIC ANALYSIS MANDATE                        ║
╠══════════════════════════════════════════════════════════════════════════════╣
║                                                                              ║
║  Scripts extract STRUCTURE:  "This file has class Foo with method bar()"    ║
║  Claude extracts MEANING:    "Foo implements the Repository pattern for     ║
║                               caching user sessions with TTL expiration"    ║
║                                                                              ║
║  NEVER stop at structure. ALWAYS pursue understanding.                      ║
║                                                                              ║
╚══════════════════════════════════════════════════════════════════════════════╝

---

The Five Layers of Code Understanding

Layer 1: WHAT (Structural) - Scripts handle this

• Classes, functions, imports
• Line counts, dependencies
• AST-extractable information

Layer 2: HOW (Mechanical) - Claude's first pass

• Algorithm implementation details
• Data flow through functions
• State transformations

Layer 3: WHY (Intent) - Claude's deep analysis

• Business purpose of the code
• Problem being solved
• Design decisions made

Layer 4: WHEN (Temporal) - Claude's behavioral analysis

• Execution conditions and triggers
• Lifecycle and state transitions
• Concurrency and timing

Layer 5: CONSEQUENCES (Impact) - Claude's systems thinking

• Side effects and mutations
• Downstream dependencies
• Failure modes and edge cases

---

Semantic Analysis Questions

When analyzing ANY code unit, Claude must answer these questions:

Identity Questions

□ What is this code's single responsibility?
□ What abstraction does it represent?
□ What would break if this code didn't exist?

Behavior Questions

□ What are ALL possible inputs?
□ What are ALL possible outputs (including side effects)?
□ What state does it read? What state does it mutate?
□ What are the preconditions for correct operation?
□ What are the postconditions guaranteed after execution?

Integration Questions

□ Who calls this code? Under what circumstances?
□ What does this code call? Why those specific dependencies?
□ What contracts/interfaces does it fulfill?
□ What would need to change if this code's signature changed?

Quality Questions

□ What could go wrong? How is failure handled?
□ Are there implicit assumptions that could break?
□ Is there hidden coupling to global state or external systems?
□ Are there race conditions or timing dependencies?

---

Analysis Patterns by Code Type

Pattern: Service/Manager Class

RECOGNIZE BY:
- Name ends in Service, Manager, Handler, Controller
- Has multiple public methods
- Coordinates between other components

ANALYZE FOR:
1. What domain concept does this service own?
2. What operations does it expose? (CRUD? Commands? Queries?)
3. What resources does it manage? (connections, state, caches)
4. What is the lifecycle? (singleton? per-request? pooled?)
5. What are the thread-safety guarantees?

DOCUMENT:
- Primary responsibility (one sentence)
- Key operations with preconditions
- Resource management strategy
- Error handling approach
- Integration points

Pattern: Data Model/Entity

RECOGNIZE BY:
- Name is a noun (User, Order, Transaction)
- Primarily contains fields/attributes
- May have validation logic

ANALYZE FOR:
1. What real-world concept does this represent?
2. What are the invariants? (fields that must always be valid)
3. What are the valid state transitions?
4. What is the identity? (which fields make it unique)
5. What are the relationships to other entities?

DOCUMENT:
- Domain meaning
- Field semantics (not just types)
- Validation rules
- State machine (if applicable)
- Relationship cardinality

Pattern: Algorithm/Processor

RECOGNIZE BY:
- Name contains process, calculate, compute, transform
- Takes input, produces output
- May be stateless

ANALYZE FOR:
1. What transformation does this perform?
2. What is the algorithmic complexity? (time/space)
3. What are the edge cases?
4. Are there numerical stability concerns?
5. Is it deterministic?

DOCUMENT:
- Input → Output transformation description
- Algorithm explanation (for non-trivial cases)
- Complexity analysis
- Edge case handling
- Example inputs and outputs

Pattern: Adapter/Integration

RECOGNIZE BY:
- Name contains Adapter, Client, Gateway, Connector
- Wraps external system
- Handles serialization/deserialization

ANALYZE FOR:
1. What external system does this wrap?
2. What is the retry/resilience strategy?
3. How are credentials managed?
4. What is the connection lifecycle?
5. How are errors from the external system translated?

DOCUMENT:
- External system and protocol
- Authentication mechanism
- Retry and timeout configuration
- Error mapping strategy
- Connection pooling details

Pattern: Event Handler/Callback

RECOGNIZE BY:
- Name contains on_, handle_, process_
- Takes event/message as parameter
- Often async

ANALYZE FOR:
1. What event triggers this handler?
2. What is the expected event frequency?
3. What happens if handling fails?
4. Is ordering guaranteed?
5. Is idempotency required/implemented?

DOCUMENT:
- Triggering event/condition
- Expected behavior
- Failure handling
- Ordering and idempotency guarantees
- Side effects produced

Pattern: Factory/Builder

RECOGNIZE BY:
- Name contains Factory, Builder, Creator
- Returns instances of other classes
- May have configuration methods

ANALYZE FOR:
1. What does this create?
2. Why is direct construction not used?
3. What configuration options exist?
4. Are created objects cached or always new?
5. What validation happens during creation?

DOCUMENT:
- What is being created and why factory pattern
- Configuration options and defaults
- Lifecycle of created objects
- Validation performed

Pattern: State Machine

RECOGNIZE BY:
- Enum of states
- Transition methods
- State-dependent behavior

ANALYZE FOR:
1. What are ALL valid states?
2. What are ALL valid transitions?
3. What triggers each transition?
4. What side effects occur on transition?
5. What is the terminal state(s)?

DOCUMENT:
- State diagram (Mermaid)
- Transition table with triggers
- Side effects per transition
- Error/recovery states

---

Flow Tracing Methodology

When tracing a flow through the system:

Step 1: Identify Entry Point

□ Where does this flow begin? (API endpoint, message handler, timer, etc.)
□ What triggers it? (user action, external event, scheduled task)
□ What data enters at this point?

Step 2: Trace Data Transformations

□ How is input data validated?
□ What transformations occur?
□ Where is data enriched with additional information?
□ Where is data persisted?

Step 3: Identify Decision Points

□ Where are conditional branches?
□ What determines which branch is taken?
□ Are there early returns or short-circuits?

Step 4: Map Side Effects

□ What external systems are called?
□ What state is mutated?
□ What events are emitted?
□ What logs are produced?

Step 5: Document Exit Points

□ What are the success outcomes?
□ What are the failure outcomes?
□ What cleanup occurs?

---

Red Flags to Identify

Claude should actively look for and document these issues:

Architecture Red Flags

⚠ GOD CLASS: Class with >10 public methods or >500 LOC
⚠ FEATURE ENVY: Method that uses more of another class than its own
⚠ SHOTGUN SURGERY: Change requires touching many files
⚠ CIRCULAR DEPENDENCY: A → B → C → A
⚠ LEAKY ABSTRACTION: Implementation details exposed in interface

Reliability Red Flags

⚠ SWALLOWED EXCEPTION: except: pass or empty catch blocks
⚠ MISSING TIMEOUT: Network/IO calls without timeout
⚠ UNBOUNDED GROWTH: Collections that grow without limit
⚠ RACE CONDITION: Shared mutable state without synchronization
⚠ RESOURCE LEAK: Opened resources not closed

Security Red Flags

⚠ HARDCODED SECRET: Passwords, API keys in code
⚠ SQL INJECTION: String concatenation in queries
⚠ MISSING VALIDATION: User input used without sanitization
⚠ OVERLY PERMISSIVE: Catch-all permissions or access
⚠ SENSITIVE LOGGING: Passwords, tokens in log output

Maintainability Red Flags

⚠ MAGIC NUMBER: Unexplained numeric constants
⚠ DEAD CODE: Unreachable or unused code
⚠ COPY-PASTE: Duplicated logic blocks
⚠ DEEP NESTING: >4 levels of indentation
⚠ LONG METHOD: >50 lines without clear sections

---

Documentation Output Standards

For Each Code Unit, Produce:

## {ClassName/FunctionName}

**Purpose:** {One sentence explaining WHY this exists}

**Responsibility:** {What this code OWNS in the system}

### Behavior

{Description of what this code DOES, not HOW it does it}

### Inputs
| Parameter | Type | Semantic Meaning | Constraints |
|-----------|------|------------------|-------------|
| ... | ... | ... | ... |

### Outputs
| Return/Effect | Type | Semantic Meaning | Conditions |
|---------------|------|------------------|------------|
| ... | ... | ... | ... |

### Dependencies
- **{Dependency}**: {WHY this dependency is needed}

### State Changes
- {What state is mutated and why}

### Error Conditions
| Condition | Behavior | Recovery |
|-----------|----------|----------|
| ... | ... | ... |

### Usage Example

Concrete example showing typical usage

### Notes
- {Any non-obvious insights, edge cases, or gotchas}

---

Incremental Understanding Protocol

As analysis progresses through the codebase:

Build Mental Model

1. Start with entry points (main, API handlers, event listeners)
2. Trace primary flows to understand core behavior
3. Map shared utilities and how they're used
4. Identify cross-cutting concerns (logging, auth, error handling)
5. Document architectural patterns in use

Cross-Reference Continuously

1. When analyzing File B, reference findings from File A
2. Update earlier documentation when new insights emerge
3. Build glossary of domain terms as they appear
4. Map acronyms and abbreviations to full meanings

Validate Understanding

1. After analyzing a subsystem, summarize it in one paragraph
2. Predict what a function does before reading it (from name/context)
3. If prediction is wrong, document the surprising behavior
4. Look for inconsistencies between similar components

---

Integration with Scripts

The mechanical scripts support AI analysis:

Workflow: 1. Run scripts to get structural overview 2. Use Claude to analyze semantics 3. Cross-reference script output with Claude insights 4. Produce final documentation combining both

---

This methodology transforms code analysis from mechanical extraction to genuine understanding.

Analysis Templates and Verification Model

Verification Trust Model

Layer 1: TOOL-VALIDATED
  - Automated checks: file exists, AST symbol exists, signature matches
  - Marker: [VALIDATED: file.py::ClassName.method_name @ 2025-12-20]

Layer 2: HUMAN-VERIFIED
  - Manual review: semantic correctness, behavior match
  - Marker: [VERIFIED: file.py::ClassName.method_name by @reviewer @ 2025-12-20]

Layer 3: RUNTIME-CONFIRMED
  - Log/trace evidence of actual behavior
  - Marker: [CONFIRMED: trace_id=abc123 @ 2025-12-20]

Tool validation catches STRUCTURAL issues (file moved, symbol renamed, signature changed).
Human verification ensures SEMANTIC correctness (code does what doc says).
Runtime confirmation proves BEHAVIORAL truth (system actually works this way).

ALL THREE LAYERS are required for critical documentation.

NOTE: Markers use qualified symbol names (Module::Class.method, file::function)
instead of line numbers. Line numbers shift on any edit; symbol names are stable
and survive refactoring as long as the symbol itself is not renamed.

The Iron Law of Documentation

DOCUMENTATION = f(SOURCE_CODE) + VERIFICATION

If NOT verified_against_code(statement) -> statement is FALSE
If NOT exists_in_codebase(reference)    -> reference is FABRICATED
If NOT traceable_to_source(claim)       -> claim is SPECULATION

The Temporal Purity Principle

Documentation = PRESENT_TENSE(current_implementation)

FORBIDDEN:
- "was/were/previously/formerly/used to"
- "deprecated since version X" -> just REMOVE it
- "changed from X to Y" -> only describe Y
- "in the old system..." -> irrelevant, delete
- inline changelogs -> use CHANGELOG.md or git

REQUIRED:
- Present tense: "The system uses..." not "The system used..."
- Current state only: Document what IS, not what WAS
- Git for archaeology: History lives in version control, not docs

The Rule:

When you find documentation containing historical language, DELETE IT.

Git blame exists for archaeology. Documentation exists for the present.

Verification Requirements

Documentation Verification Status

Every section of documentation MUST have one of these status markers:

• [VERIFIED: file.py::ClassName.method_name] - Confirmed against source code symbol
• [VERIFIED: trace_id=xyz] - Confirmed against runtime logs
• [UNVERIFIED] - Requires verification before trusting
• [DEPRECATED] - Code has changed, documentation outdated

Symbol reference format: file.py::symbol for top-level, file.py::Class.method for members. Never use line numbers in markers -- they break on any file edit.

UNVERIFIED documentation is UNTRUSTED documentation.

The Semantic Analysis Mandate

Scripts extract STRUCTURE:  "class Foo with method bar()"
Claude extracts MEANING:    "Foo implements Repository pattern for
                             caching user sessions with TTL expiration"

NEVER stop at structure. ALWAYS pursue understanding.

Semantic Analysis Template

Use templates/semantic_analysis.md for comprehensive per-file analysis that includes:

• Executive summary (purpose, responsibility, patterns)
• Behavioral analysis (triggers, processing, side effects)
• Dependency analysis (why each dependency exists)
• Quality assessment (strengths, concerns, red flags)
• Contract documentation (full interface semantics)
• Flow tracing (primary and error paths)
• Testing implications (what must be tested)

JSON Output Structure

{
  "file": "src/utils/circuit_breaker.py",
  "classification": "critical",
  "metrics": {
    "lines_of_code": 245,
    "num_classes": 2,
    "num_functions": 8,
    "num_dependencies": 12
  },
  "structure": {
    "classes": [],
    "functions": [],
    "constants": []
  },
  "dependencies": {
    "internal": [],
    "external": [],
    "external_calls": []
  },
  "usages": [],
  "verification_required": true
}

Markdown Output Format

The markdown output follows the template in templates/analysis_report.md and produces sections suitable for inclusion in phase deliverable documents.

Comment Type Classification

Comment Quality Workflow

1. SCAN
   - Run: rewrite_comments.py scan <dir> --recursive
   - Review files with most issues
   - Generate: rewrite_comments.py report <dir> --output report.md

2. TRIAGE
   - Identify high-priority files (critical modules)
   - Focus on DEBT comments (convert to issues or design docs)
   - Plan bulk TRIVIAL/BACKUP deletions

3. REWRITE
   - Run: rewrite_comments.py rewrite <file> --apply --backup
   - Review changes in diff
   - Verify no functional changes

4. VERIFY
   - Run tests to confirm no breakage
   - Re-scan to confirm improvements
   - Update comment_health.md report

Documentation Maintenance Workflow

When invoking Phase 8 documentation maintenance, follow this sequence:

1. PLANNING
   - Run: doc_review.py scan --path docs/
   - Review health report
   - Identify priority fixes (broken links, obsolete files)
   - Create todo list with specific actions

2. EXECUTION (in batches)
   - Batch 1: Fix broken links
     - Run: doc_review.py validate-links --fix
   - Batch 2: Verify critical docs against source
     - Run: doc_review.py verify --doc <file> --source <code>
   - Batch 3: Delete obsolete files
     - Manual review + deletion
   - Batch 4: Update navigation indexes
     - Run: doc_review.py update-indexes
   - Batch 5: Update timestamps
     - Set last_updated on verified files

3. VERIFICATION
   - Run: doc_review.py scan (confirm improvements)
   - Run: doc_review.py validate-links (confirm zero broken)
   - Generate final doc_health_report.json

Antirez Commenting Standards

Source: https://antirez.com/news/124

This document codifies the commenting standards from Salvatore Sanfilippo (antirez), creator of Redis. These standards form the basis for the rewrite_comments.py tool in the deep-dive-analysis skill.

---

Core Philosophy

Comments are not just for documentation. They serve multiple purposes:

1. API Documentation - Let readers treat code as black boxes 2. Design Rationale - Explain why, not what 3. Knowledge Transfer - Teach domain concepts 4. Cognitive Load Reduction - Create rhythm and structure 5. Coordination - Remind of dependent changes

The cardinal rule: A comment requiring as much effort to read as the code itself is worse than useless.

---

The Nine Comment Types

GOOD Comments (Keep and Enhance)

1. Function Comments

Purpose: Serve as inline API documentation at function/class top.

Location: Immediately before or inside function/class definition.

Goal: Allow readers to understand behavior without reading implementation.

def shutdown_all_workers(
    pool_id: str,
    reason: str,
    timeout_seconds: float = 30.0,
) -> ShutdownResult:
    """
    Gracefully shut down all workers in a pool.

    This is the nuclear option for resource management. Use only when:
    - Pool enters error state
    - Connection to backend lost > 30 seconds
    - Resource usage exceeds configured limit

    The function will:
    1. Cancel all pending tasks
    2. Drain workers in reverse allocation order (largest first)
    3. Log each shutdown with reason
    4. Broadcast worker_stopped events

    Args:
        pool_id: UUID of the worker pool
        reason: Human-readable reason for emergency shutdown
        timeout_seconds: Maximum wait time per worker (default 30s)

    Returns:
        ShutdownResult with success status and list of stopped workers

    Raises:
        ConnectionError: If backend unreachable after 3 retries
        PartialShutdownError: If some workers couldn't be stopped

    Example:
        result = shutdown_all_workers(
            pool_id="abc-123",
            reason="Memory usage exceeded 90%",
        )
        if not result.all_stopped:
            alert_operations_team(result.failed_workers)
    """

Antirez Quote:

"Function comments allow the reader to conceptually take the code and use it as aering a black box. This is the most important thing about function comments."

---

2. Design Comments

Purpose: Explain algorithms, techniques, and design decisions.

Location: At file or class top.

Goal: Show readers that non-obvious solutions were considered and justify choices.

"""
Resource Allocation Calculator

This module implements the Weighted Fair Queuing allocation model,
also known as "Proportional Share Scheduling."

DESIGN CHOICES:

1. Why Weighted Fair Queuing (not Round Robin or Priority)?
   - Round Robin ignores job importance, treats all equally
   - Pure Priority causes starvation of low-priority jobs
   - WFQ balances fairness with priority, industry standard
   - Our benchmarks show it outperforms alternatives for our workload

2. Why calculate per-job (not pool-level)?
   - System runs independent workers
   - Each worker manages its own resource allocation
   - Pool-level would require coordination layer

3. Overhead percentage assumption
   - Hardcoded to 5% because:
     a) System processes short-lived tasks (milliseconds to seconds)
     b) Context switch overhead is minimal at this scale
     c) Simplifies calculation

ALTERNATIVES CONSIDERED:
- Max-Min Fairness: Too complex for our use case
- Fixed Allocation: Doesn't scale with demand
- FIFO: No quality of service guarantees

SEE ALSO:
- Demers et al., "Analysis and Simulation of Fair Queueing"
- Parekh & Gallager, "A Generalized Processor Sharing Approach"
"""

Antirez Quote:

"Design comments are higher-level comments at the top of a file... that explain the general design of the code, typically explaining the algorithm, technique, or some kind of general idea."

---

3. Why Comments

Purpose: Explain the reasoning behind code decisions.

Location: Immediately before the code in question.

Goal: Prevent future developers from "simplifying" intentional complexity.

# We use a 100ms delay between API calls because:
# 1. The service rate-limits at 10 requests/second
# 2. During testing, we observed connection drops at higher rates
# 3. The 50ms buffer accounts for network jitter
# Issue: PROJ-892
await asyncio.sleep(0.100)

# Empty string instead of None for missing fields because
# JSON message serialization treats None as missing key,
# which breaks downstream consumers expecting the field.
comment = comment if comment else ""

# Sorting by absolute value (not signed) because we want
# to process largest items first regardless of direction.
# A -10 delta is as urgent to handle as +10 delta.
items.sort(key=lambda p: abs(p.delta), reverse=True)

Antirez Quote:

"Why comments explain the reason why the code is doing something... This is the kind of comments I love the most."

---

4. Teacher Comments

Purpose: Educate readers about domain knowledge they may lack.

Location: Before code that uses specialized concepts.

Goal: Lower barrier to entry for contributors.

# Exponential backoff calculates delay as:
#   base_delay * (2 ^ attempt_number) + jitter
# Higher attempt = longer wait. Cap prevents infinite delays.
#
# We use base=1s with max=60s (industry standard) to prevent
# thundering herd on service recovery. The jitter (±10%) prevents
# synchronized retries from multiple clients.
#
# Reference: AWS Architecture Blog, "Exponential Backoff And Jitter"
retry_delay = min(self.base_delay * (2 ** attempt), self.max_delay)

# Latency model using log-normal distribution because:
# - Cannot be negative (latency is always positive)
# - Has fat tail (high latency events are rare but occur)
# - Empirically matches our historical request data
#
# The parameters (mu=0.05, sigma=0.02) were fitted from
# 10,000 requests to the backend during 2024.
expected_latency = np.random.lognormal(mean=0.05, sigma=0.02)

Antirez Quote:

"Teacher comments explain to the reader that is not specialized in such a topic, how a given algorithm works, or how a given concept works."

---

5. Checklist Comments

Purpose: Remind developers of coordinated changes needed elsewhere.

Location: Before code that has external dependencies.

Goal: Prevent subtle bugs from incomplete refactoring.

# WARNING: If you modify these states, also update:
# - frontend/src/types/task_state.ts (TypeScript mirror)
# - src/tests/test_task_lifecycle.py (test fixtures)
# - docs/architecture/TASK_STATES.md (documentation)
# - The Mermaid diagram in README.md
#
# Failure to sync will cause deserialization errors between
# frontend and backend.
class TaskState(Enum):
    PENDING = "pending"
    QUEUED = "queued"
    RUNNING = "running"
    STOPPING = "stopping"
    ERROR = "error"


# SYNC: This routing key format must match:
# - src/messaging/publisher.py (publisher)
# - src/messaging/consumer.py (consumer)
# - docs/messaging/ROUTING_KEYS.md
EVENT_ROUTING_KEY = "event.{type}.{category}.{source_id}"

Antirez Quote:

"Checklist comments are a warning, a reminder. They are there to tell the reader that is going to modify the code that its modification, or some other action, must be performed."

---

6. Guide Comments

Purpose: Lower cognitive load through rhythm and divisions.

Location: Between logical sections of code.

Goal: Help readers navigate large files and understand structure.

class OrderExecutor:
    """Handles order execution lifecycle."""

    # ═══════════════════════════════════════════════════════════════
    # INITIALIZATION
    # ═══════════════════════════════════════════════════════════════

    def __init__(self, broker: BrokerAdapter):
        self.broker = broker
        self.pending_orders = {}

    # ═══════════════════════════════════════════════════════════════
    # ORDER SUBMISSION
    # ═══════════════════════════════════════════════════════════════

    async def submit_order(self, order: OrderRequest) -> OrderResult:
        """Submit a new order to the broker."""
        ...

    async def modify_order(self, order_id: str, changes: OrderChanges) -> bool:
        """Modify an existing pending order."""
        ...

    # ═══════════════════════════════════════════════════════════════
    # ORDER CANCELLATION
    # ═══════════════════════════════════════════════════════════════

    async def cancel_order(self, order_id: str) -> bool:
        """Cancel a pending order."""
        ...

    async def cancel_all_orders(self) -> int:
        """Cancel all pending orders. Returns count cancelled."""
        ...

Note: Guide comments are controversial. Some consider them unnecessary if code is well-structured. Use sparingly.

---

BAD Comments (Delete or Rewrite)

7. Trivial Comments

Definition: Comments that restate what the code already says.

Problem: Reading the comment requires equal effort to reading the code.

Action: Delete immediately.

# BAD - These add nothing:
i += 1  # Increment i
return result  # Return result
if user is None:  # If user is None
    raise ValueError()  # Raise an error
for item in items:  # Loop through items
    process(item)  # Process item
self.value = value  # Set value

# GOOD - These add context:
i += 1  # Move to 1-based index for API compatibility
return result  # Caller expects mutable list, not generator
if user is None:  # Unauthenticated requests get default quota
    return DEFAULT_USER

Antirez Quote:

"Trivial comments are guide comments that are completely useless... where reading the comment is not much simpler than reading the code."

---

8. Debt Comments

Definition: TODO, FIXME, XXX, HACK markers without resolution plan.

Problem: Accumulate indefinitely, become noise, never get resolved.

Action: Convert to proper documentation or create tracked issues.

# BAD:
# TODO: fix this
# FIXME: sometimes crashes
# XXX: hack
# HACK: temporary workaround
# TODO: optimize later

# BETTER - Convert to design comment:
# DESIGN DECISION: Using polling instead of webhooks
#
# Context: External API v1 doesn't support webhooks. We poll every
# 100ms which introduces latency but is reliable.
#
# Resolution Plan: API v2 (expected Q2 2025) adds webhook support.
# When upgrading, refactor to push model per PROJ-1456.
#
# Tracking: PROJ-1234
#
# Acceptance Criteria:
# - [ ] API v2 available in production
# - [ ] Webhook endpoint implemented
# - [ ] 30-day parallel run with polling as fallback

# BEST - Create issue and reference:
# See PROJ-1234 for planned migration to webhook model

Antirez Quote:

"Debt comments are sometimes acceptable. But in general, there is a better way to handle things that are problematic... If the thing is important, create an issue. If it's not, delete the comment."

---

9. Backup Comments

Definition: Commented-out code kept "just in case."

Problem: Clutters codebase, confuses readers, always outdated.

Action: Delete completely. Use git history if needed.

# BAD - Delete all of this:
# def old_calculate_price(symbol):
#     # Old implementation before v2.0
#     price = get_cached_price(symbol)
#     if price is None:
#         price = fetch_from_api(symbol)
#     return price

# class DeprecatedOrderHandler:
#     """No longer used after refactor"""
#     pass

# ACCEPTABLE (rare) - When keeping temporarily for safety:
# DEPRECATED: Remove after v2.1 stable release (target: 2025-02-01)
# Kept for emergency rollback during 2.0->2.1 migration.
# Tracking: PROJ-2001
#
# def legacy_request_handler():
#     """Old handler, kept for rollback safety only."""
#     ...

Antirez Quote:

"Backup comments are commented-out code... with modern version control systems, this is always wrong."

---

Decision Matrix

---

Integration with deep-dive-analysis

The rewrite_comments.py CLI tool uses these standards:

# Analyze comments in a file
python rewrite_comments.py analyze src/main.py --report

# Scan entire codebase
python rewrite_comments.py scan src/ --recursive

# Generate health report
python rewrite_comments.py report src/ --output comment_health.md

# Apply recommended deletions (with backup)
python rewrite_comments.py rewrite src/main.py --apply --backup

---

References

1. Original Article: https://antirez.com/news/124 2. Redis Source Code: Example of these principles in practice 3. Code Complete (McConnell): Chapter 32 on Self-Documenting Code 4. Clean Code (Martin): Chapter 4 on Comments

---

Document generated as reference for deep-dive-analysis skill

Comprehensive Codebase Analysis & Documentation Plan (v3)

1. Objective

Systematically analyze every source file in your codebase to: 1. Build a complete mental model of the system 2. Map all interactions, dependencies, and logic flows 3. Produce comprehensive, maintainable documentation 4. Identify architectural risks and improvement opportunities

Relationship to Existing Documentation:

• CONTEXT.md - High-level architecture overview (keep updated)
• This plan produces module-level deep dives that supplement the context file
• Final deliverables link back to CONTEXT.md for navigation

---

2. Methodology: "The Inverted Pyramid"

Work Bottom-Up: shared primitives → data structures → messaging → business logic → adapters → UI.

CRITICAL PRINCIPLE: ABSOLUTE SOURCE OF TRUTH

THE DOCUMENTATION PRODUCED BY THIS ANALYSIS IS THE ABSOLUTE AND UNQUESTIONABLE SOURCE OF TRUTH FOR YOUR PROJECT.

>

ANY INFORMATION NOT VERIFIED WITH IRREFUTABLE EVIDENCE FROM SOURCE CODE IS FALSE, UNRELIABLE, AND LEADS TO INEVITABLE FAILURE.

╔══════════════════════════════════════════════════════════════════════════════╗
║                        THE IRON LAW OF DOCUMENTATION                         ║
╠══════════════════════════════════════════════════════════════════════════════╣
║  DOCUMENTATION = f(SOURCE_CODE) + VERIFICATION                               ║
║                                                                              ║
║  If NOT verified_against_code(statement) → statement is FALSE                ║
║  If NOT exists_in_codebase(reference)    → reference is FABRICATED           ║
║  If NOT traceable_to_source(claim)       → claim is SPECULATION              ║
╚══════════════════════════════════════════════════════════════════════════════╝

Mandatory Rules (VIOLATION = FAILURE): 1. NEVER document anything without reading the actual source code first 2. NEVER assume any existing documentation, comment, or docstring is accurate 3. NEVER write documentation based on memory, inference, or "what should be" 4. ALWAYS derive truth EXCLUSIVELY from reading and tracing actual code 5. ALWAYS provide source file + qualified symbol name for every technical claim 6. ALWAYS verify state machines, enums, constants against actual definitions 7. TREAT all pre-existing docs as unverified claims requiring validation 8. MARK any unverifiable statement as [UNVERIFIED - REQUIRES CODE CHECK]

Why This is Non-Negotiable:

• Documentation drifts. Code is the ONLY truth.
• Starting from existing docs risks propagating lies
• Unverified documentation is worse than no documentation - it creates false confidence
• A single fabricated claim can cascade into catastrophic misunderstanding

Verification Status Markers (Required on ALL Documentation):

• [VERIFIED: file.py::ClassName.method_name] - Confirmed against source code symbol
• [VERIFIED: trace_id=xyz] - Confirmed against runtime logs
• [UNVERIFIED] - Awaiting verification, DO NOT TRUST
• [DEPRECATED] - Source code has changed, documentation is stale

Use qualified symbol names (file.py::symbol, file.py::Class.method) instead of line numbers. Line numbers shift on any edit; symbol names survive refactoring.

CRITICAL PRINCIPLE: NO HISTORICAL DEPTH

DOCUMENTATION DESCRIBES ONLY THE CURRENT STATE OF THE ART.

>

NO HISTORY. NO ARCHAEOLOGY. NO "WAS". ONLY "IS".

╔══════════════════════════════════════════════════════════════════════════════╗
║                     THE TEMPORAL PURITY PRINCIPLE                            ║
╠══════════════════════════════════════════════════════════════════════════════╣
║  Documentation = PRESENT_TENSE(current_implementation)                       ║
║                                                                              ║
║  FORBIDDEN:                                                                  ║
║  ✗ "was/were/previously/formerly/used to"                                    ║
║  ✗ "deprecated since version X" → just REMOVE it                             ║
║  ✗ "changed from X to Y" → only describe Y                                   ║
║  ✗ "in the old system..." → irrelevant, delete                               ║
║  ✗ inline changelogs → use CHANGELOG.md or git                               ║
║                                                                              ║
║  REQUIRED:                                                                   ║
║  ✓ Present tense: "The system uses..." not "The system used..."              ║
║  ✓ Current state only: Document what IS, not what WAS                        ║
║  ✓ Git for archaeology: History lives in version control, not docs           ║
╚══════════════════════════════════════════════════════════════════════════════╝

Why This is Non-Negotiable:

• Historical context in documentation creates cognitive load without actionable value
• "It used to work differently" is noise for someone trying to understand how it works NOW
• Version control exists precisely to preserve history - docs don't need to duplicate it
• Temporal language creates ambiguity: "was changed" - when? by whom? is it still valid?
• Documentation describing past states risks being mistaken for current truth

The Rule:

When you find documentation containing historical language, DELETE IT.

Git blame exists for archaeology. Documentation exists for the present.

File Classification (Apply Before Analysis)

The Analysis Loop (Per File)

┌─────────────────────────────────────────────────────────────┐
│ 1. CLASSIFY: Determine criticality & complexity             │
├─────────────────────────────────────────────────────────────┤
│ 2. READ & MAP                                               │
│    - Classes, functions, global variables                   │
│    - State mutations and side effects                       │
│    - Error handling patterns                                │
├─────────────────────────────────────────────────────────────┤
│ 3. DEPENDENCY CHECK                                         │
│    - Internal imports (within project)                      │
│    - External imports (third-party)                         │
│    - External calls (database, network, filesystem, etc.)   │
├─────────────────────────────────────────────────────────────┤
│ 4. CONTEXT ANALYSIS                                         │
│    - Where are this file's symbols used?                    │
│    - What calls INTO this file?                             │
│    - What message types flow through here?                  │
├─────────────────────────────────────────────────────────────┤
│ 5. RUNTIME VERIFICATION (if Critical/High-Complexity)       │
│    - Use log analysis to observe actual behavior            │
│    - Trace a real trace_id through this component           │
│    - Compare documented flow vs actual flow                 │
├─────────────────────────────────────────────────────────────┤
│ 6. DOCUMENTATION                                            │
│    - Internal: Verify/add docstrings                        │
│    - External: Add entry to Module Analysis Report          │
│    - Cross-reference: Link to CONTEXT.md sections           │
└─────────────────────────────────────────────────────────────┘

---

3. Progress Tracking

Progress is tracked in analysis_progress.json with the following structure:

{
  "metadata": {
    "started": "2024-XX-XX",
    "last_updated": "2024-XX-XX",
    "current_phase": 1
  },
  "files": [
    {
      "path": "src/types/enums.py",
      "phase": 1,
      "status": "pending|analyzing|done|blocked",
      "classification": "standard|critical|high-complexity|utility",
      "verification_required": true,
      "verification_done": false,
      "notes": "",
      "analyzed_at": null
    }
  ],
  "phases": {
    "1": { "name": "Foundation", "progress": "0/15", "status": "in_progress" },
    "2": { "name": "Data Layer", "progress": "0/25", "status": "pending" }
  }
}

---

4. Execution Phases (Template)

Adapt these phases to your project structure. The key principle is bottom-up analysis: start with shared utilities, then move to data models, then messaging/orchestration, then business logic, then adapters, then UI.

Phase 1: The Foundation (lib/ or common/)

Goal: Master the "language" of the system - primitives, contracts, utilities.

Deliverable: docs/01_foundation/COMMON_LIBRARY.md

Cross-Cutting Analysis:

• Document logging patterns
• Document error handling conventions
• Document ID generation (trace_id, correlation_id)

---

Phase 2: The Data Layer (models/ or entities/)

Goal: Map the data model - all entities, their relationships, and persistence.

Deliverable: docs/02_core/DATA_MODELS.md

• Entity-Relationship Diagram (Mermaid)
• State transition diagrams

---

Phase 3: The Messaging/Orchestration Layer

Goal: Understand orchestration & message routing BEFORE business logic.

Deliverable: docs/03_infrastructure/MESSAGING.md

Runtime Verification (Mandatory):

• Trace a real request through the system
• Document actual flow vs designed flow

---

Phase 4: The Business Logic (services/ or core/)

Goal: Understand the decision logic and business rules.

Deliverable: docs/02_core/BUSINESS_LOGIC.md

---

Phase 5: The Adapters (adapters/ or integrations/)

Goal: Map all external interfaces.

Deliverable: docs/03_infrastructure/ADAPTERS.md

---

Phase 6: The User Interface (frontend/ or ui/)

Goal: Map user interaction to system commands.

Deliverable: docs/06_ui/UI_ARCHITECTURE.md

---

Phase 7: Infrastructure & Operations

Goal: Operational mastery - deployment, monitoring, tooling.

Deliverable: docs/04_operations/OPERATIONAL_MANUAL.md

---

5. Cross-Cutting Concerns (Analyze Throughout)

These patterns span multiple phases - document as encountered:

---

6. Verification Checkpoints

After each phase, verify documentation accuracy:

1. Static Verification: Code review of documented flows 2. Runtime Verification: Use observability tools to trace real requests 3. Peer Review: Walk through documentation, identify gaps

Mandatory Runtime Traces:

• [ ] Phase 3: Trace a message through the messaging layer
• [ ] Phase 4: Trace a request through business logic services
• [ ] Phase 5: Trace an external API call through adapters
• [ ] Phase 5.5: Trace a REST API request end-to-end

---

7. Immediate Next Steps

1. Create analysis_progress.json with all files pre-populated 2. Begin Phase 1: Foundation (common libraries) 3. Start with config_loader.py (high-complexity, affects everything) 4. Document cross-cutting logging/telemetry patterns as encountered

---

---

8. Phase 8: Documentation Maintenance & Cleanup

Goal: Ensure all documentation under /docs is accurate, consistent, and up-to-date with verified source code.

8.1 Documentation Review Workflow

┌─────────────────────────────────────────────────────────────┐
│ 1. DISCOVERY                                                │
│    ├── Scan all .md files under /docs                       │
│    ├── Count files per directory                            │
│    ├── Identify files with TODO/FIXME/TBD markers           │
│    └── Catalog last_updated dates                           │
├─────────────────────────────────────────────────────────────┤
│ 2. LINK VALIDATION                                          │
│    ├── Extract all relative links from each doc             │
│    ├── Verify target files exist                            │
│    ├── Identify broken links to deleted files               │
│    └── Generate broken link report                          │
├─────────────────────────────────────────────────────────────┤
│ 3. SOURCE CODE VERIFICATION                                 │
│    ├── For each technical doc, identify code references     │
│    ├── Verify documented behavior matches actual code       │
│    ├── Flag documentation drift                             │
│    └── Mark as VERIFIED or NEEDS_UPDATE                     │
├─────────────────────────────────────────────────────────────┤
│ 4. MAINTENANCE ACTIONS                                      │
│    ├── Fix broken links (update or remove)                  │
│    ├── Update outdated content                              │
│    ├── Remove obsolete files                                │
│    ├── Merge redundant documentation                        │
│    ├── Split overly large files                             │
│    └── Update navigation indexes                            │
├─────────────────────────────────────────────────────────────┤
│ 5. STATISTICS UPDATE                                        │
│    ├── Update SEARCH_INDEX.md keyword counts                │
│    ├── Update BY_DOMAIN.md file references                  │
│    ├── Update version and last_updated dates                │
│    └── Generate documentation health report                 │
└─────────────────────────────────────────────────────────────┘

8.2 Documentation Categories

8.3 Maintenance Actions

8.4 Deliverables

1. doc_health_report.json - Documentation health metrics 2. Updated navigation indexes - SEARCH_INDEX.md, BY_DOMAIN.md refreshed 3. Clean documentation tree - No broken links, no obsolete files 4. Verified timestamps - All last_updated dates accurate

---

9. Success Criteria

The deep dive is complete when:

• [ ] All source files analyzed and documented (Phases 1-7)
• [ ] All mandatory runtime verifications passed
• [ ] CONTEXT.md updated with links to detailed docs
• [ ] No undocumented critical paths remain
• [ ] Documentation health check passed (Phase 8)
• [ ] All broken links fixed
• [ ] Navigation indexes up-to-date
• [ ] No obsolete documentation files
• [ ] New contributor can understand system from docs alone

Semantic Pattern Recognition Guide

Patterns Claude should recognize, understand, and document during code analysis.

---

How to Use This Guide

When analyzing code, Claude should: 1. Recognize - Identify which pattern(s) the code implements 2. Understand - Know the intent and trade-offs of the pattern 3. Document - Explain the pattern and its specific implementation 4. Evaluate - Assess if the pattern is correctly applied

---

Architectural Patterns

Repository Pattern

SIGNATURE:
- Class with get/find/save/delete methods
- Abstracts data storage
- Returns domain objects, not raw data

RECOGNIZE BY:
class UserRepository:
    def get_by_id(self, id) -> User
    def find_by_email(self, email) -> User | None
    def save(self, user: User) -> None
    def delete(self, user: User) -> None

DOCUMENT AS:
"Repository pattern abstracting {storage_type} access for {entity}.
Provides {list operations}. Uses {caching_strategy if any}."

EVALUATE:
□ Does it hide storage details completely?
□ Are queries in repository or leaking to callers?
□ Is there proper transaction handling?

Service Layer Pattern

SIGNATURE:
- Orchestrates multiple repositories/adapters
- Contains business logic
- No direct database access

RECOGNIZE BY:
class OrderService:
    def __init__(self, order_repo, inventory_service, payment_gateway):
        ...
    def place_order(self, cart, user) -> Order:
        # Coordinates multiple operations

DOCUMENT AS:
"Service layer for {domain_area}. Orchestrates {list_of_dependencies}.
Responsible for {list_business_operations}."

EVALUATE:
□ Is business logic here or scattered?
□ Does it properly coordinate transactions?
□ Are dependencies injected (testable)?

Event-Driven Architecture

SIGNATURE:
- Event classes/messages
- Publishers and subscribers
- Decoupled components

RECOGNIZE BY:
class OrderPlacedEvent:
    order_id: str
    user_id: str
    timestamp: datetime

event_bus.publish(OrderPlacedEvent(...))

@event_handler(OrderPlacedEvent)
def send_confirmation_email(event):
    ...

DOCUMENT AS:
"Event-driven flow: {trigger} → {event} → {handlers}.
{Sync/Async} delivery with {ordering_guarantee}."

EVALUATE:
□ Are events immutable?
□ Is ordering guaranteed when needed?
□ How are failed handlers retried?
□ Is there event versioning?

CQRS (Command Query Responsibility Segregation)

SIGNATURE:
- Separate read and write models
- Commands for writes, Queries for reads
- Often separate databases

RECOGNIZE BY:
# Commands (write)
class CreateOrderCommand:
    ...
def handle_create_order(cmd: CreateOrderCommand):
    ...

# Queries (read)
class OrderSummaryQuery:
    ...
def handle_order_summary(query: OrderSummaryQuery):
    ...

DOCUMENT AS:
"CQRS pattern with {write_model} and {read_model}.
Commands: {list}. Queries: {list}.
Synchronization via {mechanism}."

EVALUATE:
□ Is read/write separation consistent?
□ How is eventual consistency handled?
□ Is complexity justified by requirements?

---

Behavioral Patterns

State Machine

SIGNATURE:
- Enum of states
- Defined transitions
- State-dependent behavior

RECOGNIZE BY:
class OrderState(Enum):
    PENDING = "pending"
    CONFIRMED = "confirmed"
    SHIPPED = "shipped"
    DELIVERED = "delivered"
    CANCELLED = "cancelled"

def confirm(self):
    if self.state != OrderState.PENDING:
        raise InvalidTransition()
    self.state = OrderState.CONFIRMED

DOCUMENT AS:

stateDiagram-v2 [*] --> PENDING PENDING --> CONFIRMED: confirm() PENDING --> CANCELLED: cancel() CONFIRMED --> SHIPPED: ship() SHIPPED --> DELIVERED: deliver()

EVALUATE:
□ Are all transitions explicit?
□ Are invalid transitions prevented?
□ Is there a terminal state?
□ Can the machine get stuck?

Strategy Pattern

SIGNATURE:
- Interface defining algorithm
- Multiple implementations
- Runtime selection

RECOGNIZE BY:
class PricingStrategy(Protocol):
    def calculate(self, order: Order) -> Decimal

class StandardPricing(PricingStrategy):
    def calculate(self, order): ...

class DiscountPricing(PricingStrategy):
    def calculate(self, order): ...

# Usage
strategy = get_pricing_strategy(customer_type)
price = strategy.calculate(order)

DOCUMENT AS:
"Strategy pattern for {what_varies}.
Implementations: {list_strategies}.
Selection based on {criteria}."

EVALUATE:
□ Is the interface minimal?
□ Are strategies truly interchangeable?
□ Is selection logic centralized?

Observer Pattern

SIGNATURE:
- Subject maintains observer list
- Observers notified on state change
- Decoupled notification

RECOGNIZE BY:
class Observable:
    def __init__(self):
        self._observers = []

    def subscribe(self, observer):
        self._observers.append(observer)

    def notify(self, event):
        for observer in self._observers:
            observer.on_event(event)

DOCUMENT AS:
"Observer pattern: {subject} notifies {observer_types} on {events}.
{Sync/async} notification. {Ordering} guarantees."

EVALUATE:
□ Can observers unsubscribe?
□ Is notification order-independent?
□ Are observer exceptions isolated?

Chain of Responsibility

SIGNATURE:
- Sequence of handlers
- Each can handle or pass
- Request flows until handled

RECOGNIZE BY:
class Handler:
    def __init__(self, next_handler=None):
        self.next = next_handler

    def handle(self, request):
        if self.can_handle(request):
            return self.do_handle(request)
        elif self.next:
            return self.next.handle(request)
        raise UnhandledRequest()

DOCUMENT AS:
"Chain of responsibility for {request_type}.
Chain order: {handler1} → {handler2} → {handler3}.
Fallback behavior: {what_happens_if_unhandled}."

EVALUATE:
□ Is chain order significant?
□ Can multiple handlers act on same request?
□ Is there a default/fallback handler?

---

Resilience Patterns

Circuit Breaker

SIGNATURE:
- Tracks failures
- Opens after threshold
- Allows periodic retry

RECOGNIZE BY:
class CircuitBreaker:
    def __init__(self, failure_threshold, reset_timeout):
        self.state = "closed"
        self.failures = 0

    def call(self, func):
        if self.state == "open":
            if self.should_try_reset():
                self.state = "half-open"
            else:
                raise CircuitOpen()
        try:
            result = func()
            self.on_success()
            return result
        except Exception:
            self.on_failure()
            raise

DOCUMENT AS:
"Circuit breaker protecting {resource/service}.
Threshold: {N} failures in {time_window}.
Reset timeout: {duration}. Half-open allows {N} test requests."

EVALUATE:
□ Is threshold appropriate for the service?
□ Are the right exceptions triggering failures?
□ Is there alerting when circuit opens?

Retry with Backoff

SIGNATURE:
- Retries on failure
- Increasing delays
- Maximum attempts

RECOGNIZE BY:
@retry(
    max_attempts=3,
    backoff=exponential(base=1, max=60),
    retry_on=(ConnectionError, TimeoutError)
)
def call_external_api():
    ...

DOCUMENT AS:
"Retry pattern for {operation}.
Max attempts: {N}. Backoff: {strategy}.
Retries on: {exception_types}. Gives up after: {condition}."

EVALUATE:
□ Is the operation idempotent?
□ Is backoff capped to prevent infinite waits?
□ Are only transient errors retried?

Bulkhead

SIGNATURE:
- Isolated resource pools
- Failure containment
- Prevents cascade

RECOGNIZE BY:
class BulkheadExecutor:
    def __init__(self, name, max_concurrent):
        self.semaphore = Semaphore(max_concurrent)

    async def execute(self, func):
        async with self.semaphore:
            return await func()

# Separate bulkheads for different services
payment_bulkhead = BulkheadExecutor("payment", 10)
inventory_bulkhead = BulkheadExecutor("inventory", 20)

DOCUMENT AS:
"Bulkhead isolating {resource/service}.
Concurrency limit: {N}. Rejection policy: {behavior}.
Purpose: Prevents {failure_scenario} from affecting {protected_area}."

EVALUATE:
□ Are limits based on downstream capacity?
□ What happens when bulkhead is full?
□ Are bulkheads monitored?

Timeout

SIGNATURE:
- Time limit on operations
- Fallback on timeout
- Prevents hanging

RECOGNIZE BY:
async def fetch_with_timeout(url):
    try:
        async with asyncio.timeout(5.0):
            return await http_client.get(url)
    except asyncio.TimeoutError:
        return cached_response(url)

DOCUMENT AS:
"Timeout of {duration} on {operation}.
On timeout: {fallback_behavior}.
Rationale: {why_this_timeout_value}."

EVALUATE:
□ Is timeout value justified?
□ Does timeout include all nested operations?
□ Is the fallback safe?

---

Data Patterns

DTO (Data Transfer Object)

SIGNATURE:
- Plain data container
- No business logic
- Often used at boundaries

RECOGNIZE BY:
@dataclass
class UserDTO:
    id: str
    name: str
    email: str
    created_at: datetime

def to_dto(user: User) -> UserDTO:
    return UserDTO(
        id=str(user.id),
        name=user.display_name,
        email=user.email,
        created_at=user.created_at
    )

DOCUMENT AS:
"DTO for {purpose/boundary}.
Maps from {source_type}. Fields: {field_list}.
Used by: {consumers}."

EVALUATE:
□ Is it truly data-only (no methods)?
□ Is mapping centralized?
□ Are field names API-appropriate?

Value Object

SIGNATURE:
- Immutable
- Equality by value, not identity
- Self-validating

RECOGNIZE BY:
@dataclass(frozen=True)
class Money:
    amount: Decimal
    currency: str

    def __post_init__(self):
        if self.amount < 0:
            raise ValueError("Amount cannot be negative")

    def add(self, other: "Money") -> "Money":
        if self.currency != other.currency:
            raise CurrencyMismatch()
        return Money(self.amount + other.amount, self.currency)

DOCUMENT AS:
"Value object representing {concept}.
Invariants: {list_validation_rules}.
Operations: {list_methods}."

EVALUATE:
□ Is it immutable?
□ Is __eq__ based on values?
□ Are all invariants enforced in constructor?

Aggregate

SIGNATURE:
- Entity cluster
- Single entry point (root)
- Transactional boundary

RECOGNIZE BY:
class Order:  # Aggregate root
    def __init__(self):
        self.items: list[OrderItem] = []  # Owned entities

    def add_item(self, product, quantity):
        # All modifications through root
        item = OrderItem(product, quantity)
        self.items.append(item)

    def total(self) -> Money:
        return sum(item.subtotal for item in self.items)

DOCUMENT AS:
"Aggregate root: {root_entity}.
Contains: {child_entities}.
Invariants enforced: {list}.
Transactional boundary for: {operations}."

EVALUATE:
□ Are children only accessed through root?
□ Is the aggregate small enough?
□ Are all invariants checked on mutation?

---

Concurrency Patterns

Producer-Consumer

SIGNATURE:
- Queue between components
- Producer adds, consumer processes
- Decoupled pace

RECOGNIZE BY:
queue = asyncio.Queue(maxsize=100)

async def producer():
    while True:
        item = await generate_item()
        await queue.put(item)

async def consumer():
    while True:
        item = await queue.get()
        await process(item)
        queue.task_done()

DOCUMENT AS:
"Producer-consumer with {queue_type}.
Buffer size: {N}. Producers: {list}. Consumers: {list}.
Backpressure: {behavior_when_full}."

EVALUATE:
□ Is queue bounded?
□ What happens on producer overload?
□ Are consumers idempotent?

Worker Pool

SIGNATURE:
- Fixed number of workers
- Job queue
- Concurrent processing

RECOGNIZE BY:
class WorkerPool:
    def __init__(self, size: int):
        self.workers = [Worker() for _ in range(size)]
        self.job_queue = Queue()

    async def submit(self, job):
        await self.job_queue.put(job)

    async def run(self):
        await asyncio.gather(*[
            self._worker_loop(w) for w in self.workers
        ])

DOCUMENT AS:
"Worker pool with {N} workers for {job_type}.
Queue: {bounded/unbounded}. Max in flight: {N}.
Job timeout: {duration}. Failed job handling: {policy}."

EVALUATE:
□ Is pool size configurable?
□ How are worker failures handled?
□ Is there graceful shutdown?

---

Anti-Patterns to Flag

God Object

RECOGNIZE BY:
- >10 public methods
- >500 LOC
- Multiple unrelated responsibilities

DOCUMENT AS:
"⚠ GOD OBJECT: {class_name} has {N} responsibilities.
Should be split into: {suggested_classes}."

Anemic Domain Model

RECOGNIZE BY:
- Entities with only getters/setters
- All logic in services
- No behavior encapsulation

DOCUMENT AS:
"⚠ ANEMIC MODEL: {entity} has no behavior.
Logic scattered in: {service_list}.
Consider moving: {method_suggestions}."

Distributed Monolith

RECOGNIZE BY:
- Services with tight coupling
- Synchronous call chains
- Shared databases

DOCUMENT AS:
"⚠ DISTRIBUTED MONOLITH: {service} tightly coupled to {other_services}.
Coupling points: {list}.
Risk: {cascade_failure_scenario}."

---

Pattern Combinations

Common valid combinations:

---

Recognition is the first step. Understanding follows. Documentation captures.

Module Analysis Report: {MODULE_NAME}

Generated by deep-dive-analysis skill on {DATE}

Zero-assumptions analysis derived exclusively from source code

Overview

---

File Index

{FILE_TABLE}

---

Detailed Analysis

{FILE_ANALYSES}

---

Dependency Graph

Internal Dependencies (within project)

{INTERNAL_DEP_DIAGRAM}

External Dependencies (third-party)

{EXTERNAL_DEPS_LIST}

---

Cross-Cutting Patterns Observed

Error Handling

{ERROR_HANDLING_PATTERNS}

Logging Patterns

{LOGGING_PATTERNS}

State Management

{STATE_MANAGEMENT_PATTERNS}

---

Verification Checklist

Files requiring runtime verification:

{VERIFICATION_CHECKLIST}

---

Observations & Findings

Architecture Notes

{ARCHITECTURE_NOTES}

Potential Issues

{POTENTIAL_ISSUES}

Recommendations

{RECOMMENDATIONS}

---

Links

• CONTEXT.md - System overview
• DEEP_DIVE_PLAN.md - Analysis methodology
• analysis_progress.json - Progress tracking

---

This report was generated from source code analysis. Existing documentation was NOT consulted during analysis per the Zero Assumptions principle.