Hooks Eval

Table of Contents

• Overview
• Key Capabilities
• Core Components
• Quick Reference
• Hook Event Types
• Hook Callback Signature
• Return Values
• Quality Scoring (100 points))
• Detailed Resources
• Basic Evaluation Workflow
• Integration with Other Tools
• Related Skills

Hooks Evaluation Framework

Overview

This skill provides a detailed framework for evaluating, auditing, and implementing Claude Code hooks across all scopes (plugin, project, global) and both JSON-based and programmatic (Python SDK) hooks.

Key Capabilities

• Security Analysis: Vulnerability scanning, dangerous pattern detection, injection prevention
• Performance Analysis: Execution time benchmarking, resource usage, optimization
• Compliance Checking: Structure validation, documentation requirements, best practices
• SDK Integration: Python SDK hook types, callbacks, matchers, and patterns

Core Components

Quick Reference

Hook Event Types

HookEvent = Literal[
    "PreToolUse",       # Before tool execution
    "PostToolUse",      # After tool execution
    "UserPromptSubmit", # When user submits prompt
    "Stop",             # When stopping execution
    "SubagentStop",     # When a subagent stops
    "TeammateIdle",     # When teammate agent becomes idle (2.1.33+)
    "TaskCompleted",    # When a task finishes execution (2.1.33+)
    "PreCompact"        # Before message compaction
]

Verification: Run the command with --help flag to verify availability.

Note: Python SDK does not support SessionStart, SessionEnd, or Notification hooks due to setup limitations. However, plugins can define SessionStart hooks via hooks.json using shell commands (e.g., leyline's detect-git-platform.sh).

Plugin-Level hooks.json

Plugins can declare hooks via "hooks": "./hooks/hooks.json" in plugin.json. The evaluator validates:

• Referenced hooks.json exists and is valid JSON
• Shell commands referenced in hooks exist and are executable
• Hook matchers use valid event types

Hook Callback Signature

async def my_hook(
    input_data: dict[str, Any],    # Hook-specific input
    tool_use_id: str | None,       # Tool ID (for tool hooks)
    context: HookContext           # Additional context
) -> dict[str, Any]:               # Return decision/messages
    ...

Verification: Run the command with --help flag to verify availability.

Return Values

return {
    "hookSpecificOutput": {
        "hookEventName": "PreToolUse",       # Match hook type
        "permissionDecision": "deny",        # Optional: block action
        "permissionDecisionReason": "...",   # Reason for denial
        "additionalContext": "...",          # Optional: context added
    }
}

Verification: Run the command with --help flag to verify availability.

Quality Scoring (100 points)

Detailed Resources

• SDK Hook Types: See modules/sdk-hook-types.md for complete Python SDK type definitions, patterns, and examples
• Evaluation Criteria: See modules/evaluation-criteria.md for detailed scoring rubric and quality gates
• Security Patterns: See modules/sdk-hook-types.md for vulnerability detection and mitigation
• Performance Guide: See modules/evaluation-criteria.md for benchmarking and optimization

Basic Evaluation Workflow

# 1. Run detailed evaluation
/hooks-eval --detailed

# 2. Focus on security issues
/hooks-eval --security-only --format sarif

# 3. Benchmark performance
/hooks-eval --performance-baseline

# 4. Check compliance
/hooks-eval --compliance-report

Verification: Run the command with --help flag to verify availability.

Integration with Other Tools

# Complete plugin evaluation pipeline
/hooks-eval --detailed          # Evaluate all hooks
/analyze-hook hooks/specific.py      # Deep-dive on one hook
/validate-plugin .                   # Validate overall structure

Verification: Run the command with --help flag to verify availability.

Related Skills

• abstract:hook-scope-guide - Decide where to place hooks (plugin/project/global)
• abstract:hook-authoring - Write hook rules and patterns
• abstract:validate-plugin - Validate complete plugin structure

Troubleshooting

Common Issues

Hook not firing Verify hook pattern matches the event. Check hook logs for errors

Syntax errors Validate JSON/Python syntax before deployment

Permission denied Check hook file permissions and ownership

Hook Evaluation Criteria

Detailed scoring rubric and quality gates for hook evaluation.

Mathematical Foundation

This evaluation framework follows Multi-Criteria Decision Analysis (MCDA) best practices:

• Normalization: Vector normalization for scale invariance (full methodology)
• Weighting: Security-first weights with stakeholder validation
• Aggregation: Weighted sum with penalty-based security scoring
• Validation: Sensitivity analysis on non-security weights

Documentation: See Multi-Metric Evaluation Methodology for complete mathematical foundation.

Scoring System (100 points total)

Security Analysis (30 points)

Vulnerability Detection:

• Critical vulnerabilities: -15 points each
• High-risk issues: -8 points each
• Medium-risk issues: -4 points each
• Low-risk issues: -1 point each

Security Checklist:

Performance Analysis (25 points)

Performance Thresholds:

pre_tool_use:
  excellent: <50ms
  good: <100ms
  acceptable: <200ms
  poor: >200ms

post_tool_use:
  excellent: <100ms
  good: <200ms
  acceptable: <500ms
  poor: >500ms

memory:
  excellent: <25MB
  good: <50MB
  acceptable: <100MB
  poor: >100MB

Compliance Analysis (20 points)

Structure Requirements:

• JSON hooks: Valid JSON schema with required fields
• Python hooks: Type hints, async/await patterns
• Matcher patterns: Valid regex, appropriate scope

Reliability Analysis (15 points)

Reliability Checklist:

• [ ] Hook returns valid response on all code paths
• [ ] Exceptions are caught and handled
• [ ] Timeout is configured appropriately
• [ ] Hook can be called multiple times safely
• [ ] Failure doesn't break agent operation

Maintainability (10 points)

Quality Levels

Quality Gates

Default thresholds for CI/CD integration:

quality_gates:
  security_score: ">= 80"
  performance_score: ">= 70"
  compliance_score: ">= 85"
  reliability_score: ">= 85"
  overall_score: ">= 75"
  max_critical_issues: 0
  max_high_issues: 2

Sensitivity Analysis Requirements

Security weights are non-negotiable, but other weights should be validated:

sensitivity_analysis:
  # Security weights are fixed (non-negotiable)
  fixed_weights: ["security_analysis"]

  # Other weights tested for sensitivity
  test_weights: ["performance", "compliance", "reliability", "maintainability"]
  variation: 0.20  # ±20% weight variation

  requirements:
    stable_rankings: true  # Rankings shouldn't change (except security)
    critical_weights_identified: true  # Document sensitive weights

See Sensitivity Analysis for implementation details.

Gate Behaviors

Issue Classification

Critical Issues (Immediate Action Required)

• Dynamic code evaluation with untrusted input
• Command injection vulnerabilities
• Credential exposure
• Unhandled exceptions that break agent

High Issues (Address Before Release)

• Missing input validation
• Performance exceeds thresholds
• Missing error handling
• Insecure file operations

Medium Issues (Address Soon)

• Missing documentation
• Suboptimal patterns
• Minor performance concerns
• Code style violations

Low Issues (Nice to Fix)

• Minor documentation gaps
• Formatting inconsistencies
• Optimization opportunities
• Enhanced logging suggestions

Evaluation Report Format

Summary Format

=== Hooks Evaluation Report ===
Plugin: {name} (v{version})
Scope: {scope}
Total hooks: {count} ({json_count} JSON, {python_count} Python)

=== Scores ===
Security:      {score}/100 ({level})
Performance:   {score}/100 ({level})
Compliance:    {score}/100 ({level})
Reliability:   {score}/100 ({level})
Maintainability: {score}/100 ({level})
────────────────────────────────
Overall:       {score}/100 ({level})

=== Issues ===
Critical: {count}
High: {count}
Medium: {count}
Low: {count}

Detailed Format

Includes per-hook breakdown:

=== Hook: {hook_path} ===
Type: {json|python}
Event: {PreToolUse|PostToolUse|...}
Matcher: {pattern|universal}

Security Issues:
  [{severity}] Line {n}: {description}

Performance:
  Estimated time: {ms}ms (threshold: {threshold}ms)
  Memory usage: {mb}MB (threshold: {threshold}MB)

Recommendations:
  1. {recommendation}
  2. {recommendation}

Customization

Per-Plugin Configuration

Create .hooks-eval.yaml in plugin root:

hooks_eval:
  # Override security thresholds
  security_thresholds:
    critical_score: 80
    high_score: 70

  # Override performance thresholds
  performance_thresholds:
    pre_tool_use_max_ms: 100
    post_tool_use_max_ms: 200
    max_memory_mb: 50

  # Compliance requirements
  compliance_requirements:
    require_documentation: true
    require_error_handling: true
    require_timeout_config: true

  # Custom rules
  custom_rules:
    - name: "no-hardcoded-secrets"
      pattern: "password|secret|token"
      severity: "high"
    - name: "require-shebang"
      pattern: "^#!"
      file_types: [".sh", ".py"]
      severity: "medium"

  # Excluded paths
  exclude_paths:
    - "hooks/experimental/*"
    - "hooks/deprecated/*"

Severity Overrides

Override default severity for specific patterns:

severity_overrides:
  - pattern: "subprocess.run"
    default_severity: "high"
    override_severity: "medium"
    reason: "Safe usage verified in review"

Python SDK Hook Types

Complete reference for Claude Agent SDK hook types, callbacks, and matchers.

Hook Events

HookEvent

Supported hook event types in the Python SDK.

from typing import Literal

HookEvent = Literal[
    "Setup",             # Called when plugin installed/enabled
    "SessionStart",      # Called when session begins
    "SessionEnd",        # Called when session ends normally
    "UserPromptSubmit",  # Called when user submits a prompt
    "PreToolUse",        # Called before tool execution
    "PostToolUse",       # Called after tool execution
    "PostToolUseFailure",# Called when tool execution fails (2.1.20+)
    "PermissionRequest", # Called when permission dialog would appear
    "Notification",      # Called on system notification (2.1.20+)
    "SubagentStart",     # Called when subagent spawns (2.1.20+)
    "SubagentStop",      # Called when a subagent stops
    "Stop",              # Called when stopping execution
    "TeammateIdle",      # Called when teammate agent becomes idle (2.1.33+)
    "TaskCompleted",     # Called when a task finishes execution (2.1.33+)
    "ConfigChange",      # Called when config is modified (2.1.49+)
    "InstructionsLoaded",# Called when instructions are loaded (2.1.33+)
    "PreCompact",        # Called before message compaction
    "PostCompact",       # Called after compaction (2.1.76+)
    "WorktreeCreate",    # Called when git worktree is created (2.1.50+)
    "WorktreeRemove",    # Called when git worktree is removed (2.1.50+)
    "StopFailure",       # Called on error (2.1.78+)
    "TaskCreated",       # Called when task created (2.1.84+)
    "CwdChanged",        # Called on working dir change (2.1.83+)
    "FileChanged",       # Called on file change (2.1.83+)
    "Elicitation",       # MCP elicitation request (2.1.76+)
    "ElicitationResult", # MCP elicitation response (2.1.76+)
]

SDK vs CLI availability: Most events work in both JSON hooks (CLI) and Python SDK hooks. PermissionRequest is CLI-only. Setup, SessionStart, SessionEnd, and Notification are CLI-only (JSON hooks). WorktreeCreate and WorktreeRemove are command-only hooks (no Python SDK callback). They do not support matchers.

Event Summary

Notable Version Changes

All hook events include agent_id and agent_type as of 2.1.69+.

Type Definitions

HookCallback

from typing import Any, Awaitable, Callable

HookCallback = Callable[
    [dict[str, Any], str | None, HookContext],
    Awaitable[dict[str, Any]]
]

Returns: dict[str, Any] with optional fields: decision ("block"), systemMessage (str), hookSpecificOutput (dict).

HookMatcher

@dataclass
class HookMatcher:
    matcher: str | None = None
    hooks: list[HookCallback] = field(default_factory=list)
    timeout: float | None = None  # Default: 60s

Complete Usage Example

from claude_agent_sdk import query, ClaudeAgentOptions, HookMatcher, HookContext
from typing import Any

async def validate_bash_command(
    input_data: dict[str, Any],
    tool_use_id: str | None,
    context: HookContext
) -> dict[str, Any]:
    """Block dangerous bash commands."""
    if input_data['tool_name'] == 'Bash':
        command = input_data['tool_input'].get('command', '')
        if 'rm -rf /' in command:
            return {
                'hookSpecificOutput': {
                    'hookEventName': 'PreToolUse',
                    'permissionDecision': 'deny',
                    'permissionDecisionReason': 'Dangerous command blocked'
                }
            }
    return {}

async def log_tool_use(
    input_data: dict[str, Any],
    tool_use_id: str | None,
    context: HookContext
) -> dict[str, Any]:
    """Log all tool usage for auditing."""
    print(f"Tool used: {input_data.get('tool_name')}")
    return {}

options = ClaudeAgentOptions(
    hooks={
        'PreToolUse': [
            HookMatcher(matcher='Bash', hooks=[validate_bash_command], timeout=120),
            HookMatcher(hooks=[log_tool_use])
        ],
        'PostToolUse': [
            HookMatcher(hooks=[log_tool_use])
        ]
    }
)

async for message in query(prompt="Analyze this codebase", options=options):
    print(message)

Input Data by Event Type

PreToolUse / PostToolUse

# PreToolUse
{"tool_name": "Bash", "tool_input": {"command": "ls -la"}}

# PostToolUse (adds result)
{"tool_name": "Bash", "tool_input": {"command": "ls -la"},
 "tool_result": "file1.txt\nfile2.txt", "error": None}

PermissionRequest (CLI only)

# Input
{"session_id": "abc123", "tool_name": "Bash",
 "tool_input": {"command": "npm install"},
 "permission_mode": "default", "cwd": "/path/to/project"}

# Output: allow
{"hookSpecificOutput": {"hookEventName": "PermissionRequest",
 "decision": {"behavior": "allow"}}}

# Output: deny
{"hookSpecificOutput": {"hookEventName": "PermissionRequest",
 "decision": {"behavior": "deny", "message": "Reason"}}}

Other Events

Hook Return Patterns

# Allow (default)
return {}

# Block action
return {
    "hookSpecificOutput": {
        "hookEventName": "PreToolUse",
        "permissionDecision": "deny",
        "permissionDecisionReason": "Explanation"
    }
}

# Add system message
return {"systemMessage": "Important context added to conversation"}

Best Practices