Plugin Audit

# Plugin Audit

Audit how well vercel-plugin skill injection performs on real-world Claude Code sessions.

## Workflow

### 1. Locate conversation logs

Find JSONL conversation logs for a target project:

```bash
ls -lt ~/.claude/projects/-Users-*-<project-name>/*.jsonl
```

The path uses the project's absolute path with slashes replaced by hyphens and a leading hyphen.

### 2. Extract tool calls

Parse the JSONL log to extract all tool_use entries. Each line is a JSON object with `message.content[]` containing `type: "tool_use"` blocks. Extract `name` and `input` fields. Group by tool type (Bash, Read, Write, Edit).

### 3. Test hook matching

Use the exported pipeline functions directly — do NOT shell out to the hook script for each test. Import from the hooks directory:

```js
import { loadSkills, matchSkills } from "./hooks/pretooluse-skill-inject.mjs";
import { createLogger } from "./hooks/logger.mjs";
```

Call `loadSkills()` once, then `matchSkills(toolName, toolInput, compiledSkills)` for each tool call. This is fast and gives exact match results.

### 4. Identify gaps

Compare matched skills against what SHOULD have matched based on the project's technology stack. Common gap categories:

- **Path pattern gaps**: Files that should trigger a skill but don't (e.g., `src/db/schema.ts` not matching `vercel-storage`)
- **Bash pattern gaps**: Commands that should trigger but don't (e.g., missing package manager variants)
- **Dedup masking**: Skills that matched but were deduped before injection
- **Budget/cap drops**: Skills matched but dropped by the 12KB budget or 3-skill ceiling

### 5. Check plugin cache staleness

Compare the installed plugin cache against the dev version:

```bash
# Cache location
~/.claude/plugins/cache/vercel-labs-vercel-plugin/vercel-plugin/<version>/

# Compare skill content
diff <(grep 'pattern' skills/<skill>/SKILL.md) <(grep 'pattern' ~/.claude/plugins/cache/.../skills/<skill>/SKILL.md)
```

Check `~/.claude/plugins/installed_plugins.json` for version and git SHA.

## Report Format

Produce a structured report with:

1. **Session summary**: Project, date, tool call count, model
2. **Match matrix**: Table of tool calls × matched skills (with match type)
3. **Coverage gaps**: Unmatched tool calls that should have matched, with suggested pattern additions
4. **Dedup timeline**: Order of skill injections and what got deduped
5. **Cache status**: Whether installed version matches dev, with specific diffs

## References

- [Log format details](references/log-format.md)
- [Test script for batch matching](scripts/batch-match.mjs)


# Conversation Log Format

Claude Code stores conversation logs as JSONL files at:
```
~/.claude/projects/<encoded-project-path>/<session-id>.jsonl
```

The encoded path replaces `/` with `-` and prepends a `-`. Example:
`/Users/john/dev/my-app` → `-Users-john-dev-my-app`

## JSONL structure

Each line is a JSON object with:

```json
{
  "type": "assistant" | "user" | "system",
  "message": {
    "role": "assistant",
    "content": [
      { "type": "tool_use", "name": "Bash", "input": { "command": "..." } },
      { "type": "tool_use", "name": "Read", "input": { "file_path": "..." } },
      { "type": "tool_use", "name": "Write", "input": { "file_path": "..." } },
      { "type": "tool_use", "name": "Edit", "input": { "file_path": "..." } }
    ]
  },
  "timestamp": "ISO-8601"
}
```

## What's NOT in the log

- Hook return payloads (`hookSpecificOutput`) are NOT recorded in JSONL
- Only `hook_progress` events appear, showing that a hook was invoked
- To verify skill injection, test the hook directly against extracted tool inputs