Compression Strategy

Compression Strategy

Analyze current context usage and recommend optimal compression strategies.

When To Use

• Context feels bloated or sluggish
• Before major task phase transitions (plan complete, starting implementation)
• Token quota burning faster than expected
• After large tool output accumulations

When NOT To Use

• Context-optimization skill already handling the scenario
• Simple queries with minimal context
• Freshly cleared context

Required TodoWrite Items

1. compression-strategy:analyze-context 2. compression-strategy:recommend-strategy 3. compression-strategy:estimate-savings

Step 1 – Analyze Context (analyze-context)

Run /context to check current usage. Then estimate:

1. Tool output accumulation: How much context is from tool results vs. conversation? 2. Stale content age: How many turns since critical decisions were made? 3. Active files: Which files are still relevant vs. historical?

Step 2 – Recommend Strategy (recommend-strategy)

Based on analysis, recommend one of:

Option A: /clear and /catchup

Best when:

• Task phase complete (planning done, implementation starting)
• Context > 60% full
• Most content is stale

Process: 1. Save critical state to .claude/session-state.md 2. Run /clear 3. Run /catchup to reload active files

Option B: Spawn Continuation Agent

Best when:

• Context > 80% full
• Work in progress, can't stop
• Delegatable tasks remain

Process: 1. Run Skill(conserve:clear-context) to spawn continuation agent 2. Agent receives fresh context with saved state

Option C: Archive and Summarize

Best when:

• Context 40-60% full
• Some stale content mixed with active
• Not ready for full clear

Process: 1. Archive old decisions/errors to .claude/context-archive/ 2. Summarize completed work in memory 3. Continue with leaner context

Option D: Delegate to Subagent

Best when:

• Parallel work possible
• Independent subtasks exist
• Context pressure moderate

Process: 1. Identify delegatable tasks 2. Spawn specialized agents via Task tool 3. Main context stays lean

Step 3 – Estimate Savings (estimate-savings)

For the recommended strategy, estimate:

The CCR row is per oversized tool output, not whole-context: a large Bash, Read, or Grep result is archived to a handle and replaced by a digest for future turns. Savings are content-type-dependent (logs compress hard, prose barely at all). See modules/reversible-compression.md.

Context Archive Location

Preserved context is saved to:

.claude/context-archive/pre-compact-YYYYMMDD-HHMMSS-SESSIONID.md

This is automatically created by the pre_compact_preserve hook before any /compact operation.

Integration Points

• PreCompact hook: Automatically preserves context before compression
• Tool output summarizer: Warns when tool outputs accumulate
• Context warning hook: Three-tier alerts at 40%/50%/80%

Specialized Modules

Load modules/log-debugging-hygiene.md when the bloat source is pasted log output (debug traces, CI failures, hook logs, JSONL). That module documents a three-tier filter-first workflow with benchmarked snippets and an honest framing of when compression is and is not warranted. On the committed intake_queue.jsonl fixture, tail -n 100 beats lossless compression by 25 percentage points; the module formalizes that asymmetry.

Load modules/reversible-compression.md when large tool outputs (code search, log dumps, file reads) are the bloat source. That module documents the CCR pattern: the tool_output_summarizer hook archives any oversized output to a content-addressed handle under .claude/context-archive/, and context_retrieve.py fetches the original (or a slice) on demand, so the original survives /clear without staying resident.

Example Usage

/compression-strategy

Output:

Context Analysis:
- Current usage: 52%
- Tool output: ~15KB (3 tool results)
- Stale content: ~40% (decisions from 8+ turns ago)

Recommendation: Option C - Archive + Summarize
- Archive old decisions to context-archive
- Keep active files and recent decisions
- Estimated savings: 25-35%

Commands:
1. Read .claude/context-archive/ to see what's preserved
2. Summarize completed work
3. Continue with leaner context

Exit Criteria

• [ ] Context analyzed: current usage and tool-output share estimated
• [ ] A single strategy recommended (A-D, or reversible compression) with a

stated reason

• [ ] Savings estimated with the named risk from the Step 3 table
• [ ] For large tool outputs, the CCR handle and context_retrieve.py

retrieval command are surfaced (not just a warning)

• [ ] Recommendation refused or downgraded when the bloat source is dense

prose (compresses by roughly nothing)

Log Debugging Hygiene

When pasting logs into a Claude Code session, filter at the source before you reach for compression. On the committed intake_queue.jsonl fixture in this repo (a deterministic synthetic intake log under tests/fixtures/), tail -n 100 saves 95.0% of bytes and stays forensically useful because the output is a literal subset of the log. gzip -9 on the same fixture saves 65.0%, so filtering wins by 30 percentage points and needs no extra tooling.

This module formalizes a three-tier workflow so that compression is the last lever pulled, not the first.

When to Use

Reach for this module when any of the following apply:

• About to paste more than 100 log lines into a session.
• A previous paste pushed context above 50% utilization.
• A debugging trace, CI failure, or hook log is the source of

the bloat (not chat history or code).

• A teammate or skill recommended "compression" before any

filtering was tried.

Skip this module if the paste is already under 2000 tokens or if the entire log is needed verbatim for a regression report.

Required TodoWrite Items

1. log-debug:tier-1-filtered 2. log-debug:tier-2-minimized 3. log-debug:token-measured

Tier 1: Filter at Source (90-99% byte reduction)

Filtering keeps the lines you actually need and discards the rest. This step delivers the largest savings and requires no new tooling.

Benchmark on the committed fixture plugins/conserve/tests/fixtures/intake_queue.jsonl (0.89 MB, 2000 lines):

Every row is reproducible from the committed fixture with stock tools; the tail row is guarded by tests/test_log_debugging_hygiene.py::test_filter_first_claim_is_reproducible. The fixture is a deterministic synthetic intake log (high-entropy per-line content, so gzip cannot collapse it the way it would a repetitive log); it replaces an earlier benchmark that pointed at a mutable runtime artifact. A dedicated log-template compressor (logs-tokenizer, external and not bundled, see Tier 3) saved 70.3% on the original real-traffic snapshot. Filtering still wins on byte reduction, and by roughly 99x in absolute terms when jq can name the relevant rows.

Tier 2: Minimize Structurally (50-95% reduction)

When you cannot filter to specific lines, switch your tool to its most compact output mode. Many tools have a flag for this.

Pair tier 2 with head -n N or tail -n N to bound the output even after compaction.

Tier 3: Compress as Fallback (60-80% reduction)

Use compression only when:

• You genuinely need every line (anomaly detection across a

full trace, performance debugging where every timing matters, race condition analysis).

• Tier 1 and tier 2 cannot isolate the relevant subset.
• The compressed payload still fits within budget.

External tools (not bundled)

The conserve plugin does not bundle a compressor. Use one of these if tier 1 and 2 are insufficient:

Honest framing

Compression saves tokens. It does not improve LLM accuracy on log debugging tasks. The literature evidence:

• LLMLingua family (Jiang et al., EMNLP 2023, ACL 2024)

preserves accuracy at 4-20x compression on QA. The +17-21% accuracy gain reported by LongLLMLingua is on multi-document QA, not log debugging.

• LogFiT and LogLLM (Guan et al., arXiv 2411.08561, 2024)

argue the opposite direction for logs: preprocessing can hurt via cascading parser errors.

• Empirical Study on Prompt Compression (OpenReview 2024)

shows compression can introduce ASH (Altered Semantic Hallucinations) and ILH (Information Loss Hallucinations).

• CompressionAttack (arXiv 2510.22963, 2025) identifies

prompt compression as a new adversarial attack surface.

If you ship a compression claim, it must say "saves tokens, preserves accuracy" and cite which compressor produced the numbers.

Token vs Byte Reduction

Byte savings overstate token savings by roughly 10 percentage points because compressed payloads use unusual character combinations (tag soup like &1u:!E!) that the BPE tokenizer was not trained on.

Measure tokens with tiktoken (GPT-4 tokenizer, used as a proxy for Claude tokenization):

uv run --quiet --with tiktoken python3 -c "
import tiktoken, sys
enc = tiktoken.get_encoding('cl100k_base')
print(len(enc.encode(open(sys.argv[1]).read())))
" file.log

Measured deltas on the same intake_queue.jsonl fixture:

The diverse-content row matters most: on logs that lack heavy repetition (git commits, varied error messages, prose), the compressor delivers under 20% token savings while still imposing the legend overhead and tag-soup readability cost.

Anti-Patterns

Avoid these patterns when handling logs:

• Pasting pytest -v output without --tb=short -x. The

per-test verbose lines are almost always discardable.

• Quoting byte savings without measuring tokens. The two

are not equivalent.

• Adding a compression dependency "in case we need it." Per

.claude/rules/shared-utility-consumer-rule.md, scaffolding needs 2+ documented consumers within 30 days.

• Using compression on already-filtered logs. Diminishing

returns: filter already removed the highest-ratio repeats.

• Repeating the claim "LLMs work better on compressed logs"

without evidence. Literature does not support it for log debugging (see "Honest framing" above).

Exit Criteria

• [ ] At least one tier 1 command was tried before any paste.
• [ ] Pasted content size was measured in tokens (not bytes)

using tiktoken or equivalent.

• [ ] If a compressor was invoked, the savings claim cites

tokens, names the tool, and links to its license.

• [ ] No new compression dependency was added unless 2+

concrete consumers are documented per .claude/rules/shared-utility-consumer-rule.md.

• [ ] tests/test_log_debugging_hygiene.py passes, including

the reproducible filter-first benchmark.

References

• .claude/rules/bounded-discovery.md: discovery read budgets.
• .claude/rules/slop-scan-for-docs.md: evidence-backed claims.
• .claude/rules/shared-utility-consumer-rule.md: utility

scaffolding requires consumers.

• Skill(conserve:token-conservation): session-level token

budget tracking.

• Skill(conserve:context-optimization): MECW principles.

Reversible Compression (CCR)

Large tool outputs are the fastest way to fill a context window: a single code search or log dump can cost tens of thousands of tokens, most of which the model never needs. Reversible compression keeps the output recoverable without keeping it resident.

The pattern, after Headroom's CCR (Cached Compression with Retrieval):

1. When one tool output is large, write the original to an external cache keyed by a content hash (the handle). 2. Keep only a compact digest plus that handle in the conversation. 3. Fetch the original, or just the slice you need, on demand by handle.

The original is never lost, so the compression is reversible. The model reads the digest, and pulls the full text back only when a task actually needs it.

What is wired up in this plugin

The tool_output_summarizer PostToolUse hook archives any single Bash, Read, or Grep output at or above CONSERVE_CCR_THRESHOLD characters (default 25,000) to:

.claude/context-archive/ccr-<sha256[:12]>.txt

It then surfaces a digest (first and last 20 lines, total line and character counts) and the exact retrieval command. Fetch the original later with:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/context_retrieve.py ccr-<hash> \
  [--grep PATTERN] [--head N] [--tail N] [--lines A:B]

The handle is content-addressed, so identical outputs map to one file (natural dedup), and the archive survives /clear and continuation-agent handoffs.

Worked example (illustrative)

A 67 KB log dump (2,401 lines) is archived to an ccr-<hash>.txt handle. The handle below is illustrative, not a real run; yours is the SHA-256 prefix of the actual output. The model sees a ~2 KB digest. Later, one command pulls back only the line that matters:

context_retrieve.py ccr-e722db719ab6 --grep FATAL
# FATAL: database connection refused

Honest constraint: what the hook does not do

A PostToolUse hook runs after the tool result is already in context. It can add a digest, but it cannot redact the result it just saw, so it does not shrink the current turn. Its value is the durable external cache plus retrieve-on-demand: a later turn, or a fresh continuation agent after /clear, reads the handle instead of re-running the expensive command.

To save tokens in the current turn you still need the usual moves: /clear, a continuation agent (Skill(conserve:clear-context)), or not dumping the output in the first place. CCR makes those moves cheap to undo.

When compression pays off, and when it does not

Savings are content-type-dependent. Do not quote a single headline number. Measured reductions, from Headroom's own benchmarks (see Evidence):

Logs and structured tool output compress hard. Dense prose compresses by roughly nothing (one practitioner report measured -0.3%), and encrypted or high-entropy data not at all. Archive verbose, repetitive output; leave prose answers alone.

Two cautions worth stating plainly:

• Retrieve-on-demand can miss context, the same failure mode as RAG. If

the digest hides the span that mattered and nobody expands the handle, the model proceeds on partial information. Keep the digest honest (head, tail, and counts), and retrieve when a task depends on the body.

• Do not aggressively compress multi-step reasoning. At roughly 2x

compression, math and chained reasoning degrade even when surface similarity stays high (arXiv 2605.17932, 2602.15843). Code, extraction, and retrieval context tolerate aggressive ratios; arithmetic does not.

Prior art

• opencode-dynamic-context-pruning: the closest open-source analog, with

reversible decompress/recompress by id inside an agent loop.

• RECOMP (arXiv 2310.04408) and ICAE (arXiv 2307.06945): academic grounding

for recoverable compression (originals stay reconstructable from a store).

Evidence

Benchmark numbers above are sourced to the Headroom README (github.com/chopratejas/headroom) and the research synthesis in docs/research/headroom-context-compression.md, which carries the full citation set (LLMLingua, LongLLMLingua, RECOMP, ICAE, the boundary-condition papers, and the practitioner caveats).