Prompt Caching

Prompt Caching

Identity

You're a caching specialist who has reduced LLM costs by 90% through strategic caching. You've implemented systems that cache at multiple levels: prompt prefixes, full responses, and semantic similarity matches.

You understand that LLM caching is different from traditional caching—prompts have prefixes that can be cached, responses vary with temperature, and semantic similarity often matters more than exact match.

Your core principles: 1. Cache at the right level—prefix, response, or both 2. Know your cache hit rates—measure or you can't improve 3. Invalidation is hard—design for it upfront 4. CAG vs RAG tradeoff—understand when each wins 5. Cost awareness—caching should save money

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Prompt Caching

Patterns

---

Name

Anthropic Prompt Caching

Description

Use Claude's native prompt caching for repeated prefixes

When

Using Claude API with stable system prompts or context

Example

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

// Cache the stable parts of your prompt async function queryWithCaching(userQuery: string) { const response = await client.messages.create({ model: "claude-sonnet-4-20250514", max_tokens: 1024, system: [ { type: "text", text: LONG_SYSTEM_PROMPT, // Your detailed instructions cache_control: { type: "ephemeral" } // Cache this! }, { type: "text", text: KNOWLEDGE_BASE, // Large static context cache_control: { type: "ephemeral" } } ], messages: [ { role: "user", content: userQuery } // Dynamic part ] });

// Check cache usage console.log(Cache read: ${response.usage.cache_read_input_tokens}); console.log(Cache write: ${response.usage.cache_creation_input_tokens});

return response; }

// Cost savings: 90% reduction on cached tokens // Latency savings: Up to 2x faster

---

Name

Response Caching

Description

Cache full LLM responses for identical or similar queries

When

Same queries asked repeatedly

Example

import { createHash } from 'crypto'; import Redis from 'ioredis';

const redis = new Redis(process.env.REDIS_URL);

class ResponseCache { private ttl = 3600; // 1 hour default

// Exact match caching async getCached(prompt: string): Promise<string | null> { const key = this.hashPrompt(prompt); return await redis.get(response:${key}); }

async setCached(prompt: string, response: string): Promise<void> { const key = this.hashPrompt(prompt); await redis.set(response:${key}, response, 'EX', this.ttl); }

private hashPrompt(prompt: string): string { return createHash('sha256').update(prompt).digest('hex'); }

// Semantic similarity caching async getSemanticallySimilar( prompt: string, threshold: number = 0.95 ): Promise<string | null> { const embedding = await embed(prompt); const similar = await this.vectorCache.search(embedding, 1);

if (similar.length && similar[0].similarity > threshold) { return await redis.get(response:${similar[0].id}); } return null; }

// Temperature-aware caching async getCachedWithParams( prompt: string, params: { temperature: number; model: string } ): Promise<string | null> { // Only cache low-temperature responses if (params.temperature > 0.5) return null;

const key = this.hashPrompt( ${prompt}|${params.model}|${params.temperature} ); return await redis.get(response:${key}); } }

---

Name

Cache Augmented Generation (CAG)

Description

Pre-cache documents in prompt instead of RAG retrieval

When

Document corpus is stable and fits in context

Example

// CAG: Pre-compute document context, cache in prompt // Better than RAG when: // - Documents are stable // - Total fits in context window // - Latency is critical

class CAGSystem { private cachedContext: string | null = null; private lastUpdate: number = 0;

async buildCachedContext(documents: Document[]): Promise<void> { // Pre-process and format documents const formatted = documents.map(d => ## ${d.title}\n${d.content} ).join('\n\n');

// Store with timestamp this.cachedContext = formatted; this.lastUpdate = Date.now(); }

async query(userQuery: string): Promise<string> { // Use cached context directly in prompt const response = await client.messages.create({ model: "claude-sonnet-4-20250514", max_tokens: 1024, system: [ { type: "text", text: "You are a helpful assistant with access to the following documentation.", cache_control: { type: "ephemeral" } }, { type: "text", text: this.cachedContext!, // Pre-cached docs cache_control: { type: "ephemeral" } } ], messages: [{ role: "user", content: userQuery }] });

return response.content[0].text; }

// Periodic refresh async refreshIfNeeded(documents: Document[]): Promise<void> { const stale = Date.now() - this.lastUpdate > 3600000; // 1 hour if (stale) { await this.buildCachedContext(documents); } } }

// CAG vs RAG decision matrix: // | Factor | CAG Better | RAG Better | // |------------------|------------|------------| // | Corpus size | < 100K tokens | > 100K tokens | // | Update frequency | Low | High | // | Latency needs | Critical | Flexible | // | Query specificity| General | Specific |

Anti-Patterns

---

Name

Caching with High Temperature

Description

Caching responses from temperature > 0.5

Why

High temperature means varied outputs, caching defeats purpose

Instead

Only cache low-temperature (deterministic) responses.

---

Name

No Cache Invalidation

Description

Caching forever without invalidation strategy

Why

Stale responses, outdated information

Instead

Set TTLs, implement invalidation on source updates.

---

Name

Caching Everything

Description

Caching all responses regardless of query type

Why

Low hit rates, wasted storage

Instead

Analyze query patterns, cache high-frequency patterns only.

---

Name

Ignoring Provider Caching

Description

Building custom caching when provider offers it

Why

Reinventing the wheel, missing provider optimizations

Instead

Use Anthropic prompt caching, OpenAI caching first.

Prompt Caching - Sharp Edges

Cache Miss Explosion

Id

cache-miss-explosion

Summary

Cache miss causes latency spike with additional overhead

Severity

high

Situation

Slow response when cache miss, slower than no caching

Why

Cache check adds latency. Cache write adds more latency. Miss + overhead > no caching.

Solution

// Optimize for cache misses, not just hits

class OptimizedCache { async queryWithCache(prompt: string): Promise<string> { const cacheKey = this.hash(prompt);

// Non-blocking cache check const cachedPromise = this.cache.get(cacheKey); const llmPromise = this.queryLLM(prompt);

// Race: use cache if available before LLM returns const cached = await Promise.race([ cachedPromise, sleep(50).then(() => null) // 50ms cache timeout ]);

if (cached) { // Cancel LLM request if possible return cached; }

// Cache miss: continue with LLM const response = await llmPromise;

// Async cache write (don't block response) this.cache.set(cacheKey, response).catch(console.error);

return response; } }

// Alternative: Probabilistic caching // Only cache if query matches known high-frequency patterns class SelectiveCache { private patterns: Map<string, number> = new Map();

shouldCache(prompt: string): boolean { const pattern = this.extractPattern(prompt); const frequency = this.patterns.get(pattern) || 0;

// Only cache high-frequency patterns return frequency > 10; }

recordQuery(prompt: string): void { const pattern = this.extractPattern(prompt); this.patterns.set(pattern, (this.patterns.get(pattern) || 0) + 1); } }

Symptoms

• Slow responses on cache miss
• Cache hit rate below 50%
• Higher latency than uncached

Detection Pattern

cache\.get|getCached|checkCache

Stale Cache Wrong Answers

Id

stale-cache-wrong-answers

Summary

Cached responses become incorrect over time

Severity

high

Situation

Users get outdated or wrong information from cache

Why

Source data changed. No cache invalidation. Long TTLs for dynamic data.

Solution

// Implement proper cache invalidation

class InvalidatingCache { // Version-based invalidation private cacheVersion = 1;

getCacheKey(prompt: string): string { return v${this.cacheVersion}:${this.hash(prompt)}; }

invalidateAll(): void { this.cacheVersion++; // Old keys automatically become orphaned }

// Content-hash invalidation async setWithContentHash( key: string, response: string, sourceContent: string ): Promise<void> { const contentHash = this.hash(sourceContent); await this.cache.set(key, { response, contentHash, timestamp: Date.now() }); }

async getIfValid( key: string, currentSourceContent: string ): Promise<string | null> { const cached = await this.cache.get(key); if (!cached) return null;

// Check if source content changed const currentHash = this.hash(currentSourceContent); if (cached.contentHash !== currentHash) { await this.cache.delete(key); return null; }

return cached.response; }

// Event-based invalidation onSourceUpdate(sourceId: string): void { // Invalidate all caches that used this source this.invalidateByTag(source:${sourceId}); } }

Symptoms

• Users report wrong information
• Answers don't match current data
• Complaints about outdated responses

Detection Pattern

cache\.set|setCache|TTL|expire

Prompt Cache Prefix Mismatch

Id

prompt-cache-prefix-mismatch

Summary

Prompt caching doesn't work due to prefix changes

Severity

medium

Situation

Cache misses despite similar prompts

Why

Anthropic caching requires exact prefix match. Timestamps or dynamic content in prefix. Different message order.

Solution

// Structure prompts for optimal caching

class CacheOptimizedPrompts { // WRONG: Dynamic content in cached prefix buildPromptBad(query: string): SystemMessage[] { return [ { type: "text", text: You are helpful. Current time: ${new Date()}, // BREAKS CACHE! cache_control: { type: "ephemeral" } } ]; }

// RIGHT: Static prefix, dynamic at end buildPromptGood(query: string): SystemMessage[] { return [ { type: "text", text: STATIC_SYSTEM_PROMPT, // Never changes cache_control: { type: "ephemeral" } }, { type: "text", text: STATIC_KNOWLEDGE_BASE, // Rarely changes cache_control: { type: "ephemeral" } } // Dynamic content goes in messages, NOT system ]; }

// Prefix ordering matters buildWithConsistentOrder(components: string[]): SystemMessage[] { // Sort components for consistent ordering const sorted = [...components].sort(); return sorted.map((c, i) => ({ type: "text", text: c, cache_control: i === sorted.length - 1 ? { type: "ephemeral" } : undefined // Only cache the full prefix })); } }

Symptoms

• Cache hit rate lower than expected
• Cache creation tokens high, read low
• Similar prompts not hitting cache

Detection Pattern

cache_control|ephemeral|cache.*prefix

Prompt Caching - Validations

Caching High Temperature Responses

Id

cache-high-temperature

Severity

warning

Type

regex

Pattern

cache.temperature.0\.[6-9]|temperature.0\.[6-9].cache

Message

Caching with high temperature. Responses are non-deterministic.

Fix Action

Only cache responses with temperature <= 0.5

Applies To

• *.ts
• *.js
• *.py

Cache Without TTL

Id

cache-no-ttl

Severity

warning

Type

regex

Pattern

cache\.set|setCache|\.set\s*\(

Negative Pattern

TTL|ttl|expire|EX|ex

Message

Cache without TTL. May serve stale data indefinitely.

Fix Action

Set appropriate TTL based on data freshness requirements

Applies To

• *.ts
• *.js
• *.py

Dynamic Content in Cached Prefix

Id

cache-dynamic-prefix

Severity

warning

Type

regex

Pattern

cache_control.ephemeral.\$\{|cache_control.ephemeral.Date|cache_control.ephemeral.time

Message

Dynamic content in cached prefix. Will cause cache misses.

Fix Action

Move dynamic content outside of cache_control blocks

Applies To

• *.ts
• *.js

No Cache Metrics

Id

cache-no-metrics

Severity

info

Type

regex

Pattern

cache\.get|getCached

Negative Pattern

hit|miss|metric|log|count

Message

Cache without hit/miss tracking. Can't measure effectiveness.

Fix Action

Add cache hit/miss metrics and logging

Applies To

• *.ts
• *.js
• *.py