Ai Context Engineer

AI Context Engineer

When to Use

• Designing what enters the model context each turn
• Optimizing cost/latency via context strategy and token budgeting
• Building context pipelines for agents (prefix, retrieval, history, user input)
• Implementing summarization, compaction, or rolling history
• Debugging context-related failures (lost instructions, overflow, distraction, ignored constraints)
• Choosing delimiters, XML blocks, or structured context formats

When NOT to Use

• Persistent memory store design or long-term recall architecture → ai-memory-developer
• Full RAG ingest/chunk/embed/index pipelines → ai-engineer
• AI org operations, release governance, or SLOs → ai-lead-ops
• Structured token/cost improvement roadmaps with phased KPIs → ai-token-improvement-plan-engineer
• Commercial/enterprise AI solution architecture → applied-ai-architect-commercial-enterprise

Related skills

Core Workflows

1. Context budget and layout

Allocate tokens (example 128k window):

Use explicit XML/markdown sections: <policy>, <tools>, <context>, <history>, <user>.

See `references/context_layout.md` for templates and delimiter rules.

2. History management

Preserve: user goals, constraints, unresolved tool errors, pending confirmations.

See `references/history_compression.md` for summarization prompts and pitfalls.

3. Retrieval into context

1. Query from user message + state summary 2. Retrieve candidates (RAG chunks, memories, tool outputs) 3. Deduplicate overlapping passages 4. Order by relevance; add source labels 5. Truncate with sentence boundaries; show "[truncated]" when cut

See `references/retrieval_packing.md` for packing algorithms and citation format.

4. Caching and prefetch

• Cache stable prefix (system + tools) where provider supports prompt caching
• Prefetch retrieval while user types (optional)
• Invalidate cache on prompt version change

See `references/caching_prefetch.md` for provider notes and invalidation.

5. Debug context failures

Log token counts per block in dev/staging.

See `references/debugging_context.md` for instrumentation checklist.

When to load references

• Layout and budgets → references/context_layout.md
• Summarization → references/history_compression.md
• RAG/memory packing → references/retrieval_packing.md
• Caching → references/caching_prefetch.md
• Debugging → references/debugging_context.md

Caching and prefetch

Table of contents

1. Prompt caching 2. Invalidation 3. Prefetch

Prompt caching

Cache stable prefix: system prompt + tool definitions + static RAG boilerplate.

Measure cache hit rate and $ savings in observability.

Invalidation

Bump prompt_version in config; new version = new cache key.

Prefetch

On typing pause, start retrieval; discard if user message changes materially (hash mismatch).

Context layout

Table of contents

1. Section template 2. Delimiter rules 3. Token counting

Section template

<system>...</system>
<tools>...</tools>
<retrieved_context>...</retrieved_context>
<conversation_summary>...</conversation_summary>
<recent_messages>...</recent_messages>
<user_message>...</user_message>

Place immutable instructions in <system>. Place volatile retrieval above recent messages.

Delimiter rules

• Use rare tags unlikely in user content
• For untrusted content: wrap in <untrusted source="rag"> and instruct model not to follow instructions inside

Token counting

Pre-flight count per block; if over budget, compress in order:

1. Oldest history 2. Lowest-scoring retrieval chunks 3. Verbose tool outputs (summarize)

Never truncate <system> or current <user_message> without explicit product approval.

Debugging context

Table of contents

1. Instrumentation 2. Replay

Instrumentation

Log per request (redacted):

• Token count per block
• Retrieval IDs and scores
• Prompt version
• Model ID

Store last assembled context in secure debug mode for support (time-limited).

Replay

Re-run assembly with same inputs in staging to reproduce "ignored instruction" bugs.

Compare diffs when prompt or compression changes.

History compression

Table of contents

1. When to summarize 2. Summary prompt 3. Pitfalls

When to summarize

Trigger when history tokens > 40% of budget or message count > 30.

Summary prompt

Preserve verbatim:

• User goals and constraints
• Decisions made
• Open questions
• Tool errors and IDs
• Numbers, dates, names

Omit: pleasantries, repeated tool retries that succeeded.

Pitfalls

• Summarizing away safety refusals
• Losing "do not do X" constraints
• Double summarization drift—re-summarize from raw episodic store if available

Retrieval packing

Table of contents

1. Deduplication 2. Ordering 3. Citations

Deduplication

Merge chunks with >85% overlap; keep highest score passage.

Ordering

Sort by relevance score descending; group by source document for readability.

Citations

Each chunk: [doc_id:chunk_id] in header line.

Instruct: answer only from cited chunks; say insufficient context otherwise.