Prompt Optimizer

Prompt Optimizer

Optimize prompts with evals. Keep every instruction, example, and external context reference causal.

Load Only What You Need

Step 1: Capture Contract

Record before editing:

• task type: new, refine, port, or debug
• target model family and snapshot, if known
• prompt surface: system, developer, user, tool descriptions, examples, schemas
• layer owners: platform, deployer/persona, retrieved context, user payload
• objective and non-goals
• inputs, tools, and external files available
• required output shape
• success criteria and failure cases
• hard constraints: latency, verbosity, safety, budget, tool use, style

If success criteria or examples are missing, create a small eval set first. If the bottleneck is model choice, retrieval, tool schema, or missing evals, say so before rewriting.

Step 2: Inventory External Context

For repo or agent prompts, list stable context by exact path:

Rules:

• Reference stable files by repo-relative path instead of copying them.
• Paste only excerpts needed for the prompt or eval case.
• Mark whether a file is loaded, referenced, or out of scope.
• Avoid vague context pointers such as "read the docs".

Step 3: Choose Model Strategy

Read references/model-family-notes.md.

• Known family: optimize for that family.
• Unknown family: write a portable base plus short adapter notes.
• Snapshot changes: rerun evals.
• Cross-family divergence: specialize only the failing layer.

Step 4: Shape Prompt

Read references/core-patterns.md.

• Put stable policy in system or developer.
• Put task-local facts, retrieved context, and variables in user-facing sections.
• Keep one owner per behavior rule.
• Use headings or tags only to separate content types.
• Put tool policy in prompt text; keep schemas in provider-native tools.
• Keep persona light unless it changes behavior.
• Use the shortest wording that preserves the constraint.
• Cut filler, repeated reminders, dead examples, and rationale that does not affect evals.

Step 5: Optimize

Read references/meta-optimization-loop.md for refinements.

1. Baseline the current prompt on the same eval slice. 2. Cluster failures by root cause. 3. Write concrete edit criticisms. 4. Generate two to four candidates:

• minimal-diff repair
• structure-first rewrite
• examples-first or tool-rule variant
• provider adapter when needed

5. Compare candidates on the same cases. 6. Keep a short optimization log. 7. Validate the winner on holdout cases. 8. Stop on plateau, oscillation, overfit, excessive cost, or non-prompt bottleneck.

Step 6: Return Package

Return:

1. Target 2. Success Criteria 3. External Context 4. Optimized Prompt 5. Adapter Notes 6. Eval Set 7. Optimization Log 8. Residual Risks

For existing prompts, include a concise diff-style note of the main behavioral changes.

Failure Modes

• editing before defining the eval target
• mixing policy, examples, and raw context without boundaries
• duplicating rules across layers
• putting durable policy in user payloads
• asking for chain-of-thought
• keeping contradictory legacy instructions
• overfitting to one or two examples
• retaining examples that no longer improve evals
• fixing tool-use failures only in prompt text when tool descriptions or schemas are weak
• adding markup that does not reduce ambiguity
• using persona as a substitute for behavior rules

Core Patterns

When markers help

Use markers when the prompt mixes different content types:

• instructions
• retrieved context
• tool rules
• examples
• input payloads
• output schema

Good section names are concrete and stable:

• <role>
• <goal>
• <context>
• <tool_policy>
• <workflow>
• <constraints>
• <output_format>

Do not add markup around every sentence. Markers are useful when they carve the prompt into distinct blocks, not when they add noise.

If the target stack or model family responds better to plain markdown, use headings and bullets instead of XML-style tags. The structure matters more than the syntax.

Where rules live

Rules:

• Keep descriptive markers descriptive.
• Put directives in canonical rules sections.
• For state-conditional rules, write the directive in a rules section and reference the state by name.

Layer the prompt correctly

Keep these layers separate:

1. Stable behavior:

• role
• defaults
• safety boundaries
• tool-use policy
• output contract

2. Task-local context:

• user request
• retrieved documents
• current state
• dynamic variables

3. Examples:

• few-shot examples
• failure replays
• structured output demonstrations

If a rule should apply across tasks, keep it out of the user payload and put it in the stable layer. Keep one authoritative owner for each major behavior rule instead of repeating it across all layers.

Collapse common duplicates such as:

• verbosity rules
• "check before guessing"
• ask-first versus act-now behavior
• output format rules
• refusal and escalation boundaries

When prompts are long, separate policy from evidence explicitly:

• instructions in one block
• retrieved documents in another
• examples in another
• tool policy (when/why/whether) in its own labeled section; tool schemas stay in the provider-native tools parameter

For long-context prompts, place long evidence before the final query and keep the actual ask in a terminal section. Do not cargo-cult this ordering into short prompts that do not need it.

Layered prompts with multiple owners

When a runtime concatenates prompt layers from different owners:

Rules:

• Platform behavior must work when AGENTS.md, CLAUDE.md, or persona files are empty or customized.
• Do not delete platform rules because a deployer layer "probably covers it".
• Put load-bearing rules in the highest stable owner.

External file inventories

Use path inventories when prompts depend on repo docs, specs, or policies.

<external_files>
- `AGENTS.md` - repo agent rules; loaded
- `docs/api.md` - API contract; reference before endpoint changes
- `SECURITY.md` - security policy; reference for disclosure or auth changes
</external_files>

Rules:

• Enumerate exact repo-relative paths.
• Mark each file as loaded, referenced, or out of scope.
• Paste only excerpts needed for the prompt or eval case.
• Prefer docs/api.md over "read the docs".
• Keep stable docs outside the prompt when the runtime can retrieve files.

Tool policy placement

• Tool schemas belong in provider-native tools.
• Prompt text carries tool policy: when, why, whether, evidence, and stop conditions.
• Naming a tool in policy is fine; re-enumerating schemas is not.

High-value prompt moves

• Tell the agent what to do by default.
• State when it should ask clarifying questions and when it should infer.
• State when it should use tools and what evidence it should gather before acting.
• State what counts as completion.
• State what counts as refusal, escalation, or defer.
• Enumerate external files by path when specs, docs, or policies matter.
• Separate hard constraints from preferences.
• Keep progress-update style explicit if the user should see it.
• Use the shortest wording that preserves the intended behavioral constraint.
• Remove persona, motivation, or reminder text that does not change measured behavior.
• Place directives in canonical rules sections (<behavior>, <constraints>, <tool_policy>, <workflow>), not buried inside descriptive markers like <context>, <state>, or <turn-state>.

Symptom to fix mapping

Meta Optimization Loop

Use this file when refining an existing prompt or when a first draft needs disciplined iteration.

Inputs

Collect these before iterating:

• current prompt or first draft
• target model family
• optimizer model, if different from the deployment model
• representative eval cases
• scoring rubric
• known failures
• hard constraints

Optimization loop

1. Baseline

Run the current prompt on a representative slice.

Score at least:

• instruction following
• output shape
• tool behavior
• refusal and escalation correctness
• brevity or verbosity
• robustness on ambiguous inputs
• cost, latency, or tool-call efficiency when those matter

Preserve a structured evidence pack for each run:

• prompt version
• eval case
• model output
• relevant trace or tool behavior
• failure reason
• scores

2. Failure clustering

Group failures before editing:

• ambiguity
• missing constraints
• conflicting rules
• bad example fit
• weak tool instructions
• weak stop conditions
• provider mismatch
• prompt bloat or duplicated rules

Do not patch each failing case independently if the same root cause appears in multiple places.

3. Textual gradients

Write criticisms as concrete directional edits:

• "Clarify when to ask before acting."
• "Separate retrieved context from instructions."
• "Move tool rules into an explicit section."
• "Replace anti-pattern examples with positive demonstrations."

Avoid vague critiques like "make it better" or "be clearer."

4. Candidate beam

Generate two to four candidates:

• minimal-diff repair
• structure-first rewrite
• examples-first or tool-rule-centered variant
• provider-specific adapter when needed

Keep one low-risk candidate in the beam so you can tell whether a larger rewrite actually helped.

5. Compare on the same slice

Use the same eval cases for the candidate round.

Do not compare candidates on different inputs and assume the scores are comparable.

6. Reflective memory

Keep a short log:

This prevents looping back into edits that already failed. Record deletions and compaction edits, not just additions.

7. Holdout validation

Before finalizing:

• test the winner on holdout cases
• replay the original failure cases
• verify the fix did not regress happy-path behavior

8. Stop conditions

Stop prompt editing when:

• scores plateau across rounds
• edits oscillate between two behaviors
• the remaining failures are caused by model or tool limitations
• you are clearly overfitting to the eval slice
• cost rises without enough quality gain to justify the extra prompt complexity

Practical defaults

• Use five to ten cases for a quick pass.
• Keep at least one holdout case.
• Edit one or two causal dimensions per round.
• Optimize examples, tool descriptions, and output contracts alongside the core prompt.
• For repeated optimization, keep fixed train, validation, and holdout slices so before/after comparisons stay meaningful.
• When budget allows, consider using a stronger optimizer pass than the eventual deployment model.

What this loop is borrowing from

• score-driven candidate search rather than one linear rewrite
• explicit critique before revision
• lightweight reflection memory across rounds
• holdout validation to resist overfitting
• structured evidence capture so revisions are grounded in actual outputs and scores

Use the loop pragmatically. The point is to learn which edits change behavior, not to imitate a paper mechanically.

Model Family Notes

Use this file to adapt prompts to model behavior instead of assuming all model families respond the same way.

OpenAI

Reasoning models

• Prefer straightforward developer instructions with a clear end goal.
• Keep prompts simple and direct.
• Avoid explicitly requesting chain-of-thought or "think step by step" behavior.
• Use delimiters such as markdown headings, XML tags, or section titles when the prompt mixes multiple content blocks.
• Try zero-shot first. Add few-shot examples only when the output contract or edge cases need them.
• Be explicit about constraints, success criteria, and completion conditions.
• Tool schemas are disclosed via the Responses API tools parameter. Keep tool policy (when/why/whether to call) in the prompt; do not restate tool names or argument schemas.

GPT-style non-reasoning models

• Use more explicit instructions than you would for a reasoning model.
• Spell out the logic, data, and schema the model should use.
• Precise format instructions and tightly matched examples are especially helpful.
• Good default for tasks where deterministic formatting matters more than open-ended planning.

Anthropic Claude

• Clear, direct instructions outperform vague prompting.
• A lightweight role statement can materially improve consistency.
• XML-style tags are especially useful when mixing instructions, context, examples, and variable inputs.
• Three to five strong examples are often a good default when examples are needed.
• For long context, place long documents before the question and put the actual query near the end.
• When grounding in long documents, asking for relevant quotes first can improve downstream analysis.
• If tool use or progress-update behavior matters, specify it explicitly rather than assuming the model will infer it.
• When you call the Messages API with tools, the API injects the tool definitions into a special system prompt automatically. Keep your user-authored system prompt focused on policy; put tool detail in each tool's description field rather than re-listing schemas in prose.

Gemini

• Use clear, specific instructions.
• Few-shot examples are recommended by default; keep them structurally consistent.
• Prefer positive demonstrations over anti-pattern-only demonstrations.
• Break complex tasks into smaller components or chained prompts when one large prompt becomes hard to steer.
• Use system instructions when the target runtime supports them.
• Thinking is dynamic by default on modern Gemini thinking models; tune it only when latency or deeper reasoning warrants it.
• Gemini long-context workflows can benefit from many-shot in-context learning when you have a large bank of representative examples.
• Tool schemas are disclosed via the Gemini API tools (function declarations) parameter. Keep the prompt focused on tool policy; do not re-list function names or parameter schemas.

Cross-family adapter rules

• Keep a provider-agnostic base prompt whenever possible.
• Add a small adapter layer per family rather than forking the entire prompt immediately.
• Retest after changing model family or snapshot.
• Do not transport reasoning-model assumptions, few-shot defaults, or formatting quirks blindly across providers.
• When behavior diverges sharply, keep the shared contract stable and specialize only the parts that actually fail in evals.

Transformed Examples

Use these examples when the task is under-specified or when you need a stronger default shape.

Example 1: Happy-path new agent prompt

Input brief

"Write a system prompt for a coding agent that should act by default, use tools, and keep the user updated."

Transformed prompt

<role>
You are a pragmatic coding agent working in the user's repository.
</role>

<goal>
Implement the user's requested change end-to-end when feasible.
Do not stop at analysis if you can safely gather facts and act.
</goal>

<tool_policy>
Use tools to inspect the workspace before assuming facts.
Read before write. Validate the changed surface before finishing.
</tool_policy>

<external_files>
Reference exact repo files such as `AGENTS.md`, `CONTRIBUTING.md`, or
`docs/api.md` when they govern the task.
</external_files>

<workflow>
1. Restate the objective briefly.
2. Inspect the relevant files or state.
3. Implement the change.
4. Run the most relevant validation.
5. Report the outcome and any remaining risks.
</workflow>

<constraints>
Ask before destructive actions or risky scope expansion.
Keep progress updates brief and factual.
</constraints>

Why it is better:

• explicit default behavior
• explicit tool-use trigger
• path-backed external context
• explicit validation step
• explicit escalation boundary

Example 2: Robust variant for a weak existing prompt

Before

"Be helpful, think step by step, and suggest changes to the repo."

After

<default_behavior>
Default to implementation when the user's intent is execution rather than discussion.
Use tools to discover missing facts instead of guessing.
</default_behavior>

<tool_rules>
Use repository tools whenever correctness depends on current files, logs, or config.
If a validation command exists for the changed surface, run it before finalizing.
</tool_rules>

<progress_updates>
Send short progress updates during long tasks.
Keep them factual and tied to concrete work completed.
</progress_updates>

<ask_first>
Ask before destructive actions, network-dependent actions, or major scope changes.
</ask_first>

Why it is better:

• replaces vague helpfulness with actionable defaults
• removes unnecessary chain-of-thought instruction
• moves tool use from implication to policy
• calibrates user-facing updates

Example 3: Anti-pattern and corrected version

Anti-pattern

You are the world's best genius agent.
Think step by step and explain every internal thought.
Never ask questions.
Always ask questions before acting.
Use tools only if absolutely necessary, but always use tools before answering.
Do not be verbose.
Provide extremely detailed explanations.

Corrected version

<role>
You are a reliable implementation agent.
</role>

<goal>
Complete the user's task accurately and efficiently.
</goal>

<tool_use>
Use tools when current repository facts, logs, or external state are needed.
</tool_use>

<external_files>
List exact files for stable specs, docs, and policies. Paste excerpts only
when the runtime cannot retrieve them.
</external_files>

<clarification>
Ask only when required information is missing or the action is risky.
</clarification>

<output_format>
Keep progress updates brief.
Keep the final answer concise and include validation plus open risks.
</output_format>

Why it is better:

• removes contradictory instructions
• removes chain-of-thought demand
• replaces vague context with exact file rules
• replaces absolute slogans with operational rules
• turns style goals into specific output behavior

Sources

This file tracks the material synthesized into prompt-optimizer.

Selected profile

• skills/skill-writer/references/examples/workflow-process-skill.md

Why: this skill is a repeatable prompt-optimization workflow with explicit preconditions, ordered execution, validation, and failure handling.

Current source inventory

Decisions

1. Use an eval-first workflow before prompt edits. Status: adopted Why: Anthropic explicitly recommends success criteria, tests, and a first draft before prompt engineering, and OpenAI's prompt optimizer centers graders and annotations.

2. Keep a provider-agnostic base prompt plus model-family adapters. Status: adopted Why: OpenAI, Anthropic, and Gemini each recommend materially different prompting strategies around reasoning, XML tags, examples, and thinking controls.

3. Use markers and tags selectively, not automatically. Status: adopted Why: OpenAI and Anthropic both recommend delimiters for clarity, but over-structuring adds noise and reduces readability.

4. Use a candidate-based optimization loop instead of one linear rewrite. Status: adopted Why: OPRO, ProTeGi, and DSPy all support score-driven search across multiple prompt variants.

5. Keep an explicit optimization log and reflective memory across rounds. Status: adopted Why: Self-Refine and Reflexion both show the value of critique and retained reflection across iterations.

6. Optimize examples, tool rules, and output contracts together with the core prompt. Status: adopted Why: Provider docs and DSPy both treat examples and instructions as first-class levers, not optional decoration.

7. Make prompt compaction and de-duplication a first-class optimization step. Status: adopted Why: OpenAI recommends simple, direct prompts; Gemini explicitly says to avoid passing tokens you do not need; Anthropic frames concision as a context-budget issue.

8. Make context ordering explicit rather than implicit. Status: adopted Why: Gemini and Anthropic both document that long-context prompts perform better when evidence comes before the final query.

9. Inventory stable external context by exact path. Status: adopted Why: AGENTS.md and Codex guidance favor explicit project instruction sources, and path inventories prevent prompts from copying stale docs or using vague "read the docs" pointers.

10. Add External Context to the returned prompt package. Status: adopted Why: another engineer should see which specs, docs, policies, and examples were loaded, referenced, or left out of scope.

Coverage matrix

Description QA

Should trigger

• "Improve this system prompt for our coding agent."
• "Optimize an agent prompt for Claude."
• "Rewrite this OpenAI developer prompt."
• "Port this prompt from GPT to Gemini."
• "Make this tool-using prompt more reliable."
• "Tune this prompt wording with a proper eval loop."

Should not trigger

• "Write a new skill."
• "Review my pull request."
• "Fix this bug in the repository."
• "Summarize this document."
• "Design a new model architecture."
• "Tune only the temperature and top-p settings."

Open gaps

1. Add provider notes for additional families only when the repository has a real downstream need. 2. If this skill starts getting heavy use, add an optional script or eval template for capturing prompt scores across rounds. 3. Re-check provider docs when major model-family updates land, especially around tool use and reasoning defaults. 4. Add durable before/after examples for external context inventories if future prompt changes regress into vague file references.

Stopping rationale

Further retrieval is currently low-yield for this first version. The source pack already covers:

• official prompt guidance from the three major model families most likely to matter in practice
• current optimizer framing from a live prompt-programming framework
• foundational prompt-optimization research for candidate search, critique loops, and reflection memory
• local repository conventions for canonical skill authoring and registration

Changelog

• 2026-04-18: Created the initial prompt-optimizer skill, references, and provenance record.
• 2026-04-18: Added an explicit prompt learnings pass covering compaction, deduplication, and context ordering.
• 2026-04-18: Folded the prompt learnings back into the core shaping and iteration guidance to keep the workflow compact.
• 2026-05-04: Added exact external context inventories, External Context output, concise runtime workflow, and SPEC.md.

Prompt Optimizer Specification

Intent

prompt-optimizer improves reusable prompts through a contract-first, eval-backed workflow.

It should produce shorter, more reliable prompt packages with explicit model strategy, external context inventory, eval evidence, and residual risks.

Scope

In scope:

• New agent, system, developer, and reusable prompt templates.
• Refining existing prompts from failures or examples.
• Porting prompts across OpenAI, Claude, Gemini, or unknown model families.
• Prompt eval set design, candidate comparison, and holdout checks.
• Layering stable policy, task-local context, examples, tool policy, and external file references.

Out of scope:

• Choosing model architecture or fine-tuning strategy except to flag when prompting is not the bottleneck.
• Rewriting product docs, specs, or policies referenced by a prompt.
• Creating reusable agent skills.
• Debugging repository code unrelated to prompt behavior.
• Asking models to reveal hidden reasoning or chain-of-thought.

Users And Trigger Context

• Primary users: engineers and agents maintaining prompts for coding agents, product agents, eval harnesses, and model integrations.
• Should trigger for: improve a prompt, optimize a system prompt, rewrite an agent prompt, tune prompt wording, make prompts reliable, port prompts across model families, build prompt evals.
• Should not trigger for: normal code review, PR writing, skill authoring, generic documentation editing, or parameter-only tuning.

Runtime Contract

• Capture the prompt contract before edits: target model, prompt surfaces, layer owners, objective, non-goals, inputs, tools, output shape, success criteria, failures, and hard constraints.
• Build or request a small eval set when success criteria or examples are missing.
• Inventory stable external context by exact repo-relative path.
• Reference docs/specs/policies by path; paste only necessary excerpts.
• Keep one authoritative owner per behavior rule.
• Compare candidates on the same eval slice.
• Validate the selected prompt on holdout cases.
• Return a reusable package with target, success criteria, external context, optimized prompt, adapter notes, eval set, optimization log, and residual risks.

Source And Evidence Model

Authoritative sources:

• Local prompt-optimizer runtime files and references.
• Repository instructions and skill-writer authoring rules.
• Official OpenAI, Anthropic, and Gemini prompting and eval guidance.
• Prompt optimization research and framework docs already captured in SOURCES.md.

Useful improvement sources:

• positive examples: prompts that meet eval targets with fewer tokens and cleaner layering
• negative examples: prompts with duplicated policy, vague context, stale examples, or weak tool rules
• eval outputs: failure clusters, holdout regressions, candidate scores, and optimization logs
• model changes: provider docs or release notes showing changed behavior, tool APIs, or reasoning defaults

Do not store secrets, customer data, private policy text, or long copyrighted prompt/source excerpts in examples.

Reference Architecture

• SKILL.md contains the runtime workflow and output contract.
• SPEC.md contains this maintenance contract.
• SOURCES.md stores source inventory, decisions, coverage, gaps, and changelog.
• references/core-patterns.md covers prompt structure, layering, markers, external files, tool policy, and symptom fixes.
• references/meta-optimization-loop.md covers eval-backed iteration.
• references/model-family-notes.md covers provider adapters.
• references/transformed-examples.md contains compact examples for new prompts, repairs, and anti-pattern correction.
• scripts/ and assets/ are unused unless prompt scoring automation or reusable templates become necessary.

Evaluation

• Lightweight validation: run representative prompt tasks through the contract checklist and verify the package includes external context, eval cases, and residual risks.
• Candidate validation: compare all candidates on the same working slice and at least one holdout case.
• Trigger QA: confirm prompt optimization requests trigger this skill while skill writing, code review, and parameter-only tuning do not.
• Acceptance gates: prompt is shorter or behaviorally justified, rules have one owner, external files are exact, examples are causal, and residual risks name non-prompt bottlenecks.

Known Limitations

• Prompt behavior can change across model snapshots; the skill must recommend re-running evals after model changes.
• The skill cannot prove prompt quality without representative eval cases.
• External files referenced by path are useful only when the runtime agent can access them.
• Provider-specific advice can drift; refresh official docs when model or API behavior matters.

Maintenance Notes

• Update SKILL.md when runtime workflow, output package, or failure modes change.
• Update SPEC.md when scope, evidence policy, evaluation, or reference architecture changes.
• Update SOURCES.md when source inventory, decisions, coverage, gaps, or changelog entries change.
• Keep reference files focused; split any file that mixes unrelated lookup needs.