Arize Prompt Optimization

Arize Prompt Optimization Skill

`SPACE` — All --space flags and the ARIZE_SPACE env var accept a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list.

Concepts

Where Prompts Live in Trace Data

LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:

Finding Prompts by Span Kind

• LLM span (attributes.openinference.span.kind = 'LLM'): Check attributes.llm.input_messages for structured chat messages, OR attributes.input.value for a serialized prompt. Check attributes.llm.prompt_template.template for the template.
• Chain/Agent span: attributes.input.value contains the user's question. The actual LLM prompt lives on child LLM spans -- navigate down the trace tree.
• Tool span: attributes.input.value has tool input, attributes.output.value has tool result. Not typically where prompts live.

Performance Signal Columns

These columns carry the feedback data used for optimization:

Prerequisites

Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.

If an ax command fails, troubleshoot based on the error:

• command not found or version error → see references/ax-setup.md
• 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
• Space unknown → run ax spaces list to pick by name, or ask the user
• Project unclear → ask the user, or run ax projects list -o json --limit 100 and present as selectable options
• LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run ax ai-integrations list --space SPACE to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
• Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. If credentials are not available through these channels, ask the user.

Phase 1: Extract the Current Prompt

Find LLM spans containing prompts

# Sample LLM spans (where prompts live)
ax spans export PROJECT --filter "attributes.openinference.span.kind = 'LLM'" -l 10 --stdout

# Filter by model
ax spans export PROJECT --filter "attributes.llm.model_name = 'gpt-4o'" -l 10 --stdout

# Filter by span name (e.g., a specific LLM call)
ax spans export PROJECT --filter "name = 'ChatCompletion'" -l 10 --stdout

Export a trace to inspect prompt structure

# Export all spans in a trace
ax spans export PROJECT --trace-id TRACE_ID

# Export a single span
ax spans export PROJECT --span-id SPAN_ID

Extract prompts from exported JSON

# Extract structured chat messages (system + user + assistant)
jq '.[0] | {
  messages: .attributes.llm.input_messages,
  model: .attributes.llm.model_name
}' trace_*/spans.json

# Extract the system prompt specifically
jq '[.[] | select(.attributes.llm.input_messages.roles[]? == "system")] | .[0].attributes.llm.input_messages' trace_*/spans.json

# Extract prompt template and variables
jq '.[0].attributes.llm.prompt_template' trace_*/spans.json

# Extract from input.value (fallback for non-structured prompts)
jq '.[0].attributes.input.value' trace_*/spans.json

Reconstruct the prompt as messages

Once you have the span data, reconstruct the prompt as a messages array:

[
  {"role": "system", "content": "You are a helpful assistant that..."},
  {"role": "user", "content": "Given {input}, answer the question: {question}"}
]

If the span has attributes.llm.prompt_template.template, the prompt uses variables. Preserve these placeholders ({variable} or {{variable}}) -- they are substituted at runtime.

Phase 2: Gather Performance Data

From traces (production feedback)

# Find error spans -- these indicate prompt failures
ax spans export PROJECT \
  --filter "status_code = 'ERROR' AND attributes.openinference.span.kind = 'LLM'" \
  -l 20 --stdout

# Find spans with low eval scores
ax spans export PROJECT \
  --filter "annotation.correctness.label = 'incorrect'" \
  -l 20 --stdout

# Find spans with high latency (may indicate overly complex prompts)
ax spans export PROJECT \
  --filter "attributes.openinference.span.kind = 'LLM' AND latency_ms > 10000" \
  -l 20 --stdout

# Export error traces for detailed inspection
ax spans export PROJECT --trace-id TRACE_ID

From datasets and experiments

# Export a dataset (ground truth examples)
ax datasets export DATASET_NAME --space SPACE
# -> dataset_*/examples.json

# Export experiment results (what the LLM produced)
ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE
# -> experiment_*/runs.json

Merge dataset + experiment for analysis

Join the two files by example_id to see inputs alongside outputs and evaluations:

# Count examples and runs
jq 'length' dataset_*/examples.json
jq 'length' experiment_*/runs.json

# View a single joined record
jq -s '
  .[0] as $dataset |
  .[1][0] as $run |
  ($dataset[] | select(.id == $run.example_id)) as $example |
  {
    input: $example,
    output: $run.output,
    evaluations: $run.evaluations
  }
' dataset_*/examples.json experiment_*/runs.json

# Find failed examples (where eval score < threshold)
jq '[.[] | select(.evaluations.correctness.score < 0.5)]' experiment_*/runs.json

Identify what to optimize

Look for patterns across failures:

1. Compare outputs to ground truth: Where does the LLM output differ from expected? 2. Read eval explanations: eval.*.explanation tells you WHY something failed 3. Check annotation text: Human feedback describes specific issues 4. Look for verbosity mismatches: If outputs are too long/short vs ground truth 5. Check format compliance: Are outputs in the expected format?

Phase 3: Optimize the Prompt

The Optimization Meta-Prompt

Use this template to generate an improved version of the prompt. Fill in the three placeholders and send it to your LLM (GPT-4o, Claude, etc.):

```` You are an expert in prompt optimization. Given the original baseline prompt and the associated performance data (inputs, outputs, evaluation labels, and explanations), generate a revised version that improves results.

ORIGINAL BASELINE PROMPT ========================

{PASTE_ORIGINAL_PROMPT_HERE}

========================

PERFORMANCE DATA ================

The following records show how the current prompt performed. Each record includes the input, the LLM output, and evaluation feedback:

{PASTE_RECORDS_HERE}

================

HOW TO USE THIS DATA

1. Compare outputs: Look at what the LLM generated vs what was expected 2. Review eval scores: Check which examples scored poorly and why 3. Examine annotations: Human feedback shows what worked and what didn't 4. Identify patterns: Look for common issues across multiple examples 5. Focus on failures: The rows where the output DIFFERS from the expected value are the ones that need fixing

ALIGNMENT STRATEGY

• If outputs have extra text or reasoning not present in the ground truth,

remove instructions that encourage explanation or verbose reasoning

• If outputs are missing information, add instructions to include it
• If outputs are in the wrong format, add explicit format instructions
• Focus on the rows where the output differs from the target -- these are

the failures to fix

RULES

Maintain Structure:

• Use the same template variables as the current prompt ({var} or {{var}})
• Don't change sections that are already working
• Preserve the exact return format instructions from the original prompt

Avoid Overfitting:

• DO NOT copy examples verbatim into the prompt
• DO NOT quote specific test data outputs exactly
• INSTEAD: Extract the ESSENCE of what makes good vs bad outputs
• INSTEAD: Add general guidelines and principles
• INSTEAD: If adding few-shot examples, create SYNTHETIC examples that

demonstrate the principle, not real data from above

Goal: Create a prompt that generalizes well to new inputs, not one that memorizes the test data.

OUTPUT FORMAT

Return the revised prompt as a JSON array of messages:

[ {"role": "system", "content": "..."}, {"role": "user", "content": "..."} ]

Also provide a brief reasoning section (bulleted list) explaining:

• What problems you found
• How the revised prompt addresses each one

````

Preparing the performance data

Format the records as a JSON array before pasting into the template:

# From dataset + experiment: join and select relevant columns
jq -s '
  .[0] as $ds |
  [.[1][] | . as $run |
    ($ds[] | select(.id == $run.example_id)) as $ex |
    {
      input: $ex.input,
      expected: $ex.expected_output,
      actual_output: $run.output,
      eval_score: $run.evaluations.correctness.score,
      eval_label: $run.evaluations.correctness.label,
      eval_explanation: $run.evaluations.correctness.explanation
    }
  ]
' dataset_*/examples.json experiment_*/runs.json

# From exported spans: extract input/output pairs with annotations
jq '[.[] | select(.attributes.openinference.span.kind == "LLM") | {
  input: .attributes.input.value,
  output: .attributes.output.value,
  status: .status_code,
  model: .attributes.llm.model_name
}]' trace_*/spans.json

Applying the revised prompt

After the LLM returns the revised messages array:

1. Compare the original and revised prompts side by side 2. Verify all template variables are preserved 3. Check that format instructions are intact 4. Test on a few examples before full deployment

Phase 4: Iterate

The optimization loop

1. Extract prompt    -> Phase 1 (once)
2. Run experiment    -> ax experiments create ...
3. Export results    -> ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE
4. Analyze failures  -> jq to find low scores
5. Run meta-prompt   -> Phase 3 with new failure data
6. Apply revised prompt
7. Repeat from step 2

Measure improvement

# Compare scores across experiments
# Experiment A (baseline)
jq '[.[] | .evaluations.correctness.score] | add / length' experiment_a/runs.json

# Experiment B (optimized)
jq '[.[] | .evaluations.correctness.score] | add / length' experiment_b/runs.json

# Find examples that flipped from fail to pass
jq -s '
  [.[0][] | select(.evaluations.correctness.label == "incorrect")] as $fails |
  [.[1][] | select(.evaluations.correctness.label == "correct") |
    select(.example_id as $id | $fails | any(.example_id == $id))
  ] | length
' experiment_a/runs.json experiment_b/runs.json

A/B compare two prompts

1. Create two experiments against the same dataset, each using a different prompt version 2. Export both: ax experiments export EXP_A and ax experiments export EXP_B 3. Compare average scores, failure rates, and specific example flips 4. Check for regressions -- examples that passed with prompt A but fail with prompt B

Prompt Engineering Best Practices

Apply these when writing or revising prompts:

Variable preservation

When optimizing prompts that use template variables:

• Single braces ({variable}): Python f-string / Jinja style. Most common in Arize.
• Double braces ({{variable}}): Mustache style. Used when the framework requires it.
• Never add or remove variable placeholders during optimization
• Never rename variables -- the runtime substitution depends on exact names
• If adding few-shot examples, use literal values, not variable placeholders

Workflows

Optimize a prompt from a failing trace

1. Find failing traces:

   ax traces list PROJECT --filter "status_code = 'ERROR'" --limit 5

2. Export the trace:

   ax spans export PROJECT --trace-id TRACE_ID

3. Extract the prompt from the LLM span:

   jq '[.[] | select(.attributes.openinference.span.kind == "LLM")][0] | {
     messages: .attributes.llm.input_messages,
     template: .attributes.llm.prompt_template,
     output: .attributes.output.value,
     error: .attributes.exception.message
   }' trace_*/spans.json

4. Identify what failed from the error message or output 5. Fill in the optimization meta-prompt (Phase 3) with the prompt and error context 6. Apply the revised prompt

Optimize using a dataset and experiment

1. Find the dataset and experiment:

   ax datasets list --space SPACE
   ax experiments list --dataset DATASET_NAME --space SPACE

2. Export both:

   ax datasets export DATASET_NAME --space SPACE
   ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE

3. Prepare the joined data for the meta-prompt 4. Run the optimization meta-prompt 5. Create a new experiment with the revised prompt to measure improvement

Debug a prompt that produces wrong format

1. Export spans where the output format is wrong:

   ax spans export PROJECT \
     --filter "attributes.openinference.span.kind = 'LLM' AND annotation.format.label = 'incorrect'" \
     -l 10 --stdout > bad_format.json

2. Look at what the LLM is producing vs what was expected 3. Add explicit format instructions to the prompt (JSON schema, examples, delimiters) 4. Common fix: add a few-shot example showing the exact desired output format

Reduce hallucination in a RAG prompt

1. Find traces where the model hallucinated:

   ax spans export PROJECT \
     --filter "annotation.faithfulness.label = 'unfaithful'" \
     -l 20 --stdout

2. Export and inspect the retriever + LLM spans together:

   ax spans export PROJECT --trace-id TRACE_ID
   jq '[.[] | {kind: .attributes.openinference.span.kind, name, input: .attributes.input.value, output: .attributes.output.value}]' trace_*/spans.json

3. Check if the retrieved context actually contained the answer 4. Add grounding instructions to the system prompt: "Only use information from the provided context. If the answer is not in the context, say so."

Troubleshooting

ax Profile Setup

Consult this when authentication fails (401, missing profile, missing API key). Do NOT run these checks proactively.

Use this when there is no profile, or a profile has incorrect settings (wrong API key, wrong region, etc.).

1. Inspect the current state

ax profiles show

Look at the output to understand what's configured:

• API Key: (not set) or missing → key needs to be created/updated
• No profile output or "No profiles found" → no profile exists yet
• Connected but getting 401 Unauthorized → key is wrong or expired
• Connected but wrong endpoint/region → region needs to be updated

2. Fix a misconfigured profile

If a profile exists but one or more settings are wrong, patch only what's broken.

Never pass a raw API key value as a flag. Always reference it via the ARIZE_API_KEY environment variable. If the variable is not already set in the shell, instruct the user to set it first, then run the command:

# If ARIZE_API_KEY is already exported in the shell:
ax profiles update --api-key $ARIZE_API_KEY

# Fix the region (no secret involved — safe to run directly)
ax profiles update --region us-east-1b

# Fix both at once
ax profiles update --api-key $ARIZE_API_KEY --region us-east-1b

update only changes the fields you specify — all other settings are preserved. If no profile name is given, the active profile is updated.

3. Create a new profile

If no profile exists, or if the existing profile needs to point to a completely different setup (different org, different region):

Always reference the key via `$ARIZE_API_KEY`, never inline a raw value.

# Requires ARIZE_API_KEY to be exported in the shell first
ax profiles create --api-key $ARIZE_API_KEY

# Create with a region
ax profiles create --api-key $ARIZE_API_KEY --region us-east-1b

# Create a named profile
ax profiles create work --api-key $ARIZE_API_KEY --region us-east-1b

To use a named profile with any ax command, add -p NAME:

ax spans export PROJECT -p work

4. Getting the API key

Never ask the user to paste their API key into the chat. Never log, echo, or display an API key value.

If ARIZE_API_KEY is not already set, instruct the user to export it in their shell:

export ARIZE_API_KEY="..."   # user pastes their key here in their own terminal

They can find their key at https://app.arize.com/admin > API Keys. Recommend they create a scoped service key (not a personal user key) — service keys are not tied to an individual account and are safer for programmatic use. Keys are space-scoped — make sure they copy the key for the correct space.

Once the user confirms the variable is set, proceed with ax profiles create --api-key $ARIZE_API_KEY or ax profiles update --api-key $ARIZE_API_KEY as described above.

5. Verify

After any create or update:

ax profiles show

Confirm the API key and region are correct, then retry the original command.

Space

There is no profile flag for space. Save it as an environment variable — accepts a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list -o json.

macOS/Linux — add to ~/.zshrc or ~/.bashrc:

export ARIZE_SPACE="my-workspace"    # name or base64 ID

Then source ~/.zshrc (or restart terminal).

Windows (PowerShell):

[System.Environment]::SetEnvironmentVariable('ARIZE_SPACE', 'my-workspace', 'User')

Restart terminal for it to take effect.

Save Credentials for Future Use

At the end of the session, if the user manually provided any credentials during this conversation and those values were NOT already loaded from a saved profile or environment variable, offer to save them.

Skip this entirely if:

• The API key was already loaded from an existing profile or ARIZE_API_KEY env var
• The space was already set via ARIZE_SPACE env var
• The user only used base64 project IDs (no space was needed)

How to offer: Use AskQuestion: "Would you like to save your Arize credentials so you don't have to enter them next time?" with options "Yes, save them" / "No thanks".

If the user says yes:

1. API key — Run ax profiles show to check the current state. Then run ax profiles create --api-key $ARIZE_API_KEY or ax profiles update --api-key $ARIZE_API_KEY (the key must already be exported as an env var — never pass a raw key value).

2. Space — See the Space section above to persist it as an environment variable.

ax CLI — Troubleshooting

Consult this only when an ax command fails. Do NOT run these checks proactively.

Check version first

If ax is installed (not command not found), always run ax --version before investigating further. The version must be 0.14.0 or higher — many errors are caused by an outdated install. If the version is too old, see Version too old below.

ax: command not found

macOS/Linux: 1. Check common locations: ~/.local/bin/ax, ~/Library/Python/*/bin/ax 2. Install: uv tool install arize-ax-cli (preferred), pipx install arize-ax-cli, or pip install arize-ax-cli 3. Add to PATH if needed: export PATH="$HOME/.local/bin:$PATH"

Windows (PowerShell): 1. Check: Get-Command ax or where.exe ax 2. Common locations: %APPDATA%\Python\Scripts\ax.exe, %LOCALAPPDATA%\Programs\Python\Python*\Scripts\ax.exe 3. Install: pip install arize-ax-cli 4. Add to PATH: $env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"

Version too old (below 0.14.0)

Upgrade: uv tool install --force --reinstall arize-ax-cli, pipx upgrade arize-ax-cli, or pip install --upgrade arize-ax-cli

SSL/certificate error

• macOS: export SSL_CERT_FILE=/etc/ssl/cert.pem
• Linux: export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
• Fallback: export SSL_CERT_FILE=$(python -c "import certifi; print(certifi.where())")

Subcommand not recognized

Upgrade ax (see above) or use the closest available alternative.

Still failing

Stop and ask the user for help.