Promptic

Promptic Python SDK

SDK and CLI for the Promptic platform — LLM tracing, prompt optimization, and agent evaluation.

Installation

pip install promptic-sdk

Install extras for auto-instrumentation:

# LLM providers
pip install promptic-sdk[openai]         # OpenAI
pip install promptic-sdk[anthropic]      # Anthropic
pip install promptic-sdk[bedrock]        # AWS Bedrock
pip install promptic-sdk[vertexai]       # Google Vertex AI
pip install promptic-sdk[mistralai]      # Mistral

# Agent frameworks
pip install promptic-sdk[langchain]      # LangChain / LangGraph / create_agent / deepagents
pip install promptic-sdk[openai-agents]  # OpenAI Agents SDK
pip install promptic-sdk[claude-agent]   # Claude Agent SDK

pip install promptic-sdk[all]            # Everything above

Pydantic AI ships its own OpenTelemetry emitter — enable with Agent(..., instrument=True), no extras needed.

Authentication

# Browser login (local dev)
promptic login

# CI/CD
export PROMPTIC_API_KEY="pk_..."

Config resolution: explicit args > env vars (PROMPTIC_API_KEY, PROMPTIC_ENDPOINT) > ~/.promptic/config.toml.

Tracing

Call promptic_sdk.init() once at startup. All LLM calls from installed providers are auto-instrumented via OpenTelemetry.

import promptic_sdk
from openai import OpenAI

promptic_sdk.init()
client = OpenAI()

with promptic_sdk.ai_component("my-agent"):
    response = client.chat.completions.create(
        model="gpt-4.1-nano",
        messages=[{"role": "user", "content": "Hello!"}],
    )

init() parameters

Auto-detected instrumentors: OpenAI, Anthropic, Google Generative AI, Vertex AI, Bedrock, Mistral, Cohere, LangChain (with LangGraph / create_agent / deepagents), OpenAI Agents SDK, Claude Agent SDK. All emit the official OpenTelemetry GenAI semantic conventions (gen_ai.*).

ai_component context manager

Tag spans with an AI Component name. The platform links traces to the matching component.

with promptic_sdk.ai_component("customer-support-agent"):
    # All LLM calls here are tagged
    ...

# With dataset and run tagging for evaluation:
with promptic_sdk.ai_component("my-agent", dataset="eval-set", run="v1-baseline"):
    agent.run(test_input)

Parameters:

• name (str): AI Component name in the workspace
• dataset (str, optional): Dataset name — traces auto-added to this dataset (created if needed)
• run (str, optional): Run name — groups traces within a dataset for comparison

dataset context manager

Tag spans with a dataset name independently:

with promptic_sdk.ai_component("my-agent"):
    with promptic_sdk.dataset("eval-round-1"):
        agent.run(test_input)

Tracing workflows with custom spans

Most users don't need this. With the right [extras] installed, auto-instrumentation already creates spans for every LLM and tool call. Reach for custom spans only when you have meaningful non-LLM workflow logic (retrieval, normalization, business rules, control flow) you want represented in the trace.

When you do need it, wrap your workflow stages in custom OpenTelemetry spans. Auto-instrumented provider spans automatically nest under whichever custom span is active.

Recommended pattern:

1. Wrap the whole run in one root workflow span inside ai_component(...). 2. Add a child task span for each meaningful stage of the pipeline. 3. Record the stage's input and output as span attributes so the trace reads as a transformation, not just a list of LLM calls.

import json
import promptic_sdk
from opentelemetry import trace

promptic_sdk.init()
tracer = trace.get_tracer(__name__)

with promptic_sdk.ai_component("my-agent"):
    with tracer.start_as_current_span("run_workflow") as root:
        root.set_attribute("traceloop.span.kind", "workflow")
        root.set_attribute("traceloop.entity.input", json.dumps(user_input))

        with tracer.start_as_current_span("retrieve_context") as span:
            span.set_attribute("traceloop.span.kind", "task")
            span.set_attribute("traceloop.entity.input", json.dumps(query))
            context = retrieve(query)
            span.set_attribute("traceloop.entity.output", json.dumps(context))

        with tracer.start_as_current_span("generate_answer") as span:
            span.set_attribute("traceloop.span.kind", "task")
            # Auto-instrumented LLM call nests under this task span
            answer = llm_call(context)

        root.set_attribute("traceloop.entity.output", json.dumps(answer))

Span attribute conventions:

• traceloop.span.kind="workflow" — the top-level run
• traceloop.span.kind="task" — an internal pipeline stage
• traceloop.entity.input / traceloop.entity.output — JSON-serialized stage payloads
• gen_ai.* — reserved for LLM/tool spans; auto-instrumentors emit these

Tips:

• Use semantic span names (retrieve_context, rerank_results) instead of generic function names when several calls would otherwise collide.
• For large payloads, log a small preview plus a count rather than the full object — traces are not meant to store data:

  span.set_attribute(
      "traceloop.entity.output",
      json.dumps({
          "items": items[:5],
          "item_count": len(items),
          "additional_item_count": max(len(items) - 5, 0),
      }),
  )

Verify with promptic traces get <trace-id> --json: the root workflow span should carry structured input/output, task spans should appear as its children, and auto-instrumented LLM/tool spans should nest under the task that triggered them.

Custom OpenTelemetry instrumentors

Since Promptic uses standard OpenTelemetry, add any OTel-compatible instrumentor:

import promptic_sdk
from opentelemetry.instrumentation.requests import RequestsInstrumentor

promptic_sdk.init()
RequestsInstrumentor().instrument()  # Spans exported to Promptic

LangGraph / deepagents integration

pip install promptic-sdk[langchain] installs OpenLLMetry's opentelemetry-instrumentation-langchain (≥0.60), which covers LangChain chains, LangGraph (create_agent), and deepagents with subagents. Emits the official OpenTelemetry GenAI semantic conventions (gen_ai.tool.definitions, gen_ai.operation.name, gen_ai.usage.*), so agent-evaluation insights (loops, tool errors, unused tools) work for flat agents and multi-agent graphs uniformly.

Users who prefer the LangSmith OTel bridge (e.g. for hybrid dual-export to LangSmith) can opt in by setting LANGSMITH_TRACING=true and LANGSMITH_OTEL_ENABLED=true before calling init(). Note: the LangSmith bridge does not emit tool definitions, so the "unused tools" insight will not fire on LangSmith-bridged traces.

API Client

Both sync (PrompticClient) and async (AsyncPrompticClient) clients with identical method signatures.

from promptic_sdk import PrompticClient

with PrompticClient() as client:
    traces = client.list_traces(limit=10)

from promptic_sdk import AsyncPrompticClient

async with AsyncPrompticClient() as client:
    traces = await client.list_traces(limit=10)

Constructor args: api_key, access_token, workspace_id, endpoint, timeout (default 30s).

API reference

For detailed method signatures and parameters, see references/api.md.

Agent Evaluation Workflow

Evaluate agent performance using datasets, runs, and evaluations.

Step 1: Run agent with tracing

Instrument the agent with dataset and run tagging — traces are auto-collected:

import promptic_sdk

promptic_sdk.init()

with promptic_sdk.ai_component("my-agent", dataset="eval-set", run="v2-improved"):
    for query in test_queries:
        agent.run(query)

Step 2: Trigger evaluation

Option A — CLI (recommended for agentic workflows):

# Find the component and dataset IDs
promptic components list --json
promptic datasets list --component <comp-id> --json
promptic runs list --component <comp-id> --json

# Run evaluation (waits for completion by default)
promptic evaluations run <comp-id> --dataset <ds-id> --run <run-id> --name "v2-eval"

# Or don't wait and check later
promptic evaluations run <comp-id> --dataset <ds-id> --run <run-id> --no-wait
promptic evaluations get <eval-id> --component <comp-id>

Option B — Python API:

from promptic_sdk import PrompticClient

with PrompticClient() as client:
    components = client.list_components()
    comp_id = components["data"][0]["id"]

    datasets = client.list_datasets(comp_id)
    ds_id = datasets["data"][0]["id"]

    evaluation = client.create_evaluation(comp_id, ds_id, name="v2-eval")
    result = client.wait_for_evaluation(comp_id, evaluation["id"])

    for insight in result["results"]["insights"]:
        print(f"[{insight['severity']}] {insight['title']}: {insight['description']}")

Prompt Optimization Workflow

Optimize prompts via experiments:

from promptic_sdk import PrompticClient

with PrompticClient() as client:
    # Create experiment
    exp = client.create_experiment(
        ai_component_id="comp_...",
        target_model="gpt-4.1-nano",
        task_type="classification",  # or "textGeneration", "structuredOutput"
        initial_prompt="Classify the following text into categories.",
        optimizer="prompticV2",      # or "miproV2", "bootstrapFewShot"
    )

    # Add training observations
    client.create_observations(exp["id"], [
        {"variables": {"message": "Great product!"}, "expected": "positive"},
        {"variables": {"message": "Terrible service"}, "expected": "negative"},
    ])

    # Add evaluators
    client.create_evaluators(exp["id"], [
        {"name": "accuracy", "type": "f1", "weight": 1.0},
    ])

    # Start optimization
    client.start_experiment(exp["id"])

    # After completion, deploy the best prompt
    best = client.get_best_iteration(exp["id"])
    client.deploy("comp_...", exp["id"])

    # Fetch deployed prompt at runtime
    prompt = client.get_deployed_prompt("comp_...")
    print(prompt["prompt"])

CLI

The promptic CLI mirrors the API client. All commands support --json for JSON output.

# Auth
promptic login                      # Browser auth (device flow)
promptic logout                     # Clear saved credentials
promptic configure                  # Save API key & endpoint (CI/CD)

# Workspace
promptic workspace info             # Show current workspace details
promptic workspace list             # List accessible workspaces
promptic workspace select <id>      # Select active workspace

# Traces
promptic traces list                # List recent traces
promptic traces get <trace-id>      # Get trace with spans and events
promptic traces stats               # Aggregated tracing stats

# Components
promptic components list            # List AI components
promptic components create <name>   # Create a component
promptic components get <id>        # Get component details
promptic components delete <id>     # Delete a component

# Experiments
promptic experiments list           # List experiments
promptic experiments create         # Create experiment (interactive wizard)
promptic experiments get <id>       # Get experiment details
promptic experiments update <id>    # Update a pending experiment
promptic experiments delete <id>    # Delete an experiment
promptic experiments start <id>     # Start optimization
promptic experiments duplicate <id> [--start] [-p PROMPT]    # Clone experiment (observations + evaluators)
promptic experiments continue <id> [--start]                 # Clone, seed initial prompt from source's best iteration

# Observations (training data)
promptic observations list <exp-id>              # List observations
promptic observations add <exp-id> --from-file f # Bulk import (CSV/JSONL/JSON)
promptic observations add <exp-id> -i "..." -e "..." # Add single observation
promptic observations delete <exp-id> <obs-id>   # Delete an observation

# Evaluators
promptic evaluators list <exp-id>                # List evaluators
promptic evaluators add <exp-id> -n <name> -t <type>  # Add evaluator
promptic evaluators delete <exp-id> <eval-id>    # Delete an evaluator

# Iterations
promptic iterations list <exp-id>   # List iterations
promptic iterations get <exp-id> <iter-id>  # Get iteration with scores
promptic iterations best <exp-id>   # Get best-scoring iteration

# Deployments
promptic deployments status <comp-id>            # Show active deployment
promptic deployments deploy <comp-id> <exp-id>   # Deploy experiment
promptic deployments prompt <comp-id>            # Show deployed prompt
promptic deployments undeploy <comp-id>          # Remove deployment

# Datasets
promptic datasets create --component <id> --name <n>  # Create dataset
promptic datasets list --component <id>          # List datasets
promptic datasets get <ds-id> --component <id>   # Get dataset with items
promptic datasets delete <ds-id> --component <id>  # Delete dataset

# Runs
promptic runs create --component <id> --dataset <ds-id>  # Create run
promptic runs list --component <id>              # List runs
promptic runs get <run-id> --component <id>      # Get run with traces
promptic runs delete <run-id> --component <id>   # Delete run

# Annotations
promptic annotations create --component <id> --run <r> --trace <t>  # Annotate trace
promptic annotations list --component <id> --run <r>    # List by run
promptic annotations list --component <id> --dataset <d>  # List by dataset
promptic annotations delete <ann-id> --component <id> --run <r>  # Delete

# Evaluations
promptic evaluations run <comp-id> --dataset <ds-id> --run <run-id>  # Run evaluation (--run required)
promptic evaluations list --component <id>       # List evaluations
promptic evaluations get <eval-id> --component <id>     # Get results

Key Types

Enums (Literal types):

• ExperimentStatus: "pending" | "scheduled" | "running" | "completed" | "failed"
• ModelProvider: "openai" | "openrouter" | "custom" | "google"
• TaskType: "classification" | "textGeneration" | "structuredOutput"
• EvaluatorType: "f1" | "referenceJudge" | "comparisonJudge" | "generalJudge" | "similarity" | "structuredOutput"
• OptimizerType: "promptic" | "prompticV2" | "miproV2" | "bootstrapFewShot" | "gepa"

Promptic Skills

AI agent skills for integrating with the Promptic platform.

Install

npx skills add prompticeu/promptic-skills

Skills

What are skills?

Skills are reusable capabilities for AI coding agents (Claude Code, Cursor, Windsurf, etc.). They provide procedural knowledge that helps agents accomplish specific tasks more effectively.

Learn more at skills.sh.

License

MIT

API Reference

Complete method signatures for PrompticClient and AsyncPrompticClient. Both clients share identical signatures — async methods are prefixed with await.

Traces

client.list_traces(*, limit=50, offset=0, status=None, start_after=None, start_before=None) -> TraceList
client.get_trace(trace_id: str) -> Trace
client.get_stats(*, days_back=30) -> TracingStats

• status: "ok" or "error"
• start_after / start_before: ISO timestamp strings

Workspace

client.get_workspace() -> Workspace

Components

client.list_components() -> ComponentList
client.create_component(name: str, *, description=None) -> ComponentCreated
client.get_component(component_id: str) -> Component
client.delete_component(component_id: str) -> None

Experiments

client.list_experiments(*, component_id=None, status=None, limit=50, offset=0) -> ExperimentList
client.create_experiment(
    ai_component_id: str,
    target_model: str,
    *,
    task_type="classification",      # "classification" | "textGeneration" | "structuredOutput"
    initial_prompt=None,
    name=None,
    description=None,
    provider="openai",               # "openai" | "openrouter" | "custom" | "google"
    optimizer="prompticV2",          # "promptic" | "prompticV2" | "miproV2" | "bootstrapFewShot" | "gepa"
    hyperparameters=None,            # {"epochs": int, "trainSplitRatio": float, "numFewShots": int, "enableCot": bool}
    initial_prediction_model_schema=None,
) -> Experiment
client.get_experiment(experiment_id: str) -> Experiment
client.update_experiment(experiment_id: str, **updates) -> Experiment
client.delete_experiment(experiment_id: str) -> None
client.start_experiment(experiment_id: str) -> ExperimentStarted
client.duplicate_experiment(
    experiment_id: str,
    *,
    continue_from_optimized=False,   # True = seed new experiment from source's best optimized prompt
    initial_prompt_override=None,    # Or override the initial prompt with custom text
) -> Experiment                       # Includes ``modelUnavailable`` flag when source's model is gone

Observations

client.list_observations(experiment_id: str) -> ObservationList
client.create_observations(experiment_id: str, observations: list[dict]) -> ObservationList
client.update_observation(experiment_id: str, observation_id: int, **data) -> Observation
client.delete_observation(experiment_id: str, observation_id: int) -> None

Observation dict format: {"variables": dict[str, Any], "expected": str, "split": str (optional, default "eval")}.

Evaluators

client.list_evaluators(experiment_id: str) -> EvaluatorList
client.create_evaluators(experiment_id: str, evaluators: list[dict]) -> EvaluatorList
client.update_evaluator(experiment_id: str, evaluator_id: str, **data) -> Evaluator
client.delete_evaluator(experiment_id: str, evaluator_id: str) -> None

Evaluator dict format: {"name": str, "type": "f1"|"referenceJudge"|"comparisonJudge"|"generalJudge"|"similarity"|"structuredOutput", "weight": float, "description": str (optional), "config": dict (optional)}.

Judge evaluator configs:

• referenceJudge / comparisonJudge — config.instructions (string): rubric text. Reference judge scores predicted and expected independently and rewards matching; comparison judge scores predicted vs expected in one prompt.
• generalJudge — config.messages (list of {"role": "system"|"user"|"assistant", "content": str}): full user-defined judge prompt. Content may reference {input}, {expected}, {predicted}, or any dataset column name.

structuredOutput evaluator config

Supported config keys for the structuredOutput type:

• schema_definition (dict): JSON schema describing the prediction shape. Drives default per-field scoring — strings → embedding similarity, enums/booleans/integers → exact, numbers → tolerance, nested objects → recursive, arrays → content-aligned soft F1 (not positional).
• fields (dict, optional): per-field overrides keyed by dotted JSON path. Each entry accepts:
• include (bool, default true)
• weight (float, default 1.0)
• strategy (string): scalar comparison — "exact" | "embedding" | "contains" | "judge". The judge value enables LLM-as-judge per-pair scoring on string fields and surfaces reasoning in the observation-details sheet.
• array_strategy (string): array aggregation — "exact" | "similarity" | "judge". The judge value runs a single whole-array LLM call returning F1-compatible counts; arrays exceeding 50 items per side fall back to similarity with a warning marker.

Whether a field counts as required is read from the JSON schema's required array, not from this dict — FieldConfig rejects unknown keys.

• judge_instructions (string, optional): domain-specific guidance shared by every field configured with strategy=judge or array_strategy=judge. Appended to the built-in "do these convey the same essential information?" rubric — leave unset to use the rubric on its own.

The embedding strategy applies a calibrated cosine-similarity floor (0.15, tuned for text-embedding-3-small) so unrelated string pairs score 0.0 instead of ~0.55. Re-running older experiments may show lower scores on string-heavy schemas with unrelated content.

Iterations

client.list_iterations(experiment_id: str) -> IterationList
client.get_iteration(experiment_id: str, iteration_id: int) -> IterationWithScores
client.get_best_iteration(experiment_id: str) -> IterationWithScores

Iterations report two scores: overallNormalizedScore (train split, used to guide the search) and evalNormalizedScore (held-out eval split, None when trainSplitRatio is not configured on the experiment). get_best_iteration ranks by evalNormalizedScore when available, otherwise by overallNormalizedScore.

Deployments

client.get_deployment(component_id: str) -> Deployment | None
client.deploy(component_id: str, experiment_id: str) -> DeploymentCreated
client.undeploy(component_id: str) -> None
client.get_deployed_prompt(component_id: str) -> DeployedPrompt | None

DeployedPrompt fields: prompt, model, provider, componentId, componentName, experimentId, iterationId, score, schemaSnapshot.

Datasets

client.create_dataset(component_id: str, name: str, *, description=None, trace_ids=None) -> Dataset
client.list_datasets(component_id: str) -> DatasetList
client.get_dataset(component_id: str, dataset_id: str) -> DatasetWithItems
client.delete_dataset(component_id: str, dataset_id: str) -> None

Runs

client.create_run(component_id: str, dataset_id: str, *, name=None, trace_ids=None) -> Run
client.list_runs(component_id: str) -> RunList
client.get_run(component_id: str, run_id: str) -> RunWithTraces
client.delete_run(component_id: str, run_id: str) -> None

Annotations

client.upsert_annotation(component_id: str, run_id: str, trace_db_id: str, *, rating=None, comment=None) -> Annotation
client.list_annotations(component_id: str, run_id: str) -> AnnotationList
client.list_dataset_annotations(component_id: str, dataset_id: str) -> AnnotationList
client.delete_annotation(component_id: str, run_id: str, annotation_id: str) -> None

• rating: "positive" or "negative"

Agent Evaluations

client.create_evaluation(component_id: str, dataset_id: str, *, name=None, run_id=None) -> AgentEvaluation
client.list_evaluations(component_id: str) -> AgentEvaluationList
client.get_evaluation(component_id: str, evaluation_id: str) -> AgentEvaluation
client.wait_for_evaluation(component_id: str, evaluation_id: str, *, max_wait=300, poll_interval=2) -> AgentEvaluation

AgentEvaluation status: "pending" | "running" | "completed" | "failed". The results field contains InsightResult with insights list and meta object.