Opik

opik — Open-source LLM Observability, Evaluation & Optimization

Opik (built by Comet) streamlines the entire LLM application lifecycle: deep tracing of LLM calls and agent activity, LLM-as-a-judge evaluation, experiment management, production monitoring at scale (40M+ traces/day), plus the Opik Agent Optimizer and Opik Guardrails. This skill is the routing-first wrapper — it picks the right deployment mode, wires the SDK, and drives the trace → evaluate → monitor → optimize loop.

When to use this skill

• The user asks to install or configure Opik (pip install opik,

opik configure, ./opik.sh)

• The user wants tracing for LLM calls or agents — via @opik.track or a

framework integration (OpenAI, Anthropic, LangChain, LangGraph, LlamaIndex, CrewAI, DSPy, Haystack, Ollama, Bedrock, Vercel AI SDK, Pydantic AI, …)

• The user wants LLM-as-a-judge metrics: hallucination detection, moderation,

answer relevance, context precision/recall for RAG

• The user wants Datasets + Experiments evaluation, or PyTest-integrated

LLM evaluation in CI/CD

• The user wants production LLM monitoring dashboards, online evaluation

rules, prompt playground experiments, agent/prompt optimization, or guardrails

When not to use this skill

• The stack is LangSmith, not Opik → use langsmith
• The user needs generic service dashboards/alerts (non-LLM telemetry,

uptime, infra metrics) → use monitoring-observability

• The user wants offline dataset/KPI interpretation rather than LLM

tracing/eval plumbing → use data-analysis

• The user is doing root-cause log forensics on app/container logs →

use log-analysis

Prerequisites

Instructions

Step 1 — Choose the server mode

Docker service profiles (development scenarios):

./opik.sh              # full Opik suite (default)
./opik.sh --infra      # infrastructure only (databases, caches)
./opik.sh --backend    # infrastructure + backend services
./opik.sh --guardrails # enable guardrails with any profile
./opik.sh --help       # troubleshooting

Windows: powershell -ExecutionPolicy ByPass -c ".\opik.ps1".

Step 2 — Install and configure the SDK

pip install opik        # or: uv pip install opik
opik configure          # prompts for server address (self-hosted) or API key + workspace (cloud)

Or configure in code:

import opik
opik.configure(use_local=True)   # local self-hosted instance

TypeScript, and Ruby (via OpenTelemetry) SDKs are also available — see the client reference docs.

Step 3 — Log traces

Prefer a native integration when the framework is supported (50+ available: ADK, AG2, Agno, Anthropic, Autogen, Bedrock, CrewAI, DSPy, Dify, Flowise, Gemini, Haystack, Instructor, LangChain, LangGraph, LiteLLM, LlamaIndex, Mastra, Ollama, OpenAI, OpenAI Agents, OpenRouter, OpenTelemetry, Pydantic AI, Ragas, Semantic Kernel, Smolagents, Spring AI, Vercel AI SDK, WatsonX, …). See `references/commands.md` for the full table.

Fallback for any code path — the track decorator (nest-aware, composes with integrations):

import opik

@opik.track
def my_llm_function(user_question: str) -> str:
    # Your LLM code here
    return "Hello"

Annotate traces/spans with feedback scores via the SDK or the UI.

Step 4 — Evaluate with LLM-as-a-judge metrics

from opik.evaluation.metrics import Hallucination

metric = Hallucination()
score = metric.score(
    input="What is the capital of France?",
    output="Paris",
    context=["France is a country in Europe."],
)
print(score)

Built-in judges include Hallucination, Moderation, Answer Relevance, Context Precision/Recall; heuristic metrics and custom metrics are also supported.

Step 5 — Datasets, Experiments, and CI gates

• Manage Datasets and run Experiments to compare prompt/model

variants during development

• Wire evaluations into CI/CD with the PyTest integration so regressions

block merges

• Iterate on prompts/models in the Prompt Playground

Step 6 — Production monitoring and optimization

• Opik is built for scale: 40M+ traces/day ingestion
• Track feedback scores, trace counts, and token usage in the Opik Dashboard
• Add Online Evaluation Rules (LLM-as-a-judge on production traffic) to

catch issues live

• Use Opik Agent Optimizer (dedicated SDK) to improve prompts/agents and

Opik Guardrails for safe-AI policies

Step 7 — Plugin-style installation alongside jeo-skills

This skill folder is plugin-installable through the standard jeo-skills flow so the wrapper, references, and installer script land on disk for any supported agent runtime:

# Project install (writes into .agents/skills/opik/)
npx skills add https://github.com/akillness/jeo-skills --skill opik

# Global install for every detected agent
npx skills add -g https://github.com/akillness/jeo-skills --skill opik

# Target specific agents
npx skills add -g https://github.com/akillness/jeo-skills --skill opik -a claude-code -a codex -y

The skill also ships `scripts/install.sh` as a one-shot installer covering SDK install (uv → pip fallback) and optional local self-hosting (OPIK_INSTALL_MODE=local).

Output format

When the user asks opik for help, return a compact brief:

# opik Routing Brief

## Scope
- Server mode: cloud | docker-local | kubernetes | undecided
- SDK: python | typescript | ruby-otel
- Lifecycle stage: tracing | evaluation | ci-gate | production-monitoring | optimization | guardrails

## Recommended next move
- install-sdk | opik-configure | start-local-server | wire-integration | add-judge-metric | create-dataset-experiment | enable-online-rules

## Why
- 2-3 bullets grounded in the user's packet

## Route-outs
- `langsmith` when the observability stack is LangSmith
- `monitoring-observability` for non-LLM dashboards/alerts
- `data-analysis` for offline KPI/metric interpretation

Best practices

1. Start with cloud or `./opik.sh`, not Kubernetes — Helm is for production scale; local Docker Compose answers "does tracing work" in minutes. 2. Prefer a native integration over hand-rolled `@opik.track` when the framework is in the support table — integrations capture provider metadata (tokens, model, latency) automatically. 3. Check the changelog before upgrading a self-hosted server — e.g. v1.7.0 shipped breaking changes. 4. Judge metrics need context — Hallucination and RAG metrics score against the context you pass; empty context produces misleading scores. 5. Gate CI on small, stable datasets — PyTest-integrated experiments should be fast and deterministic; keep large sweeps in scheduled runs. 6. Turn production checks into Online Evaluation Rules instead of re-running offline experiments against live traffic.

References

• Upstream repo: <https://github.com/comet-ml/opik>
• Documentation: <https://www.comet.com/docs/opik/>
• Quickstart: <https://www.comet.com/docs/opik/quickstart/>
• Integrations overview: <https://www.comet.com/docs/opik/integrations/overview/>
• Metrics overview: <https://www.comet.com/docs/opik/evaluation/metrics/overview/>
• Self-host (local): <https://www.comet.com/docs/opik/self-host/local_deployment>
• Self-host (Kubernetes): <https://www.comet.com/docs/opik/self-host/kubernetes/>
• Installer script: `scripts/install.sh`
• Command + integration reference: `references/commands.md`
• Adjacent skills: ../langsmith/SKILL.md, ../monitoring-observability/SKILL.md,

../data-analysis/SKILL.md, ../log-analysis/SKILL.md

• License: Apache-2.0 (see upstream LICENSE)

opik command and integration reference

Upstream: <https://github.com/comet-ml/opik> · Docs: <https://www.comet.com/docs/opik/>

Server (self-hosted, Docker Compose)

Kubernetes/Helm for production scale: Kubernetes Installation Guide. Containers run as non-root users. Check the changelog before upgrades (v1.7.0 had breaking changes).

SDK (Python)

TypeScript SDK and Ruby (OpenTelemetry) are also supported — see the client reference.

LLM-as-a-judge metrics (selected)

Evaluation & CI

• Datasets + Experiments — versioned eval sets and comparison runs
• PyTest integration — run LLM evaluations as CI gates
• Prompt Playground — interactive prompt/model experimentation

Production

• 40M+ traces/day ingestion scale
• Dashboard: feedback scores, trace counts, token usage over time
• Online Evaluation Rules — LLM-as-a-judge on live traffic
• Opik Agent Optimizer — dedicated SDK to improve prompts/agents
• Opik Guardrails — safe and responsible AI policies

Framework integrations (selected, 50+ total)

ADK · AG2 · Agent Spec · AIsuite · Agno · Anthropic · Autogen · Bedrock · BeeAI (Py/TS) · BytePlus · Cloudflare Workers AI · Cohere · CrewAI · Cursor · DeepSeek · Dify · DSPy · Fireworks AI · Flowise AI · Gemini (Py/TS) · Groq · Guardrails AI · Haystack · Harbor · Instructor · LangChain (Py/JS) · LangGraph · Langflow · LiteLLM · LiveKit Agents · LlamaIndex · Mastra · Microsoft Agent Framework (Py/.NET) · Mistral AI · n8n · Novita AI · Ollama · OpenAI (Py/JS) · OpenAI Agents · OpenClaw · OpenRouter · OpenTelemetry · OpenWebUI · Pipecat · Predibase · Pydantic AI · Ragas · Semantic Kernel · Smolagents · Spring AI · Strands Agents · Together AI · Vercel AI SDK · VoltAgent · WatsonX · xAI Grok

Full table with doc links: Integrations overview. Unsupported framework → use @opik.track or open an upstream issue/PR.