arize-ai/arize-skills
12 skills21.7k installs480 starsGitHub
Install
npx skills add https://github.com/arize-ai/arize-skillsSkills in this repo
1Arize InstrumentationAdds Arize AX tracing to an LLM application for the first time. Follows a two-phase agent-assisted flow to analyze the codebase then implement instrumentation after user confirmation. Use when the user wants to instrument their app, add tracing from scratch, set up LLM observability, integrate OpenTelemetry or openinference, or get started with Arize tracing. --- name: arize-instrumentation description: Adds Arize AX tracing to an LLM application for the first time. Follows a two-phase agent-assisted flow to analyze the codebase then implement instrumentation after user confirmation. Use when the user wants to instrument their app, add tracing from scratch, set up LLM observability, integrate OpenTelemetry or openinference, or get started with Arize tracing. metadata: author: arize version: "1.0" compatibility: Python and TypeScript/JavaScript apps use openinference-instrumentation packages for auto-instrumentation. Go apps use arize-otel-go for setup plus per-provider instrumentors openinference-instrumentation-openai-go (official openai/openai-go SDK) and openinference-instrumentation-anthropic-sdk-go (anthropics/anthropic-sdk-go), or manual spans via openinference-semantic-conv.2.2kinstalls2Arize Prompt OptimizationThe arize-prompt-optimization skill Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement. It covers chain/Agent span : attributes.input.value contains the user's question. The actual LLM prompt lives on child LLM spans -- navigate down the trace tree.. Key workflows include tool span : attributes.input.value has tool input, attributes.output.value has tool result. Not typically where prompts live.. s Optimize a prompt from a failing trace 1. Find failing traces: bash ax traces list PROJECT --filter "status_code = 'ERROR'" --limit 5 2. Export the trace: bash ax spans export PROJECT --trace-id TRACE_ID 3. Extract the prompt from the LLM span: bash jq ' . | select .attributes.openinference.span.kind == "LLM" 0 | { messages: .attributes.llm.input_messages, template: .attributes.llm.prompt_templa Developers invoke arize-prompt-optimization when the task.2.1kinstalls3Arize TraceThe Arize Trace skill enables developers to download, export, and analyze production LLM traces from Arize via the ax CLI. Retrieve complete traces by ID, individual spans, or entire sessions; filter by status, latency, model, and error conditions; and investigate runtime issues at scale. Supports bulk export via Arrow Flight, time-range queries with timezone handling, and structured access to prompts, LLM I/O, errors, and cost metadata. Essential for debugging failing traces, analyzing multi-turn conversations, and performing root-cause analysis on production LLM applications.2kinstalls4Arize ExperimentThe arize-experiment skill covers experiment CRUD, exporting runs, comparing results, and evaluation workflows with the ax CLI against Arize datasets. Concepts define experiments as named evaluation runs on a dataset version, runs as per-example outputs with optional evaluations, and evaluations as named metrics like correctness or relevance. Typical flow exports a dataset, processes each example, collects outputs and evaluations, then creates an experiment with runs. SPACE flags accept workspace names or base64 IDs via --space or ARIZE_SPACE. Agents proceed directly with ax commands without upfront version checks. Use when users benchmark models, compare experiment results, run A/B evaluations, or measure accuracy in Arize. Experiment CRUD and run export via ax CLI against versioned Arize datasets Per-example runs with optional evaluations, scores, labels, and explanations SPACE config via --space flag or ARIZE_SPACE env var (name or base64 ID) Benchmark and A/B model comparison workflows without upfront env probing Dataset version binding for reproducible evaluation runs arize-experiment manages Arize evaluation experiments, runs, and comparisons via the ax CLI Experiment runs w.2kinstalls5Arize DatasetThe arize-dataset skill manages Arize datasets and examples for evaluation and experimentation using the ax CLI with a configured Arize profile. Concepts cover datasets as versioned example collections, dataset versions as snapshots, examples with user-defined fields excluding system id and timestamp fields, and spaces as organizational containers. Commands include ax datasets list with space and name filters, create, export, append examples, and version management. Prerequisites troubleshoot on failure: ax not found via references/ax-setup.md, 401 via ax profiles show, space unknown via ax spaces list. Security rule: never read .env files for credentials; use ax profiles and ax ai-integrations only. Invoke when users need test data, evaluation examples, golden datasets, or mentions create dataset, export dataset, append examples. ax CLI for dataset CRUD, export, and append examples. Datasets are versioned collections of evaluation examples in spaces.2kinstalls6Arize EvaluatorThe arize-evaluator skill handles LLM-as-judge and code evaluator workflows on Arize including create and update evaluators, evaluations on spans or experiments, task management, trigger-run operations, column mapping, and continuous monitoring hooks. Agents use ax CLI patterns with configured profiles and never read .env for secrets. Use when users mention evaluators, LLM judges, or experiment scoring in Arize. LLM-as-judge and code evaluator CRUD in Arize. Run evaluations on spans and experiments. Trigger-run, column mapping, and task management. Continuous monitoring evaluator hooks. ax CLI with profile-based authentication. Create and run Arize LLM-as-judge and code evaluators on spans, experiments, and monitoring tasks. The arize-evaluator skill handles LLM-as-judge and code evaluator workflows on Arize including create and update evaluators, evaluations on spans or experiments, task management, trigger-run operations, column mapping, a. Structured deliverables and steps from the arize-evaluator skill workflow.1.9kinstalls7Arize LinkThe arize-link skill builds clickable Arize UI deep links for traces spans sessions datasets labeling queues evaluators and annotation configs. Requires base64 org_id and space_id plus resource IDs like project_id trace_id span_id session_id dataset_id queue_id evaluator_id. Raw numeric IDs must be base64-encoded or copied from browser URLs to avoid 404s. Templates include trace views with selectedSpanId time ranges on app.arize.com with on-prem override. Use when sharing traces from exported logs opening spans in UI or linking teammates to labeling queues during LLM observability debugging and evaluation workflows. Deep links for traces spans sessions datasets queues. Base64 org_id space_id required. selectedSpanId and time range params. Raw numeric IDs cause 404 URLs. Default base https://app.arize.com. Arize UI deep link generator. User wants Arize trace or dataset link.1.9kinstalls8Arize Ai Provider IntegrationThe Arize AI Integration skill creates, reads, updates, and deletes credential stores for any LLM provider (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM, custom endpoints) within an Arize account or space. Developers use it to register provider API keys or IAM roles once, then reference the integration ID in evaluators and other Arize workflows. Key workflows include listing integrations to find existing credentials, creating new integrations with provider-specific flags (API key, base URL, provider metadata), rotating keys via update, and managing scoping rules to control which spaces can access each integration.1.9kinstalls9Arize AnnotationArize Annotation manages label schemas (categorical, continuous, freeform) and human review queues that route spans to annotators. Developers use this skill to define what feedback reviewers can provide, configure workflows with assignment methods and instructions, and programmatically apply collected annotations back to spans via the Python SDK. The workflow covers annotation config CRUD via CLI, queue management with annotator assignment, record-level annotation submission, and bulk span updates using pandas DataFrames with annotation columns.1.9kinstalls10Arize Compliance AuditThe arize-compliance-audit skill iNVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces an actionable remediation checklist tailored to the selected frameworks. # Arize Compliance Audit Skill Use this skill when the user wants to **audit their AI agent or LLM application for regulatory compliance**. The skill scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces a tailored checklist with optional remediation. **Triggers:** "audit my app for compliance", "EU AI Act requirements", "NIST AI RMF checklist", "GDPR for AI", "is my AI app compliant", "compliance checklist", "regulatory audit", "ISO 42001", "AI management system", "AIMS certification". ## Disclaimer **Before doing anything else, present this disclaimer verbatim to the user:** --- > ⚠️ **Legal disclaimer** > > This audit is for **guidance only** and does **not** constitute legal advice or a complete compliance assessment.1.5kinstalls11Arize Adminarize-admin is a skill from arize-ai/arize-skills for platform administration of Arize AI observability environments. It targets developers and ML engineers responsible for workspace structure, project boundaries, user access, and production monitoring configuration once LLM or traditional ML services are live. Typical tasks include creating or organizing spaces and projects, assigning permissions, and tuning monitoring settings so traces, evaluations, and drift signals remain visible to the right teams. The skill fits operational handoffs where observability must be governed alongside deployment rather than treated as an afterthought. Published README content is not included in the catalog excerpt, so invoke it when users explicitly reference Arize admin, workspace setup, or production monitoring configuration and confirm tenant details at runtime. Pair arize-admin with tracing or evaluation skills when moving from prototype instrumentation to governed production monitoring.1.2kinstalls12Arize Promptsarize-prompts is a Claude Code skill from arize-ai/arize-skills for Arize Prompt Hub and ax prompts CLI workflows. It covers four workflows: A builds and creates prompts with elicited system/user messages and f_string {variable} placeholders; B imports prompt text from code or exported LLM spans; C promotes versions with production and staging labels; D lists, gets, edits, duplicates, and deletes Hub entries. Each prompt version requires --provider (openAI, anthropic, awsBedrock, vertexAI, gemini, and others) and an explicit --model. The skill integrates with arize-experiment for dataset runs, arize-evaluator for LLM-as-judge scoring, and arize-prompt-optimization for trace-driven improvements. Playground test runs auto-log to Playground Traces. Developers reach for arize-prompts when saving chat templates to Hub, iterating versions with commit messages, or promoting a tested prompt to production without redeploying application code.1.2kinstalls