Elevenlabs Agents

ElevenLabs Agents Platform

Overview

ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:

1. ASR (Automatic Speech Recognition) - Converts speech to text (32+ languages, sub-second latency) 2. LLM (Large Language Model) - Reasoning and response generation (GPT, Claude, Gemini, custom models) 3. TTS (Text-to-Speech) - Converts text to speech (5000+ voices, 31 languages, low latency) 4. Turn-Taking Model - Proprietary model that handles conversation timing and interruptions

🚨 Package Updates (November 2025)

ElevenLabs migrated to new scoped packages in August 2025:

DEPRECATED (Do not use):

• @11labs/react → DEPRECATED
• @11labs/client → DEPRECATED

Current packages:

npm install @elevenlabs/react@0.9.1        # React SDK
npm install @elevenlabs/client@0.9.1       # JavaScript SDK
npm install @elevenlabs/react-native@0.5.2 # React Native SDK
npm install @elevenlabs/elevenlabs-js@2.21.0 # Base SDK
npm install -g @elevenlabs/agents-cli@0.2.0  # CLI

If you have old packages installed, uninstall them first:

npm uninstall @11labs/react @11labs/client

When to Use This Skill

Use this skill when:

• Building voice-enabled customer support agents
• Creating interactive voice response (IVR) systems
• Developing conversational AI applications
• Integrating telephony (Twilio, SIP trunking)
• Implementing voice chat in web/mobile apps
• Configuring agents via CLI ("agents as code")
• Setting up RAG/knowledge bases for agents
• Integrating MCP (Model Context Protocol) servers
• Building HIPAA/GDPR-compliant voice systems
• Optimizing LLM costs with caching strategies

Platform Capabilities

Design & Configure:

• Multi-step workflows with visual builder
• System prompt engineering (6-component framework)
• 5000+ voices across 31 languages
• Pronunciation dictionaries (IPA/CMU formats)
• Speed control (0.7x-1.2x)
• RAG-powered knowledge bases
• Dynamic variables and personalization

Connect & Deploy:

• React SDK (@elevenlabs/react)
• JavaScript SDK (@elevenlabs/client)
• React Native SDK (@elevenlabs/react-native)
• Swift SDK (iOS/macOS)
• Embeddable widget
• Telephony integration (Twilio, SIP)
• Scribe (Real-Time Speech-to-Text) - Beta

Operate & Optimize:

• Automated testing (scenario, tool call, load)
• Conversation analysis and evaluation
• Analytics dashboard (resolution rates, sentiment, compliance)
• Privacy controls (GDPR, HIPAA, SOC 2)
• Cost optimization (LLM caching, model swapping, burst pricing)
• CLI for "agents as code" workflow

---

1. Quick Start (3 Integration Paths)

Path A: React SDK (Embedded Voice Chat)

For building voice chat interfaces in React applications.

Installation:

npm install @elevenlabs/react zod

Basic Example:

import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

export default function VoiceChat() {
  const { startConversation, stopConversation, status } = useConversation({
    // Public agent (no API key needed)
    agentId: 'your-agent-id',

    // OR private agent (requires API key)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // OR signed URL (server-generated, most secure)
    signedUrl: '/api/elevenlabs/auth',

    // Client-side tools (browser functions)
    clientTools: {
      updateCart: {
        description: "Update the shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Updating cart:', item, quantity);
          return { success: true };
        }
      }
    },

    // Event handlers
    onConnect: () => console.log('Connected'),
    onDisconnect: () => console.log('Disconnected'),
    onEvent: (event) => {
      switch (event.type) {
        case 'transcript':
          console.log('User said:', event.data.text);
          break;
        case 'agent_response':
          console.log('Agent replied:', event.data.text);
          break;
      }
    },

    // Regional compliance (GDPR, data residency)
    serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
  });

  return (
    <div>
      <button onClick={startConversation}>Start Conversation</button>
      <button onClick={stopConversation}>Stop</button>
      <p>Status: {status}</p>
    </div>
  );
}

Path B: CLI ("Agents as Code")

For managing agents via code with version control and CI/CD.

Installation:

npm install -g @elevenlabs/agents-cli
# or
pnpm install -g @elevenlabs/agents-cli

Workflow:

# 1. Authenticate
elevenlabs auth login

# 2. Initialize project (creates agents.json, tools.json, tests.json)
elevenlabs agents init

# 3. Create agent from template
elevenlabs agents add "Support Agent" --template customer-service

# 4. Configure in agent_configs/support-agent.json

# 5. Push to platform
elevenlabs agents push --env dev

# 6. Test
elevenlabs agents test "Support Agent"

# 7. Deploy to production
elevenlabs agents push --env prod

Project Structure Created:

your_project/
├── agents.json              # Agent registry
├── tools.json               # Tool configurations
├── tests.json               # Test configurations
├── agent_configs/           # Individual agent files
├── tool_configs/            # Tool configuration files
└── test_configs/            # Test configuration files

Path C: API (Programmatic Agent Management)

For creating agents dynamically (multi-tenant, SaaS platforms).

Installation:

npm install elevenlabs

Example:

import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY
});

// Create agent
const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: {
        prompt: "You are a helpful customer support agent.",
        llm: "gpt-4o",
        temperature: 0.7
      },
      first_message: "Hello! How can I help you today?",
      language: "en"
    },
    tts: {
      model_id: "eleven_turbo_v2_5",
      voice_id: "your-voice-id"
    }
  }
});

console.log('Agent created:', agent.agent_id);

---

2. Agent Configuration

System Prompt Architecture (6 Components)

ElevenLabs recommends structuring agent prompts using 6 components:

1. Personality

Define the agent's identity, role, and character traits.

Example:

You are Alex, a friendly and knowledgeable customer support specialist at TechCorp.
You have 5 years of experience helping customers solve technical issues.
You're patient, empathetic, and always maintain a positive attitude.

2. Environment

Describe the communication context (phone, web chat, video call).

Example:

You're speaking with customers over the phone. Communication is voice-only.
Customers may have background noise or poor connection quality.
Speak clearly and occasionally use thoughtful pauses for emphasis.

3. Tone

Specify formality, speech patterns, humor, and verbosity.

Example:

Tone: Professional yet warm. Use contractions ("I'm" instead of "I am") to sound natural.
Avoid jargon unless the customer uses it first. Keep responses concise (2-3 sentences max).
Use encouraging phrases like "I'll be happy to help with that" and "Let's get this sorted for you."

4. Goal

Define objectives and success criteria.

Example:

Primary Goal: Resolve customer technical issues on the first call.
Secondary Goals:
- Verify customer identity securely
- Document issue details accurately
- Offer proactive solutions
- End calls with confirmation that the issue is resolved

Success Criteria: Customer verbally confirms their issue is resolved.

5. Guardrails

Set boundaries, prohibited topics, and ethical constraints.

Example:

Guardrails:
- Never provide medical, legal, or financial advice
- Do not share confidential company information
- If asked about competitors, politely redirect to TechCorp's offerings
- Escalate to a human supervisor if customer becomes abusive
- Never make promises about refunds or credits without verification

6. Tools

Describe available external capabilities and when to use them.

Example:

Available Tools:
1. lookup_order(order_id) - Fetch order details from database. Use when customer mentions an order number.
2. transfer_to_supervisor() - Escalate to human agent. Use when issue requires manager approval.
3. send_password_reset(email) - Trigger password reset email. Use when customer can't access account.

Always explain to the customer what you're doing before calling a tool.

Complete Template:

{
  "agent": {
    "prompt": {
      "prompt": "Personality:\nYou are Alex, a friendly customer support specialist.\n\nEnvironment:\nYou're speaking with customers over the phone.\n\nTone:\nProfessional yet warm. Keep responses concise.\n\nGoal:\nResolve technical issues on the first call.\n\nGuardrails:\n- Never provide medical/legal/financial advice\n- Escalate abusive customers\n\nTools:\n- lookup_order(order_id) - Fetch order details\n- transfer_to_supervisor() - Escalate to human",
      "llm": "gpt-4o",
      "temperature": 0.7,
      "max_tokens": 500
    }
  }
}

Turn-Taking Modes

Controls when the agent interrupts or waits for the user to finish speaking.

3 Modes:

Configuration:

{
  "conversation_config": {
    "turn": {
      "mode": "patient" // "eager" | "normal" | "patient"
    }
  }
}

Use Cases:

• Eager: Fast food ordering, quick FAQs, urgent notifications
• Normal: General support, product inquiries, appointment booking
• Patient: Detailed form filling, emotional support, educational tutoring

Gotchas:

• Eager mode can feel interruptive to some users
• Patient mode may feel slow in fast-paced contexts
• Can be dynamically adjusted in workflows for context-aware behavior

Workflows (Visual Builder)

Create branching conversation flows with subagent nodes and conditional routing.

Node Types: 1. Subagent Nodes - Override base agent config (change prompt, voice, turn-taking) 2. Tool Nodes - Guarantee tool execution (unlike tools in subagents)

Configuration:

{
  "workflow": {
    "nodes": [
      {
        "id": "node_1",
        "type": "subagent",
        "config": {
          "system_prompt": "You are now a technical support specialist. Ask detailed diagnostic questions.",
          "turn_eagerness": "patient",
          "voice_id": "tech_support_voice_id"
        }
      },
      {
        "id": "node_2",
        "type": "tool",
        "tool_name": "transfer_to_human"
      }
    ],
    "edges": [
      {
        "from": "node_1",
        "to": "node_2",
        "condition": "user_requests_escalation"
      }
    ]
  }
}

Use Cases:

• Multi-department routing (sales → support → billing)
• Decision trees ("press 1 for sales, 2 for support")
• Role-playing scenarios (customer vs agent voices)
• Escalation paths (bot → human transfer)

Gotchas:

• Workflows add ~100-200ms latency per node transition
• Tool nodes guarantee execution (subagents may skip tools)
• Edges can create infinite loops if not tested properly

Dynamic Variables & Personalization

Inject runtime data into prompts, first messages, and tool parameters using {{var_name}} syntax.

System Variables (Auto-Available):

{{system__agent_id}}         // Current agent ID
{{system__conversation_id}}  // Conversation ID
{{system__caller_id}}        // Phone number (telephony only)
{{system__called_number}}    // Called number (telephony only)
{{system__call_duration_secs}} // Call duration
{{system__time_utc}}         // Current UTC time
{{system__call_sid}}         // Twilio call SID (Twilio only)

Custom Variables:

// Provide when starting conversation
const conversation = await client.conversations.create({
  agent_id: "agent_123",
  dynamic_variables: {
    user_name: "John",
    account_tier: "premium",
    order_id: "ORD-12345"
  }
});

Secret Variables (For API Keys):

{{secret__stripe_api_key}}
{{secret__database_password}}

Important: Secret variables only used in headers, never sent to LLM providers.

Usage in Prompts:

{
  "agent": {
    "prompt": {
      "prompt": "You are helping {{user_name}}, a {{account_tier}} customer."
    },
    "first_message": "Hello {{user_name}}! I see you're calling about order {{order_id}}."
  }
}

Gotcha: Missing variables cause "Missing required dynamic variables" error. Always provide all referenced variables when starting conversation.

Authentication Patterns

Option 1: Public Agents (No API Key)

const { startConversation } = useConversation({
  agentId: 'your-public-agent-id' // Anyone can use
});

Option 2: Private Agents with API Key

const { startConversation } = useConversation({
  agentId: 'your-private-agent-id',
  apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY
});

⚠️ Warning: Never expose API keys in client-side code. Use signed URLs instead.

Option 3: Signed URLs (Recommended for Production)

// Server-side (Next.js API route)
import { ElevenLabsClient } from 'elevenlabs';

export async function POST(req: Request) {
  const client = new ElevenLabsClient({
    apiKey: process.env.ELEVENLABS_API_KEY // Server-side only
  });

  const signedUrl = await client.convai.getSignedUrl({
    agent_id: 'your-agent-id'
  });

  return Response.json({ signedUrl });
}

// Client-side
const { startConversation } = useConversation({
  agentId: 'your-agent-id',
  signedUrl: await fetch('/api/elevenlabs/auth').then(r => r.json()).then(d => d.signedUrl)
});

---

3. Voice & Language Features

Multi-Voice Support

Dynamically switch between different voices during a single conversation.

Use Cases:

• Multi-character storytelling (different voice per character)
• Language tutoring (native speaker voices for each language)
• Role-playing scenarios (customer vs agent)
• Emotional agents (different voices for different moods)

Configuration:

{
  "agent": {
    "prompt": {
      "prompt": "When speaking as the customer, use voice_id 'customer_voice_abc123'. When speaking as the agent, use voice_id 'agent_voice_def456'."
    }
  }
}

Gotchas:

• Voice switching adds ~200ms latency per switch
• Requires careful prompt engineering to trigger switches correctly
• Not all voices work equally well for all characters

Pronunciation Dictionary

Customize how the agent pronounces specific words or phrases.

Supported Formats:

• IPA (International Phonetic Alphabet)
• CMU (Carnegie Mellon University Pronouncing Dictionary)
• Word Substitutions (replace words before TTS)

Configuration:

{
  "pronunciation_dictionary": [
    {
      "word": "ElevenLabs",
      "pronunciation": "ɪˈlɛvənlæbz",
      "format": "ipa"
    },
    {
      "word": "API",
      "pronunciation": "ey-pee-ay",
      "format": "cmu"
    },
    {
      "word": "AI",
      "substitution": "artificial intelligence"
    }
  ]
}

Use Cases:

• Brand names (e.g., "IKEA" → "ee-KAY-uh")
• Acronyms (e.g., "API" → "A-P-I" or "ay-pee-eye")
• Technical terms
• Character names in storytelling

Gotcha: Only Turbo v2/v2.5 models support phoneme-based pronunciation. Other models silently skip phoneme entries but still process word substitutions.

Speed Control

Adjust speaking speed dynamically (0.7x - 1.2x).

Configuration:

{
  "voice_settings": {
    "speed": 1.0 // 0.7 = slow, 1.0 = normal, 1.2 = fast
  }
}

Use Cases:

• Slow (0.7x-0.9x): Accessibility, children, non-native speakers
• Normal (1.0x): Default for most use cases
• Fast (1.1x-1.2x): Urgent notifications, power users

Best Practices:

• Use 0.9x-1.1x for natural-sounding adjustments
• Extreme values (below 0.7 or above 1.2) degrade quality
• Speed can be adjusted per agent, not per utterance

Voice Design

Create custom voices using ElevenLabs Voice Design tool.

Workflow: 1. Navigate to Voice Library → Create Voice 2. Use Voice Design (text-to-voice) or Voice Cloning (sample audio) 3. Test voice with sample text 4. Save voice to library 5. Use voice_id in agent configuration

Voice Cloning Best Practices:

• Use clean audio samples (no background noise, music, or pops)
• Maintain consistent microphone distance
• Avoid extreme volumes (whispering or shouting)
• 1-2 minutes of audio recommended

Gotcha: Using English-trained voices for non-English languages causes pronunciation issues. Always use language-matched voices.

Language Configuration

Support for 32+ languages with automatic detection and in-conversation switching.

Configuration:

{
  "agent": {
    "language": "en" // ISO 639-1 code
  }
}

Multi-Language Presets (Different Voice Per Language):

{
  "conversation_config": {
    "language_presets": [
      {
        "language": "en",
        "voice_id": "en_voice_id",
        "first_message": "Hello! How can I help you today?"
      },
      {
        "language": "es",
        "voice_id": "es_voice_id",
        "first_message": "¡Hola! ¿Cómo puedo ayudarte hoy?"
      },
      {
        "language": "fr",
        "voice_id": "fr_voice_id",
        "first_message": "Bonjour! Comment puis-je vous aider aujourd'hui?"
      }
    ]
  }
}

Automatic Language Detection: Agent detects user's language and switches automatically.

Supported Languages: English, Spanish, French, German, Italian, Portuguese, Dutch, Polish, Arabic, Chinese, Japanese, Korean, Hindi, and 18+ more.

---

4. Knowledge Base & RAG

RAG (Retrieval-Augmented Generation)

Enable agents to access large knowledge bases without loading entire documents into context.

How It Works: 1. Upload documents (PDF, TXT, DOCX) to knowledge base 2. ElevenLabs automatically computes vector embeddings 3. During conversation, relevant chunks retrieved based on semantic similarity 4. LLM uses retrieved context to generate responses

Configuration:

{
  "agent": {
    "prompt": {
      "knowledge_base": ["doc_id_1", "doc_id_2"]
    }
  }
}

Upload Documents via API:

import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });

// Upload document
const doc = await client.knowledgeBase.upload({
  file: fs.createReadStream('support_docs.pdf'),
  name: 'Support Documentation'
});

// Compute RAG index
await client.knowledgeBase.computeRagIndex({
  document_id: doc.id,
  embedding_model: 'e5_mistral_7b' // or 'multilingual_e5_large'
});

Retrieval Configuration:

{
  "knowledge_base_config": {
    "max_chunks": 5,              // Number of chunks to retrieve
    "vector_distance_threshold": 0.8  // Similarity threshold
  }
}

Use Cases:

• Product documentation agents
• Customer support (FAQ, help center)
• Educational tutors (textbooks, lecture notes)
• Healthcare assistants (medical guidelines)

Gotchas:

• RAG adds ~500ms latency per query
• More chunks = higher cost but better context
• Higher vector distance = more context but potentially less relevant
• Documents must be indexed before use (can take minutes for large docs)

---

5. Tools (4 Types)

ElevenLabs supports 4 distinct tool types, each with different execution patterns.

A. Client Tools

Execute operations on the client side (browser or mobile app).

Use Cases:

• Update UI elements (shopping cart, notifications)
• Trigger navigation (redirect user to page)
• Access local storage
• Control media playback

React Example:

import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

const { startConversation } = useConversation({
  clientTools: {
    updateCart: {
      description: "Update the shopping cart with new items",
      parameters: z.object({
        item: z.string().describe("The item name"),
        quantity: z.number().describe("Quantity to add")
      }),
      handler: async ({ item, quantity }) => {
        // Client-side logic
        const cart = getCart();
        cart.add(item, quantity);
        updateUI(cart);
        return { success: true, total: cart.total };
      }
    },
    navigate: {
      description: "Navigate to a different page",
      parameters: z.object({
        url: z.string().describe("The URL to navigate to")
      }),
      handler: async ({ url }) => {
        window.location.href = url;
        return { success: true };
      }
    }
  }
});

Gotchas:

• Tool names are case-sensitive
• Must return a value (agent reads the return value)
• Handler can be async

B. Server Tools (Webhooks)

Make HTTP requests to external APIs from ElevenLabs servers.

Use Cases:

• Fetch real-time data (weather, stock prices)
• Update CRM systems (Salesforce, HubSpot)
• Process payments (Stripe, PayPal)
• Send emails/SMS (SendGrid, Twilio)

Configuration via CLI:

elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json

tool_configs/get-weather.json:

{
  "name": "get_weather",
  "description": "Fetch current weather for a city",
  "url": "https://api.weather.com/v1/current",
  "method": "GET",
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "The city name (e.g., 'London', 'New York')"
      }
    },
    "required": ["city"]
  },
  "headers": {
    "Authorization": "Bearer {{secret__weather_api_key}}"
  }
}

Dynamic Variables in Tools:

{
  "url": "https://api.crm.com/customers/{{user_id}}",
  "headers": {
    "X-API-Key": "{{secret__crm_api_key}}"
  }
}

Gotchas:

• Secret variables only work in headers (not URL or body)
• Schema description guides LLM on when to use tool

C. MCP Tools (Model Context Protocol)

Connect to external MCP servers for standardized tool access.

Use Cases:

• Access databases (PostgreSQL, MongoDB)
• Query knowledge bases (Pinecone, Weaviate)
• Integrate with IDEs (VS Code, Cursor)
• Connect to data sources (Google Drive, Notion)

Configuration: 1. Navigate to MCP server integrations in dashboard 2. Click "Add Custom MCP Server" 3. Configure:

• Name: Server identifier
• Server URL: SSE or HTTP endpoint
• Secret Token: Optional auth header

4. Test connectivity and discover tools 5. Add to agents (public or private)

Approval Modes:

• Always Ask: Maximum security, requires permission per tool call
• Fine-Grained: Per-tool approval settings
• No Approval: Auto-execute all tools

Gotchas:

• Only SSE and HTTP streamable transport supported
• MCP servers must be publicly accessible or behind auth
• Not available for Zero Retention Mode
• Not compatible with HIPAA compliance

Example: Using ElevenLabs MCP Server in Claude Desktop:

{
  "mcpServers": {
    "ElevenLabs": {
      "command": "uvx",
      "args": ["elevenlabs-mcp"],
      "env": {
        "ELEVENLABS_API_KEY": "<your-key>",
        "ELEVENLABS_MCP_OUTPUT_MODE": "files"
      }
    }
  }
}

D. System Tools

Modify the internal state of the conversation without external calls.

Use Cases:

• Update conversation context
• Switch between workflow nodes
• Modify agent behavior mid-conversation
• Track conversation state

Built-in System Tools:

• end_call - End the conversation
• detect_language - Detect user's language
• transfer_agent - Switch to different agent/workflow node
• transfer_to_number - Transfer to external phone number (telephony only)
• dtmf_playpad - Display DTMF keypad (telephony only)
• voicemail_detection - Detect voicemail (telephony only)

Configuration:

{
  "system_tools": [
    {
      "name": "update_conversation_state",
      "description": "Update the conversation context with new information",
      "parameters": {
        "key": { "type": "string" },
        "value": { "type": "string" }
      }
    }
  ]
}

Gotchas:

• System tools don't trigger external APIs
• Changes are ephemeral (lost after conversation ends)
• Useful for workflows and state management

---

6. SDK Integration

React SDK (@elevenlabs/react)

Installation:

npm install @elevenlabs/react zod

Complete Example:

import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';
import { useState } from 'react';

export default function VoiceAgent() {
  const [transcript, setTranscript] = useState<string[]>([]);

  const {
    startConversation,
    stopConversation,
    status,
    isSpeaking
  } = useConversation({
    agentId: 'your-agent-id',

    // Authentication (choose one)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // Client tools
    clientTools: {
      updateCart: {
        description: "Update shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Cart updated:', item, quantity);
          return { success: true };
        }
      }
    },

    // Events
    onConnect: () => {
      console.log('Connected to agent');
      setTranscript([]);
    },
    onDisconnect: () => console.log('Disconnected'),
    onEvent: (event) => {
      if (event.type === 'transcript') {
        setTranscript(prev => [...prev, `User: ${event.data.text}`]);
      } else if (event.type === 'agent_response') {
        setTranscript(prev => [...prev, `Agent: ${event.data.text}`]);
      }
    },
    onError: (error) => console.error('Error:', error),

    // Regional compliance
    serverLocation: 'us'
  });

  return (
    <div>
      <div>
        <button onClick={startConversation} disabled={status === 'connected'}>
          Start Conversation
        </button>
        <button onClick={stopConversation} disabled={status !== 'connected'}>
          Stop
        </button>
      </div>

      <div>Status: {status}</div>
      <div>{isSpeaking && 'Agent is speaking...'}</div>

      <div>
        <h3>Transcript</h3>
        {transcript.map((line, i) => (
          <p key={i}>{line}</p>
        ))}
      </div>
    </div>
  );
}

JavaScript SDK (@elevenlabs/client)

For vanilla JavaScript projects (no React).

Installation:

npm install @elevenlabs/client

Example:

import { Conversation } from '@elevenlabs/client';

const conversation = new Conversation({
  agentId: 'your-agent-id',
  apiKey: process.env.ELEVENLABS_API_KEY,

  onConnect: () => console.log('Connected'),
  onDisconnect: () => console.log('Disconnected'),

  onEvent: (event) => {
    switch (event.type) {
      case 'transcript':
        document.getElementById('user-text').textContent = event.data.text;
        break;
      case 'agent_response':
        document.getElementById('agent-text').textContent = event.data.text;
        break;
    }
  }
});

// Start conversation
document.getElementById('start-btn').addEventListener('click', async () => {
  await conversation.start();
});

// Stop conversation
document.getElementById('stop-btn').addEventListener('click', async () => {
  await conversation.stop();
});

Connection Types: WebRTC vs WebSocket

ElevenLabs SDKs support two connection types with different characteristics.

Comparison Table:

WebSocket Configuration (default):

import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // WebSocket uses signed URL
  signedUrl: async () => {
    const response = await fetch('/api/elevenlabs/auth');
    const { signedUrl } = await response.json();
    return signedUrl;
  },

  // Connection type (optional, defaults to 'websocket')
  connectionType: 'websocket',

  // Audio config (flexible)
  audioConfig: {
    sampleRate: 24000, // 16000, 24000, or 48000
    format: 'PCM_24000'
  }
});

WebRTC Configuration:

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // WebRTC uses conversation token (different auth flow)
  conversationToken: async () => {
    const response = await fetch('/api/elevenlabs/token');
    const { token } = await response.json();
    return token;
  },

  // Connection type
  connectionType: 'webrtc',

  // Audio format is HARDCODED to PCM_48000 (not configurable)
  // audioConfig ignored for WebRTC
});

Backend Token Endpoints:

// WebSocket signed URL (GET /v1/convai/conversation/get-signed-url)
app.get('/api/elevenlabs/auth', async (req, res) => {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/get-signed-url?agent_id=${AGENT_ID}`,
    { headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
  );
  const { signed_url } = await response.json();
  res.json({ signedUrl: signed_url });
});

// WebRTC conversation token (GET /v1/convai/conversation/token)
app.get('/api/elevenlabs/token', async (req, res) => {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/token?agent_id=${AGENT_ID}`,
    { headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
  );
  const { conversation_token } = await response.json();
  res.json({ token: conversation_token });
});

When to Use Each:

Gotchas:

• WebRTC hardcodes PCM_48000 - no way to change format
• Device switching in WebRTC limited by fixed format
• Different authentication methods (signedUrl vs conversationToken)
• WebRTC may have better performance but less flexibility

React Native SDK (Expo)

Installation:

npx expo install @elevenlabs/react-native @livekit/react-native @livekit/react-native-webrtc livekit-client

Requirements:

• Expo SDK 47+
• iOS 14.0+ / macOS 11.0+
• Custom dev build required (Expo Go not supported)

Example:

import { useConversation } from '@elevenlabs/react-native';
import { View, Button, Text } from 'react-native';
import { z } from 'zod';

export default function App() {
  const { startConversation, stopConversation, status } = useConversation({
    agentId: 'your-agent-id',
    signedUrl: 'https://api.elevenlabs.io/v1/convai/auth/...',

    clientTools: {
      updateProfile: {
        description: "Update user profile",
        parameters: z.object({
          name: z.string()
        }),
        handler: async ({ name }) => {
          console.log('Updating profile:', name);
          return { success: true };
        }
      }
    }
  });

  return (
    <View style={{ padding: 20 }}>
      <Button title="Start" onPress={startConversation} />
      <Button title="Stop" onPress={stopConversation} />
      <Text>Status: {status}</Text>
    </View>
  );
}

Gotchas:

• Requires custom dev build (not Expo Go)
• iOS/macOS only (Android support via Kotlin SDK, not yet officially released)

Swift SDK (iOS/macOS)

Installation (Swift Package Manager):

dependencies: [
  .package(url: "https://github.com/elevenlabs/elevenlabs-swift-sdk", from: "1.0.0")
]

Requirements:

• iOS 14.0+ / macOS 11.0+
• Swift 5.9+

Use Cases:

• Native iOS apps
• macOS applications
• watchOS (with limitations)

Widget (Embeddable Web Component)

Installation: Copy-paste embed code from dashboard or use this template:

<script src="https://elevenlabs.io/convai-widget/index.js"></script>
<script>
  ElevenLabsWidget.init({
    agentId: 'your-agent-id',

    // Theming
    theme: {
      primaryColor: '#3B82F6',
      backgroundColor: '#1F2937',
      textColor: '#F9FAFB'
    },

    // Position
    position: 'bottom-right', // 'bottom-left' | 'bottom-right'

    // Custom branding
    branding: {
      logo: 'https://example.com/logo.png',
      name: 'Support Agent'
    }
  });
</script>

Use Cases:

• Customer support chat bubbles
• Website assistants
• Lead capture forms

Scribe (Real-Time Speech-to-Text)

Status: Closed Beta (requires sales contact) Release: 2025

Scribe is ElevenLabs' real-time speech-to-text service for low-latency transcription.

Capabilities:

• Microphone streaming (real-time transcription)
• Pre-recorded audio file transcription
• Partial (interim) and final transcripts
• Word-level timestamps
• Voice Activity Detection (VAD)
• Manual and automatic commit strategies
• Language detection
• PCM_16000 and PCM_24000 audio formats

Authentication: Uses single-use tokens (not API keys):

// Fetch token from backend
const response = await fetch('/api/scribe/token');
const { token } = await response.json();

// Backend endpoint
const token = await client.scribe.getToken();
return { token };

React Hook (`useScribe`):

import { useScribe } from '@elevenlabs/react';

export default function Transcription() {
  const {
    connect,
    disconnect,
    startRecording,
    stopRecording,
    status,
    transcript,
    partialTranscript
  } = useScribe({
    token: async () => {
      const response = await fetch('/api/scribe/token');
      const { token } = await response.json();
      return token;
    },

    // Commit strategy
    commitStrategy: 'vad', // 'vad' (automatic) or 'manual'

    // Audio format
    sampleRate: 16000, // 16000 or 24000

    // Events
    onConnect: () => console.log('Connected to Scribe'),
    onDisconnect: () => console.log('Disconnected'),

    onPartialTranscript: (text) => {
      console.log('Interim:', text);
    },

    onFinalTranscript: (text, timestamps) => {
      console.log('Final:', text);
      console.log('Timestamps:', timestamps); // Word-level timing
    },

    onError: (error) => console.error('Error:', error)
  });

  return (
    <div>
      <button onClick={connect}>Connect</button>
      <button onClick={startRecording}>Start Recording</button>
      <button onClick={stopRecording}>Stop Recording</button>
      <button onClick={disconnect}>Disconnect</button>

      <p>Status: {status}</p>
      <p>Partial: {partialTranscript}</p>
      <p>Final: {transcript}</p>
    </div>
  );
}

JavaScript SDK (`Scribe.connect`):

import { Scribe } from '@elevenlabs/client';

const connection = await Scribe.connect({
  token: 'your-single-use-token',

  sampleRate: 16000,
  commitStrategy: 'vad',

  onPartialTranscript: (text) => {
    document.getElementById('interim').textContent = text;
  },

  onFinalTranscript: (text, timestamps) => {
    const finalDiv = document.getElementById('final');
    finalDiv.textContent += text + ' ';

    // timestamps: [{ word: 'hello', start: 0.5, end: 0.8 }, ...]
  },

  onError: (error) => {
    console.error('Scribe error:', error);
  }
});

// Start recording from microphone
await connection.startRecording();

// Stop recording
await connection.stopRecording();

// Manual commit (if commitStrategy: 'manual')
await connection.commit();

// Disconnect
await connection.disconnect();

Transcribing Pre-Recorded Files:

import { Scribe } from '@elevenlabs/client';

const connection = await Scribe.connect({ token });

// Send audio buffer
const audioBuffer = fs.readFileSync('recording.pcm');
await connection.sendAudioData(audioBuffer);

// Manually commit to get final transcript
await connection.commit();

// Wait for final transcript event

Event Types:

• SESSION_STARTED: Connection established
• PARTIAL_TRANSCRIPT: Interim transcription (unbuffered)
• FINAL_TRANSCRIPT: Complete sentence/phrase
• FINAL_TRANSCRIPT_WITH_TIMESTAMPS: Final + word timing
• ERROR: Transcription error
• AUTH_ERROR: Authentication failed
• OPEN: WebSocket opened
• CLOSE: WebSocket closed

Commit Strategies:

Audio Formats:

• PCM_16000 (16kHz, 16-bit PCM)
• PCM_24000 (24kHz, 16-bit PCM)

Gotchas:

• Token is single-use (expires after one connection)
• Closed beta - requires sales contact
• Language detection automatic (no manual override)
• No speaker diarization yet

When to Use Scribe:

• Building custom transcription UI
• Real-time captions/subtitles
• Voice note apps
• Meeting transcription
• Accessibility features

When NOT to Use: Use Agents Platform instead if you need:

• Conversational AI (LLM + TTS)
• Two-way voice interaction
• Agent responses

---

7. Testing & Evaluation

Scenario Testing (LLM-Based Evaluation)

Simulate full conversations and evaluate against success criteria.

Configuration via CLI:

elevenlabs tests add "Refund Request Test" --template basic-llm

test_configs/refund-request-test.json:

{
  "name": "Refund Request Test",
  "scenario": "Customer requests refund for defective product",
  "user_input": "I want a refund for order #12345. The product was broken when it arrived.",
  "success_criteria": [
    "Agent acknowledges the request empathetically",
    "Agent asks for order number (which was already provided)",
    "Agent verifies order details",
    "Agent provides refund timeline or next steps"
  ],
  "evaluation_type": "llm"
}

Run Test:

elevenlabs agents test "Support Agent"

Tool Call Testing

Verify that agents correctly use tools with the right parameters.

Configuration:

{
  "name": "Account Balance Test",
  "scenario": "Customer requests account balance",
  "expected_tool_call": {
    "tool_name": "get_account_balance",
    "parameters": {
      "account_id": "ACC-12345"
    }
  }
}

Load Testing

Test agent capacity under high concurrency.

Configuration:

# Spawn 100 users, 1 per second, test for 10 minutes
elevenlabs test load \
  --users 100 \
  --spawn-rate 1 \
  --duration 600

Gotchas:

• Load testing consumes real API credits
• Use burst pricing for expected traffic spikes
• Requires careful planning to avoid hitting rate limits

Simulation API (Programmatic Testing)

API Endpoint:

POST /v1/convai/agents/:agent_id/simulate

Example:

const simulation = await client.agents.simulate({
  agent_id: 'agent_123',
  scenario: 'Customer requests refund',
  user_messages: [
    "I want a refund for order #12345",
    "I ordered it last week",
    "Yes, please process it"
  ],
  success_criteria: [
    "Agent acknowledges request",
    "Agent asks for order details",
    "Agent provides refund timeline"
  ]
});

console.log('Test passed:', simulation.passed);
console.log('Criteria met:', simulation.evaluation.criteria_met);

Use Cases:

• CI/CD integration (test before deploy)
• Regression testing
• Load testing preparation

---

8. Analytics & Monitoring

Conversation Analysis

Extract structured data from conversation transcripts.

Features:

Success Evaluation (LLM-Based)

{
  "evaluation_criteria": {
    "resolution": "Was the customer's issue resolved?",
    "sentiment": "Was the conversation tone positive?",
    "compliance": "Did the agent follow company policies?"
  }
}

Data Collection

{
  "data_collection": {
    "fields": [
      { "name": "customer_name", "type": "string" },
      { "name": "issue_type", "type": "enum", "values": ["billing", "technical", "other"] },
      { "name": "satisfaction", "type": "number", "range": [1, 5] }
    ]
  }
}

Access:

• Via Post-call Webhooks (real-time)
• Via Analytics Dashboard (batch)
• Via API (on-demand)

Analytics Dashboard

Metrics:

• Resolution Rates: % of issues resolved
• CX Metrics: Sentiment, satisfaction, CSAT
• Compliance: Policy adherence, guardrail violations
• Performance: Response time, call duration, concurrency
• Tool Usage: Tool call frequency, success rates
• LLM Costs: Track costs per agent/conversation

Access: Dashboard → Analytics tab

---

9. Privacy & Compliance

Data Retention

Default: 2 years (GDPR-compliant)

Configuration:

{
  "privacy": {
    "transcripts": {
      "retention_days": 730  // 2 years (GDPR)
    },
    "audio": {
      "retention_days": 2190  // 6 years (HIPAA)
    }
  }
}

Compliance Recommendations:

• GDPR: Align with data processing purposes (typically 1-2 years)
• HIPAA: Minimum 6 years for medical records
• SOC 2: Encryption in transit and at rest (automatic)

Encryption

• In Transit: TLS 1.3
• At Rest: AES-256
• Regional Compliance: Data residency (US, EU, India)

Regional Configuration:

const { startConversation } = useConversation({
  serverLocation: 'eu-residency' // 'us' | 'global' | 'eu-residency' | 'in-residency'
});

Zero Retention Mode

For maximum privacy, enable zero retention to immediately delete all conversation data.

Limitations:

• No conversation history
• No analytics
• No post-call webhooks
• No MCP tool integrations

---

10. Cost Optimization

LLM Caching

Reduce costs by caching repeated inputs.

How It Works:

• First request: Full cost (input_cache_write)
• Subsequent requests: Reduced cost (input_cache_read)
• Automatic cache invalidation after TTL

Configuration:

{
  "llm_config": {
    "caching": {
      "enabled": true,
      "ttl_seconds": 3600  // 1 hour
    }
  }
}

Use Cases:

• Repeated system prompts
• Large knowledge bases
• Frequent tool definitions

Savings: Up to 90% on cached inputs

Model Swapping

Switch between models based on cost/performance needs.

Available Models:

• GPT-4o (high cost, high quality)
• GPT-4o-mini (medium cost, good quality)
• Claude Sonnet 4.5 (high cost, best reasoning)
• Gemini 2.5 Flash (low cost, fast)

Configuration:

{
  "llm_config": {
    "model": "gpt-4o-mini"  // Swap anytime via dashboard or API
  }
}

Burst Pricing

Temporarily exceed concurrency limits during high-demand periods.

How It Works:

• Normal: Your subscription concurrency limit (e.g., 10 simultaneous calls)
• Burst: Up to 3x your limit (e.g., 30 simultaneous calls)
• Cost: 2x the standard rate for burst calls

Configuration:

{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}

Use Cases:

• Black Friday traffic spikes
• Product launches
• Seasonal demand (holidays)

Gotchas:

• Burst calls cost 2x (plan accordingly)
• Not unlimited (3x cap)

---

11. Advanced Features

Events (WebSocket/SSE)

Real-time event streaming for live transcription, agent responses, and tool calls.

Event Types:

• audio - Audio stream chunks
• transcript - Real-time transcription
• agent_response - Agent's text response
• tool_call - Tool execution status
• conversation_state - State updates

Example:

const { startConversation } = useConversation({
  onEvent: (event) => {
    switch (event.type) {
      case 'transcript':
        console.log('User said:', event.data.text);
        break;
      case 'agent_response':
        console.log('Agent replied:', event.data.text);
        break;
      case 'tool_call':
        console.log('Tool called:', event.data.tool_name);
        break;
    }
  }
});

Custom Models (Bring Your Own LLM)

Use your own OpenAI API key or custom LLM server.

Configuration:

{
  "llm_config": {
    "custom": {
      "endpoint": "https://api.openai.com/v1/chat/completions",
      "api_key": "{{secret__openai_api_key}}",
      "model": "gpt-4"
    }
  }
}

Use Cases:

• Custom fine-tuned models
• Private LLM deployments (Ollama, LocalAI)
• Cost control (use your own credits)
• Compliance (on-premise models)

Gotchas:

• Endpoint must be OpenAI-compatible
• No official support for non-OpenAI-compatible models

Post-Call Webhooks

Receive notifications when a call ends and analysis completes.

Configuration:

{
  "webhooks": {
    "post_call": {
      "url": "https://api.example.com/webhook",
      "headers": {
        "Authorization": "Bearer {{secret__webhook_auth_token}}"
      }
    }
  }
}

Payload:

{
  "conversation_id": "conv_123",
  "agent_id": "agent_456",
  "transcript": "...",
  "duration_seconds": 120,
  "analysis": {
    "sentiment": "positive",
    "resolution": true,
    "extracted_data": {
      "customer_name": "John Doe",
      "issue_type": "billing"
    }
  }
}

Security (HMAC Verification):

import crypto from 'crypto';

export async function POST(req: Request) {
  const signature = req.headers.get('elevenlabs-signature');
  const payload = await req.text();

  const hmac = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET!)
    .update(payload)
    .digest('hex');

  if (signature !== hmac) {
    return new Response('Invalid signature', { status: 401 });
  }

  // Process webhook
  const data = JSON.parse(payload);
  console.log('Conversation ended:', data.conversation_id);

  // MUST return 200
  return new Response('OK', { status: 200 });
}

Gotchas:

• Must return 200 status code
• Auto-disabled after 10 consecutive failures (7+ days since last success)
• Retry logic: 3 attempts with exponential backoff

Chat Mode (Text-Only)

Disable voice, use text-only conversations.

Configuration:

{
  "conversation_config": {
    "chat_mode": true  // Disables audio input/output
  }
}

Benefits:

• Faster response times (~200ms saved)
• Lower costs (no ASR/TTS charges)
• Easier testing (no microphone required)

Use Cases:

• Testing agents without audio
• Building text chat interfaces
• Accessibility (text-only users)

Telephony Integration

SIP Trunking:

SIP Endpoint: sip-static.rtc.elevenlabs.io
TLS Transport: Recommended for production
SRTP Encryption: Supported

Supported Providers: Twilio, Vonage, RingCentral, Sinch, Infobip, Telnyx, Exotel, Plivo, Bandwidth

Native Twilio Integration:

{
  "telephony": {
    "provider": "twilio",
    "phone_number": "+1234567890",
    "account_sid": "{{secret__twilio_account_sid}}",
    "auth_token": "{{secret__twilio_auth_token}}"
  }
}

Use Cases:

• Customer support hotlines
• Appointment scheduling
• Order status inquiries
• IVR systems

---

12. CLI & DevOps ("Agents as Code")

Installation & Authentication

# Install globally
npm install -g @elevenlabs/cli

# Authenticate
elevenlabs auth login

# Set residency (for GDPR compliance)
elevenlabs auth residency eu-residency  # or 'in-residency' | 'global'

# Check current user
elevenlabs auth whoami

Environment Variables (For CI/CD):

export ELEVENLABS_API_KEY=your-api-key

Project Structure

Initialize Project:

elevenlabs agents init

Directory Structure Created:

your_project/
├── agents.json              # Agent registry
├── tools.json               # Tool configurations
├── tests.json               # Test configurations
├── agent_configs/           # Individual agent files (.json)
├── tool_configs/            # Tool configuration files
└── test_configs/            # Test configuration files

Agent Management Commands

# Create agent
elevenlabs agents add "Support Agent" --template customer-service

# Deploy to platform
elevenlabs agents push
elevenlabs agents push --agent "Support Agent"
elevenlabs agents push --env prod
elevenlabs agents push --dry-run  # Preview changes

# Import existing agents
elevenlabs agents pull

# List agents
elevenlabs agents list

# Check sync status
elevenlabs agents status

# Delete agent
elevenlabs agents delete <agent_id>

Tool Management Commands

# Create webhook tool
elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json

# Create client tool
elevenlabs tools add-client "Update Cart" --config-path tool_configs/update-cart.json

# Deploy tools
elevenlabs tools push

# Import existing tools
elevenlabs tools pull

# Delete tools
elevenlabs tools delete <tool_id>
elevenlabs tools delete --all

Testing Commands

# Create test
elevenlabs tests add "Refund Test" --template basic-llm

# Deploy tests
elevenlabs tests push

# Import tests
elevenlabs tests pull

# Run test
elevenlabs agents test "Support Agent"

Multi-Environment Deployment

Pattern:

# Development
elevenlabs agents push --env dev

# Staging
elevenlabs agents push --env staging

# Production (with confirmation)
elevenlabs agents push --env prod --dry-run
# Review changes...
elevenlabs agents push --env prod

Environment-Specific Configs:

agent_configs/
├── support-bot.json          # Base config
├── support-bot.dev.json      # Dev overrides
├── support-bot.staging.json  # Staging overrides
└── support-bot.prod.json     # Prod overrides

CI/CD Integration

GitHub Actions Example:

name: Deploy Agent
on:
  push:
    branches: [main]
    paths:
      - 'agent_configs/**'
      - 'tool_configs/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install CLI
        run: npm install -g @elevenlabs/cli

      - name: Test Configs
        run: elevenlabs agents push --dry-run --env prod
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}

      - name: Deploy
        run: elevenlabs agents push --env prod
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}

Version Control Best Practices

Commit:

• agent_configs/*.json
• tool_configs/*.json
• test_configs/*.json
• agents.json, tools.json, tests.json

Ignore:

# .gitignore
.env
.elevenlabs/
*.secret.json

---

13. Common Errors & Solutions

Error 1: Missing Required Dynamic Variables

Symptom: "Missing required dynamic variables" error, no transcript generated

Cause: Dynamic variables referenced in prompts/messages but not provided at conversation start

Solution:

const conversation = await client.conversations.create({
  agent_id: "agent_123",
  dynamic_variables: {
    user_name: "John",
    account_tier: "premium",
    // Provide ALL variables referenced in prompts
  }
});

Error 2: Case-Sensitive Tool Names

Symptom: Tool not executing, agent says "tool not found"

Cause: Tool name in config doesn't match registered name (case-sensitive)

Solution:

// agent_configs/bot.json
{
  "agent": {
    "prompt": {
      "tool_ids": ["orderLookup"]  // Must match exactly
    }
  }
}

// tool_configs/order-lookup.json
{
  "name": "orderLookup"  // Match case exactly
}

Error 3: Webhook Authentication Failures

Symptom: Webhook auto-disabled after failures

Cause:

• Incorrect HMAC signature verification
• Not returning 200 status code
• 10+ consecutive failures

Solution:

// Always verify HMAC signature
import crypto from 'crypto';

const signature = req.headers['elevenlabs-signature'];
const payload = JSON.stringify(req.body);

const hmac = crypto
  .createHmac('sha256', process.env.WEBHOOK_SECRET)
  .update(payload)
  .digest('hex');

if (signature !== hmac) {
  return res.status(401).json({ error: 'Invalid signature' });
}

// Process webhook
// ...

// MUST return 200
res.status(200).json({ success: true });

Error 4: Voice Consistency Issues

Symptom: Generated audio varies in volume/tone

Cause:

• Background noise in voice clone training data
• Inconsistent microphone distance
• Whispering or shouting in samples

Solution:

• Use clean audio samples (no music, noise, pops)
• Maintain consistent microphone distance
• Avoid extreme volumes
• Test voice settings before deployment

Error 5: Wrong Language Voice

Symptom: Unpredictable pronunciation, accent issues

Cause: Using English-trained voice for non-English language

Solution:

{
  "language_presets": [
    {
      "language": "es",
      "voice_id": "spanish_trained_voice_id"  // Must be Spanish-trained
    }
  ]
}

Error 6: Restricted API Keys Not Supported (CLI)

Symptom: CLI authentication fails

Cause: Using restricted API key (not currently supported)

Solution: Use unrestricted API key for CLI operations

Error 7: Agent Configuration Push Conflicts

Symptom: Changes not reflected after push

Cause: Hash-based change detection missed modification

Solution:

# Force re-sync
elevenlabs agents init --override
elevenlabs agents pull  # Re-import from platform
# Make changes
elevenlabs agents push

Error 8: Tool Parameter Schema Mismatch

Symptom: Tool called but parameters empty or incorrect

Cause: Schema definition doesn't match actual usage

Solution:

// tool_configs/order-lookup.json
{
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The order ID to look up (format: ORD-12345)"  // Clear description
      }
    },
    "required": ["order_id"]
  }
}

Error 9: RAG Index Not Ready

Symptom: Agent doesn't use knowledge base

Cause: RAG index still computing (can take minutes for large documents)

Solution:

// Check index status before using
const index = await client.knowledgeBase.getRagIndex({
  document_id: 'doc_123'
});

if (index.status !== 'ready') {
  console.log('Index still computing...');
}

Error 10: WebSocket Protocol Error (1002)

Symptom: Intermittent "protocol error" when using WebSocket connections

Cause: Network instability or incompatible browser

Solution:

• Use WebRTC instead of WebSocket (more resilient)
• Implement reconnection logic
• Check browser compatibility

Error 11: 401 Unauthorized in Production

Symptom: Works locally but fails in production

Cause: Agent visibility settings or API key configuration

Solution:

• Check agent visibility (public vs private)
• Verify API key is set in production environment
• Check allowlist configuration if enabled

Error 12: Allowlist Connection Errors

Symptom: "Host elevenlabs.io is not allowed to connect to this agent"

Cause: Agent has allowlist enabled but using shared link

Solution:

• Configure agent allowlist with correct domains
• Or disable allowlist for testing

Error 13: Workflow Infinite Loops

Symptom: Agent gets stuck in workflow, never completes

Cause: Edge conditions creating loops

Solution:

• Add max iteration limits
• Test all edge paths
• Add explicit exit conditions

Error 14: Burst Pricing Not Enabled

Symptom: Calls rejected during traffic spikes

Cause: Burst pricing not enabled in agent settings

Solution:

{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}

Error 15: MCP Server Timeout

Symptom: MCP tools not responding

Cause: MCP server slow or unreachable

Solution:

• Check MCP server URL is accessible
• Verify transport type (SSE vs HTTP)
• Check authentication token
• Monitor MCP server logs

Error 16: First Message Cutoff on Android

Symptom: First message from agent gets cut off on Android devices (works fine on iOS/web)

Cause: Android devices need time to switch to correct audio mode after connection

Solution:

import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // Add connection delay for Android
  connectionDelay: {
    android: 3_000,  // 3 seconds (default)
    ios: 0,          // No delay needed
    default: 0       // Other platforms
  },

  // Rest of config...
});

Explanation:

• Android needs 3 seconds to switch audio routing mode
• Without delay, first audio chunk is lost
• iOS and web don't have this issue
• Adjust delay if 3 seconds isn't sufficient

Testing:

# Test on Android device
npm run android

# First message should now be complete

Error 17: CSP (Content Security Policy) Violations

Symptom: "Refused to load the script because it violates the following Content Security Policy directive" errors in browser console

Cause: Applications with strict Content Security Policy don't allow data: or blob: URLs in script-src directive. ElevenLabs SDK uses Audio Worklets that are loaded as blobs by default.

Solution - Self-Host Worklet Files:

Step 1: Copy worklet files to your public directory:

# Copy from node_modules
cp node_modules/@elevenlabs/client/dist/worklets/*.js public/elevenlabs/

Step 2: Configure SDK to use self-hosted worklets:

import { useConversation } from '@elevenlabs/react';

const { startConversation } = useConversation({
  agentId: 'your-agent-id',

  // Point to self-hosted worklet files
  workletPaths: {
    'rawAudioProcessor': '/elevenlabs/rawAudioProcessor.worklet.js',
    'audioConcatProcessor': '/elevenlabs/audioConcatProcessor.worklet.js',
  },

  // Rest of config...
});

Step 3: Update CSP headers to allow self-hosted scripts:

# nginx example
add_header Content-Security-Policy "
  default-src 'self';
  script-src 'self' https://elevenlabs.io;
  connect-src 'self' https://api.elevenlabs.io wss://api.elevenlabs.io;
  worker-src 'self';
" always;

Worklet Files Location:

node_modules/@elevenlabs/client/dist/worklets/
├── rawAudioProcessor.worklet.js
└── audioConcatProcessor.worklet.js

Gotchas:

• Worklet files must be served from same origin (CORS restriction)
• Update worklet files when upgrading @elevenlabs/client
• Paths must match exactly (case-sensitive)

When You Need This:

• Enterprise applications with strict CSP
• Government/financial apps
• Apps with security audits
• Any app blocking blob: URLs

---

Integration with Existing Skills

This skill composes well with:

• cloudflare-worker-base → Deploy agents on Cloudflare Workers edge network
• cloudflare-workers-ai → Use Cloudflare LLMs as custom models in agents
• cloudflare-durable-objects → Persistent conversation state and session management
• cloudflare-kv → Cache agent configurations and user preferences
• nextjs → React SDK integration in Next.js applications
• ai-sdk-core → Vercel AI SDK provider for unified AI interface
• clerk-auth → Authenticated voice sessions with user identity
• hono-routing → API routes for webhooks and server tools

---

Additional Resources

Official Documentation:

• Platform Overview: https://elevenlabs.io/docs/agents-platform/overview
• API Reference: https://elevenlabs.io/docs/api-reference
• CLI GitHub: https://github.com/elevenlabs/cli

Examples:

• Official Examples: https://github.com/elevenlabs/elevenlabs-examples
• MCP Server: https://github.com/elevenlabs/elevenlabs-mcp

Community:

• Discord: https://discord.com/invite/elevenlabs
• Twitter: @elevenlabsio

---

Production Tested: WordPress Auditor, Customer Support Agents Last Updated: 2025-11-03 Package Versions: elevenlabs@1.59.0, @elevenlabs/cli@0.2.0

System Prompt Template

Use this template to create structured, effective agent prompts.

---

Personality

You are [NAME], a [ROLE/PROFESSION] at [COMPANY].
You have [YEARS] years of experience [DOING WHAT].
Your key traits: [LIST 3-5 PERSONALITY TRAITS].

Example:

You are Sarah, a patient and knowledgeable technical support specialist at TechCorp.
You have 7 years of experience helping customers troubleshoot software issues.
Your key traits: patient, empathetic, detail-oriented, solution-focused, friendly.

---

Environment

You're communicating via [CHANNEL: phone/chat/video].
Context: [ENVIRONMENTAL FACTORS].
Communication style: [GUIDELINES].

Example:

You're speaking with customers over the phone.
Context: Background noise and poor connections are common.
Communication style: Speak clearly, use short sentences, pause occasionally for emphasis.

---

Tone

Formality: [PROFESSIONAL/CASUAL/FORMAL].
Language: [CONTRACTIONS/JARGON GUIDELINES].
Verbosity: [SENTENCE/RESPONSE LENGTH].
Emotional Expression: [HOW TO EXPRESS EMPATHY/ENTHUSIASM].

Example:

Formality: Professional yet warm and approachable.
Language: Use contractions for natural conversation. Avoid jargon unless customer uses it first.
Verbosity: 2-3 sentences per response. Ask one question at a time.
Emotional Expression: Show empathy with phrases like "I understand how frustrating that must be."

---

Goal

Primary Goal: [MAIN OBJECTIVE]

Secondary Goals:
- [SUPPORTING OBJECTIVE 1]
- [SUPPORTING OBJECTIVE 2]
- [SUPPORTING OBJECTIVE 3]

Success Criteria:
- [MEASURABLE OUTCOME 1]
- [MEASURABLE OUTCOME 2]

Example:

Primary Goal: Resolve customer technical issues on the first call.

Secondary Goals:
- Verify customer identity securely
- Document issue details accurately
- Provide proactive tips to prevent future issues

Success Criteria:
- Customer verbally confirms issue is resolved
- Issue documented in CRM
- Customer satisfaction ≥ 4/5

---

Guardrails

Never:
- [PROHIBITED ACTION 1]
- [PROHIBITED ACTION 2]
- [PROHIBITED ACTION 3]

Always:
- [REQUIRED ACTION 1]
- [REQUIRED ACTION 2]

Escalate When:
- [ESCALATION TRIGGER 1]
- [ESCALATION TRIGGER 2]

Example:

Never:
- Provide medical, legal, or financial advice
- Share confidential company information
- Make promises about refunds without verification
- Continue if customer becomes abusive

Always:
- Verify customer identity before accessing account details
- Document all interactions
- Offer alternative solutions if first approach fails

Escalate When:
- Customer requests manager
- Issue requires account credit/refund approval
- Technical issue beyond knowledge base
- Customer exhibits abusive behavior

---

Tools

Available Tools:

1. tool_name(param1, param2)
   Purpose: [WHAT IT DOES]
   Use When: [TRIGGER CONDITION]
   Example: [SAMPLE USAGE]

2. ...

Guidelines:
- Always explain to customer before calling tool
- Wait for tool response before continuing
- If tool fails, offer alternative

Example:

Available Tools:

1. lookup_order(order_id: string)
   Purpose: Fetch order details from database
   Use When: Customer mentions order number or asks about order status
   Example: "Let me look that up for you. [Call lookup_order('ORD-12345')]"

2. send_password_reset(email: string)
   Purpose: Trigger password reset email
   Use When: Customer can't access account and identity verified
   Example: "I'll send a password reset email. [Call send_password_reset('user@example.com')]"

3. transfer_to_supervisor()
   Purpose: Escalate to human agent
   Use When: Issue requires manager approval or customer explicitly requests
   Example: "Let me connect you with a supervisor. [Call transfer_to_supervisor()]"

Guidelines:
- Always explain what you're doing before calling tool
- Wait for tool response before continuing conversation
- If tool fails, acknowledge and offer alternative solution

---

Complete Prompt

Combine all sections into your final system prompt:

Personality:
You are [NAME], a [ROLE] at [COMPANY]. You have [EXPERIENCE]. Your traits: [TRAITS].

Environment:
You're communicating via [CHANNEL]. [CONTEXT]. [COMMUNICATION STYLE].

Tone:
[FORMALITY]. [LANGUAGE]. [VERBOSITY]. [EMOTIONAL EXPRESSION].

Goal:
Primary: [PRIMARY GOAL]
Secondary: [SECONDARY GOALS]
Success: [SUCCESS CRITERIA]

Guardrails:
Never: [PROHIBITIONS]
Always: [REQUIREMENTS]
Escalate: [TRIGGERS]

Tools:
[TOOL DESCRIPTIONS WITH EXAMPLES]

---

Testing Your Prompt

1. Create test scenarios covering common use cases 2. Run conversations and analyze transcripts 3. Check for:

• Tone consistency
• Goal achievement
• Guardrail adherence
• Tool usage accuracy

4. Iterate based on findings 5. Monitor analytics dashboard for real performance

ElevenLabs Agents Platform Skill

Comprehensive skill for building production-ready conversational AI voice agents with ElevenLabs.

Auto-Trigger Keywords

This skill should be used when working with:

• ElevenLabs agents, ElevenLabs conversational AI, ElevenLabs platform
• Voice agent, voice chat, conversational AI, voice interface
• ElevenLabs React, ElevenLabs SDK, @elevenlabs/react, @elevenlabs/client
• ElevenLabs CLI, agents as code, elevenlabs agents
• Agent configuration, agent workflow, agent behavior
• System prompt, turn-taking, conversation flow
• Multi-voice, pronunciation dictionary, voice design
• RAG knowledge base, ElevenLabs RAG, knowledge base agent
• ElevenLabs MCP, MCP tools, Model Context Protocol
• Client tools, server tools, webhook tools, system tools
• Telephony integration, Twilio ElevenLabs, SIP trunk
• Voice testing, agent testing, scenario testing
• HIPAA voice agent, GDPR compliance, voice compliance
• LLM caching, cost optimization, burst pricing
• React Native voice, Swift voice agent, voice widget

What This Skill Covers

Platform Capabilities (29 Features)

Agent Configuration & Management:

• System prompt engineering (6-component framework)
• Turn-taking modes (Eager/Normal/Patient)
• Workflows (visual builder with nodes and edges)
• Dynamic variables and personalization
• Authentication patterns (public/private/signed URLs)

Voice & Language:

• Multi-voice support (5000+ voices, 31 languages)
• Pronunciation dictionaries (IPA/CMU formats)
• Speed control (0.7x-1.2x)
• Voice design and cloning
• Language presets and auto-detection

Knowledge & Tools:

• RAG (Retrieval-Augmented Generation) with knowledge bases
• 4 tool types: Client tools, Server tools (webhooks), MCP tools, System tools
• Tool parameter schemas and validation

SDKs & Integration:

• React SDK (@elevenlabs/react)
• JavaScript SDK (@elevenlabs/client)
• React Native SDK (@elevenlabs/react-native)
• Swift SDK (iOS/macOS)
• Embeddable widget
• Scribe (Real-Time Speech-to-Text) - Beta

Testing & Analytics:

• Scenario testing (LLM-based evaluation)
• Tool call testing
• Load testing
• Conversation analysis and data collection
• Analytics dashboard (resolution rates, sentiment, compliance)

Privacy & Compliance:

• Data retention policies (GDPR: 2 years, HIPAA: 6 years)
• Encryption (TLS 1.3, AES-256)
• Regional compliance (US/EU/India)
• SOC 2 compliance

Cost Optimization:

• LLM caching (up to 90% savings on cached inputs)
• Model swapping (GPT/Claude/Gemini)
• Burst pricing (3x concurrency at 2x cost)

DevOps & Advanced:

• CLI ("agents as code") with multi-environment support
• CI/CD integration (GitHub Actions examples)
• Events (WebSocket/SSE real-time streaming)
• Custom models (bring your own LLM)
• Post-call webhooks
• Chat mode (text-only)
• Telephony integration (Twilio, SIP)

Errors Prevented

This skill prevents 17+ common errors:

1. Package deprecation (@11labs/ → @elevenlabs/) 2. Android audio cutoff (connectionDelay configuration) 3. CSP violations (workletPaths self-hosting) 4. WebRTC vs WebSocket confusion (different auth flows) 5. Missing required dynamic variables 6. Case-sensitive tool names mismatch 7. Webhook authentication failures (HMAC verification) 8. Voice consistency issues (training data quality) 9. Wrong language voice (English voice for Spanish, etc.) 10. Restricted API keys in CLI 11. Agent configuration push conflicts 12. Tool parameter schema mismatches 13. RAG index not ready before use 14. WebSocket protocol errors (1002) 15. 401 Unauthorized in production (visibility settings) 16. Allowlist connection errors 17. Workflow infinite loops

Quick Start Examples

React SDK (Voice Chat UI)

import { useConversation } from '@elevenlabs/react';

const { startConversation, stopConversation, status } = useConversation({
  agentId: 'your-agent-id',
  apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,
  onConnect: () => console.log('Connected'),
  onEvent: (event) => console.log('Event:', event)
});

CLI ("Agents as Code")

elevenlabs agents init
elevenlabs agents add "Support Agent" --template customer-service
elevenlabs agents push --env prod

API (Programmatic Agent Creation)

const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: { prompt: "You are a helpful support agent.", llm: "gpt-4o" },
      language: "en"
    }
  }
});

Package Versions

🚨 IMPORTANT: ElevenLabs migrated packages in August 2025. Use these current versions:

• @elevenlabs/elevenlabs-js: 2.21.0
• @elevenlabs/agents-cli: 0.2.0
• @elevenlabs/react: 0.9.1
• @elevenlabs/client: 0.9.1
• @elevenlabs/react-native: 0.5.2

DEPRECATED (do not use):

• @11labs/react - Deprecated August 2025
• @11labs/client - Deprecated August 2025

Impact Metrics

• Token Savings: ~73% (22k → 6k tokens)
• Errors Prevented: 17+ common issues (including v1.1.0 additions)
• Time Savings: 6-8 hours → 6-8 minutes with Claude Code
• Coverage: 31 major features, 100% platform coverage (includes Scribe + WebRTC)

Templates Included

• Scripts: create-agent.sh, test-agent.sh, deploy-agent.sh, simulate-conversation.sh
• References: API reference, system prompt guide, workflow examples, tool examples, testing guide, compliance guide, cost optimization
• Assets: React boilerplate, JavaScript boilerplate, React Native boilerplate, Swift boilerplate, widget template, agent config schema, CI/CD example

Integration with Other Skills

Composes well with:

• cloudflare-worker-base - Deploy agents on edge
• cloudflare-workers-ai - Custom LLM integration
• cloudflare-durable-objects - Conversation state
• nextjs - React SDK in Next.js
• ai-sdk-core - Vercel AI SDK provider
• clerk-auth - Authenticated sessions
• hono-routing - Webhook endpoints

Documentation

• Official Docs: https://elevenlabs.io/docs/agents-platform/overview
• API Reference: https://elevenlabs.io/docs/api-reference
• CLI GitHub: https://github.com/elevenlabs/cli
• Examples: https://github.com/elevenlabs/elevenlabs-examples

Production Tested

• WordPress Auditor
• Customer Support Agents
• Multiple production deployments

License

MIT

ElevenLabs Agents API Reference

Base URL

https://api.elevenlabs.io/v1/convai

Authentication

All requests require an API key in the header:

curl -H "xi-api-key: YOUR_API_KEY" https://api.elevenlabs.io/v1/convai/agents

---

Agents

Create Agent

Endpoint: POST /agents/create

Request Body:

{
  "name": "Support Agent",
  "conversation_config": {
    "agent": {
      "prompt": {
        "prompt": "You are a helpful support agent.",
        "llm": "gpt-4o",
        "temperature": 0.7,
        "max_tokens": 500,
        "tool_ids": ["tool_123"],
        "knowledge_base": ["doc_456"]
      },
      "first_message": "Hello! How can I help?",
      "language": "en"
    },
    "tts": {
      "model_id": "eleven_turbo_v2_5",
      "voice_id": "voice_abc123",
      "stability": 0.5,
      "similarity_boost": 0.75,
      "speed": 1.0
    },
    "asr": {
      "quality": "high",
      "provider": "deepgram"
    },
    "turn": {
      "mode": "normal"
    }
  }
}

Response:

{
  "agent_id": "agent_abc123",
  "name": "Support Agent",
  "created_at": "2025-11-03T12:00:00Z"
}

Update Agent

Endpoint: PATCH /agents/:agent_id

Request Body: Same as Create Agent

Get Agent

Endpoint: GET /agents/:agent_id

Response:

{
  "agent_id": "agent_abc123",
  "name": "Support Agent",
  "conversation_config": { ... },
  "created_at": "2025-11-03T12:00:00Z",
  "updated_at": "2025-11-03T14:00:00Z"
}

List Agents

Endpoint: GET /agents

Response:

{
  "agents": [
    {
      "agent_id": "agent_abc123",
      "name": "Support Agent",
      "created_at": "2025-11-03T12:00:00Z"
    }
  ]
}

Delete Agent

Endpoint: DELETE /agents/:agent_id

Response:

{
  "success": true
}

---

Conversations

Create Conversation

Endpoint: POST /conversations/create

Request Body:

{
  "agent_id": "agent_abc123",
  "dynamic_variables": {
    "user_name": "John",
    "account_tier": "premium"
  },
  "overrides": {
    "agent": {
      "prompt": {
        "prompt": "Custom prompt override"
      }
    }
  }
}

Response:

{
  "conversation_id": "conv_xyz789",
  "signed_url": "wss://api.elevenlabs.io/v1/convai/...",
  "created_at": "2025-11-03T12:00:00Z"
}

Get Conversation

Endpoint: GET /conversations/:conversation_id

Response:

{
  "conversation_id": "conv_xyz789",
  "agent_id": "agent_abc123",
  "transcript": "...",
  "duration_seconds": 120,
  "status": "completed",
  "created_at": "2025-11-03T12:00:00Z",
  "ended_at": "2025-11-03T12:02:00Z"
}

---

Knowledge Base

Upload Document

Endpoint: POST /knowledge-base/upload

Request Body (multipart/form-data):

file: <binary>
name: "Support Documentation"

Response:

{
  "document_id": "doc_456",
  "name": "Support Documentation",
  "status": "processing"
}

Compute RAG Index

Endpoint: POST /knowledge-base/:document_id/rag-index

Request Body:

{
  "embedding_model": "e5_mistral_7b"
}

Response:

{
  "document_id": "doc_456",
  "status": "computing"
}

Get RAG Index Status

Endpoint: GET /knowledge-base/:document_id/rag-index

Response:

{
  "document_id": "doc_456",
  "status": "ready",
  "embedding_model": "e5_mistral_7b",
  "created_at": "2025-11-03T12:00:00Z"
}

---

Tools

Create Webhook Tool

Endpoint: POST /tools/webhook

Request Body:

{
  "name": "get_weather",
  "description": "Fetch current weather for a city",
  "url": "https://api.weather.com/v1/current",
  "method": "GET",
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "City name"
      }
    },
    "required": ["city"]
  },
  "headers": {
    "Authorization": "Bearer {{secret__weather_api_key}}"
  }
}

Response:

{
  "tool_id": "tool_123",
  "name": "get_weather",
  "created_at": "2025-11-03T12:00:00Z"
}

---

Testing

Simulate Conversation

Endpoint: POST /agents/:agent_id/simulate

Request Body:

{
  "scenario": "Customer requests refund",
  "user_messages": [
    "I want a refund for order #12345",
    "I ordered it last week"
  ],
  "success_criteria": [
    "Agent acknowledges request",
    "Agent provides timeline"
  ]
}

Response:

{
  "simulation_id": "sim_123",
  "passed": true,
  "transcript": "...",
  "evaluation": {
    "criteria_met": 2,
    "criteria_total": 2,
    "details": [
      {
        "criterion": "Agent acknowledges request",
        "passed": true
      },
      {
        "criterion": "Agent provides timeline",
        "passed": true
      }
    ]
  }
}

---

Error Codes

---

Rate Limits

• Standard Tier: 100 requests/minute
• Pro Tier: 500 requests/minute
• Enterprise Tier: Custom limits

Rate Limit Headers:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1730640000

---

Pagination

Query Parameters:

?page=1&per_page=50

Response Headers:

X-Total-Count: 250
X-Page: 1
X-Per-Page: 50

---

Webhook Events

Post-Call Webhook

Event Type: post_call_transcription

Payload:

{
  "type": "post_call_transcription",
  "data": {
    "conversation_id": "conv_xyz789",
    "agent_id": "agent_abc123",
    "transcript": "...",
    "duration_seconds": 120,
    "analysis": {
      "sentiment": "positive",
      "resolution": true,
      "extracted_data": {}
    }
  },
  "event_timestamp": "2025-11-03T12:02:00Z"
}

Verification (HMAC SHA-256):

import crypto from 'crypto';

const signature = request.headers['elevenlabs-signature'];
const payload = JSON.stringify(request.body);

const hmac = crypto
  .createHmac('sha256', process.env.WEBHOOK_SECRET)
  .update(payload)
  .digest('hex');

if (signature !== hmac) {
  // Invalid signature
}

---

SDK vs API

Recommendation: Use SDK for conversations, API for agent management.

CLI Commands Reference

Installation

npm install -g @elevenlabs/cli
# or
pnpm install -g @elevenlabs/cli

---

Authentication

Login

elevenlabs auth login

Check Current User

elevenlabs auth whoami

Set Residency

elevenlabs auth residency eu-residency
# Options: global | eu-residency | in-residency

Logout

elevenlabs auth logout

---

Project Initialization

Initialize New Project

elevenlabs agents init

Recreate Project Structure

elevenlabs agents init --override

---

Agent Management

Add Agent

elevenlabs agents add "Agent Name" --template TEMPLATE

Templates: default | minimal | voice-only | text-only | customer-service | assistant

Push to Platform

# Push all agents
elevenlabs agents push

# Push specific agent
elevenlabs agents push --agent "Agent Name"

# Push to environment
elevenlabs agents push --env prod

# Dry run (preview changes)
elevenlabs agents push --dry-run

Pull from Platform

# Pull all agents
elevenlabs agents pull

# Pull specific agent
elevenlabs agents pull --agent "Agent Name"

List Agents

elevenlabs agents list

Check Sync Status

elevenlabs agents status

Delete Agent

elevenlabs agents delete AGENT_ID

Generate Widget

elevenlabs agents widget "Agent Name"

---

Tool Management

Add Webhook Tool

elevenlabs tools add-webhook "Tool Name" --config-path tool_configs/tool.json

Add Client Tool

elevenlabs tools add-client "Tool Name" --config-path tool_configs/tool.json

Push Tools

elevenlabs tools push

Pull Tools

elevenlabs tools pull

Delete Tool

elevenlabs tools delete TOOL_ID

# Delete all tools
elevenlabs tools delete --all

---

Testing

Add Test

elevenlabs tests add "Test Name" --template basic-llm

Push Tests

elevenlabs tests push

Pull Tests

elevenlabs tests pull

Run Test

elevenlabs agents test "Agent Name"

---

Multi-Environment Workflow

# Development
elevenlabs agents push --env dev

# Staging
elevenlabs agents push --env staging

# Production
elevenlabs agents push --env prod --dry-run
# Review changes...
elevenlabs agents push --env prod

---

Common Workflows

Create and Deploy Agent

elevenlabs auth login
elevenlabs agents init
elevenlabs agents add "Support Bot" --template customer-service
# Edit agent_configs/support-bot.json
elevenlabs agents push --env dev
elevenlabs agents test "Support Bot"
elevenlabs agents push --env prod

Update Existing Agent

elevenlabs agents pull
# Edit agent_configs/agent-name.json
elevenlabs agents push --dry-run
elevenlabs agents push

Promote Agent to Production

# Test in staging first
elevenlabs agents push --env staging
elevenlabs agents test "Agent Name"

# If tests pass, promote to prod
elevenlabs agents push --env prod

---

Environment Variables

# For CI/CD
export ELEVENLABS_API_KEY=your-api-key

# Run commands
elevenlabs agents push --env prod

---

Troubleshooting

Reset Project

elevenlabs agents init --override
elevenlabs agents pull

Check Version

elevenlabs --version

Get Help

elevenlabs --help
elevenlabs agents --help
elevenlabs tools --help

---

File Locations

Config Files

~/.elevenlabs/api_key          # API key (if not using keychain)

Project Files

./agents.json                   # Agent registry
./tools.json                    # Tool registry
./tests.json                    # Test registry
./agent_configs/*.json          # Individual agent configs
./tool_configs/*.json           # Individual tool configs
./test_configs/*.json           # Individual test configs

---

Best Practices

1. Always use --dry-run before pushing to production 2. Commit configs to Git for version control 3. Use environment-specific configs (dev/staging/prod) 4. Test agents before deploying 5. Pull before editing to avoid conflicts 6. Use templates for consistency 7. Document changes in commit messages

Privacy & Compliance Guide

GDPR Compliance

Data Retention

Default: 2 years (730 days)

{
  "privacy": {
    "transcripts": {
      "retention_days": 730
    },
    "audio": {
      "retention_days": 730
    }
  }
}

Right to Be Forgotten

Enable data deletion requests:

await client.conversations.delete(conversation_id);

Data Residency

const { startConversation } = useConversation({
  serverLocation: 'eu-residency' // GDPR-compliant EU data centers
});

User Consent

Inform users before recording:

{
  "first_message": "This call will be recorded for quality and training purposes. Do you consent?"
}

---

HIPAA Compliance

Data Retention

Minimum: 6 years (2190 days)

{
  "privacy": {
    "transcripts": {
      "retention_days": 2190
    },
    "audio": {
      "retention_days": 2190
    }
  }
}

Encryption

• In Transit: TLS 1.3 (automatic)
• At Rest: AES-256 (automatic)

Business Associate Agreement (BAA)

Contact ElevenLabs for HIPAA BAA.

PHI Handling

Never:

• Store PHI in dynamic variables
• Log PHI in tool parameters
• Send PHI to third-party tools without BAA

Always:

• Use secure authentication
• Verify patient identity
• Document access logs

---

SOC 2 Compliance

Security Controls

✅ Encryption in transit and at rest (automatic) ✅ Access controls (API key management) ✅ Audit logs (conversation history) ✅ Incident response (automatic backups)

Best Practices

{
  "authentication": {
    "type": "signed_url", // Most secure
    "session_duration": 3600 // 1 hour max
  }
}

---

Regional Compliance

US Residency

serverLocation: 'us'

EU Residency (GDPR)

serverLocation: 'eu-residency'

India Residency

serverLocation: 'in-residency'

---

Zero Retention Mode

Maximum Privacy: Immediately delete all data after conversation ends.

{
  "privacy": {
    "zero_retention": true
  }
}

Limitations:

• No conversation history
• No analytics
• No post-call webhooks
• No MCP tool integrations

---

PCI DSS (Payment Card Industry)

Never:

❌ Store credit card numbers in conversation logs ❌ Send credit card data to LLM ❌ Log CVV or PIN numbers

Always:

✅ Use PCI-compliant payment processors (Stripe, PayPal) ✅ Tokenize payment data ✅ Use DTMF keypad for card entry (telephony)

Example: Secure Payment Collection

{
  "system_tools": [
    {
      "name": "dtmf_playpad",
      "description": "Display keypad for secure card entry"
    }
  ]
}

---

Compliance Checklist

GDPR

• [ ] Data retention ≤ 2 years (or justify longer)
• [ ] EU data residency enabled
• [ ] User consent obtained before recording
• [ ] Data deletion process implemented
• [ ] Privacy policy updated

HIPAA

• [ ] Data retention ≥ 6 years
• [ ] BAA signed with ElevenLabs
• [ ] Encryption enabled (automatic)
• [ ] Access logs maintained
• [ ] Staff trained on PHI handling

SOC 2

• [ ] API key security (never expose in client)
• [ ] Use signed URLs for authentication
• [ ] Monitor access logs
• [ ] Incident response plan documented

PCI DSS

• [ ] Never log card data
• [ ] Use tokenization for payments
• [ ] DTMF keypad for card entry
• [ ] PCI-compliant payment processor

---

Monitoring & Auditing

Access Logs

const logs = await client.conversations.list({
  agent_id: 'agent_123',
  from_date: '2025-01-01',
  to_date: '2025-12-31'
});

Compliance Reports

• Monthly conversation volume
• Data retention adherence
• Security incidents
• User consent rates

---

Incident Response

Data Breach Protocol

1. Identify affected conversations 2. Notify ElevenLabs immediately 3. Delete compromised data 4. Notify affected users (GDPR requirement) 5. Document incident 6. Review security controls

Contact

security@elevenlabs.io

Cost Optimization Guide

1. LLM Caching

How It Works

• First request: Full cost (input_cache_write)
• Subsequent requests: 10% cost (input_cache_read)
• Cache TTL: 5 minutes to 1 hour (configurable)

Configuration

{
  "llm_config": {
    "caching": {
      "enabled": true,
      "ttl_seconds": 3600
    }
  }
}

What Gets Cached

✅ System prompts ✅ Tool definitions ✅ Knowledge base context ✅ Recent conversation history

❌ User messages (always fresh) ❌ Dynamic variables ❌ Tool responses

Savings

Up to 90% on cached inputs

Example:

• System prompt: 500 tokens
• Without caching: 500 tokens × 100 conversations = 50,000 tokens
• With caching: 500 tokens (first) + 50 tokens × 99 (cached) = 5,450 tokens
• Savings: 89%

---

2. Model Swapping

Model Comparison

Configuration

{
  "llm_config": {
    "model": "gpt-4o-mini"
  }
}

Optimization Strategy

1. Start with gpt-4o-mini for all agents 2. Upgrade to gpt-4o only if:

• Complex reasoning required
• High accuracy critical
• User feedback indicates quality issues

3. Use Gemini 2.5 Flash for:

• Simple routing/classification
• FAQ responses
• Order status lookups

Savings

Up to 97% (gpt-4o → Gemini 2.5 Flash)

---

3. Burst Pricing

How It Works

• Normal: Your subscription concurrency limit (e.g., 10 calls)
• Burst: Up to 3× your limit (e.g., 30 calls)
• Cost: 2× per-minute rate for burst calls

Configuration

{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}

When to Use

✅ Black Friday traffic spikes ✅ Product launches ✅ Seasonal demand (holidays) ✅ Marketing campaigns

❌ Sustained high traffic (upgrade plan instead) ❌ Unpredictable usage patterns

Cost Calculation

Example:

• Subscription: 10 concurrent calls ($0.10/min per call)
• Traffic spike: 25 concurrent calls
• Burst calls: 25 - 10 = 15 calls
• Burst cost: 15 × $0.20/min = $3/min
• Regular cost: 10 × $0.10/min = $1/min
• Total: $4/min during spike

---

4. Prompt Optimization

Reduce Token Count

Before (500 tokens):

You are a highly experienced and knowledgeable customer support specialist with extensive training in technical troubleshooting, customer service best practices, and empathetic communication. You should always maintain a professional yet friendly demeanor while helping customers resolve their issues efficiently and effectively.

After (150 tokens):

You are an experienced support specialist. Be professional, friendly, and efficient.

Savings: 70% token reduction

Use Tools Instead of Context

Before: Include FAQ in system prompt (2,000 tokens) After: Use RAG/knowledge base (100 tokens + retrieval)

Savings: 95% for large knowledge bases

---

5. Turn-Taking Optimization

Impact on Cost

Recommendation

Use Patient mode for cost-sensitive applications where speed is less critical.

---

6. Voice Settings

Speed vs Cost

Recommendation

Use 1.1x speed for slight cost savings without compromising experience.

---

7. Conversation Duration Limits

Configuration

{
  "conversation": {
    "max_duration_seconds": 300 // 5 minutes
  }
}

Use Cases

• FAQ bots (limit: 2-3 minutes)
• Order status (limit: 1 minute)
• Full support (limit: 10-15 minutes)

Savings

Prevents unexpectedly long conversations.

---

8. Analytics-Driven Optimization

Monitor Metrics

1. Average conversation duration 2. LLM tokens per conversation 3. Tool call frequency 4. Resolution rate

Identify Issues

• Long conversations → improve prompts or add escalation
• High token count → enable caching or shorten prompts
• Low resolution rate → upgrade model or improve knowledge base

---

9. Cost Monitoring

API Usage Tracking

const usage = await client.analytics.getLLMUsage({
  agent_id: 'agent_123',
  from_date: '2025-11-01',
  to_date: '2025-11-30'
});

console.log('Total tokens:', usage.total_tokens);
console.log('Cached tokens:', usage.cached_tokens);
console.log('Cost:', usage.total_cost);

Set Budgets

{
  "cost_limits": {
    "daily_budget_usd": 100,
    "monthly_budget_usd": 2000
  }
}

---

10. Cost Optimization Checklist

Before Launch

• [ ] Enable LLM caching
• [ ] Use gpt-4o-mini (not gpt-4o)
• [ ] Optimize prompt length
• [ ] Set conversation duration limits
• [ ] Use RAG instead of large system prompts
• [ ] Configure burst pricing if needed

During Operation

• [ ] Monitor LLM token usage weekly
• [ ] Review conversation analytics monthly
• [ ] Test cheaper models quarterly
• [ ] Optimize prompts based on analytics
• [ ] Review and remove unused tools

Continuous Improvement

• [ ] A/B test cheaper models
• [ ] Analyze long conversations
• [ ] Improve resolution rates
• [ ] Reduce average conversation duration
• [ ] Increase cache hit rates

---

Expected Savings

Baseline Configuration:

• Model: gpt-4o
• No caching
• Average prompt: 1,000 tokens
• Average conversation: 5 minutes
• Cost: ~$0.50/conversation

Optimized Configuration:

• Model: gpt-4o-mini
• Caching enabled
• Average prompt: 300 tokens
• Average conversation: 3 minutes
• Cost: ~$0.05/conversation

Total Savings: 90% 🎉

System Prompt Engineering Guide

6-Component Framework

1. Personality

Define who the agent is.

Template:

You are [NAME], a [ROLE/PROFESSION] at [COMPANY].
You have [EXPERIENCE/BACKGROUND].
Your traits: [LIST PERSONALITY TRAITS].

Example:

You are Sarah, a patient and knowledgeable technical support specialist at TechCorp.
You have 7 years of experience helping customers troubleshoot software issues.
Your traits: patient, empathetic, detail-oriented, solution-focused.

2. Environment

Describe the communication context.

Template:

You're communicating via [CHANNEL: phone/chat/video].
Consider [ENVIRONMENTAL FACTORS].
Adapt your communication style to [CONTEXT].

Example:

You're speaking with customers over the phone.
Background noise and poor connections are common.
Speak clearly, use short sentences, and occasionally pause for emphasis.

3. Tone

Specify speech patterns and formality.

Template:

Tone: [FORMALITY LEVEL].
Language: [CONTRACTIONS/JARGON GUIDELINES].
Verbosity: [SENTENCE LENGTH, RESPONSE LENGTH].
Emotional Expression: [GUIDELINES].

Example:

Tone: Professional yet warm and approachable.
Language: Use contractions ("I'm", "let's") for natural conversation. Avoid technical jargon unless the customer uses it first.
Verbosity: Keep responses to 2-3 sentences. Ask one question at a time.
Emotional Expression: Express empathy with phrases like "I understand how frustrating that must be."

4. Goal

Define objectives and success criteria.

Template:

Primary Goal: [MAIN OBJECTIVE]

Secondary Goals:
- [SUPPORTING OBJECTIVE 1]
- [SUPPORTING OBJECTIVE 2]

Success Criteria:
- [MEASURABLE OUTCOME 1]
- [MEASURABLE OUTCOME 2]

Example:

Primary Goal: Resolve customer technical issues on the first call.

Secondary Goals:
- Verify customer identity securely
- Document issue details accurately
- Provide proactive tips to prevent future issues

Success Criteria:
- Customer verbally confirms their issue is resolved
- Issue documented in CRM system
- Customer satisfaction score ≥ 4/5

5. Guardrails

Set boundaries and ethical constraints.

Template:

Never:
- [PROHIBITED ACTION 1]
- [PROHIBITED ACTION 2]

Always:
- [REQUIRED ACTION 1]
- [REQUIRED ACTION 2]

Escalation Triggers:
- [CONDITION REQUIRING HUMAN INTERVENTION]

Example:

Never:
- Provide medical, legal, or financial advice
- Share confidential company information
- Make promises about refunds without verification
- Continue conversation if customer becomes abusive

Always:
- Verify customer identity before accessing account details
- Document all interactions in CRM
- Offer alternative solutions if first approach doesn't work

Escalation Triggers:
- Customer requests manager
- Issue requires account credit/refund approval
- Technical issue beyond your knowledge base
- Customer exhibits abusive behavior

6. Tools

Describe available functions and when to use them.

Template:

Available Tools:

1. tool_name(parameters)
   Purpose: [WHAT IT DOES]
   Use When: [TRIGGER CONDITION]
   Example: [SAMPLE USAGE]

2. ...

Guidelines:
- [GENERAL TOOL USAGE RULES]

Example:

Available Tools:

1. lookup_order(order_id: string)
   Purpose: Fetch order details from database
   Use When: Customer mentions an order number or asks about order status
   Example: "Let me look that up for you. [Call lookup_order(order_id='ORD-12345')]"

2. send_password_reset(email: string)
   Purpose: Trigger password reset email
   Use When: Customer can't access account and identity is verified
   Example: "I'll send you a password reset email. [Call send_password_reset(email='customer@example.com')]"

3. transfer_to_supervisor()
   Purpose: Escalate to human agent
   Use When: Issue requires manager approval or customer explicitly requests
   Example: "Let me connect you with a supervisor. [Call transfer_to_supervisor()]"

Guidelines:
- Always explain to the customer what you're doing before calling a tool
- Wait for tool response before continuing
- If tool fails, acknowledge and offer alternative

---

Complete Example Templates

Customer Support Agent

Personality:
You are Alex, a friendly and knowledgeable customer support specialist at TechCorp. You have 5 years of experience helping customers solve technical issues. You're patient, empathetic, and always maintain a positive attitude.

Environment:
You're speaking with customers over the phone. Communication is voice-only. Customers may have background noise or poor connection quality. Speak clearly and use thoughtful pauses for emphasis.

Tone:
Professional yet warm. Use contractions ("I'm", "let's") to sound natural. Avoid jargon unless the customer uses it first. Keep responses concise (2-3 sentences max). Use encouraging phrases like "I'll be happy to help with that."

Goal:
Primary: Resolve customer technical issues on the first call.
Secondary: Verify customer identity, document issues accurately, provide proactive solutions.
Success: Customer verbally confirms issue is resolved.

Guardrails:
- Never provide medical/legal/financial advice
- Don't share confidential company information
- Escalate if customer becomes abusive
- Never make promises about refunds without verification

Tools:
1. lookup_order(order_id) - Fetch order details when customer mentions order number
2. transfer_to_supervisor() - Escalate when issue requires manager approval
3. send_password_reset(email) - Trigger reset when customer can't access account
Always explain what you're doing before calling tools.

Educational Tutor

Personality:
You are Maya, a patient and encouraging math tutor. You have 10 years of experience teaching middle school students. You're enthusiastic about learning and celebrate every small victory.

Environment:
You're tutoring students via voice chat. Students may feel anxious or frustrated about math. Create a safe, judgment-free environment where mistakes are learning opportunities.

Tone:
Warm, encouraging, and patient. Never sound frustrated or disappointed. Use positive reinforcement frequently ("Great thinking!", "You're on the right track!"). Adjust complexity based on student's responses.

Goal:
Primary: Help students understand math concepts, not just get answers.
Secondary: Build confidence and reduce math anxiety.
Success: Student can explain the concept in their own words and solve similar problems independently.

Guardrails:
- Never give answers directly—guide students to discover solutions
- Don't move to next topic until current concept is mastered
- If student becomes frustrated, take a break or switch to easier problem
- Never compare students or use negative language

Tools:
1. show_visual_aid(concept) - Display diagram or graph to illustrate concept
2. generate_practice_problem(difficulty) - Create custom practice problem
3. celebrate_achievement() - Play positive feedback animation
Always make learning feel like an achievement, not a chore.

---

Prompt Engineering Tips

Do's:

✅ Use specific examples in guidelines ✅ Define success criteria clearly ✅ Include escalation conditions ✅ Explain tool usage thoroughly ✅ Test prompts with real conversations ✅ Iterate based on analytics

Don'ts:

❌ Use overly long prompts (increases cost) ❌ Be vague about goals or boundaries ❌ Include conflicting instructions ❌ Forget to test edge cases ❌ Use negative language excessively ❌ Overcomplicate simple tasks

---

Testing Your Prompts

1. Scenario Testing: Run automated tests with success criteria 2. Edge Case Testing: Test boundary conditions and unusual inputs 3. Tone Testing: Evaluate conversation tone and empathy 4. Tool Testing: Verify tools are called correctly 5. Analytics Review: Monitor real conversations for issues

---

Prompt Iteration Workflow

1. Write initial prompt using 6-component framework
2. Deploy to dev environment
3. Run 5-10 test conversations
4. Analyze transcripts for issues
5. Refine prompt based on findings
6. Deploy to staging
7. Run automated tests
8. Review analytics dashboard
9. Deploy to production
10. Monitor and iterate

Testing Guide

1. Scenario Testing (LLM-Based)

Create Test

elevenlabs tests add "Refund Request" --template basic-llm

Test Configuration

{
  "name": "Refund Request Test",
  "scenario": "Customer requests refund for defective product",
  "user_input": "I want a refund for order #12345. The product arrived broken.",
  "success_criteria": [
    "Agent acknowledges the issue empathetically",
    "Agent asks for order number or uses provided number",
    "Agent verifies order details",
    "Agent provides clear next steps or refund timeline"
  ],
  "evaluation_type": "llm"
}

Run Test

elevenlabs agents test "Support Agent"

2. Tool Call Testing

Test Configuration

{
  "name": "Order Lookup Test",
  "scenario": "Customer asks about order status",
  "user_input": "What's the status of order ORD-12345?",
  "expected_tool_call": {
    "tool_name": "lookup_order",
    "parameters": {
      "order_id": "ORD-12345"
    }
  }
}

3. Load Testing

Basic Load Test

# 100 concurrent users, spawn 10/second, run for 5 minutes
elevenlabs test load \
  --users 100 \
  --spawn-rate 10 \
  --duration 300

With Burst Pricing

{
  "call_limits": {
    "burst_pricing_enabled": true
  }
}

4. Simulation API

Programmatic Testing

const simulation = await client.agents.simulate({
  agent_id: 'agent_123',
  scenario: 'Customer requests refund',
  user_messages: [
    "I want a refund for order #12345",
    "It arrived broken",
    "Yes, process the refund"
  ],
  success_criteria: [
    "Agent shows empathy",
    "Agent verifies order",
    "Agent provides timeline"
  ]
});

console.log('Passed:', simulation.passed);
console.log('Criteria met:', simulation.evaluation.criteria_met, '/', simulation.evaluation.criteria_total);

5. Convert Real Conversations to Tests

From Dashboard

1. Navigate to Conversations 2. Select conversation 3. Click "Convert to Test" 4. Add success criteria 5. Save

From API

const test = await client.tests.createFromConversation({
  conversation_id: 'conv_123',
  success_criteria: [
    "Issue was resolved",
    "Customer satisfaction >= 4/5"
  ]
});

6. CI/CD Integration

GitHub Actions

name: Test Agent
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install CLI
        run: npm install -g @elevenlabs/cli

      - name: Push Tests
        run: elevenlabs tests push
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}

      - name: Run Tests
        run: elevenlabs agents test "Support Agent"
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}

7. Test Organization

Directory Structure

test_configs/
├── refund-tests/
│   ├── basic-refund.json
│   ├── duplicate-refund.json
│   └── expired-refund.json
├── order-lookup-tests/
│   ├── valid-order.json
│   └── invalid-order.json
└── escalation-tests/
    ├── angry-customer.json
    └── complex-issue.json

8. Best Practices

Do's:

✅ Test all conversation paths ✅ Include edge cases ✅ Test tool calls thoroughly ✅ Run tests before deployment ✅ Convert failed conversations to tests ✅ Monitor test trends over time

Don'ts:

❌ Only test happy paths ❌ Ignore failing tests ❌ Skip load testing ❌ Test only in production ❌ Write vague success criteria

9. Metrics to Track

• Pass Rate: % of tests passing
• Tool Accuracy: % of correct tool calls
• Response Time: Average time to resolution
• Load Capacity: Max concurrent users before degradation
• Error Rate: % of conversations with errors

10. Debugging Failed Tests

1. Review conversation transcript 2. Check tool calls and parameters 3. Verify dynamic variables provided 4. Test prompt clarity 5. Check knowledge base content 6. Review guardrails and constraints 7. Iterate and retest