Pipecat

Pipecat

Pipecat is an open-source Python framework for building real-time voice and multimodal bots. It composes streaming speech/LLM/TTS services into a low-latency pipeline, connected via transports (WebRTC/WebSocket) and client SDKs using the RTVI message standard.

Links

• Documentation
• Changelog
• GitHub

Quick navigation

• Installation (packages/extras/CLI): references/installation.md
• Migration to 1.0: references/migration-1-0.md
• Concepts & architecture: references/core-concepts.md
• Session initialization (runner/bot/client): references/session-initialization.md
• Pipeline & frames: references/pipeline-and-frames.md
• Transports: references/transports.md
• Speech input & turn detection: references/speech-input-and-turn-detection.md
• Client SDKs + RTVI messaging: references/client-sdks-rtvi.md
• CLI (init/tail/cloud): references/cli.md
• Function calling (server): references/function-calling.md
• Context management: references/context-management.md
• LLM inference: references/llm-inference.md
• Text to speech (TTS): references/text-to-speech.md
• Deployment (pattern/platforms): references/deployment.md
• Server APIs (supported services): references/server-services.md
• Server Utilities (runner): references/server-runner.md
• Server APIs (pipeline/task/params): references/server-pipeline-apis.md
• Pipecat Cloud ops: references/pipecat-cloud.md
• Troubleshooting: references/troubleshooting.md

Mental model (cheat sheet)

• Pipeline: ordered processors that consume/emit frames.
• Frames: the streaming units (audio/text/video/context/events) flowing through the pipeline.
• Transport: connectivity + media IO + session state (WebRTC/WebSocket/provider realtime).
• Runner: HTTP service that starts sessions and spawns a bot process with transport credentials.
• Client SDK: starts the bot, connects transport, sends messages/requests, receives events.

Recipes

1) Keep secrets server-side

• Put provider API keys (LLM/STT/TTS) only on the server/bot container.
• The client should call a server start endpoint (startBot / startBotAndConnect) to receive transport credentials (e.g., a room URL + token), not provider keys.

2) Use WebRTC for production voice

• Prefer a WebRTC transport (e.g., Daily) for resilience and media quality.
• Use a WebSocket transport mostly for server↔server, prototypes, or constrained environments.

2b) Design for streaming + overlap

• Keep the pipeline fully streaming (avoid batching whole turns when you can).
• If your services support it, start TTS from partial LLM output to reduce perceived latency.

3) Initialize and evolve context via RTVI

• Initialize the bot’s pipeline context from the server start request payload.
• For ongoing interaction, prefer a dedicated “send text” style API (when available) instead of deprecated context append methods.

4) Function calling: end-to-end flow

• LLM requests a function call.
• Client registers a handler by function name.
• Client returns a function-call result message back to the bot.

5) Pipecat Cloud deployment basics

• Build/push an image that matches the expected platform (Pipecat Cloud requires linux/arm64 in the docs).
• Use a deployment config file for repeatability.
• Configure pool sizing with min_agents (warm capacity) and max_agents (hard limit).

Critical gotchas / prohibitions

• Do not embed sensitive API keys in client apps.
• Expect and handle “at capacity” responses (HTTP 429) when the pool is exhausted.
• Plan for cold-start latency if min_agents = 0.
• Ensure secrets and image-pull credentials are created in the same region as the deployed agent.
• Do not assume deprecated import shims or service-specific context classes still exist in 1.0.0; audit imports before upgrading.
• Do not keep VAD/turn-detection logic on transport params; current releases route that control through LLMUserAggregator strategies.
• Do not assume OpenAIResponsesLLMService is HTTP-based anymore; WebSocket is now the default implementation.

Release Highlights (0.0.109 -> 1.2.0)

Runtime and service additions

• `OpenAIResponsesLLMService` now defaults to a persistent WebSocket connection; the prior HTTP behavior moved to OpenAIResponsesHttpLLMService.
• Inworld Realtime LLM adds a WebSocket cascade STT/LLM/TTS path with semantic VAD and function calling.
• `MistralTTSService` adds streaming Voxtral TTS, and TTS/STT services gained more runtime-update and sample-rate options.
• The development runner now exports a module-level FastAPI app for custom routes before main().

Tooling and context changes

• Function calling now supports grouped parallel tool batches, async tool completion after interruption, and streaming intermediate tool results.
• Context editing now has LLMMessagesTransformFrame, and the framework standardizes on universal LLMContext / LLMContextAggregatorPair.
• OpenAI tool schemas can now include provider-specific custom_tools.
• 1.2.0 adds add_tool_change_messages for LLM aggregators, widens tool_resources into deprecated app_resources, and extends async-tool compatibility across more realtime providers.

Turn-taking and client protocol

• 1.2.0 adds explicit inference/finalization turn hooks (on_user_turn_inference_triggered, LLMTurnCompletionUserTurnStopStrategy, FilterIncompleteUserTurnStrategies) for smarter end-of-turn gating.
• RTVI grows first-class UI Agent Protocol support with ui-event, ui-snapshot, ui-cancel-task, ui-command, and ui-task, bumping the protocol to 1.3.0.
• The development runner and runner arguments now carry a stable session_id, which is useful for per-session tracing across local and cloud-like flows.

Breaking migrations

• Deprecated service-specific context classes, transport params, RTVI shims, frame aliases, and interruption/VAD helpers were removed across the stack.
• Turn detection and mute behavior moved toward LLMUserAggregator strategies instead of transport-level configuration.
• Some legacy providers and helpers were removed entirely (OpenPipeLLMService, TTSService.say(), FrameProcessor.wait_for_task(), older beta/alias modules).

Release Highlights (1.3.0)

• Workers and multi-agent pipelines: PipelineTask / PipelineRunner are renamed toward PipelineWorker / WorkerRunner, and pipecat.workers makes pipelines peers on a typed-message bus for @job dispatch, handoffs, sidecars, UI workers, and distributed Redis/PGMQ patterns.
• UIWorker and RTVI UI protocol: UIWorker can observe client accessibility snapshots and drive UI commands over RTVI; the UI worker vocabulary moves from task/agent to job/worker (ui-task -> ui-job-group, cancelUITask -> cancelUIJobGroup).
• Development runner: one runner can serve WebRTC, Daily, telephony, and plain WebSocket clients; /start accepts a transport field, /ws-client supports protobuf WebSocket clients, /status reports enabled transports, and the Daily redirect moved to /daily.
• Service surface: adds Vonage Video Connector transport, Inception Mercury 2 LLM service, Cartesia turn-based STT, Rime coda TTS defaults, Soniox endpoint-delay settings, LLMService.append_system_instruction(), and STTService.supports_ttfs.
• Operational migration notes: optional service/transport extras now raise ImportError, transformers is no longer a base dependency, Rime defaults to coda, OpenRouter defaults to openai/gpt-4.1 and maps developer messages to user unless explicitly supported.

Links

• Docs: https://docs.pipecat.ai/getting-started/introduction
• Full-text extract used for this skill: https://docs.pipecat.ai/llms-full.txt
• Changelog: https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md
• GitHub: https://github.com/pipecat-ai/pipecat
• PyPI (framework): https://pypi.org/project/pipecat-ai/
• PyPI (cloud SDK): https://pypi.org/project/pipecatcloud/

CLI (pipecat / pc)

Pipecat provides a CLI for scaffolding projects (init), monitoring sessions (tail), and operating Pipecat Cloud deployments (cloud).

For installation, see: references/installation.md.

Verify:

• pipecat --version

Help:

• pipecat --help
• pipecat init --help
• pipecat tail --help
• pipecat cloud --help

Notes:

• Commands support both pipecat and the shorter pc alias.
• Requires Python 3.10+.

pipecat init

Scaffolds a new Pipecat project (server bot, optional web client, optional cloud deploy config).

Typical outputs

A generated project commonly includes:

• server/ (Python bot)
• bot.py
• pyproject.toml
• .env.example
• Dockerfile (when cloud deploy files are enabled)
• pcc-deploy.toml (when cloud deploy files are enabled)
• client/ (optional)
• package.json, src/, etc.
• Top-level helpers
• .gitignore, README.md

Modes

• Interactive: run pipecat init without --name / --config.
• Non-interactive: provide --name or --config (JSON). If required fields are missing, the command reports what’s missing.

Useful flags (selected)

• Output and naming
• --output/-o <dir>
• --name/-n <name>
• Bot shape
• --bot-type/-b web|telephony
• --transport/-t <provider> (repeatable)
• --mode/-m cascade|realtime
• Cascade services (STT/LLM/TTS)
• --stt <service>
• --llm <service>
• --tts <service>
• Realtime service (realtime mode)
• --realtime <service>
• Web client (web bots)
• --client-framework react|vanilla|none
• --client-server vite|nextjs
• Telephony-specific
• --daily-pstn-mode dial-in|dial-out
• --twilio-daily-sip-mode dial-in|dial-out
• Feature toggles (examples)
• --recording/--no-recording
• --transcription/--no-transcription
• --smart-turn/--no-smart-turn
• --observability/--no-observability
• Cloud scaffolding
• --deploy-to-cloud/--no-deploy-to-cloud
• --enable-krisp/--no-enable-krisp (cloud-related)
• Discovery/debug
• --list-options (prints available services/options as JSON and exits)
• --dry-run (prints resolved config JSON without generating files)

Examples

• Interactive wizard:
• pipecat init

• Non-interactive (cascade):
• pipecat init --name my-bot --bot-type web --transport daily --mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts

• Non-interactive (realtime):
• pipecat init --name rt-bot --bot-type web --transport smallwebrtc --mode realtime --realtime openai_realtime

pipecat tail

Tail is a terminal dashboard to monitor Pipecat sessions in real time (logs, conversation, metrics/usage, audio levels).

Requirements

• Add an observer to your server-side pipeline task:
• from pipecat_cli.tail import TailObserver
• PipelineTask(..., observers=[TailObserver()])

Usage

• Start Tail:
• pipecat tail
• Connect to a remote session:
• pipecat tail --url wss://my-bot.example.com

Selected flag:

• --url/-u <ws-url>
• Default: ws://localhost:9292

pipecat cloud

Pipecat Cloud commands cover authentication, deployments, secrets, regions, and agent operations.

Auth (pipecat cloud auth)

• pipecat cloud auth login (supports --headless/-h for remote/container environments)
• pipecat cloud auth logout
• pipecat cloud auth whoami

Config storage:

• Default path: ~/.config/pipecatcloud/pipecatcloud.toml
• Override path with PIPECAT_CONFIG_PATH
• View effective config with: pipecat cloud --config

Organizations (pipecat cloud organizations)

• pipecat cloud organizations list
• pipecat cloud organizations select (or --organization/-o <org>)

Notes:

• The currently selected organization is stored in the local config and used by default.
• Org/user management is not available via the CLI (use the dashboard).

Regions (pipecat cloud regions)

• pipecat cloud regions list

Use the region code with other commands (deploy/agent/secrets). Keep secrets in the same region as the agents that consume them.

Secrets (pipecat cloud secrets)

• List sets / keys:
• pipecat cloud secrets list
• pipecat cloud secrets list <set-name>
• Create/update a secret set:
• pipecat cloud secrets set <set-name> KEY=value ...
• pipecat cloud secrets set <set-name> --file .env
• Delete key or set:
• pipecat cloud secrets unset <set-name> <key>
• pipecat cloud secrets delete <set-name>
• Create image pull credentials (used by deploy):
• pipecat cloud secrets image-pull-secret <name> <host> [user:pass]

Notes:

• Listing shows keys, not values.
• Prefer --file to avoid leaking secrets via shell history.

Deploy (pipecat cloud deploy)

• Deploy/update an agent:
• pipecat cloud deploy <agent-name> <image>

Selected options:

• --credentials/-c <image-pull-secret-name>
• --secrets/-s <secret-set-name>
• --min-agents/-min <n> / --max-agents/-max <n>
• --profile/-p agent-1x|agent-2x|agent-3x
• --region/-r <code>
• --force/-f

Config file support:

• pcc-deploy.toml (auto-detected in the current directory)
• Precedence: CLI args > pcc-deploy.toml > defaults.

Krisp note:

• Legacy enable_krisp is documented as deprecated in favor of krisp_viva config.

Docker helper (pipecat cloud docker)

• Build and push using pcc-deploy.toml:
• pipecat cloud docker build-push

Selected options:

• --version/-v <tag>
• --no-push
• --no-latest
• Registry overrides: --registry, --registry-url, --username

Platform:

• Docs state images are built for linux/arm64 for Pipecat Cloud.

Agent ops (pipecat cloud agent)

Core subcommands:

• start (supports --data JSON, --use-daily, --daily-properties)
• stop (requires --session-id)
• status
• deployments (deployment history)
• logs (filter by --level, --deployment, --session-id, --limit)
• list (supports --region and --organization)
• sessions (can show detailed metrics with --id)
• delete (irreversible; supports --force)

Client SDKs, RTVI messaging, and function calling

Connection flow

A typical pattern in client SDKs:

1. startBot(startParams) — call your server endpoint to start a session and receive connection parameters. 2. connect(connectParams) — connect the chosen transport. 3. Or use startBotAndConnect(startParams) to do both.

Messaging primitives

• Client → server message: fire-and-forget
• Client → server request: request/response with a timeout
• Event handlers for server messages and connection state

1.2.0 note:

• RTVI adds first-class UI Agent Protocol messages. Expect client-to-server ui-event, ui-snapshot, and ui-cancel-task, plus server-to-client ui-command and ui-task.
• The RTVI protocol version moves to 1.3.0. If your client/server pins protocol behavior, upgrade both ends together.
• Default UI command payloads now cover actions such as toast, navigate, scroll, highlight, focus, click, set input value, and select text.

1.3.0 note:

• UIWorker (pipecat.workers.ui) observes the client's accessibility snapshots, routes client UI events to @ui_event handlers, drives UI commands such as scroll/highlight/click/input selection, and can answer screen-grounded questions.
• The UI worker protocol vocabulary changed from task/agent to job/worker: ui-task -> ui-job-group, ui-cancel-task -> ui-cancel-job-group, task_id -> job_id, agents -> workers, and React/JS APIs such as cancelUITask / useUITasks become cancelUIJobGroup / useUIJobGroups.
• Use ReplyToolMixin, respond_to_job(..., tts_speak=True), and ui_job_group(...) when a voice agent delegates UI-grounded work and wants cancellable progress cards in the client.

Context updates

Docs highlight a “send text” style call for appending user text to the conversation, with options like:

• Run immediately vs wait for normal turn-taking
• Whether to also produce an audio response

If you see older APIs that directly append context, treat them as deprecated and prefer the current “send text” approach.

Function calling

End-to-end shape:

• LLM emits a function call request.
• Client registers a handler for a specific function name.
• Client sends a function-call result back to the bot.

Docs mention message evolution where “in-progress” and “stopped” events replace older message types; expect version differences across SDKs.

Useful events

The docs mention callbacks/events such as:

• Bot ready/connected/disconnected
• User transcript updates
• Bot output (including whether it was spoken)
• Function call started/in-progress/stopped
• Media/device events and audio level events

When using the UI Agent Protocol additions, expect a parallel event path for UI state snapshots and task cancellation in addition to the older transcript/function-call events.

Context management

What “context” means

In Pipecat, context is the conversation history the LLM uses to respond: a sequence of role-tagged messages (system/user/assistant/tool).

As of 1.0.0, Pipecat standardizes on the universal LLMContext model. Older service-specific context classes (OpenAILLMContext, Anthropic/AWS variants) and older service-owned context aggregator helpers have been removed.

Automatic context updates

The Learn guide describes a default automatic flow:

• User audio → STT → transcription frames → user context aggregator stores a new user message.
• LLM output → TTS → TTS text frames → assistant context aggregator stores what was actually spoken.

Important detail: storing TTS text keeps context aligned with what the user heard (especially when output is interrupted).

Aggregators and placement

• Place the user context aggregator downstream from STT (so it can collect transcription frames).
• Place the assistant context aggregator after transport.output() so it can observe the post-TTS text frames and update word-by-word when supported.
• Build aggregators around LLMContextAggregatorPair(context) instead of per-provider create_context_aggregator(...) patterns.

This placement also helps keep context correct when interruptions cut off the bot mid-sentence.

Function calling and context

• Tool/function definitions live in the context as a schema.
• Tool calls and results are also stored in history so the conversation remains complete.
• Async tool completions can arrive later as developer messages; design any downstream logic that inspects history with that role in mind.

Manual control via frames

Learn docs mention control frames that let you:

• append new messages to the existing context
• replace the entire message list

Use cases:

• bot should speak first at session start
• system-mode changes (“you are now…”) applied mid-session
• injecting external events into the conversation

1.0.0 also adds LLMMessagesTransformFrame, which is safer than taking an old snapshot of messages, mutating it locally, and overwriting queued context updates.

Context summarization

For long sessions, the guide describes built-in summarization that compresses older history while keeping recent turns. Enable it via assistant aggregator params.

Pipecat 0.0.105 also exposes an on_summary_applied event on LLMAssistantAggregator, so you can observe summarization without reaching into private members.

Practical checklist

• Keep assistant aggregation after output to store what was actually spoken.
• Be explicit about whether system_instruction lives on the service, in the context, or both; service-level instruction now wins if both are present.
• If you mutate context manually, decide whether it should trigger an immediate LLM run.
• Enable summarization early for long-running agents to control token growth.
• If you are upgrading, replace any service-specific context imports before changing business logic; that migration is usually mechanical.

Core concepts & architecture (Pipecat)

Why voice AI is hard (what Pipecat abstracts)

Real-time voice bots need tight coordination across multiple streaming subsystems:

• Speech recognition (STT) while the user is speaking
• LLM generation with evolving context
• Speech synthesis (TTS) that can start early for low latency
• A transport that streams audio with minimal delay, buffering, and robust reconnect/error behavior

Key concepts

• Pipecat: Python framework for real-time voice and multimodal bots.
• RTVI: a message/event standard used between clients and servers for real-time multimedia + LLM interactions.
• Pipeline: chain of processors (STT, LLM, TTS, aggregators, observers, transports).
• Frame: the unit flowing through the pipeline (audio, text, context, events).
• Transport: manages media devices, connectivity, transmission, and lifecycle state.

Frames, processors, pipelines (one-liners)

• Frames are typed “data packets” (audio chunks, transcripts, LLM output, synthesized audio, events).
• Frame processors are single-purpose workers that transform frames (audio → text, text → audio, etc.).
• Pipelines connect processors so frames flow and orchestration happens automatically.

Typical runtime architecture

A common pattern is two processes:

• Bot runner (HTTP service)
• Receives “start session” requests from clients
• Creates/allocates a transport session (e.g., a WebRTC room)
• Spawns a bot process with transport credentials
• Returns connection parameters to the client

• Bot process
• Connects to the transport
• Runs the pipeline
• Exchanges RTVI messages/events with the client

Transport lifecycle (high-level)

Transports typically model a state machine. The docs list states like:

• Disconnected → Initializing → Initialized → Authenticating → Authenticated → Connecting → Connected → Ready
• Disconnecting / Error

Best practices

• Keep provider keys server-side; client should only receive transport credentials.
• Use a start endpoint to initialize bot context (prompt/config) before connecting.
• Prefer WebRTC transports for production voice apps.

Deployment (overview, pattern, platforms)

This page summarizes the docs under /deployment/*.

Production guidance (high-level)

• You can run Pipecat anywhere that can run Python, but for production client↔server voice, the docs recommend WebRTC transports over raw WebSockets.
• Plan for:
• A transport service (WebRTC/WebSocket) that can accept connections.
• A deployment target (VMs, containers, managed services).
• Docker (common for cloud targets).

Recommended deployment pattern

The docs describe a common split:

• Bot (e.g., bot.py): encapsulated agent code + pipeline; accepts connection/session inputs.
• Runner (e.g., bot_runner.py): small HTTP service that accepts a request (or webhook), prepares transport credentials (e.g., creates a room + token), then spawns a bot process.

Typical runner endpoint:

• POST /start_bot → returns JSON with connection details like room_url and token.

Scaling note:

• Spawning bots as subprocesses is simple for early cloud testing, but may not hold up under load.
• For higher scale, isolate bots with their own resources (containers/VMs) instead of piling subprocesses onto a single host.

Secrets note:

• Keep provider keys in server-side environment variables / secrets (runner and/or bot). Avoid putting provider API keys in the client.

Platform guides (selected)

Fly.io

What the docs show:

• fly.toml (HTTP service on port 7860), Dockerfile, .env, and a bot_runner.py that calls the Fly Machines API to spawn bot workers.

Key gotchas:

• Cache model assets (e.g., Silero VAD) in the image to avoid slow cold starts.
• Example mentions 512MB can be “just enough” for VAD; 1GB is safer.
• Spawning machines from an unauthenticated HTTP endpoint can cause uncontrolled costs; lock down / authenticate your start endpoint.

Modal

What the docs show:

• Deploy a self-hosted OpenAI-compatible LLM service (vLLM example), then deploy a Pipecat FastAPI app on Modal.

Operational notes:

• LLM cold starts can be minutes; warm the service and consider min_containers=1 to reduce latency.
• Modal apps/logs: use the Modal dashboard to inspect serve / fastapi_app / bot_runner logs.

Cerebrium

What the docs show:

• cerebrium init, configure cerebrium.toml, store secrets in the Cerebrium dashboard.
• Deploy an HTTP endpoint and pass Daily room_url + token into your agent entry.

Latency/scaling note:

• Docs emphasize cold-start performance and the option to run more components locally (LLM/TTS/STT) to reduce voice-to-voice latency.

Function calling (server-side)

This page focuses on function calling inside the server pipeline (LLM service + context aggregators).

Where it fits in the pipeline

Typical placement:

• User text enters context aggregation.
• LLM decides to call a function/tool.
• Your handler runs and returns results.
• LLM incorporates the result into the response which continues to TTS/output.
• Context aggregators store calls + results in history.

Registering function handlers

Docs describe registering a named function handler on the LLM service.

Behavioral knob:

• cancel_on_interruption: cancel the function call if the user interrupts mid-flight (docs say default enabled).
• timeout_secs: override the global function-call timeout per tool when a specific integration needs a tighter or looser budget.
• group_parallel_tools: when left at the default True, tool calls from the same LLM response batch share one group and Pipecat re-runs the LLM once after the last grouped tool completes.

FunctionCallParams: what you get

Docs show a params object with:

• function name and call/tool id
• parsed arguments from the LLM
• access to the current LLM context (conversation history)
• a reference to the LLM service
• a result_callback used to return a structured result

Returning results

• Your handler should return the final result via result_callback(result).
• Treat required configuration (API keys, endpoints) as mandatory and fail fast if missing.

For async handlers that continue after interruption (cancel_on_interruption=False), Pipecat can stream intermediate updates back into the conversation:

• call result_callback(..., properties=FunctionCallResultProperties(is_final=False)) for partial progress;
• call it once more with the final result (is_final=True, the default) when the work is complete.

When you allow a function to outlive the current turn, Pipecat injects the eventual result back as a developer message and triggers another LLM inference.

1.2.0 notes:

• LLMContextAggregatorPair(..., add_tool_change_messages=True) appends a developer-role message whenever the available standard tools change mid-conversation. Use it when tool availability is dynamic and the model tends to hallucinate removed or re-added tools.
• tool_resources has been broadened to app_resources. New code should read params.app_resources, PipelineTask.app_resources, and self.pipeline_task.app_resources; the old tool_resources aliases still work but are deprecated.
• Async tool continuation after interruption is restored/expanded across more realtime services, but streamed intermediate results are still not universally supported. Re-test cancel_on_interruption=False separately for each realtime provider you depend on.

Advanced control: chaining calls

Docs mention result properties such as:

• run_llm: if set to false, you can prevent the LLM from running immediately after a tool result (useful for back-to-back tool calls).
• on_context_updated: callback that runs after the function result has been added to the context.

If you skip LLM execution, you must explicitly trigger the next step when appropriate (otherwise the conversation may stall).

Migration notes for 1.0.0

• Single-argument function call support was removed; tools must expose named parameters.
• Prefer the async flow above instead of bespoke background-task side channels.
• If you relied on older handle_function_call* RTVI/processor helpers, move to the current processor API and universal context flow.

Practical checklist

• Keep handlers idempotent and cancel-safe.
• Set per-function timeout_secs for slow or third-party tools instead of relaxing the global timeout for everything.
• Decide whether user interruptions should cancel long-running tools.
• Log tool call ids for tracing and debugging.
• If tools appear/disappear dynamically, enable add_tool_change_messages instead of relying on prompt-only reminders.

Installation (framework, extras, CLI, cloud SDK)

This skill assumes Pipecat is already installed. Use this page only when you need to set up a fresh environment or add provider/feature extras.

Prereqs

• Python 3.10+

Framework: pipecat-ai

Base install:

• pip install pipecat-ai

Provider/feature extras (pattern used throughout the docs):

• pip install "pipecat-ai[openai]"
• pip install "pipecat-ai[runner]" (runner utilities)
• You can combine extras if needed (keep them explicit).

1.3.0 packaging notes:

• transformers is no longer a base dependency. Install the relevant extras (local-smart-turn, moondream, or other service extras) only when those features are used.
• Services/transports imported without their optional dependency now raise ImportError with the original ModuleNotFoundError as __cause__; wrap optional imports with except ImportError instead of broad except Exception.
• The deepgram extra allows deepgram-sdk>=6.1.1,<8, and the websockets-base extra no longer caps the upper bound beyond websockets>=13.1.

CLI: pipecat-ai-cli

Recommended for scaffolding/ops (pipecat init, pipecat tail, pipecat cloud ...).

• uv tool install pipecat-ai-cli
• or pipx install pipecat-ai-cli

Pipecat Cloud SDK: pipecatcloud

Install when you want to start sessions via Python code:

• pip install pipecatcloud

Verification

• CLI: pipecat --version
• Python import sanity check:
• python -c "import pipecat"

LLM inference

What the LLM service does

• Consumes an LLM context frame (conversation history)
• Streams response tokens downstream as LLM text frames
• Optionally triggers function/tool calls when tools are available

Placement in the pipeline

The Learn guide places LLM after the user context aggregator and before downstream consumers (TTS, output).

Streaming boundaries and output control

Docs describe lifecycle/boundary frames around a streamed completion:

• “full response start” marker
• streaming token/text frames
• “full response end” marker

There is also a configuration frame that can mark output as not to be spoken (skip TTS) while still flowing through the pipeline.

Function call lifecycle frames

The Learn guide mentions frames signaling:

• function calls started
• function call in progress
• function call result

This is useful for UI/UX (“thinking…”) and tracing.

Provider switching: OpenAI-compatible base URL

Docs describe an OpenAI-compatible base service pattern where you can point at an OpenAI-spec endpoint via a base_url without rewriting pipeline code.

1.0.0 changes the default OpenAI Responses integration: OpenAIResponsesLLMService now uses a persistent WebSocket connection and incremental context via previous_response_id. If you explicitly want the prior request/response model, use OpenAIResponsesHttpLLMService.

Service switching and fallback (0.0.105)

Pipecat adds ServiceSwitcherStrategyFailover, which automatically moves to the next service after a non-fatal provider error. Use the on_service_switched event to log or react to the failover.

Parallelism for tool calls

An LLM service option controls whether multiple tool calls run in parallel or sequentially. Use sequential execution for dependent tool chains.

OpenAI-oriented tool schemas can also include custom_tools when you need provider-specific capabilities alongside standard function tools.

Event handlers

Docs mention events such as:

• completion timeout
• function calls started

Use these to implement user feedback and recovery (retry/backoff/fallback) as needed.

System instruction behavior (0.0.105)

• system_instruction is now wired consistently across the OpenAI, Anthropic, and AWS Bedrock LLM services as a default system prompt.
• run_inference now accepts a one-shot system_instruction override.
• If you set both constructor-level system_instruction and a system message in context, the constructor value takes precedence and Pipecat logs a warning.

LLM service updates (1.3.0)

• LLMService.append_system_instruction(...) appends durable system text that is included on every inference and survives context resets. Prefer it when a worker needs persistent task guidance without rewriting the whole context.
• InceptionLLMService supports Inception Mercury 2 diffusion reasoning with reasoning_effort and realtime settings.
• OpenRouterLLMService now defaults to openai/gpt-4.1 and converts developer messages to user by default for broader model compatibility. Set llm.supports_developer_role = True or subclass when the target model actually supports the developer role.
• InworldRealtimeLLMService defaults STT to inworld/inworld-stt-1; verify any explicit STT override before removing old defaults.

Practical checklist

• Keep completion streaming enabled if you want low perceived latency.
• If you skip TTS for specific outputs, ensure your client still receives a useful text channel.
• Decide whether tool calls must be parallel or sequential based on dependencies.
• If you migrate from older OpenAI Responses code, verify connection lifecycle, proxy compatibility, and reconnect behavior under WebSocket transport.

Migration to 1.0

Pipecat 1.0.0 is not a routine upgrade. It removes many deprecated shims and finishes several architectural migrations that started in the 0.0.x line.

Highest-impact changes

• OpenAIResponsesLLMService is now WebSocket-based by default; use OpenAIResponsesHttpLLMService if you explicitly need the older HTTP request model.
• Universal LLMContext and LLMContextAggregatorPair replace older service-specific context classes and create_context_aggregator(...) helpers.
• Turn detection, interruption behavior, and user mute logic now belong to LLMUserAggregator strategies instead of transport parameters and older filter/helper classes.
• Deprecated module aliases and compatibility packages were removed across services, transports, RTVI, and frames.
• Some long-deprecated APIs are gone entirely: TTSService.say(), FrameProcessor.wait_for_task(), older beta realtime services, OpenPipeLLMService, and single-argument tool calls.

Upgrade checklist

1. Audit imports for removed aliases before changing runtime behavior. 2. Replace service-specific context classes with LLMContext and LLMContextAggregatorPair. 3. Review transport config for removed VAD/turn/interruption parameters. 4. Re-test function calling if you rely on background work, sequential tools, or custom tool schemas. 5. Re-test OpenAI Responses integrations under persistent WebSocket behavior. 6. Re-test TTS and interruption behavior under the current frame and user-turn APIs.

Migration hotspots to search for

• OpenAILLMContext, AnthropicLLMContext, AWSBedrockLLMContext
• create_context_aggregator(
• vad_enabled, vad_audio_passthrough, vad_analyzer, turn_analyzer
• allow_interruptions, interruption_strategies, STTMuteFilter
• TTSService.say(
• FrameProcessor.wait_for_task(
• deprecated alias packages like pipecat.services.openai_realtime, pipecat.services.google.gemini_multimodal_live, pipecat.transports.services, pipecat.transports.network

Practical rule

Do the mechanical migration first (imports, class names, config fields), then validate runtime behavior for context flow, interruptions, tool execution, and transport startup.

Links

• Release: https://github.com/pipecat-ai/pipecat/releases/tag/v1.0.0
• Changelog: https://github.com/pipecat-ai/pipecat/blob/main/CHANGELOG.md
• Docs: https://docs.pipecat.ai/getting-started/introduction

Pipecat Cloud (deploy, scaling, secrets)

Pipecat Cloud is the managed service described in the docs for deploying and operating bot images.

Core objects

• Agent image: container image for your bot.
• Agent pool: a pool of instances used to serve sessions.
• Session: a running instance handling a user interaction.

CLI (high-level)

See: references/cli.md.

Accounts, orgs, and API keys

• Pipecat Cloud has personal workspaces (not suitable for collaboration) and organizations for team workflows.
• Many operations depend on the selected organization in your local config (or an explicit --org / --organization flag).

API keys:

• Private API keys: administrative/server-to-server use; should not be shared.
• Public API keys: suitable for starting sessions (still treat as sensitive; keep server-side in production).

Base image and image constraints

The docs describe an official base image:

• dailyco/pipecat-base (multi-modal base image)
• Multiple Python tags; docs state default Python version changed to 3.12 starting at base image version 0.1.0.

Important constraints:

• Pipecat Cloud requires images built for linux/arm64.
• Avoid copying files to /app (reserved by the base image).

Reserved routes (base image):

• POST /bot (main session start entry point)
• GET|WS /ws (WebSocket endpoint)
• POST /api/offer + PATCH /api/offer (SmallWebRTC offer/ICE)

Useful env vars (selection):

• PORT (default 8080)
• PIPECAT_LOG_LEVEL (TRACE/DEBUG/INFO/WARNING/ERROR/NONE)
• PCC_LOG_FEATURES_SUMMARY=true (logs enabled features at startup)

Deploy configuration

Pipecat Cloud deploys are typically driven by:

• CLI: pipecat cloud deploy <agent-name> <image>
• Config file: pcc-deploy.toml (CLI args override file values)

Agent profiles (resource sizing):

• agent-1x: 0.5 vCPU / 1GB (docs: best for voice agents)
• agent-2x: 1 vCPU / 2GB
• agent-3x: 1.5 vCPU / 3GB

Scaling model

• min_agents: warm pool (reduces cold starts, increases reserved cost)
• max_agents: hard cap; at-capacity start requests can return HTTP 429
• Docs mention idle instances (when min_agents=0) are kept briefly (example: ~5 minutes) before scale-to-zero.

Secrets (fundamentals)

• Secrets are stored in secret sets (key/value).
• Secret keys are mounted into the agent as environment variables.
• Secret sets are region-specific and must match the deployed agent region.
• Updating a secret set requires redeploying agents that use it to pick up the new values.

Image pull secrets:

• Needed for private registries; region-specific; typically created via pipecat cloud secrets image-pull-secret.
• Docs note image-pull secrets cannot be updated in-place (delete + recreate to rotate).

Starting sessions (active sessions)

The docs describe three common ways to start sessions:

• REST (public start): POST https://api.pipecat.daily.co/v1/public/{agent_name}/start with Authorization: Bearer PUBLIC_API_KEY
• CLI: pipecat cloud agent start <agent-name> --api-key pk_... (can pass JSON data)
• Python SDK (pipecatcloud): Session(...).start() with SessionParams(use_daily=..., data=..., daily_room_properties=...)

Your agent receives arguments (examples in docs):

• DailyRunnerArguments (room_url, token, body, session_id)
• WebSocketRunnerArguments (websocket, body, session_id)

Logging / observability (fundamentals)

• Docs note the logging/observability section is WIP, but call out:
• PIPECAT_LOG_LEVEL to control verbosity
• View logs via CLI: pipecat cloud agent logs <agent> (filter by level)
• Session CPU/memory metrics available via dashboard and pipecat cloud agent sessions --id <session-id>
• Use loguru so logs are associated to session IDs.

Managed API keys

• A deployment can enable “managed keys” (enable_managed_keys=true).
• Docs show usage pattern where you pass the magic string PIPECATCLOUD as the API key (service-specific), and Pipecat Cloud supplies the managed key at runtime when no local env var is set.

Error codes (selected)

Docs list error codes you should handle in automation and UIs:

• PCC-1002: start without public API key
• PCC-1004: billing not configured
• PCC_INVALID_IMAGE_PLATFORM: image not linux/arm64
• PCC_IMAGE_PULL_UNAUTHORIZED: private registry auth missing/invalid
• PCC-AGENT-AT-CAPACITY: pool at capacity (429)

Deployment config file

Docs mention a pcc-deploy.toml with fields such as:

• Required: agent_name, image
• Optional: region, secret set and image credentials, agent profile
• Scaling: min_agents, max_agents

Scaling model

• min_agents: warm reserved instances (lower latency, higher cost)
• max_agents: hard cap
• Expect short “idle instance creation delay”; design for bursts.

Capacity handling

• Start API can return HTTP 429 when at capacity.
• Client code should treat capacity errors as a normal failure mode and retry/backoff or show UI.

Secrets & regions

• Secret sets and image-pull secrets are region-specific.
• Keep secrets, image credentials, and the deployed agent in the same region.

Image constraints

Docs explicitly mention building for linux/arm64.

Pipeline & frame model

Pipeline model

• A Pipeline is a sequence of processors.
• Each processor consumes and emits frames.
• A PipelineWorker runs the pipeline (previously PipelineTask, often under a WorkerRunner), and you can attach observers.

Workers and multi-agent pipelines (1.3.0)

• PipelineTask, PipelineTaskParams, and pipecat.pipeline.task are renamed toward PipelineWorker, WorkerParams, and pipecat.pipeline.worker. Old names still resolve but emit DeprecationWarning; migrate imports before they become removals.
• PipelineRunner is renamed to WorkerRunner under pipecat.workers.runner. Register workers with WorkerRunner.add_workers() before run() instead of passing a worker directly to run().
• FrameProcessor.pipeline_task is deprecated; use FrameProcessor.pipeline_worker.
• The new pipecat.workers framework lets PipelineWorker peers exchange typed messages, dispatch @job work, and coordinate handoffs, debates, sidecars, hardware controllers, UI workers, and distributed workers over Redis or PGMQ.

Typical voice processing flow

A common real-time flow described in the docs:

1. Transport receives streaming audio and emits audio frames 2. STT processor emits transcript/text frames 3. Context processor aggregates text + history into the LLM input 4. LLM processor emits streaming response text frames 5. TTS processor converts response text frames into audio frames 6. Transport streams output audio frames back to the user

Key insight: these steps can overlap. While later parts of the LLM response are still generating, earlier parts can already be synthesized and played.

Frames

Frames represent “what is happening right now” in the conversation:

• Audio/video chunks
• Partial and final transcriptions
• LLM context updates
• LLM run requests
• Client/server RTVI messages
• Settings updates (e.g., STT settings)

Frame classes and ordering

The Learn guide distinguishes frame “kinds” that affect scheduling:

• System-like frames: handled immediately (useful for interruptions, errors, speech start/stop events).
• Queued frames: processed in guaranteed order (typical audio/text/control boundaries).

Practical consequence: you can enqueue “speak then stop” and rely on the order being respected.

The docs mention example frame types such as:

• LLMContextFrame, LLMRunFrame
• RTVIClientMessageFrame, RTVIServerResponseFrame
• STTUpdateSettingsFrame

Context + aggregation

Common pattern:

• Maintain an LLMContext (system/user/assistant messages).
• Use aggregators to merge streaming fragments into stable “turns”.
• Separate aggregation for user input vs assistant output is a common design.

VAD / turn detection

The docs highlight:

• VAD analyzers (e.g., Silero) to decide “is the user speaking”.
• Smart turn analyzers for better end-of-turn decisions.

Practical checklist

• Decide your frame boundary (token, sentence, turn) for downstream TTS.
• If your TTS supports streaming, emit smaller chunks to reduce latency.
• Keep processors single-purpose and composable.

Parallel pipelines (branching)

Use a parallel/branching pattern when multiple processors need the same upstream frames (e.g., multi-language TTS, extra recording, side-channel analytics). Each branch receives the frames and can filter/gate them.

Execution building blocks

• PipelineWorker: wraps a pipeline plus execution params (sample rates, metrics flags) and observers. Older docs may call this PipelineTask.
• WorkerRunner: runs workers and can optionally handle OS signals for graceful shutdown. Older docs may call this PipelineRunner.
• Observers: monitor protocol events and custom metrics (useful for debugging and production visibility).

For exact parameter names (metrics, heartbeats, idle timeout), see: references/server-pipeline-apis.md.

Note: the Server API docs mark an older pipeline-level interruption flag as deprecated; prefer configuring interruptions via user turn strategies.

Lifecycle patterns

• Stop on client disconnect by cancelling the running task.
• Expect a lifecycle: starting → running → stopping → stopped (with cleanup).

Termination (graceful vs immediate)

Learn guide describes two main shutdown modes:

• Graceful: enqueue an “end” control frame so pending output can finish (use when the bot should say goodbye).
• Immediate: cancel the task to force fast shutdown and discard pending frames (use on disconnects or fatal errors).

If you need to end the task from inside the pipeline (e.g., from a tool/function handler), use the task-level termination signal and push it upstream so the pipeline source can terminate the whole chain correctly.

Idle detection

Docs mention built-in idle detection that can auto-cancel a task after a timeout to avoid leaked sessions.

Gotcha

Custom processors must propagate frames. If a processor forgets to push frames onward, termination frames may never reach the end of the pipeline and shutdown can hang.

Server APIs: pipeline/worker/params

This reference summarizes the Server API pages for operating pipelines.

WorkerParams / PipelineParams (worker-wide knobs)

1.3.0 renames PipelineTaskParams toward WorkerParams; older docs and examples may still say PipelineParams / task params. Treat the old names as migration aliases that emit deprecation warnings.

Highlights from the reference:

• Audio:
• audio_in_sample_rate (input)
• audio_out_sample_rate (output)

Setting these at the pipeline/task level helps keep all services consistent.

• Metrics:
• enable_metrics
• enable_usage_metrics
• report_only_initial_ttfb
• send_initial_empty_metrics

• Heartbeats:
• enable_heartbeats
• heartbeats_period_secs

• start_metadata: arbitrary serializable metadata attached to the start frame.

• allow_interruptions: deprecated (use user turn strategies / enable_interruptions on start strategies instead).

PipelineWorker (execution + lifecycle)

The Server API describes PipelineWorker (previously PipelineTask) as the center of execution:

• Queue work:
• queue_frame() / queue_frames()
• Stop and cancel:
• stop_when_done() (graceful end after queued frames)
• cancel() (immediate shutdown)
• has_finished()

Idle detection settings (task-level)

The task supports:

• idle_timeout_secs (disable by setting to None)
• idle_timeout_frames (what counts as “activity”)
• cancel_on_idle_timeout (if true, auto-cancel after handler runs)

Event handlers

Docs list events such as:

• on_pipeline_started / on_pipeline_finished
• on_pipeline_error (fatal errors may cancel after handler)
• on_idle_timeout
• reached-upstream / reached-downstream events for registered frame types

This is useful for cleanup, logging, instrumentation, and debugging frame flow.

Pipeline idle detection (semantics)

Server docs describe idle detection as a safeguard against leaked sessions:

• Activity is determined by a set of frame types.
• Timer resets when an activity frame occurs.
• You can choose to auto-cancel or handle it yourself (e.g., speak a prompt then end gracefully).

Heartbeats (stall detection)

Server docs describe periodic HeartbeatFrames traversing the entire pipeline:

• Enable via PipelineParams.
• Warnings are logged when heartbeats stop returning within a threshold, indicating a stall/blockage.

ParallelPipeline (branching)

Server docs describe ParallelPipeline as:

• multiple branches that each receive the same downstream frames
• branch results merged back into a single stream
• system frames (start/end) synchronized across branches

Common patterns:

• multi-agent branching
• redundancy/failover paths
• cross-branch communication via producer/consumer style processors

Practical checklist

• Turn on metrics/usage metrics early in development to spot latency regressions.
• Use heartbeats when you suspect stalls or processors blocking.
• Keep idle detection enabled unless you have a managed external lifecycle.
• Prefer turn strategies for interruption behavior (do not rely on deprecated flags).

Server Utilities: runner (development runner + transport utils)

What a runner is

Server docs describe a “bot runner” as an HTTP gateway that:

• creates transport sessions (rooms/tokens)
• spawns a bot process per user session
• passes connection details to the bot
• manages lifecycle and cleanup

Development runner: why it matters

• Lets you run the same bot logic across multiple transports.
• Handles setup for WebRTC (local), Daily rooms/tokens, and telephony webhooks.
• Encourages a pattern: keep core bot logic transport-agnostic, create the transport in the entry point based on runner args.
• 1.0.0 exports a module-level FastAPI app from the development runner, so you can register custom routes/middleware before calling main().

Installation (extras)

Runner support is provided via the pipecat-ai[runner] extra. See: references/installation.md.

CLI patterns

Docs show a single entry point you run with transport selection flags:

• WebRTC local: -t webrtc
• Daily rooms: -t daily
• Telephony: -t twilio|telnyx|plivo|exotel (requires a public proxy hostname)

1.3.0 runner update:

• Omitting -t now enables all supported transports instead of defaulting to WebRTC only.
• POST /start accepts a transport field to select WebRTC, Daily, telephony, or plain WebSocket per request.
• Plain WebSocket clients can connect through /ws-client, which is useful for browser apps using protobuf framing or non-telephony WebSocket clients.
• GET /status reports which transports the running instance accepts and helps client bootstraps choose the correct path.
• The Daily browser redirect route moved from GET / to GET /daily.

The runner also documents extra switches for direct Daily testing and dial-in webhook handling.

Runner arguments (what your bot receives)

The runner passes transport-specific data to the bot entry point, e.g.:

• Daily room URL and token
• WebRTC connection object
• WebSocket stream for telephony

1.2.0 note:

• Runner arguments now carry a session_id consistently across local development flows, Daily /start, and dial-in style entry points. Reuse that id for logs, traces, and cross-service correlation instead of minting a second session identifier in user code.

Implementation note: some signal-handling args are development-only and not available on Pipecat Cloud.

Built-in endpoints and RTVI /start

Docs mention an RTVI-compatible POST /start endpoint for Daily flows that:

• creates room + token
• spawns a bot instance with request body data
• returns connection info to the client

Use this pattern to keep provider secrets on the server.

Security note from the 1.2.0 line: if you expose file-download helpers around the development runner, validate resolved paths against the allowed base directory; the upstream runner patched a traversal issue in its /files/{filename:path} endpoint.

Transport utilities (for custom runners)

Docs describe utilities for advanced setups:

• create_transport(runner_args, transport_params): build the correct transport without manual conditionals
• parse_telephony_websocket(websocket): detect provider from initial messages and extract call/session data
• daily/livekit configure() helpers to create rooms and tokens
• helper functions to get transport-specific client ids and optionally capture participant camera/screen when supported

Practical checklist

• Start with the development runner, then build a custom runner only if you need custom auth/endpoints.
• Treat telephony/provider credentials as required configuration; validate them explicitly.
• Use lazy imports inside transport-specific branches to keep optional dependencies truly optional.

Server APIs: supported services (how to think about integrations)

Service categories

Server docs organize integrations into categories such as:

• Transports (audio/video exchange)
• Serializers (frame ↔ media stream conversion for websockets)
• Speech-to-text (STT)
• Large language models (LLM)
• Text-to-speech (TTS)
• Speech-to-speech (multimodal realtime models)
• Image generation
• Video/avatar
• Memory
• Vision
• Analytics & monitoring

Setup pattern: optional dependencies via extras

Integrations are typically installed via provider-specific extras (e.g. pipecat-ai[openai]). See: references/installation.md.

This implies:

• base install may be minimal
• each provider integration can pull its own dependency set

Practical checklist

• Decide your transport first (WebRTC vs telephony websocket) because it shapes latency and reliability.
• Then pick STT/LLM/TTS based on whether you need streaming + timestamps, and how you will handle interruptions.
• Keep dependencies explicit by using the documented extras rather than ad-hoc installs.

Links

• Server API reference: https://reference-server.pipecat.ai/

Session initialization (runner, bot, client)

This page adds practical guidance for how users and bots connect before any audio can flow.

Roles

• Runner: an HTTP server (docs use FastAPI) that accepts connection/start requests and coordinates session setup.
• Bot: your Pipecat pipeline code, typically started as a separate server-side process.
• Client: the user app (web/mobile) that captures audio and connects via a transport.

Recommendation: start with the development runner

Docs describe a built-in “development runner” that:

• Spins up the HTTP server and endpoints for you
• Manages connection setup and bot process lifecycle
• Can provide a web UI for WebRTC flows

Your bot exposes a single async entry point that receives runner arguments (including transport connection info), then you create the appropriate transport and run your pipeline.

Connection patterns (under the hood)

1) P2P WebRTC

• Runner serves a local client page.
• The browser generates a WebRTC offer.
• Runner starts the bot and passes the negotiated connection.
• Browser and bot exchange audio directly over WebRTC.

Use when: local development, direct browser-to-bot experiments.

2) Room-based WebRTC (Daily)

• Runner creates a room + token via Daily API.
• Both the client and the bot join the same room.
• A handshake event (client readiness) indicates the client can receive the first bot messages.

Use when: production deployments, richer call scenarios.

3) WebSocket (telephony)

• A telephony provider connects to your runner’s websocket/webhook.
• Runner parses provider-specific frames/messages.
• Bot starts immediately with the parsed connection data.

Use when: phone bots and PSTN/SIP integrations.

Starting the conversation: timing matters

Docs distinguish:

• Immediate start (P2P WebRTC / WebSocket): you can start the LLM run once the client is connected.
• Handshake-required start (room-based client/server): wait for a “client ready” signal, then mark the bot ready and start the first turn.

If you start too early in a room-based flow, the client may miss the beginning of the bot’s first message.

Process isolation (one bot per session)

Docs recommend per-session bot instances for:

• Resource management (CPU/memory per session)
• Failure isolation
• Cleaner teardown

When to build a custom runner

Use a custom runner when you need:

• Custom auth or endpoints
• Deeper integration with an existing backend
• Non-standard session lifecycle

Docs suggest using the development runner’s source code as a reference, since it handles real-world edge cases.

Speech input & turn detection

This page explains how Pipecat decides when the user starts/stops a “turn”, and how interruptions work.

Turn strategies: start vs end

Pipecat separates:

• Turn start detection: decide when the user started speaking
• Turn end detection: decide when the user finished the thought and expects a response

Signals can combine:

• VAD (voice activity)
• Transcription events (as fallback)
• Minimum words / gating (to avoid triggering on noise)

VAD (Voice Activity Detection)

Docs highlight a local Silero VAD analyzer (CPU-friendly). Configuration is typically provided via user aggregator params.

1.0.0 completes the move away from transport-owned VAD/turn analysis: vad_analyzer, turn_analyzer, and older transport-side mute/interruption helpers were removed.

Key tuning knobs (names from docs):

• start_secs: how long speech must continue to confirm a start
• stop_secs: how much silence is needed to confirm a stop
• confidence and min_volume: sensitivity thresholds

Rule of thumb from docs: defaults are usually good; tune only for specific audio conditions and validate with real recordings.

Turn boundary frames

Docs describe multiple layers of “speech vs turn”:

• Raw VAD speech/silence events (VAD-level start/stop)
• Higher-level turn start/stop decisions:
• UserStartedSpeakingFrame
• UserStoppedSpeakingFrame

Turn end strategies

Docs mention:

• Smart Turn (default): a turn analyzer model that decides when the user is done (better conversational feel).
• Speech timeout: simpler strategy that triggers end after a silence timeout.

1.2.0 adds a cleaner separation between “start inference now” and “finalize the turn now”:

• on_user_turn_inference_triggered fires as soon as a strategy decides the LLM can start thinking.
• LLMTurnCompletionUserTurnStopStrategy waits for UserTurnInferenceCompletedFrame before emitting the final stop event, with a timeout safety net.
• FilterIncompleteUserTurnStrategies() is the new high-level preset when you want tentative stop detection filtered through an LLM completion check.

Runtime STT updates (0.0.105)

• Runtime STTUpdateSettingsFrame updates now reconnect correctly for a wider set of STT/TTS services instead of only mutating local state.
• Deepgram Flux settings can be updated mid-stream without a reconnect.
• BaseWhisperSTTService and OpenAISTTService can optionally push empty transcripts downstream when VAD fires but no speech was actually transcribed.
• AssemblyAIConnectionParams adds vad_threshold for U3 Pro, which helps align provider-side detection with external VAD.

These changes matter when you tune speech sensitivity live or need the agent to resume speaking cleanly after a false-positive VAD event.

STT and turn updates (1.3.0)

• CartesiaTurnsSTTService supports Cartesia Streaming ASR v2 turn-based WebSocket sessions and maps turn.start, turn.update, turn.end, eager-end, and resume events into Pipecat speech/turn frames.
• SonioxSTTService.Settings.max_endpoint_delay_ms controls the maximum endpointing delay before a turn finalizes, and Soniox settings updates now reconnect gracefully instead of hard disconnecting.
• STTService.supports_ttfs lets turn-based STT services opt out of TTFS latency semantics; when false, STTMetadataFrame uses ttfs_p99_latency=0.0 without noisy warnings.
• Smart Turn v3 no longer imports transformers at module import time; cold start and memory footprint are much lower, and transformers is no longer part of the base install.

Interruptions

When interruptions are enabled (docs say default enabled), starting a user turn can:

• stop the bot from speaking
• clear pending audio/text output
• allow natural “barge-in” behavior

You can disable interruptions on the start strategy for experiences where barge-in is undesirable.

In 1.0.0, interruptions are effectively always allowed at the pipeline level. Control now lives in LLMUserAggregator strategy selection (user_turn_strategies, user_mute_strategies) instead of the older allow_interruptions, interruption_strategies, STTMuteFilter, or transport params.

Practical checklist

• Keep interruptions enabled unless you have a strong UX reason to disable them.
• Prefer Smart Turn when you want fewer awkward cutoffs / long waits.
• If your environment is noisy, adjust start_secs and min_volume conservatively and re-test.
• If you are upgrading, migrate configuration before tuning thresholds; otherwise you may be changing knobs that no longer exist.
• For half-finished utterances or barge-in heavy UX, prefer FilterIncompleteUserTurnStrategies over legacy filter_incomplete_user_turns flags.

Text to speech (TTS)

Placement

Learn guide places TTS:

• after the LLM (so it can consume streamed LLM text frames)
• before transport.output() (to produce audio frames)
• before the assistant context aggregator (so spoken text can be captured accurately)

Two common input modes

• Streamed LLM output: TTS aggregates streaming LLM tokens into speakable chunks (often sentence-like), sends to the provider, and streams audio back.
• Direct speak: a dedicated “speak this text now” frame bypasses the LLM/context (useful for system prompts and immediate cues).

Typical outputs

Docs describe TTS emitting:

• raw audio frames for playback
• TTS text frames representing what was actually spoken
• boundary frames that mark speech start/stop

Word timestamps

Some providers expose word timestamps. The guide emphasizes these for:

• accurate context updates when output is interrupted
• tighter sync for captions/subtitles and other post-output processing

As of 0.0.105, word timestamp handling is effectively built into the base TTS flow rather than something you opt into with older word/audio-context subclasses.

Pipeline-level audio configuration

Prefer setting output sample rate and related audio settings at the pipeline/task level so all processors stay consistent.

Text shaping: what gets spoken

The guide outlines multiple ways to control spoken content:

• customize aggregation before TTS (e.g., group URLs/code separately)
• skip selected aggregated types (do not speak them)
• apply just-in-time text transforms for pronunciation/clarity (numbers, acronyms, URLs)

Note: transforms may affect what ends up in assistant context when context is based on spoken output.

Skipping TTS (voice ↔ text toggles)

Docs describe a skip_tts flag that can be applied:

• globally for a stretch of conversation (via an LLM configuration frame)
• per-frame for selective silencing

Useful for:

• structured metadata that should be processed but not spoken
• text-only replies
• audio-less testing pipelines

Dynamic updates

The guide shows a settings-update frame to change TTS parameters mid-conversation.

Recent service notes (1.3.0 line):

• Rime RimeTTSService / RimeHttpTTSService default to the coda model instead of arcana; set model="arcana" explicitly to preserve old behavior.
• Rime coda ignores temperature, top_p, and repetition_penalty, while timeScaleFactor controls playback speed for arcana and coda.
• Gradium defaults to voice _6Aslh2DxfmnRLmP.
• Azure TTS completion now waits for the word-boundary queue so the final word is observed before TTSStoppedFrame.
• Skipped TTS frames keep their order until previous spoken frames finish, and TTSTextFrame.raw_text preserves original LLM text structure when word timestamps are enabled.

Recent service notes (1.0.0 line):

• MistralTTSService adds SSE-based streaming TTS with automatic resampling.
• ElevenLabs services now support pcm_32000 / pcm_48000 and an enable_logging=False zero-retention mode.

Audio context changes (0.0.105)

• Audio context management now lives in TTSService rather than AudioContextTTSService.
• WebSocket TTS providers now inherit from WebsocketTTSService directly.
• AudioContextTTSService, AudioContextWordTTSService, WordTTSService, WebsocketWordTTSService, and InterruptibleWordTTSService are deprecated.
• supports_word_timestamps was removed from TTSService.__init__(); do not pass it from custom subclasses anymore.

If you maintain custom TTS classes, update inheritance and constructor calls before upgrading.

Concurrent audio contexts (Cartesia, 0.0.105)

CartesiaTTSService can synthesize the next sentence while the previous one is still playing by disabling frame-processing pauses and routing each sentence through its own audio context queue. Use this when you want lower perceived latency without waiting for the prior sentence to finish playback.

Practical checklist

• Use WebSocket TTS providers when latency is critical.
• Capture spoken text (not just LLM text) in context for correctness under interruptions.
• Decide upfront how you will handle URLs/code/structured output so the bot doesn’t read garbage aloud.
• If you still call TTSService.say(), migrate to pushing TTSSpeakFrame into the pipeline.

Transports

Transports handle connectivity, media IO, and session state.

Pipeline integration

Transports typically expose two processors:

• transport.input() to inject user media frames into the pipeline
• transport.output() to send bot media frames back to the user

You do not have to put transport.output() as the final processor. Placing processors after output enables tightly synchronized work (recording, subtitles, timing-aligned context updates).

When to choose which

• WebRTC: best for production voice UX (latency, jitter handling, audio quality).
• WebSocket: good for server↔server, prototypes, and simpler integration.
• Direct provider realtime: connects the transport directly to a provider’s realtime endpoint (useful for fast prototyping).

Examples mentioned in docs

• DailyTransport (WebRTC)
• “Production-ready” WebRTC transport.
• Often connects using { url, token } returned from a start endpoint.

• SmallWebRTCTransport
• Lightweight peer-to-peer WebRTC.
• Typically used with a matching server implementation.

• WebSocketTransport
• WebSocket-based transport.
• Can use different serializers (the docs mention Protobuf and Twilio-oriented serialization).

• OpenAIRealtimeWebRTCTransport
• WebRTC directly to OpenAI Realtime (construct with an API key + session config).

• GeminiLiveWebsocketTransport
• WebSocket transport to Gemini Live / multimodal realtime (API key + generation config).

The Learn guide also calls out additional transports you may encounter:

• LiveKit-based WebRTC transports
• Vonage Video Connector transport for real-time Vonage WebRTC sessions
• FastAPI-oriented websocket transports for telephony/webhooks
• Video/avatar generation transports (e.g., HeyGen, Tavus)

Common configuration surface

The Learn guide describes a shared TransportParams structure with flags for:

• audio in/out enablement
• video in/out enablement
• video output sizing / bitrate / framerate

Transport-specific parameter types may extend this base.

Migration note for 1.0.0: deprecated transport-level VAD/turn parameters are gone, and the older camera_* compatibility params were removed in favor of the video_in_* / video_out_* names.

Daily transport updates (0.0.105)

• DailyParams can publish custom video tracks via video_out_destinations, mirroring the existing multi-destination audio model.
• Daily recording supports a cloud-audio-only mode when you need cloud recording without storing video.

Use these options when the bot needs to publish more than one visual stream or when compliance/cost requirements make audio-only recording preferable.

Telephony over WebSocket

Telephony providers typically stream media over WebSockets using provider-specific framing/serialization.

Docs mention provider serializers for:

• Twilio
• Telnyx
• Plivo
• Exotel

Implementation tip: treat provider credentials as required configuration and fail fast if they are missing (avoid empty-string fallbacks).

Multi-transport bots (selection by runner args)

The Learn guide shows a practical pattern:

• The bot entry point receives runner arguments that describe the connection type.
• Construct the correct transport implementation based on those args.
• Then run the same pipeline logic regardless of transport.

Operational gotchas

• Model the transport as a state machine; do not start streaming audio until the bot is “ready”.
• Buffer local audio until the bot is ready if the transport supports it.
• Prefer a server “start” endpoint that creates the transport session and returns connection params to the client.
• Do not keep transport-specific workarounds for interruption/VAD behavior if you are upgrading; that policy now belongs with user-turn aggregation.

WebRTC vs WebSocket (rules of thumb)

• Prefer WebRTC for client applications: better resilience, built-in audio processing, and quality telemetry.
• Prefer WebSocket for telephony and server-to-server; expect to implement more reconnection/timestamping/observability yourself.
• In 1.3.0, the development runner can expose a plain WebSocket /ws-client endpoint alongside WebRTC/Daily/telephony. Use it for non-telephony clients, but keep production readiness checks and auth around the custom runner surface.

Troubleshooting

First steps

• Check agent logs for the specific session.
• Confirm your transport credentials and room/session creation.
• Verify secrets exist in the same region as the agent.

Common Pipecat Cloud errors (from docs)

• Missing/invalid API key
• Billing not set
• Image not found / unauthorized to pull image / rate limited pulls
• Image too large
• Wrong image platform (docs say linux/arm64)
• Agent at capacity (HTTP 429)

Logging

Docs mention an environment variable to control log verbosity (levels like TRACE/DEBUG/INFO/WARNING/ERROR/NONE).

Pipelines that don’t terminate

Learn docs warn about a common bug:

• A custom frame processor that does not propagate frames can block termination frames.

If shutdown hangs, verify that every custom processor pushes frames onward (including end/cancel signals).

Also note the framework can use idle timeouts as a safety net; if your sessions are terminating unexpectedly, double-check idle timeout configuration.

Capacity & cold starts

• If min_agents = 0, be ready for cold starts.
• Use min_agents to keep warm capacity for low-latency UX.
• Treat 429 “at capacity” as a normal operational response.