Omnivoice

OmniVoice

Overview

Generate audio locally via the OmniVoice Studio MCP server. Tools: generate_speech, list_voices, list_personalities, list_languages, check_health. Resources: voice://{id}, history://recent.

Prerequisites — Backend Must Be Running

The MCP tools all hit $OMNIVOICE_API_URL (default http://localhost:3900). If the backend is down, every tool returns a connection error. Install + boot:

git clone https://github.com/debpalash/OmniVoice-Studio.git "$OMNIVOICE_HOME"
cd "$OMNIVOICE_HOME"
uv sync
VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]'

Then:

scripts/check-health.sh        # exit 0 if up
scripts/start-backend.sh       # boot in background (MPS/CUDA auto-detected)

First synthesis call lazy-downloads the k2-fsa/OmniVoice model (~2.4 GB) from HuggingFace — cached on subsequent boots.

Task Index — Pick the Right Tool

For non-trivial decisions (which engine to use, when to pick OmniVoice over kokoro / Edge TTS / ElevenLabs), see references/engines-comparison.md.

For MCP wiring details, backend lifecycle, troubleshooting, and a clean teardown, see references/mcp-setup.md.

Common Workflows

1. One-shot narration with the demo voice

# As called through the MCP client (your agent will do this for you):
result = generate_speech(
    text="Hello — this is OmniVoice generating speech locally.",
    profile_id="demo0001",
    language="English",
    steps=16,                   # 8 = fast/draft · 16 = balanced · 32 = quality
)
# result is JSON with audio_id, generation_time_s, audio_duration_s, format, wav_base64

Benchmark: 4.2 s of audio in ~24 s server-side on Apple Silicon MPS at 16 diffusion steps.

2. Save the WAV to disk and play

Tool returns base64 PCM WAV (16-bit, mono, 24 kHz). Decode + write:

import base64, json
payload = json.loads(result_text)            # parse JSON the tool returns
open("out.wav","wb").write(base64.b64decode(payload["wav_base64"]))

On macOS: afplay out.wav. Convert to MP3 with ffmpeg -i out.wav -codec:a libmp3lame -b:a 128k out.mp3.

3. Voice clone — end-to-end recipe

Cloning needs a 3-10 second reference clip the model will use as a speaker embedding. The MCP server does NOT expose profile creation — it only reads existing profiles. Two paths to create one:

Path A — bundled helper (macOS, recommended for fresh clones):

scripts/record-reference.sh ~/Downloads/my-ref.wav 12 1
# args: output_path raw_duration_sec mic_index
# Default mic_index=1 (MacBook built-in); list devices via:
#   ffmpeg -f avfoundation -list_devices true -i ""

The script gives audible countdown + start/stop cues via macOS say + /System/Library/Sounds/Ping.aiff so the user knows when to speak (terminal stdout is buffered — text "speak now" prompts arrive too late). It records a longer raw window, then trims to ~10 seconds of speech via silenceremove + atrim, plays back for verification, and prints the next-step curl command.

Path B — manual:

# 1. Record (mono, 24 kHz native — matches model's internal rate)
ffmpeg -f avfoundation -i ":1" -t 12 -ac 1 -ar 24000 raw.wav

# 2. Trim leading silence + take first 10 sec of speech
ffmpeg -i raw.wav \
  -af "silenceremove=start_periods=1:start_silence=0.05:start_threshold=-40dB,atrim=end=10" \
  -ac 1 -ar 24000 ref.wav

# 3. Verify
ffmpeg -i ref.wav -af volumedetect -f null - 2>&1 | grep volume   # max should be > -20 dB
afplay ref.wav

POST to /profiles (multipart/form-data — required fields: name, ref_audio):

curl -X POST http://127.0.0.1:3900/profiles \
  -F "name=carlos-clone" \
  -F "ref_audio=@ref.wav" \
  -F "ref_text=The exact text spoken in the clip" \
  -F "language=English" \
  | python3 -m json.tool
# returns { "id": "abc12345", "name": "carlos-clone" }

Once created, pass profile_id to generate_speech (via MCP) or directly via POST /generate. Profiles persist in SQLite + reference-audio files at ~/Library/Application Support/OmniVoice/voices/<id>.<ext> (the backend preserves the uploaded extension — .wav if you uploaded a WAV, .mp3 if MP3, etc.). State persists across backend restarts.

Reference clip tips that materially affect quality:

4. Voice design (no reference clip)

Skip profile_id; provide an instruct string describing the desired voice:

generate_speech(
    text="Welcome to the future of agentic systems.",
    instruct="warm middle-aged female narrator, calm authoritative pace, documentary style",
)

Get pre-made instructs via list_personalities and copy the one matching the brief (narrator, casual, news-anchor, etc.).

5. Video dubbing (web UI only)

The MCP server does not expose the dubbing endpoint. The full transcribe → translate → re-voice → mux pipeline lives behind the desktop UI (bun run desktop in $OMNIVOICE_HOME) and the /dub/* REST routes. When the user asks to dub a video, point them to the UI; surface this skill only for the synthesis primitives above.

When NOT to use OmniVoice

• Fast English-only narration on weak hardware → kokoro-tts is ~10× smaller and 2× realtime on CPU (see references/engines-comparison.md)
• Lowest-friction one-off TTS → Edge TTS needs no install or backend
• Highest possible quality regardless of cost → ElevenLabs still wins on English narration polish; OmniVoice ties or wins on multilingual + cloning
• Real-time streaming dictation → use the OmniVoice desktop widget (⌘+⇧+Space), not the MCP server

Resources

• references/engines-comparison.md — Decision tree across OmniVoice / kokoro / Voicebox / Edge TTS / ElevenLabs / cloud APIs
• references/mcp-setup.md — MCP wiring, backend lifecycle, env vars, troubleshooting
• scripts/check-health.sh — curl /health, exit 0/1
• scripts/start-backend.sh — Start uvicorn on 127.0.0.1:3900 with health probe
• scripts/stop-backend.sh — Clean shutdown via kill -TERM on the bound PID
• scripts/record-reference.sh — macOS-only: record + trim + verify a reference clip for cloning, with audible cues (say + system beeps) that bypass terminal output buffering

Backend Swagger / OpenAPI: http://127.0.0.1:3900/docs (when backend is up).

Upstream: github.com/debpalash/OmniVoice-Studio — FSL-1.1-ALv2 (free for personal/internal/non-commercial; auto-converts to Apache-2.0 two years after each release).

omnivoice-skill (DEPRECATED — migrated to broomva/skills monorepo)

Status: This repository is in a 6-month deprecation window (until 2026-11-26). The skill has migrated to the broomva/skills monorepo as a Tier-2 vendored skill, renamed to omnivoice per ecosystem norm (drops the -skill suffix).

New install command

npx skills add broomva/skills --skill omnivoice

Skill home (new)

broomva/skills/skills/omnivoice

After 2026-11-26

This repository will be archived (read-only on GitHub permanently). All updates ship from broomva/skills.

License

MIT — unchanged.

TTS Engine Selection — Decision Tree

When to pick OmniVoice vs other engines available in this workspace. Match the user's constraint to the right column.

Decision tree

Is voice cloning required?
├─ yes → OmniVoice (3-sec ref clip, zero-shot, 646 langs)
└─ no →
   Is the language non-English?
   ├─ yes → OmniVoice (646 langs) or Edge TTS (subset, cloud)
   └─ no (English) →
      Is privacy required (no cloud)?
      ├─ yes →
      │  Is GPU available?
      │  ├─ yes (CUDA/MPS) → OmniVoice (best quality) or Voicebox
      │  └─ no (CPU only) → kokoro-tts (2× realtime CPU) or OmniVoice on CPU (slow)
      └─ no (cloud OK) →
         Is cost-no-object?
         ├─ yes → ElevenLabs (best polish), then OpenAI TTS
         └─ no → Edge TTS (free, unofficial, MS Azure neural)

Full comparison

*Edge TTS is unofficial. Microsoft could block it at any time.

When OmniVoice wins decisively

1. Voice cloning — 3-sec reference clip, zero-shot, no fine-tuning. ElevenLabs is the only competitor; OmniVoice is free and local. 2. Long-tail languages — 646 supported. ElevenLabs covers 32; everything else fewer. 3. Privacy / regulatory — Nothing leaves the machine. ElevenLabs and OpenAI ship audio to their servers. 4. No-API-key constraint — Local-first. No accounts. 5. Bulk generation without metered cost — ElevenLabs bills per character. OmniVoice is free at any volume.

When OmniVoice loses

1. Lowest-friction one-off TTS — Backend install + ~3 GB model + uvicorn boot. Edge TTS or OpenAI TTS is one command. 2. Fast English narration on weak hardware — kokoro-tts is ~30 MB vs OmniVoice's 2.4 GB and runs 2× realtime on CPU. Use kokoro for blog-narration batch jobs unless you need cloning. 3. Streaming real-time TTS — OmniVoice is diffusion-based and not streaming. Use Edge TTS or cloud APIs for true streaming. 4. Apple Silicon-only specialized voices — mlx-audio ships 14 engines (Kokoro, CSM, Dia, Qwen3-TTS, etc.) that may match a specific voice better.

Composition with content pipelines

OmniVoice fits between visual asset generation and video assembly:

research → narrative → visual assets → AUDIO (OmniVoice) → video assembly → distribution

Default for blog-post audio narration:

• English, no cloning needed, fast → kokoro-tts (cheap CPU)
• English, want a specific cloned voice → OmniVoice with a saved profile
• Non-English → OmniVoice
• One-time, no install → Edge TTS

For Remotion-based video pipelines that previously required ElevenLabs, OmniVoice closes the last cloud dependency — pair it with any local image/video generator for a fully self-hosted multimedia stack.

OmniVoice MCP Setup, Lifecycle, Troubleshooting

Install

# Pick any location. The scripts in this skill default to ~/OmniVoice-Studio if
# $OMNIVOICE_HOME is unset.
export OMNIVOICE_HOME="${HOME}/OmniVoice-Studio"

git clone https://github.com/debpalash/OmniVoice-Studio.git "$OMNIVOICE_HOME"
cd "$OMNIVOICE_HOME"
uv sync                                                  # ~1.6 GB venv on darwin arm64
VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]'     # SDK not in their lockfile yet

Any non-default install location works as long as $OMNIVOICE_HOME is set in the env that launches the MCP server.

MCP Wiring

Drop into your MCP client config (Claude Desktop, Claude Code at ~/.claude.json, Cursor, OpenClaw, etc.). Replace <OMNIVOICE_HOME> with the absolute path:

{
  "mcpServers": {
    "omnivoice": {
      "type": "stdio",
      "command": "uv",
      "args": [
        "--directory", "<OMNIVOICE_HOME>",
        "run", "python", "-m", "backend.mcp_server"
      ],
      "env": { "OMNIVOICE_API_URL": "http://localhost:3900" }
    }
  }
}

Restart the MCP client. The server only starts at client launch — in-session edits do not hot-reload.

Note (mcp SDK ≥ 1.10): If you see TypeError: FastMCP.__init__() got an unexpected keyword argument 'version', your OmniVoice-Studio checkout is older than debpalash/OmniVoice-Studio#112. Either git pull once that PR lands, or apply the 3-line patch manually: replace version="…", description=(…) with instructions=(…) in backend/mcp_server.py.

Backend Lifecycle

The MCP server needs the FastAPI backend running:

# Foreground (logs in terminal)
cd "$OMNIVOICE_HOME"
uv run uvicorn main:app --app-dir backend --host 127.0.0.1 --port 3900

# Detached, via the helper script in this skill
scripts/start-backend.sh                                  # idempotent
scripts/check-health.sh                                   # exit 0/1
scripts/stop-backend.sh                                   # graceful SIGTERM

127.0.0.1 keeps the API local-only. The project's package.json defaults to 0.0.0.0 which exposes the API on all interfaces — wider than needed for personal use.

First boot runs alembic migrations on the SQLite settings DB at <data_dir>/omnivoice.db. Idempotent — safe to re-run.

First synthesis call lazy-downloads the k2-fsa/OmniVoice model (~2.4 GB) into the HuggingFace cache. Path varies by OS:

• macOS / Linux: ~/.cache/huggingface/hub/
• Windows: %LOCALAPPDATA%\OmniVoice\hf_cache (OmniVoice redirects via backend/core/config.py to keep the cache off the system drive root)

Cached on subsequent boots.

Idle Behavior

GET /system/info exposes idle_timeout_seconds: 900. After 15 min of no synthesis, the diffusion model is evicted from GPU memory but the FastAPI server stays up. Next call pays ~5-10 s warm-up.

Environment Variables

Troubleshooting

Clean teardown

scripts/stop-backend.sh                                   # graceful shutdown
# Uninstall: rm -rf "$OMNIVOICE_HOME" ~/.cache/huggingface/hub/models--k2-fsa--OmniVoice
# Remove the `omnivoice` entry from your MCP client config

User profiles + history live in the platform data dir (~/Library/Application Support/OmniVoice/ on macOS; ~/.local/share/OmniVoice/ on Linux). Preserve across reinstalls if you want to keep your saved voice profiles.