Realtime Audio Architecture

Real-Time Audio Architecture on macOS

Battle-tested patterns and anti-patterns for jitter-free audio playback on macOS Apple Silicon, learned from building the Kokoro TTS pipeline.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

Decision Framework

When building audio playback in Python on macOS, choose based on this hierarchy:

1. Write-based sd.OutputStream     ← DEFAULT CHOICE
2. Callback-based sd.OutputStream  ← Only if you need sample-level control
3. afplay subprocess               ← Only for one-shot playback of existing files
4. macOS say                       ← NEVER for production TTS

Patterns (DO)

Pattern 1: Write-Based sounddevice.OutputStream

The default choice for Python audio playback. stream.write() blocks in PortAudio's C code until the device buffer has space. No Python code runs on the audio thread, so the GIL is irrelevant.

import sounddevice as sd
import numpy as np

def open_audio_stream() -> sd.OutputStream:
    # Refresh PortAudio to discover hot-plugged devices (Bluetooth, HDMI)
    sd._terminate()
    sd._initialize()
    stream = sd.OutputStream(
        samplerate=24000,
        channels=1,
        dtype="float32",
        blocksize=2048,    # ~85ms blocks at 24kHz
        latency="high",    # large internal buffer (not live, so latency is fine)
    )
    stream.start()
    return stream

# Open per request — close after each to follow device changes
stream = open_audio_stream()

# Play audio — blocks in C code, no GIL contention
audio = np.array([...], dtype=np.float32).reshape(-1, 1)
WRITE_BLOCK = 4096  # ~170ms — responsive to stop, smooth playback
for i in range(0, len(audio), WRITE_BLOCK):
    if interrupted:
        break
    stream.write(audio[i:i + WRITE_BLOCK])

stream.close()  # close after request so next open uses current default device

Why this works:

• stream.write() calls into PortAudio's C layer → no Python on the audio thread
• PortAudio handles all buffering, timing, and device interaction internally
• GIL held by CPU-intensive work (MLX inference, numpy ops) cannot affect audio timing
• Writing in ~170ms blocks allows responsive interrupt checking
• Stream opened per request (not at startup) to follow device changes

Stop mechanism: stream.abort() immediately stops playback and unblocks write(). Reopen the stream for next playback.

Reference: write-based-stream.md

Pattern 2: Pipeline Synthesis (Synthesize N+1 While Playing N)

For chunked TTS, overlap synthesis and playback:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=1) as pool:
    ahead = pool.submit(synthesize, chunks[0])
    for i in range(len(chunks)):
        audio = ahead.result()
        if i + 1 < len(chunks):
            ahead = pool.submit(synthesize, chunks[i + 1])
        stream.write(audio)  # plays while next chunk synthesizes

Why: Synthesis takes 500-2000ms per chunk. Without pipelining, there's dead silence between chunks while waiting for synthesis. With pipelining, chunk N+1 is ready by the time chunk N finishes playing (since playback is typically longer than synthesis).

Pattern 3: Float32 PCM as Native Format

CoreAudio's native sample format is 32-bit float. Use it end-to-end:

# Synthesis output → float32 directly
audio = model.synthesize(text)
if audio.dtype != np.float32:
    audio = audio.astype(np.float32)
    if np.max(np.abs(audio)) > 2.0:  # int16 range
        audio = audio / 32768.0

Why: Avoids WAV encode/decode overhead. No temp files. No format conversion at playback time. CoreAudio receives the data in its preferred format.

Pattern 4: Boundary Fades (2ms)

Apply tiny fade-in/out at chunk boundaries to prevent click artifacts:

FADE_SAMPLES = 48  # 2ms at 24kHz

def apply_boundary_fades(audio: np.ndarray) -> np.ndarray:
    if len(audio) < FADE_SAMPLES * 2:
        return audio
    audio = audio.copy()
    audio[:FADE_SAMPLES] *= np.linspace(0, 1, FADE_SAMPLES, dtype=np.float32)
    audio[-FADE_SAMPLES:] *= np.linspace(1, 0, FADE_SAMPLES, dtype=np.float32)
    return audio

Why: Adjacent chunks may have different DC offsets or phase. A 2ms fade is inaudible but prevents the discontinuity click. Simpler and more reliable than inter-chunk crossfade.

Pattern 5: launchd QoS for Audio Processes

<!-- CORRECT: Audio process gets CPU priority -->
<key>Nice</key>
<integer>-10</integer>
<key>ProcessType</key>
<string>Adaptive</string>

Why:

• Nice: -10 gives higher CPU scheduling priority (range: -20 highest to 20 lowest)
• ProcessType: Adaptive lets macOS boost priority when the process is actively working
• launchd CAN set negative nice values for user agents (runs as root)

Pattern 6: Centralized Audio Server

One server, one speak queue, shared across all clients (BTT, Telegram bot, CLI):

BTT shortcut  →  POST /v1/audio/speak  →  [server queue]  →  synthesize  →  play
Telegram bot  →  POST /v1/audio/speak  →  [server queue]  →  synthesize  →  play

Why: Prevents audio conflicts. One lock protocol. One process to tune. Clients are thin HTTP POST callers.

Pattern 7: Audio Device Hot-Switching

PortAudio caches the device list at Pa_Initialize() time. Bluetooth devices (AirPods) connecting later are invisible. Two-layer strategy:

def _refresh_audio_devices():
    """Re-init PortAudio to discover hot-plugged devices (~1ms)."""
    sd._terminate()
    sd._initialize()

def open_audio_stream():
    """Open stream with fresh device discovery."""
    _refresh_audio_devices()  # ← discovers AirPods, new HDMI, etc.
    stream = sd.OutputStream(samplerate=24000, channels=1, dtype="float32",
                             blocksize=2048, latency="high")
    stream.start()
    return stream

def maybe_reopen_stream(stream):
    """Between-chunk check for device switching (cached devices only).

    CRITICAL: Do NOT call _refresh_audio_devices() here — it invalidates
    the active stream pointer (PaErrorCode -9988).
    """
    current_default = sd.query_devices(kind='output')['index']
    if stream.device != current_default:
        stream.close()
        return open_audio_stream()
    return stream

Two layers:

CRITICAL: Never call sd._terminate() while a stream is active — it invalidates all PortAudio stream pointers.

Reference: device-routing.md

Anti-Patterns (DON'T)

Anti-Pattern 1: Callback-Based sd.OutputStream with Python Queue

# DON'T — GIL contention causes jitter
def callback(outdata, frames, time_info, status):
    data = audio_queue.get_nowait()  # needs GIL!
    outdata[:, 0] = data

stream = sd.OutputStream(callback=callback, ...)

Why it fails: The callback runs on PortAudio's real-time audio thread, but queue.get_nowait() acquires Python's GIL to execute. When MLX synthesis (or any CPU-intensive Python work) holds the GIL — even for 10ms — the callback is delayed, causing buffer underruns → audible glitches.

The callback itself is C-level, but the Python code inside it needs the GIL. This is the fundamental trap: the sounddevice docs say "callback runs on real-time thread" which is true for the C wrapper, but your Python code inside still contends for the GIL.

Anti-Pattern 2: Subprocess Per Chunk (afplay)

# DON'T — process spawn + device acquisition per chunk = jitter
for chunk in chunks:
    wav_path = write_temp_wav(chunk)
    subprocess.run(["afplay", wav_path])  # new process each time!
    os.unlink(wav_path)

Why it fails:

1. Process spawn overhead: fork() + exec() for each chunk 2. Audio device re-acquisition: Each afplay opens the audio device, negotiates format, starts playback, then releases. Gap between chunks = silence + click. 3. File I/O overhead: Write WAV to disk, read it back. Unnecessary when you have numpy arrays in memory. 4. No pipeline: Can't synthesize next chunk while current plays (process is blocking).

When afplay IS appropriate: One-shot playback of an existing file (e.g., notification sound). Not for streaming/chunked audio.

Anti-Pattern 3: launchd Background QoS for Audio

<!-- DON'T — macOS actively throttles CPU and I/O -->
<key>Nice</key>
<integer>5</integer>
<key>ProcessType</key>
<string>Background</string>

Why it fails: ProcessType: Background tells macOS this process doesn't need timely CPU access. macOS will:

• Deprioritize CPU scheduling
• Throttle I/O bandwidth
• Potentially defer execution during high system load

For audio playback, this causes sporadic jitter that's hard to reproduce — it only happens when other processes are active.

Anti-Pattern 4: macOS say as TTS Fallback

# DON'T — quality cliff, unexpected behavior
if ! kokoro_synthesize "$text"; then
    say "$text"  # "fallback"
fi

Why it fails:

• Massive quality difference (robotic vs neural) confuses users
• say has different timing, volume, and behavior
• Creates a "works but badly" state that's harder to debug than a clean failure
• Multiple TTS engines = multiple lock protocols, process management, edge cases

Instead: Fail loudly with a notification. Let the user know the TTS server is down and how to fix it.

Anti-Pattern 5: Static Stream Opened at Startup

# DON'T — stream binds to whatever device was default at process start
stream = sd.OutputStream(samplerate=24000, channels=1, dtype="float32")
stream.start()
# ... reuse forever, never close/reopen

Why it fails:

1. Device lock-in: Stream binds to the default device at open time. Switching system default later has no effect — audio keeps going to the old device. 2. launchd boot timing: Server starts at login when MacBook Speakers may be default. External monitor / Bluetooth not yet connected. 3. PortAudio device cache: Pa_Initialize() scans devices once. Bluetooth devices connecting later are invisible — stream open to them fails silently or crashes the playback worker.

Instead: Open stream lazily per request, close after each. Call sd._terminate() + sd._initialize() before opening to refresh the device list.

Quick Diagnostic

If you hear jitter/choppiness:

1. Check process priority: ps -o pid,nice,pri,command -p $(pgrep -f tts_server)

• Nice should be ≤ 0 (not 5 or higher)

2. Check playback method: grep -c afplay ~/.local/state/launchd-logs/kokoro-tts-server/stdout.log

• Should be 0 (no afplay spawning)

3. Check for GIL contention: Look for audio callback status: output underflow in logs

• If present → switch from callback to write-based stream

4. Check launchd QoS: plutil -p ~/Library/LaunchAgents/com.terryli.kokoro-tts-server.plist | grep -E 'Nice|ProcessType'

• Should be Nice: -10, ProcessType: Adaptive

If audio goes to wrong device:

1. Check stream device in logs: grep "Audio stream opened" ~/.local/state/launchd-logs/kokoro-tts-server/stdout.log | tail -3

• Should show the expected device name

2. Check for PortAudio errors: grep "PaErrorCode\|PortAudio error" ~/.local/state/launchd-logs/kokoro-tts-server/stdout.log | tail -5

• PaErrorCode -9988 = stream pointer invalidated (device refresh while stream active)

3. Check system default: ~/.local/share/kokoro/.venv/bin/python3 -c "import sounddevice as sd; print(sd.query_devices(kind='output'))"

References

• Write-based stream implementation
• launchd QoS reference
• Pipeline synthesis pattern
• Device routing and hot-switching

See Also

• `devops-tools:macbook-desktop-mode` — Complementary skill covering USB device _resilience_ (sleep/wake recovery, uhubctl port cycling, battery longevity, pmset desktop configuration). This skill handles the application/playback layer; that one handles the system/USB layer.

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure. 2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match. 3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

Audio Device Routing and Hot-Switching

The Problem

A TTS server running as a launchd daemon opens its audio stream at startup. The stream binds to whatever output device was default at that moment (typically MacBook Pro Speakers at login). When the user later switches to an external monitor (HDMI) or Bluetooth headphones (AirPods), audio keeps going to the old device.

Two independent issues:

1. PortAudio device cache: Pa_Initialize() scans devices once. Bluetooth devices connecting later are invisible. 2. Stream device binding: sd.OutputStream binds to a specific device at creation. Changing the system default has no effect on an already-open stream.

Solution: Two-Layer Device Detection

Layer 1: Between Requests (Full Refresh)

Before each speak request, re-initialize PortAudio to discover new devices:

def _refresh_audio_devices():
    """Re-init PortAudio to pick up hot-plugged devices (~1ms)."""
    sd._terminate()
    sd._initialize()

def _open_audio_stream():
    _refresh_audio_devices()
    device = _get_output_device()  # None = system default, or env override
    stream = sd.OutputStream(
        samplerate=24000, channels=1, dtype="float32",
        blocksize=2048, latency="high", device=device,
    )
    stream.start()
    dev_info = sd.query_devices(stream.device, kind='output')
    print(f"Audio stream opened → {dev_info['name']}")
    return stream

The stream is closed after each request completes:

finally:
    if _stream is not None:
        _stream.close()
        _stream = None

This guarantees the next request opens a fresh stream on the current default device, with a fresh PortAudio device scan that sees newly-connected Bluetooth devices.

Layer 2: Between Chunks (Cached Check)

For long multi-chunk playback, check between chunks if the default device changed among already-known devices:

def _maybe_reopen_stream(stream):
    """Check if default output changed. Uses cached device list only."""
    try:
        current_default = sd.query_devices(kind='output')['index']
    except sd.PortAudioError:
        return stream
    if stream.device != current_default:
        stream.close()
        return _open_audio_stream()  # refresh + open on new device
    return stream

CRITICAL: Do NOT call _refresh_audio_devices() inside _maybe_reopen_stream(). The sd._terminate() call invalidates all active PortAudio stream pointers, causing PaErrorCode -9988 (invalid stream pointer) on the next stream.write().

What Each Layer Handles

Explicit Device Override

For cases where the system default isn't what you want:

# In tts_server.py
def _get_output_device():
    """KOKORO_AUDIO_DEVICE env: integer index or device name substring."""
    env_dev = os.environ.get("KOKORO_AUDIO_DEVICE", "").strip()
    if env_dev:
        try:
            return int(env_dev)
        except ValueError:
            return env_dev  # name substring, e.g., "AirPods"
    return None  # system default

When an explicit device is set, _maybe_reopen_stream() skips the default-device check (the user chose a specific device).

Why Not CoreAudio Directly?

Using CoreAudio's AudioObjectGetPropertyData via ctypes would give real-time device change notifications without PortAudio re-initialization. However:

1. Complexity: Requires ctypes structs, callback registration, run loop integration 2. Sufficient: The two-layer approach covers all practical scenarios 3. PortAudio compatibility: Mixing CoreAudio and PortAudio device IDs is error-prone (different numbering)

Diagnostic

# Check which device the server is using
grep "Audio stream opened" ~/.local/state/launchd-logs/kokoro-tts-server/stdout.log | tail -3

# Check for device switch events
grep "Output device changed" ~/.local/state/launchd-logs/kokoro-tts-server/stdout.log | tail -5

# Check for PortAudio errors (stream pointer invalidation)
grep "PaErrorCode" ~/.local/state/launchd-logs/kokoro-tts-server/stdout.log | tail -5

# Query current default from server's venv
~/.local/share/kokoro/.venv/bin/python3 -c "import sounddevice as sd; print(sd.query_devices(kind='output'))"

launchd QoS for Audio Processes

The Problem

macOS uses Quality of Service (QoS) classes to schedule process priority. A TTS server configured as Background gets actively throttled — macOS treats it as unimportant work that can be deferred.

QoS Classes (Apple's Hierarchy)

Correct Configuration

<key>Nice</key>
<integer>-10</integer>
<key>ProcessType</key>
<string>Adaptive</string>

Nice Value

• Range: -20 (highest priority) to 20 (lowest priority)
• 0 = normal user process
• -10 = elevated priority (appropriate for audio)
• -20 = maximum priority (usually reserved for system processes)
• launchd user agents CAN use negative values (launchd runs as root)

ProcessType: Adaptive

Adaptive is ideal for audio because:

1. High priority when active: When the process is doing work (synthesizing, playing audio), macOS boosts it to User Initiated QoS 2. Low priority when idle: When waiting for requests, drops to Utility QoS — doesn't waste resources 3. No UI impact: Unlike Interactive, doesn't affect macOS responsiveness metrics

Symptoms of Wrong QoS

• Jitter that only appears during system load (other apps active)
• Inconsistent synthesis times (same text takes 500ms sometimes, 2000ms other times)
• Audio that sounds fine in isolation but glitches during normal use
• Hard to reproduce in testing (test conditions usually have low system load)

Verification

# Check process nice value
ps -o pid,nice,pri,command -p $(pgrep -f tts_server.py)

# Check plist settings
plutil -p ~/Library/LaunchAgents/com.terryli.kokoro-tts-server.plist | grep -E 'Nice|ProcessType'

# Expected output:
#   "Nice" => -10
#   "ProcessType" => "Adaptive"

SoftResourceLimits

The memory limit in the plist should accommodate model loading:

<key>SoftResourceLimits</key>
<dict>
    <key>MemoryLimit</key>
    <integer>4294967296</integer>  <!-- 4GB — Kokoro-82M needs ~150MB, but MLX allocates more -->
</dict>

Pipeline Synthesis for Gapless TTS

The Gap Problem

Without pipelining, chunked TTS has silence between paragraphs:

[synth 700ms][play 10s][synth 700ms][play 8s][synth 600ms][play 9s]
                       ↑ 700ms gap!          ↑ 600ms gap!

Pipeline Solution

Synthesize chunk N+1 while chunk N plays:

[synth chunk 1 700ms][play chunk 1 ───────────────── 10s ──────────────────]
                     [synth chunk 2 700ms][play chunk 2 ─────────── 8s ────]
                                          [synth chunk 3 600ms][play chunk 3]

No gaps — synthesis completes well before the current chunk finishes playing.

Implementation

from concurrent.futures import ThreadPoolExecutor, Future

def playback_worker(model, stream, speak_queue, interrupted):
    with ThreadPoolExecutor(max_workers=1, thread_name_prefix="synth-ahead") as pool:
        while True:
            item = speak_queue.get()
            if item is None:
                break

            text, voice, lang, speed = item
            interrupted.clear()
            chunks = chunk_paragraphs(text)
            n = len(chunks)

            # Submit first chunk
            ahead: Future | None = pool.submit(synthesize, model, chunks[0], voice, lang, speed)

            for i in range(n):
                if interrupted.is_set():
                    if ahead and not ahead.done():
                        ahead.cancel()
                    break

                audio, gen_ms, char_count, _ = ahead.result()
                ahead = None

                # Pipeline: start next synthesis while current plays
                if i + 1 < n and not interrupted.is_set():
                    ahead = pool.submit(synthesize, model, chunks[i + 1], voice, lang, speed)

                # Apply boundary fades and play
                audio = apply_boundary_fades(audio)
                write_audio(stream, audio, interrupted)

Why Single-Threaded Synthesis Pool

ThreadPoolExecutor(max_workers=1) — only one synthesis at a time because:

1. MLX Metal is single-device: Multiple concurrent syntheses don't parallelize on GPU 2. Memory: Each synthesis allocates GPU memory; concurrent runs could OOM 3. Simplicity: One-ahead is sufficient since playback >> synthesis time

When Pipeline Isn't Enough

If chunks are very short (< 1 second of audio), synthesis of the next chunk may not complete before the current one finishes. Solutions:

1. Batch small chunks: Merge consecutive short paragraphs before synthesis 2. Pre-synthesize buffer: Synthesize 2-3 chunks before starting playback 3. Chunk sizing: Target 100-300 chars per chunk (~4-12 seconds of audio at 24kHz)

Chunk Paragraph Strategy

The chunk_paragraphs() function splits text at paragraph boundaries (\n\n) while keeping each chunk under the model's comfortable synthesis length (~400 chars). Long paragraphs are further split at sentence boundaries.

def chunk_paragraphs(text: str, max_len: int = 400) -> list[str]:
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks = []
    current = ""
    for p in paragraphs:
        if len(current) + len(p) + 1 <= max_len:
            current = f"{current} {p}" if current else p
        else:
            if current:
                chunks.append(current)
            current = p
    if current:
        chunks.append(current)
    return chunks

Write-Based sounddevice.OutputStream

Why Write-Based Over Callback-Based

The sounddevice library offers two approaches:

1. Callback-based: You provide a Python function that PortAudio calls on its real-time thread 2. Write-based: You call stream.write(data) which blocks in C until the device buffer has space

The GIL Problem with Callbacks

Audio thread (PortAudio)     Python main thread (MLX synthesis)
─────────────────────────    ──────────────────────────────────
callback() called            model.synthesize(chunk)
  → needs GIL                  → holds GIL for 10-50ms
  → BLOCKED                    → numpy operations
  → buffer underrun!           → finally releases GIL
  → silence/glitch           callback() finally runs (too late)

Even though the callback is invoked from C, the Python code inside (queue.get_nowait()) needs the GIL. When MLX Metal inference holds the GIL (common during wrapper calls), the callback is delayed past its deadline.

Write-Based: No GIL on Audio Thread

Playback thread              Python synthesis thread
──────────────────           ──────────────────────
stream.write(block)          model.synthesize(chunk)
  → enters C code              → holds GIL
  → blocks in PortAudio        → numpy operations
  → NO GIL NEEDED              → releases GIL
  → audio flows smoothly     next chunk ready

stream.write() passes the data to PortAudio's internal buffer in C and blocks until there's space. The audio thread is managed entirely by PortAudio in C — no Python code runs on it.

Implementation

import sounddevice as sd
import numpy as np

# Opened lazily per request — closed after each to follow device changes
_stream: sd.OutputStream | None = None

def open_audio_stream() -> sd.OutputStream:
    # Refresh PortAudio to discover hot-plugged devices (Bluetooth, HDMI)
    sd._terminate()
    sd._initialize()
    stream = sd.OutputStream(
        samplerate=24000,   # Kokoro outputs 24kHz
        channels=1,
        dtype="float32",    # CoreAudio native format
        blocksize=2048,     # ~85ms — good balance
        latency="high",     # large buffer = fewer underruns
    )
    stream.start()
    return stream

def write_audio(stream: sd.OutputStream, audio: np.ndarray, interrupted) -> None:
    """Write audio in ~170ms blocks for responsive interrupt checking."""
    WRITE_BLOCK = 4096
    audio_2d = audio.reshape(-1, 1)  # write() expects (frames, channels)
    for i in range(0, len(audio_2d), WRITE_BLOCK):
        if interrupted.is_set():
            return
        stream.write(audio_2d[i:i + WRITE_BLOCK])

def stop_audio(stream: sd.OutputStream) -> None:
    """Immediately stop playback. stream.abort() unblocks write()."""
    if stream and stream.active:
        stream.abort()  # raises PortAudioError in write() — catch in caller

Tuning Parameters

Stop/Resume Lifecycle

1. Normal stop: stream.abort() → unblocks write() → PortAudioError in caller 2. Caller catches PortAudioError, checks _interrupted flag 3. Stream closed in finally block after each request 4. Next request: stream = open_audio_stream() (fresh device scan + open) 5. Stream stays open between chunks within the same speak request

Compared to afplay Subprocess