Voice Audio Engineer

Voice & Audio Engineer: Voice Synthesis, TTS & Speech Processing

Expert in voice synthesis, speech processing, and vocal production using ElevenLabs and professional audio techniques. Specializes in TTS, voice cloning, podcast production, and voice UI design.

When to Use This Skill

✅ Use for:

• Text-to-speech (TTS) generation
• Voice cloning and voice design
• Speech-to-speech voice transformation
• Podcast production and editing
• Audiobook production
• Voice UI/conversational AI audio
• Dialogue mixing and processing
• Loudness normalization (LUFS)
• Voice quality enhancement (de-essing, compression)
• Transcription and speech-to-text

❌ Do NOT use for:

• Spatial audio (HRTF, Ambisonics) → sound-engineer
• Sound effects generation → sound-engineer (ElevenLabs SFX)
• Game audio middleware (Wwise, FMOD) → sound-engineer
• Music composition/production → DAW tools
• Live concert/event audio → specialized domain

MCP Integrations

Expert vs Novice Shibboleths

Common Anti-Patterns

Anti-Pattern: Uploading Noisy Audio for Voice Cloning

What it looks like: Voice clone from phone recording with background noise, echo Why it's wrong: Clone learns the noise; output has artifacts What to do instead: Use isolate_audio first; record in quiet space; provide 1-3 min of varied speech

Anti-Pattern: Ignoring Loudness Standards

What it looks like: Podcast at -6 LUFS, then normalized by platform → crushed dynamics Why it's wrong: Each platform normalizes differently; too loud = distortion, too quiet = inaudible What to do instead: Master to -16 LUFS for podcasts; -14 LUFS for streaming; always check true peak < -1 dBTP

Anti-Pattern: TTS Without Voice Matching

What it looks like: Using default robotic voice for premium product Why it's wrong: Voice IS brand; wrong voice = wrong emotional connection What to do instead: search_voices to find matching tone; consider custom clone for brand consistency

Anti-Pattern: No De-essing on Processed Voice

What it looks like: "SSSSibilant" speech after compression and EQ boost Why it's wrong: Compression brings up sibilance; EQ boost at 3-5kHz makes it worse What to do instead: De-ess at 5-8kHz before compression; use frequency-selective compression

Anti-Pattern: Single Take, No Editing

What it looks like: Podcast with 20 "ums", breath sounds, long pauses Why it's wrong: Listeners fatigue; unprofessional; reduces engagement What to do instead: Edit out filler words; gate or manually cut breaths; tighten pacing

Evolution Timeline

Pre-2020: Robotic TTS

• Concatenative synthesis (spliced recordings)
• Obvious robotic quality
• Limited voice options

2020-2022: Neural TTS Emerges

• Tacotron, WaveNet improve naturalness
• Still detectable as synthetic
• Voice cloning requires hours of data

2023-2024: AI Voice Revolution

• ElevenLabs instant voice cloning (30 seconds)
• Near-human quality in TTS
• Real-time voice transformation
• Voice agents for customer service

2025+: Current Best Practices

• Emotional TTS (control tone, pace, emotion)
• Cross-lingual voice cloning
• Real-time voice transformation in apps
• Personalized voice agents
• Voice authentication integration

Core Concepts

ElevenLabs Voice Selection

Model comparison:

Voice parameters:

# Stability: 0-1 (lower = more expressive, higher = more consistent)
# Similarity boost: 0-1 (higher = closer to original voice)
# Style: 0-1 (higher = more exaggerated style)

# For natural speech:
stability = 0.5       # Balanced expression
similarity = 0.75     # Close to voice but natural
style = 0.0           # Neutral (increase for dramatic)

Voice Cloning Best Practices

Audio requirements:

• Duration: 1-3 minutes (more = better, diminishing returns after 3min)
• Quality: Clean, no background noise, no reverb
• Content: Varied speech (questions, statements, emotions)
• Format: WAV/MP3, 44.1kHz or higher

Cloning workflow: 1. isolate_audio to clean source material 2. voice_clone with cleaned audio 3. Test with varied prompts 4. Adjust stability/similarity for output quality

Voice Processing Chain

Standard voice chain (order matters!):

[Raw Recording]
    ↓
[High-Pass Filter @ 80Hz]  ← Remove rumble, plosives
    ↓
[De-esser @ 5-8kHz]        ← Before compression!
    ↓
[Compressor 3:1, 10ms/100ms] ← Smooth dynamics
    ↓
[EQ: +2dB @ 3kHz presence] ← Clarity boost
    ↓
[Limiter -1 dBTP]          ← Prevent clipping
    ↓
[Loudness Norm -16 LUFS]   ← Target loudness

Loudness Standards

Measurement:

• LUFS = Loudness Units Full Scale (integrated)
• True Peak = Maximum level including intersample peaks
• Always measure with K-weighting (ITU-R BS.1770)

Conversational AI Agents

ElevenLabs agent configuration:

create_agent(
    name="Support Agent",
    first_message="Hi, how can I help you today?",
    system_prompt="You are a helpful customer support agent...",
    voice_id="your_voice_id",
    language="en",
    llm="gemini-2.0-flash-001",  # Fast for conversation
    temperature=0.5,
    asr_quality="high",          # Speech recognition quality
    turn_timeout=7,              # Seconds before agent responds
    max_duration_seconds=300     # 5 minute call limit
)

Voice UI considerations:

• Use fast model (eleven_flash_v2_5) for real-time
• Keep responses concise (&lt; 30 seconds)
• Add pauses for natural conversation flow
• Handle interruptions gracefully

Quick Reference

Voice Selection Decision Tree

• Brand/professional content? → Custom clone or curated voice
• Real-time/interactive? → eleven_flash_v2_5 model
• Quality-critical? → eleven_multilingual_v2 model
• Multiple languages? → Check language support per voice

Processing Decision Tree

• Voice sounds muddy? → HPF at 80Hz, boost 3kHz
• Sibilance harsh? → De-ess at 5-8kHz
• Inconsistent volume? → Compress 3:1, then limit
• Too quiet? → Normalize to target LUFS
• Background noise? → Use isolate_audio first

Common Settings

De-esser: 5-8kHz, -6dB reduction, Q=2
Compressor: 3:1 ratio, -20dB threshold, 10ms attack, 100ms release
EQ presence: +2-3dB shelf at 3kHz
HPF: 80-100Hz, 12dB/oct
Limiter: -1 dBTP ceiling

Working With Speech Disfluencies

Cluttering vs Stuttering

ASR Challenges with Disfluent Speech

Most ASR models trained on fluent speech. Disfluencies cause:

• Word boundary detection errors
• Repetitions transcribed literally ("I I I want" vs "I want")
• Collapsed syllables missed entirely
• Timing models confused by irregular pace

Solutions & Workarounds

1. Model selection (best to worst for disfluencies):

• Whisper large-v3 - Most robust to disfluencies
• ElevenLabs speech_to_text - Good with varied speech
• Google Speech-to-Text - Decent with enhanced models
• Fast/lightweight models - Usually worst

2. Pre-processing:

# Normalize speech rate before ASR
# Use librosa to stretch irregular segments toward target rate
import librosa
y, sr = librosa.load("disfluent.wav")
y_stretched = librosa.effects.time_stretch(y, rate=0.9)  # Slow down

3. Post-processing:

• Remove duplicate words: "I I I want" → "I want"
• Filter common fillers: "um", "uh", "like", "you know"
• Use LLM to clean transcripts while preserving meaning

4. Fine-tuning Whisper (advanced):

# Fine-tune on disfluent speech dataset
# Datasets: FluencyBank, UCLASS, SEP-28k (stuttering)
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
# Fine-tune on your speech samples with corrected transcripts
# Training loop with disfluent audio → fluent transcript pairs

5. ElevenLabs voice cloning approach:

• Clone your voice from fluent segments
• Use TTS for fluent output with your voice
• Great for pre-recorded content, not live

Accessibility Considerations

• Always provide manual transcript correction option
• Consider hybrid: ASR + human review
• For voice UI: longer timeout, confirmation prompts
• Test with actual users from target population

Performance Targets

Integrates With

• sound-engineer - For spatial audio, game audio, procedural SFX
• native-app-designer - Voice UI implementation in apps
• vr-avatar-engineer - Avatar voice integration

---

For detailed implementations: See /references/implementations.md

Remember: Voice is intimate—it speaks directly to the listener's brain. Match voice to brand, process for clarity not loudness, and always respect the platform's loudness standards. With ElevenLabs, you have instant access to professional voice synthesis; use it thoughtfully.

Voice Audio Implementation Reference

Detailed code implementations for voice processing, loudness measurement, and speech analysis.

Biquad Filter (Audio EQ Cookbook)

import numpy as np
from scipy import signal

class AudioFilters:
    """Production-ready filter implementations."""

    @staticmethod
    def biquad_coefficients(filter_type: str, fc: float, fs: float,
                           Q: float = 0.707, gain_db: float = 0) -> tuple:
        """
        Calculate biquad filter coefficients.
        Uses Audio EQ Cookbook formulas (Robert Bristow-Johnson)
        """
        A = 10 ** (gain_db / 40)
        w0 = 2 * np.pi * fc / fs
        cos_w0 = np.cos(w0)
        sin_w0 = np.sin(w0)
        alpha = sin_w0 / (2 * Q)

        if filter_type == 'lowpass':
            b0 = (1 - cos_w0) / 2
            b1 = 1 - cos_w0
            b2 = (1 - cos_w0) / 2
            a0 = 1 + alpha
            a1 = -2 * cos_w0
            a2 = 1 - alpha

        elif filter_type == 'highpass':
            b0 = (1 + cos_w0) / 2
            b1 = -(1 + cos_w0)
            b2 = (1 + cos_w0) / 2
            a0 = 1 + alpha
            a1 = -2 * cos_w0
            a2 = 1 - alpha

        elif filter_type == 'peaking':
            b0 = 1 + alpha * A
            b1 = -2 * cos_w0
            b2 = 1 - alpha * A
            a0 = 1 + alpha / A
            a1 = -2 * cos_w0
            a2 = 1 - alpha / A

        elif filter_type == 'highshelf':
            b0 = A * ((A + 1) + (A - 1) * cos_w0 + 2 * np.sqrt(A) * alpha)
            b1 = -2 * A * ((A - 1) + (A + 1) * cos_w0)
            b2 = A * ((A + 1) + (A - 1) * cos_w0 - 2 * np.sqrt(A) * alpha)
            a0 = (A + 1) - (A - 1) * cos_w0 + 2 * np.sqrt(A) * alpha
            a1 = 2 * ((A - 1) - (A + 1) * cos_w0)
            a2 = (A + 1) - (A - 1) * cos_w0 - 2 * np.sqrt(A) * alpha

        # Normalize
        b = np.array([b0/a0, b1/a0, b2/a0])
        a = np.array([1, a1/a0, a2/a0])
        return b, a

LUFS Loudness Measurement

import numpy as np
from scipy import signal

def measure_lufs(audio: np.ndarray, fs: int) -> float:
    """
    Measure integrated loudness per ITU-R BS.1770-4.
    """
    # Stage 1: K-weighting filter
    b1, a1 = signal.butter(2, 1500 / (fs/2), btype='high')
    b2, a2 = signal.butter(2, 38 / (fs/2), btype='high')

    filtered = signal.lfilter(b1, a1, audio)
    filtered = signal.lfilter(b2, a2, filtered)

    # Stage 2: Mean square with gating
    block_size = int(0.4 * fs)  # 400ms blocks
    hop_size = int(0.1 * fs)    # 100ms overlap

    block_loudness = []
    for i in range(0, len(filtered) - block_size, hop_size):
        block = filtered[i:i+block_size]
        mean_square = np.mean(block ** 2)
        block_loudness.append(-0.691 + 10 * np.log10(mean_square + 1e-10))

    # Absolute threshold gate (-70 LUFS)
    gated = [l for l in block_loudness if l > -70]
    if not gated:
        return -70.0

    # Relative threshold gate (-10 LU below ungated mean)
    ungated_mean = np.mean(gated)
    relative_threshold = ungated_mean - 10
    final_gated = [l for l in gated if l > relative_threshold]

    return np.mean(final_gated) if final_gated else ungated_mean

Compressor Implementation

import numpy as np

class Compressor:
    """Production-quality dynamics compressor."""

    def __init__(self, fs: int):
        self.fs = fs
        self.envelope = 0.0

    def process(self, audio: np.ndarray,
                threshold_db: float = -20,
                ratio: float = 4.0,
                attack_ms: float = 10,
                release_ms: float = 100,
                knee_db: float = 6,
                makeup_db: float = 0) -> np.ndarray:
        """Apply compression to audio signal."""
        attack_coef = np.exp(-1 / (self.fs * attack_ms / 1000))
        release_coef = np.exp(-1 / (self.fs * release_ms / 1000))

        output = np.zeros_like(audio)

        for i in range(len(audio)):
            input_level = 20 * np.log10(abs(audio[i]) + 1e-10)

            # Envelope follower
            if input_level > self.envelope:
                self.envelope = attack_coef * self.envelope + (1 - attack_coef) * input_level
            else:
                self.envelope = release_coef * self.envelope + (1 - release_coef) * input_level

            # Gain computer with soft knee
            over_threshold = self.envelope - threshold_db

            if knee_db > 0 and over_threshold > -knee_db/2 and over_threshold < knee_db/2:
                knee_factor = (over_threshold + knee_db/2) ** 2 / (2 * knee_db)
                gain_db = -knee_factor * (1 - 1/ratio)
            elif over_threshold >= knee_db/2:
                gain_db = -(over_threshold - knee_db/2) * (1 - 1/ratio) - (knee_db/2) * (1 - 1/ratio)
            else:
                gain_db = 0

            gain_linear = 10 ** ((gain_db + makeup_db) / 20)
            output[i] = audio[i] * gain_linear

        return output

De-esser Implementation

import numpy as np
from scipy import signal

class DeEsser:
    """Frequency-selective de-esser for sibilance control."""

    def __init__(self, fs: int):
        self.fs = fs

    def process(self, audio: np.ndarray,
                frequency: float = 6000,
                threshold_db: float = -20,
                reduction_db: float = 6,
                q: float = 2.0) -> np.ndarray:
        """
        Reduce sibilance in voice recordings.

        Args:
            frequency: Center frequency for sibilance detection (5-8kHz typical)
            threshold_db: Level above which de-essing activates
            reduction_db: Maximum gain reduction
            q: Bandwidth (higher = narrower)
        """
        # Create bandpass to detect sibilance
        nyq = self.fs / 2
        low = (frequency - frequency/q) / nyq
        high = (frequency + frequency/q) / nyq
        b, a = signal.butter(2, [low, high], btype='band')

        # Detect sibilance envelope
        sibilance = signal.lfilter(b, a, audio)
        envelope = np.abs(sibilance)

        # Smooth envelope
        smooth_coef = 0.99
        smoothed = np.zeros_like(envelope)
        smoothed[0] = envelope[0]
        for i in range(1, len(envelope)):
            smoothed[i] = smooth_coef * smoothed[i-1] + (1 - smooth_coef) * envelope[i]

        # Calculate gain reduction
        threshold_linear = 10 ** (threshold_db / 20)
        reduction_linear = 10 ** (-reduction_db / 20)

        output = audio.copy()
        for i in range(len(audio)):
            if smoothed[i] > threshold_linear:
                gain = 1.0 - (1.0 - reduction_linear) * (smoothed[i] - threshold_linear) / smoothed[i]
                output[i] *= gain

        return output

Voice Activity Detection (VAD)

import numpy as np

class VoiceActivityDetector:
    """Simple energy-based VAD for voice detection."""

    def __init__(self, fs: int, frame_ms: float = 20):
        self.fs = fs
        self.frame_size = int(fs * frame_ms / 1000)

    def detect(self, audio: np.ndarray,
               energy_threshold_db: float = -40,
               min_speech_ms: float = 100) -> list[tuple[int, int]]:
        """
        Detect speech segments in audio.

        Returns: List of (start_sample, end_sample) tuples
        """
        num_frames = len(audio) // self.frame_size
        is_speech = np.zeros(num_frames, dtype=bool)

        for i in range(num_frames):
            frame = audio[i * self.frame_size:(i + 1) * self.frame_size]
            energy_db = 20 * np.log10(np.sqrt(np.mean(frame ** 2)) + 1e-10)
            is_speech[i] = energy_db > energy_threshold_db

        # Merge short gaps, remove short segments
        min_frames = int(min_speech_ms / (self.frame_size / self.fs * 1000))

        # Simple hangover
        for i in range(1, len(is_speech) - 1):
            if is_speech[i-1] and is_speech[i+1]:
                is_speech[i] = True

        # Extract segments
        segments = []
        in_segment = False
        start = 0

        for i, speech in enumerate(is_speech):
            if speech and not in_segment:
                start = i * self.frame_size
                in_segment = True
            elif not speech and in_segment:
                end = i * self.frame_size
                if (end - start) / self.fs * 1000 >= min_speech_ms:
                    segments.append((start, end))
                in_segment = False

        if in_segment:
            segments.append((start, len(audio)))

        return segments

Audio Analysis Report Generator

import numpy as np
from scipy.fft import rfft, rfftfreq

def analyze_voice_audio(audio: np.ndarray, fs: int) -> dict:
    """Comprehensive voice audio analysis."""

    # Mono for analysis
    if len(audio.shape) > 1:
        mono = np.mean(audio, axis=1)
    else:
        mono = audio

    # Level measurements
    peak_db = 20 * np.log10(np.max(np.abs(mono)) + 1e-10)
    rms_db = 20 * np.log10(np.sqrt(np.mean(mono**2)) + 1e-10)
    crest_factor = peak_db - rms_db
    lufs = measure_lufs(mono, fs)
    dc_offset = np.mean(mono)

    # Spectral analysis
    spectrum = np.abs(rfft(mono))
    freqs = rfftfreq(len(mono), 1/fs)
    spectral_centroid = np.sum(freqs * spectrum) / np.sum(spectrum)

    # Voice-specific metrics
    # Fundamental frequency estimation (simple autocorrelation)
    autocorr = np.correlate(mono[:4096], mono[:4096], mode='full')
    autocorr = autocorr[len(autocorr)//2:]
    # Find first peak after initial decay
    min_lag = int(fs / 500)  # 500Hz max
    max_lag = int(fs / 50)   # 50Hz min
    peak_lag = np.argmax(autocorr[min_lag:max_lag]) + min_lag
    f0_estimate = fs / peak_lag if peak_lag > 0 else 0

    return {
        'peak_db': peak_db,
        'rms_db': rms_db,
        'crest_factor': crest_factor,
        'lufs': lufs,
        'dc_offset': dc_offset,
        'spectral_centroid': spectral_centroid,
        'f0_estimate': f0_estimate,
        'duration_seconds': len(mono) / fs,
        'sample_rate': fs
    }

def generate_recommendations(analysis: dict) -> list[str]:
    """Generate processing recommendations from analysis."""
    recs = []

    if analysis['peak_db'] > -0.5:
        recs.append("Peaks near 0dBFS - risk of clipping; add limiter")
    if analysis['lufs'] > -14:
        recs.append("Too loud for streaming (-14 LUFS target)")
    if analysis['lufs'] < -20:
        recs.append("Consider increasing overall level")
    if analysis['crest_factor'] < 6:
        recs.append("Low crest factor - may sound over-compressed")
    if abs(analysis['dc_offset']) > 0.01:
        recs.append("DC offset detected - apply high-pass filter at 80Hz")
    if analysis['spectral_centroid'] < 1500:
        recs.append("Voice sounds muddy - consider high shelf boost at 3kHz")
    if analysis['spectral_centroid'] > 4000:
        recs.append("Voice sounds harsh - consider reducing 2-4kHz")

    return recs if recs else ["Audio looks good!"]

Loudness Standards Reference

LOUDNESS UNITS (ITU-R BS.1770)

LUFS (Loudness Units Full Scale)
├── Integrated: Average loudness over entire program
├── Short-term: 3-second sliding window
├── Momentary: 400ms sliding window
└── True Peak: Maximum sample value with intersample peaks

DELIVERY STANDARDS
├── Streaming (Spotify, Apple): -14 LUFS, -1 dBTP
├── Broadcast (EBU R128): -23 LUFS ±1, -1 dBTP
├── Broadcast (ATSC A/85): -24 LKFS ±2, -2 dBTP
├── Podcast: -16 to -19 LUFS (dialogue norm)
├── YouTube: -14 LUFS (normalized)
└── Audiobook (ACX): -18 to -23 dBFS RMS, -3 dBFS peak

LOUDNESS RANGE (LRA)
├── Classical: 15-20 LU
├── Film: 10-15 LU
├── Pop music: 5-8 LU
└── Broadcast speech: 3-6 LU

Digital Audio Theory Reference

SAMPLE RATES
├── 44.1kHz: CD standard, captures up to 22.05kHz
├── 48kHz: Video standard (cleaner for frame sync)
├── 96kHz: High-resolution, better for processing headroom
└── Why 44.1kHz? Derived from video: 44100 = 3×3×5×5×7×7×2

BIT DEPTH → DYNAMIC RANGE
├── Dynamic Range (dB) ≈ 6.02 × bits + 1.76
├── 16-bit: ~96 dB (CD quality)
├── 24-bit: ~144 dB (professional)
└── 32-bit float: ~1528 dB (effectively infinite)

DITHERING
├── Required when reducing bit depth (24→16)
├── TPDF (triangular): Standard, mathematically optimal
└── Shaped: Noise pushed above hearing range

Key References

• ITU-R BS.1770: "Algorithms to measure audio programme loudness"
• EBU R128: "Loudness normalisation and permitted maximum level"
• AES-6id: "Personal Sound Exposure"
• Bristow-Johnson, R.: "Audio EQ Cookbook" (filter formulas)
• Blauert, J. (1997): Spatial Hearing (MIT Press)