Gemini Audio

Gemini Audio API Skill

Process audio with transcription, analysis, and understanding, plus generate natural speech using Google's Gemini API. Supports up to 9.5 hours of audio per request with multiple formats.

When to Use This Skill

Use this skill when you need to:

• Transcribe audio files to text with timestamps
• Summarize audio content and extract key points
• Analyze speech, music, or environmental sounds
• Generate speech from text with controllable voice and style
• Process podcasts, interviews, meetings, or any audio content
• Understand non-speech audio (birdsong, sirens, music)

Prerequisites

API Key Setup

The skill supports both Google AI Studio and Vertex AI endpoints.

Option 1: Google AI Studio (Default)

The skill automatically detects your GEMINI_API_KEY in this order:

1. Process environment: export GEMINI_API_KEY="your-key" 2. Project root: .env 3. .claude directory: .claude/.env 4. .claude/skills directory: .claude/skills/.env 5. Skill directory: .claude/skills/gemini-audio/.env

Get your API key: Visit Google AI Studio

Create .env file with:

GEMINI_API_KEY=your_api_key_here

Option 2: Vertex AI

To use Vertex AI instead:

# Enable Vertex AI
export GEMINI_USE_VERTEX=true
export VERTEX_PROJECT_ID=your-gcp-project-id
export VERTEX_LOCATION=us-central1  # Optional, defaults to us-central1

Or in .env file:

GEMINI_USE_VERTEX=true
VERTEX_PROJECT_ID=your-gcp-project-id
VERTEX_LOCATION=us-central1

Python Setup

Install required package:

pip install google-genai

Quick Start

Audio Analysis (Transcription, Summarization)

from google import genai
import os

# API key auto-detected from environment
client = genai.Client(api_key=os.getenv('GEMINI_API_KEY'))

# Upload audio file
myfile = client.files.upload(file='podcast.mp3')

# Transcribe
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=['Generate a transcript of the speech.', myfile]
)
print(response.text)

# Summarize
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=['Summarize the key points in 5 bullets.', myfile]
)
print(response.text)

Using Helper Scripts

# Transcribe audio
python .claude/skills/gemini-audio/scripts/transcribe.py audio.mp3

# Summarize audio
python .claude/skills/gemini-audio/scripts/analyze.py audio.mp3 \
  "Summarize key points"

# Analyze specific segment (timestamps in MM:SS format)
python .claude/skills/gemini-audio/scripts/analyze.py audio.mp3 \
  "What is discussed from 02:30 to 05:15?"

# Generate speech
python .claude/skills/gemini-audio/scripts/generate-speech.py \
  "Welcome to our podcast" \
  --output welcome.wav

Audio Understanding Capabilities

Supported Formats

Audio Specifications

• Maximum length: 9.5 hours per request
• Multiple files: Unlimited count, combined max 9.5 hours
• Token rate: 32 tokens/second (1 minute = 1,920 tokens)
• Processing: Auto-downsampled to 16 Kbps mono
• File size limits:
• Inline: 20 MB max total request
• File API: 2 GB per file, 20 GB project quota
• Retention: 48 hours auto-delete

Analysis Features

• Transcription: Full text with punctuation
• Timestamps: Reference segments (MM:SS format)
• Multi-speaker: Identify different speakers
• Non-speech: Analyze music, sounds, ambient audio
• Languages: Support for multiple languages

Speech Generation (TTS)

Available TTS Models

Controllable Voice Options

• Style: Professional, casual, narrative, conversational
• Pace: Slow, normal, fast
• Tone: Friendly, serious, enthusiastic
• Accent: Natural language control

TTS Example

response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents='Generate audio: Welcome to today\'s episode, in a warm, friendly tone.'
)

# Save audio output
with open('output.wav', 'wb') as f:
    f.write(response.audio_data)

Input Methods

Method 1: File Upload (Recommended for >20MB)

# Upload and reuse
myfile = client.files.upload(file='large-audio.mp3')

# Use file multiple times
response1 = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=['Transcribe this', myfile]
)

response2 = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=['Summarize this', myfile]
)

Method 2: Inline Data (<20MB)

from google.genai import types

with open('small-audio.mp3', 'rb') as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Describe this audio',
        types.Part.from_bytes(data=audio_bytes, mime_type='audio/mp3')
    ]
)

Common Use Cases

Transcription

python scripts/transcribe.py meeting.mp3 --include-timestamps

Summary with Key Points

python scripts/analyze.py interview.wav "Extract main topics and key quotes"

Speaker Identification

python scripts/analyze.py discussion.mp3 "Identify speakers and extract dialogue"

Segment Analysis

python scripts/analyze.py podcast.mp3 "Summarize content from 10:30 to 15:45"

Non-Speech Analysis

python scripts/analyze.py ambient.wav "Identify all sounds: voices, music, ambient"

Best Practices

File Management

• Use File API for files >20MB or repeated usage
• Files auto-delete after 48 hours
• Manage quota (20 GB project limit)

Prompt Engineering

• Be specific: "Transcribe from 02:30 to 03:29"
• Use timestamps for segment analysis (MM:SS format)
• Combine tasks: "Transcribe and summarize"
• Provide context: "This is a medical interview"

Cost Optimization

• Use gemini-2.5-flash ($1/1M tokens) for most tasks
• Upgrade to gemini-2.5-pro ($3/1M tokens) for complex analysis
• Check token count: 1 min audio = 1,920 tokens

Error Handling

• Validate file format and size before upload
• Implement exponential backoff for rate limits
• Handle 48-hour file expiration

Token Costs & Pricing

Audio Input (32 tokens/second):

• 1 minute = 1,920 tokens
• 1 hour = 115,200 tokens
• 9.5 hours = 1,094,400 tokens

Model Pricing:

• Gemini 2.5 Flash: $1.00/1M input, $0.10/1M output
• Gemini 2.5 Pro: $3.00/1M input, $12.00/1M output
• Gemini 1.5 Flash: $0.70/1M input, $0.175/1M output

TTS Pricing:

• Flash TTS: $10/1M tokens
• Pro TTS: $20/1M tokens

Reference Documentation

For detailed information, see:

• references/api-reference.md - Complete API specifications
• references/code-examples.md - Comprehensive code examples
• references/tts-guide.md - Text-to-speech implementation guide
• references/best-practices.md - Advanced optimization strategies

Scripts Overview

All scripts support 3-step API key detection:

• transcribe.py: Generate transcripts with optional timestamps
• analyze.py: General audio analysis with custom prompts
• generate-speech.py: Text-to-speech generation
• manage-files.py: Upload, list, and delete audio files

Run any script with --help for detailed usage.

Resources

• Audio Understanding Docs
• Speech Generation Docs
• API Reference
• Get API Key

Gemini Audio API Reference

Complete API reference for audio understanding and speech generation with Google Gemini API.

Table of Contents

• Audio Understanding API
• Speech Generation API
• Files API
• Common Parameters
• Response Formats
• Error Handling

Audio Understanding API

Generate Content with Audio

Endpoint: POST https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent

Authentication: x-goog-api-key: YOUR_API_KEY

Request Format (File API)

{
  "contents": [{
    "parts": [
      {"text": "Transcribe this audio"},
      {
        "file_data": {
          "mime_type": "audio/mp3",
          "file_uri": "https://generativelanguage.googleapis.com/v1beta/files/abc123"
        }
      }
    ]
  }],
  "generationConfig": {
    "temperature": 0.4,
    "topP": 0.95,
    "maxOutputTokens": 8192
  }
}

Request Format (Inline Data)

{
  "contents": [{
    "parts": [
      {"text": "Describe this audio"},
      {
        "inline_data": {
          "mime_type": "audio/mp3",
          "data": "base64_encoded_audio_data"
        }
      }
    ]
  }]
}

Python SDK

from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_API_KEY")

# Method 1: Upload file
audio_file = client.files.upload(file="audio.mp3")
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=['Transcribe this audio', audio_file]
)

# Method 2: Inline data
with open('audio.mp3', 'rb') as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Describe this audio',
        types.Part.from_bytes(data=audio_bytes, mime_type='audio/mp3')
    ]
)

JavaScript SDK

import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";

const ai = new GoogleGenAI({ apiKey: "YOUR_API_KEY" });

// Method 1: Upload file
const myfile = await ai.files.upload({
  file: "audio.mp3",
  config: { mimeType: "audio/mp3" }
});

const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: [
    { text: "Transcribe this audio" },
    { fileData: { fileUri: myfile.uri, mimeType: myfile.mimeType } }
  ]
});

// Method 2: Inline data
const base64Audio = fs.readFileSync("audio.mp3", { encoding: "base64" });

const response2 = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: [
    { text: "Describe this audio" },
    { inlineData: { mimeType: "audio/mp3", data: base64Audio } }
  ]
});

Supported MIME Types

Audio Constraints

• Maximum duration: 9.5 hours (34,200 seconds) per request
• Token rate: 32 tokens per second
• Multiple files: Combined duration must not exceed 9.5 hours
• File size: 2 GB max per file (Files API), 20 MB max total (inline)
• Processing: Auto-downsampled to 16 Kbps mono

Speech Generation API

Text-to-Speech with Native Audio Model

Model: gemini-2.5-flash-native-audio-preview-09-2025

Python SDK

from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_API_KEY")

response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents='Generate audio: Welcome to our podcast, in a warm, friendly tone.',
    config=types.GenerateContentConfig(
        response_modalities=['audio']
    )
)

# Extract audio data
for part in response.candidates[0].content.parts:
    if part.inline_data:
        with open('output.wav', 'wb') as f:
            f.write(part.inline_data.data)

JavaScript SDK

const ai = new GoogleGenAI({ apiKey: "YOUR_API_KEY" });

const response = await ai.models.generateContent({
  model: "gemini-2.5-flash-native-audio-preview-09-2025",
  contents: [
    { text: "Generate audio: Welcome to our podcast, in a warm, friendly tone." }
  ],
  generationConfig: {
    responseModalities: ['audio']
  }
});

// Extract and save audio
const audioPart = response.candidates[0].content.parts[0];
if (audioPart.inlineData) {
  const buffer = Buffer.from(audioPart.inlineData.data, 'base64');
  fs.writeFileSync('output.wav', buffer);
}

Voice Control Parameters

Control voice characteristics through natural language in the prompt:

# Style examples
"Generate audio: [text], in a professional style"
"Generate audio: [text], casually and conversationally"
"Generate audio: [text], as a narrative storytelling"

# Pace examples
"Generate audio: [text], speaking slowly and clearly"
"Generate audio: [text], at a normal conversational pace"
"Generate audio: [text], quickly and energetically"

# Tone examples
"Generate audio: [text], with a friendly and warm tone"
"Generate audio: [text], seriously and authoritatively"
"Generate audio: [text], enthusiastically and upbeat"

# Combined
"Generate audio: [text], in a professional style, at a normal pace, with a friendly tone"

Files API

Upload File

Endpoint: POST https://generativelanguage.googleapis.com/upload/v1beta/files

Method: Resumable upload protocol

Python SDK

client = genai.Client(api_key="YOUR_API_KEY")

uploaded_file = client.files.upload(
    file="audio.mp3",
    display_name="My Audio File"  # Optional
)

print(f"URI: {uploaded_file.uri}")
print(f"Name: {uploaded_file.name}")
print(f"Size: {uploaded_file.size_bytes}")
print(f"MIME: {uploaded_file.mime_type}")

JavaScript SDK

const ai = new GoogleGenAI({ apiKey: "YOUR_API_KEY" });

const uploadedFile = await ai.files.upload({
  file: "audio.mp3",
  config: {
    displayName: "My Audio File"  // Optional
  }
});

console.log("URI:", uploadedFile.uri);
console.log("Name:", uploadedFile.name);

List Files

# Python
files = list(client.files.list())
for file in files:
    print(f"{file.name}: {file.display_name}")

// JavaScript
const files = await ai.files.list();
for await (const file of files) {
  console.log(`${file.name}: ${file.displayName}`);
}

Get File Metadata

# Python
file = client.files.get(name="files/abc123")
print(f"State: {file.state}")
print(f"Created: {file.create_time}")

// JavaScript
const file = await ai.files.get("files/abc123");
console.log("State:", file.state);

Delete File

# Python
client.files.delete(name="files/abc123")

// JavaScript
await ai.files.delete("files/abc123");

Common Parameters

Generation Config

from google.genai import types

config = types.GenerateContentConfig(
    temperature=0.4,          # Creativity (0.0-2.0)
    top_p=0.95,              # Nucleus sampling
    top_k=40,                # Top-k sampling
    max_output_tokens=8192,  # Max response length
    response_modalities=['text'],  # Or ['audio'] for TTS
    stop_sequences=["END"]   # Stop generation at these
)

Safety Settings

config = types.GenerateContentConfig(
    safety_settings=[
        types.SafetySetting(
            category=types.HarmCategory.HARM_CATEGORY_HARASSMENT,
            threshold=types.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE
        )
    ]
)

Response Formats

Audio Understanding Response

{
  "candidates": [{
    "content": {
      "parts": [{
        "text": "This is a transcript of the audio..."
      }],
      "role": "model"
    },
    "finishReason": "STOP",
    "safetyRatings": [...]
  }],
  "usageMetadata": {
    "promptTokenCount": 1920,
    "candidatesTokenCount": 150,
    "totalTokenCount": 2070
  }
}

Speech Generation Response

{
  "candidates": [{
    "content": {
      "parts": [{
        "inlineData": {
          "mimeType": "audio/wav",
          "data": "base64_encoded_audio..."
        }
      }],
      "role": "model"
    }
  }]
}

Error Handling

Common HTTP Status Codes

Error Response Format

{
  "error": {
    "code": 400,
    "message": "Invalid file format",
    "status": "INVALID_ARGUMENT",
    "details": [...]
  }
}

Python Error Handling

from google.api_core import exceptions

try:
    response = client.models.generate_content(...)
except exceptions.InvalidArgument as e:
    print(f"Invalid request: {e}")
except exceptions.ResourceExhausted as e:
    print(f"Rate limit exceeded: {e}")
except exceptions.NotFound as e:
    print(f"File not found: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

JavaScript Error Handling

try {
  const response = await ai.models.generateContent(...);
} catch (error) {
  if (error.status === 429) {
    console.error("Rate limit exceeded");
  } else if (error.status === 400) {
    console.error("Invalid request:", error.message);
  } else {
    console.error("Error:", error);
  }
}

Rate Limits

Rate limits vary by tier and model. Check current limits:

Python:

# Implement exponential backoff
import time

max_retries = 5
for attempt in range(max_retries):
    try:
        response = client.models.generate_content(...)
        break
    except exceptions.ResourceExhausted:
        if attempt < max_retries - 1:
            wait_time = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait_time)
        else:
            raise

Token Counting

# Python
response = client.models.count_tokens(
    model='gemini-2.5-flash',
    contents=[audio_file]
)
print(f"Total tokens: {response.total_tokens}")

// JavaScript
const countResponse = await ai.models.countTokens({
  model: "gemini-2.5-flash",
  contents: [{ fileData: { fileUri: myfile.uri } }]
});
console.log("Total tokens:", countResponse.totalTokens);

Additional Resources

• Official API Reference
• Audio Understanding Docs
• Speech Generation Docs
• Files API Docs

Gemini Audio Best Practices

Comprehensive guide for optimizing audio processing with Google Gemini API.

Table of Contents

• File Management
• Cost Optimization
• Prompt Engineering
• Performance Optimization
• Quality Guidelines
• Security Best Practices
• Error Handling

File Management

Choosing Upload Method

Use File API when:

• File size > 20 MB
• Reusing audio across multiple requests
• Processing long-form content (>5 minutes)
• Need to reference same file multiple times

Use Inline Data when:

• File size < 20 MB total request
• Single-use audio clips
• Quick one-off analysis
• Prototyping and testing

File Lifecycle Management

from google import genai
import time

client = genai.Client(api_key="YOUR_API_KEY")

# Upload with tracking
uploaded_files = []

def upload_with_tracking(file_path: str):
    file = client.files.upload(file=file_path)
    uploaded_files.append({
        'name': file.name,
        'uploaded_at': time.time()
    })
    return file

# Cleanup after use
def cleanup_old_files(max_age_hours: int = 24):
    """Delete files older than max_age_hours"""
    current_time = time.time()

    for file_info in uploaded_files[:]:
        age_hours = (current_time - file_info['uploaded_at']) / 3600

        if age_hours > max_age_hours:
            try:
                client.files.delete(name=file_info['name'])
                uploaded_files.remove(file_info)
                print(f"Deleted old file: {file_info['name']}")
            except Exception as e:
                print(f"Error deleting {file_info['name']}: {e}")

# Reuse uploaded files efficiently
def process_multiple_tasks(audio_file):
    """Use same file for multiple analyses"""

    # Upload once
    uploaded = client.files.upload(file=audio_file)

    # Transcribe
    transcript = client.models.generate_content(
        model='gemini-2.5-flash',
        contents=['Transcribe', uploaded]
    )

    # Summarize
    summary = client.models.generate_content(
        model='gemini-2.5-flash',
        contents=['Summarize key points', uploaded]
    )

    # Extract speakers
    speakers = client.models.generate_content(
        model='gemini-2.5-flash',
        contents=['Identify speakers', uploaded]
    )

    # Cleanup
    client.files.delete(name=uploaded.name)

    return {
        'transcript': transcript.text,
        'summary': summary.text,
        'speakers': speakers.text
    }

Managing Project Quota

Quota limits:

• 20 GB total per project
• 2 GB per file
• 48-hour retention

Best practices:

def check_quota_before_upload(file_size_bytes: int) -> bool:
    """Check if upload fits within quota"""

    # List current files
    files = list(client.files.list())

    # Calculate total used
    total_used = sum(f.size_bytes for f in files)

    # Check if new file fits (leave 1 GB buffer)
    quota_limit = 20 * 1024 * 1024 * 1024  # 20 GB
    buffer = 1 * 1024 * 1024 * 1024  # 1 GB

    available = quota_limit - total_used - buffer

    return file_size_bytes < available

def cleanup_if_needed(new_file_size: int):
    """Cleanup old files if quota is tight"""

    if not check_quota_before_upload(new_file_size):
        # Delete oldest files until space available
        files = sorted(
            client.files.list(),
            key=lambda f: f.create_time
        )

        for file in files:
            client.files.delete(name=file.name)
            if check_quota_before_upload(new_file_size):
                break

Cost Optimization

Model Selection Strategy

Token Optimization

# Calculate cost before processing
def estimate_cost(audio_duration_seconds: int, model: str = 'gemini-2.5-flash'):
    """Estimate processing cost"""

    # 32 tokens per second
    tokens = audio_duration_seconds * 32

    # Pricing per model (per 1M tokens)
    pricing = {
        'gemini-1.5-flash': 0.70,
        'gemini-2.5-flash': 1.00,
        'gemini-2.5-pro': 3.00
    }

    input_cost = (tokens / 1_000_000) * pricing.get(model, 1.00)

    # Assume 10% output tokens (conservative)
    output_pricing = {
        'gemini-1.5-flash': 0.175,
        'gemini-2.5-flash': 0.10,
        'gemini-2.5-pro': 12.00
    }

    output_tokens = tokens * 0.1
    output_cost = (output_tokens / 1_000_000) * output_pricing.get(model, 0.10)

    return {
        'input_tokens': tokens,
        'estimated_input_cost': input_cost,
        'estimated_output_cost': output_cost,
        'total_cost': input_cost + output_cost
    }

# Example
duration_minutes = 60
cost = estimate_cost(duration_minutes * 60, 'gemini-2.5-flash')
print(f"60-minute audio: ${cost['total_cost']:.4f}")
# Output: ~$0.1152 for 1-hour audio

Batch Processing for Cost Efficiency

def batch_process_with_caching(audio_files: list[str]):
    """Process multiple files with shared context caching"""

    # Upload all files first (reuse across tasks)
    uploaded = [client.files.upload(file=f) for f in audio_files]

    # Process all with single model instance
    results = []

    for audio_file in uploaded:
        response = client.models.generate_content(
            model='gemini-2.5-flash',
            contents=['Transcribe', audio_file]
        )
        results.append(response.text)

    # Cleanup all at once
    for file in uploaded:
        client.files.delete(name=file.name)

    return results

Reducing Output Tokens

# Inefficient: verbose output
prompt = "Transcribe this audio and provide detailed analysis with examples"

# Efficient: concise output
prompt = "Transcribe this audio. Use concise language."

# Structured output reduces tokens
prompt = """
Provide JSON output:
{
  "transcript": "text here",
  "speakers": ["A", "B"],
  "duration": "MM:SS"
}
"""

Prompt Engineering

Effective Prompts for Audio

Structure: Context + Task + Format + Constraints

# Poor prompt
"Transcribe this"

# Better prompt
"Generate a complete transcript with accurate punctuation and capitalization."

# Best prompt
"""
Generate a complete transcript of this audio with:
- Accurate punctuation and capitalization
- Speaker labels (Speaker A, B, etc.)
- Timestamps in [MM:SS] format at paragraph breaks
- Preserve filler words like "um" and "uh"
"""

Timestamp References

# Specific segment
"Transcribe from 02:30 to 05:15"

# Multiple segments
"""
Analyze these segments:
- 00:00-02:00: Introduction
- 02:00-10:00: Main discussion
- 10:00-end: Conclusion
"""

# Find specific content
"At what timestamp does the speaker mention 'artificial intelligence'?"

Multi-Task Prompts

# Combine tasks to reduce API calls
prompt = """
Perform these tasks:
1. Generate complete transcript
2. Summarize in 5 bullet points
3. Identify speakers
4. Extract action items
5. List key timestamps

Format output with clear sections.
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Structured Output Requests

# Request JSON format
prompt = """
Analyze this audio and return JSON:
{
  "transcript": "full transcript text",
  "summary": "brief summary",
  "speakers": ["Speaker 1", "Speaker 2"],
  "key_quotes": [
    {"speaker": "Speaker 1", "quote": "...", "timestamp": "MM:SS"}
  ],
  "topics": ["topic1", "topic2"],
  "sentiment": "positive|neutral|negative"
}
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

# Parse JSON response
import json
data = json.loads(response.text)

Performance Optimization

Parallel Processing

import concurrent.futures

def process_audio_file(file_path: str) -> dict:
    """Process single audio file"""
    client = genai.Client(api_key="YOUR_API_KEY")

    uploaded = client.files.upload(file=file_path)
    response = client.models.generate_content(
        model='gemini-2.5-flash',
        contents=['Transcribe', uploaded]
    )

    client.files.delete(name=uploaded.name)

    return {
        'file': file_path,
        'transcript': response.text
    }

def batch_process_parallel(file_paths: list[str], max_workers: int = 5):
    """Process multiple files in parallel"""

    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = [
            executor.submit(process_audio_file, path)
            for path in file_paths
        ]

        results = []
        for future in concurrent.futures.as_completed(futures):
            try:
                results.append(future.result())
            except Exception as e:
                print(f"Error processing file: {e}")

        return results

# Process 10 files in parallel
files = ['audio1.mp3', 'audio2.mp3', ...]
results = batch_process_parallel(files, max_workers=5)

Rate Limit Handling

import time
import random
from google.api_core import exceptions

class RateLimitedClient:
    """Gemini client with automatic rate limiting"""

    def __init__(self, api_key: str, requests_per_minute: int = 60):
        self.client = genai.Client(api_key=api_key)
        self.rpm = requests_per_minute
        self.min_interval = 60.0 / requests_per_minute
        self.last_request = 0

    def _wait_if_needed(self):
        """Enforce rate limit"""
        elapsed = time.time() - self.last_request
        if elapsed < self.min_interval:
            time.sleep(self.min_interval - elapsed)
        self.last_request = time.time()

    def generate_content_with_retry(self, **kwargs):
        """Generate content with rate limiting and retry"""
        max_retries = 5

        for attempt in range(max_retries):
            try:
                self._wait_if_needed()
                return self.client.models.generate_content(**kwargs)

            except exceptions.ResourceExhausted:
                if attempt < max_retries - 1:
                    wait = (2 ** attempt) + random.uniform(0, 1)
                    print(f"Rate limited. Waiting {wait:.2f}s...")
                    time.sleep(wait)
                else:
                    raise

# Usage
client = RateLimitedClient(api_key="YOUR_API_KEY", requests_per_minute=60)
response = client.generate_content_with_retry(
    model='gemini-2.5-flash',
    contents=['Transcribe', audio_file]
)

Chunking Long Audio

def process_long_audio(file_path: str, chunk_minutes: int = 30):
    """
    Process audio longer than 9.5 hours by chunking

    Args:
        file_path: Path to audio file
        chunk_minutes: Minutes per chunk (max 570 for 9.5 hours)
    """
    from pydub import AudioSegment

    audio = AudioSegment.from_file(file_path)
    chunk_ms = chunk_minutes * 60 * 1000

    chunks = []
    for i in range(0, len(audio), chunk_ms):
        chunk = audio[i:i + chunk_ms]
        chunk_path = f"chunk_{i//chunk_ms}.mp3"
        chunk.export(chunk_path, format="mp3")
        chunks.append(chunk_path)

    # Process each chunk
    transcripts = []
    for chunk_path in chunks:
        uploaded = client.files.upload(file=chunk_path)
        response = client.models.generate_content(
            model='gemini-2.5-flash',
            contents=['Transcribe', uploaded]
        )
        transcripts.append(response.text)

        # Cleanup
        os.remove(chunk_path)
        client.files.delete(name=uploaded.name)

    # Combine transcripts
    return "\n\n".join(transcripts)

Quality Guidelines

Audio File Preparation

Format recommendations:

• Lossless: FLAC for highest quality
• Compressed: MP3 at 128+ kbps, AAC at 96+ kbps
• Uncompressed: WAV for editing workflow

Quality checks:

from pydub import AudioSegment
import librosa

def validate_audio_quality(file_path: str) -> dict:
    """Check audio quality metrics"""

    audio = AudioSegment.from_file(file_path)

    # Load with librosa for analysis
    y, sr = librosa.load(file_path, sr=None)

    # Calculate metrics
    duration = len(audio) / 1000.0  # seconds
    channels = audio.channels
    sample_rate = audio.frame_rate
    bit_depth = audio.sample_width * 8

    # Check for silence
    silence_thresh = -50  # dB
    silence_chunks = librosa.effects.split(y, top_db=abs(silence_thresh))

    return {
        'duration': duration,
        'channels': channels,
        'sample_rate': sample_rate,
        'bit_depth': bit_depth,
        'silent_segments': len(silence_chunks),
        'passed': sample_rate >= 16000 and duration > 0
    }

Handling Poor Quality Audio

# Use higher-quality model for difficult audio
prompt = """
This audio has poor quality (background noise, mumbling).
Please transcribe carefully, marking unclear sections as [inaudible].
"""

response = client.models.generate_content(
    model='gemini-2.5-pro',  # Better for challenging audio
    contents=[prompt, audio_file]
)

Security Best Practices

API Key Management

import os
from pathlib import Path

def load_api_key_securely() -> str:
    """Load API key using secure 3-step lookup"""

    # 1. Environment variable (most secure)
    api_key = os.getenv('GEMINI_API_KEY')
    if api_key:
        return api_key

    # 2. User config file (chmod 600)
    config_file = Path.home() / '.gemini_config'
    if config_file.exists():
        # Ensure restrictive permissions
        if oct(config_file.stat().st_mode)[-3:] == '600':
            return config_file.read_text().strip()

    # 3. Project .env (add to .gitignore)
    env_file = Path('.env')
    if env_file.exists():
        for line in env_file.read_text().splitlines():
            if line.startswith('GEMINI_API_KEY='):
                return line.split('=', 1)[1].strip()

    raise ValueError("GEMINI_API_KEY not found")

Sanitizing Inputs

def sanitize_prompt(user_input: str) -> str:
    """Sanitize user input for prompts"""

    # Remove potential injection attempts
    sanitized = user_input.replace('"', '\\"')

    # Limit length
    max_length = 5000
    if len(sanitized) > max_length:
        sanitized = sanitized[:max_length]

    return sanitized

def safe_transcribe(audio_path: str, user_prompt: str):
    """Transcribe with sanitized user input"""

    safe_prompt = sanitize_prompt(user_prompt)

    response = client.models.generate_content(
        model='gemini-2.5-flash',
        contents=[safe_prompt, audio_file]
    )

    return response.text

Content Safety

from google.genai import types

# Enable safety filters
config = types.GenerateContentConfig(
    safety_settings=[
        types.SafetySetting(
            category=types.HarmCategory.HARM_CATEGORY_HARASSMENT,
            threshold=types.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE
        ),
        types.SafetySetting(
            category=types.HarmCategory.HARM_CATEGORY_HATE_SPEECH,
            threshold=types.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE
        )
    ]
)

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file],
    config=config
)

# Check if blocked
if response.prompt_feedback.block_reason:
    print(f"Content blocked: {response.prompt_feedback.block_reason}")

Error Handling

Comprehensive Error Handling

from google.api_core import exceptions
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def robust_transcribe(audio_path: str) -> dict:
    """Transcribe with comprehensive error handling"""

    result = {
        'success': False,
        'transcript': None,
        'error': None
    }

    try:
        # Validate file exists
        if not Path(audio_path).exists():
            raise FileNotFoundError(f"Audio file not found: {audio_path}")

        # Upload file
        logger.info(f"Uploading {audio_path}...")
        uploaded = client.files.upload(file=audio_path)

        # Transcribe
        logger.info("Transcribing...")
        response = client.models.generate_content(
            model='gemini-2.5-flash',
            contents=['Transcribe', uploaded]
        )

        result['success'] = True
        result['transcript'] = response.text

        # Cleanup
        client.files.delete(name=uploaded.name)

    except FileNotFoundError as e:
        result['error'] = f"File error: {e}"
        logger.error(result['error'])

    except exceptions.InvalidArgument as e:
        result['error'] = f"Invalid request: {e}"
        logger.error(result['error'])

    except exceptions.ResourceExhausted as e:
        result['error'] = f"Rate limit exceeded: {e}"
        logger.warning(result['error'])

    except exceptions.PermissionDenied as e:
        result['error'] = f"Permission denied: {e}"
        logger.error(result['error'])

    except Exception as e:
        result['error'] = f"Unexpected error: {e}"
        logger.exception("Unexpected error occurred")

    return result

Additional Resources

• See api-reference.md for complete API documentation
• See code-examples.md for implementation examples
• See tts-guide.md for speech generation best practices

Gemini Audio Code Examples

Comprehensive code examples for audio understanding and speech generation.

Table of Contents

• Audio Understanding Examples
• Speech Generation Examples
• Advanced Use Cases
• Integration Patterns

Audio Understanding Examples

Basic Transcription

Python:

from google import genai

client = genai.Client(api_key="YOUR_API_KEY")

# Upload audio
audio_file = client.files.upload(file="meeting.mp3")

# Transcribe
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Generate a complete transcript with accurate punctuation.',
        audio_file
    ]
)

print(response.text)

JavaScript:

import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "YOUR_API_KEY" });

const audioFile = await ai.files.upload({
  file: "meeting.mp3",
  config: { mimeType: "audio/mp3" }
});

const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: [
    { text: "Generate a complete transcript with accurate punctuation." },
    { fileData: { fileUri: audioFile.uri, mimeType: audioFile.mimeType } }
  ]
});

console.log(response.text);

Transcription with Timestamps

from google import genai

client = genai.Client(api_key="YOUR_API_KEY")
audio_file = client.files.upload(file="podcast.mp3")

prompt = """
Generate a detailed transcript with timestamps in [MM:SS] format.
Format each paragraph with its timestamp at the beginning.
Example:
[00:15] First speaker introduces the topic...
[02:30] Discussion of key points begins...
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

print(response.text)

Audio Summarization

# Simple summary
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Summarize the key points from this audio in 5 bullet points.',
        audio_file
    ]
)

# Structured summary
prompt = """
Analyze this audio and provide:
1. Main topic (1 sentence)
2. Key points (3-5 bullets)
3. Action items mentioned (if any)
4. Speakers/participants identified
5. Overall tone and context
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Segment Analysis with Timestamps

# Analyze specific time range
prompt = "What topics are discussed between 05:30 and 10:15? Provide quotes."

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

# Multiple segments
prompt = """
Analyze these time segments:
- 00:00-02:30: Introduction
- 02:30-10:00: Main discussion
- 10:00-12:00: Q&A

For each segment, provide:
- Key topics covered
- Important quotes
- Speaker identification
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Multi-Speaker Dialogue Extraction

prompt = """
Extract dialogue from this audio:
1. Identify each speaker (Speaker A, B, etc.)
2. Format as:
   Speaker A: "Dialogue text"
   Speaker B: "Response"
3. Include timestamps for speaker changes
4. Note tone and emotion when relevant
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Non-Speech Audio Analysis

# Identify all sounds
prompt = """
Analyze all sounds in this audio:
- Speech (what is being said)
- Music (type, instruments, mood)
- Ambient sounds (traffic, nature, etc.)
- Sound effects
Provide timestamps for each sound type.
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

# Music analysis
prompt = """
Analyze the music in this audio:
- Genre and style
- Tempo and mood
- Instruments used
- Vocal characteristics (if any)
- Similar artists or songs
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Inline Audio for Small Files

from google.genai import types

# For files under 20MB
with open('short_clip.mp3', 'rb') as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Transcribe this audio',
        types.Part.from_bytes(data=audio_bytes, mime_type='audio/mp3')
    ]
)

Multiple Audio Files

# Upload multiple files
file1 = client.files.upload(file="part1.mp3")
file2 = client.files.upload(file="part2.mp3")

# Analyze together (ensure combined duration < 9.5 hours)
prompt = "Compare these two audio clips. What are the differences?"

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, file1, file2]
)

Speech Generation Examples

Basic Text-to-Speech

from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_API_KEY")

response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents='Generate audio: Welcome to our podcast.',
    config=types.GenerateContentConfig(
        response_modalities=['audio']
    )
)

# Save audio
for part in response.candidates[0].content.parts:
    if part.inline_data:
        with open('welcome.wav', 'wb') as f:
            f.write(part.inline_data.data)

Controlled Voice Style

# Professional narration
response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents='Generate audio: In today\'s episode, we explore the future of AI, in a professional, authoritative tone.',
    config=types.GenerateContentConfig(response_modalities=['audio'])
)

# Casual conversation
response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents='Generate audio: Hey everyone, welcome back! in a casual, friendly style.',
    config=types.GenerateContentConfig(response_modalities=['audio'])
)

# Slow and clear
response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents='Generate audio: Follow these steps carefully, speaking slowly and clearly.',
    config=types.GenerateContentConfig(response_modalities=['audio'])
)

Multi-Paragraph Speech

script = """
Generate audio:
Welcome to the Daily Tech Update. I'm your host, and today we have some exciting news.

First up, recent developments in artificial intelligence have shown remarkable progress.
Researchers have achieved new breakthroughs in natural language processing.

Stay tuned for more updates throughout the week.

This has been your Daily Tech Update.

Narrate this in a professional news anchor style, with appropriate pacing and emphasis.
"""

response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents=script,
    config=types.GenerateContentConfig(response_modalities=['audio'])
)

Batch Speech Generation

# Generate multiple audio segments
segments = [
    "Welcome to chapter one: Introduction",
    "In this chapter, we will explore the basics.",
    "Chapter two begins our deep dive into advanced topics."
]

for i, text in enumerate(segments):
    response = client.models.generate_content(
        model='gemini-2.5-flash-native-audio-preview-09-2025',
        contents=f'Generate audio: {text}, in a narrative storytelling style.',
        config=types.GenerateContentConfig(response_modalities=['audio'])
    )

    # Save each segment
    for part in response.candidates[0].content.parts:
        if part.inline_data:
            with open(f'segment_{i+1}.wav', 'wb') as f:
                f.write(part.inline_data.data)

Advanced Use Cases

Meeting Transcription with Action Items

prompt = """
Analyze this meeting recording:

1. Provide complete transcript with timestamps
2. Identify all speakers
3. Extract action items in format:
   - Action: [description]
   - Owner: [speaker]
   - Deadline: [if mentioned]
4. Summarize key decisions made
5. Note any follow-up items
"""

response = client.models.generate_content(
    model='gemini-2.5-pro',  # Use Pro for complex analysis
    contents=[prompt, audio_file]
)

Podcast Editing Assistant

prompt = """
Analyze this podcast episode:

1. Identify intro (0:00-?), main content, and outro
2. Flag any long pauses or "um/uh" sections with timestamps
3. Suggest cuts or edits for better flow
4. Identify best quotable moments
5. Suggest chapter markers with titles
6. Rate audio quality and note any issues
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Interview Analysis

prompt = """
Analyze this interview:

1. Interview structure:
   - Introduction
   - Main questions and answers
   - Conclusion

2. For each question:
   - Question text with timestamp
   - Answer summary
   - Key insights or quotes

3. Interviewee insights:
   - Main themes discussed
   - Notable quotes
   - Expertise areas demonstrated

4. Overall assessment:
   - Interview quality
   - Best moments
   - Suggested highlights for promotional clips
"""

response = client.models.generate_content(
    model='gemini-2.5-pro',
    contents=[prompt, audio_file]
)

Accessibility: Audio Description Generation

# Analyze video audio, generate descriptions
prompt = """
From this audio track:
1. Transcribe all dialogue
2. Identify sound effects and their timing
3. Generate audio description script for visual scenes based on audio cues
4. Format for screen reader compatibility
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Language Learning: Pronunciation Analysis

# Student recording analysis
prompt = """
Analyze this language learning recording:

1. Transcribe what was said
2. Identify pronunciation issues
3. Compare to target pronunciation
4. Suggest specific improvements
5. Rate overall clarity (1-10)
6. Highlight words to practice
"""

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[prompt, audio_file]
)

Integration Patterns

Flask Web API

from flask import Flask, request, jsonify
from google import genai
import tempfile

app = Flask(__name__)
client = genai.Client(api_key="YOUR_API_KEY")

@app.route('/transcribe', methods=['POST'])
def transcribe():
    if 'audio' not in request.files:
        return jsonify({'error': 'No audio file'}), 400

    audio = request.files['audio']

    # Save temporarily
    with tempfile.NamedTemporaryFile(delete=False, suffix='.mp3') as tmp:
        audio.save(tmp.name)

        # Upload to Gemini
        uploaded = client.files.upload(file=tmp.name)

        # Transcribe
        response = client.models.generate_content(
            model='gemini-2.5-flash',
            contents=['Transcribe this audio', uploaded]
        )

        return jsonify({
            'transcript': response.text,
            'file_id': uploaded.name
        })

if __name__ == '__main__':
    app.run(debug=True)

Batch Processing with Progress Tracking

import os
from pathlib import Path
from tqdm import tqdm

def batch_transcribe(audio_dir: str, output_dir: str):
    """Transcribe all audio files in directory"""

    client = genai.Client(api_key=os.getenv('GEMINI_API_KEY'))
    audio_files = list(Path(audio_dir).glob('*.mp3'))

    for audio_path in tqdm(audio_files, desc="Transcribing"):
        # Upload
        uploaded = client.files.upload(file=str(audio_path))

        # Transcribe
        response = client.models.generate_content(
            model='gemini-2.5-flash',
            contents=['Generate complete transcript', uploaded]
        )

        # Save transcript
        output_path = Path(output_dir) / f"{audio_path.stem}.txt"
        with open(output_path, 'w') as f:
            f.write(response.text)

        # Cleanup
        client.files.delete(name=uploaded.name)

# Usage
batch_transcribe('audio_files/', 'transcripts/')

Streaming Response Handler

def process_audio_stream(audio_path: str, chunk_duration: int = 300):
    """
    Process long audio in chunks (for audio > 9.5 hours)

    Args:
        audio_path: Path to audio file
        chunk_duration: Duration per chunk in seconds
    """
    from pydub import AudioSegment

    audio = AudioSegment.from_file(audio_path)
    duration_ms = len(audio)
    chunk_ms = chunk_duration * 1000

    transcripts = []

    for i in range(0, duration_ms, chunk_ms):
        chunk = audio[i:i+chunk_ms]

        # Export chunk
        chunk_path = f"temp_chunk_{i}.mp3"
        chunk.export(chunk_path, format="mp3")

        # Transcribe chunk
        uploaded = client.files.upload(file=chunk_path)
        response = client.models.generate_content(
            model='gemini-2.5-flash',
            contents=[f'Transcribe from {i//1000}s:', uploaded]
        )

        transcripts.append({
            'start_time': i // 1000,
            'transcript': response.text
        })

        # Cleanup
        os.remove(chunk_path)
        client.files.delete(name=uploaded.name)

    return transcripts

Error Handling and Retry Logic

import time
import random
from google.api_core import exceptions

def transcribe_with_retry(audio_path: str, max_retries: int = 5):
    """Transcribe with exponential backoff retry"""

    client = genai.Client(api_key=os.getenv('GEMINI_API_KEY'))

    for attempt in range(max_retries):
        try:
            uploaded = client.files.upload(file=audio_path)

            response = client.models.generate_content(
                model='gemini-2.5-flash',
                contents=['Transcribe this audio', uploaded]
            )

            return response.text

        except exceptions.ResourceExhausted:
            if attempt < max_retries - 1:
                wait = (2 ** attempt) + random.uniform(0, 1)
                print(f"Rate limited. Waiting {wait:.2f}s...")
                time.sleep(wait)
            else:
                raise

        except exceptions.InvalidArgument as e:
            print(f"Invalid request: {e}")
            raise

        except Exception as e:
            print(f"Unexpected error: {e}")
            if attempt < max_retries - 1:
                time.sleep(2)
            else:
                raise

Additional Resources

• See api-reference.md for complete API documentation
• See best-practices.md for optimization strategies
• See tts-guide.md for advanced speech generation techniques

Gemini Audio Reference Documentation

This directory contains comprehensive reference documentation for the Gemini Audio skill.

Files

api-reference.md

Complete API reference including:

• Audio Understanding API endpoints and methods
• Speech Generation API
• Files API for audio uploads
• Request/response formats for Python, JavaScript, Go, and REST
• Common parameters and configurations
• Error handling and rate limits
• Token counting

code-examples.md

Comprehensive code examples including:

• Basic and advanced audio understanding
• Transcription with timestamps
• Audio summarization and analysis
• Speech generation examples
• Multi-file processing
• Integration patterns (Flask, batch processing)
• Production-ready implementations

best-practices.md

Optimization and best practices covering:

• File management strategies
• Cost optimization techniques
• Prompt engineering for audio
• Performance optimization
• Quality guidelines
• Security best practices
• Error handling patterns

tts-guide.md

Advanced text-to-speech guide including:

• Voice control techniques
• Prompt engineering for TTS
• Style, pace, tone, and emotion control
• Multi-paragraph speech generation
• Quality optimization
• Common use cases (podcasts, audiobooks, e-learning)
• Production best practices

Quick Navigation

Getting Started: 1. Start with the main SKILL.md for overview 2. Check api-reference.md for API details 3. Review code-examples.md for implementations

For Audio Analysis:

• code-examples.md - Audio Understanding Examples
• best-practices.md - Prompt Engineering section

For Speech Generation:

• tts-guide.md - Complete TTS guide
• code-examples.md - Speech Generation Examples

For Production:

• best-practices.md - All sections
• api-reference.md - Error Handling

External Resources

• Official Audio Documentation
• Speech Generation Documentation
• API Reference
• Get API Key

Gemini Text-to-Speech Guide

Advanced guide for generating natural speech using Google Gemini API.

Table of Contents

• Overview
• Voice Control Techniques
• Prompt Engineering for TTS
• Quality Optimization
• Common Use Cases
• Production Best Practices

Overview

Gemini provides controllable text-to-speech through natural language instructions embedded in the prompt.

Available Models

Basic Usage

from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_API_KEY")

response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents='Generate audio: Hello, welcome to our podcast.',
    config=types.GenerateContentConfig(
        response_modalities=['audio']
    )
)

# Save audio
for part in response.candidates[0].content.parts:
    if part.inline_data:
        with open('output.wav', 'wb') as f:
            f.write(part.inline_data.data)

Voice Control Techniques

Style Control

Control voice characteristics through natural language:

# Professional/Formal
"Generate audio: Welcome to the quarterly earnings call, in a professional, formal tone."

# Casual/Conversational
"Generate audio: Hey everyone, what's up? in a casual, friendly style."

# Narrative/Storytelling
"Generate audio: Once upon a time, in a distant land, in a narrative storytelling style."

# Instructional/Educational
"Generate audio: Today we'll learn how to..., in a clear, instructional tone."

# Enthusiastic/Energetic
"Generate audio: Welcome back to the show! in an enthusiastic, high-energy style."

# Calm/Soothing
"Generate audio: Take a deep breath and relax, in a calm, soothing voice."

Pace Control

# Slow pace
"Generate audio: Follow these steps carefully, speaking slowly and clearly."

# Normal pace
"Generate audio: This is a standard narration, at a normal conversational pace."

# Fast pace
"Generate audio: Breaking news update, speaking quickly with urgency."

# Variable pace
"""Generate audio:
Speak slowly here for emphasis.
Then speed up for this exciting part!
And return to normal pace for conclusion.
"""

Tone Control

# Friendly/Warm
"Generate audio: Thank you for joining us, with a warm, friendly tone."

# Serious/Authoritative
"Generate audio: This is a critical announcement, in a serious, authoritative voice."

# Playful/Lighthearted
"Generate audio: Let's have some fun! in a playful, lighthearted manner."

# Empathetic/Compassionate
"Generate audio: We understand this is difficult, with an empathetic, compassionate tone."

# Confident/Assertive
"Generate audio: We will succeed, speaking confidently and assertively."

Accent and Language

# Specify accent
"Generate audio: Cheerio, old chap! with a British English accent."

"Generate audio: G'day mate! with an Australian accent."

# Different languages (check model support)
"Generate audio: Bonjour, comment allez-vous? in French with native accent."

Emotion Control

# Happy/Joyful
"Generate audio: We did it! speaking happily and joyfully."

# Excited
"Generate audio: This is amazing! with excitement and enthusiasm."

# Sad/Somber
"Generate audio: We'll miss you, in a sad, somber tone."

# Angry/Frustrated
"Generate audio: This is unacceptable, with controlled frustration."

# Surprised
"Generate audio: Wow, I didn't expect that! with genuine surprise."

Prompt Engineering for TTS

Effective Prompt Structure

Template: Generate audio: [text], [style], [pace], [tone], [additional modifiers]

# Simple
"Generate audio: Hello world."

# With modifiers
"Generate audio: Hello world, in a professional tone."

# Multiple modifiers
"Generate audio: Hello world, in a professional tone, speaking clearly at normal pace."

# Complex instruction
"""Generate audio:
Welcome to our podcast. Today we have an exciting episode.
Speak professionally with a warm, welcoming tone at a moderate pace.
Emphasize the word 'exciting' with enthusiasm.
"""

Multi-Paragraph Speech

script = """
Generate audio:

[Intro - warm and welcoming]
Welcome to Tech Talks, your weekly dose of technology news.
I'm your host, and today we have an incredible lineup.

[Main content - professional and informative]
First up, let's discuss the latest developments in artificial intelligence.
Recent breakthroughs have shown remarkable progress in natural language processing.

[Conclusion - friendly and upbeat]
That's all for today's episode. Thanks for listening!
Join us next week for more tech insights.

Narrate this as a professional podcast host, varying tone and energy
appropriately for each section. Use natural pacing with brief pauses
between sections.
"""

response = client.models.generate_content(
    model='gemini-2.5-flash-native-audio-preview-09-2025',
    contents=script,
    config=types.GenerateContentConfig(response_modalities=['audio'])
)

Emphasis and Pronunciation

# Emphasis
"""Generate audio:
This is VERY important. Please emphasize the word 'VERY'.
"""

# Pronunciation guidance
"""Generate audio:
The company name is Anthropic (an-THROP-ik).
Pronounce it as specified in the phonetic guide.
"""

# Acronyms
"""Generate audio:
We'll discuss AI (spell out as 'A-I'), API (spell as 'A-P-I'),
and NASA (pronounce as 'NASA', not spelled out).
"""

Pauses and Timing

# Natural pauses
"""Generate audio:
First point. [pause] Second point. [pause] Third point.
Include natural pauses between each point.
"""

# Dramatic pauses
"""Generate audio:
The answer is... [dramatic pause] ... yes!
Add a suspenseful pause before revealing the answer.
"""

# Breathing points
"""Generate audio:
This is a long sentence that needs breathing points [breath]
so the listener can follow along comfortably [breath]
without feeling rushed or overwhelmed.
Include natural breathing pauses where indicated.
"""

Quality Optimization

Audio Post-Processing

from pydub import AudioSegment
from pydub.effects import normalize, compress_dynamic_range

def enhance_audio(input_path: str, output_path: str):
    """Enhance generated audio quality"""

    # Load audio
    audio = AudioSegment.from_file(input_path)

    # Normalize volume
    audio = normalize(audio)

    # Light compression
    audio = compress_dynamic_range(audio, threshold=-20.0, ratio=2.0)

    # Export with high quality
    audio.export(
        output_path,
        format='wav',
        parameters=['-ar', '44100', '-ac', '2']  # 44.1kHz stereo
    )

Combining Multiple Segments

def generate_segmented_audio(segments: list[dict], output_path: str):
    """
    Generate audio for multiple segments and combine

    Args:
        segments: List of {'text': str, 'style': str}
        output_path: Output file path
    """
    from pydub import AudioSegment

    audio_segments = []

    for i, segment in enumerate(segments):
        prompt = f"Generate audio: {segment['text']}, {segment.get('style', '')}"

        response = client.models.generate_content(
            model='gemini-2.5-flash-native-audio-preview-09-2025',
            contents=prompt,
            config=types.GenerateContentConfig(response_modalities=['audio'])
        )

        # Extract audio
        for part in response.candidates[0].content.parts:
            if part.inline_data:
                # Save temporary file
                temp_path = f'temp_segment_{i}.wav'
                with open(temp_path, 'wb') as f:
                    f.write(part.inline_data.data)

                # Load with pydub
                segment_audio = AudioSegment.from_file(temp_path)
                audio_segments.append(segment_audio)

                # Cleanup
                os.remove(temp_path)

    # Combine all segments with silence between
    silence = AudioSegment.silent(duration=500)  # 500ms pause
    combined = audio_segments[0]

    for segment in audio_segments[1:]:
        combined += silence + segment

    # Export
    combined.export(output_path, format='wav')

# Usage
segments = [
    {'text': 'Welcome to our podcast', 'style': 'warm and welcoming'},
    {'text': 'Today we discuss AI', 'style': 'professional and clear'},
    {'text': 'Thanks for listening', 'style': 'friendly and upbeat'}
]

generate_segmented_audio(segments, 'podcast_intro.wav')

Common Use Cases

Podcast Production

def generate_podcast_intro(
    title: str,
    host_name: str,
    episode_number: int,
    topic: str
) -> bytes:
    """Generate podcast introduction"""

    script = f"""
Generate audio:

Welcome to {title}, episode {episode_number}.
I'm your host, {host_name}.

Today, we're diving into {topic}.
This is going to be an exciting episode, so let's get started!

Narrate this as a professional podcast host with energy and enthusiasm.
Use a warm, welcoming tone with natural pacing.
"""

    response = client.models.generate_content(
        model='gemini-2.5-flash-native-audio-preview-09-2025',
        contents=script,
        config=types.GenerateContentConfig(response_modalities=['audio'])
    )

    # Extract and return audio bytes
    for part in response.candidates[0].content.parts:
        if part.inline_data:
            return part.inline_data.data

# Usage
intro_audio = generate_podcast_intro(
    title="Tech Insights",
    host_name="Alex",
    episode_number=42,
    topic="the future of artificial intelligence"
)

with open('podcast_intro.wav', 'wb') as f:
    f.write(intro_audio)

Audiobook Narration

def generate_audiobook_chapter(chapter_text: str, output_path: str):
    """Generate audiobook narration for chapter"""

    # Split into paragraphs for better control
    paragraphs = chapter_text.split('\n\n')

    audio_parts = []

    for para in paragraphs:
        prompt = f"""
Generate audio:
{para}

Narrate this as an audiobook reader with clear, expressive delivery.
Use appropriate pacing for narrative fiction. Add natural pauses
between sentences. Convey emotion and tone based on the content.
"""

        response = client.models.generate_content(
            model='gemini-2.5-pro',  # Use Pro for higher quality
            contents=prompt,
            config=types.GenerateContentConfig(response_modalities=['audio'])
        )

        # Extract audio
        for part in response.candidates[0].content.parts:
            if part.inline_data:
                audio_parts.append(part.inline_data.data)

    # Combine and save
    combine_audio_parts(audio_parts, output_path)

E-Learning Content

def generate_tutorial_audio(
    lesson_title: str,
    steps: list[str],
    output_path: str
):
    """Generate instructional audio for e-learning"""

    script_parts = [
        f"Generate audio: Welcome to this lesson on {lesson_title}. "
        "Let's begin. Speak in a clear, instructional tone."
    ]

    for i, step in enumerate(steps, 1):
        script_parts.append(
            f"Generate audio: Step {i}: {step}. "
            "Speak slowly and clearly, emphasizing key terms."
        )

    script_parts.append(
        "Generate audio: Great work! You've completed this lesson. "
        "Speak encouragingly with a warm tone."
    )

    # Generate each part
    audio_bytes_list = []

    for script in script_parts:
        response = client.models.generate_content(
            model='gemini-2.5-flash-native-audio-preview-09-2025',
            contents=script,
            config=types.GenerateContentConfig(response_modalities=['audio'])
        )

        for part in response.candidates[0].content.parts:
            if part.inline_data:
                audio_bytes_list.append(part.inline_data.data)

    # Combine with pauses
    combine_with_pauses(audio_bytes_list, output_path, pause_ms=1000)

Voice Announcements

def generate_announcement(
    message: str,
    style: str = 'professional',
    urgency: str = 'normal'
) -> bytes:
    """Generate public announcement audio"""

    urgency_styles = {
        'low': 'in a calm, relaxed manner',
        'normal': 'in a clear, professional tone',
        'high': 'with urgency and importance',
        'critical': 'with immediate urgency and seriousness'
    }

    prompt = f"""
Generate audio:
{message}

Speak {urgency_styles.get(urgency, 'clearly')}, {style}.
Ensure the message is clearly understandable.
"""

    response = client.models.generate_content(
        model='gemini-2.5-flash-native-audio-preview-09-2025',
        contents=prompt,
        config=types.GenerateContentConfig(response_modalities=['audio'])
    )

    for part in response.candidates[0].content.parts:
        if part.inline_data:
            return part.inline_data.data

# Usage examples
normal_announcement = generate_announcement(
    "The meeting will begin in 5 minutes.",
    style='professional',
    urgency='normal'
)

urgent_announcement = generate_announcement(
    "Please evacuate the building immediately.",
    style='authoritative',
    urgency='critical'
)

Production Best Practices

Caching for Reusable Content

import hashlib
from pathlib import Path

class AudioCache:
    """Cache generated audio to avoid regeneration"""

    def __init__(self, cache_dir: str = '.audio_cache'):
        self.cache_dir = Path(cache_dir)
        self.cache_dir.mkdir(exist_ok=True)

    def get_cache_path(self, text: str, style: str = '') -> Path:
        """Generate cache file path from text hash"""
        content = f"{text}|{style}"
        hash_key = hashlib.md5(content.encode()).hexdigest()
        return self.cache_dir / f"{hash_key}.wav"

    def get_or_generate(self, text: str, style: str = '') -> bytes:
        """Get cached audio or generate new"""
        cache_path = self.get_cache_path(text, style)

        if cache_path.exists():
            print(f"Using cached audio: {cache_path}")
            return cache_path.read_bytes()

        # Generate new
        prompt = f"Generate audio: {text}, {style}" if style else f"Generate audio: {text}"

        response = client.models.generate_content(
            model='gemini-2.5-flash-native-audio-preview-09-2025',
            contents=prompt,
            config=types.GenerateContentConfig(response_modalities=['audio'])
        )

        audio_bytes = None
        for part in response.candidates[0].content.parts:
            if part.inline_data:
                audio_bytes = part.inline_data.data
                break

        if audio_bytes:
            cache_path.write_bytes(audio_bytes)
            print(f"Cached audio: {cache_path}")

        return audio_bytes

# Usage
cache = AudioCache()

# First call generates
audio1 = cache.get_or_generate("Welcome to our podcast", "warm and friendly")

# Second call uses cache
audio2 = cache.get_or_generate("Welcome to our podcast", "warm and friendly")

Quality Assurance

def validate_audio_output(audio_bytes: bytes) -> dict:
    """Validate generated audio quality"""

    import tempfile
    from pydub import AudioSegment

    # Save to temp file
    with tempfile.NamedTemporaryFile(suffix='.wav', delete=False) as tmp:
        tmp.write(audio_bytes)
        tmp_path = tmp.name

    # Load and analyze
    audio = AudioSegment.from_file(tmp_path)

    # Check duration
    duration_ms = len(audio)

    # Check silence
    silence_thresh = -50  # dB
    non_silent = audio.strip_silence(silence_thresh=silence_thresh)

    # Calculate metrics
    silence_ratio = 1 - (len(non_silent) / duration_ms)

    # Cleanup
    os.remove(tmp_path)

    # Validation
    passed = (
        duration_ms > 100 and  # At least 100ms
        silence_ratio < 0.5 and  # Less than 50% silence
        audio.frame_rate >= 16000  # Min sample rate
    )

    return {
        'passed': passed,
        'duration_ms': duration_ms,
        'silence_ratio': silence_ratio,
        'sample_rate': audio.frame_rate,
        'channels': audio.channels
    }

Error Handling for TTS

def generate_speech_robust(text: str, style: str = '', max_retries: int = 3):
    """Generate speech with robust error handling"""

    for attempt in range(max_retries):
        try:
            prompt = f"Generate audio: {text}, {style}" if style else f"Generate audio: {text}"

            response = client.models.generate_content(
                model='gemini-2.5-flash-native-audio-preview-09-2025',
                contents=prompt,
                config=types.GenerateContentConfig(response_modalities=['audio'])
            )

            # Extract audio
            for part in response.candidates[0].content.parts:
                if part.inline_data:
                    audio_bytes = part.inline_data.data

                    # Validate quality
                    validation = validate_audio_output(audio_bytes)

                    if validation['passed']:
                        return audio_bytes
                    else:
                        print(f"Audio quality check failed: {validation}")
                        if attempt < max_retries - 1:
                            continue
                        else:
                            return audio_bytes  # Return anyway on last attempt

            raise ValueError("No audio data in response")

        except Exception as e:
            print(f"Attempt {attempt + 1} failed: {e}")
            if attempt == max_retries - 1:
                raise

            time.sleep(2 ** attempt)  # Exponential backoff

    raise RuntimeError("Failed to generate speech after all retries")

Additional Resources

• See api-reference.md for complete API documentation
• See code-examples.md for more implementation examples
• See best-practices.md for general optimization strategies