Ai Music Audio

Ai Music Audio

Identity

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

AI Music & Audio Generation

Patterns

---

Name

Music Generation with MusicGen/Replicate

Description

Generate music from text descriptions using Meta's MusicGen

When

User needs AI-generated background music or soundtracks

Implementation

import Replicate from "replicate";

const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });

interface MusicOptions { prompt: string; duration?: number; // seconds (max 30) modelVersion?: "stereo-large" | "melody-large" | "large"; inputAudio?: string; // for melody conditioning temperature?: number; // 0-1, higher = more random topK?: number; // token sampling topP?: number; // nucleus sampling cfgCoefficient?: number; // classifier-free guidance seed?: number; // for reproducibility }

async function generateMusic(options: MusicOptions): Promise<string> { const { prompt, duration = 8, modelVersion = "stereo-large", inputAudio, temperature = 1.0, topK = 250, topP = 0, cfgCoefficient = 3, seed, } = options;

// Validate duration (MusicGen max is 30 seconds) const safeDuration = Math.min(duration, 30);

const input: Record<string, unknown> = { prompt, duration: safeDuration, model_version: modelVersion, output_format: "mp3", temperature, top_k: topK, top_p: topP, classifier_free_guidance: cfgCoefficient, };

if (inputAudio) { input.input_audio = inputAudio; input.continuation = false; // Use as melody reference }

if (seed !== undefined) { input.seed = seed; }

const output = await replicate.run( "meta/musicgen:671ac645ce5e552cc63a54a2bbff63fcf798043055f2b43c18a9f16e11e82434", { input } );

return output as string; }

// Usage const audioUrl = await generateMusic({ prompt: "upbeat electronic dance music with synths and drums, 120 BPM", duration: 15, modelVersion: "stereo-large", });

---

Name

Text-to-Speech with ElevenLabs

Description

High-quality voice synthesis for narration and dialogue

When

User needs natural-sounding speech from text

Implementation

import { ElevenLabsClient } from "elevenlabs"; import { Readable } from "stream";

const elevenlabs = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY, });

interface TTSOptions { text: string; voiceId: string; // Use premade or cloned voice modelId?: string; // eleven_multilingual_v2, eleven_turbo_v2 stability?: number; // 0-1, lower = more expressive similarityBoost?: number; // 0-1, higher = closer to original style?: number; // 0-1, style exaggeration useSpeakerBoost?: boolean; // enhance voice clarity }

async function textToSpeech(options: TTSOptions): Promise<Buffer> { const { text, voiceId, modelId = "eleven_multilingual_v2", stability = 0.5, similarityBoost = 0.75, style = 0, useSpeakerBoost = true, } = options;

const audioStream = await elevenlabs.textToSpeech.convert(voiceId, { text, model_id: modelId, voice_settings: { stability, similarity_boost: similarityBoost, style, use_speaker_boost: useSpeakerBoost, }, });

// Convert stream to buffer const chunks: Buffer[] = []; for await (const chunk of audioStream) { chunks.push(Buffer.from(chunk)); }

return Buffer.concat(chunks); }

// Streaming version for real-time playback async function streamTextToSpeech( options: TTSOptions, onChunk: (chunk: Buffer) => void ): Promise<void> { const audioStream = await elevenlabs.textToSpeech.convertAsStream( options.voiceId, { text: options.text, model_id: options.modelId || "eleven_turbo_v2", voice_settings: { stability: options.stability || 0.5, similarity_boost: options.similarityBoost || 0.75, }, } );

for await (const chunk of audioStream) { onChunk(Buffer.from(chunk)); } }

// Get available voices async function getVoices() { const response = await elevenlabs.voices.getAll(); return response.voices.map((voice) => ({ id: voice.voice_id, name: voice.name, category: voice.category, description: voice.description, previewUrl: voice.preview_url, })); }

---

Name

Voice Cloning with ElevenLabs

Description

Create custom voice clones from audio samples

When

User wants to clone a voice for consistent narration

Implementation

import { ElevenLabsClient } from "elevenlabs"; import * as fs from "fs";

const elevenlabs = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY, });

// Instant Voice Clone (from 1-3 minute sample) async function createInstantVoiceClone( name: string, samplePaths: string[], description?: string ) { // Read audio files const files = samplePaths.map((path) => fs.createReadStream(path));

const voice = await elevenlabs.voices.add({ name, description: description || Instant clone of ${name}, files, labels: { type: "instant_clone", }, });

return { voiceId: voice.voice_id, name: voice.name, }; }

// Professional Voice Clone (requires 1-3 hours of audio) // This initiates the process - actual training takes time async function initiateProfessionalClone( name: string, sampleUrls: string[], // Links to high-quality audio description: string ) { // PVC requires verification - user must agree to terms // and may need to complete identity verification

const response = await fetch( "https://api.elevenlabs.io/v1/voices/add/professional", { method: "POST", headers: { "xi-api-key": process.env.ELEVENLABS_API_KEY!, "Content-Type": "application/json", }, body: JSON.stringify({ name, description, sample_urls: sampleUrls, }), } );

if (!response.ok) { throw new Error(PVC initiation failed: ${response.statusText}); }

return response.json(); }

// Best practices for voice samples: // - Use high-quality recording (XLR mic, treated room) // - Aim for -6dB to -3dB peak levels // - No background noise, reverb, or music // - For instant: 1-3 minutes // - For professional: 1-3 hours // - Include varied emotional range and speaking styles

---

Name

Sound Effects Generation

Description

Generate sound effects using AudioGen

When

User needs custom sound effects for games or videos

Implementation

import Replicate from "replicate";

const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });

interface SFXOptions { prompt: string; // Description of sound duration?: number; // seconds guidanceScale?: number; // 1-20, higher = closer to prompt temperature?: number; // randomness }

async function generateSoundEffect(options: SFXOptions): Promise<string> { const { prompt, duration = 5, guidanceScale = 3, temperature = 1.0, } = options;

// Using AudioGen via Replicate const output = await replicate.run( "meta/audiogen:f8a0bac6-4f97-4e62-ad89-5e5c89c5d5f2", { input: { prompt, duration: Math.min(duration, 10), // AudioGen max 10s guidance_scale: guidanceScale, temperature, }, } );

return output as string; }

// Alternative: Using Bark for short sound effects with speech async function generateSpeechWithEffects(text: string): Promise<string> { // Bark supports sound effects in text // [laughs], [sighs], [clears throat], [music], etc. const output = await replicate.run( "suno-ai/bark:b76242b40d67c76ab6742e987628a2a9ac019e11d56ab96c4e91ce03b79b2787", { input: { prompt: text, text_temp: 0.7, waveform_temp: 0.7, }, } );

return output.audio_out as string; }

// SFX prompting tips: // - Be specific: "footsteps on wooden floor" not "walking" // - Include environment: "rain on metal roof in warehouse" // - Describe qualities: "deep rumbling thunder, distant" // - Avoid music terms for non-music sounds

---

Name

Audio Watermarking with AudioSeal

Description

Embed imperceptible watermarks for content provenance

When

User needs to mark AI-generated audio for detection

Implementation

// AudioSeal is typically run server-side with PyTorch // This shows the integration pattern

import { spawn } from "child_process";

interface WatermarkResult { watermarkedAudioPath: string; secret: string; // The embedded message }

interface DetectionResult { isWatermarked: boolean; confidence: number; decodedMessage?: string; }

// Python script for AudioSeal (run on server) const AUDIOSEAL_SCRIPT = ` import torch import torchaudio from audioseal import AudioSeal import sys import json

def watermark_audio(input_path, output_path, message=None):

Load model

model = AudioSeal.load_generator("audioseal_wm_16bits")

Load audio

audio, sr = torchaudio.load(input_path) if sr != 16000: resampler = torchaudio.transforms.Resample(sr, 16000) audio = resampler(audio)

Generate watermark

if message:

Convert message to bits

msg_bits = torch.tensor([int(b) for b in format(int(message, 16), '016b')]) watermarked = model.embed(audio.unsqueeze(0), msg_bits.unsqueeze(0)) else: watermarked = model.embed(audio.unsqueeze(0))

Save

torchaudio.save(output_path, watermarked.squeeze(0), 16000) return {"success": True}

def detect_watermark(audio_path): detector = AudioSeal.load_detector("audioseal_detector_16bits")

audio, sr = torchaudio.load(audio_path) if sr != 16000: resampler = torchaudio.transforms.Resample(sr, 16000) audio = resampler(audio)

result = detector.detect(audio.unsqueeze(0))

return { "is_watermarked": result.score > 0.5, "confidence": float(result.score), "message": result.message if hasattr(result, 'message') else None }

if __name__ == "__main__": action = sys.argv[1] if action == "watermark": result = watermark_audio(sys.argv[2], sys.argv[3], sys.argv[4] if len(sys.argv) > 4 else None) elif action == "detect": result = detect_watermark(sys.argv[2]) print(json.dumps(result)) `;

// Node.js wrapper async function watermarkAudio( inputPath: string, outputPath: string, message?: string ): Promise<WatermarkResult> { return new Promise((resolve, reject) => { const args = ["watermark", inputPath, outputPath]; if (message) args.push(message);

const process = spawn("python", ["-c", AUDIOSEAL_SCRIPT, ...args]);

let output = ""; process.stdout.on("data", (data) => (output += data)); process.stderr.on("data", (data) => console.error(data.toString()));

process.on("close", (code) => { if (code === 0) { resolve({ watermarkedAudioPath: outputPath, secret: message || "default", }); } else { reject(new Error("Watermarking failed")); } }); }); }

async function detectWatermark(audioPath: string): Promise<DetectionResult> { return new Promise((resolve, reject) => { const process = spawn("python", [ "-c", AUDIOSEAL_SCRIPT, "detect", audioPath, ]);

let output = ""; process.stdout.on("data", (data) => (output += data));

process.on("close", (code) => { if (code === 0) { const result = JSON.parse(output); resolve({ isWatermarked: result.is_watermarked, confidence: result.confidence, decodedMessage: result.message, }); } else { reject(new Error("Detection failed")); } }); }); }

---

Name

Streaming Audio Response

Description

Stream generated audio for real-time playback

When

User needs audio to start playing before generation completes

Implementation

import { NextRequest, NextResponse } from "next/server"; import { ElevenLabsClient } from "elevenlabs";

const elevenlabs = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY, });

// Next.js streaming TTS endpoint export async function POST(request: NextRequest) { const { text, voiceId } = await request.json();

// Get streaming audio const audioStream = await elevenlabs.textToSpeech.convertAsStream( voiceId, { text, model_id: "eleven_turbo_v2", // Faster for streaming voice_settings: { stability: 0.5, similarity_boost: 0.75, }, } );

// Create readable stream for response const stream = new ReadableStream({ async start(controller) { for await (const chunk of audioStream) { controller.enqueue(chunk); } controller.close(); }, });

return new NextResponse(stream, { headers: { "Content-Type": "audio/mpeg", "Transfer-Encoding": "chunked", }, }); }

// Frontend: Play streaming audio async function playStreamingAudio(text: string, voiceId: string) { const response = await fetch("/api/tts/stream", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ text, voiceId }), });

// Use Media Source Extensions for streaming playback const mediaSource = new MediaSource(); const audio = new Audio(); audio.src = URL.createObjectURL(mediaSource);

mediaSource.addEventListener("sourceopen", async () => { const sourceBuffer = mediaSource.addSourceBuffer("audio/mpeg"); const reader = response.body!.getReader();

while (true) { const { done, value } = await reader.read(); if (done) { mediaSource.endOfStream(); break; }

// Wait for buffer to be ready await new Promise((resolve) => { if (sourceBuffer.updating) { sourceBuffer.addEventListener("updateend", resolve, { once: true }); } else { resolve(undefined); } });

sourceBuffer.appendBuffer(value); } });

await audio.play(); }

---

Name

Lyria for High-Quality Music

Description

Use Fal.ai's Lyria model for production music

When

User needs higher quality AI music than MusicGen

Implementation

// Using Fal.ai's Lyria model for music generation // Lyria2 provides higher quality output

interface LyriaOptions { prompt: string; duration?: number; negativePrompt?: string; }

async function generateMusicWithLyria(options: LyriaOptions) { const response = await fetch("https://fal.run/fal-ai/lyria2", { method: "POST", headers: { Authorization: Key ${process.env.FAL_KEY}, "Content-Type": "application/json", }, body: JSON.stringify({ prompt: options.prompt, duration_seconds: options.duration || 30, negative_prompt: options.negativePrompt, }), });

if (!response.ok) { throw new Error(Lyria generation failed: ${response.statusText}); }

const result = await response.json(); return { audioUrl: result.audio.url, duration: result.audio.duration, }; }

// Music prompting best practices: // - Include genre: "synthwave", "lo-fi hip hop", "orchestral" // - Specify tempo: "120 BPM", "slow tempo", "upbeat" // - Describe mood: "melancholic", "energetic", "peaceful" // - Add instruments: "piano and strings", "electric guitar solo" // - Reference styles: "in the style of 80s synth pop"

Anti-Patterns

---

Name

Exposing audio API keys client-side

Why Bad

Audio generation is expensive - leaked keys cause massive bills

Example Bad

// Frontend code with exposed key const elevenlabs = new ElevenLabsClient({ apiKey: "sk-..." // Exposed in browser! });

Example Good

// Server-side only // app/api/tts/route.ts const elevenlabs = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY, });

---

Name

No character/duration limits

Why Bad

Single long text can cost $10+ in TTS charges

Example Bad

async function synthesize(text: string) { return elevenlabs.textToSpeech.convert(voiceId, { text }); }

Example Good

async function synthesize(text: string) { if (text.length > 5000) { throw new Error("Text exceeds 5000 character limit"); } // Also implement cost tracking await recordCost(userId, text.length * 0.00003); // ~$0.03/1000 chars return elevenlabs.textToSpeech.convert(voiceId, { text }); }

---

Name

Voice cloning without consent

Why Bad

Creates deepfakes, legal liability, platform bans

Example Bad

// Clone any uploaded voice await createVoiceClone(userUploadedAudio);

Example Good

// Require explicit consent const consent = await db.voiceConsent.findUnique({ where: { voiceOwnerEmail, voiceId }, });

if (!consent || !consent.verified) { throw new Error("Voice consent not verified"); }

await createVoiceClone(userUploadedAudio);

---

Name

No content moderation on generated audio

Why Bad

AI can generate harmful content (hate speech, misinformation)

Example Bad

// Generate without checking const audio = await textToSpeech(userInput); return audio;

Example Good

// Moderate text before synthesis const moderation = await openai.moderations.create({ input: userInput }); if (moderation.results[0].flagged) { throw new Error("Content violates policy"); } const audio = await textToSpeech(userInput);

---

Name

Synchronous long audio generation

Why Bad

Music generation takes 30-120+ seconds, blocks requests

Example Bad

// Blocks for 2+ minutes app.post("/api/music", async (req, res) => { const audio = await generateMusic(req.body.prompt); res.json({ audio }); });

Example Good

// Queue-based async app.post("/api/music", async (req, res) => { const job = await musicQueue.add("generate", req.body); res.json({ jobId: job.id, status: "pending" }); });

Ai Music Audio - Sharp Edges

Voice Cloning Legal Liability

Id

voice-cloning-legal-liability

Summary

Voice cloning without consent creates deepfakes

Severity

critical

Situation

Cloning voices from public audio without explicit permission

Why

Voice cloning technology can perfectly replicate anyone's voice. Using someone's voice without consent:

• Creates potential deepfakes for fraud/misinformation
• Violates right of publicity laws in most jurisdictions
• Results in platform bans (ElevenLabs actively monitors)
• Can be used for voice phishing (vishing) attacks
• Major lawsuits pending against AI companies over this

Detection Pattern

createVoiceClone|add_voice|clone.voice(?!.consent|verify)

Solution

Always require and verify consent: 1. Implement voice consent verification flow 2. Require email/identity verification for voice owner 3. Store consent records with timestamps 4. Limit cloning to user's own voice or verified consents 5. Add voice watermarking for traceability

  // Consent verification required
  const consent = await db.voiceConsent.findUnique({
    where: { voiceOwnerId: ownerId, projectId },
  });
  
  if (!consent?.verified || !consent?.signature) {
    throw new Error("Voice cloning requires verified consent");
  }
  
  // Log for audit trail
  await db.voiceCloneAudit.create({
    data: {
      clonedBy: userId,
      voiceOwnerId: ownerId,
      consentId: consent.id,
      purpose: purpose,
    },
  });

Tts Cost Explosion

Id

tts-cost-explosion

Summary

Unbounded TTS text causes massive bills

Severity

high

Situation

User-provided text passed directly to TTS without limits

Why

TTS pricing is per character (e.g., ElevenLabs ~$0.03/1000 chars). A single novel (~500K characters) = $15+ per generation. Malicious users can exploit unlimited endpoints:

• Submit entire books repeatedly
• Bot attacks generating millions of characters
• Single request can consume monthly budget

Detection Pattern

textToSpeech\(.text(?!.slice|substring|length.*<|limit)

Solution

Implement multiple safeguards:

  const MAX_CHARS = 5000;
  const MAX_MONTHLY_CHARS_PER_USER = 100000;
  
  async function safeTTS(userId: string, text: string) {
    // 1. Hard character limit
    if (text.length > MAX_CHARS) {
      throw new Error(`Text exceeds ${MAX_CHARS} character limit`);
    }
  
    // 2. Check monthly usage
    const monthlyUsage = await getMonthlyCharUsage(userId);
    if (monthlyUsage + text.length > MAX_MONTHLY_CHARS_PER_USER) {
      throw new Error("Monthly character limit reached");
    }
  
    // 3. Rate limit per minute
    await rateLimit(userId, "tts", { maxRequests: 10, window: 60 });
  
    // 4. Track and charge
    await recordUsage(userId, text.length);
  
    return elevenlabs.textToSpeech.convert(voiceId, { text });
  }

Music Gen 30 Second Limit

Id

music-gen-30-second-limit

Summary

MusicGen silently clips audio at 30 seconds

Severity

medium

Situation

Requesting music duration longer than model supports

Why

MusicGen (via Replicate) has a hard 30-second limit. If you request 60 seconds, you get 30 seconds (no error). User expects full length, receives half. Wastes money on clipped generations.

Detection Pattern

musicgen.duration.[3-9][0-9]|duration.*[1-9][0-9][0-9]

Solution

Validate duration and implement concatenation for longer audio:

  const MUSICGEN_MAX_DURATION = 30;
  
  async function generateLongMusic(prompt: string, duration: number) {
    // For short music, generate directly
    if (duration <= MUSICGEN_MAX_DURATION) {
      return generateMusic({ prompt, duration });
    }
  
    // For longer, generate segments with continuation
    const segments: string[] = [];
    let remaining = duration;
  
    while (remaining > 0) {
      const segmentDuration = Math.min(remaining, MUSICGEN_MAX_DURATION);
      const previousAudio = segments[segments.length - 1];
  
      const segment = await generateMusic({
        prompt,
        duration: segmentDuration,
        inputAudio: previousAudio, // Use as reference
        continuation: true,
      });
  
      segments.push(segment);
      remaining -= segmentDuration;
    }
  
    // Concatenate with crossfade
    return concatenateWithCrossfade(segments);
  }

Voice Settings Instability

Id

voice-settings-instability

Summary

Low stability settings produce inconsistent output

Severity

medium

Situation

Using ElevenLabs with stability < 0.3

Why

ElevenLabs stability setting (0-1) controls voice consistency:

• Low stability = more expressive but unpredictable
• Can produce different voices on same text
• May include artifacts, breathing, emotional breaks
• Problematic for long-form content or audiobooks
• Inconsistent between API calls

Detection Pattern

stability.0\.[0-2]|stability.0(?!\.)|voice_settings.stability.0\.[0-2]

Solution

Use appropriate stability for use case:

  const STABILITY_PRESETS = {
    // High consistency for narration/audiobooks
    narration: { stability: 0.75, similarityBoost: 0.75 },
  
    // Medium for conversational content
    conversational: { stability: 0.5, similarityBoost: 0.75 },
  
    // Low for expressive/emotional content (with review)
    expressive: { stability: 0.35, similarityBoost: 0.5 },
  };
  
  function getVoiceSettings(useCase: keyof typeof STABILITY_PRESETS) {
    return STABILITY_PRESETS[useCase];
  }
  
  // Always use presets, avoid raw low values
  const settings = getVoiceSettings("narration");

Audio Format Compatibility

Id

audio-format-compatibility

Summary

Generated audio format incompatible with target platform

Severity

medium

Situation

Using mp3/wav without checking platform requirements

Why

Different platforms/browsers have format limitations:

• Safari has limited WebM/Opus support
• Some mobile browsers can't play certain codecs
• Streaming requires specific container formats
• Sample rate mismatches cause playback issues
• Bitrate affects both quality and file size

Detection Pattern

output_format.*(?!mp3|wav)|audio\/(?!mpeg|wav|mp3)

Solution

Use widely compatible formats and transcode when needed:

  // Default to MP3 for maximum compatibility
  const SAFE_FORMATS = {
    web: { format: "mp3", bitrate: 128 },
    mobile: { format: "mp3", bitrate: 96 },
    highQuality: { format: "wav", bitrate: null },
    streaming: { format: "mp3", bitrate: 64 },
  };
  
  async function generateCompatibleAudio(
    prompt: string,
    platform: keyof typeof SAFE_FORMATS
  ) {
    const { format, bitrate } = SAFE_FORMATS[platform];
  
    const audio = await generate(prompt, { outputFormat: format });
  
    // Verify format
    const mimeType = await detectAudioFormat(audio);
    if (!isSupportedOn(mimeType, platform)) {
      return transcodeAudio(audio, format, bitrate);
    }
  
    return audio;
  }

Prompt Injection Audio

Id

prompt-injection-audio

Summary

Malicious prompts generate harmful audio content

Severity

high

Situation

User prompts passed to TTS without sanitization

Why

TTS can be used to generate:

• Hate speech and slurs
• Misinformation/fake news
• Instructions for harm
• Impersonation of real people
• Political manipulation content

Content moderation on text is essential before synthesis.

Detection Pattern

textToSpeech\(.(?:req|request|user)\.(?:body|query|input)(?!.moderat)

Solution

Always moderate text before synthesis:

  import OpenAI from "openai";
  
  const openai = new OpenAI();
  
  async function moderatedTTS(text: string) {
    // 1. OpenAI moderation
    const moderation = await openai.moderations.create({ input: text });
    if (moderation.results[0].flagged) {
      throw new Error("Content violates content policy");
    }
  
    // 2. Check for impersonation patterns
    const impersonationPatterns = [
      /this is (president|senator|ceo|celebrity name)/i,
      /I am (elon musk|joe biden|famous person)/i,
      /speaking as the (ceo|president|official)/i,
    ];
  
    for (const pattern of impersonationPatterns) {
      if (pattern.test(text)) {
        throw new Error("Potential impersonation detected");
      }
    }
  
    // 3. Generate only if passes checks
    return textToSpeech(text);
  }

No Watermarking

Id

no-watermarking

Summary

AI audio released without provenance markers

Severity

medium

Situation

Generated audio shared publicly without watermarking

Why

Without watermarking:

• Cannot prove audio is AI-generated
• Enables deepfakes and misinformation
• No accountability for misuse
• Increasingly required by regulations
• Platforms may require C2PA compliance

Detection Pattern

generate.audio.return(?!.*watermark|seal)

Solution

Embed AudioSeal or similar watermark before distribution:

  async function generateWatermarkedAudio(prompt: string) {
    // 1. Generate audio
    const audio = await generateAudio(prompt);
  
    // 2. Add watermark with provenance info
    const watermarked = await embedWatermark(audio, {
      source: "ai-generated",
      model: "elevenlabs-v2",
      timestamp: Date.now(),
      creator: userId,
    });
  
    // 3. Add C2PA manifest for platforms that support it
    const withManifest = await addC2PAManifest(watermarked, {
      assertions: [
        { label: "c2pa.ai_generative_training", data: { model: "elevenlabs" } },
      ],
    });
  
    return withManifest;
  }

Streaming Memory Leak

Id

streaming-memory-leak

Summary

Audio streams not properly closed cause memory leaks

Severity

medium

Situation

Streaming TTS without proper stream handling

Why

Audio streams consume memory until closed:

• Unclosed streams accumulate
• Server memory exhaustion
• Connection pool exhaustion
• Especially bad with long audio or errors mid-stream

Detection Pattern

convertAsStream\((?!.*finally|close|destroy)

Solution

Always properly handle and close streams:

  async function safeStreamTTS(text: string, voiceId: string) {
    let stream: AsyncIterable<Buffer> | null = null;
  
    try {
      stream = await elevenlabs.textToSpeech.convertAsStream(voiceId, {
        text,
      });
  
      const chunks: Buffer[] = [];
      for await (const chunk of stream) {
        chunks.push(Buffer.from(chunk));
      }
  
      return Buffer.concat(chunks);
    } catch (error) {
      console.error("TTS stream error:", error);
      throw error;
    } finally {
      // Cleanup - stream should auto-close but be explicit
      if (stream && typeof (stream as any).destroy === "function") {
        (stream as any).destroy();
      }
    }
  }
  
  // For HTTP responses, pipe properly
  export async function GET(req: NextRequest) {
    const stream = await elevenlabs.textToSpeech.convertAsStream(...);
  
    return new NextResponse(
      new ReadableStream({
        async start(controller) {
          try {
            for await (const chunk of stream) {
              controller.enqueue(chunk);
            }
          } finally {
            controller.close();
          }
        },
        cancel() {
          // Handle client disconnect
          if (typeof (stream as any).destroy === "function") {
            (stream as any).destroy();
          }
        },
      }),
      { headers: { "Content-Type": "audio/mpeg" } }
    );
  }

Sample Rate Mismatch

Id

sample-rate-mismatch

Summary

Mixed sample rates cause audio artifacts

Severity

low

Situation

Concatenating or mixing audio with different sample rates

Why

Different models output different sample rates:

• MusicGen: 32kHz
• ElevenLabs: 44.1kHz or 22.05kHz
• AudioSeal: 16kHz
• Bark: 24kHz

Mixing without resampling causes pitch/speed issues.

Detection Pattern

concat.audio|merge.audio|mix.audio(?!.resample|rate)

Solution

Normalize sample rates before combining:

  import ffmpeg from "fluent-ffmpeg";
  
  const TARGET_SAMPLE_RATE = 44100;
  
  async function normalizeAndConcatenate(audioPaths: string[]) {
    const normalizedPaths: string[] = [];
  
    // Normalize each to target sample rate
    for (const path of audioPaths) {
      const normalized = path.replace(".mp3", "_normalized.mp3");
  
      await new Promise((resolve, reject) => {
        ffmpeg(path)
          .audioFrequency(TARGET_SAMPLE_RATE)
          .audioChannels(2) // Stereo
          .audioBitrate("128k")
          .output(normalized)
          .on("end", resolve)
          .on("error", reject)
          .run();
      });
  
      normalizedPaths.push(normalized);
    }
  
    // Now safe to concatenate
    return concatenateAudio(normalizedPaths);
  }

Unofficial Api Instability

Id

unofficial-api-instability

Summary

Unofficial Suno/Udio APIs break frequently

Severity

high

Situation

Using unofficial wrapper APIs for Suno or Udio

Why

Suno and Udio don't offer official APIs. Unofficial wrappers:

• Scrape web interfaces (break with UI changes)
• Require solving CAPTCHAs
• Violate Terms of Service
• Can get your account banned
• Major lawsuit risk (Sony, Universal, Warner suing both)

Detection Pattern

suno-api|udio.*api|gcui-art/suno|SUNO_COOKIE

Solution

Use officially supported alternatives:

  // AVOID: Unofficial Suno API
  // const suno = new SunoAPI({ cookie: process.env.SUNO_COOKIE });
  
  // USE: Official APIs with proper licensing
  // Option 1: Meta MusicGen (MIT license, CC-BY-NC for weights)
  import Replicate from "replicate";
  const music = await replicate.run("meta/musicgen");
  
  // Option 2: Stable Audio (commercial license available)
  // Option 3: Soundraw API (commercial license)
  // Option 4: AIVA API (commercial license)
  
  // For production, only use services with:
  // - Official API documentation
  // - Clear licensing terms
  // - Commercial use rights

Ai Music Audio - Validations

Audio API Key in Client Code

Id

audio-api-key-exposed

Severity

error

Description

Audio generation API keys must be server-side only

Pattern

(NEXT_PUBLIC|REACT_APP|VITE).(ELEVENLABS|REPLICATE|FAL|OPENAI).KEY

Message

Audio API key exposed to client. Use server-side routes only.

Autofix

Hardcoded Audio API Key

Id

hardcoded-audio-api-key

Severity

error

Description

API keys should use environment variables

Pattern

(xi-|sk-|r8_|fal_)[A-Za-z0-9]{20,}

Message

Hardcoded API key detected. Use environment variables.

Autofix

Missing TTS Content Moderation

Id

no-tts-moderation

Severity

error

Description

Text must be moderated before speech synthesis

Pattern

textToSpeech\(.(?:req|request|user)\.(?:body|query)(?!.moderat)

Message

User text passed to TTS without moderation. Add content check.

Autofix

Voice Cloning Without Consent Check

Id

voice-clone-no-consent

Severity

error

Description

Voice cloning requires verified consent

Pattern

createVoiceClone|add.voice|clone.voice(?!.*consent|verify|permission)

Message

Voice cloning without consent verification. Implement consent flow.

Autofix

Missing Deepfake Prevention

Id

no-deepfake-prevention

Severity

warning

Description

Check for impersonation attempts in TTS requests

Pattern

textToSpeech.(?:as|voice.of|impersonate).*(?:president|ceo|celebrity)

Message

Potential impersonation content. Add impersonation detection.

Autofix

TTS Without Rate Limiting

Id

no-tts-rate-limiting

Severity

warning

Description

TTS endpoints should be rate limited

Pattern

async.textToSpeech.request(?!.*rateLimit|limit)

Message

TTS endpoint without rate limiting.

Autofix

TTS Without Character Limit

Id

no-character-limit

Severity

warning

Description

Text length should be limited to prevent cost abuse

Pattern

textToSpeech\(.text(?!.length.*<|slice|substring|limit)

Message

TTS without text length limit. Add maximum character check.

Autofix

Missing Audio Cost Tracking

Id

no-audio-cost-tracking

Severity

warning

Description

Audio generation costs should be tracked per user

Pattern

(textToSpeech|generateMusic)\((?!.*cost|credits|budget)

Message

No cost tracking for audio generation. Add usage tracking.

Autofix

Unbounded Music Duration

Id

unbounded-music-duration

Severity

warning

Description

Music duration should be capped

Pattern

duration.request\.(body|query)(?!.Math\.min|clamp|max)

Message

User-controlled duration without limit. Cap at maximum allowed.

Autofix

Missing Sample Rate Normalization

Id

no-sample-rate-normalization

Severity

warning

Description

Audio should be normalized before mixing/concatenation

Pattern

concat.audio|merge.audio(?!.*resample|normalize|rate)

Message

Audio concatenation without sample rate normalization.

Autofix

Missing Audio Format Validation

Id

no-audio-format-validation

Severity

warning

Description

Uploaded audio should be validated

Pattern

upload.audio|audio.file(?!.*validate|check|format)

Message

Audio upload without format validation.

Autofix

Audio Stream Not Properly Closed

Id

stream-not-closed

Severity

warning

Description

Audio streams should be properly closed

Pattern

convertAsStream\((?!.*finally|close|destroy)

Message

Audio stream without proper cleanup. Add finally block.

Autofix

Streaming Without Timeout

Id

no-stream-timeout

Severity

warning

Description

Audio streams should have timeouts

Pattern

Stream.audio(?!.timeout|AbortController)

Message

Audio streaming without timeout. Add timeout handling.

Autofix

Missing Audio Watermark

Id

no-audio-watermark

Severity

warning

Description

AI-generated audio should be watermarked

Pattern

generate.audio.return(?!.*watermark|seal|c2pa)

Message

AI audio returned without watermark. Add AudioSeal or similar.

Autofix