Heygen Best Practices

When to use

Use this skill whenever you are dealing with HeyGen API code to obtain domain-specific knowledge for creating AI avatar videos, managing avatars, handling video generation workflows, and integrating with HeyGen's services.

How to use

Read individual rule files for detailed explanations and code examples:

Foundation

• rules/authentication.md - API key setup, X-Api-Key header, and authentication patterns
• rules/quota.md - Credit system, usage limits, and checking remaining quota
• rules/video-status.md - Polling patterns, status types, and retrieving download URLs
• rules/assets.md - Uploading images, videos, and audio for use in video generation

Core Video Creation

• rules/avatars.md - Listing avatars, avatar styles, and avatar_id selection
• rules/voices.md - Listing voices, locales, speed/pitch configuration
• rules/scripts.md - Writing scripts, pauses/breaks, pacing, and structure templates
• rules/video-generation.md - POST /v2/video/generate workflow and multi-scene videos
• rules/video-agent.md - One-shot prompt video generation with Video Agent API
• rules/dimensions.md - Resolution options (720p/1080p) and aspect ratios

Video Customization

• rules/backgrounds.md - Solid colors, images, and video backgrounds
• rules/text-overlays.md - Adding text with fonts and positioning
• rules/captions.md - Auto-generated captions and subtitle options

Advanced Features

• rules/templates.md - Template listing and variable replacement
• rules/video-translation.md - Translating videos, quality/fast modes, and dubbing
• rules/streaming-avatars.md - Real-time interactive avatar sessions
• rules/photo-avatars.md - Creating avatars from photos (talking photos)
• rules/webhooks.md - Registering webhook endpoints and event types

Integration

• rules/remotion-integration.md - Using HeyGen avatar videos in Remotion compositions

Asset Upload and Management

HeyGen allows you to upload custom assets (images, videos, audio) for use in video generation, such as backgrounds, talking photo sources, and custom audio.

Upload Flow

Asset uploads use a two-step process: 1. Get a presigned upload URL from HeyGen 2. Upload the file to the presigned URL

Getting an Upload URL

Request Fields

curl

curl -X POST "https://api.heygen.com/v1/asset" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"content_type": "image/jpeg"}'

TypeScript

interface AssetUploadRequest {
  content_type: string;                        // Required
}

interface AssetUploadResponse {
  error: null | string;
  data: {
    url: string;
    asset_id: string;
  };
}

async function getUploadUrl(contentType: string): Promise<AssetUploadResponse["data"]> {
  const response = await fetch("https://api.heygen.com/v1/asset", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ content_type: contentType }),
  });

  const json: AssetUploadResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data;
}

Python

import requests
import os

def get_upload_url(content_type: str) -> dict:
    response = requests.post(
        "https://api.heygen.com/v1/asset",
        headers={
            "X-Api-Key": os.environ["HEYGEN_API_KEY"],
            "Content-Type": "application/json"
        },
        json={"content_type": content_type}
    )

    data = response.json()
    if data.get("error"):
        raise Exception(data["error"])

    return data["data"]

Supported Content Types

Uploading Files

TypeScript

import fs from "fs";

async function uploadFile(filePath: string, contentType: string): Promise<string> {
  // 1. Get upload URL
  const { url, asset_id } = await getUploadUrl(contentType);

  // 2. Read file
  const fileBuffer = fs.readFileSync(filePath);

  // 3. Upload to presigned URL
  const uploadResponse = await fetch(url, {
    method: "PUT",
    headers: {
      "Content-Type": contentType,
    },
    body: fileBuffer,
  });

  if (!uploadResponse.ok) {
    throw new Error(`Upload failed: ${uploadResponse.status}`);
  }

  return asset_id;
}

// Usage
const imageAssetId = await uploadFile("./background.jpg", "image/jpeg");
console.log(`Uploaded image asset: ${imageAssetId}`);

TypeScript (with streams for large files)

import fs from "fs";
import { stat } from "fs/promises";

async function uploadLargeFile(filePath: string, contentType: string): Promise<string> {
  const { url, asset_id } = await getUploadUrl(contentType);

  const fileStats = await stat(filePath);
  const fileStream = fs.createReadStream(filePath);

  const uploadResponse = await fetch(url, {
    method: "PUT",
    headers: {
      "Content-Type": contentType,
      "Content-Length": fileStats.size.toString(),
    },
    body: fileStream as any,
    // @ts-ignore - duplex is needed for streaming
    duplex: "half",
  });

  if (!uploadResponse.ok) {
    throw new Error(`Upload failed: ${uploadResponse.status}`);
  }

  return asset_id;
}

Python

import requests

def upload_file(file_path: str, content_type: str) -> str:
    # 1. Get upload URL
    upload_data = get_upload_url(content_type)
    url = upload_data["url"]
    asset_id = upload_data["asset_id"]

    # 2. Upload file
    with open(file_path, "rb") as f:
        response = requests.put(
            url,
            headers={"Content-Type": content_type},
            data=f
        )

    if not response.ok:
        raise Exception(f"Upload failed: {response.status_code}")

    return asset_id


# Usage
image_asset_id = upload_file("./background.jpg", "image/jpeg")
print(f"Uploaded image asset: {image_asset_id}")

Uploading from URL

If your asset is already hosted online:

async function uploadFromUrl(sourceUrl: string, contentType: string): Promise<string> {
  // 1. Download the file
  const sourceResponse = await fetch(sourceUrl);
  const buffer = await sourceResponse.arrayBuffer();

  // 2. Get HeyGen upload URL
  const { url, asset_id } = await getUploadUrl(contentType);

  // 3. Upload to HeyGen
  await fetch(url, {
    method: "PUT",
    headers: { "Content-Type": contentType },
    body: buffer,
  });

  return asset_id;
}

Using Uploaded Assets

As Background Image

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Hello, this is a video with a custom background!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
      background: {
        type: "image",
        url: `https://files.heygen.ai/asset/${imageAssetId}`,
      },
    },
  ],
};

As Talking Photo Source

const talkingPhotoConfig = {
  video_inputs: [
    {
      character: {
        type: "talking_photo",
        talking_photo_id: photoAssetId,
      },
      voice: {
        type: "text",
        input_text: "Hello from my talking photo!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
    },
  ],
};

As Audio Input

const audioConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "audio",
        audio_url: `https://files.heygen.ai/asset/${audioAssetId}`,
      },
    },
  ],
};

Complete Upload Workflow

async function createVideoWithCustomBackground(
  backgroundPath: string,
  script: string
): Promise<string> {
  // 1. Upload background
  console.log("Uploading background...");
  const backgroundId = await uploadFile(backgroundPath, "image/jpeg");

  // 2. Create video config
  const config = {
    video_inputs: [
      {
        character: {
          type: "avatar",
          avatar_id: "josh_lite3_20230714",
          avatar_style: "normal",
        },
        voice: {
          type: "text",
          input_text: script,
          voice_id: "1bd001e7e50f421d891986aad5158bc8",
        },
        background: {
          type: "image",
          url: `https://files.heygen.ai/asset/${backgroundId}`,
        },
      },
    ],
    dimension: { width: 1920, height: 1080 },
  };

  // 3. Generate video
  console.log("Generating video...");
  const response = await fetch("https://api.heygen.com/v2/video/generate", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(config),
  });

  const { data } = await response.json();
  return data.video_id;
}

Asset Limitations

• File size: Varies by asset type (typically 10-100MB max)
• Image dimensions: Recommended to match video dimensions
• Audio duration: Should match expected video length
• Retention: Assets may be deleted after a period of inactivity

Best Practices

1. Optimize images - Resize to match video dimensions before uploading 2. Use appropriate formats - JPEG for photos, PNG for graphics with transparency 3. Validate before upload - Check file type and size locally first 4. Handle upload errors - Implement retry logic for failed uploads 5. Cache asset IDs - Reuse assets across multiple video generations

HeyGen Authentication

All HeyGen API requests require authentication using an API key passed in the X-Api-Key header.

Getting Your API Key

1. Log in to your HeyGen account at https://app.heygen.com 2. Navigate to Settings > API 3. Copy your API key

Environment Setup

Store your API key securely as an environment variable:

export HEYGEN_API_KEY="your-api-key-here"

For .env files:

HEYGEN_API_KEY=your-api-key-here

Making Authenticated Requests

curl

curl -X GET "https://api.heygen.com/v2/avatars" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

TypeScript/JavaScript (fetch)

const response = await fetch("https://api.heygen.com/v2/avatars", {
  headers: {
    "X-Api-Key": process.env.HEYGEN_API_KEY!,
  },
});
const { data } = await response.json();

TypeScript/JavaScript (axios)

import axios from "axios";

const client = axios.create({
  baseURL: "https://api.heygen.com",
  headers: {
    "X-Api-Key": process.env.HEYGEN_API_KEY,
  },
});

const { data } = await client.get("/v2/avatars");

Python (requests)

import os
import requests

response = requests.get(
    "https://api.heygen.com/v2/avatars",
    headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()

Python (httpx)

import os
import httpx

async with httpx.AsyncClient() as client:
    response = await client.get(
        "https://api.heygen.com/v2/avatars",
        headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
    )
    data = response.json()

Creating a Reusable API Client

TypeScript

class HeyGenClient {
  private baseUrl = "https://api.heygen.com";
  private apiKey: string;

  constructor(apiKey: string) {
    this.apiKey = apiKey;
  }

  async request<T>(endpoint: string, options: RequestInit = {}): Promise<T> {
    const response = await fetch(`${this.baseUrl}${endpoint}`, {
      ...options,
      headers: {
        "X-Api-Key": this.apiKey,
        "Content-Type": "application/json",
        ...options.headers,
      },
    });

    if (!response.ok) {
      const error = await response.json();
      throw new Error(error.message || `HTTP ${response.status}`);
    }

    return response.json();
  }

  get<T>(endpoint: string): Promise<T> {
    return this.request<T>(endpoint);
  }

  post<T>(endpoint: string, body: unknown): Promise<T> {
    return this.request<T>(endpoint, {
      method: "POST",
      body: JSON.stringify(body),
    });
  }
}

// Usage
const client = new HeyGenClient(process.env.HEYGEN_API_KEY!);
const avatars = await client.get("/v2/avatars");

API Response Format

All HeyGen API responses follow this structure:

interface ApiResponse<T> {
  error: null | string;
  data: T;
}

Successful response example:

{
  "error": null,
  "data": {
    "avatars": [...]
  }
}

Error response example:

{
  "error": "Invalid API key",
  "data": null
}

Error Handling

Common authentication errors:

Handling Errors

async function makeRequest(endpoint: string) {
  const response = await fetch(`https://api.heygen.com${endpoint}`, {
    headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
  });

  const json = await response.json();

  if (!response.ok || json.error) {
    throw new Error(json.error || `HTTP ${response.status}`);
  }

  return json.data;
}

Rate Limiting

HeyGen enforces rate limits on API requests:

• Standard rate limits apply per API key
• Some endpoints (like video generation) have stricter limits
• Use exponential backoff when receiving 429 errors

async function requestWithRetry(
  fn: () => Promise<Response>,
  maxRetries = 3
): Promise<Response> {
  for (let i = 0; i < maxRetries; i++) {
    const response = await fn();

    if (response.status === 429) {
      const waitTime = Math.pow(2, i) * 1000;
      await new Promise((resolve) => setTimeout(resolve, waitTime));
      continue;
    }

    return response;
  }

  throw new Error("Max retries exceeded");
}

Security Best Practices

1. Never expose API keys in client-side code - Always make API calls from a backend server 2. Use environment variables - Don't hardcode API keys in source code 3. Rotate keys periodically - Generate new API keys regularly 4. Monitor usage - Check your HeyGen dashboard for unusual activity

HeyGen Avatars

Avatars are the AI-generated presenters in HeyGen videos. You can use public avatars provided by HeyGen or create custom avatars.

Previewing Avatars Before Generation

Always preview avatars before generating a video to ensure they match user preferences. Each avatar has preview URLs that can be opened directly in the browser - no downloading required.

Quick Preview: Open URL in Browser (Recommended)

The fastest way to preview avatars is to open the URL directly in the default browser. Do not download the image first - just pass the URL to open:

# macOS: Open URL directly in default browser (no download)
open "https://files.heygen.ai/avatar/preview/josh.jpg"

# Open preview video to see animation
open "https://files.heygen.ai/avatar/preview/josh.mp4"

# Linux: Use xdg-open
xdg-open "https://files.heygen.ai/avatar/preview/josh.jpg"

# Windows: Use start
start "https://files.heygen.ai/avatar/preview/josh.jpg"

The open command on macOS opens URLs directly in the default browser - it does not download the file. This is the quickest way to let users see avatar previews.

List Avatars and Open Previews

async function listAndPreviewAvatars(openInBrowser = true): Promise<void> {
  const response = await fetch("https://api.heygen.com/v2/avatars", {
    headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
  });
  const { data } = await response.json();

  for (const avatar of data.avatars.slice(0, 5)) {
    console.log(`\n${avatar.avatar_name} (${avatar.gender})`);
    console.log(`  ID: ${avatar.avatar_id}`);
    console.log(`  Preview: ${avatar.preview_image_url}`);
  }

  // Open preview URLs directly in browser (no download needed)
  if (openInBrowser) {
    const { execSync } = require("child_process");
    for (const avatar of data.avatars.slice(0, 3)) {
      // 'open' on macOS opens the URL in default browser - doesn't download
      execSync(`open "${avatar.preview_image_url}"`);
    }
  }
}

Note: The open command passes the URL to the browser - it does not download. The browser fetches and displays the image directly.

Workflow: Preview Before Generate

1. List available avatars - get names, genders, and preview URLs 2. Open previews in browser - open <preview_image_url> for quick visual check 3. User selects preferred avatar by name or ID 4. Get avatar details for default_voice_id 5. Generate video with selected avatar

# Example workflow in terminal
# 1. List avatars (agent shows options)
# 2. Open preview for candidate
open "https://files.heygen.ai/avatar/preview/josh.jpg"
# 3. User says "use Josh"
# 4. Agent gets details and generates

Preview Fields in API Response

Both URLs are publicly accessible - no authentication needed to view.

Listing Available Avatars

curl

curl -X GET "https://api.heygen.com/v2/avatars" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

TypeScript

interface Avatar {
  avatar_id: string;
  avatar_name: string;
  gender: "male" | "female";
  preview_image_url: string;
  preview_video_url: string;
}

interface AvatarsResponse {
  error: null | string;
  data: {
    avatars: Avatar[];
    talking_photos: TalkingPhoto[];
  };
}

async function listAvatars(): Promise<Avatar[]> {
  const response = await fetch("https://api.heygen.com/v2/avatars", {
    headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
  });

  const json: AvatarsResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data.avatars;
}

Python

import requests
import os

def list_avatars() -> list:
    response = requests.get(
        "https://api.heygen.com/v2/avatars",
        headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
    )

    data = response.json()
    if data.get("error"):
        raise Exception(data["error"])

    return data["data"]["avatars"]

Response Format

{
  "error": null,
  "data": {
    "avatars": [
      {
        "avatar_id": "josh_lite3_20230714",
        "avatar_name": "Josh",
        "gender": "male",
        "preview_image_url": "https://files.heygen.ai/...",
        "preview_video_url": "https://files.heygen.ai/..."
      },
      {
        "avatar_id": "angela_expressive_20231010",
        "avatar_name": "Angela",
        "gender": "female",
        "preview_image_url": "https://files.heygen.ai/...",
        "preview_video_url": "https://files.heygen.ai/..."
      }
    ],
    "talking_photos": []
  }
}

Avatar Types

Public Avatars

HeyGen provides a library of public avatars that anyone can use:

// List only public avatars
const avatars = await listAvatars();
const publicAvatars = avatars.filter((a) => !a.avatar_id.startsWith("custom_"));

Private/Custom Avatars

Custom avatars created from your own training footage:

const customAvatars = avatars.filter((a) => a.avatar_id.startsWith("custom_"));

Instant Avatars

Quick avatars created from a single photo or short video:

// List instant avatars
const response = await fetch("https://api.heygen.com/v1/instant_avatar.list", {
  headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});

const { data } = await response.json();
console.log(data.avatars);

Avatar Styles

Avatars support different rendering styles:

When to Use Each Style

Using Avatar Styles

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal", // "normal" | "closeUp" | "circle" | "voice_only"
      },
      voice: {
        type: "text",
        input_text: "Hello, world!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
    },
  ],
};

Circle Style for Talking Heads

Circle style is ideal for overlay compositions:

// Circle avatar for picture-in-picture
{
  character: {
    type: "avatar",
    avatar_id: "josh_lite3_20230714",
    avatar_style: "circle",
  },
  voice: { ... },
  background: {
    type: "color",
    value: "#00FF00", // Green for chroma key, or use webm endpoint
  },
}

Searching and Filtering Avatars

By Gender

function filterByGender(avatars: Avatar[], gender: "male" | "female"): Avatar[] {
  return avatars.filter((a) => a.gender === gender);
}

const maleAvatars = filterByGender(avatars, "male");
const femaleAvatars = filterByGender(avatars, "female");

By Name

function searchByName(avatars: Avatar[], query: string): Avatar[] {
  const lowerQuery = query.toLowerCase();
  return avatars.filter((a) =>
    a.avatar_name.toLowerCase().includes(lowerQuery)
  );
}

const results = searchByName(avatars, "josh");

Avatar Groups

Avatars are organized into groups for better management. Use the v3 APIs for paginated listing with search capabilities.

List Avatar Groups (v3 - Recommended)

The v3 API provides pagination and search capabilities:

curl -X GET "https://api.heygen.com/v3/avatar_group.list?page=1&page_size=20&include_public=true" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

Query Parameters

TypeScript

interface AvatarGroupItem {
  id: string;
  name: string;
  created_at: number;
  num_looks: number;
  preview_image: string;
  group_type: string;
  train_status: string;
  default_voice_id: string | null;
}

interface AvatarGroupListResponse {
  error: null | string;
  data: {
    avatar_group_list: AvatarGroupItem[];
    total_count: number;
    current_page: number;
    page_size: number;
  };
}

async function listAvatarGroups(
  page = 1,
  pageSize = 20,
  includePublic = true,
  query?: string
): Promise<AvatarGroupListResponse["data"]> {
  const params = new URLSearchParams({
    page: page.toString(),
    page_size: pageSize.toString(),
    include_public: includePublic.toString(),
  });

  if (query) {
    params.set("query", query);
  }

  const response = await fetch(
    `https://api.heygen.com/v3/avatar_group.list?${params}`,
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  const json: AvatarGroupListResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data;
}

Search Public Avatars (v3)

Search and filter public avatars with pagination:

curl -X GET "https://api.heygen.com/v3/avatar_groups/search?page=1&page_size=20&query=professional" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

Query Parameters

TypeScript

async function searchPublicAvatars(
  page = 1,
  pageSize = 20,
  query?: string,
  searchTags?: string[]
): Promise<AvatarGroupListResponse["data"]> {
  const params = new URLSearchParams({
    page: page.toString(),
    page_size: pageSize.toString(),
  });

  if (query) {
    params.set("query", query);
  }

  if (searchTags?.length) {
    params.set("search_tags", searchTags.join(","));
  }

  const response = await fetch(
    `https://api.heygen.com/v3/avatar_groups/search?${params}`,
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  const json: AvatarGroupListResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data;
}

// Usage examples
const businessAvatars = await searchPublicAvatars(1, 20, "business");
const femaleAvatars = await searchPublicAvatars(1, 20, undefined, ["female"]);

List Avatar Groups (v2 - Legacy)

curl -X GET "https://api.heygen.com/v2/avatar_group.list" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

Get Avatars in a Group

curl -X GET "https://api.heygen.com/v2/avatar_group/{group_id}/avatars" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

Using Avatars in Video Generation

Basic Avatar Usage

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Welcome to our product demo!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
    },
  ],
  dimension: { width: 1920, height: 1080 },
};

Multiple Scenes with Different Avatars

const multiSceneConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Hi, I'm Josh. Let me introduce my colleague.",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
    },
    {
      character: {
        type: "avatar",
        avatar_id: "angela_expressive_20231010",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Hello! I'm Angela. Nice to meet you!",
        voice_id: "2d5b0e6a8c3f47d9a1b2c3d4e5f60718",
      },
    },
  ],
};

Using Avatar's Default Voice

Many avatars have a default_voice_id that's pre-matched for natural results. This is the recommended approach rather than manually selecting voices.

Recommended Flow

1. GET /v2/avatars           → Get list of avatar_ids
2. GET /v2/avatar/{id}/details → Get default_voice_id for chosen avatar
3. POST /v2/video/generate   → Use avatar_id + default_voice_id

Get Avatar Details (v2 API)

Given an avatar_id, fetch its details including the default voice:

curl -X GET "https://api.heygen.com/v2/avatar/{avatar_id}/details" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

Response Format

{
  "error": null,
  "data": {
    "type": "avatar",
    "id": "josh_lite3_20230714",
    "name": "Josh",
    "gender": "male",
    "preview_image_url": "https://files.heygen.ai/...",
    "preview_video_url": "https://files.heygen.ai/...",
    "premium": false,
    "is_public": true,
    "default_voice_id": "1bd001e7e50f421d891986aad5158bc8",
    "tags": ["AVATAR_IV"]
  }
}

TypeScript

interface AvatarDetails {
  type: "avatar";
  id: string;
  name: string;
  gender: "male" | "female";
  preview_image_url: string;
  preview_video_url: string;
  premium: boolean;
  is_public: boolean;
  default_voice_id: string | null;
  tags: string[];
}

async function getAvatarDetails(avatarId: string): Promise<AvatarDetails> {
  const response = await fetch(
    `https://api.heygen.com/v2/avatar/${avatarId}/details`,
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  const json = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data;
}

// Usage: Get default voice for a known avatar
const details = await getAvatarDetails("josh_lite3_20230714");
if (details.default_voice_id) {
  console.log(`Using ${details.name} with default voice: ${details.default_voice_id}`);
} else {
  console.log(`${details.name} has no default voice, select manually`);
}

Complete Example: Generate Video with Any Avatar's Default Voice

async function generateWithAvatarDefaultVoice(
  avatarId: string,
  script: string
): Promise<string> {
  // 1. Get avatar details to find default voice
  const avatar = await getAvatarDetails(avatarId);

  if (!avatar.default_voice_id) {
    throw new Error(`Avatar ${avatar.name} has no default voice`);
  }

  // 2. Generate video with the avatar's default voice
  const videoId = await generateVideo({
    video_inputs: [{
      character: {
        type: "avatar",
        avatar_id: avatar.id,
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: script,
        voice_id: avatar.default_voice_id,
      },
    }],
    dimension: { width: 1920, height: 1080 },
  });

  return videoId;
}

Why Use Default Voice?

1. Guaranteed gender match - Avatar and voice are pre-paired 2. Natural lip sync - Default voices are optimized for the avatar 3. Simpler code - No need to fetch and match voices separately 4. Better quality - HeyGen has tested this combination

Selecting the Right Avatar

Avatar Categories

HeyGen avatars fall into distinct categories. Match the category to your use case:

Selection Guidelines

For business/professional content:

• Choose avatars with neutral attire (business casual or formal)
• Avoid themed or seasonal avatars (holiday costumes, casual clothing)
• Preview the avatar to verify professional appearance
• Consider your audience demographics when selecting gender and appearance

For casual/social content:

• More flexibility in avatar choice
• Themed avatars can work for specific campaigns
• Match avatar energy to content tone

Common Mistakes to Avoid

1. Using themed avatars for business content - A holiday-themed avatar looks unprofessional in a product demo 2. Not previewing before generation - Always open <preview_url> to verify appearance 3. Ignoring avatar style - A circle style avatar may not work for full-screen presentations 4. Mismatched voice gender - Always use the avatar's default_voice_id or match genders manually

Selection Checklist

Before generating a video:

• [ ] Previewed avatar image/video in browser
• [ ] Avatar appearance matches content tone (professional vs casual)
• [ ] Avatar style (normal, closeUp, circle) fits the video format
• [ ] Voice gender matches avatar gender
• [ ] Using default_voice_id when available

Helper Functions

Get Avatar by ID

async function getAvatarById(avatarId: string): Promise<Avatar | null> {
  const avatars = await listAvatars();
  return avatars.find((a) => a.avatar_id === avatarId) || null;
}

Validate Avatar ID

async function isValidAvatarId(avatarId: string): Promise<boolean> {
  const avatar = await getAvatarById(avatarId);
  return avatar !== null;
}

Get Random Avatar

async function getRandomAvatar(gender?: "male" | "female"): Promise<Avatar> {
  let avatars = await listAvatars();

  if (gender) {
    avatars = avatars.filter((a) => a.gender === gender);
  }

  const randomIndex = Math.floor(Math.random() * avatars.length);
  return avatars[randomIndex];
}

Common Avatar IDs

Some commonly used public avatar IDs (availability may vary):

Always verify avatar availability by calling the list endpoint before using.

Video Backgrounds

HeyGen supports various background types to customize the appearance of your avatar videos.

Background Types

Color Backgrounds

The simplest option - use a solid color:

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Hello with a colored background!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
      background: {
        type: "color",
        value: "#FFFFFF", // White background
      },
    },
  ],
};

Common Color Values

Using Transparent/Green Screen

For compositing in post-production:

background: {
  type: "color",
  value: "#00FF00", // Green screen
}

Image Backgrounds

Use a static image as background:

From URL

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Check out this custom background!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
      background: {
        type: "image",
        url: "https://example.com/my-background.jpg",
      },
    },
  ],
};

From Uploaded Asset

First upload your image, then use the asset URL:

// 1. Upload the image
const assetId = await uploadFile("./background.jpg", "image/jpeg");

// 2. Use in video config
const videoConfig = {
  video_inputs: [
    {
      character: {...},
      voice: {...},
      background: {
        type: "image",
        url: `https://files.heygen.ai/asset/${assetId}`,
      },
    },
  ],
};

Image Requirements

• Formats: JPEG, PNG
• Recommended size: Match video dimensions (e.g., 1920x1080 for 1080p)
• Aspect ratio: Should match video aspect ratio
• File size: Under 10MB recommended

Video Backgrounds

Use a looping video as background:

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Dynamic video background!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
      background: {
        type: "video",
        url: "https://example.com/background-loop.mp4",
      },
    },
  ],
};

Video Requirements

• Format: MP4 (H.264 codec recommended)
• Looping: Video will loop if shorter than avatar content
• Audio: Background video audio is typically muted
• File size: Under 100MB recommended

Different Backgrounds Per Scene

Use different backgrounds for each scene:

const multiBackgroundConfig = {
  video_inputs: [
    // Scene 1: Office background
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Let me start with an introduction.",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
      background: {
        type: "image",
        url: "https://example.com/office-bg.jpg",
      },
    },
    // Scene 2: Product showcase
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "closeUp",
      },
      voice: {
        type: "text",
        input_text: "Now let me show you our product.",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
      background: {
        type: "image",
        url: "https://example.com/product-bg.jpg",
      },
    },
    // Scene 3: Call to action
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Get started today!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
      background: {
        type: "color",
        value: "#1a1a2e",
      },
    },
  ],
};

Background Helper Functions

TypeScript

type BackgroundType = "color" | "image" | "video";

interface Background {
  type: BackgroundType;
  value?: string;
  url?: string;
}

function createColorBackground(hexColor: string): Background {
  return { type: "color", value: hexColor };
}

function createImageBackground(imageUrl: string): Background {
  return { type: "image", url: imageUrl };
}

function createVideoBackground(videoUrl: string): Background {
  return { type: "video", url: videoUrl };
}

// Preset backgrounds
const backgrounds = {
  white: createColorBackground("#FFFFFF"),
  black: createColorBackground("#000000"),
  greenScreen: createColorBackground("#00FF00"),
  corporate: createColorBackground("#0066CC"),
};

Best Practices

1. Match dimensions - Background should match video dimensions 2. Consider avatar position - Leave space where avatar will appear 3. Use contrasting colors - Ensure avatar is visible against background 4. Optimize file sizes - Compress images/videos for faster processing 5. Test with green screen - For professional post-production workflows 6. Keep backgrounds simple - Avoid distracting elements behind the avatar

Common Issues

Background Not Showing

// Wrong: missing url/value
background: {
  type: "image"
}

// Correct
background: {
  type: "image",
  url: "https://example.com/bg.jpg"
}

Aspect Ratio Mismatch

If your background doesn't match the video dimensions, it may be cropped or stretched. Always match your background aspect ratio to your video dimensions:

// For 1920x1080 video
// Use 1920x1080 background image

// For 1080x1920 portrait video
// Use 1080x1920 background image

Video Background Audio

Background video audio is typically muted to avoid conflicting with the avatar's voice. If you need background music, add it as a separate audio track in post-production.

Video Captions

HeyGen can automatically generate captions (subtitles) for your videos, improving accessibility and engagement.

Enabling Captions

Captions can be enabled when generating a video:

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "Hello! This video will have automatic captions.",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
    },
  ],
  // Caption settings (availability varies by plan)
  caption: true,
};

Caption Configuration Options

interface CaptionConfig {
  // Enable/disable captions
  enabled: boolean;

  // Caption style
  style?: {
    font_family?: string;
    font_size?: number;
    font_color?: string;
    background_color?: string;
    position?: "top" | "bottom";
  };

  // Language for caption generation
  language?: string;
}

Caption Styles

Basic Captions

const config = {
  video_inputs: [...],
  caption: true, // Enable with default styling
};

Styled Captions

const config = {
  video_inputs: [...],
  caption: {
    enabled: true,
    style: {
      font_family: "Arial",
      font_size: 32,
      font_color: "#FFFFFF",
      background_color: "rgba(0, 0, 0, 0.7)",
      position: "bottom",
    },
  },
};

Multi-Language Captions

For videos in different languages, captions are generated based on the voice language:

// Spanish video with Spanish captions
const spanishConfig = {
  video_inputs: [
    {
      character: {
        type: "avatar",
        avatar_id: "josh_lite3_20230714",
        avatar_style: "normal",
      },
      voice: {
        type: "text",
        input_text: "¡Hola! Este video tendrá subtítulos en español.",
        voice_id: "spanish_voice_id",
      },
    },
  ],
  caption: true,
};

Working with SRT Files

Downloading SRT

After video generation, you may be able to download the SRT file:

async function downloadSrt(videoId: string): Promise<string> {
  const response = await fetch(
    `https://api.heygen.com/v1/video/${videoId}/srt`,
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  if (!response.ok) {
    throw new Error("Failed to download SRT");
  }

  return response.text();
}

SRT File Format

Standard SRT format:

1
00:00:00,000 --> 00:00:03,000
Hello! This video will have

2
00:00:03,000 --> 00:00:06,000
automatic captions generated.

3
00:00:06,000 --> 00:00:09,000
They sync with the audio.

Using Custom SRT

For video translation, you can provide your own SRT:

const translationConfig = {
  input_video_id: "original_video_id",
  output_languages: ["es-ES", "fr-FR"],
  srt_key: "path/to/custom.srt", // Custom SRT file
  srt_role: "input", // "input" or "output"
};

Caption Positioning

Bottom (Default)

Standard position for most videos:

caption: {
  enabled: true,
  style: {
    position: "bottom"
  }
}

Top

For videos where bottom space is occupied:

caption: {
  enabled: true,
  style: {
    position: "top"
  }
}

Accessibility Best Practices

1. Always enable captions - Improves accessibility for deaf/hard-of-hearing viewers 2. Use high contrast - White text on dark background or vice versa 3. Readable font size - At least 24px for standard video, larger for mobile 4. Don't cover important content - Position captions away from key visual elements 5. Sync timing - Ensure captions match audio timing accurately

Caption Helper Functions

interface CaptionStyle {
  font_family: string;
  font_size: number;
  font_color: string;
  background_color: string;
  position: "top" | "bottom";
}

const captionPresets: Record<string, CaptionStyle> = {
  default: {
    font_family: "Arial",
    font_size: 32,
    font_color: "#FFFFFF",
    background_color: "rgba(0, 0, 0, 0.7)",
    position: "bottom",
  },
  minimal: {
    font_family: "Arial",
    font_size: 28,
    font_color: "#FFFFFF",
    background_color: "transparent",
    position: "bottom",
  },
  bold: {
    font_family: "Arial",
    font_size: 36,
    font_color: "#FFFFFF",
    background_color: "rgba(0, 0, 0, 0.9)",
    position: "bottom",
  },
  branded: {
    font_family: "Roboto",
    font_size: 30,
    font_color: "#00D1FF",
    background_color: "rgba(26, 26, 46, 0.9)",
    position: "bottom",
  },
};

function createCaptionConfig(preset: keyof typeof captionPresets) {
  return {
    enabled: true,
    style: captionPresets[preset],
  };
}

Social Media Caption Considerations

TikTok / Instagram Reels

• Position captions in center or upper portion
• Avoid bottom 20% (covered by UI elements)
• Use larger font sizes for mobile viewing

const socialCaptions = {
  enabled: true,
  style: {
    font_size: 42,
    position: "top", // Avoid bottom UI elements
  },
};

YouTube

• Standard bottom captions work well
• YouTube also supports closed captions upload

LinkedIn

• Captions highly recommended (many watch without sound)
• Professional styling preferred

Limitations

• Caption styles may be limited depending on your subscription tier
• Some advanced caption features may require the web interface
• Multi-speaker caption detection may have limited availability
• Caption accuracy depends on audio quality and speech clarity

Integration with Video Translation

When using video translation, captions are automatically handled:

// Video translation includes caption generation
const translationConfig = {
  input_video_id: "original_video_id",
  output_languages: ["es-ES"],
  // Captions generated in target language
};

See video-translation.md for more details.

Video Dimensions and Resolution

HeyGen supports various video dimensions and aspect ratios to fit different platforms and use cases.

Standard Resolutions

Landscape (16:9)

Portrait (9:16)

Square (1:1)

Setting Dimensions

TypeScript

// Landscape 1080p
const landscapeConfig = {
  video_inputs: [...],
  dimension: {
    width: 1920,
    height: 1080
  }
};

// Portrait 1080p
const portraitConfig = {
  video_inputs: [...],
  dimension: {
    width: 1080,
    height: 1920
  }
};

// Square 1080p
const squareConfig = {
  video_inputs: [...],
  dimension: {
    width: 1080,
    height: 1080
  }
};

curl

# Landscape 1080p
curl -X POST "https://api.heygen.com/v2/video/generate" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "video_inputs": [...],
    "dimension": {
      "width": 1920,
      "height": 1080
    }
  }'

Dimension Helper Functions

type AspectRatio = "16:9" | "9:16" | "1:1" | "4:3" | "4:5";
type Quality = "720p" | "1080p";

interface Dimensions {
  width: number;
  height: number;
}

function getDimensions(aspectRatio: AspectRatio, quality: Quality): Dimensions {
  const configs: Record<AspectRatio, Record<Quality, Dimensions>> = {
    "16:9": {
      "720p": { width: 1280, height: 720 },
      "1080p": { width: 1920, height: 1080 },
    },
    "9:16": {
      "720p": { width: 720, height: 1280 },
      "1080p": { width: 1080, height: 1920 },
    },
    "1:1": {
      "720p": { width: 720, height: 720 },
      "1080p": { width: 1080, height: 1080 },
    },
    "4:3": {
      "720p": { width: 960, height: 720 },
      "1080p": { width: 1440, height: 1080 },
    },
    "4:5": {
      "720p": { width: 576, height: 720 },
      "1080p": { width: 864, height: 1080 },
    },
  };

  return configs[aspectRatio][quality];
}

// Usage
const youTubeDimensions = getDimensions("16:9", "1080p");
const tikTokDimensions = getDimensions("9:16", "1080p");
const instagramDimensions = getDimensions("1:1", "1080p");

Platform-Specific Recommendations

YouTube

const youtubeConfig = {
  video_inputs: [...],
  dimension: { width: 1920, height: 1080 }, // 16:9 landscape
};

TikTok / Instagram Reels / YouTube Shorts

const shortFormConfig = {
  video_inputs: [...],
  dimension: { width: 1080, height: 1920 }, // 9:16 portrait
};

Instagram Feed Post

const instagramFeedConfig = {
  video_inputs: [...],
  dimension: { width: 1080, height: 1080 }, // 1:1 square
};

LinkedIn

const linkedinConfig = {
  video_inputs: [...],
  dimension: { width: 1920, height: 1080 }, // 16:9 landscape preferred
};

Twitter/X

const twitterConfig = {
  video_inputs: [...],
  dimension: { width: 1280, height: 720 }, // 16:9, 720p is common
};

Avatar IV Dimensions

For Avatar IV (photo-based avatars), dimensions are set via orientation:

type VideoOrientation = "portrait" | "landscape" | "square";

function getAvatarIVDimensions(orientation: VideoOrientation): Dimensions {
  switch (orientation) {
    case "portrait":
      return { width: 720, height: 1280 };
    case "landscape":
      return { width: 1280, height: 720 };
    case "square":
      return { width: 720, height: 720 };
  }
}

Custom Dimensions

HeyGen supports custom dimensions within limits:

const customConfig = {
  video_inputs: [...],
  dimension: {
    width: 1600,
    height: 900  // Custom 16:9 at non-standard resolution
  }
};

Dimension Constraints

• Minimum: 128px on any side
• Maximum: 4096px on any side
• Must be even numbers: Both width and height must be divisible by 2

function validateDimensions(width: number, height: number): boolean {
  if (width < 128 || height < 128) {
    throw new Error("Dimensions must be at least 128px");
  }
  if (width > 4096 || height > 4096) {
    throw new Error("Dimensions cannot exceed 4096px");
  }
  if (width % 2 !== 0 || height % 2 !== 0) {
    throw new Error("Dimensions must be even numbers");
  }
  return true;
}

Resolution vs. Credit Cost

Higher resolutions may consume more credits:

Consider using 720p for drafts and testing, then 1080p for final output.

Background Considerations

Match background image/video dimensions to your video dimensions:

// For 1080p landscape video
const config = {
  video_inputs: [
    {
      character: {...},
      voice: {...},
      background: {
        type: "image",
        url: "https://example.com/1920x1080-background.jpg" // Match video dimensions
      }
    }
  ],
  dimension: { width: 1920, height: 1080 }
};

Creating a Video Config Factory

interface VideoConfigOptions {
  script: string;
  avatarId: string;
  voiceId: string;
  platform: "youtube" | "tiktok" | "instagram_feed" | "instagram_story" | "linkedin";
  quality?: "720p" | "1080p";
}

function createVideoConfig(options: VideoConfigOptions) {
  const platformDimensions: Record<string, Dimensions> = {
    youtube: { width: 1920, height: 1080 },
    tiktok: { width: 1080, height: 1920 },
    instagram_feed: { width: 1080, height: 1080 },
    instagram_story: { width: 1080, height: 1920 },
    linkedin: { width: 1920, height: 1080 },
  };

  const dimension = platformDimensions[options.platform];

  // Scale down for 720p if requested
  if (options.quality === "720p") {
    dimension.width = Math.round((dimension.width * 720) / 1080);
    dimension.height = Math.round((dimension.height * 720) / 1080);
  }

  return {
    video_inputs: [
      {
        character: {
          type: "avatar",
          avatar_id: options.avatarId,
          avatar_style: "normal",
        },
        voice: {
          type: "text",
          input_text: options.script,
          voice_id: options.voiceId,
        },
      },
    ],
    dimension,
  };
}

// Usage
const tiktokVideo = createVideoConfig({
  script: "Hey everyone! Check this out!",
  avatarId: "josh_lite3_20230714",
  voiceId: "1bd001e7e50f421d891986aad5158bc8",
  platform: "tiktok",
  quality: "1080p",
});

Photo Avatars (Talking Photos)

Photo avatars allow you to animate a static photo and make it speak. This is useful for creating personalized video content from portraits, headshots, or any suitable image.

Overview

HeyGen offers several approaches to photo-based avatars:

Uploading a Photo for Talking Photo

Step 1: Upload the Image

// Upload photo as an asset
const assetId = await uploadFile("./portrait.jpg", "image/jpeg");

Step 2: Create Talking Photo

curl -X POST "https://api.heygen.com/v2/talking_photo" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image_url": "https://files.heygen.ai/asset/your_asset_id"
  }'

TypeScript

interface TalkingPhotoResponse {
  error: null | string;
  data: {
    talking_photo_id: string;
  };
}

async function createTalkingPhoto(imageUrl: string): Promise<string> {
  const response = await fetch("https://api.heygen.com/v2/talking_photo", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ image_url: imageUrl }),
  });

  const json: TalkingPhotoResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data.talking_photo_id;
}

Using Talking Photo in Video

const videoConfig = {
  video_inputs: [
    {
      character: {
        type: "talking_photo",
        talking_photo_id: "your_talking_photo_id",
      },
      voice: {
        type: "text",
        input_text: "Hello! This is my talking photo speaking!",
        voice_id: "1bd001e7e50f421d891986aad5158bc8",
      },
    },
  ],
  dimension: { width: 1920, height: 1080 },
};

const videoId = await generateVideo(videoConfig);

Avatar IV API

Avatar IV is HeyGen's latest photo avatar technology with improved quality and natural motion.

Generate Avatar IV Video

curl -X POST "https://api.heygen.com/v2/video/av4/generate" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "photo_s3_key": "path/to/uploaded/photo.jpg",
    "script": "Hello! This is Avatar IV with enhanced quality.",
    "voice_id": "1bd001e7e50f421d891986aad5158bc8",
    "video_orientation": "portrait",
    "video_title": "My Avatar IV Video"
  }'

TypeScript

interface AvatarIVRequest {
  photo_s3_key: string;
  script: string;
  voice_id: string;
  video_orientation?: "portrait" | "landscape" | "square";
  video_title?: string;
  fit?: "cover" | "contain";
  custom_motion_prompt?: string;
  enhance_custom_motion_prompt?: boolean;
}

interface AvatarIVResponse {
  error: null | string;
  data: {
    video_id: string;
  };
}

async function generateAvatarIVVideo(config: AvatarIVRequest): Promise<string> {
  const response = await fetch(
    "https://api.heygen.com/v2/video/av4/generate",
    {
      method: "POST",
      headers: {
        "X-Api-Key": process.env.HEYGEN_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(config),
    }
  );

  const json: AvatarIVResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data.video_id;
}

Avatar IV Options

Video Orientation

Fit Options

Custom Motion Prompts

Add specific motion or expression:

const config = {
  photo_s3_key: "path/to/photo.jpg",
  script: "Let me tell you about our product.",
  voice_id: "voice_id",
  custom_motion_prompt: "nodding head and smiling",
  enhance_custom_motion_prompt: true,
};

Photo Avatar Groups

Organize multiple photo looks:

Create Photo Avatar Group

interface PhotoAvatarGroupRequest {
  image_key: string;
  name: string;
  generation_id?: string;
}

async function createPhotoAvatarGroup(
  imageKey: string,
  name: string
): Promise<string> {
  const response = await fetch(
    "https://api.heygen.com/v2/photo_avatar/avatar_group/create",
    {
      method: "POST",
      headers: {
        "X-Api-Key": process.env.HEYGEN_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ image_key: imageKey, name }),
    }
  );

  const json = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data.id;
}

Add Photos to Group

async function addPhotosToGroup(
  groupId: string,
  imageKeys: string[],
  name: string
): Promise<void> {
  const response = await fetch(
    "https://api.heygen.com/v2/photo_avatar/avatar_group/add",
    {
      method: "POST",
      headers: {
        "X-Api-Key": process.env.HEYGEN_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        group_id: groupId,
        image_keys: imageKeys,
        name,
      }),
    }
  );

  const json = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }
}

Photo Requirements

For best results, use photos that meet these criteria:

Technical Requirements

Quality Guidelines

1. Lighting - Even, natural lighting on face 2. Expression - Neutral or slight smile 3. Background - Simple, uncluttered 4. Face position - Centered, not cut off 5. Clarity - Sharp, in focus 6. Angle - Straight-on or slight angle

Generating AI Photo Avatars

Generate synthetic photo avatars from text descriptions.

IMPORTANT: All 8 fields are REQUIRED. The API will reject requests missing any field.

You cannot simply provide a text description - you MUST specify each enum field explicitly.

When a user asks to "generate an AI avatar of a professional man", you need to ask for or select values for ALL fields below.

Required Fields (ALL must be provided)

curl Example

curl -X POST "https://api.heygen.com/v2/photo_avatar/photo/generate" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Sarah Product Demo",
    "age": "Young Adult",
    "gender": "Woman",
    "ethnicity": "White",
    "orientation": "horizontal",
    "pose": "half_body",
    "style": "Realistic",
    "appearance": "Professional woman with a friendly smile, wearing a navy blue blazer over a white blouse, soft studio lighting, clean neutral background"
  }'

TypeScript

// All fields are REQUIRED - the API will reject requests with missing fields
interface GeneratePhotoAvatarRequest {
  name: string;                    // Name for the avatar
  age: "Young Adult" | "Early Middle Age" | "Late Middle Age" | "Senior" | "Unspecified";
  gender: "Woman" | "Man" | "Unspecified";
  ethnicity: "White" | "Black" | "Asian American" | "East Asian" | "South East Asian" | "South Asian" | "Middle Eastern" | "Pacific" | "Hispanic" | "Unspecified";
  orientation: "square" | "horizontal" | "vertical";
  pose: "half_body" | "close_up" | "full_body";
  style: "Realistic" | "Pixar" | "Cinematic" | "Vintage" | "Noir" | "Cyberpunk" | "Unspecified";
  appearance: string;              // Max 1000 characters
  callback_url?: string;           // Optional: webhook for completion notification
  callback_id?: string;            // Optional: custom ID for tracking
}

interface GeneratePhotoAvatarResponse {
  error: string | null;
  data: {
    generation_id: string;
  };
}

async function generatePhotoAvatar(
  config: GeneratePhotoAvatarRequest
): Promise<string> {
  const response = await fetch(
    "https://api.heygen.com/v2/photo_avatar/photo/generate",
    {
      method: "POST",
      headers: {
        "X-Api-Key": process.env.HEYGEN_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(config),
    }
  );

  const json: GeneratePhotoAvatarResponse = await response.json();

  if (json.error) {
    throw new Error(`Photo avatar generation failed: ${json.error}`);
  }

  return json.data.generation_id;
}

Example: Generate Professional Avatar

const generationId = await generatePhotoAvatar({
  name: "Tech Demo Presenter",
  age: "Early Middle Age",
  gender: "Man",
  ethnicity: "East Asian",
  orientation: "horizontal",
  pose: "half_body",
  style: "Realistic",
  appearance: "Professional man in a modern office setting, wearing a dark gray suit with no tie, confident and approachable expression, soft natural lighting from a window, clean minimalist background"
});

console.log(`Generation started: ${generationId}`);
// Save generationId to poll for status later

Check Generation Status

interface PhotoGenerationStatus {
  error: string | null;
  data: {
    status: "pending" | "processing" | "completed" | "failed";
    image_url?: string;        // Available when completed
    image_key?: string;        // S3 key for use in video generation
  };
}

async function checkPhotoGeneration(generationId: string): Promise<PhotoGenerationStatus> {
  const response = await fetch(
    `https://api.heygen.com/v2/photo_avatar/generation/${generationId}`,
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  return response.json();
}

// Poll for completion
async function waitForPhotoGeneration(generationId: string): Promise<string> {
  const maxAttempts = 60;
  const pollIntervalMs = 5000; // 5 seconds

  for (let i = 0; i < maxAttempts; i++) {
    const status = await checkPhotoGeneration(generationId);

    if (status.error) {
      throw new Error(status.error);
    }

    if (status.data.status === "completed") {
      return status.data.image_key!;
    }

    if (status.data.status === "failed") {
      throw new Error("Photo generation failed");
    }

    console.log(`Status: ${status.data.status}, waiting...`);
    await new Promise(r => setTimeout(r, pollIntervalMs));
  }

  throw new Error("Photo generation timed out");
}

Pre-Generation Checklist

Before calling the API, ensure you have values for ALL fields:

If the user only provides a vague request like "create a professional looking man", ask them to specify the missing fields OR make reasonable defaults (e.g., "Early Middle Age", "Realistic" style, "half_body" pose, "horizontal" orientation).

Appearance Prompt Tips

The appearance field is a text prompt - be descriptive:

Good prompts:

• "Professional woman with shoulder-length brown hair, wearing a light blue button-down shirt, warm friendly smile, soft studio lighting, clean white background"
• "Young man with short black hair, casual tech startup style, wearing a dark hoodie, confident expression, modern office background with plants"

Avoid:

• Vague descriptions: "a nice person"
• Conflicting attributes
• Requesting specific real people

Complete Workflow: Photo to Video

async function createVideoFromPhoto(
  photoPath: string,
  script: string,
  voiceId: string
): Promise<string> {
  // 1. Upload photo
  console.log("Uploading photo...");
  const assetId = await uploadFile(photoPath, "image/jpeg");

  // 2. Create talking photo
  console.log("Creating talking photo...");
  const talkingPhotoId = await createTalkingPhoto(
    `https://files.heygen.ai/asset/${assetId}`
  );

  // 3. Generate video
  console.log("Generating video...");
  const videoId = await generateVideo({
    video_inputs: [
      {
        character: {
          type: "talking_photo",
          talking_photo_id: talkingPhotoId,
        },
        voice: {
          type: "text",
          input_text: script,
          voice_id: voiceId,
        },
      },
    ],
    dimension: { width: 1920, height: 1080 },
  });

  // 4. Wait for completion
  console.log("Processing video...");
  const videoUrl = await waitForVideo(videoId);

  return videoUrl;
}

Managing Photo Avatars

Get Photo Avatar Details

async function getPhotoAvatar(id: string) {
  const response = await fetch(
    `https://api.heygen.com/v2/photo_avatar/${id}`,
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  return response.json();
}

Delete Photo Avatar

async function deletePhotoAvatar(id: string): Promise<void> {
  const response = await fetch(
    `https://api.heygen.com/v2/photo_avatar/${id}`,
    {
      method: "DELETE",
      headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
    }
  );

  if (!response.ok) {
    throw new Error("Failed to delete photo avatar");
  }
}

Best Practices

1. Use high-quality photos - Better input = better output 2. Front-facing portraits - Work best for animation 3. Neutral expressions - Allow for more natural animation 4. Test different photos - Results vary by image 5. Consider Avatar IV - For highest quality results 6. Organize with groups - Keep photo avatars organized

Limitations

• Photo quality significantly affects output
• Side-profile photos have limited support
• Full-body photos may not animate properly
• Some expressions may look unnatural
• Processing time varies by complexity

HeyGen Quota and Credits

HeyGen uses a credit-based system for video generation. Understanding quota management helps prevent failed video generation requests.

Checking Remaining Quota

curl

curl -X GET "https://api.heygen.com/v1/video_generate.quota" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

TypeScript

interface QuotaResponse {
  error: null | string;
  data: {
    remaining_quota: number;
    used_quota: number;
  };
}

const response = await fetch("https://api.heygen.com/v1/video_generate.quota", {
  headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});

const { data }: QuotaResponse = await response.json();
console.log(`Remaining credits: ${data.remaining_quota}`);

Python

import requests
import os

response = requests.get(
    "https://api.heygen.com/v1/video_generate.quota",
    headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)

data = response.json()["data"]
print(f"Remaining credits: {data['remaining_quota']}")

Response Format

{
  "error": null,
  "data": {
    "remaining_quota": 450,
    "used_quota": 50
  }
}

Credit Consumption

Different operations consume different amounts of credits:

Pre-Generation Quota Check

Always verify sufficient quota before generating videos:

async function generateVideoWithQuotaCheck(videoConfig: VideoConfig) {
  // Check quota first
  const quotaResponse = await fetch(
    "https://api.heygen.com/v1/video_generate.quota",
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  const { data: quota } = await quotaResponse.json();

  // Estimate required credits (rough estimate: 1 credit per minute)
  const estimatedMinutes = videoConfig.estimatedDuration / 60;
  const requiredCredits = Math.ceil(estimatedMinutes);

  if (quota.remaining_quota < requiredCredits) {
    throw new Error(
      `Insufficient credits. Need ${requiredCredits}, have ${quota.remaining_quota}`
    );
  }

  // Proceed with video generation
  return generateVideo(videoConfig);
}

Quota Management Best Practices

1. Monitor Usage Regularly

async function logQuotaUsage() {
  const response = await fetch(
    "https://api.heygen.com/v1/video_generate.quota",
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  const { data } = await response.json();

  console.log({
    remaining: data.remaining_quota,
    used: data.used_quota,
    percentUsed: (
      (data.used_quota / (data.remaining_quota + data.used_quota)) *
      100
    ).toFixed(1),
  });
}

2. Set Up Alerts

const QUOTA_WARNING_THRESHOLD = 50;

async function checkQuotaWithAlert() {
  const response = await fetch(
    "https://api.heygen.com/v1/video_generate.quota",
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  const { data } = await response.json();

  if (data.remaining_quota < QUOTA_WARNING_THRESHOLD) {
    // Send alert (email, Slack, etc.)
    await sendAlert(`Low HeyGen quota: ${data.remaining_quota} credits remaining`);
  }

  return data;
}

3. Use Test Mode for Development

When available, use test mode to avoid consuming credits during development:

const videoConfig = {
  test: true, // Use test mode during development
  video_inputs: [...],
};

// Test videos may have watermarks but don't consume credits

Subscription Tiers

Different subscription tiers have different quota allocations and features:

API access typically requires Enterprise tier or higher.

Error Handling for Quota Issues

async function handleQuotaError(error: any) {
  if (error.message.includes("quota") || error.message.includes("credit")) {
    console.error("Quota exceeded. Consider:");
    console.error("1. Upgrading your subscription");
    console.error("2. Waiting for quota reset");
    console.error("3. Purchasing additional credits");

    // Check current quota
    const quota = await getQuota();
    console.error(`Current remaining: ${quota.remaining_quota}`);
  }

  throw error;
}

HeyGen + Remotion Integration

This guide covers workflows for generating HeyGen avatar videos and using them in Remotion compositions.

Quick Start

// 1. Get avatar with default voice
const avatar = await getAvatarDetails(avatarId);

// 2. Generate video (MP4 with background - most common)
const videoId = await generateVideo({
  video_inputs: [{
    character: { type: "avatar", avatar_id: avatar.id, avatar_style: "normal" },
    voice: { type: "text", input_text: script, voice_id: avatar.default_voice_id },
    background: { type: "color", value: "#1a1a2e" },
  }],
  dimension: { width: 1920, height: 1080 },
});

// 3. Poll for completion (10-15+ min)
// 4. Use in Remotion with motion graphics overlaid on top

Overview

A typical workflow: 1. Generate avatar video with HeyGen 2. Wait for completion and get video URL 3. Download or use URL directly in Remotion 4. Compose with other elements (backgrounds, overlays, animations)

Choosing the Right Output Format

Use MP4 with background for most cases. Use WebM when you need to see content behind the avatar.

Note: WebM only supports normal and closeUp styles. For circular framing, use CSS border-radius: 50% in Remotion.

Recommended: Parallel Development Workflow

HeyGen video generation takes 10-15+ minutes. Don't wait - work in parallel:

1. Start HeyGen generation - save video_id to a file, exit immediately 2. Build Remotion composition - use a placeholder or the avatar's preview_video_url (a short loop) 3. Check HeyGen status periodically or when done building 4. Swap placeholder for real video URL once ready

Estimate duration from script: ~150 words/minute speech rate, so wordCount / 150 * 60 * fps gives approximate frames.

Composition tip: Design components to work with or without the avatar video, so motion graphics can be tested independently.

Dimension Alignment

Critical: Match HeyGen output dimensions to your Remotion composition.

Common Dimension Presets

// Shared dimension constants for both HeyGen and Remotion
const DIMENSIONS = {
  landscape_1080p: { width: 1920, height: 1080 },
  landscape_720p: { width: 1280, height: 720 },
  portrait_1080p: { width: 1080, height: 1920 },
  portrait_720p: { width: 720, height: 1280 },
  square_1080p: { width: 1080, height: 1080 },
  square_720p: { width: 720, height: 720 },
} as const;

type DimensionPreset = keyof typeof DIMENSIONS;

HeyGen Video Generation

// Generate HeyGen video with specific dimensions
async function generateHeyGenVideo(
  script: string,
  avatarId: string,
  voiceId: string,
  preset: DimensionPreset
): Promise<string> {
  const dimension = DIMENSIONS[preset];

  const response = await fetch("https://api.heygen.com/v2/video/generate", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      video_inputs: [
        {
          character: {
            type: "avatar",
            avatar_id: avatarId,
            avatar_style: "normal",
          },
          voice: {
            type: "text",
            input_text: script,
            voice_id: voiceId,
          },
          background: {
            type: "color",
            value: "#00FF00", // Green screen for compositing
          },
        },
      ],
      dimension,
    }),
  });

  const { data } = await response.json();
  return data.video_id;
}

Remotion Composition Setup

// remotion/src/Root.tsx
import { Composition } from "remotion";
import { AvatarComposition } from "./AvatarComposition";

const DIMENSIONS = {
  landscape_1080p: { width: 1920, height: 1080 },
  // ... same as above
};

export const RemotionRoot: React.FC = () => {
  return (
    <>
      <Composition
        id="AvatarVideo"
        component={AvatarComposition}
        durationInFrames={300} // Will be set dynamically
        fps={30}
        width={DIMENSIONS.landscape_1080p.width}
        height={DIMENSIONS.landscape_1080p.height}
        defaultProps={{
          avatarVideoUrl: "",
        }}
      />
    </>
  );
};

Generating Avatar Video for Remotion

Standard: MP4 with Background

Most Remotion compositions work best with MP4 + background. Overlays and motion graphics go on top:

async function generateAvatarForRemotion(
  script: string,
  avatarId: string,
  voiceId: string,
  options: {
    style?: "normal" | "closeUp" | "circle";
    backgroundColor?: string;
  } = {}
): Promise<string> {
  const { style = "normal", backgroundColor = "#1a1a2e" } = options;

  const response = await fetch("https://api.heygen.com/v2/video/generate", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      video_inputs: [{
        character: {
          type: "avatar",
          avatar_id: avatarId,
          avatar_style: style,
        },
        voice: {
          type: "text",
          input_text: script,
          voice_id: voiceId,
        },
        background: {
          type: "color",
          value: backgroundColor,
        },
      }],
      dimension: { width: 1920, height: 1080 },
    }),
  });

  const { data } = await response.json();
  return data.video_id;
}

Transparent Background (WebM)

Only use when you need to see content behind the avatar (e.g., avatar overlaid on screen recording):

// Use /v1/video.webm endpoint for transparent background
// Note: Different structure than /v2/video/generate
const response = await fetch("https://api.heygen.com/v1/video.webm", {
  method: "POST",
  headers: {
    "X-Api-Key": process.env.HEYGEN_API_KEY!,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    avatar_pose_id: avatarPoseId,  // Required: avatar pose ID
    avatar_style: "normal",        // Required: "normal" or "closeUp" only
    input_text: script,            // Required (with voice_id)
    voice_id: voiceId,             // Required (with input_text)
    dimension: { width: 1920, height: 1080 },
  }),
});

Using HeyGen Video in Remotion

Important: Use OffthreadVideo for Frame-Accurate Rendering

Always use `OffthreadVideo` instead of `Video` for HeyGen avatar videos. The basic Video component uses the browser's video decoder which isn't frame-accurate, causing jitter during rendering. OffthreadVideo extracts frames via FFmpeg for smooth, accurate playback.

OffthreadVideo is included in the core remotion package - no additional install needed.

Basic Usage

// remotion/src/AvatarComposition.tsx
import { OffthreadVideo, useVideoConfig } from "remotion";

interface AvatarCompositionProps {
  avatarVideoUrl: string;
}

export const AvatarComposition: React.FC<AvatarCompositionProps> = ({
  avatarVideoUrl,
}) => {
  return (
    <div style={{ flex: 1, backgroundColor: "#1a1a2e" }}>
      <OffthreadVideo
        src={avatarVideoUrl}
        style={{
          width: "100%",
          height: "100%",
          objectFit: "contain",
        }}
      />
    </div>
  );
};

WebM with Transparent Background (Recommended)

Using WebM from /v1/video.webm - no chroma keying needed:

import { OffthreadVideo, AbsoluteFill, Sequence } from "remotion";

export const AvatarWithMotionGraphics: React.FC<{
  avatarWebmUrl: string
}> = ({ avatarWebmUrl }) => {
  return (
    <AbsoluteFill>
      {/* Layer 1: Your background/content */}
      <AbsoluteFill style={{ backgroundColor: "#1a1a2e" }}>
        <YourMotionGraphics />
      </AbsoluteFill>

      {/* Layer 2: Avatar with transparent background - use OffthreadVideo for frame-accurate rendering */}
      <OffthreadVideo
        src={avatarWebmUrl}
        transparent
        style={{
          position: "absolute",
          bottom: 0,
          right: 0,
          width: "50%",
          height: "auto",
        }}
      />

      {/* Layer 3: Overlays on top of avatar */}
      <Sequence from={30}>
        <AnimatedTitle text="Welcome!" />
      </Sequence>
    </AbsoluteFill>
  );
};

Loom-Style: Circle Avatar Over Screen Recording

Use closeUp style + WebM, then apply circular mask in Remotion:

import { OffthreadVideo, AbsoluteFill } from "remotion";

export const LoomStyleComposition: React.FC<{
  screenRecordingUrl: string;
  avatarWebmUrl: string; // Generated with avatar_style: "closeUp" via /v1/video.webm
}> = ({ screenRecordingUrl, avatarWebmUrl }) => {
  return (
    <AbsoluteFill>
      {/* Screen recording fills the frame */}
      <OffthreadVideo src={screenRecordingUrl} style={{ width: "100%", height: "100%" }} />

      {/* Avatar with circular mask - transparent bg shows screen behind */}
      <OffthreadVideo
        src={avatarWebmUrl}
        transparent
        style={{
          position: "absolute",
          bottom: 40,
          left: 40,
          width: 180,
          height: 180,
          borderRadius: "50%", // Circular mask applied in CSS
          overflow: "hidden",
          objectFit: "cover",
        }}
      />
    </AbsoluteFill>
  );
};

Note: WebM doesn't support circle style - use normal or closeUp and apply circular masking via CSS.

Legacy: Green Screen with Chroma Key

If using MP4 with green background (not recommended - use WebM instead):

// Note: True chroma key requires WebGL or post-processing
// WebM transparent background is much simpler
<OffthreadVideo
  src={avatarVideoUrl}
  style={{
    mixBlendMode: "multiply", // Basic compositing only
  }}
/>

Layered Composition

import { OffthreadVideo, Sequence, useVideoConfig, Img } from "remotion";

interface LayeredAvatarProps {
  avatarVideoUrl: string;
  backgroundUrl: string;
  logoUrl: string;
  title: string;
}

export const LayeredAvatarComposition: React.FC<LayeredAvatarProps> = ({
  avatarVideoUrl,
  backgroundUrl,
  logoUrl,
  title,
}) => {
  const { fps } = useVideoConfig();

  return (
    <div style={{ position: "relative", width: "100%", height: "100%" }}>
      {/* Layer 1: Background */}
      <Img
        src={backgroundUrl}
        style={{
          position: "absolute",
          width: "100%",
          height: "100%",
          objectFit: "cover",
        }}
      />

      {/* Layer 2: Avatar video - use OffthreadVideo to prevent jitter */}
      <OffthreadVideo
        src={avatarVideoUrl}
        style={{
          position: "absolute",
          bottom: 0,
          right: 0,
          width: "40%",
          height: "auto",
        }}
      />

      {/* Layer 3: Title (appears after 1 second) */}
      <Sequence from={fps}>
        <div
          style={{
            position: "absolute",
            top: 50,
            left: 50,
            color: "white",
            fontSize: 48,
            fontWeight: "bold",
          }}
        >
          {title}
        </div>
      </Sequence>

      {/* Layer 4: Logo */}
      <Img
        src={logoUrl}
        style={{
          position: "absolute",
          top: 20,
          right: 20,
          width: 100,
          height: "auto",
        }}
      />
    </div>
  );
};

Complete Workflow

Generate and Compose

import { bundle } from "@remotion/bundler";
import { renderMedia, selectComposition } from "@remotion/renderer";

async function generateAvatarVideoForRemotion(
  script: string,
  outputPath: string
) {
  // 1. Generate HeyGen video
  console.log("Generating HeyGen avatar video...");
  const videoId = await generateHeyGenVideo(
    script,
    "josh_lite3_20230714",
    "1bd001e7e50f421d891986aad5158bc8",
    "landscape_1080p"
  );

  // 2. Wait for completion
  console.log("Waiting for HeyGen video...");
  const avatarVideoUrl = await waitForVideo(videoId);
  console.log(`HeyGen video ready: ${avatarVideoUrl}`);

  // 3. Get video duration for Remotion
  const avatarDuration = await getVideoDuration(avatarVideoUrl);
  const durationInFrames = Math.ceil(avatarDuration * 30); // 30 fps

  // 4. Bundle Remotion project
  console.log("Bundling Remotion project...");
  const bundleLocation = await bundle({
    entryPoint: "./remotion/src/index.ts",
  });

  // 5. Select composition
  const composition = await selectComposition({
    serveUrl: bundleLocation,
    id: "AvatarVideo",
    inputProps: {
      avatarVideoUrl,
    },
  });

  // 6. Render final video
  console.log("Rendering final composition...");
  await renderMedia({
    composition: {
      ...composition,
      durationInFrames,
    },
    serveUrl: bundleLocation,
    codec: "h264",
    outputLocation: outputPath,
    inputProps: {
      avatarVideoUrl,
    },
  });

  console.log(`Final video rendered: ${outputPath}`);
  return outputPath;
}

Dynamic Duration with calculateMetadata

// remotion/src/AvatarComposition.tsx
import { CalculateMetadataFunction } from "remotion";

export const calculateAvatarMetadata: CalculateMetadataFunction<
  AvatarCompositionProps
> = async ({ props }) => {
  // Fetch video duration from HeyGen video
  const duration = await getVideoDurationInSeconds(props.avatarVideoUrl);

  return {
    durationInFrames: Math.ceil(duration * 30),
    fps: 30,
    width: 1920,
    height: 1080,
  };
};

// In Root.tsx
<Composition
  id="AvatarVideo"
  component={AvatarComposition}
  calculateMetadata={calculateAvatarMetadata}
  defaultProps={{
    avatarVideoUrl: "",
  }}
/>

Best Practices

1. Use Green Screen for Flexibility

Generate HeyGen videos with green screen background when you want to composite:

background: {
  type: "color",
  value: "#00FF00", // Pure green for chroma key
}

2. Match Frame Rates

HeyGen default is 25 fps. Consider this when setting Remotion fps:

// Option 1: Match HeyGen's 25 fps
fps: 25

// Option 2: Use 30 fps with playback rate adjustment
<OffthreadVideo
  src={avatarVideoUrl}
  playbackRate={25/30} // Slow down slightly to match
/>

3. URL vs Download: When to Use Each

Use URL directly when:

• Previewing in Remotion Studio (npm run dev)
• URL won't expire before render completes
• You want faster iteration during development

// Direct URL usage - simpler, faster for dev
<OffthreadVideo src={avatarVideoUrl} />

Download first when:

• URL has expiration (HeyGen URLs expire after ~24 hours)
• Rendering will happen later or repeatedly
• Network reliability is a concern
• You need offline rendering

// Download with retry for reliability
async function downloadVideoWithRetry(
  url: string,
  outputPath: string,
  maxRetries = 5
): Promise<string> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await fetch(url);
      if (!response.ok) throw new Error(`HTTP ${response.status}`);

      const buffer = await response.arrayBuffer();
      await fs.promises.writeFile(outputPath, Buffer.from(buffer));
      return outputPath;
    } catch (error) {
      const delay = 2000 * Math.pow(2, attempt);
      console.log(`Retry ${attempt + 1}/${maxRetries} in ${delay}ms...`);
      await new Promise((r) => setTimeout(r, delay));
    }
  }
  throw new Error("Download failed after retries");
}

// Use local file in Remotion
const localPath = await downloadVideoWithRetry(avatarVideoUrl, "./public/avatar.mp4");

Hybrid approach (recommended for production):

// Save both URL and local path in metadata
const metadata = {
  videoUrl: result.video_url,           // For quick preview
  localPath: "./public/avatar.mp4",     // For reliable rendering
  expiresAt: Date.now() + 24 * 60 * 60 * 1000, // URL expiration
};

// In Remotion component, prefer local if available
const videoSrc = fs.existsSync(localPath) ? staticFile("avatar.mp4") : avatarVideoUrl;

4. Handle Avatar Positioning

Common avatar positions in compositions:

const AVATAR_POSITIONS = {
  fullscreen: { width: "100%", height: "100%", position: "center" },
  bottomRight: { width: "40%", bottom: 0, right: 0 },
  bottomLeft: { width: "40%", bottom: 0, left: 0 },
  pictureInPicture: { width: "25%", bottom: 20, right: 20 },
  leftThird: { width: "33%", left: 0, height: "100%" },
};

Output Formats

HeyGen Output

• Format: MP4 (H.264)
• Audio: AAC
• Resolution: As specified in request

Remotion Output

• Codec: H.264 (default), VP8, VP9, ProRes
• Match or exceed HeyGen quality settings

await renderMedia({
  codec: "h264",
  crf: 18, // High quality
  // ...
});

Troubleshooting

Video Not Playing in Remotion

1. Check URL accessibility (CORS issues) 2. Verify video format compatibility 3. Try downloading locally first

Dimension Mismatch

Ensure both HeyGen and Remotion use identical dimensions:

// Shared config
const VIDEO_CONFIG = {
  width: 1920,
  height: 1080,
  fps: 30,
};

// HeyGen
dimension: { width: VIDEO_CONFIG.width, height: VIDEO_CONFIG.height }

// Remotion
<Composition width={VIDEO_CONFIG.width} height={VIDEO_CONFIG.height} />

Video Jitter During Rendering

If avatar video appears jittery or stuttery in rendered output:

1. Use `OffthreadVideo` instead of `Video` - The basic Video component uses the browser's video decoder which isn't frame-accurate 2. Update imports (no additional install needed - it's in core remotion):

   // Before (causes jitter)
   import { Video } from "remotion";

   // After (frame-accurate)
   import { OffthreadVideo } from "remotion";

3. For WebM with transparency, add the transparent prop:

   <OffthreadVideo src={avatarWebmUrl} transparent />

Audio Sync Issues

If avatar audio drifts:

• Verify source video frame rate
• Check for encoding issues
• Consider re-encoding with consistent settings

Writing Scripts for HeyGen Videos

Scripts for AI avatar videos have different requirements than scripts for human presenters. This guide covers best practices for writing scripts that sound natural and render well.

Script Basics

Speech Rate and Duration

Typical speech is approximately 150 words per minute at normal speed (1.0x). Use this as a rough estimate for planning script length.

// Estimate video duration from script
function estimateDuration(script: string, speed: number = 1.0): number {
  const words = script.split(/\s+/).filter(w => w.length > 0).length;
  const wordsPerMinute = 150 * speed;
  return words / wordsPerMinute * 60; // seconds
}

// Estimate frames for Remotion
function estimateFrames(script: string, fps: number = 30, speed: number = 1.0): number {
  const durationSeconds = estimateDuration(script, speed);
  return Math.ceil(durationSeconds * fps);
}

Sentence Structure

Keep sentences short. AI voices handle shorter sentences more naturally.

Punctuation Affects Delivery

Adding Pauses with Break Tags

Use SSML-style <break> tags for precise pause control:

<break time="Xs"/>

Where X is seconds (e.g., 0.5s, 1s, 1.5s, 2s).

Formatting Rules

When to Use Pauses

Pause Duration Guide

Examples

// Section transitions
const script = `
Welcome to our product overview. <break time="1s"/>

Today I'll cover three key features. <break time="0.5s"/>
First, let's look at the dashboard. <break time="1.5s"/>

As you can see, it's designed for simplicity. <break time="0.5s"/>
Every action is just one click away.
`;

// Building suspense
const announcement = `
We've been working on something special. <break time="1s"/>
After months of development... <break time="1.5s"/>
I'm excited to announce <break time="0.5s"/> our new AI assistant.
`;

// List with rhythm
const features = `
Our platform offers three core benefits. <break time="0.5s"/>
Speed. <break time="0.5s"/>
Reliability. <break time="0.5s"/>
And simplicity. <break time="1s"/>
Let me show you each one.
`;

Consecutive Breaks

Multiple consecutive breaks are combined:

// These two breaks:
"Hello <break time=\"1s\"/> <break time=\"0.5s\"/> world"

// Are treated as a single 1.5s pause

Script Structure Templates

Product Demo (60 seconds, ~150 words)

const productDemo = `
Hi, I'm [Name], and I'm excited to show you [Product]. <break time="1s"/>

[Product] helps you [main benefit] in just [timeframe]. <break time="0.5s"/>

Here's how it works. <break time="1s"/>

First, [step 1]. <break time="0.5s"/>
Then, [step 2]. <break time="0.5s"/>
And finally, [step 3]. <break time="1s"/>

What used to take [old time] now takes [new time]. <break time="0.5s"/>

Ready to get started? <break time="0.5s"/>
Visit [website] today.
`;

Tutorial Introduction (90 seconds, ~225 words)

const tutorial = `
Welcome to this tutorial on [topic]. <break time="0.5s"/>
I'm [Name], and I'll guide you through everything you need to know. <break time="1s"/>

By the end of this video, you'll be able to [outcome 1], [outcome 2], and [outcome 3]. <break time="1s"/>

Let's start with the basics. <break time="1.5s"/>

[Section 1 content - 2-3 sentences] <break time="1s"/>

Now that you understand [concept], let's move on to [next topic]. <break time="1.5s"/>

[Section 2 content - 2-3 sentences] <break time="1s"/>

And finally, let's cover [last topic]. <break time="1.5s"/>

[Section 3 content - 2-3 sentences] <break time="1s"/>

That's everything you need to get started. <break time="0.5s"/>
If you have questions, leave a comment below. <break time="0.5s"/>
Thanks for watching!
`;

Announcement (30 seconds, ~75 words)

const announcement = `
Big news! <break time="0.5s"/>

We're thrilled to announce [announcement]. <break time="1s"/>

This means [benefit 1] and [benefit 2] for all our users. <break time="0.5s"/>

Starting [date], you'll be able to [new capability]. <break time="1s"/>

Head to [location] to learn more. <break time="0.5s"/>
We can't wait to hear what you think!
`;

Writing Tips for AI Voices

Do

• Write conversationally - Read it aloud to check flow
• Use contractions - "We're" not "We are", "It's" not "It is"
• Break up long sentences - Split at natural pause points
• Spell out abbreviations - "API" may sound like "a pee eye"
• Add pauses for emphasis - Guide the listener's attention
• End sections clearly - Don't trail off mid-thought

Avoid

• Jargon without context - Explain technical terms
• Long parentheticals - Move to separate sentences
• Ambiguous pronunciations - "read" (present) vs "read" (past)
• Excessive exclamation marks - One per script is usually enough
• Run-on sentences - Break into digestible chunks
• Dense information - Space out facts with pauses

Pronunciation Hints

For words that might be mispronounced, spell phonetically or add hints:

// Technical terms
const script1 = "Our API (A-P-I) handles authentication...";

// Ambiguous words
const script2 = "I read (red) the documentation yesterday...";

// Brand names
const script3 = "Welcome to HeyGen (hey-jen)...";

Multi-Scene Scripts

When splitting scripts across scenes (for different backgrounds or avatars):

const multiSceneVideo = {
  video_inputs: [
    {
      // Scene 1: Introduction
      character: { type: "avatar", avatar_id: "josh_lite3_20230714", avatar_style: "normal" },
      voice: {
        type: "text",
        input_text: "Welcome to our quarterly update. <break time=\"1s\"/> I'm Josh, and I'll walk you through the highlights.",
        voice_id: "voice_id_here",
      },
      background: { type: "color", value: "#1a1a2e" },
    },
    {
      // Scene 2: Main content (different background)
      character: { type: "avatar", avatar_id: "josh_lite3_20230714", avatar_style: "normal" },
      voice: {
        type: "text",
        input_text: "Let's start with revenue. <break time=\"0.5s\"/> We grew 25 percent quarter over quarter. <break time=\"1s\"/> Here's what drove that growth.",
        voice_id: "voice_id_here",
      },
      background: { type: "image", url: "https://..." },
    },
    // ... more scenes
  ],
};

Scene Transition Tips

• End each scene with a complete thought
• Start new scenes with brief context
• Maintain consistent tone across scenes
• Use pauses at scene starts to let visuals register

Testing Your Script

Before generating the full video:

1. Read aloud - Time yourself, check for awkward phrasing 2. Count words - Verify expected duration 3. Check break tags - Ensure proper spacing and syntax 4. Preview with short clip - Generate a 10-second test if unsure about pronunciation

// Test a small portion first
const testScript = script.split('.').slice(0, 2).join('.') + '.';
const testVideoId = await generateVideo({
  video_inputs: [{
    character: { type: "avatar", avatar_id: avatarId, avatar_style: "normal" },
    voice: { type: "text", input_text: testScript, voice_id: voiceId },
  }],
  dimension: { width: 1280, height: 720 }, // Lower res for test
});

Voice Speed Adjustment

Adjust delivery speed in the voice configuration:

voice: {
  type: "text",
  input_text: script,
  voice_id: "voice_id",
  speed: 1.1,  // Slightly faster (range: 0.5 - 2.0)
}

See voices.md for full voice configuration options.

Streaming Avatars

HeyGen's Streaming Avatar feature enables real-time interactive avatar experiences, perfect for live customer service, virtual assistants, and interactive applications.

Overview

Streaming avatars provide:

• Real-time rendering - Avatar responds immediately to input
• Interactive dialogue - Two-way conversation capability
• WebRTC streaming - Low-latency video delivery
• Session management - Create and manage live sessions

Creating a Streaming Session

Request Fields

curl

curl -X POST "https://api.heygen.com/v1/streaming.new" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "avatar_id": "josh_lite3_20230714",
    "voice_id": "1bd001e7e50f421d891986aad5158bc8",
    "quality": "high"
  }'

TypeScript

interface StreamingSessionRequest {
  avatar_id: string;                           // Required
  voice_id: string;                            // Required
  quality?: "low" | "medium" | "high";
  video_encoding?: "H264" | "VP8";
}

interface StreamingSessionResponse {
  error: null | string;
  data: {
    session_id: string;
    access_token: string;
    url: string;
    ice_servers: IceServer[];
  };
}

interface IceServer {
  urls: string[];
  username?: string;
  credential?: string;
}

async function createStreamingSession(
  config: StreamingSessionRequest
): Promise<StreamingSessionResponse["data"]> {
  const response = await fetch("https://api.heygen.com/v1/streaming.new", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(config),
  });

  const json: StreamingSessionResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data;
}

Python

import requests
import os

def create_streaming_session(config: dict) -> dict:
    response = requests.post(
        "https://api.heygen.com/v1/streaming.new",
        headers={
            "X-Api-Key": os.environ["HEYGEN_API_KEY"],
            "Content-Type": "application/json"
        },
        json=config
    )

    data = response.json()
    if data.get("error"):
        raise Exception(data["error"])

    return data["data"]

Session Quality Options

Sending Text to Avatar

Make the avatar speak in real-time:

Request Fields

curl

curl -X POST "https://api.heygen.com/v1/streaming.task" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "your_session_id",
    "text": "Hello! How can I help you today?",
    "task_type": "talk"
  }'

TypeScript

interface StreamingTaskRequest {
  session_id: string;                          // Required
  text: string;                                // Required
  task_type: "talk" | "repeat";                // Required
  task_mode?: "sync" | "async";
}

async function sendTextToAvatar(
  sessionId: string,
  text: string
): Promise<void> {
  const response = await fetch("https://api.heygen.com/v1/streaming.task", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      session_id: sessionId,
      text,
      task_type: "talk",
    }),
  });

  const json = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }
}

Stopping a Session

curl

curl -X POST "https://api.heygen.com/v1/streaming.stop" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "your_session_id"
  }'

TypeScript

async function stopStreamingSession(sessionId: string): Promise<void> {
  const response = await fetch("https://api.heygen.com/v1/streaming.stop", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ session_id: sessionId }),
  });

  const json = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }
}

WebRTC Integration

Connect to the streaming avatar using WebRTC:

class StreamingAvatarClient {
  private peerConnection: RTCPeerConnection | null = null;
  private sessionId: string | null = null;

  async connect(avatarId: string, voiceId: string): Promise<MediaStream> {
    // 1. Create session
    const session = await createStreamingSession({
      avatar_id: avatarId,
      voice_id: voiceId,
      quality: "high",
    });

    this.sessionId = session.session_id;

    // 2. Set up WebRTC peer connection
    this.peerConnection = new RTCPeerConnection({
      iceServers: session.ice_servers,
    });

    // 3. Handle incoming video stream
    const mediaStream = new MediaStream();

    this.peerConnection.ontrack = (event) => {
      event.streams[0].getTracks().forEach((track) => {
        mediaStream.addTrack(track);
      });
    };

    // 4. Create and set offer
    const offer = await this.peerConnection.createOffer();
    await this.peerConnection.setLocalDescription(offer);

    // 5. Exchange SDP with server (implementation depends on signaling method)
    // ...

    return mediaStream;
  }

  async speak(text: string): Promise<void> {
    if (!this.sessionId) {
      throw new Error("Not connected");
    }
    await sendTextToAvatar(this.sessionId, text);
  }

  async disconnect(): Promise<void> {
    if (this.sessionId) {
      await stopStreamingSession(this.sessionId);
      this.sessionId = null;
    }

    if (this.peerConnection) {
      this.peerConnection.close();
      this.peerConnection = null;
    }
  }
}

React Integration Example

import React, { useRef, useEffect, useState } from "react";

interface StreamingAvatarProps {
  avatarId: string;
  voiceId: string;
}

function StreamingAvatar({ avatarId, voiceId }: StreamingAvatarProps) {
  const videoRef = useRef<HTMLVideoElement>(null);
  const [client] = useState(() => new StreamingAvatarClient());
  const [connected, setConnected] = useState(false);
  const [inputText, setInputText] = useState("");

  useEffect(() => {
    async function connect() {
      try {
        const stream = await client.connect(avatarId, voiceId);
        if (videoRef.current) {
          videoRef.current.srcObject = stream;
        }
        setConnected(true);
      } catch (error) {
        console.error("Failed to connect:", error);
      }
    }

    connect();

    return () => {
      client.disconnect();
    };
  }, [avatarId, voiceId]);

  const handleSend = async () => {
    if (inputText.trim()) {
      await client.speak(inputText);
      setInputText("");
    }
  };

  return (
    <div>
      <video
        ref={videoRef}
        autoPlay
        playsInline
        style={{ width: "100%", maxWidth: 640 }}
      />
      {connected && (
        <div>
          <input
            type="text"
            value={inputText}
            onChange={(e) => setInputText(e.target.value)}
            placeholder="Type message..."
          />
          <button onClick={handleSend}>Send</button>
        </div>
      )}
    </div>
  );
}

Session Management

Listing Active Sessions

async function listActiveSessions(): Promise<string[]> {
  const response = await fetch("https://api.heygen.com/v1/streaming.list", {
    headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
  });

  const json = await response.json();
  return json.data.sessions;
}

Session Timeout

Sessions automatically timeout after inactivity. Send periodic keep-alive messages:

class StreamingAvatarClient {
  private keepAliveInterval: NodeJS.Timer | null = null;

  startKeepAlive() {
    this.keepAliveInterval = setInterval(async () => {
      if (this.sessionId) {
        await this.ping();
      }
    }, 30000); // Every 30 seconds
  }

  private async ping(): Promise<void> {
    // Implementation depends on API
  }

  stopKeepAlive() {
    if (this.keepAliveInterval) {
      clearInterval(this.keepAliveInterval);
      this.keepAliveInterval = null;
    }
  }
}

Interrupting Speech

Interrupt current speech to start new content:

async function interruptAndSpeak(
  sessionId: string,
  newText: string
): Promise<void> {
  // Interrupt current speech
  await fetch("https://api.heygen.com/v1/streaming.interrupt", {
    method: "POST",
    headers: {
      "X-Api-Key": process.env.HEYGEN_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ session_id: sessionId }),
  });

  // Send new text
  await sendTextToAvatar(sessionId, newText);
}

Use Cases

Virtual Customer Service

async function handleCustomerQuery(
  sessionId: string,
  query: string
): Promise<void> {
  // Process query with your AI/logic
  const response = await processCustomerQuery(query);

  // Have avatar speak the response
  await sendTextToAvatar(sessionId, response);
}

Interactive Training

async function conductTrainingSession(
  sessionId: string,
  trainingScript: string[]
): Promise<void> {
  for (const segment of trainingScript) {
    await sendTextToAvatar(sessionId, segment);

    // Wait for avatar to finish speaking
    await waitForSpeechComplete(sessionId);

    // Pause between segments
    await new Promise((r) => setTimeout(r, 1000));
  }
}

Best Practices

1. Handle disconnections - Implement reconnection logic 2. Manage bandwidth - Adjust quality based on connection 3. Limit session duration - Close unused sessions to save credits 4. Test latency - Ensure acceptable response times 5. Provide fallback - Have backup for streaming failures 6. Monitor usage - Track session duration and costs

Limitations

• Session duration limits vary by plan
• Concurrent session limits apply
• Credits consumed based on session time
• WebRTC requires modern browser support
• Network quality affects experience

Video Templates

HeyGen templates allow you to create reusable video structures with variable placeholders, enabling personalized video generation at scale.

Listing Templates

curl

curl -X GET "https://api.heygen.com/v2/templates" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

TypeScript

interface Template {
  template_id: string;
  name: string;
  thumbnail_url: string;
  variables: TemplateVariable[];
}

interface TemplateVariable {
  name: string;
  type: "text" | "image" | "audio";
  properties?: {
    max_length?: number;
    default_value?: string;
  };
}

interface TemplatesResponse {
  error: null | string;
  data: {
    templates: Template[];
  };
}

async function listTemplates(): Promise<Template[]> {
  const response = await fetch("https://api.heygen.com/v2/templates", {
    headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
  });

  const json: TemplatesResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data.templates;
}

Python

import requests
import os

def list_templates() -> list:
    response = requests.get(
        "https://api.heygen.com/v2/templates",
        headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
    )

    data = response.json()
    if data.get("error"):
        raise Exception(data["error"])

    return data["data"]["templates"]

Response Format

{
  "error": null,
  "data": {
    "templates": [
      {
        "template_id": "template_abc123",
        "name": "Product Announcement",
        "thumbnail_url": "https://files.heygen.ai/...",
        "variables": [
          {
            "name": "product_name",
            "type": "text",
            "properties": {
              "max_length": 50
            }
          },
          {
            "name": "presenter_script",
            "type": "text",
            "properties": {
              "max_length": 500
            }
          },
          {
            "name": "product_image",
            "type": "image"
          }
        ]
      }
    ]
  }
}

Getting Template Details

curl

curl -X GET "https://api.heygen.com/v2/template/{template_id}" \
  -H "X-Api-Key: $HEYGEN_API_KEY"

TypeScript

async function getTemplate(templateId: string): Promise<Template> {
  const response = await fetch(
    `https://api.heygen.com/v2/template/${templateId}`,
    { headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
  );

  const json = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data;
}

Generating Video from Template

Request Fields

Note: The variables object keys must match the template's defined variable names. Check template details to see which variables are defined.

curl

curl -X POST "https://api.heygen.com/v2/template/{template_id}/generate" \
  -H "X-Api-Key: $HEYGEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "test": false,
    "variables": {
      "product_name": "SuperWidget Pro",
      "presenter_script": "Introducing our latest innovation!",
      "product_image": "https://example.com/product.jpg"
    }
  }'

TypeScript

interface TemplateGenerateRequest {
  variables: Record<string, string>;           // Required
  test?: boolean;
  title?: string;
  callback_id?: string;
  callback_url?: string;
}

interface TemplateGenerateResponse {
  error: null | string;
  data: {
    video_id: string;
  };
}

async function generateFromTemplate(
  templateId: string,
  variables: Record<string, string>,
  test: boolean = false
): Promise<string> {
  const response = await fetch(
    `https://api.heygen.com/v2/template/${templateId}/generate`,
    {
      method: "POST",
      headers: {
        "X-Api-Key": process.env.HEYGEN_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ test, variables }),
    }
  );

  const json: TemplateGenerateResponse = await response.json();

  if (json.error) {
    throw new Error(json.error);
  }

  return json.data.video_id;
}

Python

def generate_from_template(template_id: str, variables: dict, test: bool = False) -> str:
    response = requests.post(
        f"https://api.heygen.com/v2/template/{template_id}/generate",
        headers={
            "X-Api-Key": os.environ["HEYGEN_API_KEY"],
            "Content-Type": "application/json"
        },
        json={
            "test": test,
            "variables": variables
        }
    )

    data = response.json()
    if data.get("error"):
        raise Exception(data["error"])

    return data["data"]["video_id"]

Variable Types

Text Variables

For dynamic text content:

const variables = {
  customer_name: "John Smith",
  product_name: "SuperWidget Pro",
  price: "$99.99",
  cta_text: "Order Now!",
};

Image Variables

For dynamic images (backgrounds, product shots):

const variables = {
  product_image: "https://example.com/product.jpg",
  logo: "https://example.com/logo.png",
  background: "https://example.com/bg.jpg",
};

Audio Variables

For custom audio content:

const variables = {
  background_music: "https://example.com/music.mp3",
  custom_voiceover: "https://example.com/voiceover.mp3",
};

Batch Video Generation

Generate multiple personalized videos from a template:

interface PersonalizationData {
  name: string;
  email: string;
  company: string;
  customMessage: string;
}

async function batchGenerateVideos(
  templateId: string,
  recipients: PersonalizationData[]
): Promise<string[]> {
  const videoIds: string[] = [];

  for (const recipient of recipients) {
    const variables = {
      recipient_name: recipient.name,
      company_name: recipient.company,
      personalized_message: recipient.customMessage,
    };

    const videoId = await generateFromTemplate(templateId, variables);
    videoIds.push(videoId);

    // Rate limiting: add delay between requests
    await new Promise((r) => setTimeout(r, 1000));
  }

  return videoIds;
}

// Usage
const recipients = [
  {
    name: "John Smith",
    email: "john@example.com",
    company: "Acme Inc",
    customMessage: "Thanks for your interest in our product!",
  },
  {
    name: "Jane Doe",
    email: "jane@example.com",
    company: "Tech Corp",
    customMessage: "We'd love to show you a demo!",
  },
];

const videoIds = await batchGenerateVideos("template_abc123", recipients);

Template Validation

Validate variables before generating:

function validateTemplateVariables(
  template: Template,
  variables: Record<string, string>
): { valid: boolean; errors: string[] } {
  const errors: string[] = [];

  for (const templateVar of template.variables) {
    const value = variables[templateVar.name];

    // Check if required variable is provided
    if (!value) {
      errors.push(`Missing required variable: ${templateVar.name}`);
      continue;
    }

    // Check text length limits
    if (templateVar.type === "text" && templateVar.properties?.max_length) {
      if (value.length > templateVar.properties.max_length) {
        errors.push(
          `Variable "${templateVar.name}" exceeds max length of ${templateVar.properties.max_length}`
        );
      }
    }

    // Validate image URLs
    if (templateVar.type === "image") {
      try {
        new URL(value);
      } catch {
        errors.push(`Variable "${templateVar.name}" is not a valid URL`);
      }
    }
  }

  return {
    valid: errors.length === 0,
    errors,
  };
}

Complete Template Workflow

async function createPersonalizedVideo(
  templateId: string,
  personalization: Record<string, string>
): Promise<string> {
  // 1. Get template details
  const template = await getTemplate(templateId);
  console.log(`Using template: ${template.name}`);

  // 2. Validate variables
  const validation = validateTemplateVariables(template, personalization);
  if (!validation.valid) {
    throw new Error(`Validation errors: ${validation.errors.join(", ")}`);
  }

  // 3. Generate video
  console.log("Generating video...");
  const videoId = await generateFromTemplate(templateId, personalization);
  console.log(`Video ID: ${videoId}`);

  // 4. Wait for completion
  const videoUrl = await waitForVideo(videoId);
  console.log(`Video ready: ${videoUrl}`);

  return videoUrl;
}

// Usage
const videoUrl = await createPersonalizedVideo("template_abc123", {
  customer_name: "John Smith",
  product_name: "SuperWidget Pro",
  offer_details: "Get 20% off your first order!",
});

Best Practices

1. Design for flexibility - Create templates with generic placeholders 2. Set reasonable limits - Define max lengths for text variables 3. Validate inputs - Check variable values before generating 4. Use test mode - Test with test: true to verify before production 5. Implement rate limiting - Add delays for batch generation 6. Cache template data - Reduce API calls by caching template details 7. Error handling - Gracefully handle generation failures

Use Cases

• Sales outreach - Personalized prospect videos
• Customer onboarding - Welcome videos with customer name
• Product updates - Announcements with dynamic content
• Training - Customized training modules
• Marketing campaigns - Targeted promotional videos