Comfyui Character Gen

ComfyUI Character Generation Expert

Build production-ready ComfyUI workflows for consistent character generation across image, video, and voice modalities.

Quick Decision: Which Approach?

Starting from reference images (like 3D renders)? → InfiniteYou (state-of-the-art 2025) or InstantID + IP-Adapter (proven, lower VRAM)

Need highest identity fidelity? → FLUX.2 (NEW 2026: up to 10 ref images) or PuLID Flux II (no model pollution)

Want iterative editing without retraining? → FLUX Kontext (context-aware, maintains consistency across edits)

Creating video content? → LTX-2 (NEW 2026: 4K production-ready), Wan 2.2 MoE (film-level), or FramePack (60-sec on 6GB!)

Need voice for character? → TTS Audio Suite (unified platform, 23 languages) or F5-TTS Cross-Lingual (NEW 2026)

Core Workflow Patterns

Pattern 1: Zero-Shot Character Generation (No Training)

Best for: Quick iteration, 3D-to-photorealism conversion, limited reference images

Load Reference Face → InstantID + IP-Adapter FaceID → ControlNet Pose → KSampler → FaceDetailer → Upscale

Critical settings:

• CFG: 4-5 (prevents burning with InstantID)
• Resolution: 1016×1016 (avoids watermark artifacts)
• IP-Adapter weight: 0.6-0.8
• InstantID noise injection: 35% to negative

See references/workflows.md for complete node configurations.

Pattern 2: LoRA + Identity Methods (Maximum Consistency)

Best for: Production work, character series, video generation base

Train LoRA → Load LoRA + Checkpoint → Add InstantID/PuLID → Generate → FaceDetailer → ReActor (optional) → Upscale

Training requirements:

• 15-30 images, varied poses/expressions/lighting
• Unique trigger word (e.g., "sage_character")
• See references/lora-training.md for full parameters

Pattern 3: Video Generation Pipeline

Best for: Talking heads, character animation, promotional content

Generate/Load Hero Image → Wan 2.1 I2V OR AnimateDiff → FaceDetailer per frame → Frame Interpolation → Video Combine

Model selection:

• Wan 2.1 14B: Best quality, 24GB+ VRAM, slower
• Wan 2.1 1.3B: 8GB VRAM, good quality, faster
• AnimateDiff Lightning: Fastest, best for iteration

Pattern 4: Talking Head with Voice

Best for: Character dialogue, presentations, social content

Two approaches available:

Approach 1 (Image → Talking Head):
Character Portrait → Generate Audio → SadTalker/LivePortrait → CodeFormer Enhancement → Final Video

Approach 2 (Video → Add Voice):
Existing Video → Generate Audio → Wav2Lip Lip-Sync → CodeFormer Enhancement → Final Video

See references/talking-head-workflows.md for complete workflows and references/voice-synthesis.md for voice creation options.

Model Recommendations (2026 Updated)

Image Generation

Identity Preservation (2026 State-of-Art)

Video Generation

Voice/TTS

Essential Custom Nodes

Install via ComfyUI-Manager:

ComfyUI-Manager              # Must install first
ComfyUI_IPAdapter_plus       # IP-Adapter and FaceID
ComfyUI_InstantID            # InstantID workflow
ComfyUI-Impact-Pack          # FaceDetailer
ComfyUI-ReActor              # Face swapping
ComfyUI-AnimateDiff-Evolved  # Video generation
ComfyUI-VideoHelperSuite     # Video I/O
comfyui_controlnet_aux       # Pose/depth preprocessors
ComfyUI_UltimateSDUpscale    # Tiled upscaling
ComfyUI-Frame-Interpolation  # Smooth video

RTX 50 Series Optimization (NEW 2026)

With 32GB VRAM on RTX 5090, run most workflows without optimization. ComfyUI v0.8.1 adds major RTX 50 Series enhancements:

Launch flags: --highvram --fp8_e4m3fn-unet

NEW v0.8.1 Features:

• NVFP4/NVFP8 precision formats: 3x faster performance, 60% VRAM reduction on RTX 50 Series
• Weight streaming: Uses system RAM when VRAM exhausted, enables larger models on mid-range GPUs
• Enable tiled VAE for 8K+ upscaling
• Batch 4× 1024×1024 generations in parallel
• Run Wan 2.2 14B + LTX-2 natively
• Use FP8 quantization for FLUX (50% VRAM reduction)

Workflow Generation Process

When building a workflow for a user:

1. Clarify the goal: Image only? Video? With voice? What's the source material?

2. Select the pipeline pattern from above based on requirements

3. Generate the workflow following node configurations in references/workflows.md

4. Include model downloads with exact filenames and paths from references/models.md

5. Provide parameter recommendations specific to their hardware/use case

Reference Files

• references/research-log.md - Latest techniques: InfiniteYou, FLUX Kontext, PuLID Flux II, Wan 2.2 MoE, FramePack, FLUX.2, LTX-2.3, Wan 2.6, Qwen3-TTS, and more
• references/models.md - Complete model list with HuggingFace/Civitai links, file paths, and compatibility notes
• references/workflows.md - Detailed node-by-node workflow templates for each pattern
• references/lora-training.md - LoRA training guide with Kohya/AI-Toolkit parameters
• references/voice-synthesis.md - Voice cloning, TTS, and lip-sync pipeline details
• references/talking-head-workflows.md - Complete talking head workflows: Image→Talking Head (SadTalker, LivePortrait) and Video→Add Voice (Wav2Lip) with production scripts
• references/evolution.md - Update sources, changelog, and user-specific learnings

Skill Evolution

This skill is designed to evolve. When helping the user:

Before starting a workflow:

• Check if new models have dropped that might be better (search HuggingFace/Civitai if uncertain)
• Consider if user's past successes/failures inform the approach

After completing a workflow:

• Note what worked well or poorly for future reference
• If user discovers better settings, update the relevant reference file

Proactive updates:

• When the user mentions a new model or technique, research and integrate it
• Periodically suggest checking for updates to key dependencies

See references/evolution.md for monitoring sources and update protocols.

Example: 3D Render to Photorealistic Character

For converting stylized 3D renders (like game/VN characters) to photorealistic images:

Recommended approach: InstantID + IP-Adapter FaceID on FLUX

1. Load 3D render reference (best quality, front-facing)
2. Apply InstantID (extracts identity + facial keypoints)
3. Apply IP-Adapter FaceID Plus V2 (weight 0.7)
4. Use FLUX.1-dev checkpoint
5. Prompt: "photorealistic portrait, detailed skin texture, natural lighting, [character description]"
6. CFG: 4-5, Steps: 25-30
7. FaceDetailer pass (denoise 0.35)
8. Upscale with 4x-UltraSharp

This converts the stylized look to photorealism while preserving the core identity features.

comfyui-character-gen — Eval Configuration

Classification

• Type: Capability Uplift
• Category: Identity-preserving character generation pipeline design with method selection

What "Good" Looks Like

1. Correct identity preservation method selected for the use case (InfiniteYou for 3D-to-photo, FLUX Kontext for iterative editing, PuLID for style transfer, InstantID for direct likeness) 2. CFG scale appropriate for the selected method (InstantID needs 4-5, not the typical 7-8) 3. Resolution matches the base model requirements (FLUX at 1024x1024, SDXL at 1024x1024, SD1.5 at 512x512) 4. Pipeline ordering is correct — identity injection at the right stage, not conflicting with other conditioning 5. Multi-reference handling follows correct strategy when multiple identity images are provided

Known Limitations

• Cannot evaluate actual likeness preservation quality — only structural/parameter correctness
• New identity methods emerge frequently; skill may not cover the latest releases
• Quality of input reference images significantly affects results but is outside skill scope

Benchmark Strategy

• Without skill: Base Claude recommends IP-Adapter or generic img2img for everything, uses default CFG values, misses method-specific parameter requirements
• With skill: Selects the optimal identity method per use case, sets method-specific parameters correctly, orders pipeline stages properly
• Key differentiator: Identity method selection logic — knowing when InfiniteYou beats InstantID, and why CFG must be lowered for certain methods

Security — Eval Sandboxing

Eval runs use real tool access and may expose secrets in output. Results are gitignored. Use --allowedTools "Read,Glob,Grep" to prevent modification during eval runs.

Running Evals

bash eval/run-eval.sh                  # Full run (with-skill + baseline)
bash eval/run-eval.sh --skill-only     # With-skill only
bash eval/run-eval.sh --case TC-001    # Single test case

Retirement Signal

When base Claude can reliably distinguish between identity preservation methods (InfiniteYou, FLUX Kontext, PuLID, InstantID) and correctly recommends method-specific parameters without the skill's decision tree.

Skill Evolution & Update Protocol

This skill is designed to evolve continuously. Monitor these sources and update accordingly.

---

Sources to Monitor

Model Releases (Check Weekly)

HuggingFace Trending

• https://huggingface.co/models?sort=trending
• Filter: diffusers, video, audio
• Watch for: New FLUX variants, Wan updates, voice models

Civitai New Models

• https://civitai.com/models?sort=Newest
• Filter: Checkpoint, LoRA, ControlNet
• Watch for: New photorealistic checkpoints, face LoRAs

GitHub Releases

• https://github.com/comfyanonymous/ComfyUI/releases
• https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/releases
• https://github.com/cubiq/ComfyUI_IPAdapter_plus/releases
• https://github.com/cubiq/ComfyUI_InstantID/releases
• https://github.com/ltdrdata/ComfyUI-Impact-Pack/releases

Research & Techniques (Check Monthly)

Papers

• https://huggingface.co/papers (daily ML papers)
• https://arxiv.org/list/cs.CV/recent (computer vision)
• Search terms: "identity preservation", "video generation", "voice cloning", "diffusion"

Community Knowledge

• r/StableDiffusion
• r/comfyui
• ComfyUI Discord
• Civitai articles/guides

Video Model Landscape (Check Bi-Weekly)

Open Source

• Wan series (Alibaba) - watch for 2.3+
• HunyuanVideo (Tencent)
• CogVideoX (Zhipu)
• LTX Video (Lightricks)
• Mochi (Genmo)

Commercial (for benchmarking)

• Kling (Kuaishou)
• Runway Gen-3
• Pika
• Sora (OpenAI)

Voice/Audio (Check Monthly)

• https://github.com/SWivid/F5-TTS/releases
• https://github.com/resemble-ai/chatterbox/releases
• ElevenLabs blog for new features
• Fish Audio, Cartesia for emerging options

---

Update Protocol

When New Model Drops

1. Evaluate relevance: Does it improve on current recommendations? 2. Test compatibility: Works with ComfyUI? Required nodes available? 3. Benchmark: Compare quality/speed vs current options 4. Update files:

• models.md: Add download links, paths, requirements
• workflows.md: Add/modify workflow if needed
• SKILL.md: Update recommendation tables if it becomes new default

5. Log change: Add to changelog below

When User Reports Issue

1. Document the issue: What workflow, what settings, what result 2. Research solutions: Community fixes, parameter adjustments 3. Test fix: Verify solution works 4. Update relevant file: Add troubleshooting entry or modify settings 5. Log change: Add to changelog

When User Discovers Better Approach

1. Document the discovery: What worked better and why 2. Validate: Test in multiple scenarios 3. Integrate: Update workflows.md with new approach 4. Promote if significant: Update SKILL.md recommendations

---

Changelog

v1.2.0 (2026-02-18 Research Run)

NEW MODELS:

• Z-Image: Day-0 ComfyUI support (Feb 2, 2026) — non-distilled, flexible quality, NVFP4/NVFP8
• Hunyuan 3D 3.0: Text/image/sketch → 3D assets via Partner Nodes (Feb 16, 2026)
• Kling 3.0: Commercial-quality video via Partner Nodes (Feb 16, 2026)
• Stable Video Infinity 2.0 Pro: Infinite-length video with Wan 2.2 I2V A14B

IDENTITY & CHARACTER:

• PuLID Flux Chroma: New fork extending PuLID to FLUX + Chroma models
• USO (ByteDance): FLUX.1-dev based unified style+subject generation — no more fighting the model for style fidelity
• FLUX Kontext: Confirmed community gold standard for single-reference consistent character editing

PERFORMANCE (CRITICAL):

• NVFP4 requires PyTorch cu130 — without it, NVFP4 is up to 2x SLOWER than FP8
• ComfyUI +40% on all NVIDIA GPUs (async offload + pinned memory now default)
• NVFP4 = 3x faster / 60% VRAM reduction on RTX 50 Series
• NVFP8 = 2x faster / 40% VRAM reduction (any NVIDIA GPU)
• AMD ROCm native integration: 5.4x faster for AMD GPUs

NEW FEATURE:

• Added comfyui-prompt-interview skill — conversational guided interview that synthesizes a perfect model-appropriate prompt from user answers

FILES UPDATED:

• references/staleness-report.md — first full research run logged
• foundation/model-landscape.md — Z-Image, SVI 2.0 Pro, Kling 3.0, Hunyuan 3D 3.0 added
• references/research-log.md — 2026 section appended
• foundation/skill-registry.md — new prompt-interview skill registered

v1.1.0 (February 2026 Update)

NEW MODELS & FEATURES:

• FLUX.2: Up to 10 reference images for maximum identity consistency
• LTX-2: First production-ready open-source 4K audio+video generation
• AuraFace: Commercial-friendly identity encoder (ArcFace alternative)
• F5-TTS Cross-Lingual: Cross-lingual voice cloning without transcripts
• DiffSwap++: 3D latent-controlled face swapping

PLATFORM UPDATES:

• ComfyUI v0.8.1: NVFP4/NVFP8 support (3x faster, 60% VRAM reduction on RTX 50 Series)
• Native AMD ROCm integration for AMD GPUs
• Audio recording node for direct audio capture
• Weight streaming memory management

DOCUMENTATION:

• Added 2026 updates section to research-log.md
• Updated all model recommendation tables
• Updated Quick Decision guide with 2026 options
• Updated RTX optimization section for v0.8.1

v1.0.0 (Initial Release)

• Core workflows: InstantID, IP-Adapter, PuLID, AnimateDiff, Wan 2.1
• Model reference with 2024-2025 state of the art
• LoRA training guide for SDXL and FLUX
• Voice synthesis covering Chatterbox, F5-TTS, RVC, ElevenLabs
• Lip-sync pipelines: Wav2Lip, SadTalker, LivePortrait

---

User-Specific Learnings

Track what works best for this user's specific setup and preferences.

Hardware Profile

• GPU: RTX 5090 (32GB VRAM)
• Can run: All models natively, including Wan 14B, FLUX FP16
• Optimization: Use --highvram, batch processing enabled

Project: Sage Character

• Source: 3D renders (visual novel style)
• Target: Photorealistic output
• Key features: Auburn hair, green eyes, freckles, fair skin
• Recommended approach: InstantID + IP-Adapter → test → train LoRA if needed

Workflow Preferences

(To be filled as user works with skill)

• Preferred checkpoint: [TBD]
• Preferred upscaler: [TBD]
• CFG sweet spot: [TBD]
• Face detailer settings: [TBD]

What Worked Well

(Document successful approaches)

What Didn't Work

(Document failed approaches to avoid repeating)

---

Scheduled Research Tasks

Weekly

• [ ] Check HuggingFace trending for new models
• [ ] Check ComfyUI releases for breaking changes
• [ ] Review any user feedback from recent sessions

Monthly

• [ ] Deep dive on any major new releases
• [ ] Review video model landscape
• [ ] Check voice synthesis developments
• [ ] Update changelog if needed

Quarterly

• [ ] Full skill audit - are recommendations still current?
• [ ] Remove deprecated models/methods
• [ ] Add any new paradigm shifts

LoRA Training Guide for Character Consistency

Training custom LoRAs from reference images (including 3D renders) for maximum character consistency.

---

When to Train vs Zero-Shot

Train a LoRA when:

• Need absolute consistency across many images
• Building a character series or ongoing project
• Zero-shot methods don't capture specific features
• Want to combine with prompts for varied outputs

Use zero-shot (InstantID/PuLID) when:

• Quick one-off generation
• Limited reference images (1-5)
• Need style transfer (3D→realistic)
• Testing concepts before committing to training

---

Dataset Preparation

Minimum Viable Dataset

• 10-15 images minimum for recognizable results
• 20-30 images optimal for strong identity
• 50+ images for highly flexible, robust LoRA

Image Requirements

Technical specs:

• Resolution: 512×512 minimum, 1024×1024 preferred
• Format: PNG or high-quality JPEG
• File size: No compression artifacts

Content diversity:

• Multiple angles (front, 3/4, profile)
• Various expressions (neutral, smile, serious, etc.)
• Different lighting conditions
• Varied backgrounds (or transparent/solid)
• Multiple outfits/contexts

Preprocessing 3D Renders

Problem: Training on 3D renders can bake in the "3D render" aesthetic.

Solution: Convert renders to varied styles before training:

# Use img2img to create style variations
styles = [
    "photorealistic portrait, dslr photo",
    "oil painting portrait", 
    "digital illustration",
    "pencil sketch",
    "watercolor portrait"
]
# Generate 3-5 variations per base render
# Include original renders + variations in training set

This teaches the model the identity, not the style.

Cropping Strategy

For face-focused LoRA:

• Square crop centered on face
• Face should occupy 50-70% of frame
• Include some neck/shoulders for context
• Consistent framing across dataset

For full-body LoRA:

• Include full body shots
• Mix with face close-ups (70/30 ratio)
• Maintain consistent proportions

Captioning

Method 1: Auto-caption with JoyCaption + Manual Edit

# Run JoyCaption
python caption.py --input ./images --output ./captions

# Then manually add trigger word and remove identity descriptions

Method 2: WD14 Tags + Sentence

sage_character, 1woman, green eyes, freckles, looking at viewer, 
white shirt, indoor, soft lighting

Critical captioning rules:

1. Always prefix with unique trigger word:

• Good: sage_character, ohwx_sage, sks_person
• Bad: woman, redhead, character (too generic)

2. Don't describe the face in detail:

• Bad: "woman with green eyes, freckles, auburn hair, defined cheekbones"
• Good: "sage_character, woman, indoor portrait"
• Let the model learn features organically

3. Do describe everything else:

• Clothing, pose, background, lighting, camera angle
• Expression (smiling, serious, etc.)
• Style cues (photorealistic, illustration, etc.)

4. Keep captions consistent in structure:

   [trigger], [subject], [clothing], [pose], [setting], [lighting], [style]

---

Training Parameters

SDXL LoRA (Kohya_ss)

Config file settings:

# Basic settings
pretrained_model_name_or_path: "./models/RealVisXL_V5.0.safetensors"
train_data_dir: "./dataset/sage"
output_dir: "./output"
output_name: "sage_character_sdxl"

# Network settings
network_module: "networks.lora"
network_dim: 32          # Rank - 16-64, higher = more capacity
network_alpha: 16        # Usually dim/2

# Training settings  
resolution: "1024,1024"
train_batch_size: 1      # 2 if VRAM allows
gradient_accumulation_steps: 4
learning_rate: 0.0001    # 1e-4, can try 4e-4
lr_scheduler: "cosine_with_restarts"
lr_scheduler_num_cycles: 3
max_train_epochs: 10
optimizer_type: "AdamW8bit"

# Quality settings
mixed_precision: "bf16"
save_precision: "fp16"
cache_latents: true
cache_latents_to_disk: true
enable_bucket: true
min_bucket_reso: 512
max_bucket_reso: 2048

# Regularization (optional but recommended)
prior_loss_weight: 1.0
min_snr_gamma: 5

Recommended steps calculation:

total_steps = (num_images × repeats × epochs) / batch_size
target: 1500-3000 steps for SDXL

Example: 20 images × 10 repeats × 5 epochs / 1 batch = 1000 steps

FLUX LoRA (AI-Toolkit)

Config YAML:

job: extension
config:
  name: "sage_character_flux"
  process:
    - type: sd_trainer
      training_folder: "./output"
      device: cuda:0
      
      network:
        type: lora
        linear: 16           # Rank
        linear_alpha: 16     # Alpha = rank for FLUX
        
      model:
        name_or_path: "black-forest-labs/FLUX.1-dev"
        assistant_lora_path: null
        quantize: true       # Enable for <24GB VRAM
        
      datasets:
        - folder_path: "./dataset/sage"
          caption_ext: "txt"
          default_caption: "sage_character"
          resolution: [1024]
          
      train:
        batch_size: 1
        gradient_accumulation_steps: 4
        steps: 1500          # FLUX converges faster
        lr: 4e-4             # Higher LR for FLUX
        optimizer: adamw8bit
        dtype: bf16
        
      sample:
        sample_every: 250
        prompts:
          - "sage_character, photorealistic portrait, green eyes"

FLUX training notes:

• Converges 2-3x faster than SDXL
• Use higher learning rate (4e-4 vs 1e-4)
• 1000-2000 steps usually sufficient
• Quality plateaus earlier, watch for overfitting

Low VRAM Training (FluxGym)

For 12-16GB VRAM:

# Enable all memory optimizations
use_8bit_adam: true
gradient_checkpointing: true
cache_latents_to_disk: true
max_data_loader_n_workers: 0

# Reduce batch processing
train_batch_size: 1
gradient_accumulation_steps: 8

# Use NF4 quantization (SimpleTuner)
quantize_base_model: nf4

---

Training Workflow

Step 1: Prepare Dataset

# Create folder structure
mkdir -p dataset/sage/10_sage_character

# Place images (naming: 001.png, 002.png, ...)
# Place captions (naming: 001.txt, 002.txt, ...)

Folder naming: [repeats]_[trigger word]

• 10_sage_character = each image repeated 10 times per epoch

Step 2: Run Training (Kohya)

# Navigate to kohya_ss
cd kohya_ss

# Start training with config
accelerate launch train_network.py --config_file ./config/sage_train.yaml

Step 3: Test Checkpoints

• Training saves checkpoints every N steps
• Test each checkpoint with identical prompts
• Look for: identity accuracy, flexibility, overfitting signs

Step 4: Select Best Epoch

Signs of good training:

• Character recognizable from trigger word alone
• Responds well to different prompts/contexts
• Doesn't always produce same pose/expression

Signs of overfitting:

• Same exact pose/expression regardless of prompt
• Background elements from training appearing
• Ignores clothing/setting prompts

---

Recommended Settings by Use Case

3D Render → Photorealistic Character

# Preprocessing
- Convert 3D renders to varied styles first
- Mix: 60% style variations, 40% original renders

# Training
network_dim: 32
learning_rate: 0.0001
epochs: 8-10
steps: ~2000

# Testing
- Test with InstantID/PuLID combination
- LoRA provides base identity, zero-shot methods add realism

Anime/Stylized Character

# Use anime-focused base model
pretrained_model: "animagineXL" OR "ponyDiffusion"

# Training
network_dim: 48-64  # Higher for style complexity
learning_rate: 0.00008
epochs: 12-15

# Include style tags
captions: "sage_character, anime style, 1girl, ..."

Video-Ready Character LoRA

# Include motion diversity in training
- Standing, sitting, walking poses
- Various arm positions
- Multiple head angles

# Training (slightly overfit for consistency)
network_dim: 24
epochs: 12
# Test with AnimateDiff to verify motion compatibility

---

Combining LoRA with Zero-Shot Methods

Best practice: LoRA as base, zero-shot for enhancement

[Load Checkpoint]
      ↓
[Load Character LoRA (0.7-0.9 strength)]
      ↓
[Apply PuLID or IP-Adapter (0.5-0.7 weight)]
      ↓
[Generate]

This approach:

• LoRA provides learned identity from training
• Zero-shot method reinforces specific reference features
• Lower weights on both prevent conflict
• More robust than either method alone

---

Troubleshooting

LoRA not activating

• Verify trigger word exactly matches training
• Check LoRA strength (start at 0.8)
• Ensure LoRA loaded BEFORE KSampler

Identity drift at different angles

• Add more varied angles to dataset
• Reduce network_dim (16-24)
• Lower learning rate

Overfitting (same output every time)

• Reduce epochs
• Increase dataset size
• Lower network_dim
• Add more caption variety

Style contamination (looks like training images)

• Better caption diversity
• Don't describe style in captions
• Use style-varied preprocessing

Poor quality/artifacts

• Check training images for compression artifacts
• Reduce learning rate
• Enable min_snr_gamma
• Try different optimizer

Model Reference Guide

Complete specifications for all recommended models. Download links, file paths, and compatibility notes.

Directory Structure

ComfyUI/
├── models/
│   ├── checkpoints/          # Base models (FLUX, SDXL, SD1.5)
│   ├── loras/                # LoRA adapters
│   ├── controlnet/           # ControlNet models
│   ├── clip_vision/          # CLIP vision encoders
│   ├── ipadapter/            # IP-Adapter models
│   ├── instantid/            # InstantID models
│   ├── insightface/          # Face analysis models
│   ├── facerestore_models/   # Face restoration (GFPGAN, CodeFormer)
│   ├── ultralytics/bbox/     # Detection models
│   ├── upscale_models/       # Upscaler models
│   └── diffusion_models/     # Video diffusion models (Wan)
└── custom_nodes/
    └── ComfyUI-AnimateDiff-Evolved/models/  # Motion modules

---

Checkpoint Models

FLUX.1-dev (Recommended for Photorealism)

• Download: https://huggingface.co/black-forest-labs/FLUX.1-dev
• Files: flux1-dev.safetensors (23.8GB)
• Path: models/checkpoints/
• Also requires:
• T5 encoder: t5xxl_fp16.safetensors → models/clip/
• CLIP-L: clip_l.safetensors → models/clip/
• VAE: ae.safetensors → models/vae/
• VRAM: 16GB+ (FP16), 10GB (FP8)
• Notes: Best photorealism, slow generation. Use --fp8_e4m3fn-unet for VRAM savings.

FLUX Kontext (Character Editing)

• Download: https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev
• Path: models/checkpoints/
• Use case: Edit existing character images without retraining
• VRAM: 16GB+

RealVisXL V5.0 (Fast SDXL Photorealism)

• Download: https://civitai.com/models/139562/realvisxl-v50
• File: RealVisXL_V5.0.safetensors
• Path: models/checkpoints/
• VRAM: 8GB+
• Notes: Good balance of speed and quality for SDXL workflows

Juggernaut XL Ragnarok

• Download: https://civitai.com/models/133005/juggernaut-xl
• Path: models/checkpoints/
• VRAM: 8GB+
• Notes: Excellent for diverse human subjects

---

Identity Preservation Models

InstantID

• IP-Adapter: https://huggingface.co/InstantX/InstantID
• File: ip-adapter.bin → models/instantid/
• ControlNet: Same repo
• File: ControlNetModel/diffusion_pytorch_model.safetensors → models/controlnet/
• Rename to: instantid_controlnet.safetensors
• Requires: InsightFace antelopev2 model
• VRAM: 8GB+ additional

IP-Adapter FaceID Plus V2

• Download: https://huggingface.co/h94/IP-Adapter-FaceID
• Files:
• ip-adapter-faceid-plusv2_sd15.bin → models/ipadapter/
• ip-adapter-faceid-plusv2_sd15_lora.safetensors → models/loras/
• VRAM: 6GB+ additional
• Notes: Faster than InstantID, auto-loads paired LoRA

IP-Adapter SDXL

• Download: https://huggingface.co/h94/IP-Adapter
• Files:
• ip-adapter_sdxl_vit-h.safetensors → models/ipadapter/
• ip-adapter-plus_sdxl_vit-h.safetensors → models/ipadapter/
• ip-adapter-plus-face_sdxl_vit-h.safetensors → models/ipadapter/
• CLIP Vision: CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors → models/clip_vision/

PuLID Flux II

• Download: https://huggingface.co/guozinan/PuLID
• File: pulid_flux_v0.9.1.safetensors → models/pulid/
• Requires: EVA-CLIP model
• VRAM: 16GB+
• Notes: Highest fidelity, no model pollution, slowest

InsightFace (Required for all face methods)

• Download: https://huggingface.co/datasets/Gourieff/ReActor/tree/main/models
• Files:
• inswapper_128.onnx → models/insightface/
• buffalo_l/ folder → models/insightface/models/buffalo_l/
• antelopev2/ folder → models/insightface/models/antelopev2/

---

ControlNet Models

SDXL ControlNet

• OpenPose: control-lora-openposeXL2-rank256.safetensors
• Depth: control-lora-depth-rank256.safetensors
• Canny: control-lora-canny-rank256.safetensors
• Download: https://huggingface.co/stabilityai/control-lora
• Path: models/controlnet/

FLUX ControlNet (Union)

• Download: https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union
• File: diffusion_pytorch_model.safetensors
• Path: models/controlnet/
• Notes: Single model handles multiple control types

SD 1.5 ControlNet

• Download: https://huggingface.co/lllyasviel/ControlNet-v1-1
• Files: control_v11p_sd15_openpose.pth, etc.
• Path: models/controlnet/

---

Face Restoration Models

CodeFormer

• Download: https://github.com/sczhou/CodeFormer/releases
• File: codeformer.pth → models/facerestore_models/

GFPGAN

• Download: https://github.com/TencentARC/GFPGAN/releases
• File: GFPGANv1.4.pth → models/facerestore_models/

RestoreFormer

• Download: https://github.com/wzhouxiff/RestoreFormer
• File: RestoreFormer.pth → models/facerestore_models/

---

Detection Models (for FaceDetailer)

YOLO Face Detection

• Download: https://huggingface.co/Bingsu/adetailer/tree/main
• Files:
• face_yolov8m.pt → models/ultralytics/bbox/
• face_yolov8n.pt → models/ultralytics/bbox/
• hand_yolov8n.pt → models/ultralytics/bbox/

SAM (Segment Anything)

• Download: https://huggingface.co/spaces/abhishek/StableSAM/tree/main
• File: sam_vit_b_01ec64.pth → models/sams/

---

Upscale Models

4x-UltraSharp

• Download: https://civitai.com/models/116225/4x-ultrasharp
• Path: models/upscale_models/
• Notes: Best for faces and fine detail

4x-Foolhardy-Remacri

• Download: https://civitai.com/models/40067
• Path: models/upscale_models/
• Notes: Good general purpose

SUPIR

• Download: https://huggingface.co/Kijai/SUPIR_pruned
• Path: models/upscale_models/
• Notes: AI-enhanced upscaling, slower but better quality

---

Video Models

Wan 2.1 (Recommended)

• 14B T2V: wan2.1_t2v_14b_bf16.safetensors
• 14B I2V: wan2.1_i2v_720p_14b_bf16.safetensors
• 1.3B T2V: wan2.1_t2v_1.3b_bf16.safetensors
• Download: https://huggingface.co/Wan-AI/Wan2.1-Preview
• Path: models/diffusion_models/
• Text Encoder: umt5_xxl_fp8_e4m3fn_scaled.safetensors → models/clip/
• CLIP: open_clip_vit_h_14.safetensors → models/clip_vision/
• VAE: wan_2.1_vae.safetensors → models/vae/

AnimateDiff

• Motion Module V3: v3_sd15_mm.ckpt
• Lightning (fast): animatediff_lightning_4step.safetensors
• Download: https://huggingface.co/guoyww/animatediff
• Path: custom_nodes/ComfyUI-AnimateDiff-Evolved/models/

Motion LoRAs

• Download: https://huggingface.co/guoyww/animatediff/tree/main/motion_lora
• Files: v2_lora_ZoomIn.ckpt, v2_lora_PanLeft.ckpt, etc.
• Path: custom_nodes/ComfyUI-AnimateDiff-Evolved/motion_lora/

---

Voice/Lip-Sync Models

Wav2Lip

• Download: https://github.com/Rudrabha/Wav2Lip
• Files:
• wav2lip_gan.pth (better quality)
• wav2lip.pth (faster)
• Path: Varies by ComfyUI node implementation

SadTalker

• Download: https://github.com/OpenTalker/SadTalker
• Required models: Multiple, see repo releases
• Notes: Generates head motion + expressions from audio

LivePortrait

• Download: https://github.com/KwaiVGI/LivePortrait
• Notes: Best expression control, non-commercial license for InsightFace components

---

LoRA Training Tools

Kohya_ss (SDXL/SD1.5)

• Repo: https://github.com/bmaltais/kohya_ss
• Notes: Most mature, GUI available

AI-Toolkit (FLUX)

• Repo: https://github.com/ostris/ai-toolkit
• Notes: Beginner-friendly, 24GB VRAM for standard training

SimpleTuner (FLUX, Advanced)

• Repo: https://github.com/bghira/SimpleTuner
• Notes: NF4 quantization enables 9GB VRAM training

FluxGym (Low VRAM FLUX)

• Repo: https://github.com/cocktailpeanut/fluxgym
• Notes: Optimized for 12-16GB VRAM

---

Skin Texture LoRAs (Combine with Character LoRA)

Realistic Skin Texture

• Download: https://civitai.com/models/238756
• File: Realistic_Skin_Texture_SkinF1dV2.5.safetensors
• Path: models/loras/
• Strength: 0.2-0.4 (subtle enhancement)

Detail Tweaker

• Download: https://civitai.com/models/122359
• Path: models/loras/
• Notes: Enhances fine detail without changing style

ComfyUI Character Generation: 2024-2025 Comprehensive Techniques Guide

The ComfyUI ecosystem has undergone a dramatic transformation since 2024, with InfiniteYou, FLUX Kontext, and PuLID Flux II emerging as the new gold standards for character consistency—while many foundational repositories from cubiq (InstantID, PuLID, IP-Adapter) entered maintenance mode in April 2025. Video generation has been revolutionized by Wan 2.2's MoE architecture and FramePack's 6GB VRAM breakthrough for 60-second videos. Voice cloning integration reached maturity with unified platforms like TTS Audio Suite supporting 23+ languages and emotion control. This report synthesizes the latest techniques, workflows, and community discoveries to update character generation capabilities.

---

Pixaroma's ComfyUI mastery: Accessibility-first workflows

Pixaroma has established itself as the definitive beginner-to-advanced ComfyUI resource, run by a graphic designer who emphasizes GGUF quantized models for users with 8-12GB VRAM. The channel's structured episode format builds systematically from installation basics through advanced video generation.

The character consistency methodology from Episode 28 follows a multi-step pipeline: generate character sheets showing multiple angles and poses, train LoRAs online using these generated images, then deploy locally with OpenPose and ControlNet for pose control. This approach prioritizes accessibility over cutting-edge methods, making it ideal for creators with consumer hardware.

Key innovations include the Pixaroma Upscaler workflow using Flux Dev GGUF Q8, which employs dual outputs—one from KSampler at 0.75-0.80 denoise (preserving character features) and one further upscaled without KSampler. The 260+ art styles collection in CSV format provides curated SDXL-compatible styles organized by category, with the recommendation to mix painting and photography styles for optimal results.

For video generation, Pixaroma's Wan 2.1 tutorials provide precise specifications: 832×480px for landscape, 480×832px for portrait, with frame calculations at 16 FPS (17 frames = 1 second, 81 frames = 5 seconds). The ComfyUI-Easy-Install Pixaroma Community Edition bundles essential nodes including GGUF support, VideoHelperSuite, WanVideoWrapper, and Impact Pack for one-click setup.

---

Character consistency enters a new era with InfiniteYou and FLUX Kontext

The character consistency landscape fundamentally shifted in 2024-2025. InfiniteYou from ByteDance (ICCV 2025 Highlight) introduces InfuseNet architecture with residual connections and multi-stage training using single-person-multiple-sample data. Two model variants serve different needs: SIM maximizes identity matching while AES prioritizes visual quality. FP8 and GGUF support enable lower VRAM requirements with multi-ID masked generation.

FLUX.1 Kontext from Black Forest Labs represents perhaps the most significant advancement—a context-aware editing model with built-in character consistency that maintains features across multiple editing steps. The model excels at localized editing (targeting specific elements without affecting others) and iterative refinement with minimal drift. Three tiers exist: Kontext [dev] at 12B parameters for research, Kontext [pro] running 8x faster than GPT-Image, and Kontext [max] with improved prompt adherence. FP8 versions reduce requirements to ~12GB VRAM.

PuLID Flux II solved the critical "model pollution" issue using contrastive alignment techniques that minimize disruption to the original FLUX model. The dual-character support enables generating scenes with two consistent people—previously extremely difficult. Mickmumpitz's "Flux Consistent Characters" workflow remains the community gold standard, though it requires A100-40GB GPU at approximately 7.5 minutes per generation.

For SDXL workflows, the proven combination of InstantID + IP-Adapter + ControlNet + FaceDetailer continues to deliver excellent results, though these cubiq repositories are now in maintenance mode. Key optimizations include using 1016×1016 resolution (avoiding 1024×1024 watermark artifacts), lowering CFG to 4-5, and applying 35% noise injection in negative embeds to reduce the "burn" effect.

---

Video generation transforms with Wan 2.2 and FramePack

Wan 2.2 from Alibaba introduces a Mixture of Experts architecture where high-noise and low-noise expert models collaborate, enabling film-level aesthetic control over lighting, color, and composition. The groundbreaking first-and-last frame generation controls both start and end frames for precise video planning. Models range from the consumer-friendly 1.3B version requiring ~8.19GB VRAM to the 14B version for maximum quality.

The FramePack breakthrough from Dr. Lvmin Zhang (ControlNet author) makes VRAM usage invariant to video length—generating 60-second videos at 30fps (1800 frames) on just 6GB VRAM via an RTX 3060. Dynamic context compression assigns 1536 markers to key frames versus 192 for transitional frames, while bidirectional memory with reverse generation prevents the drift that plagued earlier methods.

AnimateDiff V3 with motion modules (mm_sd_v15_v2) supports camera controls through Motion LoRAs—pan, zoom, tilt, rolling—while effect LoRAs enable shatter, smoke, explosion, and liquid motion effects. The sliding context window system enables infinite animation length, though the model works best at 512x512 resolution (its training resolution). For SDXL, HotshotXL provides temporal layer support.

Lip-sync technology matured significantly with LatentSync 1.6 from ByteDance training at 512×512 for improved clarity, using TREPA modules for temporal consistency. InfiniteTalk enables unlimited-length talking avatar videos with WAN integration, while the emerging Character AI Ovi pipeline generates joint video and audio with natural lip synchronization from a single image input.

---

Voice cloning reaches production quality with unified platforms

TTS Audio Suite emerged as the definitive multi-engine platform, integrating F5-TTS, Chatterbox (23 languages), Higgs Audio 2, VibeVoice (90-minute generation), IndexTTS-2 (8-emotion vector control), and RVC for real-time conversion. Advanced features include character switching with [CharacterName] tags, language switching ([de:Alice], [fr:Bob]), pause control ([pause:1s]), and SRT subtitle timing synchronization.

F5-TTS delivers zero-shot voice cloning from samples under 15 seconds with multi-language support across English, German, Spanish, French, Japanese, Hindi, Thai, and Portuguese. Voice samples must be paired with .wav and .txt files containing matching transcriptions. The model integrates Whisper for automatic transcription.

Chatterbox from ResembleAI supports paralinguistic tags ([laugh], [sigh], [gasp]) and multi-speaker dialog synthesis for up to 4 voices per generation. The exaggeration parameter (0.25-2.0) controls expressiveness, while a 40-second generation limit requires splitting longer content.

For emotion control, IndexTTS-2 provides an 8-emotion vector system covering happy, angry, sad, surprised, afraid, disgusted, calm, and melancholic states with per-segment parameter control.

---

LoRA training evolves with FLUX-specific optimizations

FLUX LoRA training differs substantially from SDXL approaches. FluxGym provides the most beginner-friendly path with web-based Gradio interface, integrated Florence 2 auto-captioning, and RunPod templates. Recommended settings: 15-20 high-quality images, unique trigger tokens (e.g., ch3rrybl0nde), network rank 16-32, and 2000 steps.

SimpleTuner targets production environments with support for FLUX.1 Dev and FLUX.2 with Mistral-3 encoder. The int8/nf4 quantization enables training on 20GB VRAM via Optimum-Quanto. Key parameters: learning rate 1e-4 to 1e-5 (lower than SD), network rank 4-64 (start small), polynomial LR schedule.

For SDXL, Kohya_ss remains the gold standard with Prodigy optimizer for auto-tuning, block-wise dims/lrs support, and recommended rank of 64-96 with alpha at half the rank value. Dataset preparation requires high-resolution images (1024×1024 minimum), multiple angles and expressions (10-20 images), and captions with trigger words first followed by detailed descriptions—but excluding unchangeable features like eye color.

---

Memory optimization unlocks consumer hardware potential

VRAM management fundamentally determines workflow feasibility. Command-line flags provide the first optimization layer: --lowvram for 6GB systems trades speed for memory efficiency, --reserve-vram [amount] prevents system freezing by reserving VRAM for OS functions, and --use-xformers changes attention scaling from quadratic to near-linear—reducing 16GB requirements to 4GB for 1024×1024 generation.

FP8 precision for SDXL/Flux models halves memory usage with minimal quality loss—FLUX checkpoints use ~6GB VRAM versus much more for FP16/BF16. The VRAM_Debug node from KJNodes monitors free VRAM before/after operations with built-in garbage collection and model unloading. Restarting ComfyUI between significantly different workloads clears memory fragmentation.

Sampling optimization delivers dramatic speed improvements: 15-25 steps suffices for most generations since quality plateaus around 25-30 steps. DPM++ 2M Karras achieves excellent results at 20 steps, while DEIS sampler achieves high-fidelity in just 10 steps. Lightning/Turbo models enable 4-8 step generation; LCM achieves 4 steps for rapid iteration.

A critical community discovery: updating cuDNN to version 8800 can double speed on RTX 4070Ti from 8.5it/s to 17.6it/s. Reserving 7GB VRAM in ComfyUI settings made workflows run 20x faster on 5070ti in community testing.

---

Essential custom nodes for character generation pipelines

ComfyUI Manager remains the foundational requirement—enabling one-click installation, auto-updates, and missing node detection. Impact Pack provides FaceDetailer for automatic face enhancement and image segmentation. WAS Node Suite delivers 300+ nodes for image processing, text manipulation, and file operations.

For upscaling, SUPIR (Scaling-UP Image Restoration) uses diffusion-based enhancement achieving photo-realistic results—512p to 2048 with under 10GB VRAM in FP8 mode. Key parameters: CFG scale 2.5→1.5, EDM s_churn ~5. The 4x Foolhardy Remacri model provides superior texture reconstruction, while 4x AnimeSharp optimizes for illustration styles.

New samplers and schedulers emerged for specific use cases. The Beta scheduler (α=1.0, β=0.6) delivers improved details for Flux.1 [dev] with Euler sampler. ComfyUI-CapitanFlowMatch provides optimal samplers for rectified flow models (Flux, SD3, Lumina2), preventing the "burning" issues with turbo models at higher step counts.

For ControlNet, FLUX ControlNet Inpainting from Alimama (beta model trained on 15M images at 1024×1024) outperforms SDXL inpainting with conditioning scale between 0.9-0.95 for best results.

---

Community workflows and cutting-edge discoveries

Consistent Character Creator 3.0 using the Qwen Image Edit model generates multi-view character sheets (front, side, back views, expressions, turntable rotations) while preserving identity. The Character AI Ovi pipeline enables single-image-to-talking-avatar with joint video and audio generation and natural lip synchronization.

StoryDiffusion (NeurIPS 2024 Spotlight) provides training-free consistent text-to-image generation through Consistent Self-Attention, requiring minimum 3 text prompts (5-6 recommended) for character consistency across sequences. It works as a hot-pluggable module compatible with both SD1.5 and SDXL.

ComfyUI Copilot from Alibaba offers AI-assisted workflow generation with claimed 10x faster workflow creation. ACE Plus technology achieves 99% facial consistency for background replacement (versus previous 90%). HyperLoRA (CVPR 2025) introduces parameter-efficient adaptive generation for portrait synthesis.

For workflow discovery, OpenArt provides curated workflows with previews, Civitai offers model-specific workflows, and awesome-comfyui on GitHub maintains a daily-updated collection. The Searge SDXL Evolved v4.32 workflow on Civitai represents one of the most optimized community workflows with ControlNet, Revision, and multi-LoRA support.

---

2026 Updates: FLUX.2, LTX-2, and Platform Optimizations

FLUX.2 (Flux2) from Black Forest Labs represents a major evolution in identity preservation, supporting up to 10 reference images in a single generation for unprecedented identity consistency. This multi-reference capability is especially valuable for branded content, recurring characters, and multi-scene creative workflows where maintaining visual consistency across diverse contexts is critical. The model demonstrates strong preservation of character identity, product appearance, and visual style across complex scenarios.

LTX-2 from Lightricks (January 2026) achieved a significant milestone as the first production-ready open-source model combining truly open audio and video generation with native 4K output. This represents a substantial leap in quality and accessibility for high-resolution video generation workflows, eliminating previous limitations around resolution ceiling or proprietary audio systems.

AuraFace emerged as a commercially-friendly identity encoder—an open-source alternative to ArcFace specifically designed for commercial applications without licensing restrictions. This addresses a long-standing pain point where ArcFace's licensing prevented commercial use, making advanced face recognition and identity preservation technologies accessible for production environments. AuraFace integrates seamlessly with existing InstantID and IP-Adapter workflows as a drop-in replacement for identity encoding.

F5-TTS Cross-Lingual framework (February 2026) enables cross-lingual voice cloning without requiring audio prompt transcripts, matching F5-TTS performance while unlocking multilingual voice transfer capabilities. This dramatically simplifies workflows where reference audio lacks transcription or involves code-switching between languages—common scenarios in international content production.

ComfyUI v0.8.1 (January 8, 2026) introduced critical performance optimizations for RTX 50 Series GPUs with NVFP4 and NVFP8 data format support, achieving 3x faster performance and 60% VRAM reduction compared to previous precision formats. Native AMD ROCm integration significantly improved generation speeds on AMD graphics cards and SoCs, expanding hardware compatibility. The new audio recording node enables direct audio capture within ComfyUI, streamlining voice synthesis workflows. Weight streaming memory management allows using system RAM when VRAM is exhausted, enabling larger models on mid-range GPUs.

DiffSwap++ introduces the first face-swapping framework leveraging 3D facial latent features to guide diffusion-based generation, incorporating 3D geometric cues during training while remaining fully 2D at inference. This demonstrates superior identity preservation and structural consistency compared to 2D-only approaches, particularly for challenging angles and expressions.

---

Conclusion: Building the complete character pipeline

The 2024-2025 ComfyUI ecosystem enables truly professional character generation workflows. For new projects, InfiniteYou or FLUX Kontext represent the state-of-the-art, while SDXL workflows should continue leveraging InstantID + IP-Adapter combinations despite maintenance mode status. Production environments benefit from FLUX Kontext Pro/Max via API for maximum consistency.

The optimal multi-modal pipeline follows this sequence: generate consistent characters with LoRA-enhanced FLUX/SDXL models → clone voices with F5-TTS or Chatterbox using 10-30 second references → create video with Wan 2.2 or FramePack → apply lip-sync with LatentSync 1.6 or InfiniteTalk → post-process with RIFE interpolation and SUPIR upscaling.

Key architectural insights: FramePack's VRAM-invariant approach enables 60-second videos on consumer GPUs; TTS Audio Suite's unified platform eliminates fragmented voice cloning setups; PuLID Flux II's contrastive alignment solves model pollution for dual-character generation. The field continues rapid evolution—monitor the community forks emerging from maintenance-mode repositories and emerging models like CharaConsist's point-tracking attention for next-generation consistency.

Talking Head Workflow Reference

Complete production workflows for creating talking head videos using two main approaches.

---

Table of Contents

1. Approach 1: Image → Talking Head 2. Approach 2: Video → Add Voice/Lip-Sync 3. Hybrid Approach 4. Parameter Reference 5. Troubleshooting

---

Approach 1: Image → Talking Head

Use Case: Generate talking head video from single character portrait.

Input: 1 image (1024×1024 recommended) + audio file Output: Talking head video with natural head movement and lip-sync Time: 2-5 minutes per 10-second clip (GPU-dependent)

Method 1A: SadTalker (Most Popular)

Best For: Natural head movement, expressions, eye blink

Installation

# Clone SadTalker
cd E:/ComfyUI-Easy-Install/ComfyUI/custom_nodes
git clone https://github.com/Winfredy/SadTalker

# Install dependencies
cd SadTalker
pip install -r requirements.txt

# Download checkpoints
python scripts/download_models.py

Required Models:

• checkpoints/SadTalker_V0.0.2_512.safetensors (4.5GB)
• gfpgan/weights/ (face enhancement)
• checkpoints/mapping_00109-model.pth.tar (expression)
• checkpoints/mapping_00229-model.pth.tar (pose)

Python Script

#!/usr/bin/env python3
"""
SadTalker: Image to Talking Head Video
Generates natural talking head with head movement from single portrait
"""

import os
import sys
sys.path.append('E:/ComfyUI-Easy-Install/ComfyUI/custom_nodes/SadTalker/src')

from facerender.animate import AnimateFromCoeff
from test_audio2coeff import Audio2Coeff
from face3d.models import networks

class SadTalkerPipeline:
    def __init__(self, checkpoint_dir='checkpoints'):
        self.checkpoint_dir = checkpoint_dir
        self.device = 'cuda'

        # Load models
        self.audio2coeff = Audio2Coeff(checkpoint_dir, self.device)
        self.animate = AnimateFromCoeff(checkpoint_dir, self.device)

    def generate(
        self,
        source_image,
        driven_audio,
        output_path,
        preprocess='crop',  # 'crop', 'resize', 'full'
        still_mode=False,   # Less head movement
        use_enhancer=True,  # GFPGAN enhancement
        batch_size=2,
        fps=25
    ):
        """
        Generate talking head video

        Args:
            source_image: Path to portrait (1024x1024 recommended)
            driven_audio: Path to audio file (.wav, .mp3)
            output_path: Output video path
            preprocess: How to handle input image
            still_mode: True for minimal head movement
            use_enhancer: Apply GFPGAN for face quality
            batch_size: Processing batch size
            fps: Output video frame rate
        """

        print(f"[SadTalker] Processing: {source_image}")
        print(f"[SadTalker] Audio: {driven_audio}")

        # 1. Extract audio coefficients
        print("[SadTalker] Extracting audio features...")
        coeff_path = self.audio2coeff.generate(
            audio_path=driven_audio,
            save_dir=os.path.dirname(output_path)
        )

        # 2. Generate video
        print("[SadTalker] Generating video...")
        self.animate.generate(
            source_image=source_image,
            coeff_path=coeff_path,
            output_path=output_path,
            preprocess=preprocess,
            still_mode=still_mode,
            use_enhancer=use_enhancer,
            batch_size=batch_size,
            fps=fps
        )

        print(f"[SadTalker] ✓ Video saved: {output_path}")
        return output_path

# Example usage
if __name__ == "__main__":
    pipeline = SadTalkerPipeline()

    pipeline.generate(
        source_image="E:/ComfyUI-Easy-Install/ComfyUI/input/sage_portrait.png",
        driven_audio="E:/ComfyUI-Easy-Install/ComfyUI/input/sage_dialogue.wav",
        output_path="E:/ComfyUI-Easy-Install/ComfyUI/output/sage_talking.mp4",
        still_mode=False,      # Natural head movement
        use_enhancer=True,     # GFPGAN enhancement
        fps=25
    )

ComfyUI Workflow (SadTalker Node)

{
  "1": {
    "class_type": "LoadImage",
    "inputs": {
      "image": "sage_portrait.png"
    }
  },
  "2": {
    "class_type": "LoadAudio",
    "inputs": {
      "audio": "sage_dialogue.wav"
    }
  },
  "3": {
    "class_type": "SadTalker",
    "inputs": {
      "source_image": ["1", 0],
      "driven_audio": ["2", 0],
      "preprocess": "crop",
      "still_mode": false,
      "use_enhancer": true,
      "expression_scale": 1.0,
      "fps": 25
    }
  },
  "4": {
    "class_type": "SaveVideo",
    "inputs": {
      "video": ["3", 0],
      "filename": "sage_talking",
      "fps": 25
    }
  }
}

Parameter Guide: SadTalker

Recommended Settings:

# Natural conversation (YouTube, podcast)
still_mode=False, expression_scale=1.0

# Professional presentation (corporate)
still_mode=True, expression_scale=0.8

# Animated character (exaggerated)
still_mode=False, expression_scale=1.5

---

Method 1B: LivePortrait (Premium Quality)

Best For: Fine control over expressions, eye gaze, head pose

Installation

cd E:/ComfyUI-Easy-Install/ComfyUI/custom_nodes
git clone https://github.com/KwaiVGI/LivePortrait

cd LivePortrait
pip install -r requirements.txt
python scripts/download_models.py

Python Script

#!/usr/bin/env python3
"""
LivePortrait: Advanced Talking Head Generation
Fine control over expressions, eye gaze, and head movements
"""

import sys
sys.path.append('E:/ComfyUI-Easy-Install/ComfyUI/custom_nodes/LivePortrait')

from liveportrait.live_portrait_pipeline import LivePortraitPipeline

class LivePortraitGenerator:
    def __init__(self):
        self.pipeline = LivePortraitPipeline(
            model_path='checkpoints/liveportrait',
            device='cuda'
        )

    def generate(
        self,
        source_image,
        driving_audio,
        output_path,
        expression_strength=1.0,
        head_pose_strength=1.0,
        eye_retargeting=True,
        lip_retargeting=True,
        fps=25
    ):
        """
        Generate talking head with fine control

        Args:
            source_image: Portrait image path
            driving_audio: Audio file path
            output_path: Output video path
            expression_strength: Expression intensity (0-2)
            head_pose_strength: Head movement intensity (0-2)
            eye_retargeting: Enable eye movement
            lip_retargeting: Enable lip-sync
            fps: Output frame rate
        """

        print(f"[LivePortrait] Processing: {source_image}")

        result = self.pipeline.execute(
            source_image=source_image,
            driving_audio=driving_audio,
            output_path=output_path,
            flag_pasteback=True,           # Paste face back to original
            flag_do_crop=True,             # Crop face region
            flag_stitching=True,           # Smooth face boundary
            expression_friendly=expression_strength,
            pose_friendly=head_pose_strength,
            flag_eye_retargeting=eye_retargeting,
            flag_lip_retargeting=lip_retargeting,
            fps=fps
        )

        print(f"[LivePortrait] ✓ Video saved: {output_path}")
        return output_path

# Example usage
if __name__ == "__main__":
    generator = LivePortraitGenerator()

    generator.generate(
        source_image="E:/ComfyUI-Easy-Install/ComfyUI/input/character.png",
        driving_audio="E:/ComfyUI-Easy-Install/ComfyUI/input/dialogue.wav",
        output_path="E:/ComfyUI-Easy-Install/ComfyUI/output/character_talking.mp4",
        expression_strength=1.2,    # Slightly exaggerated
        head_pose_strength=0.8,     # Subtle head movement
        eye_retargeting=True,       # Natural eye movement
        lip_retargeting=True        # Precise lip-sync
    )

---

Method 1C: EMO (State-of-Art 2024)

Best For: Most natural results, viral quality demos

Note: EMO is research code, less production-ready than SadTalker/LivePortrait

# EMO is typically run via research codebase
# Not recommended for production yet but worth monitoring
# See: https://github.com/HumanAIGC/EMO

---

Approach 2: Video → Add Voice/Lip-Sync

Use Case: Add voice/dialogue to existing video footage.

Input: Video (any source) + audio file Output: Same video with synchronized lip movements Time: 1-3 minutes per 10-second clip

Method 2A: Wav2Lip (Industry Standard)

Best For: Best lip-sync accuracy, works with any video

Installation

cd E:/ComfyUI-Easy-Install/ComfyUI/custom_nodes
git clone https://github.com/Rudrabha/Wav2Lip

cd Wav2Lip
pip install -r requirements.txt

# Download checkpoint
# Manual download: https://github.com/Rudrabha/Wav2Lip
# Place in: Wav2Lip/checkpoints/wav2lip_gan.pth

Python Script

#!/usr/bin/env python3
"""
Wav2Lip: Add Lip-Sync to Existing Video
Industry-standard lip synchronization for any video
"""

import sys
import subprocess
import os

class Wav2LipPipeline:
    def __init__(self, wav2lip_path='E:/ComfyUI-Easy-Install/ComfyUI/custom_nodes/Wav2Lip'):
        self.wav2lip_path = wav2lip_path
        self.checkpoint = os.path.join(wav2lip_path, 'checkpoints/wav2lip_gan.pth')

    def apply_lipsync(
        self,
        video_path,
        audio_path,
        output_path,
        quality='improved',  # 'improved' or 'enhanced'
        face_det_batch_size=16,
        wav2lip_batch_size=128,
        resize_factor=1,
        fps=25
    ):
        """
        Apply lip-sync to video

        Args:
            video_path: Input video with face(s)
            audio_path: Audio to sync (.wav recommended)
            output_path: Output synced video
            quality: 'improved' (GAN) or 'enhanced' (expert discriminator)
            face_det_batch_size: Face detection batch (lower if OOM)
            wav2lip_batch_size: Processing batch (lower if OOM)
            resize_factor: Resize input (1=original, 2=half size)
            fps: Output frame rate
        """

        print(f"[Wav2Lip] Syncing: {video_path}")
        print(f"[Wav2Lip] Audio: {audio_path}")

        # Build command
        cmd = [
            'python',
            os.path.join(self.wav2lip_path, 'inference.py'),
            '--checkpoint_path', self.checkpoint,
            '--face', video_path,
            '--audio', audio_path,
            '--outfile', output_path,
            '--fps', str(fps),
            '--face_det_batch_size', str(face_det_batch_size),
            '--wav2lip_batch_size', str(wav2lip_batch_size),
            '--resize_factor', str(resize_factor)
        ]

        if quality == 'improved':
            cmd.append('--nosmooth')  # GAN checkpoint

        # Run Wav2Lip
        result = subprocess.run(cmd, cwd=self.wav2lip_path, capture_output=True)

        if result.returncode == 0:
            print(f"[Wav2Lip] ✓ Synced video saved: {output_path}")
            return output_path
        else:
            print(f"[Wav2Lip] ✗ Error: {result.stderr.decode()}")
            raise RuntimeError("Wav2Lip failed")

# Example usage
if __name__ == "__main__":
    pipeline = Wav2LipPipeline()

    # Apply lip-sync to existing video
    pipeline.apply_lipsync(
        video_path="E:/ComfyUI-Easy-Install/ComfyUI/output/sage_video_clip1.mp4",
        audio_path="E:/ComfyUI-Easy-Install/ComfyUI/input/sage_dialogue.wav",
        output_path="E:/ComfyUI-Easy-Install/ComfyUI/output/sage_lipsync_clip1.mp4",
        quality='improved',
        face_det_batch_size=16,
        wav2lip_batch_size=128,
        fps=25
    )

Full Production Pipeline: Video → Voice → Enhance

#!/usr/bin/env python3
"""
Complete Production Pipeline: Existing Video → Voice → Lip-Sync → Enhance
For adding professional dialogue to pre-generated video clips
"""

import os
from pathlib import Path

# Import previous components
from wav2lip_pipeline import Wav2LipPipeline

class VideoVoicePipeline:
    def __init__(self):
        self.wav2lip = Wav2LipPipeline()

    def enhance_faces(self, video_path, output_path, fidelity=0.7):
        """
        Enhance face quality with CodeFormer

        Args:
            video_path: Input video (after Wav2Lip)
            output_path: Enhanced output video
            fidelity: Balance between quality and fidelity (0.5-1.0)
                     0.5 = creative quality, 1.0 = exact restoration
        """
        print(f"[CodeFormer] Enhancing faces: {video_path}")

        import subprocess

        cmd = [
            'python',
            'E:/ComfyUI-Easy-Install/ComfyUI/custom_nodes/CodeFormer/inference_codeformer.py',
            '-i', video_path,
            '-o', output_path,
            '--fidelity_weight', str(fidelity),
            '--bg_upsampler', 'realesrgan',  # Background upscaling
            '--face_upsample',
            '--draw_box',
            '--detection_model', 'retinaface'
        ]

        result = subprocess.run(cmd, capture_output=True)

        if result.returncode == 0:
            print(f"[CodeFormer] ✓ Enhanced video: {output_path}")
            return output_path
        else:
            print(f"[CodeFormer] Warning: Enhancement may have issues")
            return output_path

    def process_clip(
        self,
        video_path,
        audio_path,
        output_dir,
        clip_name,
        enhance=True,
        fidelity=0.7
    ):
        """
        Complete pipeline for one video clip

        Args:
            video_path: Input video clip
            audio_path: Dialogue audio
            output_dir: Output directory
            clip_name: Base filename
            enhance: Apply CodeFormer enhancement
            fidelity: CodeFormer fidelity weight

        Returns:
            Path to final video
        """

        os.makedirs(output_dir, exist_ok=True)

        # Step 1: Apply lip-sync
        lipsync_path = os.path.join(output_dir, f"{clip_name}_lipsync.mp4")
        self.wav2lip.apply_lipsync(
            video_path=video_path,
            audio_path=audio_path,
            output_path=lipsync_path,
            quality='improved'
        )

        if not enhance:
            return lipsync_path

        # Step 2: Enhance faces
        final_path = os.path.join(output_dir, f"{clip_name}_final.mp4")
        self.enhance_faces(
            video_path=lipsync_path,
            output_path=final_path,
            fidelity=fidelity
        )

        return final_path

    def process_batch(self, video_audio_pairs, output_dir):
        """
        Process multiple video clips

        Args:
            video_audio_pairs: List of (video_path, audio_path, clip_name) tuples
            output_dir: Output directory

        Returns:
            List of final video paths
        """

        results = []

        for i, (video, audio, name) in enumerate(video_audio_pairs):
            print(f"\n[Batch] Processing {i+1}/{len(video_audio_pairs)}: {name}")

            final = self.process_clip(
                video_path=video,
                audio_path=audio,
                output_dir=output_dir,
                clip_name=name,
                enhance=True
            )

            results.append(final)
            print(f"[Batch] ✓ Completed: {final}")

        return results

# Example usage: Process 5 Sage video clips
if __name__ == "__main__":
    pipeline = VideoVoicePipeline()

    # Define your video/audio pairs
    clips = [
        ("output/sage_clip1.mp4", "input/audio/sage_dialogue1.wav", "sage_01"),
        ("output/sage_clip2.mp4", "input/audio/sage_dialogue2.wav", "sage_02"),
        ("output/sage_clip3.mp4", "input/audio/sage_dialogue3.wav", "sage_03"),
        ("output/sage_clip4.mp4", "input/audio/sage_dialogue4.wav", "sage_04"),
        ("output/sage_clip5.mp4", "input/audio/sage_dialogue5.wav", "sage_05"),
    ]

    # Process all clips
    final_videos = pipeline.process_batch(
        video_audio_pairs=clips,
        output_dir="E:/ComfyUI-Easy-Install/ComfyUI/output/sage_final"
    )

    print("\n[Complete] All clips processed:")
    for video in final_videos:
        print(f"  - {video}")

    # Next step: Concatenate with scripts/video/concat_videos.py

Wav2Lip Parameter Guide

VRAM Requirements:

• 6GB: face_det_batch_size=4, wav2lip_batch_size=32, resize_factor=2
• 12GB: face_det_batch_size=8, wav2lip_batch_size=64, resize_factor=1
• 24GB+: face_det_batch_size=16, wav2lip_batch_size=128, resize_factor=1

---

Method 2B: MuseTalk (Faster Alternative)

Best For: Real-time processing, lower VRAM usage

#!/usr/bin/env python3
"""
MuseTalk: Faster Lip-Sync Alternative
Lower VRAM, faster inference than Wav2Lip
"""

# Installation
# pip install musetalk

from musetalk import MuseTalkModel

class MuseTalkPipeline:
    def __init__(self):
        self.model = MuseTalkModel(device='cuda')

    def apply_lipsync(self, video_path, audio_path, output_path):
        """Apply lip-sync with MuseTalk"""

        result = self.model.inference(
            video_path=video_path,
            audio_path=audio_path,
            output_path=output_path
        )

        return output_path

# Usage
pipeline = MuseTalkPipeline()
pipeline.apply_lipsync(
    video_path="input.mp4",
    audio_path="audio.wav",
    output_path="output.mp4"
)

MuseTalk vs Wav2Lip:

• ✅ Faster (3-5x speed)
• ✅ Lower VRAM (6GB vs 12GB)
• ❌ Slightly less accurate sync
• ❌ Less battle-tested

---

Hybrid Approach

For maximum production quality, combine both methods:

Workflow: Video + Selective Talking Head Enhancement

#!/usr/bin/env python3
"""
Hybrid: Use video motion + add talking head refinement for speaking sections
Best of both worlds
"""

import cv2
import numpy as np
from pathlib import Path

class HybridPipeline:
    def __init__(self):
        from sadtalker_pipeline import SadTalkerPipeline
        from wav2lip_pipeline import Wav2LipPipeline

        self.sadtalker = SadTalkerPipeline()
        self.wav2lip = Wav2LipPipeline()

    def process_hybrid(
        self,
        base_video,
        character_portrait,
        audio_segments,  # List of (start_time, end_time, audio_file)
        output_path
    ):
        """
        Apply SadTalker to specific speaking segments, Wav2Lip to full video

        Args:
            base_video: Full video with motion
            character_portrait: Character headshot
            audio_segments: Speaking segments with timestamps
            output_path: Final video output
        """

        # Step 1: Apply Wav2Lip to full video for baseline sync
        full_audio = self._combine_audio_segments(audio_segments)
        baseline_sync = "temp_baseline_sync.mp4"

        self.wav2lip.apply_lipsync(
            video_path=base_video,
            audio_path=full_audio,
            output_path=baseline_sync
        )

        # Step 2: Generate high-quality talking segments with SadTalker
        talking_segments = []
        for start, end, audio_file in audio_segments:
            segment_video = f"temp_segment_{start}_{end}.mp4"

            self.sadtalker.generate(
                source_image=character_portrait,
                driven_audio=audio_file,
                output_path=segment_video
            )

            talking_segments.append((start, end, segment_video))

        # Step 3: Composite - replace speaking sections with SadTalker output
        self._composite_segments(
            base_video=baseline_sync,
            talking_segments=talking_segments,
            output_path=output_path
        )

        return output_path

This is advanced production technique - most users should stick to Approach 1 or 2.

---

Parameter Reference

When to Use Each Method

Quality vs Speed Tradeoffs

Fastest → Slowest (for 10-second clip): 1. MuseTalk: ~30 seconds 2. Wav2Lip: ~60 seconds 3. SadTalker: ~120 seconds 4. LivePortrait: ~180 seconds 5. Hybrid: ~300 seconds

Quality Ranking (subjective): 1. Hybrid (best) 2. LivePortrait 3. SadTalker 4. Wav2Lip 5. MuseTalk (still good)

VRAM Requirements Summary

---

Troubleshooting

Issue: Face Not Detected

Symptoms: "No face found in image/video"

Solutions: 1. Ensure face is clearly visible, front-facing 2. Increase image resolution (min 512×512) 3. Use preprocess='crop' to auto-detect face region 4. Try different face detection models (retinaface, blazeface)

# SadTalker: Try different preprocess modes
preprocess='crop'   # Auto-detect and crop face (recommended)
preprocess='full'   # Use entire image
preprocess='resize' # Resize to expected size

Issue: Blurry/Low Quality Output

Symptoms: Output video looks blurry or degraded

Solutions: 1. Always use CodeFormer after Wav2Lip:

   self.enhance_faces(video_path, output_path, fidelity=0.7)

2. Adjust fidelity weight:

• fidelity=0.5: More enhancement, less faithful
• fidelity=0.7: Balanced (recommended)
• fidelity=1.0: Most faithful, least enhancement

3. Use higher input resolution:

• Source image: 1024×1024 minimum
• Video: 1080p minimum

4. SadTalker: Enable enhancer:

   use_enhancer=True  # Applies GFPGAN

Issue: Lip-Sync Accuracy Poor

Symptoms: Lips don't match audio timing

Solutions:

1. Check audio quality:

• Use clean audio without background noise
• Ensure clear speech, not overlapping
• Use .wav format (44.1kHz, 16-bit)

2. Wav2Lip: Adjust batch sizes:

   # Larger batch = more context = better sync
   wav2lip_batch_size=128  # Try 256 if you have VRAM

3. Try different quality modes:

   quality='improved'  # GAN (better quality)
   quality='enhanced'  # Expert discriminator (better sync)

4. Frame rate match:

• Ensure video and output fps match (25 or 30)
• Don't mix frame rates

Issue: Head Movement Too Stiff/Unnatural

Symptoms: SadTalker output looks robotic

Solutions:

1. Disable still_mode:

   still_mode=False  # Allows natural movement

2. Increase expression scale:

   expression_scale=1.2  # More expressive (range: 0.5-2.0)

3. Try LivePortrait for more control:

   head_pose_strength=1.0  # Natural head movement
   expression_strength=1.2  # Expressive faces

Issue: Out of Memory (OOM)

Symptoms: CUDA out of memory error

Solutions:

1. Reduce batch sizes:

   # SadTalker
   batch_size=1  # Minimum

   # Wav2Lip
   face_det_batch_size=4
   wav2lip_batch_size=16

2. Resize input:

   # Wav2Lip
   resize_factor=2  # Half resolution

3. Process shorter segments:

• Split video into 10-second clips
• Process individually
• Concatenate after

4. Enable gradient checkpointing (if available):

   use_gradient_checkpointing=True

Issue: Audio-Video Desync Over Time

Symptoms: Sync is good at start but drifts by end

Solutions:

1. Match frame rates exactly:

• Check input video fps: ffprobe input.mp4
• Set output fps to match: fps=25 or fps=30

2. Re-encode audio to exact duration:

   # Get video duration
   duration=$(ffprobe -v error -show_entries format=duration \
              -of default=noprint_wrappers=1:nokey=1 input.mp4)

   # Stretch/compress audio to match
   ffmpeg -i audio.wav -af atempo=$tempo -t $duration output.wav

3. Use constant frame rate video:

   # Convert variable frame rate to constant
   ffmpeg -i input.mp4 -vsync cfr -r 25 output.mp4

Issue: Face Replacement Visible/Jarring

Symptoms: Can see edges of face replacement, doesn't blend

Solutions:

1. Enable stitching (LivePortrait):

   flag_stitching=True  # Smooth boundaries
   flag_pasteback=True  # Blend with original

2. Use feathering:

   # Post-process with face boundary blur
   import cv2

   # Apply Gaussian blur to face mask edges
   mask = create_face_mask(frame)
   mask_blurred = cv2.GaussianBlur(mask, (21, 21), 11)

3. Match color/lighting:

• Apply color correction to match original
• Use histogram matching for lighting consistency

---

Next Steps After Generation

After generating talking head videos:

1. Concatenate clips (if multiple):

   python E:/ComfyUI-Easy-Install/ComfyUI/scripts/video/concat_videos.py

2. Add music/background audio:

   ffmpeg -i video.mp4 -i music.mp3 -c:v copy \
          -filter_complex "[0:a][1:a]amerge=inputs=2[a]" \
          -map 0:v -map "[a]" output.mp4

3. Color grading (optional):

• Use DaVinci Resolve (free)
• Or ffmpeg LUTs

4. Add subtitles (recommended):

• Use Whisper for transcription
• Burn-in with ffmpeg

5. Export for platform:

• YouTube: H.264, 1080p, 25fps
• Instagram: H.264, 1080×1350, 30fps
• TikTok: H.264, 1080×1920, 30fps

---

Production Checklist

Before deploying workflows:

• [ ] Test on 5-second clips first
• [ ] Verify face detection works on all frames
• [ ] Check audio is clean (no background noise)
• [ ] Match frame rates (video and output)
• [ ] Use proper audio format (.wav, 44.1kHz)
• [ ] Apply enhancement (CodeFormer) after Wav2Lip
• [ ] Validate output quality on target platform
• [ ] Batch process efficiently (don't overload GPU)
• [ ] Keep checkpoints/models organized
• [ ] Test full pipeline end-to-end before batch

---

References

• SadTalker: https://github.com/Winfredy/SadTalker
• LivePortrait: https://github.com/KwaiVGI/LivePortrait
• Wav2Lip: https://github.com/Rudrabha/Wav2Lip
• MuseTalk: https://github.com/TMElyralab/MuseTalk
• CodeFormer: https://github.com/sczhou/CodeFormer
• EMO (research): https://github.com/HumanAIGC/EMO

Last Updated: February 2026

Voice Synthesis & Lip-Sync Guide

Creating character voices and synchronizing them with video.

---

Voice Creation Decision Tree

Have reference audio of target voice?

• Yes → Voice cloning (RVC, ElevenLabs, XTTS)
• No → Voice design or TTS with persona tuning

Need commercial license?

• Yes → Chatterbox (MIT), F5-TTS (MIT), ElevenLabs (paid)
• No → RVC, XTTS (non-commercial), any

Quality priority vs speed?

• Quality → ElevenLabs Professional, Chatterbox
• Speed → F5-TTS, StyleTTS2

---

Voice Cloning Options

Chatterbox (Recommended Open-Source)

Why Chatterbox:

• Beat ElevenLabs in 63.8% of blind preference tests
• MIT license (commercial use OK)
• Only needs 5-second voice sample
• Emotion exaggeration control
• Native paralinguistic tags ([laugh], [sigh], etc.)
• Sub-200ms latency

Installation:

pip install chatterbox-tts

Basic usage:

from chatterbox import ChatterboxTTS

# Initialize
tts = ChatterboxTTS()

# Clone from audio file
tts.clone_voice("reference_audio.wav", voice_name="sage_voice")

# Generate speech
tts.synthesize(
    text="Hello, I'm Sage. *laughs* Nice to meet you.",
    voice="sage_voice",
    emotion_scale=1.2,  # 0.5-2.0, higher = more expressive
    output_path="output.wav"
)

Emotion tags:

[laugh] [chuckle] [sigh] [gasp] [cough] [clear throat]
[whisper] [excited] [sad] [angry] [surprised]

F5-TTS

Strengths:

• Zero-shot cloning from seconds of audio
• Code-switching (multiple languages)
• Very fast (RTF 0.15)
• MIT license

Installation:

pip install f5-tts

Usage:

from f5_tts import F5TTS

model = F5TTS()

# Generate with voice reference
audio = model.generate(
    text="This is Sage speaking.",
    ref_audio="voice_sample.wav",
    ref_text="Hello, my name is Sage.",  # Transcript of ref audio
    speed=1.0
)
audio.save("output.wav")

RVC (Retrieval-based Voice Conversion)

Use case: Convert any voice to your target voice

How it works: 1. Train RVC model on 10+ minutes of target voice audio 2. Generate base speech with any TTS 3. Convert base speech through RVC to target voice

Training requirements:

• 10+ minutes of clean speech (ideally 30+ minutes)
• Single speaker only
• No background music/noise
• Consistent recording quality

Training with RVC WebUI:

# Clone RVC
git clone https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI
cd Retrieval-based-Voice-Conversion-WebUI

# Install
pip install -r requirements.txt

# Launch
python infer-web.py

Training settings:

Training epochs: 300-500
Batch size: Based on VRAM (8 for 8GB, 16 for 16GB)
Save frequency: Every 50 epochs
Feature extraction: RMVPE (best quality)

Inference pipeline:

[Text] → [Any TTS] → [Base Audio] → [RVC Model] → [Character Voice]

XTTS-v2 (Coqui)

Note: Coqui shut down late 2024, but code is community-maintained.

Strengths:

• Clones from 6-second samples
• 17 language support
• <150ms streaming latency
• 85-95% similarity with 10 seconds

Limitation: Non-commercial license (CPML)

Usage:

from TTS.api import TTS

tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")

tts.tts_to_file(
    text="Hello, I'm Sage.",
    speaker_wav="reference.wav",
    language="en",
    file_path="output.wav"
)

ElevenLabs (Commercial)

Tiers:

• Instant Voice Cloning: 1-minute sample, good quality
• Professional Voice Cloning: 30+ minutes (3 hours ideal), near-indistinguishable

Voice Design (no sample needed):

Describe in 20-1000 characters:
"A warm, witty female voice in her late 20s. Slight Italian-American 
undertones. Confident but approachable. Quick, intelligent delivery 
with natural pauses. Occasionally playful. Sounds like she could be 
from Manhattan but educated at an Ivy League school."

API usage:

from elevenlabs import generate, clone

# Clone voice
voice = clone(
    name="Sage",
    files=["sample1.mp3", "sample2.mp3", "sample3.mp3"]
)

# Generate
audio = generate(
    text="Hello, I'm Sage.",
    voice=voice,
    model="eleven_multilingual_v2"
)

---

Creating Voice Without Reference Audio

Option 1: Voice Design (ElevenLabs)

Write detailed description matching character:

For Sage:

"A sophisticated, warm female voice, late 20s. Italian-American heritage 
subtly evident in certain vowel sounds. Intelligent, quick-witted delivery 
with natural rhythm. Confident but not arrogant. Capable of sharp sarcasm 
but more often warm and engaging. Higher education background evident in 
vocabulary and diction. Think a young Diane Lane meets Marisa Tomei - 
East Coast, cultured, but with authentic warmth."

Option 2: Find Similar Voice + RVC

1. Search ElevenLabs Voice Library (10,000+ voices) 2. Find voice with similar qualities 3. Use that as base, then train RVC on synthetic samples 4. Convert base voice through RVC for final output

Option 3: StyleTTS2 with Persona Prompting

from styletts2 import StyleTTS2

model = StyleTTS2()

# Use style reference audio that matches desired qualities
audio = model.synthesize(
    text="Hello, I'm Sage.",
    style_ref="warm_confident_female.wav",
    alpha=0.3,  # Style strength
    beta=0.7    # Prosody matching
)

---

Lip-Sync Technologies

Wav2Lip (Recommended for Accuracy)

Strengths:

• Best lip accuracy
• Works with any face
• Handles various angles

Limitations:

• Minimal head movement
• Requires face enhancement post-processing

ComfyUI integration:

# Install node
cd custom_nodes
git clone https://github.com/ShmuelRonen/ComfyUI_wav2lip

Node settings:

face_detect_batch: 16
nosmooth: false
wav2lip_model: "wav2lip_gan.pth"  # Better quality than wav2lip.pth
pad_bottom: 10  # Helps with chin visibility

Post-processing (required):

[Wav2Lip Output] → [CodeFormer (fidelity 0.7)] → [Final Video]

SadTalker

Strengths:

• Generates head movement
• Creates expressions from audio
• Single image input

Limitations:

• Less accurate lips than Wav2Lip
• Can look artificial with extreme movements

Command line:

python inference.py \
    --driven_audio audio.wav \
    --source_image character.png \
    --preprocess full \
    --enhancer gfpgan \
    --pose_style 0

Pose styles (0-45):

• 0: Minimal movement
• 10-20: Natural conversation
• 30+: Expressive/animated

LivePortrait

Strengths:

• Full expression control
• Pitch/yaw/roll adjustment
• Lip zero parameter reduces artifacts

Best for:

• Premium avatar creation
• Expression transfer from video
• Fine-grained control needs

Key parameters:

lip_zero: 0.03          # Reduces unnatural lip movements
stitching: true         # Seamless face blending
relative_motion_mode: "source_video_smoothed"

MuseTalk (Tencent)

Strengths:

• Fast processing
• Good quality at 256×256 face region
• Real-time capable

Best for:

• Large volume processing
• Real-time applications

---

Complete Talking Head Pipeline

Pipeline A: Image → Talking Video (Simple)

1. Generate character audio
   [Text] → [Chatterbox/F5-TTS] → audio.wav

2. Apply lip-sync to still image
   [Character Image] + [audio.wav] → [SadTalker] → video.mp4

3. Enhance
   [video.mp4] → [GFPGAN/CodeFormer per frame] → final.mp4

Best when: Quick turnaround, acceptable head movement

Pipeline B: Image → Video → Lip-Sync (Quality)

1. Generate base video with motion
   [Character Image] → [Wan I2V OR AnimateDiff] → base_video.mp4
   Prompt: "person talking, slight head movement, indoor"

2. Generate character audio
   [Text] → [Chatterbox] → audio.wav

3. Apply lip-sync to video
   [base_video.mp4] + [audio.wav] → [Wav2Lip] → lipsync.mp4

4. Enhance faces
   [lipsync.mp4] → [FaceDetailer batch] → enhanced.mp4

5. Final polish
   [enhanced.mp4] → [Color correct + Deflicker] → final.mp4

Best when: Production quality needed

Pipeline C: Expression Transfer (Premium)

1. Record driving video (actor performing lines)
   [Actor Video] → driving.mp4

2. Generate character audio
   [Text] → [Voice Clone TTS] → audio.wav

3. Transfer expressions to character
   [Character Image] + [driving.mp4] → [LivePortrait] → expression_video.mp4

4. Apply precise lip-sync
   [expression_video.mp4] + [audio.wav] → [Wav2Lip] → lipsync.mp4

5. Enhance
   [lipsync.mp4] → [CodeFormer] → final.mp4

Best when: Maximum realism, acting performance needed

---

Audio Requirements

Recording Quality Guidelines

For voice cloning source:

• Sample rate: 44.1kHz or 48kHz
• Bit depth: 16-bit minimum, 24-bit preferred
• Format: WAV (uncompressed)
• Environment: Quiet room, minimal reverb
• Microphone: XLR condenser preferred, USB acceptable
• Distance: 6-12 inches from mic
• Pop filter: Recommended

For TTS output (lip-sync input):

• Sample rate: 16-24kHz (model dependent)
• Format: WAV
• Mono channel
• Trim leading silence
• Add 0.2s trailing silence
• Normalize to -3dB peak

Audio Preprocessing

import librosa
import soundfile as sf

# Load and preprocess
audio, sr = librosa.load("raw.wav", sr=24000, mono=True)

# Trim silence
audio_trimmed, _ = librosa.effects.trim(audio, top_db=20)

# Normalize
audio_norm = librosa.util.normalize(audio_trimmed) * 0.7  # -3dB headroom

# Add tail silence
silence = np.zeros(int(0.2 * sr))
audio_final = np.concatenate([audio_norm, silence])

# Save
sf.write("processed.wav", audio_final, sr)

---

Sync Troubleshooting

Lips out of sync with audio

Causes:

• Frame rate mismatch
• Audio/video length mismatch
• Processing delay

Solutions:

# Offset audio (if lips are early)
ffmpeg -i video.mp4 -itsoffset 0.1 -i audio.wav -c:v copy -c:a aac output.mp4

# Offset audio (if lips are late)
ffmpeg -i video.mp4 -itsoffset -0.1 -i audio.wav -c:v copy -c:a aac output.mp4

Mouth movements too subtle

Solutions:

• Use wav2lip_gan.pth instead of wav2lip.pth
• Increase audio volume before processing
• Check face detection is accurate

Face artifacts after lip-sync

Solutions:

• Always run face enhancement after Wav2Lip
• CodeFormer fidelity: 0.6-0.8 (not too high)
• Ensure source image resolution matches output

Unnatural head movement (SadTalker)

Solutions:

• Lower pose_style value (0-10)
• Use preprocess: "crop" for stability
• Provide front-facing source image