Video Toolkit

Video Toolkit

Create professional explainer videos from a text brief. The toolkit uses open-source AI models on cloud GPUs (Modal or RunPod) for voiceover, image generation, music, and talking head animation. Remotion (React) handles composition and rendering.

CRITICAL: Toolkit Path

The toolkit lives at a fixed path. ALWAYS `cd` here before running any tool command.

TOOLKIT=~/.openclaw/workspace/claude-code-video-toolkit
cd $TOOLKIT

NEVER run tool commands from inside a project directory. Tools resolve paths relative to the toolkit root.

CRITICAL: Progress Reporting

ALWAYS add `--progress json` to every cloud GPU tool command. This gives you structured JSON Lines on stderr so you can monitor job status, detect stuck jobs, and report progress to the user in real-time.

# CORRECT — always include --progress json
python3 tools/music_gen.py --preset corporate-bg --duration 60 --output bg.mp3 --progress json

# WRONG — no visibility into job status
python3 tools/music_gen.py --preset corporate-bg --duration 60 --output bg.mp3

Tools that support --progress json: music_gen.py, qwen3_tts.py, flux2.py, upscale.py, sadtalker.py, image_edit.py, dewatermark.py, ltx2.py, chain_video.py.

See the Progress Reporting section below for output format and stage definitions.

CRITICAL: Long-Running Tasks — Use yieldMs, Not background:true

Any tool command that takes more than 30 seconds MUST use `exec` with `yieldMs` so you can report progress to the user live. This includes: batch FLUX generation, chain_video, SadTalker, music generation, and any multi-scene pipeline.

exec command:"cd ~/.openclaw/workspace/claude-code-video-toolkit && python3 tools/chain_video.py --output-dir /path/ --progress json ..." yieldMs:10000

The polling loop: 1. exec with yieldMs:10000 starts the command and returns control to you every 10 seconds 2. Read the --progress json output — look for "stage":"item" (scene complete) or "stage":"complete" (all done) 3. Report progress to the user ("Scene 05/30 complete, 17%") 4. Poll again: process action:poll sessionId:<id> 5. Repeat until "stage":"complete"

Why: Your agent run ends when you finish responding. If you use bash background:true, you lose the ability to report progress — the user sees silence until they nudge you. With yieldMs, you stay in the loop.

NEVER do this:

• bash background:true command:"long running thing" then promise to "monitor" — you can't, your run ends
• Break a batch into individual tool calls across separate messages — your run ends between each one
• Promise to "continue autonomously" — you literally cannot without an external trigger

Setup

Step 1: Check Current State

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/verify_setup.py

If everything shows [x], skip to "Quick Test" below. Otherwise continue setup.

Step 2: Install Python Dependencies

cd ~/.openclaw/workspace/claude-code-video-toolkit
pip3 install --break-system-packages -r tools/requirements.txt

Note: --break-system-packages is needed on Debian/Ubuntu with managed Python (PEP 668). Safe inside containers.

Step 3: Configure Cloud GPU Endpoints

The toolkit needs cloud GPU endpoint URLs in .env. Check if .env exists and has Modal endpoints:

cat ~/.openclaw/workspace/claude-code-video-toolkit/.env | grep MODAL

If Modal endpoints are configured, you're ready. If not, ask the user to provide Modal endpoint URLs or set up Modal:

pip3 install --break-system-packages modal
python3 -m modal setup   # Opens browser for authentication

# Deploy each tool — capture the endpoint URL from output
cd ~/.openclaw/workspace/claude-code-video-toolkit
modal deploy docker/modal-qwen3-tts/app.py
modal deploy docker/modal-flux2/app.py
modal deploy docker/modal-music-gen/app.py
modal deploy docker/modal-sadtalker/app.py
modal deploy docker/modal-image-edit/app.py
modal deploy docker/modal-upscale/app.py
modal deploy docker/modal-propainter/app.py
modal deploy docker/modal-ltx2/app.py      # Requires: modal secret create huggingface-token HF_TOKEN=hf_...

LTX-2 prerequisite: Before deploying LTX-2, create a HuggingFace secret and accept the Gemma 3 license:

modal secret create huggingface-token HF_TOKEN=hf_your_read_access_token

Add each URL to .env:

ACEMUSIC_API_KEY=...                          # Free key from acemusic.ai/api-key (best music quality)
MODAL_QWEN3_TTS_ENDPOINT_URL=https://...modal.run
MODAL_FLUX2_ENDPOINT_URL=https://...modal.run
MODAL_MUSIC_GEN_ENDPOINT_URL=https://...modal.run
MODAL_SADTALKER_ENDPOINT_URL=https://...modal.run
MODAL_IMAGE_EDIT_ENDPOINT_URL=https://...modal.run
MODAL_UPSCALE_ENDPOINT_URL=https://...modal.run
MODAL_DEWATERMARK_ENDPOINT_URL=https://...modal.run
MODAL_LTX2_ENDPOINT_URL=https://...modal.run

Optional but recommended — Cloudflare R2 for reliable file transfer:

R2_ACCOUNT_ID=...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=video-toolkit

Step 4: Verify and Quick Test

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/verify_setup.py

All tools should show [x]. Then run a quick test to confirm the GPU pipeline works:

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/qwen3_tts.py --text "Hello, this is a test." --speaker Ryan --tone warm --output /tmp/video-toolkit-test.mp3 --cloud modal

If you get a valid .mp3 file, setup is complete. If it fails, check:

• .env has the correct MODAL_QWEN3_TTS_ENDPOINT_URL
• Run python3 tools/verify_setup.py --json and check modal_tools for which endpoints are missing

Cost: Modal includes $30/month free compute. A typical 60s video costs $1-3.

---

Creating a Video

Step 1: Create Project

cd ~/.openclaw/workspace/claude-code-video-toolkit
cp -r templates/product-demo projects/PROJECT_NAME
cd projects/PROJECT_NAME
npm install

Templates: product-demo (marketing/explainer), sprint-review, sprint-review-v2 (composable scenes).

Step 2: Write Config

Edit projects/PROJECT_NAME/src/config/demo-config.ts:

export const demoConfig: ProductDemoConfig = {
  product: {
    name: 'My Product',
    tagline: 'What it does in one line',
    website: 'example.com',
  },
  scenes: [
    { type: 'title', durationSeconds: 9, content: { headline: '...', subheadline: '...' } },
    { type: 'problem', durationSeconds: 14, content: { headline: '...', problems: ['...', '...'] } },
    { type: 'solution', durationSeconds: 13, content: { headline: '...', highlights: ['...', '...'] } },
    { type: 'stats', durationSeconds: 12, content: { stats: [{value: '99%', label: '...'}, ...] } },
    { type: 'cta', durationSeconds: 10, content: { headline: '...', links: ['...'] } },
  ],
  audio: {
    backgroundMusicFile: 'audio/bg-music.mp3',
    backgroundMusicVolume: 0.12,
  },
};

Scene types: title, problem, solution, demo, feature, stats, cta.

Duration rule: Estimate durationSeconds as ceil(word_count / 2.5) + 2. You will adjust this after generating audio in Step 4.

Step 3: Write Voiceover Script

Create projects/PROJECT_NAME/VOICEOVER-SCRIPT.md:

## Scene 1: Title (9s, ~17 words)
Build videos with AI. The product name toolkit makes it easy.

## Scene 2: Problem (14s, ~30 words)
The problem statement goes here. Keep it punchy and relatable.

Word budget per scene: (durationSeconds - 2) * 2.5 words. The -2 accounts for 1s audio delay + 1s padding.

Step 4: Generate Assets

CRITICAL: All commands below MUST be run from the toolkit root, not the project directory.

cd ~/.openclaw/workspace/claude-code-video-toolkit

4a. Background Music

Default provider is acemusic (official cloud API, free key). No GPU required. Falls back to Modal/RunPod for self-hosted.

cd ~/.openclaw/workspace/claude-code-video-toolkit

# Using acemusic cloud API (default — best quality, XL Turbo 4B model)
python3 tools/music_gen.py \
  --preset corporate-bg \
  --duration 90 \
  --output projects/PROJECT_NAME/public/audio/bg-music.mp3 \
  --progress json

# Or with custom prompt and thinking mode
python3 tools/music_gen.py \
  --prompt "Subtle ambient tech, soft synth pads" \
  --duration 90 \
  --output projects/PROJECT_NAME/public/audio/bg-music.mp3 \
  --progress json

# Fall back to self-hosted Modal if no acemusic key
python3 tools/music_gen.py \
  --preset corporate-bg \
  --duration 90 \
  --output projects/PROJECT_NAME/public/audio/bg-music.mp3 \
  --cloud modal --progress json

Presets: corporate-bg, upbeat-tech, ambient, dramatic, tension, hopeful, cta, lofi.

Setup: echo "ACEMUSIC_API_KEY=your_key" >> .env (get free key at acemusic.ai/api-key).

4b. Voiceover (per-scene)

Generate ONE .mp3 file PER SCENE. Do NOT generate a single voiceover file.

cd ~/.openclaw/workspace/claude-code-video-toolkit

# Scene 01
python3 tools/qwen3_tts.py \
  --text "The voiceover text for scene one." \
  --speaker Ryan --tone warm \
  --output projects/PROJECT_NAME/public/audio/scenes/01.mp3 \
  --cloud modal --progress json

# Scene 02
python3 tools/qwen3_tts.py \
  --text "The voiceover text for scene two." \
  --speaker Ryan --tone warm \
  --output projects/PROJECT_NAME/public/audio/scenes/02.mp3 \
  --cloud modal --progress json

# ... repeat for each scene

Speakers: Ryan, Aiden, Vivian, Serena, Uncle_Fu, Dylan, Eric, Ono_Anna, Sohee Tones: neutral, warm, professional, excited, calm, serious, storyteller, tutorial

For voice cloning (needs a reference recording):

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/qwen3_tts.py \
  --text "Text to speak" \
  --ref-audio assets/voices/reference.m4a \
  --ref-text "Exact transcript of the reference audio" \
  --output projects/PROJECT_NAME/public/audio/scenes/01.mp3 \
  --cloud modal --progress json

4c. Scene Images

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/flux2.py \
  --prompt "Dark tech background with blue geometric grid, cinematic lighting" \
  --width 1920 --height 1080 \
  --output projects/PROJECT_NAME/public/images/title-bg.png \
  --cloud modal --progress json

Image presets (use --preset instead of --prompt --width --height): title-bg, problem, solution, demo-bg, stats-bg, cta, thumbnail, portrait-bg

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/flux2.py \
  --preset title-bg \
  --output projects/PROJECT_NAME/public/images/title-bg.png \
  --cloud modal --progress json

4d. Video Clips — B-Roll & Animated Backgrounds (optional)

Generate AI video clips for b-roll cutaways, animated slide backgrounds, or intro/outro sequences:

cd ~/.openclaw/workspace/claude-code-video-toolkit

# B-roll clip from text
python3 tools/ltx2.py \
  --prompt "Aerial drone shot over a European city at golden hour, cinematic wide angle" \
  --output projects/PROJECT_NAME/public/videos/broll-europe.mp4 \
  --cloud modal --progress json

# Animate a slide/screenshot (image-to-video)
python3 tools/ltx2.py \
  --prompt "Gentle particle effects, soft ambient light shifts, very slight camera drift" \
  --input projects/PROJECT_NAME/public/images/title-bg.png \
  --output projects/PROJECT_NAME/public/videos/animated-title.mp4 \
  --cloud modal --progress json

# Abstract intro/outro background
python3 tools/ltx2.py \
  --prompt "Dark moody abstract background with flowing blue light streaks, bokeh particles, cinematic" \
  --output projects/PROJECT_NAME/public/videos/intro-bg.mp4 \
  --cloud modal --progress json

Use in Remotion compositions with <OffthreadVideo>:

<OffthreadVideo src={staticFile('videos/broll-europe.mp4')} />

LTX-2 rules:

• Max ~8 seconds per clip (193 frames at 24fps). Default is ~5s (121 frames).
• Width/height must be divisible by 64. Default: 768x512.
• ~$0.20-0.25 per clip, ~2.5 min generation time.
• Cold start ~60-90s. Subsequent clips on warm GPU are faster.
• Generated audio is ambient only — use voiceover/music tools for speech and music.
• ~30% of generations may have training data artifacts (logos/text). Re-run with --seed to vary.

4d-chain. Chained Video Sequences (visual continuity)

Generate a sequence of video clips where each scene flows from the last frame of the previous one. This runs as a single command — no manual nudging between scenes.

cd ~/.openclaw/workspace/claude-code-video-toolkit

# Chain scenes 1-30 from a directory of FLUX images
python3 tools/chain_video.py \
  --scenes-dir projects/PROJECT_NAME/public/images/scenes/ \
  --output-dir projects/PROJECT_NAME/public/videos/chain/ \
  --prompt "Cinematic continuation, flowing transition" \
  --start 1 --end 30 \
  --progress json

# Resume from scene 10 (skips existing files automatically)
python3 tools/chain_video.py \
  --scenes-dir projects/PROJECT_NAME/public/images/scenes/ \
  --output-dir projects/PROJECT_NAME/public/videos/chain/ \
  --start 10 --end 30 \
  --progress json

# Per-scene prompts from JSON file
python3 tools/chain_video.py \
  --scenes-dir projects/PROJECT_NAME/public/images/scenes/ \
  --output-dir projects/PROJECT_NAME/public/videos/chain/ \
  --prompts-file projects/PROJECT_NAME/scenes.json \
  --progress json

# Chain from an existing clip (no scene images needed)
python3 tools/chain_video.py \
  --first-clip output/chain-04.mp4 \
  --output-dir output/ \
  --start 5 --end 30 \
  --prompt "Celtic mythology, flowing transition" \
  --progress json

Prompts file format (scenes.json):

{"1": "Ancient stone circle at dawn", "2": "Celtic spirals emerge from stone", "3": "Portal opens with golden light"}

Chain rules:

• Extracts last frame from scene N, feeds as --input to scene N+1 via LTX-2
• Skips scenes that already exist on disk (safe to resume)
• Falls back to scene images from --scenes-dir if chaining fails
• Use --prefix to set output filename prefix (default: chain)
• ~2.5 min per scene, ~$0.20-0.25 per clip
• Extra args (e.g. --negative-prompt, --seed) are passed through to ltx2.py

CRITICAL: Style drift in chained sequences. LTX-2 has ~30% training data contamination (anime/Asian content). Generic prompts like "cinematic transition" will drift toward anime aesthetics within 5-10 chained scenes. To prevent this:

1. ALWAYS use `--prompts-file` with specific per-scene prompts — never a single generic prompt for the whole chain 2. ALWAYS add `--negative-prompt` to exclude unwanted styles:

   --negative-prompt "anime, manga, asian, cartoon, illustration, watermark, text, logo"

3. Each per-scene prompt should include strong style anchors (e.g. "Irish landscape, Celtic knotwork, oil painting style") not just subject descriptions

CRITICAL: Run with `yieldMs` for live progress reporting. Don't break it into per-scene tool calls — OpenClaw's agent run ends between calls, causing the sequence to stall. Instead, use exec with yieldMs so you stay in the loop and can relay progress to the user:

exec command:"cd ~/.openclaw/workspace/claude-code-video-toolkit && python3 tools/chain_video.py --scenes-dir /path/to/images/ --output-dir /path/to/output/ --prompts-file scenes.json --progress json" yieldMs:10000

How this works:

• yieldMs:10000 returns control to you every 10 seconds
• You read the --progress json output (JSON Lines on stderr with stage/pct/msg)
• Report progress to the user ("Scene 05/30 complete, 17%")
• Then poll again: process action:poll sessionId:<id>
• Repeat until "stage":"complete" appears

This is the correct pattern for ALL long-running tool commands (chain_video, batch flux, batch sadtalker, etc.). Never use bash background:true and forget about it — use exec + yieldMs + process poll loop so you can report progress live.

4e. Talking Head Narrator (optional)

Generate a presenter portrait, then animate per-scene clips:

cd ~/.openclaw/workspace/claude-code-video-toolkit

# 1. Generate portrait
python3 tools/flux2.py \
  --prompt "Professional presenter portrait, clean style, dark background, facing camera, upper body" \
  --width 1024 --height 576 \
  --output projects/PROJECT_NAME/public/images/presenter.png \
  --cloud modal --progress json

# 2. Generate per-scene narrator clips (one per scene, NOT one long video)
python3 tools/sadtalker.py \
  --image projects/PROJECT_NAME/public/images/presenter.png \
  --audio projects/PROJECT_NAME/public/audio/scenes/01.mp3 \
  --preprocess full --still --expression-scale 0.8 \
  --output projects/PROJECT_NAME/public/narrator-01.mp4 \
  --cloud modal --progress json

# Repeat for each scene that needs a narrator

SadTalker rules — follow these exactly:

• ALWAYS use --preprocess full (default crop outputs a square, wrong aspect ratio)
• ALWAYS use --still (reduces head movement, looks professional)
• ALWAYS generate per-scene clips (6-15s each), NEVER one long video
• Processing: ~3-4 min per 10s of audio on Modal A10G
• --expression-scale 0.8 keeps expressions subtle (range 0.0-1.5)

4e. Image Editing (optional)

Create scene variants from existing images:

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/image_edit.py \
  --input projects/PROJECT_NAME/public/images/title-bg.png \
  --prompt "Make it darker with red tones, more ominous" \
  --output projects/PROJECT_NAME/public/images/problem-bg.png \
  --cloud modal --progress json

4f. Upscaling (optional)

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/upscale.py \
  --input projects/PROJECT_NAME/public/images/some-image.png \
  --output projects/PROJECT_NAME/public/images/some-image-4x.png \
  --scale 4 --cloud modal --progress json

Step 5: Sync Timing

ALWAYS do this after generating voiceover. Audio duration differs from estimates.

cd ~/.openclaw/workspace/claude-code-video-toolkit
for f in projects/PROJECT_NAME/public/audio/scenes/*.mp3; do
  echo "$(basename $f): $(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")s"
done

Update each scene's durationSeconds in demo-config.ts to: ceil(actual_audio_duration + 2).

Example: if 01.mp3 is 6.8s, set scene 1 durationSeconds to 9 (ceil(6.8 + 2) = 9).

Step 6: Review Still Frames

cd ~/.openclaw/workspace/claude-code-video-toolkit/projects/PROJECT_NAME
npx remotion still src/index.ts ProductDemo --frame=100 --output=/tmp/review-scene1.png
npx remotion still src/index.ts ProductDemo --frame=400 --output=/tmp/review-scene2.png

Check: text truncation, animation timing, narrator PiP positioning, background contrast.

Step 7: Render

cd ~/.openclaw/workspace/claude-code-video-toolkit/projects/PROJECT_NAME
npm run render

Output: out/ProductDemo.mp4

---

Composition Patterns

Per-Scene Audio

Use per-scene audio with a 1-second delay (from={30} = 30 frames = 1s at 30fps):

<Sequence from={30}>
  <Audio src={staticFile('audio/scenes/01.mp3')} volume={1} />
</Sequence>

Per-Scene Narrator PiP

<Sequence from={30}>
  <OffthreadVideo
    src={staticFile('narrator-01.mp4')}
    style={{ width: 320, height: 180, objectFit: 'cover' }}
    muted
  />
</Sequence>

ALWAYS use `<OffthreadVideo>`, NEVER `<video>`. Remotion requires its own component for frame-accurate rendering.

Transitions

import { TransitionSeries, linearTiming } from '@remotion/transitions';
import { fade } from '@remotion/transitions/fade';
import { glitch } from '../../../lib/transitions/presentations/glitch';
import { lightLeak } from '../../../lib/transitions/presentations/light-leak';

NEVER import from `lib/transitions` barrel — import custom transitions from lib/transitions/presentations/ directly.

---

Progress Reporting

All cloud GPU tools support structured progress output for automated monitoring.

Usage

Add --progress json to any tool command to get JSON Lines on stderr:

cd ~/.openclaw/workspace/claude-code-video-toolkit
python3 tools/music_gen.py \
  --preset corporate-bg --duration 60 \
  --output projects/PROJECT_NAME/public/audio/bg-music.mp3 \
  --progress json

Output Format

Each line on stderr is a JSON object:

{"ts":"14:23:15","stage":"submit","msg":"Sending to acemusic.ai (XL Turbo 4B, thinking: on)...","pct":null,"elapsed":0.0}
{"ts":"14:23:30","stage":"waiting","msg":"Waiting for acemusic.ai response... (15s)","pct":null,"elapsed":15.0}
{"ts":"14:23:45","stage":"waiting","msg":"Waiting for acemusic.ai response... (30s)","pct":null,"elapsed":30.0}
{"ts":"14:24:02","stage":"complete","msg":"Saved: bg-music.mp3 (245 KB, 60.1s)","pct":100,"elapsed":47.3}

Stages

Behaviour by Provider

• acemusic: Emits submit → periodic waiting heartbeats (every 15s) → complete
• RunPod: Emits submit → queue → processing → complete (on each poll)
• Modal: Emits submit → periodic waiting heartbeats → complete

Default mode (--progress human) shows the same events as colored terminal output — no change to existing behaviour.

---

Error Recovery

---

Cost Estimates (Modal)

Total for a 60s video: ~$1-3 depending on scenes and narrator clips.

Modal Starter plan: $30/month free compute. Apps scale to zero when idle.