
Video Editing
- 1 installs
- Updated April 18, 2026
- darsh20009/genmz-shop
Helps with ai & agent building tasks.
About
video-editing is a Claude Code skill for ai & agent building. It helps developers move faster with AI-assisted coding.
- video-editing
- AI & Agent Building
- AI-coding skill
Video Editing by the numbers
- 1 all-time installs (skills.sh)
- Ranked #14,098 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Jul 7, 2026 (Skillselion catalog sync)
npx skills add https://github.com/darsh20009/genmz-shop --skill video-editingAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1 |
|---|---|
| Last updated | April 18, 2026 |
| Repository | darsh20009/genmz-shop ↗ |
What it does
Helps with ai & agent building tasks.
Files
Video Editing
Edit existing video files server-side using FFmpeg. This skill covers the full editing workflow: trimming, splicing, concatenating, transitions, text overlays, subtitles, audio mixing, effects, and format conversion.
When to Use
Trimming & Cutting
- "Trim this video to 0:30-1:15" / "Cut out the first 10 seconds"
- "Extract a clip from 2:00 to 3:30"
- "Remove the intro" / "Cut the ending"
Merging & Compositing
- "Merge these clips together" / "Concatenate these videos"
- "Put two videos side by side" / "Split screen"
- "Picture-in-picture"
Text, Titles & Subtitles
- "Add subtitles" / "Add captions"
- "Put a title card at the beginning"
- "Add text overlay"
Audio
- "Add background music" / "Mix in this audio track"
- "Add a voiceover" / "Narrate this video" (uses ElevenLabs)
- "Mute this video" / "Remove the audio"
- "Replace the audio with this track"
Speed & Motion
- "Speed up this video 2x" / "Slow motion"
- "Make this part slow-mo"
- "Reverse this clip"
- "Loop this clip 3 times"
Visual Effects & Fixes
- "Rotate this video 90 degrees" / "Flip horizontally"
- "Stabilize this shaky footage"
- "Crop out the watermark" / "Remove the watermark"
- "Add my logo to this video" (image overlay / branding)
- "Color correct" / "Make it brighter" / "Adjust contrast"
Format & Size
- "Convert this MP4 to WebM" / "Change format"
- "Compress this video" / "Make this file smaller"
- "Resize to 720p" / "Downscale" / "Change resolution"
- "Extract the audio as MP3"
- "Create a GIF from this clip"
- "Make a thumbnail from this video" (frame extraction as image)
Social Media & Platform
- "Make TikTok clips from this" / "Break this into Reels"
- "Make this vertical for Shorts"
- "Reframe for Instagram" / "LinkedIn" / "Pinterest" / "Facebook" / "Snapchat"
- "Convert this to square for Instagram feed"
AI-Powered Analysis
- "Find the best moments" / "Auto-trim for virality"
- "Extract the highlights" / "Score these clips"
- "Which parts are most engaging?"
Cleanup & Optimization
- "Remove dead space" / "Remove silence"
- "Shorten this by cutting out boring parts"
- "Tighten up this video"
Chunking & Splitting
- "Split this into short clips" / "Chunk this for TikTok"
- "Break this talk into Reels-length videos"
- "Make 15-second clips from this"
No Direction (user just uploads a video)
- Probe the file, report what it is, and ask what they want to do
When NOT to Use
- User wants to create an animated video from scratch using code (use the
video-jsskill instead) - User wants a browser-based video editor UI (build as a
react-viteartifact with this skill powering the backend)
User Interaction — ALWAYS Ask Before Acting
When a user uploads a video without specific instructions, you MUST ask what they want to do with it. Do NOT assume a workflow or start processing automatically.
Probe the video first (to report basic info), then ask the user what they'd like. Here's a good pattern:
1. Probe the file and tell the user what they uploaded (duration, resolution, format) 2. Ask what they want to do. Offer common options like:
- Trim or cut specific sections
- Find the best/most viral moments
- Break it into short clips for a specific platform (TikTok, Reels, Shorts, X)
- Remove dead space / silence
- Add text, subtitles, or a voiceover
- Reframe for a different aspect ratio
- Convert to a different format
- Something else
When the user specifies a platform, the full pipeline includes reframing to that platform's native format. Do not just chunk the video and leave it in the original aspect ratio. Always deliver platform-ready output — correct aspect ratio, resolution, and duration range.
Platform Specifications
| Platform | Aspect Ratio | Resolution | Duration Range | Ideal Length | Max File Size |
|----------|-------------|------------|---------------|-------------|--------------|
| TikTok | 9:16 vertical | 1080×1920 | 10–45s | 25s | 287 MB |
| Instagram Reels | 9:16 vertical | 1080×1920 | 10–30s | 20s | 250 MB |
| Instagram Stories | 9:16 vertical | 1080×1920 | 1–60s | 15s | 250 MB |
| Instagram Feed | 1:1 square or 4:5 portrait | 1080×1080 or 1080×1350 | 3–60s | 30s | 250 MB |
| YouTube Shorts | 9:16 vertical | 1080×1920 | 15–60s | 40s | 256 MB |
| YouTube (standard) | 16:9 landscape | 1920×1080 | any | any | 256 GB |
| X / Twitter | 16:9 landscape | 1280×720 | 10–45s | 25s | 512 MB |
| X / Twitter (square) | 1:1 square | 720×720 | 10–45s | 25s | 512 MB |
| Facebook Reels | 9:16 vertical | 1080×1920 | 10–30s | 20s | 250 MB |
| Facebook Feed | 16:9 or 1:1 | 1920×1080 or 1080×1080 | any | 15–60s | 4 GB |
| LinkedIn | 16:9 landscape or 1:1 | 1920×1080 or 1080×1080 | 10–60s | 30s | 5 GB |
| Pinterest | 9:16 or 2:3 vertical | 1080×1920 or 1000×1500 | 6–60s | 15–30s | 2 GB |
| Snapchat Spotlight | 9:16 vertical | 1080×1920 | 5–60s | 15–30s | 250 MB |
When the user says a platform name, apply both the correct duration targets AND the correct aspect ratio/resolution. For example:
- "Make TikTok clips" → chunk to 10-45s AND reframe to 9:16 (1080×1920)
- "Instagram feed posts" → chunk to 3-60s AND reframe to 1:1 (1080×1080) or 4:5 (1080×1350)
- "YouTube Shorts" → chunk to 15-60s AND reframe to 9:16 (1080×1920)
- "Clips for X" → chunk to 10-45s, keep 16:9 (1280×720)
When the user asks for clips or chunking without specifying a platform, ask which platform they're targeting so you can apply the right duration targets and reframing.
When you deliver results, always present the output files so the user can download and preview them. Never just describe what was done — show the files.
Environment
FFmpeg 6.1.2 is pre-installed in Replit with full codec support:
- Video codecs: H.264 (libx264), H.265 (libx265), VP8/VP9 (libvpx), AV1 (libaom, libsvtav1, librav1e), Theora
- Audio codecs: AAC, MP3 (libmp3lame), Opus, Vorbis, FLAC, WAV
- Image formats: PNG, JPEG, WebP, JPEG XL (libjxl)
- Subtitle rendering: libass (for styled subtitles), libfreetype + libfontconfig (for drawtext)
- Filters: All standard filters available — scale, crop, overlay, drawtext, concat, fade, crossfade, etc.
- Tools:
ffmpeg,ffprobe(both available on PATH)
No additional installation is needed for FFmpeg itself.
Architecture Decisions
Script-based editing (simple tasks)
For one-off or single-operation tasks (trim a clip, convert a format, extract audio), write a Node.js script in scripts/src/usingfluent-ffmpegor directchild_process.execFile calls to FFmpeg.
pnpm --filter @workspace/scripts add fluent-ffmpeg
pnpm --filter @workspace/scripts add -D @types/fluent-ffmpeg
API-based editing (complex workflows or UI)
For multi-step editing workflows or when building a video editor UI, add routes to the API server (artifacts/api-server/) that accept video files, process them with FFmpeg, and return results.
pnpm --filter @workspace/api-server add fluent-ffmpeg multer
pnpm --filter @workspace/api-server add -D @types/fluent-ffmpeg @types/multer
Choosing the right approach
| Scenario | Approach |
|----------|----------|
| "Trim this video to 0:30-1:15" | Script in scripts/src/ |
| "Convert this MP4 to WebM" | Script in scripts/src/ |
| "Build me a video editor app" | React-vite artifact + API routes |
| "Add subtitles to all my videos" | Script in scripts/src/ |
| "Merge these 5 clips with transitions" | Script in scripts/src/ (or API if repeated) |
Instructions
1. Probe Before Processing
Always probe the input file first to understand its properties. This prevents errors from wrong assumptions about codecs, resolution, frame rate, or duration.
import { execFile } from 'child_process';
import { promisify } from 'util';
const execFileAsync = promisify(execFile);
async function probeVideo(filePath: string) {
const { stdout } = await execFileAsync('ffprobe', [
'-v', 'quiet',
'-print_format', 'json',
'-show_format',
'-show_streams',
filePath
]);
return JSON.parse(stdout);
}
2. Use fluent-ffmpeg for Complex Operations
fluent-ffmpeg provides a chainable API that is easier to maintain than raw command strings. Use it for multi-step operations.
import ffmpeg from 'fluent-ffmpeg';
function trimVideo(input: string, output: string, start: string, duration: string): Promise<void> {
return new Promise((resolve, reject) => {
ffmpeg(input)
.setStartTime(start)
.setDuration(duration)
.output(output)
.on('end', resolve)
.on('error', reject)
.run();
});
}
3. Use execFile for Simple One-Liners
For straightforward operations where the FFmpeg command is well-known, calling FFmpeg directly is cleaner.
await execFileAsync('ffmpeg', [
'-i', inputPath,
'-vf', 'scale=1280:720',
'-c:a', 'copy',
outputPath
]);
4. File Management
- Store uploaded/input files in a
tmp/oruploads/directory - Store output files where the user can access them — either
public/for web serving or a known output directory
- Clean up temporary files after processing
- For large files, stream the output rather than buffering in memory
5. Progress Reporting
FFmpeg outputs progress to stderr. Parse it for progress reporting in longer operations:
ffmpeg(input)
.output(output)
.on('progress', (progress) => {
console.log(`Processing: ${progress.percent?.toFixed(1)}% done`);
})
.on('end', () => console.log('Done'))
.run();
6. Error Handling
FFmpeg errors are often cryptic. Always:
- Log the full stderr output on failure
- Validate input files exist before processing
- Check that output codecs are compatible with the output container
- Use
-yflag to overwrite output files without prompting
AI-Powered Virality Scoring
Use AI vision to analyze video content and score it on viral potential. This feature has two modes depending on the workflow.
This feature requires an AI integration (OpenAI or Gemini) to be set up. Set up the AI integration following the ai-integrations-openaiorai-integrations-gemini skill before proceeding. OpenAI is preferred because gpt-5.2 handles image analysis well and is the default for most tasks. Gemini (gemini-2.5-flash) is a good alternative, especially for high-volume analysis since it supports video/image input natively.
Two Scoring Modes
Mode 1: Segment scoring (find the best moments) — For when the user says "find the best clips" or "auto-trim for virality." Detects scenes, scores small segments, and exports the top moments. Good for finding highlights in raw footage.
Mode 2: Clip scoring (rank ready-to-post clips)— For when clips already exist (from chunking, manual trimming, etc.) and the user wants to know which ones to post first. Scores each complete clip with a richer evaluation including narrative arc and standalone quality.This is the preferred mode when used with the chunking pipeline
Recommended Pipeline
When the user has a long video and wants social-ready clips, use this order:
Long Video → Dead Space Removal → Chunking (10-60s clips) → Clip-Level Scoring → Reframe for Platform → Output
This way the AI scores complete, self-contained clips rather than tiny scene fragments. The scores are much more meaningful because each clip is what will actually be posted.
For "find the best moment in this video" requests (no chunking), use the original segment-level pipeline:
Video → Scene Detection → Frame Extraction → Segment Scoring → Rank → Trim/Export
When the user asks for auto-trim, ask which output they want
Present these three options and let the user choose each time. Do not assume a default:
- Best clip — The single highest-scoring segment/clip, trimmed and exported
- Multiple clips — Several top segments/clips exported as separate files, ranked by score
- Highlight reel — Top moments stitched together with crossfade transitions into one video
Scoring Criteria
Segment scoring evaluates: visual dynamism, emotional impact, hook potential, pacing/energy, uniqueness, and shareability. Each factor gets a 1-10 score, weighted to produce a final virality score.
Clip scoring adds three additional criteria for complete clips:
- Narrative completeness (weight: 15%) — Does the clip tell a complete micro-story? Does it have a clear beginning, middle, and end? Would a viewer feel satisfied or intrigued, not confused?
- Hook-to-payoff flow (weight: 10%) — Does the clip open with something that grabs attention and deliver on that promise? Or does it start slow and meander?
- Standalone quality (weight: 10%) — Would this clip make sense to someone who hasn't seen the full video? Can it be posted without context?
These three criteria reduce the weights of the original six factors proportionally so the total still sums to 100%.
Frame Extraction
- Segments (2-10s): Extract 3 frames evenly spread across the segment.
- Clips (10-60s): Extract 5-8 frames to capture the full arc. Always include the first frame (hook evaluation) and last frame (payoff evaluation).
Workflow
For the complete step-by-step implementation with code examples, see:
virality-scoring.md— Full virality analysis pipeline, AI prompts, scoring criteria, and output assembly
Key Gotchas (learned from testing)
- Scene detection: Use
ffmpegwithselect+showinfofilters and parse stderr — theffprobe -f lavfiapproach doesn't work reliably in Replit. - Content safety: OpenAI vision may reject frames from documentary/medical/news content. Always wrap AI calls in try/catch and skip failed segments gracefully.
- Segment indexing: Use
index: segments.lengthwhen filtering short segments, not the loop counteri. - Package setup: For scripts, install
openaidirectly (pnpm add -w openai) and create the client withAI_INTEGRATIONS_OPENAI_BASE_URL/AI_INTEGRATIONS_OPENAI_API_KEYenv vars. No need for the full workspace integration library.
- Timeouts: A 90-second video with ~13 segments takes 1-2 minutes for AI analysis. Warn the user and set generous timeouts.
- Minimum segment duration: Use 2.0s minimum — segments under 2 seconds waste API calls and don't produce useful scores.
- Prefer clip-level scoring: When the video has already been chunked, always score the chunks rather than re-running segment detection. The clips are what will actually be posted.
Dead Space Removal
Automatically detect and remove silence, dead air, and filler from a video to produce a tighter, more engaging cut. This is especially useful for talks, interviews, podcasts, tutorials, and raw footage.
How It Works
1. Silence detection — Use FFmpeg's silencedetect filter to find all silent intervals (configurable threshold and minimum duration) 2. Segment extraction — Extract all non-silent segments as individual clips
3. Reassembly — Concatenate the non-silent segments back together with optional brief crossfade transitions 4. Optional: AI-assisted filler removal — For more aggressive editing (removing "ums", filler words, repetitive sections), combine silence detection with AI transcription analysis
When the user asks to remove dead space
Ask these clarifying questions if not clear from context:
- How aggressive? Light (remove only true silence, > 1s gaps) vs. aggressive (remove short pauses > 0.3s for a fast-paced edit)
- Transitions? Hard cuts between segments (default) or brief crossfades (0.2-0.5s) for smoother flow
Configuration
| Preset | Silence Threshold | Min Silence Duration | Use Case |
|--------|------------------|---------------------|----------|
| Light | -40dB | 1.0s | Talks, interviews — remove obvious dead air |
| Medium | -35dB | 0.5s | Podcasts, tutorials — tighter pacing |
| Aggressive | -30dB | 0.3s | Fast-paced edits, social media — maximum tightness |
Implementation
For the complete FFmpeg commands and Node.js pipeline, see:
dead-space-and-chunking.md— See the "Dead Space Removal" section
Quick Summary
Video → Silence Detection (FFmpeg silencedetect) → Identify Non-Silent Ranges → Extract Segments → Concatenate → Output
Social Media Chunking
Break a longer video (ad, promo, talk, presentation, livestream) into self-contained clips suitable for TikTok, Reels, and Shorts. Each clip should be 10-60 seconds and feel complete on its own.
How It Works (2)
1. Scene detection + silence detection — Find natural break points using both visual scene changes and audio silence gaps 2. Smart boundary selection — Merge adjacent segments into clips targeting 10-60 seconds, preferring to break at silence or scene changes rather than mid-sentence
3. AI content analysis (optional) — Score each potential clip with AI to identify which ones are worth posting, suggest captions, and recommend clip order 4. Export — Output each clip as a separate file, numbered and ranked
Clip Length Guidelines
| Platform | Ideal Length | Max Length | Notes |
|----------|-------------|------------|-------|
| TikTok | 15-45s | 10 min | 15-30s performs best for new accounts |
| Instagram Reels | 15-30s | 90s | Under 30s gets more reach |
| YouTube Shorts | 30-60s | 60s | Hard cap at 60 seconds |
| X / Twitter | 15-45s | 2m 20s | Shorter gets more engagement |
When the user asks to chunk a video
Ask which platform they're targeting (or default to TikTok at 15-45s). The chunking algorithm should:
- Never cut mid-sentence — Always break at silence gaps or natural pauses
- Prefer scene boundaries — Break where the visual content changes
- Ensure each clip stands alone — Each clip should have a clear beginning, not start mid-thought
- Target the platform's sweet spot — Not just "under 60s" but actually the ideal range for that platform
Implementation (2)
For the complete pipeline with code examples, see:
dead-space-and-chunking.md— See the "Social Media Chunking" section
Quick Summary (2)
Video → Scene Detection + Silence Detection → Merge into 10-60s Clips at Natural Boundaries → (Optional) AI Scoring → Export Individual Clips
Combining with Other Features
These features work well together in a pipeline:
Long Video → Dead Space Removal → Chunking into Clips → Virality Scoring → Reframe for TikTok → Output
This takes a raw long-form video and produces polished, platform-ready social clips end to end.
Voiceovers with ElevenLabs
Add AI-generated voiceovers to videos using ElevenLabs text-to-speech, then mix the audio into the video with FFmpeg.
Prerequisites
ElevenLabs is available as a Replit integration (connector). The user needs to connect their ElevenLabs account through Replit's integration system — no manual API keys needed. Search for the integration and propose it to the user:
const results = await searchIntegrations("elevenlabs");
// Then propose the connector so the user can authorize
await proposeIntegration("connector:ccfg_elevenlabs_...");
After authorization, use addIntegrationto wire it to the project, then uselistConnections('elevenlabs') in the code execution sandbox to get the credentials.
Workflow (2)
1. Generate voiceover audio — Send text to ElevenLabs TTS API, receive audio file 2. Mix into video — Use FFmpeg to add the voiceover as a new audio track, optionally ducking (lowering volume of) existing audio
Voiceover Modes
When the user asks for a voiceover, they may want one of these:
- Replace audio — Remove existing audio entirely, use only the voiceover
- Mix over — Layer voiceover on top of existing audio (with existing audio volume reduced)
- Add as track — Keep existing audio at full volume, add voiceover on top
Ask which mode the user prefers if not clear from context.
Implementation (3)
For the complete step-by-step implementation with code examples, see:
voiceover.md— ElevenLabs TTS integration, voice selection, audio mixing with FFmpeg, and timed voiceover segments
Quick Summary (3)
Text Script → ElevenLabs TTS → Audio File → FFmpeg Mix with Video → Output
Key capabilities:
- Choose from ElevenLabs' voice library or cloned voices
- Control voice settings (stability, similarity boost, style)
- Generate voiceover for the full video or specific time segments
- Duck existing audio under voiceover sections
- Support for multiple languages
Social Media Reframing
Automatically reframe videos for different social media platforms. Each platform has a preferred aspect ratio and resolution — this feature handles the conversion intelligently.
Platform Specs
| Platform | Aspect Ratio | Resolution | Notes |
|----------|-------------|------------|-------|
| YouTube | 16:9 | 1920x1080 | Standard landscape, also supports 4K (3840x2160) |
| TikTok / Reels | 9:16 | 1080x1920 | Vertical video, max 60s for Reels, 10min for TikTok |
| Instagram Feed | 1:1 | 1080x1080 | Square format |
| X / Twitter | 16:9 or 1:1 | 1280x720 or 720x720 | Supports both, 2min 20s max |
Reframing Strategies
When converting between aspect ratios, there are three strategies:
1. Center crop (default) — Crop from the center to fill the target ratio. Fast, works well for most content. May cut off edges. 2. Letterbox/pillarbox — Add black bars (or blurred background) to fit without cropping. Preserves all content but adds empty space.
3. Blurred fill — Use a blurred, scaled-up version of the video as the background behind the original. Looks much better than black bars, especially for vertical reframing. This is the recommended approach for going from 16:9 to 9:16.
When the user asks to reframe, ask which platform they're targeting. Use blurred fillby default for aspect ratio changes that would lose significant content (e.g., landscape to portrait). Usecenter cropwhen the content is center-focused. Useletterbox only if the user explicitly asks for it.
Implementation (4)
For the complete FFmpeg commands for each reframing strategy, see:
operations.md— See the "Social Media Reframing" section
Multi-platform Export
When the user wants to export for multiple platforms at once, generate all versions in a batch:
Input (16:9) → YouTube (copy) + TikTok (9:16 blurred fill) + Instagram (1:1 center crop) + X (1280x720 copy)
This is a common workflow — the user shoots in landscape and needs versions for every platform.
Common Operations Reference
For detailed FFmpeg commands and patterns for each operation type, see:
operations.md— Complete command reference for all supported editing operationsvirality-scoring.md— AI-powered virality analysis and auto-trim pipeline
voiceover.md— ElevenLabs voiceover generation and audio mixingdead-space-and-chunking.md— Dead space removal and social media chunking pipelines
Building a Video Editor UI
If the user wants a visual editor interface:
1. Create a react-vite artifact for the frontend 2. Add API routes to artifacts/api-server/ for video processing
3. Use multer for file uploads to the API 4. Process videos with FFmpeg on the server
5. Return processed files for download or preview 6. Use the object-storage skill for persisting uploaded and processed files if needed
Key frontend considerations:
- Use
<video>element for preview playback - Show a timeline UI for trim/cut operations
- Display progress during processing (poll an API status endpoint or use Server-Sent Events)
- Provide download links for processed output
Tested Bugs & Caveats (Do Not Regress)
These issues were discovered during real-world testing and are documented here to prevent regressions:
- Scene detection: The
ffprobe -f lavfiapproach does NOT work in Replit. Useffmpegwithselect='gt(scene,threshold)',showinfoand parsepts_timefrom stderr. - Segment indexing: Use
index: segments.lengthwhen building segments, NOT the loop counteri— filtered segments cause index mismatches.
- Content safety: OpenAI vision may reject frames (medical/documentary content). Always wrap AI calls in try/catch and skip gracefully.
- Frame extraction timestamp clamping: Clamp the last frame timestamp to
segment.endTime - 0.1. Without this, the last clip in a video fails becausestartTime + 1.0 * durationcan exceed the actual file duration due to floating point arithmetic.
- `silenceremove` filter: Only strips audio silence, NOT video frames. Must use segmented extract+concat approach for proper dead space removal.
- Dead space with many segments: Videos with lots of short dialogue (50+ speaking segments) can be very slow to process with re-encoding. Use
-preset ultrafastfor testing, or skip dead space removal for dialogue-heavy content where there's little dead space anyway.
- OpenAI for scripts: Install with
pnpm add -w openai, create client withAI_INTEGRATIONS_OPENAI_BASE_URLandAI_INTEGRATIONS_OPENAI_API_KEYenv vars. - AI scoring duration: A 90-second video with ~13 segments takes 1-2 minutes for AI analysis. A 3.5-minute video with 9 clips takes ~3 minutes. Always warn the user.
Limitations
- No GPU acceleration — Replit containers do not have GPU access, so hardware encoding (NVENC, VAAPI) is unavailable. Use software encoding (libx264, libx265, libvpx-vp9).
- Memory and CPU — Very large files or high-resolution encoding (4K+) may be slow or hit memory limits. For large files, prefer stream-copy (
-c copy) where possible and avoid unnecessary re-encoding.
- Disk space — Video files are large. Clean up temporary files after processing. Monitor disk usage for batch operations.
- No real-time preview — FFmpeg processes offline. Users cannot preview effects in real-time during editing (unlike desktop NLEs). Generate short previews of segments instead.
Mac OS X 2��ATTR��9�9com.apple.quarantine0281;69e40224;Chrome;CDF017FA-52AE-4A15-8BE5-98EF895E9AB7This resource fork intentionally left blank ��Mac OS X 2��ATTR��9�9com.apple.quarantine0281;69e40224;Chrome;CDF017FA-52AE-4A15-8BE5-98EF895E9AB7This resource fork intentionally left blank ��
Mac OS X 2��ATTR��9�9com.apple.quarantine0281;69e40224;Chrome;CDF017FA-52AE-4A15-8BE5-98EF895E9AB7This resource fork intentionally left blank ��
Mac OS X 2��ATTR��9�9com.apple.quarantine0281;69e40224;Chrome;CDF017FA-52AE-4A15-8BE5-98EF895E9AB7This resource fork intentionally left blank ��
Mac OS X 2��ATTR��9�9com.apple.quarantine0281;69e40224;Chrome;CDF017FA-52AE-4A15-8BE5-98EF895E9AB7This resource fork intentionally left blank ��
Mac OS X 2��ATTR��9�9com.apple.quarantine0281;69e40224;Chrome;CDF017FA-52AE-4A15-8BE5-98EF895E9AB7This resource fork intentionally left blank ��
6240cf8f21b790a53ab9bc6c47136c51Dead Space Removal & Social Media Chunking
Complete implementation guide for removing silence/dead space from videos and chunking longer videos into social media-ready clips.
Table of Contents
1. Dead Space Removal 2. Social Media Chunking
---
Dead Space Removal
Remove silence, dead air, and filler from a video to produce a tighter cut.
Step 1: Detect Silent Intervals
Use FFmpeg's silencedetect filter to find all silent sections. Parse the output from stderr.
import { execFile } from 'child_process';
import { promisify } from 'util';
const execFileAsync = promisify(execFile);
interface SilentInterval {
start: number;
end: number;
duration: number;
}
interface SpeakingInterval {
start: number;
end: number;
duration: number;
}
async function detectSilence(
videoPath: string,
noiseThresholdDb = -35,
minSilenceDuration = 0.5
): Promise<SilentInterval[]> {
const { stderr } = await execFileAsync('ffmpeg', [
'-i', videoPath,
'-af', `silencedetect=noise=${noiseThresholdDb}dB:d=${minSilenceDuration}`,
'-f', 'null', '-'
], { maxBuffer: 10 * 1024 * 1024 });
const silentIntervals: SilentInterval[] = [];
const lines = stderr.split('\n');
let currentStart: number | null = null;
for (const line of lines) {
const startMatch = line.match(/silence_start:\s*([0-9.]+)/);
const endMatch = line.match(/silence_end:\s*([0-9.]+)\s*\\\s*silence_duration:\s*([0-9.]+)/);
if (startMatch) {
currentStart = parseFloat(startMatch[1]);
}
if (endMatch && currentStart !== null) {
const end = parseFloat(endMatch[1]);
const duration = parseFloat(endMatch[2]);
silentIntervals.push({ start: currentStart, end, duration });
currentStart = null;
}
}
return silentIntervals;
}
Presets
const SILENCE_PRESETS = {
light: { noiseThresholdDb: -40, minSilenceDuration: 1.0 },
medium: { noiseThresholdDb: -35, minSilenceDuration: 0.5 },
aggressive: { noiseThresholdDb: -30, minSilenceDuration: 0.3 },
} as const;
type SilencePreset = keyof typeof SILENCE_PRESETS;
Step 2: Calculate Non-Silent (Speaking) Intervals
Invert the silent intervals to find the parts worth keeping. Add a small padding around each speaking segment so words don't get clipped.
async function getSpeakingIntervals(
videoPath: string,
silentIntervals: SilentInterval[],
paddingSeconds = 0.1
): Promise<SpeakingInterval[]> {
const { stdout } = await execFileAsync('ffprobe', [
'-v', 'quiet', '-print_format', 'json', '-show_format', videoPath
]);
const totalDuration = parseFloat(JSON.parse(stdout).format.duration);
if (silentIntervals.length === 0) {
return [{ start: 0, end: totalDuration, duration: totalDuration }];
}
const speaking: SpeakingInterval[] = [];
let cursor = 0;
for (const silence of silentIntervals) {
const segStart = Math.max(0, cursor);
const segEnd = Math.max(segStart, silence.start + paddingSeconds);
if (segEnd - segStart >= 0.1) {
speaking.push({ start: segStart, end: segEnd, duration: segEnd - segStart });
}
cursor = Math.max(cursor, silence.end - paddingSeconds);
}
if (cursor < totalDuration) {
speaking.push({
start: cursor,
end: totalDuration,
duration: totalDuration - cursor,
});
}
return speaking;
}
Step 3: Extract and Reassemble
Extract each speaking segment and concatenate them into the final output.
import * as fs from 'fs';
import * as path from 'path';
async function removeDeadSpace(
videoPath: string,
outputPath: string,
preset: SilencePreset = 'medium',
useTransitions = false,
transitionDuration = 0.3
): Promise<{ outputPath: string; originalDuration: number; newDuration: number; removedSeconds: number }> {
const config = SILENCE_PRESETS[preset];
console.log(`Detecting silence (preset: ${preset}, threshold: ${config.noiseThresholdDb}dB, min duration: ${config.minSilenceDuration}s)...`);
const silentIntervals = await detectSilence(videoPath, config.noiseThresholdDb, config.minSilenceDuration);
console.log(`Found ${silentIntervals.length} silent intervals`);
const speakingIntervals = await getSpeakingIntervals(videoPath, silentIntervals);
console.log(`Found ${speakingIntervals.length} speaking segments`);
if (speakingIntervals.length === 0) {
throw new Error('No speaking segments found — the entire video appears silent');
}
if (speakingIntervals.length === 1 && silentIntervals.length === 0) {
console.log('No silence detected — video is already tight');
await fs.promises.copyFile(videoPath, outputPath);
const dur = speakingIntervals[0].duration;
return { outputPath, originalDuration: dur, newDuration: dur, removedSeconds: 0 };
}
const tmpDir = path.join(path.dirname(outputPath), `tmp_deadspace_${Date.now()}`);
await fs.promises.mkdir(tmpDir, { recursive: true });
const clipPaths: string[] = [];
for (let i = 0; i < speakingIntervals.length; i++) {
const seg = speakingIntervals[i];
const clipPath = path.join(tmpDir, `seg_${i.toString().padStart(4, '0')}.mp4`);
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-ss', seg.start.toFixed(3),
'-to', seg.end.toFixed(3),
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
clipPath
]);
clipPaths.push(clipPath);
}
const concatList = path.join(tmpDir, 'concat.txt');
const lines = clipPaths.map(p => `file '${p}'`).join('\n');
await fs.promises.writeFile(concatList, lines);
await execFileAsync('ffmpeg', [
'-y',
'-f', 'concat', '-safe', '0',
'-i', concatList,
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outputPath
]);
for (const cp of clipPaths) await fs.promises.unlink(cp).catch(() => {});
await fs.promises.rm(tmpDir, { recursive: true, force: true });
const { stdout: probeOrig } = await execFileAsync('ffprobe', [
'-v', 'quiet', '-print_format', 'json', '-show_format', videoPath
]);
const { stdout: probeNew } = await execFileAsync('ffprobe', [
'-v', 'quiet', '-print_format', 'json', '-show_format', outputPath
]);
const originalDuration = parseFloat(JSON.parse(probeOrig).format.duration);
const newDuration = parseFloat(JSON.parse(probeNew).format.duration);
return {
outputPath,
originalDuration,
newDuration,
removedSeconds: originalDuration - newDuration,
};
}
Quick FFmpeg-only approach (no Node.js)
For a simple one-off dead space removal, you can use FFmpeg directly with a two-pass approach:
# Step 1: Detect silence and note timestamps
ffmpeg -i input.mp4 -af "silencedetect=noise=-35dB:d=0.5" -f null - 2>&1 | grep "silence_"
# Step 2: Manually construct a trim filter from the timestamps, or use the select filter:
# This keeps only non-silent frames (audio-based)
ffmpeg -i input.mp4 \\
-af "silenceremove=stop_periods=-1:stop_duration=0.5:stop_threshold=-35dB" \\
-c:v libx264 -crf 23 -preset medium \\
output_no_silence.mp4
Note: The silenceremove filter only removes silence from the audio track — the video frames during silence are NOT removed. For proper video dead space removal (where both audio and video are cut), you must use the segmented extraction approach above.
---
Social Media Chunking
Break a longer video into self-contained clips of 10-60 seconds for TikTok, Reels, and Shorts.
Step 1: Find Natural Break Points
Combine scene detection and silence detection to find the best places to split.
interface BreakPoint {
time: number;
type: 'scene' | 'silence' | 'both';
strength: number;
}
async function findBreakPoints(
videoPath: string,
sceneThreshold = 0.3,
silenceThresholdDb = -35,
minSilenceDuration = 0.3
): Promise<BreakPoint[]> {
const sceneChanges = await detectSceneChangeTimes(videoPath, sceneThreshold);
const silentIntervals = await detectSilence(videoPath, silenceThresholdDb, minSilenceDuration);
const breakPoints: BreakPoint[] = [];
for (const sceneTime of sceneChanges) {
const nearbySilence = silentIntervals.find(
s => Math.abs(s.start - sceneTime) < 1.0 || Math.abs(s.end - sceneTime) < 1.0
);
if (nearbySilence) {
breakPoints.push({
time: nearbySilence.end,
type: 'both',
strength: 1.0,
});
} else {
breakPoints.push({ time: sceneTime, type: 'scene', strength: 0.6 });
}
}
for (const silence of silentIntervals) {
const alreadyCovered = breakPoints.some(bp => Math.abs(bp.time - silence.end) < 1.0);
if (!alreadyCovered) {
breakPoints.push({
time: silence.end,
type: 'silence',
strength: 0.4 + Math.min(silence.duration / 3, 0.4),
});
}
}
breakPoints.sort((a, b) => a.time - b.time);
return breakPoints;
}
async function detectSceneChangeTimes(videoPath: string, threshold = 0.3): Promise<number[]> {
const { stderr } = await execFileAsync('ffmpeg', [
'-i', videoPath,
'-filter:v', `select='gt(scene,${threshold})',showinfo`,
'-f', 'null', '-'
], { maxBuffer: 10 * 1024 * 1024 });
const times: number[] = [];
for (const line of stderr.split('\n')) {
const match = line.match(/pts_time:([0-9.]+)/);
if (match) {
const pts = parseFloat(match[1]);
if (pts > 0) times.push(pts);
}
}
return times;
}
Step 2: Build Clips from Break Points
Group segments between break points into clips targeting the desired length range.
interface ClipRange {
index: number;
start: number;
end: number;
duration: number;
breakType: string;
}
interface ChunkingConfig {
minClipDuration: number;
maxClipDuration: number;
idealClipDuration: number;
}
const PLATFORM_CONFIGS: Record<string, ChunkingConfig> = {
tiktok: { minClipDuration: 10, maxClipDuration: 45, idealClipDuration: 25 },
reels: { minClipDuration: 10, maxClipDuration: 30, idealClipDuration: 20 },
instagram_reels: { minClipDuration: 10, maxClipDuration: 30, idealClipDuration: 20 },
instagram_stories: { minClipDuration: 1, maxClipDuration: 60, idealClipDuration: 15 },
instagram_feed: { minClipDuration: 3, maxClipDuration: 60, idealClipDuration: 30 },
shorts: { minClipDuration: 15, maxClipDuration: 60, idealClipDuration: 40 },
youtube_shorts: { minClipDuration: 15, maxClipDuration: 60, idealClipDuration: 40 },
youtube: { minClipDuration: 30, maxClipDuration: 300, idealClipDuration: 120 },
twitter: { minClipDuration: 10, maxClipDuration: 45, idealClipDuration: 25 },
x: { minClipDuration: 10, maxClipDuration: 45, idealClipDuration: 25 },
facebook_reels: { minClipDuration: 10, maxClipDuration: 30, idealClipDuration: 20 },
facebook_feed: { minClipDuration: 15, maxClipDuration: 60, idealClipDuration: 30 },
linkedin: { minClipDuration: 10, maxClipDuration: 60, idealClipDuration: 30 },
pinterest: { minClipDuration: 6, maxClipDuration: 60, idealClipDuration: 20 },
snapchat: { minClipDuration: 5, maxClipDuration: 60, idealClipDuration: 20 },
general: { minClipDuration: 10, maxClipDuration: 60, idealClipDuration: 30 },
};
const PLATFORM_REFRAME: Record<string, { width: number; height: number; ratio: string }> = {
tiktok: { width: 1080, height: 1920, ratio: '9:16' },
reels: { width: 1080, height: 1920, ratio: '9:16' },
instagram_reels: { width: 1080, height: 1920, ratio: '9:16' },
instagram_stories: { width: 1080, height: 1920, ratio: '9:16' },
instagram_feed: { width: 1080, height: 1080, ratio: '1:1' },
shorts: { width: 1080, height: 1920, ratio: '9:16' },
youtube_shorts: { width: 1080, height: 1920, ratio: '9:16' },
youtube: { width: 1920, height: 1080, ratio: '16:9' },
twitter: { width: 1280, height: 720, ratio: '16:9' },
x: { width: 1280, height: 720, ratio: '16:9' },
facebook_reels: { width: 1080, height: 1920, ratio: '9:16' },
facebook_feed: { width: 1920, height: 1080, ratio: '16:9' },
linkedin: { width: 1920, height: 1080, ratio: '16:9' },
pinterest: { width: 1080, height: 1920, ratio: '9:16' },
snapchat: { width: 1080, height: 1920, ratio: '9:16' },
};
async function buildClips(
videoPath: string,
breakPoints: BreakPoint[],
platform: string = 'tiktok'
): Promise<ClipRange[]> {
const config = PLATFORM_CONFIGS[platform] ?? PLATFORM_CONFIGS.general;
const { stdout } = await execFileAsync('ffprobe', [
'-v', 'quiet', '-print_format', 'json', '-show_format', videoPath
]);
const totalDuration = parseFloat(JSON.parse(stdout).format.duration);
const allPoints = [0, ...breakPoints.map(bp => bp.time), totalDuration];
const uniquePoints = [...new Set(allPoints)].sort((a, b) => a - b);
const clips: ClipRange[] = [];
let clipStart = 0;
let clipIndex = 0;
for (let i = 1; i < uniquePoints.length; i++) {
const candidateEnd = uniquePoints[i];
const candidateDuration = candidateEnd - clipStart;
if (candidateDuration >= config.idealClipDuration) {
clips.push({
index: clipIndex++,
start: clipStart,
end: candidateEnd,
duration: candidateDuration,
breakType: breakPoints.find(bp => Math.abs(bp.time - candidateEnd) < 0.5)?.type ?? 'end',
});
clipStart = candidateEnd;
} else if (candidateDuration > config.maxClipDuration) {
const bestBreak = findBestBreakInRange(
breakPoints,
clipStart + config.minClipDuration,
clipStart + config.maxClipDuration
);
const splitPoint = bestBreak?.time ?? (clipStart + config.idealClipDuration);
clips.push({
index: clipIndex++,
start: clipStart,
end: splitPoint,
duration: splitPoint - clipStart,
breakType: bestBreak?.type ?? 'forced',
});
clipStart = splitPoint;
i--;
}
}
if (clipStart < totalDuration) {
const remainingDuration = totalDuration - clipStart;
if (remainingDuration >= config.minClipDuration) {
clips.push({
index: clipIndex++,
start: clipStart,
end: totalDuration,
duration: remainingDuration,
breakType: 'end',
});
} else if (clips.length > 0) {
const lastClip = clips[clips.length - 1];
lastClip.end = totalDuration;
lastClip.duration = totalDuration - lastClip.start;
}
}
return clips;
}
function findBestBreakInRange(
breakPoints: BreakPoint[],
rangeStart: number,
rangeEnd: number
): BreakPoint | null {
const candidates = breakPoints.filter(bp => bp.time >= rangeStart && bp.time <= rangeEnd);
if (candidates.length === 0) return null;
return candidates.sort((a, b) => b.strength - a.strength)[0];
}
Step 3: Export Clips
async function exportChunkedClips(
videoPath: string,
clips: ClipRange[],
outputDir: string
): Promise<string[]> {
await fs.promises.mkdir(outputDir, { recursive: true });
const exportedPaths: string[] = [];
for (const clip of clips) {
const outputPath = path.join(
outputDir,
`clip_${(clip.index + 1).toString().padStart(2, '0')}_${formatTime(clip.start).replace(':', 'm')}s-${formatTime(clip.end).replace(':', 'm')}s.mp4`
);
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-ss', clip.start.toFixed(3),
'-to', clip.end.toFixed(3),
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outputPath
]);
exportedPaths.push(outputPath);
console.log(` Clip ${clip.index + 1}: ${formatTime(clip.start)} - ${formatTime(clip.end)} (${clip.duration.toFixed(1)}s) → ${path.basename(outputPath)}`);
}
return exportedPaths;
}
function formatTime(seconds: number): string {
const m = Math.floor(seconds / 60);
const s = Math.floor(seconds % 60);
return `${m}:${s.toString().padStart(2, '0')}`;
}
Complete Chunking Pipeline
async function chunkVideoForSocialMedia(
videoPath: string,
outputDir: string,
platform: string = 'tiktok'
): Promise<string[]> {
console.log(`Chunking ${videoPath} for ${platform}...`);
const config = PLATFORM_CONFIGS[platform] ?? PLATFORM_CONFIGS.general;
console.log(`Target clip length: ${config.minClipDuration}-${config.maxClipDuration}s (ideal: ${config.idealClipDuration}s)\n`);
console.log('Finding natural break points (scenes + silence)...');
const breakPoints = await findBreakPoints(videoPath);
console.log(`Found ${breakPoints.length} break points\n`);
console.log('Building clips...');
const clips = await buildClips(videoPath, breakPoints, platform);
console.log(`Generated ${clips.length} clips:\n`);
for (const clip of clips) {
console.log(` Clip ${clip.index + 1}: ${formatTime(clip.start)}-${formatTime(clip.end)} (${clip.duration.toFixed(1)}s, break: ${clip.breakType})`);
}
console.log('\nExporting clips...');
const paths = await exportChunkedClips(videoPath, clips, outputDir);
console.log(`\nDone! ${paths.length} clips exported to ${outputDir}/`);
return paths;
}
---
Combined Pipeline
For the full workflow of taking a raw long-form video and producing polished social clips.
IMPORTANT: When a user specifies a platform, reframing is NOT optional — the output must be in the platform's native format. TikTok/Reels = 9:16 (1080x1920), YouTube = 16:9 (1920x1080), Instagram Feed = 1:1 (1080x1080). Never deliver landscape clips when the user asked for TikTok content.
async function fullSocialMediaPipeline(
videoPath: string,
outputDir: string,
options: {
platform?: string;
removeDeadSpace?: boolean;
deadSpacePreset?: SilencePreset;
scoreWithAI?: boolean;
} = {}
) {
const {
platform = 'tiktok',
removeDeadSpace: shouldRemoveDeadSpace = true,
deadSpacePreset = 'medium',
scoreWithAI = false,
} = options;
let workingPath = videoPath;
if (shouldRemoveDeadSpace) {
console.log('=== Step 1: Removing dead space ===\n');
const tightPath = path.join(outputDir, 'tightened.mp4');
const result = await removeDeadSpace(workingPath, tightPath, deadSpacePreset);
console.log(`Removed ${result.removedSeconds.toFixed(1)}s of dead space (${result.originalDuration.toFixed(1)}s → ${result.newDuration.toFixed(1)}s)\n`);
workingPath = tightPath;
}
console.log('=== Step 2: Chunking into clips ===\n');
const clipPaths = await chunkVideoForSocialMedia(workingPath, path.join(outputDir, 'clips'), platform);
if (scoreWithAI) {
console.log('\n=== Step 3: AI Virality Scoring (clip-level) ===\n');
// Score each exported clip using the clip-level prompt (Mode 2)
// This uses CLIP_SCORING_PROMPT which evaluates narrative arc,
// hook-to-payoff, and standalone quality — not just visual moments.
// See virality-scoring.md for the full scoring pipeline.
//
// Key differences from segment scoring:
// - Extracts 5-8 frames per clip (not 3) including first and last frame
// - Uses CLIP_SCORING_PROMPT which adds narrativeCompleteness,
// hookStrength, and standaloneQuality factors
// - Returns postingRecommendation: 'post as-is', 'post with minor edits', or 'consider skipping'
//
// Import and use:
// import { extractFrames, analyzeSegment, getScoringPrompt } from virality pipeline
// Score each clip file, rank by overallScore, present ranked results to user
console.log('Scoring clips with AI (clip-level prompt — evaluates narrative arc, hook, and standalone quality)...');
}
// Step 4: Reframe ALL clips for the target platform
// This is NOT optional when a platform is specified — users expect platform-ready output.
// See operations.md "Social Media Reframing" section for FFmpeg commands.
// Preferred strategy: blurred fill for landscape→portrait (looks polished).
// Alternative: center crop if the subject is center-framed throughout.
console.log('\n=== Step 4: Reframing clips for platform ===\n');
const reframeSpec = PLATFORM_REFRAME[platform];
if (reframeSpec) {
const reframedDir = path.join(outputDir, 'reframed');
await fs.promises.mkdir(reframedDir, { recursive: true });
const { width, height, ratio } = reframeSpec;
console.log(`Reframing all clips to ${ratio} (${width}×${height}) for ${platform}...`);
// Use blurred fill strategy (Strategy 3 from operations.md) for each clip
// See operations.md for the exact FFmpeg filter_complex commands
for (const clipPath of clipPaths) {
const outName = path.basename(clipPath, '.mp4') + `_${platform}.mp4`;
const outPath = path.join(reframedDir, outName);
await execFileAsync('ffmpeg', [
'-y', '-i', clipPath,
'-filter_complex',
`[0:v]scale=${width}:${height}:force_original_aspect_ratio=increase,crop=${width}:${height},gblur=sigma=50[bg];` +
`[0:v]scale=${width}:${height}:force_original_aspect_ratio=decrease[fg];` +
`[bg][fg]overlay=(W-w)/2:(H-h)/2[outv]`,
'-map', '[outv]', '-map', '0:a?',
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outPath
]);
console.log(` ${path.basename(clipPath)} → ${outName}`);
}
} else {
console.log(`No reframing needed for platform: ${platform}`);
}
return clipPaths;
}
Example Usage
await fullSocialMediaPipeline('raw_video.mp4', 'output/social', {
platform: 'tiktok',
removeDeadSpace: true,
deadSpacePreset: 'medium',
scoreWithAI: true,
});
This produces:
1. output/social/tightened.mp4 — the video with dead space removed 2. output/social/clips/clip_01_...mp4 through clip_N_...mp4 — individual clips chunked to platform duration targets
3. output/social/reframed/clip_01_tiktok.mp4 — each clip reframed to 9:16 vertical (1080x1920) for TikTok 4. Each clip scored with clip-level AI analysis (narrative arc, hook strength, standalone quality)
5. Clips ranked with posting recommendations ('post as-is', 'post with minor edits', 'consider skipping') 6. The reframed clips are the final deliverables — always present these to the user, not the intermediate landscape clips
Why clip-level scoring matters
The original segment-level scoring (Mode 1) evaluates tiny 2-10 second fragments on visual dynamism and emotional moments. This is useful for finding highlights, but it doesn't tell you whether a 30-second clip will actually perform well on social media.
Clip-level scoring (Mode 2) evaluates the full clip as a piece of content that will be posted. It checks:
- Does the clip open strong (hook evaluation on the first frame)?
- Does it tell a complete story in 15-45 seconds?
- Would it make sense without context from the full video?
- Does the ending satisfy or intrigue the viewer?
A clip can have great individual moments but still fail as a social post if it starts slow, ends mid-sentence, or requires context the viewer doesn't have. Clip-level scoring catches these issues.
Key Implementation Notes
- `silenceremove` vs segmented approach: FFmpeg's
silenceremovefilter only strips silence from the audio — it does NOT cut the corresponding video frames. For proper dead space removal where both audio and video are cut, always use the segmented extract-and-concatenate approach. - Padding around speaking segments: Always add 0.05-0.15s padding around each speaking segment. Without padding, words get clipped at the start/end.
- Break point priority: When building clips, prefer to split at points that are both a scene change AND a silence gap (strength 1.0). Pure scene changes (0.6) are better than pure silence (0.4), because visual continuity matters more than audio gaps.
- Short remainders: If the last clip would be under
minClipDuration, merge it into the previous clip rather than exporting a too-short orphan.
- Re-encoding: Each extract-and-concat pass re-encodes the video. For the combined pipeline (dead space removal + chunking), consider running dead space removal first to produce one clean file, then chunking that — this minimizes total re-encodes to 2 passes instead of more.
FFmpeg Operations Reference
Complete command reference for video editing operations in Replit. All examples use Node.js with child_process.execFileorfluent-ffmpeg.
Table of Contents
1. Trimming and Cutting 2. Concatenation
3. Transitions 4. Text Overlays
7. Effects 8. Format Conversion
9. Image Extraction 10. GIF Creation
11. Resizing and Cropping 12. Watermarks and Overlays
13. Compression 14. Stabilization
15. Looping 16. Rotation and Flipping
17. Split Screen 18. Muting and Audio Removal
19. Thumbnail Generation 20. Watermark Removal (Crop)
21. Logo Overlay (Branding) 22. Social Media Reframing
---
Trimming and Cutting
Trim without re-encoding (fast, frame-accurate at keyframes)
ffmpeg -i input.mp4 -ss 00:00:30 -to 00:01:15 -c copy output.mp4
Place -ssbefore-ifor faster seeking (input seeking), or after-ifor frame-accurate seeking (output seeking). Using-c copy avoids re-encoding but may have imprecise start points at non-keyframe positions.
Trim with re-encoding (slower, frame-accurate)
ffmpeg -i input.mp4 -ss 00:00:30 -to 00:01:15 -c:v libx264 -c:a aac output.mp4
Remove a section (keep before and after)
Split into two parts and concatenate:
ffmpeg -i input.mp4 -t 00:00:30 -c copy part1.mp4
ffmpeg -i input.mp4 -ss 00:01:00 -c copy part2.mp4
Then concatenate (see Concatenation section).
---
Concatenation
Concat demuxer (same codec, resolution, frame rate)
Create a text file concat_list.txt:
file 'clip1.mp4'
file 'clip2.mp4'
file 'clip3.mp4'
ffmpeg -f concat -safe 0 -i concat_list.txt -c copy output.mp4
This is the fastest method because it avoids re-encoding, but all clips must share the same codec, resolution, and frame rate.
Concat filter (different codecs/resolutions)
ffmpeg -i clip1.mp4 -i clip2.mp4 \\
-filter_complex "[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1[outv][outa]" \\
-map "[outv]" -map "[outa]" output.mp4
This re-encodes everything but handles mismatched inputs. Scale inputs to a common resolution first if they differ.
---
Transitions
Crossfade between two clips
ffmpeg -i clip1.mp4 -i clip2.mp4 \\
-filter_complex "\\
[0:v]trim=duration=5,setpts=PTS-STARTPTS[v0]; \\
[1:v]trim=duration=5,setpts=PTS-STARTPTS[v1]; \\
[v0][v1]xfade=transition=fade:duration=1:offset=4[outv]; \\
[0:a]atrim=duration=5,asetpts=PTS-STARTPTS[a0]; \\
[1:a]atrim=duration=5,asetpts=PTS-STARTPTS[a1]; \\
[a0][a1]acrossfade=d=1[outa]" \\
-map "[outv]" -map "[outa]" output.mp4
Available xfade transitions
fade, wipeleft, wiperight, wipeup, wipedown, slideleft, slideright, slideup, slidedown, circlecrop, rectcrop, distance, fadeblack, fadewhite, radial, smoothleft, smoothright, smoothup, smoothdown, circleopen, circleclose, vertopen, vertclose, horzopen, horzclose, dissolve, pixelize, diagtl, diagtr, diagbl, diagbr, hlslice, hrslice, vuslice, vdslice, hblur, fadegrays, squeezev, squeezeh, zoomin, hlwind, hrwind, vuwind, vdwind, coverleft, coverright, coverup, coverdown, revealleft, revealright, revealup, revealdown
Fade in/out
# Fade in first 2 seconds
ffmpeg -i input.mp4 -vf "fade=t=in:st=0:d=2" -c:a copy output.mp4
# Fade out last 2 seconds (requires knowing duration)
ffmpeg -i input.mp4 -vf "fade=t=out:st=8:d=2" -c:a copy output.mp4
# Audio fade in/out
ffmpeg -i input.mp4 -af "afade=t=in:st=0:d=2,afade=t=out:st=8:d=2" output.mp4
---
Text Overlays
Static text overlay
ffmpeg -i input.mp4 \\
-vf "drawtext=text='Hello World':fontsize=48:fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2:fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf" \\
output.mp4
Text with background box
ffmpeg -i input.mp4 \\
-vf "drawtext=text='Title':fontsize=64:fontcolor=white:box=1:boxcolor=black@0.6:boxborderw=10:x=(w-text_w)/2:y=50" \\
output.mp4
Timed text (appears and disappears)
ffmpeg -i input.mp4 \\
-vf "drawtext=text='Scene 1':fontsize=48:fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2:enable='between(t,2,5)'" \\
output.mp4
Font discovery
List available fonts in the Replit environment:
fc-list : family style | sort
Or use a custom font file by downloading it to the project and referencing it with fontfile=.
---
Subtitles
Burn subtitles from SRT file
ffmpeg -i input.mp4 -vf "subtitles=subs.srt" output.mp4
Burn subtitles with styling
ffmpeg -i input.mp4 \\
-vf "subtitles=subs.srt:force_style='FontSize=24,FontName=Arial,PrimaryColour=&Hffffff&,OutlineColour=&H000000&,Outline=2'" \\
output.mp4
Burn ASS/SSA subtitles (preserves advanced styling)
ffmpeg -i input.mp4 -vf "ass=subs.ass" output.mp4
Embed subtitles as a stream (soft subs, MKV only)
ffmpeg -i input.mp4 -i subs.srt -c copy -c:s srt output.mkv
---
Audio
Add background music
ffmpeg -i video.mp4 -i music.mp3 \\
-filter_complex "[1:a]volume=0.3[music];[0:a][music]amix=inputs=2:duration=first[outa]" \\
-map 0:v -map "[outa]" -c:v copy output.mp4
Replace audio entirely
ffmpeg -i video.mp4 -i new_audio.mp3 \\
-map 0:v -map 1:a -c:v copy -shortest output.mp4
Extract audio
ffmpeg -i video.mp4 -vn -c:a libmp3lame -q:a 2 output.mp3
ffmpeg -i video.mp4 -vn -c:a copy output.aac
Remove audio
ffmpeg -i input.mp4 -an -c:v copy output.mp4
Adjust volume
# Set to 50% volume
ffmpeg -i input.mp4 -af "volume=0.5" -c:v copy output.mp4
# Increase by 10dB
ffmpeg -i input.mp4 -af "volume=10dB" -c:v copy output.mp4
# Normalize audio
ffmpeg -i input.mp4 -af "loudnorm" -c:v copy output.mp4
Audio delay/offset
# Delay audio by 1 second
ffmpeg -i input.mp4 -itsoffset 1 -i input.mp4 \\
-map 0:v -map 1:a -c copy output.mp4
---
Effects
Speed change
# 2x speed
ffmpeg -i input.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" \\
-map "[v]" -map "[a]" output.mp4
# 0.5x speed (slow motion)
ffmpeg -i input.mp4 -filter_complex "[0:v]setpts=2.0*PTS[v];[0:a]atempo=0.5[a]" \\
-map "[v]" -map "[a]" output.mp4
Note: atempoonly accepts values between 0.5 and 100.0. For extreme slow-motion, chain multiple atempo filters:atempo=0.5,atempo=0.5 for 0.25x.
Reverse
ffmpeg -i input.mp4 -vf reverse -af areverse output.mp4
Warning: reverse loads the entire video into memory. Only use on short clips.
Rotate
# 90 degrees clockwise
ffmpeg -i input.mp4 -vf "transpose=1" output.mp4
# 90 degrees counter-clockwise
ffmpeg -i input.mp4 -vf "transpose=2" output.mp4
# 180 degrees
ffmpeg -i input.mp4 -vf "transpose=1,transpose=1" output.mp4
# Arbitrary angle (in radians)
ffmpeg -i input.mp4 -vf "rotate=PI/6" output.mp4
Color adjustments
# Brightness, contrast, saturation
ffmpeg -i input.mp4 -vf "eq=brightness=0.1:contrast=1.2:saturation=1.3" output.mp4
# Convert to grayscale
ffmpeg -i input.mp4 -vf "hue=s=0" output.mp4
# Color curves (cinematic look)
ffmpeg -i input.mp4 -vf "curves=preset=cross_process" output.mp4
Available curve presets: none, color_negative, cross_process, darker, increase_contrast, lighter, linear_contrast, medium_contrast, negative, strong_contrast, vintage
Blur
# Box blur
ffmpeg -i input.mp4 -vf "boxblur=5:1" output.mp4
# Gaussian blur
ffmpeg -i input.mp4 -vf "gblur=sigma=10" output.mp4
Picture-in-picture
ffmpeg -i main.mp4 -i overlay.mp4 \\
-filter_complex "[1:v]scale=320:240[pip];[0:v][pip]overlay=W-w-10:H-h-10" \\
output.mp4
Stabilize shaky video
# Step 1: Analyze
ffmpeg -i input.mp4 -vf "vidstabdetect=shakiness=5:accuracy=15" -f null -
# Step 2: Apply stabilization
ffmpeg -i input.mp4 -vf "vidstabtransform=smoothing=10:input=transforms.trf" output.mp4
---
Format Conversion
MP4 (H.264 + AAC — most compatible)
ffmpeg -i input.webm -c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k output.mp4
CRF values: 18 (high quality, large file) to 28 (lower quality, small file). 23 is a good default.
WebM (VP9 + Opus — web-optimized)
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus output.webm
MOV (ProRes — editing friendly)
ffmpeg -i input.mp4 -c:v prores_ks -profile:v 3 -c:a pcm_s16le output.mov
Audio only
ffmpeg -i input.mp4 -vn -c:a libmp3lame -q:a 2 output.mp3
ffmpeg -i input.mp4 -vn -c:a libopus -b:a 128k output.opus
ffmpeg -i input.mp4 -vn -c:a flac output.flac
---
Image Extraction
Extract a single frame
ffmpeg -i input.mp4 -ss 00:00:05 -frames:v 1 frame.png
Extract frames at interval
# One frame per second
ffmpeg -i input.mp4 -vf "fps=1" frames/frame_%04d.png
# One frame every 10 seconds
ffmpeg -i input.mp4 -vf "fps=1/10" frames/frame_%04d.png
Create a thumbnail sheet (contact sheet)
ffmpeg -i input.mp4 -frames:v 1 -vf "select=not(mod(n\\100)),scale=160:90,tile=5x4" thumbnail.png
---
GIF Creation
Basic GIF
ffmpeg -i input.mp4 -ss 00:00:02 -t 5 -vf "fps=15,scale=480:-1" output.gif
High-quality GIF with palette
# Step 1: Generate palette
ffmpeg -i input.mp4 -ss 2 -t 5 -vf "fps=15,scale=480:-1:flags=lanczos,palettegen" palette.png
# Step 2: Use palette for GIF
ffmpeg -i input.mp4 -i palette.png -ss 2 -t 5 \\
-filter_complex "[0:v]fps=15,scale=480:-1:flags=lanczos[v];[v][1:v]paletteuse" output.gif
The two-pass palette method produces significantly better GIFs with less banding.
---
Resizing and Cropping
Scale to specific dimensions
# Exact size
ffmpeg -i input.mp4 -vf "scale=1280:720" -c:a copy output.mp4
# Scale width, auto height (maintain aspect ratio)
ffmpeg -i input.mp4 -vf "scale=1280:-2" -c:a copy output.mp4
# Scale to fit within bounds (maintain aspect ratio)
ffmpeg -i input.mp4 -vf "scale='min(1280,iw)':'min(720,ih)':force_original_aspect_ratio=decrease" output.mp4
Crop
# Crop to 1280x720 from center
ffmpeg -i input.mp4 -vf "crop=1280:720" -c:a copy output.mp4
# Crop with offset (x=100, y=50)
ffmpeg -i input.mp4 -vf "crop=1280:720:100:50" -c:a copy output.mp4
# Crop to 16:9 from center
ffmpeg -i input.mp4 -vf "crop=ih*16/9:ih" -c:a copy output.mp4
Add padding/letterbox
# Add black bars to make 16:9
ffmpeg -i input.mp4 -vf "pad=ih*16/9:ih:(ow-iw)/2:(oh-ih)/2:black" output.mp4
---
Watermarks and Overlays
Image watermark
# Bottom-right corner with padding
ffmpeg -i video.mp4 -i watermark.png \\
-filter_complex "overlay=W-w-10:H-h-10" output.mp4
# With opacity
ffmpeg -i video.mp4 -i watermark.png \\
-filter_complex "[1:v]format=rgba,colorchannelmixer=aa=0.5[wm];[0:v][wm]overlay=W-w-10:H-h-10" output.mp4
Timed overlay (appears at specific time)
ffmpeg -i video.mp4 -i overlay.png \\
-filter_complex "overlay=10:10:enable='between(t,3,8)'" output.mp4
---
Compression
Reduce file size while maintaining acceptable quality.
Quick compress (reduce bitrate)
# Moderate compression — good balance of size vs quality
ffmpeg -y -i input.mp4 \\
-c:v libx264 -crf 28 -preset medium \\
-c:a aac -b:a 96k \\
output_compressed.mp4
# Aggressive compression — much smaller file, noticeable quality loss
ffmpeg -y -i input.mp4 \\
-c:v libx264 -crf 32 -preset slow \\
-c:a aac -b:a 64k \\
output_small.mp4
# Target a specific file size (e.g., 25MB for Discord/email)
# Calculate bitrate: bitrate = (target_size_MB * 8192) / duration_seconds
# For a 60s video targeting 25MB: bitrate = (25 * 8192) / 60 ≈ 3413 kbps
ffmpeg -y -i input.mp4 \\
-c:v libx264 -b:v 3000k -maxrate 3400k -bufsize 6800k -preset medium \\
-c:a aac -b:a 128k \\
output_25mb.mp4
Two-pass encoding (best quality at target size)
ffmpeg -y -i input.mp4 \\
-c:v libx264 -b:v 2000k -preset medium -pass 1 \\
-an -f null /dev/null
ffmpeg -y -i input.mp4 \\
-c:v libx264 -b:v 2000k -preset medium -pass 2 \\
-c:a aac -b:a 128k \\
output_2pass.mp4
Reduce resolution for smaller file
# Downscale to 720p (keeps aspect ratio)
ffmpeg -y -i input.mp4 \\
-vf "scale=-2:720" \\
-c:v libx264 -crf 23 -preset medium \\
-c:a aac -b:a 128k \\
output_720p.mp4
# Downscale to 480p
ffmpeg -y -i input.mp4 \\
-vf "scale=-2:480" \\
-c:v libx264 -crf 23 -preset medium \\
-c:a aac -b:a 96k \\
output_480p.mp4
CRF guide: 18 = visually lossless, 23 = default (good quality), 28 = moderate compression, 32 = heavy compression, 40+ = very low quality. Each +6 roughly halves the file size.
---
Stabilization
Stabilize shaky footage using FFmpeg's vidstab filter (two-pass process).
Basic stabilization
# Pass 1: Analyze motion (creates transforms.trf)
ffmpeg -y -i shaky.mp4 \\
-vf vidstabdetect=shakiness=5:accuracy=15:result=transforms.trf \\
-f null -
# Pass 2: Apply stabilization
ffmpeg -y -i shaky.mp4 \\
-vf vidstabtransform=input=transforms.trf:smoothing=10:crop=black:zoom=1 \\
-c:v libx264 -crf 23 -preset medium \\
-c:a copy \\
stabilized.mp4
Node.js wrapper
async function stabilizeVideo(inputPath: string, outputPath: string, smoothing = 10) {
const transformsFile = inputPath + '.transforms.trf';
// Pass 1: detect motion
await execFileAsync('ffmpeg', [
'-y', '-i', inputPath,
'-vf', `vidstabdetect=shakiness=5:accuracy=15:result=${transformsFile}`,
'-f', 'null', '-'
]);
// Pass 2: apply stabilization
await execFileAsync('ffmpeg', [
'-y', '-i', inputPath,
'-vf', `vidstabtransform=input=${transformsFile}:smoothing=${smoothing}:crop=black:zoom=1`,
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'copy',
outputPath
]);
await fs.promises.unlink(transformsFile).catch(() => {});
}
Parameters
shakiness(1-10): How shaky the video is. Default 5, use 8-10 for very shaky footage.
-smoothing(0-30): How much to smooth the camera path. Higher = smoother but may crop more. Default 10.
-zoom (0-10): Extra zoom to hide black borders from stabilization. 1 = slight zoom, 0 = no zoom (may show borders).
---
Looping
Repeat a clip multiple times.
# Loop a clip 3 times using stream_loop
ffmpeg -y -stream_loop 2 -i input.mp4 \\
-c copy \\
output_looped.mp4
# Loop with re-encoding (more reliable for some formats)
ffmpeg -y -stream_loop 2 -i input.mp4 \\
-c:v libx264 -crf 23 -preset medium \\
-c:a aac -b:a 128k \\
output_looped.mp4
Note: -stream_loop 2means play the input 3 times total (original + 2 loops). Use-stream_loop 4 for 5 times, etc.
Create a boomerang effect (forward + reverse)
# Forward then reverse
ffmpeg -y -i input.mp4 \\
-filter_complex "[0:v]reverse[rv];[0:v][rv]concat=n=2:v=1:a=0[outv]" \\
-map "[outv]" \\
-c:v libx264 -crf 23 -preset medium \\
boomerang.mp4
---
Rotation and Flipping
Rotate (2)
# Rotate 90 degrees clockwise
ffmpeg -y -i input.mp4 \\
-vf "transpose=1" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
rotated_90.mp4
# Rotate 90 degrees counter-clockwise
ffmpeg -y -i input.mp4 \\
-vf "transpose=2" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
rotated_270.mp4
# Rotate 180 degrees
ffmpeg -y -i input.mp4 \\
-vf "transpose=1,transpose=1" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
rotated_180.mp4
# Rotate by arbitrary angle (e.g., 15 degrees) — adds black borders
ffmpeg -y -i input.mp4 \\
-vf "rotate=15*PI/180:c=black:ow=rotw(15*PI/180):oh=roth(15*PI/180)" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
rotated_15deg.mp4
Flip
# Flip horizontally (mirror)
ffmpeg -y -i input.mp4 \\
-vf "hflip" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
flipped_horizontal.mp4
# Flip vertically
ffmpeg -y -i input.mp4 \\
-vf "vflip" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
flipped_vertical.mp4
`transpose` values: 0 = 90° counter-clockwise + vertical flip, 1 = 90° clockwise, 2 = 90° counter-clockwise, 3 = 90° clockwise + vertical flip.
---
Split Screen
Put two or more videos side by side or in a grid.
Two videos side by side (horizontal)
ffmpeg -y -i left.mp4 -i right.mp4 \\
-filter_complex "\\
[0:v]scale=640:360[left];\\
[1:v]scale=640:360[right];\\
[left][right]hstack=inputs=2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
split_screen.mp4
Two videos stacked (vertical)
ffmpeg -y -i top.mp4 -i bottom.mp4 \\
-filter_complex "\\
[0:v]scale=1080:540[top];\\
[1:v]scale=1080:540[bottom];\\
[top][bottom]vstack=inputs=2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
stacked.mp4
2x2 grid (four videos)
ffmpeg -y -i v1.mp4 -i v2.mp4 -i v3.mp4 -i v4.mp4 \\
-filter_complex "\\
[0:v]scale=540:360[a];[1:v]scale=540:360[b];\\
[2:v]scale=540:360[c];[3:v]scale=540:360[d];\\
[a][b]hstack=inputs=2[top];\\
[c][d]hstack=inputs=2[bottom];\\
[top][bottom]vstack=inputs=2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
grid_2x2.mp4
Picture-in-picture (2)
# Small video in bottom-right corner (25% size)
ffmpeg -y -i main.mp4 -i pip.mp4 \\
-filter_complex "\\
[1:v]scale=iw/4:ih/4[pip];\\
[0:v][pip]overlay=W-w-20:H-h-20[outv]" \\
-map "[outv]" -map 0:a \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
pip_output.mp4
# PiP with rounded corners (border radius)
ffmpeg -y -i main.mp4 -i pip.mp4 \\
-filter_complex "\\
[1:v]scale=320:180,format=yuva420p,\\
geq='lum=lum(X,Y):a=if(gt(abs(X-W/2),W/2-10)*gt(abs(Y-H/2),H/2-10),0,255)'[pip];\\
[0:v][pip]overlay=W-w-20:H-h-20[outv]" \\
-map "[outv]" -map 0:a \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
pip_rounded.mp4
---
Muting and Audio Removal
# Remove audio track entirely
ffmpeg -y -i input.mp4 -an \\
-c:v copy \\
output_muted.mp4
# Replace audio with silence (keeps audio stream but silent)
ffmpeg -y -i input.mp4 \\
-f lavfi -i anullsrc=channel_layout=stereo:sample_rate=44100 \\
-c:v copy -c:a aac -shortest \\
output_silent.mp4
# Keep only audio (extract as MP3)
ffmpeg -y -i input.mp4 \\
-vn -c:a libmp3lame -q:a 2 \\
audio_only.mp3
# Keep only audio (extract as AAC/M4A)
ffmpeg -y -i input.mp4 \\
-vn -c:a copy \\
audio_only.m4a
---
Thumbnail Generation
Extract a single frame as an image file.
# Frame at a specific timestamp
ffmpeg -y -ss 00:00:05 -i input.mp4 \\
-frames:v 1 -q:v 2 \\
thumbnail.jpg
# Frame at 25% through the video
# (calculate timestamp from duration first with ffprobe)
ffmpeg -y -ss 30 -i input.mp4 \\
-frames:v 1 -q:v 2 \\
thumbnail_mid.jpg
# Multiple thumbnails at regular intervals (e.g., every 10 seconds)
ffmpeg -y -i input.mp4 \\
-vf "fps=1/10" -q:v 2 \\
thumbnails_%03d.jpg
# Best quality PNG thumbnail
ffmpeg -y -ss 00:00:05 -i input.mp4 \\
-frames:v 1 \\
thumbnail.png
Node.js: Smart thumbnail (pick the most interesting frame)
async function generateThumbnail(
videoPath: string,
outputPath: string,
timestamp?: number
): Promise<string> {
if (timestamp === undefined) {
// Default to 25% through the video — usually more interesting than the first frame
const { stdout } = await execFileAsync('ffprobe', [
'-v', 'quiet', '-print_format', 'json', '-show_format', videoPath
]);
const duration = parseFloat(JSON.parse(stdout).format.duration);
timestamp = duration * 0.25;
}
await execFileAsync('ffmpeg', [
'-y', '-ss', timestamp.toFixed(3), '-i', videoPath,
'-frames:v', '1', '-q:v', '2',
outputPath
]);
return outputPath;
}
---
Watermark Removal (Crop)
Remove a watermark by cropping it out. This is the only reliable FFmpeg approach — there is no "inpainting" filter.
# Crop out a watermark in the bottom-right corner
# Removes 60px from the bottom and 120px from the right
ffmpeg -y -i input.mp4 \\
-vf "crop=iw-120:ih-60:0:0" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
output_no_watermark.mp4
# Crop out a watermark at the top-left
# Removes 80px from the top and 200px from the left
ffmpeg -y -i input.mp4 \\
-vf "crop=iw-200:ih-80:200:80" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
output_no_watermark.mp4
# Crop + scale back to original resolution (fills the gap)
ffmpeg -y -i input.mp4 \\
-vf "crop=iw-120:ih-60:0:0,scale=1920:1080" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
output_cleaned.mp4
Crop filter syntax: crop=out_w:out_h:x:y — output width, output height, x offset from left, y offset from top.
Note: Cropping changes the aspect ratio/resolution. If the user needs the original dimensions, scale back up after cropping, though this may introduce slight softness.
---
Logo Overlay (Branding)
Add a persistent logo/brand image to a video.
# Logo in the top-left corner with padding
ffmpeg -y -i video.mp4 -i logo.png \\
-filter_complex "overlay=20:20" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
branded.mp4
# Logo in the bottom-right corner
ffmpeg -y -i video.mp4 -i logo.png \\
-filter_complex "overlay=W-w-20:H-h-20" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
branded.mp4
# Logo with transparency (50% opacity)
ffmpeg -y -i video.mp4 -i logo.png \\
-filter_complex "[1:v]format=rgba,colorchannelmixer=aa=0.5[logo];[0:v][logo]overlay=W-w-20:20" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
branded_subtle.mp4
# Logo scaled to a specific size (e.g., 100px wide, keep aspect ratio)
ffmpeg -y -i video.mp4 -i logo.png \\
-filter_complex "[1:v]scale=100:-1[logo];[0:v][logo]overlay=W-w-20:20" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
branded_small.mp4
# Logo that appears only during the first 5 seconds
ffmpeg -y -i video.mp4 -i logo.png \\
-filter_complex "[1:v]scale=100:-1[logo];[0:v][logo]overlay=20:20:enable='between(t,0,5)'" \\
-c:v libx264 -crf 23 -preset medium -c:a copy \\
branded_intro.mp4
Overlay position shortcuts
- Top-left:
overlay=20:20 - Top-right:
overlay=W-w-20:20
- Bottom-left:
overlay=20:H-h-20 - Bottom-right:
overlay=W-w-20:H-h-20
- Center:
overlay=(W-w)/2:(H-h)/2
---
Social Media Reframing
Reframe videos for different platforms with the correct aspect ratio and resolution.
Platform Reference
| Platform | Aspect Ratio | Resolution | Duration | Codec |
|----------|-------------|------------|----------|-------|
| TikTok | 9:16 | 1080×1920 | 10–45s | H.264 |
| Instagram Reels | 9:16 | 1080×1920 | 10–30s | H.264 |
| Instagram Stories | 9:16 | 1080×1920 | 1–60s | H.264 |
| Instagram Feed | 1:1 or 4:5 | 1080×1080 or 1080×1350 | 3–60s | H.264 |
| YouTube Shorts | 9:16 | 1080×1920 | 15–60s | H.264 |
| YouTube (standard) | 16:9 | 1920×1080 | any | H.264 |
| X / Twitter | 16:9 | 1280×720 | 10–45s | H.264 |
| X / Twitter (square) | 1:1 | 720×720 | 10–45s | H.264 |
| Facebook Reels | 9:16 | 1080×1920 | 10–30s | H.264 |
| Facebook Feed | 16:9 or 1:1 | 1920×1080 or 1080×1080 | any | H.264 |
| LinkedIn | 16:9 or 1:1 | 1920×1080 or 1080×1080 | 10–60s | H.264 |
| Pinterest | 9:16 or 2:3 | 1080×1920 or 1000×1500 | 6–60s | H.264 |
| Snapchat Spotlight | 9:16 | 1080×1920 | 5–60s | H.264 |
Strategy selection by conversion type
- Landscape → Vertical (9:16): Use blurred fill (Strategy 3) — looks polished on TikTok/Reels/Shorts
- Landscape → Square (1:1): Use blurred fill or center crop depending on framing
- Vertical → Landscape (16:9): Use blurred fill to avoid large black bars
- Same orientation: Use center crop or scale directly
- 4:5 portrait (Instagram Feed): Use letterbox or blurred fill from 16:9
Strategy 1: Center Crop
Crops from the center to fill the target ratio. Best when the subject is center-framed.
# 16:9 landscape → 9:16 vertical (TikTok/Reels) — center crop
ffmpeg -y -i input.mp4 \\
-vf "crop=ih*9/16:ih,scale=1080:1920" \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_tiktok.mp4
# 16:9 landscape → 1:1 square (Instagram) — center crop
ffmpeg -y -i input.mp4 \\
-vf "crop=ih:ih,scale=1080:1080" \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_instagram.mp4
# 9:16 vertical → 16:9 landscape — center crop
ffmpeg -y -i input.mp4 \\
-vf "crop=iw:iw*9/16,scale=1920:1080" \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_youtube.mp4
# 9:16 vertical → 1:1 square — center crop
ffmpeg -y -i input.mp4 \\
-vf "crop=iw:iw,scale=1080:1080" \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_square.mp4
Strategy 2: Letterbox / Pillarbox (black bars)
Fits the entire video inside the target frame with black bars filling the gaps. No content is lost.
# 16:9 → 9:16 with black bars (pillarbox)
ffmpeg -y -i input.mp4 \\
-vf "scale=1080:-2,pad=1080:1920:(ow-iw)/2:(oh-ih)/2:black" \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_tiktok_letterbox.mp4
# 16:9 → 1:1 with black bars
ffmpeg -y -i input.mp4 \\
-vf "scale=1080:-2,pad=1080:1080:(ow-iw)/2:(oh-ih)/2:black" \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_square_letterbox.mp4
# 9:16 → 16:9 with black bars (pillarbox)
ffmpeg -y -i input.mp4 \\
-vf "scale=-2:1080,pad=1920:1080:(ow-iw)/2:(oh-ih)/2:black" \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_youtube_letterbox.mp4
Strategy 3: Blurred Fill (recommended for landscape → portrait)
Uses a blurred, scaled-up version of the video as background behind the original. Looks much more polished than black bars, especially for landscape-to-portrait conversion.
# 16:9 → 9:16 with blurred background fill (best for TikTok/Reels)
ffmpeg -y -i input.mp4 \\
-filter_complex "\\
[0:v]scale=1080:1920:force_original_aspect_ratio=increase,crop=1080:1920,gblur=sigma=50[bg];\\
[0:v]scale=1080:1920:force_original_aspect_ratio=decrease[fg];\\
[bg][fg]overlay=(W-w)/2:(H-h)/2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_tiktok_blur.mp4
# 16:9 → 1:1 with blurred background fill
ffmpeg -y -i input.mp4 \\
-filter_complex "\\
[0:v]scale=1080:1080:force_original_aspect_ratio=increase,crop=1080:1080,gblur=sigma=50[bg];\\
[0:v]scale=1080:1080:force_original_aspect_ratio=decrease[fg];\\
[bg][fg]overlay=(W-w)/2:(H-h)/2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_square_blur.mp4
# 9:16 → 16:9 with blurred background fill
ffmpeg -y -i input.mp4 \\
-filter_complex "\\
[0:v]scale=1920:1080:force_original_aspect_ratio=increase,crop=1920:1080,gblur=sigma=50[bg];\\
[0:v]scale=1920:1080:force_original_aspect_ratio=decrease[fg];\\
[bg][fg]overlay=(W-w)/2:(H-h)/2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_youtube_blur.mp4
# 16:9 → 4:5 portrait with blurred fill (Instagram Feed — preferred over 1:1)
ffmpeg -y -i input.mp4 \\
-filter_complex "\\
[0:v]scale=1080:1350:force_original_aspect_ratio=increase,crop=1080:1350,gblur=sigma=50[bg];\\
[0:v]scale=1080:1350:force_original_aspect_ratio=decrease[fg];\\
[bg][fg]overlay=(W-w)/2:(H-h)/2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_instagram_feed_4x5.mp4
# 16:9 → 2:3 portrait with blurred fill (Pinterest)
ffmpeg -y -i input.mp4 \\
-filter_complex "\\
[0:v]scale=1000:1500:force_original_aspect_ratio=increase,crop=1000:1500,gblur=sigma=50[bg];\\
[0:v]scale=1000:1500:force_original_aspect_ratio=decrease[fg];\\
[bg][fg]overlay=(W-w)/2:(H-h)/2[outv]" \\
-map "[outv]" -map 0:a? \\
-c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k \\
output_pinterest_blur.mp4
Multi-platform Export (Node.js)
Export to all platforms in one go:
import { execFile } from 'child_process';
import { promisify } from 'util';
import * as path from 'path';
const execFileAsync = promisify(execFile);
interface PlatformSpec {
name: string;
width: number;
height: number;
suffix: string;
}
const PLATFORMS: Record<string, PlatformSpec> = {
tiktok: { name: 'TikTok', width: 1080, height: 1920, suffix: 'tiktok' },
instagram_reels: { name: 'Instagram Reels', width: 1080, height: 1920, suffix: 'ig_reels' },
reels: { name: 'Instagram Reels', width: 1080, height: 1920, suffix: 'ig_reels' },
instagram_stories: { name: 'Instagram Stories', width: 1080, height: 1920, suffix: 'ig_stories' },
instagram_feed: { name: 'Instagram Feed', width: 1080, height: 1080, suffix: 'ig_feed' },
instagram_feed_45: { name: 'Instagram Feed 4:5', width: 1080, height: 1350, suffix: 'ig_feed_4x5' },
youtube_shorts: { name: 'YouTube Shorts', width: 1080, height: 1920, suffix: 'yt_shorts' },
shorts: { name: 'YouTube Shorts', width: 1080, height: 1920, suffix: 'yt_shorts' },
youtube: { name: 'YouTube', width: 1920, height: 1080, suffix: 'youtube' },
twitter: { name: 'X / Twitter', width: 1280, height: 720, suffix: 'twitter' },
x: { name: 'X / Twitter', width: 1280, height: 720, suffix: 'twitter' },
twitter_sq: { name: 'X / Twitter Square', width: 720, height: 720, suffix: 'twitter_sq' },
facebook_reels: { name: 'Facebook Reels', width: 1080, height: 1920, suffix: 'fb_reels' },
facebook_feed: { name: 'Facebook Feed', width: 1920, height: 1080, suffix: 'fb_feed' },
linkedin: { name: 'LinkedIn', width: 1920, height: 1080, suffix: 'linkedin' },
pinterest: { name: 'Pinterest', width: 1080, height: 1920, suffix: 'pinterest' },
pinterest_2x3: { name: 'Pinterest 2:3', width: 1000, height: 1500, suffix: 'pinterest_2x3' },
snapchat: { name: 'Snapchat Spotlight', width: 1080, height: 1920, suffix: 'snapchat' },
};
async function reframeForPlatform(
inputPath: string,
platform: PlatformSpec,
outputDir: string,
strategy: 'crop' | 'letterbox' | 'blur' = 'blur'
): Promise<string> {
const ext = path.extname(inputPath);
const base = path.basename(inputPath, ext);
const outputPath = path.join(outputDir, `${base}_${platform.suffix}${ext}`);
const { width, height } = platform;
let filterComplex: string;
switch (strategy) {
case 'crop':
filterComplex = `scale=${width}:${height}:force_original_aspect_ratio=increase,crop=${width}:${height}`;
break;
case 'letterbox':
filterComplex = `scale=${width}:${height}:force_original_aspect_ratio=decrease,pad=${width}:${height}:(ow-iw)/2:(oh-ih)/2:black`;
break;
case 'blur':
filterComplex = [
`[0:v]scale=${width}:${height}:force_original_aspect_ratio=increase,crop=${width}:${height},gblur=sigma=50[bg]`,
`[0:v]scale=${width}:${height}:force_original_aspect_ratio=decrease[fg]`,
`[bg][fg]overlay=(W-w)/2:(H-h)/2[outv]`
].join(';');
break;
}
const args = ['-y', '-i', inputPath];
if (strategy === 'blur') {
args.push('-filter_complex', filterComplex, '-map', '[outv]', '-map', '0:a?');
} else {
args.push('-vf', filterComplex);
}
args.push(
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outputPath
);
await execFileAsync('ffmpeg', args);
return outputPath;
}
async function exportAllPlatforms(
inputPath: string,
outputDir: string,
platforms: string[] = ['youtube', 'tiktok', 'instagram', 'twitter'],
strategy: 'crop' | 'letterbox' | 'blur' = 'blur'
) {
const results: Array<{ platform: string; path: string }> = [];
for (const key of platforms) {
const spec = PLATFORMS[key];
if (!spec) continue;
console.log(`Reframing for ${spec.name} (${spec.width}x${spec.height})...`);
const outPath = await reframeForPlatform(inputPath, spec, outputDir, strategy);
results.push({ platform: spec.name, path: outPath });
console.log(` → ${outPath}`);
}
return results;
}
Virality Scoring Pipeline
Complete implementation guide for AI-powered video analysis and auto-trimming based on virality potential.
Table of Contents
1. Overview 2. Step 1: Scene Detection
3. Step 2: Frame Extraction 4. Step 3: AI Analysis
5. Step 4: Scoring and Ranking 6. Step 5: Output Assembly
---
Overview
The virality scoring pipeline takes a video file as input, breaks it into segments, analyzes each segment using AI vision, scores them on engagement potential, and outputs the best content in the user's preferred format.
Input Video
↓
Scene Detection (FFmpeg)
↓
Frame Extraction (FFmpeg) — 2-3 keyframes per segment
↓
AI Vision Analysis (OpenAI gpt-5.2 or Gemini gemini-2.5-flash)
↓
Score & Rank Segments
↓
Ask User: Best clip / Multiple clips / Highlight reel
↓
Trim & Assemble (FFmpeg)
↓
Output File(s)
---
Step 1: Scene Detection
Use FFmpeg's scene detection filter to find natural cut points in the video. This splits the video into segments based on visual changes.
Important: The ffprobe -f lavfiapproach does not work reliably in Replit's environment. Instead, useffmpegwith theselectandshowinfofilters, then parsepts_time values from stderr output. This is the tested, working approach.
import { execFile } from 'child_process';
import { promisify } from 'util';
const execFileAsync = promisify(execFile);
interface Segment {
index: number;
startTime: number;
endTime: number;
duration: number;
}
async function detectScenes(videoPath: string, threshold = 0.3): Promise<Segment[]> {
const { stderr } = await execFileAsync('ffmpeg', [
'-i', videoPath,
'-filter:v', `select='gt(scene,${threshold})',showinfo`,
'-f', 'null', '-'
], { maxBuffer: 10 * 1024 * 1024 });
const sceneChanges: number[] = [0];
const lines = stderr.split('\n');
for (const line of lines) {
const match = line.match(/pts_time:([0-9.]+)/);
if (match) {
const pts = parseFloat(match[1]);
if (pts > 0) sceneChanges.push(pts);
}
}
const { stdout: probeOut } = await execFileAsync('ffprobe', [
'-v', 'quiet',
'-print_format', 'json',
'-show_format',
videoPath
]);
const totalDuration = parseFloat(JSON.parse(probeOut).format.duration);
sceneChanges.push(totalDuration);
const segments: Segment[] = [];
for (let i = 0; i < sceneChanges.length - 1; i++) {
const start = sceneChanges[i];
const end = sceneChanges[i + 1];
const duration = end - start;
if (duration >= 2.0) {
segments.push({ index: segments.length, startTime: start, endTime: end, duration });
}
}
return segments;
}
Note on segment indexing: Use index: segments.length(notindex: i) when building segments. When short segments are filtered out, i no longer matches the array position, which causes index mismatches during the export step.
Alternative: Fixed-interval segmentation
For videos without clear scene changes (talking head, screencasts), split into fixed-length chunks:
async function splitFixed(videoPath: string, chunkSeconds = 10): Promise<Segment[]> {
const { stdout } = await execFileAsync('ffprobe', [
'-v', 'quiet', '-print_format', 'json', '-show_format', videoPath
]);
const totalDuration = parseFloat(JSON.parse(stdout).format.duration);
const segments: Segment[] = [];
let start = 0;
let index = 0;
while (start < totalDuration) {
const end = Math.min(start + chunkSeconds, totalDuration);
segments.push({ index, startTime: start, endTime: end, duration: end - start });
start = end;
index++;
}
return segments;
}
Choosing the right approach
- Use scene detection for edited content, vlogs, multi-shot videos
- Use fixed-interval for single-camera footage, lectures, screencasts, livestreams
- If scene detection produces too many or too few segments, adjust the
threshold(lower = more sensitive, higher = fewer scenes). Default 0.3 works well for most content.
---
Step 2: Frame Extraction
Extract representative frames for AI analysis. The number of frames should scale with the content duration:
- Short segments (2-10s): 3 frames, evenly spread
- Medium clips (10-30s): 5 frames, including the first and last frame
- Long clips (30-60s): 8 frames, including the first and last frame
Including the first and last frame is important for clip-level scoring — the AI needs to evaluate the hook (opening) and payoff (ending).
import * as fs from 'fs';
import * as path from 'path';
function getFrameCount(durationSeconds: number): number {
if (durationSeconds <= 10) return 3;
if (durationSeconds <= 30) return 5;
return 8;
}
async function extractFrames(
videoPath: string,
segment: Segment,
outputDir: string,
framesPerSegment?: number
): Promise<string[]> {
await fs.promises.mkdir(outputDir, { recursive: true });
const count = framesPerSegment ?? getFrameCount(segment.duration);
const framePaths: string[] = [];
for (let i = 0; i < count; i++) {
const fraction = count === 1 ? 0.5 : i / (count - 1);
const timestamp = Math.min(segment.startTime + fraction * segment.duration, segment.endTime - 0.1);
const framePath = path.join(outputDir, `segment_${segment.index}_frame_${i + 1}.jpg`);
await execFileAsync('ffmpeg', [
'-y',
'-ss', timestamp.toFixed(3),
'-i', videoPath,
'-frames:v', '1',
'-q:v', '2',
framePath
]);
framePaths.push(framePath);
}
return framePaths;
}
Notes
- The frame extraction uses
i / (count - 1)for timestamp calculation, which ensures the first frame is at the very start and the last frame is at the very end of the segment. This is critical for clip-level scoring where the AI evaluates the opening hook and closing payoff. - The timestamp is clamped to
segment.endTime - 0.1to prevent seeking past the end of the video. Without this, the last clip in a video can fail becausestartTime + 1.0 * durationmay exceed the actual file duration due to floating point arithmetic.
---
Step 3: AI Analysis
Send extracted frames to an AI vision model to analyze each segment's virality potential. The AI evaluates visual dynamism, emotional impact, hook quality, pacing, and shareability.
Content Safety Handling
OpenAI's vision API may reject certain frames that trigger content safety filters (e.g., medical imagery, sensitive content involving minors or elderly people). Always wrap AI calls in try/catch and skip segments that fail rather than crashing the entire pipeline:
try {
const score = await analyzeSegment(framePaths, segment, videoContext);
scores.push(score);
} catch (err: any) {
if (err?.code === 'content_policy_violation') {
console.log(`Segment ${segment.index + 1}: Skipped (content safety filter)`);
} else {
console.log(`Segment ${segment.index + 1}: Error — ${err?.message}`);
}
}
This is especially common with documentary, medical, or news content. The pipeline should continue and rank whatever segments it can analyze.
OpenAI Client Setup
For script-based usage (not using the full @workspace/integrations-openai-ai-serverlibrary), installopenai directly and create the client with the AI Integrations environment variables:
import OpenAI from 'openai';
const openai = new OpenAI({
baseURL: process.env.AI_INTEGRATIONS_OPENAI_BASE_URL,
apiKey: process.env.AI_INTEGRATIONS_OPENAI_API_KEY,
});
Install with: pnpm add -w openai
If using the full workspace integration, import from @workspace/integrations-openai-ai-serverinstead (see theai-integrations-openai skill for setup).
Using OpenAI (gpt-5.2)
import OpenAI from 'openai';
import * as fs from 'fs';
const openai = new OpenAI({
baseURL: process.env.AI_INTEGRATIONS_OPENAI_BASE_URL,
apiKey: process.env.AI_INTEGRATIONS_OPENAI_API_KEY,
});
interface ViralityScore {
segmentIndex: number;
overallScore: number;
factors: {
visualDynamism: number;
emotionalImpact: number;
hookPotential: number;
pacingEnergy: number;
uniqueness: number;
shareability: number;
};
reasoning: string;
suggestedCaption: string;
}
async function analyzeSegment(
framePaths: string[],
segment: Segment,
videoContext: string
): Promise<ViralityScore> {
const imageContents = framePaths.map(fp => ({
type: 'image_url' as const,
image_url: {
url: `data:image/jpeg;base64,${fs.readFileSync(fp).toString('base64')}`,
detail: 'low' as const
}
}));
const response = await openai.chat.completions.create({
model: 'gpt-5.2',
max_completion_tokens: 1024,
messages: [
{
role: 'system',
content: getScoringPrompt(segment.duration)
},
{
role: 'user',
content: [
{
type: 'text',
text: `Analyze this video segment for virality potential.
Video context: ${videoContext}
Segment: ${segment.index + 1}
Timestamp: ${formatTime(segment.startTime)} - ${formatTime(segment.endTime)}
Duration: ${segment.duration.toFixed(1)}s
These ${framePaths.length} frames are sampled evenly across the segment. Score this segment's virality potential.`
},
...imageContents
]
}
],
response_format: { type: 'json_object' }
});
const result = JSON.parse(response.choices[0]?.message?.content ?? '{}');
return {
segmentIndex: segment.index,
overallScore: result.overallScore,
factors: result.factors,
reasoning: result.reasoning,
suggestedCaption: result.suggestedCaption
};
}
function formatTime(seconds: number): string {
const m = Math.floor(seconds / 60);
const s = Math.floor(seconds % 60);
return `${m}:${s.toString().padStart(2, '0')}`;
}
Using Gemini (gemini-2.5-flash)
import { ai } from '@workspace/integrations-gemini-ai';
import * as fs from 'fs';
async function analyzeSegmentGemini(
framePaths: string[],
segment: Segment,
videoContext: string
): Promise<ViralityScore> {
const imageParts = framePaths.map(fp => ({
inlineData: {
mimeType: 'image/jpeg',
data: fs.readFileSync(fp).toString('base64')
}
}));
const response = await ai.models.generateContent({
model: 'gemini-2.5-flash',
contents: [{
role: 'user',
parts: [
{ text: `${getScoringPrompt(segment.duration)}\n\nAnalyze this video clip for virality potential.\n\nVideo context: ${videoContext}\nClip: ${segment.index + 1}\nTimestamp: ${formatTime(segment.startTime)} - ${formatTime(segment.endTime)}\nDuration: ${segment.duration.toFixed(1)}s\n\nThese ${framePaths.length} frames are sampled across the clip (first frame = opening hook, last frame = ending). Score this clip's virality potential. Respond in JSON format.` },
...imageParts
]
}],
config: { responseMimeType: 'application/json' }
});
const result = JSON.parse(response.text ?? '{}');
return {
segmentIndex: segment.index,
overallScore: result.overallScore,
factors: result.factors,
reasoning: result.reasoning,
suggestedCaption: result.suggestedCaption
};
}
Scoring Prompts
There are two prompts — one for scoring short segments (Mode 1) and one for scoring complete clips (Mode 2). Use the clip-level prompt when clips are 10+ seconds.
Segment Scoring Prompt (Mode 1 — finding best moments)
Use this when scoring raw scene-detected segments (typically 2-10 seconds). This is the original prompt.
const SEGMENT_SCORING_PROMPT = `You are a social media virality analyst. Your job is to score video segments on their potential to go viral on platforms like TikTok, Instagram Reels, YouTube Shorts, and X/Twitter.
You will receive 2-3 frames from a video segment. Analyze them and score the segment on these factors (each 1-10):
1. **Visual Dynamism** (weight: 20%) — How visually interesting and dynamic is the content? Look for: action/movement implied between frames, striking compositions, dramatic lighting, vibrant colors, visual contrast. Low scores: static shots, bland compositions, poor lighting.
2. **Emotional Impact** (weight: 25%) — Does this segment evoke a strong emotional response? Look for: facial expressions, dramatic moments, humor potential, surprise elements, awe-inspiring visuals, relatable situations. This is the strongest predictor of shares.
3. **Hook Potential** (weight: 20%) — Would this segment work as the opening of a viral clip? Could it stop someone mid-scroll in the first 1-2 seconds? Look for: immediate visual intrigue, curiosity gap, unexpected elements, bold visuals.
4. **Pacing & Energy** (weight: 15%) — Does the visual pacing feel energetic and engaging? Look for: variety between frames (different angles, scales, subjects), implied motion, dynamic framing changes. Monotonous sameness across frames scores low.
5. **Uniqueness** (weight: 10%) — Is this something unusual, novel, or different from typical content? Everyday/generic scenes score low. Unusual situations, rare moments, or distinctive aesthetics score high.
6. **Shareability** (weight: 10%) — Would someone send this to a friend? Look for: "you have to see this" factor, relatable moments, impressive skills/feats, beautiful or shocking visuals.
Respond with JSON in this exact format:
{
"factors": {
"visualDynamism": <1-10>,
"emotionalImpact": <1-10>,
"hookPotential": <1-10>,
"pacingEnergy": <1-10>,
"uniqueness": <1-10>,
"shareability": <1-10>
},
"overallScore": <weighted average, 1-10, one decimal>,
"reasoning": "<2-3 sentences explaining why this segment scored the way it did>",
"suggestedCaption": "<a short, punchy caption that could accompany this clip on social media>"
}
Be honest and critical. Most segments in a typical video are NOT viral-worthy (score 3-5). Reserve scores above 7 for genuinely compelling moments. A score of 9-10 means this could realistically trend on social media.`;
Clip Scoring Prompt (Mode 2 — ranking ready-to-post clips)
Use this when scoring complete clips (10-60 seconds) that have been chunked from a longer video. This prompt adds narrative arc evaluation and is the preferred prompt for the chunking pipeline.
const CLIP_SCORING_PROMPT = `You are a social media content strategist and virality analyst. You are evaluating complete video clips (10-60 seconds) that are ready to be posted on TikTok, Instagram Reels, YouTube Shorts, or X/Twitter.
You will receive 5-8 frames sampled across the entire clip, including the very first and very last frame. The first frame represents the hook (what viewers see first in their feed), and the last frame represents the payoff/ending.
Score this clip on these factors (each 1-10):
### Visual & Production Quality (35% total)
1. **Visual Dynamism** (weight: 13%) — How visually interesting and dynamic is the content across the full clip? Look for: variety of shots, movement, striking compositions, dynamic lighting, color. Score the overall visual journey, not just one frame.
2. **Pacing & Energy** (weight: 12%) — Does the clip maintain engaging pacing throughout? Look for: variety between frames (different angles, scales, subjects), good rhythm. Does it drag anywhere or rush past key moments?
3. **Uniqueness** (weight: 10%) — Is this clip distinctive? Would it stand out in a crowded feed? Everyday/generic content scores low. Fresh angles, unusual moments, or distinctive aesthetics score high.
### Emotional & Engagement Power (35% total)
4. **Emotional Impact** (weight: 20%) — Does this clip evoke a strong emotional response? This is the \#1 predictor of shares. Look for: relatable moments, humor, surprise, awe, empathy, or tension. A clip that makes you feel something deeply scores high.
5. **Shareability** (weight: 15%) — Would someone send this to a friend or repost it? Look for: "you have to see this" factor, quotable moments, impressive feats, beautiful or shocking visuals, relatable situations.
### Clip Structure (30% total)
6. **Hook Strength** (weight: 12%) — Look at the FIRST frame specifically. Would this stop someone mid-scroll in the first 1-2 seconds? Does the clip open with intrigue, a bold visual, or an attention-grabbing moment? A weak opening kills a clip regardless of how good the rest is.
7. **Narrative Completeness** (weight: 10%) — Does this clip tell a complete micro-story? Does it have a clear arc — a setup, development, and resolution (or intentional cliffhanger)? Would a viewer feel satisfied, intrigued, or moved at the end? Clips that end abruptly mid-thought or feel like random fragments score low.
8. **Standalone Quality** (weight: 8%) — Would this clip make sense to someone who has never seen the full video? Can it be posted without any additional context or explanation? Does it work as an independent piece of content?
Respond with JSON in this exact format:
{
"factors": {
"visualDynamism": <1-10>,
"pacingEnergy": <1-10>,
"uniqueness": <1-10>,
"emotionalImpact": <1-10>,
"shareability": <1-10>,
"hookStrength": <1-10>,
"narrativeCompleteness": <1-10>,
"standaloneQuality": <1-10>
},
"overallScore": <weighted average using the weights above, 1-10, one decimal>,
"reasoning": "<2-3 sentences explaining this clip's strengths and weaknesses as a standalone social media post>",
"suggestedCaption": "<a short, punchy caption optimized for the platform>",
"postingRecommendation": "<one of: 'post as-is', 'post with minor edits', 'consider skipping'>"
}
Be honest and critical. A mediocre clip is worse than no post at all — it hurts the algorithm. Most clips from a chunked video will score 3-5. Scores above 7 mean this clip could genuinely perform well. Scores 8+ mean this is strong enough to lead a content calendar.`;
Choosing the Right Prompt
function getScoringPrompt(durationSeconds: number): string {
return durationSeconds >= 10 ? CLIP_SCORING_PROMPT : SEGMENT_SCORING_PROMPT;
}
Use SEGMENT_SCORING_PROMPTfor scene-detected segments under 10 seconds (Mode 1: finding best moments). UseCLIP_SCORING_PROMPTfor clips 10 seconds or longer (Mode 2: ranking chunks for posting).
---
Step 4: Scoring and Ranking
After all segments are analyzed, rank them by overall score.
Important: Each segment requires an AI API call with image attachments. For a 90-second video with ~13 segments, expect the analysis to take 1-2 minutes. For longer videos (5+ minutes), this can take much longer. Always warn the user that analysis will take a moment.
If the@workspace/integrations-openai-ai-server/batch utilities are available, use them for rate limiting. Otherwise, a simple sequential loop with try/catch works well:
async function scoreAllSegments(
videoPath: string,
segments: Segment[],
videoContext: string,
tmpDir: string
): Promise<ViralityScore[]> {
const scores: ViralityScore[] = [];
for (const segment of segments) {
const framePaths = await extractFrames(videoPath, segment, tmpDir);
try {
const score = await analyzeSegment(framePaths, segment, videoContext);
scores.push(score);
} catch (err: any) {
if (err?.code === 'content_policy_violation') {
console.log(`Segment ${segment.index + 1}: Skipped (content safety filter)`);
} else {
console.log(`Segment ${segment.index + 1}: Error — ${err?.message}`);
}
}
for (const fp of framePaths) {
await fs.promises.unlink(fp).catch(() => {});
}
}
return scores.sort((a, b) => b.overallScore - a.overallScore);
}
---
Step 5: Output Assembly
Based on the user's choice, produce one of three outputs.
Option A: Best Clip (single highest-scoring segment)
async function exportBestClip(
videoPath: string,
segments: Segment[],
scores: ViralityScore[],
outputPath: string
): Promise<{ path: string; score: ViralityScore }> {
const best = scores[0];
const segment = segments[best.segmentIndex];
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-ss', segment.startTime.toFixed(3),
'-to', segment.endTime.toFixed(3),
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outputPath
]);
return { path: outputPath, score: best };
}
Option B: Multiple Clips (ranked by score)
async function exportTopClips(
videoPath: string,
segments: Segment[],
scores: ViralityScore[],
outputDir: string,
count = 5
): Promise<Array<{ path: string; score: ViralityScore }>> {
await fs.promises.mkdir(outputDir, { recursive: true });
const topScores = scores.slice(0, count);
const results: Array<{ path: string; score: ViralityScore }> = [];
for (let i = 0; i < topScores.length; i++) {
const score = topScores[i];
const segment = segments[score.segmentIndex];
const outputPath = path.join(outputDir, `clip_${i + 1}_score_${score.overallScore.toFixed(1)}.mp4`);
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-ss', segment.startTime.toFixed(3),
'-to', segment.endTime.toFixed(3),
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outputPath
]);
results.push({ path: outputPath, score });
}
return results;
}
Option C: Highlight Reel (top moments stitched together)
async function exportHighlightReel(
videoPath: string,
segments: Segment[],
scores: ViralityScore[],
outputPath: string,
topCount = 5,
transitionDuration = 0.5
): Promise<string> {
const topScores = scores.slice(0, topCount);
const sortedByTime = [...topScores].sort(
(a, b) => segments[a.segmentIndex].startTime - segments[b.segmentIndex].startTime
);
const tmpDir = path.join(path.dirname(outputPath), 'tmp_highlight');
await fs.promises.mkdir(tmpDir, { recursive: true });
const clipPaths: string[] = [];
for (let i = 0; i < sortedByTime.length; i++) {
const segment = segments[sortedByTime[i].segmentIndex];
const clipPath = path.join(tmpDir, `clip_${i}.mp4`);
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-ss', segment.startTime.toFixed(3),
'-to', segment.endTime.toFixed(3),
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
clipPath
]);
clipPaths.push(clipPath);
}
if (clipPaths.length === 1) {
await fs.promises.copyFile(clipPaths[0], outputPath);
} else {
const filterParts: string[] = [];
let lastVideo = '';
let lastAudio = '';
for (let i = 0; i < clipPaths.length; i++) {
filterParts.push(`[${i}:v]setpts=PTS-STARTPTS[v${i}];`);
filterParts.push(`[${i}:a]asetpts=PTS-STARTPTS[a${i}];`);
}
// Chain xfade transitions between clips
if (clipPaths.length === 2) {
filterParts.push(
`[v0][v1]xfade=transition=fade:duration=${transitionDuration}:offset=auto[outv];`
);
filterParts.push(
`[a0][a1]acrossfade=d=${transitionDuration}[outa]`
);
lastVideo = 'outv';
lastAudio = 'outa';
} else {
// For 3+ clips, use concat with crossfade between each pair
const concatList = path.join(tmpDir, 'concat.txt');
const lines = clipPaths.map(p => `file '${p}'`).join('\n');
await fs.promises.writeFile(concatList, lines);
await execFileAsync('ffmpeg', [
'-y',
'-f', 'concat', '-safe', '0',
'-i', concatList,
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outputPath
]);
// Clean up
for (const cp of clipPaths) await fs.promises.unlink(cp).catch(() => {});
await fs.promises.rm(tmpDir, { recursive: true, force: true });
return outputPath;
}
const inputs = clipPaths.flatMap(p => ['-i', p]);
await execFileAsync('ffmpeg', [
'-y',
...inputs,
'-filter_complex', filterParts.join(''),
'-map', `[${lastVideo}]`,
'-map', `[${lastAudio}]`,
'-c:v', 'libx264', '-crf', '23', '-preset', 'medium',
'-c:a', 'aac', '-b:a', '128k',
outputPath
]);
}
// Clean up
for (const cp of clipPaths) await fs.promises.unlink(cp).catch(() => {});
await fs.promises.rm(tmpDir, { recursive: true, force: true });
return outputPath;
}
---
Complete Pipeline Example
Here's how to tie everything together in a script:
import * as path from 'path';
import * as os from 'os';
async function viralityAutoTrim(
videoPath: string,
outputMode: 'best' | 'multiple' | 'highlight',
videoContext = ''
) {
const tmpDir = path.join(os.tmpdir(), `virality_${Date.now()}`);
console.log('Step 1: Detecting scenes...');
let segments = await detectScenes(videoPath);
// Fall back to fixed intervals if scene detection yields too few segments
if (segments.length < 3) {
console.log('Few scenes detected, using fixed-interval segmentation...');
segments = await splitFixed(videoPath, 10);
}
console.log(`Found ${segments.length} segments`);
console.log('Step 2: Analyzing segments with AI...');
const scores = await scoreAllSegments(videoPath, segments, videoContext, tmpDir);
console.log('\nSegment Rankings:');
for (const score of scores) {
const seg = segments[score.segmentIndex];
console.log(
`\#${score.segmentIndex + 1} [${formatTime(seg.startTime)}-${formatTime(seg.endTime)}]` +
`Score: ${score.overallScore.toFixed(1)}/10 — ${score.reasoning}`
);
}
console.log(`\nStep 3: Generating output (mode: ${outputMode})...`);
const outputDir = path.dirname(videoPath);
switch (outputMode) {
case 'best': {
const { path: outPath, score } = await exportBestClip(
videoPath, segments, scores,
path.join(outputDir, 'best_clip.mp4')
);
console.log(`Best clip exported: ${outPath} (score: ${score.overallScore.toFixed(1)})`);
console.log(`Suggested caption: ${score.suggestedCaption}`);
break;
}
case 'multiple': {
const clips = await exportTopClips(
videoPath, segments, scores,
path.join(outputDir, 'top_clips')
);
console.log(`Exported ${clips.length} clips to ${path.join(outputDir, 'top_clips')}/`);
for (const clip of clips) {
console.log(` ${path.basename(clip.path)} — ${clip.score.reasoning}`);
}
break;
}
case 'highlight': {
const reelPath = await exportHighlightReel(
videoPath, segments, scores,
path.join(outputDir, 'highlight_reel.mp4')
);
console.log(`Highlight reel exported: ${reelPath}`);
break;
}
}
// Clean up temp directory
await fs.promises.rm(tmpDir, { recursive: true, force: true });
console.log('Done!');
}
Scoring Calibration Notes
- Most content scores 3-5. This is normal. Everyday footage, filler shots, and transitional moments are not viral.
- Scores 6-7 indicate genuinely interesting moments worth keeping.
- Scores 8+ are rare and represent truly compelling, share-worthy content.
- If every segment scores above 7, the entire video is likely high-quality content — suggest keeping more segments or using a longer highlight reel.
- If no segment scores above 4, be honest with the user — the video may not have strong viral moments. Suggest what would make it more engaging (better lighting, more action, stronger hook, etc.).
Performance Tips
- Timeout awareness — For a 90-second video with ~13 segments, expect analysis to take 1-2 minutes. A 5-minute video with fixed 10s segments = 30 API calls = 3-5 minutes. Warn the user and set generous script timeouts.
- For long videos (30+ minutes), use fixed-interval segmentation with larger chunks (15-20s) to keep the number of AI calls manageable.
- Use
concurrency: 2in batch processing to avoid rate limits while maintaining speed. - Extract frames at reduced quality (
-q:v 5) if API costs are a concern — the AI can still analyze lower-quality frames effectively.
- Use
detail: 'low'for OpenAI vision to reduce token usage per frame. - Content safety skips — Expect some segments to be skipped due to content safety filters, especially with documentary, medical, or news content. The pipeline should gracefully continue.
- Minimum segment duration — Use 2.0s as the minimum segment threshold (not 1.0s). Very short segments (< 2s) don't produce useful virality scores and waste API calls.
Voiceover with ElevenLabs
Complete implementation guide for adding AI-generated voiceovers to videos using ElevenLabs TTS and FFmpeg audio mixing.
Table of Contents
1. Setup 2. Generating Voiceover Audio
3. Voice Selection 4. Mixing Audio into Video
5. Timed Voiceover Segments 6. Complete Pipeline Example
---
Setup
1. Connect ElevenLabs via Replit Integration
ElevenLabs is available as a Replit connector. Set it up through the integration system:
// In the code execution sandbox:
const results = await searchIntegrations("elevenlabs");
console.log(results);
// Propose the connector to the user (they'll complete OAuth)
await proposeIntegration("connector:ccfg_elevenlabs_...");
After the user authorizes, on the next loop:
// Add the connection to the project
const addResult = await addIntegration("connection:conn_elevenlabs_...");
console.log(addResult);
2. Access Credentials
Use listConnections in the code execution sandbox to get the API key:
const conns = await listConnections('elevenlabs');
if (conns.length > 0) {
const apiKey = conns[0].settings.api_key;
// Use this key for ElevenLabs API calls
}
3. Install Dependencies
No special SDK is needed — ElevenLabs has a straightforward REST API. Use fetch or install the official SDK:
pnpm --filter @workspace/scripts add elevenlabs
# or for the API server:
pnpm --filter @workspace/api-server add elevenlabs
---
Generating Voiceover Audio
Using the ElevenLabs REST API directly
import * as fs from 'fs';
async function generateVoiceover(
text: string,
outputPath: string,
apiKey: string,
voiceId = 'JBFqnCBsd6RMkjVDRZzb', // Default: "George" - warm, narrative voice
options: {
stability?: number;
similarityBoost?: number;
style?: number;
modelId?: string;
} = {}
): Promise<string> {
const {
stability = 0.5,
similarityBoost = 0.75,
style = 0.0,
modelId = 'eleven_multilingual_v2'
} = options;
const response = await fetch(
`https://api.elevenlabs.io/v1/text-to-speech/${voiceId}`,
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
'xi-api-key': apiKey
},
body: JSON.stringify({
text,
model_id: modelId,
voice_settings: {
stability,
similarity_boost: similarityBoost,
style,
use_speaker_boost: true
}
})
}
);
if (!response.ok) {
const error = await response.text();
throw new Error(`ElevenLabs TTS failed: ${response.status} ${error}`);
}
const buffer = Buffer.from(await response.arrayBuffer());
await fs.promises.writeFile(outputPath, buffer);
return outputPath;
}
Using the ElevenLabs SDK
import { ElevenLabsClient } from 'elevenlabs';
import * as fs from 'fs';
async function generateVoiceoverSDK(
text: string,
outputPath: string,
apiKey: string,
voiceId = 'JBFqnCBsd6RMkjVDRZzb'
): Promise<string> {
const client = new ElevenLabsClient({ apiKey });
const audioStream = await client.textToSpeech.convert(voiceId, {
text,
model_id: 'eleven_multilingual_v2',
voice_settings: {
stability: 0.5,
similarity_boost: 0.75,
style: 0.0
}
});
const chunks: Buffer[] = [];
for await (const chunk of audioStream) {
chunks.push(Buffer.from(chunk));
}
await fs.promises.writeFile(outputPath, Buffer.concat(chunks));
return outputPath;
}
---
Voice Selection
Listing Available Voices
async function listVoices(apiKey: string) {
const response = await fetch('https://api.elevenlabs.io/v1/voices', {
headers: { 'xi-api-key': apiKey }
});
const data = await response.json();
return data.voices.map((v: any) => ({
id: v.voice_id,
name: v.name,
category: v.category,
labels: v.labels,
previewUrl: v.preview_url
}));
}
Popular Pre-made Voices
These are commonly available ElevenLabs voices good for voiceovers:
| Voice | ID | Best For |
|-------|-----|----------|
| George | JBFqnCBsd6RMkjVDRZzb | Warm narration, documentaries |
| Rachel | 21m00Tcm4TlvDq8ikWAM | Professional, clear narration |
| Adam | pNInz6obpgDQGcFmaJgB | Deep, authoritative narration |
| Bella | EXAVITQu4vr4xnSDxMaL | Friendly, conversational |
| Antoni | ErXwobaYiN019PkySvjV | Calm, measured narration |
| Domi | AZnzlk1XvdvUeBnXmlld | Energetic, youthful |
When the user doesn't specify a voice, ask what tone they want (professional, warm, energetic, etc.) and suggest an appropriate voice.
Voice Settings Explained
- Stability (0.0–1.0): Higher = more consistent/monotone, Lower = more expressive/varied. Default 0.5 is balanced. Use 0.7+ for narration, 0.3 for dramatic reads.
- Similarity Boost (0.0–1.0): How closely to match the original voice. 0.75 is a good default. Higher values may introduce artifacts.
- Style (0.0–1.0): Exaggeration of the voice's style. 0.0 is neutral. Higher values amplify the voice's character but can sound unnatural. Keep at 0.0 for voiceovers.
---
Mixing Audio into Video
Mode 1: Replace Audio Entirely
Remove existing audio and use only the voiceover:
ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
-map 0:v -map 1:a \\
-c:v copy -c:a aac -b:a 192k \\
-shortest output.mp4
import { execFile } from 'child_process';
import { promisify } from 'util';
const execFileAsync = promisify(execFile);
async function replaceAudio(videoPath: string, voiceoverPath: string, outputPath: string) {
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-i', voiceoverPath,
'-map', '0:v',
'-map', '1:a',
'-c:v', 'copy',
'-c:a', 'aac', '-b:a', '192k',
'-shortest',
outputPath
]);
}
Mode 2: Mix Over (duck existing audio)
Layer voiceover on top of existing audio, reducing the existing audio volume:
ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
-filter_complex "[0:a]volume=0.2[bg];[bg][1:a]amix=inputs=2:duration=first[outa]" \\
-map 0:v -map "[outa]" \\
-c:v copy -c:a aac -b:a 192k \\
output.mp4
async function mixOverAudio(
videoPath: string,
voiceoverPath: string,
outputPath: string,
bgVolume = 0.2 // How much to reduce existing audio (0.0 = silent, 1.0 = full)
) {
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-i', voiceoverPath,
'-filter_complex',
`[0:a]volume=${bgVolume}[bg];[bg][1:a]amix=inputs=2:duration=first[outa]`,
'-map', '0:v',
'-map', '[outa]',
'-c:v', 'copy',
'-c:a', 'aac', '-b:a', '192k',
outputPath
]);
}
Mode 3: Add as Track (full volume both)
Keep existing audio at full volume, add voiceover on top:
ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
-filter_complex "[0:a][1:a]amix=inputs=2:duration=first[outa]" \\
-map 0:v -map "[outa]" \\
-c:v copy -c:a aac -b:a 192k \\
output.mp4
async function addAudioTrack(videoPath: string, voiceoverPath: string, outputPath: string) {
await execFileAsync('ffmpeg', [
'-y',
'-i', videoPath,
'-i', voiceoverPath,
'-filter_complex',
'[0:a][1:a]amix=inputs=2:duration=first[outa]',
'-map', '0:v',
'-map', '[outa]',
'-c:v', 'copy',
'-c:a', 'aac', '-b:a', '192k',
outputPath
]);
}
Adding Voiceover to a Video with No Audio
If the input video has no audio stream:
ffmpeg -y -i video.mp4 -i voiceover.mp3 \\
-map 0:v -map 1:a \\
-c:v copy -c:a aac -b:a 192k \\
-shortest output.mp4
---
Timed Voiceover Segments
For videos that need voiceover at specific timestamps (not continuous narration), generate separate audio clips and position them on a timeline.
Step 1: Define the Script with Timestamps
interface VoiceoverSegment {
text: string;
startTime: number; // seconds
voiceId?: string; // optional per-segment voice
}
const script: VoiceoverSegment[] = [
{ text: "Welcome to our product overview.", startTime: 0 },
{ text: "Here's how it works in three simple steps.", startTime: 5.5 },
{ text: "Step one: upload your content.", startTime: 10 },
{ text: "Step two: customize your settings.", startTime: 18 },
{ text: "Step three: share with the world.", startTime: 25 },
];
Step 2: Generate Each Segment
async function generateSegments(
segments: VoiceoverSegment[],
tmpDir: string,
apiKey: string,
defaultVoiceId: string
): Promise<string[]> {
const paths: string[] = [];
for (let i = 0; i < segments.length; i++) {
const outputPath = path.join(tmpDir, `segment_${i}.mp3`);
await generateVoiceover(
segments[i].text,
outputPath,
apiKey,
segments[i].voiceId || defaultVoiceId
);
paths.push(outputPath);
}
return paths;
}
Step 3: Create a Combined Audio Track with Positioning
Use FFmpeg to position each segment at its target timestamp and mix them into a single audio track:
async function assembleTimedVoiceover(
segments: VoiceoverSegment[],
segmentPaths: string[],
totalDuration: number,
outputPath: string
) {
const inputs: string[] = [];
const filterParts: string[] = [];
for (let i = 0; i < segmentPaths.length; i++) {
inputs.push('-i', segmentPaths[i]);
const delay = Math.round(segments[i].startTime * 1000);
filterParts.push(`[${i}:a]adelay=${delay}|${delay}[a${i}]`);
}
const mixInputs = segments.map((_, i) => `[a${i}]`).join('');
const filterComplex = [
...filterParts,
`${mixInputs}amix=inputs=${segments.length}:normalize=0[mixed]`,
`[mixed]apad=whole_dur=${totalDuration}[out]`
].join(';');
await execFileAsync('ffmpeg', [
'-y',
...inputs,
'-filter_complex', filterComplex,
'-map', '[out]',
'-c:a', 'aac', '-b:a', '192k',
outputPath
]);
}
Step 4: Mix the Assembled Voiceover into the Video
Use one of the mixing modes above (replace, mix over, or add as track) with the assembled voiceover file.
---
Complete Pipeline Example
import * as path from 'path';
import * as os from 'os';
async function addVoiceoverToVideo(
videoPath: string,
script: string | VoiceoverSegment[],
apiKey: string,
options: {
voiceId?: string;
mode?: 'replace' | 'mix' | 'add';
bgVolume?: number;
} = {}
) {
const {
voiceId = 'JBFqnCBsd6RMkjVDRZzb',
mode = 'mix',
bgVolume = 0.2
} = options;
const tmpDir = path.join(os.tmpdir(), `voiceover_${Date.now()}`);
await fs.promises.mkdir(tmpDir, { recursive: true });
const outputPath = videoPath.replace(/(\\\w+)$/, '_voiceover$1');
if (typeof script === 'string') {
// Simple full-video voiceover
console.log('Generating voiceover audio...');
const voiceoverPath = path.join(tmpDir, 'voiceover.mp3');
await generateVoiceover(script, voiceoverPath, apiKey, voiceId);
console.log(`Mixing audio (mode: ${mode})...`);
switch (mode) {
case 'replace':
await replaceAudio(videoPath, voiceoverPath, outputPath);
break;
case 'mix':
await mixOverAudio(videoPath, voiceoverPath, outputPath, bgVolume);
break;
case 'add':
await addAudioTrack(videoPath, voiceoverPath, outputPath);
break;
}
} else {
// Timed segments
console.log(`Generating ${script.length} voiceover segments...`);
const segmentPaths = await generateSegments(script, tmpDir, apiKey, voiceId);
// Get video duration
const { stdout } = await execFileAsync('ffprobe', [
'-v', 'quiet', '-print_format', 'json', '-show_format', videoPath
]);
const duration = parseFloat(JSON.parse(stdout).format.duration);
console.log('Assembling timed voiceover...');
const assembledPath = path.join(tmpDir, 'assembled_voiceover.aac');
await assembleTimedVoiceover(script, segmentPaths, duration, assembledPath);
console.log(`Mixing audio (mode: ${mode})...`);
switch (mode) {
case 'replace':
await replaceAudio(videoPath, assembledPath, outputPath);
break;
case 'mix':
await mixOverAudio(videoPath, assembledPath, outputPath, bgVolume);
break;
case 'add':
await addAudioTrack(videoPath, assembledPath, outputPath);
break;
}
}
// Clean up
await fs.promises.rm(tmpDir, { recursive: true, force: true });
console.log(`Voiceover added: ${outputPath}`);
return outputPath;
}
Tips
- Character limits: ElevenLabs has a per-request character limit (typically 5,000 characters). For long narrations, split the text into chunks and concatenate the resulting audio files.
- Audio format: ElevenLabs returns MP3 by default. You can request other formats via the
output_formatparameter (e.g.,pcm_16000for raw PCM,mp3_44100_128for specific bitrate).
- Pacing: If the voiceover needs to match specific video timing, adjust the text length per segment rather than trying to speed up/slow down the generated audio. Shorter text = shorter audio.
- Silence padding: Use
apadoradelayin FFmpeg to position audio precisely. ElevenLabs doesn't add leading silence, so timing is exact.
- Cost awareness: ElevenLabs charges per character. Long narrations can add up. Let the user know the approximate character count before generating.
- Multilingual: The
eleven_multilingual_v2model supports 29 languages. The text language is auto-detected. No special configuration needed — just write the text in the target language.