
Video Use
- 2.8k installs
- 19.4k repo stars
- Updated July 1, 2026
- browser-use/video-use
video-use is an agent skill that Edit any video by conversation. Transcribe, cut, color grade, generate overlay animations, burn subtitles — for talking .
About
1 LLM reasons from raw transcript on demand visuals The only derived artifact that earns its keep is a packed phrase level transcript takes_packed md Everything else filler tagging retake detection shot classification emphasis scoring you derive at decision time 2 Audio is primary visuals follow Cut candidates come from speech boundaries and silence gaps Drill into visuals only at decision points 3 Ask confirm execute iterate persist Never touch the cut until the user has confirmed the strategy in plain English 4 Generalize Do not assume what kind of video this is Look at the material ask the user then edit 5 Artistic freedom is the default Every specific value preset font color duration pitch structure and technique in this document is a worked example from one proven video not a mandate Read them to understand what s possible and why each worked Then make your own taste calls based on what the material actually is and what the user actually wants The only things you MUST do are
- description: Edit any video by conversation. Transcribe, cut, color grade, generate overlay animations, burn subtitles —
- 1. **LLM reasons from raw transcript + on-demand visuals.** The only derived artifact that earns its keep is a packed ph
- 2. **Audio is primary, visuals follow.** Cut candidates come from speech boundaries and silence gaps. Drill into visuals
- Follow video-use SKILL.md steps and documented constraints.
- Follow video-use SKILL.md steps and documented constraints.
Video Use by the numbers
- 2,842 all-time installs (skills.sh)
- +237 installs in the week ending Aug 4, 2026 (Skillselion tracking)
- Ranked #276 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Security screen: HIGH risk (skills.sh audit)
- Data as of Aug 5, 2026 (Skillselion catalog sync)
video-use capabilities & compatibility
- Capabilities
- description: edit any video by conversation. tra · 1. **llm reasons from raw transcript + on demand · 2. **audio is primary, visuals follow.** cut can · follow video use skill.md steps and documented c
- Use cases
- orchestration
What video-use says it does
description: Edit any video by conversation. Transcribe, cut, color grade, generate overlay animations, burn subtitles — for talking heads, montages, tutorials, travel, interviews. No presets, no menu
1. **LLM reasons from raw transcript + on-demand visuals.** The only derived artifact that earns its keep is a packed phrase-level transcript (`takes_packed.md`). Everything else — filler tagging, ret
2. **Audio is primary, visuals follow.** Cut candidates come from speech boundaries and silence gaps. Drill into visuals only at decision points.
npx skills add https://github.com/browser-use/video-use --skill video-useAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 2.8k |
|---|---|
| repo stars | ★ 19.4k |
| Security audit | 2 / 3 scanners passed |
| Last updated | July 1, 2026 |
| Repository | browser-use/video-use ↗ |
When should an agent use video-use and what problem does it solve?
Edit any video by conversation. Transcribe, cut, color grade, generate overlay animations, burn subtitles — for talking heads, montages, tutorials, travel, interviews. No presets, no menus. Ask questi
Who is it for?
Developers invoking video-use as documented in the skill source.
Skip if: Skip when requirements fall outside video-use documented scope.
When should I use this skill?
Edit any video by conversation. Transcribe, cut, color grade, generate overlay animations, burn subtitles — for talking heads, montages, tutorials, travel, interviews. No presets, no menus. Ask questi
What you get
Outputs aligned with the video-use SKILL.md workflow and stated deliverables.
- takes_packed.md
- edited video renders
- burned subtitles
Files
Video Use
Principle
1. LLM reasons from raw transcript + on-demand visuals. The only derived artifact that earns its keep is a packed phrase-level transcript (takes_packed.md). Everything else — filler tagging, retake detection, shot classification, emphasis scoring — you derive at decision time. 2. Audio is primary, visuals follow. Cut candidates come from speech boundaries and silence gaps. Drill into visuals only at decision points. 3. Ask → confirm → execute → iterate → persist. Never touch the cut until the user has confirmed the strategy in plain English. 4. Generalize. Do not assume what kind of video this is. Look at the material, ask the user, then edit. 5. Artistic freedom is the default. Every specific value, preset, font, color, duration, pitch structure, and technique in this document is a worked example from one proven video — not a mandate. Read them to understand what's possible and why each worked. Then make your own taste calls based on what the material actually is and what the user actually wants. The only things you MUST do are in the Hard Rules section below. Everything else is yours. 6. Invent freely. If the material calls for a technique not described here — split-screen, picture-in-picture, lower-third identity cards, reaction cuts, speed ramps, freeze frames, crossfades, match cuts, L-cuts, J-cuts, speed ramps over breath, whatever — build it. The helpers are ffmpeg and PIL. They can do anything the format supports. Do not wait for permission. 7. Verify your own output before showing it to the user. If you wouldn't ship it, don't present it.
Hard Rules (production correctness — non-negotiable)
These are the things where deviation produces silent failures or broken output. They are not taste, they are correctness. Memorize them.
1. Subtitles are applied LAST in the filter chain, after every overlay. Otherwise overlays hide captions. Silent failure. 2. Per-segment extract → lossless `-c copy` concat, not single-pass filtergraph. Otherwise you double-encode every segment when overlays are added. 3. 30ms audio fades at every segment boundary (afade=t=in:st=0:d=0.03,afade=t=out:st={dur-0.03}:d=0.03). Otherwise audible pops at every cut. 4. Overlays use `setpts=PTS-STARTPTS+T/TB` to shift the overlay's frame 0 to its window start. Otherwise you see the middle of the animation during the overlay window. 5. Master SRT uses output-timeline offsets: output_time = word.start - segment_start + segment_offset. Otherwise captions misalign after segment concat. 6. Never cut inside a word. Snap every cut edge to a word boundary from the Scribe transcript. 7. Pad every cut edge. Working window: 30–200ms. Scribe timestamps drift 50–100ms — padding absorbs the drift. Tighter for fast-paced, looser for cinematic. 8. Word-level verbatim ASR only. Never SRT/phrase mode (loses sub-second gap data). Never normalized fillers (loses editorial signal). 9. Cache transcripts per source. Never re-transcribe unless the source file itself changed. 10. Parallel sub-agents for multiple animations. Never sequential. Spawn N at once via the Agent tool; total wall time ≈ slowest one. 11. Strategy confirmation before execution. Never touch the cut until the user has approved the plain-English plan. 12. All session outputs in `<videos_dir>/edit/`. Never write inside the video-use/ project directory.
Everything else in this document is a worked example. Deviate whenever the material calls for it.
Directory layout
The skill lives in video-use/. User footage lives wherever they put it. All session outputs go into <videos_dir>/edit/.
<videos_dir>/
├── <source files, untouched>
└── edit/
├── project.md ← memory; appended every session
├── takes_packed.md ← phrase-level transcripts, the LLM's primary reading view
├── edl.json ← cut decisions
├── transcripts/<name>.json ← cached raw Scribe JSON
├── animations/slot_<id>/ ← per-animation source + render + reasoning
├── clips_graded/ ← per-segment extracts with grade + fades
├── master.srt ← output-timeline subtitles
├── downloads/ ← yt-dlp outputs
├── verify/ ← debug frames / timeline PNGs
├── preview.mp4
└── final.mp4Setup
First-time install lives in install.md (clone, deps, ffmpeg, skill registration, API key). Don't re-run it every session; on cold start just verify:
ELEVENLABS_API_KEYresolves — either in the environment or in.envat the video-use repo root. If missing, ask the user to paste one and write it to.env(never to the user's<videos_dir>).ffmpeg+ffprobeon PATH.- Python deps installed (
uv syncorpip install -e .inside the repo). - Node.js + npm available if the session needs HyperFrames or Remotion slots. HyperFrames currently requires Node.js 22+.
yt-dlp, HyperFrames, Remotion, Manim installed only on first use.- First-use animation setup happens inside the slot directory, never at the video-use repo root. HyperFrames can be invoked with
npx --yes hyperframes ...; Remotion can be scaffolded withnpx create-video@latestor installed as a project-local dependency before using itsremotion rendercommand. - This skill vendors
skills/manim-video/. Read its SKILL.md when building a Manim slot.
Helpers (helpers/transcribe.py, helpers/render.py, etc.) live alongside this SKILL.md. Resolve their paths relative to the directory containing this file — the skill is typically symlinked at ~/.claude/skills/video-use/ or ~/.codex/skills/video-use/.
Helpers
- `transcribe.py <video>` — single-file Scribe call.
--num-speakers Noptional. Cached. - `transcribe_batch.py <videos_dir>` — 4-worker parallel transcription. Use for multi-take.
- `pack_transcripts.py --edit-dir <dir>` —
transcripts/*.json→takes_packed.md(phrase-level, break on silence ≥ 0.5s). - `timeline_view.py <video> <start> <end>` — filmstrip + waveform PNG. On-demand visual drill-down. Not a scan tool — use it at decision points, not constantly.
- `render.py <edl.json> -o <out>` — per-segment extract → concat → overlays (PTS-shifted) → subtitles LAST.
--previewfor 720p fast.--build-subtitlesto generate master.srt inline. - `grade.py <in> -o <out>` — ffmpeg filter chain grade. Presets +
--filter '<raw>'for custom.
For animations, create <edit>/animations/slot_<id>/ with Bash and spawn a sub-agent via the Agent tool.
The process
1. Inventory. ffprobe every source. transcribe_batch.py on the directory. pack_transcripts.py to produce takes_packed.md. Sample one or two timeline_views for a visual first impression. 2. Pre-scan for problems. One pass over takes_packed.md to note verbal slips, obvious mis-speaks, or phrasings to avoid. Plain list, feed into the editor brief. 3. Converse. Describe what you see in plain English. Ask questions shaped by the material. Collect: content type, target length/aspect, aesthetic/brand direction, pacing feel, must-preserve moments, must-cut moments, animation and grade preferences, subtitle needs. Do not use a fixed checklist — the right questions are different every time. 4. Propose strategy. 4–8 sentences: shape, take choices, cut direction, animation plan, grade direction, subtitle style, length estimate. Wait for confirmation. 5. Execute. Produce edl.json via the editor sub-agent brief. Drill into timeline_view at ambiguous moments. Build animations in parallel sub-agents. Apply grade per-segment. Compose via render.py. 6. Preview. render.py --preview. 7. Self-eval (before showing the user). Run timeline_view on the rendered output (not the sources) at every cut boundary (±1.5s window). Check each image for:
- Visual discontinuity / flash / jump at the cut
- Waveform spike at the boundary (audio pop that slipped past the 30ms fade)
- Subtitle hidden behind an overlay (Rule 1 violation)
- Overlay misaligned or showing wrong frames (Rule 4 violation)
Also sample: first 2s, last 2s, and 2–3 mid-points — check grade consistency, subtitle readability, overall coherence. Run ffprobe on the output to verify duration matches the EDL expectation.
If anything fails: fix → re-render → re-eval. Cap at 3 self-eval passes — if issues remain after 3, flag them to the user rather than looping forever. Only present the preview once the self-eval passes. 8. Iterate + persist. Natural-language feedback, re-plan, re-render. Never re-transcribe. Final render on confirmation. Append to project.md.
Cut craft (techniques)
- Audio-first. Candidate cuts from word boundaries and silence gaps.
- Preserve peaks. Laughs, punchlines, emphasis beats. Extend past punchlines to include reactions — the laugh IS the beat.
- Speaker handoffs benefit from air between utterances. Common values: 400–600ms. Less for fast-paced, more for cinematic. Taste call.
- Audio events as signals.
(laughs),(sighs),(applause)mark beats. Extend past them. - Silence gaps are cut candidates. Silences ≥400ms are usually the cleanest. 150–400ms phrase boundaries are usable with a visual check. <150ms is unsafe (mid-phrase).
- Example cut padding (the launch video shipped with this): 50ms before the first kept word, 80ms after the last. Tighter for montage energy, looser for documentary. Stay in the 30–200ms working window (Hard Rule 7).
- Never reason audio and video independently. Every cut must work on both tracks.
The packed transcript (primary reading view)
pack_transcripts.py reads all transcripts/*.json and produces one markdown file where each take is a list of phrase-level lines, each prefixed with its [start-end] time range. Phrases break on any silence ≥ 0.5s OR speaker change. This is the artifact the editor sub-agent reads to pick cuts — it gives word-boundary precision from text alone at 1/10 the tokens of raw JSON.
Example line:
## C0103 (duration: 43.0s, 8 phrases)
[002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
[006.08-006.74] S0 We fixed this.Editor sub-agent brief (for multi-take selection)
When the task is "pick the best take of each beat across many clips," spawn a dedicated sub-agent with a brief shaped like this. The structure is load-bearing; the pitch-shape example is not.
You are editing a <type> video. Pick the best take of each beat and
assemble them chronologically by beat, not by source clip order.
INPUTS:
- takes_packed.md (time-annotated phrase-level transcripts of all takes)
- Product/narrative context: <2 sentences from the user>
- Speaker(s): <name, role, delivery style note>
- Expected structure: <pick an archetype or invent one>
- Verbal slips to avoid: <list from the pre-scan pass>
- Target runtime: <seconds>
Common structural archetypes (pick, adapt, or invent):
- Tech launch / demo: HOOK → PROBLEM → SOLUTION → BENEFIT → EXAMPLE → CTA
- Tutorial: INTRO → SETUP → STEPS → GOTCHAS → RECAP
- Interview: (QUESTION → ANSWER → FOLLOWUP) repeat
- Travel / event: ARRIVAL → HIGHLIGHTS → QUIET MOMENTS → DEPARTURE
- Documentary: THESIS → EVIDENCE → COUNTERPOINT → CONCLUSION
- Music / performance: INTRO → VERSE → CHORUS → BRIDGE → OUTRO
- Or invent your own.
RULES:
- Start/end times must fall on word boundaries from the transcript.
- Pad cut boundaries (working window 30–200ms).
- Prefer silences ≥ 400ms as cut targets.
- Unavoidable slips are kept if no better take exists. Note them in "reason".
- If over budget, revise: drop a beat or trim tails. Report total and self-correct.
OUTPUT (JSON array, no prose):
[{"source": "C0103", "start": 2.42, "end": 6.85, "beat": "HOOK",
"quote": "...", "reason": "..."}, ...]
Return the final EDL and a one-line total runtime check.Color grade (when requested)
Your job is to reason about the image, not apply a preset. Look at a frame (via timeline_view), decide what's wrong, adjust one thing, look again.
Mental model is ASC CDL. Per channel: out = (in * slope + offset) ** power, then global saturation. slope → highlights, offset → shadows, power → midtones.
Example filter chains (grade.py has --list-presets; use them as starting points or mix your own):
- `warm_cinematic` — retro/technical, subtle teal/orange split, desaturated. Shipped in a real launch video. Safe for talking heads.
- `neutral_punch` — minimal corrective: contrast bump + gentle S-curve. No hue shifts.
- `none` — straight copy. Default when the user hasn't asked.
For anything else — portraiture, nature, product, music video, documentary — invent your own chain. grade.py --filter '<raw ffmpeg>' accepts any filter string.
Hard rules: apply per-segment during extraction (not post-concat, which re-encodes twice). Never go aggressive without testing skin tones.
Subtitles (when requested)
Subtitles have three dimensions worth reasoning about: chunking (1/2/3/sentence per line), case (UPPER/Title/Natural), and placement (margin from bottom). The right combo depends on content.
Worked styles — pick, adapt, or invent:
`bold-overlay` — short-form tech launch, fast-paced social. 2-word chunks, UPPERCASE, break on punctuation, Helvetica 18 Bold, white-on-outline, MarginV=35. render.py ships with this as SUB_FORCE_STYLE.
FontName=Helvetica,FontSize=18,Bold=1,
PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,BackColour=&H00000000,
BorderStyle=1,Outline=2,Shadow=0,
Alignment=2,MarginV=35`natural-sentence` (if you invent this mode) — narrative, documentary, education. 4–7 word chunks, sentence case, break on natural pauses, MarginV=60–80, larger font for readability, slightly wider max-width. No shipped force_style — design one if you need it.
Invent a third style if neither fits. Hard rules: subtitles LAST (Rule 1), output-timeline offsets (Rule 5).
Animations (when requested)
Animations match the content and the brand. Get the palette, font, and visual language from the conversation — never assume a default. If the user hasn't told you, propose a palette in the strategy phase and wait for confirmation before building anything.
Tool options:
Pick the engine per animation slot. Do not default to Remotion just because the animation is web-adjacent.
- HyperFrames — Browser-native HTML/CSS/GSAP video compositions: product UI motion, website-to-video or mockup-to-video captures, kinetic typography, landing-page/storyboard promos, data-driven UI states, transparent WebM overlays, and clips that need deterministic frame capture plus HyperFrames lint/validate/render checks. Best when the animation should be authored and verified like a web composition instead of a React component tree.
- Remotion — React/CSS compositions with component state, reusable React primitives, or an existing Remotion brand system. Best when the user specifically asks for React/Remotion or when React composition is the simpler authoring model.
- Manim — formal diagrams, state machines, equation derivations, graph morphs. Read
skills/manim-video/SKILL.mdand its references for depth. - PIL + PNG sequence + ffmpeg — simple overlay cards: counters, typewriter text, single bar reveals, progressive draws. Fast to iterate, any aesthetic you want. The launch video used this.
For HyperFrames slots, scaffold the slot inside edit/animations/slot_<id>/ with npx --yes hyperframes init . --example blank --non-interactive --skip-skills, build the HTML composition there, run the HyperFrames checks that fit the slot (lint, validate, and a draft render when practical), then produce the final overlay video with npx --yes hyperframes render . -o render.mp4 or --format webm -o render.webm when alpha is required. Point the EDL overlay file at the actual rendered path.
For Remotion slots, keep the Remotion project isolated inside the same slot directory, scaffold with npx create-video@latest or install Remotion locally there, render the composition to render.mp4 with the project-local remotion render command, and verify duration and dimensions with ffprobe.
None is mandatory. Invent hybrids if useful (e.g., PIL background with a HyperFrames or Remotion layer on top).
Duration rules of thumb, context-dependent:
- Sync-to-narration explanations. A viewer needs to parse the content at 1×. Rough floor 3s, typical 5–7s for simple cards, 8–14s for complex diagrams. The launch video shipped at 5–7s per simple card.
- Beat-synced accents (music video, fast montage). 0.5–2s is fine — they're visual accents, not information. The "readable at 1×" rule becomes "recognizable at 1×", not "fully parseable."
- Hold the final frame ≥ 1s before the cut (universal).
- Over voiceover: total duration ≥
narration_length + 1s(universal). - Never parallel-reveal independent elements — the eye can't track two new things at once. One thing, pause, next thing.
Animation payoff timing (rule for sync-to-narration): get the payoff word's timestamp. Start the overlay reveal_duration seconds earlier so the landing frame coincides with the spoken payoff word. Without this sync the animation feels disconnected.
Easing (universal — never linear, it looks robotic):
def ease_out_cubic(t): return 1 - (1 - t) ** 3
def ease_in_out_cubic(t):
if t < 0.5: return 4 * t ** 3
return 1 - (-2 * t + 2) ** 3 / 2ease_out_cubic for single reveals (slow landing). ease_in_out_cubic for continuous draws.
Typing text anchor trick: center on the FULL string's width, not the partial-string width — otherwise text slides left during reveal.
Example palette (the launch video — one aesthetic among infinite):
- Background
(10, 10, 10)near-black - Accent
#FF5A00/(255, 90, 0)orange - Labels
(110, 110, 110)dim gray - Font: Menlo Bold at
/System/Library/Fonts/Menlo.ttc(index 1) - ≤ 2 accent colors, ~40% empty space, minimal chrome
- Result: terminal / retro tech feel
This is one style. If the brand is warm and serif, use that. If it's colorful and playful, use that. If the user handed you a style guide, follow it. If they didn't, propose one and confirm.
Parallel sub-agent brief — each animation is one sub-agent spawned via the Agent tool. Each prompt is self-contained (sub-agents have no parent context). Include:
1. One-sentence goal: "Build ONE animation: [spec]. Nothing else." 2. Absolute output path (<edit>/animations/slot_<id>/render.mp4) 3. Exact technical spec: resolution, fps, codec, pix_fmt, CRF, duration 4. Style palette as concrete values (RGB tuples, hex, or reference to a design system) 5. Font path with index 6. Frame-by-frame timeline (what happens when, with easing) 7. Anti-list ("no chrome, no extras, no titles unless specified") 8. Code pattern reference (copy helpers inline, don't import across slots) 9. Deliverable checklist (script, render, verify duration via ffprobe, report) 10. "Do not ask questions. If anything is ambiguous, pick the most obvious interpretation and proceed."
One sub-agent = one file (unique filenames, parallel agents don't overwrite each other).
Output spec
Match the source unless the user asked for something specific. Common targets: 1920×1080@24 cinematic, 1920×1080@30 screen content, 1080×1920@30 vertical social, 3840×2160@24 4K cinema, 1080×1080@30 square. render.py defaults the scale to 1080p from any source; pass --filter or edit the extract command for other targets. Worth asking the user which delivery format matters.
EDL format
{
"version": 1,
"sources": {"C0103": "/abs/path/C0103.MP4", "C0108": "/abs/path/C0108.MP4"},
"ranges": [
{"source": "C0103", "start": 2.42, "end": 6.85,
"beat": "HOOK", "quote": "...", "reason": "Cleanest delivery, stops before slip at 38.46."},
{"source": "C0108", "start": 14.30, "end": 28.90,
"beat": "SOLUTION", "quote": "...", "reason": "Only take without the false start."}
],
"grade": "warm_cinematic",
"overlays": [
{"file": "edit/animations/slot_1/render.mp4", "start_in_output": 0.0, "duration": 5.0}
],
"subtitles": "edit/master.srt",
"total_duration_s": 87.4
}grade is a preset name or raw ffmpeg filter. overlays are rendered animation clips. subtitles is optional and applied LAST.
Memory — project.md
Append one section per session at <edit>/project.md:
## Session N — YYYY-MM-DD
**Strategy:** one paragraph describing the approach
**Decisions:** take choices, cuts, grades, animations + why
**Reasoning log:** one-line rationale for non-obvious decisions
**Outstanding:** deferred itemsOn startup, read project.md if it exists and summarize the last session in one sentence before asking whether to continue.
Anti-patterns
Things that consistently fail regardless of style:
- Hierarchical pre-computed codec formats with USABILITY / tone tags / shot layers. Over-engineering. Derive from the transcript at decision time.
- Hand-tuned moment-scoring functions. The LLM picks better than any heuristic you'll write.
- Whisper SRT / phrase-level output. Loses sub-second gap data. Always word-level verbatim.
- Running Whisper locally on CPU. Slow and it normalizes fillers. Use hosted Scribe.
- Burning subtitles into base before compositing overlays. Overlays hide them. (Hard Rule 1.)
- Single-pass filtergraph when you have overlays. Double re-encodes. Use per-segment extract → concat.
- Linear animation easing. Looks robotic. Always cubic.
- Hard audio cuts at segment boundaries. Audible pops. (Hard Rule 3.)
- Typing text centered on the partial string. Text slides left as it grows.
- Sequential sub-agents for multiple animations. Always parallel.
- Editing before confirming the strategy. Never.
- Re-transcribing cached sources. Immutable outputs of immutable inputs.
- Assuming what kind of video it is. Look first, ask second, edit last.
ELEVENLABS_API_KEY=
# Secrets
.env
.env.local
*.pem
*.key
# Python
__pycache__/
*.py[cod]
*$py.class
*.egg
*.egg-info/
.eggs/
build/
dist/
.pytest_cache/
.mypy_cache/
.ruff_cache/
# Virtual environments
.venv/
venv/
env/
ENV/
# Editors / IDEs
.vscode/
.idea/
*.swp
*.swo
.DS_Store
Thumbs.db
# Node / Remotion (installed on demand, not vendored)
node_modules/
package-lock.json
yarn.lock
.npm/
# Manim render artifacts (when running skills/manim-video scripts locally)
media/
*.log
# Session artifacts if someone invokes the skill inside the project dir
# (normally outputs go to <videos_dir>/edit/ — keep these as a safety net)
edit/
transcripts/
animations/
clips_graded/
clips_preview/
downloads/
verify/
preview.mp4
final.mp4
base.mp4
base_preview.mp4
master.srt
takes_packed.md
edl.json
project.md
"""Apply a color grade to a video via ffmpeg filter chain.
Two modes:
1. Preset mode — pick a named preset (e.g. `warm_cinematic`, `neutral_punch`).
Simple fixed filter chain applied uniformly.
2. Auto mode (DEFAULT) — analyze the clip mathematically and emit a subtle
per-clip correction. Samples N frames via ffmpeg, computes mean brightness,
RMS contrast, saturation. Emits a bounded filter string that corrects
under-exposure, flatness, and mild desaturation without applying any
creative color shift. All adjustments capped at ±8% on any axis.
The goal is "make it look clean without looking graded". Never applies
creative LUTs, teal/orange splits, or filmic curves. For creative looks,
use `--preset warm_cinematic` explicitly.
Usage:
python helpers/grade.py <input> -o <output> # auto mode
python helpers/grade.py <input> -o <output> --preset warm_cinematic
python helpers/grade.py <input> -o <output> --filter 'eq=contrast=1.1'
python helpers/grade.py --print-preset warm_cinematic # print filter only
python helpers/grade.py --analyze <input> # print auto-grade analysis
Can also be imported by render.py: `get_preset(name)` and `auto_grade_for_clip(path, edl_range)`.
"""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import tempfile
from pathlib import Path
PRESETS: dict[str, str] = {
# Subtle baseline — barely perceptible cleanup. No color shift.
# Use when auto-analysis isn't available or when you want a safe floor.
"subtle": "eq=contrast=1.03:saturation=0.98",
# Minimal corrective grade: light contrast + subtle S-curve, no color shifts.
"neutral_punch": (
"eq=contrast=1.06:brightness=0.0:saturation=1.0,"
"curves=master='0/0 0.25/0.23 0.75/0.77 1/1'"
),
# OPT-IN creative preset for retro/cinematic looks ONLY. Not a default.
# +12% contrast, crushed blacks, -12% sat, warm shadows + cool highs, filmic curve.
# Originally from HEURISTICS §6 — too aggressive for standard launch content.
"warm_cinematic": (
"eq=contrast=1.12:brightness=-0.02:saturation=0.88,"
"colorbalance="
"rs=0.02:gs=0.0:bs=-0.03:"
"rm=0.04:gm=0.01:bm=-0.02:"
"rh=0.08:gh=0.02:bh=-0.05,"
"curves=master='0/0 0.25/0.22 0.75/0.78 1/1'"
),
# Flat — no grade. Useful as a sentinel for "skip grading this source".
"none": "",
}
def get_preset(name: str) -> str:
"""Return the ffmpeg filter string for a preset name. Empty string for 'none'."""
if name not in PRESETS:
raise KeyError(
f"unknown preset '{name}'. Available: {', '.join(sorted(PRESETS))}"
)
return PRESETS[name]
# -------- Auto grade (data-driven, per-clip) --------------------------------
def _sample_frame_stats(
video: Path,
start: float,
duration: float,
n_samples: int = 10,
) -> dict[str, float]:
"""Sample N frames from a range and compute brightness/contrast/saturation stats.
Uses ffmpeg's `signalstats` filter which gives us YMIN, YMAX, YAVG, SATAVG
etc. in the metadata. We average across the sample range.
Returns:
{
"y_mean": mean Y (luma) in 0..1,
"y_std": approximate stddev of Y across samples (0..1),
"sat_mean": mean saturation in 0..1,
}
"""
# Use signalstats + metadata=print to get per-frame stats
# Sample fps = n_samples / duration, clamped so we don't over-sample short clips
fps = max(0.5, min(n_samples / max(duration, 0.1), 10.0))
with tempfile.NamedTemporaryFile(mode="w+", suffix=".txt", delete=False) as f:
metadata_path = f.name
try:
cmd = [
"ffmpeg", "-y", "-hide_banner", "-nostats",
"-ss", f"{start:.3f}",
"-i", str(video),
"-t", f"{duration:.3f}",
"-vf", f"fps={fps:.2f},signalstats,metadata=print:file={metadata_path}",
"-f", "null", "-",
]
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
# Parse signalstats metadata. Signalstats reports values in the NATIVE
# bit depth of the decoded frame (8-bit → 0-255, 10-bit → 0-1023). We
# read YBITDEPTH and normalize by (2^depth - 1) so downstream math is
# in 0..1 regardless of source bit depth.
y_avgs: list[float] = []
y_mins: list[float] = []
y_maxs: list[float] = []
sat_avgs: list[float] = []
bit_depth: int = 8
def _parse_value(line: str) -> float | None:
try:
return float(line.rsplit("=", 1)[1])
except (ValueError, IndexError):
return None
with open(metadata_path) as f:
for line in f:
line = line.strip()
if "lavfi.signalstats.YBITDEPTH" in line:
v = _parse_value(line)
if v is not None:
bit_depth = int(v)
elif "lavfi.signalstats.YAVG" in line:
v = _parse_value(line)
if v is not None:
y_avgs.append(v)
elif "lavfi.signalstats.YMIN" in line:
v = _parse_value(line)
if v is not None:
y_mins.append(v)
elif "lavfi.signalstats.YMAX" in line:
v = _parse_value(line)
if v is not None:
y_maxs.append(v)
elif "lavfi.signalstats.SATAVG" in line:
v = _parse_value(line)
if v is not None:
sat_avgs.append(v)
if not y_avgs:
# Analysis failed — return neutral defaults (no correction)
return {"y_mean": 0.5, "y_std": 0.18, "sat_mean": 0.25}
# Normalize by native bit-depth max value
max_val = (2 ** bit_depth) - 1
y_mean = (sum(y_avgs) / len(y_avgs)) / max_val
y_range = (
((sum(y_maxs) / len(y_maxs)) - (sum(y_mins) / len(y_mins))) / max_val
if y_maxs and y_mins
else 0.7
)
sat_mean = ((sum(sat_avgs) / len(sat_avgs)) / max_val) if sat_avgs else 0.25
return {
"y_mean": y_mean,
"y_std": y_range / 4.0, # range ÷ 4 ≈ stddev for normal-ish distributions
"sat_mean": sat_mean,
}
finally:
Path(metadata_path).unlink(missing_ok=True)
def auto_grade_for_clip(
video: Path,
start: float = 0.0,
duration: float | None = None,
verbose: bool = False,
) -> tuple[str, dict[str, float]]:
"""Analyze a clip range and emit a subtle per-clip correction filter.
Returns (filter_string, stats_dict). The filter is bounded to ±8% on any axis
and applies NO color shift. It only addresses:
- Underexposure (lift gamma slightly if too dark)
- Flatness (tiny contrast boost if range is narrow)
- Desaturation (tiny sat boost if extremely flat)
If the clip is already well-balanced, returns the baseline `subtle` preset.
"""
if duration is None:
# Probe duration
probe_cmd = [
"ffprobe", "-v", "error",
"-show_entries", "format=duration",
"-of", "default=noprint_wrappers=1:nokey=1",
str(video),
]
try:
duration = float(subprocess.check_output(probe_cmd).decode().strip())
except Exception:
duration = 10.0
stats = _sample_frame_stats(video, start, duration)
y_mean = stats["y_mean"]
y_range = stats["y_std"] * 4.0 # back to range
sat_mean = stats["sat_mean"]
# ------ Decision rules ---------------------------------------------------
# All caps bounded to ±8%. Target "clean, not graded".
# Contrast: target y_range ≈ 0.72. Boost gently if flat, never reduce.
contrast_adj = 1.0
if y_range < 0.65:
# Map [0.50, 0.65] → [1.08, 1.03]
t = max(0.0, min(1.0, (y_range - 0.50) / 0.15))
contrast_adj = 1.08 - 0.05 * t
else:
contrast_adj = 1.03 # subtle baseline
# Gamma: target y_mean ≈ 0.48. Lift gently if too dark.
gamma_adj = 1.0
if y_mean < 0.42:
# Map [0.30, 0.42] → [1.10, 1.02]
t = max(0.0, min(1.0, (y_mean - 0.30) / 0.12))
gamma_adj = 1.10 - 0.08 * t
elif y_mean > 0.60:
# Slightly overexposed — tiny pullback
gamma_adj = 0.97
# Saturation: target sat_mean ≈ 0.25. Never desaturate aggressively;
# modest boost if very flat. Default to 0.98 (tiny pullback — most digital
# video is slightly over-saturated on consumer displays).
sat_adj = 0.98
if sat_mean < 0.18:
# Very flat — tiny boost
sat_adj = 1.04
elif sat_mean > 0.38:
# Already punchy — hold
sat_adj = 0.96
# Clamp all adjustments hard
contrast_adj = max(0.94, min(1.08, contrast_adj))
gamma_adj = max(0.94, min(1.10, gamma_adj))
sat_adj = max(0.94, min(1.06, sat_adj))
# Build filter string
eq_parts = []
if abs(contrast_adj - 1.0) > 0.005:
eq_parts.append(f"contrast={contrast_adj:.3f}")
if abs(gamma_adj - 1.0) > 0.005:
eq_parts.append(f"gamma={gamma_adj:.3f}")
if abs(sat_adj - 1.0) > 0.005:
eq_parts.append(f"saturation={sat_adj:.3f}")
if not eq_parts:
filter_string = ""
else:
filter_string = "eq=" + ":".join(eq_parts)
if verbose:
print(f" auto-grade stats:")
print(f" y_mean={y_mean:.3f} y_range={y_range:.3f} sat_mean={sat_mean:.3f}")
print(f" → contrast={contrast_adj:.3f} gamma={gamma_adj:.3f} sat={sat_adj:.3f}")
print(f" → filter: {filter_string or '(empty)'}")
return filter_string, stats
def apply_grade(input_path: Path, output_path: Path, filter_string: str) -> None:
output_path.parent.mkdir(parents=True, exist_ok=True)
if not filter_string:
cmd = [
"ffmpeg", "-y", "-i", str(input_path),
"-c", "copy", str(output_path),
]
else:
cmd = [
"ffmpeg", "-y", "-i", str(input_path),
"-vf", filter_string,
"-c:v", "libx264", "-preset", "fast", "-crf", "18",
"-pix_fmt", "yuv420p",
"-c:a", "copy",
"-movflags", "+faststart",
str(output_path),
]
subprocess.run(cmd, check=True)
def main() -> None:
ap = argparse.ArgumentParser(description="Apply a color grade via ffmpeg filter chain")
ap.add_argument("input", type=Path, nargs="?", help="Input video")
ap.add_argument("-o", "--output", type=Path, help="Output video")
ap.add_argument(
"--preset",
type=str,
default=None,
choices=list(PRESETS.keys()),
help="Grade preset. Omit for auto mode (default).",
)
ap.add_argument(
"--filter",
type=str,
default=None,
help="Raw ffmpeg filter string. Overrides --preset.",
)
ap.add_argument(
"--analyze",
type=Path,
default=None,
help="Analyze a clip and print the auto-grade filter it would produce. No output written.",
)
ap.add_argument(
"--print-preset",
type=str,
default=None,
help="Print the filter string for a preset and exit. No input/output needed.",
)
ap.add_argument(
"--list-presets",
action="store_true",
help="List available presets and exit.",
)
args = ap.parse_args()
if args.list_presets:
for name, f in PRESETS.items():
print(f"{name}:")
print(f" {f}" if f else " (no filter)")
print()
return
if args.print_preset is not None:
print(get_preset(args.print_preset))
return
if args.analyze is not None:
if not args.analyze.exists():
sys.exit(f"input not found: {args.analyze}")
filter_string, stats = auto_grade_for_clip(args.analyze, verbose=True)
print(f"\nfilter: {filter_string or '(none)'}")
print(f"stats: {json.dumps(stats, indent=2)}")
return
if not args.input or not args.output:
ap.error("input and -o/--output are required unless using --analyze/--print-preset/--list-presets")
if not args.input.exists():
sys.exit(f"input not found: {args.input}")
# Decide filter string
if args.filter is not None:
filter_string = args.filter
elif args.preset is not None:
filter_string = get_preset(args.preset)
else:
# Auto mode (default)
filter_string, _ = auto_grade_for_clip(args.input, verbose=True)
print(f"grading {args.input.name} → {args.output.name}")
if filter_string:
print(f" filter: {filter_string[:120]}{'...' if len(filter_string) > 120 else ''}")
else:
print(" filter: (none — copy)")
apply_grade(args.input, args.output, filter_string)
print(f"done: {args.output}")
if __name__ == "__main__":
main()
"""Pack all Scribe transcripts in <edit>/transcripts/ into one readable markdown.
Groups word-level entries into phrase-level lines, breaking on any silence
>= 0.5s OR speaker change. Each phrase gets a [start-end] prefix. This is
the PRIMARY artifact the editor sub-agent reads to pick cuts — it fits one
hour of takes in a tenth the tokens of raw Scribe JSON and gives
word-boundary precision from text alone.
Output: <edit>/takes_packed.md
Usage:
python helpers/pack_transcripts.py --edit-dir <edit_dir>
python helpers/pack_transcripts.py --edit-dir <edit_dir> --silence-threshold 0.5
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
def format_time(seconds: float) -> str:
"""Format a time in seconds as "NNN.NN" with fixed 6-char width for alignment."""
return f"{seconds:06.2f}"
def format_duration(seconds: float) -> str:
"""Format a duration as "Ms" or "Mm SSs"."""
if seconds < 60:
return f"{seconds:.1f}s"
m = int(seconds // 60)
s = seconds - m * 60
return f"{m}m {s:04.1f}s"
def group_into_phrases(
words: list[dict],
silence_threshold: float = 0.5,
) -> list[dict]:
"""Walk a Scribe word list, break into phrases on silence >= threshold
OR speaker change. Returns list of {start, end, text, speaker_id}.
Scribe `words` entries have types 'word', 'spacing', or 'audio_event'.
We keep 'word' and 'audio_event' content in phrase text. 'spacing'
entries carry the silence information via their start/end times.
"""
phrases: list[dict] = []
current_words: list[dict] = []
current_start: float | None = None
current_speaker: str | None = None
def flush() -> None:
nonlocal current_words, current_start, current_speaker
if not current_words:
return
text_parts: list[str] = []
for w in current_words:
t = w.get("type", "word")
raw = (w.get("text") or "").strip()
if not raw:
continue
if t == "audio_event":
if not raw.startswith("("):
raw = f"({raw})"
text_parts.append(raw)
if not text_parts:
current_words = []
current_start = None
current_speaker = None
return
text = " ".join(text_parts)
text = text.replace(" ,", ",").replace(" .", ".").replace(" ?", "?").replace(" !", "!")
end_time = current_words[-1].get("end", current_words[-1].get("start", current_start or 0.0))
phrases.append({
"start": current_start,
"end": end_time,
"text": text,
"speaker_id": current_speaker,
})
current_words = []
current_start = None
current_speaker = None
prev_end: float | None = None
for w in words:
t = w.get("type", "word")
if t == "spacing":
# spacing entries mark the gaps between words; if the gap is long,
# flush the current phrase.
start = w.get("start")
end = w.get("end")
if start is not None and end is not None:
gap = end - start
if gap >= silence_threshold:
flush()
continue
# 'word' or 'audio_event'
start = w.get("start")
if start is None:
continue
speaker = w.get("speaker_id")
# Flush on speaker change
if current_speaker is not None and speaker is not None and speaker != current_speaker:
flush()
# Flush on a long gap from the previous kept token
if prev_end is not None and start - prev_end >= silence_threshold:
flush()
if current_start is None:
current_start = start
current_speaker = speaker
current_words.append(w)
prev_end = w.get("end", start)
flush()
return phrases
def pack_one_file(json_path: Path, silence_threshold: float) -> tuple[str, float, list[dict]]:
"""Return (header_name, duration, phrases) for one transcript file."""
data = json.loads(json_path.read_text())
words = data.get("words", [])
phrases = group_into_phrases(words, silence_threshold)
if phrases:
duration = phrases[-1]["end"] - phrases[0]["start"]
else:
duration = 0.0
return json_path.stem, duration, phrases
def render_markdown(entries: list[tuple[str, float, list[dict]]], silence_threshold: float) -> str:
lines: list[str] = []
lines.append("# Packed transcripts")
lines.append("")
lines.append(f"Phrase-level, grouped on silences ≥ {silence_threshold:.1f}s or speaker change.")
lines.append("Use `[start-end]` ranges to address cuts in the EDL.")
lines.append("")
for name, duration, phrases in entries:
lines.append(f"## {name} (duration: {format_duration(duration)}, {len(phrases)} phrases)")
if not phrases:
lines.append(" _no speech detected_")
lines.append("")
continue
for p in phrases:
spk = p.get("speaker_id")
if spk is not None:
# Scribe returns IDs like "speaker_0" — strip the prefix for readability
spk_str = str(spk)
if spk_str.startswith("speaker_"):
spk_str = spk_str[len("speaker_"):]
spk_tag = f" S{spk_str}"
else:
spk_tag = ""
lines.append(f" [{format_time(p['start'])}-{format_time(p['end'])}]{spk_tag} {p['text']}")
lines.append("")
return "\n".join(lines)
def main() -> None:
ap = argparse.ArgumentParser(description="Pack Scribe transcripts into takes_packed.md")
ap.add_argument("--edit-dir", type=Path, required=True, help="Edit directory containing transcripts/")
ap.add_argument(
"--silence-threshold",
type=float,
default=0.5,
help="Break phrases on silences >= this (seconds). Default 0.5.",
)
ap.add_argument(
"-o", "--output",
type=Path,
default=None,
help="Output path (default: <edit-dir>/takes_packed.md)",
)
args = ap.parse_args()
edit_dir = args.edit_dir.resolve()
transcripts_dir = edit_dir / "transcripts"
if not transcripts_dir.is_dir():
sys.exit(f"no transcripts directory at {transcripts_dir}")
json_files = sorted(transcripts_dir.glob("*.json"))
if not json_files:
sys.exit(f"no .json files in {transcripts_dir}")
entries = [pack_one_file(p, args.silence_threshold) for p in json_files]
markdown = render_markdown(entries, args.silence_threshold)
out_path = args.output or (edit_dir / "takes_packed.md")
out_path.write_text(markdown, encoding="utf-8")
total_phrases = sum(len(e[2]) for e in entries)
total_duration = sum(e[1] for e in entries)
kb = out_path.stat().st_size / 1024
print(f"packed {len(entries)} transcripts → {out_path}")
print(f" {total_phrases} phrases, {format_duration(total_duration)} total runtime")
print(f" {kb:.1f} KB")
if __name__ == "__main__":
main()
"""Render a video from an EDL.
Implements the HEURISTICS render pipeline in the correct order:
1. Per-segment extract with color grade + 30ms audio fades baked in
2. Lossless -c copy concat into base.mp4
3. If overlays or subtitles: single filter graph that overlays animations
(with PTS shift so frame 0 lands at the overlay window start)
and applies `subtitles` filter LAST → final.mp4
Optionally builds a master SRT from the per-source transcripts + EDL
output-timeline offsets, applies the proven force_style (2-word
UPPERCASE chunks, Helvetica 18 Bold, MarginV=35).
Usage:
python helpers/render.py <edl.json> -o final.mp4
python helpers/render.py <edl.json> -o preview.mp4 --preview
python helpers/render.py <edl.json> -o final.mp4 --build-subtitles
python helpers/render.py <edl.json> -o final.mp4 --no-subtitles
"""
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
from pathlib import Path
try:
from grade import get_preset, auto_grade_for_clip # same directory
except Exception:
def get_preset(name: str) -> str:
return ""
def auto_grade_for_clip(video, start=0.0, duration=None, verbose=False): # type: ignore
return "eq=contrast=1.03:saturation=0.98", {}
# -------- Subtitle style (bold-overlay, proven at 1920×1080 and 1080×1920) --
#
# MarginV is NOT taste — it is a platform safe-zone rule.
# TikTok / IG Reels / Shorts UI (caption, username, music, right-rail actions)
# covers roughly the bottom ~25–30% of a 1080×1920 frame. Captions placed near
# the bottom edge get clipped or obscured by the UI. libass auto-scales the
# render canvas relative to PlayResY=288, so MarginV=90 lands the caption
# baseline roughly 30% up from the bottom on any aspect — clear of the UI on
# every major vertical-video platform. Do not drop this below ~75 without a
# specific reason.
SUB_FORCE_STYLE = (
"FontName=Helvetica,FontSize=18,Bold=1,"
"PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,BackColour=&H00000000,"
"BorderStyle=1,Outline=2,Shadow=0,"
"Alignment=2,MarginV=90"
)
# -------- Helpers ------------------------------------------------------------
def run(cmd: list[str], quiet: bool = False) -> None:
if not quiet:
print(f" $ {' '.join(str(c) for c in cmd[:6])}{' …' if len(cmd) > 6 else ''}")
subprocess.run(cmd, check=True)
def resolve_grade_filter(grade_field: str | None) -> str:
"""The EDL's 'grade' field can be a preset name, a raw ffmpeg filter, or 'auto'.
Returns the filter string to embed into the per-segment -vf chain.
For 'auto', returns the sentinel "__AUTO__" which is resolved per-segment.
"""
if not grade_field:
return ""
if grade_field == "auto":
return "__AUTO__"
# Preset names are short identifiers, filter strings contain '=' or ','.
if re.fullmatch(r"[a-zA-Z0-9_\-]+", grade_field):
try:
return get_preset(grade_field)
except KeyError:
print(f"warning: unknown preset '{grade_field}', using as raw filter")
return grade_field
return grade_field
def resolve_path(maybe_path: str, base: Path) -> Path:
"""Resolve a path that may be absolute or relative to `base`."""
p = Path(maybe_path)
if p.is_absolute():
return p
return (base / p).resolve()
# -------- HDR → SDR tone mapping (HLG / PQ sources) --------------------------
#
# iPhone defaults to HLG HDR in Rec.2020 (and many mirrorless cameras ship PQ).
# If the source is HDR and we only downconvert bit depth (yuv420p10le → yuv420p)
# without tone-mapping, the output is 8-bit but still carries HLG/PQ transfer
# metadata. Players that honor the metadata (screen recorders, most social
# upload re-encodes) interpret 8-bit values in an HDR container and the result
# looks oversaturated / blown out. QuickTime on macOS can hide this locally —
# screen recording and uploaded renders cannot.
#
# Fix: detect HDR via color_transfer and prepend a zscale+tonemap chain to the
# vf graph so the output is clean Rec.709 SDR.
HDR_TRANSFERS = {"smpte2084", "arib-std-b67"} # PQ (HDR10) and HLG
TONEMAP_CHAIN = (
"zscale=t=linear:npl=100,"
"format=gbrpf32le,"
"zscale=p=bt709,"
"tonemap=tonemap=hable:desat=0,"
"zscale=t=bt709:m=bt709:r=tv,"
"format=yuv420p"
)
def is_hdr_source(video: Path) -> bool:
"""Return True if the source uses a PQ or HLG transfer function."""
try:
out = subprocess.run(
["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_entries", "stream=color_transfer",
"-of", "default=noprint_wrappers=1:nokey=1", str(video)],
capture_output=True, text=True, check=True,
)
return out.stdout.strip() in HDR_TRANSFERS
except subprocess.CalledProcessError:
return False
def is_portrait_source(video: Path) -> bool:
"""Return True if the video's height > width (portrait / vertical)."""
try:
out = subprocess.run(
["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_entries", "stream=width,height",
"-of", "csv=p=0", str(video)],
capture_output=True, text=True, check=True,
)
w, h = map(int, out.stdout.strip().split(","))
return h > w
except Exception:
return False
# -------- Per-segment extraction (Rule 2 + Rule 3) --------------------------
def extract_segment(
source: Path,
seg_start: float,
duration: float,
grade_filter: str,
out_path: Path,
preview: bool = False,
draft: bool = False,
) -> None:
"""Extract a cut range as its own MP4 with grade + 30ms audio fades baked in.
`-ss` before `-i` for fast accurate seeking. Scale to 1080p from 4K.
Portrait sources (height > width) are scaled by height to preserve orientation.
Quality ladder:
- final (default): 1080p libx264 fast CRF 20
- preview: 1080p libx264 medium CRF 22 (evaluable for QC)
- draft: 720p libx264 ultrafast CRF 28 (cut-point check only)
"""
out_path.parent.mkdir(parents=True, exist_ok=True)
portrait = is_portrait_source(source)
if draft:
scale = "scale=-2:1280" if portrait else "scale=1280:-2"
else:
scale = "scale=-2:1920" if portrait else "scale=1920:-2"
vf_parts: list[str] = []
if is_hdr_source(source):
vf_parts.append(TONEMAP_CHAIN)
vf_parts.append(scale)
if grade_filter:
vf_parts.append(grade_filter)
vf = ",".join(vf_parts)
# 30ms audio fades at both edges (Rule 3) — prevent pops
fade_out_start = max(0.0, duration - 0.03)
af = f"afade=t=in:st=0:d=0.03,afade=t=out:st={fade_out_start:.3f}:d=0.03"
if draft:
preset, crf = "ultrafast", "28"
elif preview:
preset, crf = "medium", "22"
else:
preset, crf = "fast", "20"
cmd = [
"ffmpeg", "-y",
"-ss", f"{seg_start:.3f}",
"-i", str(source),
"-t", f"{duration:.3f}",
"-vf", vf,
"-af", af,
"-c:v", "libx264", "-preset", preset, "-crf", crf,
"-pix_fmt", "yuv420p", "-r", "24",
"-c:a", "aac", "-b:a", "192k", "-ar", "48000",
"-movflags", "+faststart",
str(out_path),
]
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
def extract_all_segments(
edl: dict,
edit_dir: Path,
preview: bool,
draft: bool = False,
) -> list[Path]:
"""Extract every EDL range into edit_dir/clips_graded/seg_NN.mp4.
Returns the ordered list of segment paths.
If the EDL `grade` is "auto", analyze each segment range with
`auto_grade_for_clip` and apply a per-segment subtle correction.
Otherwise, apply the same preset/raw filter to every segment.
"""
resolved = resolve_grade_filter(edl.get("grade"))
is_auto = resolved == "__AUTO__"
clips_dir = edit_dir / (
"clips_draft" if draft else ("clips_preview" if preview else "clips_graded")
)
clips_dir.mkdir(parents=True, exist_ok=True)
ranges = edl["ranges"]
sources = edl["sources"]
seg_paths: list[Path] = []
print(f"extracting {len(ranges)} segment(s) → {clips_dir.name}/")
if is_auto:
print(" (auto-grade per segment: analyzing each range)")
for i, r in enumerate(ranges):
src_name = r["source"]
src_path = resolve_path(sources[src_name], edit_dir)
start = float(r["start"])
end = float(r["end"])
duration = end - start
out_path = clips_dir / f"seg_{i:02d}_{src_name}.mp4"
if is_auto:
seg_filter, _stats = auto_grade_for_clip(src_path, start=start, duration=duration, verbose=False)
else:
seg_filter = resolved
note = r.get("beat") or r.get("note") or ""
print(f" [{i:02d}] {src_name} {start:7.2f}-{end:7.2f} ({duration:5.2f}s) {note}")
if is_auto:
print(f" grade: {seg_filter or '(none)'}")
extract_segment(src_path, start, duration, seg_filter, out_path, preview=preview, draft=draft)
seg_paths.append(out_path)
return seg_paths
# -------- Lossless concat ----------------------------------------------------
def concat_segments(segment_paths: list[Path], out_path: Path, edit_dir: Path) -> None:
"""Lossless concat via the concat demuxer. No re-encode."""
out_path.parent.mkdir(parents=True, exist_ok=True)
concat_list = edit_dir / "_concat.txt"
concat_list.write_text("".join(f"file '{p.resolve()}'\n" for p in segment_paths))
cmd = [
"ffmpeg", "-y",
"-f", "concat", "-safe", "0",
"-i", str(concat_list),
"-c", "copy",
"-movflags", "+faststart",
str(out_path),
]
print(f"concat → {out_path.name}")
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
concat_list.unlink(missing_ok=True)
# -------- Master SRT (Rule 5) ------------------------------------------------
PUNCT_BREAK = set(".,!?;:")
def _srt_timestamp(seconds: float) -> str:
total_ms = int(round(seconds * 1000))
h, rem = divmod(total_ms, 3600_000)
m, rem = divmod(rem, 60_000)
s, ms = divmod(rem, 1000)
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
def _words_in_range(transcript: dict, t_start: float, t_end: float) -> list[dict]:
out: list[dict] = []
for w in transcript.get("words", []):
if w.get("type") != "word":
continue
ws = w.get("start")
we = w.get("end")
if ws is None or we is None:
continue
if we <= t_start or ws >= t_end:
continue
out.append(w)
return out
def build_master_srt(edl: dict, edit_dir: Path, out_path: Path) -> None:
"""Build an output-timeline SRT from per-source transcripts.
- 2-word chunks (break on any punctuation in between)
- UPPERCASE text
- Output times computed as word.start - segment_start + segment_offset
"""
transcripts_dir = edit_dir / "transcripts"
sources = edl["sources"]
entries: list[tuple[float, float, str]] = []
seg_offset = 0.0
for r in edl["ranges"]:
src_name = r["source"]
seg_start = float(r["start"])
seg_end = float(r["end"])
seg_duration = seg_end - seg_start
tr_path = transcripts_dir / f"{src_name}.json"
if not tr_path.exists():
print(f" no transcript for {src_name}, skipping captions for this segment")
seg_offset += seg_duration
continue
transcript = json.loads(tr_path.read_text())
words_in_seg = _words_in_range(transcript, seg_start, seg_end)
# Group into 2-word chunks, break on punctuation
chunks: list[list[dict]] = []
current: list[dict] = []
for w in words_in_seg:
text = (w.get("text") or "").strip()
if not text:
continue
current.append(w)
# Break if the current text ends in punctuation or we hit 2 words
ends_in_punct = bool(text) and text[-1] in PUNCT_BREAK
if len(current) >= 2 or ends_in_punct:
chunks.append(current)
current = []
if current:
chunks.append(current)
for chunk in chunks:
local_start = max(seg_start, chunk[0].get("start", seg_start))
local_end = min(seg_end, chunk[-1].get("end", seg_end))
out_start = max(0.0, local_start - seg_start) + seg_offset
out_end = max(0.0, local_end - seg_start) + seg_offset
if out_end <= out_start:
out_end = out_start + 0.4
text = " ".join((w.get("text") or "").strip() for w in chunk)
text = re.sub(r"\s+", " ", text).strip()
# Strip trailing punctuation for cleaner uppercase look
text = text.rstrip(",;:")
text = text.upper()
entries.append((out_start, out_end, text))
seg_offset += seg_duration
# Sort and write as SRT
entries.sort(key=lambda e: e[0])
lines: list[str] = []
for i, (a, b, t) in enumerate(entries, start=1):
lines.append(str(i))
lines.append(f"{_srt_timestamp(a)} --> {_srt_timestamp(b)}")
lines.append(t)
lines.append("")
out_path.write_text("\n".join(lines))
print(f"master SRT → {out_path.name} ({len(entries)} cues)")
# -------- Loudness normalization (social-ready audio) -----------------------
# Social-media standard: -14 LUFS integrated, -1 dBTP peak, LRA 11 LU.
# Matches YouTube / Instagram / TikTok / X / LinkedIn normalization targets.
LOUDNORM_I = -14.0
LOUDNORM_TP = -1.0
LOUDNORM_LRA = 11.0
def measure_loudness(video_path: Path) -> dict[str, str] | None:
"""Run ffmpeg loudnorm first pass and parse the JSON measurement.
Returns a dict with measured_i, measured_tp, measured_lra, measured_thresh,
target_offset, or None if measurement failed.
"""
filter_str = (
f"loudnorm=I={LOUDNORM_I}:TP={LOUDNORM_TP}:LRA={LOUDNORM_LRA}:print_format=json"
)
cmd = [
"ffmpeg", "-y", "-hide_banner", "-nostats",
"-i", str(video_path),
"-af", filter_str,
"-vn", "-f", "null", "-",
]
proc = subprocess.run(cmd, capture_output=True, text=True)
# loudnorm prints the JSON to stderr at the end of the run
stderr = proc.stderr
# Find the JSON block — loudnorm output contains a `{ ... }` block
start = stderr.rfind("{")
end = stderr.rfind("}")
if start == -1 or end == -1 or end <= start:
return None
try:
data = json.loads(stderr[start : end + 1])
except json.JSONDecodeError:
return None
needed = {"input_i", "input_tp", "input_lra", "input_thresh", "target_offset"}
if not needed.issubset(data.keys()):
return None
return data
def apply_loudnorm_two_pass(
input_path: Path,
output_path: Path,
preview: bool = False,
) -> bool:
"""Run two-pass loudnorm on input_path, write normalized copy to output_path.
Returns True on success, False if measurement failed (caller should fall
back to copying the input unchanged).
In preview mode, skips the measurement pass and uses a one-pass approximation
for speed. Final mode always does the proper two-pass.
"""
if preview:
# One-pass approximation — faster, slightly less accurate.
filter_str = f"loudnorm=I={LOUDNORM_I}:TP={LOUDNORM_TP}:LRA={LOUDNORM_LRA}"
cmd = [
"ffmpeg", "-y", "-hide_banner", "-nostats",
"-i", str(input_path),
"-c:v", "copy",
"-af", filter_str,
"-c:a", "aac", "-b:a", "192k", "-ar", "48000",
"-movflags", "+faststart",
str(output_path),
]
print(f" loudnorm (1-pass preview) → {output_path.name}")
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
return True
# Full two-pass
print(f" loudnorm pass 1: measuring {input_path.name}")
measurement = measure_loudness(input_path)
if measurement is None:
print(" loudnorm measurement failed — falling back to 1-pass")
return apply_loudnorm_two_pass(input_path, output_path, preview=True)
print(f" measured: I={measurement['input_i']} LUFS "
f"TP={measurement['input_tp']} LRA={measurement['input_lra']}")
filter_str = (
f"loudnorm=I={LOUDNORM_I}:TP={LOUDNORM_TP}:LRA={LOUDNORM_LRA}"
f":measured_I={measurement['input_i']}"
f":measured_TP={measurement['input_tp']}"
f":measured_LRA={measurement['input_lra']}"
f":measured_thresh={measurement['input_thresh']}"
f":offset={measurement['target_offset']}"
f":linear=true"
)
cmd = [
"ffmpeg", "-y", "-hide_banner", "-nostats",
"-i", str(input_path),
"-c:v", "copy",
"-af", filter_str,
"-c:a", "aac", "-b:a", "192k", "-ar", "48000",
"-movflags", "+faststart",
str(output_path),
]
print(f" loudnorm pass 2: normalizing → {output_path.name}")
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
return True
# -------- Final compositing (Rule 1 + Rule 4) -------------------------------
def build_final_composite(
base_path: Path,
overlays: list[dict],
subtitles_path: Path | None,
out_path: Path,
edit_dir: Path,
) -> None:
"""Final pass: base → overlays (PTS-shifted) → subtitles LAST → out.
If there are no overlays and no subtitles, just copy base to out.
"""
has_overlays = bool(overlays)
has_subs = subtitles_path is not None and subtitles_path.exists()
if not has_overlays and not has_subs:
# Nothing to do — just rename/copy base to final name
run(["ffmpeg", "-y", "-i", str(base_path), "-c", "copy", str(out_path)], quiet=True)
return
inputs: list[str] = ["-i", str(base_path)]
for ov in overlays:
ov_path = resolve_path(ov["file"], edit_dir)
inputs += ["-i", str(ov_path)]
filter_parts: list[str] = []
# PTS-shift every overlay so its frame 0 lands at start_in_output
for idx, ov in enumerate(overlays, start=1):
t = float(ov["start_in_output"])
filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")
# Chain overlays on top of base
current = "[0:v]"
for idx, ov in enumerate(overlays, start=1):
t = float(ov["start_in_output"])
dur = float(ov["duration"])
end = t + dur
next_label = f"[v{idx}]"
filter_parts.append(
f"{current}[a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'{next_label}"
)
current = next_label
# Subtitles LAST — Rule 1
if has_subs:
subs_abs = str(subtitles_path.resolve()).replace(":", r"\:").replace("'", r"\'")
filter_parts.append(
f"{current}subtitles='{subs_abs}':force_style='{SUB_FORCE_STYLE}'[outv]"
)
out_label = "[outv]"
else:
# Rename the last overlay output to [outv] for consistency
if has_overlays:
filter_parts.append(f"{current}null[outv]")
out_label = "[outv]"
else:
out_label = "[0:v]"
filter_complex = ";".join(filter_parts)
cmd = [
"ffmpeg", "-y",
*inputs,
"-filter_complex", filter_complex,
"-map", out_label,
"-map", "0:a",
"-c:v", "libx264", "-preset", "fast", "-crf", "18",
"-pix_fmt", "yuv420p",
"-c:a", "copy",
"-movflags", "+faststart",
str(out_path),
]
print(f"compositing → {out_path.name}")
print(f" overlays: {len(overlays)}, subtitles: {'yes' if has_subs else 'no'}")
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
# -------- Main ---------------------------------------------------------------
def main() -> None:
ap = argparse.ArgumentParser(description="Render a video from an EDL")
ap.add_argument("edl", type=Path, help="Path to edl.json")
ap.add_argument("-o", "--output", type=Path, required=True, help="Output video path")
ap.add_argument(
"--preview",
action="store_true",
help="Preview mode: 1080p, medium, CRF 22 — evaluable for QC, faster than final.",
)
ap.add_argument(
"--draft",
action="store_true",
help="Draft mode: 720p, ultrafast, CRF 28 — cut-point verification only.",
)
ap.add_argument(
"--build-subtitles",
action="store_true",
help="Build master.srt from transcripts + EDL offsets before compositing",
)
ap.add_argument(
"--no-subtitles",
action="store_true",
help="Skip subtitles even if the EDL references one",
)
ap.add_argument(
"--no-loudnorm",
action="store_true",
help="Skip audio loudness normalization. Default is on (-14 LUFS, -1 dBTP, LRA 11).",
)
args = ap.parse_args()
edl_path = args.edl.resolve()
if not edl_path.exists():
sys.exit(f"edl not found: {edl_path}")
edl = json.loads(edl_path.read_text())
edit_dir = edl_path.parent
out_path = args.output.resolve()
# 1. Extract per-segment (auto-grade per range if EDL grade is "auto")
segment_paths = extract_all_segments(
edl, edit_dir, preview=args.preview, draft=args.draft
)
# 2. Concat → base
if args.draft:
base_name = "base_draft.mp4"
elif args.preview:
base_name = "base_preview.mp4"
else:
base_name = "base.mp4"
base_path = edit_dir / base_name
concat_segments(segment_paths, base_path, edit_dir)
# 3. Subtitles: build if requested, resolve final path
subs_path: Path | None = None
if not args.no_subtitles:
if args.build_subtitles:
subs_path = edit_dir / "master.srt"
build_master_srt(edl, edit_dir, subs_path)
elif edl.get("subtitles"):
subs_path = resolve_path(edl["subtitles"], edit_dir)
if not subs_path.exists():
print(f"warning: subtitles path in EDL does not exist: {subs_path}")
subs_path = None
# 4. Composite (overlays + subtitles LAST) → intermediate (pre-loudnorm) path
overlays = edl.get("overlays") or []
if args.no_loudnorm:
# Composite directly to final output
build_final_composite(base_path, overlays, subs_path, out_path, edit_dir)
else:
# Composite to a temp file, then run loudnorm → final output
tmp_composite = out_path.with_suffix(".prenorm.mp4")
build_final_composite(base_path, overlays, subs_path, tmp_composite, edit_dir)
print("loudness normalization → social-ready (-14 LUFS / -1 dBTP / LRA 11)")
apply_loudnorm_two_pass(tmp_composite, out_path, preview=args.draft)
tmp_composite.unlink(missing_ok=True)
size_mb = out_path.stat().st_size / (1024 * 1024)
print(f"\ndone: {out_path} ({size_mb:.1f} MB)")
if __name__ == "__main__":
main()
"""Filmstrip + waveform composite PNG for a time range of a video.
The only visual drill-down tool. Given a video and a [start, end] range,
extracts N evenly spaced frames via ffmpeg, composites them into a
horizontal filmstrip, and renders a waveform ribbon below with word
labels overlaid from the transcript (if available) and silence gaps
shaded.
Use this at decision points — ambiguous pauses, retake disambiguation,
cut-point sanity checks. Do NOT call it in a scan loop over every
utterance; it's an on-demand drill-down, not a background index.
Usage:
python helpers/timeline_view.py <video> <start> <end>
python helpers/timeline_view.py <video> <start> <end> -o out.png
python helpers/timeline_view.py <video> <start> <end> --n-frames 12
python helpers/timeline_view.py <video> <start> <end> --transcript <path>
python helpers/timeline_view.py --edl <edl.json> (full-project view — not yet)
"""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import tempfile
from pathlib import Path
import numpy as np
from PIL import Image, ImageDraw, ImageFont
# -------- Frame extraction ---------------------------------------------------
def extract_frames(video: Path, start: float, end: float, n: int, dest_dir: Path) -> list[Path]:
"""Extract N frames evenly spaced across [start, end]. Returns paths in order."""
dest_dir.mkdir(parents=True, exist_ok=True)
if n < 1:
n = 1
if n == 1:
times = [(start + end) / 2.0]
else:
step = (end - start) / (n - 1)
times = [start + i * step for i in range(n)]
paths: list[Path] = []
for i, t in enumerate(times):
out = dest_dir / f"f_{i:03d}.jpg"
cmd = [
"ffmpeg", "-y",
"-ss", f"{t:.3f}",
"-i", str(video),
"-frames:v", "1",
"-q:v", "4",
"-vf", "scale=320:-2",
str(out),
]
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
paths.append(out)
return paths
# -------- Audio envelope (librosa if available, ffmpeg fallback) ------------
def compute_envelope(video: Path, start: float, end: float, samples: int = 2000) -> np.ndarray:
"""Extract the audio segment and return an RMS envelope of length `samples`.
Uses ffmpeg to dump mono 16kHz PCM to a temp wav, then computes a
windowed RMS. Falls back gracefully if the source has no audio.
"""
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
wav = Path(f.name)
try:
cmd = [
"ffmpeg", "-y",
"-ss", f"{start:.3f}",
"-i", str(video),
"-t", f"{(end - start):.3f}",
"-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le",
str(wav),
]
r = subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if r.returncode != 0 or not wav.exists() or wav.stat().st_size == 0:
return np.zeros(samples)
# Read the WAV manually — avoid librosa as a hard dep
import wave
with wave.open(str(wav), "rb") as w:
frames = w.readframes(w.getnframes())
pcm = np.frombuffer(frames, dtype=np.int16).astype(np.float32) / 32768.0
if pcm.size == 0:
return np.zeros(samples)
# Windowed RMS → envelope of length `samples`
n = pcm.size
window = max(1, n // samples)
usable = (n // window) * window
reshaped = pcm[:usable].reshape(-1, window)
env = np.sqrt(np.mean(reshaped ** 2, axis=1))
if env.size < samples:
env = np.pad(env, (0, samples - env.size))
elif env.size > samples:
env = env[:samples]
# Normalize to [0, 1]
if env.max() > 0:
env = env / env.max()
return env
finally:
wav.unlink(missing_ok=True)
# -------- Transcript word overlays ------------------------------------------
def words_in_range(transcript_path: Path, start: float, end: float) -> list[dict]:
if not transcript_path.exists():
return []
data = json.loads(transcript_path.read_text())
out: list[dict] = []
for w in data.get("words", []):
t = w.get("type", "word")
ws = w.get("start")
we = w.get("end")
if ws is None or we is None:
continue
if we <= start or ws >= end:
continue
out.append(w)
return out
def find_silences(words: list[dict], start: float, end: float, threshold: float = 0.4) -> list[tuple[float, float]]:
"""Find gaps >= threshold seconds inside [start, end] between kept tokens."""
gaps: list[tuple[float, float]] = []
prev_end = start
for w in words:
if w.get("type") == "spacing":
continue
ws = max(start, w.get("start", start))
if ws - prev_end >= threshold:
gaps.append((prev_end, ws))
prev_end = max(prev_end, w.get("end", ws))
if end - prev_end >= threshold:
gaps.append((prev_end, end))
return gaps
# -------- Font loading -------------------------------------------------------
FONT_CANDIDATES = [
"/System/Library/Fonts/Menlo.ttc",
"/System/Library/Fonts/Helvetica.ttc",
"/System/Library/Fonts/SFNSMono.ttf",
"/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf",
"/usr/share/fonts/truetype/liberation/LiberationMono-Regular.ttf",
]
def load_font(size: int) -> ImageFont.ImageFont:
for fp in FONT_CANDIDATES:
if Path(fp).exists():
try:
return ImageFont.truetype(fp, size)
except Exception:
continue
return ImageFont.load_default()
# -------- Composite ----------------------------------------------------------
BG = (18, 18, 22)
FG = (235, 235, 235)
DIM = (110, 110, 120)
ACCENT = (255, 140, 60)
SILENCE = (50, 80, 120, 120) # muted blue, semi-transparent
WAVE = (140, 180, 255)
def render_timeline(
video: Path,
start: float,
end: float,
out_path: Path,
n_frames: int,
transcript: Path | None,
) -> None:
# Frame extraction
with tempfile.TemporaryDirectory() as tmp:
tmp_dir = Path(tmp)
print(f"extracting {n_frames} frames from {start:.2f}s to {end:.2f}s")
frame_paths = extract_frames(video, start, end, n_frames, tmp_dir)
# Layout metrics
canvas_width = 1920
frame_h = 180
filmstrip_y = 50
filmstrip_h = frame_h
wave_y = filmstrip_y + filmstrip_h + 20
wave_h = 220
label_y = wave_y + wave_h + 10
canvas_height = label_y + 60
# Load + resize frames to uniform height and compute total width
imgs: list[Image.Image] = []
for fp in frame_paths:
img = Image.open(fp).convert("RGB")
aspect = img.width / img.height
new_w = int(frame_h * aspect)
imgs.append(img.resize((new_w, frame_h), Image.LANCZOS))
total_frame_w = sum(img.width for img in imgs) + (len(imgs) - 1) * 4
content_w = max(1400, total_frame_w)
canvas_width = max(canvas_width, content_w + 100)
canvas = Image.new("RGB", (canvas_width, canvas_height), BG)
draw = ImageDraw.Draw(canvas, "RGBA")
header_font = load_font(22)
label_font = load_font(14)
small_font = load_font(12)
# Header — time range
draw.text(
(50, 12),
f"{video.name} {start:.2f}s → {end:.2f}s ({(end - start):.2f}s, {n_frames} frames)",
fill=FG,
font=header_font,
)
# Filmstrip
x = 50
strip_width = canvas_width - 100
if total_frame_w <= strip_width:
cursor = 50
for img in imgs:
canvas.paste(img, (cursor, filmstrip_y))
cursor += img.width + 4
draw_width = cursor - 50
else:
scale = strip_width / total_frame_w
new_h = int(frame_h * scale)
cursor = 50
for img in imgs:
new_w = int(img.width * scale)
scaled = img.resize((new_w, new_h), Image.LANCZOS)
canvas.paste(scaled, (cursor, filmstrip_y + (filmstrip_h - new_h) // 2))
cursor += new_w + max(2, int(4 * scale))
draw_width = cursor - 50
strip_x0 = 50
strip_x1 = 50 + draw_width
strip_span = strip_x1 - strip_x0
def time_to_x(t: float) -> int:
frac = (t - start) / max(1e-6, (end - start))
return int(strip_x0 + frac * strip_span)
# Waveform background
draw.rectangle((strip_x0, wave_y, strip_x1, wave_y + wave_h), fill=(28, 28, 34))
# Silence shading (under the waveform)
words = words_in_range(transcript, start, end) if transcript else []
silences = find_silences(words, start, end, threshold=0.4) if words else []
for a, b in silences:
xa = time_to_x(a)
xb = time_to_x(b)
draw.rectangle((xa, wave_y, xb, wave_y + wave_h), fill=SILENCE)
# Waveform envelope
env = compute_envelope(video, start, end, samples=max(strip_span, 200))
mid_y = wave_y + wave_h // 2
max_amp = wave_h // 2 - 8
points_top: list[tuple[int, int]] = []
points_bot: list[tuple[int, int]] = []
for i, v in enumerate(env):
xi = strip_x0 + int(i * strip_span / max(1, len(env) - 1))
a = int(v * max_amp)
points_top.append((xi, mid_y - a))
points_bot.append((xi, mid_y + a))
if points_top:
draw.line(points_top, fill=WAVE, width=1, joint="curve")
draw.line(points_bot, fill=WAVE, width=1, joint="curve")
# Fill between
poly = points_top + list(reversed(points_bot))
draw.polygon(poly, fill=(*WAVE, 60))
# Word labels above the waveform (only words lasting ≥ 120ms to avoid clutter)
last_label_x = -9999
for w in words:
if w.get("type") != "word":
continue
ws = w.get("start")
we = w.get("end")
text = (w.get("text") or "").strip()
if not text or ws is None or we is None:
continue
if (we - ws) < 0.05:
continue
cx = (time_to_x(ws) + time_to_x(we)) // 2
if cx - last_label_x < 28:
continue
# Tiny tick on the waveform
draw.line((cx, wave_y - 4, cx, wave_y), fill=DIM, width=1)
# Text above the waveform
draw.text((cx + 2, wave_y - 18), text, fill=FG, font=small_font)
last_label_x = cx
# Time ruler below waveform
ruler_y = wave_y + wave_h + 2
n_ticks = 6
for i in range(n_ticks + 1):
frac = i / n_ticks
t = start + frac * (end - start)
xi = strip_x0 + int(frac * strip_span)
draw.line((xi, ruler_y, xi, ruler_y + 6), fill=DIM, width=1)
draw.text((xi - 20, ruler_y + 8), f"{t:.2f}s", fill=DIM, font=label_font)
# Silences legend if any
if silences:
txt = f"shaded bands = silences ≥ 400ms ({len(silences)} gap(s))"
draw.text((strip_x0, label_y + 30), txt, fill=DIM, font=label_font)
out_path.parent.mkdir(parents=True, exist_ok=True)
canvas.save(out_path, "PNG", optimize=True)
print(f"saved: {out_path} ({out_path.stat().st_size // 1024} KB)")
def main() -> None:
ap = argparse.ArgumentParser(description="Filmstrip + waveform composite for a video range")
ap.add_argument("video", type=Path, nargs="?", help="Source video")
ap.add_argument("start", type=float, nargs="?", help="Start time in seconds")
ap.add_argument("end", type=float, nargs="?", help="End time in seconds")
ap.add_argument("-o", "--output", type=Path, default=None, help="Output PNG path")
ap.add_argument("--n-frames", type=int, default=10, help="Number of frames in the filmstrip (default 10)")
ap.add_argument(
"--transcript",
type=Path,
default=None,
help="Path to transcript.json for word labels + silence shading. "
"If omitted, will auto-resolve to <video_parent>/edit/transcripts/<video_stem>.json",
)
ap.add_argument(
"--edl",
type=Path,
default=None,
help="(Not yet implemented) Render a full-project timeline from an EDL",
)
args = ap.parse_args()
if args.edl:
sys.exit("--edl mode is not implemented yet; use range mode")
if not args.video or args.start is None or args.end is None:
ap.error("video, start, and end are required")
video = args.video.resolve()
if not video.exists():
sys.exit(f"video not found: {video}")
if args.end <= args.start:
sys.exit("end must be > start")
# Auto-resolve transcript if not given
transcript = args.transcript
if transcript is None:
auto = video.parent / "edit" / "transcripts" / f"{video.stem}.json"
if auto.exists():
transcript = auto
out_path = args.output
if out_path is None:
out_dir = video.parent / "edit" / "verify"
out_dir.mkdir(parents=True, exist_ok=True)
out_path = out_dir / f"{video.stem}_{args.start:.2f}-{args.end:.2f}.png"
render_timeline(
video=video,
start=args.start,
end=args.end,
out_path=out_path,
n_frames=args.n_frames,
transcript=transcript,
)
if __name__ == "__main__":
main()
"""Batch-transcribe every video in a directory with 4 parallel workers.
Walks <videos_dir> for common video extensions, runs ElevenLabs Scribe on
each, writes transcripts to <videos_dir>/edit/transcripts/<name>.json.
Cached per-file: any source that already has a transcript is skipped.
Usage:
python helpers/transcribe_batch.py <videos_dir>
python helpers/transcribe_batch.py <videos_dir> --workers 4
python helpers/transcribe_batch.py <videos_dir> --num-speakers 2
python helpers/transcribe_batch.py <videos_dir> --edit-dir /custom/edit
"""
from __future__ import annotations
import argparse
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from transcribe import load_api_key, transcribe_one
VIDEO_EXTS = {".mp4", ".MP4", ".mov", ".MOV", ".mkv", ".MKV", ".avi", ".AVI", ".m4v"}
def find_videos(videos_dir: Path) -> list[Path]:
videos = sorted(
p for p in videos_dir.iterdir()
if p.is_file() and p.suffix in VIDEO_EXTS
)
return videos
def main() -> None:
ap = argparse.ArgumentParser(description="Parallel batch transcription of a videos directory")
ap.add_argument("videos_dir", type=Path, help="Directory containing source videos")
ap.add_argument(
"--edit-dir",
type=Path,
default=None,
help="Edit output directory (default: <videos_dir>/edit)",
)
ap.add_argument("--workers", type=int, default=4, help="Parallel workers (default: 4)")
ap.add_argument(
"--language",
type=str,
default=None,
help="Optional ISO language code. Omit to auto-detect per file.",
)
ap.add_argument(
"--num-speakers",
type=int,
default=None,
help="Optional number of speakers. Improves diarization when known.",
)
args = ap.parse_args()
videos_dir = args.videos_dir.resolve()
if not videos_dir.is_dir():
sys.exit(f"not a directory: {videos_dir}")
edit_dir = (args.edit_dir or (videos_dir / "edit")).resolve()
(edit_dir / "transcripts").mkdir(parents=True, exist_ok=True)
videos = find_videos(videos_dir)
if not videos:
sys.exit(f"no videos found in {videos_dir}")
already_cached = [v for v in videos if (edit_dir / "transcripts" / f"{v.stem}.json").exists()]
pending = [v for v in videos if v not in already_cached]
print(f"found {len(videos)} videos ({len(already_cached)} cached, {len(pending)} to transcribe)")
if not pending:
print("nothing to do")
return
api_key = load_api_key()
print(f"transcribing {len(pending)} files with {args.workers} parallel workers")
t0 = time.time()
errors: list[tuple[Path, str]] = []
with ThreadPoolExecutor(max_workers=args.workers) as pool:
futures = {
pool.submit(
transcribe_one,
video=v,
edit_dir=edit_dir,
api_key=api_key,
language=args.language,
num_speakers=args.num_speakers,
verbose=False,
): v
for v in pending
}
for fut in as_completed(futures):
v = futures[fut]
try:
out = fut.result()
print(f" + {v.stem} → {out.name}")
except Exception as e:
errors.append((v, str(e)))
print(f" x {v.stem} FAILED: {e}")
dt = time.time() - t0
print(f"\ndone in {dt:.1f}s")
if errors:
print(f"{len(errors)} failures:")
for v, msg in errors:
print(f" {v.name}: {msg}")
sys.exit(1)
if __name__ == "__main__":
main()
"""Transcribe a video with ElevenLabs Scribe.
Extracts mono 16kHz audio via ffmpeg, uploads to Scribe with verbatim +
diarize + audio events + word-level timestamps, writes the full response
to <edit_dir>/transcripts/<video_stem>.json.
Cached: if the output file already exists, the upload is skipped.
Usage:
python helpers/transcribe.py <video_path>
python helpers/transcribe.py <video_path> --edit-dir /custom/edit
python helpers/transcribe.py <video_path> --language en
python helpers/transcribe.py <video_path> --num-speakers 2
"""
from __future__ import annotations
import argparse
import json
import os
import subprocess
import sys
import tempfile
import time
from pathlib import Path
import requests
SCRIBE_URL = "https://api.elevenlabs.io/v1/speech-to-text"
def load_api_key() -> str:
for candidate in [Path(__file__).resolve().parent.parent / ".env", Path(".env")]:
if candidate.exists():
for line in candidate.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
k, v = line.split("=", 1)
if k.strip() == "ELEVENLABS_API_KEY":
return v.strip().strip('"').strip("'")
v = os.environ.get("ELEVENLABS_API_KEY", "")
if not v:
sys.exit("ELEVENLABS_API_KEY not found in .env or environment")
return v
def extract_audio(video_path: Path, dest: Path) -> None:
cmd = [
"ffmpeg", "-y", "-i", str(video_path),
"-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le",
str(dest),
]
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
def call_scribe(
audio_path: Path,
api_key: str,
language: str | None = None,
num_speakers: int | None = None,
) -> dict:
data: dict[str, str] = {
"model_id": "scribe_v1",
"diarize": "true",
"tag_audio_events": "true",
"timestamps_granularity": "word",
}
if language:
data["language_code"] = language
if num_speakers:
data["num_speakers"] = str(num_speakers)
with open(audio_path, "rb") as f:
resp = requests.post(
SCRIBE_URL,
headers={"xi-api-key": api_key},
files={"file": (audio_path.name, f, "audio/wav")},
data=data,
timeout=1800,
)
if resp.status_code != 200:
raise RuntimeError(f"Scribe returned {resp.status_code}: {resp.text[:500]}")
return resp.json()
def transcribe_one(
video: Path,
edit_dir: Path,
api_key: str,
language: str | None = None,
num_speakers: int | None = None,
verbose: bool = True,
) -> Path:
"""Transcribe a single video. Returns path to transcript JSON.
Cached: returns existing path immediately if the transcript already exists.
"""
transcripts_dir = edit_dir / "transcripts"
transcripts_dir.mkdir(parents=True, exist_ok=True)
out_path = transcripts_dir / f"{video.stem}.json"
if out_path.exists():
if verbose:
print(f"cached: {out_path.name}")
return out_path
if verbose:
print(f" extracting audio from {video.name}", flush=True)
t0 = time.time()
with tempfile.TemporaryDirectory() as tmp:
audio = Path(tmp) / f"{video.stem}.wav"
extract_audio(video, audio)
size_mb = audio.stat().st_size / (1024 * 1024)
if verbose:
print(f" uploading {video.stem}.wav ({size_mb:.1f} MB)", flush=True)
payload = call_scribe(audio, api_key, language, num_speakers)
out_path.write_text(json.dumps(payload, indent=2))
dt = time.time() - t0
if verbose:
kb = out_path.stat().st_size / 1024
print(f" saved: {out_path.name} ({kb:.1f} KB) in {dt:.1f}s")
if isinstance(payload, dict) and "words" in payload:
print(f" words: {len(payload['words'])}")
return out_path
def main() -> None:
ap = argparse.ArgumentParser(description="Transcribe a video with ElevenLabs Scribe")
ap.add_argument("video", type=Path, help="Path to video file")
ap.add_argument(
"--edit-dir",
type=Path,
default=None,
help="Edit output directory (default: <video_parent>/edit)",
)
ap.add_argument(
"--language",
type=str,
default=None,
help="Optional ISO language code (e.g., 'en'). Omit to auto-detect.",
)
ap.add_argument(
"--num-speakers",
type=int,
default=None,
help="Optional number of speakers when known. Improves diarization accuracy.",
)
args = ap.parse_args()
video = args.video.resolve()
if not video.exists():
sys.exit(f"video not found: {video}")
edit_dir = (args.edit_dir or (video.parent / "edit")).resolve()
api_key = load_api_key()
transcribe_one(
video=video,
edit_dir=edit_dir,
api_key=api_key,
language=args.language,
num_speakers=args.num_speakers,
)
if __name__ == "__main__":
main()
video-use install
Use this file only for first-time install or reconnect. For daily editing, read SKILL.md. Always read helpers/ — that's where the scripts live.
What you're doing
You're setting up a conversation-driven video editor for the user. After install, the user drops raw footage into any folder, runs their agent (claude, codex, etc.) there, and says "edit these into a launch video." You do the rest by reading SKILL.md.
Three things must exist on this machine:
1. The video-use repo cloned somewhere stable. 2. ffmpeg on $PATH (plus optional yt-dlp for online sources). 3. An ElevenLabs API key in .env at the repo root (for Scribe transcription).
And one thing must be true about the current agent:
4. It can discover SKILL.md — either via a global skills directory (~/.claude/skills/, ~/.codex/skills/) or via a CLAUDE.md / system-prompt import.
Install prompt contract
- Do everything yourself. Only ask the user for things you cannot generate — the ElevenLabs API key, and confirmation before
brew install. - Prefer a stable clone path like
~/Developer/video-use(not/tmp, not~/Downloads). - The skill references helpers by bare name (
transcribe.py,render.py). That works because SKILL.md andhelpers/ship together — keep them as siblings when you register the skill. - After install, verify by running one real command against one real file. Don't declare success on file-existence checks alone.
Steps
1. Clone to a stable path
test -d ~/Developer/video-use || git clone https://github.com/browser-use/video-use ~/Developer/video-use
cd ~/Developer/video-useIf the repo is already there, git pull --ff-only and continue.
2. Install Python deps
# Prefer uv if available; fall back to pip.
command -v uv >/dev/null && uv sync || pip install -e .pyproject.toml lists requests, librosa, matplotlib, pillow, numpy. No console scripts — helpers are invoked directly as python helpers/<name>.py.
3. Install ffmpeg (+ optional yt-dlp)
ffmpeg and ffprobe are hard requirements. yt-dlp is only needed if the user wants to pull sources from URLs. Animation engines such as HyperFrames, Remotion, and Manim are installed lazily the first time a project actually needs them.
# macOS
command -v ffmpeg >/dev/null || brew install ffmpeg
command -v yt-dlp >/dev/null || brew install yt-dlp # optional
# Debian / Ubuntu
# sudo apt-get update && sudo apt-get install -y ffmpeg
# pip install yt-dlp
# Arch
# sudo pacman -S ffmpeg yt-dlpIf brew / apt / pacman requires a sudo prompt, tell the user the exact command and wait. Do not invent a password.
4. Register the skill with the current agent
Figure out which agent you are running under, and register once. A symlink of the whole repo directory is the right shape — helpers/ needs to sit next to SKILL.md.
- Claude Code (
~/.claude/present):
mkdir -p ~/.claude/skills
ln -sfn ~/Developer/video-use ~/.claude/skills/video-use- Codex (
$CODEX_HOMEset, or~/.codex/present):
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills"
ln -sfn ~/Developer/video-use "${CODEX_HOME:-$HOME/.codex}/skills/video-use"- Hermes / Openclaw / another agent with a skills directory: symlink
~/Developer/video-useinto that agent's skills directory under the namevideo-use. If the agent has no skills directory, add a line to its system prompt / config pointing at~/Developer/video-use/SKILL.md(e.g. an@~/Developer/video-use/SKILL.mdimport in aCLAUDE.md-equivalent).
If you can't tell which agent you're in, ask the user once: "which agent am I running under — Claude Code, Codex, or something else?" Then pick the right target.
5. ElevenLabs API key
Scribe (ElevenLabs) does all transcription. Without a key, nothing transcribes.
1. Check existing state in this order and stop at the first hit:
# a) env var already exported
[ -n "$ELEVENLABS_API_KEY" ] && echo "env"
# b) .env at repo root already has it
grep -q '^ELEVENLABS_API_KEY=..' ~/Developer/video-use/.env 2>/dev/null && echo "dotenv"2. If neither is set, ask the user exactly once:
I need an ElevenLabs API key for transcription (word-level timestamps, speaker diarization, filler tagging). Grab one at https://elevenlabs.io/app/settings/api-keys and paste it here — I'll write it to~/Developer/video-use/.env. Or if you already have it exported asELEVENLABS_API_KEY, say "use env" and I'll skip.
When the user pastes a key, write it to ~/Developer/video-use/.env:
printf 'ELEVENLABS_API_KEY=%s\n' "$KEY" > ~/Developer/video-use/.env
chmod 600 ~/Developer/video-use/.envNever echo the key back in tool output. Never commit .env.
3. Sanity check with a cheap, quota-free call:
curl -s -o /dev/null -w '%{http_code}\n' \
-H "xi-api-key: $(sed -n 's/^ELEVENLABS_API_KEY=//p' ~/Developer/video-use/.env)" \
https://api.elevenlabs.io/v1/user200 means the key works. 401 means the user pasted a wrong/expired key — ask once more and stop. Anything else (network, 5xx), move on and verify during first real transcription.
6. Verify end-to-end
Run one real thing. Prefer the lightest verification that still proves the pipeline is wired up:
python ~/Developer/video-use/helpers/timeline_view.py --help >/dev/null && echo "helpers OK"
ffprobe -version | head -1Full transcription test is optional at install time — it burns Scribe credits. Better to wait until the user hands you their first clip.
7. Hand off
Tell the user, in one short message:
- Where the skill is installed (
~/Developer/video-use). - That they should
cdinto their footage folder and start their agent there (e.g.claude). - That a good first message is: "edit these into a launch video" or "inventory these takes and propose a strategy."
- That all outputs land in
<videos_dir>/edit/— the repo stays clean.
Keeping the skill current
cd ~/Developer/video-use && git pull --ff-onlypulls the latest code. The symlink auto-picks it up on the next run.- If
pyproject.tomlchanged deps, re-runuv sync/pip install -e .after pulling.
Cold-start reminders
- Symlink the whole directory, not just
SKILL.md. The helpers need to sit next to it. - If
.envexists but the key is empty, treat it the same as missing — don't assume existence means validity. ffmpegfrom static builds works fine. Any modern (≥ 4.x) build is enough.yt-dlpis optional. Don't block install on it; install lazily the first time a user asks to pull from a URL.- Node.js/npm are only needed for HyperFrames or Remotion slots. HyperFrames currently requires Node.js 22+.
- HyperFrames, Remotion, and Manim are optional animation engines. Don't install or prefer one globally during setup; pick the engine per animation slot in
SKILL.md. HyperFrames can run throughnpx --yes hyperframes ...in the slot directory. Remotion can be scaffolded withnpx create-video@latestor installed inside the slot before rendering. - Never run transcription as part of install verification unless the user explicitly asks — Scribe costs real money.
- If the user is on Linux without a package manager Claude recognizes, print the manual
ffmpeginstall URL and wait rather than guessing.
MIT License
Copyright (c) 2026 Browser Use
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>How an LLM Understands Video</title>
<script src="https://cdn.tailwindcss.com"></script>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800;900&family=JetBrains+Mono:wght@400;500;600;700&display=swap" rel="stylesheet">
<script>
tailwind.config = {
theme: {
extend: {
fontFamily: {
sans: ['Inter', 'system-ui', 'sans-serif'],
mono: ['JetBrains Mono', 'monospace'],
},
colors: {
accent: '#ff6b35',
}
}
}
}
</script>
</head>
<body class="bg-stone-200 flex justify-center py-10">
<div class="max-w-[1400px] w-full bg-stone-50 border border-stone-300 shadow-sm">
<!-- Header -->
<div class="bg-stone-900 px-10 py-7">
<div class="flex items-end justify-between">
<div>
<div class="text-[11px] font-bold uppercase tracking-[3px] text-accent mb-2">The Harness</div>
<div class="text-[32px] font-extrabold text-white tracking-tight leading-none">How an LLM Understands Video</div>
</div>
<div class="text-right text-stone-500 text-[11px] font-mono">
video-use / SKILL.md
</div>
</div>
</div>
<!-- Insight bar -->
<div class="bg-stone-800 px-10 py-3 border-b border-stone-700">
<div class="text-stone-300 text-[13px] text-center">
Don't show pixels. Show a <span class="text-white font-bold">text transcript</span> from audio + <span class="text-accent font-bold">visual composites</span> on demand. The LLM never watches the video. It reads it.
</div>
</div>
<!-- Section label -->
<div class="px-10 pt-7 pb-3">
<div class="text-[10px] font-extrabold uppercase tracking-[3px] text-stone-400">What the LLM actually sees</div>
</div>
<!-- ============ HERO: The combined timeline_view visualization ============ -->
<div class="px-10 pb-8">
<div class="bg-stone-900 rounded-xl overflow-hidden border border-stone-700">
<!-- Top bar mimicking an editor UI -->
<div class="bg-stone-800 px-5 py-2.5 flex items-center justify-between border-b border-stone-700">
<div class="flex items-center gap-3">
<div class="flex gap-1.5">
<div class="w-2.5 h-2.5 rounded-full bg-stone-600"></div>
<div class="w-2.5 h-2.5 rounded-full bg-stone-600"></div>
<div class="w-2.5 h-2.5 rounded-full bg-stone-600"></div>
</div>
<div class="font-mono text-[11px] text-stone-400">timeline_view.py — C0103.MP4 — 42.0s → 48.0s</div>
</div>
<div class="font-mono text-[10px] text-accent font-bold">ON-DEMAND COMPOSITE PNG</div>
</div>
<div class="p-5">
<svg viewBox="0 0 1100 460" class="w-full" xmlns="http://www.w3.org/2000/svg">
<!-- ===== FILMSTRIP ROW ===== -->
<rect x="0" y="0" width="1100" height="135" rx="6" fill="#1a1a20"/>
<!-- 10 frame thumbnails with slight color variation to look like real frames -->
<rect x="8" y="8" width="100" height="119" rx="4" fill="#252530"/>
<rect x="20" y="30" width="30" height="30" rx="15" fill="#353540"/>
<rect x="55" y="35" width="40" height="20" rx="3" fill="#2d2d38"/>
<rect x="25" y="80" width="60" height="8" rx="2" fill="#2d2d38"/>
<rect x="35" y="95" width="40" height="6" rx="2" fill="#252530"/>
<rect x="116" y="8" width="100" height="119" rx="4" fill="#282832"/>
<rect x="128" y="30" width="30" height="30" rx="15" fill="#383842"/>
<rect x="163" y="35" width="40" height="20" rx="3" fill="#303038"/>
<rect x="133" y="80" width="60" height="8" rx="2" fill="#303038"/>
<rect x="143" y="95" width="40" height="6" rx="2" fill="#282832"/>
<rect x="224" y="8" width="100" height="119" rx="4" fill="#262630"/>
<rect x="236" y="30" width="30" height="30" rx="15" fill="#363640"/>
<rect x="271" y="35" width="40" height="20" rx="3" fill="#2e2e36"/>
<rect x="241" y="80" width="60" height="8" rx="2" fill="#2e2e36"/>
<rect x="251" y="95" width="40" height="6" rx="2" fill="#262630"/>
<rect x="332" y="8" width="100" height="119" rx="4" fill="#252530"/>
<rect x="344" y="30" width="30" height="30" rx="15" fill="#353540"/>
<rect x="379" y="35" width="40" height="20" rx="3" fill="#2d2d38"/>
<rect x="349" y="80" width="60" height="8" rx="2" fill="#2d2d38"/>
<rect x="440" y="8" width="100" height="119" rx="4" fill="#282832"/>
<rect x="452" y="30" width="30" height="30" rx="15" fill="#383842"/>
<rect x="487" y="35" width="40" height="20" rx="3" fill="#303038"/>
<rect x="457" y="80" width="60" height="8" rx="2" fill="#303038"/>
<rect x="548" y="8" width="100" height="119" rx="4" fill="#262630"/>
<rect x="560" y="30" width="30" height="30" rx="15" fill="#363640"/>
<rect x="595" y="35" width="40" height="20" rx="3" fill="#2e2e36"/>
<rect x="565" y="80" width="60" height="8" rx="2" fill="#2e2e36"/>
<rect x="656" y="8" width="100" height="119" rx="4" fill="#252530"/>
<rect x="668" y="30" width="30" height="30" rx="15" fill="#353540"/>
<rect x="703" y="35" width="40" height="20" rx="3" fill="#2d2d38"/>
<rect x="673" y="80" width="60" height="8" rx="2" fill="#2d2d38"/>
<rect x="764" y="8" width="100" height="119" rx="4" fill="#282832"/>
<rect x="776" y="30" width="30" height="30" rx="15" fill="#383842"/>
<rect x="811" y="35" width="40" height="20" rx="3" fill="#303038"/>
<rect x="781" y="80" width="60" height="8" rx="2" fill="#303038"/>
<rect x="872" y="8" width="100" height="119" rx="4" fill="#262630"/>
<rect x="884" y="30" width="30" height="30" rx="15" fill="#363640"/>
<rect x="919" y="35" width="40" height="20" rx="3" fill="#2e2e36"/>
<rect x="980" y="8" width="112" height="119" rx="4" fill="#252530"/>
<rect x="992" y="30" width="30" height="30" rx="15" fill="#353540"/>
<rect x="1027" y="35" width="40" height="20" rx="3" fill="#2d2d38"/>
<!-- ===== SPEAKER TRACK ===== -->
<rect x="0" y="145" width="1100" height="30" rx="0" fill="#13131a"/>
<rect x="8" y="150" width="620" height="20" rx="4" fill="rgba(255,107,53,0.15)" stroke="rgba(255,107,53,0.3)" stroke-width="1"/>
<text x="310" y="164" text-anchor="middle" font-family="JetBrains Mono" font-size="10" font-weight="600" fill="#ff6b35">SPEAKER 0</text>
<rect x="680" y="150" width="410" height="20" rx="4" fill="rgba(255,107,53,0.08)" stroke="rgba(255,107,53,0.15)" stroke-width="1"/>
<text x="885" y="164" text-anchor="middle" font-family="JetBrains Mono" font-size="10" font-weight="600" fill="rgba(255,107,53,0.5)">SPEAKER 1</text>
<!-- ===== WAVEFORM ===== -->
<rect x="0" y="180" width="1100" height="190" rx="0" fill="#0f0f16"/>
<!-- Center line -->
<line x1="0" y1="275" x2="1100" y2="275" stroke="#1a1a24" stroke-width="1"/>
<!-- Silence gap shading -->
<rect x="370" y="183" width="80" height="184" fill="rgba(255,107,53,0.06)"/>
<rect x="630" y="183" width="100" height="184" fill="rgba(255,107,53,0.06)"/>
<!-- Waveform — top half -->
<polygon points="
8,275
30,260 50,250 70,235 90,220 110,230 130,220 150,210 170,225 190,240 210,250
230,260 250,265 270,268 290,270 310,268 330,265 350,258 370,272
380,275 390,275 400,275 410,275 420,275 430,275 440,275 450,272
470,260 490,248 510,232 530,218 540,225 550,235 560,228 570,215 580,222
590,235 600,248 610,258 620,268 630,274
640,275 650,275 660,275 670,275 680,275 690,275 700,275 710,275 720,275 730,274
750,268 770,255 790,240 810,228 830,218 850,225 870,235 890,242
910,250 930,258 950,262 970,268 990,272 1010,274 1030,275 1050,275 1080,275 1092,275
" fill="rgba(140,180,255,0.12)" stroke="none"/>
<polyline points="
8,275
30,260 50,250 70,235 90,220 110,230 130,220 150,210 170,225 190,240 210,250
230,260 250,265 270,268 290,270 310,268 330,265 350,258 370,272
380,275 390,275 400,275 410,275 420,275 430,275 440,275 450,272
470,260 490,248 510,232 530,218 540,225 550,235 560,228 570,215 580,222
590,235 600,248 610,258 620,268 630,274
640,275 650,275 660,275 670,275 680,275 690,275 700,275 710,275 720,275 730,274
750,268 770,255 790,240 810,228 830,218 850,225 870,235 890,242
910,250 930,258 950,262 970,268 990,272 1010,274 1030,275 1050,275 1080,275 1092,275
" fill="none" stroke="#6b9fff" stroke-width="1.5" stroke-linejoin="round"/>
<!-- Waveform — bottom half (mirror) -->
<polygon points="
8,275
30,290 50,300 70,315 90,330 110,320 130,330 150,340 170,325 190,310 210,300
230,290 250,285 270,282 290,280 310,282 330,285 350,292 370,278
380,275 390,275 400,275 410,275 420,275 430,275 440,275 450,278
470,290 490,302 510,318 530,332 540,325 550,315 560,322 570,335 580,328
590,315 600,302 610,292 620,282 630,276
640,275 650,275 660,275 670,275 680,275 690,275 700,275 710,275 720,275 730,276
750,282 770,295 790,310 810,322 830,332 850,325 870,315 890,308
910,300 930,292 950,288 970,282 990,278 1010,276 1030,275 1050,275 1080,275 1092,275
" fill="rgba(140,180,255,0.06)" stroke="none"/>
<polyline points="
8,275
30,290 50,300 70,315 90,330 110,320 130,330 150,340 170,325 190,310 210,300
230,290 250,285 270,282 290,280 310,282 330,285 350,292 370,278
380,275 390,275 400,275 410,275 420,275 430,275 440,275 450,278
470,290 490,302 510,318 530,332 540,325 550,315 560,322 570,335 580,328
590,315 600,302 610,292 620,282 630,276
640,275 650,275 660,275 670,275 680,275 690,275 700,275 710,275 720,275 730,276
750,282 770,295 790,310 810,322 830,332 850,325 870,315 890,308
910,300 930,292 950,288 970,282 990,278 1010,276 1030,275 1050,275 1080,275 1092,275
" fill="none" stroke="#6b9fff" stroke-width="1" stroke-linejoin="round" opacity="0.4"/>
<!-- Word labels above waveform -->
<text x="40" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">Ninety</text>
<text x="120" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">percent</text>
<text x="210" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">of</text>
<text x="260" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">what</text>
<text x="320" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">...wasted.</text>
<text x="480" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">We</text>
<text x="540" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">fixed</text>
<text x="600" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">this.</text>
<text x="760" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">When</text>
<text x="830" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">a</text>
<text x="870" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">web</text>
<text x="930" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">agent</text>
<text x="1000" y="198" font-family="JetBrains Mono" font-size="10" fill="#6b9fff" font-weight="500">comes...</text>
<!-- Silence gap labels -->
<text x="395" y="215" text-anchor="middle" font-family="JetBrains Mono" font-size="9" fill="#ff6b35" font-weight="600">620ms</text>
<text x="395" y="227" text-anchor="middle" font-family="Inter" font-size="8" fill="rgba(255,107,53,0.6)">silence</text>
<text x="680" y="215" text-anchor="middle" font-family="JetBrains Mono" font-size="9" fill="#ff6b35" font-weight="600">880ms</text>
<text x="680" y="227" text-anchor="middle" font-family="Inter" font-size="8" fill="rgba(255,107,53,0.6)">silence</text>
<!-- Cut candidate lines -->
<line x1="410" y1="183" x2="410" y2="367" stroke="#059669" stroke-width="2" stroke-dasharray="6 4"/>
<line x1="680" y1="183" x2="680" y2="367" stroke="#059669" stroke-width="2" stroke-dasharray="6 4"/>
<!-- Audio event pill -->
<rect x="610" y="348" width="85" height="22" rx="11" fill="rgba(255,107,53,0.2)" stroke="#ff6b35" stroke-width="1"/>
<text x="652" y="363" text-anchor="middle" font-family="Inter" font-size="9" font-weight="700" fill="#ff6b35">(laughter)</text>
<!-- ===== TIME RULER ===== -->
<rect x="0" y="375" width="1100" height="30" rx="0" fill="#13131a"/>
<text x="8" y="394" font-family="JetBrains Mono" font-size="10" fill="#4a4a58">42.00s</text>
<line x1="8" y1="378" x2="8" y2="385" stroke="#4a4a58" stroke-width="1"/>
<text x="228" y="394" font-family="JetBrains Mono" font-size="10" fill="#4a4a58">43.20s</text>
<line x1="228" y1="378" x2="228" y2="385" stroke="#4a4a58" stroke-width="1"/>
<text x="448" y="394" font-family="JetBrains Mono" font-size="10" fill="#4a4a58">44.40s</text>
<line x1="448" y1="378" x2="448" y2="385" stroke="#4a4a58" stroke-width="1"/>
<text x="668" y="394" font-family="JetBrains Mono" font-size="10" fill="#4a4a58">45.60s</text>
<line x1="668" y1="378" x2="668" y2="385" stroke="#4a4a58" stroke-width="1"/>
<text x="888" y="394" font-family="JetBrains Mono" font-size="10" fill="#4a4a58">46.80s</text>
<line x1="888" y1="378" x2="888" y2="385" stroke="#4a4a58" stroke-width="1"/>
<text x="1048" y="394" font-family="JetBrains Mono" font-size="10" fill="#4a4a58">48.00s</text>
<line x1="1048" y1="378" x2="1048" y2="385" stroke="#4a4a58" stroke-width="1"/>
<!-- ===== CUT CANDIDATE LABELS ===== -->
<rect x="375" y="410" width="70" height="22" rx="4" fill="#059669"/>
<text x="410" y="425" text-anchor="middle" font-family="Inter" font-size="9" font-weight="800" fill="white">CUT</text>
<rect x="645" y="410" width="70" height="22" rx="4" fill="#059669"/>
<text x="680" y="425" text-anchor="middle" font-family="Inter" font-size="9" font-weight="800" fill="white">CUT</text>
<text x="550" y="450" text-anchor="middle" font-family="Inter" font-size="11" fill="#4a4a58">Silences ≥ 400ms = safe cut candidates. Words give boundary precision. Audio events mark beats to preserve.</text>
</svg>
</div>
</div>
</div>
<!-- ============ PIPELINE ============ -->
<div class="px-10 pb-8">
<div class="text-[10px] font-extrabold uppercase tracking-[3px] text-stone-400 mb-4">How it gets there</div>
<svg viewBox="0 0 1100 100" class="w-full" xmlns="http://www.w3.org/2000/svg">
<defs>
<marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" fill="#78716c"/></marker>
<marker id="arr-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" fill="#059669"/></marker>
</defs>
<rect x="0" y="15" width="145" height="55" rx="8" fill="white" stroke="#d6d3d1" stroke-width="1.5"/>
<text x="72" y="40" text-anchor="middle" font-family="Inter" font-size="12" font-weight="800" fill="#1c1917">Transcribe</text>
<text x="72" y="55" text-anchor="middle" font-family="JetBrains Mono" font-size="9" fill="#a8a29e">Scribe, parallel</text>
<line x1="150" y1="42" x2="178" y2="42" stroke="#78716c" stroke-width="1.5" marker-end="url(#arr)"/>
<rect x="183" y="15" width="120" height="55" rx="8" fill="white" stroke="#d6d3d1" stroke-width="1.5"/>
<text x="243" y="40" text-anchor="middle" font-family="Inter" font-size="12" font-weight="800" fill="#1c1917">Pack</text>
<text x="243" y="55" text-anchor="middle" font-family="JetBrains Mono" font-size="9" fill="#a8a29e">phrase-level .md</text>
<line x1="308" y1="42" x2="336" y2="42" stroke="#78716c" stroke-width="1.5" marker-end="url(#arr)"/>
<rect x="341" y="5" width="175" height="75" rx="10" fill="#1c1917"/>
<text x="428" y="32" text-anchor="middle" font-family="Inter" font-size="13" font-weight="900" fill="white">LLM Reasons</text>
<text x="428" y="50" text-anchor="middle" font-family="Inter" font-size="9" fill="#a8a29e">reads transcript</text>
<text x="428" y="64" text-anchor="middle" font-family="Inter" font-size="9" fill="#ff6b35" font-weight="600">calls timeline_view at decision points</text>
<line x1="521" y1="42" x2="549" y2="42" stroke="#059669" stroke-width="1.5" marker-end="url(#arr-g)"/>
<rect x="554" y="15" width="100" height="55" rx="8" fill="#f0fdf4" stroke="#059669" stroke-width="1.5"/>
<text x="604" y="40" text-anchor="middle" font-family="Inter" font-size="12" font-weight="800" fill="#059669">EDL</text>
<text x="604" y="55" text-anchor="middle" font-family="JetBrains Mono" font-size="9" fill="#059669">.json</text>
<line x1="659" y1="42" x2="687" y2="42" stroke="#78716c" stroke-width="1.5" marker-end="url(#arr)"/>
<rect x="692" y="15" width="120" height="55" rx="8" fill="white" stroke="#d6d3d1" stroke-width="1.5"/>
<text x="752" y="40" text-anchor="middle" font-family="Inter" font-size="12" font-weight="800" fill="#1c1917">Render</text>
<text x="752" y="55" text-anchor="middle" font-family="JetBrains Mono" font-size="9" fill="#a8a29e">ffmpeg pipeline</text>
<line x1="817" y1="42" x2="845" y2="42" stroke="#78716c" stroke-width="1.5" marker-end="url(#arr)"/>
<rect x="850" y="5" width="145" height="75" rx="10" fill="#ff6b35"/>
<text x="922" y="32" text-anchor="middle" font-family="Inter" font-size="13" font-weight="900" fill="white">Self-Eval</text>
<text x="922" y="50" text-anchor="middle" font-family="Inter" font-size="9" fill="rgba(255,255,255,0.8)">timeline_view on output</text>
<text x="922" y="64" text-anchor="middle" font-family="Inter" font-size="9" fill="rgba(255,255,255,0.8)">at every cut point</text>
<!-- Loop back -->
<path d="M 922 82 L 922 96 L 428 96 L 428 82" fill="none" stroke="#dc2626" stroke-width="1.5" stroke-dasharray="5 3"/>
<text x="675" y="94" text-anchor="middle" font-family="Inter" font-size="9" font-weight="700" fill="#dc2626">issue? fix → re-render (max 3)</text>
</svg>
</div>
<!-- Bottom bar -->
<div class="bg-stone-900 px-10 py-4 text-center">
<span class="text-stone-400 text-[12px]"><strong class="text-white">Audio transcript</strong> is the primary surface.</span>
<span class="text-[12px] text-stone-500 mx-3">•</span>
<span class="text-stone-400 text-[12px]"><strong class="text-accent">Visual composites</strong> are on-demand at decision points.</span>
<span class="text-[12px] text-stone-500 mx-3">•</span>
<span class="text-stone-400 text-[12px]"><strong class="text-white">Verify output</strong> before presenting.</span>
</div>
</div>
</body>
</html>
[project]
name = "video-use"
version = "0.1.0"
description = "Conversation-driven video editor skill for Claude Code"
license = { file = "LICENSE" }
requires-python = ">=3.10"
dependencies = [
"requests",
"librosa",
"matplotlib",
"pillow",
"numpy",
]
[project.optional-dependencies]
animations = ["manim"]
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
py-modules = []
<p align="center"> <img src="static/video-use-banner.png" alt="video-use" width="100%"> </p>
video-use
Introducing video-use — edit videos with Claude Code. 100% open source.
Drop raw footage in a folder, chat with Claude Code, get final.mp4 back. Works for any content — talking heads, montages, tutorials, travel, interviews — without presets or menus.
What it does
- Cuts out filler words (
umm,uh, false starts) and dead space between takes - Auto color grades every segment (warm cinematic, neutral punch, or any custom ffmpeg chain)
- 30ms audio fades at every cut so you never hear a pop
- Burns subtitles in your style — 2-word UPPERCASE chunks by default, fully customizable
- Generates animation overlays via HyperFrames, Remotion, Manim, or PIL — spawned in parallel sub-agents, one per animation
- Self-evaluates the rendered output at every cut boundary before showing you anything
- Persists session memory in
project.mdso next week's session picks up where you left off
Setup prompt
Paste into Claude Code, Codex, Hermes, Openclaw, or any agent with shell access:
Set up https://github.com/browser-use/video-use for me.
Read install.md first to install this repo, wire up ffmpeg, register the skill with whichever agent you're running under, and set up the ElevenLabs API key — ask me to paste it when you need it. Then read SKILL.md for daily usage, and always read helpers/ because that's where the editing scripts live. After install, don't transcribe anything on your own — just tell me it's ready and wait for me to drop footage into a folder.The agent handles the clone, dependencies, skill registration, and prompts you once for your ElevenLabs API key (grab one at elevenlabs.io/app/settings/api-keys).
Then point your agent at a folder of raw takes:
cd /path/to/your/videos
claude # or codex, hermes, etc.For always-on editing from your own VPS or Telegram, run the agent through Browser Use Box. Watch the 15-second demo.
And in the session:
edit these into a launch video
It inventories the sources, proposes a strategy, waits for your OK, then produces edit/final.mp4 next to your sources. All outputs live in <videos_dir>/edit/ — the skill directory stays clean.
Manual install
If you'd rather do it by hand:
# 1. Clone and symlink into your agent's skills directory
git clone https://github.com/browser-use/video-use ~/Developer/video-use
ln -sfn ~/Developer/video-use ~/.claude/skills/video-use # Claude Code
# ln -sfn ~/Developer/video-use ~/.codex/skills/video-use # Codex
# 2. Install deps
cd ~/Developer/video-use
uv sync # or: pip install -e .
brew install ffmpeg # required
brew install yt-dlp # optional, for downloading online sources
# 3. Add your ElevenLabs API key
cp .env.example .env
$EDITOR .env # ELEVENLABS_API_KEY=...How it works
The LLM never watches the video. It reads it — through two layers that together give it everything it needs to cut with word-boundary precision.
<p align="center"> <img src="static/timeline-view.svg" alt="timeline_view composite — filmstrip + speaker track + waveform + word labels + silence-gap cut candidates" width="100%"> </p>
Layer 1 — Audio transcript (always loaded). One ElevenLabs Scribe call per source gives word-level timestamps, speaker diarization, and audio events ((laughter), (applause), (sigh)). All takes pack into a single ~12KB takes_packed.md — the LLM's primary reading view.
## C0103 (duration: 43.0s, 8 phrases)
[002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
[006.08-006.74] S0 We fixed this.Layer 2 — Visual composite (on demand). timeline_view produces a filmstrip + waveform + word labels PNG for any time range. Called only at decision points — ambiguous pauses, retake comparisons, cut-point sanity checks.
Naive approach: 30,000 frames × 1,500 tokens = 45M tokens of noise.
Video Use: 12KB text + a handful of PNGs.
Same idea as browser-use giving an LLM a structured DOM instead of a screenshot — but for video.
Pipeline
Transcribe ──> Pack ──> LLM Reasons ──> EDL ──> Render ──> Self-Eval
│
└─ issue? fix + re-render (max 3)The self-eval loop runs timeline_view on the _rendered output_ at every cut boundary — catches visual jumps, audio pops, hidden subtitles. You see the preview only after it passes.
Design principles
1. Text + on-demand visuals. No frame-dumping. The transcript is the surface. 2. Audio is primary, visuals follow. Cuts come from speech boundaries and silence gaps. 3. Ask → confirm → execute → self-eval → persist. Never touch the cut without strategy approval. 4. Zero assumptions about content type. Look, ask, then edit. 5. 12 hard rules, artistic freedom elsewhere. Production-correctness is non-negotiable. Taste isn't.
See `SKILL.md` for the full production rules and editing craft.
Manim Video Skill
Production pipeline for mathematical and technical animations using Manim Community Edition.
What it does
Creates 3Blue1Brown-style animated videos from text prompts. The agent handles the full pipeline: creative planning, Python code generation, rendering, scene stitching, and iterative refinement.
Use cases
- Concept explainers — "Explain how neural networks learn"
- Equation derivations — "Animate the proof of the Pythagorean theorem"
- Algorithm visualizations — "Show how quicksort works step by step"
- Data stories — "Animate our before/after performance metrics"
- Architecture diagrams — "Show our microservice architecture building up"
Prerequisites
Python 3.10+, Manim CE (pip install manim), LaTeX, ffmpeg.
bash skills/creative/manim-video/scripts/setup.shCamera and 3D Reference
MovingCameraScene (2D Camera Control)
class ZoomExample(MovingCameraScene):
def construct(self):
circle = Circle(radius=2, color=BLUE)
self.play(Create(circle))
# Zoom in
self.play(self.camera.frame.animate.set(width=4).move_to(circle.get_top()), run_time=2)
self.wait(2)
# Zoom back out
self.play(self.camera.frame.animate.set(width=14.222).move_to(ORIGIN), run_time=2)Camera Operations
self.camera.frame.animate.set(width=6) # zoom in
self.camera.frame.animate.set(width=20) # zoom out
self.camera.frame.animate.move_to(target) # pan
self.camera.frame.save_state() # save
self.play(Restore(self.camera.frame)) # restoreThreeDScene
class ThreeDExample(ThreeDScene):
def construct(self):
self.set_camera_orientation(phi=60*DEGREES, theta=-45*DEGREES)
axes = ThreeDAxes()
surface = Surface(
lambda u, v: axes.c2p(u, v, np.sin(u) * np.cos(v)),
u_range=[-PI, PI], v_range=[-PI, PI], resolution=(30, 30)
)
surface.set_color_by_gradient(BLUE, GREEN, YELLOW)
self.play(Create(axes), Create(surface))
self.begin_ambient_camera_rotation(rate=0.2)
self.wait(5)
self.stop_ambient_camera_rotation()Camera Control in 3D
self.set_camera_orientation(phi=70*DEGREES, theta=-45*DEGREES)
self.move_camera(phi=45*DEGREES, theta=30*DEGREES, run_time=2)
self.begin_ambient_camera_rotation(rate=0.2)3D Mobjects
sphere = Sphere(radius=1).set_color(BLUE).set_opacity(0.7)
cube = Cube(side_length=2, fill_color=GREEN, fill_opacity=0.5)
arrow = Arrow3D(start=ORIGIN, end=[2, 1, 1], color=RED)
# 2D text facing camera:
label = Text("Label", font_size=30)
self.add_fixed_in_frame_mobjects(label)Parametric Curves
helix = ParametricFunction(
lambda t: [np.cos(t), np.sin(t), t / (2*PI)],
t_range=[0, 4*PI], color=YELLOW
)When to Use 3D
- Surfaces, vector fields, spatial geometry, 3D transforms
When NOT to Use 3D
- 2D concepts, text-heavy scenes, flat data (bar charts, time series)
ZoomedScene — Inset Zoom
Show a magnified inset of a detail while keeping the full view visible:
class ZoomExample(ZoomedScene):
def __init__(self, **kwargs):
super().__init__(
zoom_factor=0.3, # how much of the scene the zoom box covers
zoomed_display_height=3, # size of the inset
zoomed_display_width=3,
zoomed_camera_frame_starting_position=ORIGIN,
**kwargs
)
def construct(self):
self.camera.background_color = BG
# ... create your scene content ...
# Activate the zoom
self.activate_zooming()
# Move the zoom frame to a point of interest
self.play(self.zoomed_camera.frame.animate.move_to(detail_point))
self.wait(2)
# Deactivate
self.play(self.get_zoomed_display_pop_out_animation(), rate_func=lambda t: smooth(1-t))Use cases: zooming into a specific term in an equation, showing fine detail in a diagram, magnifying a region of a plot.
LinearTransformationScene — Linear Algebra
Pre-built scene with basis vectors and grid for visualizing matrix transformations:
class LinearTransformExample(LinearTransformationScene):
def __init__(self, **kwargs):
super().__init__(
show_coordinates=True,
show_basis_vectors=True,
**kwargs
)
def construct(self):
matrix = [[2, 1], [1, 1]]
# Add a vector before applying the transform
vector = self.get_vector([1, 2], color=YELLOW)
self.add_vector(vector)
# Apply the transformation — grid, basis vectors, and your vector all transform
self.apply_matrix(matrix)
self.wait(2)This produces the signature 3Blue1Brown "Essence of Linear Algebra" look — grid lines deforming, basis vectors stretching, determinant visualized through area change.
#!/usr/bin/env bash
set -euo pipefail
G="\033[0;32m"; R="\033[0;31m"; N="\033[0m"
ok() { echo -e " ${G}+${N} $1"; }
fail() { echo -e " ${R}x${N} $1"; }
echo ""; echo "Manim Video Skill — Setup Check"; echo ""
errors=0
command -v python3 &>/dev/null && ok "Python $(python3 --version 2>&1 | awk '{print $2}')" || { fail "Python 3 not found"; errors=$((errors+1)); }
python3 -c "import manim" 2>/dev/null && ok "Manim $(manim --version 2>&1 | head -1)" || { fail "Manim not installed: pip install manim"; errors=$((errors+1)); }
command -v pdflatex &>/dev/null && ok "LaTeX (pdflatex)" || { fail "LaTeX not found (macOS: brew install --cask mactex-no-gui)"; errors=$((errors+1)); }
command -v ffmpeg &>/dev/null && ok "ffmpeg" || { fail "ffmpeg not found"; errors=$((errors+1)); }
echo ""
[ $errors -eq 0 ] && echo -e "${G}All prerequisites satisfied.${N}" || echo -e "${R}$errors prerequisite(s) missing.${N}"
echo ""
Related skills
How it compares
Pick video-use when transcript-driven conversational iteration beats hand-writing FFmpeg filter graphs for creator-style edits.
FAQ
What is video-use?
Edit any video by conversation. Transcribe, cut, color grade, generate overlay animations, burn subtitles — for talking heads, montages, tutorials, travel, interviews. No presets,
When should I use video-use?
Edit any video by conversation. Transcribe, cut, color grade, generate overlay animations, burn subtitles — for talking heads, montages, tutorials, travel, interviews. No presets,
Is video-use safe to install?
Review the Security Audits panel on this page before production use.