
Oma Video
- 19 installs
- 41 repo stars
- Updated August 4, 2026
- gracefullight/stock-checker
Generates narrated, captioned mp4 videos in shorts, explainer, or demo modes via a key-optional provider router and a vendored Remotion compositor.
About
A video generator that composes scripts, narration, visuals, and captions into reproducible run directories, routing across shorts (9:16), explainer (16:9), and screen-capture demo modes. A developer uses it to turn a topic, README, or capture into a finished short-form or explainer video.
- Key-optional dispatch with a key-free fallback for every capability
- Deterministic render-spec makes re-renders byte-stable
Oma Video by the numbers
- 19 all-time installs (skills.sh)
- Ranked #1,001 of 1,335 Generative Media skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/gracefullight/stock-checker --skill oma-videoAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 19 |
|---|---|
| repo stars | ★ 41 |
| Last updated | August 4, 2026 |
| Repository | gracefullight/stock-checker ↗ |
What it does
Generates narrated, captioned mp4 videos in shorts, explainer, or demo modes via a key-optional provider router and a vendored Remotion compositor.
Files
Video Agent - Short-form, Explainer & Demo Router
Scheduling
Goal
Generate finished .mp4 videos through a key-optional, 3-tier (CLI-first / MCP / guided) provider router while preserving deterministic asset buses (script -> timing -> render-spec), reproducible manifests, cost controls, and capture-path safety.
Intent signature
- User asks for a short-form video, shorts/reels clip, TikTok/YouTube Short, explainer, demo, walkthrough, or screencast.
- User wants a topic, README, code, or data turned into a narrated, captioned video.
- Another skill needs shared video-generation infrastructure (script -> assets -> render).
When to use
- Generating short-form video (shorts / reels) from a topic or brief (
--mode shorts, 9:16) - Generating an explainer from a README, code, or data set (
--mode explainer, 16:9 / 9:16) - Producing a demo / walkthrough from a screen capture file (
--mode demo --source file, 16:9) - Supervised headed web-app capture of any URL (
--mode demo --source web --url <url>) — a human drives the on-screen flow; the tool only opens a headed browser and records. Example categories are equal and illustrative only: demo, walkthrough, onboarding clip, bug repro, app-review screencast. - Re-rendering an existing run deterministically from
render-spec.json - Other skills needing video-generation infrastructure (shared invocation via
--format json)
When NOT to use
- Generating a single still image -> use
oma-image - Generating a slide deck / presentation -> use
oma-slide(this skill calls it internally for explainer frames) - Generating speech audio only (no video) -> use
oma-voice - Non-linear video editing of an existing finished mp4 -> out of scope (OpenCut-MCP deferred)
- Supervised headed web capture is in-scope (
--source web); live streaming is out of scope
Expected inputs
- A brief (topic / README path / data) plus optional mode, aspect, locale, captions, visual, voice, music, duration, compositor, capture path, seed
- For
demo--source file: a screen-capture file path (--capture) or Cap availability - For
demo--source web: a target--url(any URL — local/staging/prod), optional--device/--ready-selector/--show-cursor/--polish/--capture-timeout; capture size is derived from--aspect/--device(no hardcoded size); a resolvable Playwright + an interactive TTY (else the run falls back to the guided protocol) - Authentication/environment state for oma-voice (Voicebox MCP), oma-image vendors, and optional Pexels / Pixelle keys
Expected outputs
- A run directory under
.agents/results/videos/<timestamp>-<shortid>-<mode>/ - Deterministic asset bus:
script.json,timing.json,render-spec.json audio/,visuals/,captions.srt/captions.vtt, the rendered<mode>-<slug>.mp4manifest.jsonwith providers, asset hashes, cost breakdown, and exit code
Dependencies
oma video generateCLI + central error module (exit codes aligned withoma search fetch)- oma-voice (Voicebox MCP), oma-image, oma-slide as key-free fallback providers
- Vendored Remotion project at
resources/remotion/(compositor) resources/vendor-matrix.md,resources/execution-protocol.md,resources/prompt-tips.md,config/video-config.yaml
Control-flow features
- Branches by mode (shorts / explainer / demo), aspect, visual strategy, provider availability, cost threshold, capture requirement, and path safety
- Runs a per-capability fallback chain (real key/resource -> key-free fallback) per backend rule 11
- Reads briefs/captures and writes assets, render-spec, and manifests
- Calls external resources: Voicebox MCP, oma-image vendors, Remotion toolchain, optional Pexels / Pixelle
Structural Flow
Entry
1. Validate that the brief carries enough mode/topic signal (or infer the mode from keywords). 2. For demo, confirm a capture path exists (or Cap is available); otherwise enter the guided protocol. 3. Resolve defaults from config/video-config.yaml -> env vars -> CLI flags; check output path safety and limits.
Scenes
1. PREPARE: Resolve mode/aspect/locale, clarify or amplify the brief, choose the visual + compositor strategy. 2. ACQUIRE: Probe provider availability (voice / visual / caption / compositor), validate capture path, check cost. 3. ACT: Run the mode pipeline — script -> (voice ∥ visuals ∥ captions) -> render-spec -> compositor render. 4. VERIFY: Validate every asset-bus schema, manifest hashes, exit code, and the output mp4. 5. FINALIZE: Return the run-dir path, the mp4 path, and any provider/coverage warnings.
Transitions
- If the brief lacks a clear mode, infer from keywords (shorts/reels -> shorts; README/code -> explainer; capture -> demo) and show the user the inferred plan before generating.
- If the selected visual provider key is absent (Pexels / Pixelle), fall through the chain to the key-free oma-image stills + Ken Burns fallback and annotate coverage.
- If
demo--source webhas a--url, dispatch the headed web-capture path (human-driven flow, ENTER to stop); if Playwright is unresolvable OR there is no interactive TTY, fall back to the guided protocol (no hang). - If
demo--source filehas no capture and Cap is unavailable, emit the guided capture protocol and stop (exit code maps to capture-required). - If estimated cost (Pixelle / RunningHub credits) exceeds the guardrail, require confirmation unless bypassed.
Failure and recovery
- If a provider is unavailable, try the next provider in the capability's
order; only chain exhaustion is a stage failure. - If the Remotion toolchain is not bootstrapped, point the user to
oma video doctor; fall back to the MPT compositor where applicable. - If Voicebox MCP is down, fall back through voicebox-stt -> whisper.cpp -> estimated timing (still produces captions).
- If the brief locale is non-source, translate via oma-translator (key-free); if absent, warn and keep source text.
Exit
- Success:
<mode>-<slug>.mp4andmanifest.jsonexist in the run directory; all schemas validate. - Partial success: video renders with a key-free fallback in place of a paid provider; coverage is annotated in warnings.
- Failure: no video is produced and the route/cost/capture/auth/safety blocker is explicit in the exit code + manifest.
Logical Operations
Actions
| Action | SSL primitive | Evidence |
|---|---|---|
| Validate brief + mode | VALIDATE | Clarification protocol, mode inference |
| Select provider strategy | SELECT | Vendor matrix, providers.*.order, availability |
| Read brief / capture | READ | Brief text, --capture path |
| Generate script | CALL_TOOL | AgentScriptProvider -> script.json |
| Synthesize narration + timing | CALL_TOOL | oma-voice -> audio/*.wav + timing.json |
| Produce visuals | CALL_TOOL | oma-image / oma-slide / stock -> visuals/* |
| Build captions | WRITE | key-free captions.srt / .vtt from timing |
| Compose render-spec | WRITE | render-spec.json (determinism boundary) |
| Render video | CALL_TOOL | Remotion / MPT compositor -> <mode>-<slug>.mp4 |
| Validate result | VALIDATE | Schema parse, manifest hashes, exit code |
| Report output | NOTIFY | Run-dir + mp4 path summary |
Tools and instruments
oma video generate,oma video doctor,oma video list-providers,oma video render- Provider adapters: AgentScript, oma-voice, oma-image, oma-slide, Pexels, Pixelle, oma-captions, Cap, Remotion, MPT
- Vendored Remotion project (
resources/remotion/), prompt tips, vendor matrix, video config
Canonical command path
oma video doctor
oma video generate "<brief>" --mode shorts --aspect auto --captions tiktok --format jsonExplainer from a README, with a deterministic seed:
oma video generate "explain this project" --mode explainer --aspect 16:9 --seed 42 --out ./outDemo from a screen capture:
oma video generate "feature walkthrough" --mode demo --capture <absolute-path>.mp4Deterministic re-render from an existing run:
oma video render .agents/results/videos/20260603-143052-ab12cd-shortsResource scope
| Scope | Resource target |
|---|---|
LOCAL_FS | Briefs, captures, assets, render-spec, run dir, manifests |
PROCESS | oma-image / oma-slide CLIs, Remotion / MPT, Cap CLI, Playwright web-capture driver (subprocess) |
NETWORK | Voicebox MCP (localhost), oma-image vendor APIs, the user-supplied --url for web capture (masked in logs/manifest), optional Pexels / Pixelle / RunningHub |
CREDENTIALS | oma-image vendor auth, optional PEXELS_API_KEY / RUNNINGHUB_API_KEY. Web capture handles NO credentials — a human logs in if the flow needs it; nothing is stored or printed. |
Preconditions
- Brief carries enough signal for the mode, or the user approves the inferred/amplified plan.
- Output path is inside
$PWD(or--allow-external-outis set). - For
demo--source file: the capture path exists, is absolute/$PWD-guarded, and is a valid format. - For
demo--source web: a--urlis supplied (elseSchemaValidationError); a resolvable Playwright + an interactive TTY exist (else the run falls back to the guided protocol). - Required provider availability holds for the chosen (non-fallback) path, or the fallback is acceptable.
Effects and side effects
- Creates a run directory with assets, render-spec, captions, mp4, and manifest.
- oma-voice plays narration on the speakers as a side effect of synthesis.
- May call paid or rate-limited providers (Pexels / Pixelle / RunningHub) only when keys are present.
Guardrails
1. Clarify or infer before invoking: if the mode/topic is ambiguous, infer the mode from keywords and show the user the plan, or ask. Do NOT silently render from a vague brief. See Clarification Protocol below. 2. Key-optional dispatch (backend rule 11): every external capability has a real (key/resource) path AND a key-free fallback. Paid providers (Pexels, Pixelle) auto-enable only when their env key is present; otherwise the chain falls through to oma-image stills + Ken Burns. Default providers (oma-voice local, oma-image, oma-slide, Remotion) are key-free, so auto-triggering on keywords is safe. 3. Cost guardrail: confirm before runs whose estimated cost is >= cost.guardrail_usd ($0.20, configurable) or --max-usd. --yes / OMA_VIDEO_YES=1 bypass. Local/free paths carry zero cost. 4. Path safety: output paths outside $PWD require --allow-external-out. --capture is absolutized, $PWD-guarded, and format-validated; external assets are copied into the run dir and hashed (no URL refs). 5. Cancellable: SIGINT/SIGTERM aborts in-flight provider calls, the render, and the orchestrator. 6. Deterministic outputs: render-spec.json + asset files (+ seed + embedded Pretendard font) are the determinism boundary. Re-rendering the same render-spec is byte-stable; OMA_VIDEO_MOCK=1 replays golden fixtures. 7. Limits: limits.max_duration_sec = 180, limits.max_scenes = 40 (wall-time + memory bound). 8. Community-MCP consent: Pixelle-MCP is off by default and requires one-time explicit consent + source review before connecting; RunningHub credits gate on --max-usd. 9. Demo is human-in-the-loop: capture is performed by a human. For --source file the skill guides but does not screen-record autonomously; for --source web the tool only opens a headed browser and records while the human drives the entire on-screen flow (interactive ENTER to stop). The mechanism prescribes nothing about what the flow is or what the recording is for. 10. Web-capture security: NO credential automation of any kind — if a flow needs a login, a human performs it. The driver runs as a subprocess under the resolved Playwright (never imported into the CLI). The --url and any query tokens are masked in logs and in manifest.json; credentials are never stored or printed. Recording and all outputs are confined to the run dir. On-screen sensitive input is captured as-is — the user controls the flow. Multi-page navigation (popup / new tab / redirect) is recorded generically, with no assumption about the flow's shape. 11. Web capture is key-optional + non-blocking: web capture is the real branch; the guided protocol is the fallback when Playwright is unresolvable OR there is no interactive TTY (CI / -y / no stdin) — the run falls back to guided and never hangs. Live capture is outside the determinism boundary, so the manifest records nondeterministic: true. 12. Exit codes align with `oma search fetch` (0 ok, 1 generic, 2 safety, 3 not-found, 4 invalid-input, 5 auth-required, 6 timeout).
Clarification Protocol
Before invoking oma video generate, the calling agent runs this checklist. If any answer is "no / unknown", clarify or infer-and-confirm with the user first.
Required signal (must be present or inferable):
- [ ] Mode: shorts / explainer / demo? Infer from keywords (shorts/reels/쇼츠/릴스 -> shorts; README/code/data/explain/설명 -> explainer; demo/walkthrough/capture/데모 -> demo).
- [ ] Topic / source: what is the video about? (a topic, a README/code path, a capture file, or — for
demo--source web— a--url) - [ ] For `demo`:
--source file(a--capturepath) or--source web(a--url)? For--source web, state up front that a human drives the on-screen flow and that the tool never automates any login.
Strongly recommended (ask if absent AND not inferable):
- [ ] Aspect:
9:16(shorts/reels),16:9(explainer/demo),1:1, orauto(snaps to the mode default). - [ ] Locale: narration + caption language (default from config; translated via oma-translator when non-source).
- [ ] Captions:
tiktok(centered, animated),lower-third, ornone. - [ ] Duration: target seconds (<= 180) or
auto(derived from the script). - [ ] Voice / music: voice profile or
none; musicupbeat/calm/none.
Amplification shortcut. For a one-line brief (e.g. "shorts about Jeju coffee"), do not pop a questionnaire if the request is genuinely simple. Instead amplify inline and show the user the inferred plan before invoking:
User: "make a short about Jeju coffee"
Agent: "I'll generate this as: mode `shorts`, 9:16, ~30s, oma-image stills with Ken Burns, TikTok captions, locale `en`, calm music. Proceed, or adjust mode/aspect/voice?"
Skip clarification when the user authored a full brief (mode + topic + aspect + captions). Respect their flags verbatim.
Output language. Narration and on-screen text are authored in the requested locale. Image-generation prompts passed to oma-image are sent in English (image models are trained predominantly on English captions); translate the user's request and show the translated version during amplification.
Modes
| Mode | Aspect | Source | Default visual | Compositor | Output |
|---|---|---|---|---|---|
shorts | 9:16 | synthetic (topic) | oma-image stills + Ken Burns; Pexels (key) · Pixelle AIGC (key) opt | Remotion · MPT alt | shorts-<slug>.mp4 |
explainer | 16:9 / 9:16 | README · code · data | oma-slide frames + oma-image diagrams + code | Remotion (deterministic) | explainer-<slug>.mp4 |
demo | 16:9 | --source file (Cap / capture file) · --source web (headed browser at --url) | raw footage (default) · Remotion intro · zoom · callouts (--polish) | Remotion polish | demo-<slug>.mp4 |
3-Tier Integration
| Tier | Surface | Providers | Trigger |
|---|---|---|---|
| 1 | CLI-first (subprocess, deterministic) | Remotion render, MPT, oma-image, oma-slide, oma-voice | always available (key-free defaults) |
| 2 | MCP | Voicebox MCP (voice/timing), Pixelle-MCP (AIGC, off by default) | MCP server reachable; Pixelle needs explicit consent + key |
| 3 | Guided (human-in-the-loop) | Playwright headed web capture (--source web, human drives the flow), Cap (capture), openscreen fallback | demo mode; web capture needs a resolvable Playwright + a TTY (else guided protocol) |
Invocation
Standalone
/oma-video make a 30s short about Jeju coffee
/oma-video --mode explainer --aspect 16:9 explain this project from the README
/oma-video --mode demo --source file --capture ~/recordings/walkthrough.mp4 feature demo
/oma-video --mode demo --source web --url http://localhost:3000 record my app flow
/oma-video --mode demo --source web --url <url> --ready-selector "#app" --polish onboarding clipShell CLI
oma video generate "<brief>" [--mode shorts|explainer|demo] \
[--aspect 9:16|16:9|1:1|auto] [--locale <lang>] \
[--captions tiktok|lower-third|none] \
[--visual auto|generate|stock|aigc|slide] \
[--voice <profile>|none] [--music upbeat|calm|none] \
[--duration <sec>|auto] [--compositor remotion|mpt] \
[--capture <path>] \
[--source file|web] [--url <url>] [--device <name>] \
[--ready-selector <css>] [--show-cursor] [--polish] \
[--capture-timeout <sec>] [--capture-stop duration:<sec>|selector:<css>] \
[--out <dir>] [--allow-external-out] \
[--max-usd <n>] [--seed <n>] [--timeout 600] [-y] \
[--dry-run] [--format text|json] [--no-brief-in-manifest]
# --source web: headed browser at --url; capture size derived from --aspect/--device (no hardcoded size).
# A human drives the on-screen flow; press ENTER to stop. NO credential automation. --url/tokens masked.
# Non-interactive (CI / -y / no TTY) or unresolvable Playwright -> falls back to the guided protocol (no hang).
# --capture-stop gives CI a non-interactive stop (duration / selector) in place of the ENTER prompt.
oma video doctor # Node/Chromium/FFmpeg · Voicebox MCP · oma-image vendors · Pixelle-MCP · Cap · Playwright (web capture)
oma video doctor --install-playwright # one-time: npm i playwright + chromium (web capture)
oma video list-providers # availability + key/fallback status
oma video render <runDir> # re-render from render-spec.json (deterministic)Shared Infrastructure (from other skills)
Other skills call oma video generate --format json and parse the JSON envelope ({exitCode, runDir, manifestPath, outputs}) from stdout. The deterministic boundary is render-spec.json + assets, so a downstream consumer can re-render via oma video render <runDir> without re-running script/voice/visual generation.
Output Layout
.agents/results/videos/
└── 20260603-143052-ab12cd-shorts/ # {timestamp}-{shortid}-{mode}
├── script.json # determinism boundary start
├── timing.json
├── render-spec.json # deterministic compute boundary
├── audio/
│ └── narration-01.wav …
├── visuals/
│ └── scene-01.jpg …
├── captions.srt
├── captions.vtt
├── shorts-<slug>.mp4
└── manifest.json # reproducibility recordReferences
Follow resources/execution-protocol.md step by step. See resources/vendor-matrix.md for provider precheck + fallback-chain rules. Use resources/prompt-tips.md for writing effective briefs per mode. Before submitting, run resources/checklist.md. The vendored Remotion compositor lives at resources/remotion/ (see its README.md). The web-capture driver lives at resources/playwright/record.mjs (runs as a subprocess under the resolved Playwright; never imported into the CLI). The MPT fallback compositor driver lives at resources/mpt/driver.py (consumed by the CLI's mpt-project internals).
Configuration
Project-specific settings: config/video-config.yaml. Env vars: OMA_VIDEO_DEFAULT_MODE, OMA_VIDEO_DEFAULT_OUT, OMA_VIDEO_YES, PEXELS_API_KEY, RUNNINGHUB_API_KEY (+ POLLINATIONS_API_KEY via oma-image), OMA_VIDEO_MOCK, OMA_VIDEO_PLAYWRIGHT_DIR (web-capture Playwright override), OMA_VIDEO_PWTEST (opt-in web-capture e2e).
- Execution steps:
resources/execution-protocol.md - Vendor matrix:
resources/vendor-matrix.md - Prompt tips:
resources/prompt-tips.md - Checklist:
resources/checklist.md - Remotion compositor:
resources/remotion/README.md - Context loading:
../_shared/core/context-loading.md
# oma-video skill configuration.
# Mirrors cli/commands/video/config/video-config.yaml (the CLI is the source of
# truth; keep these in sync). Precedence at runtime: this config -> env vars ->
# CLI flags (lowest to highest).
default_output_dir: .agents/results/videos
default_mode: shorts
default_aspect: auto
default_locale: en
default_captions: tiktok
default_visual: auto
default_voice: none
default_music: none
default_compositor: remotion
default_timeout_sec: 600
providers:
script:
order: [agent-script]
voice:
order: [oma-voice]
visual:
order: [oma-image, pexels, pixelle]
caption:
order: [oma-captions]
capture:
order: [cap]
compositor:
order: [remotion, mpt]
# Paid providers — auto-enabled only when their env key is present (backend
# rule 11: real path gated on key, key-free fallback otherwise).
pexels:
enabled: false
envVar: PEXELS_API_KEY
pixelle:
enabled: false
envVar: RUNNINGHUB_API_KEY
cost:
guardrail_usd: 0.20
limits:
max_duration_sec: 180
max_scenes: 40
naming:
single_folder_pattern: "{timestamp}-{shortid}-{mode}"
Checklist: before you run oma video generate
- [ ] Mode is set or inferable (shorts / explainer / demo) and the topic/source is clear (see
prompt-tips.md). - [ ]
--aspectmatches the mode (9:16 shorts, 16:9 explainer/demo) or isauto. - [ ] Duration ≤ 180s and the script stays ≤ 40 scenes.
- [ ]
--outis inside the project, or--allow-external-outis set. - [ ] For
demo:--capture <path>exists, is absolute + inside$PWD, and is a valid video format — or you accept the guided protocol. - [ ] Provider readiness checked with
oma video doctor(Node/Chromium/FFmpeg · Voicebox MCP · oma-image vendors · Pixelle-MCP · Cap). - [ ] Paid visuals (Pexels / Pixelle) have their env key set, OR you accept the key-free oma-image fallback.
- [ ] Pixelle-MCP, if used: one-time explicit consent + source review done;
--max-usdset for RunningHub credits. - [ ] Estimated cost is acceptable. Run
--dry-runfirst for unfamiliar combinations. - [ ] Secrets are not in the brief, or
--no-brief-in-manifestis set.
Checklist: after the run
- [ ] The run directory contains
script.json,timing.json,render-spec.json,captions.srt/.vtt, the<mode>-<slug>.mp4, andmanifest.json. - [ ] Every asset-bus schema validates (
schemaVersion: "1.0"). - [ ]
manifest.jsonrecords each provider, assetsha256hashes, cost breakdown, and the exit code. - [ ]
warnings[]annotates any fallback taken (e.g. Pexels key absent -> oma-image stills, translator absent -> source locale). - [ ] External assets were copied into the run dir and hashed (no URL refs).
- [ ] The mp4 plays (or, on the toolchain-free path, the deterministic placeholder is present and reproducible).
- [ ] Re-rendering with
oma video render <runDir>reproduces the same output fromrender-spec.json. - [ ] If results are consumed downstream, the consumer parses
--format jsonstdout rather than re-reading the manifest file.
Video Agent - Execution Protocol
Step -1: Clarify / Infer Mode (agent-side, before oma video generate)
Run the Clarification Protocol in SKILL.md before shelling out. Infer the mode from keywords (shorts/reels/쇼츠/릴스 -> shorts; README/code/data/explain/설명 -> explainer; demo/walkthrough/capture/데모 -> demo) and show the user the inferred plan when the brief is a one-liner.
Step 0: Parse Request
1. Extract the brief and flags from the invocation. 2. Resolve defaults from config/video-config.yaml -> env vars -> CLI flags (lowest to highest precedence). 3. Validate:
mode∈ {shorts,explainer,demo}.aspect∈ {9:16,16:9,1:1,auto} (autosnaps to the mode default: shorts -> 9:16, explainer/demo -> 16:9).captions∈ {tiktok,lower-third,none};visual∈ {auto,generate,stock,aigc,slide}.music∈ {upbeat,calm,none};compositor∈ {remotion,mpt}.duration≤limits.max_duration_sec(180); resultingscenes≤limits.max_scenes(40).outis inside$PWDunless--allow-external-out.- For
demo:--capture(if given) exists, is absolute +$PWD-guarded, and is a valid video format.
4. If invalid: exit code 4 with a message identifying the offending field.
Step 0.5: Mode Routing
- shorts (9:16): synthetic from the topic. Default visual = oma-image stills + Ken Burns.
- explainer (16:9 / 9:16): from README / code / data. Default visual = oma-slide frames + oma-image diagrams.
- demo (16:9): from a screen capture. If no
--captureand Cap is unavailable -> emit the guided capture protocol (Step 4b) and stop.
Step 1: Provider Availability + Selection
1. Call available() on every registered provider in parallel (oma video list-providers). 2. For each capability, walk providers.<capability>.order:
- The first available provider wins.
- Paid providers (
pexels,pixelle) are skipped unless their env key is present (enabledgate). - On chain exhaustion for a required capability -> exit 5 naming the capability.
3. Log using: <provider> per capability to stderr before generation.
Step 2: Cost Guardrail
1. Estimate cost = sum of each selected provider's estimateCost() (most are $0; Pixelle/RunningHub credits are non-zero). 2. If --dry-run: emit script.json / render-spec.json / manifest.json, skip rendering, exit 0. 3. If estimate ≥ cost.guardrail_usd (or --max-usd) and not --yes / OMA_VIDEO_YES=1:
- Prompt on stderr:
Estimated cost $X.XX. Proceed? (y/N). Decline -> exit 1.
Step 3: Cancellation Setup
1. Install SIGINT / SIGTERM handlers that call AbortController.abort(). 2. Thread the signal into every provider call and into the render subprocess.
Step 4: Pipeline (asset bus)
brief ─► [ScriptProvider] ─► script.json {scenes[], narration[], onScreenText[]}
script.json ─► parallel:
├► [VoiceProvider] ─► audio/narration-*.wav + timing.json
├► [VisualProvider] ─► visuals/scene-NN.*
└► [CaptionProvider] ─► captions.srt / captions.vtt
all assets ─► render-spec.json ─► [Compositor: Remotion] ─► <mode>-<slug>.mp41. Script: AgentScriptProvider writes script.json (start of the determinism boundary). 2. Voice: oma-voice synthesizes narration -> audio/narration-NN.wav + timing.json. Fallback: estimated timing (no wav). 3. Visuals: walk the visual chain. oma-image stills (key-free default) / oma-slide frames (explainer) / Pexels (key) / Pixelle (key). Aspect -> 16-multiple size; Remotion crops to the exact frame. 4. Captions: oma-captions builds captions.srt + captions.vtt from timing.json. For a non-source locale, translate via oma-translator (key-free); absent -> warn + keep source. 5. render-spec: compose render-spec.json (the deterministic compute boundary) from the assets + seed. 6. Render: the compositor consumes render-spec.json.
Step 4b: Guided Capture Protocol (demo, no capture)
State plainly to the user: "Demo capture is performed by a human." Then: 1. Instruct: record the walkthrough with Cap (or any recorder) at 16:9. 2. Ask the user to re-run with --capture <absolute-path>.mp4. 3. Stop without rendering (capture-required is a guided stop, not a hard error).
Step 5: Compositor Render
- Remotion (default): when the vendored toolchain (Node + Chromium + FFmpeg) is bootstrapped, run
npx remotion render <entry> <CompId> <mode>-<slug>.mp4 --props=render-spec.jsonfromresources/remotion/. The live invocation is deferred at the CLI adapter (TODO(oma-deferred): remotion render). - Fallback: write a deterministic placeholder mp4 derived from the render-spec so the run dir + manifest are still well-formed with zero toolchain.
- MPT (`--compositor mpt`): inject the agent-written script (custom-script mode); keys env-only + log masking.
Step 6: Write Artifacts
1. Copy every external asset into the run dir and hash it (sha256); no URL refs. 2. Validate each asset-bus schema (script / timing / render-spec / manifest) — schemaVersion must be "1.0". 3. Build manifest.json: runId, mode, providers{...}, assets[{path,sha256,bytes,seed}], outputs{video,durationSec,sha256}, cost{usd,breakdown}, warnings[], exitCode. 4. If --no-brief-in-manifest: replace prompt with promptSha256.
Step 7: Report
1. Print a one-line status per capability to stderr:
[oma video] <capability> <provider> ok (Xs)[oma video] <capability> <provider> fallback -> <fallback>
2. Print the run-dir path + the mp4 path. 3. For --format json: write {exitCode, runDir, manifestPath, outputs} to stdout as one JSON object.
Step 8: Exit Code Aggregation (aligned with oma search fetch)
- Success (mp4 + valid manifest) -> exit 0 (fallbacks recorded in
warnings). - Otherwise pick the most specific code:
safety-refused-> 2not-found(profile/asset) -> 3schema-validation/ invalid input -> 4provider-unavailable/auth-required-> 5timeout-> 6- otherwise -> 1
On Error
| Situation | Action |
|---|---|
| No provider for a required capability | Exit 5, print Run: oma video doctor |
| Remotion toolchain not bootstrapped | Exit 1 (CompositorBootstrapError) + doctor remediation; MPT fallback where applicable |
| Voicebox MCP down | Fall back voicebox-stt -> whisper.cpp -> estimated timing; still emit captions |
| Pexels / Pixelle key absent | Skip provider; fall through to oma-image stills; annotate coverage in warnings |
demo with no capture + no Cap | Guided protocol (Step 4b); stop without rendering |
--capture outside $PWD or wrong format | Exit 4 with the path/format problem |
| Cost over guardrail, declined | Exit 1 |
| Timeout | Exit 6, manifest records after_ms |
| Cancelled (Ctrl+C) | Exit 130 (signal); no manifest if abort was pre-write |
#!/usr/bin/env python3
# pyright: reportMissingImports=false
# The `app.*` imports below are MoneyPrinterTurbo's modules. MPT is never
# vendored into this repo; this driver only ever runs under the MPT checkout's
# own venv (`<MPT>/.venv/bin/python`), where `app` is importable. The repo's
# Python analyzer cannot see them, so this third-party-runtime directive is
# expected — it is not a suppression of any first-party issue.
"""Headless MoneyPrinterTurbo (MPT) driver — boundary-safe subprocess entrypoint.
oma-video never imports MPT. The TypeScript compositor (`internal/mpt-project.ts`
+ `providers/compositor.ts`) spawns *this* script with the MPT venv's python and
a single JSON argument describing the run. The driver loads MPT in-process here
(inside MPT's own venv, never inside oma's runtime) and drives the full pipeline
via `app.services.task.start(...)`, then copies the produced mp4 to the caller's
output path.
Key-free by construction (design 013 §5, backend rule 11):
* video_script is injected directly -> MPT's custom-script mode, NO LLM key.
* voice defaults to an edge-tts voice -> NO TTS key.
* subtitle_provider="edge" (MPT config) reuses the edge sub_maker -> no
faster-whisper model download.
* video_source defaults to "local": the driver synthesizes a few ffmpeg
test-pattern clips into MPT's storage/local_videos so composition needs NO
Pexels key. video_source="pexels" is used only when a key is provided.
Contract — stdin/argv is one JSON object, stdout's LAST line is one JSON result:
IN {
"mpt_dir": "<abs path to MPT repo>", # required
"script": "<narration text, one line per scene>", # required
"subject": "<short subject>", # optional, default "video"
"out_path": "<abs output mp4 path>", # required
"aspect": "9:16" | "16:9" | "1:1", # optional, default 9:16
"voice_name": "<edge voice>", # optional, default en-US
"video_source":"local" | "pexels", # optional, default local
"pexels_api_key": "<key>", # optional (pexels source)
"materials": ["<abs clip path>", ...], # optional, local source
"clip_duration": 5, # optional
"subtitle": true | false # optional, default true
}
OUT {"ok": true, "output": "<abs mp4>", "duration": <float>, "source": "..."}
| {"ok": false, "error": "<message>"}
Exit code is 0 on success, 1 on failure; the JSON result line is authoritative.
"""
from __future__ import annotations
import json
import os
import shutil
import subprocess
import sys
import traceback
import uuid
# Default key-free edge-tts voice. parse_voice_name() strips the gender suffix.
DEFAULT_VOICE = "en-US-AvaNeural-Female"
# MPT's preprocess_video rejects materials below 480x480; portrait shorts need
# height >= width, so 1080x1920 is both safe and matches the design's frame.
RESOLUTIONS = {
"9:16": (1080, 1920),
"16:9": (1920, 1080),
"1:1": (1080, 1080),
}
# A small palette of solid colors so synthesized local clips are visually
# distinct (deterministic order — same input yields the same clips).
COLORS = ["0x1a2332", "0x2d3a4f", "0x3f5168", "0x52687f", "0x6a7f96"]
def _emit(result: dict) -> None:
"""Print the single authoritative JSON result line and flush."""
sys.stdout.write(json.dumps(result) + "\n")
sys.stdout.flush()
def _ffmpeg_bin() -> str:
return os.environ.get("IMAGEIO_FFMPEG_EXE") or shutil.which("ffmpeg") or "ffmpeg"
def _make_test_clip(out_path: str, width: int, height: int, seconds: int,
color: str, label: str) -> None:
"""Synthesize a solid-color test clip with ffmpeg (key-free local material).
Drawn at the target resolution so MPT's >=480x480 check passes and no
upscaling/letterboxing is required. libx264 + yuv420p keeps the clip a
standard, broadly decodable mp4.
"""
ffmpeg = _ffmpeg_bin()
cmd = [
ffmpeg, "-y",
"-f", "lavfi",
"-i", f"color=c={color}:s={width}x{height}:d={seconds}:r=30",
"-vf", (
f"drawtext=text='{label}':fontcolor=white:fontsize={max(width, height)//22}"
":x=(w-text_w)/2:y=(h-text_h)/2"
),
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-t", str(seconds),
out_path,
]
proc = subprocess.run(cmd, capture_output=True, text=True)
if proc.returncode != 0 or not os.path.isfile(out_path):
# Fallback without drawtext (some ffmpeg builds lack the freetype filter).
cmd_plain = [
ffmpeg, "-y",
"-f", "lavfi",
"-i", f"color=c={color}:s={width}x{height}:d={seconds}:r=30",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-t", str(seconds),
out_path,
]
proc2 = subprocess.run(cmd_plain, capture_output=True, text=True)
if proc2.returncode != 0 or not os.path.isfile(out_path):
raise RuntimeError(
f"ffmpeg failed to synthesize local material: "
f"{(proc2.stderr or proc.stderr or '').strip()[-400:]}"
)
def _prepare_local_materials(spec: dict, width: int, height: int,
clip_duration: int):
"""Resolve local material clips into MPT's storage/local_videos directory.
MPT's preprocess_video resolves each material.url *within* storage/local_videos
(file_security.resolve_path_within_directory). So provided clips are copied in
and synthesized clips are written there. Returns a list of MaterialInfo.
"""
from app.models.schema import MaterialInfo # noqa: E402 (MPT import)
from app.utils import utils # noqa: E402
local_dir = utils.storage_dir("local_videos", create=True)
materials = []
# MaterialInfo.url must be an ABSOLUTE path inside storage/local_videos.
# preprocess_video only rewrites .url for *image* materials; for video clips
# it returns the original url unchanged, and combine_videos then opens that
# url directly. A bare filename would be opened relative to cwd and fail
# ("'oma-synth-00.mp4' not found"). Absolute paths still pass MPT's
# resolve_path_within_directory guard because they live under local_videos.
provided = spec.get("materials") or []
if provided:
for idx, src in enumerate(provided):
if not os.path.isfile(src):
continue
ext = os.path.splitext(src)[1] or ".mp4"
dst = os.path.join(local_dir, f"oma-material-{idx:02d}{ext}")
shutil.copyfile(src, dst)
materials.append(MaterialInfo(provider="local", url=dst, duration=0))
if not materials:
# Synthesize a few deterministic test-pattern clips (key-free path).
count = 3
for idx in range(count):
dst = os.path.join(local_dir, f"oma-synth-{idx:02d}.mp4")
_make_test_clip(
dst, width, height, clip_duration,
COLORS[idx % len(COLORS)], f"scene {idx + 1}",
)
materials.append(MaterialInfo(provider="local", url=dst, duration=0))
return materials
def run(spec: dict) -> dict:
mpt_dir = spec.get("mpt_dir")
if not mpt_dir or not os.path.isdir(mpt_dir):
return {"ok": False, "error": f"mpt_dir not found: {mpt_dir!r}"}
script = (spec.get("script") or "").strip()
if not script:
return {"ok": False, "error": "script is required and must be non-empty"}
out_path = spec.get("out_path")
if not out_path:
return {"ok": False, "error": "out_path is required"}
aspect = spec.get("aspect") or "9:16"
if aspect not in RESOLUTIONS:
aspect = "9:16"
width, height = RESOLUTIONS[aspect]
voice_name = spec.get("voice_name") or DEFAULT_VOICE
video_source = spec.get("video_source") or "local"
clip_duration = int(spec.get("clip_duration") or 5)
subtitle = bool(spec.get("subtitle", True))
subject = spec.get("subject") or "video"
# Make MPT importable. This driver runs under MPT's OWN venv python, so its
# third-party deps resolve; we only need MPT's package root on sys.path.
if mpt_dir not in sys.path:
sys.path.insert(0, mpt_dir)
# Pexels key (only when the pexels source is selected) — env-only, never
# logged. MPT reads pexels keys from its config; set it on config in-memory.
from app.config import config # noqa: E402
if video_source == "pexels":
key = spec.get("pexels_api_key") or os.environ.get("PEXELS_API_KEY")
if not key:
return {"ok": False, "error": "video_source=pexels but no PEXELS_API_KEY"}
config.app["pexels_api_keys"] = [key]
from app.models.schema import VideoConcatMode, VideoParams # noqa: E402
from app.services import task # noqa: E402
from app.utils import utils # noqa: E402
video_materials = None
if video_source == "local":
video_materials = _prepare_local_materials(
spec, width, height, clip_duration
)
if not video_materials:
return {"ok": False, "error": "no local materials available"}
params = VideoParams(
video_subject=subject,
# Injecting video_script puts MPT in custom-script mode: generate_script
# returns it verbatim, no LLM call (backend rule 11 key-free path).
video_script=script,
# Empty terms + local source avoids generate_terms' LLM call too.
video_terms=[] if video_source == "local" else None,
video_aspect=aspect,
video_concat_mode=VideoConcatMode.sequential.value,
video_clip_duration=clip_duration,
video_count=1,
video_source=video_source,
video_materials=video_materials,
voice_name=voice_name,
voice_rate=1.0,
bgm_type="", # no background music (key-free, deterministic)
bgm_volume=0.0,
subtitle_enabled=subtitle,
n_threads=2,
paragraph_number=1,
)
task_id = "oma-" + uuid.uuid4().hex[:12]
result = task.start(task_id=task_id, params=params, stop_at="video")
if not result or not result.get("videos"):
return {
"ok": False,
"error": "MPT task.start produced no videos (see stderr for the MPT log)",
}
final = result["videos"][0]
if not os.path.isfile(final):
return {"ok": False, "error": f"MPT reported video but file missing: {final}"}
os.makedirs(os.path.dirname(os.path.abspath(out_path)), exist_ok=True)
shutil.copyfile(final, out_path)
duration = 0.0
try:
duration = float(result.get("audio_duration") or 0.0)
except (TypeError, ValueError):
duration = 0.0
# Clean up MPT's per-task storage so the cache clone does not grow unbounded.
try:
shutil.rmtree(utils.task_dir(task_id), ignore_errors=True)
except Exception: # noqa: BLE001 (cleanup is best-effort)
pass
return {
"ok": True,
"output": os.path.abspath(out_path),
"duration": duration,
"source": video_source,
}
def main() -> int:
# Spec JSON comes from argv[1] (a path or inline JSON) or stdin.
raw = None
if len(sys.argv) > 1:
arg = sys.argv[1]
if os.path.isfile(arg):
with open(arg, "r", encoding="utf-8") as fh:
raw = fh.read()
else:
raw = arg
if raw is None:
raw = sys.stdin.read()
try:
spec = json.loads(raw)
except Exception as exc: # noqa: BLE001
_emit({"ok": False, "error": f"invalid spec JSON: {exc}"})
return 1
try:
result = run(spec)
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
_emit({"ok": False, "error": f"{type(exc).__name__}: {exc}"})
return 1
_emit(result)
return 0 if result.get("ok") else 1
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env node
// Headed web-app capture driver — boundary-safe subprocess entrypoint.
//
// oma-video NEVER imports Playwright. The TypeScript provider
// (`providers/capture-playwright.ts` + `internal/playwright-project.ts`) spawns
// *this* script with the node interpreter of a resolved Playwright install
// (its node_modules reachable via cwd / NODE_PATH). The driver drives a real
// headed Chromium, records a live human-driven flow, and muxes the result to a
// single mp4 — then prints ONE authoritative JSON result line on stdout.
//
// MECHANISM ONLY. The driver assumes NOTHING about the flow or its purpose:
// * any URL (local / staging / prod),
// * any number of pages (popups / new tabs / cross-origin redirects are
// recorded generically — never tied to a specific auth or app shape),
// * NO credential handling or automation of any kind — a human drives the
// on-screen flow and logs in themselves if the flow needs it.
//
// SECURITY:
// * the URL + any query tokens are MASKED in every stderr log line,
// * the recording + all outputs are confined to --out's directory,
// * credentials are never read, stored, or printed; on-screen sensitive input
// is captured as-is (the human controls what is on screen).
//
// Contract — flags in, ONE JSON result line out (stdout's LAST line):
// IN --url <url> target (required)
// --out <abs mp4 path> output, inside a run dir (required)
// --size <WxH> recording frame size (required; derived upstream)
// --headless <0|1> headed (0, default, real use) | headless (1, CI)
// --ready-selector <css> await before the meaningful capture (optional)
// --show-cursor overlay a visible cursor (optional)
// --timeout <ms> hard ceiling for the whole capture (optional)
// --stop <mode> NON-interactive stop for CI/tests:
// duration:<sec> stop after N seconds
// selector:<css> stop when the selector appears
// omitted -> interactive ENTER prompt (real path)
// OUT {"ok":true, "output":"<abs mp4>", "pages":<n>, "durationSec":<float>}
// | {"ok":false,"error":"<message>","code":"<reason>"}
//
// Exit code is 0 on success, 1 on failure; the JSON result line is authoritative.
import { spawn } from "node:child_process";
import { createRequire } from "node:module";
import {
existsSync,
mkdtempSync,
readdirSync,
rmSync,
statSync,
} from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import process from "node:process";
import { pathToFileURL } from "node:url";
// All mutable state + scratch handle are declared FIRST, so the early-`fail`
// path (and its `cleanup()`) never touches a binding in its temporal dead zone.
let scratch = null;
let browser;
let hardTimer;
let aborted = false;
// Pages recorded, in chronological open order. Each entry: { video, openedAt }.
const recorded = [];
// Stop coordinators — resolved by ENTER / duration / selector / timeout / SIGINT.
const stopWaiters = [];
const args = parseArgs(process.argv.slice(2));
const url = args.url;
const out = args.out;
const size = parseSize(args.size);
const headless = args.headless === "1";
const readySelector = args["ready-selector"];
const showCursor = args["show-cursor"] === true || args["show-cursor"] === "1";
const timeoutMs = Number.parseInt(args.timeout ?? "", 10);
const stopMode = parseStop(args.stop);
if (!url) fail("missing --url", "bad-args");
if (!out) fail("missing --out", "bad-args");
if (!size) fail("missing or invalid --size (expected WxH)", "bad-args");
const outDir = path.dirname(path.resolve(out));
if (!existsSync(outDir)) fail(`--out directory does not exist: ${mask(outDir)}`, "bad-out");
// Resolve Playwright from the install the provider passes via --playwright-dir
// (its node_modules), falling back to a bare resolution from cwd. ESM ignores
// NODE_PATH, so we resolve the package's absolute entry explicitly and import
// it by file URL — boundary-safe (this script is never imported into the CLI).
const chromium = await resolveChromium(args["playwright-dir"]);
// Recordings land in a private temp dir we own, then get muxed into --out.
// Confinement: nothing is written outside outDir (final mp4) or this scratch
// dir (intermediate webms, cleaned up at the end).
scratch = mkdtempSync(path.join(tmpdir(), "oma-video-pwcap-"));
log(`opening ${mask(url)} (${size.width}x${size.height}, ${headless ? "headless" : "headed"})`);
try {
browser = await chromium.launch({ headless });
const context = await browser.newContext({
viewport: { width: size.width, height: size.height },
recordVideo: { dir: scratch, size: { width: size.width, height: size.height } },
});
// Record EVERY page generically — popups, new tabs, cross-origin redirects.
// We never inspect or interpret the page; we only track open order so the
// final concat is chronological.
context.on("page", (page) => {
recorded.push({ page, openedAt: Date.now() });
if (showCursor) injectCursor(page).catch(() => undefined);
});
const firstPage = await context.newPage();
if (recorded.length === 0) {
recorded.push({ page: firstPage, openedAt: Date.now() });
}
if (showCursor) await injectCursor(firstPage).catch(() => undefined);
// Hard ceiling: abort the whole capture cleanly if it runs past --timeout.
if (Number.isFinite(timeoutMs) && timeoutMs > 0) {
hardTimer = setTimeout(() => {
aborted = true;
log(`capture timeout (${timeoutMs}ms) reached — stopping`);
stopWaiters.forEach((resolve) => {
resolve("timeout");
});
}, timeoutMs);
hardTimer.unref?.();
}
await firstPage.goto(url, { waitUntil: "load", timeout: 60_000 }).catch((err) => {
log(`navigation warning: ${stringifyError(err)}`);
});
// Hydration: settle the network, then await an optional readiness selector.
await firstPage
.waitForLoadState("networkidle", { timeout: 30_000 })
.catch(() => log("networkidle not reached within 30s — continuing"));
if (readySelector) {
await firstPage
.waitForSelector(readySelector, { timeout: 30_000 })
.catch(() => log(`ready-selector not found within 30s — continuing`));
}
// STOP: interactive ENTER (real human path) OR a non-interactive mode (CI).
const reason = await waitForStop(firstPage, stopMode);
log(`stopping (${reason})`);
// Close the context to flush every page's video to disk.
await context.close();
await browser.close();
browser = undefined;
// Collect the produced webms in chronological page order. Playwright names
// them unpredictably, so we sort scratch entries by mtime to approximate the
// open order (the first page is always first).
const webms = collectWebms(scratch);
if (webms.length === 0) {
fail("no video produced (0-frame recording)", "empty-recording");
}
const durationSec = await muxToMp4(webms, path.resolve(out));
if (durationSec === null || !existsSync(path.resolve(out))) {
fail("ffmpeg produced no output mp4", "mux-failed");
}
emit({
ok: true,
output: path.resolve(out),
pages: webms.length,
durationSec,
});
cleanup();
process.exit(0);
} catch (err) {
cleanup();
// Ctrl-C / SIGINT surfaces here as an abort; report it without partial output.
emit({
ok: false,
code: aborted ? "aborted" : "capture-error",
error: stringifyError(err),
});
if (browser) await browser.close().catch(() => undefined);
process.exit(1);
}
// ---------------------------------------------------------------------------
// Stop coordination
// ---------------------------------------------------------------------------
/**
* Resolve when the capture should stop. Interactive: the first ENTER on stdin.
* Non-interactive: `duration:<sec>` after N seconds, or `selector:<css>` when
* the selector appears on the first page. The hard --timeout also resolves it.
*/
function waitForStop(firstPage, stop) {
return new Promise((resolve) => {
let settled = false;
const done = (reason) => {
if (settled) return;
settled = true;
resolve(reason);
};
stopWaiters.push(done);
if (stop?.kind === "duration") {
log(`non-interactive stop: after ${stop.seconds}s`);
const t = setTimeout(() => done("duration"), stop.seconds * 1000);
t.unref?.();
return;
}
if (stop?.kind === "selector") {
log(`non-interactive stop: when selector appears`);
firstPage
.waitForSelector(stop.selector, { timeout: 0 })
.then(() => done("selector"))
.catch(() => undefined);
return;
}
// Interactive ENTER. The prompt goes to stderr so stdout stays pure JSON.
process.stderr.write(
"\n[oma-video] Browser is open. Perform your flow, then press ENTER here to stop recording.\n",
);
process.stdin.resume();
const onData = () => {
process.stdin.off("data", onData);
process.stdin.pause();
done("enter");
};
process.stdin.on("data", onData);
// Ctrl-C: clean stop, no partial output (caller sees a non-ok result).
process.once("SIGINT", () => {
aborted = true;
done("sigint");
});
});
}
// ---------------------------------------------------------------------------
// Media helpers
// ---------------------------------------------------------------------------
/** Sorted list of produced webm paths (chronological by mtime). */
function collectWebms(dir) {
let entries;
try {
entries = readdirSync(dir);
} catch {
return [];
}
return entries
.filter((name) => name.toLowerCase().endsWith(".webm"))
.map((name) => path.join(dir, name))
.sort((a, b) => mtime(a) - mtime(b));
}
function mtime(file) {
try {
return statSync(file).mtimeMs;
} catch {
return 0;
}
}
/**
* Mux the recorded webms into a single mp4 at `outPath`. One page -> a straight
* transcode; multiple pages -> a chronological concat (filtergraph, so differing
* codecs/sizes are normalized to the first page's frame). Returns the muxed
* duration (seconds) or null on failure.
*/
async function muxToMp4(webms, outPath) {
if (webms.length === 1) {
const code = await runFfmpeg([
"-y",
"-i",
webms[0],
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
outPath,
]);
if (code !== 0) return null;
return probeDuration(outPath);
}
// Multi-page: concat via the concat filter so heterogeneous inputs are safe.
const inputs = [];
for (const webm of webms) {
inputs.push("-i", webm);
}
const labels = webms.map((_, i) => `[${i}:v:0]`).join("");
const filter = `${labels}concat=n=${webms.length}:v=1:a=0[outv]`;
const code = await runFfmpeg([
"-y",
...inputs,
"-filter_complex",
filter,
"-map",
"[outv]",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
outPath,
]);
if (code !== 0) return null;
return probeDuration(outPath);
}
function runFfmpeg(ffArgs) {
return new Promise((resolve) => {
const ff = spawn(ffmpegBin(), ffArgs, { stdio: ["ignore", "ignore", "pipe"] });
let stderr = "";
ff.stderr?.on("data", (c) => {
stderr += c.toString();
});
ff.on("error", () => resolve(1));
ff.on("close", (code) => {
if (code !== 0) log(`ffmpeg exit ${code}: ${stderr.trim().split("\n").slice(-1)[0] ?? ""}`);
resolve(code ?? 1);
});
});
}
function probeDuration(file) {
return new Promise((resolve) => {
const ff = spawn(
ffprobeBin(),
[
"-v",
"error",
"-show_entries",
"format=duration",
"-of",
"default=noprint_wrappers=1:nokey=1",
file,
],
{ stdio: ["ignore", "pipe", "ignore"] },
);
let stdout = "";
ff.stdout?.on("data", (c) => {
stdout += c.toString();
});
ff.on("error", () => resolve(null));
ff.on("close", () => {
const seconds = Number.parseFloat(stdout.trim());
resolve(Number.isFinite(seconds) && seconds > 0 ? seconds : null);
});
});
}
function ffmpegBin() {
return process.env.OMA_FFMPEG?.trim() || "ffmpeg";
}
function ffprobeBin() {
return process.env.OMA_FFPROBE?.trim() || "ffprobe";
}
/** Inject a small visible-cursor overlay that follows the mouse, for clarity. */
async function injectCursor(page) {
await page.addInitScript(() => {
const dot = document.createElement("div");
dot.style.cssText =
"position:fixed;z-index:2147483647;width:18px;height:18px;margin:-9px 0 0 -9px;border-radius:50%;background:rgba(255,80,80,.65);box-shadow:0 0 0 2px rgba(255,255,255,.9);pointer-events:none;transition:transform .03s linear;";
const mount = () => {
if (document.body) document.body.appendChild(dot);
};
if (document.body) mount();
else document.addEventListener("DOMContentLoaded", mount);
document.addEventListener("mousemove", (e) => {
dot.style.left = `${e.clientX}px`;
dot.style.top = `${e.clientY}px`;
});
});
}
// ---------------------------------------------------------------------------
// Arg parsing + masking + result emission
// ---------------------------------------------------------------------------
/**
* Resolve `chromium` from a Playwright install. `dir` is the directory whose
* `node_modules` holds `playwright` (or `@playwright/test`); when omitted we try
* a bare resolution from the current working directory. Imports the package's
* absolute entry by file URL because ESM does not honor NODE_PATH. Emits a
* masked `playwright-unresolved` result and exits 1 on failure.
*/
async function resolveChromium(dir) {
const anchor = dir
? path.join(path.resolve(dir), "node_modules", "__anchor__.js")
: path.join(process.cwd(), "__anchor__.js");
const require = createRequire(anchor);
for (const pkg of ["playwright", "@playwright/test"]) {
// Playwright is CommonJS; `require` yields the real module.exports whose
// named `chromium` is the launcher. (An ESM `import` of the CJS entry only
// interops `default`, dropping the named exports — so we require here.)
try {
const mod = require(pkg);
if (mod?.chromium) return mod.chromium;
} catch {
// try the next package specifier
}
// Fallback: resolve the absolute entry and import by file URL (covers
// installs whose package "exports" map only exposes an ESM entry).
try {
const entry = require.resolve(pkg);
const mod = await import(pathToFileURL(entry).href);
const chromium = mod?.chromium ?? mod?.default?.chromium;
if (chromium) return chromium;
} catch {
// try the next package specifier
}
}
emit({
ok: false,
code: "playwright-unresolved",
error: `could not load Playwright from ${mask(dir ?? process.cwd())}`,
});
cleanup();
process.exit(1);
return undefined; // unreachable; keeps the type obvious
}
function parseArgs(argv) {
const out = {};
for (let i = 0; i < argv.length; i++) {
const token = argv[i];
if (!token.startsWith("--")) continue;
const key = token.slice(2);
const next = argv[i + 1];
if (next === undefined || next.startsWith("--")) {
out[key] = true; // boolean flag
} else {
out[key] = next;
i++;
}
}
return out;
}
function parseSize(value) {
if (!value || typeof value !== "string") return null;
const m = /^(\d+)x(\d+)$/.exec(value.trim());
if (!m) return null;
const width = Number.parseInt(m[1], 10);
const height = Number.parseInt(m[2], 10);
if (!(width > 0 && height > 0)) return null;
return { width, height };
}
function parseStop(value) {
if (!value || typeof value !== "string") return null;
const [kind, ...rest] = value.split(":");
const arg = rest.join(":");
if (kind === "duration") {
const seconds = Number.parseFloat(arg);
if (Number.isFinite(seconds) && seconds > 0) return { kind: "duration", seconds };
return null;
}
if (kind === "selector" && arg.length > 0) {
return { kind: "selector", selector: arg };
}
return null;
}
/**
* Mask a URL for logging: keep scheme + host + path shape, strip the query/hash
* entirely (it may carry tokens), and redact userinfo. Non-URL strings get a
* coarse redaction of anything that looks like a token/query.
*/
function mask(value) {
if (typeof value !== "string") return String(value);
try {
const u = new URL(value);
const auth = u.username ? "***@" : "";
const query = u.search ? "?<redacted>" : "";
const hash = u.hash ? "#<redacted>" : "";
return `${u.protocol}//${auth}${u.host}${u.pathname}${query}${hash}`;
} catch {
// Not a URL — redact any "?...="/"&...=" token-ish tails and long hex/jwt.
return value
.replace(/([?&][^=\s]+=)[^&\s]+/g, "$1<redacted>")
.replace(/\b[A-Za-z0-9_-]{24,}\b/g, "<redacted>");
}
}
function log(message) {
process.stderr.write(`[oma-video] ${mask(message)}\n`);
}
function stringifyError(err) {
const msg = err instanceof Error ? err.message : String(err);
return mask(msg);
}
function emit(result) {
process.stdout.write(`${JSON.stringify(result)}\n`);
}
function fail(message, code) {
cleanup();
emit({ ok: false, code: code ?? "error", error: mask(message) });
process.exit(1);
}
function cleanup() {
if (hardTimer) clearTimeout(hardTimer);
if (!scratch) return;
try {
rmSync(scratch, { recursive: true, force: true });
} catch {
// best-effort scratch cleanup
}
}
Prompt Tips
A good video brief specifies the mode, the topic/source, and the arc (what the viewer should feel/learn). The brief drives the script; the script's per-scene visual.prompt then drives oma-image (English prompts — image models are trained predominantly on English captions).
Brief structure
Mode + audience → Topic/source → Arc (hook → body → payoff) → Tone/pacingExample (shorts): 30s vertical short for a dev audience: how oma-video turns a topic into a finished clip; hook with the pain, show the 3-step flow, end on the one-command CTA; upbeat, fast cuts
Per-mode guidance
shorts (9:16)
- Hook in the first 1.5s — the first scene's on-screen text must earn the swipe.
- Keep scenes 2–4s; favor 6–10 scenes for a 30s clip (≤
max_scenes40). - oma-image stills get Ken Burns motion — write prompts that frame a clear subject with headroom for the pan.
- TikTok captions are centered and animated; keep narration lines short (≤ ~8 words/segment) so caption pages switch cleanly.
explainer (16:9 / 9:16)
- Source is a README / code / data — point the brief at the file and the one thing to teach.
- Mix oma-slide frames (structure, bullet beats) with oma-image diagrams (concepts) and code frames.
- Code frames use one fixed deterministic theme (v1). Keep snippets short and legible at 1920×1080.
- Narration should explain why, not read the code line-by-line.
demo (16:9)
- The visual is a human-recorded capture (
--capture). The brief drives the intro card, zoom/callout beats, and outro. - Call out where to look on screen — Remotion adds zoom + callouts over the capture.
- Keep the intro ≤ 3s; viewers came for the product, not the title card.
Per-scene image prompts (oma-image)
The script's scenes[].visual.prompt is forwarded to oma-image. Write them with the same structure oma-image expects:
Scene/backdrop → Subject → Details → Constraints| Mode | Example per-scene prompt |
|---|---|
| shorts | Aerial drone shot of Jeju coastline, turquoise water meeting volcanic rock, golden hour, vertical composition with headroom for a slow zoom |
| shorts | Close-up of a hand pouring espresso into a glass over ice, warm cafe light, shallow depth of field |
| explainer | Clean isometric diagram of a 3-stage pipeline (script → assets → render), flat vector, labeled boxes, neutral background |
| demo | Minimal title card: product name centered, dark UI background, subtle accent gradient, 16:9 |
Do's
- Anchor the arc: hook → body → payoff. A short without a hook gets swiped past.
- Match aspect to mode (9:16 shorts, 16:9 explainer/demo) or use
auto. - Keep narration per-scene and short so caption pages and scene boundaries align.
- Pick music that matches pacing (
upbeatfor shorts,calmfor explainer).
Don'ts
- Don't write a brief with no mode and no topic — the agent will have to clarify.
- Don't request paid stock/AIGC visuals without the key — the chain silently falls back to oma-image stills (annotated in
warnings), which may surprise you. - Don't exceed
max_duration_sec(180) ormax_scenes(40). - Don't put secrets in the brief unless
--no-brief-in-manifestis set.
Localization
- Narration + on-screen text are authored in
--locale. For a non-source locale, oma-translator translates the script text before TTS/captions (key-free). If the translator is absent, the source text is kept and a warning is recorded. - oma-image prompts are sent in English regardless of
--locale; translate the user's subject and show the translated prompt during amplification.
Determinism
--seed <n>makes the render reproducible. The samerender-spec.json+ assets + seed + embedded Pretendard font produce a byte-stable render.- Re-render an existing run with
oma video render <runDir>— it consumesrender-spec.jsononly, so script/voice/visual generation is not re-run. OMA_VIDEO_MOCK=1replays golden fixtures for deterministic tests.
node_modules/
out/
.remotion/
# Fonts are provisioned by `oma video doctor`, not committed (see public/fonts/README.md).
public/fonts/*.woff2
public/fonts/*.ttf
public/fonts/*.otf
{
"name": "oma-video-remotion",
"version": "1.0.0",
"private": true,
"description": "Vendored Remotion compositor for oma-video. Registers Shorts/Explainer/Demo compositions that consume render-spec.json as input props. Lockfile-pinned; install once via `oma video doctor`, never during a run.",
"license": "UNLICENSED",
"type": "module",
"scripts": {
"studio": "remotion studio src/index.ts",
"render": "remotion render src/index.ts",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@remotion/captions": "4.0.380",
"@remotion/cli": "4.0.380",
"@remotion/fonts": "4.0.380",
"react": "19.2.0",
"react-dom": "19.2.0",
"remotion": "4.0.380",
"zod": "3.23.8"
},
"devDependencies": {
"@types/react": "19.2.0",
"typescript": "5.6.3"
}
}
Embedded fonts (Pretendard)
src/load-fonts.ts embeds Pretendard Variable via @remotion/fonts loadFont(). Embedding the font locally — instead of relying on a system font or a network fetch — is what makes a render byte-identical across machines (design 013 §5; design rule 2: CJK-ready font priority).
What goes here
public/fonts/PretendardVariable.woff2staticFile("fonts/PretendardVariable.woff2") resolves to this path at render time. The .woff2 is not committed to keep the skill tree light.
How it is provisioned
oma video doctor fetches the font once into this directory (lockfile-pinned, no fetch during a render). Until then, ensurePretendard() swallows the missing file and the browser falls back to the system stack defined in FONT_STACK (system-ui, -apple-system, …). The render still succeeds, but is not guaranteed byte-identical across machines.
Source: Pretendard (OFL-1.1) — https://github.com/orioncactus/pretendard Mirror the exact release pinned by oma video doctor so renders stay reproducible.
oma-video Remotion compositor (vendored)
The universal compositor for oma-video. It registers three compositions — Shorts (9:16), Explainer (16:9 / 9:16), and Demo (16:9) — that each consume render-spec.json as input props and emit an .mp4.
This project is lockfile-pinned and installed once via `oma video doctor` — never installed during a render (design 013 §5, §7 Tier-1).
Layout
resources/remotion/
├── package.json # pinned remotion + @remotion/cli + @remotion/captions + @remotion/fonts
├── tsconfig.json # strict, noEmit (typecheck-shaped)
├── remotion.config.ts # deterministic encoder defaults (h264 mp4)
├── public/fonts/ # Pretendard embed (provisioned by doctor; see README)
└── src/
├── index.ts # registerRoot(RemotionRoot)
├── Root.tsx # <Composition> Shorts / Explainer / Demo + calculateMetadata
├── render-spec.ts # Zod mirror of the CLI RenderSpec (schemaVersion "1.0")
├── load-fonts.ts # Pretendard via @remotion/fonts loadFont()
├── compositions/ # Shorts.tsx · Explainer.tsx · Demo.tsx (thin wrappers)
└── components/
├── VideoBase.tsx # shared timeline: scenes + audio + captions
├── Scene.tsx # one render-spec scene (image/video/slide/capture) + Ken Burns
└── Captions.tsx # @remotion/captions: parseSrt -> createTikTokStyleCaptionsrender-spec.json as input props
render-spec.json is the deterministic compute boundary (design 013 §4.1). The CLI writes it into the run dir; the compositions read it via --props. Root.tsx validates the props with RenderSpecSchema (Zod) and derives the real width / height / fps / durationInFrames from the spec via calculateMetadata. An invalid render-spec fails fast (maps to the CLI's SchemaValidationError, exit 4) instead of rendering garbage.
The Zod schema here mirrors cli/commands/video/types.ts RenderSpecSchema. The CLI is the source of truth — keep the two in sync.
Rendering
Install once (via doctor), then render a run:
# Shorts (9:16). <entry> = src/index.ts, <CompId> = Shorts
npx remotion render src/index.ts Shorts out.mp4 --props=render-spec.json
# Explainer (16:9)
npx remotion render src/index.ts Explainer out.mp4 --props=render-spec.json
# Demo (16:9)
npx remotion render src/index.ts Demo out.mp4 --props=render-spec.jsonWhere render-spec.json is the run dir's spec (the CLI passes an absolute path). Preview interactively with npm run studio.
Live render is deferred (key-optional, backend rule 11)
Actual rendering needs Node + Chromium + FFmpeg, bootstrapped by oma video doctor. The CLI adapter (cli/commands/video/providers/compositor.ts) gates the real invocation on FFmpeg availability and, until the toolchain is wired, falls back to a deterministic placeholder mp4 derived from the render-spec so the run dir + manifest are always well-formed:
// real : when FFmpeg + this vendored project are present, invoke
// `npx remotion render src/index.ts <CompId> out.mp4 --props=render-spec.json`
// fallback : deterministic placeholder mp4 from the render-spec (zero toolchain)
//
// TODO(oma-deferred): remotion render (F3) — wire the CLI adapter to spawn the
// `npx remotion render` subprocess against this project once doctor guarantees
// Node/Chromium/FFmpeg. The placeholder branch stays as the key-free fallback.The fallback is itself a pure function of the render-spec, so it is reproducible from the same spec.
Determinism
render-spec.json+ asset files +seed+ the embedded Pretendard font
are the determinism boundary. The same inputs render byte-stable across machines.
- Ken Burns and all motion are driven purely by the frame (no randomness).
loadFont(..., { display: "block" })blocks the render until Pretendard is
ready, so text never flashes a fallback face mid-render.
Dependency notes (lockfile)
remotion,@remotion/cli,@remotion/captions,@remotion/fontsare pinned
to the same exact version (Remotion requires lockstep versions across its packages). Bump them together.
- Generate and commit a lockfile (
package-lock.json/bun.lock) when this
project is first installed so doctor installs the exact pinned tree. Do not run install during a render.
// Remotion CLI config for the vendored oma-video compositor.
// Kept minimal + deterministic: H.264 mp4, overwrite output, color-managed.
// The CLI adapter passes --props=render-spec.json; this file only fixes the
// encoder/output defaults so renders are reproducible across machines.
import { Config } from "@remotion/cli/config";
Config.setVideoImageFormat("jpeg");
Config.setCodec("h264");
Config.setOverwriteOutput(true);
Config.setChromiumOpenGlRenderer("angle");
// Captions.tsx — render captions.srt over the video using @remotion/captions.
//
// The CLI's oma-captions provider writes captions.srt (+ .vtt) into the run
// dir; render-spec.captions.file points at it. Here we fetch + parseSrt() the
// .srt, then createTikTokStyleCaptions() to page it, and display the active
// page at the current frame. Two styles map from render-spec.captions.style:
// - "tiktok" : centered, large, animated pop, sits in the lower-third
// above the safe-area bottom margin.
// - "lower-third" : smaller, left-aligned band near the bottom.
// - "none" : nothing rendered (handled by the caller).
import { useCallback, useEffect, useMemo, useState } from "react";
import {
createTikTokStyleCaptions,
parseSrt,
type Caption,
type TikTokPage,
} from "@remotion/captions";
import {
AbsoluteFill,
staticFile,
useCurrentFrame,
useVideoConfig,
delayRender,
continueRender,
cancelRender,
} from "remotion";
import { FONT_STACK } from "../load-fonts";
import type { CaptionStyleSchema, SafeArea } from "../render-spec";
import type { z } from "zod";
type CaptionStyle = z.infer<typeof CaptionStyleSchema>;
// How often TikTok-style caption pages switch (ms). Higher = more words/page.
const SWITCH_CAPTIONS_EVERY_MS = 1200;
export const Captions: React.FC<{
file?: string;
style: CaptionStyle;
maxWidthPct: number;
safeArea: SafeArea;
}> = ({ file, style, maxWidthPct, safeArea }) => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const [pages, setPages] = useState<TikTokPage[] | null>(null);
const [handle] = useState(() => delayRender("Loading captions"));
const fetchCaptions = useCallback(async () => {
if (!file) {
setPages([]);
continueRender(handle);
return;
}
try {
const res = await fetch(staticFile(file));
const text = await res.text();
const { captions } = parseSrt({ input: text }) as { captions: Caption[] };
const { pages: built } = createTikTokStyleCaptions({
captions,
combineTokensWithinMilliseconds: SWITCH_CAPTIONS_EVERY_MS,
});
setPages(built);
continueRender(handle);
} catch (err) {
cancelRender(err);
}
}, [file, handle]);
useEffect(() => {
fetchCaptions();
}, [fetchCaptions]);
const activePage = useMemo<TikTokPage | null>(() => {
if (!pages) return null;
const nowMs = (frame / fps) * 1000;
return (
pages.find(
(page) => nowMs >= page.startMs && nowMs < page.startMs + page.durationMs,
) ?? null
);
}, [pages, frame, fps]);
if (style === "none" || !activePage) return null;
const isTikTok = style === "tiktok";
return (
<AbsoluteFill
style={{
justifyContent: "flex-end",
alignItems: isTikTok ? "center" : "flex-start",
paddingBottom: `${safeArea.bottomPct}%`,
paddingLeft: `${safeArea.leftPct}%`,
paddingRight: `${safeArea.rightPct}%`,
pointerEvents: "none",
}}
>
<div
style={{
maxWidth: `${maxWidthPct}%`,
fontFamily: FONT_STACK,
fontWeight: 800,
textAlign: isTikTok ? "center" : "left",
color: "#ffffff",
textShadow: "0 2px 8px rgba(0,0,0,0.85)",
fontSize: isTikTok ? 64 : 40,
lineHeight: 1.15,
letterSpacing: -0.5,
background: isTikTok ? "transparent" : "rgba(0,0,0,0.55)",
padding: isTikTok ? 0 : "12px 20px",
borderRadius: isTikTok ? 0 : 12,
}}
>
{activePage.text}
</div>
</AbsoluteFill>
);
};
// Scene.tsx — render one render-spec scene: a visual (image / video / slide /
// capture / placeholder) plus its on-screen text, with optional Ken Burns.
//
// Each scene is placed on the timeline by the composition using <Sequence>; this
// component only draws the visual for the duration it is mounted. Ken Burns is a
// deterministic slow zoom driven purely by the frame (no randomness), so the
// output is reproducible from the render-spec + seed.
import {
AbsoluteFill,
Img,
OffthreadVideo,
staticFile,
interpolate,
useCurrentFrame,
} from "remotion";
import { FONT_STACK } from "../load-fonts";
import type { RenderSpecScene } from "../render-spec";
const isColor = (src: string): boolean => src.startsWith("#");
export const Scene: React.FC<{ scene: RenderSpecScene }> = ({ scene }) => {
const frame = useCurrentFrame();
const { type, src, kenBurns } = scene.visual;
// Deterministic slow zoom over the scene's local frame range.
const scale = kenBurns
? interpolate(frame, [0, scene.durationInFrames], [1, 1.08], {
extrapolateRight: "clamp",
})
: 1;
return (
<AbsoluteFill>
<AbsoluteFill style={{ transform: `scale(${scale})` }}>
{renderVisual(type, src)}
</AbsoluteFill>
{scene.onScreenText.length > 0 ? (
<AbsoluteFill
style={{
justifyContent: "flex-start",
alignItems: "center",
paddingTop: "10%",
}}
>
<div
style={{
fontFamily: FONT_STACK,
fontWeight: 800,
fontSize: 56,
color: "#ffffff",
textShadow: "0 2px 10px rgba(0,0,0,0.8)",
textAlign: "center",
maxWidth: "86%",
}}
>
{scene.onScreenText.join("\n")}
</div>
</AbsoluteFill>
) : null}
</AbsoluteFill>
);
};
function renderVisual(
type: RenderSpecScene["visual"]["type"],
src: string,
): React.ReactNode {
if (type === "placeholder" || (type === "image" && isColor(src))) {
const color = isColor(src) ? src : "#0f1117";
return <AbsoluteFill style={{ backgroundColor: color }} />;
}
if (type === "video" || type === "capture") {
return (
<OffthreadVideo
src={staticFile(src)}
style={{ width: "100%", height: "100%", objectFit: "cover" }}
/>
);
}
// image | slide -> still frame
return (
<Img
src={staticFile(src)}
style={{ width: "100%", height: "100%", objectFit: "cover" }}
/>
);
}
// VideoBase.tsx — the shared timeline used by all three modes. It lays out the
// render-spec scenes with <Sequence>, draws the background, mounts narration +
// optional music <Audio>, and overlays captions. Shorts/Explainer/Demo are thin
// wrappers that only differ in default framing; the timeline logic is one place.
import {
AbsoluteFill,
Audio,
Sequence,
staticFile,
} from "remotion";
import { Scene } from "./Scene";
import { Captions } from "./Captions";
import { ensurePretendard } from "../load-fonts";
import type { RenderSpec } from "../render-spec";
// Block the render until Pretendard is ready (deterministic glyphs).
void ensurePretendard();
export const VideoBase: React.FC<RenderSpec> = (spec) => {
const bgColor =
spec.background.type === "color" ? spec.background.src ?? "#0f1117" : "#000";
return (
<AbsoluteFill style={{ backgroundColor: bgColor }}>
{spec.background.type !== "color" && spec.background.src ? (
<AbsoluteFill>
<Scene
scene={{
id: "bg",
fromFrame: 0,
durationInFrames: spec.durationInFrames,
visual: {
type: spec.background.type === "video" ? "video" : "image",
src: spec.background.src,
kenBurns: false,
},
onScreenText: [],
}}
/>
</AbsoluteFill>
) : null}
{spec.scenes.map((scene) => (
<Sequence
key={scene.id}
from={scene.fromFrame}
durationInFrames={scene.durationInFrames}
name={scene.id}
>
<Scene scene={scene} />
</Sequence>
))}
{spec.audio.narration ? (
<Audio src={staticFile(spec.audio.narration)} />
) : null}
{spec.audio.music ? (
<Audio
src={staticFile(spec.audio.music)}
volume={dbToGain(spec.audio.musicGainDb ?? -18)}
/>
) : null}
<Captions
file={spec.captions.file}
style={spec.captions.style}
maxWidthPct={spec.captions.maxWidthPct}
safeArea={spec.captions.safeArea}
/>
</AbsoluteFill>
);
};
function dbToGain(db: number): number {
return Math.min(1, Math.max(0, 10 ** (db / 20)));
}
// Demo — 16:9 demo/walkthrough composition over a human-recorded capture.
// The capture arrives as a render-spec scene/background (visual.type "capture");
// intro card + callout scenes are additional render-spec scenes. Zoom/callout
// motion is expressed as Ken Burns + on-screen text on those scenes, so the
// timeline stays a pure function of the render-spec.
import { VideoBase } from "../components/VideoBase";
import type { RenderSpec } from "../render-spec";
export const Demo: React.FC<RenderSpec> = (props) => {
return <VideoBase {...props} />;
};
// Explainer — 16:9 (or 9:16) explainer composition built from README/code/data.
// Slide + diagram + code frames arrive as render-spec scenes (visual.type
// "slide" | "image"); VideoBase renders them on the timeline.
import { VideoBase } from "../components/VideoBase";
import type { RenderSpec } from "../render-spec";
export const Explainer: React.FC<RenderSpec> = (props) => {
return <VideoBase {...props} />;
};
// Shorts — 9:16 short-form composition. Thin wrapper over VideoBase; all
// timeline/visual/caption logic lives there. The render-spec drives dimensions,
// fps, and duration via the <Composition> calculateMetadata in Root.tsx.
import { VideoBase } from "../components/VideoBase";
import type { RenderSpec } from "../render-spec";
export const Shorts: React.FC<RenderSpec> = (props) => {
return <VideoBase {...props} />;
};
// index.ts — Remotion entry point. registerRoot() is what `npx remotion render
// src/index.ts <CompId> ...` and `remotion studio src/index.ts` look for.
import { registerRoot } from "remotion";
import { RemotionRoot } from "./Root";
registerRoot(RemotionRoot);
// load-fonts.ts — embed Pretendard for cross-machine identical renders.
//
// Pretendard is a CJK-ready variable font (design rule 2: CJK services
// prioritize Pretendard Variable). Embedding it locally — rather than relying
// on a system font or a network fetch — is what makes a render byte-stable
// across machines (design 013 §5). The .woff2 is NOT committed to git here to
// keep the skill tree light; `oma video doctor` fetches it once into
// `public/fonts/` (see public/fonts/README.md), then this loader picks it up.
//
// loadFont() blocks the render until the font is ready, so captions/on-screen
// text never flash a fallback face mid-render.
import { loadFont } from "@remotion/fonts";
import { staticFile } from "remotion";
export const PRETENDARD_FAMILY = "Pretendard";
// staticFile() resolves against public/. doctor writes the woff2 here.
const PRETENDARD_URL = staticFile("fonts/PretendardVariable.woff2");
let fontPromise: Promise<void> | null = null;
/**
* Idempotently load Pretendard. Compositions call this at module scope so the
* render is delayed until the font is ready. If the embedded woff2 is missing
* (doctor not run), the browser falls back to the system stack — the render
* still succeeds, but is not guaranteed byte-identical across machines.
*
* `@remotion/fonts` `loadFont()` calls Remotion's `cancelRender()` internally
* when the font URL fails to load (e.g. a 404 because `oma video doctor` has not
* fetched the woff2 yet). `cancelRender()` aborts the WHOLE render, so a plain
* `.catch()` on the returned promise is not enough — the render is already
* cancelled. We therefore probe the URL with `fetch` first and only call
* `loadFont()` when the asset is actually present. A missing font then degrades
* gracefully to the system `FONT_STACK` instead of hard-failing the render.
*/
export function ensurePretendard(): Promise<void> {
if (!fontPromise) {
fontPromise = (async () => {
try {
const probe = await fetch(PRETENDARD_URL, { method: "HEAD" });
if (!probe.ok) return;
} catch {
// Network/probe error -> system fallback.
return;
}
await loadFont({
family: PRETENDARD_FAMILY,
url: PRETENDARD_URL,
format: "woff2",
weight: "100 900",
display: "block",
}).catch(() => undefined);
})();
}
return fontPromise;
}
// System fallback stack (design rule 1 + 2): CJK-ready first, then system-ui.
export const FONT_STACK =
`"${PRETENDARD_FAMILY}", "Noto Sans CJK KR", system-ui, ` +
`-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif`;
// render-spec.ts — the Remotion-side mirror of the CLI's RenderSpec schema
// (cli/commands/video/types.ts, schemaVersion "1.0"). render-spec.json is the
// deterministic compute boundary: the CLI writes it, `npx remotion render`
// reads it as `--props`. Keep this in sync with the CLI schema; the CLI remains
// the source of truth.
import { z } from "zod";
export const VIDEO_SCHEMA_VERSION = "1.0" as const;
export const CaptionStyleSchema = z.enum(["tiktok", "lower-third", "none"]);
export const SafeAreaSchema = z.object({
topPct: z.number().nonnegative(),
bottomPct: z.number().nonnegative(),
leftPct: z.number().nonnegative(),
rightPct: z.number().nonnegative(),
});
export const RenderSpecSceneSchema = z.object({
id: z.string().min(1),
fromFrame: z.number().int().nonnegative(),
durationInFrames: z.number().int().positive(),
visual: z.object({
type: z.enum(["image", "video", "slide", "capture", "placeholder"]),
src: z.string().min(1),
kenBurns: z.boolean().default(false),
}),
onScreenText: z.array(z.string()).default([]),
transitionOut: z.string().optional(),
});
export const RenderSpecSchema = z.object({
schemaVersion: z.literal(VIDEO_SCHEMA_VERSION),
compositor: z.enum(["remotion", "mpt"]),
composition: z.string().min(1),
fps: z.number().int().positive(),
dimensions: z.object({
width: z.number().int().positive(),
height: z.number().int().positive(),
}),
durationInFrames: z.number().int().nonnegative(),
audio: z.object({
narration: z.string().optional(),
music: z.string().optional(),
musicGainDb: z.number().optional(),
}),
scenes: z.array(RenderSpecSceneSchema),
captions: z.object({
file: z.string().optional(),
style: CaptionStyleSchema,
fontFamily: z.string(),
maxWidthPct: z.number().positive().max(100),
safeArea: SafeAreaSchema,
}),
background: z.object({
type: z.enum(["color", "image", "video"]),
src: z.string().optional(),
}),
seed: z.number().int(),
});
export type RenderSpec = z.infer<typeof RenderSpecSchema>;
export type RenderSpecScene = z.infer<typeof RenderSpecSceneSchema>;
export type SafeArea = z.infer<typeof SafeAreaSchema>;
// Default props used by the Remotion Studio preview + as the schema fallback.
// A real render overrides every field via --props=render-spec.json.
export const PLACEHOLDER_RENDER_SPEC: RenderSpec = {
schemaVersion: VIDEO_SCHEMA_VERSION,
compositor: "remotion",
composition: "Shorts",
fps: 30,
dimensions: { width: 1080, height: 1920 },
durationInFrames: 90,
audio: {},
scenes: [
{
id: "scene-01",
fromFrame: 0,
durationInFrames: 90,
visual: { type: "placeholder", src: "#0f1117", kenBurns: false },
onScreenText: ["oma-video"],
},
],
captions: {
style: "tiktok",
fontFamily: "Pretendard",
maxWidthPct: 86,
safeArea: { topPct: 8, bottomPct: 18, leftPct: 7, rightPct: 7 },
},
background: { type: "color", src: "#0f1117" },
seed: 1,
};
// Root.tsx — registers the Shorts / Explainer / Demo compositions.
//
// Each <Composition> reads render-spec.json (passed via `--props`) as input
// props. dimensions/fps/durationInFrames come FROM the render-spec, so we use
// calculateMetadata to override the static defaults at render time. The Zod
// `schema` validates the props, giving the CLI adapter a typed contract: an
// invalid render-spec fails fast (maps to the CLI's SchemaValidationError / exit
// 4) instead of rendering garbage.
import { Composition, type CalculateMetadataFunction } from "remotion";
import { Shorts } from "./compositions/Shorts";
import { Explainer } from "./compositions/Explainer";
import { Demo } from "./compositions/Demo";
import {
RenderSpecSchema,
PLACEHOLDER_RENDER_SPEC,
type RenderSpec,
} from "./render-spec";
// Derive real dimensions/fps/duration from the render-spec props.
const calculateMetadata: CalculateMetadataFunction<RenderSpec> = ({ props }) => {
return {
width: props.dimensions.width,
height: props.dimensions.height,
fps: props.fps,
durationInFrames: Math.max(1, props.durationInFrames),
};
};
export const RemotionRoot: React.FC = () => {
return (
<>
<Composition
id="Shorts"
component={Shorts}
schema={RenderSpecSchema}
defaultProps={{ ...PLACEHOLDER_RENDER_SPEC, composition: "Shorts" }}
calculateMetadata={calculateMetadata}
// Static fallbacks (9:16); overridden by calculateMetadata at render.
width={1080}
height={1920}
fps={30}
durationInFrames={90}
/>
<Composition
id="Explainer"
component={Explainer}
schema={RenderSpecSchema}
defaultProps={{
...PLACEHOLDER_RENDER_SPEC,
composition: "Explainer",
dimensions: { width: 1920, height: 1080 },
}}
calculateMetadata={calculateMetadata}
width={1920}
height={1080}
fps={30}
durationInFrames={90}
/>
<Composition
id="Demo"
component={Demo}
schema={RenderSpecSchema}
defaultProps={{
...PLACEHOLDER_RENDER_SPEC,
composition: "Demo",
dimensions: { width: 1920, height: 1080 },
}}
calculateMetadata={calculateMetadata}
width={1920}
height={1080}
fps={30}
durationInFrames={90}
/>
</>
);
};
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "Bundler",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"jsx": "react-jsx",
"strict": true,
"noUncheckedIndexedAccess": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"noEmit": true
},
"include": ["src"]
}
Vendor Matrix
oma-video is a key-optional router (backend rule 11). Each capability has a provider order (a fallback chain). The orchestrator probes availability and walks the chain: the first available provider wins; only chain exhaustion is a stage failure. Paid providers auto-enable only when their env key is set; otherwise the chain falls through to a key-free default.
Capabilities -> providers
| Capability | order (config) | real (key/resource) | key-free fallback | marker |
|---|---|---|---|---|
| script | [agent-script] | agent-authored script (agent-as-key) | — | — |
| voice | [oma-voice] | Voicebox MCP TTS + STT timing | estimated timing (no wav) | — |
| visual | [oma-image, pexels, pixelle] | Pexels stock · Pixelle AIGC | oma-image stills + Ken Burns | TODO(oma-deferred): pexels / pixelle |
| caption | [oma-captions] | oma-translator for non-source locale | source-locale text from timing | TODO(oma-deferred): oma-translator |
| capture | [cap] | Cap CLI trigger | guided protocol + --capture <path> | TODO(oma-deferred): cap |
| compositor | [remotion, mpt] | Remotion render · MPT custom-script | deterministic placeholder mp4 | TODO(oma-deferred): remotion render |
Tier model
| Tier | Surface | Providers | Notes |
|---|---|---|---|
| 1 | CLI-first (subprocess) | Remotion, MPT, oma-image, oma-slide, oma-voice (REST) | deterministic; preferred whenever a CLI can drive the work |
| 2 | MCP | Voicebox MCP, Pixelle-MCP | localhost MCP; Pixelle off by default, community-MCP consent + key |
| 3 | Guided (human) | Cap, openscreen | demo capture is performed by a human |
oma-voice (VoiceProvider + timing)
| Field | Value |
|---|---|
| Surface | Voicebox MCP at 127.0.0.1:17493 |
| Synthesize | voicebox_speak{text, profile, language} -> generation_id |
| Retrieve wav | REST GET /audio/{generation_id} (MCP has no save-to-disk) |
| Timing (real) | voicebox_transcribe{audio_path} on the wav -> source: voicebox-stt |
| Timing (fallback) | whisper.cpp -> estimated (no wav written, audio field empty) |
| Side effect | Narration plays on the speakers during synthesis |
| Health | exit 5 if MCP down; exit 3 if the named profile is missing |
oma-image (VisualProvider: generate) — default key-free visual
| Field | Value |
|---|---|
| Transport | oma image generate "<prompt>" --vendor auto --size <16-multiple> --format json --out <runDir>/visuals |
| Aspect -> size | snapped to nearest 16-multiple: 9:16 -> 1088×1920, 16:9 -> 1920×1088, 1:1 -> 1088×1088 |
| Crop | Remotion crops the still to the exact frame; Ken Burns adds motion |
| Cost | free defaults (pollinations / antigravity); codex per-image per oma-image config |
oma-slide (VisualProvider: slide, explainer) — key-free
| Field | Value |
|---|---|
| Transport | oma slide generate deck -> oma slide export --format png -> 1920×1080 frames |
| Layering | oma-slide internally calls oma-image (same key-free chain) |
| Use | explainer code/diagram frames |
Pexels (VisualProvider: stock) — paid, opt-in
| Field | Value |
|---|---|
| Auth | PEXELS_API_KEY env var |
| Enabled | only when the key is present (providers.pexels.enabled gate) |
| Fallback | absent key -> skip; chain falls to oma-image stills + Ken Burns |
| Marker | TODO(oma-deferred): pexels on the real-call branch until the key is provisioned |
Pixelle-MCP + RunningHub (VisualProvider: aigc) — paid, off by default
| Field | Value |
|---|---|
| Surface | MCP http://localhost:9004/pixelle/mcp; RunningHub cloud or local ComfyUI |
| Setup | uvx pixelle@latest + wizard; one-time explicit consent + source review |
| Auth | RUNNINGHUB_API_KEY env var; cost gates on RunningHub credits via --max-usd |
| Enabled | off by default (providers.pixelle.enabled); never auto-connects |
| Fallback | absent/declined -> oma-image stills |
| Marker | TODO(oma-deferred): pixelle on the real-call branch |
Cap (CaptureProvider, demo) — Tier 3 guided
| Field | Value |
|---|---|
| Real | Cap CLI trigger when installed |
| Fallback | guided protocol: instruct the human to record, then pass --capture <path> |
| Path safety | --capture is absolutized, $PWD-guarded, existence + format validated |
| Marker | TODO(oma-deferred): cap on the CLI-trigger branch |
Compositor: Remotion (default) / MPT (alt)
| Field | Value |
|---|---|
| Real | vendored resources/remotion/ -> npx remotion render <entry> <CompId> out.mp4 --props=render-spec.json |
| Requires | Node + Chromium + FFmpeg (bootstrapped once via oma video doctor) |
| Fallback | deterministic placeholder mp4 derived from the render-spec (well-formed run dir + manifest, zero toolchain) |
| Determinism | render-spec + assets + seed + embedded Pretendard; re-render is byte-stable |
| Marker | TODO(oma-deferred): remotion render on the live-render branch (CLI adapter) |
| MPT alt | inject the agent-written script (custom-script mode); keys env-only + log masking; --compositor mpt |
Error Classification
| Error kind | Retry policy | Exit code when solo |
|---|---|---|
provider-unavailable | try next provider in order; chain-exhaustion fails | 5 |
auth-required | fail; hint tells the user how to authenticate | 5 |
compositor-bootstrap | fail; point to oma video doctor (+ MPT fallback) | 1 |
cost-guardrail | confirm; decline -> stop | 1 |
capture-required | guided protocol; not a hard error | (guided) |
schema-validation | fail; identify the offending field | 4 |
safety-refused | short-circuit (no fallback) | 2 |
not-found | fail; missing profile / asset | 3 |
timeout | record; fail | 6 |