
Screen Studio Editor
- 9 installs
- 22 repo stars
- Updated July 15, 2026
- oil-oil/screen-studio-editor
Helps with ai & agent building tasks.
About
screen-studio-editor is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted coding.
- screen-studio-editor
- AI & Agent Building
- AI-coding skill
Screen Studio Editor by the numbers
- 9 all-time installs (skills.sh)
- +3 installs in the week ending Aug 4, 2026 (Skillselion tracking)
- Ranked #12,152 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/oil-oil/screen-studio-editor --skill screen-studio-editorAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 9 |
|---|---|
| repo stars | ★ 22 |
| Last updated | July 15, 2026 |
| Repository | oil-oil/screen-studio-editor ↗ |
What it does
Helps with ai & agent building tasks.
Files
Screen Studio Auto-Editor
Mode A — Full editing of a .screenstudio project: remove pauses, cut repeated narration, burn subtitles Mode B — Standalone: burn accurate subtitles onto any .mp4 Mode C — Merge two .screenstudio projects (for supplementary re-recordings)
---
Prerequisites
Platform: macOS on Apple Silicon (M1/M2/M3) only — mlx-whisper does not run on Intel Macs or Linux.
First-time setup (run once after installing the skill):
bash <skill-directory>/setup.shThis installs ffmpeg (via Homebrew if missing), creates a Python venv, installs mlx-whisper, and pre-downloads the Whisper large-v3 model (~3 GB). Takes ~5 minutes on first run.
`SKILL_DIR`: Throughout this skill, SKILL_DIR refers to the skill's own directory. Claude Code injects this as "Base directory" in the skill header — use that value. Never hardcode a user-specific path.
# At the start of every session, set this variable from the injected base directory:
SKILL_DIR="<base directory shown in skill header>"
PYTHON="$SKILL_DIR/.venv/bin/python3"---
Mode A — Full .screenstudio workflow
A .screenstudio bundle is a directory with project.json (the editing timeline as scenes[].slices) and recording/ (raw audio/video per session). We modify the slices to remove silences and repeated narration, then burn subtitles onto the exported video.
Critical: `sourceStartMs`/`sourceEndMs` coordinate system — slice timestamps are in the merged audio timeline (all recording sessions concatenated back-to-back, gaps between sessions excluded). This is identical to the timestamps Whisper produces. Do NOT add processTimeStartMs offsets. The original single-slice project always has sourceEndMs ≈ sum of all session durations — this is the proof.
Critical: slice IDs must be unique — when splitting one slice into many, each new slice needs a fresh random ID or Screen Studio collapses duplicates and ignores all cuts. process.py handles this automatically.
Step 1: Validate inputs
Confirm the .screenstudio path exists and contains a recording/ folder. Ask the user to confirm settings if not already provided.
Defaults:
pause_threshold_ms: 800 — pauses longer than this feel awkward on screenmin_pause_to_keep_ms: 300 — leave a small gap so cuts don't sound abrupt
Step 2: Transcribe and remove pauses
# SKILL_DIR and PYTHON are defined in the Prerequisites section above
$PYTHON $SKILL_DIR/scripts/process.py \
--project "/path/to/Project.screenstudio" \
--pause-threshold 800 \
--min-pause 300 \
--language zh # use 'en' for English recordings, 'None' to auto-detectThis backs up project.json, transcribes microphone audio with mlx-whisper large-v3 (local, ~10-30s), removes pause gaps from the timeline, enables improveMicrophoneAudio (noise reduction + normalization), and saves transcript.json in the project folder.
After running, check for noise fragments: Whisper sometimes transcribes ambient noise during long silent waits (e.g. waiting for a build or page load) as single CJK characters like "坐". These become tiny spurious slices in the timeline — too short to contain real speech. Inspect project.json's slices and remove any that are under ~600ms or fall entirely within a known silent wait zone. These slices are harmless but create micro-gaps in the exported video that look like glitches.
Step 3: Remove repeated content
Read transcript.json. Look for:
- False starts: speaker begins a sentence, stops mid-way, then restarts it more cleanly
- Repeated explanations: the same idea said multiple times — keep the clearest version
- Duplicate closings: same sign-off said twice
For each repeat, note its timestamps (audio seconds from the Whisper transcript map 1:1 to sourceStartMs in slices — they use the same merged-audio coordinate). Write to /tmp/cuts.json:
[{"start_ms": 123000, "end_ms": 131500, "removed_text": "the repeated text"}]Then run Phase 2 to apply the cuts:
$PYTHON $SKILL_DIR/scripts/process.py \
--project "/path/to/Project.screenstudio" \
--skip-transcribe "/path/to/Project.screenstudio/transcript.json" \
--cuts-file "/tmp/cuts.json"Note: process.py always loads from project.json.bak (created on first run, never overwritten). Re-running is safe and idempotent — pauses + cuts are always applied to the original state.
Session boundary snap: process.py snaps slice endpoints to 10ms before the session boundary (not exactly at it). Snapping to the exact boundary triggers a Screen Studio audio-composer bug: metadata durationMs differs from the actual WAV length by ~0.3µs, causing a sub-millisecond audio segment that the composer rejects. The 10ms margin keeps the snap point safely inside session N's audio while still producing gap=0 for spring transition animation.
Step 4: Report and wait for export
Tell the user: how many pauses were removed, total time saved, which repeated segments were cut. Ask them to open Screen Studio, preview, and export the video.
Step 5: Burn subtitles
Once the user provides the exported video path, follow the Subtitle workflow below.
---
Mode B — Standalone subtitle burning
When the user has a video but no .screenstudio project, transcribe first, then follow the same subtitle workflow.
# SKILL_DIR and PYTHON are defined in the Prerequisites section above
# Extract audio
ffmpeg -i "/path/to/video.mp4" -ar 16000 -ac 1 /tmp/audio_for_transcribe.wav -y
# Check for AAC timestamp drift BEFORE transcribing
# If WAV duration > video duration by more than 0.5%, apply piecewise correction after transcription
python3 -c "
import subprocess, json
wav = float(subprocess.check_output(['ffprobe','-v','quiet','-show_entries','format=duration','-of','default=noprint_wrappers=1:nokey=1','/tmp/audio_for_transcribe.wav']).strip())
vid = float(subprocess.check_output(['ffprobe','-v','quiet','-show_entries','format=duration','-of','default=noprint_wrappers=1:nokey=1','/path/to/video.mp4']).strip())
drift_pct = (wav - vid) / vid * 100
print(f'WAV: {wav:.3f}s Video: {vid:.3f}s Drift: {drift_pct:+.2f}%')
if abs(drift_pct) > 0.5:
print('⚠️ Significant AAC drift detected — timestamps will need correction after transcription')
"
# Transcribe
# Use language='zh' for Mandarin/Chinese content to prevent Traditional Chinese output.
# Use language=None for English or mixed/unknown language content.
$PYTHON -c "
import mlx_whisper, json
result = mlx_whisper.transcribe('/tmp/audio_for_transcribe.wav',
path_or_hf_repo='mlx-community/whisper-large-v3-mlx',
word_timestamps=True, language='zh') # change to None for non-Chinese content
with open('/tmp/transcript.json', 'w') as f:
json.dump(result['segments'], f, ensure_ascii=False, indent=2)
print(f'Transcribed {len(result[\"segments\"])} segments')
"Then follow the Subtitle workflow below with /tmp/transcript.json as the transcript path.
---
Subtitle workflow
0. Apply glossary before reviewing
SKILL_DIR/glossary.json is user-specific and gitignored — it won't exist on a fresh install. If it doesn't exist, skip this step. If it does, apply all entries as automated corrections before manual review.
The glossary is case-insensitive. Format: [{"wrong": "...", "correct": "..."}]
import json, re
from pathlib import Path
glossary_path = Path(SKILL_DIR) / "glossary.json"
if glossary_path.exists():
glossary = json.loads(glossary_path.read_text())
for seg in segments:
for entry in glossary:
seg["text"] = re.sub(re.escape(entry["wrong"]), entry["correct"], seg["text"], flags=re.IGNORECASE)1. Semantic review — correct transcript.json
Read the full transcript text first. Look for words that are semantically inconsistent with the surrounding context — Whisper mishears phonetically similar words. For every suspicious word, classify it:
High-confidence corrections (fix immediately, no frames needed):
- Clear phonetic substitutions with obvious right answer given context (
Scream Studio→Screen Studio,cloud call→Claude Code,Nordic.js→Node.js) - Wrong capitalization of well-known proper nouns (
minimax→MiniMax,github→GitHub) - Nonsense words where the correct word is unambiguous from context
Low-confidence items (flag for visual verification):
- English proper nouns you don't recognize — a product name, tool, or brand that appeared on screen but you can't confidently spell or capitalize
- Commands or filenames that seem partially garbled
- Version numbers or org names (e.g.
oyo-oyo/something— is thisoil-oil?oiloil?)
Apply high-confidence fixes immediately. Then decide:
- No low-confidence items → skip frame extraction entirely, proceed to preview editor
- Low-confidence items exist → extract targeted frames around those timestamps only:
# Extract a few frames near the uncertain segment (e.g. segment at ~42s)
mkdir -p /tmp/frames
ffmpeg -i "/path/to/video.mp4" -ss 30 -t 30 -vf "fps=1/5" /tmp/frames/frame_%04d.jpg -yUse the Read tool to view those frames, identify the correct term, then apply the fix.
Full-video frame extraction (every 30s) is only warranted for dense screen-recording demos where many unknown technical terms appear throughout. For talking-head or pure voiceover videos, it wastes time and adds no value — skip it.
Edit the "text" fields in transcript.json using a Python find-replace script. The subtitle script reads text fields directly, so corrections apply immediately — no need to touch word-level tokens or the .ass file.
3. Preview and edit subtitles
Before burning, always launch the preview editor so the user can review subtitles synced with video playback. This is the mandatory quality gate — never skip straight to burning.
# Kill any previous instance on port 8765
lsof -ti :8765 | xargs kill -9 2>/dev/null; sleep 1
$PYTHON $SKILL_DIR/scripts/preview_editor.py \
"/path/to/exported.mp4" \
"/path/to/transcript.json"Run this in the background (run_in_background=true, do NOT append &). The script opens http://localhost:8765 in the browser automatically.
What the user sees: left side = video player with live subtitle overlay; right side = editable subtitle list. Features:
- Click a subtitle → video seeks to that timestamp
- Double-click text → inline edit mode (Enter to save, Esc to cancel)
- Checkbox → mark subtitles for deletion
- Ctrl/Cmd+F → find & replace across all subtitles
- "保存并关闭" → saves edits back to
transcript.jsonand closes
Tell the user: "已在浏览器中打开字幕预览编辑器,请检查字幕是否准确。可以双击编辑文字、勾选删除不需要的条目。确认无误后点击「保存并关闭」,然后告诉我继续烧录。"
Wait for the user to confirm before proceeding to burn. Do not burn until they say the subtitles look good.
Note on save format: The preview editor saves transcript.json wrapped as {"segments": [...]}. The burn script handles both this format and plain arrays, so no conversion is needed.
3.5 Update glossary after user saves
After the user confirms the subtitles look good, diff the original snapshot against the saved transcript to find what the user changed. The preview editor saves a .orig.json backup automatically at startup.
import json, difflib
with open("/path/to/transcript.json.orig.json") as f:
orig = json.load(f)
with open("/path/to/transcript.json") as f:
edited = json.load(f)
orig_segs = orig.get("segments", orig) if isinstance(orig, dict) else orig
edit_segs = edited.get("segments", edited) if isinstance(edited, dict) else edited
orig_by_start = {s["start"]: s["text"].strip() for s in orig_segs}
for s in edit_segs:
orig_text = orig_by_start.get(s["start"], "")
if orig_text and orig_text != s["text"].strip():
print(f'CHANGED [{s["start"]:.1f}s]')
print(f' before: {orig_text}')
print(f' after: {s["text"].strip()}')Read the diff output and decide which changes represent systematic Whisper misrecognitions (i.e., patterns that will recur in future videos). Add those to SKILL_DIR/glossary.json. Do NOT add:
- one-time content fixes (wrong date, specific name that won't repeat)
- deletions (the user removed a subtitle entirely)
- minor punctuation or style tweaks
Glossary format: {"wrong": "whisper_output", "correct": "true_term"} — case-insensitive, one canonical entry per pattern.
4. Burn subtitles
$PYTHON $SKILL_DIR/scripts/burn_subtitles.py \
--video "/path/to/exported.mp4" \
--transcript "/path/to/transcript.json"Output is saved as <video>_subtitled.mp4 next to the source. The script prints a real-time progress bar during encoding. Subtitle style: semi-transparent dark background box with white text.
Portrait video (e.g. iPhone): handled automatically. ffmpeg's autorotate corrects the orientation and clears the rotation metadata in the output, so no manual flags are needed. The script detects display dimensions correctly from the stored rotation metadata.
AAC timestamp drift (VFR / mobile recordings): Videos from iPhone or Screen Studio exports often have an AAC audio stream whose raw sample count doesn't match the container-declared duration. Example: container says 417.56s but AAC has 19852 frames × 1024 samples ÷ 48000 Hz = 423.51s. ffmpeg decodes all AAC frames faithfully, so the extracted WAV is 423.51s. Whisper timestamps follow the WAV — causing subtitles to drift ahead of the video as playback progresses.
Attempts to clamp the WAV with -t <duration> or -af atrim=end=<duration> do not work — both operate on container PTS timestamps (which already go 0→417.56s), so all samples pass through unchanged.
Diagnosis: Ask the user to identify 2–3 sync points — e.g. "beginning is fine, middle (~2 min) drifts by 3s, end is fine". This reveals whether the drift is linear or concentrated in a region.
Fix — piecewise linear scaling: If the user says drift starts at time B (e.g. 108s) and before that everything is in sync:
import json
BOUNDARY = 0.0 # ← video time (seconds) where drift begins; 0 if drift starts from the start
WAV_DUR = 0.0 # ← ffprobe -show_entries format=duration /tmp/audio_for_transcribe.wav
VIDEO_DUR = 0.0 # ← ffprobe -show_entries format=duration /path/to/video.mp4
ratio2 = (VIDEO_DUR - BOUNDARY) / (WAV_DUR - BOUNDARY)
with open('/tmp/transcript.json') as f:
segs = json.load(f)
for s in segs:
for key in ('start', 'end'):
t = s[key]
if t > BOUNDARY:
s[key] = round(BOUNDARY + (t - BOUNDARY) * ratio2, 3)
with open('/tmp/transcript_fixed.json', 'w') as f:
json.dump(segs, f, ensure_ascii=False, indent=2)Then burn subtitles using transcript_fixed.json. Verify: last segment end should be within ~1s of VIDEO_DUR.
If drift is purely linear (starts from t=0), set BOUNDARY = 0 and the formula reduces to uniform scaling: ratio = VIDEO_DUR / WAV_DUR.
---
Mode C — Merge Projects
Screen Studio does not support importing external videos. If you recorded supplementary content in a separate .screenstudio session (补录), use this mode to merge the two projects before editing.
How it works: The merge script copies recording files from both projects into a new output project, offsets the supplement's slice timestamps by the base project's total audio duration, and combines the slices. The result opens directly in Screen Studio for preview and rearrangement.
Step 1: Ask for the two project paths
Confirm:
- Base project: the original recording (e.g.
Tutorial_Part1.screenstudio) - Supplement project: the re-recorded / additional content (e.g.
Tutorial_Supplement.screenstudio) - Merge mode: append supplement at end (default), or insert after a specific slice
Step 2: Run the merge
# SKILL_DIR and PYTHON are defined in the Prerequisites section above
# Default: append supplement at end, output named {base}_Merged.screenstudio
$PYTHON $SKILL_DIR/scripts/merge_projects.py \
--base "/path/to/ProjectA.screenstudio" \
--supplement "/path/to/ProjectB.screenstudio"
# Custom output path
$PYTHON $SKILL_DIR/scripts/merge_projects.py \
--base "/path/to/ProjectA.screenstudio" \
--supplement "/path/to/ProjectB.screenstudio" \
--output "/path/to/ProjectMerged.screenstudio"
# Insert supplement after slice index N (0-based)
# Use when you know approximately where in the timeline the supplement belongs
$PYTHON $SKILL_DIR/scripts/merge_projects.py \
--base "/path/to/ProjectA.screenstudio" \
--supplement "/path/to/ProjectB.screenstudio" \
--insert-after-slice 5To find the right slice index: Read the base project's project.json and count the scenes[0].slices array. Each slice is a continuous segment. The slice index corresponds to the position in the timeline where you want the supplement to appear.
Step 3: Open in Screen Studio and arrange
Open the merged project in Screen Studio. The supplement content appears as additional clips at the end (or at the insertion point). Drag and reorder slices in the Screen Studio timeline as needed, then export.
Step 4: Continue with Mode A editing (optional)
After merging, you can run process.py on the merged project to remove pauses and repeated content across both recordings as if it were a single recording:
$PYTHON $SKILL_DIR/scripts/process.py \
--project "/path/to/ProjectMerged.screenstudio" \
--language zhMerge notes
- Originals are NOT modified — output is always a new directory
- If
--outputis omitted, the merged project is named{base_stem}_Merged.screenstudioin the same folder - File conflicts: if both projects have a recording file with the same name (e.g.
microphone_session_0.m4a), the supplement's file is automatically renamed (microphone_session_0_s1.m4a) — the metadata is updated accordingly - Slice ordering: slices are combined in array order. After merging, open Screen Studio and drag slices to the correct position if needed
- After merging you can still edit: the merged project behaves exactly like any other
.screenstudioproject — you can remove pauses, cut repeats, and burn subtitles normally
---
Notes
- If
project.json.bakalready exists, ask the user before overwriting - Camera config:
process.pysetscameraAspectRatio = "square"(宽高一致). Do NOT overridecameraRoundness— Screen Studio's default corner roundness looks natural; forcing it to 0 creates an unnaturally sharp square that clashes with the rest of the UI. - Subtitle line breaks follow Whisper segment boundaries (natural speech pauses), so each
textfield in transcript.json maps to one subtitle block. Long segments are split at punctuation. - Whisper sometimes splits technical terms into sub-tokens in the
wordsarray (e.g.BulkGen→["bug", "gem"]), but since the script now usestextfields for display, only thetextfield needs to be corrected — ignorewords. - Running the burn command in background:
burn_subtitles.pycorrectly waits for ffmpeg internally. When using the Bash tool'srun_in_backgroundparameter, do NOT add&to the shell command —&backgrounds the Python process itself, the shell exits immediately, and the "task completed" notification fires before ffmpeg finishes. The output MP4 will appear but havemoov atom not found(unplayable). Userun_in_background=trueon the tool call alone, without&in the command. - Screen Studio transition mechanism: Screen Studio applies its spring-based transition animation (zoom/pan) between consecutive slices only when `gap = 0ms` (i.e.,
slice[n].sourceEndMs == slice[n+1].sourceStartMs). Slices with any gap > 0ms get a hard cut. Pause-removal cuts always create gaps (audio was removed), so transitions can never be applied to within-session cuts. However,process.pynow callssnap_to_session_boundaries()after cutting, which restores gap=0 at inter-session boundaries (the natural seams Screen Studio originally created), giving smooth transitions at recording-session joins even after editing.
.venv/
__pycache__/
*.pyc
*.pyo
归档.zip
glossary.json
screen-studio-editor
A Claude Code skill for editing Screen Studio recordings and burning AI-corrected subtitles onto any video.
What it does
Mode A — Full .screenstudio editing
- Automatically removes awkward pauses from the recording timeline
- Detects and cuts repeated narration / false starts (Claude reads the transcript and decides what to remove)
- Enables noise reduction and volume normalization
- Burns accurate subtitles onto the exported video
Mode B — Subtitle burning for any .mp4
- Works with any video, not just Screen Studio exports
- Transcribes audio locally with Whisper (no API calls, no cost)
- Launches a live preview editor in the browser so you can review and fix subtitles before burning
- Handles iPhone portrait videos, AAC timestamp drift, and mixed CJK/Latin content
Mode C — Merge two .screenstudio projects
- Merges a supplementary re-recording into an existing project
- Supports inserting at a specific point in the timeline
Requirements
- macOS on Apple Silicon (M1/M2/M3) —
mlx-whisperrequires Apple Silicon - Claude Code
- Homebrew (for ffmpeg)
Installation
1. Install the skill into Claude Code:
~/.agents/skills/screen-studio-editor/2. Run the one-time setup (installs ffmpeg, Python venv, and downloads the Whisper large-v3 model ~3 GB):
bash ~/.agents/skills/screen-studio-editor/setup.shUsage
Once installed, just describe what you want to Claude Code in natural language:
- "帮我处理这个录屏 ~/Recordings/Tutorial.screenstudio"
- "给这个视频加字幕 ~/Desktop/demo.mp4"
- "把这两个工程合并 ProjectA.screenstudio ProjectB.screenstudio"
Claude will handle the rest — transcribing, editing the timeline, launching the subtitle preview, and burning the final video.
How subtitles work
Transcription runs fully locally using mlx-whisper (Whisper large-v3 on Apple Silicon). No data leaves your machine.
Before burning, a browser-based preview editor opens so you can:
- Review all subtitles synced with video playback
- Double-click to edit any line
- Check/uncheck to delete lines
- Find & replace across all subtitles
Scripts
| Script | Purpose |
|---|---|
scripts/process.py | Edit .screenstudio timeline (remove pauses, apply cuts) |
scripts/burn_subtitles.py | Burn ASS subtitles onto a video with ffmpeg |
scripts/preview_editor.py | Local HTTP server for the subtitle preview/edit UI |
scripts/merge_projects.py | Merge two .screenstudio projects |
setup.sh | One-time environment setup |
License
MIT
#!/usr/bin/env python3
"""
Burn subtitles onto an exported video using ffmpeg.
Converts transcript.json (from mlx-whisper) → ASS subtitle file → burned video.
Style: white text, subtle shadow, no border, centered bottom, clean sans-serif.
"""
import argparse
import json
import re
import subprocess
import sys
import textwrap
from pathlib import Path
def log(msg):
print(f"[burn-subtitles] {msg}", flush=True)
def add_cjk_spacing(text: str) -> str:
"""Add spaces between CJK characters and Latin/numeric characters for readability."""
text = re.sub(r'([\u4e00-\u9fff\u3400-\u4dbf])([A-Za-z0-9])', r'\1 \2', text)
text = re.sub(r'([A-Za-z0-9])([\u4e00-\u9fff\u3400-\u4dbf])', r'\1 \2', text)
return text
def _visual_len(text: str) -> float:
"""Visual width estimate: CJK = 1.0, Latin/digits/punct = 0.55, space = 0.5."""
w = 0.0
for c in text:
if '\u4e00' <= c <= '\u9fff' or '\u3400' <= c <= '\u4dbf' or '\u3000' <= c <= '\u303f':
w += 1.0
elif c == ' ':
w += 0.5
else:
w += 0.55
return w
def _split_text(text: str, max_chars: int) -> list[str]:
"""
Split text into subtitle-sized chunks.
Uses visual width (CJK=1.0, Latin=0.55) so mixed lines don't overflow.
Tries to break at sentence-end punctuation first, then soft punctuation,
then cuts at word boundaries as a last resort.
"""
if _visual_len(text) <= max_chars:
return [text]
result = []
def split_at(chunk: str, pattern: str) -> list[str]:
parts = re.split(pattern, chunk)
return [p.strip() for p in parts if p.strip()]
# Pass 1: split at sentence-ending punctuation
chunks = split_at(text, r'(?<=[。!?!?])\s*')
if len(chunks) == 1:
chunks = [text] # no hard punct found
# Pass 2: split oversized chunks at soft punctuation
mid = []
for c in chunks:
if _visual_len(c) <= max_chars:
mid.append(c)
else:
sub = split_at(c, r'(?<=[,,、;;])\s*')
mid.extend(sub if len(sub) > 1 else [c])
# Pass 3: cut at word boundaries, using visual width to find the split point
for c in mid:
while _visual_len(c) > max_chars:
# Walk forward to find the last space whose prefix fits within max_chars
cut_at = 0
best_space = -1
vw = 0.0
for i, ch in enumerate(c):
if ch == ' ' and vw <= max_chars:
best_space = i
vw += _visual_len(ch)
if vw > max_chars:
break
cut_at = i + 1
if best_space > len(c) // 4:
cut_at = best_space
# cut_at may be 0 if the very first char exceeds budget; force at least 1
cut_at = max(cut_at, 1)
result.append(c[:cut_at].rstrip())
c = c[cut_at:].lstrip()
if c:
result.append(c)
return result or [text]
def segments_to_lines(segments: list[dict], max_chars: int = 16) -> list[dict]:
"""
Convert Whisper segments to subtitle lines using segment-level text.
- Uses seg["text"] directly, so text corrections apply immediately
(no need to touch the word-level tokens at all)
- Respects segment boundaries as natural break points — Whisper segments
correspond to breath groups / pauses, giving much more natural phrasing
than re-grouping word tokens by character count
- Long segments are split at punctuation first, then hard-cut;
timing within a segment is interpolated proportionally by character count
"""
lines = []
for seg in segments:
text = seg["text"].strip()
if not text:
continue
# Apply CJK spacing before splitting so word-boundary detection
# can see spaces at CJK↔Latin boundaries (e.g. "这个Screen" → "这个 Screen")
text = add_cjk_spacing(text)
start = seg["start"]
end = seg["end"]
sub_lines = _split_text(text, max_chars)
if len(sub_lines) == 1:
lines.append({"start": start, "end": end, "text": sub_lines[0]})
else:
# Join as a single multi-line subtitle using \n (converted to \N in ASS)
# This keeps the original segment timing intact and avoids time-splitting words
lines.append({"start": start, "end": end, "text": "\n".join(sub_lines)})
return lines
def seconds_to_ass_time(s: float) -> str:
"""Convert seconds to ASS timestamp: H:MM:SS.cc"""
h = int(s // 3600)
m = int((s % 3600) // 60)
sec = s % 60
cs = int((sec - int(sec)) * 100)
return f"{h}:{m:02d}:{int(sec):02d}.{cs:02d}"
def generate_ass(lines: list[dict], output_path: Path, video_width: int = 1920, video_height: int = 1080):
"""Generate an ASS subtitle file with clean white + shadow style."""
is_portrait = video_height > video_width
# Portrait: 3.8% of display height (already large due to tall frame)
# Landscape: 5.5% of height — previous 3.8% was too small on widescreen
font_size = max(40, int(video_height * (0.038 if is_portrait else 0.055)))
# Portrait videos need a higher bottom margin so subtitles don't sit at the very edge.
margin_v = int(video_height * (0.20 if is_portrait else 0.06))
ass_header = textwrap.dedent(f"""\
[Script Info]
ScriptType: v4.00+
PlayResX: {video_width}
PlayResY: {video_height}
WrapStyle: 0
[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,PingFang SC,{font_size},&H00FFFFFF,&H000000FF,&H40202020,&H40202020,0,0,0,0,100,100,0,0,3,3,0,2,20,20,{margin_v},1
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
""")
event_lines = []
for line in lines:
start = seconds_to_ass_time(line["start"])
end = seconds_to_ass_time(line["end"])
text = add_cjk_spacing(line["text"]).replace("\n", "\\N")
event_lines.append(f"Dialogue: 0,{start},{end},Default,,0,0,0,,{text}")
with open(output_path, "w", encoding="utf-8") as f:
f.write(ass_header)
f.write("\n".join(event_lines))
f.write("\n")
log(f"✅ Generated ASS subtitle file: {output_path.name} ({len(event_lines)} lines)")
def _render_progress(elapsed_us: int, total_s: float, speed: float):
"""Print a single-line progress bar, overwriting the previous one."""
elapsed_s = elapsed_us / 1_000_000
pct = min(elapsed_s / total_s, 1.0) if total_s > 0 else 0
filled = int(pct * 20)
bar = "█" * filled + "░" * (20 - filled)
elapsed_fmt = f"{int(elapsed_s // 60)}:{int(elapsed_s % 60):02d}"
total_fmt = f"{int(total_s // 60)}:{int(total_s % 60):02d}"
speed_str = f"{speed:.1f}x" if speed > 0 else "..."
sys.stdout.write(f"\r {bar} {pct:>3.0%} {elapsed_fmt} / {total_fmt} {speed_str} ")
sys.stdout.flush()
def burn_subtitles(video_path: Path, ass_path: Path, output_path: Path,
scale_to: tuple[int, int] | None = None, total_duration_s: float = 0):
"""Burn ASS subtitles into video using ffmpeg.
scale_to: (width, height) to scale before rendering subtitles.
total_duration_s: video duration for progress display.
Rotation is handled automatically by ffmpeg's built-in autorotate. It both physically
corrects the frame orientation AND clears the Display Matrix in the output, so no manual
transpose filter or metadata patching is needed.
"""
log(f"Burning subtitles into video...")
# Escape special chars in path for ffmpeg filter
ass_str = str(ass_path).replace("\\", "/").replace(":", "\\:").replace("'", "\\'")
if scale_to:
w, h = scale_to
vf = f"scale={w}:{h},ass='{ass_str}'"
# H264 at downscaled resolution (faster, smaller file)
video_codec = ["-c:v", "h264_videotoolbox", "-b:v", "8M"]
log(f"Scaling to {w}x{h}")
else:
vf = f"ass='{ass_str}'"
# HEVC quality-based encoding — matches original iPhone/Screen Studio quality
# -q:v 65 on hevc_videotoolbox ≈ visually lossless for 4K source
# -tag:v hvc1 ensures broad player compatibility (hev1 tag breaks QPlayer etc.)
video_codec = ["-c:v", "hevc_videotoolbox", "-q:v", "65", "-tag:v", "hvc1"]
log("Keeping original resolution (HEVC quality mode)")
progress_path = Path("/tmp/ffmpeg_burn_progress.txt")
progress_path.unlink(missing_ok=True)
cmd = [
"ffmpeg",
"-i", str(video_path),
"-vf", vf,
"-c:a", "copy",
*video_codec,
"-progress", str(progress_path),
"-loglevel", "error",
str(output_path),
"-y",
]
import time
proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, text=True)
while proc.poll() is None:
time.sleep(0.5)
if not progress_path.exists():
continue
data = {}
for line in progress_path.read_text().splitlines():
if "=" in line:
k, _, v = line.partition("=")
data[k.strip()] = v.strip()
raw_us = data.get("out_time_us", "0")
elapsed_us = int(raw_us) if raw_us and raw_us.lstrip("-").isdigit() else 0
speed_str = data.get("speed", "0x").replace("x", "")
try:
speed = float(speed_str)
except ValueError:
speed = 0.0
_render_progress(elapsed_us, total_duration_s, speed)
sys.stdout.write("\n")
if proc.returncode != 0:
err = (proc.stderr.read() if proc.stderr else "")
raise RuntimeError(f"ffmpeg failed:\n{err[-1000:]}")
size_mb = output_path.stat().st_size / 1024 / 1024
log(f"Output: {output_path.name} ({size_mb:.1f} MB)")
def main():
parser = argparse.ArgumentParser(description="Burn subtitles onto exported Screen Studio video")
parser.add_argument("--video", required=True, help="Path to exported video file (.mp4)")
parser.add_argument("--transcript", required=True, help="Path to transcript.json from mlx-whisper")
parser.add_argument("--output", default=None, help="Output video path (default: input_subtitled.mp4)")
parser.add_argument("--max-chars", type=int, default=18, help="Max chars per subtitle line (default: 18)")
parser.add_argument("--ass-only", action="store_true", help="Only generate .ass file, don't burn")
parser.add_argument("--output-height", type=int, default=0,
help="Scale output: for landscape use height (e.g. 1440), for portrait use width (e.g. 1440). Default 0 = keep original resolution.")
args = parser.parse_args()
video_path = Path(args.video)
transcript_path = Path(args.transcript)
if not video_path.exists():
print(f"❌ Video not found: {video_path}")
sys.exit(1)
if not transcript_path.exists():
print(f"❌ Transcript not found: {transcript_path}")
sys.exit(1)
# Output paths
if args.output:
output_path = Path(args.output)
else:
output_path = video_path.with_name(video_path.stem + "_subtitled.mp4")
ass_path = output_path.with_suffix(".ass")
# Load transcript (supports both plain array and {"segments": [...]} from preview editor)
with open(transcript_path, encoding="utf-8") as f:
raw = json.load(f)
segments = raw.get("segments", raw) if isinstance(raw, dict) else raw
log(f"📝 Loaded {len(segments)} transcript segments")
# Get video dimensions
probe = subprocess.run(
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_streams", str(video_path)],
capture_output=True, text=True
)
video_w, video_h, video_duration, video_rotation = 1920, 1080, 0.0, 0
if probe.returncode == 0:
for stream in json.loads(probe.stdout).get("streams", []):
if stream.get("codec_type") == "video":
video_w = stream.get("width", 1920)
video_h = stream.get("height", 1080)
video_duration = float(stream.get("duration", 0) or 0)
# Detect rotation metadata (e.g. iPhone portrait stored as landscape + rotate)
for sd in stream.get("side_data_list", []):
if "rotation" in sd:
try:
video_rotation = int(sd["rotation"])
except (ValueError, TypeError):
pass
break
# Swap to display dimensions so portrait detection and ASS layout are correct
if abs(video_rotation) in (90, 270):
video_w, video_h = video_h, video_w
break
log(f"📐 Video display resolution: {video_w}x{video_h}"
+ (f" (stored rotated {video_rotation}°)" if video_rotation else ""))
# Compute output resolution (scale if requested)
# For portrait video (height > width), scale by width to avoid tiny output.
# --output-height 1440 on landscape 3840x2160 → 2560x1440 (2K)
# --output-height 1440 on portrait 2160x3840 → would be 810x1440 (blurry)
# So for portrait, treat output_height as the target for the SHORT side (width).
is_portrait = video_h > video_w
scale_to = None
if args.output_height and args.output_height > 0:
if is_portrait:
# Scale by width: output_height arg acts as target width
target_w = args.output_height
if target_w < video_w:
out_w = target_w if target_w % 2 == 0 else target_w + 1
out_h = round(video_h * out_w / video_w)
out_h = out_h if out_h % 2 == 0 else out_h + 1
scale_to = (out_w, out_h)
log(f"Portrait video detected — scaling by width to {out_w}x{out_h}")
else:
if args.output_height < video_h:
out_h = args.output_height
out_w = round(video_w * out_h / video_h)
out_w = out_w if out_w % 2 == 0 else out_w + 1
scale_to = (out_w, out_h)
ass_w = scale_to[0] if scale_to else video_w
ass_h = scale_to[1] if scale_to else video_h
# Convert segments to subtitle lines
lines = segments_to_lines(segments, args.max_chars)
log(f"🔤 Generated {len(lines)} subtitle lines")
# Generate ASS (at output resolution so font size is correct)
generate_ass(lines, ass_path, ass_w, ass_h)
if args.ass_only:
log(f"Done. ASS file: {ass_path}")
return
# Burn subtitles
burn_subtitles(video_path, ass_path, output_path, scale_to=scale_to,
total_duration_s=video_duration)
log("")
log("=" * 50)
log("✅ Done!")
log(f" Output: {output_path}")
log(f" Subtitle file: {ass_path}")
log("")
log("Tip: Edit the .ass file to tweak font/size/position, then re-run with --ass-only skipped.")
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Screen Studio Project Merger
Screen Studio stores each recording session as channel files:
channel-3-microphone-0.m4a, channel-3-microphone-0.m3u8,
channel-3-microphone-0-0001.m4s, ...
When merging, supplement's session 0 becomes session N
(where N = number of existing sessions in the base for that channel).
All related segment files are renamed accordingly, and .m3u8 content is updated.
Usage:
merge_projects.py --base A.screenstudio --supplement B.screenstudio
merge_projects.py --base A.screenstudio --supplement B.screenstudio --output Merged.screenstudio
merge_projects.py --base A.screenstudio --supplement B.screenstudio --insert-after-slice 5
"""
import argparse
import copy
import json
import random
import re
import shutil
import string
import sys
from pathlib import Path
def log(msg):
print(f"[merge] {msg}", flush=True)
def random_id(k=10):
return "".join(random.choices(string.ascii_letters + string.digits, k=k))
def load_json(path: Path) -> dict:
with open(path) as f:
return json.load(f)
def save_json(path: Path, data: dict):
with open(path, "w") as f:
json.dump(data, f, ensure_ascii=False, separators=(",", ":"))
def get_mic_sessions(metadata: dict) -> list[dict]:
"""Return microphone sessions sorted by processTimeStartMs."""
for recorder in metadata.get("recorders", []):
if recorder.get("type") == "microphone":
return sorted(recorder.get("sessions", []),
key=lambda s: s["processTimeStartMs"])
# Fallback: input recorder
for recorder in metadata.get("recorders", []):
if recorder.get("type") == "input":
return sorted(recorder.get("sessions", []),
key=lambda s: s["processTimeStartMs"])
return []
def get_total_mic_duration_ms(metadata: dict) -> float:
"""Sum of all microphone session durations = audio coordinate space size."""
return sum(s.get("durationMs", 0) for s in get_mic_sessions(metadata))
def get_max_process_time_end_ms(metadata: dict) -> float:
"""Maximum processTimeEndMs across all recorders."""
max_end = 0.0
for recorder in metadata.get("recorders", []):
for s in recorder.get("sessions", []):
end = s.get("processTimeEndMs", 0)
max_end = max(max_end, end)
return max_end
def build_channel_session_count(metadata: dict) -> dict[str, int]:
"""Return {channel_id: session_count} for all recorders with sessions."""
counts = {}
for recorder in metadata.get("recorders", []):
rid = recorder.get("id", "")
sessions = recorder.get("sessions", [])
if sessions:
counts[rid] = len(sessions)
return counts
def rename_channel_files(
src_recording_dir: Path,
dst_recording_dir: Path,
channel_id: str,
old_idx: int,
new_idx: int,
file_rename_map: dict[str, str],
):
"""
Copy all files belonging to channel_id session old_idx into dst_recording_dir
with session index new_idx. Update .m3u8 content.
File patterns:
{channel_id}-{old_idx}.m4a → {channel_id}-{new_idx}.m4a
{channel_id}-{old_idx}.mp4 → {channel_id}-{new_idx}.mp4
{channel_id}-{old_idx}.m3u8 → {channel_id}-{new_idx}.m3u8
{channel_id}-{old_idx}-NNNN.mp4 → {channel_id}-{new_idx}-NNNN.mp4
{channel_id}-{old_idx}-NNNN.m4s → {channel_id}-{new_idx}-NNNN.m4s
"""
old_prefix = f"{channel_id}-{old_idx}"
new_prefix = f"{channel_id}-{new_idx}"
copied = 0
for src_file in sorted(src_recording_dir.iterdir()):
if not src_file.is_file():
continue
name = src_file.name
if name == old_prefix + ".m4a" or name == old_prefix + ".mp4":
# Simple rename: channel-X-type-0.m4a → channel-X-type-2.m4a
new_name = new_prefix + src_file.suffix
shutil.copy2(src_file, dst_recording_dir / new_name)
file_rename_map[name] = new_name
copied += 1
elif name == old_prefix + ".m3u8":
# Update .m3u8 content: replace all occurrences of old_prefix with new_prefix
content = src_file.read_text()
new_content = content.replace(old_prefix + "-", new_prefix + "-")
new_name = new_prefix + ".m3u8"
(dst_recording_dir / new_name).write_text(new_content)
file_rename_map[name] = new_name
copied += 1
elif name.startswith(old_prefix + "-"):
# Segment files: channel-X-type-0-0001.m4s → channel-X-type-2-0001.m4s
suffix = name[len(old_prefix):] # e.g. "-0001.m4s"
new_name = new_prefix + suffix
shutil.copy2(src_file, dst_recording_dir / new_name)
file_rename_map[name] = new_name
copied += 1
return copied
def rename_input_files(
src_recording_dir: Path,
dst_recording_dir: Path,
old_idx: int,
new_idx: int,
file_rename_map: dict[str, str],
):
"""Rename input recorder session files: keystrokes-N, mouseclicks-N, mousemoves-N."""
for prefix in ("keystrokes", "mouseclicks", "mousemoves"):
for src_file in src_recording_dir.iterdir():
if src_file.is_file() and src_file.stem == f"{prefix}-{old_idx}":
new_name = f"{prefix}-{new_idx}{src_file.suffix}"
shutil.copy2(src_file, dst_recording_dir / new_name)
file_rename_map[src_file.name] = new_name
def merge_projects(
base_dir: Path,
supplement_dir: Path,
output_dir: Path,
insert_after_slice: int | None,
):
log(f"Base: {base_dir.name}")
log(f"Supplement: {supplement_dir.name}")
log(f"Output: {output_dir.name}")
# ── Validate ──────────────────────────────────────────────────────────────
for p, label in [(base_dir, "base"), (supplement_dir, "supplement")]:
if not (p / "project.json").exists():
log(f"❌ {label}: project.json not found"); sys.exit(1)
if not (p / "recording" / "metadata.json").exists():
log(f"❌ {label}: recording/metadata.json not found"); sys.exit(1)
# ── Load data ─────────────────────────────────────────────────────────────
base_project = load_json(base_dir / "project.json")
supp_project = load_json(supplement_dir / "project.json")
base_meta = load_json(base_dir / "recording" / "metadata.json")
supp_meta = load_json(supplement_dir / "recording" / "metadata.json")
audio_offset_ms = get_total_mic_duration_ms(base_meta)
supp_dur_ms = get_total_mic_duration_ms(supp_meta)
max_base_end = get_max_process_time_end_ms(base_meta)
log(f"Base audio duration: {audio_offset_ms/1000:.1f}s")
log(f"Supplement audio duration: {supp_dur_ms/1000:.1f}s")
log(f"Supplement slice offset: +{audio_offset_ms/1000:.1f}s")
# ── Create output from base ───────────────────────────────────────────────
if output_dir.exists():
log(f"⚠️ Output already exists: {output_dir}")
resp = input("Overwrite? [y/N]: ").strip().lower()
if resp != "y":
log("Aborted."); sys.exit(0)
shutil.rmtree(output_dir)
shutil.copytree(base_dir, output_dir)
log("✅ Copied base project to output.")
out_rec = output_dir / "recording"
supp_rec = supplement_dir / "recording"
# ── Determine session index mapping ───────────────────────────────────────
# For each channel, supplement's session 0 → base_session_count
base_counts = build_channel_session_count(base_meta)
log(f"Base session counts: {base_counts}")
# ── Copy supplement recording files with correct session indices ──────────
file_rename_map: dict[str, str] = {} # old_name → new_name
updated_supp_meta = copy.deepcopy(supp_meta)
pts_shift = max_base_end + 1000 # supplement sessions come after base ends
for i, recorder in enumerate(updated_supp_meta.get("recorders", [])):
rid = recorder.get("id", "")
rtype = recorder.get("type", "")
sessions = recorder.get("sessions", [])
if not sessions:
continue
base_count = base_counts.get(rid, 0)
for j, session in enumerate(sessions):
old_idx = j
new_idx = base_count + j
# Shift processTimeStartMs/End so this session sorts after base
min_supp_pts = min(s.get("processTimeStartMs", 0)
for r in supp_meta.get("recorders", [])
for s in r.get("sessions", []))
session["processTimeStartMs"] = (
session.get("processTimeStartMs", 0) - min_supp_pts + pts_shift
)
session["processTimeEndMs"] = (
session.get("processTimeEndMs", 0) - min_supp_pts + pts_shift
)
if rtype in ("systemAudio", "display", "microphone", "webcam"):
n = rename_channel_files(supp_rec, out_rec, rid, old_idx, new_idx, file_rename_map)
log(f" {rid}: session {old_idx}→{new_idx}, {n} files copied")
# Update outputFilename in session metadata
old_fn = session.get("outputFilename", "")
if old_fn in file_rename_map:
session["outputFilename"] = file_rename_map[old_fn]
elif rtype == "input":
rename_input_files(supp_rec, out_rec, old_idx, new_idx, file_rename_map)
session["keyStrokesFilename"] = f"keystrokes-{new_idx}.json"
session["mouseClicksFilename"] = f"mouseclicks-{new_idx}.json"
session["mouseMovesFilename"] = f"mousemoves-{new_idx}.json"
log(f" {rid}: input session {old_idx}→{new_idx}")
# ── Merge metadata ────────────────────────────────────────────────────────
output_meta = copy.deepcopy(base_meta)
for supp_rec_entry in updated_supp_meta.get("recorders", []):
rid = supp_rec_entry.get("id", "")
supp_sessions = supp_rec_entry.get("sessions", [])
if not supp_sessions:
continue
matched = False
for out_rec_entry in output_meta.get("recorders", []):
if out_rec_entry.get("id") == rid:
out_rec_entry.setdefault("sessions", []).extend(
copy.deepcopy(supp_sessions)
)
matched = True
break
if not matched:
output_meta.setdefault("recorders", []).append(
copy.deepcopy(supp_rec_entry)
)
# Merge top-level sessions array if present
if "sessions" in supp_meta:
supp_top_sessions = copy.deepcopy(supp_meta["sessions"])
min_supp_pts = min(s.get("processTimeStartMs", 0) for s in supp_top_sessions)
for s in supp_top_sessions:
s["processTimeStartMs"] = s.get("processTimeStartMs", 0) - min_supp_pts + pts_shift
s["processTimeEndMs"] = s.get("processTimeEndMs", 0) - min_supp_pts + pts_shift
output_meta.setdefault("sessions", []).extend(supp_top_sessions)
# ── Offset supplement slices ──────────────────────────────────────────────
supp_slices = supp_project["json"]["scenes"][0]["slices"]
offset_slices = []
for sl in supp_slices:
new_sl = copy.deepcopy(sl)
new_sl["sourceStartMs"] = sl["sourceStartMs"] + audio_offset_ms
new_sl["sourceEndMs"] = sl["sourceEndMs"] + audio_offset_ms
new_sl["id"] = random_id()
offset_slices.append(new_sl)
# ── Combine slices ────────────────────────────────────────────────────────
base_slices = base_project["json"]["scenes"][0]["slices"]
if insert_after_slice is None:
merged_slices = base_slices + offset_slices
placement = "at the END of the timeline"
log(f"Appending {len(offset_slices)} supplement slice(s) after {len(base_slices)} base slice(s).")
else:
pos = min(insert_after_slice + 1, len(base_slices))
merged_slices = base_slices[:pos] + offset_slices + base_slices[pos:]
placement = f"after slice {insert_after_slice}"
log(f"Inserting {len(offset_slices)} supplement slice(s) at position {pos}.")
# ── Write output ──────────────────────────────────────────────────────────
out_project = copy.deepcopy(base_project)
out_project["json"]["scenes"][0]["slices"] = merged_slices
save_json(output_dir / "project.json", out_project)
save_json(output_dir / "recording" / "metadata.json", output_meta)
log("")
log("=" * 52)
log("✅ Merge complete!")
log(f" Base slices: {len(base_slices)}")
log(f" Supplement slices: {len(offset_slices)}")
log(f" Total slices: {len(merged_slices)}")
log(f" Base duration: {audio_offset_ms/1000:.1f}s")
log(f" Supplement duration: {supp_dur_ms/1000:.1f}s")
log(f" Combined duration: {(audio_offset_ms+supp_dur_ms)/1000:.1f}s")
log("")
log(f" Open Screen Studio: {output_dir}")
log(f" Supplement appears {placement}.")
def main():
parser = argparse.ArgumentParser(description="Merge two Screen Studio projects")
parser.add_argument("--base", required=True)
parser.add_argument("--supplement", required=True)
parser.add_argument("--output", default=None)
parser.add_argument("--insert-after-slice", type=int, default=None, metavar="N")
args = parser.parse_args()
base_dir = Path(args.base)
supp_dir = Path(args.supplement)
if not base_dir.exists():
print(f"❌ Base not found: {base_dir}"); sys.exit(1)
if not supp_dir.exists():
print(f"❌ Supplement not found: {supp_dir}"); sys.exit(1)
output_dir = (
Path(args.output) if args.output
else base_dir.parent / f"{base_dir.stem}_Merged.screenstudio"
)
merge_projects(base_dir, supp_dir, output_dir, args.insert_after_slice)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Local HTTP server for subtitle preview + editing.
Serves HTML page (left=video, right=subtitle list), backed by Flask.
Save writes transcript.json and signals exit.
"""
import json
import os
import sys
import threading
import webbrowser
from pathlib import Path
from flask import (
Flask,
Response,
jsonify,
request,
send_file,
)
# ----------------------------------------------------------------------------- #
# Config
# ----------------------------------------------------------------------------- #
PORT = 8765
TRANSCRIPT_PATH = None
VIDEO_PATH = None
RESULT_DONE = threading.Event()
# ----------------------------------------------------------------------------- #
# Flask app
# ----------------------------------------------------------------------------- #
app = Flask(__name__, static_folder=None)
HTML_TEMPLATE = r"""<!DOCTYPE html>
<html lang="zh">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>字幕编辑器</title>
<style>
:root {
--bg: #f5f5f3;
--surface: #ffffff;
--border: #e4e4e0;
--text: #1a1a18;
--muted: #8a8a82;
--accent: #2563eb;
--accent2: #dc2626;
--hover: #f0f0ec;
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
background: var(--bg);
color: var(--text);
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
height: 100vh;
display: flex;
flex-direction: column;
}
/* ---- panel header ---- */
.panel-header {
display: flex;
align-items: center;
gap: 6px;
padding: 10px 12px 8px;
border-bottom: 1px solid var(--border);
flex-shrink: 0;
}
.panel-title { font-size: 12px; font-weight: 600; color: var(--muted); letter-spacing: 0.04em; text-transform: uppercase; flex: 1; }
.btn {
padding: 4px 10px;
border-radius: 6px;
border: 1px solid var(--border);
background: transparent;
color: var(--text);
font-size: 12px;
cursor: pointer;
transition: background 0.12s;
white-space: nowrap;
}
.btn:hover { background: var(--hover); }
.btn.danger { color: var(--accent2); border-color: transparent; }
.btn.danger:hover { background: #fef2f2; border-color: var(--accent2); }
.btn.save {
background: var(--accent);
border-color: var(--accent);
color: #fff;
font-weight: 600;
padding: 6px 16px;
font-size: 13px;
width: 100%;
border-radius: 8px;
}
.btn.save:hover { background: #1d4ed8; }
/* ---- main ---- */
.main { display: flex; flex: 1; overflow: hidden; position: relative; }
/* ---- video pane ---- */
.video-pane {
flex: 1;
display: flex;
flex-direction: column;
margin-right: 380px;
}
.video-wrap {
flex: 1;
background: #000;
display: flex;
align-items: center;
justify-content: center;
overflow: hidden;
position: relative;
}
.video-wrap video { width: 100%; height: 100%; object-fit: contain; }
.current-subtitle {
position: absolute;
bottom: 6%;
left: 50%;
transform: translateX(-50%);
max-width: 90%;
text-align: center;
font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", sans-serif;
font-size: clamp(13px, 2.2vw, 22px);
line-height: 1.5;
color: #ffffff;
background: rgba(32, 32, 32, 0.62);
padding: 4px 14px 6px;
border-radius: 4px;
pointer-events: none;
white-space: pre-wrap;
word-break: break-word;
text-shadow: 0 1px 3px rgba(0,0,0,0.8);
display: none;
}
.current-subtitle.visible { display: block; }
/* ---- list pane: always-visible right panel ---- */
.list-pane {
position: absolute;
top: 0; right: 0; bottom: 0;
width: 380px;
display: flex;
flex-direction: column;
overflow: hidden;
background: var(--surface);
border-left: 1px solid var(--border);
z-index: 10;
}
/* ---- find bar: hidden by default, shown with Ctrl+F ---- */
.find-bar {
display: none;
padding: 8px 12px;
background: var(--surface);
border-bottom: 1px solid var(--border);
gap: 6px;
align-items: center;
flex-wrap: wrap;
}
.find-bar.show { display: flex; }
.list-header {
display: flex;
align-items: center;
gap: 6px;
padding: 6px 12px;
background: var(--bg);
border-bottom: 1px solid var(--border);
flex-shrink: 0;
}
.list-header span { font-size: 11px; color: var(--muted); }
/* ---- panel footer ---- */
.panel-footer {
padding: 10px 12px;
border-top: 1px solid var(--border);
flex-shrink: 0;
display: flex;
flex-direction: column;
gap: 6px;
}
.panel-status { font-size: 11px; color: var(--muted); text-align: center; min-height: 14px; }
.subtitle-list { flex: 1; overflow-y: auto; padding: 6px; }
.subtitle-list::-webkit-scrollbar { width: 6px; }
.subtitle-list::-webkit-scrollbar-thumb { background: var(--border); border-radius: 3px; }
/* ---- subtitle item ---- */
.sub-item {
display: flex;
align-items: flex-start;
gap: 8px;
padding: 8px 10px;
border-radius: 8px;
margin-bottom: 3px;
cursor: pointer;
transition: background 0.1s;
position: relative;
user-select: none;
-webkit-user-select: none;
}
.sub-item:hover { background: var(--hover); }
.sub-item.deleted { opacity: 0.35; text-decoration: line-through; }
.sub-item mark {
background: #fbbf24;
color: #1a1a18;
border-radius: 3px;
padding: 0 1px;
}
.sub-check { flex-shrink: 0; margin-top: 4px; width: 16px; height: 16px; cursor: pointer; }
.sub-times {
flex-shrink: 0;
font-size: 11px;
color: var(--muted);
font-variant-numeric: tabular-nums;
min-width: 110px;
margin-top: 3px;
line-height: 1.6;
}
.sub-text-wrap { flex: 1; min-width: 0; }
.sub-text {
font-size: 14px;
line-height: 1.55;
white-space: pre-wrap;
word-break: break-word;
border-radius: 4px;
padding: 1px 3px;
margin: -1px -3px;
outline: none;
user-select: text;
-webkit-user-select: text;
}
.sub-text:focus {
background: rgba(37,99,235,0.06);
box-shadow: 0 0 0 2px rgba(37,99,235,0.2);
}
.sub-text[contenteditable="true"] { cursor: text; }
.sub-actions {
flex-shrink: 0;
display: flex;
gap: 3px;
margin-top: 1px;
opacity: 0;
transition: opacity 0.15s;
}
.sub-item:hover .sub-actions { opacity: 1; }
.icon-btn {
width: 26px; height: 26px;
border-radius: 5px;
border: 1px solid var(--border);
background: var(--surface);
color: var(--muted);
cursor: pointer;
font-size: 13px;
display: flex;
align-items: center;
justify-content: center;
transition: all 0.15s;
}
.icon-btn:hover { background: var(--hover); color: var(--text); }
.icon-btn.del:hover { background: #fef2f2; color: var(--accent2); border-color: var(--accent2); }
/* ---- find bar ---- */
.find-bar {
display: none;
padding: 8px 12px;
background: var(--surface);
border-bottom: 1px solid var(--border);
gap: 6px;
align-items: center;
flex-wrap: wrap;
}
.find-bar.show { display: flex; }
.find-bar input {
background: #f9f9f7;
border: 1px solid var(--border);
border-radius: 6px;
color: var(--text);
padding: 4px 10px;
font-size: 13px;
min-width: 160px;
}
.find-bar input:focus { outline: none; border-color: var(--accent); }
.find-bar label { font-size: 12px; color: var(--muted); }
/* ---- status bar ---- */
.status {
padding: 6px 16px;
background: var(--surface);
border-top: 1px solid var(--border);
font-size: 11px;
color: var(--muted);
flex-shrink: 0;
display: flex;
gap: 16px;
}
/* ---- mobile layout: video top, subtitles bottom ---- */
@media (max-width: 768px) {
body { height: 100dvh; overflow: hidden; }
.main { flex-direction: column; overflow: hidden; }
.video-pane { margin-right: 0; flex: 0 0 auto; height: 40dvh; }
.video-wrap video { width: 100%; height: 100%; object-fit: contain; }
.list-pane {
position: relative;
top: auto; right: auto; bottom: auto;
width: 100%;
flex: 1;
border-left: none;
border-top: 1px solid var(--border);
}
.sub-times { min-width: 80px; font-size: 10px; }
.current-subtitle { font-size: clamp(11px, 3.5vw, 16px); }
}
</style>
</head>
<body>
<div class="main">
<div class="video-pane" id="videoPaneEl">
<div class="video-wrap">
<video id="vid" controls src="/video"></video>
<div class="current-subtitle" id="curSub"></div>
</div>
</div>
<div class="list-pane" id="listPane">
<!-- Panel header: title + bulk actions -->
<div class="panel-header">
<span class="panel-title">字幕</span>
<button class="btn" id="btnSelectAll">全选</button>
<button class="btn danger" id="btnDeleteSelected">删除</button>
</div>
<!-- find bar: hidden by default, Ctrl+F to show -->
<div class="find-bar" id="findBar">
<label>查找</label><input id="findInput" placeholder="大小写不敏感…">
<label>替换</label><input id="replaceInput" placeholder="新文本…">
<button class="btn" id="btnReplaceOne">替换下一个</button>
<button class="btn" id="btnReplaceAll">全部替换</button>
</div>
<!-- subtitle count row -->
<div class="list-header">
<input type="checkbox" id="selectAllCheck" title="全选">
<span id="listInfo"></span>
</div>
<div class="subtitle-list" id="list"></div>
<!-- Panel footer: status + save -->
<div class="panel-footer">
<div class="panel-status" id="statusText"></div>
<button class="btn save" id="btnSave">保存并关闭</button>
</div>
</div>
</div>
<script>
const LS_KEY = 'subtitle_editor_v1';
// ── state ────────────────────────────────────────────────────────────────────
let segments = [];
let deletedIds = new Set();
let editMode = null;
let findIndex = -1;
let currentNeedle = '';
// ── DOM ─────────────────────────────────────────────────────────────────────
const vid = document.getElementById('vid');
const curSub = document.getElementById('curSub');
const listEl = document.getElementById('list');
const findInput = document.getElementById('findInput');
const repInput = document.getElementById('replaceInput');
const selAll = document.getElementById('selectAllCheck');
const listInfo = document.getElementById('listInfo');
const statusTxt = document.getElementById('statusText');
const listPane = document.getElementById('listPane');
const videoPaneEl = document.getElementById('videoPaneEl');
// ── find bar toggle (Ctrl+F) ──────────────────────────────────────────────────
const findBarEl = document.getElementById('findBar');
function openFindBar() {
findBarEl.classList.add('show');
findInput.focus();
findInput.select();
}
function closeFindBar() {
findBarEl.classList.remove('show');
currentNeedle = '';
render();
}
document.addEventListener('keydown', e => {
if ((e.ctrlKey || e.metaKey) && e.key === 'f') {
e.preventDefault();
findBarEl.classList.contains('show') ? closeFindBar() : openFindBar();
}
if (e.key === 'Escape' && findBarEl.classList.contains('show')) {
closeFindBar();
}
});
document.getElementById('btnClosePanel')?.addEventListener('click', closeFindBar);
// ── helpers ─────────────────────────────────────────────────────────────────
function escapeHtml(s) {
return s.replace(/&/g,'&').replace(/</g,'<').replace(/>/g,'>');
}
function escapeRe(s) {
return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
function insertMarks(text, needle) {
if (!needle) return escapeHtml(text);
const re = new RegExp(escapeRe(needle), 'gi');
return escapeHtml(text).replace(re, m => `<mark>${m}</mark>`);
}
function fmtSeg(seg) {
const fmt = s => {
const m = Math.floor(s / 60);
const sec = (s % 60).toFixed(2).padStart(5, '0');
return `${m}:${sec}`;
};
return `${fmt(seg.start)} → ${fmt(seg.end)}`;
}
function getVisibleSegments() {
return segments.filter(s => !deletedIds.has(s._id));
}
// ── boot ────────────────────────────────────────────────────────────────────
async function init() {
const cached = localStorage.getItem(LS_KEY);
if (cached) {
try {
const data = JSON.parse(cached);
segments = data.segments || [];
deletedIds = new Set(data.deletedIds || []);
statusTxt.textContent = '已从 localStorage 恢复编辑进度';
} catch {
await loadFromServer();
}
} else {
await loadFromServer();
}
render();
updateInfo();
}
async function loadFromServer() {
const r = await fetch('/api/transcript');
const data = await r.json();
segments = data.segments || [];
deletedIds = new Set();
}
init();
// ── video sync ─────────────────────────────────────────────────────────────
vid.addEventListener('timeupdate', () => {
const t = vid.currentTime;
let active = null;
for (const s of segments) {
if (!deletedIds.has(s._id) && t >= s.start && t <= s.end) {
active = s; break;
}
}
if (active) {
curSub.textContent = active.text.trim();
curSub.classList.add('visible');
} else {
curSub.textContent = '';
curSub.classList.remove('visible');
}
document.querySelectorAll('.sub-item').forEach(el => {
const isActive = active && el.dataset.id == active._id;
el.classList.toggle('current', !!isActive);
if (isActive) el.scrollIntoView({ block: 'nearest', behavior: 'smooth' });
});
});
// ── render ──────────────────────────────────────────────────────────────────
function render() {
listEl.innerHTML = '';
const vis = getVisibleSegments();
selAll.checked = vis.length > 0 && vis.every(s => !deletedIds.has(s._id));
segments.forEach(seg => {
const id = seg._id || (seg._id = Math.random().toString(36).slice(2));
const isDel = deletedIds.has(id);
const isEdit = editMode === id;
const item = document.createElement('div');
item.className = 'sub-item' + (isDel ? ' deleted' : '') + (isEdit ? ' current' : '');
item.dataset.id = id;
const times = document.createElement('div');
times.className = 'sub-times';
times.textContent = fmtSeg(seg);
// Checkbox — stop all propagation so clicks don't bubble to row seek
const check = document.createElement('input');
check.type = 'checkbox';
check.className = 'sub-check';
check.checked = isDel;
check.addEventListener('click', e => { e.stopPropagation(); toggleDelete(id); });
check.addEventListener('mousedown', e => e.stopPropagation());
check.addEventListener('touchstart', e => e.stopPropagation());
const textWrap = document.createElement('div');
textWrap.className = 'sub-text-wrap';
const textEl = document.createElement('div');
textEl.className = 'sub-text';
textEl.contentEditable = 'false';
// When needle active show highlights; otherwise plain text
if (currentNeedle) {
textEl.innerHTML = insertMarks(seg.text, currentNeedle);
} else {
textEl.textContent = seg.text;
}
// Click row → seek video (only when not editing this item)
textEl.addEventListener('mousedown', e => {
if (textEl.contentEditable === 'true') {
e.stopPropagation(); // don't seek while editing
}
});
textEl.addEventListener('dblclick', e => {
e.stopPropagation();
startEdit(id, textEl);
});
textEl.addEventListener('blur', () => {
if (textEl.contentEditable === 'true') {
finishEdit(id, textEl.innerText.trim());
textEl.contentEditable = 'false';
// Restore highlights if needle still active
if (currentNeedle) textEl.innerHTML = insertMarks(seg.text, currentNeedle);
else textEl.textContent = seg.text;
}
});
textEl.addEventListener('keydown', e => {
if (e.key === 'Escape') {
textEl.contentEditable = 'false';
textEl.textContent = seg.text;
editMode = null;
}
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault();
textEl.blur();
}
});
textWrap.appendChild(textEl);
// Action buttons
const actions = document.createElement('div');
actions.className = 'sub-actions';
const editBtn = document.createElement('button');
editBtn.className = 'icon-btn';
editBtn.textContent = '✎';
editBtn.title = '编辑';
editBtn.addEventListener('click', e => { e.stopPropagation(); startEdit(id, textEl); });
const delBtn = document.createElement('button');
delBtn.className = 'icon-btn del';
delBtn.textContent = '✕';
delBtn.title = '删除';
delBtn.addEventListener('click', e => { e.stopPropagation(); toggleDelete(id); });
actions.appendChild(editBtn);
actions.appendChild(delBtn);
// Click row → seek video (only when text not in edit mode)
item.addEventListener('click', e => {
const el = listEl.querySelector(`[data-id="${id}"] .sub-text`);
if (el && el.contentEditable === 'true') return;
vid.currentTime = seg.start;
// Immediately show this subtitle without waiting for timeupdate
curSub.textContent = seg.text.trim();
curSub.classList.add('visible');
});
item.appendChild(check);
item.appendChild(times);
item.appendChild(textWrap);
item.appendChild(actions);
listEl.appendChild(item);
});
}
function startEdit(id, textEl) {
editMode = id;
textEl.contentEditable = 'true';
textEl.innerHTML = '';
const seg = segments.find(s => s._id === id);
if (seg) textEl.textContent = seg.text;
textEl.focus();
// Move cursor to end
const range = document.createRange();
range.selectNodeContents(textEl);
range.collapse(false);
const sel = window.getSelection();
sel.removeAllRanges();
sel.addRange(range);
}
function finishEdit(id, value) {
const seg = segments.find(s => s._id === id);
if (seg) seg.text = value;
editMode = null;
saveLS();
}
function toggleDelete(id) {
deletedIds.has(id) ? deletedIds.delete(id) : deletedIds.add(id);
saveLS();
render();
updateInfo();
}
function updateInfo() {
const total = segments.length;
const vis = getVisibleSegments().length;
listInfo.textContent = `共 ${total} 条 | 显示 ${vis} | 已删除 ${total - vis}`;
}
// ── localStorage ────────────────────────────────────────────────────────────
function saveLS() {
localStorage.setItem(LS_KEY, JSON.stringify({
segments,
deletedIds: [...deletedIds]
}));
}
// ── find / replace ──────────────────────────────────────────────────────────
// Update highlights on every keystroke in find input
findInput.addEventListener('input', () => {
currentNeedle = findInput.value;
render();
});
function doFind(replaceVal, replaceOne) {
const needle = findInput.value;
if (!needle) return;
const re = new RegExp(escapeRe(needle), 'i');
const visible = getVisibleSegments();
let startIdx = findIndex < 0 ? 0 : findIndex + (replaceOne ? 1 : 0);
for (let i = 0; i < visible.length; i++) {
const idx = (startIdx + i) % visible.length;
if (re.test(visible[idx].text)) {
findIndex = idx;
const el = listEl.querySelector(`[data-id="${visible[idx]._id}"]`);
if (el) {
el.scrollIntoView({ block: 'nearest' });
el.style.outline = '2px solid var(--accent)';
setTimeout(() => { if (el) el.style.outline = ''; }, 1500);
}
if (replaceOne) {
const seg = segments.find(s => s._id === visible[idx]._id);
if (seg) {
seg.text = seg.text.replace(new RegExp(escapeRe(needle), 'gi'), replaceVal);
saveLS();
render();
}
} else if (replaceVal) {
let count = 0;
segments.forEach(s => {
if (deletedIds.has(s._id)) return;
const next = s.text.replace(new RegExp(escapeRe(needle), 'gi'), replaceVal);
if (next !== s.text) { s.text = next; count++; }
});
statusTxt.textContent = `已替换 ${count} 处`;
saveLS();
render();
return;
}
return;
}
}
statusTxt.textContent = '未找到: ' + needle;
}
document.getElementById('btnReplaceOne').addEventListener('click', () => doFind(repInput.value, true));
document.getElementById('btnReplaceAll').addEventListener('click', () => doFind(repInput.value, false));
findInput.addEventListener('keydown', e => { if (e.key === 'Enter') doFind(repInput.value, true); });
// ── bulk select / delete ────────────────────────────────────────────────────
document.getElementById('btnSelectAll').addEventListener('click', () => {
const allDel = segments.every(s => deletedIds.has(s._id));
if (allDel) { deletedIds.clear(); statusTxt.textContent = '已取消全部删除'; }
else { segments.forEach(s => deletedIds.add(s._id)); statusTxt.textContent = '已选中全部'; }
saveLS();
render();
updateInfo();
});
document.getElementById('btnDeleteSelected').addEventListener('click', () => {
const sel = segments.filter(s => deletedIds.has(s._id));
if (!sel.length) { statusTxt.textContent = '请先勾选要删除的条目'; return; }
sel.forEach(s => deletedIds.add(s._id));
saveLS();
render();
updateInfo();
statusTxt.textContent = `已删除 ${sel.length} 条`;
});
selAll.addEventListener('change', () => {
if (selAll.checked) segments.forEach(s => deletedIds.add(s._id));
else deletedIds.clear();
saveLS();
render();
updateInfo();
});
// ── save & close ─────────────────────────────────────────────────────────────
document.getElementById('btnSave').addEventListener('click', async () => {
// Flush any in-progress contenteditable edit before saving
const active = document.activeElement;
if (active && active.classList.contains('sub-text')) active.blur();
const toSave = segments
.filter(s => !deletedIds.has(s._id))
.map(({ _id, ...rest }) => rest);
const res = await fetch('/api/transcript', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ segments: toSave })
});
if (res.ok) {
localStorage.removeItem(LS_KEY);
statusTxt.textContent = `✅ 已保存 ${toSave.length} 条字幕,可以关闭此标签页`;
window.close(); // may be blocked by browser; user can close manually
} else {
statusTxt.textContent = '保存失败,请重试';
}
});
</script>
</body>
</html>"""
# ----------------------------------------------------------------------------- #
# Routes
# ----------------------------------------------------------------------------- #
@app.route("/")
def index():
return Response(HTML_TEMPLATE, content_type="text/html; charset=utf-8")
@app.route("/video")
def video():
if not os.path.exists(VIDEO_PATH):
return "Video not found", 404
return send_file(VIDEO_PATH, mimetype="video/mp4")
@app.route("/api/transcript", methods=["GET"])
def get_transcript():
with open(TRANSCRIPT_PATH, encoding="utf-8") as f:
data = json.load(f)
# Support both {"segments": [...]} and plain [...]
segs = data.get("segments", data) if isinstance(data, dict) else data
return jsonify({"segments": segs})
@app.route("/api/transcript", methods=["POST"])
def post_transcript():
body = request.get_json()
output = {"segments": body["segments"]}
with open(TRANSCRIPT_PATH, "w", encoding="utf-8") as f:
json.dump(output, f, ensure_ascii=False, indent=2)
RESULT_DONE.set()
return jsonify({"ok": True})
# ----------------------------------------------------------------------------- #
# Main
# ----------------------------------------------------------------------------- #
def main():
global VIDEO_PATH, TRANSCRIPT_PATH
if len(sys.argv) < 3:
print("Usage: preview_editor.py <video.mp4> <transcript.json>")
sys.exit(1)
VIDEO_PATH = os.path.abspath(sys.argv[1])
TRANSCRIPT_PATH = os.path.abspath(sys.argv[2])
for p in [VIDEO_PATH, TRANSCRIPT_PATH]:
if not Path(p).exists():
print(f"❌ File not found: {p}")
sys.exit(1)
# Save pre-edit snapshot for Claude to diff after the session
import shutil
orig_backup = TRANSCRIPT_PATH + ".orig.json"
shutil.copy2(TRANSCRIPT_PATH, orig_backup)
print(f"[preview] Snapshot saved: {orig_backup}")
url = f"http://localhost:{PORT}"
print(f"[preview] Video: {VIDEO_PATH}")
print(f"[preview] Transcript: {TRANSCRIPT_PATH}")
print(f"[preview] Opening {url} …")
webbrowser.open(url)
app.run(host="0.0.0.0", port=PORT, debug=False, use_reloader=False, threaded=True)
print("[preview] Exiting.")
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Screen Studio Auto-Editor
Removes pauses and repeated narration from a .screenstudio project,
and enables native captions.
"""
import argparse
import json
import random
import shutil
import string
import subprocess
import sys
import tempfile
from pathlib import Path
def log(msg):
print(f"[screen-studio-editor] {msg}", flush=True)
def backup_project(project_json_path: Path):
bak = project_json_path.with_suffix(".json.bak")
if bak.exists():
log(f"⚠️ Backup already exists at {bak}, skipping backup.")
return
shutil.copy2(project_json_path, bak)
log(f"✅ Backed up project.json → {bak.name}")
def load_metadata(project_dir: Path) -> dict:
"""Load recording/metadata.json to get session timing info."""
metadata_path = project_dir / "recording" / "metadata.json"
with open(metadata_path) as f:
return json.load(f)
def get_mic_sessions(metadata: dict) -> list[dict]:
"""Return microphone sessions sorted by processTimeStartMs."""
sessions = []
for recorder in metadata.get("recorders", []):
if recorder.get("type") == "microphone" or "microphone" in recorder.get("id", ""):
for s in recorder.get("sessions", []):
sessions.append(s)
if not sessions:
# fallback: look for sessions in input recorder
for recorder in metadata.get("recorders", []):
if "input" in recorder.get("id", ""):
for s in recorder.get("sessions", []):
sessions.append(s)
return sorted(sessions, key=lambda s: s["processTimeStartMs"])
def merge_audio(project_dir: Path, sessions: list[dict], output_path: Path) -> list[dict]:
"""
Merge all microphone .m4a segments into a single WAV (concatenated, no gap padding).
Returns session offset mappings for timestamp conversion:
[{"processTimeStartMs": ..., "audioOffsetMs": ..., "durationMs": ...}, ...]
audioOffsetMs = where this session starts in the merged audio (cumulative ms).
"""
recording_dir = project_dir / "recording"
valid_sessions = []
current_offset_ms = 0.0
for session in sessions:
filename = session.get("outputFilename", "")
mic_file = recording_dir / filename
if not mic_file.exists():
log(f"⚠️ Audio file not found: {filename}, skipping.")
continue
valid_sessions.append({
"processTimeStartMs": session["processTimeStartMs"],
"durationMs": session["durationMs"],
"audioOffsetMs": current_offset_ms,
"file": str(mic_file),
})
current_offset_ms += session["durationMs"]
if not valid_sessions:
raise RuntimeError("No microphone audio files found in this project.")
n = len(valid_sessions)
inputs = []
for s in valid_sessions:
inputs.extend(["-i", s["file"]])
concat_filter = "".join([f"[{j}:a]" for j in range(n)])
concat_filter += f"concat=n={n}:v=0:a=1[outa]"
cmd = inputs + [
"-filter_complex", concat_filter,
"-map", "[outa]",
"-ar", "16000",
"-ac", "1",
str(output_path),
"-y",
]
log(f"🎵 Merging {n} audio segment(s)...")
result = subprocess.run(["ffmpeg", "-loglevel", "error"] + cmd, capture_output=True, text=True)
if result.returncode != 0:
raise RuntimeError(f"ffmpeg failed: {result.stderr}")
log(f"✅ Merged audio → {output_path.name}")
return valid_sessions
def transcribe(audio_path: Path, language: str | None = "zh") -> list[dict]:
"""
Run mlx-whisper large-v3 on the audio file.
Returns list of segments with word-level timestamps:
[{"start": float, "end": float, "text": str, "words": [{"word": str, "start": float, "end": float}]}]
language: ISO 639-1 code (e.g. 'zh', 'en') or None for auto-detect.
Defaults to 'zh' to prevent Whisper from outputting Traditional Chinese for Mandarin content.
Pass --language en for English recordings, or --language None to auto-detect.
"""
lang_display = language if language else "auto-detect"
log(f"🎙️ Transcribing with mlx-whisper large-v3 (language={lang_display}, ~10-30s)...")
import mlx_whisper
result = mlx_whisper.transcribe(
str(audio_path),
path_or_hf_repo="mlx-community/whisper-large-v3-mlx",
word_timestamps=True,
language=language,
)
segments = result.get("segments", [])
log(f"✅ Transcribed {len(segments)} segments, {sum(len(s.get('words',[])) for s in segments)} words.")
return segments
def map_audio_time_to_process_time(audio_offset_s: float, session_offsets: list[dict]) -> float:
"""
Convert a timestamp in the merged audio (seconds) to slice coordinate.
The sourceStartMs / sourceEndMs fields in Screen Studio slices use the
*merged audio timeline* as their coordinate system: sessions are laid out
back-to-back starting at 0, with no gaps between them. That means the
slice coordinate equals the merged-audio offset in milliseconds — which is
simply the audio timestamp itself.
"""
return audio_offset_s * 1000.0
def detect_pauses(segments: list[dict], threshold_ms: float, min_pause_ms: float, session_offsets: list[dict]) -> list[dict]:
"""
Detect pauses between words longer than threshold_ms.
Returns list of {"start_process_ms": ..., "end_process_ms": ..., "duration_ms": ...}
representing ranges to cut (leaving min_pause_ms of silence).
"""
words = []
for seg in segments:
for w in seg.get("words", []):
words.append(w)
if not words:
return []
pauses = []
for i in range(len(words) - 1):
gap_start_s = words[i]["end"]
gap_end_s = words[i + 1]["start"]
gap_ms = (gap_end_s - gap_start_s) * 1000.0
if gap_ms > threshold_ms:
# We'll cut from (gap_start + min_pause_ms/2) to (gap_end - min_pause_ms/2)
# to leave a natural-sounding gap.
# Extra 80ms padding guards against Whisper timestamp inaccuracy (±50-200ms):
# Whisper often reports word boundaries slightly early or late, so leaving
# an extra buffer prevents accidentally clipping the tail of the preceding
# word or the onset of the following word.
SPEECH_PAD_S = 0.08
cut_start_s = gap_start_s + (min_pause_ms / 2000.0) + SPEECH_PAD_S
cut_end_s = gap_end_s - (min_pause_ms / 2000.0) - SPEECH_PAD_S
cut_duration_ms = (cut_end_s - cut_start_s) * 1000.0
# Skip cuts shorter than 300ms: Whisper timestamp inaccuracy (±50-200ms)
# means a tiny cut is more likely to clip real speech than remove silence.
if cut_end_s > cut_start_s and cut_duration_ms >= 300:
start_pm = map_audio_time_to_process_time(cut_start_s, session_offsets)
end_pm = map_audio_time_to_process_time(cut_end_s, session_offsets)
pauses.append({
"start_ms": start_pm,
"end_ms": end_pm,
"duration_ms": gap_ms,
"text_before": words[i]["word"].strip(),
"text_after": words[i + 1]["word"].strip(),
})
log(f"🔍 Found {len(pauses)} pause(s) > {threshold_ms}ms to cut.")
return pauses
def detect_silence_regions(audio_path: Path, noise_db: float = -28.0, min_dur: float = 0.3) -> list[tuple[float, float]]:
"""
Use ffmpeg silencedetect to find actual silent regions in the merged audio.
Returns list of (start_s, end_s) tuples.
noise_db: threshold in dB (default -28dB, matches auto-editor's default)
min_dur: minimum silence duration in seconds (default 0.3s)
"""
cmd = [
"ffmpeg", "-i", str(audio_path),
"-af", f"silencedetect=noise={noise_db}dB:d={min_dur}",
"-f", "null", "-",
]
result = subprocess.run(cmd, capture_output=True, text=True)
output = result.stderr
regions = []
silence_start = None
for line in output.splitlines():
if "silence_start" in line:
try:
silence_start = float(line.split("silence_start:")[1].strip())
except (IndexError, ValueError):
pass
elif "silence_end" in line and silence_start is not None:
try:
silence_end = float(line.split("silence_end:")[1].split("|")[0].strip())
regions.append((silence_start, silence_end))
silence_start = None
except (IndexError, ValueError):
pass
log(f"🔇 silencedetect ({noise_db}dB, min {min_dur}s): found {len(regions)} region(s).")
return regions
def filter_pauses_by_silence(pauses: list[dict], silence_regions: list[tuple[float, float]]) -> list[dict]:
"""
Validate each pause cut against actual silence regions.
A cut is 'confirmed' if its interval overlaps with a silence region by >= 50% of the cut's duration.
Unconfirmed cuts are skipped to avoid clipping real speech.
"""
if not silence_regions:
log("⚠️ No silence regions detected — skipping silence validation.")
return pauses
confirmed = []
skipped = []
for p in pauses:
cut_start_s = p["start_ms"] / 1000.0
cut_end_s = p["end_ms"] / 1000.0
cut_dur = cut_end_s - cut_start_s
# Find max overlap with any silence region
max_overlap = 0.0
for (sr_start, sr_end) in silence_regions:
overlap = max(0.0, min(cut_end_s, sr_end) - max(cut_start_s, sr_start))
max_overlap = max(max_overlap, overlap)
overlap_ratio = max_overlap / cut_dur if cut_dur > 0 else 0.0
if overlap_ratio >= 0.5:
confirmed.append(p)
else:
skipped.append(p)
log(f" ⏭️ Skip (overlap={overlap_ratio:.0%}): [{cut_start_s:.2f}s→{cut_end_s:.2f}s] "
f"「{p['text_before']}」...「{p['text_after']}」")
log(f"✅ Silence validation: {len(confirmed)} confirmed, {len(skipped)} skipped.")
return confirmed
def detect_repeats(cuts_file: str) -> list[dict]:
"""
Load repeat cuts from a JSON file produced by Claude in the conversation.
Format: [{"start_ms": ..., "end_ms": ..., "removed_text": "..."}]
"""
if not cuts_file:
return []
import os
if not os.path.exists(cuts_file):
log(f"⚠️ Cuts file not found: {cuts_file}, skipping repeat detection.")
return []
with open(cuts_file) as f:
repeats = json.load(f)
log(f"✂️ Loaded {len(repeats)} repeat cut(s) from {cuts_file}")
for r in repeats:
log(f" Remove: \"{r.get('removed_text', '')[:70]}\"")
return repeats
def get_session_boundaries_ms(session_offsets: list[dict]) -> list[float]:
"""
Return the merged-audio timestamps (in ms) at which each new recording session begins.
These are the "natural" transition points that Screen Studio originally placed between sessions.
The first session always starts at 0 so only boundaries *between* sessions are returned.
"""
boundaries = []
cumulative = 0.0
for s in session_offsets:
cumulative += s["durationMs"]
boundaries.append(cumulative)
return boundaries[:-1] # exclude the very end; only inter-session boundaries
_SNAP_MARGIN_MS = 10 # snap 10ms before boundary; see snap_to_session_boundaries docstring
def snap_to_session_boundaries(slices: list[dict], boundaries_ms: list[float]) -> tuple[list[dict], int]:
"""
After pause-removal cuts, some inter-session boundaries end up inside a gap between
slices (because the removed pause straddled the boundary). Screen Studio originally
rendered a smooth spring transition at those boundaries; the cuts destroy it.
Fix: if a boundary B falls inside a gap [slice_n.sourceEndMs, slice_{n+1}.sourceStartMs],
clamp both endpoints to (B - MARGIN). This makes the pair source-contiguous
(gap = 0 ms) which re-enables Screen Studio's spring animation, while keeping
the snap point safely inside session N's audio range.
Why the margin is necessary: metadata durationMs and the actual WAV file length
differ by ~0.3 µs. Snapping to the exact boundary causes Screen Studio's audio
composer to generate a sub-millisecond segment at the session seam, which it
rejects ("Invalid time range: duration must be positive"). A 10 ms margin
produces a short but valid segment (~10 ms of near-silence from the tail of
session N) that the audio composer accepts. 10 ms is imperceptible.
Returns (updated_slices, num_boundaries_snapped).
"""
if not boundaries_ms or len(slices) < 2:
return slices, 0
snapped = 0
for b in boundaries_ms:
snap_point = b - _SNAP_MARGIN_MS
for i in range(len(slices) - 1):
end_i = slices[i]["sourceEndMs"]
start_n = slices[i + 1]["sourceStartMs"]
if end_i <= snap_point <= start_n:
slices[i]["sourceEndMs"] = snap_point
slices[i + 1]["sourceStartMs"] = snap_point
snapped += 1
log(f"🔗 Session boundary {b/1000:.3f}s snapped between slice {i} and {i+1} → gap restored to 0 (margin={_SNAP_MARGIN_MS}ms)")
break # each boundary can only fall in one gap
return slices, snapped
def apply_cuts(slices: list[dict], cuts: list[dict]) -> tuple[list[dict], int]:
"""
Given the existing slices and a list of time ranges to cut,
return updated slices with those ranges removed.
Each cut: {"start_ms": ..., "end_ms": ...}
Each slice: {"sourceStartMs": ..., "sourceEndMs": ..., ...}
Returns (new_slices, num_cuts_applied).
"""
if not cuts:
return slices, 0
# Sort cuts
cuts = sorted(cuts, key=lambda c: c["start_ms"])
new_slices = []
cuts_applied = 0
for sl in slices:
sl_start = sl["sourceStartMs"]
sl_end = sl["sourceEndMs"]
remaining = [(sl_start, sl_end)]
for cut in cuts:
cut_start = cut["start_ms"]
cut_end = cut["end_ms"]
new_remaining = []
for (rs, re) in remaining:
if cut_end <= rs or cut_start >= re:
# No overlap
new_remaining.append((rs, re))
elif cut_start <= rs and cut_end >= re:
# Cut removes entire segment
cuts_applied += 1
else:
# Partial overlap
if cut_start > rs:
new_remaining.append((rs, cut_start))
if cut_end < re:
new_remaining.append((cut_end, re))
cuts_applied += 1
remaining = new_remaining
for (start, end) in remaining:
if end - start > 100: # Skip tiny fragments < 100ms
new_slice = dict(sl)
new_slice["sourceStartMs"] = start
new_slice["sourceEndMs"] = end
# Each slice must have a unique id or Screen Studio collapses them
# into a single uneditable block. Transitions in Screen Studio are
# triggered by consecutive same-ID slices, but that only works for
# small groups (2-3 slices). With many cuts, unique IDs are required
# to preserve editability in the UI.
new_slice["id"] = ''.join(random.choices(string.ascii_letters + string.digits, k=10))
new_slices.append(new_slice)
return new_slices, cuts_applied
def main():
parser = argparse.ArgumentParser(description="Screen Studio Auto-Editor")
parser.add_argument("--project", required=True, help="Path to .screenstudio directory")
parser.add_argument("--pause-threshold", type=float, default=800, help="Pause threshold in ms (default: 800)")
parser.add_argument("--min-pause", type=float, default=800, help="Minimum pause to keep in ms (default: 800)")
parser.add_argument("--cuts-file", default=None, help="JSON file with repeat cuts (produced by Claude in conversation)")
parser.add_argument("--skip-transcribe", help="Path to existing transcript JSON to reuse")
parser.add_argument("--language", default="zh",
help="Whisper language code (default: zh). Use 'en' for English, 'None' to auto-detect.")
args = parser.parse_args()
project_dir = Path(args.project)
if not project_dir.exists() or not project_dir.is_dir():
print(f"❌ Project not found: {project_dir}")
sys.exit(1)
project_json_path = project_dir / "project.json"
backup_path = project_dir / "project.json.bak"
# Backup (no-op if backup already exists)
backup_project(project_json_path)
# Always load from backup — it is created on the first run and never
# overwritten, so it always holds the original unedited project.json.
# This makes every run idempotent: pauses + repeats are applied to the
# original single slice, never to already-cut slices from a prior run.
with open(backup_path) as f:
project_data = json.load(f)
# Load metadata
metadata = load_metadata(project_dir)
mic_sessions = get_mic_sessions(metadata)
if not mic_sessions:
log("❌ No microphone sessions found in metadata.")
sys.exit(1)
log(f"📁 Found {len(mic_sessions)} recording session(s).")
with tempfile.TemporaryDirectory() as tmpdir:
tmp = Path(tmpdir)
# Merge audio
merged_audio = tmp / "merged_mic.wav"
session_offsets = merge_audio(project_dir, mic_sessions, merged_audio)
# Transcribe
if args.skip_transcribe:
with open(args.skip_transcribe) as f:
segments = json.load(f)
log(f"♻️ Loaded existing transcript from {args.skip_transcribe}")
else:
lang = None if args.language == "None" else args.language
segments = transcribe(merged_audio, language=lang)
# Save transcript for debugging
transcript_out = project_dir / "transcript.json"
with open(transcript_out, "w") as f:
json.dump(segments, f, indent=2, ensure_ascii=False)
log(f"💾 Saved transcript → transcript.json")
# Detect pauses
pauses = detect_pauses(segments, args.pause_threshold, args.min_pause, session_offsets)
# Validate pause cuts against actual audio silence regions.
# This filters out cuts where Whisper detected a gap but the audio is not
# actually silent (timestamp inaccuracy). Uses -28dB threshold (same as
# auto-editor default), which is lenient enough to catch real inter-word
# pauses without triggering on background room noise.
silence_regions = detect_silence_regions(merged_audio)
pauses = filter_pauses_by_silence(pauses, silence_regions)
# Load repeat cuts (produced by Claude in the conversation, if any)
repeats = detect_repeats(args.cuts_file)
# Add 150ms inward padding to repeat cuts to protect neighboring speech from clipping.
# Whisper start/end timestamps for a segment have ±100-200ms inaccuracy — the "start_ms"
# of the region to remove might land slightly inside the last good word. Shrinking each
# repeat cut inward by 150ms on both ends preserves the natural word onset/offset.
REPEAT_PAD_MS = 150
padded_repeats = []
for r in repeats:
padded = dict(r)
padded["start_ms"] = r["start_ms"] + REPEAT_PAD_MS
padded["end_ms"] = r["end_ms"] - REPEAT_PAD_MS
if padded["end_ms"] > padded["start_ms"] + 200:
padded_repeats.append(padded)
else:
log(f"⚠️ Repeat cut too short after padding, skipping: \"{r.get('removed_text','')[:50]}\"")
repeats = padded_repeats
all_cuts = pauses + repeats
original_slices = project_data["json"]["scenes"][0]["slices"]
new_slices, cuts_applied = apply_cuts(original_slices, all_cuts)
session_boundaries = get_session_boundaries_ms(session_offsets)
new_slices, snapped = snap_to_session_boundaries(new_slices, session_boundaries)
# Calculate time saved
original_duration = sum(s["sourceEndMs"] - s["sourceStartMs"] for s in original_slices)
new_duration = sum(s["sourceEndMs"] - s["sourceStartMs"] for s in new_slices)
saved_ms = original_duration - new_duration
# Update project
project_data["json"]["scenes"][0]["slices"] = new_slices
project_data["json"]["config"]["showTranscript"] = True
project_data["json"]["config"]["backgroundPaddingRatio"] = 1.02 # 2% padding
project_data["json"]["config"]["cameraAspectRatio"] = "square" # 宽高一致(正方形)
project_data["json"]["config"]["improveMicrophoneAudio"] = True # 降噪 + 音量均一化
# Note: cameraRoundness is intentionally NOT set — keep Screen Studio's default roundness
# Write updated project.json
with open(project_json_path, "w") as f:
json.dump(project_data, f, ensure_ascii=False, separators=(",", ":"))
log("")
log("=" * 50)
log("✅ Done! Summary:")
log(f" Pauses removed: {len(pauses)}")
log(f" Repeats removed: {len(repeats)}")
log(f" Total cuts: {cuts_applied}")
log(f" Original duration: {original_duration/1000:.1f}s")
log(f" New duration: {new_duration/1000:.1f}s")
log(f" Time saved: {saved_ms/1000:.1f}s ({saved_ms/original_duration*100:.1f}%)")
log(f" Captions: enabled ✓")
log("")
log("Open Screen Studio to preview the result.")
log("Backup saved as project.json.bak")
if repeats:
log("")
log("Removed repeated segments:")
for r in repeats:
log(f" ✂️ \"{r.get('removed_text', '')[:80]}\"")
if __name__ == "__main__":
main()
#!/bin/bash
# Screen Studio Editor — one-time setup
# Run once after installing the skill: bash setup.sh
# Requirements: macOS on Apple Silicon (M1/M2/M3), Homebrew
set -e
SKILL_DIR="$(cd "$(dirname "$0")" && pwd)"
cd "$SKILL_DIR"
echo "=== Screen Studio Editor Setup ==="
echo "Skill directory: $SKILL_DIR"
echo ""
# 1. Check platform
if [[ "$(uname)" != "Darwin" ]]; then
echo "ERROR: This skill requires macOS (Apple Silicon)."
exit 1
fi
if [[ "$(uname -m)" != "arm64" ]]; then
echo "ERROR: This skill requires Apple Silicon (M1/M2/M3)."
echo " mlx-whisper does not run on Intel Macs."
exit 1
fi
echo "[1/4] Platform: macOS Apple Silicon OK"
# 2. Check ffmpeg
if ! command -v ffmpeg &>/dev/null; then
echo ""
echo "ffmpeg not found. Installing via Homebrew..."
if ! command -v brew &>/dev/null; then
echo "ERROR: Homebrew is required. Install from https://brew.sh then re-run this script."
exit 1
fi
brew install ffmpeg
fi
echo "[2/4] ffmpeg: $(ffmpeg -version 2>&1 | head -1 | cut -d' ' -f1-3) OK"
# 3. Create Python venv and install dependencies
echo "[3/4] Setting up Python environment..."
if [[ ! -d ".venv" ]]; then
python3 -m venv .venv
fi
.venv/bin/pip install --quiet --upgrade pip
.venv/bin/pip install --quiet mlx-whisper
echo " Python venv ready"
# 4. Pre-download Whisper large-v3 model (~3 GB, one-time)
echo "[4/4] Downloading Whisper large-v3 model (~3 GB, one-time)..."
echo " This may take a few minutes on first run."
"$SKILL_DIR/.venv/bin/python3" -c "
import mlx_whisper, tempfile, subprocess, os
tmp = tempfile.mktemp(suffix='.wav')
subprocess.run(['ffmpeg','-f','lavfi','-i','anullsrc=r=16000:cl=mono','-t','1',tmp,'-y','-loglevel','quiet'])
mlx_whisper.transcribe(tmp, path_or_hf_repo='mlx-community/whisper-large-v3-mlx', language='zh')
os.unlink(tmp)
print('Model downloaded and verified.')
"
echo ""
echo "=== Setup complete ==="
echo ""
echo "You're ready to use the screen-studio-editor skill."
echo "Point Claude Code at this skill directory and start editing your recordings."