Ffmpeg Fundamentals 2025

FFmpeg Fundamentals (2025)

Foundational FFmpeg knowledge for 7.1 LTS and 8.0.1. This SKILL is a lean orchestrator; detailed recipes and tables live under references/.

When to Use This Skill

Use for foundational FFmpeg operations:

• First-time FFmpeg users needing syntax help
• Basic transcoding between formats (MP4, WebM, MKV, MOV)
• Understanding codec options and quality settings
• Simple filter operations (scale, crop, rotate)
• File analysis with ffprobe

For specialized topics see related skills: hardware encoding -> ffmpeg-hardware-acceleration, streaming -> ffmpeg-streaming, Docker -> ffmpeg-docker-containers, advanced filter graphs -> ffmpeg-filter-complex-patterns.

Quick Reference

Core Workflow

1. Verify your FFmpeg version with ffmpeg -version. Prefer the latest stable (8.0.1) or the 7.1 LTS. 2. Probe the input with ffprobe to inspect codecs, container, resolution, and duration. 3. Decide whether to remux (-c copy) or transcode (specify encoders, quality). 4. Pick encoders and quality settings (see Codec Reference below). 5. Add filters as needed (see references/filter-essentials.md and references/frame-manipulation.md). 6. Verify the output: re-probe, check duration, A/V sync, and pixel format.

Basic Command Syntax

ffmpeg [global_options] {[input_options] -i input} ... {[output_options] output} ...

Common examples:

# Basic transcode
ffmpeg -i input.mp4 output.mkv

# Specify codecs
ffmpeg -i input.mp4 -c:v libx264 -c:a aac output.mp4

# Copy streams without re-encoding (remux)
ffmpeg -i input.mkv -c copy output.mp4

# CRF quality mode
ffmpeg -i input.mp4 -c:v libx264 -crf 23 output.mp4

# Bitrate mode
ffmpeg -i input.mp4 -c:v libx264 -b:v 5M output.mp4

# Overwrite without prompting
ffmpeg -y -i input.mp4 output.mp4

# Show progress as parseable stats
ffmpeg -progress - -i input.mp4 output.mp4

For exhaustive option ordering rules, stream specifiers, and per-format option tables, see the ffmpeg-command-syntax skill.

Codec Reference (2025)

Video Codecs (Software)

Hardware Encoders

Hardware Decoders (FFmpeg 8.0+)

Audio Codecs

Probing & Analysis

# Get media info as JSON
ffprobe -v quiet -print_format json -show_format -show_streams input.mp4

# Get duration
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 input.mp4

# Get resolution
ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=p=0 input.mp4

# Get codec
ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 input.mp4

# Validate file
ffprobe -v error input.mp4 && echo "Valid" || echo "Invalid"

Best Practices

Performance

1. Use -threads 0 to auto-detect optimal thread count. 2. Place -ss before -i for fast seeking (may be less accurate); after -i for frame-accurate seek. 3. Use -c copy when possible to avoid re-encoding. 4. Use hardware acceleration when available.

Quality

1. Prefer CRF over bitrate for single-pass encoding. 2. Use -preset slow or slower for better compression. 3. Use -tune options (film, animation, grain) when appropriate. 4. Keep -pix_fmt yuv420p for broad compatibility.

Compatibility

1. Use -movflags +faststart for web MP4 files. 2. Use -vtag hvc1 for HEVC on Apple devices. 3. Test on target devices before mass conversion. 4. Provide fallback formats for older browsers.

Error Prevention

1. Test commands on a sample file first. 2. Use -report for detailed logging. 3. Check output file size and duration. 4. Verify A/V sync after conversion.

Capability Discovery

ffmpeg -encoders          # List encoders
ffmpeg -decoders          # List decoders
ffmpeg -formats           # List formats
ffmpeg -filters           # List filters
ffmpeg -hwaccels          # List hardware acceleration methods
ffmpeg -h encoder=libx264 # Encoder-specific options

Exit codes:

• 0: success
• 1: error (check stderr)
• 255: interrupted (Ctrl+C)

Reference Map

Detailed recipes, option tables, and FFmpeg 8.0-specific guides:

• references/version-history.md - Full FFmpeg 8.0 / 7.1 LTS feature lists, update strategy, official resources
• references/common-operations.md - Format conversion, scaling, trimming, concat, audio, subtitles, Whisper recipe, image ops, output optimization (web/iOS/Android)
• references/filter-essentials.md - Video filters (-vf), audio filters (-af), complex filtergraphs, filter_complex syntax reference
• references/frame-manipulation.md - Frame rate, setpts, select, thumbnails, tiles, freezing, looping, reversing
• references/whisper-transcription.md - Whisper AI filter for speech recognition and subtitle generation
• references/vvc-h266-encoding.md - VVC/H.266 encoding with libvvenc plus hardware decode
• references/vulkan-compute-codecs.md - Vulkan compute-based codecs (FFv1, AV1, VP9, ProRes RAW)

For deeper dives: ffmpeg-filter-complex-patterns, ffmpeg-hardware-acceleration, ffmpeg-streaming, ffmpeg-captions-subtitles, ffmpeg-audio-processing.

Common FFmpeg Operations

Recipes for format conversion, scaling, trimming, concatenation, audio operations, subtitles, Whisper transcription, and image operations.

Format Conversion

# MP4 to WebM
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus output.webm

# MKV to MP4 (no re-encoding)
ffmpeg -i input.mkv -c copy output.mp4

# AVI to MP4 with H.265
ffmpeg -i input.avi -c:v libx265 -crf 28 -c:a aac output.mp4

# GIF to MP4
ffmpeg -i input.gif -movflags +faststart -pix_fmt yuv420p output.mp4

# MP4 to GIF (optimized)
ffmpeg -i input.mp4 -vf "fps=15,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif

Resolution & Scaling

# Scale to 1080p (maintain aspect ratio)
ffmpeg -i input.mp4 -vf "scale=1920:-2" output.mp4

# Scale to 720p
ffmpeg -i input.mp4 -vf "scale=-2:720" output.mp4

# Scale to fit within bounds
ffmpeg -i input.mp4 -vf "scale='min(1920,iw)':'min(1080,ih)':force_original_aspect_ratio=decrease" output.mp4

# Scale with padding (letterbox)
ffmpeg -i input.mp4 -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" output.mp4

Trimming & Splitting

# Extract segment (fast seek)
ffmpeg -ss 00:01:00 -i input.mp4 -t 00:00:30 -c copy output.mp4

# Extract from timestamp to end
ffmpeg -ss 00:05:00 -i input.mp4 -c copy output.mp4

# Extract first 10 seconds
ffmpeg -i input.mp4 -t 10 -c copy output.mp4

# Split into segments
ffmpeg -i input.mp4 -c copy -map 0 -segment_time 60 -f segment output_%03d.mp4

# Accurate trim (re-encode)
ffmpeg -i input.mp4 -ss 00:01:00 -t 00:00:30 -c:v libx264 -c:a aac output.mp4

Concatenation

# Create file list
echo "file 'part1.mp4'" > list.txt
echo "file 'part2.mp4'" >> list.txt
echo "file 'part3.mp4'" >> list.txt

# Concatenate (same codecs)
ffmpeg -f concat -safe 0 -i list.txt -c copy output.mp4

# Concatenate with re-encoding
ffmpeg -f concat -safe 0 -i list.txt -c:v libx264 -c:a aac output.mp4

Audio Operations

# Extract audio
ffmpeg -i input.mp4 -vn -c:a copy output.aac
ffmpeg -i input.mp4 -vn -c:a libmp3lame -b:a 320k output.mp3

# Replace audio
ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -c:a aac -map 0:v:0 -map 1:a:0 output.mp4

# Add audio to video
ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -c:a aac -shortest output.mp4

# Remove audio
ffmpeg -i input.mp4 -an -c:v copy output.mp4

# Adjust volume
ffmpeg -i input.mp4 -af "volume=1.5" output.mp4
ffmpeg -i input.mp4 -af "volume=6dB" output.mp4

# Normalize audio (EBU R128)
ffmpeg -i input.mp4 -af loudnorm=I=-16:TP=-1.5:LRA=11 output.mp4

Subtitles

# Burn subtitles (hardcode)
ffmpeg -i input.mp4 -vf "subtitles=subs.srt" output.mp4

# Add subtitle track
ffmpeg -i input.mp4 -i subs.srt -c copy -c:s mov_text output.mp4

# Extract subtitles
ffmpeg -i input.mkv -map 0:s:0 output.srt

Whisper AI Transcription (FFmpeg 8.0+)

# Generate SRT subtitles from video using Whisper
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=auto:queue=3:destination=output.srt:format=srt" \
  -f null -

# Live transcription from microphone
ffmpeg -loglevel warning -f pulse -i default \
  -af "highpass=f=200,lowpass=f=3000,whisper=model=ggml-medium-q5_0.bin:language=en:queue=10:destination=-:format=json:vad_model=for-tests-silero-v5.1.2-ggml.bin" \
  -f null -

# Display live subtitles on video (reads from frame metadata)
ffmpeg -i input.mp4 \
  -af "whisper=model=ggml-base.en.bin:language=en" \
  -vf "drawtext=text='%{metadata\:lavfi.whisper.text}':fontsize=24:fontcolor=white:x=10:y=h-th-10" \
  output_with_subtitles.mp4

Whisper Model Sizes

Whisper Filter Parameters

• model: Path to GGML model file
• language: Language code or "auto" for detection
• format: Output format - "text", "srt", or "json"
• destination: Output file path (or "-" for stdout)
• queue: Buffer size (increase for VAD, e.g., 20)
• vad_model: Path to Silero VAD model for voice detection
• use_gpu: Set to "false" to disable GPU acceleration

Image Operations

# Video to images
ffmpeg -i input.mp4 -vf "fps=1" frame_%04d.png

# Images to video
ffmpeg -framerate 30 -i frame_%04d.png -c:v libx264 -pix_fmt yuv420p output.mp4

# Extract single frame
ffmpeg -ss 00:00:10 -i input.mp4 -vframes 1 thumbnail.jpg

# Create video from single image
ffmpeg -loop 1 -i image.jpg -c:v libx264 -t 10 -pix_fmt yuv420p output.mp4

Output Optimization

Web Delivery

# MP4 for web (faststart)
ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k -movflags +faststart output.mp4

# WebM for web
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus -b:a 128k output.webm

iOS/Apple Compatibility

# H.265 with hvc1 tag (required for Apple)
ffmpeg -i input.mp4 -c:v libx265 -vtag hvc1 -c:a aac output.mp4

# Baseline profile for older iOS
ffmpeg -i input.mp4 -c:v libx264 -profile:v baseline -level 3.0 -c:a aac output.mp4

Android Compatibility

# Widely compatible H.264
ffmpeg -i input.mp4 -c:v libx264 -profile:v main -level 4.0 -c:a aac output.mp4

Filter Essentials

Video filters (-vf), audio filters (-af), complex filtergraphs, and filter_complex quick reference.

Video Filters (-vf)

# Chain multiple filters
ffmpeg -i input.mp4 -vf "scale=1280:720,fps=30,format=yuv420p" output.mp4

# Crop
ffmpeg -i input.mp4 -vf "crop=640:480:100:50" output.mp4

# Rotate
ffmpeg -i input.mp4 -vf "transpose=1" output.mp4  # 90 deg clockwise
ffmpeg -i input.mp4 -vf "transpose=2" output.mp4  # 90 deg counter-clockwise
ffmpeg -i input.mp4 -vf "hflip" output.mp4        # Horizontal flip
ffmpeg -i input.mp4 -vf "vflip" output.mp4        # Vertical flip

# Overlay/Watermark
ffmpeg -i video.mp4 -i logo.png -filter_complex "overlay=10:10" output.mp4
ffmpeg -i video.mp4 -i logo.png -filter_complex "overlay=W-w-10:H-h-10" output.mp4

# Text overlay
ffmpeg -i input.mp4 -vf "drawtext=text='Hello World':x=10:y=10:fontsize=24:fontcolor=white" output.mp4

# Fade in/out
ffmpeg -i input.mp4 -vf "fade=t=in:st=0:d=1,fade=t=out:st=9:d=1" output.mp4

# Speed up/slow down
ffmpeg -i input.mp4 -vf "setpts=0.5*PTS" -af "atempo=2.0" output.mp4  # 2x speed
ffmpeg -i input.mp4 -vf "setpts=2.0*PTS" -af "atempo=0.5" output.mp4  # 0.5x speed

# Deinterlace
ffmpeg -i input.mp4 -vf "yadif" output.mp4

# Denoise
ffmpeg -i input.mp4 -vf "nlmeans" output.mp4
ffmpeg -i input.mp4 -vf "hqdn3d" output.mp4

# Sharpen
ffmpeg -i input.mp4 -vf "unsharp=5:5:1.0:5:5:0.0" output.mp4

# Color correction
ffmpeg -i input.mp4 -vf "eq=brightness=0.1:contrast=1.2:saturation=1.3" output.mp4

Audio Filters (-af)

# Multiple audio filters
ffmpeg -i input.mp4 -af "volume=1.5,highpass=f=200,lowpass=f=3000" output.mp4

# Noise reduction
ffmpeg -i input.mp4 -af "afftdn=nf=-25" output.mp4

# Compressor/limiter
ffmpeg -i input.mp4 -af "acompressor=threshold=-20dB:ratio=4:attack=5:release=50" output.mp4

# Equalizer
ffmpeg -i input.mp4 -af "equalizer=f=1000:width_type=h:width=200:g=-10" output.mp4

# Fade audio
ffmpeg -i input.mp4 -af "afade=t=in:ss=0:d=3,afade=t=out:st=57:d=3" output.mp4

# Channel remix (stereo to mono)
ffmpeg -i input.mp4 -af "pan=mono|c0=0.5*c0+0.5*c1" output.mp4

Complex Filtergraphs

# Picture-in-picture
ffmpeg -i main.mp4 -i pip.mp4 -filter_complex "[1:v]scale=320:240[pip];[0:v][pip]overlay=W-w-10:H-h-10" output.mp4

# Side by side
ffmpeg -i left.mp4 -i right.mp4 -filter_complex "[0:v]pad=iw*2:ih[bg];[bg][1:v]overlay=W/2:0" output.mp4

# Grid layout (2x2)
ffmpeg -i 1.mp4 -i 2.mp4 -i 3.mp4 -i 4.mp4 -filter_complex \
  "[0:v]scale=640:360[v0];[1:v]scale=640:360[v1];[2:v]scale=640:360[v2];[3:v]scale=640:360[v3];\
   [v0][v1]hstack[top];[v2][v3]hstack[bottom];[top][bottom]vstack" output.mp4

filter_complex Quick Reference

Syntax Components

Common Patterns

# Video transition (xfade)
ffmpeg -i a.mp4 -i b.mp4 -filter_complex \
  "[0:v][1:v]xfade=transition=fade:duration=1:offset=4[v]" \
  -map "[v]" output.mp4

# Audio mixing
ffmpeg -i video.mp4 -i music.mp3 -filter_complex \
  "[0:a]volume=1.0[v0];[1:a]volume=0.3[v1];[v0][v1]amix=inputs=2[a]" \
  -map 0:v -map "[a]" output.mp4

# Multi-output (ABR ladder)
ffmpeg -i input.mp4 -filter_complex \
  "[0:v]split=2[v1][v2];[v1]scale=1920:1080[hd];[v2]scale=1280:720[sd]" \
  -map "[hd]" hd.mp4 -map "[sd]" sd.mp4

For comprehensive filter_complex patterns, see the ffmpeg-filter-complex-patterns skill for PiP, grids, transitions, audio mixing, and GPU-accelerated filtergraphs.

Frame Manipulation

Frame rate conversion, timestamp manipulation, frame selection, thumbnails, tiles, freezing, looping, and reversing.

Frame Rate Conversion

# Change frame rate (drop/duplicate frames)
ffmpeg -i input.mp4 -vf "fps=30" output.mp4
ffmpeg -i input.mp4 -vf "fps=24000/1001" output.mp4  # 23.976 fps

# Motion-interpolated frame rate conversion (smoother)
ffmpeg -i input.mp4 -vf "framerate=fps=60" output.mp4

# High-quality motion interpolation (minterpolate)
ffmpeg -i input.mp4 -vf "minterpolate=fps=60:mi_mode=mci:mc_mode=aobmc:me_mode=bidir:vsbmc=1" output.mp4

# Simple motion interpolation (faster)
ffmpeg -i input.mp4 -vf "minterpolate=fps=60:mi_mode=blend" output.mp4

minterpolate Modes

Timestamp Manipulation (setpts)

# Speed up 2x (halve timestamps)
ffmpeg -i input.mp4 -vf "setpts=0.5*PTS" output.mp4

# Slow down 2x (double timestamps)
ffmpeg -i input.mp4 -vf "setpts=2.0*PTS" output.mp4

# Start from timestamp 0 (reset PTS)
ffmpeg -i input.mp4 -vf "setpts=PTS-STARTPTS" output.mp4

# Time-lapse: keep every 10th frame, at 30fps output
ffmpeg -i input.mp4 -vf "select='not(mod(n,10))',setpts=N/30/TB" output.mp4

# Variable speed (accelerate over time)
ffmpeg -i input.mp4 -vf "setpts='PTS/(1+0.001*N)'" output.mp4

setpts Variables

Frame Selection (select)

# Select every 5th frame
ffmpeg -i input.mp4 -vf "select='not(mod(n,5))',setpts=N/FRAME_RATE/TB" output.mp4

# Select I-frames only (keyframes)
ffmpeg -i input.mp4 -vf "select='eq(pict_type,I)',setpts=N/FRAME_RATE/TB" keyframes.mp4

# Select frames with scene change
ffmpeg -i input.mp4 -vf "select='gt(scene,0.4)',setpts=N/FRAME_RATE/TB" scenes.mp4

# Select frames based on time range
ffmpeg -i input.mp4 -vf "select='between(t,5,10)',setpts=N/FRAME_RATE/TB" output.mp4

# Select frames by expression (dark frames)
ffmpeg -i input.mp4 -vf "select='lt(avg(scene),0.1)'" output.mp4

select Expressions

Thumbnail Generation

# Generate thumbnail from best frame
ffmpeg -i input.mp4 -vf "thumbnail" -frames:v 1 thumbnail.jpg

# Thumbnail with custom frames to analyze
ffmpeg -i input.mp4 -vf "thumbnail=n=100" -frames:v 1 thumbnail.jpg

# Generate multiple thumbnails (every 10 seconds)
ffmpeg -i input.mp4 -vf "fps=1/10,thumbnail=n=1" thumb_%04d.jpg

# Thumbnail at specific time
ffmpeg -ss 00:00:30 -i input.mp4 -vframes 1 thumbnail.jpg

Tile / Contact Sheet

# Create 4x4 tile mosaic
ffmpeg -i input.mp4 -vf "fps=1/5,scale=320:180,tile=4x4" -frames:v 1 contact_sheet.jpg

# Create tile with padding
ffmpeg -i input.mp4 -vf "fps=1/10,scale=256:144,tile=5x4:margin=5:padding=5" contact.png

# Video tile mosaic (animated)
ffmpeg -i input.mp4 -vf "scale=320:180,tile=2x2" tiled.mp4

tile Parameters

Frame Freezing

# Freeze first frame for 3 seconds, then play
ffmpeg -i input.mp4 -vf "tpad=start_mode=clone:start_duration=3" output.mp4

# Freeze last frame for 3 seconds
ffmpeg -i input.mp4 -vf "tpad=stop_mode=clone:stop_duration=3" output.mp4

# Add black frames at start
ffmpeg -i input.mp4 -vf "tpad=start=90:color=black" output.mp4

# Freeze specific frame range (frames 100-150)
ffmpeg -i input.mp4 -vf "freezeframes=first=100:last=150:replace=100" output.mp4

Loop and Reverse

# Loop video 3 times
ffmpeg -stream_loop 3 -i input.mp4 -c copy output.mp4

# Loop filter (within filter graph)
ffmpeg -i input.mp4 -vf "loop=loop=3:size=30:start=0" output.mp4

# Reverse video
ffmpeg -i input.mp4 -vf "reverse" reversed.mp4

# Reverse audio too
ffmpeg -i input.mp4 -vf "reverse" -af "areverse" reversed.mp4

# Boomerang effect (forward then reverse)
ffmpeg -i input.mp4 -filter_complex "[0:v]split[v1][v2];[v2]reverse[r];[v1][r]concat=n=2:v=1:a=0" boomerang.mp4

Frame Extraction Patterns

# Extract all frames
ffmpeg -i input.mp4 frames/frame_%05d.png

# Extract at specific FPS
ffmpeg -i input.mp4 -vf "fps=1" frames/frame_%04d.jpg

# Extract keyframes only
ffmpeg -i input.mp4 -vf "select='eq(pict_type,I)'" -vsync vfr keyframe_%04d.png

# Extract frames with timestamp filename
ffmpeg -i input.mp4 -vf "fps=1" -frame_pts 1 "frame_%d.jpg"

FFmpeg Version History

Checking Your Version

# Check installed version
ffmpeg -version

# Check build configuration
ffmpeg -buildconf

Official Sources for Latest Version

• Official Download Page: https://ffmpeg.org/download.html
• Official Git Repository: https://git.ffmpeg.org/gitweb/ffmpeg.git
• GitHub Releases: https://github.com/FFmpeg/FFmpeg/releases

FFmpeg 8.0.1 (Released 2025-11-20) - Current Latest Stable

The latest stable release from the 8.0 "Huffman" branch (originally cut from master on 2025-08-09). This patch release contains important bug fixes and security updates.

FFmpeg 8.0 is one of the largest releases to date, named after the Huffman code algorithm invented in 1952.

AI and Transcription

• Whisper AI Filter: Built-in speech recognition via whisper.cpp for live subtitle generation and transcription
• Supports 99 languages with automatic language detection
• Outputs to SRT, JSON, or plain text formats
• Voice Activity Detection (VAD) support with Silero VAD

Vulkan Compute Codecs (Cross-Platform GPU)

• FFv1 Vulkan: Encode and decode via Vulkan 1.3 compute shaders
• ProRes RAW Vulkan: Hardware-accelerated decode
• AV1 Vulkan Encoder: GPU-accelerated AV1 encoding
• VP9 Vulkan Decoder: Hardware-accelerated VP9 decode
• Works on any Vulkan 1.3 implementation (AMD, Intel, NVIDIA)

New Codecs

• APV Codec: Samsung Advanced Professional Video encoder (via libopenapv) and native decoder
• ProRes RAW Decoder: Native Apple ProRes RAW decode
• RealVideo 6.0 Decoder: Native RV6 decode support
• G.728 Decoder: Low-delay CELP audio codec
• Sanyo LD-ADPCM Decoder: Sanyo audio format
• ADPCM IMA Xbox Decoder: Xbox audio format
• libx265 Alpha Layer Encoding: HEVC with alpha channel

Hardware Acceleration

• VVC VA-API Decoding: H.266/VVC hardware decode on Intel/AMD Linux
• VVC QSV Decoding: Intel Quick Sync VVC hardware decode
• OpenHarmony Support: H.264/H.265 encode/decode for HarmonyOS platform

New Features

• WHIP Muxer: Sub-second latency WebRTC ingestion
• Animated JPEG XL Encoding: Via libjxl library
• FLV v2 Support: Multitrack audio/video and modern codecs
• VVC in Matroska: H.266/VVC support in MKV container
• VVC SCC Support: Screen Content Coding with IBC, Palette Mode, ACT
• TLS Verification: Enabled by default for HTTPS connections

New Filters

• whisper: AI speech recognition filter
• colordetect: Detect JPEG/MPEG range and alpha channel properties
• pad_cuda: GPU-accelerated padding filter
• scale_d3d11: Direct3D 11 hardware scaling filter

Breaking Changes

• Dropped: OpenSSL 1.1.0 support (requires 1.1.1+)
• Dropped: yasm assembler support (use nasm instead)
• Deprecated: OpenMAX encoders
• Changed: Default PNG prediction method set to PAETH
• Changed: GCC autovectorization no longer disabled on x86/ARM/AArch64

Infrastructure

• New Forgejo-based code hosting at code.ffmpeg.org
• Modernized mailing list infrastructure

FFmpeg 7.1 "Péter" LTS (September 2024)

• VVC Decoder Stable: Full native H.266/VVC decoder (production-ready)
• MV-HEVC Decoder: 3D HEVC for Apple Vision Pro / iPhone spatial video
• LC-EVC Decoder: MPEG-5 Part 2 enhancement layer
• xHE-AAC Decoder: Extended High Efficiency AAC
• Vulkan H.264/H.265 Encoders: GPU encoding via Vulkan Video
• Native Intel QSV VVC Decoder: Hardware-accelerated VVC
• AVX2 VVC Optimizations: Faster software VVC decoding
• Full-Range Color Fixes: Correct color range handling throughout pipeline

Staying Up-to-Date

Why Version Matters

• Security fixes: Patch releases address vulnerabilities
• Bug fixes: Issues with specific codecs or filters get resolved
• New features: Major versions add codec support and filters
• Performance: Optimizations improve encoding speed

Recommended Update Strategy

1. Production: Use LTS releases (7.1) for stability, apply patch updates 2. Development: Use latest stable (8.0.1) for new features 3. Always: Check release notes before upgrading

Official Resources

• Release Notes: https://ffmpeg.org/download.html
• Security Advisories: https://ffmpeg.org/security.html
• Mailing Lists: https://ffmpeg.org/contact.html

Vulkan Compute Codecs in FFmpeg 8.0

FFmpeg 8.0 introduced Vulkan compute-based video codecs, enabling cross-platform GPU acceleration without vendor-specific APIs.

What's New in FFmpeg 8.0

• FFv1 Vulkan: Encode and decode via Vulkan 1.3 compute shaders
• ProRes RAW Vulkan: Hardware-accelerated decode
• AV1 Vulkan Encoder: GPU-accelerated AV1 encoding
• VP9 Vulkan Decoder: Hardware-accelerated VP9 decode

Requirements

• Vulkan 1.3 compatible GPU and drivers
• FFmpeg 8.0+ built with Vulkan support

Check Vulkan Support

# Check for Vulkan hardware acceleration
ffmpeg -hwaccels | grep vulkan

# List Vulkan devices
ffmpeg -init_hw_device vulkan -v verbose 2>&1 | grep -i vulkan

# Check Vulkan encoders
ffmpeg -encoders | grep vulkan

# Check Vulkan decoders
ffmpeg -decoders | grep vulkan

FFv1 Vulkan Codec

FFv1 is a lossless video codec, now with Vulkan acceleration.

Encoding

# FFv1 Vulkan encoding
ffmpeg -i input.mp4 \
  -init_hw_device vulkan=vk -filter_hw_device vk \
  -c:v ffv1_vulkan \
  output.mkv

# With specific GPU selection
ffmpeg -i input.mp4 \
  -init_hw_device vulkan=vk:0 -filter_hw_device vk \
  -c:v ffv1_vulkan \
  output.mkv

Decoding

# FFv1 Vulkan decoding
ffmpeg -hwaccel vulkan \
  -i input_ffv1.mkv \
  -c:v libx264 -crf 18 \
  output.mp4

AV1 Vulkan Encoder

# AV1 Vulkan encoding
ffmpeg -i input.mp4 \
  -init_hw_device vulkan=vk -filter_hw_device vk \
  -c:v av1_vulkan \
  output.webm

# With quality settings
ffmpeg -i input.mp4 \
  -init_hw_device vulkan=vk -filter_hw_device vk \
  -c:v av1_vulkan -b:v 5M \
  output.webm

VP9 Vulkan Decoder

# VP9 Vulkan decoding to H.264
ffmpeg -hwaccel vulkan \
  -i input.webm \
  -c:v libx264 -crf 20 \
  output.mp4

# Keep in GPU for further processing
ffmpeg -hwaccel vulkan -hwaccel_output_format vulkan \
  -i input.webm \
  -vf "scale_vulkan=w=1920:h=1080" \
  -c:v av1_vulkan \
  output.mp4

ProRes RAW Vulkan Decoder

# ProRes RAW Vulkan decoding
ffmpeg -hwaccel vulkan \
  -i input.braw \
  -c:v libx264 -crf 18 \
  output.mp4

Hardware Device Initialization

Auto-select GPU

ffmpeg -init_hw_device vulkan=vk -filter_hw_device vk \
  -i input.mp4 ...

Select Specific GPU

# First GPU (index 0)
ffmpeg -init_hw_device vulkan=vk:0 -filter_hw_device vk \
  -i input.mp4 ...

# Second GPU (index 1)
ffmpeg -init_hw_device vulkan=vk:1 -filter_hw_device vk \
  -i input.mp4 ...

List Available GPUs

# Show Vulkan device info
ffmpeg -init_hw_device vulkan=vk -v verbose 2>&1 | grep -E "device|GPU"

Vulkan Filters

FFmpeg includes Vulkan-based video filters:

# Vulkan scale filter
ffmpeg -hwaccel vulkan -hwaccel_output_format vulkan \
  -i input.mp4 \
  -vf "scale_vulkan=w=1920:h=1080" \
  -c:v av1_vulkan \
  output.mp4

# Vulkan color conversion
ffmpeg -hwaccel vulkan -hwaccel_output_format vulkan \
  -i input.mp4 \
  -vf "format_vulkan=pix_fmts=p010" \
  -c:v av1_vulkan \
  output.mp4

Cross-Platform Compatibility

Vulkan works on:

Performance Comparison

Best Practices

1. Use Vulkan when:

• You need cross-platform GPU acceleration
• Vendor-specific APIs (NVENC, QSV) aren't available
• Working with FFv1 lossless compression

2. Prefer vendor APIs when:

• Maximum performance is critical
• You're on a single platform
• Using established codecs (H.264, H.265)

3. Memory considerations:

   # Keep processing on GPU to avoid transfers
   ffmpeg -hwaccel vulkan -hwaccel_output_format vulkan \
     -i input.mp4 \
     -vf "scale_vulkan=1920:1080" \
     -c:v av1_vulkan \
     output.mp4

Troubleshooting

"Vulkan not available"

• Install/update GPU drivers
• Ensure Vulkan 1.3 support
• Check FFmpeg was built with --enable-vulkan

Poor performance

• Ensure hwaccel_output_format vulkan to keep data on GPU
• Use Vulkan filters instead of CPU filters
• Try different GPU if multiple available

Encoding errors

• Check input pixel format compatibility
• Try adding explicit format conversion
• Verify GPU memory availability

Example Pipelines

Lossless Archival with FFv1

# Fast lossless encoding for archival
ffmpeg -i source.mp4 \
  -init_hw_device vulkan=vk -filter_hw_device vk \
  -c:v ffv1_vulkan \
  -c:a flac \
  archive.mkv

Web Delivery with AV1

# AV1 encoding for web
ffmpeg -i source.mp4 \
  -init_hw_device vulkan=vk -filter_hw_device vk \
  -c:v av1_vulkan -b:v 3M \
  -c:a libopus -b:a 128k \
  output.webm

Transcode VP9 to AV1

# VP9 decode + AV1 encode (all on GPU)
ffmpeg -hwaccel vulkan -hwaccel_output_format vulkan \
  -i input.webm \
  -c:v av1_vulkan -b:v 3M \
  -c:a copy \
  output.mp4

VVC/H.266 Encoding with FFmpeg

Versatile Video Coding (VVC/H.266) is the latest video coding standard, offering 25-50% better compression than HEVC at similar quality.

FFmpeg VVC Support Timeline

• FFmpeg 7.0: Native VVC decoder added (experimental)
• FFmpeg 7.1 LTS: Production-ready VVC decoder, QSV hardware decoding
• FFmpeg 8.0: VA-API hardware decoding, stable native decoder

Encoding with libvvenc

# Basic VVC encoding
ffmpeg -i input.mp4 \
  -c:v libvvenc -qp 32 \
  -c:a aac -b:a 128k \
  output.mp4

# High quality encoding
ffmpeg -i input.mp4 \
  -c:v libvvenc -qp 24 \
  -vvenc-params "preset=medium" \
  -c:a aac -b:a 192k \
  output_hq.mp4

# 4K encoding with tiling (parallel processing)
ffmpeg -i input_4k.mp4 \
  -c:v libvvenc -qp 28 \
  -vvenc-params "preset=fast:tiles=4x4:threads=16" \
  -c:a copy \
  output_4k_vvc.mp4

# Specify exact bitrate
ffmpeg -i input.mp4 \
  -c:v libvvenc -b:v 5M \
  -vvenc-params "preset=medium" \
  output_cbr.mp4

Quality Parameters

Presets Comparison

Hardware Decoding

VA-API (FFmpeg 8.0+, Intel/AMD on Linux)

# VVC hardware decoding with VA-API
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
  -i input_vvc.mp4 \
  -c:v libx264 -crf 20 \
  output.mp4

# Hardware decode and scale
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi \
  -i input_vvc.mp4 \
  -vf "scale_vaapi=w=1920:h=1080" \
  -c:v h264_vaapi \
  output.mp4

Intel QSV (FFmpeg 7.1+)

# VVC hardware decoding with QSV
ffmpeg -hwaccel qsv \
  -i input_vvc.mp4 \
  -c:v libx264 -crf 20 \
  output.mp4

# Full QSV pipeline
ffmpeg -hwaccel qsv \
  -i input_vvc.mp4 \
  -c:v h264_qsv -preset fast \
  output.mp4

Container Support

• MP4/MOV: Full support
• MKV: Full support (FFmpeg 8.0+)
• TS: Experimental

# VVC in MKV container (FFmpeg 8.0+)
ffmpeg -i input.mp4 \
  -c:v libvvenc -qp 28 \
  -c:a copy \
  output.mkv

Screen Content Coding (SCC)

FFmpeg 8.0 adds VVC Screen Content Coding support:

# Screen content (presentations, desktop capture)
ffmpeg -i screen_capture.mp4 \
  -c:v libvvenc -qp 26 \
  -vvenc-params "preset=medium:scc=1" \
  output_screen.mp4

SCC features:

• Intra Block Copy (IBC)
• Palette Mode
• Adaptive Color Transform (ACT)

Comparison with HEVC

Best Practices

1. Use VVC for:

• Archival where storage matters
• 4K/8K content where bandwidth is limited
• Future-proofing content libraries

2. Avoid VVC for:

• Live streaming (too slow)
• Web delivery (limited browser support)
• Quick turnaround projects

3. Encode workflow:

   # Test settings on sample first
   ffmpeg -i input.mp4 -t 30 \
     -c:v libvvenc -qp 28 -vvenc-params "preset=medium" \
     test_output.mp4

   # Verify quality before full encode
   ffplay test_output.mp4

Decoding VVC Files

# Software decode (always available in FFmpeg 7.1+)
ffmpeg -i input.vvc -c:v libx264 -crf 18 output.mp4

# Play VVC file directly
ffplay input.vvc

# Get VVC file info
ffprobe -v error -show_streams input.vvc

Whisper AI Transcription with FFmpeg 8.0

FFmpeg 8.0 introduced the whisper audio filter for automatic speech recognition, powered by whisper.cpp.

Requirements

• FFmpeg 8.0 or later built with whisper.cpp support
• GGML model file (download from Hugging Face)

Verify Support

# Check FFmpeg version
ffmpeg -version  # Should show 8.0+

# Check for whisper filter
ffmpeg -filters | grep whisper

Download Models

Models are in GGML format from whisper.cpp:

# Base model (~150MB, good balance)
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.bin

# Medium model (~500MB, better accuracy)
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.bin

# Large model (~1.5GB, best accuracy)
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large.bin

# Quantized models (smaller, faster)
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base-q5_0.bin
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium-q5_0.bin

Model Comparison

Basic Usage

Generate SRT Subtitles

# From video file
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=auto:format=srt" \
  -f srt output.srt

# Specify language for better accuracy
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=en:format=srt" \
  -f srt output.srt

Generate VTT Subtitles

ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=auto:format=vtt" \
  -f webvtt output.vtt

Generate JSON Output

ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=auto:format=json" \
  -f null - 2>&1 | grep -o '{.*}'

Filter Parameters

Advanced Examples

Translate to English

# Translate any language to English
ffmpeg -i foreign_video.mp4 -vn \
  -af "whisper=model=ggml-medium.bin:language=auto:translate=1:format=srt" \
  -f srt english_subs.srt

With Voice Activity Detection (VAD)

# Download Silero VAD model
wget https://github.com/snakers4/silero-vad/raw/master/files/for-tests-silero-v5.1.2-ggml.bin

# Use VAD for better segmentation
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-medium.bin:language=en:queue=20:vad_model=for-tests-silero-v5.1.2-ggml.bin:format=srt" \
  -f srt output.srt

Live Transcription from Microphone

# Linux (PulseAudio)
ffmpeg -loglevel warning -f pulse -i default \
  -af "highpass=f=200,lowpass=f=3000,whisper=model=ggml-base.bin:language=en:format=text" \
  -f null -

# macOS (AVFoundation)
ffmpeg -loglevel warning -f avfoundation -i ":0" \
  -af "whisper=model=ggml-base.bin:language=en:format=text" \
  -f null -

Burn Subtitles into Video

# Two-pass approach (recommended)
# 1. Generate SRT
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=auto:format=srt" \
  -f srt subs.srt

# 2. Burn into video
ffmpeg -i input.mp4 -vf "subtitles=subs.srt" \
  -c:v libx264 -crf 20 -c:a copy \
  output_with_subs.mp4

Display Live Subtitles (Frame Metadata)

# Whisper writes text to frame metadata
ffmpeg -i input.mp4 \
  -af "whisper=model=ggml-base.en.bin:language=en" \
  -vf "drawtext=text='%{metadata\\:lavfi.whisper.text}':fontsize=24:fontcolor=white:x=10:y=h-th-10:box=1:boxcolor=black@0.5" \
  -c:v libx264 -crf 20 -c:a aac \
  output_with_live_subs.mp4

Performance Optimization

For Long Videos

# Extract audio first (faster processing)
ffmpeg -i input.mp4 -vn -c:a pcm_s16le -ar 16000 -ac 1 audio.wav

# Process audio separately
ffmpeg -i audio.wav \
  -af "whisper=model=ggml-base.bin:language=en:format=srt" \
  -f srt output.srt

# Clean up
rm audio.wav

GPU Acceleration

# Enable GPU (if supported)
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=en:use_gpu=1:format=srt" \
  -f srt output.srt

# Disable GPU (force CPU)
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-base.bin:language=en:use_gpu=0:format=srt" \
  -f srt output.srt

Use Quantized Models

# Quantized models are smaller and faster
ffmpeg -i input.mp4 -vn \
  -af "whisper=model=ggml-medium-q5_0.bin:language=en:format=srt" \
  -f srt output.srt

Audio Preprocessing

For better accuracy, clean up audio before transcription:

# Noise reduction and normalization
ffmpeg -i input.mp4 -vn \
  -af "afftdn=nf=-25,loudnorm=I=-16:TP=-1.5:LRA=11,whisper=model=ggml-base.bin:language=en:format=srt" \
  -f srt output.srt

# High-pass and low-pass for speech
ffmpeg -i input.mp4 -vn \
  -af "highpass=f=200,lowpass=f=3000,whisper=model=ggml-base.bin:language=en:format=srt" \
  -f srt output.srt

Supported Languages

Whisper supports 99 languages. Common codes:

• en: English
• es: Spanish
• fr: French
• de: German
• zh: Chinese
• ja: Japanese
• ko: Korean
• pt: Portuguese
• ru: Russian
• ar: Arabic
• auto: Automatic detection

Troubleshooting

"Filter whisper not found"

FFmpeg not built with whisper.cpp support. Use a build that includes it or compile from source.

Poor accuracy

• Try a larger model (medium or large)
• Specify the language explicitly
• Apply audio preprocessing
• Use VAD for better segmentation

Slow processing

• Use a smaller or quantized model
• Enable GPU acceleration
• Extract audio first for long videos

Memory errors

• Use a smaller model
• Process in segments
• Disable GPU if VRAM is limited