ffmpeg-opencv-integration

FFmpeg + OpenCV Integration Guide

Use this skill when FFmpeg handles video I/O and OpenCV handles image processing. This SKILL is a lean orchestrator; full pipe patterns, library examples, and Modal recipes are preserved in references/opencv-pipelines-and-libraries.md.

When to Use

• Decode with FFmpeg and process frames with OpenCV
• Encode OpenCV-generated/processed frames with FFmpeg
• Compare cv2.VideoCapture, subprocess pipes, PyAV, ffmpegcv, VidGear, and Decord
• Fix RGB/BGR color bugs or (height, width) dimension-order bugs
• Preserve audio while replacing processed video frames
• Build GPU-assisted video I/O around CPU OpenCV processing

Library Selection

Need	Best option	Why
Simple local file	cv2.VideoCapture	Built-in and simple
Full FFmpeg format/protocol support	subprocess pipe	Exact CLI behavior
GPU video I/O	ffmpegcv	NVDEC/NVENC with OpenCV-like API
Network/RTSP streaming	VidGear	Threaded capture and stream helpers
ML batch loading	Decord	Fast random/batch frame access
Frame-level libav control	PyAV	Direct FFmpeg library access

Critical Gotchas

1. OpenCV is BGR. FFmpeg/PyAV/PIL/Decord often produce RGB. Convert explicitly.
2. NumPy dimensions are (height, width, channels). Pixel access is img[y, x], not img[x, y].
3. Video filters can drop audio. Preserve or remux original audio intentionally.
4. Release resources. Always close VideoCapture, pipes, writers, and PyAV containers.
5. Rawvideo pipes require exact frame size. width * height * channels must match -pix_fmt.

Minimal Pipe Patterns

FFmpeg to OpenCV using BGR frames:

cmd = [
    "ffmpeg", "-i", input_path,
    "-f", "rawvideo", "-pix_fmt", "bgr24", "-"
]

OpenCV to FFmpeg using BGR frames:

cmd = [
    "ffmpeg", "-y",
    "-f", "rawvideo", "-vcodec", "rawvideo",
    "-s", f"{width}x{height}", "-pix_fmt", "bgr24",
    "-r", str(fps), "-i", "-",
    "-c:v", "libx264", "-preset", "fast", "-crf", "23",
    "-pix_fmt", "yuv420p", output_path
]

Core Workflow

1. Probe input dimensions, fps, duration, and audio streams.
2. Pick the I/O library based on the selection table.
3. Lock pixel format at boundaries (bgr24 for OpenCV pipes; yuv420p for final H.264 output).
4. Process frames in generators/batches; avoid loading full videos unless they are small.
5. Preserve or re-encode audio explicitly.
6. Verify output duration, fps, resolution, codec, and A/V sync.

Reference Map

• references/opencv-pipelines-and-libraries.md - Full preserved reference: color/dimension gotchas, cleanup patterns, FFmpeg-to-OpenCV and OpenCV-to-FFmpeg pipes, bidirectional pipeline, ffmpegcv, VidGear, Decord, PyAV, Modal.com examples, GPU pipeline, cheat sheets, sources.

Related Skills

• ffmpeg-python-integration-reference - Type-safe parameters, colors, time units
• ffmpeg-pyav-integration - PyAV API details
• ffmpeg-hardware-acceleration - GPU decode/encode and filter pipelines