video-vision

Video Vision

Give Claude the ability to watch and understand a local video file. The skill turns the video into two streams Claude can reason over directly:

1. Sampled frame images on disk — read on demand via the Read tool (Claude is multimodal).
2. A timestamped transcript from local Whisper — fully offline after the first model download.

Claude is the interpreter. The scripts are a deterministic perception layer.

When to use

Trigger whenever the user points at a video file and wants it understood: summarization, Q&A, captioning, finding a moment, explaining what's on screen, transcribing dialogue, or describing visuals. Also trigger when the user says "watch this" with a path, or drops a video path into the chat with any implied question.

Do not use for: streaming URLs (YouTube etc. — ask the user to download first with yt-dlp), live camera feeds, or pure audio files (use a transcription-only tool instead).

Requirements

• ffmpeg and ffprobe on PATH (brew install ffmpeg / apt install ffmpeg)
• Python 3.9+
• pip install -U openai-whisper (first run downloads ~150 MB for the base model to ~/.cache/whisper)

If openai-whisper is missing, scripts/transcribe_audio.py exits with code 2 and prints the install command. Surface that to the user, offer to install, then continue.

How to run

The orchestrator runs frame extraction and transcription in parallel and prints a JSON manifest to stdout:

python3 scripts/analyze_video.py --video /abs/path/to/file.mp4 \
    [--fps 1] [--max-width 1024] [--whisper-model base] [--no-transcribe]

Manifest shape:

{
  "video": "/abs/path/file.mp4",
  "duration_seconds": 612.4,
  "frames": {
    "out_dir": "/tmp/video_frames_XXXX",
    "fps": 1.0,
    "frame_count": 612,
    "frames": [{"path": "...frame_00001.jpg", "t": 0.5}, ...]
  },
  "transcript": {
    "model": "base",
    "language": "en",
    "text": "full concatenated transcript...",
    "segments": [{"start": 0.0, "end": 3.2, "text": "..."}, ...]
  }
}

For partial work, call the sub-scripts directly:

• scripts/extract_frames.py --video X --fps 2 --max-width 1024 — frames only.
• scripts/transcribe_audio.py --video X --model base — transcript only.

Choosing parameters

Do not blindly default. Read the user's intent first, then pick:

Intent	--fps	--max-width	Whisper model
Summarize a long lecture / meeting (>20 min)	0.25	768	base
General Q&A, short clip (<5 min)	1	1024	base
Detailed visual question, reading on-screen text	2	1280	base
Noisy audio, accents, multilingual	—	—	small or medium
Audio-only question ("what did they say about X?")	skip frames	—	base
Visual-only question ("what's on screen at 1:30?")	1–2	1024	--no-transcribe

Bumping Whisper to medium pulls ~1.5 GB; large-v3 is ~3 GB. Warn the user before escalating.

Reasoning over the manifest

1. For visual questions at a specific time — look up frames.frames[i] where t is closest to the target timestamp, then Read that single frame path. Do not Read all frames; pull only what you need.
2. For audio questions — search transcript.segments[].text. Quote with [MM:SS-MM:SS] timestamps.
3. For synthesis / summaries — skim the full transcript.text, then sample 5–10 frames spread across the duration (out_dir glob or evenly-spaced indices). Cite timestamps in the answer.
4. Cross-reference — when the user asks "what is shown while they talk about X?", find the segment, then pull the frame whose t falls in [segment.start, segment.end].

Always cite timestamps as [MM:SS] so the user can verify.

Edge cases

• No audio track → transcript.segments is empty. Proceed with frames only.
• Silent stretches → expected; transcript will have gaps.
• Very long video (>1 hr) → start at --fps 0.25 --whisper-model base. Re-sample a narrow window by pre-trimming: ffmpeg -ss 600 -to 720 -i in.mp4 -c copy clip.mp4.
• Frame timestamps are the midpoint of each sampling interval ((i + 0.5) / fps), not exact decode PTS. Good enough for grounding; don't claim frame-accurate precision.
• Frames dir is ephemeral — lives under $TMPDIR. The OS reaps it; delete manually if the user is privacy-sensitive.
• Whisper model cache — ~/.cache/whisper. Safe to delete to reclaim disk.

Self-annealing

If something breaks (ffmpeg missing codec, Whisper OOM on a large model, unsupported container), read the stderr, fix the root cause (install codec, drop to smaller model, remux the container), and note the learning. Do not silently swallow the failure.