SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsvideo-debugger

video-debugger

Testing & quality

View repository

Updated 10-16 06:14

About this skill

Video Debugger

Extract frames from screen recordings so Claude can visually analyze animations, transitions, and web behaviors frame-by-frame.

Phase 0: Check dependencies

which ffmpeg && ffmpeg -version | head -1
which ffprobe
which python3

If any dependency is missing:

ffmpeg + ffprobe:
  macOS:   brew install ffmpeg
  Linux:   sudo apt install ffmpeg
  Windows: winget install ffmpeg  (o choco install ffmpeg)

python3: macOS: included, or brew install python Linux: sudo apt install python3 Windows: winget install Python.Python.3


**No continuar hasta que las 3 dependencias esten disponibles.**

### Dependencias opcionales (para --vision)

mlx-vlm (macOS Apple Silicon — default, no app needed): pip3 install mlx-vlm

LM Studio (any OS — fallback): Download from https://lmstudio.ai, open at least once to register the CLI


### Modelos de vision (descargar uno)

Recommended — Qwen3-VL-30B-A3B (MLX, 17 GB, macOS Apple Silicon): From the LM Studio UI search for "Qwen3-VL-30B" or: huggingface-cli download mlx-community/Qwen3-VL-30B-A3B-Instruct-4bit

Alternative — Qwen3.5-35B-A3B (GGUF, 22 GB, any OS via LM Studio): From the LM Studio UI search for "Qwen3.5-35B-A3B"


**Nota**: La version MLX del Qwen3.5 NO funciona con mlx-vlm (processor incompatible con transformers v5). Usar la version GGUF via LM Studio.

## Fase 0.5: Detectar capacidad de vision local

```bash
bash scripts/analyze-frames.sh --check-vision

Possible results:

  • "available": true → local model available, use --vision in Phase 4.5
  • "available": false → no local model, use agent frame-analyzer (Sonnet) in Phase 4.5

Save the result to decide in Phase 4.5.

Phase 1: Receive video and get metadata

The user provides the path to the video. Run:

ffprobe -v quiet -print_format json -show_format -show_streams <video_path>

Extract and display summary:

Video: 12.3s | 1920x1080 | 60fps | 8.2 MB

Relevant data: duration, width, height, r_frame_rate, file size.

Phase 2: Configure extraction

Ask the user which mode they prefer:

Mode FPS Use
Fast (default) 4 fps General overview, slow transitions
Detailed 10 fps Precise frame-by-frame, fast animations
Changes only Auto (scene detect) Automatically detects where there are visual changes
Specific segment 10 fps Analyze only a time range (e.g., "from second 2 to 5")

Default format: JPEG q:v 2 (~150-300 KB/frame, legible text).

For format comparison and when to use each one: references/format-guide.md

Phase 3: Extract frames

Output path

Save to ./debug-frames/{YYYY-MM-DD_HH-MM-SS}/:

OUTPUT_DIR="./debug-frames/$(date +%Y-%m-%d_%H-%M-%S)"
mkdir -p "$OUTPUT_DIR"

Add to .gitignore

grep -qxF 'debug-frames/' .gitignore 2>/dev/null || echo 'debug-frames/' >> .gitignore

Commands by mode

See references/ffmpeg-commands.md for the exact commands for each mode.

Fast:

ffmpeg -i <video> -vf "fps=4" -q:v 2 "$OUTPUT_DIR/frame_%04d.jpg"

Detailed:

ffmpeg -i <video> -vf "fps=10" -q:v 2 "$OUTPUT_DIR/frame_%04d.jpg"

Changes only:

ffmpeg -i <video> -vf "select=gt(scene\,0.3)" -vsync vfr -q:v 2 "$OUTPUT_DIR/scene_%04d.jpg"

Specific segment:

ffmpeg -i <video> -ss <inicio> -to <fin> -vf "fps=10" -q:v 2 "$OUTPUT_DIR/frame_%04d.jpg"

Show the number of extracted frames and total size:

echo "Frames: $(ls "$OUTPUT_DIR"/*.jpg 2>/dev/null | wc -l)"
echo "Tamaño: $(du -sh "$OUTPUT_DIR" | cut -f1)"

Phase 4: Generate contact sheet

Always generate automatically after extracting frames.

Calculate grid

FRAME_COUNT=$(ls "$OUTPUT_DIR"/frame_*.jpg "$OUTPUT_DIR"/scene_*.jpg 2>/dev/null | wc -l)
Frames Columns Scale
≤ 12 4 400:-1
13-30 5 320:-1
31-60 6 256:-1
> 60 8 200:-1

Generate with ffmpeg tile

ffmpeg -i <video> -vf "fps=<fps_usado>,scale=<escala>,tile=<cols>x0" -frames:v 1 -q:v 3 "$OUTPUT_DIR/contact_sheet.jpg"

Example for 24 frames at 4fps:

ffmpeg -i <video> -vf "fps=4,scale=320:-1,tile=5x0" -frames:v 1 -q:v 3 "$OUTPUT_DIR/contact_sheet.jpg"

Phase 4.5: Frame Diff & Vision Analysis

After extracting frames, offer additional analysis with visual diff and local vision model.

Visual diff (always available)

Run the analysis script that generates diffs with tblend and detects changes with scdet:

# Desde la raiz del repo llm-toolkit (o donde este el script)
bash scripts/analyze-frames.sh <video_path> "$OUTPUT_DIR" --threshold 8

For GIFs (Gifox, etc): use --threshold 12 or --threshold 14 because Bayer dithering generates noise that confuses scene detect.

This produces:

  • diffs/ — temporal difference frames (black=no change, bright=movement)
  • diff-motion.mp4 — amplified motion video
  • diff-contact.jpg — contact sheet of the diffs
  • diff-report.txt — manifest with frame index, timestamp, score, and role (before/change/after)
  • scores.txt — raw scdet scores per frame

Read diff-report.txt to identify where the significant changes are before the visual analysis phase.

Vision analysis (optional)

If the user wants semantic analysis (motion description, glitch detection), ask if they want to use a local vision model:

# macOS Apple Silicon (default — usa mlx-vlm, no necesita LM Studio)
bash scripts/analyze-frames.sh <video_path> "$OUTPUT_DIR" --vision --threshold 8

# Cualquier OS (fallback — requiere LM Studio corriendo)
bash scripts/analyze-frames.sh <video_path> "$OUTPUT_DIR" --vision --backend lmstudio --threshold 8

mlx backend (default, macOS Apple Silicon):

  1. Run mlx-vlm directly with the MLX model (no server or app needed)
  2. Default: Qwen3-VL-30B-A3B (17 GB, MoE, ~23s for 14 frames)
  3. Auto-detects the model in ~/.lmstudio/models/

lmstudio backend (fallback, any OS):

  1. Starts LM Studio server if it is not running
  2. Loads the GGUF model (Qwen3.5-35B-A3B by default, auto-downloads in 2 min)
  3. Unloads the model when finished

Both backends:

  • Send selected frames to the model (all if ≤30, or windows ±3 around each change)
  • Generate vision-analysis.txt with motion description, glitches, transitions

Important: If a local model generated vision-analysis.txt, read that file as text (there is no point in re-analyzing the images). If there is no local model, prefer the frame-analyzer agent (Sonnet) — it does the same visual analysis but costs less than loading frames into Opus. If for some reason frames need to be loaded directly into context, that also works.

Additional options:

  • --backend mlx|lmstudio — vision backend (default: mlx)
  • --model "path/o/key" — MLX path or LM Studio key (auto-detected if not specified)
  • --context 5 — more context frames around each change (default: 3)
  • --max-frames 15 — threshold for short video mode vs windows (default: 30)
  • --keep-model — do not unload the model when finished (lmstudio only)
  • --timeout 600 — longer timeout for slow models (lmstudio only, default: 300s)

Without a local vision model (fallback with Sonnet agent)

If --check-vision returned available: false, run only the diff (without --vision):

bash scripts/analyze-frames.sh <video_path> "$OUTPUT_DIR" --threshold 8

Then launch the frame-analyzer agent with this prompt:

Analiza los frames en {OUTPUT_DIR}. Lee diff-report.txt para identificar frames de cambio.
Lee el contact sheet y los frames individuales de las zonas de cambio.
Diagnostica bugs visuales, layout shifts, y problemas de animacion.

The agent (Sonnet with multimodal vision) returns textual analysis equivalent to vision-analysis.txt.

For reference on diff commands: references/diff-tools.md

Phase 5: Analysis

  1. If there is diff-report.txt: read it first → gives the frames with significant changes and their scores
  2. If there is vision-analysis.txt: read it → gives a semantic description of the local vision model (do NOT load images, only the text)
  3. Read contact sheet → quick overview of the whole sequence
  4. Identify problematic zone → use diff-report + vision-analysis + contact sheet to triangulate
  5. Read individual frames of the problematic zone in sequence (the diff frames in diffs/ show which pixels changed)
  6. Diagnose with reference to specific frame numbers:
    • Incorrect timing (transition too fast/slow)
    • Easing problems (jumps, unwanted bounces)
    • Layout shifts (elements that move unexpectedly)
    • Z-index issues (badly stacked layers)
    • Opacity/visibility glitches (flickering)
    • Reflow/repaint artifacts
  7. Report findings with: frame number, approximate timestamp, problem description, probable cause

Troubleshooting

Problem Cause Solution
"No such file or directory" Incorrect path or path with spaces Use double quotes in the path
Black frames Video with unsupported codec ffmpeg -i <video> to see codec, install missing codec
Too many frames FPS too high for long video Use "Changes only" or "Specific range" mode
Blurry frames Source video compression It is not an ffmpeg problem, it is the quality of the recording
Contact sheet cut off Too many frames for tile Reduce fps or use a specific range
ffmpeg slow/hung Video too large (>1GB) Add -threads 4 to the command
Many false positives in scene detect GIF with Bayer dithering Raise --threshold to 12-14
--check-vision says unavailable There is no mlx-vlm or LM Studio A Sonnet agent is used automatically; optionally install mlx-vlm (macOS) or LM Studio (any OS)
Vision analysis fails (mlx) mlx-vlm not installed or model not found pip3 install mlx-vlm, verify model in ~/.lmstudio/models/
Vision analysis fails (lmstudio) LM Studio not running or model without vision Verify lms server status and that the model has vision capability
mlx-vlm not supported Not macOS Apple Silicon Use --backend lmstudio
Qwen3.5 MLX does not work with mlx-vlm Processor incompatible with transformers v5 Use GGUF version via --backend lmstudio
Slow vision analysis Large model + many frames Use --max-frames 15 or reduce frames with --context 1