SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscompare-with-reference-video

compare-with-reference-video

Design & creation

View repository

Updated 10-15 21:54

About this skill

compare-with-reference-video

Purpose

After a target master video is rendered, the operator's first instinct is "this feels worse than the reference but I can't tell why." This atom answers that question structurally — by extracting matched frames, computing comparable metrics, and forcing a checklist review across 7 dimensions the LLM eyeball would otherwise skip.

The mechanical work (frame extraction, metric computation, grid HTML) is automated. The qualitative analysis is LLM/human driven against an explicit rubric. Output is a prioritized P0/P1/P2 fix list that auto-fix-from-review-notes can consume.

This skill is general-purpose — the dimensions and rubric work for ANY video format (music-video ad, narrated comic-strip, UGC, talking-head). The reference video acts as the "north star" for whatever format the operator is shipping.

When to use

  • At State 8 of video-orchestrator/orchestrator when a reference video for the target format exists (most common: when a molecule documents a reference run and you have a master to compare against it).
  • After a re-roll pass to verify the fixes closed the gap.
  • Standalone, any time the operator says "this feels off vs [reference]".

Skip when:

  • No reference video exists (use review-video-for-ad-quality instead — generic quality review).
  • Reference and target are different lengths by > 2× (comparison breaks down at extreme length deltas).
  • Reference is a 30s ad and target is a 4s loop — they're different formats, not comparable.

Inputs

  • --reference <path> — the reference video (required). Local mp4/mov.
  • --target <path> — the target master to evaluate (required). Local mp4/mov.
  • --output-dir <path> — default <target_folder>/review-notes/vs-reference-<ts>/. Holds the grid, frames, metrics, and report.
  • --fps <float> — sampling rate for both videos (default 2). Higher fps = more frames = more LLM tokens at review time.
  • --max-frames <int> — cap per video (default 60). At fps=2 this covers up to 30s; bump for longer videos.
  • --align <auto|relative|absolute> — how to match frames across the two videos:
    • auto (default): if durations within 10% match, use absolute timestamps; else relative (target_time = ref_time × duration_ratio).
    • relative: always normalize to 0–1 timeline position.
    • absolute: never normalize — only useful when both videos are the same nominal length.

Workflow

Step 1 — Frame extraction (mechanical)

Use the watch skill's pipeline (scripts/watch.py) under the hood, OR drive ffmpeg directly:

ffmpeg -i <video> -vf "fps=<fps>,scale=512:-1" <out>/frames/frame_%03d.jpg

Run on both reference and target. Save to <output_dir>/ref-frames/ and <output_dir>/target-frames/.

Step 2 — Metric computation (mechanical)

For each video compute and persist to <output_dir>/metrics.json:

Metric How
duration_s ffprobe -show_entries format=duration
aspect_ratio ffprobe -show_entries stream=width,height
cuts_total ffmpeg scene-detect filter (select='gt(scene,0.4)')
cuts_per_minute cuts_total / duration_min
avg_cut_interval_s mean of cut-to-cut intervals
cut_interval_stdev_s stdev — high values = irregular pacing
motion_mag_mean per-frame avg pixel difference (frame N+1 vs frame N), normalized 0–1. Proxy for "how much is happening per frame"
motion_mag_p90 90th percentile — captures peak motion intensity
dominant_palette top 5 RGB hex colors via PIL quantize
avg_luminance mean Y channel value
caption_box_estimate OCR a few mid-frames to roughly detect caption presence + bounding box dims (optional, requires tesseract)

Compute these for BOTH videos. The diff between them surfaces the structural deltas the operator can't see frame-by-frame.

Step 3 — Aligned grid HTML (mechanical)

Build <output_dir>/grid.html. Layout:

  • Top: metrics side-by-side as a comparison table (target / reference / delta)
  • Body: a vertical scroll of paired frames (target left, reference right) at matched timeline positions
  • Each pair labeled with timestamp and a notes textarea
  • Footer: a checklist of the 7 review dimensions (below)

The grid is the artifact the user opens. It also stays in the project folder so the LLM can re-open it on follow-ups.

Step 4 — Qualitative review (LLM-driven)

For each dimension, the LLM reviewer:

  1. Reads sampled frames from BOTH videos (use Read tool on the frame paths in <output_dir>/{ref,target}-frames/)
  2. Compares against the rubric below
  3. Names 0–N findings with severity

Dimension 1 — Character framing

  • What fraction of the frame does the protagonist occupy across shots?
  • Is the character always facing camera or is the back-of-head common?
  • Establishing wides vs character-dominant: what ratio?
  • Rubric: if target has the character at <30% of frame in >40% of shots while reference has them at >40% — that's a P0 character-framing issue.

Dimension 2 — Caption treatment

  • Size: % of frame width occupied by typical caption
  • Background: pill / none / drop-shadow / outline
  • Weight contrast within a single chunk: regular only, or regular+bold mix?
  • Punctuation: leaked from transcript or stripped?
  • Position: mid / lower-third / top / variable?
  • Rubric: P0 if target caption width < 60% of reference caption width, or if punctuation noise leaks ("Super Bowl Sunday,").

Dimension 3 — Camera motion

  • Is each shot's motion deliberately authored or ambient-only?
  • Per-shot motion vector — push-in / parallax / static / tracking?
  • Motion magnitude (use metric motion_mag_mean)
  • Rubric: P1 if target motion_mag_mean is <50% of reference; P0 if <25%.

Dimension 4 — Cut rhythm

  • Cuts per minute, mean and stdev
  • Are cuts on a musical grid (bar boundaries) or arbitrary?
  • Section breaks marked visually (transitions / freeze frames / palette shift)?
  • Rubric: P1 if avg_cut_interval_s is >2× the reference, or if cut_interval_stdev_s is >2× (irregular pacing).

Dimension 5 — Color & lighting

  • Dominant palette overlap (Jaccard similarity of top-5 colors)
  • Saturation distribution
  • Cool/warm scene contrast variety
  • Rubric: P2 mostly — flag if palette divergence is the operator's intent (different brand). P1 if avg luminance is >25% off (target much darker/lighter).

Dimension 6 — Background complexity

  • Are background elements supporting the character or competing for attention?
  • "Dead-zone" shots (no character, no clear focal point)?
  • Rubric: P1 if target has 2+ character-less shots that don't earn their place (vs the reference which keeps almost every shot character-driven).

Dimension 7 — Energy & cohesion

  • Does the video sustain attention end-to-end?
  • Is there a clear arc (build → climax → resolution)?
  • Climax bar/shot — does it land harder than its neighbors visually?
  • Rubric: P0 if the climax frame is visually indistinguishable from non-climax frames (in the reference the climax is usually marked by color saturation OR a distinct motion event).

Step 5 — Write the review report

Output <output_dir>/report.md in the standard review-notes format that auto-fix-from-review-notes consumes:

---
status: NEEDS_REVISION
reviewer: compare-with-reference-video
reference: <path>
target: <path>
metrics_delta:
  cuts_per_minute: target=18.4 ref=17.8 delta=+0.6
  motion_mag_mean: target=0.06 ref=0.14 delta=-58%
  avg_caption_width_pct: target=20 ref=32 delta=-12pp
---

## TL;DR
<one paragraph: top 3 reasons reference feels better>

## P0
- [Dimension] <issue> · evidence: <frame_path> @ <timestamp>
  - auto-fix: <atom name> --<arg> "<value>"

## P1
- ...

## P2
- ...

## Proposed rebuild plan
| Pass | What | Time | Cost | Impact |
|---|---|---|---|---|
| A | ... | ... | ... | ... |

## Recommendation
<one sentence pointing at the cheapest pass with highest impact>

Step 6 — Embed in storyboard

If a storyboard.html exists in the target's project folder, append a "Review · target vs reference" section above the master video panel:

  • TL;DR paragraph
  • Two-column P0/P1 fix card
  • Rebuild-plan card with Pass A/B/C/D summary
  • Link to the full report and the grid HTML

This follows the single-review-surface rule from STORYBOARD.md — the operator should see the review at the top of the same page they reviewed the master from.

Output

  • <output_dir>/grid.html — paired-frame visual comparison + metrics table
  • <output_dir>/report.md — prioritized review notes (P0/P1/P2 + rebuild plan)
  • <output_dir>/metrics.json — raw mechanical metrics for both videos
  • <output_dir>/ref-frames/frame_*.jpg — sampled reference frames
  • <output_dir>/target-frames/frame_*.jpg — sampled target frames
  • Inline edit to <target_project>/storyboard.html — review summary section above the master panel

Quality Checks

  • Both videos have ≥ 20 sampled frames each (otherwise the dimension review is sparse).
  • metrics.json has all 10 metric fields populated for both videos.
  • report.md frontmatter status is set (APPROVED or NEEDS_REVISION).
  • Every P0/P1 finding cites at least one frame path + timestamp as evidence.
  • Every P0/P1 finding names an auto-fixable atom OR explicitly says "manual re-roll required".
  • The rebuild plan table covers ≥ 3 passes (otherwise the review wasn't deep enough).
  • Storyboard.html was updated if it exists.

Known metric quirks

  • motion_mag_mean includes the big pixel diffs that happen at cut boundaries. A video with many hard cuts will register high motion even if every individual shot is static. Cross-check the quantitative number against the qualitative review of within-shot motion. (Future: filter pixel-diffs within ±1 frame of detected cuts.)
  • cuts_per_minute relies on ffmpeg scene-detect threshold 0.4. Hard-cut concatenated clips with similar palettes (e.g. felt-craft scenes that all share warm tones) may score below threshold — lower the threshold to 0.25 in those cases (currently a code change, future: CLI flag).
  • avg_luminance is sampled at one-in-twenty frames for speed — outliers in dark/light transitions can skew it. For precise lighting comparison, compute on all frames.

Failure Modes

  • Reference is a placeholder, not a real reference run. If the reference video is a brand sizzle or unrelated format, the comparison is misleading. Skill should refuse to run or warn loudly if duration delta > 100% or aspect ratio differs.
  • Both videos are bad and the diff is moot. If target is broken (corrupt encoding, missing audio, wrong aspect), bail before comparison. The skill is for evaluating a finished master vs a known-good reference.
  • Operator over-weights one dimension. A target that wins on captions but loses on cut rhythm shouldn't get APPROVED just because the operator likes the captions. The rubric is multi-axis on purpose.
  • OCR for caption box estimation is brittle. When tesseract isn't installed, skip Dimension 2's automated measurement and rely on visual review only. Don't fail the whole skill.
  • Reference video is private / unviewable. If the reference URL 404s or local path is missing, return a clear error with the missing path. Don't fall back to a generic review.

Composition

  • Wraps the watch skill (claude-video plugin) for frame extraction. Reuses its ffmpeg + Whisper pipeline.
  • Feeds auto-fix-from-review-notes via the report.md output format.
  • Sits alongside review-video-pacing-rhythm (cut/beat alignment), review-video-hook-strength (first-3s test), review-video-character-consistency (per-shot drift), review-music-video-beat-sync (caption-onset + bar-grid). This atom is the umbrella comparison — the others are deep-dives on single axes.
  • Consumed by video-orchestrator/review-video Phase 7b if a reference_path is set in implementation-brief.md.

Example invocation

python3 skills/atoms/review/compare-with-reference-video/scripts/compare.py \
  --reference reference-videos/gum-of-gods-music-video.mp4 \
  --target   coinbase/video-01-music-video-debut/edits/master-final.mp4 \
  --output-dir coinbase/video-01-music-video-debut/review-notes/vs-reference-01/ \
  --fps 2 --max-frames 80

Then the LLM reads frames from both output dirs, fills in the qualitative review against the 7-dimension rubric, and writes the report.

Reference run

The canonical first execution of this skill was the Coinbase music-video v1 review: coinbase/video-01-music-video-debut/review-notes/01-vs-reference.md. That review found 3 P0s (character drift in bar 13, pencil artifact in scene 05, caption chunk bleed at 0:14), 3 P1s (character framing too small, caption presentation feels like subtitles, camera motion invisible), and proposed a 4-pass rebuild plan (A: captions only/free, B: re-roll 3 broken keyframes, C: re-roll establishing shots, D: re-animate weak motion). Pass A was recommended as the cheapest highest-impact starting move.

What NOT to do

  • Don't grade subjectively without naming a frame. Every finding cites evidence.
  • Don't fuse this with review-video-for-ad-quality — those are different reviews (generic quality vs reference-anchored).
  • Don't auto-trigger fixes from this skill. It writes review-notes and stops. auto-fix-from-review-notes is the one that acts.
  • Don't compare across formats. A talking-head ad vs a music video is not a reference-comparison.