video-sync

video-sync

Combine shots.json + original footage into one video: picture on one side,the current shot's breakdown infoon the other, when the shot cuts the info cuts with it, the shot table auto-scrolls and highlights.

The layout depends only on the aspect ratio, nothing else:

Original footage	Layout	Compositing method
Landscape (width > height), square	Picture on top, info below	vstack
Portrait (height > width)	Picture on left, info on right	hstack

Both directions guaranteethe picture is scaled as-is, not cropped or stretched, and the panel fills the remaining space.

Only three images are captured; the moving parts are left to ffmpeg. Only two things change on the panel: where it scrolls to, and which row is lit. So measure the row positions once, capture one base plate, and capture two long images with the shot table laid out (one dimmed, one fully lit); scrolling = crop a viewport from the dim long image, highlighting = crop one row from the bright strip long image and overlay it, with the position driven by sendcmd sending a small expression at each cut point. For a 53-shot film, captures drop from 53 to 3 (12 seconds).

The current shot is pinned to the second row: shots 1 and 2 do not scroll (the highlight moves down one row), and from shot 3 on, each cut scrolls up one row. Motion at cut points, stillness within shots—at the cut point it scrolls for --ease(default 0.45) seconds to the position and then stops; in a 16-second long take, if you let the list keep crawling, the audience has to stare at it wobbling for 16 seconds.

Row height is determined by content, so a long description makes a tall row; the highlight strip is layered by row height (crop 's height cannot be changed at runtime).

{baseDir} = the directory containing this file. The script {baseDir}/scripts/video-sync.mjs, zero dependencies,node runs directly.

Boundaries (things it does not do): it does not do shot breakdown (that is the job of video-shots , this skill consumes its shots.json), does not edit, add transitions, or add music, does not burn subtitles onto the picture, and does not do frame-by-frame rendering (motion effects are made by ffmpeg cropping a window by time).

---

Step 0 — First have shot breakdown data

Needs a shots.json（video-shots 's output) andthe corresponding original footage. Having frames/(keyframe for each shot) is even better—the shot table will use the opening frame as a thumbnail, and if there is none it leaves a blank.

shots.json must match this film：meta.durationSeconds if it does not match the original footage, do not composite; go back and get the shot breakdown right first.

Step 1 — Check the geometry first, then touch the video

node {baseDir}/scripts/video-sync.mjs plan shots.json --video <片>

Only compute, do not composite; results in a few seconds: layout, picture area, panel area, output size.Confirm these three numbers are reasonable before going further— compositing a 3-minute film takes one or two minutes, and redoing it after a miscalculation is not worth it.

If unsatisfied, adjust:--panel <比例>(for landscape it is panel height ÷ picture height, default 0.8; for portrait it is panel width ÷ picture width, default 1.6), --width / --height(the upper limit of the picture area, default 1920 / 1080).

If the source clip is too small, scale up the picture area: the panel is calculated from the picture area; for a 640×360 clip the panel is only 640×288, and four columns squeezed together are unreadable.--scale 2 Set the picture area to 1280×720 and the panel to 1280×576, keeping the aspect ratio. The default is 1 (shrink only, no upscaling).plan / panels / compose / export must all carry the same --scale。

Step 2 — Render the panel

node {baseDir}/scripts/video-sync.mjs panels shots.json --video <片> \
  --frames frames --lang zh|en [--out panels] [--chrome <路径>]

Produces static.png(base plate),list-dim.png / list-lit.png(a long image with the whole shot table laid out, one dark and one light),layout.json(the position of each row, used to calculate scrolling during compositing), plus a panels/panel.html—— Open it in a browser to preview the panel，#S07 Switch shots to see the effect (the scrolling animation in the preview matches the export).

To change the layout, change {baseDir}/scripts/panel.css, then rerun panels to get the new layout; no need to touch the script or rethink the screenshot logic. For style conventions see {baseDir}/references/panel-style.md。

Step 3 — Composite

node {baseDir}/scripts/video-sync.mjs compose shots.json --video <片> \
  [--panels panels] [-o out.mp4] [--crf 20] [--ease 0.45]

The audio track is copied from the source clip (silent clips automatically get no audio track). After compositing, stderr reports the output size, duration, and layout;give it a quick check。

The two steps can also be done in one go:

node {baseDir}/scripts/video-sync.mjs export shots.json --video <片> -o out.mp4 --frames frames

Step 4 — Extract frames for acceptance ⛔ Do not skip

Don't just see that compositing produced no error and call it done. Extract at least three frames and check whether the info matches the picture:

for t in 5 30 60; do ffmpeg -v error -y -ss $t -i out.mp4 -frames:v 1 -q:v 3 /tmp/check-$t.jpg; done

Check three things:which shot the frame's picture belongs to, whether the shot number on the panel is the same one, and whether the highlighted row follows along. If they don't match, it's usually because shots.json is not from the same clip, or --video points to the wrong thing.

Report in one sentence: output path, size, duration, layout, how many shots, and which panel images failed to render (if any).

---

Boundaries

• Scrolling only happens at cut points: it scrolls for --ease seconds to position and then stops; within a shot the panel is static. If you really want "constant crawling throughout" that's possible too (change the segments of motionPlan ), but long shots will keep making your eyes wobble
• A headless browser is required(Chrome / Chromium / Edge). This is the price of "the layout can be changed freely": switching to ffmpeg's drawtext to draw the panel would mean implementing Chinese line wrapping, thumbnails, and highlighting all yourself, and changing a layout would mean changing the filter graph
• The word lists (shot size / category / camera movement / transition) are in this skillbundled with a copy, not imported across directories— the skill must be able to be copied away whole.video-shots If new words are added over there, they must be added here too, otherwise they display as enum keys
• Rendering the panel always takes three screenshots, regardless of the number of shots: about 12 seconds for 53 shots. Compositing follows the clip length; about 75 seconds for a 203-second clip

Self-test

node {baseDir}/scripts/selftest.mjs

122 assertions, without touching ffmpeg or opening a browser: geometry (landscape/square/portrait, even side lengths, upper and lower limits, knobs), the panel page's data contract (word list delivery, rhythm roles, thumbnails only given if present, escaping), animation commands (each must be short—if the expression gets long ffmpeg fails configuration outright), ffmpeg parameters. After changing the script, run this first.