video-shots

video-shots

For a finished filmShot breakdown— split a film into a shot-by-shot analysis table:duration, shot size, category, camera movement, frame。

The premise is carved into the bone: shot boundaries are measured, not seen. The least reliable thing a model does when watching video is reporting time — "this shot is about 3 seconds" and "2.97 seconds" differ not in precision, but in whether this table is usable. So a hard line is drawn here:

Who decides	What	How it is decided
Code	Cut points, duration, film length, frame rate	ffmpeg scene detection + ffprobe, two decimal places
Code	Each shot's measured motion amount	The interval median of the frame-by-frame difference curve (trimming both ends to avoid cut-point spikes)
Model	Shot size, category, camera movement, frame, rhythm	Judge by looking at keyframes —Only these few things are the model's job
Code	Whether the judgment is correct	15 quality gates, checked one by one

The hardest gate iscamera movement measured reconciliation: if the camera really moved, the pixels cannot stay unchanged. So "claiming push/pull/pan/track/follow, with measured inter-frame change close to 0" is directly blocked — this is the most common hallucination in model shot breakdowns. Conversely (claiming fixed, with measured motion high) is not blocked, only a note is given: if someone is dancing in front of a fixed camera, the inter-frame difference will also spike.

{baseDir} = the directory where this file is located. The script {baseDir}/scripts/video-shots.mjs, zero dependencies,node run directly.

Boundaries (things not done): no speech transcription (no ASR; dialogue is read from subtitles burned into the frame, and if unreadable it is left blank with an explanation), no face recognition or automatic person merging (cast is manually numbered), no judging whether the film is good or bad (the report only gives facts and statistics), no editing or exporting clips, no object detection within shots.

---

Step 0 — Define input and scope

Only one video file is needed. First clarify two things; if they cannot be clarified, go with defaults and explain in the report:

• Break down the whole film or a segment: the whole film is the default. If only a certain segment is needed, first cut it out with ffmpeg and then break it down; do not mark half of the whole film.
• What the breakdown is for: as imitation-writing reference (emphasize frame and camera movement), for editing rhythm analysis (emphasize duration and category), for inventorying ad materials (emphasize product shots and text cards). Different uses mean different things should be recorded more innote —the table structure is the same。

Step 1 — seed working draft ⛔ cut points are fixed at this step

cd <输出目录>
node {baseDir}/scripts/video-shots.mjs seed <video> --track track.json --title "<片名>" > shots.json

stderr will report: film length, frame rate, resolution, how many cut points were detected, and how many shots after merging.Look at this line first before continuing：

• Average shot length of over ten seconds and an obviously low shot count → the threshold is too high,--threshold 0.15 Re-run (dark scenes, slow films, and films with lots of same-camera dialogue shots all need to be turned down)
• The number of shots is noticeably more than what you can count by eye → the threshold is too low, turn it toward 0.4 , or leave it to Step 4 and use recut --merge and
• A 3-minute film finishes in just a few seconds,running it a couple more times is more worthwhile than settling for a bad draft

In the draft, start / end / seconds / motion are already filled in,size / category / camera / frame are empty— those four blank cells are the model's job。

Step 2 — Extract keyframes and contact sheets

node {baseDir}/scripts/video-shots.mjs frames shots.json --video <video>
node {baseDir}/scripts/video-shots.mjs sheet shots.json --cols 4 --rows 6
node {baseDir}/scripts/video-shots.mjs sheet shots.json --cols 4 --rows 6 --pick b

Two per shot:frames/S01a.jpg(at the 15% starting point) and S01b.jpg(at the 85% ending point). The contact sheet stitches them each into one large image (row-major, S01 at the top left),sheet a shows content, sheet b shows camera movement— Compare the same cell before and after, and you can tell at a glance whether the framing changed.

Look at the contact sheets first, then the individual frames. Flipping through the whole film one by one wastes quota: one contact sheet = twenty-some shots, only go back to look at individual frames for the few shots you can't judge confidently (Read frames/S07a.jpg）。

Step 3 — Fill in the four fields batch by batch

One batch ≤ 25 shots (exactly one contact sheet). Each batch gets:

• {baseDir}/references/taxonomy.md(four word lists + criteria,fill in accordingly) and {baseDir}/references/analysis-pass.md(how to look, common problems)
• this batch's shot drafts (shot number, start/end, duration,measured motion）
• this batch's a / b contact sheets

Order of filling:shot size → category → camera movement → image → rhythm. For camera movement, look at the a/b framing difference + measured motion value, when the two conflicttrust the measurement. By the way, record subjects / onscreenText / audio—— dialogue subtitles burned into the image count as speech, put into audio and include the speaker; film title, title cards, and interface text go into onscreenText。

Rhythm (rhythm + rhythmNote) answers "why hasn't the audience scrolled away yet": hook / setup / escalation / accent / turn / payoff / breath / close, choose one of the eight roles, then write clearly in one sentencewhat the audience sees at this moment(for criteria see taxonomy.md Section 5). Optional field: if you don't do rhythm analysis, don't label a single shot; once you label it, you have to label the whole film, the gate will block half a sheet. After labeling everything, validation will also remind you of three things: whether the opening has a hook, whether there is setup before a payoff, and whether six shots in a row have flat rhythm.

When editing shots.json ,only touch those fields：start / end / seconds / motion / seedCuts / meta are machine fields; changing them is forging evidence, and the gate will call it out.

Step 4 — Patch cuts and merge cuts (if you find missed cuts, fix them, don't settle)

Scene detection inevitably makes mistakes in two places: dissolves and dark-to-dark scenesmissed cuts, handheld shake and flashesextra cuts. If frame a and frame b are fundamentally two scenes, that's ironclad proof of a missed cut.

node {baseDir}/scripts/video-shots.mjs recut shots.json --track track.json \
  --split 63.5 --split 127.37 --merge 45.97 > shots.new.json && mv shots.new.json shots.json

Automatically renumber, recompute duration and measured motion, record patched cuts into manualCuts（boundary the gate recognizes it). For shots whose boundaries were not changed, annotations are kept as is; for shots split or merged, annotations are cleared and written in note to state their origin— whether these two halves are the same thing has to be re-examined in the image; you're not allowed to carry the old description over.

After changes, re-extract frames for these shots (frames will overwrite the entire directory, just rerun it), fill in the cleared cells.

Step 5 — Validate ⛔ cannot skip

node {baseDir}/scripts/video-shots.mjs validate shots.json --track track.json --frames frames

All 15 gates are code: timeline continuity (in order, end-to-end, from 0 to the end of the film), duration consistency (seconds = end − start, short shots must carry a note), shot numbers sequential,shot size/category/camera movement three word lists, transition enum,visual description verifiable (Chinese ≥12 characters / English ≥8 words + filler-word list + must not start with "this shot…" or "This shot…"),visual descriptions not repeated, subject reconciliation cast、 category must have evidence(dialogue must have lines, title cards must have on-screen text, reactions must state who, empty shots must not contain people), camera movement measured reconciliation、boundaries come from detection(cuts you added yourself must be declared in manualCuts ), keyframes complete, rhythm analysis verifiable(optional field: mark the whole film or mark none of it; if marked, must clearly state why, filler words are still blocked).

Fix violations one by one, rerun after fixing, until it passes. Skipped gates will state the reason clearly (didn't provide --track, didn't create cast, keyframe directory does not exist)——skipping is not passing, must be mentioned in the report. For English films add --lang en, gate names and violation messages will all be spoken in English.

The "Hints (non-blocking)" column is not an error, it is where human judgment is needed: a fixed camera position measured as high is mostly likely because the subject is moving, or you may have missed a slow push, go back and look at that shot yourself.

Step 6 — Produce report and report out

node {baseDir}/scripts/video-shots.mjs render shots.json --md --track track.json > shots.md
node {baseDir}/scripts/video-shots.mjs render shots.json --html --track track.json \
  --video <原片相对报告的路径> > shots-report.html

--video point the player in the report to the original film (by default use the sourcefrom the JSON; viewers can also select a local file on the page). UI language uses --lang zh|en(default Chinese).render automatically goes to frames/ to find keyframes, extract frames first then render, if images are missing say so clearly, do not pad with placeholder images.

The report is asingle-file interactive page(styles and interactions come from {baseDir}/scripts/report.css and report.js, the entire section is inlined at render time; these three files must be copied together):

• Player: during playback, synchronously highlight the current shot and fill the timeline progress; click any shot or timeline segment to jump there
• Shot rhythm band: band width = duration proportion, color depth = shot size distance
• Shot table: list / card two views, first and last keyframes side by side (viewing them together is the camera movement), searchable (shot number, visual, lines, characters), filterable by category, sortable by duration; click a keyframe to open a large image
• Statistics distribution / characters appearing / quality check: collapsed by default, expand as needed; click a character card to filter out all their shots
• The header gives the conclusion in one line (14 项通过 · 1 条提示), shot numbers named in the hints can be clicked to jump there

Report out in one sentence clearly:how many shots, average shot length, cuts per minute, the main bulk of shot sizes and camera movements, where the longest and shortest shots are, report path; how many cuts were added and how many merged, which gates were skipped, which hints need human review, state clearly.

Final deliverables:

<输出目录>/
├── shots.json              ← 拉片主数据
├── track.json              ← 运动曲线（机器证据，别手改）
├── shots.md
├── shots-report.html       ← 双击就能开
├── frames/                 ← S01a.jpg / S01b.jpg …
└── sheets/                 ← sheet-a01.jpg / sheet-b01.jpg …（联系表）

---

Boundary

• No speech transcription. Dialogue comes only from subtitles burned into the picture; for footage with no subtitles, audio leaving a large area blank is normal, in which case mark dialogue as subject is more honest—the category evidence gate will force you to make this choice
• The measured motion does not distinguish between camera movement and subject movement. So the camera movement gate only blocks the direction of "claiming large movement but measuring no movement"; the reverse only produces a warning. To adjust strictness, change params.staticMaxMotion / busyMinMotion
• Scene detection does not recognize dissolves. For dissolve segments, take the midpoint as the cut point,transitionIn and write dissolve
• The upper limit on shot count depends on patience, not the script. A 90-minute film can be processed, but that means dozens of contact sheets; for long films, it is recommended to split into chapters and process in batches
• Bilingual Chinese-English：--lang zh|en runs through all commands—gate names, violation messages, command-line output, report interface, and all four word lists switch along with it (priority --lang > JSON top-level lang > Chinese).What switches is labels, the visual description, dialogue, and character names stay unchanged—that is content, not labels. seed --lang en will write lang: "en" into the draft, and later recut / validate / render without --lang also follows English (even the note written as a patch and the punctuation in English tables are half-width)
• The criteria for visual descriptions follow the language of the description itself: Chinese counts characters (≥12 characters), English counts words (≥8 words), and the empty-talk word list and filler opening list each have their own set. When processing an English film, write in English; the gate will still block it
• To view video in the report, you need the original file:--video point to the path, or select the file on the page. The report itself does not embed video data

Self-test

node {baseDir}/scripts/selftest.mjs

449 assertions, no model calls, no quota spent, no ffmpeg touched.Each of the 15 gates has a breach test case—proving it really blocks. Run this first after changing the script.

Built-in sample

{baseDir}/examples/demo-shots.json + demo-track.json: a complete breakdown of a 202.9-second AI short film— 53 shots, average shot length 3.83 seconds, 15.7 cuts per minute; dialogue accounts for 58%, fixed camera 55%; shortest 0.33 seconds (a flash cut of running in the snow), longest 16.06 seconds (a long take under the light pillar at the end). seedCuts 10 more cuts were added beyond that (dissolves and end credits title cards each account for half, all recorded in manualCuts ), all 15 gates green, with two warnings kept as examples (measured camera movement on the high side, six consecutive shots at the end with flat pacing). It is a quality baseline and also a self-test fixture.