video-understanding

1. Positioning

This skill turns a source video into an understanding index readable by the Agent and downstream stages. Its creative role ismaterial observer / script supervisor, not director:

• observe first, then interpret; keep facts and inferences separate.
• Beyond "what happened," it must also let downstream see at which moment knowledge, power, goals, relationships, or emotions change.
• Mark whose POV carries the change, which reaction or performance is irreplaceable, and where natural edit boundaries exist for complete lines/actions.
• When evidence is insufficient, retain uncertainty and do not manufacture dramatic conclusions.

2. Processing stages

1. Scene detection: write scenes.json, including cut points, durations, and waste-segment filtering results.
2. Frame extraction: extract representative frames for visual analysis.
3. ASR: via mimo-v2.5-asr write roughly segmented dialogue asr_result.json, and write asr_timing_evidence.json to explain availability, limited temporal precision, and sources of text correction.
4. Silence detection: write silence_periods.json, annotating quiet windows and has_speech。
5. VLM observation: write vlm_analysis.json, including scene descriptions, deep analysis, and frame_facts。
6. Timeline fusion and writing brief: write timeline_fusion.json、asr_writing_chunks.json and agent_narration_brief.md。

Each stage is reused only when both the output artifact and the provenance sidecar match the current video and the settings that affect results;--force forces recomputation.

3. Environment requirements

# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
export MIMO_API_KEY=***

ASR uses mimo-v2.5-asr; VLM uses mimo-v2.5。--skip-asr . Dialogue transcription can be skipped, but complete understanding still requires MIMO_API_KEY to run VLM.--mimo-video-overview can enable per-scene-block video overview. When no key is set, a rerun reuses cached transcription, frame analysis, overview, and story index (the key-determined default endpoint does not participate in comparison); consolidation requiring a model request is recorded as skipped_no_key, and no request is sent. When the cache does not match (for example, file timestamps were not preserved when copying work_dir) but a transcription already exists, the run stops and keeps the transcription: use cp -p / cp -Rp / rsync -t to copy again while preserving timestamps, or set the key and rerun (it will re-transcribe). Do not use --skip-asr to bypass it; it will replace the existing transcription with []。

If work_dir/background_research.json exists, this skill folds the plot synopsis and character names into the VLM context;--context can supplement with a brief hint.

The following scripts/... are all relative to this skill directory. If the executor starts from the repository root, prepend the absolute directory of this skill to the script path.

4. Run commands

python3 scripts/understand.py <video> --work-dir <work_dir> [选项]

Option	Default	Purpose
<video>	Required	Source video
--work-dir	Required	Artifact directory; created if it does not exist
--context "..."	Empty	Brief context to supplement the VLM (program name, character names), merged with background_research.json , merged
--scene-threshold	0.1	Scene detection threshold
--style	纪录片	Narration style written into the creation brief
--edit-mode full|cut	Not set	recap mode written into the brief;cut estimate the narration budget by the post-cut duration; when there is already a edited_source.mp4 use post-cut time for sentence-end anchors
--target-duration	Not set	cut target duration written into the brief; when there is not yet a clip_plan_validated.json use it to estimate the narration budget
--skip-asr	Off	do not transcribe dialogue, write asr_result.json as [](existing transcriptions will be overwritten), mark ASR evidence as explicitly skipped
--mimo-video-overview	Off	run the MiMo video overview by scene blocks, and use it as the per-scene primary description
--force	Off	ignore the cache, recompute everything
--brief-only	Off	rebuild using only existing artifacts agent_narration_brief.md, without extracting frames or calling APIs
--edited-storyboard-only	Off	write the post-cut timeline only according to clip_plan_validated.json Post-cut timeline storyboard/edited_storyboard.*, and add storyboard guidance at the top of the existing agent_narration_brief.md ; for multi-source plans (with sources) take frames from each source source_work_dir 's frames/ according to its frames_manifest.json fps, label tiles S1/S2…; do not extract frames or call APIs, and only log if storyboard generation fails. Mutually exclusive with --brief-only Mutually exclusive
--consolidate / --no-consolidate	On	generate the global story index understanding_index.*
--consolidate-asr	Off	additionally clean the ASR text, write asr_clean.json

5. Output contract

A default run writes the artifacts in the table below; next to each stage's artifacts, *.meta.json is the cache provenance sidecar.

File	Content
frames/frame_*.jpg、frames_manifest.json	Frames extracted by fps and their manifest
scenes.json	Scene cut points, start/end times, and durations
audio.wav	16 kHz mono audio, used for ASR, silence detection, and sentence-end anchors
asr_result.json	[{start, end, text}] timestamped dialogue
asr_timing_evidence.json	ASR availability status, coarse-window precision, text before and after glossary, and the source video/audio/result files it describes (path existence + size/mtime)
silence_periods.json	[{start, end, duration, has_speech}] quiet windows
speech_boundary_anchors.json	sentence-end anchors aligning ASR sentence-final punctuation to short pauses; when audio or ASR is missing status: unavailable
vlm_analysis.json	per-scene descriptions, deep analysis, and frame_facts
mimo_video_overview.status.json	video overview status (when not enabled, disabled); if enabled successfully, additionally write mimo_video_overview.json
understanding_index.json、understanding_index.md	global story index (--no-consolidate not written when)
consolidation.status.json	Run status of story indexing and ASR cleaning
storyboard/source_storyboard.{json,jpg}	Original-film timeline contact sheet (continue writing when exceeding a page _001.jpg , etc.); when there is already a clip_plan_validated.json additionally write edited_storyboard.*；STORYBOARD=0 off
timeline_fusion.json	Unified timeline of VLM, ASR, and silence information
asr_writing_chunks.json	ASR writing blocks split by sentence boundaries and scenes
agent_narration_brief.md	The creative brief that the Agent reads first

Subsequent writing stages formulate a plan based on the creative brief and index and write narration.json。

6. Reference materials

• Background research:references/research-guide.md, producing background_research.json。
• JSON structure:references/data-schema.md。

7. Capability boundaries

• Does not write narration, nor does it score narration; it is only responsible for generating the understanding index and creative brief.
• Does not fabricate plot that the signals cannot support; when ASR / VLM is too thin, outputs a material warning.
• MiMo ASR's start/end is formed by fixed segmentationCoarse window, not word-level alignment; empty text only indicates the reason is unknown, and cannot be treated as confirmed silence.asr_timing_evidence.json See the status field of references/data-schema.md。