video-understanding

1. Positioning

This skill turns a source video into an understanding index readable by the Agent and downstream stages. Its creative role ismaterial observer / continuity clerk, not director:

• Observe first, then interpret; keep facts and inferences separate.
• Beyond "what happened," also let downstream see at which moment knowledge, power, goals, relationships, or emotions change.
• Mark whose POV carries the change, which reaction or performance is irreplaceable, and where natural editing boundaries for complete dialogue/action exist.
• When evidence is insufficient, preserve uncertainty; do not manufacture dramatic conclusions.

2. Processing stages

1. Scene detection: write scenes.json, including cut points, durations, and waste-segment filtering results.
2. Frame extraction: extract representative frames for visual analysis.
3. ASR: via mimo-v2.5-asr write coarse-segmented dialogue asr_result.json, and write asr_timing_evidence.json stating availability, limited time precision, and text correction sources.
4. Silence detection: write silence_periods.json, marking quiet windows and has_speech。
5. VLM observation: write vlm_analysis.json, including scene descriptions, deeper analysis, and frame_facts。
6. Timeline fusion and creative brief: write timeline_fusion.json、asr_writing_chunks.json and agent_narration_brief.md。

Each stage is reused only when the output artifacts and provenance sidecar both match the current video and the settings that affect results;--force forces recomputation.

3. Environment requirements

# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
export MIMO_API_KEY=***

ASR uses mimo-v2.5-asr; VLM uses mimo-v2.5。--skip-asr can skip dialogue transcription, but full understanding still requires MIMO_API_KEY to run VLM.--mimo-video-overview can enable per-scene-block video overview. When no key is set, reruns reuse cached transcription, visual analysis, overview, and story index (the key-determined default endpoint does not participate in comparison); consolidation that would require a model request is recorded as skipped_no_key, with no request sent. If the cache does not match (for example, copying work_dir without preserving file times) but a transcription already exists, the run stops and preserves the transcription: use cp -p / cp -Rp / rsync -t to copy again with times preserved, or set the key and rerun (which will retranscribe). Do not use --skip-asr to bypass, as it will replace the existing transcription with []。

If work_dir/background_research.json exists, this skill folds the plot synopsis and character names into the VLM context;--context can add a short hint.

The following scripts/... are all relative to this skill's directory. If the executor starts from the repository root, prepend this skill's absolute directory to the script paths.

4. Run commands

python3 scripts/understand.py <video> --work-dir <work_dir> [选项]

Option	Default	Purpose
<video>	Required	Source video
--work-dir	Required	Artifact directory; created if it does not exist
--context "..."	Empty	Short context supplied to the VLM (program name, character names), merged with background_research.json merge
--scene-threshold	0.1	Scene detection threshold
--style	纪录片	Narration style written into the creative brief
--edit-mode full|cut	Not set	recap mode written into the brief;cut estimate the narration budget by the post-cut duration; when edited_source.mp4 is available, sentence-end anchors use post-cut time
--target-duration	Not set	cut target duration written into the brief; when clip_plan_validated.json is not yet available, use it to estimate the narration budget
--skip-asr	Off	Do not transcribe dialogue; write asr_result.json as [](existing transcripts will be overwritten); ASR evidence is marked as explicitly skipped
--mimo-video-overview	Off	Run the MiMo video overview by scene block and use it as the primary per-scene description
--force	Off	Ignore the cache and recompute everything
--brief-only	Off	Rebuild only from existing artifacts agent_narration_brief.md, without extracting frames or calling APIs
--edited-storyboard-only	Off	Only by clip_plan_validated.json write the post-cut timeline storyboard/edited_storyboard.*, and add storyboard guidance at the top of the existing agent_narration_brief.md ; for multi-source plans (with sources) extract frames from each source's source_work_dir of frames/ by its frames_manifest.json fps, with tiles labeled S1/S2…; no frame extraction, no API calls; storyboard generation failures are only logged. Mutually exclusive with --brief-only mutually exclusive with
--consolidate / --no-consolidate	On	Generate the global story index understanding_index.*
--consolidate-asr	Off	Additionally clean the ASR text and write asr_clean.json

5. Output contract

A default run writes the artifacts in the table below; next to each stage's artifacts, *.meta.json is the cache provenance sidecar.

File	Content
frames/frame_*.jpg、frames_manifest.json	Frames extracted by fps and their manifest
scenes.json	Scene cut points, start/end times, and durations
audio.wav	16 kHz mono audio for ASR, silence detection, and sentence-end anchors
asr_result.json	[{start, end, text}] timestamped dialogue
asr_timing_evidence.json	ASR availability status, coarse window precision, glossary pre/post text, and the source video/audio/result files it describes (path existence + size/mtime)
silence_periods.json	[{start, end, duration, has_speech}] quiet windows
speech_boundary_anchors.json	ASR sentence-end punctuation aligned to sentence-end anchors at short pauses; when audio or ASR is missing, status: unavailable
vlm_analysis.json	Per-scene descriptions, deep analysis, and frame_facts
mimo_video_overview.status.json	video overview status (when not enabled, disabled); if enabled successfully, also write mimo_video_overview.json
understanding_index.json、understanding_index.md	global story index (--no-consolidate not written when
consolidation.status.json	Run status of the story index and ASR cleaning
storyboard/source_storyboard.{json,jpg}	Contact sheet for the original timeline (continued as _001.jpg when over page limit, etc.); if clip_plan_validated.json exists, also write edited_storyboard.*；STORYBOARD=0 off
timeline_fusion.json	Unified timeline of VLM, ASR, and silence information
asr_writing_chunks.json	ASR writing blocks split by sentence boundaries and scene cuts
agent_narration_brief.md	The creative brief the Agent reads first

Subsequent writing stages devise a plan from the creative brief and index and write narration.json。

6. Reference materials

• Background research:references/research-guide.md, producing background_research.json。
• JSON structure:references/data-schema.md。

7. Capability boundaries

• Does not write narration or score narration; only generates the understanding index and creative brief.
• Does not fabricate plots unsupported by the signals; when ASR / VLM is too thin, outputs a material warning.
• MiMo ASR's start/end is formed by fixed shardingcoarse window, not word-level alignment; empty text only indicates the cause is unknown, and cannot be treated as confirmed silence.asr_timing_evidence.json For the status fields, see references/data-schema.md。