1. Positioning and workflow
This skill is a lightweight orchestrator of five independent skills. The skills communicate only through the JSON / MP4 artifacts in work_dir , and do not share code:
video-understanding ─▶ Agent 按 video-script 制定方案并写稿 ─▶ [video-cut] ─▶ video-voiceover ─▶ video-assemble
Breaking a final cut into production references is not the production path. When the user requests it: run video-understanding on the final cut (recommended ASR_SEGMENT_SECONDS=5),
then use video-reference to do measure, annotation, check, and export, and copy the exported production_reference.json into the next run's
work_dir, or register it as a resource library's production_reference template, and after adoption bind it via --project . recap does not run it automatically, nor does it compare a new final cut against the reference.
The workflow supports resuming from a checkpoint: after writing narration.json , repeat the same command to continue. The second stage compares
recap_run_manifest.json the recorded source video path, file size/modification time, and run parameters, and refuses to reuse an old working directory from another source video or other parameters; video-understanding artifacts are also reused only when the source matches.
The resume command printed on pause is the original command: keep the original wording, convert the video and path parameters to absolute paths, add --work-dir and the settings from environment variables, and it can be run directly from any directory; the same set of parameters is written in the manifest's argv。
Visual workflow --edit-mode full|cut|dub and sound strategy --audio-mode are two independent switches;
narration keeps the above narration workflow,source-mix does no dubbing,adopted-packet-copy freezes the adopted AAC audio track of the current input.
There are only the following combinations of paths; all other combinations error out immediately at startup:
| Input | --edit-mode |
--audio-mode |
Agent pause point | Workflow | See |
|---|---|---|---|---|---|
| Single video | full | narration | 1:narration.json |
understanding → scriptwriting → validation → dubbing → compositing | §4 |
| Single video | cut | narration | 2:clip_plan.json, then write against the final cut narration.json |
understanding → editing → rebuild output timing brief → scriptwriting → dubbing → compositing | §4 |
| Multiple videos | cut | narration | 2: same as above, clip must carry source_id |
per-source understanding → editing → scriptwriting → dubbing → compositing | §4.3 |
| Single video | full | source-mix / adopted-packet-copy | None | directly composite the current full segment | §4.7 |
| Single / multiple videos | cut | source-mix / adopted-packet-copy | 1:clip_plan.json |
understanding → editing → compositing, no scriptwriting | §4.7 |
| Single video | dub | narration | 1:dub_script.json |
English transcription → translated script → cloned voice full-track replacement | §5 |
| Pre-edited master | full | narration + three adopted JSON | None | strict compositing only | below |
All full/cut paths share the same closing segment: (when there is narration) review → TTS → compositing → final-cut QC. When using original-audio mode, read
references/audio-routing.md。
When there are existing pre-made visuals and the complete locally adopted three-piece sound set, the strict assembly-only path can be used:
python3 scripts/recap.py picture.mp4 --edit-mode full --work-dir NEW_WORK \
--output-dir DELIVERY \
--tts-meta tts_meta.json \
--narration-adoption narration_adoption.json \
--audio-mix-adoption audio_mix_adoption.json
The three JSON parameters must appear at the same time. This entry point only accepts a single video, full, narration, audio track 0, a new working directory, and a non-existent delivery file; it does not run understanding, scriptwriting, narration review, TTS, cut, or CapCut export. Semantics and media shape are still strictly validated by video-assemble; recap only verifies that the sub-skill binding record references the same batch of adopted files and master path, and does not declare the caller-adopted voice or mix as automatic creation or publication approval. See references/audio-routing.md。
The single video here isan already-cut master. After re-cutting, the unchanged WAV and tts_meta.jsonmay be reused, but the placement points and prepared audio bed in the mix adoption file must be rewritten according to the new master; for the handoff steps see references/audio-routing.md 's "Keep adopted voice after a cut".
2. Creative Responsibilities
This is not a mere JSON / rendering pipeline. The Agent is the creative owner of this content. First determine this round'screative control mode(CREATE / DIRECTED / REVISION, unrelated to --edit-mode ), then complete five judgments before entering expensive downstream processing:
- Directorial judgment: audience promise, POV, dramatic question, emotional endpoint, and reveal pacing.
- Story editing: a beat is defined as "what changed", not a scene summary.
- Picture editing: select the specific moments, reactions, in-points and out-points truly worth keeping.
- Sound/narration: first assign the tasks of picture, original sound, silence, and narration, then write the narration script.
- Audience review: separately check the experience without narration, with audio only, and on first viewing.
The definitions of the three control modes, the modification/freeze rules of REVISION, the creative method, and the writing of recap_story_plan.json / visual_audio_board.json / style_card.json are all executed according to video-script ; it will require reading the creative manual first. These files only record auditable current decisions and do not add service or rendering dependencies.
3. Environment and Script Paths
# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
export MIMO_API_KEY=***
The same MiMo key drives:
- ASR:
mimo-v2.5-asr - VLM:
mimo-v2.5 - TTS:
mimo-v2.5-tts
The TTS provider is passed through to the dubbing skill by --tts-provider mimo-tts|fish-audio|index-tts(or TTS_PROVIDER); the respective environment variables, default voices, and capability limits of Fish Audio and self-hosted index-tts are described in that skill. ASR/VLM always uses MiMo.--doctor only performs offline configuration checks.
tp-* Token Plan keys default to the China cluster and can be overridden with MIMO_TOKEN_PLAN_CLUSTER .
Optional capabilities:
--mimo-video-overview: supplement MiMo video understanding by scene blocks.
For overridable configuration see references/config-playbook.md,final_qc.json 's fields see references/data-schema.md。
The scripts/... below are all relative to this skill's directory. If the executor is started from the repository root, add this skill's absolute directory to the script paths. After startup the script will locate sibling skills and resources by itself.
4. Standard Narration Flow (audio-mode narration)
4.1 Background Research
If the film, series, or theme can be identified, first conduct research according to the research guide of the video-understanding skill research-guide.md and write to
work_dir/background_research.json. Video understanding folds character names and plot background into the VLM context to avoid getting only vague descriptions like "man in black". If the source cannot be identified, this can be skipped.
Multi-video runs are likewise written under the project work_dir : recap copies it to sources/<source_id>/(Copy only when it is missing from the source directory or older than the project file, so when a particular episode needs separate research, just write that source directory after the project file).
4.2 Analysis and pause for creation
python3 scripts/recap.py <video> --work-dir <work_dir> --context "背景"
The command completes video understanding, writes out agent_narration_brief.md, then pauses. At this point, execute in the following order video-script:
- Review the creation brief and the original film's storyboard.
- Write
recap_story_plan.jsonandvisual_audio_board.json。 - In full mode, write
narration.json; in cut mode, the first stage writes onlyclip_plan.json。 - In the second stage of cut mode, review the edited storyboard, add the output timing and audio division of labor, then write
narration.json。
Do not start from the title or a narration sentence; first lock down the story experience and material selection.
The timeline has two hard constraints that cannot be downgraded: original audio can only be cut in, cut out, or restored at reliable sentence ends/silence boundaries; narration must use
complete segment-by-segment audio, and any clip mapping trimming, TTS tail trimming, or CapCut referencing longer pre-acceleration material is blocked. When the Agent receives
interrupts_source_sentence / unsafe_clip_sentence_boundary / no_safe_fit /
timeline_audio_mismatch , it should move the boundary, shorten the whole sentence, or delete that block, rather than adding a snatch override;
unsafe_clip_sentence_boundary the accompanying nearest_safe gives the nearest safe boundary times before and after.
4.3 Multi-video and material library
Multi-video only supports cut mode. The project brief will list stable source_id,clip_plan.json Every clip in it must fill in the source:
python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --target-duration 10m --work-dir work_dir_multi_ep
The second stage generates the edited storyboard just like single video storyboard/edited_storyboard.*, taken from each source sources/<source_id>/frames/, and the top of the brief lists S1/S2… corresponding source_id; sources recovered from the material library have no extracted frames, so these clips do not appear in the storyboard; use inspect clip-map to verify.
Optional filesystem material library:
python3 scripts/recap.py ep1.mp4 --material-library-dir .video-materials --save-materials
python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --material-library-dir .video-materials --use-materials
Material retrieval is just grep over JSON / MD / JSONL, for example grep -R "keyword" .video-materials. The current version does not copy original media, nor does it provide database, vector, or semantic search.
The same root directory can also register reusable resources (BGM, sound effects, voices, fonts, images), templates with versions and adoption records (subtitle styles, packaging layers, production references), and sample clips.
Format and scripts/library.py check|list|show For read-only tools, see references/resource-library.md. Use --project recap_project.json to bind the adopted subtitle styles, packaging, production references, voices, and BGM to this run;
after each full / cut composition work_dir/resource_lock.json record the resources actually used and their authorization status.
4.4 Continue generating the final video
After writing the required artifacts, repeat the same command:
python3 scripts/recap.py <video> --work-dir <work_dir> # 可追加 --edit-mode cut / --no-burn-subtitles
The process will validate the hard inputs of the current stage (clip_plan.json / narration.json); the two creation plans are still working records used by the Agent and advisory reviewers, not rendering gates. Cut mode then generates edited_source.mp4, then synthesizes narration and outputs recap_<name>.mp4。
Validation never rewrites the narration script (it only updates the measured overlaps_speech); in full mode, when the text cannot fit the time window, it is returned to the Agent as a over_budget error. When there is already an approved narration script, add --preserve-approved-text, and TTS also preserves the approved script as-is; if it cannot fit the time window, it fails, without shrinking the script or degrading to partial success.
4.5 Subtitles and cloned narration
If you want to fix the narration subtitles in the original film's subtitle area, first run at the repository root:
python3 tools/measure_subtitle.py <video>
Then pass in the measured --subtitle-y-top/--subtitle-y-bot. The coordinates are based on the display canvas after ffmpeg automatic rotation, with a half-open interval [top, bot), and require a bottom-aligned ASS style; after explicit setting, this area uses a 60% opacity narration window mask by default.
In commentary mode, if you need to clone a reference voice, use --voice-ref <audio>; it differs from dub mode.
4.6 Final viewing and delivery review
Scripts, cut-point detection, sample frames, and QC reports cannot replace viewing. Before preparing each round of delivery, you must checkthe final files actually to be delivered in this round, not old aliases, subtitle-free masters, or intermediate proxies:
- Play the short film through once completely at normal speed, without editing while watching; first record the real viewing issues.
- Play about 0.5–1 second before and after each splice point, checking for flash frames, truncated dissolves from the original footage, action jumps, and half-sentence original audio.
- Listen to the audio once completely on its own, checking whether the narration is chopped into one sentence with a pause, whether the sound connects between scenes, and whether key original audio is complete.
- Separately review the opening, core emotional/performance points, and ending, confirming that the entry timing, payoff hold, and closure all work.
- For REVISION, separately verify that this round's changes have actually changed and that frozen items have not changed unexpectedly; then perform mechanical checks such as decoding, duration, and audio-video specifications.
Scene score, brightness statistics, contact sheet, and automated QC are only responsible for locating candidate issues; the final judgment is based on actual playback. The source judgment and handling rules for dense cut points follow the editing skill. When a fix fails, return to the cut point, sound, or copy layer; do not use more packaging to cover it up.
For full/cut delivery, if you want the deterministic final check to affect the command exit status, explicitly pass
--require-final-qc. Only when the summary of final_qc.json is ok: true and the integer
blocker_count: 0 will it print completion and return success; missing, malformed, or blocker
will retain the report and rendered diagnostic media, but the command exits nonzero and does not print completion. The default is still report-only, non-blocking.
This parameter does not support --edit-mode dub; the preparation and rendering behavior when dub does not pass this parameter remains unchanged.
At delivery time, read final_qc.json the metadata.warnings, and if there is content, tell the user as-is; you must not report only "complete".subtitle_burn_degraded
indicates that ffmpeg lacks libass and the default burn-in has been downgraded: there are no subtitles in the final film, and the subtitles are in the same-named file next to the final film .srt. If the user wants burned-in subtitles, ask them to install
ffmpeg with libass and rerun; when explicitly passing --burn-subtitles , missing libass will error out before starting. If the narration contains visual_overlays and ffmpeg lacks
drawtext, the process stops before dubbing; follow the error to remove the overlay or switch ffmpeg and continue.
4.7 Films that do not need commentary
# 对当前整段输入直接合成;不隐式跑理解/ASR/TTS
python3 scripts/recap.py locked_picture.mp4 --work-dir source_work --audio-mode source-mix
# 剪辑计划仍按 cut 流程产生,剪完不再暂停等待 narration.json
python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --work-dir cut_work --audio-mode source-mix
# 只换包装时冻结当前整片 AAC;不允许同时加 BGM/TTS
python3 scripts/recap.py adopted.mp4 --work-dir packaging_work --audio-mode adopted-packet-copy
source-mix will still be mixed and re-encoded;cut + adopted-packet-copy what is frozen is the audio of the edited intermediate film, not the AAC packets of the original film.
The current strict subtitle track only supports adopted mode; other subtitle sources have not thereby become precisely aligned. Switching sound mode requires a new working directory; old TTS, QC, or automatically generated commentary captions must not be mixed into this round's original-audio production. For details, see references/audio-routing.md。
5. English-to-Chinese original-voice replication mode
--edit-mode dub is an experimental feature: it translates an English video into Chinese and replaces the human voice with the original speaker's cloned timbre; it is not adding commentary over lowered original audio. Use it only when the user explicitly requests English-to-Chinese dubbing.
dub sends the source video audio track in windows to MiMo ASR for transcription, and sends the captured approximately 10 seconds of the original speaker's voice as a reference, together with each sentence's translation, to MiMo voiceclone. Therefore it must include --confirm-voice-rights: first confirm with the user that he has the right to use this video and audio and that the speaker agrees to have his voice cloned, and only add this parameter after confirmation; if missing, it errors out at startup, extracts no audio, and sends no request. This parameter is only for dub; passing it in other modes will error.
python3 scripts/recap.py <video> --edit-mode dub --confirm-voice-rights --work-dir <work_dir>
The preparation stage transcribes the English, extracts a reference audio clip, and writes dub_brief.md and dub_transcript.json. The Agent follows the translation requirements in dub_brief.md (faithful sentence by sentence, time windows, speech rate) to write dub_script.json:
[{"start": 0.0, "end": 2.0, "zh": "中文译文"}]
After repeating the same command, it first runs a deterministic lint (dub_lint.json, aborting on blank lines, overlaps, or out-of-bounds), then outputs dub_<name>.mp4. Each sentence is cloned individually and pasted back onto the original timeline; local speedup is applied only when it is about to overlap the next sentence. The current version supports only a single speaker and whole-track replacement, and does not separate background music.
6. Self-check and read-only dashboard
python3 scripts/recap.py --doctor
6.1 Read-only dashboard
python3 scripts/dashboard_server.py --root <目录> [--port 0] [--open]
Runs in the foreground and prints the local address (binds only to 127.0.0.1; the Agent puts it in the background at startup). Under --root it discovers the resource library, projects, and runs by library.json、recap_project.json、
recap_run_manifest.json , and displays by stage the editing rhythm, narration, final cut and timeline, QC, and resource_lock.json.
Strictly read-only: accepts only GET / HEAD, writes no files; the "copy to assistant" on the page copies only a single request, and changes are made back in the conversation.
7. Outputs and parameters
Main outputs:
recap_<video>.mp4: the final cut.subtitles.srt/subtitles.ass: subtitles.work_dir/: all intermediate products, see the contract inreferences/data-schema.md。work_dir/recap_story_plan.json/visual_audio_board.json: the Agent's creative intent and editing decisions.
The full parameter list is subject to python3 scripts/recap.py --help .--style is free-text guidance passed verbatim to the Agent, not a preset, enum, switch, or limited style category.
8. Capability boundaries
- Semantic review is advisory by default and fails open; only when the caller explicitly enables strict commentary review will factual contradictions, incomplete sentences, or review unavailability block before TTS. The deterministic validation stage is always responsible for hard validation.
- For promotional titles, on-screen text, or external copy backfilling, see
video-script's references/promotional-copy.md.