SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsvideo-recap

video-recap

Design & creation

View repository

Updated 10-16 04:59

About this skill

1. Positioning and Workflow

This skill is a lightweight orchestrator of five independent skills. Each skill communicates only through work_dir JSON / MP4 artifacts, and they do not share code:

video-understanding ─▶ Agent 按 video-script 制定方案并写稿 ─▶ [video-cut] ─▶ video-voiceover ─▶ video-assemble

Decomposing a finished video into production references is not a production path. When the user requests it: run video-understanding on the finished video (recommended ASR_SEGMENT_SECONDS=5), then use video-reference to do measure, annotation, check, and export, and copy the exported production_reference.json into the next run's work_dir, or register it as a resource library's production_reference template, and after adoption bind it via --project . recap does not run it automatically, nor does it compare a new finished video against the reference.

The workflow supports resuming from checkpoints: after writing narration.json , repeating the same command will continue. The second phase compares recap_run_manifest.json the source video path, file size/modification time, and run parameters recorded; it refuses to reuse old working directories from other source videos or other parameters; video understanding artifacts are also reused only when the source is consistent. The resume command printed on pause is the original command: keep the original wording, convert the video and path parameters to absolute paths, add --work-dir and the settings from environment variables, and it can be run directly from any directory; the same set of parameters is written in the manifest's argv。

Visual workflow --edit-mode full|cut|dub and audio strategy --audio-mode are two independent switches; narration keeps the above commentary workflow,source-mix does not do voiceover,adopted-packet-copy freezes the already adopted AAC audio track of the current input. There are only the following combinations of paths, and other combinations error out directly at startup:

Input --edit-mode --audio-mode Agent pause point Workflow See
Single video full narration 1：narration.json understanding → scriptwriting → validation → voiceover → compositing §4
Single video cut narration 2：clip_plan.json, then write against the finished video narration.json understanding → editing → rebuild output timeline brief → scriptwriting → voiceover → compositing §4
Multiple videos cut narration 2: same as above, clip must include source_id per-source understanding → editing → scriptwriting → voiceover → compositing §4.3
Single video full source-mix / adopted-packet-copy None Directly composite the current entire segment §4.7
Single / multiple videos cut source-mix / adopted-packet-copy 1：clip_plan.json understanding → editing → compositing, no scriptwriting §4.7
Single video dub narration 1：dub_script.json English transcription → translated script → cloned voice full-track replacement §5
Already edited master full narration + three adopted JSONs None Strict compositing only Below

All full/cut paths share the same final stage: (when narration is present) review → TTS → compositing → finished video QC. When using original-audio mode, read references/audio-routing.md。

When there are already pre-made visuals and the complete locally adopted three-piece audio set, the strict assembly-only path can be used:

python3 scripts/recap.py picture.mp4 --edit-mode full --work-dir NEW_WORK \
  --output-dir DELIVERY \
  --tts-meta tts_meta.json \
  --narration-adoption narration_adoption.json \
  --audio-mix-adoption audio_mix_adoption.json

The three JSON parameters must appear together. This entry point only accepts a single video, full, narration, audio track 0, a new working directory, and a non-existent deliverable file; it does not run understanding, scriptwriting, narration review, TTS, cut, or CapCut export. Semantics and media shape are still strictly validated by video-assemble; recap only verifies that the sub-skill binding record references the same batch of adopted files and master path, and does not declare the caller's adopted voice or mix as automatic creation or publication approval. See references/audio-routing.md。

Here the single video isan already-cut master. After re-cutting, unchanged WAVs and tts_meta.jsonmay be reused, but the placement points and prepared audio bed in the mix adoption file must be rewritten according to the new master; for the handoff steps, see references/audio-routing.md 's “Keep adopted voice after a cut”.

2. Creative Responsibilities

This is not a mere JSON / rendering pipeline. The Agent is the creative lead for this content. First determine this round'screative control mode(CREATE / DIRECTED / REVISION, unrelated to --edit-mode ), then complete five judgments before entering expensive downstream processing:

  1. Director's judgment: audience promise, POV, dramatic question, emotional endpoint, and reveal pacing.
  2. Story editor: a beat is defined as “what changed”, not a scene summary.
  3. Picture editing: choose the specific moments, reactions, in-points, and out-points truly worth keeping.
  4. Sound/narration: first assign the roles of picture, original sound, silence, and narration, then write the commentary script.
  5. Audience review: separately check the experience without narration, with audio only, and on first viewing.

The definitions of the three control modes, the modification/freeze rules of REVISION, the creative method, and how to write recap_story_plan.json / visual_audio_board.json / style_card.json are all executed according to video-script ; it will require reading the creative manual first. These files only record the auditable current decisions and do not add service or rendering dependencies.

3. Environment and Script Paths

# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
export MIMO_API_KEY=***

The same MiMo key drives:

  • ASR：mimo-v2.5-asr
  • VLM：mimo-v2.5
  • TTS：mimo-v2.5-tts

The TTS provider is passed through by --tts-provider mimo-tts|fish-audio|index-tts(or TTS_PROVIDER) to the voiceover skill; for Fish Audio and self-hosted index-tts, see that skill for their respective environment variables, default voices, and capability limits. ASR/VLM always uses MiMo.--doctor only performs offline configuration checks.

tp-* Token Plan keys use the China cluster by default, which can be overridden with MIMO_TOKEN_PLAN_CLUSTER .

Optional capabilities:

  • --mimo-video-overview: supplement MiMo video understanding by scene blocks.

For overridable configuration, see references/config-playbook.md，final_qc.json 's fields, see references/data-schema.md。

Below, the scripts/... are all relative to this skill directory. If the executor is started from the repository root directory, add this skill's absolute directory to the script path. After startup, the script locates sibling skills and resources on its own.

4. Standard Narration Flow (audio-mode narration)

4.1 Background Research

If the film, series, or topic can be identified, first research according to the video-understanding skill's research guide research-guide.md and write to work_dir/background_research.json. Video understanding folds character names and plot background into the VLM context, avoiding vague descriptions like “a man in black”. If the source cannot be identified, it may be skipped. Multi-video runs are likewise written under the project work_dir : recap copies it before each source understanding to sources/<source_id>/(Copy only when it is not in the source directory or is older than the project file, so when a certain episode needs separate research, just write that source directory after the project file).

4.2 Analyze and pause creation

python3 scripts/recap.py <video> --work-dir <work_dir> --context "背景"

The command completes video understanding, writes out agent_narration_brief.md, and then pauses. At this point, execute the following in order video-script：

  1. Review the creation brief and the original film's storyboard.
  2. Write recap_story_plan.json and visual_audio_board.json。
  3. In full mode, write narration.json; in the first stage of cut mode, only write clip_plan.json。
  4. In the second stage of cut mode, review the post-cut storyboard, supplement the output timing and sound division of labor, then write narration.json。

Do not start from the title or narration sentences; first lock down the story experience and material selection.

The timeline has two hard constraints that cannot be downgraded: the original sound can only be cut in, cut out, or restored at reliable sentence ends/silence boundaries; narration must use complete paragraph-by-paragraph audio, and any clip mapping segment trimming, TTS tail trimming, or Jianying referencing longer pre-acceleration material is blocked. When the Agent receives interrupts_source_sentence / unsafe_clip_sentence_boundary / no_safe_fit / timeline_audio_mismatch , it should move the boundary, shorten the entire sentence, or delete that block, rather than adding a snatch override; unsafe_clip_sentence_boundary The accompanying nearest_safe gives the nearest safe boundary times before and after.

4.3 Multiple videos and material library

Multiple videos only support cut mode. The project brief will list stable source_id，clip_plan.json Every segment in must have its source filled in:

python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --target-duration 10m --work-dir work_dir_multi_ep

The second stage, like single video, generates the post-cut storyboard storyboard/edited_storyboard.*, taken from each source sources/<source_id>/frames/, and the top of the brief lists S1/S2… corresponding to source_id; sources restored from the material library have no extracted frames, these segments do not appear in the storyboard, use inspect clip-map to verify.

Optional file system material library:

python3 scripts/recap.py ep1.mp4 --material-library-dir .video-materials --save-materials
python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --material-library-dir .video-materials --use-materials

Material retrieval is just grep on JSON / MD / JSONL, for example grep -R "keyword" .video-materials. The current version does not copy original media, nor does it provide a database, vector, or semantic search.

The same root directory can also register reusable resources (BGM, sound effects, voice timbres, fonts, images), templates with versions and adoption records (subtitle styles, packaging layers, production references), and sample clips. For the format and scripts/library.py check|list|show read-only tools see references/resource-library.md. Use --project recap_project.json to bind the adopted subtitle styles, packaging, production references, voice timbres, and BGM to this run; after each full / cut composition work_dir/resource_lock.json record the resources actually used and their authorization status.

4.4 Continue generating the final video

After writing the required artifacts, repeat the same command:

python3 scripts/recap.py <video> --work-dir <work_dir>  # 可追加 --edit-mode cut / --no-burn-subtitles

The process will validate the hard inputs of the current stage (clip_plan.json / narration.json); the two creation plans are still working records used by the Agent and advisory reviewers, not rendering gates. Cut mode then generates edited_source.mp4, then synthesizes the narration and outputs recap_<name>.mp4。

Validation never rewrites the narration script (it only updates the measured overlaps_speech); in full mode, if the text cannot fit the time window, it is returned to the Agent as a over_budget error. When there is already an approved narration script, add --preserve-approved-text, and TTS also preserves the approved script as-is; if it cannot fit the time window, it fails, without shrinking the script or degrading to partial success.

4.5 Subtitles and cloned narration

If you want to fix the narration subtitles in the original film's subtitle area, first run at the repository root:

python3 tools/measure_subtitle.py <video>

Then pass in the measured --subtitle-y-top/--subtitle-y-bot. The coordinates are based on the display canvas after ffmpeg automatic rotation, the interval is half-open [top, bot), and a bottom-aligned ASS style is required; after explicit setting, this area uses a 60% opacity narration window mask by default.

Narration mode: if you need to clone a reference voice, use --voice-ref <audio>; it differs from dub mode.

4.6 Final viewing and delivery review

Scripts, cut-point detection, sample frames, and QC reports cannot replace viewing. Before preparing each round of delivery, you must checkthe final files actually to be delivered in this round, not old aliases, subtitle-free masters, or intermediate proxies:

  1. Play the short film once in full at normal speed, without editing while watching; first record the real viewing problems.
  2. Play about 0.5–1 second before and after each splice point, checking for flash frames, truncated dissolves from the original footage, motion jumps, and half-sentence original audio.
  3. Listen to the audio once in full on its own, checking whether the narration is broken into one sentence with a pause after each, whether the sound connects between scenes, and whether the key original audio is complete.
  4. Review the opening, the core emotional/performance points, and the ending separately to confirm that the entry timing, payoff hold, and closing all work.
  5. REVISION: separately verify that this round's modified items have actually changed and that frozen items have not changed unexpectedly; then perform mechanical checks such as decoding, duration, and audio-video specifications.

The scene score, brightness statistics, contact sheet, and automated QC are only responsible for locating candidate problems; the final judgment is based on real playback. Source judgment and handling rules for dense cut points are executed according to the editing skill. When a fix fails, return to the cut point, sound, or copy layer, and do not cover it up with more packaging.

For full/cut delivery, if you want the deterministic final check to affect the command exit status, explicitly pass --require-final-qc. Only when final_qc.json 's summary is ok: true and the integer blocker_count: 0 will completion be printed and success returned; missing, malformed, or blocker will retain the report and the rendered diagnostic media, but the command exits nonzero and does not print completion. The default remains report-only and non-blocking. This parameter does not support --edit-mode dub; when dub does not pass this parameter, the preparation and rendering behavior is unchanged.

At delivery, read final_qc.json 's metadata.warnings, and if there is content, tell the user exactly as is; you must not merely report “done.”subtitle_burn_degraded indicates that ffmpeg lacks libass and the default burn-in was downgraded: there are no subtitles in the final video, and the subtitles are in the same-named file next to the final video .srt. If the user wants burned-in subtitles, ask them to install ffmpeg with libass and rerun; when explicitly passing --burn-subtitles , missing libass will error out before the run starts. If the narration contains visual_overlays and ffmpeg lacks drawtext, the process stops before dubbing; follow the error to remove the overlay or switch ffmpeg and then continue the run.

4.7 Films that do not need narration

# 对当前整段输入直接合成；不隐式跑理解/ASR/TTS
python3 scripts/recap.py locked_picture.mp4 --work-dir source_work --audio-mode source-mix
# 剪辑计划仍按 cut 流程产生，剪完不再暂停等待 narration.json
python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --work-dir cut_work --audio-mode source-mix
# 只换包装时冻结当前整片 AAC；不允许同时加 BGM/TTS
python3 scripts/recap.py adopted.mp4 --work-dir packaging_work --audio-mode adopted-packet-copy

source-mix will still be mixed and re-encoded;cut + adopted-packet-copy freezes the audio of the intermediate film after editing, not the AAC packets of the original film. The current strict subtitle track only supports adopted mode; other subtitle sources have not thereby become precisely aligned. Switching sound modes requires a new working directory, and old TTS, QC, or automatically generated narration overlay text must not be mixed into this round's original-audio production. For details, see references/audio-routing.md。

5. English-to-Chinese original-voice replication mode

--edit-mode dub is an experimental feature: it translates an English video into Chinese and replaces the human voice with the cloned voice of the original speaker; it is not narration overlaid on top of ducked original audio. Use it only when the user explicitly requests English-to-Chinese dubbing.

dub sends the source video audio track in windows to MiMo ASR for transcription, and sends the captured approximately 10-second original speaker voice as a reference, together with each sentence's translation, to MiMo voiceclone. Therefore it must include --confirm-voice-rights: first confirm with the user that they have the right to use this video and audio and that the speaker consents to their voice being cloned; only after confirmation add this parameter; if it is missing, startup errors out immediately, without extracting audio or sending requests. This parameter is only used for dub; passing it in other modes will error.

python3 scripts/recap.py <video> --edit-mode dub --confirm-voice-rights --work-dir <work_dir>

The preparation stage transcribes the English, extracts a reference audio segment, and writes out dub_brief.md and dub_transcript.json. The Agent follows the translation requirements in dub_brief.md (faithful sentence by sentence, time window, speech rate) to write dub_script.json：

[{"start": 0.0, "end": 2.0, "zh": "中文译文"}]

After repeating the same command, it first runs a deterministic lint (dub_lint.json, aborting on blank lines, overlaps, or out-of-bounds), then outputs dub_<name>.mp4. Each sentence is cloned individually and pasted back onto the original timeline; local speed-up is applied only when it is about to overlap the next sentence. The current version supports only a single speaker and whole-track replacement, and does not separate background music.

6. Self-check and read-only dashboard

python3 scripts/recap.py --doctor

6.1 Read-only dashboard

python3 scripts/dashboard_server.py --root <目录> [--port 0] [--open]

Runs in the foreground and prints the local address (bound only to 127.0.0.1; the Agent puts it in the background at startup). Under --root it discovers resource libraries, projects, and runs by library.json、recap_project.json、 recap_run_manifest.json , and displays by stage the editing rhythm, narration, final cut and timeline, QC, and resource_lock.json. Strictly read-only: accepts only GET / HEAD, writes no files; the "copy for assistant" on the page copies only a single request, and changes are made back in the conversation.

7. Outputs and parameters

Main outputs:

  • recap_<video>.mp4: the final cut.
  • subtitles.srt / subtitles.ass: subtitles.
  • work_dir/: all intermediate artifacts, with the contract in references/data-schema.md。
  • work_dir/recap_story_plan.json / visual_audio_board.json: the Agent's creative intent and editing decisions.

The complete parameter list is subject to python3 scripts/recap.py --help .--style is free-text guidance passed to the Agent as-is, not a preset, enumeration, switch, or limited style category.

8. Capability boundaries

  • Semantic review is advisory by default and fails open; only when the caller explicitly enables strict commentary review will factual contradictions, incomplete sentences, or review unavailability block before TTS. The deterministic validation stage is always responsible for hard validation.
  • For backfilling promotional titles, on-screen text, or external copy, see video-script 's references/promotional-copy.md.