SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsvideo-assemble

video-assemble

Design & creation

View repository

Updated 10-16 09:30

About this skill

1. Positioning

This skill handles final assembly:

  1. Place each narration audio segment onto the video timeline.
  2. Within narration windows, use a fixed envelope to duck the original sound (one volume level when covering original dialogue and another when falling on quiet passages), and restore the original sound in the gaps.
  3. Generate based on narration positions subtitles.srt; by default also generate and burn in subtitles.ass，--no-burn-subtitles can be disabled. When not burning in (disabled or degraded),subtitles.srt copy next to the cut, named recap_<stem>.srt; when burning in, delete the old external subtitle with the same name.
  4. Optionally normalize the final loudness to the target LUFS: two-pass loudnorm, using only a constant gain, with no dynamic compression; when the mix's true peak cannot accommodate such a large gain, first pass through a 4x oversampled true-peak limiter (cutting at most LOUDNESS_LIMITER_MAX_DB, default 6 dB), and only lower the target loudness for the excess. The true-peak target is for the delivered AAC file: AAC encoding can make the true peak 0.3–1.4 dB higher than the PCM mix, so the first render aims for 0.5 dB below TP, and after rendering it is decoded and measured; if it still exceeds -1 dBTP, lower by another 0.1 dB based on the excess and re-encode only the audio (video stream copied), at most twice; if it still exceeds, block with delivered_true_peak_over_target . The mode actually used by ffmpeg, the limiter amount, and the measured values of the final cut are recorded in assembly_qc.json in loudness_mode and loudnorm_final_pass（delivered）。
  5. The final cut does not carry the source's container metadata (title、comment and other tags and chapters).

2. Sound finishing contract

The assembly stage only implements creative decisions, it does not create decisions out of thin air. Before writing narration positions, the Agent has already, in visual_audio_board.json for each beat, specified audio_owner：

  • original_dialogue
  • action_sound
  • ambience / music
  • silence
  • narration

Therefore, narration gaps are active choices, not blanks that must be filled. Do not add generic BGM to make it "fuller," suppress lines that must be heard, or eliminate meaningful silence.

The current renderer does not parse visual_audio_board.json; the Agent implements these decisions through narration timing,overlaps_speech, original-sound pauses, and existing mixing parameters.

3. Input contract

  • <video>: source video; in cut mode it is edited_source.mp4。
  • work_dir/tts_meta.json: default narration required for mode; written by the dubbing stage {segments: [...]}. Each segment contains audio_path, time,pause_after_ms、overlaps_speech and positions used for mixing/subtitles. Explicit source-mix / adopted-packet-copy mode does not read it.
  • Adopted dubbing uses explicit --tts-meta and --narration-adoption: the latter is independently confirmed by the caller regarding text, requested engine/voice, and speed policy, and cannot be automatically "approved" from metadata to be consumed. For the complete format and record boundaries, see references/narration-adoption.md。
  • The adopted complete sound bed and per-segment dubbing can additionally pass --audio-mix-adoption; for the strict format, 48 kHz channel matrix, and dual binding transaction, see references/explicit-audio-mix.md。

Below, the scripts/... are all relative to this skill directory. If the executor is started from the repository root, prepend the skill's absolute directory to the script path.

4. Run command

python3 scripts/assemble.py <video> --work-dir <work_dir> \
  [--audio-mode narration|source-mix|adopted-packet-copy] [--audio-stream-index <N>] \
  [--tts-meta <tts_meta.json> --narration-adoption <narration_adoption.json>] \
  [--audio-mix-adoption <audio_mix_adoption.json>] \
  [--recap-stem <name>] [--output-dir <dir>] [--no-burn-subtitles] \
  [--subtitle-y-top <inclusive-y> --subtitle-y-bot <exclusive-y>] \
  [--source-video <orig.mp4>] [--export-jianying [--jianying-out <dir>]]

5. Output contract

  • recap_<stem>.mp4: stable final output alias; overwritten and updated on each run.
  • work_dir/output.mp4: the cut within the working directory.
  • subtitles.srt: narration subtitles; when burning in, there is also subtitles.ass。
  • timeline.json: Backend-agnostic multi-track model, including video, original sound, narration, BGM, subtitles, and ducking automation.
  • _placed_*.wav: The complete segment-by-segment narration PCM actually written to the main mix; the timeline and CapCut only reference these files.
  • narration_input_binding.json: Consumption record of narration input, conversion, actual placement, narration master track, and final audio track (paths, PCM parameters, packet counts). Distinguishes the two states of not adopted and bound-adopted decisions; it is not equal to voice identification or listening review.
  • audio_mix_binding.json: Record of the picture clock, base track, 48 kHz dubbing placement, premaster, fixed master gain, final PCM/AAC facts, and narration binding paths explicitly consuming the complete audio branch.
  • assembly_manifest.json: Input source, cut source identifiers (path, size, mtime), render settings, and final output path.
  • assembly_qc.json: Release gates for narration integrity, original-sound sentence-end handoff, timeline asset duration, and delivery quality.
  • CapCut draft directory: generated only when --export-jianying , including draft_content.json、draft_info.json and draft_meta_info.json。

6. Composition rules

  • For the processing and freezing semantics of audio modes, see references/audio-modes.md. The default remains narration; the other two modes must be explicitly selected.
  • --audio-mix-adoption is used only together with explicit --tts-meta、--narration-adoption ; it retains the narration mode name, but skips the old speed/fitting, original-sound handoff, ambient BGM, duck, loudnorm, and limiter.
  • Audio is mixed by track: original sound, optional BGM, and narration are each independent.
  • For a segment's first block (with more than 1.6 seconds of author-intended blank space from the previous block), place from the written start , but no earlier than the previous block's actual end plus its pause_after_ms; if the previous block runs overtime, this pushes it later; subsequent blocks within the segment follow immediately after the previous block's actual end (0.35 seconds apart), at most 1.2 seconds earlier than the written start . This advancement happens after narration validation, so the advanced segment cannot enter the original-sound dialogue: when there is dialogue, it can be advanced at most to the end of the last dialogue segment. Blocks that were actually advanced record source_entry_status in paragraph_tightened, and record the written written_start in start。
  • Narration does not undergo any tolerance-based tail trimming; if it still does not fit after gentle speed-up, then no_safe_fit. Each segment's _placed_*.wav must be equal in length to or shorter than the serialized timeline interval, otherwise timeline_audio_mismatch blocks it.
  • The v1 contract for adopted dubbing supports only original speed and forbids in-segment speed fitting; the environment default 1.15x speed or old cache must not override it. If it does not fit, revise the arrangement; do not trim the tail. Strict runs use a new working directory and new output path; when inputs/actual mix sources change or QC fails, do not release a candidate final cut. Old entry points without adopted files are still compatibility mode and do not automatically gain equivalent evidence.
  • The original sound is ducked for at most another 3 seconds after the narration ends, waiting for the next sentence-end anchor within this period: duck to its pause_start, ramp up only within the measured pause, and return to full at source_restore_at (when the anchor is unverified , the state is recorded as sentence_boundary_unverified). When there is no anchor within 3 seconds, if less than 3 seconds remain to the timeline end, duck to the end (held_to_timeline_end), otherwise return to full directly at the narration end (bounded_release), and do not hold down the original-sound dialogue for a long time for a distant anchor.assembly_qc.json 's summary.max_source_duck_hold_seconds records the longest ducking continuation.
  • --export-jianying / EXPORT_JIANYING=1 can export timeline.json as an editable draft. Cut mode should pass --source-video <orig>, so the draft references the real original-film intervals.
  • CapCut export by default copies video, audio, and images to Resources/local/{video,audio,image}, keeping the draft relocatable;--jianying-no-bundle-media only suitable when the original path is always accessible.
  • Overlapping overlays are split into numbered tracks; a non-empty target directory is not overwritten, but instead a numbered sibling directory is created.
  • Export only accepts schema_version: 2 of timeline.json, mapping only video, audio, subtitles and image overlays (scale / position); hand-written extension fields such as speed changes, transitions, masks, rich text and effect tracks are explicitly rejected.
  • The Jianying draft references the unburned source video, so hard subtitles in the original footage remain; mask them separately in Jianying if necessary.
  • Subtitle appearance can be controlled with SUBTITLE_FONT_SIZE、SUBTITLE_MARGIN_V、SUBTITLE_MAX_CHARS etc.
  • SUBTITLE_Y_TOP/BOT places the ASS baseline into the measured subtitle area of the original footage, with coordinates as a half-open [top, bot)on the display canvas, accepting only square or near-square pixels (SAR within 2% of 1:1; unlabeled 0:1 is treated as square); under the explicit mask policy the default SUBTITLE_MASK_OPACITY=0.6，SOURCE_SUBTITLE_MASK_TIMING=narration。
  • The original audio returns to IDLE_ORIG_VOLUMEin narration gaps, and narration is ducked to SPEECH_DUCKING_VOLUME；DUCK_FADE_SECONDS controls the transition. DUCK_BRIDGE_SECONDS、ZONE_DUCKING_VOLUME、FINAL_LOUDNORM、TARGET_LUFS and LOUDNESS_LIMITER_MAX_DB。
  • BGM can be specified via BGM_PATH ; it loops to the finished length and is mixed according to BGM_VOLUME / BGM_DUCKING_VOLUME . Do not set generic BGM without a creative basis.
  • Burning subtitles requires ffmpeg with subtitles / libass, prechecked before rendering in the compositing stage. When burning is explicitly required (--burn-subtitles or environment variable BURN_SUBTITLES), missing libass fails directly; when it is merely enabled by default, it degrades: no burn-in, delivering external .srt(the 「」 original dialogue in the blank space is still written in as usual only when there is user_subtitles.* ; dialogue originally triggered by the mask is turned off together with the mask, and the original hard subtitles are visible),visual_qc.json of warnings and assembly_manifest.json of warnings records a subtitle_burn_degraded，subtitles.burn_degraded_reason writing the reason. After degradation the mask is still turned off (mask.trigger is burn_subtitles_degraded）。
  • visual_overlays.json text overlays use ffmpeg drawtext(libfreetype). When drawtext is missing, compositing fails before rendering; overlays are content explicitly added when writing the script and will not be silently dropped.
  • The finished picture is always H.264 8-bit 4:2:0 with +faststart: stream copy and deliver as-is only when no filters are needed and the source picture is already H.264 8-bit 4:2:0 (yuv420p, or full-range yuvj420p) with even width and height; otherwise re-encode to yuv420p. Color tagging: if the source is untagged or already BT.709, tag as BT.709 (-colorspace/-color_primaries/-color_trc bt709); other declared color spaces are preserved as-is; full-range (pc) sources keep pc, the rest are tagged tv. YUV sources only have their tags changed, without pixel conversion; RGB sources (reported by ffprobe as gbr, such as PNG/QuickTime RLE wrapped MOV) have no YUV matrix to preserve and are converted as BT.709 limited. The result is recorded in assembly_qc.json of delivery_qc.color_tags(RGB sources have one extra from_rgb: true）。
  • The dialogue subtitles in the original-audio blank space preferentially read the Agent-proofread original_subtitles.json; otherwise the ASR is mapped conservatively. Original dialogue is burned only when the mask covers the blank space or the user subtitles explicitly require replacement, and is distinguished from narration with 「」 .

Prepare the sound according to the original intervals, rather than lowering the old finished video as a whole

When there are already multiple original-audio choices and independent BGM decisions, first use references/source-score.md of the independent source_score.py Reconstruct the original sound track, music track, and the sum of both from the original audio stream by precise frame intervals; it outputs only the sound base track and source receipt. To synthesize with the already-adopted dubbing segment by segment, the caller then provides references/explicit-audio-mix.md strict adoption; do not stuff the base track into the old entry for automatic ducking, and do not take the entire sound from a finished video containing old narration and pass it off as clean original audio.

7. Subtitles and optional packaging

Lock the picture, cut points, narration, and mix first, then invest in subtitle animation or frame packaging; subtitle styling must not mask problems in narrative, cut points, or sound. For ordinary delivery, preferentially use the existing ASS path; for frame wrapping, title bars, and corner marks that stay fixed across the whole video, use packaging_layers.json static overlay; only when the user needs per-cue layout, animation, or transparent layers, use a project-level code renderer, and have that project itself composite the transparent layer onto the locked master. For the packaging order and sample-frame spot-check list, see references/packaging.md。

8. Capability boundaries

  • Does not generate narration text, does not synthesize TTS, does not re-transcribe the video.
  • Subtitle burn-in is enabled by default; when disabled, the subtitle drawing area will not be re-encoded.

For the independent contract of the explicit output-axis subtitle track, the full replacement semantics, and the current boundaries, see references/subtitle-track.md。