SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsvideo-assemble

video-assemble

Design & creation

View repository

Updated 10-16 05:04

About this skill

1. Positioning

This skill handles final synthesis:

  1. Place each narration audio segment onto the video timeline.
  2. Within narration windows, use a fixed envelope to duck the original audio (one volume level each for when it covers original dialogue and when it falls in quiet passages), and restore the original audio in the gaps.
  3. Generate based on narration positions subtitles.srt; by default also generate and burn in subtitles.ass，--no-burn-subtitles can be disabled. When not burning in (disabled or degraded),subtitles.srt copy next to the final cut, named recap_<stem>.srt; when burning in, delete the old external subtitle with the same name.
  4. Optionally normalize the final loudness to the target LUFS: two-pass loudnorm, using only a single constant gain, with no dynamic compression; when the mix's true peak cannot accommodate such a large gain, first pass through a 4x oversampled true-peak limiter (cutting at most LOUDNESS_LIMITER_MAX_DB, default 6 dB), and only lower the target loudness for the excess. The true-peak target is for the delivered AAC file: AAC encoding makes the true peak 0.3–1.4 dB higher than the PCM mix, so the first render aims for 0.5 dB below TP, and after rendering it is decoded and measured on the final cut; if it still exceeds -1 dBTP, lower by an additional 0.1 dB based on the excess and re-encode only the audio (video stream copied), at most twice; if it still exceeds, block with delivered_true_peak_over_target The mode actually used by ffmpeg, the limiter amount, and the measured final-cut values are recorded in assembly_qc.json of loudness_mode and loudnorm_final_pass（delivered）。
  5. The final cut does not carry the original film's container metadata (title、comment and other tags and chapters).

2. Sound finalization contract

The synthesis stage only implements creative decisions; it does not invent decisions out of thin air. Before writing narration positions, the Agent has already, in visual_audio_board.json for each beat specified audio_owner：

  • original_dialogue
  • action_sound
  • ambience / music
  • silence
  • narration

Therefore, narration gaps are active choices, not blanks that must be filled. Do not add generic BGM for the sake of "fuller" sound, duck dialogue that must be heard, or eliminate meaningful silence.

The current renderer does not parse visual_audio_board.json; the Agent implements these decisions through narration timing,overlaps_speech, original-audio blank space, and existing mixing parameters.

3. Input contract

  • <video>: source video; in cut mode, it is edited_source.mp4。
  • work_dir/tts_meta.json: default narration required in mode; written by the dubbing stage {segments: [...]}. Each segment contains audio_path, time,pause_after_ms、overlaps_speech and the positions used for mixing/subtitles. Explicit source-mix / adopted-packet-copy mode does not read it.
  • Adopted dubbing uses explicit --tts-meta and --narration-adoption: the latter independently confirms the text, requested engine/voice, and speed policy by the caller, and cannot be automatically "approved" from the metadata to be consumed. For the complete format and recording boundaries, see references/narration-adoption.md。
  • The adopted complete sound bed and per-segment dubbing can be passed again --audio-mix-adoption; for the strict format, 48 kHz channel matrix, and dual binding transaction, see references/explicit-audio-mix.md。

The following scripts/... are all relative to this skill directory. If the executor is started from the repository root, add this skill's absolute directory to the script path.

4. Run commands

python3 scripts/assemble.py <video> --work-dir <work_dir> \
  [--audio-mode narration|source-mix|adopted-packet-copy] [--audio-stream-index <N>] \
  [--tts-meta <tts_meta.json> --narration-adoption <narration_adoption.json>] \
  [--audio-mix-adoption <audio_mix_adoption.json>] \
  [--recap-stem <name>] [--output-dir <dir>] [--no-burn-subtitles] \
  [--subtitle-y-top <inclusive-y> --subtitle-y-bot <exclusive-y>] \
  [--source-video <orig.mp4>] [--export-jianying [--jianying-out <dir>]]

5. Output contract

  • recap_<stem>.mp4: stable final output alias; overwritten and updated on each run.
  • work_dir/output.mp4: final cut within the working directory.
  • subtitles.srt: narration subtitles; when burning in, there is also subtitles.ass。
  • timeline.json: Backend-agnostic multitrack model, including video, original audio, narration, BGM, subtitles, and ducking automation.
  • _placed_*.wav: The complete segment-by-segment narration PCM actually written into the main mix; the timeline and Jianying only reference these files.
  • narration_input_binding.json: The consumption record of narration input, conversion, actual placement, narration main track, and final audio track (paths, PCM parameters, packet counts). Distinguishes between the two states of not adopted and bound-adopted decision; not equivalent to voice identification or audition.
  • audio_mix_binding.json: The record of the picture clock, base track, 48 kHz voice placement, premaster, fixed master gain, final PCM/AAC facts, and narration binding paths for the explicit complete audio branch consumption.
  • assembly_manifest.json: Input source, cut source identification (path, size, mtime), render settings, and final output path.
  • assembly_qc.json: Release gates for narration integrity, original-audio sentence-end handoff, timeline material duration, and delivery quality.
  • Jianying draft directory: generated only when --export-jianying , including draft_content.json、draft_info.json and draft_meta_info.json。

6. Composition rules

  • For the processing and freezing semantics of audio modes, see references/audio-modes.md. The default remains narration; the other two modes must be explicitly selected.
  • --audio-mix-adoption is used only together with explicit --tts-meta、--narration-adoption ; it retains the narration mode name, but skips the old speed/fit, original-audio handoff, ambient BGM, duck, loudnorm, and limiter.
  • Audio is mixed by track: original audio, optional BGM, and narration are each independent.
  • The first block of a paragraph (with more than 1.6 seconds of author whitespace from the previous block) is placed from the written start , but not earlier than the previous block's actual end plus its pause_after_ms; if the previous block times out, it pushes it later; subsequent blocks within the paragraph follow immediately after the previous block's actual end (0.35 second gap), at most 1.2 seconds earlier than the written start . This advance occurs after narration validation, so the advanced segment cannot enter the original-audio dialogue: when there is dialogue, it can advance at most to the end of the last dialogue segment. Blocks that actually advanced record source_entry_status in paragraph_tightened, and record the written written_start in start。
  • Narration does not undergo any tolerance tail trimming; if it still does not fit after gentle speeding up, then no_safe_fit. Each segment's _placed_*.wav must be equal in length to or shorter than the serialized timeline interval, otherwise timeline_audio_mismatch blocks it.
  • The v1 contract with adopted voiceover supports only original speed and prohibits in-segment speed fitting; the environment default of 1.15x speed or an old cache must not override it. If it does not fit, revise the arrangement; do not trim the tail. Strict runs use a new working directory and new output path; when inputs/actual mix sources change or QC fails, do not publish a candidate film. The old entry point without an adopted file is still compatibility mode and does not automatically obtain equivalent evidence.
  • The original audio is ducked for at most another 3 seconds after the narration ends, waiting for the next sentence-end anchor within this period: duck to its pause_start, fade in only within the measured pause, and return to full at source_restore_at (when the anchor is unverified , the state is recorded as sentence_boundary_unverified). If there is no anchor within 3 seconds, when less than 3 seconds remain to the timeline end, duck to the end (held_to_timeline_end), otherwise return to full directly at the end of the narration (bounded_release), and do not hold the original-audio dialogue down for a long time for a distant anchor.assembly_qc.json 's summary.max_source_duck_hold_seconds records the longest ducking continuation.
  • --export-jianying / EXPORT_JIANYING=1 can export timeline.json as an editable draft. Cut mode should pass --source-video <orig>, so that the draft references the real original-film intervals.
  • Jianying export by default copies video, audio, and images to Resources/local/{video,audio,image}, keeping the draft relocatable;--jianying-no-bundle-media only suitable when the original path is always accessible.
  • Overlapping overlays are split onto numbered tracks; a non-empty target directory is not overwritten, but a numbered sibling directory is created instead.
  • Export only accepts schema_version: 2 of timeline.json, mapping only video, audio, subtitles, and image overlays (scale / position); handwritten extension fields such as speed changes, transitions, masks, rich text, and effect tracks are explicitly rejected.
  • Jianying drafts reference unbaked source video, so hardcoded subtitles in the original footage remain; mask them separately in Jianying if necessary.
  • Subtitle appearance can be controlled with SUBTITLE_FONT_SIZE、SUBTITLE_MARGIN_V、SUBTITLE_MAX_CHARS , etc.
  • SUBTITLE_Y_TOP/BOT places the ASS baseline into the measured subtitle region of the original footage, with coordinates as a half-open [top, bot)on the display canvas, accepting only square or near-square pixels (SAR differing from 1:1 by no more than 2%; unlabeled 0:1 is treated as square); under the explicit mask strategy, the default SUBTITLE_MASK_OPACITY=0.6，SOURCE_SUBTITLE_MASK_TIMING=narration。
  • Original audio returns to IDLE_ORIG_VOLUMEduring narration gaps, and narration is ducked to SPEECH_DUCKING_VOLUME；DUCK_FADE_SECONDS controls the transition. You can also configure DUCK_BRIDGE_SECONDS、ZONE_DUCKING_VOLUME、FINAL_LOUDNORM、TARGET_LUFS and LOUDNESS_LIMITER_MAX_DB。
  • You can specify BGM via BGM_PATH ; it loops to the final video length and is mixed according to BGM_VOLUME / BGM_DUCKING_VOLUME . Do not set generic BGM without a creative basis.
  • Burning subtitles requires ffmpeg with subtitles / libass, and the compositing stage pre-checks before rendering. When burning is explicitly required (--burn-subtitles or environment variable BURN_SUBTITLES), missing libass fails directly; when it is merely enabled by default, it degrades: no burning, delivering external .srt(the 「」 original dialogue in the blank space is written in as usual only when there is user_subtitles.* ; dialogue originally triggered by the mask is turned off together with the mask, and the original hardcoded subtitles remain visible),visual_qc.json of warnings and assembly_manifest.json of warnings records a subtitle_burn_degraded，subtitles.burn_degraded_reason writing the reason. After degradation, the mask is turned off as before (mask.trigger is burn_subtitles_degraded）。
  • visual_overlays.json text overlays use ffmpeg drawtext(libfreetype). When drawtext is missing, compositing fails before rendering; overlays are content explicitly added when writing the script and will not be silently dropped.
  • The final video picture is always H.264 8-bit 4:2:0 with +faststart: stream copy is used only when no filters are needed, the source picture is already H.264 8-bit 4:2:0 (yuv420p, or full-range yuvj420p), and the width and height are even, delivering it as-is; otherwise it is re-encoded to yuv420p. Color tagging: when the source is untagged or already BT.709, it is tagged as BT.709 (-colorspace/-color_primaries/-color_trc bt709); other declared color spaces are preserved as-is; full-range (pc) sources preserve pc, and the rest are tagged tv. YUV sources only have their tags changed, without pixel conversion; RGB sources (ffprobe reports gbr, such as MOV wrapped in PNG/QuickTime RLE) have no YUV matrix to preserve and are converted as BT.709 limited. The result is recorded in assembly_qc.json of delivery_qc.color_tags(RGB sources have one extra from_rgb: true）。
  • Dialogue subtitles in the original-audio blank space preferentially read the Agent-proofread original_subtitles.json; otherwise ASR is mapped conservatively. Original dialogue is burned only when the mask covers the blank space or the user subtitles explicitly require replacement, and is distinguished from narration using 「」 .

Prepare the sound according to the original footage intervals, rather than lowering the entire old final video

When there are already multiple original-audio trade-off and independent BGM decisions, first use the independent of references/source-score.md independent source_score.py Reconstruct the original audio track, music track, and the sum of both from the original film's audio stream according to precise frame intervals; it outputs only the sound base track and a source receipt. To synthesize with the dubbing already adopted segment by segment, the caller must then provide references/explicit-audio-mix.md strict adoption; do not stuff the base track into the old entry point for automatic ducking, and do not take the entire audio from a finished film containing old narration and pass it off as clean original audio.

7. Subtitles and optional packaging

First lock the picture, cut points, narration, and mix, then invest in subtitle animation or border packaging; subtitle styling must not mask problems with the narrative, cut points, or sound. For ordinary delivery, prefer the existing ASS path; for full-film static frames, title bars, and corner marks, use packaging_layers.json static overlay; only when the user needs per-cue layout, animation, or transparent layers should a project-level code renderer be used, with that project compositing the transparent layer onto the locked master itself. For the packaging order and sample-frame spot-check list, see references/packaging.md。

8. Capability boundaries

  • Does not generate narration text, does not synthesize TTS, and does not re-transcribe the video.
  • Subtitle burn-in is enabled by default; when disabled, the subtitle drawing area will not be re-encoded.

For the independent contract of explicitly outputting an axis subtitle track, the complete replacement semantics, and the current boundaries, see references/subtitle-track.md。