SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsvideo-voiceover

video-voiceover

Design & creation

View repository

Updated 10-16 06:49

About this skill

1. Positioning

This skill reads a timestamped narration script, generates separate audio for each segment, adapts the speech to the corresponding time window, and then records the placement metadata needed for downstream synthesis. The default engine is MiMo TTS (mimo-v2.5-tts); you may also explicitly select Fish Audio (default model s2.1-pro-free）。

2. Remote services and data egress

This skill only connects to the network on the paths selected in the table below, and the content sent on each path is as follows. Actual read text means the text of the narration.json segment after removing formatting and stage directions.

Path Service and credentials Content sent
Default narration TTS (--tts-provider auto|mimo-tts） MiMo chat endpoint <MIMO_TTS_API_URL 或 MIMO_API_URL>/chat/completions，MIMO_TTS_API_KEY or MIMO_API_KEY, model mimo-v2.5-tts the actual read text of each segment, a natural-language tone/speed instruction (computed from the segment's emotion and time window), the built-in voice name (by default 冰糖）；auto when the MiMo key is missing and FISH_API_KEY is set, switch to the Fish Audio row
Narration voice cloning (--voice-ref <audio> / VOICE_REF） same as above, model mimo-v2.5-tts-voiceclone the above content, plus the base64 of the reference audio (converted to 24 kHz mono WAV, max 30 seconds), included with every segment request
Fish Audio（--tts-provider fish-audio） FISH_TTS_API_URL(default https://api.fish.audio/v1/tts），FISH_API_KEY the actual read text of each segment, numeric speaking rate, voice ID FISH_TTS_REFERENCE_ID; no local audio is sent
Self-hosted IndexTTS (--tts-provider index-tts） an HTTP(S) service deployed by the user and specified by INDEX_TTS_ENDPOINT Specified HTTP(S) service {"voice": INDEX_TTS_VOICE, "text": 实读文本}; no local audio is sent
Experimental dub (see §8) MiMo ASR（MIMO_API_URL，MIMO_API_KEY，mimo-v2.5-asr) and MiMo voiceclone (TTS endpoint and credentials,mimo-v2.5-tts-voiceclone） the entire source video audio track is sent to ASR in 6-second windows; each Chinese translation is sent to voiceclone together with about 10 seconds of the original speaker's voice clipped from the source audio (clone reference)

--voice-ref and dub both send a real person's voice to MiMo for cloning: use only when the user has the right to use that audio and the voice owner consents to being cloned.

3. Environment requirements

export MIMO_API_KEY=***  # 也可使用仅供 TTS 的 MIMO_TTS_API_KEY

# 或改用 Fish Audio TTS
export TTS_PROVIDER=fish-audio
export FISH_API_KEY=***
export FISH_TTS_REFERENCE_ID=<voice-model-id>  # 可选；覆盖内置“娱乐扒妹”音色

# 或显式选择自托管 index-tts 端点，配置见 references/index-tts.md
export TTS_PROVIDER=index-tts

The following scripts/... are all relative to this skill's directory. If the executor starts from the repository root, prepend this skill's absolute directory to the script path.

4. Input contract

The default input is work_dir/narration.json. Each segment must contain start、end and narration, and optional fields include pause_after_ms and overlaps_speech. Times uniformly represent the seconds on the output timeline where the audio is ultimately placedoutput timeline seconds。

The cut flow trims first and then dubs:narration.json itself is written according to the output time of the trimmed final cut, and there is no separate mapping script.

5. Run command

python3 scripts/voiceover.py --work-dir <work_dir> --narration <narration.json> \
  [--tts-provider auto|mimo-tts|fish-audio|index-tts] \
  [--mimo-voice 冰糖 | --voice-ref <reference-audio>] \
  [--preserve-approved-text] [--allow-partial-tts]

When run alone and --narration is omitted, the default is to read work_dir/narration.json；--narration is only used to specify a script in the same format at another path.

6. Output contract

  • tts_segments/*.wav: each narration segment corresponds to one audio file.
  • tts_meta.json: contains segments、engine、voice(the provider, model, voice, or reference audio actually used) and narration. Each segment records audio_path, time, pause_after_ms and placement fields.
  • Clean run writes partial: false and failures: []。
  • When using --allow-partial-tts to skip failed segments, writes partial: true and failures: [{index,start,end,text,error}], keeping missing speech visible.
  • --preserve-approved-text is an explicit approved-script protection policy. Each segment retains the original authored_text evidence; the spoken_text actually read by TTS only undergoes the existing format/stage-direction cleanup. If the complete speech exceeds the time window and the cumulative speech-rate budget, the command fails and reports the segment index, original script, actually-read text, speech duration, and window evidence, without writing a successful tts_meta.json. In strict mode, any required segment failure (including provider failure) cannot be --allow-partial-tts downgraded to a deliverable partial success; the exception record is marked as required: true with the policy ID. Author segments that contain only cleanup markers such as [停顿] and have no actually-read text after cleanup are also required-segment errors.

7. Run rules

  • Segmented audio is cached by content in tts_segments/cache/: the key is the actually-read text, the tone request actually sent to the provider (for MiMo it is that natural-language instruction, and the speech rate changes the wording only at ≥+6% or ≤-3%; for Fish Audio it is the numeric speed; index-tts has no segment-level control) and the TTS settings, excluding the segment index and time window; therefore, when a segment position change makes the nominal speech rate go from +5% to -2%, MiMo does not re-synthesize; the cached WAV is invalidated as soon as its size/mtime_ns changes.narr_NNN.wav is a hard link to the cache (or a copy when unsupported), andtts_meta.json still references it as before. After deleting, inserting, or moving a segment, only regenerate the segments whose text or request sent to the provider changed (the nominal speech rate changes with the position of the first segment and the last two segments); the old version's narr_NNN.wav.cache.json is no longer read.
  • The approved-script protection policy is part of the cache settings: strict mode adds the policy and original script to the cache key, and does not hit the default policy (report-over-budget-v2, which is not included in the key); the old version's per-segment cache (including auto-shortened audio) is no longer read; only caches under the same strict policy with a spoken_text complete match and a non-empty WAV can be reused offline; when reused, they are still checked against the current time window, and if they do not fit, they still fail.
  • Before synthesis in this round, the strict CLI archives the old tts_meta.json by timestamp to tts_meta.history/, so on failure the current path will not continue to masquerade as this round's success; successful metadata is atomically replaced via a temporary file in the same directory.
  • auto preferentially uses the configured MiMo; when the MiMo key is missing and FISH_API_KEY is set, Fish Audio is used; when a reproducible provider choice is needed, explicitly pass --tts-provider。
  • The self-hosted index-tts endpoint can only be explicitly selected by --tts-provider index-tts or TTS_PROVIDER=index-tts , andauto is never chosen as a fallback. For the protocol, request body, receipt semantics, and cache invalidation rules, see references/index-tts.md。
  • Fish Audio directly requests WAV; by default it uses the "娱乐扒妹" voice (5653cea4ac83480aaf2bf45406556185），FISH_TTS_REFERENCE_ID can override it. The cache is regenerated when the model, voice ID, API URL, normalization settings, or content-derived speech-rate change (Fish does not receive pitch and emotion, so changes to them do not trigger regeneration). The current free model has no SLA and is subject to Fair Use and the official free-period limits.
  • --voice-ref is used only for full/cut commentary cloning; switch to mimo-v2.5-tts-voiceclone. Lazily normalize once only when new synthesis is truly needed; changes to the reference audio path,size/mtime_ns , or the preprocessed version invalidate the old cache. Use only after obtaining authorization; the reference audio is sent to MiMo.
  • The dub voiceclone original WAV is also cached by model, prompt, script, and reference audio; size/mtime_ns a matching rerun no longer re-requests or re-bills, dub_manifest.json and each line is logged tts_cache=hit|miss。
  • When the synthesized segment audio is longer than the upper limit of reading the full text at TTS_MIN_SPEECH_RATE(default 2.5 characters/second, English counted as 1.5 characters per word) plus pauses and leading/trailing silence, it is treated as TTS hallucination (inventing an extra passage after finishing the script), retried as a failure, not cached, and not handed to assemble; if retries are exhausted, that segment fails, the error states the duration and the upper limit, and the last rejected audio is kept in tts_segments/narr_NNN.rejected.wav for listening. Digits (half-width/full-width) each count as 1 character,% counts as 3 characters (percent). dub's voiceclone lines go through the same check and retry (rejected ones are kept in dub_tts/line_NNN_raw.rejected.wav). Such WAVs under older-version caches are no longer reused on rerun and will be re-synthesized. Set to 0 to turn off this check.
  • TTS_WORKERS、TTS_TIMEOUT、TTS_RETRIES、ALLOW_PARTIAL_TTS is used to adjust concurrency, timeouts, retries, and partial-success policy.

8. Experimental dub dubbing (English to Chinese, cloned original voice)

dub.py is an experimental feature: it translates the English original audio into Chinese and replaces the vocals across the whole track with the original speaker's cloned timbre. It is an independent path from the narration dubbing above,voiceover.py never calls it.

  • Trigger: only when the user explicitly requests English-to-Chinese dubbing, run by the orchestration entry point with --edit-mode dub --confirm-voice-rights ; the orchestration entry point passes the confirmation parameters as-is to dub.py 's preparation and rendering stages. There is no separate manual stage.
  • Confirmation gate：dub.py : both stages of --confirm-voice-rights; when missing, exit before extracting audio and sending any request, and explain what will be sent externally. The orchestration entry point likewise rejects runs missing this parameter in dub mode, and rejects this parameter in other modes.
  • Externally sent content: the preparation stage sends the source video audio track in 6-second windows to MiMo ASR (mimo-v2.5-asr) to transcribe English; the rendering stage sends each Chinese translated sentence together with about 10 seconds of the original speaker's voice captured starting from the 2nd second of the source audio (dub_reference.wav) to MiMo voiceclone (mimo-v2.5-tts-voiceclone）。
  • Rights and consent: use only for videos and audio the user has the right to use, and the speaker whose voice is cloned has consented. The Agent must first confirm these two points with the user; only after the user confirms can --confirm-voice-rightsbe added; when unable to confirm, do not run dub, and do not use it to impersonate others speaking.
  • Determinism gate: the rendering stage writes dub_lint.jsonbefore voice cloning; blank lines, overlapping, or out-of-range translations abort immediately, and no voiceclone request is sent.
  • Local artifacts：dub_source.wav、dub_transcript.json、dub_brief.md、dub_reference.wav, the Agent-written dub_script.json、dub_lint.json、dub_tts/、dub_manifest.json and dub_<name>.mp4, are all written only in work_dir。

9. Capability boundaries

  • When over window, keep the script and log it; if assemble's bounded speed-up cannot fit it, block before rendering with no_safe_fit . For approved scripts add --preserve-approved-text; if over window, fail at the TTS stage.
  • No mixing, no lowering of the original audio, no subtitle rendering.
  • Does not analyze the video, nor choose time points; it only dubs the established segments in the input script.
  • Neither the Fish Audio nor the IndexTTS path accepts a local --voice-ref; the former uses the already-created FISH_TTS_REFERENCE_ID to select the timbre.