SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsvideo-voiceover

video-voiceover

Office & productivity

View repository

Updated 10-16 09:30

About this skill

1. Positioning

This skill reads a timestamped narration script, generates an independent audio file for each segment, and adapts the speech to the corresponding time window, then records the placement metadata needed for downstream composition. The default engine is MiMo TTS (mimo-v2.5-tts); you can also explicitly choose Fish Audio (default model s2.1-pro-free）。

2. Remote services and data egress

This skill only connects to the network on the paths selected in the table below, and the content sent on each path is as follows. Spoken text refers to the narration.json text of a segment after removing formatting and stage directions.

Path Service and credentials Content sent
Default narration TTS (--tts-provider auto|mimo-tts） MiMo chat interface <MIMO_TTS_API_URL 或 MIMO_API_URL>/chat/completions，MIMO_TTS_API_KEY or MIMO_API_KEY, model mimo-v2.5-tts the spoken text of each segment, a natural-language tone/speaking-rate instruction (calculated from the segment's emotion and time window), and the built-in voice name (by default 冰糖）；auto when the MiMo key is missing and FISH_API_KEY is set, switch to the Fish Audio row
Narration voice cloning (--voice-ref <audio> / VOICE_REF） same as above, model mimo-v2.5-tts-voiceclone the above content, plus the base64 of the reference audio (converted to 24 kHz mono WAV, maximum 30 seconds), carried with every segment request
Fish Audio（--tts-provider fish-audio） FISH_TTS_API_URL(default https://api.fish.audio/v1/tts），FISH_API_KEY the spoken text of each segment, numeric speaking rate, voice ID FISH_TTS_REFERENCE_ID; local audio is not sent
Self-hosted IndexTTS (--tts-provider index-tts） an HTTP(S) service deployed by the user themselves and specified by INDEX_TTS_ENDPOINT the specified HTTP(S) service {"voice": INDEX_TTS_VOICE, "text": 实读文本}; local audio is not sent
Experimental dub (see §8) MiMo ASR（MIMO_API_URL，MIMO_API_KEY，mimo-v2.5-asr) and MiMo voiceclone (TTS interface and credentials,mimo-v2.5-tts-voiceclone） the entire source video audio track is sent to ASR in 6-second windows; each Chinese translation sentence, together with about 10 seconds of the original speaker's voice clipped from the source audio (cloning reference), is sent to voiceclone

--voice-ref and dub both send a real person's voice to MiMo for cloning: use them only when the user has the right to use that audio and the voice owner consents to being cloned.

3. Environment requirements

export MIMO_API_KEY=***  # 也可使用仅供 TTS 的 MIMO_TTS_API_KEY

# 或改用 Fish Audio TTS
export TTS_PROVIDER=fish-audio
export FISH_API_KEY=***
export FISH_TTS_REFERENCE_ID=<voice-model-id>  # 可选；覆盖内置“娱乐扒妹”音色

# 或显式选择自托管 index-tts 端点，配置见 references/index-tts.md
export TTS_PROVIDER=index-tts

The following scripts/... are all relative to this skill directory. If the executor starts from the repository root, prepend the absolute directory of this skill to the script paths.

4. Input contract

The default input is work_dir/narration.json. Each segment must contain start、end and narration, and optional fields include pause_after_ms and overlaps_speech. Time uniformly represents the audio's final placement on theoutput timeline in seconds.。

The cut workflow is cut first, then dub:narration.json is itself written according to the output time of the cut final video, and there is no separate mapping script.

5. Run command

python3 scripts/voiceover.py --work-dir <work_dir> --narration <narration.json> \
  [--tts-provider auto|mimo-tts|fish-audio|index-tts] \
  [--mimo-voice 冰糖 | --voice-ref <reference-audio>] \
  [--preserve-approved-text] [--allow-partial-tts]

When run standalone and --narration is omitted, the default read is work_dir/narration.json；--narration is only used to specify a script in the same format at another path.

6. Output contract

  • tts_segments/*.wav: each narration segment corresponds to one audio file.
  • tts_meta.json: contains segments、engine、voice(the actually used provider, model, voice, or reference audio) and narration. Each segment records audio_path, time, pause_after_ms and placement fields.
  • Clean run writes partial: false and failures: []。
  • When using --allow-partial-tts to skip failed segments, write partial: true and failures: [{index,start,end,text,error}], keeping missing speech visible.
  • --preserve-approved-text is an explicit approved-script protection policy. Each segment retains the original authored_text evidence; the spoken_text actually read by TTS only undergoes the existing format/stage-direction cleanup. If the complete speech exceeds the time window and the cumulative speech-rate budget, the command fails and reports the segment index, original script, actually-read text, speech duration, and window evidence, without writing a successful tts_meta.json. In strict mode, any required segment failure (including provider failure) cannot be --allow-partial-tts downgraded to a deliverable partial success; the exception record is marked as required: true with the policy ID. Author segments that contain only cleanup markers such as [停顿] and have no actually-read text after cleanup are also required-segment errors.

7. Run rules

  • Segmented audio is cached by content in tts_segments/cache/: the key is the actually-read text, the tone request actually sent to the provider (for MiMo it is that natural-language instruction, and the speech rate changes the wording only at ≥+6% or ≤-3%; for Fish Audio it is the numeric speed; index-tts has no segment-level control) and the TTS settings, excluding the segment index and time window; therefore, when a segment position change makes the nominal speech rate go from +5% to -2%, MiMo does not re-synthesize; the cached WAV is invalidated as soon as its size/mtime_ns changes.narr_NNN.wav is a hard link to the cache (or a copy when hard links are unsupported), andtts_meta.json still references it as before. After deleting, inserting, or moving a segment, only regenerate the segments whose text or request sent to the provider changed (the nominal speech rate changes with the position of the first segment and the last two segments); the old version's narr_NNN.wav.cache.json is no longer read.
  • The approved-script protection policy is part of the cache settings: strict mode adds the policy and original script to the cache key, and does not hit the default policy (report-over-budget-v2, which is not included in the key); the old version's per-segment cache (including auto-shortened-script audio) is no longer read; only caches under the same strict policy with a spoken_text complete match and a non-empty WAV present can be reused offline; reuse is still checked against the current time window, and it still fails if it does not fit.
  • Before synthesis in this round, the strict CLI archives the old tts_meta.json by timestamp to tts_meta.history/, so on failure the current path will not continue to masquerade as this round's success; successful metadata is atomically replaced via a temporary file in the same directory.
  • auto preferentially uses the configured MiMo; when the MiMo key is missing and FISH_API_KEY is set, Fish Audio is used; when a reproducible provider choice is needed, explicitly pass --tts-provider。
  • The self-hosted index-tts endpoint can only be explicitly selected by --tts-provider index-tts or TTS_PROVIDER=index-tts , andauto is never chosen as a fallback. For the protocol, request body, receipt semantics, and cache invalidation rules, see references/index-tts.md。
  • Fish Audio requests WAV directly; by default it uses the "娱乐扒妹" voice (5653cea4ac83480aaf2bf45406556185），FISH_TTS_REFERENCE_ID can override it. The cache is regenerated when the model, voice ID, API URL, normalization settings, or content-derived speech-rate change (Fish does not receive pitch and emotion, so changes to them do not trigger regeneration). The current free model has no SLA and is subject to Fair Use and the official free-period limits.
  • --voice-ref is used only for full/cut commentary cloning; switch to mimo-v2.5-tts-voiceclone. Lazily normalize once only when new synthesis is truly needed; changes to the reference audio path,size/mtime_ns , or the preprocessed version invalidate the old cache. Use only after obtaining authorization; the reference audio is sent to MiMo.
  • dub voiceclone raw WAV is also cached by model, prompt, lines, and reference audio size/mtime_ns ; a matching rerun will not repeat the request or incur charges, dub_manifest.json line by line records tts_cache=hit|miss。
  • The synthesized segment audio is longer than the upper limit of reading the full text at the rate of TTS_MIN_SPEECH_RATE(default 2.5 characters/second, English at 1.5 characters per word) plus pauses and leading/trailing silence, it is considered TTS hallucination (after finishing the original script, it fabricates another passage), is retried as a failure, is not cached, and is not handed to assemble; if retries are exhausted, that segment fails, and the error states the duration and upper limit, the last rejected audio is left in tts_segments/narr_NNN.rejected.wav for preview. Digits (half-width/full-width) count as 1 character each,% counts as 3 characters (percent). dub voiceclone lines go through the same check and retry (rejected ones are left in dub_tts/line_NNN_raw.rejected.wav). This kind of WAV under older caches is no longer reused on rerun and will be synthesized again. Set to 0 to disable this check.
  • TTS_WORKERS、TTS_TIMEOUT、TTS_RETRIES、ALLOW_PARTIAL_TTS is used to adjust concurrency, timeouts, retries, and partial-success policy.

8. Experimental dub voiceover (English to Chinese, cloned original voice)

dub.py is an experimental feature: it translates the original English audio into Chinese and replaces the vocals across the entire track with the cloned timbre of the original speaker. It is a separate path from the commentary voiceover above,voiceover.py never calls it.

  • Trigger: only when the user explicitly requests English-to-Chinese dubbing, it is run by the orchestration entry point with --edit-mode dub --confirm-voice-rights ; the orchestration entry point passes the confirmation parameters as-is to the two stages of dub.py preparation and rendering. There is no separate manual stage.
  • Confirmation gate：dub.py Both stages of --confirm-voice-rightsmust include
  • Outbound content: the preparation stage sends the source video audio track in 6-second windows to MiMo ASR (mimo-v2.5-asr) to transcribe English; the rendering stage sends each Chinese translated sentence together with about 10 seconds of the original speaker's voice captured starting from the 2nd second of the source audio (dub_reference.wav) to MiMo voiceclone (mimo-v2.5-tts-voiceclone）。
  • Rights and consent: it may only be used for videos and audio the user has the right to use, and the speaker whose voice is being cloned has consented. The Agent must first confirm these two points with the user, and only after the user confirms may --confirm-voice-rightsbe added; if this cannot be confirmed, do not run dub, and do not use it to impersonate another person's speech.
  • Determinism gate: before voice cloning, the rendering stage writes dub_lint.json, and empty lines, overlapping, or out-of-bounds translations cause an abort, with no voiceclone request sent.
  • Local artifacts：dub_source.wav、dub_transcript.json、dub_brief.md、dub_reference.wav, the Agent-written dub_script.json、dub_lint.json、dub_tts/、dub_manifest.json and dub_<name>.mp4, are all written only in work_dir。

9. Capability boundaries

  • When over-window, keep the original script and log it; if bounded speed-up for assemble cannot fit it, block before rendering with no_safe_fit . For approved scripts, add --preserve-approved-text, and if over-window it fails at the TTS stage.
  • No mixing, no lowering of the original audio, no subtitle rendering.
  • It does not analyze the video or choose time points; it only dubs the fixed segments in the input script.
  • Neither the Fish Audio nor IndexTTS path accepts local --voice-ref; the former uses the already-created FISH_TTS_REFERENCE_ID to select a timbre.