SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsvoice-to-video

voice-to-video

Design & creation

View repository

Updated 10-16 08:49

About this skill

voiceover-to-video

Turn a voiceover script into an script ↔ voiceover ↔ visuals sentence-by-sentence correspondence MP4:

script.txt ──tts.py──▶ voice.mp3 + timeline.json(逐句/逐词毫秒时间戳) + subtitles.srt
                              │
                              ▼
              composition.html（一句口播 = 一个场景，时间戳来自 timeline）
                              │ render.py（无头浏览器逐帧截图 = 确定性渲染）
                              ▼
                        final.mp4（配音音轨已混入）

Core principles (must follow):audio is the clock, visuals follow the audio. Scene transitions, element appearances, and subtitle highlighting are all driven by timestamps in timeline.json; rendering is seek-style frame-by-frame screenshots, and the same input always renders the same every frame.

Working directory conventions

Each video gets its own independent directory, with outputs stored together:

<video_dir>/
├── script.txt          口播稿（纯文本）
├── kit.css engine.js   从技能 assets/ 拷贝
├── composition.html    你编写的合成页
├── build/              tts.py 产物: voice.mp3 / timeline.json / subtitles.srt
├── frames_final/       渲染中间帧（默认渲完删除）
└── final.mp4           成片

Step 0 · Environment self-check

Check on first use or when errors occur (versions confirmed available on this machine: ffmpeg 6.1.1 / edge-tts 7.2.8 / playwright 1.55):

ffmpeg -version | head -1 && python -c "import edge_tts, playwright" \
  && python -m playwright install chromium --dry-run

If Chromium is missing, run python -m playwright install chromium. edge-tts needs access to Microsoft services (failures are usually network/proxy issues; retry or switch networks).

Step 1 · Voiceover script script.txt

  • Plain text, one continuous passage, standard punctuation (sentence segmentation based on 。！？；!? and line breaks).
  • If the user provided a script → use it as-is, only correcting obvious slips of the tongue/typos.
  • If the user only provided a topic → first write a script based on the topic (colloquial short sentences, ≤ 40 characters per Chinese sentence),and proceed to composition only after user confirmation; Chinese speech rate is about 4.2 characters/second, which can be used to estimate the final video length.
  • For long scripts (>5 minutes), warn the user about rendering time (1080p30 is about 6-10 times the video length); you can change to --fast or 720p.

Step 2 · Generate voiceover and timeline

python <skill>/scripts/tts.py --script script.txt --out build --voice xiaoxiao
  • Voice short names:xiaoxiao(female · general default) xiaoyi(female · lively) yunxi(male · sunny) yunjian(male · steady) yunyang(male · news broadcast); or pass a full edge-tts voice name. Available --rate +10% to adjust speech rate.
  • The output build/timeline.json is the "clock" for the whole workflow; read it first: confirm sentence segmentation is reasonable and durations match expectations, then proceed to Step 3.
  • User-provided voiceover/SRT (e.g. ListenHub export) → skip this step, and convert the SRT into timeline.json's sentences format ({i,text,start,end,words?}, milliseconds; words can be omitted).

Step 3 · Write the composition page composition.html

  1. Choose a visual style: first read references/styles.md(style catalog + layout DNA + selection rules). If the user specifies a style, use the corresponding kit (assets/kits/ contains dark-hud / editorial / terminal / whiteboard / swiss / ink / glass / clay / blueprint / pixel / stage); if not specified, automatically match by subject matter; if unsure, use the default assets/kit.css(BlockFrame block style). When the visuals need embedded video assets, use the stage style + HF.bindVideo, convert assets to webm, and add --wait-videos。
  2. copy assets/engine.js、assets/template.html(rename to composition.html) andthe selected kit(e.g. assets/kits/ink.css) to the video directory, and point composition.html's <link rel="stylesheet"> to that kit filename. Note: the safety hook will intercept copying source files with Bash cp ; use the Read + Write tools to complete the copy (write engine.js / kit contents as-is).
  3. First read references/composition-guide.md(alignment rules, animation properties, scene types, and the self-check list are all in there), key points:
    • one spoken line = one .scene，data-start = sentence start，data-end = sentence hold_end；
    • scene content must correspond to the current sentence; keyword appearance timing uses data-delay ≈ 词时间 - 场景start；
    • follow the selected style's references/styles.md "layout DNA" to rearrange the scene structure(layout skeleton/composition/decoration/motion signature change with the style); only swapping CSS colors does not count as meeting the bar;
    • subtitle data is introduced with <script src="build/subtitles.js"></script> (produced by tts.py, i.e. the timeline's sentences; can also be manually pasted as const SUBTITLES）；
    • CSS animation/transition, setTimeout, and Math.random are prohibited (they break determinism).
  4. Visuals: by default use kit.css's BlockFrame design system (cream background/candy colors/black outlines with hard shadows). If the user gives brand/style requirements → override :root variables and component styles, but keep the whole set consistent.

Step 4 · Preview (required after compositing)

python -m http.server 8765   # 在视频目录下执行

Give http://localhost:8765/composition.html to the user, or self-check with browser tools:space to play, confirm (1) subtitles switch sentence by sentence and word highlighting keeps up with the speech rate; (2) the scene switches at the instant the sentence changes; (3) element appearance timing matches the semantics. If problems are found, go back to Step 3 and fix them (no rendering during the preview stage, so fixes are fastest).

Step 5 · Render the final video

python <skill>/scripts/render.py --html composition.html --audio build/voice.mp3 --out final.mp4
  • Default 1920×1080 @30fps, CRF18. For drafts use --fast, for samples use --width 1280 --height 720。
  • Author attribution watermark:--watermark laohaibao2025(small signature text in the lower right corner, injected at render time, does not modify the composition page; the engine also supports HF.init({ watermark: "..." })）。
  • Resume rendering from a breakpoint: after an interruption, rerunning the original command automatically skips existing frames;--keep-frames keep intermediate frames.
  • Estimated time ≈ video length × 6-10 (1080p30); for long videos, inform the user first.
  • After rendering, use ffmpeg to spot-check frame alignment with the timeline:ffmpeg -ss <句中时间> -i final.mp4 -frames:v 1 check.jpg, confirm that the frame/subtitles at that moment match the sentence.

Step 6 · Delivery

Report to the user:final.mp4(final video),build/subtitles.srt(external subtitles for platform upload),composition.html(source files, can be modified and re-rendered), duration/size. In one sentence, explain "which two steps to rerun after changing the script/changing the visuals".

Troubleshooting

Symptom Handling
edge-tts 429/timeout Wait a few seconds and retry; switch networks; or the user brings their own TTS+SRT (using the timeline.json format)
Chromium fails to start python -m playwright install chromium
Wrong fonts in the visuals/Chinese tofu blocks Confirm Microsoft YaHei is available on this machine; kit.css's --f-body fallback chain already includes system fonts
Rendered visuals do not match the preview Check whether CSS animation/random was used; all animations must go through data-anim / HF.on
Audio and video are half a beat off render.py uses the ffprobe audio duration as authoritative; confirm that composition does not hardcode durationMs to override it
Frame rendering interrupted Rerun the original command (auto-resume), or --from-frame N