SuperAPI AI Directory

qwen-subtitle

Coding & development

View repository

Updated 10-15 17:26

About this skill

qwen-subtitle — Bailian subtitle intelligent error correction + multilingual going-global dubbing

What problem does it solve

For subtitles of screen-recording / tutorial videos, no matter which ASR is used,proper nouns will always be misheard:Claude heard as cloud、Codex heard as class q、html2pptx heard as html to ppt、OOXML heard as OOCML. Relying on a manually maintained glossary to replace them is both tiring and incomplete.

The core insight of this skill:In screen-recording videos, the correct words are often clearly written on the screen. So — have the vision model look at that frame according to the timestamp, and read the real text on the screen to correct it. This is something single-channel (audio-only) tools cannot do.

Judgment criterion (to avoid overfitting to literal examples): any term / product name / code / command / file name / path that is "written on the screen but heard as a homophone" belongs to this category.

Which Qwen capabilities are combined

Capability Model Command used Role
🎙️ Speech recognition FunAudio-ASR bl speech recognize Produces subtitles, withsentence-level + word-level millisecond timestamps— subsequent frame extraction, sentence splitting, and dubbing alignment all rely on it
🧠 Error marking qwen3.7-max bl text chat High-precision scanning for "obviously misheard" points, categorized into screen / semantic types
👁️ Screen-reading correction qwen3-vl-plus bl vision describe For the screen type, extract frames by timestamp and read the real text on the screen to decide, with visual evidence
✨ Filler word removal qwen-plus bl text chat Delete uh, ah, um, stutters and repetitions, for smoothness

The model division of labor ismeasured in practiceand finalized: use qwen3.7-max for error marking (accuracy is the key, plus will miss real errors and re-mark them incorrectly); use qwen-plus for filler word removal (low risk, 5 times faster). These are in the FLAG_MODEL / VL_MODEL / FILLER_MODEL constants at the top of the script, and can be changed directly when needed.

Process (5 steps, run in one command)

视频
 ├─[1] FunAudio ASR ───────► 字幕轨道(句+词级毫秒时间戳)
 ├─[2] qwen3.7-max 标错 ───► 可疑点(screen=看画面 / semantic=纯语义)
 ├─[3] qwen3-vl-plus 看帧 ─► screen 类逐帧读屏纠正(带证据)
 │                           └ 取不到证据 → 保留原文+标记待确认〔零误改〕
 ├─[4] 词级时间戳断句 ─────► 按标点拆长句(>24字 / >6秒)
 ├─[5] qwen-plus 去水词 ───► 删语气词/口吃,顺滑
 └─► corrected.srt + transcript.json + report.md(画面证据)

Preparation

Authentication:bl If already logged in, nothing needs to be set — the script directly uses bl's own authentication (bl auth status verification). The cloned dubbing step requires curl to directly call the original API, and the script willautomatically from ~/.bailian/config.json read the key(or environment variable DASHSCOPE_API_KEY, either is fine).No need to manually export every time, and the key is never written into any file. Only when bl has never been configured, export DASHSCOPE_API_KEY=sk-xxx。

Dependencies:

  • bl(Bailian CLI, v1.4+;bl --version verification)
  • ffmpeg(audio extraction + frame extraction; the script automatically finds it from PATH, or set the FFMPEG environment variable)
  • python3
  • flask(Only needed for the preview page:pip install flask; not needed for error correction/translation/dubbing)

bl speech recognize and bl vision describe bothdirectly accept local file paths(uploading is already handled internally by the CLI), so there is no need to manually upload files to get URLs.

Step one: first ask which languages are needed (default behavior,the menu always asks in Chinese)

Each run,first ask in Chinesewhich languages the user wants (give numbers, multiple selection allowed e.g. 1,3,5)。both Q&A and tab labels use Chinese(the user may not necessarily understand foreign languages/native spellings):

请选择要生成的语言(可多选,用逗号分隔):

  〔配音 + 字幕〕用你的克隆声音说外语
    1. 中文(原声,仅纠错字幕)
    2. 英语    3. 日语    4. 韩语

  〔仅字幕〕qwen 翻译,视频保留原声
    5. 西班牙语   6. 葡萄牙语   7. 阿拉伯语
    8. 印尼语     9. 越南语    10. 泰语
  • select only "Chinese" → pure error correction (A). If selecteddubbing language(English/Japanese/Korean) → A + cloned dubbing (B). If selectedsubtitle-only language → A + translated subtitles (B, no dubbing, video keeps original audio).
  • Dubbing only supports Chinese/English/Japanese/Korean(CosyVoice limitation); other languages only produce subtitles (qwen supports 92 languages, here are a few mainstream overseas ones, add others as needed).
  • code reference: English en / Japanese ja / Korean ko / Spanish es / Portuguese pt / Arabic ar / Indonesian id / Vietnamese vi / Thai th (in the script LANGS table).

A. Error-correct the Chinese subtitles (baseline, run first in all cases)

python3 scripts/subfix.py <video.mp4> [--out DIR]

Parameters:--out DIR(default <视频名>.subfix/)、--max-seconds N(trial run the first N seconds),--lang zh、--reuse(reuse asr.json/suspects.json to rerun only the subsequent part).

Output:corrected.srt(Chinese subtitles with error correction + sentence segmentation + filler word removal),transcript.json([{start,end,text}])、report.md(each change + visual evidence + a "keep original text pending confirmation when evidence cannot be obtained" list); additionally corrected.json(sentence-level structured results) and report.json(machine-readable evidence). Intermediate artifacts:asr.json/suspects.json。

When showing results to the user: read out report.md 's changes + evidence (most persuasive); honestly distinguish "already changed (with solid visual evidence)" from "kept original text pending confirmation", and don't claim the latter has been fixed.

B. Multilingual overseas: clone original audio + translate + dub

When the user selects a foreign language, run on top of A:

python3 scripts/dub_multi.py <video.mp4> \
  --transcript <A的transcript.json> --langs en,ja,es,pt --out <DIR> [--clip-seconds N] [--voice-id ID]

⚠️ --clip-seconds default 0 = full video; it willlimit the actual range of subtitles/dubbing(not just trimming the preview), set only when doing a trial run of the first N seconds. If you want the full video, don't include it (and the upstream subfix.py also should not include --max-seconds)。

--langs pass code (en/ja/ko/es/pt/ar/id/vi/th…). The script automatically determines the tier:

  1. dubbing language (DUB={en,ja,ko}): clone original audio (extract 18s → bl file upload → clone API → voice_id,--voice-id can be skipped) → translate by whole sentence, with word count constraints(dubbing=subtitles use the same copy, must be spoken within the shot duration) → cloned voice TTS → atempo check duration → output <code>.json + <code>.m4a(dubbing track).
  2. subtitle-only language (the rest): translate only → output <code>.json; video keeps original audio.
  3. When all selected are subtitle-onlydo not clone(saves a step). Output manifest.json + transcripts for each language (+dubbing track) + clip.mp4。

Translation has two paths (script translate() automatically selects by for_dub ):dubbing languages use qwen-plus——the same script is both dubbing and subtitles, compressed by word count (≈时长×2.6) so it can be spoken within the duration;pure subtitle languages use qwen-mt-turbo(TRANSLATE_MODEL constant, Bailian's dedicated translation model, faithful, brand/model names naturally preserved). Supported:dubbing=Chinese/English/Japanese/Korean(CosyVoice v2 official);subtitles=92 languages(LANGS add code in the table). English dubbing is fully verified; Japanese/Korean dubbing uses the same cloned voice, quality needs actual-run confirmation.

Three key designs of dubbing (learned from pitfalls, understand before changing):

  • Read by whole sentences, not by subtitle fragments—fragments get read as half-sentences and sound "unfinished"; so first merge the subtitle fragments back into whole sentences before dubbing.
  • Subtitles = the same translation as the dubbing, timed by sentence—otherwise the text on screen and what's spoken won't match up.
  • Compress only, never slow down(atempo max 1.5): if the translation is too short, let it play naturally with silence, don't slow it down and drag.

Multilingual preview (tab switches subtitles + switches audio track)

preview pagecomes built-in(scripts/preview_editor.py, a local interactive page based on Flask). Requires flask(pip install flask; error correction/translation/dubbing itself doesn't need it). Hand the manifest to it:

python3 scripts/preview_editor.py <out>/manifest.json        # 多语言(manifest)
python3 scripts/preview_editor.py <video> <transcript.json>  # 单语言

In the top right of the preview page, output by language tab, switch tab → the right side switches to that language's subtitles, the left side video switches to that language's audio track (source language uses the original audio, foreign languages use the cloned dubbing track, synced with the video frames). Review it manually, then save, then hand it off for burning.

Design principle: zero false edits first

In subtitle tools,a wrong "correction" is worse than no correction—changing a correct word into a wrong one actively pollutes the subtitles. So every step of this pipeline follows "rather not change than change wrongly":

  • Don't overstep when flagging errors: only flag what's clearly misheard; Chinese that is itself fluent and looks like a name (even if a suspected transliteration) and words that are themselves correct just "not specific enough" are not flagged.
  • No VL hallucinations: only recognize text that can be read character by character on the screen; anything inferred from icons/logos/common sense doesn't count.
  • Minimal replacement + pronunciation consistency: only replace the few misheard characters, without bringing in adjacent path segments; the changed word must match the pronunciation of the original text.
  • If it can't be proven, keep the original text: for screen-type cases where no on-screen evidence can be obtained, keep the original text + mark as pending confirmation, never apply a blind model guess.

These gates were added only after going through a round of pitfalls (mis-flagging ooxml, changing "超级麦吉" to "超级码力/Supermario"). For the ins and outs of each gate and the corresponding prompt wording, see references/design-gates.md—be sure to read it before changing prompts, otherwise it's easy to tune the precision back to its original form.

Tuning parameters (constants at the top of the script)

  • FLAG_MODEL / VL_MODEL / FILLER_MODEL — three-stage model
  • MAX_CHARS=24 / MAX_DUR_MS=6000 — sentence-splitting thresholds (max character count/duration for a single subtitle)
  • MIN_BREAK_CHARS=12 — break at punctuation once it's already this long
  • Sweet spot: screen recording / screen sharing / tutorial type (terminology is all written on screen). For pure voiceover, talking head (terminology not on screen), VL can't get evidence and will keep more original text—this is expected behavior, not a bug.