What is Dubbing v2
Dubbing v2 is an AI dubbing model launched by ElevenLabs, supporting automatic translation and dubbing in 29 languages while preserving the original speaker's timbre and emotion. The model offers two workflow modes: Auto Dub for quickly generating previews, and Dubbing Project for entering the timeline editor to fine-tune segment by segment. Dubbing v2 supports multi-speaker separation,voice cloning, multi-format import and export, and API batch processing, and can process content up to 2.5 hours long.
Main features of Dubbing v2
-
AI automatic dubbing: Supports 29 languages, automatically detects multiple speakers and separates voices while preserving original vocal characteristics.
-
Voice cloning: Provides three modes: segment-level cloning, track-level cloning, and voice library selection.
-
Timeline editor: Allows segment-by-segment editing of transcribed text, adjusting translations, fine-tuning the timeline, and regenerating segments.
-
Multi-format support: Import supports MP3/MP4/WAV/MOV and YouTube/TikTok/Vimeo/X links; export supports MP4/AAC/WAV/SRT/AAF.
-
Dual workflow modes: Auto Dub quickly generates automatically, while Dubbing Project supports detailed editing.
-
API integration: Supports batch processing and automated workflows, and can process content up to 2.5 hours long.
Technical principles of Dubbing v2
- Multilingual speech recognition: A deep learning-based ASR model automatically transcribes source-language content, identifies multiple speakers, and separates voice tracks.
- Neural machine translation: Uses a context-aware translation engine to preserve colloquial expressions and cultural context, avoiding distortions from literal translation.
- Voice cloning and synthesis: Extracts speaker timbre features through a Speaker Encoder and combines them with a TTS model to generate speech in the target language while maintaining the original prosody and emotion.
- Timeline alignment algorithm: A dynamic programming algorithm matches translated text with original timestamps and supports segment-by-segment fine-tuning and regeneration.
- Multimodal processing pipeline: Audio-video separation → speech recognition → translation → speech synthesis → mixing output, supporting up to 2.5 hours of continuous processing.
How to use Dubbing v2
- Visit the official website: Visit the Dubbing v2 official website https://elevenlabs.io/dubbing-studio，登录 ElevenLabs account.
- Upload source files: Directly upload MP3/MP4/WAV/MOV files, or paste links from YouTube/TikTok/Vimeo/X platforms.
- Select target languages: Multiple target languages can be selected at the same time for parallel processing.
- Choose a workflow: Auto Dub quickly generates a preview automatically, or Dubbing Project enters detailed editing mode.
- Review and edit: In the timeline editor, check translation accuracy segment by segment, adjust timeline alignment, and regenerate unsatisfactory segments.
- Export the final product: Choose MP4 (with video), AAC/WAV (audio only), or SRT subtitle format for download.
Core advantages of Dubbing v2
-
High timbre fidelity: The cloned dubbing is highly consistent with the original speaker's timbre, and emotional expression is natural
-
Multi-speaker support: Automatically identifies and separates different speakers, and can handle even overlapping dialogue
-
Strong editing controllability: The timeline editor provides segment-by-segment fine-tuning capability, rather than an "all-or-nothing" output
-
Cost efficiency: Traditional dubbing for a single 30-second ad in 10 languages can cost $10,000-$30,000, while ElevenLabs can complete it in minutes at a significantly lower cost.
Dubbing v2 project address
- Project official website：https://elevenlabs.io/dubbing-studio
Comparison of Dubbing v2 with similar competitors
| Dimension | Dubbing v2 | Speech Synthesis |
|---|---|---|
| Core features | Video/audio translation + dubbing + voice cloning | Text-to-speech, multiple voice options |
| Translation capability | Built-in automatic translation for 29 languages | No translation feature |
| Voice preservation | Preserves the original speaker's voice and emotion | Uses preset voices or custom cloning |
| Multiple speakers | Automatic detection and separation | Single voice output |
| Timeline editing | Fine-grained segment-by-segment editing | No timeline concept |
| Input method | Audio/video files/platform links | Plain text input |
| Use cases | Content localization, multilingual distribution | Audiobooks, navigation, customer service voice |
Dubbing v2 application scenarios
-
Podcast localization: Simultaneously translate and dub a single episode into 29 languages, covering the global market without re-recording.
-
Cross-border e-commerce ads: Quickly generate multilingual versions from a single video asset, greatly reducing the production cost of ad campaigns.
-
Online education: Batch translate course videos while preserving the instructor's original voice characteristics, enhancing the learning immersion for non-native learners.
-
Film and television content distribution: Independent creators or small studios can achieve multilingual distribution of film and television works at low cost.
-
Corporate training: Multilingualize internal training video materials, unify brand voice output, and accelerate knowledge transfer across global teams.