What is Fun-CineForge
Fun-CineForge is the first film-grade multimodal dubbing large model open-sourced by Tongyi Lab, built onCosyVoice3It innovatively introduces the "temporal modality" to achieve precise audio-visual synchronization. The model supports monologues, narration, dialogues, and multi-person scenes, solvinglip synchronization, emotional expression, timbre consistency, and temporal alignment. Fun-CineForge comes with an open-source CineDub dataset construction pipeline, covering 350+ film and TV dramas, with a Chinese character error rate as low as 1.49%. Even in complex scenarios such as facial occlusion and shot switching, it can maintain high-quality dubbing effects.
Main Features of Fun-CineForge
-
Lip Synchronization: The model supports highly synchronized synthesis of speech and lip movements of characters in the video, achieving precise audio-visual alignment.
-
Emotional Expression: Based on the character's facial appearance and instruction descriptions, it achieves anthropomorphic presentation and free control of emotional tones.
-
Timbre Cloning: The model can reference the timbre characteristics of input audio to synthesize highly similar personalized speech.
-
Temporal Alignment: It controls the start and end of speech based on timestamps, and can generate speech at the correct time period even when the speaker is occluded.
-
Multi-Scenario Adaptation: It supports complex film and TV dubbing scenarios such as monologues, narration, two-person dialogues, and multi-person dialogues.
Technical Principles of Fun-CineForge
- Multimodal Fusion Architecture: The model can simultaneously process four types of information. The visual modality learns lip movements and facial expressions, the text modality provides dialogue content and character emotion cues, the audio modality serves as the prediction target, and the temporal modality controls when speech appears and indicates speaker identity. The four complement each other to achieve precise dubbing.
- Temporal Modality Innovation: For the first time, temporal information is introduced as an independent modality into a dubbing model. Through strong supervision signals such as start time, duration, and speaker identity, the model understands "when and who is speaking," and can accurately locate speech periods during facial occlusion or shot switching.
- Data-Driven Training: Trained on the automatically constructed CineDub dataset, which is extracted from film and television materials through processes such as vocal separation, text transcription, and speaker separation. It contains frame-level lip data, millisecond-level timestamps, and emotion annotations, providing multimodal supervision signals for the model.
Fun-CineForge Project Address
- Project Official Website：https://funcineforge.github.io/
- GitHub Repository：https://github.com/FunAudioLLM/FunCineForge
- HuggingFace Model Library：https://huggingface.co/FunAudioLLM/Fun-CineForge
Application Scenarios of Fun-CineForge
-
Film and TV Post-Production: Provides multilingual dubbing for films and TV dramas, precisely matching lip movements and emotions, and handling complex scenarios such as shot switching and facial occlusion.
-
Animation and Game Development: Generates audio-visually synchronized speech for animated characters, supports timbre differentiation for multiple characters, and reduces the cost of game story dubbing.
-
Content Localization: Translates and dubs overseas film and television works into other languages, preserving the emotional rhythm of the original film, and supporting conversion of long segments such as narration and monologues.
-
Advertising Short Videos: The model can quickly generate voiceovers for talking-head videos, adjust tone according to the emotions in the video, and clone specific timbres to maintain brand consistency.
-
Accessibility Assistance:The model can generate synchronized narration speech for silent videos, assisting visually impaired users in understanding the visuals and providing accurate subtitle-audio pairing.