What is ControlFoley
ControlFoley is Xiaomi's open-source controllable video sound effect generation model, capable of solving the controllability challenges in the V2A field. The model uniformly supports three types of video dubbing tasks: text-guided, text-controlled, and reference audio-controlled. Through the self-developed spatiotemporal audio-visual encoder CAV-MAE-ST, temporal-timbre disentanglement, and modality-robust training, it achieves comprehensive improvements in semantic alignment, temporal synchronization, and audio quality. The model reaches open-source SOTA on multiple benchmarks, and the code and model weights have been released.
Main features of ControlFoley
-
TV2A (text-guided video dubbing): Generates synchronized sound effects based on video and text prompts, with text supplementing the visual sound semantics.
-
TC-V2A (text-controlled video dubbing): When text and video semantics conflict, prioritizes the text intent while maintaining temporal synchronization.
-
AC-V2A (reference audio-controlled video dubbing): Controls the timbre style based on reference audio without disrupting the video rhythm.
Technical principles of ControlFoley
-
Joint visual encoding: The self-developed CAV-MAE-ST spatiotemporal audio-visual encoder focuses on audio-visual spatiotemporal correspondence, enhancing understanding of action rhythm and temporal synchronization, and combines with CLIP to balance semantics and synchronization.
-
Temporal-timbre disentanglement: Suppresses temporal information in the reference audio, retains global timbre features, and avoids reference audio interfering with video synchronization.
-
Modality-robust training: Random modality dropout + unified multimodal representation alignment, adapting to various input combinations; improves semantic consistency through the REPA alignment objective.
How to use ControlFoley
- Obtain open-source resources: Visit the GitHub repository to download the code and model weights, or directly use the official online Demo for a trial experience.
- Environment configuration: Install the dependency environment according to the repository instructions, and configure the Python runtime environment and necessary audio-video processing libraries.
- Choose a task mode: Choose one of the three task types according to your creation needs: TV2A (text-guided), TC-V2A (text-controlled), or AC-V2A (reference audio-controlled).
- Prepare input conditions: Import the video file; if TV2A is selected, attach a text prompt; if TC-V2A is selected, input a text instruction that conflicts with the visuals; if AC-V2A is selected, upload a reference audio file.
- Execute generative inference: Run the model inference script, and ControlFoley will generate sound effects synchronized with the video based on joint visual encoding and the temporal-timbre disentanglement mechanism.
- Export and post-processing: After obtaining the generated audio, output the final audio track through the VAE Decoder and Vocoder, and complete dubbing by compositing it with the video.
Core advantages of ControlFoley
-
Unified framework: A single model covers three types of tasks: text-guided, text-controlled, and reference audio-controlled, with no need to switch between multiple tools.
-
Precise synchronization: The self-developed CAV-MAE-ST spatiotemporal encoder enhances audio-visual temporal understanding, and its audio-visual alignment accuracy leads open-source competitors.
-
Timbre Decoupling: The time-timbre decoupling technique ensures that the reference audio only affects the timbre style without interfering with the original rhythm of the video.
-
Robust Control: Random modality dropout and unified representation alignment training enable the model to output stably under either unimodal or multimodal input.
-
Open-Source SOTA: Leads comprehensively in semantic alignment and sound quality on multiple benchmarks such as VGGSound-Test and Kling-Audio-Eval.
ControlFoley's project address
- Project Website：https://yjx-research.github.io/ControlFoley_web_page/
- GitHub Repository：https://github.com/xiaomi-research/controlfoley
- HuggingFace Model Hub：https://huggingface.co/YJX-Xiaomi/ControlFoley
- arXiv Technical Paper：https://arxiv.org/abs/2604.15086
Comparison of ControlFoley with Similar Competitors
| Comparison Dimension | ControlFoley | MMAudio | HunyuanVideo-Foley |
|---|---|---|---|
| Task Coverage | Uniformly supports three types of controllable tasks: TV2A / TC-V2A / AC-V2A | Mainly supports basic TV2A video dubbing | Mainly supports basic TV2A video dubbing |
| Text Conflict Handling | Strong: In conflict scenarios, DeSync is only 0.36-0.38, prioritizing text intent | Weak: Text is easily overridden by visual information | Weak: Limited text control capability |
| Reference Audio Control | Supported; time-timbre decoupling does not break synchronization | Not supported | Not supported |
| Audio-Video Synchronization | Excellent: CAV-MAE-ST enhances spatiotemporal correspondence | Good | Good |
| Open-Source Status | Code, weights, and technical report are all open source | Open source | Open source |
ControlFoley's Application Scenarios
-
Short Video Creation: Add customized sound effects that match the creator's intent to silent footage, avoiding deviations from the model's automatic guesses.
-
Animation and Gaming: Generate style-specific impact sounds or ambient sounds for character actions, such as replacing an ordinary knock with the sound of a mallet hitting a drum.
-
Film and Television Post-Production: Unify the timbre style of sound effects across an entire film based on reference audio assets, maintaining sound consistency for a brand or series of works.
-
Advertising and Marketing: Quickly generate synchronized dubbing that matches brand tonality according to text instructions, strengthening rhythm and emotional expression.
-
Self-Media Livestreaming: Supplement livestream clips or derivative videos with multimodal controllable immersive audio, improving content completeness.