What Is MultiFoley
MultiFoley is a sound effect generation system jointly launched by Adobe Research and the University of Michigan. It can generate Foley sound effects based on multimodal control from text, audio, and video. The system supports users in customizing and generating sounds synchronized with video based on text prompts, reference audio, or partial video, enhancing the video viewing experience. MultiFoley is jointly trained on internet video datasets and professional sound effect recordings, achieving high-quality, full-bandwidth (48kHz) audio generation. MultiFoley can provide flexible sound design control for video production, helping users create both clean and creative sound effects.
MultiFoley's Main Features
- Text-controlled Foley generation: Use text prompts to guide and generate sound effects synchronized with video, whether realistic sounds or creative sounds.
- Audio-controlled Foley generation: Supports users in selecting reference audio from a sound effect library, applying the sound to silent video, and synchronizing it with the video.
- Foley audio extension: Extend part of an audio track to produce complete Foley sound.
- Quality control: Generate high-quality full-band (48kHz) audio by adding quality tags to the text.
- Multimodal control: Combine conditional signals from text, audio, and video to provide detailed sound design control.
MultiFoley's Technical Principles
- Joint training: Train on internet video datasets (low-quality audio) and professional sound effect (SFX) recordings to generate high-quality full-band audio.
- Diffusion Transformer: Generate new samples from random noise based on a diffusion model, used in video-guided Foley sound generation and combined with multimodal control.
- High-quality audio autoencoder (DAC-VAE): Based on a variational autoencoder (VAE), it encodes 48kHz audio waveforms into 40Hz latent features, used for audio-video synchronization.
- Frozen video encoder: Used for audio-video synchronization, encoding video into features and using them together with audio latent codes.
- Multi-condition training strategy: Enables the model to flexibly support downstream tasks, such as audio extension and text-driven sound design.
- Multi-head attention mechanism: Enhances the model's expressive ability and learns different types of features or dependencies in parallel.
MultiFoley's Project Address
- Project official website：ificl.github.io/MultiFoley
- arXiv technical paper：https://arxiv.org/pdf/2411.17698
MultiFoley's Application Scenarios
- Film and video production: In film production, generate sound effects synchronized with on-screen actions, such as footsteps and door closing sounds, enhancing the audience's sense of immersion.
- Game development: In games, generate realistic sounds for different game environments and actions, improving the gaming experience.
- Animation production: For animation, generate corresponding sounds based on the actions of animated characters, making the animation more vivid.
- Advertising production: In the advertising industry, generate eye-catching sound effects based on advertising creativity, increasing the appeal of advertisements.
- Virtual reality (VR): In VR experiences, generate sounds synchronized with the virtual environment, improving users' immersion and experience quality.