What is Kling-Foley
Kling-Foley isKling AI 's multimodal video-to-sound-effect model. The model takes video and text prompts as conditional inputs, and can generate high-quality stereo audio that is semantically related to the video content and temporally synchronized, covering various types of sound content such as sound effects and background music, and supporting audio generation of arbitrary duration. The model is based on a multimodal-controlled flow matching architecture, using multimodal feature fusion and dedicated modules to accurately achieve audio-video alignment. The model is trained on a large-scale self-built multimodal dataset, demonstrating excellent audio generation performance and reaching industry-leading levels in the field of sound effect generation, bringing more efficient and high-quality audio solutions to video content creation.
Kling-Foley's Main Features
- High-Quality Sound Effect Generation: Based on the input video content and optional text prompts, it generates high-quality stereo audio that is semantically related to the video frames and temporally synchronized, covering various types of sound content such as sound effects and background music, meeting audio needs in different scenarios.
- Arbitrary-Duration Audio Generation: Supports generating audio content of arbitrary duration and can dynamically adapt to the length of the input video.
- Stereo Rendering: Has stereo rendering capability and supports spatially directed sound source modeling and rendering, giving the generated audio a stronger sense of space and immersion.
Kling-Foley's Technical Principles
- Multimodal-Controlled Flow Matching Model: Kling-Foley is a multimodal-controlled flow matching model. Its core is to take text, video, and video frames extracted over time as conditional inputs, fuse them based on a multimodal joint conditioning module, and input them into the MMDit module for processing. The multimodal control approach enables the model to better understand and generate audio that matches the video content.
- Modular Processing Pipeline: The model's processing pipeline includes multiple key modules. Multimodal features are fused based on the multimodal joint conditioning module and input into the MMDit module to predict VAE latent features. A pretrained mel decoder reconstructs the latent features into a mono mel spectrogram. Based on the Mono2Stereo module, the mono spectrogram is rendered into a stereo spectrogram, and a vocoder is used to generate the output waveform.
- Visual Semantic Representation and Audio-Video Synchronization Module: The Kling-Foley architecture introduces a visual semantic representation module and an audio-video synchronization module, supporting frame-level alignment between video conditions and audio latent elements, improving video semantic alignment and audio-video synchronization, and ensuring that the generated audio is highly matched with the video in time and content.
- Discrete Duration Embedding: Kling-Foley introduces discrete duration embedding as part of the global conditioning mechanism. This enables the model to better handle video inputs of different lengths and generate audio content suitable for the video length.
- Universal Latent Audio Codec: At the audio latent representation level, Kling-Foley applies a universal latent audio codec, which can achieve high-quality modeling in diverse scenarios such as sound effects, speech, singing, and music. Its main body is Mel-VAE, which jointly trains the Mel encoder, Mel decoder, and discriminator, allowing the model to learn a continuous and complete latent space distribution and significantly enhancing audio representation capability.
Kling-Foley's Project Address
- Project Website：https://klingfoley.github.io/Kling-Foley/
- GitHub Repository：https://github.com/klingfoley/Kling-Foley
- arXiv Technical Paper：https://www.arxiv.org/pdf/2506.19774
Kling-Foley's Application Scenarios
- Video Content Creation: Provide precisely matched sound effects and background music for video production such as animation, short videos, and advertisements, enhancing the appeal and professionalism of videos and improving creative efficiency.
- Game Development: Generate realistic scene sound effects and background music, such as weapon firing, character actions, and ambient sound effects, enhancing the immersion of games and the player experience.
- Education and Training: Add appropriate sound effects and background music to instructional videos and virtual training environments, enhancing the realism and appeal of teaching and training and improving learning outcomes.
- Film and Television Production: Generate high-quality sound effects and soundtracks for film, television, and other audiovisual works, improving the sound quality and dramatic impact of the works.
- Social Media: Users can quickly add matching sound effects and background music to shared videos, enhancing the appeal of the content.