In the field of artificial intelligence, Google's DeepMind has once again made a breakthrough with the launch of an AI model called V2A (Video-to-Audio). This technology can combine video pixels with text prompts to generate detailed audio tracks containing dialogue, sound effects, and music for silent videos.
V2A technology can not only be used in conjunction with DeepMind's own video generation modelVeobut can also be combined withSora、KlingorGen 3and other competitors' video generation models to add dramatic music, realistic sound effects, or dialogue that matches the characters and emotions in the video. Of course, this technology is also suitable for adding sound to traditional film materials, such as archival footage and silent films. The power of V2A lies in its ability to create an unlimited number of audio tracks for each video input.
The DeepMind team said the V2A model is based on a diffusion model and provides the most realistic and satisfying results in synchronizing video and audio. The V2A system first encodes the video input into a compressed representation, and then the diffusion model gradually refines audio from random noise under the guidance of visual input and text prompts. Finally, the audio output is decoded, converted into an audio waveform, and combined with the video data.
To improve audio quality, DeepMind added additional information during training, including AI-generated sound descriptions and transcriptions of dialogue. In this way, V2A learned to associate certain audio events with different visual scenes and to respond to information contained in the descriptions or transcriptions.
However, V2A technology also has some limitations. For example, the quality of the audio output depends on the quality of the video input. Artifacts or distortions in the video, if beyond the model's training distribution, may cause a significant decline in audio quality. In addition, lip synchronization in the video is still not stable enough.
Currently, V2A has not been publicly released. DeepMind is collecting feedback from leading creatives and filmmakers to ensure that V2A can have a positive impact on the creative community. The company said that before considering broader access, V2A will undergo rigorous safety evaluations and testing.
(More information:Google DeepMind – Generating audio for video）