What is OmniAudio
OmniAudio is a technology launched by the speech team of Alibaba Tongyi Lab for generating spatial audio (FOA) from 360° video. It provides a more realistic audio experience for virtual reality and immersive entertainment. By building the large-scale dataset Sphere360, which contains more than 103,000 video clips, covers 288 audio events, and has a total duration of 288 hours, it provides abundant resources for model training. OmniAudio's training is divided into two stages: self-supervised coarse-to-fine flow matching pretraining, based on large-scale non-spatial audio resources for self-supervised learning; and supervised fine-tuning based on dual-branch video representations, strengthening the model's ability to represent sound source direction.
OmniAudio's main features
-
Generate spatial audio: It can directly generate FOA audio from 360° video. This audio is a standard 3D spatial audio format that can capture the directionality of sound and achieve realistic 3D audio reproduction. It uses four channels (W, X, Y, Z) to represent sound, where the W channel is responsible for capturing overall sound pressure, and the X, Y, and Z channels capture sound information in the front-back, left-right, and vertical directions respectively. When the head rotates, it can ensure that the accuracy of sound localization is maintained.
-
Enhance immersive experience: It brings new possibilities for virtual reality and immersive entertainment. It can solve the problem that traditional video-to-audio generation technology mainly generates non-spatial audio and cannot meet the demand for 3D sound localization in immersive experiences.
OmniAudio's technical principles
- Self-supervised coarse-to-fine flow matching pretraining
-
Data processing: Because real FOA data is scarce, the research team uses large-scale non-spatial audio resources (such as FreeSound, AudioSet, VGGSound, etc.) to convert stereo into a "pseudo-FOA" format. Specifically, the W channel is the sum of the left and right channels, the X channel is the difference between the left and right channels, and the Y and Z channels are set to zero.
-
Model training: The converted "pseudo-FOA" audio is fed into a four-channel VAE encoder to obtain a latent representation, and then random time-window masking is performed with a certain probability, and the masked latent sequence together with the complete sequence are input as conditions to the flow matching model. By minimizing the difference in the velocity field of the latent states before and after masking, the model achieves self-supervised learning of audio timing and structure. This stage enables the model to master general audio features and macroscopic temporal patterns, providing a foundation for the subsequent refinement of spatial audio.
-
- Supervised fine-tuning based on dual-branch video representations
-
Data utilization: Only real FOA audio data is used, and the masked flow matching training framework continues to be used, but at this time all of the model's attention is focused on the spatial characteristics of the four channels.
-
Model strengthening: By applying a higher probability of masking to real FOA latent sequences, it strengthens the ability to represent sound source direction (the complementary relationship among the four channels W/X/Y/Z), and improves the reconstruction of high-fidelity spatial audio details on the decoding side.
-
Dual-branch combination: After completing self-supervised pretraining, the model is combined with a dual-branch video encoder. For the input 360° panoramic video, a frozen MetaCLIP-Huge image encoder is used to extract global features; at the same time, local FOV views are cropped from the same video, and local detail representations are also obtained through this encoder. The global features are max-pooled and used as the global condition of the Transformer, and the local features, after temporal upsampling, are added element-wise to the audio latent sequence as the local condition in the step-by-step generation process.
-
Fine-tuning and Output: While maintaining the general direction of the pretrained initialization parameters, efficiently fine-tune the conditional flow field to precisely "sculpt" FOA latent trajectories that conform to visual instructions from noise. After fine-tuning is complete, in the inference stage it is only necessary to sample the learned velocity field and then restore the waveform through the VAE decoder, thereby outputting four-channel spatial audio that is highly aligned with the 360° video and has a precise sense of direction.
-
OmniAudio's project address
- Project Website：https://omniaudio-360v2sa.github.io/
- Github Repository：https://github.com/liuhuadai/OmniAudio
- arXiv Technical Paper：https://arxiv.org/pdf/2504.14906
OmniAudio's Application Scenarios
-
Virtual Reality (VR) and Immersive Experiences: OmniAudio can generate spatial audio for VR content that highly matches the visual scene, enhancing the user's sense of immersion.
-
360° Video Soundtrack: Automatically generate immersive sound effects for 360° panoramic videos, enabling viewers to have a more realistic auditory experience while watching the video.
-
Smart Voice Assistants: Integrated into smart home devices, such as smart speakers, smart appliances, etc., to enable voice control and interaction. Users can control the on/off of home appliances, adjust temperature, query information, and more through voice commands.
-
Robotics and Autonomous Driving: OmniAudio can be applied in the fields of robotics and autonomous driving to provide these systems with more accurate sound localization and environmental perception.