PrismAudio - Alibaba Tongyi's Video-to-Audio Generation Framework

What is PrismAudio

PrismAudio is a Video-to-Audio framework launched by Alibaba Tongyi Lab, which can automatically add environmental sound effects to silent videos. The model pioneers the "decomposed chain-of-thought" technique, allowing the model to first think about sound content, timing, texture, and spatial position before generating audio, and introduces four "teachers" (semantics, timing, aesthetics, and space) for multi-dimensional scoring optimization. The model has only 518 million parameters and takes only 0.63 seconds to generate 9 seconds of audio, comprehensively surpassing existing methods in performance, and has now been accepted by ICLR 2026.

PrismAudio's Main Features

Video-to-Audio: The model can automatically generate environmental sound effects that match the visuals for silent videos (such as horse hooves, wind and rain, etc.).
Semantic Alignment: The model can ensure that the generated sound content accurately corresponds to the objects and actions in the video, avoiding audio-visual mismatch.
Temporal Synchronization: Supports precise control of the timing of sounds and visual events, achieving perfectly synchronized effects.
Aesthetic Optimization: The model generates high-quality audio that is natural, layered, and free of electronic artifacts, enhancing the listening experience.
Spatial Localization: Supports stereo output, automatically adjusting the left and right channels according to the position of the sound source in the frame, enabling sound localization.
Chain-of-Thought Reasoning: Adopts a decomposed chain-of-thought of "think first, then sound," making the generation process interpretable and controllable.

PrismAudio's Key Information and Usage Requirements

Developer: Alibaba Tongyi Lab (Tongyi Fun Team)
Technology Type: Video-to-Audio (V2A) framework
Core Innovation: Decomposed chain-of-thought + multi-dimensional reinforcement learning
Model Scale: 518 million parameters
Output Specification: 44kHz stereo
Inference Speed: Only 0.63 seconds to generate 9 seconds of audio
Input Format: Silent video (supports common video formats)
Content Restriction: Only generates environmental sound/sound effects, does not support character dubbing
Optional Input: Can be paired with text descriptions to assist generation (not required)
Hardware Requirements: Supports GPU acceleration, can also run on CPU

PrismAudio's Core Advantages

Four-Dimensional Collaborative Optimization: Pioneers the decomposed chain-of-thought, independently modeling and collaboratively optimizing the four dimensions of semantics, timing, aesthetics, and space, avoiding the drawbacks of traditional models "favoring one over the other," and achieving high audio-visual unity.
Think First, Then Sound: Breaks through the end-to-end black-box generation mode; the model first outputs structured reasoning text (sound content, timing, texture, direction), then generates audio, making the process interpretable and controllable.
Efficient and Lightweight: Only 518 million parameters, generating 9 seconds of audio in just 0.63 seconds, nearly twice as fast as similar models, making it more suitable for real-time application scenarios.
Robust in Complex Scenarios: Performs far better than existing methods on the self-built AudioCanvas complex scenario benchmark, maintaining stable output even in multi-event, multi-sound-source scenarios.

How to Use PrismAudio

Online Experience (Recommended for Beginners): Visit the Hugging Face online experience Demo, upload a silent video, optionally input a text description to assist generation, and AI automatically generates an audio file.
Local Deployment: Download the open-source code and model weights from GitHub or Hugging Face, install the dependency environment, then load the pretrained model, input the video path to call the inference interface to generate audio, and support custom adjustment of chain-of-thought parameters or reward weights.

PrismAudio's project address

Project Official Website：https://prismaudio-project.github.io/
GitHub Repository：https://github.com/FunAudioLLM/ThinkSound/tree/prismaudio
HuggingFace Model Library：https://huggingface.co/FunAudioLLM/PrismAudio
arXiv Technical Paper：https://arxiv.org/pdf/2511.18833
Online Experience Demo：https://huggingface.co/spaces/FunAudioLLM/PrismAudio

Comparison of PrismAudio with Similar Competing Products

Comparison Dimension	PrismAudio	MMAudio	ThinkSound
Developer	Alibaba Tongyi Lab	Nanyang Technological University, Singapore, etc.	Alibaba Tongyi Lab
Core Method	Decomposed Chain-of-Thought + Multi-dimensional Reinforcement Learning	Multimodal Transformer	Monolithic Chain-of-Thought
Parameter Count	518 million	About 1 billion	Billions
Inference Speed	0.63 seconds/9 seconds of audio	1.30 seconds/9 seconds of audio	1.07 seconds/9 seconds of audio
Output Audio Quality	44kHz stereo	44kHz mono	44kHz stereo
Semantic Consistency (CLAP)	0.47	0.40	0.43
Temporal Synchronization (DeSync)	0.41	0.46	0.55
Spatial Accuracy (CRW)	7.72	—	13.47
Aesthetic Quality (MOS-Q)	4.21	3.95	4.05

PrismAudio's Application Scenarios

Film and Television Post-production: Automatically generate ambient sound effects for movies, documentaries, and trailers, replacing traditional Foley work and reducing post-production cost and time.
Short Video Creation: Quickly add atmospheric sound to silent videos such as Vlogs, food, and travel, enhancing the immersion and传播 effect of ASMR and healing content.
Game Development: Generate dynamic sound effects for cutscenes and CG promotional videos, matching ambient sound in real time according to scenes such as forests, cities, and battlefields, reducing repetitive work for sound designers.
Advertising and Marketing: Automatically add operation sound effects to product display videos, support rapid iteration of multiple versions of audio tracks, and improve advertising testing efficiency and creative flexibility.
Education and Training: Supplement teaching videos and operation demonstrations with prompt sounds and background sounds, enriching the auditory experience of multimedia courseware and improving learning focus and information absorption.