What is AudioStory
AudioStory is an audio generation technology released by Tencent ARC Lab, capable of generating high-quality long-form narrative audio based on natural language descriptions. It adopts a divide-and-conquer strategy, breaking down complex narrative requests into ordered subtasks, and through a decoupled bridging mechanism, precisely coordinates semantics and sound effect details. The end-to-end training approach enhances model collaboration, and the generated audio has temporal logic and emotional layering.
Main features of AudioStory
-
Automatic video dubbing: Users upload silent videos and describe the sound effect style, and AudioStory can automatically analyze the video content and generate a synchronized background audio track with a consistent style.
-
Intelligent audio continuation: Given a segment of audio, AudioStory can intelligently infer subsequent scenes and automatically supplement reasonable audio continuations, such as adding player footsteps and basketball bouncing sounds based on the coach's voice in basketball training.
-
Audiobook creation: Provides high-quality audio content for audiobooks, generating audio with temporal logic and emotional layering based on text descriptions, allowing listeners to be better immersed in the story.
-
Game sound effect production: Generates immersive sound effects for games, creating matching audio based on game scene descriptions to enhance the player's gaming experience.
-
Intelligent podcasting: Helps podcast creators quickly generate audio content, producing corresponding audio clips based on topic descriptions to improve creation efficiency.
Technical principles of AudioStory
-
Divide-and-conquer strategy: Breaks down complex narrative requests into sequential subtasks, generates corresponding audio clips separately, and then precisely arranges them along the timeline to ensure the coherence and logic of the overall audio.
-
Decoupled bridging mechanism: Decomposes the collaboration between the large language model and the audio generator into two components, bridge query and residual query, which are used respectively for intra-event semantic alignment and cross-event consistency preservation, improving generation results.
-
End-to-end training: Adopts a unified training approach, simultaneously optimizing both instruction understanding and audio generation, enhancing collaboration among model components and improving overall performance.
-
Dual-channel mechanism of semantic tokens and residual tokens: Processes macro-narrative and micro-level sound effect details through two channels respectively, precisely coordinating the relationship between the two, so that the generated audio both conforms to the overall narrative logic and possesses rich detailed expression.
-
Three-stage progressive training: From single-sound generation to audio collaboration and then to long-form narration, gradually improves the model's performance and adaptability, enabling it to better handle complex long-form narrative audio generation tasks.
AudioStory project address
-
Github repository：https://github.com/TencentARC/AudioStory。
-
Paper address：https://arxiv.org/pdf/2508.20088。
Application scenarios of AudioStory
-
Video dubbing: Based on the silent video and sound effect style description provided by the user, automatically analyzes the video content and generates a matching background audio track.
-
Audio continuation: Based on a given audio clip, infers subsequent scenes and supplements reasonable audio continuations, such as adding player footsteps to basketball training audio.
-
Audiobook creation: Generates audio with temporal logic and emotional layering based on text descriptions, improving the auditory experience of audiobooks.
-
Game sound effect generation: Generate immersive sound effects based on game scene descriptions to enhance the player's gaming experience.