SemanticAudio - An audio generation and editing framework launched by CUHK et al.

What is SemanticAudio

SemanticAudio is an audio generation and editing framework jointly launched by The Chinese University of Hong Kong, LIGHTSPEED, and Shanghai Jiao Tong University. The framework splits text-to-audio generation into two stages: "semantic planning" and "acoustic synthesis." It plans the identity, timing, and structure of sound events in a high-level semantic space, then renders them into high-quality audio. The framework supports training-free text-guided audio editing and significantly outperforms mainstream methods such as TangoFlux on the AudioCaps and TTABench benchmarks, achieving dual improvements in semantic alignment and generation quality.

Main features of SemanticAudio

• Text-to-audio generation: Input a natural language description to generate high-quality audio such as environmental sounds, action sounds, and complex sound scenes.
• Semantic planning generation: Plan the global layout of sound events in a high-level semantic space, then synthesize acoustic details.
• Training-free audio editing: Through the FlowEdit ODE mechanism, modify sound attributes directly in the semantic space, supporting operations such as replacement and adjustment.
• Frame-level semantic embeddings: Extract semantic representations that preserve temporal structure, accurately describing event order and local changes in complex audio.

Technical principles of SemanticAudio

• Two-stage Flow Matching architecture: SemanticAudio splits audio generation into two stages: semantic planning and acoustic synthesis. The Semantic Planner first generates a compact semantic representation from text, depicting the global layout of sound events. The Acoustic Synthesizer then uses this semantic plan as a condition to generate high-quality acoustic latents and decode them into audio. This decoupling gives the models clear division of labor and avoids the coupling problem in single-stage models that simultaneously handle semantic understanding and acoustic rendering in acoustic space.
• Construction and compression of the semantic space: Use the Perception Encoder to extract frame-level semantic embeddings to preserve temporal structure, then compress them to 128 dimensions through a lightweight MLP. The low-dimensional semantic space not only preserves key sound identity and timing information, but can also be efficiently learned by the Flow Matching model, becoming a key intermediate layer connecting text semantics and acoustic details.
• FlowEdit ODE: Use the velocity field difference between the source text and target text to guide semantic trajectory editing, encode the source audio into semantic latents, compute the velocity field difference of the two texts to determine the editing direction, execute ODE stepping to obtain the target semantic representation, and then restore it through the acoustic synthesizer. Editing is performed in the high-level semantic space, enabling attribute-level modification without additional training, inversion, or paired data.
• The core value of semantic-acoustic decoupling: Explicit decoupling enables the model to first plan the identities and timing of multiple events in semantic space, then synthesize acoustic details, avoiding missing events and ordering errors under complex prompts. The semantic space provides a clearer and more interpretable object of operation for editing, making modifications closer to "changing the sound content itself" and achieving more stable and flexible text-guided modification.

Follow WeChat and reply "open source" to jointhe AI Open Source Project Communication Group

How to use SemanticAudio

• Visit the Demo page: Open https://semanticaudio1.github.io/ to experience the online demo.
• Enter a text prompt: Describe the target sound scene, such as "a car horn sounds after a dog barks."
• Semantic planning: Semantic Planner generates a compact semantic representation and plans the layout of sound events.
• Acoustic Synthesis: Acoustic Synthesizer renders the semantic plan into high-quality audio.
• Audio Editing (Optional): Input source audio and target text, and use FlowEdit ODE to modify sound attributes in semantic space.

Core Advantages of SemanticAudio

• Significant improvement in semantic alignment: By first planning the semantic layout and then synthesizing acoustic details, AudioCaps LAION-CLAP reaches 0.381, outperforming TangoFlux's 0.361, with more accurate understanding of complex prompts.
• Editing without training: The FlowEdit ODE mechanism uses velocity field differences to guide trajectories in semantic space, supports attribute-level modifications, improves CLAP by +0.094, outperforms similar methods, and requires no additional training.
• Balancing generation quality and controllability: FD 19.1, MOS 3.72, maintaining high-fidelity perceptual quality while significantly improving text-audio semantic alignment.
• Strong handling of complex prompts: Explicitly plans multi-event identities and temporal relationships, effectively avoiding issues such as missing events, incorrect order, and insufficient text alignment.
• More in line with human creative workflows: Adopts the paradigm of first planning the structure of the sound scene and then supplementing acoustic details, realizing a generation approach of thinking clearly before speaking.

Comparison of SemanticAudio with Similar Competitors

Application Scenarios of SemanticAudio

• Film and Television Post-Production: Based on natural language descriptions in the script, SemanticAudio can generate complex sound effect scenes containing multiple sound events with precise timing, improving post-production efficiency.
• Game Development: For different game scenarios, developers can dynamically generate matching ambient sounds and action sound effects through text instructions, enabling rapid iteration of audio content.
• Content Creation: Creators of short videos, podcasts, and more can quickly obtain customized audio materials by entering descriptions, lowering the threshold for professional audio production.
• Virtual Reality: The system generates immersive ambient sounds in real time according to users' natural language instructions, enhancing the sense of presence and interactive experience in virtual spaces.
• Audio Editing Tools: Users can directly modify the attributes of existing audio or replace sound events using natural language, completing professional-level editing without training.