Stable Audio 3 - Stability AI's Open-Source Audio Generation Model Series

What is Stable Audio 3

Stable Audio 3 is a new-generation open-source audio generation model series launched by Stability AI, based on a flow-matching latent diffusion architecture, supporting text-to-music and sound effects, audio editing, and continuation. Stable Audio 3 offers Small, Medium, and Large multi-scale models. The Small version can run locally on a MacBook Pro and generate audio up to 2 minutes long, while Medium and Large support high-quality generation of over 6 minutes. All models in the series are trained on fully licensed data, with Small and Medium weights open-sourced, supporting LoRA fine-tuning and achieving fast inference in under 2 seconds.

Main Features of Stable Audio 3

• Text-to-Audio Generation: Users can input English text prompts to generate instrumental music or ambient sound effects, with precise control over output duration down to the second.
• Variable-Length Audio Synthesis: The model allocates latent sequence length proportionally to the requested duration, avoiding computational and memory waste caused by a fixed maximum length.
• Local Audio Editing and Repair: Achieves local audio inpainting through single or multiple masks, replacing target regions while preserving the original segments.
• Intelligent Audio Continuation and Extension: Uses a causal masking mechanism to coherently continue existing audio, extending short recordings into complete works of over 6 minutes.
• LoRA Style Fine-Tuning: For the first time, LoRA training documentation and Small and Medium weights are open-sourced, supporting users to efficiently adapt custom styles using their own audio libraries.
• Full Local Deployment: 3.0 Small supports fully offline operation on consumer-grade devices such as MacBook Pro, enabling privacy-preserving creation without network access.

Technical Principles of Stable Audio 3

• Semantic-Acoustic Autoencoder: Based on the SAME architecture, it maps 44.1kHz stereo to a 256-dimensional latent space through 4096x downsampling, balancing high-fidelity reconstruction and semantic structure encoding.
• Flow-Matching Latent Diffusion: Adopts the flow-matching training paradigm to perform diffusion generation in a compact latent space, combined with mini-batch optimal transport coupling to improve training stability and efficiency.
• Adversarial Post-Training Acceleration: Introduces adversarial post-training after pretraining and ODE distillation warm-up, compressing inference steps to an extremely low level, achieving generation of long audio in under 2 seconds on H200.
• Differential Attention Transformer: The diffusion Transformer integrates differential attention, adaptive layer normalization condition injection, and memory embeddings to enhance modeling accuracy for long-sequence audio.
• Variable-Length Inference Mechanism: Breaks through the fixed sequence length limitation of traditional diffusion models, making the latent space length proportional to the actual requested duration, significantly reducing the cost of short audio generation.

How to Use Stable Audio 3

• Obtain Model Weights: Visit the Stability-AI/stable-audio-3 repository on Hugging Face and download the weight files for 3.0 Small, Small SFX, or Medium.
• Configure Runtime Environment: Clone the stable-audio-tools code repository and install dependencies, ensuring PyTorch and CUDA or Apple Metal backend are correctly configured.
• Load Model and Encoder: In the script, instantiate the SAME autoencoder and the diffusion Transformer of the corresponding scale, and load the pretrained weights into GPU memory.
• Write Generation Prompts: Input an English prompt describing the target audio style, instruments, and mood, and set the precise output duration (in seconds).
• Execute inference generation: Call the model generation interface; the model will perform variable-length inference based on the duration parameter and output a 44.1kHz stereo WAV file.

Core advantages of Stable Audio 3

• Fully licensed and commercially friendly: All models are trained on licensed and Creative Commons data, and the Community License allows users to freely own and commercially use generated content.
• Native support on consumer-grade hardware: 3.0 Small is open-weight and can run offline on a MacBook Pro M4, making it the first lightweight model to support full-song composition locally.
• Ultra-long high-quality generation: The Medium and Large models support generation of high-quality audio longer than 6 minutes, an order-of-magnitude breakthrough over the 47-second limit of the previous generation Stable Audio Open.
• Extreme inference efficiency: After adversarial post-training optimization, the Large model takes less than 2 seconds to generate 6 minutes and 20 seconds of audio on an H200 GPU, and only a few seconds on a MacBook Pro.
• Zero-annotation flexible editing: Without additional training data annotation, single-segment editing, multi-segment editing, and continuation can be achieved through random and causal masking, integrating directly into real creative workflows.

Project addresses for Stable Audio 3

• Project official website：https://stability.ai/news-updates/meet-stable-audio-3-the-model-family-built-for-artistic-experimentation-with-open-weight-models
• GitHub repository：https://github.com/Stability-AI/stable-audio-3
• HuggingFace model library：https://huggingface.co/collections/stabilityai/stable-audio-3
• arXiv technical paper：https://arxiv.org/pdf/2605.17991

Comparison of Stable Audio 3 with similar competitors

Application scenarios of Stable Audio 3

• Game and film/TV sound effects: Quickly generate interactive sound effects, ambient atmosphere sounds, and background music, and precisely match the rhythm and emotional needs of the visuals through local editing.
• Short video and advertising music: Generate customized instrumental clips according to precise durations, avoiding manual trimming, and directly adapt them to short videos, promotional films, and podcast transitions.
• Music creation assistance: Help musicians generate motivic inspiration, expand unfinished drafts, or replace local sections of songs, significantly accelerating the arrangement iteration cycle.
• Local privacy-sensitive creation: 3.0 Small supports fully offline operation, meeting the strict requirements of film and television studios and independent musicians for data privacy and network isolation.
• Personalized brand sound: Through LoRA fine-tuning of the enterprise's own audio assets, create consistent UI sound effects, brand notification sounds, and exclusive music styles.