What Is ACE-Step 1.5
ACE-Step 1.5 is an open-sourcemusic generationfoundation model jointly launched by ACE Studio and StepFun, capable of achieving commercial-grade music generation on consumer-grade hardware. The model adopts a hybrid architecture, with a language model serving as the planner that transforms user prompts into song blueprints, and a Diffusion Transformer responsible for acoustic rendering. Through 4-8 step distilled inference, generating a 4-minute song takes only 2 seconds on an A100 and about 10 seconds on an RTX 3090, with VRAM requirements below 4GB. ACE-Step 1.5 supports 50+ languages, precise style control, and editing functions such as cover generation, repainting, and vocal-to-instrumental conversion. Users can train LoRA on a small number of songs to achieve personalized style transfer.
Main Features of ACE-Step 1.5
- Music Generation: Supports generating complete songs from text prompts, covering lyric singing in more than 50 languages, and can flexibly extend to music of any duration from 10-second short loops to 10-minute long-form works.
- Editing Features: Provides six major editing capabilities: audio repainting, cover generation, vocal-to-instrumental conversion, stem separation, layered arrangement, and continuation completion, enabling fine-grained manipulation and re-creation of existing audio.
- Style Control: Can accurately parse and execute complex prompts containing professional music terminology, achieving zero-shot timbre cloning and strict style adherence.
- Personalization: Users only need to provide a small number of reference songs to perform lightweight LoRA fine-tuning and quickly train a customized model that captures their unique personal style.
- Efficiency Features: The model can run locally on consumer-grade GPUs with less than 4GB of VRAM, achieving sub-second high-speed generation and supporting batch parallel sampling to explore diverse creative candidates.
Technical Principles of ACE-Step 1.5
- Hybrid Reasoning-Diffusion Architecture: ACE-Step 1.5 adopts a two-component collaborative architecture that decouples music generation into two stages: planning and rendering. The language model (based on Qwen3-0.6B) acts as a "composer agent," using chain-of-thought reasoning to transform user prompts into a YAML-format blueprint containing BPM, key, duration, lyrics, and acoustic descriptions; the Diffusion Transformer (about 2 billion parameters) acts as the acoustic renderer, receiving standardized conditions and focusing on generating high-fidelity audio. This division of labor frees the DiT from the burden of semantic understanding, while the LM's multitask training ensures robust alignment across more than 50 languages.
- Efficient Inference Optimization: To achieve real-time generation on consumer-grade hardware, the team introduced adversarial dynamic shift distillation technology. Using Decoupled DMD2 as the foundation, it introduces a GAN objective and a latent-space discriminator. By randomly sampling shift parameters from {1,2,3}, the model is exposed to diverse denoising states, avoiding overfitting caused by fixed step sizes. This approach compresses inference steps from 50 to 4-8, requiring only about 1 second to generate a 240-second audio track on an A100, achieving a 200x speedup, and the adversarial feedback helps the student model surpass the teacher's audio quality performance.
- Intrinsic Reinforcement Learning Alignment: The system establishes a unified intrinsic reinforcement learning framework to avoid external bias. For the DiT, an Attention Alignment Score (AAS) is proposed as an intrinsic reward, measuring lyric token coverage, attention monotonicity, and path confidence through dynamic time warping. After optimization, lyric-audio synchronization correlates with human judgment by over 95%. For the LM, the GRPO algorithm is adopted, using pointwise mutual information to build a reward model, treating the LM as both "composer" and "listener." PMI penalizes generic descriptions and rewards specific annotations, and the final reward is dynamically weighted as 50% style and atmosphere, 30% lyric content, and 20% metadata constraints.
- Unified Mask Generation Framework: Continuous audio latents are discretized into a 5Hz codebook representation via Finite Scalar Quantization (FSQ), building a flexible mask generation paradigm. By manipulating the source latent and mask configuration, a single model can support six modalities: text-to-music, cover, repaint, track extraction, stacking, and completion. FSQ compresses the 25Hz latent space into a structured source latent via attention pooling, which is concatenated with the noisy target and mask before being processed by the patchify layer. The unified representation simplifies multi-task training and ensures high-fidelity preservation of melodic and rhythmic elements during transformation through quantized latents.
Project Address for ACE-Step 1.5
- Project Website：https://ace-step.github.io/ace-step-v1.5.github.io/
- GitHub Repository：https://github.com/ace-step/ACE-Step-1.5
- arXiv Technical Paper：https://arxiv.org/pdf/2602.00744
- Online Demo：https://huggingface.co/spaces/ACE-Step/Ace-Step-v1.5
Application Scenarios of ACE-Step 1.5
- Music Creation and Production: Musicians and producers can use ACE-Step 1.5 as an inspiration generation tool to quickly turn text descriptions into complete song drafts, breaking through creative bottlenecks.
- Personalized Content Creation: Content creators can train personal style models through LoRA fine-tuning to generate customized background music in batches for projects such as videos, podcasts, and games, maintaining timbre consistency across works.
- Multilingual Music Production: The model supports accurate lyric generation and singing in more than 50 languages, making it suitable for global music distribution, cross-cultural collaboration projects, and content production for minority-language music markets.
- Education and Learning: Music learners can input professional terms (such as specific modes and chord progressions) and observe the model's generated results, intuitively understanding music theory concepts.