Magenta RealTime 2 - Google Magenta's Open-Source Real-Time Music Generation Model

What is Magenta RealTime 2

Magenta RealTime 2 (MRT2) is the second-generation open-source local real-timemusic generationmodel launched by Google's Magenta team. MRT2 is based on a frame-by-frame generation architecture, with each frame lasting 40 milliseconds, achieving actual control latency of approximately 200 milliseconds, allowing AI to react to input changes at a speed close to the experience of playing a real instrument. The model offers two versions: mrt2_base (2.4B parameters, high quality) and mrt2_small (230M parameters, high speed), both optimized for Apple Silicon, with the small model capable of running in real time on M1 and subsequent chips. MRT2 adopts an open-source weights strategy and is released alongside the free app Jam and DAW plugin, lowering the barrier to entry for music creators and developers.

Main Features of Magenta RealTime 2

• Low-Latency Real-Time Generation: Generates audio frame by frame, with each frame lasting 40 milliseconds and actual control latency of approximately 200 milliseconds, reduced to 1/15 of the previous generation's 3-second latency, enabling true real-time interaction.
• Multimodal Control Input: Supports text descriptions, audio style prompts, MIDI note input, and drum kit on/off control, allowing users to guide musical style and direction in real time through multiple methods.
• Continuous Note Following: The model can continuously follow changes in note input and adjust the generated content in real time, adapting to dynamic changes in the user's performance.
• Auto-Strum Intelligent Strumming: In Auto-Strum mode, the AI automatically decides the timing of strumming or note onset; when disabled, users can precisely specify the start time of notes, balancing automation and fine-grained control.
• Drum Track Toggle: Supports switching between output with or without drums, suitable for multi-track arrangement scenarios, flexibly adapting to different instrumentation needs.
• Dual-Scale Model Selection: Offers two specifications: mrt2_base (2.4B parameters, audio quality priority) and mrt2_small (230M parameters, speed priority), adapting to different hardware performance.
• Native Apple Silicon Optimization: Deeply optimized for M-series chips, mrt2_small can run in real time on M1 and subsequent models, while mrt2_base supports M2 Max and above.

Technical Principles of Magenta RealTime 2

• Codec Language Model Architecture: MRT2 belongs to the codec language model paradigm, whose core idea is to discretize continuous audio signals into predictable token sequences and generate them frame by frame through autoregression. The model is based on the SpectroStream audio codec to compress 48kHz stereo audio into a compact latent representation, outputting audio tokens at a frame rate of 25Hz. The discretization strategy greatly reduces sequence length and computational complexity, making real-time inference possible on consumer-grade hardware.
• Frame-by-Frame Autoregressive Generation Mechanism: Unlike the previous generation, which used 2-second audio block batch processing and had a latency of about 3 seconds, MRT2 adopts a frame-by-frame generation architecture. Each frame is only 40 milliseconds; after receiving the current input conditions (MIDI, text, audio style), the model immediately predicts the audio token distribution of the next frame and decodes the output. The streaming autoregressive approach compresses actual control latency to approximately 200 milliseconds, reduced to 1/15 of the previous generation, approaching the threshold of human perception for instrument response.
• SpectroStream Audio Codec: As the audio frontend and backend of MRT2, SpectroStream is responsible for encoding raw 48kHz stereo waveforms into discrete tokens that the model can process, and decoding them back into audible audio after generation. The codec is optimized for the high-frequency structure and stereo phase information of music signals, ensuring that timbre, spatiality, and harmonic detail are preserved even in a highly compressed latent space, providing a foundational guarantee for audio quality in real-time scenarios.

How to use Magenta RealTime 2

• Experience via the Jam app: Download the free Jam app, enter a music style description, and the model starts generating; adjust the pitch via the keyboard below, with support for mouse, MacBook keyboard, or MIDI keyboard input.
• Call in a DAW: Install the MRT2 plugin to directly call the model in common digital audio workstations (DAWs), integrating AI generation into existing arrangement workflows.
• Python library development: Developers run pip install magenta-rt to install the Python library, and integrate MRT2 into custom music applications or interactive installations via the API.
• Local on-device deployment: Use the C++/MLX inference engine to achieve fully offline local inference on Apple Silicon Macs, with no cloud dependency.

Core advantages of Magenta RealTime 2

• Extremely low latency: 200 ms latency brings AI response speed close to the threshold of human perception when playing an instrument, enabling truly "improvised jamming."
• Fully local operation: Optimized based on Apple Silicon and the MLX framework, all inference is completed locally with no network connection required, ensuring privacy and stable low latency.
• Open-source and open ecosystem: Open-source model weights, with three access methods: a Python library, a standalone app, and a DAW plugin, covering the full-chain needs from ordinary users to professional developers.
• Multi-dimensional real-time control: Simultaneously supports text, audio, MIDI, and drum kit toggle control, providing the richest interactive control dimensions currently available in the field of real-time generation.

Project addresses of Magenta RealTime 2

• Project official website：https://magenta.withgoogle.com/magenta-realtime-2
• GitHub repository：https://github.com/magenta/magenta-realtime
• HuggingFace model library：https://huggingface.co/google/magenta-realtime-2

Comparison of Magenta RealTime 2 with similar competitors

Application scenarios of Magenta RealTime 2

• Live improvisation: Musicians play together with MRT2 in real time via a MIDI keyboard; the AI generates accompaniment or responding phrases on the fly based on the played notes and style prompts, for strongly improvisational music scenarios such as jazz and electronic music.
• Real-time arrangement assistance: Load the MRT2 plugin in a DAW; when creators adjust chord progressions or style descriptions, the AI provides instant feedback on the arrangement effect, accelerating creative iteration.
• Interactive music installations: Developers use the open-source Python library and low-latency features to build interactive sound installations in museums, exhibitions, or on stage, turning audience movements into music in real time.
• Music education and practice: Students enter style descriptions through the Jam app, and the AI generates accompaniment in real time for solo practice, or simulates the ensemble experience under different band configurations.