What is MuScriptor
MuScriptor is an open-source multi-instrument music transcription model jointly launched by Kyutai and Mirelo, capable of automatically transcribing real-world music audio across multiple genres into MIDI. MuScriptor is the first model of its kind trained on a large-scale real-world dataset covering 170,000 songs. It adopts a pure decoder-only Transformer architecture, offers four parameter scales from 60M to 1.4B, supports instrument conditioning, and its code is open-sourced under the MIT license.
MuScriptor's Main Features
- Multi-Instrument Audio Transcription: Automatically converts real-world music audio containing multiple instruments into standard MIDI format, supporting 36 instrument categories including piano, guitar, drums, and bass.
- Multi-Genre Music Support: Covers multiple music genres such as pop, classical, rock, and heavy metal, adapting to complex real-world mixing scenarios.
- Instrument Conditioning: Supports users specifying the target instrument set to be transcribed, enabling customized output and maintaining stable cross-segment instrument recognition consistency.
- Multiple Model Size Options: Offers four parameter scales: 60M, 103M, 307M, and 1.4B, balancing lightweight deployment and high-precision requirements.
- End-to-End Automatic Processing: Using 5-second audio slices as input units, the model automatically completes the entire process of framing, feature extraction, note detection, and MIDI generation.
MuScriptor's Technical Principles
- Pure Decoder-Only Transformer Architecture: The model takes 5-second audio slices as input, converts 16kHz mono audio into mel-spectrograms, and predicts MIDI-like token sequences in an autoregressive manner, achieving end-to-end multi-instrument transcription.
- Three-Stage Training Strategy: The model is first pre-trained on 1.5 million synthetic MIDI data, then fine-tuned on 170,000 real music tracks, and finally post-trained with 300 high-quality annotated data samples through GRPO-style reinforcement learning to align with high-quality output.
- Instrument Conditioning: The model supports specifying the target instrument set through prefix embeddings. Users can customize the instrument types to be transcribed, while classifier-free guidance stabilizes the consistency of instrument assignment across segments.
- Data Synthesis and Augmentation Pipeline: The pre-training stage employs a dynamic synthesis process, applying symbolic-level augmentations to MIDI such as pitch shifting, tempo changes, and instrument randomization, synthesizing audio with over 250 soundfonts, and adding random detuning to improve generalization ability.
Follow on WeChat and reply "Open Source" to join theAI Open-Source Project Exchange Group
How to Use MuScriptor
- Environment Preparation: Clone the GitHub repository and install dependencies, ensuring the system supports Python and deep learning frameworks such as PyTorch.
- Download Weights: Obtain the model weight files corresponding to the parameter scale (103M, 307M, or 1.4B) from official channels.
- Load the Model: Initialize the model with the provided inference code, and choose whether to enable instrument conditioning.
- Input Audio: Input the audio file to be transcribed (supporting multiple formats) into the model, and the model automatically splits it into 5-second segments for processing.
- Get MIDI: The model outputs standard MIDI files, which can be directly imported into a DAW or notation software for subsequent editing.
MuScriptor's Core Advantages
- Strongest Real-World Generalization: Compared with models trained on synthetic data (such as MT3), MuScriptor is trained on 170,000 real multi-instrument music tracks, significantly reducing error rates in complex real-world mixing scenarios. It is officially described as the best open-source multi-instrument transcription model to date.
- Reinforcement Learning Alignment for High-Quality Output: Through GRPO-style reinforcement learning in the post-training stage, the model achieves further improvements in F1 scores at the note onset, offset, and frame levels, reducing missed detections and false detections.
- Flexible Instrument Conditioning: Users can specify the types of instruments to transcribe, which streamlines the output and maintains consistency in instrument recognition across audio segments, avoiding fluctuations in instrument assignment between segments.
- Multiple Open-Source Configurations: Provides four model scales ranging from 60M to 1.4B, balancing edge device deployment and high-precision needs. The code is open-sourced under MIT, and the weights are available under CC BY-NC 4.0.
MuScriptor's Project Addresses
- GitHub Repository：https://github.com/muscriptor/muscriptor
- HuggingFace Model Library：https://huggingface.co/MuScriptor
- arXiv Technical Paper：https://arxiv.org/pdf/2607.08168v1
Comparison of MuScriptor with Similar Competing Products
| Comparison Dimension | MuScriptor | YourMT3+ |
|---|---|---|
| Publisher | Kyutai / Mirelo | Academic Research Team |
| Architecture | Pure Decoder Transformer | Hierarchical Attention Transformer + MoE |
| Training Data | 1.5 million synthetic MIDI + 170,000 real music tracks + 300 RL data samples | Small-scale synthetic data + limited real data |
| Core Strategy | Real data fine-tuning + reinforcement learning post-training | Architecture improvement + cross-dataset enhancement |
| Instrument Control | Supports explicit instrument conditioning | Not supported |
| Open-Source License | Code MIT / Weights CC BY-NC 4.0 | Partially open-source |
| Real Audio Performance | Multi F1 approximately 41.6, significantly leading | Surpassed as a baseline |
| Velocity Information | Tokenizer does not currently support it | Depends on the specific implementation |
MuScriptor's Application Scenarios
- Music Production Assistance: Producers can quickly convert existing audio without MIDI into editable MIDI tracks for rearrangement, mixing, or sampling.
- Automatic Score Generation: Music education institutions and publishers can automatically convert recordings into sheet music, reducing the cost of manual transcription.
- Music Information Retrieval: Provides high-quality symbolic input for downstream tasks such as chord recognition, key analysis, and style classification.
- Generative Music Modeling: Provides large-scale, high-quality MIDI training data for music generation models, advancing research in symbolic music generation.
- Music Archaeology and Archiving: Converts unstructured audio such as historical recordings and live performances into structured MIDI data, facilitating digital preservation and analysis.