SALMONN-2 - A General-Purpose Audio Large Language Model Open-Sourced by Tsinghua University and Others

What is SALMONN-2

SALMONN-2 is a general-purpose audio large language model jointly open-sourced by Tsinghua University, Shanghai AI Laboratory, and the University of Cambridge. It is built on ByteDance's SPEAR unified self-supervised audio encoder and a multi-layer feature fusion adapter, with Qwen3 as the text backbone. Using about 18,000 hours of supervised data, it achieves the best performance among open-source models of the same scale on the three comprehensive benchmarks MMAU-Pro, MMAR, and MMSU. It is the first to systematically realize advanced capabilities in a general-purpose ALLM, including sound event detection, audio forgery detection, speech quality assessment, and multimodal in-context speech recognition.

Main Features of SALMONN-2

• General Audio Understanding: Supports cross-domain tasks such as speech recognition, audio captioning, music understanding, emotion recognition, and speaker verification.
• Sound Event Detection (SED): Can locate and output the start and end time segments of environmental sound events, such as dog barking and footsteps.
• Audio Deepfake Detection: Determines whether speech is authentic and locates the time intervals of local segments suspected of being synthesized or edited.
• Speech Quality Assessment (SQA): Perceives low-level acoustic features such as noise, distortion, and clarity, and provides quality scores and explanations.
• Multimodal In-Context Speech Recognition (MICL): Combines text spelling and reference pronunciation audio to improve recognition accuracy for uncommon words and rare personal names.

Technical Principles of SALMONN-2

• Unified Self-Supervised Audio Encoder: SALMONN-2 uses SPEAR as a single audio frontend. The encoder is trained through self-supervised learning on large-scale unlabeled audio data and can simultaneously capture semantics, pronunciation, timbre, speaker, and acoustic environment information. It replaces the traditional Whisper+BEATs dual-encoder scheme, simplifying the architecture while maintaining more balanced cross-domain capabilities.
• Multi-Layer Feature Fusion (MLF) Adapter: The information encoded at different layers of SPEAR has different levels of hierarchy—shallow layers retain acoustic/pronunciation details, middle layers carry timbre/speaker information, and deep layers accumulate semantic concepts. The MLF adapter first performs layer normalization and concatenation on the hidden states of each layer, then applies learnable down-projection and frame grouping compression, and finally maps the fused hierarchical representation into the language model embedding space, enabling the model to flexibly invoke acoustic cues at different levels according to task requirements.
• Timestamp Injection Mechanism: The model inserts timestamps directly into the audio sequence in the form of natural language text (e.g., <2.0 seconds>), interleaved with audio embeddings as input to the language model, allowing the model to complete temporal localization tasks within a unified generation framework without an additional timestamp-specific embedding layer.
• Multimodal In-Context Learning (MICL) Training: In the in-context speech recognition task, the model not only receives a text bias word list but also simultaneously obtains the reference pronunciation audio of these words as multimodal context. During training, a strategy of randomly discarding distractor words and target words is introduced to prevent over-biasing, enabling the model to learn to reasonably use contextual cues with support from acoustic evidence.

Follow WeChat and reply “Open Source” to join theAI Open Source Project Communication Group

How to Use SALMONN-2

• Visit the repository and clone the code: Visit the GitHub repository https://github.com/bytedance/SALMONN/tree/salmonn2, and use git clone to download the code locally.
• Create a runtime environment: Execute conda create -n salmonn2 python=3.10 Create and activate a virtual environment, then upgrade pip, setuptools, and wheel.
• Install dependencies and the project: Run in the repository root pip install -r requirements.txt and pip install -e . --no-depsto complete the installation of SALMONN-2 and its runtime dependencies.
• Download pretrained weights: Download the model checkpoint via the HuggingFace CLI:hf download marcoyang/SALMONN-2-8B --repo-type model --local-dir /path/to/salmonn-2-hf。
• Load the model and run inference: Use AutoProcessor and AutoModelForCausalLM to load the local weights, pass in the audio file and instruction text, and call model.generate() to obtain the audio understanding result.

Core Advantages of SALMONN-2

• Data-efficient: Uses about 18,000 hours of supervised data, far less than the hundreds of thousands to millions of hours commonly used by competitors of the same scale, significantly reducing annotation costs.
• More balanced unified front end: A single SPEAR self-supervised encoder replaces dual encoders, achieving more balanced performance across speech, environmental sound, music, and paralinguistic tasks.
• Full utilization of hierarchical information: The MLF adapter fuses features from all encoder layers, avoiding the loss of key details such as acoustics and timbre that occurs when only the last layer is used.
• Expanded audio analysis capabilities: In a general ALLM, it is the first to systematically support previously overlooked sound analysis tasks such as SED, forgery detection, and SQA.
• Scalable: After scaling the text backbone from 8B to 30B-A3B, scores on all three comprehensive benchmarks continue to improve, verifying that the architecture has good scale-up potential.

SALMONN-2 Project Address

• GitHub repository：https://github.com/bytedance/SALMONN/tree/salmonn2
• HuggingFace model library：https://huggingface.co/marcoyang/SALMONN-2-8B
• arXiv technical paper：https://arxiv.org/pdf/2607.17079

Comparison of SALMONN-2 with Similar Competitors

SALMONN-2 Application Scenarios

• Intelligent meeting assistant: Transcribes meeting content in real time and accurately recognizes foreign names and technical terms by combining participants' pronunciation examples.
• Audio content moderation: Automatically detects abnormal sound events in livestreams or podcasts and identifies deepfake speech to prevent fraud.
• Speech quality monitoring: Automatically scores the quality of customer service calls and recording studio material, locating noise and distortion segments.
• Multimedia archive retrieval: Generates timestamped event descriptions for historical audio and radio programs, enabling precise content-based retrieval.
• Assistive hearing devices: Alerts hearing-impaired users in real time to environmental sound events (such as doorbells and alarms) in the form of text and timestamps.