FireRedAudio - A General-Purpose Audio Language Model Launched by Xiaohongshu FireRed

What is FireRedAudio

FireRedAudio is a general-purpose audio language model launched by the Xiaohongshu FireRed team, based on a 9B-parameter Qwen3.5 autoregressive LLM. The model uses independent continuous representation channels for understanding and generation respectively (Audio Encoder and RedAE), which then feed into a shared LLM for unified inference. The model simultaneously supports ASR, audio question answering, understanding of recordings up to 1 hour long, Zero-shot TTS, instruction TTS, and speech editing, and has achieved leading performance in evaluations such as MMAU, MMSU, multilingual ASR, and Instruct TTS.

Main Features of FireRedAudio

• Audio Understanding: Achieves the best results in the table on comprehensive audio question answering evaluations such as MMAU and MMSU, covering understanding of speech, music, and environmental sounds.
• Multilingual ASR: Supports recognition of 102 languages, with an average error rate of 14.94% on FLEURS-102, showing particularly outstanding performance on low-resource languages.
• Zero-shot TTS: Given any reference speech, it can synthesize new content, with Chinese and English content accuracy leading in the table.
• Instruct TTS: Follows natural language instructions such as "read at a slower speaking rate," leading on all six evaluation metrics.
• Speech Editing: Supports semantic-level deletion/insertion/replacement, as well as acoustic-level speed/pitch/volume changes.
• Long Audio Processing: Supports recordings up to 1 hour long and can generate timeline-structured summaries with second-level precision.

Technical Principles of FireRedAudio

• Decoupled Continuous Representations: FireRedAudio does not force understanding and generation to share the same audio representation, instead designing independent continuous vector channels for each: the understanding side needs compact semantic features to support long-context modeling, while the generation side needs reconstructable fine-grained acoustic features to preserve timbre and prosody; the two complete unified inference within a shared LLM.
• Understanding Path: Audio is encoded by an Audio Encoder initialized from Whisper-large-v3, compressed by an Adapter into 12.5 continuous representations per second, and then fed into the 9B LLM, which directly outputs recognized text or understanding answers, adapting to ASR, audio question answering, and long-audio structuring tasks.
• Generation Path: Conditional audio is first compressed by RedAE (a deterministic autoencoder) into 25Hz, 64-dimensional continuous latents; during pretraining, semantic distillation is performed through a frozen teacher encoder to preserve content cues; a Patch Encoder aggregates every 4 frames into one audio step (6.25Hz) before sending it to the LLM; for each audio-step token the LLM predicts, its hidden state drives a flow-matching DiT to generate new latents, which are finally restored by the RedAE Decoder into 24kHz waveforms.
• Five-Stage Progressive Training: Training successively completes Adapter alignment, Audio Encoder adaptation, joint training of understanding and generation, multi-task post-training, and 200k long-context extension, accumulating approximately 2.66T non-padding multimodal tokens, progressively stacking capabilities rather than mixing all tasks at once.
• Shared Inference Hub: The 9B Qwen3.5 autoregressive LLM serves as the only trainable backbone, responsible for cross-modal reasoning, task planning, and context modeling; understanding and generation are decoupled only at the input representation layer and unified at the inference layer, enabling the model to both "understand" and "speak" without sacrificing information fidelity on either side.

Follow on WeChat and reply "Open Source" to join theAI Open Source Project Communication Group

How to Use FireRedAudio

• Environment Setup: Read the environment and dependency installation instructions from the GitHub repository, and prepare the runtime environment.
• Obtain weights: Download the model weights and configuration files from the official Hugging Face page.
• Run the example: First run the minimal inference example provided officially to verify the model version, audio sampling rate, and output path.
• Verify understanding: Test ASR and audio question answering separately to confirm that the understanding path works properly.
• Verify generation: Test Zero-shot TTS and Instruct TTS separately to confirm that the generation path works properly.
• Audio format: In generation tasks, keep the input and output at a 24 kHz sampling rate, and record the number of DiT sampling steps and peak VRAM usage.
• TTS settings: For Zero-shot TTS, initialize the acoustic history with the last two audio steps of the reference speech; other generation tasks use zero initialization.
• Hardware evaluation: Evaluate VRAM requirements based on the 9B backbone weights, KV cache, and DiT sampling overhead; long-audio tasks require additional reserved context cache space.