Muse Voice Transcribe - Meta's first real-time audio perception model

What is Muse Voice Transcribe

Muse Voice Transcribe is Meta's first real-time audio perception model, integrating streaming automatic speech recognition, speaker diarization, and endpoint detection into one system. The model supports over 70 languages and native code-switching, can handle audio longer than 1 hour and simultaneous conversations with more than 20 people, and ranks first on both streaming transcription and speech segmentation benchmarks. Muse Voice Transcribe is based on reinforcement learning-based adaptive latency technology, which can dynamically balance speed and accuracy. Muse Voice Transcribe has been integrated into Meta AI and Muse Code, positioned as the real-time perception layer of AI systems.

Main features of Muse Voice Transcribe

Real-time streaming ASR: The model processes audio in real time in 80ms units and outputs text without waiting for the entire audio segment to end.
Speaker diarization: Automatically identifies and labels more than 20 different speakers, distinguishing who is speaking and when.
Voice endpoint detection: Accurately determines when speech starts and ends, supporting natural pauses and continuous conversation.
Multilingual and code-switching: Covers more than 70 languages and natively supports arbitrary code-switching such as mixed Chinese and English.
Long audio and multi-person meetings: Can continuously process more than 1 hour of audio and output a complete transcript without post-processing.

Technical principles of Muse Voice Transcribe

Autoregressive multimodal architecture: Based on the autoregressive model of the Muse Spark series, it converts the audio stream frame by frame into soft tokens and models them uniformly with text tokens, achieving end-to-end generation from audio to text.
Streaming audio token mechanism: Each 80ms audio segment is encoded as a soft token, and the model predicts <|next_audio|> to decide whether to continue listening to the next segment; upon receiving <|empty_audio|> it immediately outputs the remaining text, enabling true streaming inference.
Task-extended special tokens: On top of ASR, it introduces <|start_of_turn|> and <|speaker_{A-Z}|> to implement speaker-change recognition, and introduces <|speech_onset|> and <|speech_endpoint|> to implement endpoint detection; all tasks share the same streaming training framework.
Reinforcement learning adaptive latency: Through RL, the word error rate reward and the latency reward are combined multiplicatively, enabling the model to dynamically decide, based on the recognition difficulty of each word, whether to wait for more context or output immediately, achieving Pareto optimality between speed and accuracy.

How to use Muse Voice Transcribe

Choose an entry point: Access the service through the Meta Model API, the Mac version of Meta AI or Muse Code in three ways.
Start voice: Press ⌘ + v in Muse Code, or click the voice button in the Mac version of Meta AI to start dictation.
Cross-app use: Press and hold the Fn key in any Mac app to invoke Meta AI for speech transcription.
API Access: Pay-as-you-go via the Meta Model API ($3 per 1,000 minutes), integrated into your own app or workflow.

Core Advantages of Muse Voice Transcribe

Real-Time Streaming ASR: Processes audio in 80ms units, enabling true low-latency real-time transcription.
Multi-Speaker Diarization: Natively supports distinguishing more than 20 speakers, accurately attributing each sentence without post-processing.
Adaptive Latency: Dynamically adjusts the wait time for each word via reinforcement learning, achieving Pareto optimality between speed and accuracy.
Seamless Code-Switching: Natively supports arbitrary language switching within or between sentences, accurately recognizing mixed expressions such as Chinese-English code-mixing.
Ultra-Long Audio Processing: Supports continuous long audio input of more than 1 hour, suitable for long meetings or interview scenarios.
Voice Activity Detection: Automatically marks speech start (<|speech_onset|>) and end (<|speech_endpoint|>), accurately determining when the user has finished speaking.
Ranked First in Benchmarks: Ranked first in both the Artificial Analysis streaming speech-to-text and public speech diarization benchmarks.

Muse Voice Transcribe Project Address

Project Website：https://research.meta.ai/blog/introducing-muse-voice-transcribe

Comparison of Muse Voice Transcribe with Similar Competing Products

Comparison Dimension	Muse Voice Transcribe (Meta)	Gemini 3.1 Flash Live (Google)
Product Positioning	A dedicated real-time audio perception and ASR model	A general-purpose real-time multimodal dialogue model, with ASR being only one of its input capabilities
Core Capabilities	Streaming ASR, speaker diarization, voice activity detection	Real-time bidirectional voice dialogue (ASR + TTS), multimodality (audio/video/image/text), function calling
Word Error Rate	AA-WER Streaming Index 3.1%, ranked first in Artificial Analysis	Specific ASR WER figures have not been published; focuses on low-latency streaming ASR and dialogue fluency
Speaker Diarization	Natively supports 20+ speakers, with a DER of only 17.5%	Does not explicitly support multi-speaker diarization, focusing on single-turn/two-turn real-time dialogue
Latency Strategy	Adaptive latency, dynamically adjusting the wait time for each word via reinforcement learning	Sub-200ms Fixed first-token latency, pursuing extremely fast responses
Long audio processing	Native support for 1 hour+ continuous audio, no segmentation required	Pure audio sessions default to a 15-minutelimit; a session continuation mechanism is required
Code-switching	Native support for arbitrary code-switching within and between sentences, accurately recognizing mixed Chinese and English	Supports 90+ languages, but does not emphasize special optimization for code-switching
Pure transcription cost	$0.18/hour	Audio input $0.30/hour + text output approximately $0.07/hour, totaling approximately $0.37/hour
Output capabilities	Text transcription + speaker labels only (`<\|speaker_A\|>`）	text + audio synthesis (TTS) + tool calling + structured output + real-time search

Muse Voice Transcribe application scenarios

Real-time minutes for multinational meetings: Supports long meetings with 20+ simultaneous speakers and over 1 hour, automatically distinguishing speakers and accurately transcribing discussions that mix Chinese and English.
AI voice assistant/Agent: As a real-time perception layer, it provides low-latency streaming ASR and precise voice activity detection, letting the AI assistant know when the user starts and stops speaking.
Podcast and interview subtitle generation: Processes audio material lasting several hours, natively supports multi-speaker dialogue segmentation, and requires no manual speaker annotation afterward.
Multilingual customer service quality inspection: Transcribes mixed-language conversations between customer service and customers in real time, uses context preferences to improve recognition accuracy of specialized terminology, and automatically archives them.
Developer voice input tool : Globally invoked via the Fn key in the Mac version of Meta AI or Muse Code, enabling voice transcription for programming, writing documents, or sending messages in any application.