Overview
VibeVoice is Microsoft's open-source family of voice models (MIT licence). As of October 2026 the repository ships:
- VibeVoice-ASR (7B): transcribes up to 60 minutes of audio in one pass and returns who spoke, when, and what. Over 50 languages, no language flag needed, custom hotwords supported.
- VibeVoice-ASR-Streaming (7B): emits text chunk by chunk while audio is still arriving, with hotwords and 10 languages (released September 2026).
- VibeVoice-ASR-BitNet: a quantized CPU build (about 1.6 GB, run through the separate
microsoft/VibeASR.cppengine), no GPU needed. - VibeVoice-Realtime-0.5B: single-speaker streaming TTS, roughly 200-300 ms to first audio, about 10 minutes of speech per generation. English is the supported language; nine other languages are experimental.
- VibeVoice-TTS-1.5B: long-form multi-speaker TTS (up to 90 minutes, 4 speakers). The weights are on Hugging Face, but Microsoft removed the TTS inference code from the repository in September 2025 after misuse, so the repo cannot run it today.
Important: there is no VibeVoiceTTS or VibeVoiceASR Python class and no pip install vibevoice release from Microsoft. The PyPI package of that name is not the Microsoft project. Install from the GitHub clone, or use the Hugging Face Transformers integration for ASR.
Instructions
Speech recognition with Transformers (simplest path)
VibeVoice-ASR is in Transformers 5.3 and later. The repo itself pins transformers<5, so keep the two in separate virtual environments.
python -m venv .venv-asr && source .venv-asr/bin/activate
pip install "transformers>=5.3.0" accelerate librosa
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
inputs = processor.apply_transcription_request(
audio="team_standup.wav",
prompt="Participants: Priya Raman, Tomasz Wozniak. Terms: Kubernetes, Grafana",
).to(model.device, model.dtype)
output_ids = model.generate(**inputs)
generated = output_ids[:, inputs["input_ids"].shape[1]:]
segments = processor.decode(generated, return_format="parsed")[0]
for s in segments:
print(f"[{s['Start']:.1f}-{s['End']:.1f}s] Speaker {s['Speaker']}: {s['Content']}")
return_formatis"parsed"(list of dicts withStart,End,Speaker,Content),"transcription_only"(plain text) or raw. If the model output is not valid JSON, the raw string comes back unchanged.- The
promptargument is the context/hotword channel. Pass a list of audio files and a list of prompts of equal length for batches (useNonefor no prompt). - If you run out of memory, the model processes audio in 60-second tokenizer chunks; the model card describes adjusting the chunk size.
Running from the repository
git clone https://github.com/microsoft/VibeVoice.git && cd VibeVoice
pip install -e . # realtime TTS needs: pip install -e .[streamingtts]
apt install ffmpeg # the demos use it to decode audio
python demo/vibevoice_asr_inference_from_file.py \
--model_path microsoft/VibeVoice-ASR --audio_files team_standup.wav
python demo/vibevoice_asr_gradio_demo.py --model_path microsoft/VibeVoice-ASR --share
NVIDIA's PyTorch container (24.07 to 25.12) is the verified environment; install flash-attn if the container lacks it.
Streaming recognition
python demo/vibevoice_asr_streaming_inference_from_file.py \
--model_path microsoft/VibeVoice-ASR-Streaming-7B --audio_files call.wav \
--context_info "Microsoft,VibeVoice"
python demo/vibevoice_asr_streaming_fastapi_demo.py --model_path microsoft/VibeVoice-ASR-Streaming-7B
The FastAPI demo serves a page on http://localhost:7870 that keeps a WebSocket open while you speak. Chunk size is read from the checkpoint's preprocessor_config.json.
Real-time TTS
bash demo/download_experimental_voices.sh # optional extra voices
python demo/realtime_model_inference_from_file.py \
--model_path microsoft/VibeVoice-Realtime-0.5B \
--txt_path demo/text_examples/1p_vibevoice.txt --speaker_name Carter
python demo/vibevoice_realtime_demo.py --model_path microsoft/VibeVoice-Realtime-0.5B # websocket demo
Voices are shipped as embedded prompts; custom voice cloning is not offered by Microsoft for this model.
Serving ASR with vLLM
docs/vibevoice-vllm-asr.md describes a plugin with an OpenAI-compatible /v1/chat/completions endpoint. From a clone of the repo:
docker run -d --gpus all --name vibevoice-vllm --ipc=host -p 127.0.0.1:8000:8000 \
-v $(pwd):/app -w /app --entrypoint bash vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py"
docker exec -it vibevoice-vllm python3 vllm_plugin/tests/test_api.py /app/audio.wav --hotwords "Microsoft,VibeVoice"
Add --dp N for N replicas or --tp N to split one model over N GPUs. Audio files must sit inside the mounted directory.
Examples
Example 1: Meeting transcript with speaker labels
Request: "Transcribe standup.wav and tell me who said what, and make sure Grafana and Kubernetes are spelled right."
Use the Transformers snippet above with prompt="Terms: Grafana, Kubernetes" and return_format="parsed". Result: a list of segments such as [0.0-3.4s] Speaker 0: Let's review the Q3 numbers. Speaker ids are numbers, not names; map them yourself afterwards.
Example 2: Live captions for a call
Request: "I want text on screen while the customer is still talking."
Start demo/vibevoice_asr_streaming_fastapi_demo.py with the streaming checkpoint, open http://localhost:7870, and record from the microphone. Text arrives once per chunk over the WebSocket. For a file test, run the ..._streaming_inference_from_file.py script; each chunk prints as it is emitted.
Guidelines
- Hardware: the 7B ASR models need a large GPU (the repo does not publish a minimum; plan for 16 GB or more). The 0.5B realtime model runs on a T4 or an Apple M4 Pro in real time according to the docs.
- The realtime TTS model is English-focused, single speaker, and unstable for inputs of three words or fewer. It cannot read code, formulas or unusual symbols; normalize the text first.
- Microsoft states the models are for research and development and does not recommend commercial use without further testing. Disclose AI-generated audio and never use it to impersonate anyone; this is why the TTS code was withdrawn.
- ASR can fall into repetition loops on very long audio; the vLLM plugin ships
test_api_auto_recover.pyfor that. - Do not copy API calls from older tutorials (
VibeVoiceTTS.from_pretrained,synthesize_conversation): they do not exist in the repository. - Resources: project page, Hugging Face collection, ASR report.