whisperX

WhisperX is an open-source tool for automatic speech recognition (ASR), whose main features include word-level timestamps and speaker diarization. It is based on OpenAI's Whisper model, but addresses the issue that the original timestamps only go to the utterance level and may deviate by several seconds by using wav2vec2 forced alignment to achieve precise word-level timestamps.

Main capabilities:

• Batch inference: with the whisper large-v2 model, it can achieve transcription speeds of up to 70x real time.
• Efficient backend: it uses the faster-whisper backend; the large-v2 model requires less than 8GB of VRAM at beam_size=5.
• Word-level timestamps: it uses wav2vec2 alignment to provide accurate word-level timestamps.
• Speaker diarization: it integrates pyannote-audio's speaker diarization functionality and can output multi-speaker transcripts with speaker ID labels.
• VAD preprocessing: it uses voice activity detection to reduce hallucinations and supports batching without degrading WER.
• Context-aware batching: released in 2026, it enables condition_on_previous_context in the batch transcription framework, improving the accuracy of punctuation and proper nouns.

Usage: WhisperX can be installed via PyPI (recommended for simple installation), or via uvx or from the GitHub repository. Developers can clone the repository to make modifications. If using GPU acceleration, the CUDA toolkit 12.8 must be installed first; if using only CPU, this can be skipped. ffmpeg, rust, etc. may also need to be installed; refer to OpenAI whisper's installation instructions. Command-line usage: run the whisper example (default parameters, whisper small model); you can add --highlight_words True to visualize word times in the .srt file. Enabling speaker diarization: a Hugging Face access token (read permission) must be provided, passed after the --hf_token parameter, and the user agreement for the speaker-diarization-community-1 model must be accepted. To improve timestamp accuracy (at the cost of higher VRAM usage), a larger model can be used; the number of speakers can also be set (e.g., --min_speakers 2 --max_speakers 2) to label speaker IDs. To maintain continuous context (suitable for technical language/punctuation), you can set interleaved_context=True. It can run on CPU (also works on Mac OS X). With --condition_on_prev_text, it can be based on the preceding context (helpful for long audio effects).

Multilingual support: The phoneme ASR alignment model is language-specific. For tested languages, the model is automatically selected from torchaudio pipelines or Hugging Face. Just pass in the --language code and use whisper --model large. Currently, languages such as {en, fr, de, es, it} are provided by default, and other languages can be found in the supported list in DEFAULT_ALIGN_MODELS_HF in alignment.py. If the detected language is not in the list, you need to find a phoneme-based ASR model from the Hugging Face model hub and test it on the data.

Technical details and limitations:

• Transcription differences: WhisperX enables single-batch processing, and whisper inference uses --without_timestamps True, which may cause differences from the default whisper output.
• VAD-based segment transcription, unlike OpenAI's buffered transcription, can reduce WER and support accurate batch inference.
• --condition_on_prev_text defaults to False (reduces hallucinations).
• Transcribed words that do not contain characters in the alignment model dictionary (such as "2014." or "£13.60") cannot be aligned, so no timestamps are provided.
• Overlapping speech is handled poorly.
• Speaker diarization is far from perfect.
• A language-specific wav2vec2 model is required.
• Ways to reduce VRAM: reduce batch_size (e.g. --batch_size 4), use a smaller ASR model (--model base), use a lighter compute type (--compute_type int8) (the latter two may affect quality).

Use cases: Suitable for speech transcription tasks that require high-precision word-level timestamps, such as subtitle generation, meeting minutes, and multi-speaker conversation transcription. Won first place in the Ego4d transcription challenge and was included in INTERSPEECH 2023.

Contact and support: Contact maxhbain@gmail.com. The project is supported by VGG (Visual Geometry Group) and the University of Oxford, is built on top of OpenAI whisper, borrows PyTorch forced alignment tutorial code, and uses pyannote VAD/speaker diarization and the faster-whisper and CTranslate2 backends.