WhisperS2T is an open-source speech-to-text (ASR) pipeline optimized for Whisper models, supporting multiple inference engines. It is not a speech recognition model itself, but a transcription pipeline built around the OpenAI Whisper model, designed to provide faster Whisper transcription speed and better accuracy. According to the official description, it is 2.3 times faster than WhisperX and 3 times faster than the HuggingFace Pipeline using FlashAttention 2 (Insanely Fast Whisper). Whisper is a general-purpose speech recognition model developed by OpenAI, trained on large-scale diverse audio datasets, with multi-task capabilities such as multilingual speech recognition, speech translation, and language identification.
In terms of main capabilities, WhisperS2T provides multi-backend support, including the original OpenAI model, the HuggingFace model using FlashAttention2, and the CTranslate2 model. It supports convenient integration of custom VAD (voice activity detection) models to enhance the control and accuracy of speech recognition. For audio file processing, it can intelligently batch smaller speech segments from different files to ensure performance; for large audio files, it supports transcription of segmented batches while asynchronously loading in the background, significantly reducing loading time. It also supports batch multilingual/task decoding, allowing decoding of multiple languages in a single batch or simultaneous transcription and translation (best with the CTranslate2 backend). In addition, it reduces repetitive text output or hallucinations through optimized parameters and heuristics, some of which apply only to the CTranslate2 backend. There is also experimental dynamic duration support, which can handle variable-length inputs within a given input batch instead of a fixed 30 seconds, thereby providing flexibility during transcription and saving computation time. This feature applies only to the CTranslate2 backend.
In terms of feature updates, on February 25, 2024, prebuilt Docker images and transcription exporters were added, allowing export to txt, json, tsv, srt, vtt; on January 28, 2024, support for the TensorRT-LLM backend was added; on December 23, 2023, support for CTranslate2 backend word alignment was added; on December 19, 2023, support for Whisper-Large-V3 and Distil-Whisper-Large-V2 was added; WhisperS2T was released on December 17, 2023.
In terms of usage, the official provides two approaches: Docker containers and local installation. The Docker image can be found in the Dockerhub repository (https://hub.docker.com/r/shashikg/whisper_s2t/tags）获取，支持从主分支或特定版本 Built with v1.3.0, it also supports building containers with TensorRT-LLM support. Local installation requires first installing the audio package used for resampling and loading audio files, then installing or updating to the latest released version of WhisperS2T via command, or installing from the latest commit in the repository. If CUDNN and CUBLAS are installed via pip wheel, you can run a command to add the CUDNN path to LD_LIBRARY_PATH. Using the TensorRT-LLM backend requires installing TensorRT and TensorRT-LLM. The official project provides a bash script that works on most Debian systems; if it does not apply, refer to the official TensorRT-LLM instructions. To use word alignment, the model must be loaded in a specific way. The official project also provides Google Colab notebooks for reference. Note that model inference may be slightly slower on the first run; after 1-2 runs, inference speed improves due to JIT tracing of the VAD model.
In terms of use cases, WhisperS2T is suitable for developers and users who need fast, accurate speech-to-text, especially in scenarios involving large numbers or long audio files, multilingual transcription and translation, and requirements on transcription latency. It relies on the capabilities of the Whisper model itself and supports multilingual speech recognition, speech translation, and language identification.
In terms of limitations, the official documentation states that some optimization heuristics apply only to the CTranslate2 backend, and dynamic duration support is experimental and limited to the CTranslate2 backend. The benchmark tests were conducted on an A30 GPU with the without_timestamps parameter set to True; adjusting this parameter may improve the word error rate (WER) of the HuggingFace pipeline but increases inference time. The speed improvements are achieved only through better pipeline design, without specific optimization of the backend inference engine. Using the TensorRT-LLM backend requires additionally installing TensorRT and TensorRT-LLM, which imposes certain requirements on the system environment. This project uses the MIT license.
The future roadmap includes: a usable Docker container, WhisperS2T-Server (an optimized server codebase for end-to-end deployment), in-depth documentation (hosted using GitHub Pages), exploring integration of Meta's SeamlessM4T model, and adding more datasets for WER benchmarking. The official project also plans to release a technical report comparing WhisperS2T with other Whisper pipelines. For now, you can view the quick benchmark on an A30 GPU, and the benchmark scripts are in the scripts/ directory.
