Whisper

Whisper is an open-source Windows desktop application based on OpenAI's Whisper automatic speech recognition (ASR) model, using DirectCompute (Direct3D 11 compute shaders) for high-performance GPGPU inference. This project is a Windows port of whisper.cpp (a C++ port of OpenAI Whisper).

Purpose

• Transcribe speech from audio files to text.
• Capture real-time audio from a microphone and transcribe or translate it.

Main capabilities

• Vendor-agnostic GPGPU: based on DirectCompute (Direct3D 11 compute shaders).
• Pure C++ implementation with no runtime dependencies other than necessary operating system components.
• Performance significantly better than OpenAI's PyTorch/CUDA implementation: using the medium model on a GeForce 1080Ti, 3 minutes 24 seconds of speech takes 45 seconds with PyTorch/CUDA, while this implementation takes only 19 seconds; runtime dependencies are 9.63 GB versus 431 KB for Whisper.dll.
• Mixed F16/F32 precision (Windows has required support for R16_FLOAT buffers since D3D 10.0).
• Built-in performance profiler that can measure the execution time of each compute shader.
• Low memory usage.
• Uses Media Foundation for audio processing, supporting most audio and video formats (except Ogg Vorbis) and most Windows audio capture devices (except some professional devices that only implement the ASIO API).
• Includes voice activity detection during audio capture, based on the 2009 article "A simple but efficient real-time voice activity detection algorithm" by Mohammad Moattar and Mahdi Homayoonpoor.
• Provides an easy-to-use COM-style API; an idiomatic C# wrapper is available on NuGet. Version 1.10 introduced PowerShell 5.1 (Windows PowerShell) script support.
• Provides precompiled binaries.

Usage

• From the repository's Releases section, download WhisperDesktop.zip, extract it, and run WhisperDesktop.exe.
• On first launch, the interface will ask you to download a model; ggml-medium.bin (1.42 GB) is recommended because the author mainly tested with that model.
• The next screen can transcribe audio files.
• There is also a screen that can capture and transcribe or translate real-time audio from a microphone.
• Developer build steps: clone the repository, open WhisperCpp.sln with Visual Studio 2022, switch to the Release configuration, build and run the CompressShaders C# project in the Tools subfolder (set it as the startup project, then "Debug / Start Without Debugging"), and after success you should see output similar to "Compressed 46 compute shaders, 123.5 kb -> 18.0 kb". Then build the Whisper project to generate the native DLL, or WhisperNet to generate the C# wrapper and NuGet package, or the sample project. If you use the library in software built with Visual C++ 2022 or newer, you may need to redistribute the Visual C++ runtime DLLs as an .msm merge module or vc_redist.x64.exe; if you do so, change C/C++ -> Code Generation -> Runtime Library in the Whisper project properties from "Multi-threaded (/MT)" to "Multi-threaded DLL (/MD)" and rebuild. The library includes RenderDoc GPU debugger integration; when launching the program from RenderDoc, press F12 to capture compute calls; to debug HLSL shaders, use a debug build of the DLL. The repository also contains a lot of code used only for development (alternative model implementations, some compute shaders compatible with FP64 versions, debug tracing and trace comparison tools, etc.), disabled via preprocessor macros or constexpr flags.

Suitable scenarios

• Users who need to quickly transcribe audio files on Windows.
• Users who need real-time transcription or translation of speech from a microphone.
• Developers with Direct3D 11.0-capable GPUs who want to integrate speech recognition functionality through a COM-style API or C# wrapper.

Limitations

• Has not been updated since 2023.
• The only supported platform is 64-bit Windows; it should run on Windows 8.1 or newer, but has only been tested on Windows 10.
• The library requires a GPU with Direct3D 11.0 capability (in 2023, that is "any hardware GPU"; the most recent GPU that does not support D3D 11.0 is Intel Sandy Bridge from 2011).
• On the CPU side, AVX1 and F16C support are required.
• Automatic language detection is not implemented.
• The current version has high latency for real-time audio capture, approximately 5-10 seconds depending on voice detection.
• The author notes that the code may contain bugs, and the software is provided "as is", without any warranty.
• The website whisperdesktop.com impersonates this project and should not be trusted; although it currently links to the GitHub download, the site operators may replace it with malware at any time.
• The author has built a commercial successor (closed source, 30-day free trial, then a one-time purchase), which uses a new speech-to-text engine, is implemented mainly in AOT-compiled C#, includes Direct3D 12 and Vulkan 1.3 compute backends, and supports inference of the Voxtral Mini 4B Realtime model.
• Performance optimization is limited: the author optimized for nVidia 1080Ti, the Radeon Vega 8 of Ryzen 7 5700G, and the Radeon Vega 7 of Ryzen 5 5600U. nVidia relative speed: large model 5.8, medium model 10.6; on the Ryzen 5 5600U APU, the medium model is about 2.2; nVidia 1650 is slower than 1080Ti but far faster than real time; on Intel HD Graphics 4000 (Core i7-3612QM), the medium model has a relative speed of 0.14, and the small model 0.44, far slower than real time but able to run. No specific optimization was done for discrete AMD GPUs or integrated Intel GPUs. The bottleneck may be memory rather than computation.