Whissle Gateway is a self-hosted gateway for the Whissle voice stack, packaging the same engine behind the hosted platform as a component that can run on your own infrastructure. It is essentially a reverse proxy positioned in front of the real-time WebRTC voice pipeline and the agent runtime.
Purpose
- Provide a unified entry point for voice AI applications; frontends (browser, telephony, macOS, embedded) connect to the gateway's :9000 port via HTTPS and WebRTC.
- Support running the full voice stack on your own infrastructure, or running only a lightweight backend and calling external APIs.
Key capabilities
- Streaming speech recognition (STT): with endpoint detection, noise gate, and recipient gating, implementable via Deepgram / Sarvam API or local Whissle ASR.
- Reasoning and tools: the agent runtime supports custom prompts, knowledge bases, and real tools (booking, logging, SMS, Webhook), based on Gemini or Claude.
- Text-to-speech (TTS): supports emotion and mid-call language switching, with optional Deepgram Aura, Sarvam, Cartesia, ElevenLabs, or local Kokoro.
- Avatar (optional): lip-synced video via Simli, or client-side rendering, which can be enabled or disabled per agent.
- Privacy by default: after adding local model services, audio, transcripts, and structured metadata do not leave the network; hosted databases can be replaced, bring your own keys.
- Advanced turn control: optional speculative tool warm-up, shadow LLM draft headers, real-time turn integrity, etc., disabled by default, documented and reversible.
How to use
- The backend is always a small stateless container and does not include model weights.
- Two operating modes:
- Backend using external APIs: STT/TTS/LLM/avatar run through provider APIs, requiring only a stateless container on a small VM.
- Backend + your own models: run independent Whissle model services (ASR/TTS/video) on your own machines or GPUs, and point the backend to them via MODELS_HOST, so audio and transcripts do not leave the network.
- Frontends connect to the gateway's :9000 port via HTTPS and WebRTC.
- Voice models, LLMs, and databases are all pluggable and configurable.
Applicable scenarios
- Deployments that require a self-hosted voice AI gateway and have data privacy and compliance requirements.
- Scenarios where you want voicerecognition, TTS, and agent runtime to be managed uniformly, with flexible replacement of components at each stage.
- For single-machine development, you can use the convenience database inside the container; for production environments, an external stateless database is recommended.
Limitations
- The gateway itself does not carry model weights, and you need to bring your own LLM keys.
- Local voice models require additional deployment of Whissle model services.
- Specific pricing, platform support, performance metrics, and other materials are not yet provided.
