dots.tts - A speech synthesis foundation model open-sourced by Xiaohongshu and Shanghai Jiao Tong University

What is dots.tts

dots.tts is a 2-billion-parameter fully continuous autoregressivespeech synthesisfoundation model jointly open-sourced by Xiaohongshu's dots team and Shanghai Jiao Tong University's X-LANCE Lab. The model directly generates 48kHz audio block by block in a continuous latent space, achieving SOTA timbre similarity and content accuracy on benchmarks such as Seed-TTS-Eval, and supports zero-shot cloning, streaming output, and low-latency duplex conversation. The project fully open-sources the code and weights, and extends the precise speech editing capability of dots.tts.edit.

Main features of dots.tts

• Zero-shot timbre cloning: With only 3–10 seconds of reference audio, it can replicate that timbre to read any new text, supports cross-lingual cloning, and achieves SOTA performance in content accuracy and speaker similarity on the Seed-TTS-Eval benchmark.
• Text-to-speech synthesis: Provides high-quality speech generation from random timbre sampling to a specified speaker, directly outputting 48 kHz audio based on a 2B-parameter continuous autoregressive model.
• Streaming generation and low-latency interaction: Naturally supports streaming output, generating audio block by block after the text prefix is input; in the 1T1A dual-stream mode, the speech side responds immediately each time the upstream LLM outputs a text Token, with audio first-packet latency as low as 54.4 milliseconds, suitable for real-time speech agents and duplex conversations.
• Precise speech editing: Supports text replacement, emotion adjustment, pitch and speaking rate modification, pause insertion or deletion on existing recordings, and supports combining multiple operations in a single instruction at once, with unedited regions remaining unchanged.
• Multilingual support: Covers 24 languages, with an average speaker similarity of 83.9 in the MiniMax multilingual evaluation, among which 19 languages rank first in individual categories.
• Multiple inference speeds: Provides multiple checkpoints such as Base, SOAR, MeanFlow (4-step), two-step sCM, and single-step DMD, allowing users to choose as needed between audio quality and inference speed.

Technical principles of dots.tts

• Fully continuous autoregressive architecture: dots.tts abandons the traditional approach of quantizing speech into discrete Tokens; the entire generation pipeline operates completely in a continuous latent space: the model autoregressively predicts acoustic segments block by block and restores them to 48 kHz waveforms through a BigVGAN-style decoder, fundamentally avoiding the information bottleneck and timbre detail loss caused by discrete quantization.
• Semantic continuous representation: The model uses an AudioVAE designed based on HoliTok to compress raw waveforms into continuous latent variables at 25 FPS. The encoder achieves a balance between reconstruction fidelity and latent space structure through semantic training, bringing the speaker similarity after reconstruction to 0.969, preserving a rich upper limit of timbre and acoustic details for subsequent autoregressive generation.
• Error control mechanism: To address the inherent problem in continuous autoregression where small deviations in previous steps accumulate and spread backward, dots.tts introduces a causal semantic encoder that compresses generated VAE patches in real time into a compact semantic history and feeds it back to the large language model; by stripping high-variance local acoustic variations and retaining only long-range semantic summaries, it effectively suppresses error propagation, howling, and timbre drift in long-sequence generation.
• Generation pipeline: The generation process is centered on a large language model, which is initialized based on Qwen2.5-1.5B, directly consumes BPE text tokens, and outputs the hidden state at each step; then an autoregressive flow matching head (18-layer DiT), conditioned on the LLM hidden states and the autoregressive prefix, performs denoising generation for the next acoustic patch, while a global x-vector provided by a frozen CAM++ speaker encoder ensures timbre consistency.
• Post-training acceleration: The team uses SOAR self-corrective alignment to let the model simulate deviation states in real inference during training and learn self-correction, improving stability without a reward model; MeanFlow distillation is used to compress the teacher's multi-step trajectory into an average velocity, achieving 4-step high-quality generation; it is compressed to 2 steps via a simplified consistency model (sCM); through reward-aware distribution matching distillation (DMD) combined with a dual-timescale update strategy, a natural, clear, high-quality one-step model is obtained.

Follow on WeChat and reply "Open Source" to jointhe AI Open Source Project Exchange Group

How to use dots.tts

• Installation: Run pip install dots.tts to install, or clone the source code from GitHub and use pip install -e . for local development installation; for high-concurrency scenarios, SGLang Omni can additionally be deployed to obtain CUDA Graph acceleration and continuous batching support.
• Command-line synthesis: Use the dots.tts entry point and specify --model-name-or-path、--text and the target --prompt-audio and --prompt-textto achieve zero-shot timbre cloning; omitting --prompt-text degrades to x-vector cloning, and omitting --prompt-audio performs random timbre sampling.
• Command-line editing: Use the dots.tts.edit entry point, provide --source-audio and XML-style --instruction, and perform precise edits such as text replacement, emotion, or prosody modification on existing recordings.
• Basic Python API call: After loading the model via DotsTtsRuntime.from_pretrained() , call generate(), pass in text and an optional reference audio to obtain the synthesized audio tensor and save it as a file.
• Python API streaming output: Calling generate_stream() allows audio tensors to be obtained chunk by chunk and pushed to a player or WebSocket in real time, enabling a low-latency experience of generating and playing simultaneously.
• Python API dual-stream dialogue: After starting a session with DotsTtsRuntimeDoubleStreaming , push text token by token, and the speech side starts generating immediately; the first audio packet latency can be as low as 54.4 milliseconds, making it suitable for real-time voice agents.
• Fine-tuning: Run accelerate launch scripts/train_dots_tts.py and configure your own data path to perform timbre or domain adaptation fine-tuning based on a public checkpoint.
• Distillation acceleration: Through scripts/train_dots_tts_meanflow.py , use the SOAR checkpoint as the teacher for MeanFlow distillation to obtain 4-step, 2-step, or 1-step high-quality student models to suit different latency and concurrency requirements.
• Web interface: Run python apps/gradio/app.py to start the visual synthesis interface, or run apps/edit_playground/app.py to start the Edit Playground, where you can upload audio, annotate edit intervals, and preview results in real time in the browser.

Core advantages of dots.tts

• Fully continuous latent space modeling: Abandons discrete acoustic tokens and directly performs autoregressive generation in a continuous latent space, preserving rich timbre details such as spectral texture and breathiness, with reconstructed speaker similarity reaching as high as 0.969.
• SOTA synthesis quality: Achieves SOTA in both content accuracy and timbre similarity on the Seed-TTS-Eval benchmark, with an average SIM of 83.9 in MiniMax evaluation across 24 languages, surpassing mainstream solutions in overall performance.
• Native streaming low latency: In 1T1A dual-stream mode, the audio first-packet latency is as low as 54.4 milliseconds, and with SGLang Omni on a single H100, concurrent throughput for 16 streams reaches 4.76 req/s, meeting the needs of real-time voice agents.
• Unified editing capability: dots.tts.edit is based on the same foundation, supports XML-structured instructions to precisely edit text, emotion, prosody, and pauses, and leads other open-source systems overall in the doteBench evaluation.

dots.tts project address

• GitHub repository:https://github.com/studio-dots-ai/dots.tts
• arXiv technical paper:https://arxiv.org/pdf/2608.02673

Comparison of dots.tts with similar competitors

Application scenarios of dots.tts

• Real-time voice Agent: 1T1A dual-stream mode has first-packet latency as low as 54.4 milliseconds, suitable for LLM-driven real-time duplex dialogue and voice assistants.
• Zero-shot timbre cloning: Only a few seconds of reference audio are needed to replicate any timbre, supports cross-language cloning, suitable for audiobooks, dubbing, and personalized content creation.
• Multilingual content generation: Covers 24 languages while maintaining high speaker similarity, suitable for multilingual voice announcements and localization services for global products.
• Precise speech editing: Through dots.tts.edit, existing recordings can undergo text replacement, emotion adjustment, prosody modification, and pause adjustment, suitable for podcast post-production and fine editing of audio content.
• Livestreaming and real-time broadcasting: Works with LLM streaming output to generate and play simultaneously, suitable for news broadcasting, livestream e-commerce, and real-time information broadcasting scenarios.