SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

MOSS-TTSD - An Open-Source Spoken Dialogue Speech Generation Model from a Tsinghua Laboratory

Author: AI小集 · Original source: AI工具集

What is MOSS-TTSD

MOSS-TTSD (Text to Spoken Dialogue) is an open-source spoken dialogue speech generation model developed by the Speech and Language Laboratory of Tsinghua University (Tencent AI Lab). It can convert text dialogue scripts into natural, fluent, and expressive conversational speech, and supports bilingual generation in Chinese and English. The model is based on an advanced semantic-acoustic neural network audio codec and a large-scale pretrained language model, and was trained on more than 1 million hours of single-speaker speech data and 400,000 hours of conversational speech data. It supports zero-shot voice cloning and can generate accurate speaker-switching speech based on dialogue scripts, making it suitable for AI podcasts, interviews, news reports, and many other scenarios.

MOSS-TTSD

Main Features of MOSS-TTSD

  • Highly Expressive Dialogue Speech Generation: It can convert dialogue scripts into natural and expressive conversational speech, accurately capturing characteristics such as prosody and intonation in the dialogue.
  • Zero-Shot Multi-Speaker Timbre Cloning: It supports generating accurate speaker-switching speech based on dialogue scripts, enabling timbre cloning of two speakers without additional samples.
  • Chinese-English Bilingual Support: It can generate high-quality conversational speech in both Chinese and English.
  • Long-Form Speech Generation: Based on a low-bitrate codec and an optimized training framework, it can generate extremely long speech in one pass, avoiding the unnatural transitions caused by concatenating speech segments.
  • Fully Open Source and Commercially Ready: The model weights, inference code, and API interface are all open source and support free commercial use.

Technical Principles of MOSS-TTSD

  • Base Model Architecture: MOSS-TTSD is continuously trained based on the Qwen3-1.7B-base model and adopts a discretized speech sequence modeling method. The model discretizes speech through an eight-layer RVQ (Residual Vector Quantization) codebook, converting continuous speech signals into discrete token sequences. These token sequences are generated through autoregression with Delay Pattern, and finally the tokens are restored to speech through the Tokenizer's decoder.
  • Speech Discretization and Encoder Innovation: One of the core innovations is XY-Tokenizer, a specially designed speech discretization encoder. XY-Tokenizer is trained using a two-stage multi-task learning approach:
    • First Stage: Train the automatic speech recognition (ASR) task and the reconstruction task, allowing the encoder to retain coarse-grained acoustic information while encoding semantic information.
    • Second Stage: Fix the encoder and quantization layer, and train only the decoder part, supplementing fine-grained acoustic information through reconstruction loss and GAN loss. XY-Tokenizer can simultaneously model semantic and acoustic information at a bitrate of 1kbps and a frame rate of 12.5Hz, outperforming other similar Codecs.
  • Data Processing and Pretraining: MOSS-TTSD was trained using approximately 1 million hours of single-speaker speech data and 400,000 hours of conversational speech data. The team designed an efficient data processing pipeline to select high-quality single-speaker speech and multi-speaker conversational speech from massive raw audio and annotate them. The model underwent TTS pretraining, using 1.1 million hours of Chinese and English TTS data to significantly enhance speech prosody and expressiveness.
  • Long-Form Speech Generation Capability: Based on an ultra-low bitrate Codec, MOSS-TTSD supports audio generation of up to 960 seconds and can generate ultra-long speech in one go, avoiding unnatural transitions between spliced speech segments.

MOSS-TTSD project address

  • Project official website：https://www.open-moss.com/en/moss-ttsd/
  • Github repository：https://github.com/OpenMOSS/MOSS-TTSD
  • HuggingFace model library：https://huggingface.co/fnlp/MOSS-TTSD-v0.5
  • Online experience Demo：https://huggingface.co/spaces/fnlp/MOSS-TTSD

Application scenarios of MOSS-TTSD

  • AI podcast production: MOSS-TTSD can generate natural and fluent conversational speech, making it especially suitable for AI podcast production. It can simulate real dialogue scenarios and generate high-quality podcast content.
  • Film and television dubbing: The model supports highly expressive conversational speech generation in both Chinese and English, and can perform zero-shot voice cloning, making it suitable for dialogue dubbing in film and television works.
  • Long-form interviews: MOSS-TTSD supports audio generation of up to 960 seconds and can generate ultra-long speech in one go, avoiding unnatural transitions between spliced speech segments, making it very suitable for speech generation for long-form interviews.
  • News reporting: In news reporting, MOSS-TTSD can generate natural conversational speech for broadcasting news content, enhancing the appeal of the news.
  • E-commerce livestreaming: The model can be used in e-commerce livestreaming scenarios such as digital human conversational selling, attracting viewers by generating natural conversational speech.