SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

DiaMoE-TTS - A Multi-Dialect TTS Framework Open-Sourced by Tsinghua University and Giant Network

Author: AI小集 · Original source: AI工具集

What is DiaMoE-TTS

DiaMoE-TTS is a multi-dialectspeech synthesis(TTS) framework jointly launched by Tsinghua University and Giant Network. The framework is based on a unified International Phonetic Alphabet (IPA) input system, combined with a dialect-aware Mixture-of-Experts (MoE) architecture and low-resource adaptation strategies (PEFT), to achieve low-cost, low-barrier multi-dialect speech synthesis. The framework supports multiple dialects and low-resource languages, and can quickly model with a small amount of data, combining both efficiency and flexibility. DiaMoE-TTS's data, code, and methods are fully open-source, promoting dialect preservation and cultural heritage, and letting minority languages be heard in the digital world.

DiaMoE-TTS

Main Features of DiaMoE-TTS

  • Multi-Dialect Speech Synthesis: Supports speech synthesis for multiple dialects and low-resource languages, covering Cantonese, Hokkien, Wu, etc., and can be extended to special types such as Peking Opera recitative, allowing minority languages to speak.
  • Low-Resource Adaptation: Through PEFT strategies and data augmentation, under low-resource conditions with only a few hours of corpus, it can quickly adapt to new dialects and synthesize natural and fluent speech.
  • High Scalability: Fully open-source across the entire pipeline, providing complete data preprocessing, training, and inference code, supporting multiple languages, making it convenient for researchers and developers to reproduce and extend.
  • High-Naturalness Speech Synthesis: Uses a dialect-aware MoE architecture, selecting expert networks through a dynamic gating mechanism, preserving the unique timbre and prosody of each dialect, and improving speech naturalness.

Technical Principles of DiaMoE-TTS

  • Unified IPA Front-End: Uses the International Phonetic Alphabet (IPA) as a unified input system, mapping the speech of all dialects to the same phoneme space, eliminating differences between dialects and ensuring consistency and generalization ability in model training.
  • Dialect-Aware MoE Architecture: Introduces multiple expert networks, each expert focusing on learning the features of one or several dialects, avoiding the "style averaging" problem of a single network. : Automatically selects the most suitable expert network for speech synthesis based on the input IPA, while adding a dialect classification auxiliary loss to enhance the discriminative ability of the expert networks.
  • Low-Resource Adaptation Strategy (PEFT): Incorporates Conditioning Adapter and LoRA into the text embedding layer and attention layer, requiring only fine-tuning a small number of parameters to complete dialect extension, while the backbone network and MoE modules remain frozen. Uses techniques such as pitch perturbation and speaking rate perturbation to improve the model's synthesis performance under low-resource conditions.
  • Multi-Stage Training Method: Based on the original F5-TTS checkpoint, introduces data converted to IPA phonemes for warm-up training, achieving smooth migration of input format. Uses multiple open-source dialect datasets for joint modeling, activating the MoE structure, learning shared features, and distinguishing pronunciation patterns of different dialects. Through the dynamic gating mechanism and dialect classification auxiliary loss, further optimizes the MoE routing effect and captures the unique features of each dialect. For new dialects with only a few hours of corpus, uses PEFT strategies combined with data augmentation to achieve efficient transfer and keep existing knowledge from being forgotten.

Project Addresses of DiaMoE-TTS

  • GitHub Repository：https://github.com/GiantAILab/DiaMoE-TTS
  • HuggingFace Model Library：https://huggingface.co/RICHARD12369/DiaMoE_TTS
  • arXiv Technical Paper：https://www.arxiv.org/pdf/2509.22727

Application Scenarios of DiaMoE-TTS

  • Education: Provides vivid speech synthesis tools for dialect and low-resource language teaching, helping students better learn and master the pronunciation of different languages.
  • Cultural Preservation: Helping to protect and pass on dialects and minority languages, documenting and recreating endangered dialects through speech synthesis technology, and preserving cultural diversity.
  • Virtual Humans and Digital Content: Generating diverse dialect speech for virtual humans, digital assistants, and more, enriching the expressiveness of virtual characters and improving the user experience.
  • Digital Cultural Tourism: Providing audio guides in multiple dialects at tourist attractions, enhancing visitors' sense of identity with and affinity for the local culture.
  • Cross-Border Communication: Supporting speech synthesis in multiple languages and dialects, promoting communication and understanding among people from different language backgrounds.