SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Ming-omni-tts - Ant Group's Open-Source Unified Audio Generation Model

Author: AI小集 · Original source: AI工具集

What is Ming-omni-tts

Ming-omni-tts is an open-source unified audio generation model from Ant Group, employing an autoregressive architecture to achieve joint generation of speech, music, and sound effects. The model supports fine-grained control over speech rate, pitch, volume, emotion, and dialect through natural language instructions, achieving 93% accuracy in Cantonese dialect control and 46.7% accuracy in emotion control, surpassing CosyVoice3. Technically, it adopts a unified continuous audio Tokenizer and Diffusion Transformer architecture, processing multimodal audio at a 12.5Hz frame rate, and through a "Patch-by-Patch" compression strategy reduces the LLM inference frame rate to 3.1Hz, maintaining audio quality while reducing latency. The 16.8B parameter version achieves a WER of only 0.83% on the Seed-tts-eval Chinese test set, surpassing SeedTTS and GLM-TTS. The model comes with over 100 high-quality voices built in, supports zero-shot voice design, provides Docker images and a Gradio demo, and is suitable for scenarios such as audiobooks, podcasts, and multilingual content creation.

Ming-omni-tts

Main Features of Ming-omni-tts

  • Unified Multimodal Audio Generation: The industry's first autoregressive model that can jointly generate speech, ambient sound, and music in a single channel, delivering an "immersive" auditory experience.
  • Fine-Grained Speech Control: Supports precise control over speech rate, pitch, volume, emotion, and dialect through simple instructions, with Cantonese dialect control accuracy reaching 93% and emotion control accuracy reaching 46.7%.
  • Intelligent Voice Design: Comes with over 100 high-quality voices built in and supports zero-shot voice design through natural language descriptions.
  • Efficient Inference Optimization: Adopts a "Patch-by-Patch" compression strategy, reducing the LLM inference frame rate to 3.1Hz and significantly lowering latency.
  • Professional Text Normalization: Accurately parses and reads complex mathematical expressions, chemical equations, and other professional formats, with a CER of only 1.97% on the internal test set.
  • Multilingual Support: Supports speech synthesis and cross-lingual transfer for Chinese, English, and other languages.
  • Zero-Shot TTS: Clones any voice with just 3-10 seconds of reference audio, achieving a WER as low as 0.83% on Seed-tts-eval.

Technical Principles of Ming-omni-tts

  • Unified Continuous Audio Tokenizer: A VAE-based continuous Tokenizer that integrates speech, music, and general audio into a unified latent space at a 12.5Hz frame rate, supporting joint modeling of multimodal audio.
  • Diffusion Transformer (DiT) Head: Adopts a diffusion head architecture to enhance audio generation quality, improving the fineness and naturalness of the sound.
  • Patch Generation Strategy: Adopts a generation strategy with a patch size of 4 and a lookback history of 32, striking a balance between local acoustic details and long-term structural coherence.
  • Autoregressive Generation Architecture: The industry's first autoregressive model to jointly generate speech, music, and sound effects in a single channel, achieving unified audio generation.
  • "Patch-by-Patch" Compression Mechanism: Through the compression strategy, the LLM inference frame rate is reduced from the original frequency to 3.1Hz, greatly lowering computational latency and inference cost.
  • Instruction Fine-Tuning Alignment: Achieves fine-grained control over speech rate, pitch, volume, emotion, and dialect through instruction fine-tuning, supporting natural language instruction parsing.

Project Addresses of Ming-omni-tts

  • GitHub Repository:https://github.com/inclusionAI/Ming-omni-tts
  • Hugging Face Model Library：
    • https://modelscope.cn/models/inclusionAI/Ming-omni-tts-16.8B-A3B
    • https://huggingface.co/inclusionAI/Ming-omni-tts-0.5B

Application Scenarios of Ming-omni-tts

  • Audiobook and Podcast Production: Supports long-text speech synthesis, with only 1.84% CER on Podcast TTS tasks, suitable for audiobooks, news broadcasting, and podcast content generation.
  • Multilingual Content Creation: Supports multilingual speech synthesis such as Chinese and English as well as cross-lingual timbre transfer, meeting the needs of global content production.
  • Game Sound Effect Design: Can jointly generate speech, ambient sounds, and music, providing immersive audio experiences for game scenarios.
  • Education and Training: Accurately reads professional content such as complex mathematical expressions and chemical equations, suitable for online education courseware and academic explanations.
  • Intelligent Customer Service and Assistants: Built in with 100+ high-quality timbres and supports zero-shot voice cloning, enabling rapid customization of brand-exclusive voice assistants.
  • Advertising and Marketing Voiceovers: Through emotion control and dialect support, generates highly expressive advertising voiceovers and localized marketing content.