SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Tongyi Bailing - Alibaba Tongyi's Enterprise-Level Speech Foundation Large Model

Author: AI小集 · Original source: AI工具集

What is Tongyi Bailing

Tongyi Bailing is an enterprise-level speech foundation large model launched by Alibaba Tongyi Lab, integrating Fun-ASR speech recognition and Fun-CosyVoice speech synthesis, two major models, designed specifically for speech applications in complex environments. Through the Context-enhanced architecture, it significantly reduces the hallucination rate, solves the language-mixing problem, and supports dynamic injection of hotwords and precise recognition of industry terminology. The model's speech synthesis capability supports cross-lingual cloning, with leading voice similarity. Trained on massive real audio, it covers multiple industries such as finance and education, enabling rapid deployment and helping enterprises efficiently implement speech applications.

Tongyi Bailing's latest upgrade: After the Fun-CosyVoice3 model upgrade, the first-packet latency is reduced by 50%, and the accuracy of mixed Chinese-English characters is doubled. It supports 9 common languages, 18 dialect accents, cross-lingual cloning, and emotional control, and has zero-shot voice cloning capability, enabling more efficient and natural speech synthesis. At the same time, the Fun-ASR model's capabilities are significantly enhanced. In noisy scenarios, the recognition accuracy is increased to 93%, it supports free mixing of 31 languages, coverage of dialect accents, and newly adds lyrics and rap recognition capabilities. The first-character latency for streaming recognition is reduced to 160ms, making speech recognition more accurate and faster.

Tongyi Bailing

Main Features of Tongyi Bailing

  • Significant reduction in hallucination rate: Through the Context-enhanced architecture (CTC+LLM+RAG), the CTC preliminary screening results are used as LLM context, reducing the hallucination rate from 78.5% to 10.7%, making output more stable and reliable.
  • Completely solves the language-mixing problem: CTC-decoded text is input into the LLM Prompt, greatly alleviating the "automatic translation" phenomenon, for example, avoiding English recordings being output as Chinese.
  • Strong customization capability: Introduces the RAG mechanism to dynamically inject terminology libraries, supporting precise recognition of personal names, brands, and industry slang (such as "ROI" and "private domain user acquisition"), with configuration completed in 5 minutes.
  • Cross-lingual voice cloning: Based on a multi-stage training method, one voice can support multiple languages, with industry-leading voice similarity.
  • Full coverage of industry scenarios: Based on tens of millions of hours of real audio training, covering 10+ industries such as finance, education, manufacturing, internet, and animal husbandry, deeply involved in frontline industries.

Technical Principles of Tongyi Bailing

  • Fun-ASR Speech Recognition Large Model: The Fun-ASR speech recognition large model based on Bailing adopts an innovative Context-enhanced architecture (CTC+LLM+RAG). It uses CTC technology for preliminary speech-to-text conversion and uses LLM to optimize the context of the generated text, significantly reducing the hallucination rate from 78.5% to 10.7%, making output more stable and reliable. Based on the RAG mechanism, it dynamically injects terminology libraries, supports precise recognition of personal names, brands, industry slang, etc., and completes configuration within 5 minutes, meeting the personalized needs of different enterprises.
  • Fun-CosyVoice Speech Synthesis Large Model: The Fun-CosyVoice speech synthesis large model is based on an innovative speech decoupling training method, separating and independently training features such as voice timbre, speaking rate, and intonation, and combining them to generate high-quality speech, making synthesized speech more natural and fluent. The model supports cross-lingual voice cloning. Through a multi-stage training method, one voice can support multiple languages, achieving "one voice speaks globally," with industry-leading voice similarity.

Tongyi Bailing Project Address

Application Scenarios of Tongyi Bailing

  • Financial Industry: Used in intelligent customer service, voice transactions, and risk monitoring to improve service efficiency and risk prevention and control capabilities.
  • Education industry: Helps online education platforms, intelligent tutoring systems, and voice homework grading, optimizing the teaching and learning experience.
  • Manufacturing industry: Enables voice control of industrial equipment, production process monitoring, and quality inspection, improving production efficiency and safety.
  • Internet industry: Supports voice search, intelligent assistants, and content creation, enhancing user experience and content diversity.
  • Livestock industry: Applied in intelligent breeding systems, animal health monitoring, and breeding environment management, improving breeding efficiency and animal health management.