SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Fun-CosyVoice3.5 - A speech generation model launched by Alibaba Tongyi Lab

Author: AI小集 · Original source: AI工具集

What is Fun-CosyVoice3.5

Fun-CosyVoice3.5 is the latest release from the speech team at Alibaba Tongyi Labspeech generationmodel, focusing on multilingual voice cloning and fine-grained expression control. The model's most prominent innovation is its FreeStyle natural language control capability—users do not need to master professional parameters and can precisely regulate speech effects directly with colloquial instructions such as "make the tone a bit more firm" or "slow down the speaking rate," achieving "free speech generation with a single sentence." Fun-CosyVoice3.5 adds support for Thai, Indonesian, Portuguese, and Vietnamese, covering 13 languages, reduces the mispronunciation rate of rare characters from 15.2% to 5.3%, and lowers first-packet latency by 35% through Tokenizer frame rate optimization.

Fun-CosyVoice3.5

Main features of Fun-CosyVoice3.5

  • FreeStyle natural language control: Supports directly describing speech effects with colloquial instructions, such as "make the tone a bit more firm," "lower the pitch slightly and slow down the speaking rate," or "add a bit of emotional variation," enabling precise control without mastering professional parameters.
  • Multilingual voice cloning: Adds support for Thai, Indonesian, Portuguese, and Vietnamese, covering 13 languages in total, with word error rate (WER) and speaker similarity (SpkSim) metrics maintaining industry-leading levels.
  • Improved pronunciation accuracy: The mispronunciation rate of rare characters is greatly reduced from 15.2% to 5.3%, making long-text reading more stable and fluent and reducing stutters and errors.
  • Low-latency optimization: The Tokenizer frame rate is halved, and first-packet latency is reduced by 35%, making it more suitable for real-time speech interaction scenarios.
  • Reinforcement learning-driven optimization: The language model uses DiffRO+GRPO strategies to optimize prosody, while audio generation uses Flow-GRPO technology to improve audio quality and voice cloning similarity.

Technical principles of Fun-CosyVoice3.5

  • DiffRO + GRPO prosody optimization: Reinforcement learning is introduced into the language model part, adopting DiffRO (Differential Reward Optimization) combined with the GRPO (Generalized Reward-Penalty Optimization) strategy, and adding a multi-channel reward mechanism for duration and prosody to improve the naturalness and rhythm of speech.
  • Flow-GRPO audio quality improvement: Flow-GRPO technology is used in the audio generation part, optimizing through a Flow Matching framework combined with reinforcement learning, significantly improving the similarity of voice cloning and overall audio quality.
  • Tokenizer frame rate optimization: Halves the Tokenizer frame rate, effectively reducing computational overhead and achieving a 35% reduction in first-packet latency while maintaining generation quality.
  • End-to-end speech synthesis architecture: Based on the end-to-end speech synthesis framework of the CosyVoice series, it integrates text encoding, acoustic modeling, and a vocoder to achieve direct generation from text to speech, reducing errors in intermediate steps.
  • Multi-task joint training: Through a multi-task learning framework, it simultaneously optimizes multiple objectives such as speech recognition, voice cloning, and style control, improving the model's generalization ability in complex scenarios.

How to use Fun-CosyVoice3.5

  • Call via the Alibaba Cloud Bailian platform: Visit theAlibaba Cloud Bailianconsole, activate the speech synthesis service, obtain an API Key for calls, and enjoy support for online debugging and batch generation.
  • API interface calls: Use the official API provided, pass in the text content, target voice ID, and FreeStyle control instructions (such as "make the tone gentler"), and return the generated audio file.

Application Scenarios of Fun-CosyVoice3.5

  • Intelligent Customer Service and Voice Assistants: The low-latency feature supports real-time conversations, and FreeStyle control can adjust the tone based on user emotions, enhancing service warmth and user satisfaction.
  • Audio Content Creation: In scenarios such as podcasts, audiobooks, and news broadcasting, natural language instructions can quickly adjust the reading style, enabling one person to voice multiple roles or maintain a unified brand voice.
  • Virtual Streamers and Digital Humans: Precise voice cloning capability can replicate a specific person's voice, and combined with emotion control, generate expressive dubbing for livestreams/short videos.
  • Game and Animation Voiceover: Supports localization in 13 languages, quickly generates character voices, and reduces the cost and time of producing multilingual versions.
  • Education and Language Learning: Accurate reading of rare characters assists teaching, and multilingual support can be used for foreign language pronunciation demonstrations and speaking practice.
  • Accessibility Services: Provides high-quality voice reading for visually impaired individuals, with support for personalized voice customization and speech rate adjustment.