SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Fun-Audio-Chat - Alibaba Tongyi's Open-Source End-to-End Voice Interaction Model

Author: AI小集 · Original source: AI工具集

What is Fun-Audio-Chat

Fun-Audio-Chat is a new-generation end-to-end voice interaction model open-sourced by Alibaba Cloud's Tongyi Bailing team, featuring voice understanding, emotion perception, and task execution capabilities. The model adopts a dual-resolution design, working collaboratively through 5Hz and 25Hz frame rates, saving nearly 50% of GPU computing resources compared to similar products. The core innovation lies in the Core-Cocktail two-stage training strategy, which effectively avoids catastrophic forgetting while supporting multilingual voice translation and role-playing functions. In international evaluations such as OpenAudioBench, the open-sourced Fun-Audio-Chat-8B surpasses competitors like GLM4-Voice in tasks such as voice dialogue and emotion recognition. It has been applied in scenarios such as intelligent customer service and emotional companionship, and users can download and use it for free via the ModelScope and HuggingFace platforms.

Fun-Audio-Chat

Main Features of Fun-Audio-Chat

  • End-to-End Voice Interaction: It adopts an end-to-end voice interaction mode, directly generating voice output from voice input, without the traditional multi-module concatenation of Automatic Speech Recognition (ASR) + Large Language Model (LLM) + Text-to-Speech (TTS), achieving higher efficiency and lower latency.
  • Emotion Perception: It can perceive the user's emotional state through details such as semantics, tone, speaking rate, and pauses, even if the user does not directly express emotions.
  • Emotional Response: When the user is angry, anxious, or happy, the model provides just the right comfort, companionship, or resonance, offering a conversation experience similar to that of a friend.
  • Natural Voice Command Execution: The model supports the Speech Function Call feature, allowing users to issue commands with natural voice, and the system can automatically call relevant functions to complete complex tasks.
  • Open Source and Ease of Use: The 8B model weights, inference code, and Function Call integration examples are all open-sourced, making it convenient for developers to download and use.

Technical Principles of Fun-Audio-Chat

  • End-to-End S2S Architecture: Fun-Audio-Chat adopts a Speech-to-Speech (S2S) end-to-end architecture, directly generating voice output from voice input, without the traditional multi-module concatenation of ASR (Automatic Speech Recognition) + LLM (Large Language Model) + TTS (Text-to-Speech). It significantly improves processing efficiency, reduces system latency, and achieves a smoother voice interaction experience.
  • Dual-Resolution Design: The model adopts an innovative dual-resolution processing mechanism: the Shared LLM layer performs efficient semantic processing at a 5Hz frame rate, while the SRH (Speech Reconstruction Head) generates high-quality speech at a 25Hz frame rate. While ensuring speech quality, it reduces GPU computing overhead by nearly 50%, achieving a balance between performance and efficiency.
  • Trained on One Million Hours of Multi-Task Data: The model has been trained on one million hours of multi-task data, covering real-world scenarios such as audio understanding, voice question answering, emotion recognition, and tool invocation. It can understand user intent in a more "down-to-earth" way. On multiple authoritative leaderboards such as OpenAudioBench, MMAU, Speech-ACEBench, and VStyle, it ranks first among models of the same size, with comprehensive performance surpassing competitors such as GLM4-Voice, Kimi-Audio, and Baichuan-Omni.
  • Emotion Perception Capability: Fun-Audio-Chat has excellent emotion recognition capability, able to perceive emotions from paralinguistic cues such as the user's tone, speaking rate, and pauses. Even if the user does not directly express emotions, it can accurately identify them and give an appropriate response. This makes the conversation experience more natural and humanized.
  • Speech Function Call Feature: The model supports function calling under natural voice commands; users only need to issue commands by voice, and the system can automatically call the relevant functions to complete complex tasks. This expands the model's application scenarios, enabling it not only to chat but also to truly "get things done."

Fun-Audio-Chat project address

  • Project official website：https://funaudiollm.github.io/funaudiochat/
  • Github repository：https://github.com/FunAudioLLM/Fun-Audio-Chat
  • HuggingFace model library：https: //huggingface.co/FunAudioLLM/Fun-Audio-Chat-8B
  • Technical paper：https://github.com/FunAudioLLM/Fun-Audio-Chat/blob/main/Fun-Audio-Chat-Technical-Report.pdf

Fun-Audio-Chat application scenarios

  • Voice chat: Fun-Audio-Chat can engage in natural and fluent voice conversations with users, providing an experience similar to talking with a real person, and is suitable for daily chatting and social interaction.
  • Emotional companionship: The model can perceive the user's emotions and respond, such as comforting, encouraging, or empathizing, and is suitable for use when the user feels lonely, anxious, or needs to talk.
  • Smart device control: Users can control smart devices through voice commands, such as smart home devices and smart wearables, enabling more convenient operation.
  • Voice customer service: In customer service scenarios, Fun-Audio-Chat can understand users' questions and provide accurate answers, improving customer service efficiency and user experience.
  • Role-playing: Users can specify the model to play a specific role, such as an esports commentator or virtual assistant, to meet different entertainment or work needs.