What is FunAudioLLM
FunAudioLLM is an open-source speech large model project launched by Alibaba's Tongyi Lab, including two models, SenseVoice and CosyVoice. SenseVoice excels at multilingual speech recognition and emotion recognition, supports more than 50 languages, and performs particularly well in Chinese and Cantonese. CosyVoice focuses on natural speech generation, can control timbre and emotion, and supports five languages: Chinese, English, Japanese, Cantonese, and Korean. FunAudioLLM is suitable for scenarios such as multilingual translation and emotional voice conversations. The related models and code have been open-sourced on the Modelscope and Huggingface platforms.
Main features of FunAudioLLM
- SenseVoice model：
- Focuses on high-precision multilingual speech recognition.
- Supports more than 50 languages, and its recognition performance in Chinese and Cantonese is better than that of existing models.
- Has emotion recognition capabilities and can identify various human-computer interaction events.
- Provides two versions, lightweight and large, to adapt to different application scenarios.
- CosyVoice model：
- Focuses on natural speech generation and supports multilingual, timbre, and emotion control.
- Can quickly generate simulated timbres based on a small amount of original audio, including prosody and emotional details.
- Supports cross-lingual speech generation and fine-grained emotion control.
Project address of FunAudioLLM
- Project official website：https://fun-audio-llm.github.io/
- CosyVoice online experience:https://www.modelscope.cn/studios/iic/CosyVoice-300M
- SenseVoice online experience:https://www.modelscope.cn/studios/iic/SenseVoice
- GitHub repository：https://github.com/FunAudioLLM
- arXiv technical paper:https://arxiv.org/abs/2407.04051
Application scenarios of FunAudioLLM
- Developers and researchers: Use FunAudioLLM for research and development in fields such as speech recognition, speech synthesis, and sentiment analysis.
- Enterprise users: Apply FunAudioLLM in business scenarios such as customer service, intelligent assistants, and multilingual translation to improve efficiency and user experience.
- Content creators: Use FunAudioLLM to generate audiobooks or podcasts, enrich content formats, and attract more listeners.
- Education sector: Used in educational applications such as language learning and listening training to improve learning efficiency and interest.
- People with disabilities: Help visually impaired people obtain information through voice interaction and improve convenience in life.