What is JiaoJiao
JiaoJiao is the world's first spoken dialogue emotional large model independently developed purely within academia, launched by the Auditory Cognition and Computational Acoustics Laboratory of Shanghai Jiao Tong University. JiaoJiao has powerful functions including multi-person dialogue, multilingual communication, dialect understanding, role-playing, emotional interaction, and knowledge question answering. It supports multiple languages such as Chinese, English, Japanese, and French, and can accurately recognize Chinese dialects. Based on innovative technology, JiaoJiao achieves end-to-end spoken dialogue, multilingual understanding, multi-person interaction, and real-time timbre cloning. JiaoJiao demonstrates powerful voice interaction capabilities and brings new breakthroughs to the field of intelligent voice assistants.
Main Functions of JiaoJiao
- Multi-person Dialogue: Conduct natural and fluent conversations with multiple users simultaneously, accurately identify each person's identity and spoken content, and provide personalized responses.
- Multilingual Communication: Supports the four mainstream languages of Chinese, English, Japanese, and French, and has cross-language reply capabilities.
- Role-playing and Emotional Interaction: Understands user emotions based on dialogue content and scenarios, and generates emotionally rich responses.
- Knowledge Question Answering: Covers a wide range of knowledge domains, such as reciting ancient poetry, explaining scientific principles, and interpreting literary classics.
- Real-time Timbre Cloning: Provides high-fidelity voice imitation technology, supports multi-character voice acting styles, and enables real-time seamless switching with the user's own voice.
Technical Principles of JiaoJiao
- End-to-end Spoken Dialogue: Based on a robust audio encoder, the audio input is streamingly encoded into discrete sequences and aligned to the text sequence space. Without large-scale high-quality data fine-tuning, it can maintain and leverage the basic generalization ability of large text models to achieve real-time knowledge question answering.
- Multilingual Understanding and Generation: Based on an innovative cross-modal alignment mechanism, multilingual speech signals and corresponding texts are precisely mapped in the feature space. Implicit representation learning is used to preserve language-specific information, and combined with the context modeling capability of deep language models, it achieves seamless switching and efficient semantic understanding in cross-language scenarios.
- Multi-person Dialogue Modeling: Constructs multi-person dialogue data, simulates real scenarios, and enhances the model's dialogue processing capability. Uses an end-to-end model to fuse context information, generate personalized responses and summaries, and achieve natural and coherent multi-party interaction.
- Emotional Understanding and Expression: Based on context information, uses chain-of-thought technology to generate global emotional representations that fit the dialogue scenario, and applies them to vivid emotional speech response generation, enhancing the realism of conversational communication.
- Real-time Timbre Cloning and Switching: Provides high-fidelity voice imitation technology, performs control signal reasoning based on chain-of-thought technology, and supports multi-character voice acting styles and real-time seamless switching with the user's own voice.
- Flexible Expansion: A powerful alignment strategy supports arbitrary concatenation and fusion of text and audio modalities, providing a unified and extensible interface for integrating various enhancement mechanisms in large-scale text models, such as web search and RAG retrieval-augmented generation.
JiaoJiao Project Address
- Experience Application Address：https://wj.sjtu.edu.cn/q/4FiP8hsB
Application Scenarios of JiaoJiao
- Educational Tutoring: Provides personalized learning guidance for students, answers questions, and assists teachers in teaching.
- Family Interaction: For entertainment and adding fun at family gatherings, and for everyday companionship to chat with family and relieve boredom.
- Business Communication: Assists with meeting notes and summaries, and supports cross-language business communication.
- Customer Service Support: Quickly responds to customer inquiries, provides professional answers, and improves service efficiency.
- Entertainment Companion: Participates in role-playing, provides emotional support, and adds fun to life.