What is Open-LLM-VTuber
Open-LLM-VTuber is an open-source cross-platform voice interaction AI companion project. It supports real-time voice dialogue and visual perception, is equipped with a vivid Live2D dynamic avatar, and can run completely offline to protect privacy. Users use it as avirtual girlfriend, boyfriendor pet, enjoying a personalized interactive experience. The project integrates multiple large language models (LLM), automatic speech recognition (ASR), and text-to-speech (TTS) solutions, and supports users in customizing character avatars, voices, and interaction features according to their needs.
Main features of Open-LLM-VTuber
- Voice interaction: Supports real-time voice dialogue, allowing users to communicate with the AI by voice without manual input.
- Visual perception: Supports camera input, screen recording, and screenshots, so the AI can "see" the user and screen content.
- Live2D dynamic avatar: Equipped with a vivid Live2D dynamic character, supporting changes in facial expressions and movements.
- Offline operation: All features support fully offline operation locally, protecting user privacy.
- Cross-platform support: Compatible with Windows, macOS, and Linux, and supports GPU acceleration and CPU operation.
- Personalized customization: Users can customize character avatars, voices, and interaction features, including cloning a specific voice.
- Rich interaction features: Supports voice interruption, touch feedback, chat history saving, multilingual TTS, and more.
- Desktop pet mode: Supports transparent background, always-on-top, and mouse click-through, so the AI can move anywhere on the desktop.
Technical principles of Open-LLM-VTuber
- Large language model: As the core interaction engine, the LLM is responsible for understanding user input (voice or text) and generating responses. The project supports multiple LLMs, such as Ollama、OpenAI、Gemini , etc., and users can choose different models according to their needs.
- Speech recognition: Converts the user's voice input into text for the LLM to process. It supports multiple ASR solutions, such as Whisper,FunASR , etc., ensuring the accuracy and efficiency of speech recognition.
- Speech synthesis: Converts the text generated by the LLM into voice output, supports multiple TTS engines, such as MeloTTS、Bark , etc., and supports multilingual synthesis.
- Live2D dynamic avatar: Uses Live2D technology to generate a dynamic character avatar. Based on expression mapping and motion control, the character dynamically displays expressions and movements according to the conversation content or emotional changes.
- Visual perception: Based on camera or screen recording functionality, the AI obtains visual information to achieve a richer interactive experience, such as recognizing the user's facial expressions or screen content.
- Modular design: The project uses a modular architecture, allowing users to switch between different functional modules by modifying simple configuration files without delving into the code.
Project address of Open-LLM-VTuber
- GitHub repository：https://github.com/t41372/Open-LLM-VTuber
Application scenarios of Open-LLM-VTuber
- Virtual companion: Users set it as a virtual girlfriend, boyfriend, or pet, enjoying emotional companionship and personalized interaction to meet emotional needs.
- Office assistant: In desktop pet mode, it provides real-time assistance such as information queries, voice reminders, and document reading to improve office efficiency.
- Learning Assistance: Helps users learn languages and answer questions, assisting learning based on screen sharing.
- Entertainment and Interaction: Users engage in voice games, role-playing, and other entertainment activities with the AI, adding fun.
- Technical Demonstration and Development: Developers develop and demonstrate AI interaction technologies, exploring more application scenarios.