What is Ichigo
Ichigo is an open-source multimodal AI voice assistant that uses a hybrid-modal model and can process interleaved sequences of speech and text in real time. Based on directly quantizing speech into discrete tokens, it uses a unified transformer architecture to process both speech and text simultaneously, enabling joint cross-modal reasoning and generation. Based on this method, it improves processing speed and reduces computational power consumption, keeping the first-token generation latency at only 111 milliseconds, far lower than existing models, allowing Ichigo to provide a near-real-time voice interaction experience.
Ichigo's main features
- Real-time voice processing: Ichigo processes voice input in real time, converting it into discrete tokens to achieve fast responses.
- Cross-modal interaction: Supports interleaved sequence processing of speech and text, enabling true cross-modal interaction.
- Multi-turn dialogue management: Maintains contextual understanding across multi-turn dialogues and provides accurate and personalized answers.
- Ambiguous input handling: When faced with unclear voice input or background noise, it asks the user to repeat, ensuring interaction accuracy.
- Multilingual support: Thanks to pretraining on multilingual speech recognition datasets, Ichigo supports processing in multiple languages.
Ichigo's technical principles
- Hybrid-modal early fusion: Ichigo is based on early fusion technology, merging speech and text data during the input stage to improve efficiency.
- Unified transformer architecture: Uses a unified transformer architecture to process quantized speech and text tokens, supporting cross-modal learning and feature sharing.
- Speech-to-token conversion: Uses WhisperVQ technology to convert continuous speech signals into discrete tokens for the model to process.
- Low-latency real-time performance: The average latency for first-token generation is only 111 milliseconds, providing excellent real-time processing capability.
- Multilingual pretraining: Uses multilingual speech recognition datasets during the pretraining stage, giving the model the ability to handle multiple languages.
Ichigo's project address
- GitHub repository：https://github.com/homebrewltd/ichigo
- HuggingFace model library：https://huggingface.co/collections/homebrewltd/ichigo-66ffc7484ef31ec5596ef6d0
- arXiv technical paper：https://arxiv.org/pdf/2410.15316
Ichigo's application scenarios
- Smart home control: Ichigo can be integrated into smart home systems to control smart devices in the home with voice commands, such as lights, temperature, security systems, etc.
- Virtual personal assistant: As a personal assistant, Ichigo helps users manage schedules, remind them of important events, query information, send messages, etc.
- Customer service: In the customer service field, Ichigo acts as a chatbot, providing 24*7 automated customer support and handling common questions and requests.
- Education and training: As an educational assistance tool, Ichigo provides language learning support, course content explanations, and interactive learning experiences.
- Health consultation: In the healthcare field, Ichigo provides basic health consultation services, such as symptom checks, health advice, and initial responses to emergencies.