What is MiniCPM-o 4.5
MiniCPM-o 4.5 is ModelBest's open-source 9B-parameter omni-modal flagship model, adopting an end-to-end architecture that integrates SigLip2、Whisper、CosyVoice2 and Qwen3-8B. As the industry's first model to support "instant free conversation," the model achieves full-duplex interaction—it can see, listen, and speak simultaneously, breaking away from the traditional turn-based "walkie-talkie" mode. The model features leading visual understanding, hyper-realistic speech generation, andvoice cloningcapabilities, supports proactive interaction and real-time streaming media processing, can run on edge devices, has been adapted for Ascend, Hygon, and other domestic chips, and achieves efficient deployment through frameworks such as llama.cpp and vLLM.
Main Features of MiniCPM-o 4.5
-
Full-duplex real-time interaction: The model can simultaneously process visual and audio input and generate speech output, achieving parallel perception and expression of seeing, listening, and speaking.
-
Proactive intelligent interaction: The model autonomously monitors environmental changes once per second, proactively determines when to speak, and achieves human-like interactive behaviors such as proactive reminders and real-time commentary.
-
Hyper-realistic speech synthesis: Supports end-to-end speech generation with rich emotion and natural timbre, can clone customized voices based on a few seconds of audio samples, and maintains stable consistency in long speech synthesis.
-
Leading visual understanding: Surpasses GPT-4o and Gemini 2.0 Pro with 9B parameters in the OpenCompass evaluation, and supports high-resolution image parsing and high-frame-rate video real-time understanding.
-
End-to-end document parsing: Achieves industry-best performance on the OmniDocBench benchmark, and can efficiently handle English document understanding and structured extraction tasks with complex layouts.
Technical Principles of MiniCPM-o 4.5
- End-to-end omni-modal architecture design: MiniCPM-o 4.5 performs end-to-end joint training of the SigLip2 vision encoder, Whisper-medium audio encoder, CosyVoice2 speech decoder, and Qwen3-8B language model through dense feature connections. The tightly coupled design allows information from each modality to flow freely within the model, avoiding information loss and error accumulation in traditional pipeline architectures, and achieving more accurate multimodal understanding and generation control.
- Full-duplex multimodal real-time streaming mechanism: The model transforms offline modality codecs into online versions that support streaming input and output, and the speech decoder uses interleaved modeling of text and speech tokens to achieve full-duplex generation. During inference, a time-division multiplexing mechanism divides parallel multimodal data streams into sequential information groups within millisecond-level time slices, enabling the language model backbone to uniformly schedule and process them, efficiently completing synchronized perception and response to real-time audio-video streams within a single architecture.
- Proactive interaction decision mechanism: The language model module continuously monitors the input video stream and audio stream, automatically triggering speech decisions at a frequency of 1 Hz. The high-frequency decision-making capability, combined with full-duplex characteristics, enables the model to autonomously choose the most appropriate timing and content for replies based on dynamic environmental changes, breaking through the limitation of traditional models passively waiting for user instructions.
- Configurable speech modeling design: The model continues the design paradigm of multimodal system prompts, supporting both text system prompts and audio system prompts as dual inputs. Audio system prompts are used to specify target timbre characteristics. This design allows the model to complete voice cloning and role-playing during inference by providing only a short reference audio sample.
Project Address of MiniCPM-o 4.5
- GitHub Repository：https://github.com/OpenBMB/MiniCPM-o
- HuggingFace Model Hub：https://huggingface.co/openbmb/MiniCPM-o-4_5
- Online Demo：https://huggingface.co/spaces/openbmb/minicpm-omni
Application Scenarios of MiniCPM-o 4.5
- Intelligent Assistant and Companionship: As an all-around AI assistant, the model can perceive the user's environment and emotions in real time, proactively provide reminders, suggestions, or emotional companionship, and support personalized voice cloning to create a tailored interactive experience.
- Real-Time Video Interaction: Suitable for scenarios such as video surveillance analysis, live real-time commentary, and remote teaching and tutoring, it can simultaneously understand visual content and voice commands and provide immediate voice feedback.
- Intelligent Customer Service and Shopping Guidance: It provides natural and fluent voice services in fields such as e-commerce, finance, and government affairs, supports multi-turn dialogue and proactive recommendations, and improves user service experience and business conversion efficiency.
- Education and Training: Used for language learning practice, virtual teachers, skills training, and more, it achieves immersive interactive teaching through the combination of visual demonstration and voice explanation.
- Content Creation and Entertainment: It supports audiobook generation, virtual character dubbing, game NPC interaction, and more. The voice cloning feature can quickly replicate a specific person's voice for role-playing.