What is Ovis2
Ovis2 is a new multimodal large language model launched by the Alibaba International team, based on structured embedding alignment to resolve the differences between visual and text modalities. Ovis2 inherits and optimizes the Ovis series architecture, strengthens the capability density of small-scale models, and significantly improves chain-of-thought (CoT) reasoning ability based on instruction fine-tuning and preference learning. Ovis2 introduces video and multi-image processing capabilities, enhances multilingual support and OCR capability in complex scenarios. The Ovis2 series includes six model versions with different parameter scales: 1B, 2B, 4B, 8B, 16B, and 34B, all demonstrating outstanding performance on the OpenCompass multimodal evaluation leaderboard, with particularly strong performance in mathematical reasoning and video understanding. The open-sourcing of Ovis2 provides new directions and tools for the research and application of multimodal large models.
Main Features of Ovis2
- Multimodal understanding and generation: Processes multiple input modalities such as text, images, and video, generates high-quality text output, and supports visual and language tasks in complex scenarios.
- Enhanced reasoning ability: Based on improved chain-of-thought (CoT) reasoning ability, solves complex logic and math problems, and provides step-by-step reasoning solutions.
- Video and multi-image processing: Introduces video understanding capability, supports key frame selection and multi-image input, and processes complex visual information across frames.
- Multilingual support and OCR capability: Supports text processing in multiple languages and extracts structured data from complex visual elements (such as tables and charts).
- Small model optimization: Based on optimized training strategies, enables small-scale models to achieve high capability density and meet the needs of different application scenarios.
Technical Principles of Ovis2
- Structured embedding alignment: Based on a visual tokenizer, images are split into patches, features are extracted and mapped to 'visual words' to form probabilistic visual tokens. Visual tokens are input into the LLM together with text tokens to achieve structured alignment between modalities.
- Four-stage training strategy：
- Stage 1: Freeze the LLM, train the visual module, and learn the transformation of visual features into embeddings.
- Stage 2: Further train the visual module to enhance high-resolution image understanding and multilingual OCR capabilities.
- Stage 3: Use dialogue-form visual data to align visual embeddings with the LLM's dialogue format.
- Stage 4: Conduct multimodal instruction training and preference learning to improve the model's ability to follow user instructions and output quality.
- Enhanced video understanding: Use the MDP3 algorithm (based on frame-text relevance, compositional diversity, and sequentiality) to select key frames and improve video understanding ability.
- Based on Transformer architecture: Combines powerful visual encoders (such as ViT) and language models (such as Qwen) to achieve efficient multimodal fusion and generation.
Ovis2 Project Address
- GitHub repository：https://github.com/AIDC-AI/Ovis
- HuggingFace model library：https://huggingface.co/collections/AIDC-AI/ovis2
Application Scenarios of Ovis2
- Researchers and developers: Professionals engaged in artificial intelligence and multimodal technology research, as well as developers who need to build intelligent applications, for model optimization, algorithm improvement, or developing multimodal applications.
- Content creators: Professionals in news media, advertising, marketing, and other industries can quickly generate descriptions, copy, titles, and more for images or videos, improving creative efficiency.
- Educators and students: Teachers generate explanatory text for images or videos to help students understand complex content; students use the visual question answering feature to solve problems in their learning.
- Enterprise users: Professionals in finance, law, healthcare, and other industries process complex documents, images, or video data, extract key information, and support decision-making.
- General users and technology enthusiasts: People interested in artificial intelligence can perform simple multimodal tasks, such as generating image descriptions or conducting visual question answering, to explore the application of technology in daily life.