dots.vlm1 - The first multimodal large model open-sourced by Xiaohongshu hi lab

What is dots.vlm1

dots.vlm1 is the first multimodal large model open-sourced by Xiaohongshu hi lab. Built on a 1.2-billion-parameter vision encoder NaViT trained from scratch and the DeepSeek V3 large language model (LLM), it possesses strong visual perception and text reasoning capabilities. The model performs excellently on visual understanding and reasoning tasks, approaching the level of closed-source SOTA models, and remains competitive on text tasks. dots.vlm1's vision encoder NaViT is trained entirely from scratch, natively supports dynamic resolution, and adds pure visual supervision on top of text supervision, enhancing perception capabilities. The training data introduces multiple synthetic data ideas, covering diverse image types and their descriptions, significantly improving data quality.

Main features of dots.vlm1

• Powerful visual understanding capability: Can accurately recognize and understand content in images, including complex charts, tables, documents, graphics, etc., supports dynamic resolution, and is suitable for a variety of visual tasks.
• Efficient text generation and reasoning: Based on DeepSeek V3 LLM, it can generate high-quality text descriptions and performs excellently in text reasoning tasks such as mathematics and code.
• Multimodal data processing: Supports interleaved image-text data processing and can combine visual and textual information for comprehensive reasoning, suitable for multimodal application scenarios.
• Flexible adaptation and extension: Connects the vision encoder and language model through a lightweight MLP adapter, facilitating flexible adaptation and extension across different tasks.
• Open source and openness: Provides complete open-source code and models, supports developers in research and application development, and promotes the advancement of multimodal technology.

Technical principles of dots.vlm1

• NaViT vision encoder: dots.vlm1 uses a 1.2-billion-parameter vision encoder NaViT trained from scratch, rather than fine-tuning based on an existing mature model. It natively supports dynamic resolution, can handle image inputs of different resolutions, and adds pure visual supervision on top of text supervision, enhancing the model's image perception capability.
• Multimodal data training: The model uses diverse multimodal training data, including ordinary images, complex charts, tables, documents, graphics, etc., as well as corresponding text descriptions (such as Alt Text, Dense Caption, Grounding, etc.). It introduces synthetic data ideas and interleaved image-text data from web pages, PDFs, etc., and improves data quality through rewriting and cleaning, enhancing the model's multimodal understanding capability.
• Fusion of vision and language models: dots.vlm1 combines the vision encoder with the DeepSeek V3 large language model (LLM), connected through a lightweight MLP adapter, achieving effective fusion of visual information and linguistic information and supporting the processing of multimodal tasks.
• Three-stage training process: The model's training is divided into three stages: vision encoder pre-training, VLM pre-training, and VLM post-training. By gradually increasing image resolution and introducing diverse training data, it enhances the model's generalization ability and multimodal task processing capability.

Project address of dots.vlm1

• GitHub repository：https://github.com/rednote-hilab/dots.vlm1
• Hugging Face model library：https://huggingface.co/rednote-hilab/dots.vlm1.inst
• Online experience Demo：https://huggingface.co/spaces/rednote-hilab/dots-vlm1-demo

Application scenarios of dots.vlm1

• Complex chart reasoning: Can analyze and reason about complex charts, helping users better understand and interpret the information in the charts.
• STEM problem solving: In the fields of science, technology, engineering, and mathematics (STEM), the model can assist in solving related problems and provide problem-solving approaches.
• Long-tail recognition: For some categories or objects that appear less frequently, dots.vlm1 also has relatively good recognition capabilities.
• Visual reasoning: It can handle reasoning tasks involving visual information, such as obstacle recognition, product comparison and analysis, etc.
• Image-text question answering and interaction: It supports question-answering tasks that combine images and text, can conduct multi-turn conversations, and can give coherent answers based on context.
• Content recommendation: Based on multimodal data, it provides users with personalized content recommendations, such as recommending related images, text, or videos on the Xiaohongshu platform.