What is LLaDA-V
LLaDA-V is a multimodal large language model (MLLM) launched by the Gaoling School of Artificial Intelligence at Renmin University of China and Ant Group. It is based on a pure diffusion model architecture and focuses on visual instruction tuning. Building on LLaDA, the model introduces a vision encoder and an MLP connector to map visual features into the language embedding space, achieving effective multimodal alignment. LLaDA-V reaches state-of-the-art performance in multimodal understanding, surpassing existing hybrid autoregressive-diffusion and pure diffusion models.
Main Features of LLaDA-V
- Image Caption Generation: Generates detailed description text based on the input image.
- Visual Question Answering: Answers questions related to image content.
- Multi-turn Multimodal Dialogue: Conducts multi-turn dialogues within the context of a given image, understanding and generating responses related to the image and dialogue history.
- Complex Reasoning Tasks:Performs reasoning in composite tasks involving images and text, such as solving image-related math problems or logic problems.
Technical Principles of LLaDA-V
- Diffusion Models: Diffusion models generate data by gradually removing noise. In LLaDA-V, masked diffusion models are used, where some words in a sentence are randomly masked (replaced with a special token [M]) to train the model to predict the original content of the masked words.
- Visual Instruction Tuning: Based on a visual instruction tuning framework, the framework includes a Vision Tower and an MLP Connector. The Vision Tower uses the SigLIP 2 model to convert images into visual representations. The MLP Connector maps the visual representations into the word embedding space of the language model, enabling effective alignment and fusion of visual features and language features.
- Multi-stage Training Strategy: In the first stage, the MLP Connector is trained to align visual representations and language embeddings. In the second stage, the entire model is fine-tuned to understand and follow visual instructions. In the third stage, the model's multimodal reasoning ability is further enhanced by training it to handle complex multimodal reasoning tasks.
- Bidirectional Attention Mechanism: In multi-turn dialogues, LLaDA-V uses a bidirectional attention mechanism, allowing the model to consider the entire dialogue context when predicting masked words, which helps the model better understand the overall logic and content of the dialogue.
Project Addresses of LLaDA-V
- Project Official Website：https://ml-gsai.github.io/LLaDA-V
- GitHub Repository：https://github.com/ML-GSAI/LLaDA-V
- arXiv Technical Paper：https://arxiv.org/pdf/2505.16933
Application Scenarios of LLaDA-V
- Image Caption Generation: Automatically generates detailed descriptions of images to help understand image content.
- Visual Question Answering: Answers image-related questions, suitable for fields such as education and tourism.
- Multi-turn Dialogue: Conducts multi-turn multimodal dialogues in scenarios such as intelligent customer service and virtual assistants.
- Complex Reasoning: Solves complex reasoning tasks involving images and text, such as math problem solving.
- Multi-image and Video Understanding: Analyzes multi-image and video content, suitable for scenarios such as video analysis and surveillance.