What is OmniAlign-V
OmniAlign-V is a high-quality dataset jointly launched by Shanghai Jiao Tong University, Shanghai AI Lab, Nanjing University, Fudan University, and Zhejiang University, specifically designed to enhance the alignment capabilities of multimodal large language models (MLLMs) with human preferences. OmniAlign-V contains approximately 200,000 multimodal training samples, covering natural images and infographics, combined with open-ended, knowledge-rich question-answer pairs. The design of OmniAlign-V emphasizes task diversity, including knowledge question answering, reasoning tasks, and creative tasks, enhancing the model's alignment capabilities based on complex questions and diverse answer formats. OmniAlign-V introduces an image screening strategy to ensure that semantically rich and complex images are used for data generation.
Main features of OmniAlign-V
- Provides high-quality multimodal training data: Contains approximately 200,000 multimodal training samples, covering natural images and infographics (such as posters, charts, etc.), combined with complex questions and diverse answer formats, helping the model better understand human preferences and needs.
- Enhances the model's open-ended question answering capabilities: The dataset design emphasizes open-ended questions, interdisciplinary knowledge, and comprehensive answers, enabling the model to generate responses that better align with human preferences.
- Improves the model's reasoning and creative capabilities: Trains the model to engage in more complex thinking and creation, thereby enhancing its performance in multimodal interactions.
- Optimizes multimodal instruction tuning: Based on high-quality instruction tuning data, helps the model better follow human instructions and maintain foundational capabilities (such as object recognition, OCR, etc.).
- Supports continuous optimization of multimodal models: OmniAlign-V is used for supervised fine-tuning (SFT), combined with direct preference optimization (DPO) to further enhance the model's alignment capabilities.
Technical principles of OmniAlign-V
- Image screening and classification: Based on image complexity (IC) scoring and object category (OC) filtering, screens out semantically rich and complex images. Images are classified into natural images and infographics, and different tasks are designed for different types of images.
- Task design and data generation: Natural image tasks include knowledge question answering, reasoning tasks, and creative tasks, enhancing the model's understanding and generation capabilities for real-world scenarios. Infographic tasks are designed specifically for charts, posters, etc., requiring the model to have the ability to understand and interpret complex information. Uses advanced models such as GPT-4o to generate high-quality question-answer pairs, with data quality optimized based on post-processing.
- Post-processing optimization: Post-processes the generated question-answer pairs, including instruction enhancement, reasoning enhancement, and refined processing of infographic answers, ensuring data diversity and high quality.
- Multimodal training and optimization: Enhances the model's alignment capabilities based on supervised fine-tuning (SFT) and direct preference optimization (DPO). The dataset design emphasizes diversity and complexity, enabling the model to better understand human preferences in multimodal interactions.
- Benchmark testing and evaluation: Introduces the MM-AlignBench benchmark to evaluate the performance of MLLMs in human preference alignment, ensuring the model's applicability in real-world scenarios.
Project address of OmniAlign-V
- Project official website：https://phoenixz810.github.io/OmniAlign-V
- GitHub repository：https://github.com/PhoenixZ810/OmniAlign-V
- HuggingFace model library：https://huggingface.co/collections/PhoenixZ/omnialign-v
- arXiv technical paper：https://arxiv.org/pdf/2502.18411
Application scenarios of OmniAlign-V
- Multimodal dialogue systems: Improve the quality of interaction between intelligent assistants and users, making responses more aligned with human preferences.
- Image-assisted question answering: Provide more comprehensive and accurate question-answering services by combining image information, suitable for fields such as education and tourism.
- Creative content generation: Help users quickly generate high-quality creative text, such as advertising copy and story creation.
- Education and learning assistance: Provide students with richer learning materials and assist in understanding complex charts and illustrations.
- Infographic interpretation: Help users interpret complex charts, provide background knowledge and reasoning results, and improve data comprehension.