What is BLIP3-o
BLIP3-o is an innovative multimodal model launched by Salesforce Research and other institutions, integrating the reasoning and instruction-following capabilities of autoregressive models with the powerful generation capabilities of diffusion models. The model is based on diffusing semantically rich CLIP image features, rather than relying on traditional VAE features or raw pixels, and performs excellently in image understanding and generation. BLIP3-o uses a sequential pretraining strategy, first performing image understanding training, then performing image generation training, preserving image understanding capabilities while developing powerful image generation capabilities. The model achieves excellent results on multiple image understanding and generation benchmarks, and is fully open source, including code, model weights, pretraining and instruction tuning datasets.
Main features of BLIP3-o
- Text-to-text: Generates descriptive text related to images.
- Image-to-text: Understands input images and generates descriptive text, supporting various image understanding tasks such as visual question answering (VQA) and image classification.
- Text-to-image: Generates high-quality images based on input text descriptions.
- Image-to-image: Edits and modifies input images to generate new images.
- Mixed training: Supports mixed training of image generation and understanding tasks, improving the model's overall performance.
Technical principles of BLIP3-o
- Combination of autoregressive models and diffusion models: The autoregressive model generates intermediate visual features and captures semantic information in text descriptions. The diffusion model generates the final image. The diffusion model generates images by gradually removing noise, producing high-quality and diverse images.
- CLIP feature diffusion: Uses the CLIP model to encode images and generate semantically rich feature vectors. The feature vectors are more compact and more informative than traditional VAE features. Based on the diffusion model, CLIP features are modeled to generate feature vectors similar to the target image features, achieving high-quality image generation.
- Sequential pretraining strategy: First, the model is pretrained on image understanding tasks to ensure that the model has strong image understanding capabilities. On the basis of image understanding pretraining, the weights of the autoregressive model are frozen, and only the diffusion model is trained, achieving efficient image generation.
- Flow matching loss function: Uses the flow matching loss function to train the diffusion model. The loss function can better capture the distribution of image features and generate higher-quality images. The flow matching loss function introduces randomness, allowing the model to generate diverse images, not just a single output.
- Instruction tuning dataset: Based onGPT-4ogenerated diverse prompts, creates a dataset containing 60k high-quality prompt-image pairs, used to fine-tune the model and improve instruction-following ability and visual aesthetic quality.
Project address of BLIP3-o
- GitHub repository：https://github.com/JiuhaiChen/BLIP3o
- HuggingFace model library：https://huggingface.co/BLIP3o
- arXiv technical paper：https://arxiv.org/pdf/2505.09568
Application scenarios of BLIP3-o
- Image generation and editing: Generates or modifies images based on text descriptions, assisting design and creative work.
- Visual question answering: Understands image content and answers related questions, and can be used in education and intelligent customer service.
- Multimodal dialogue: Combines images and text for dialogue, improving the interactive experience.
- Image annotation and classification: Automatically generate image tags and perform classification, optimizing image management.
- Art and creativity: Generate artistic images, inspire creative inspiration, and meet personalized needs.