Qwen VLo - A multimodal unified understanding and generation model launched by Tongyi Qianwen

What is Qwen VLo

Qwen VLo is a multimodal unified understanding and generation model launched by the Tongyi Qianwen team. It has been comprehensively upgraded on the basis of multimodal large models, can "understand" the world, and can perform high-quality re-creation based on understanding, achieving a leap from perception to generation. It can accurately understand image content and, on this basis, perform consistent and high-quality generation. Users can use natural language instructions to ask the model to perform style transfer, scene reconstruction, or detail refinement on images, and the model can respond flexibly and generate results that meet expectations. Qwen VLo supports multilingual instructions, breaking down language barriers and providing a convenient interactive experience for users worldwide. It has the capability for dynamic resolution training and generation, supports image generation at arbitrary resolutions and aspect ratios, and is suitable for a variety of scenarios.

Main features of Qwen VLo

• Accurate content understanding and re-creation: Qwen VLo can accurately understand image content and maintain a high degree of semantic consistency during the generation process. For example, users can upload a photo of a car and ask to "change the color," and the model can accurately identify the car model, preserve the structural features of the original image, and complete a natural transformation of the color style.
• Open instruction editing and modification: Users can propose various creative instructions through natural language, such as "change this painting style to Van Gogh style" or "add a clear sky to this image." The model can flexibly respond to these instructions and complete tasks such as artistic style transfer, scene reconstruction, and detail refinement, and can complete complex instructions containing multiple operations at once.
• Multilingual instruction support: Qwen VLo supports instructions in multiple languages such as Chinese and English, breaking down language barriers and providing a convenient interactive experience for users worldwide.
• Dynamic resolution generation: The model uses dynamic resolution training and supports image generation at arbitrary resolutions and aspect ratios, making it suitable for scenarios such as posters, illustrations, and web banners.
• Progressive generation mechanism: Qwen VLo generates images in a progressive manner from left to right and top to bottom, allowing the generation process to be observed and adjusted in real time for a more flexible and controllable creative experience.
• Image detection and annotation: Qwen VLo can complete annotation tasks for existing information, such as detection, segmentation, and edge detection.
• Text-to-image generation: Qwen VLo supports directly generating images from text descriptions, including general images and Chinese and English posters.

Technical principles of Qwen VLo

• Model architecture: The architecture of Qwen VLo mainly consists of the following key modules:
  • Visual Encoder: It uses the Vision Transformer (ViT) architecture to split the input image into multiple patches of fixed size and convert them into serialized feature vectors. To support dynamic resolution, Qwen VLo modified ViT by removing the original absolute position embeddings and introducing 2D-RoPE (Rotary Position Embedding) to capture the two-dimensional position information of the image.
  • Input Projector: Through a single-layer cross-attention module, the visual feature sequence is compressed to a fixed length (for example, 256) to improve processing efficiency. At the same time, this module integrates two-dimensional absolute position encoding to preserve position information.
  • Large Language Model (LLM): Based on Qwen-7B, initialized with pre-trained weights, responsible for handling input in the language modality.
  • Output Projector: Maps the features generated by the LLM to a feature space understandable by the modality generator, usually a simple Transformer layer or MLP layer.
  • Modality Generator: A derivative model based on LDM (Latent Diffusion Models), responsible for generating the final image output.
• Dynamic Resolution Mechanism: Qwen VLo introduces a dynamic resolution mechanism that can handle image inputs of any resolution:
  • Dynamic Visual Token Conversion: The model dynamically generates a variable number of visual tokens based on the resolution of the input image, avoiding information loss caused by scaling high-resolution images to low resolution.
  • Smart Resize: During inference, images are resized to dimensions that are integer multiples of 28, preserving the aspect ratio as much as possible and avoiding distortion.
  • Token Compression: Through a simple MLP layer, adjacent 2×2 tokens are compressed into a single token to reduce the sequence length of the visual input.
• Training Method: Qwen VLo's training is divided into three stages:
  • Stage 1: Single-Task Large-Scale Pre-training: Pre-training with a large amount of image-text pair data, with training images uniformly processed to a size of 224×224. This stage mainly trains the model's ability to align the visual modality with the language model.
  • Stage 2: Multi-Task Pre-training: Using higher-resolution (448×448) data and introducing multiple visual and text generation tasks to enhance the model's multimodal task processing capability.
  • Stage 3: Instruction Fine-Tuning (SFT): Constructing multimodal multi-turn conversation data through manual annotation, model generation, etc., to enhance the model's instruction-following and dialogue capabilities.
• Progressive Generation Mechanism: Qwen VLo adopts a progressive generation approach from left to right and top to bottom, gradually constructing the image content. During the generation process, the model continuously adjusts and optimizes the predicted content to ensure the final result is harmonious and consistent. It is suitable for long-paragraph text generation tasks that require fine control, allowing users to observe the generation process in real time and make adjustments.
• Multimodal Fusion: Qwen VLo achieves unified processing of multimodal data by fusing visual features and language features. The model can perform operations such as editing, style transfer, and generation on images according to the text instructions input by the user, and supports multilingual instructions.

How to Use Qwen VLo

• Visit Qwen Chat: Visit Qwen Chat 's official website.
• Upload an Image or Enter Text: Upload an image to the platform, or enter a text instruction.
• Enter an Instruction: Enter a natural language instruction as needed, for example, "change this painting's style to Van Gogh's style" or "add a clear sky to this image".
• View the Generated Result: The model will generate an image or perform editing based on the instruction, and display the result.

Application Scenarios of Qwen VLo

• Image Editing and Generation: Qwen VLo can convert an image's style from one to another, for example converting a cartoon style to a realistic style.
• Visual Question Answering (VQA): Qwen VLo can answer questions related to image content, such as describing the scene in an image, identifying objects in an image, and so on.
• Document Parsing: Qwen VLo can parse image-based documents (such as scans or image PDFs), identifying the positional information of elements such as text, images, and tables within them.
• Text Recognition and Information Extraction: Supports recognizing text and formulas from images, or extracting information from receipts, certificates, and forms.
• Video Understanding: Qwen VLo can analyze video content, for example locating events in a video and obtaining timestamps, or generating summaries of key time periods.
• Design and Creativity: Qwen VLo can provide powerful tool support for designers, marketers, educators, and others, quickly bringing ideas to life, for example generating posters, illustrations, and so on.