What is UniFluid
UniFluid is a unified autoregressive framework jointly launched by Google DeepMind and MIT, used for joint visual generation and understanding tasks. Based on continuous visual tokens, it processes multimodal image and text inputs, generating discrete text tokens and continuous image tokens. The framework is based on the pretrained Gemma model, trained with paired image-text data, allowing generation and understanding tasks to mutually reinforce each other. UniFluid uses standard SentencePiece as the text tokenizer, a continuous variational autoencoder (VAE) as the tokenizer for image generation, and combines the SigLIP image encoder for understanding tasks. Based on a carefully tuned training recipe and balanced loss weights, UniFluid achieves results comparable to or better than single-task baselines on both image generation and understanding tasks, demonstrating strong downstream task transfer capabilities, including image editing, visual captioning, and question answering.
UniFluid's Main Features
- Joint Visual Generation and Understanding: Simultaneously handles image generation (such as generating images from text descriptions) and visual understanding (such as image captioning and visual question answering) tasks.
- Multimodal Input Processing: Supports multimodal input of images and text, embedding images and text into the same space for joint training.
- High-Quality Image Generation: Generates high-quality images based on continuous visual tokens, and supports random generation order to improve generation results.
- Powerful Visual Understanding Capabilities: Performs excellently on tasks such as visual question answering and image captioning, and supports a variety of downstream tasks.
- Efficient Downstream Task Transfer: Quickly adapts to downstream tasks such as image editing and visual question answering, demonstrating good generality and scalability.
UniFluid's Technical Principles
- Unified Autoregressive Framework: UniFluid uses an autoregressive model to embed image and text inputs into the same space, and uniformly handles visual generation and understanding tasks based on "next-token prediction".
- Continuous Visual Tokens: Images are encoded into continuous tokens using a continuous variational autoencoder (VAE), avoiding the information loss caused by discrete tokens and preserving the continuity of images.
- Modality-Specific Prediction Heads: The classification head handles text generation tasks, and the diffusion head handles image generation tasks, ensuring effective training and inference under different modalities.
- Random Generation Order: In image generation tasks, image tokens are generated in random order, avoiding generation problems caused by fixed order (such as raster order).
- Loss Function Balancing: Adjusts the loss weights of image generation and text understanding tasks to achieve a balance between the two tasks, allowing the model to perform excellently on both generation and understanding tasks.
- Pretrained Large Language Model (LLM): Based on the pretrained Gemma model, it uses its powerful language and visual understanding capabilities to improve the model's overall performance.
UniFluid's Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2503.13436
UniFluid's Application Scenarios
- Image Generation: In fields such as creative design, advertising production, and game development, quickly generate corresponding image content based on user-input text descriptions.
- Image Editing: In scenarios such as photo editing and digital art creation, adding or removing objects, changing styles, adjusting colors, etc.
- Visual Question Answering (VQA): In fields such as education, intelligent customer service, and assisting visually impaired people, it helps users better understand and interpret image information.
- Image Captioning): Automatically generates descriptive text for images, used in social media content generation, image search engine optimization, assisting visually impaired people in understanding images, etc.
- Multimodal Content Creation: In video script creation and virtual reality (VR) and augmented reality (AR) content development, it provides a more vivid visual and linguistic experience.