What is Liquid
Liquid is a minimalist unified multimodal generation framework jointly launched by Huazhong University of Science and Technology, ByteDance, and the University of Hong Kong. Based on VQGAN, it encodes images into discrete visual tokens that share the same vocabulary space as text tokens, enabling large language models (LLMs) to achieve visual generation and understanding without modifying their structure. Liquid abandons traditional external visual modules and uses the semantic understanding capabilities of LLMs for multimodal tasks, significantly reducing training costs (saving 100 times compared to training from scratch) and performing excellently in visual generation and understanding tasks, surpassing some diffusion models. Liquid reveals the scaling laws of multimodal tasks, proving that as model scale increases, the conflict between visual and language tasks gradually disappears, and the two can mutually promote each other.
Main features of Liquid
- Visual generation: Generates high-quality images based on text descriptions, supporting multiple resolutions and styles.
- Visual understanding: Handles image-related tasks such as visual question answering (VQA) and image caption generation.
- Multimodal fusion: Seamlessly combines visual and language tasks, supporting simultaneous processing of text generation, image generation, and visual understanding tasks.
- Efficient scaling: Based on existing large language models (LLMs), it rapidly scales multimodal capabilities with a small amount of data and low-cost training.
- Preservation of language capabilities: While adding visual generation capabilities, it maintains strong language generation and understanding capabilities, suitable for mixed multimodal tasks.
Technical principles of Liquid
- Image Tokenizer: VQGAN (Vector Quantized Generative Adversarial Network) encodes images into discrete visual tokens. Visual tokens share the same vocabulary space as text tokens, allowing images and text to be processed in a unified way.
- Unified feature space: Visual tokens and text tokens are learned in the same feature space and trained based on the "next token prediction" task. This supports the model in seamlessly switching and optimizing between visual and language tasks.
- LLM-based generation: Liquid extends existing large language models (LLMs) and performs visual generation and understanding based on powerful semantic understanding capabilities. It adds visual token embeddings to the LLM to handle visual tasks without additional visual modules (such as CLIP or diffusion models).
- Multimodal data training: It uses mixed data (text data, image-text pair data) for pretraining, and the model simultaneously learns language and visual tasks. By adjusting the data ratio, it optimizes the model's performance on different tasks.
- Bidirectional promotion mechanism: Visual generation and visual understanding tasks share a unified token space with consistent optimization objectives, so the two can mutually promote each other. Increasing the data for visual generation or understanding tasks improves the model's performance on the other task.
Project addresses of Liquid
- Project official website：https://foundationvision.github.io/Liquid/
- GitHub repository：https://github.com/FoundationVision/Liquid
- HuggingFace model library：https://huggingface.co/Junfeng5/Liquid
- arXiv technical paper：https://arxiv.org/pdf/2412.04332
- Online experience Demo：https://huggingface.co/spaces/Junfeng5/Liquid_demo
Application scenarios of Liquid
- Creative design: Generates high-quality images based on text descriptions, assisting artistic creation, advertising design, and game art.
- Content creation: Automatically generates images related to text, used for social media, blogs, and news reports.
- Visual Question Answering: Understand image content and answer related questions, used in education, customer service, and intelligent assistants.
- Multimodal Dialogue: Combine images and text for intelligent interaction, enhancing the naturalness and practicality of dialogue systems.
- VR/AR Applications: Generate virtual scenes and objects, enhancing immersion and interactive experience.