What Is UniTok
UniTok is a unified visual tokenizer jointly launched by ByteDance, the University of Hong Kong, and Huazhong University of Science and Technology, capable of supporting both visual generation and understanding tasks simultaneously. Based on multi-codebook quantization technology, it partitions visual features into multiple small blocks, each quantized with an independent sub-codebook, greatly expanding the representational capacity of discrete tokenization and resolving the contradiction between detail capture and semantic understanding in traditional tokenizers. UniTok achieves a zero-shot classification accuracy of 78.6% on ImageNet, with a reconstruction quality (rFID) of only 0.38, significantly outperforming existing tokenizers. Multimodal large language models (MLLMs) built on UniTok perform excellently in visual question answering and image generation tasks, demonstrating strong potential in multimodal tasks.
Main Features of UniTok
- Unified Visual Representation: Encodes images into discrete visual tokens that can be used for image generation tasks (e.g., text-to-image) as well as visual understanding tasks (e.g., visual question answering).
- High-Quality Image Reconstruction: Performs efficient image reconstruction while preserving image details.
- Semantic Alignment: Combines contrastive learning and reconstruction loss to ensure the generated visual tokens align with text descriptions, enhancing visual understanding capabilities.
- Support for Multimodal Large Language Models (MLLMs): Serves as the visual input module for multimodal large language models, supporting unified processing and generation in multimodal tasks.
Technical Principles of UniTok
- Multi-Codebook Quantization: UniTok splits visual tokens into multiple small blocks, each quantized with an independent sub-codebook. For example, a 64-dimensional visual feature vector is split into 8 8-dimensional blocks, each quantized based on a sub-codebook of 4096 codewords. In this way, the theoretical vocabulary size of UniTok can grow exponentially, greatly expanding the representational capacity of discrete tokens.
- Attention Decomposition: Replaces traditional linear projection layers with multi-head attention modules for token decomposition, better preserving semantic information in the original tokens and enhancing the expressiveness of the decomposed features. UniTok uses causal attention to ensure compatibility with autoregressive generation tasks.
- Unified Training Objective: The reconstruction loss based on VQVAE ensures that image details are accurately reconstructed. The loss includes pixel-level reconstruction error, perceptual loss, discriminator loss, and vector quantization loss. UniTok introduces a CLIP-like contrastive loss to ensure the generated visual tokens align with text descriptions, enhancing visual understanding capabilities. The final total loss is a weighted sum of the reconstruction loss and the contrastive loss. Based on this approach, UniTok can simultaneously optimize generation and understanding tasks.
- Integration with Multimodal Large Language Models (MLLMs): The generated visual tokens are mapped to the token space of the multimodal large language model through an MLP projection layer, achieving unified processing of vision and language. To simplify the input to the MLLM, UniTok merges tokens generated from multiple sub-codebooks into a single visual token input to the MLLM. When visual tokens need to be predicted, the MLLM autoregressively predicts the multiple sub-codebook tokens corresponding to the next position, achieving efficient visual generation.
UniTok Project Address
- Project Website：https://foundationvision.github.io/UniTok/
- GitHub Repository：https://github.com/FoundationVision/UniTok
- HuggingFace Model Library：https://huggingface.co/FoundationVision/unitok_tokenizer
- arXiv Technical Paper：https://arxiv.org/pdf/2502.20321
Application Scenarios of UniTok
- Visual Input for Multimodal Models: As the visual module of a multimodal large language model, it helps the model process image and text information simultaneously and improves overall performance.
- High-Quality Image Generation: Generates detailed images based on text descriptions, suitable for fields such as creative design and advertising production.
- Visual Question Answering and Understanding: Assists the model in understanding image content and answering vision-related questions, used in education, medical image analysis, and more.
- Multimodal Content Creation: Quickly generates image and text content, used in news reporting, social media, and more, improving creative efficiency.
- Cross-Modal Retrieval and Recommendation: Performs retrieval and recommendation based on text or images, improving the user experience of e-commerce platforms and multimedia platforms.