Tuna-2 is the official implementation of the paper "Pixel Embeddings Beat Vision Encoders for Unified Understanding and Generation," jointly completed by researchers from Meta, the University of Hong Kong, and the University of Waterloo. The project aims to simplify the visual encoding design of unified multimodal models (UMMs).
Purpose
Tuna-2 is used for unified multimodal understanding and generation tasks. It simplifies the model by progressively stripping away visual encoding components: first, the VAE is removed, yielding Tuna-R, a pixel-space unified multimodal model that relies only on a representation encoder; Tuna-2 further bypasses the representation encoder and directly uses a patch embedding layer to process raw image inputs. According to the project description, Tuna-2 using pixel embeddings outperforms Tuna-R and Tuna on a variety of multimodal benchmarks.
Main Capabilities
- Unified understanding and generation: The model supports both multimodal understanding and image generation tasks.
- Pixel embeddings: Directly performs patch embedding on raw images without the need for a traditional vision encoder.
- Single inference script: All inference is completed through one unified script.
- Support for multiple resolutions: The project lists the supported resolutions.
- Video training and inference code: Due to policy restrictions, the video generation model is not being released for now, but a complete video training and inference codebase is provided, including the training configuration configs/train/video_t2v.yaml and the inference configuration configs/predict/t2v_2b.yaml.
Usage
The project provides an official code repository containing training and inference code. Inference is performed through a single unified script. Example prompts can be found in assets/prompts.txt. For training guidance, refer to the training guide in the project.
Model Release Notes
Due to organizational policy restrictions, the fully production-trained model weights cannot be released. The project plans to release a base checkpoint in which a small number of layers have been removed from the LLM backbone and the diffusion head (flow head), while the remaining layers and all other components (vision encoder, projection, embeddings, etc.) are fully preserved. Users can perform brief fine-tuning on their own data to quickly relearn the removed layers and restore the model's full quality. The project is also actively fine-tuning the removed layers using external data and plans to release the full weights as soon as possible.
Applicable Scenarios
Suitable for research on multimodal understanding and generation, especially academic and engineering scenarios that explore unifiedmodels in pixel space and simplify the visual encoding process. It can also serve as a starting point for training custom video models.
Limitations
- The fully production-trained model weights have not yet been released.
- The video generation model has not yet been released due to policy restrictions; only the codebase is provided.
- The released base checkpoint is missing some layers and requires users to fine-tune it themselves to restore them.
- The project uses the Apache License 2.0 license.
