SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsUAE

UAE

UAE is a unified multimodal framework proposed by PKU-YuanGroup at Peking University, aiming to make image understanding and generation truly mutually reinforcing through reinforcement learning, rather than simply coexisting. The framework treats understanding as an encoder (image-to-text) and generation as a decoder (text-to-image), and uses reconstruction similarity as an explicit objective to quantify cross-modal consistency. The accompanying Unified-GRPO is the first reinforcement learning scheme to jointly optimize understanding and generation modules, forming a positive feedback loop through two complementary steps: "generation promotes understanding" and "understanding promotes generation," and observes an "epiphany moment" in multimodal learning. In addition, the project also released Unified-Bench, the first benchmark specifically designed to measure the degree of unification of multimodal models, rather than separately evaluating generation or understanding capabilities. The repository provides training code, model checkpoints, inference demos, and evaluation tools, suitable for research and development in unified multimodal learning.

Open website

About this tool

UAE is a unified multimodal framework developed by PKU-YuanGroup, with the paper titled "Can Understanding and Generation Truly Benefit Together — or Just Coexist?". The framework aims to address how multimodal understanding and generation can truly work together.

Purpose: UAE is used for unified multimodal learning, that is, simultaneously performing image understanding (image-to-text) and image generation (text-to-image) within the same framework, and quantifying the degree of unification between the two.

Main capabilities:

  • UAE is based on the autoencoder idea, treating understanding as an encoder (I2T) and generation as a decoder (T2I), using reconstruction similarity as an explicit objective to quantify cross-modal consistency.
  • Unified-GRPO is the first reinforcement learning scheme to jointly optimize understanding and generation modules, including two complementary steps: generation promotes understanding (training the encoder to generate higher-quality descriptions to improve reconstruction quality) and understanding promotes generation (using these descriptions to optimize decoder reconstruction), forming a positive feedback loop.
  • An "epiphany moment" in multimodal learning is observed: as reinforcement learning progresses, the encoder autonomously generates longer, more descriptive text, while the decoder achieves highly realistic reconstruction.
  • Unified-Bench is the first benchmark specifically designed to measure the degree of unification of multimodal models, rather than separately evaluating generation or understanding capabilities.

How to use:

  • Download the required model checkpoints: Stable Diffusion 3.5 Large model, UAE fine-tuned weights, vision-language model checkpoints (Qwen-2.5-VL-3B and its LoRA).
  • Update the model paths in demo.py, including "SD3" (official Stable Diffusion 3-Large weight path), "dit" (pretrained DiT weights), "dit_lora" (DiT LoRA obtained in the third stage of Unified-GRPO), "llm" (pretrained Qwen-2.5-VL-3B weights), "llm_lora" (Qwen-2.5-VL-3B LoRA obtained in the second stage of Unified-GRPO), "llm_processor" (official Qwen-2.5-VL-3B configuration, located at ./Checkpoints/llm_processor).
  • demo.py supports two modes: text-to-image generation and image-to-textunderstanding/description.
  • The evaluation framework is located in the Unified-Bench/ directory and supports similarity metrics such as CLIP, DINO v2, DINO v3, and LongCLIP. Run the unified evaluation script for evaluation.
  • Unified-Bench/UniBench/ contains the evaluation benchmark, and the data in the Image folder can be downloaded from the link. Evaluation generates statistics and supports custom evaluation; simply organize images according to the UniBench structure and ensure the names match.

Applicable scenarios: suitable for unified multimodal learning research, joint training of image generation and understanding, applications of reinforcement learning in multimodality, and benchmarking to evaluate the degree of unification of multimodal models.

Limitations: the project does not yet provide specific charging, pricing, platform, or commercial use information, and all content is subject to the official repository release.

Related tools

Browse category