Muse: Text-To-Image Generation via Masked Generative Transformers
AI imagesMuse is a text-to-image generation and editing model based on a masked generative Transformer. It is trained with masked modeling in a discrete token space: given text embeddings extracted from a pretrained large language model, Muse learns to predict randomly masked image tokens, thereby enabling high-quality text-to-image generation, zero-shot image editing (such as local redrawing, outpainting, and mask-free editing), and other capabilities. Compared with diffusion models and autoregressive models, Muse has clear advantages in sampling efficiency and inference speed; for example, on TPUv4 it takes only 1.3 seconds to generate a 512x512 image and only 0.5 seconds for 256x256. Two parameter versions are currently provided, 900M and 3B, among which the 900M model reaches 6.06 FID on CC3M, and the 3B model reaches 7.88 FID and a 0.32 CLIP score on zero-shot COCO evaluation.