What is PUMA
PUMA is an advanced multimodal large language model (MLLM) designed to unify and enhance visual generation and understanding tasks based on the integration of multi-granularity visual features. PUMA can handle text-to-image generation, detailed image editing, and other visual tasks, adapting to different levels of detail requirements. Based on multimodal pretraining and fine-tuning techniques, PUMA demonstrates cutting-edge capabilities in diverse applications such as text-to-image generation, image editing, conditional image generation, and visual language understanding. The project was updated in October 2024 and is ongoing, jointly launched by researchers from CUHK MMLab, HKU MMLab, SenseTime, Shanghai AI Laboratory, and Tsinghua University. The PUMA project pushes the boundaries of AI visual language models, providing a flexible and powerful solution for the future exploration of multimodal AI.
Main Features of PUMA
- Diverse Text-to-Image Generation: PUMA can generate diverse and high-quality images based on text prompts, enhancing creativity and consistency based on coarse-grained visual features.
- Image Editing: PUMA uses fine-grained image features to achieve precise image editing, including adding or removing objects, style adjustments, etc., while maintaining the fidelity of the original image.
- Conditional Image Generation: PUMA excels at image generation tasks based on specific conditional inputs, such as generating images from edge maps, image inpainting, or colorization, ensuring accurate and contextually appropriate results.
- Multi-Granularity Visual Decoding: PUMA uses five different granularities of image representations and corresponding decoders to achieve a wide range of visual decoding capabilities, from precise image reconstruction to semantically guided generation.
Technical Principles of PUMA
- Multi-Granularity Image Encoding: PUMA uses an image encoder to process input images, extracting multi-level visual features from fine-grained to coarse-grained, providing a foundation for generating diverse and controllable images.
- Autoregressive MLLM: PUMA's autoregressive multimodal large language model (MLLM) can process and generate multi-scale text and visual tokens, suitable for the needs of different tasks.
- Diffusion-Based Decoders: PUMA uses a set of diffusion-based decoders corresponding to different feature granularities for visual decoding of images, supporting visual outputs with high controllability or high diversity.
- Two-Stage Training Strategy: PUMA uses multimodal pretraining and task-specific instruction fine-tuning to optimize the model's performance in multi-task processing, enabling the model to perform excellently in various visual tasks.
Project Addresses of PUMA
- Project Official Website：rongyaofang.github.io/puma
- GitHub Repository：https://github.com/rongyaofang/PUMA
- arXiv Technical Paper：https://arxiv.org/pdf/2410.13861
Application Scenarios of PUMA
- Art Creation and Design: PUMA generates diverse images based on text descriptions, providing inspiration for artists and designers or directly creating artworks with specific styles and themes.
- Media and Entertainment: In film, game, and animation production, it generates backgrounds, scenes, or concept art, accelerating the creative realization process.
- Advertising and Marketing: PUMA can quickly generate attractive advertising images based on marketing copy, helping brands produce visual content at lower cost and faster speed.
- Education and Training: PUMA can generate illustrations and example images in teaching materials, making educational content more vivid and interactive.
- E-commerce: Online retailers create visual displays of products, for example, generating product images based on descriptions or changing product colors and styles.