WeGen - A Unified Multimodal Generation Model Jointly Launched by USTC, Shanghai Jiao Tong University, and Others

What Is WeGen

WeGen is a unified multimodal generation model jointly launched by the University of Science and Technology of China together with Shanghai Jiao Tong University, the WeChat team, the Chinese Academy of Sciences, and other institutions. It enables diverse visual generation tasks based on natural dialogue. WeGen combines a multimodal large language model (MLLM) with a diffusion model to handle a variety of tasks such as text-to-image generation, condition-driven generation, image editing, and style transfer. WeGen's core advantage is its ability to provide diverse creative outputs when user instructions are relatively vague, while maintaining consistency between the generated results and the instructions and reference images when users have specific needs. Based on the Dynamic Instance Identity Consistency (DIIC) data pipeline and the Prompt Self-Rewriting (PSR) mechanism, WeGen addresses the two major challenges of instance identity consistency and generation diversity, demonstrating its potential as a user-friendly design assistant.

WeGen's Main Features

• Text-to-Image Generation: Generate high-quality images based on text descriptions.
• Condition-Driven Generation: Generate images based on specific conditions (such as edge maps, depth maps, and pose maps).
• Image Editing and Restoration: Modify, restore, or extend existing images.
• Style Transfer: Apply the style of one image to another image.
• Multi-Subject Generation: Preserve the key features of multiple reference objects when generating images.
• Interactive Generation: Gradually optimize the generated results based on natural dialogue and user interaction.
• Creative Design Assistance: Provide users with diverse generation options to inspire creativity.

WeGen's Technical Principles

• Combination of Multimodal Large Language Models (MLLM) and Diffusion Models: CLIP is used as the visual encoder to transform images into semantic features; a diffusion model (such as SDXL) is used as the decoder to generate high-quality images. An LLM (such as LLaMA) processes natural language instructions to achieve the fusion of text and visual information.
• Dynamic Instance Identity Consistency (DIIC): Use video sequences to track the natural changes of objects and maintain their identity consistency. The DIIC data pipeline addresses the shortcomings of traditional methods in preserving instance identity, allowing the model to retain key features when modifying images.
• Prompt Self-Rewriting (PSR) Mechanism: Rewrite text prompts based on a language model and introduce randomness to generate diverse images. PSR uses discrete text sampling, allowing the model to explore different interpretations while maintaining semantic consistency.
• Unified Framework and Interactive Generation: WeGen integrates multiple visual generation tasks into a single framework, gradually optimizing the generated results based on natural dialogue and user interaction while retaining the parts the user is satisfied with.
• Support from Large-Scale Datasets: WeGen is trained on a large-scale dataset extracted from internet videos. The dataset contains rich object dynamics and automatically annotated descriptions, helping the model learn consistency and diversity.

WeGen's Project Address

• GitHub Repository：https://github.com/hzphzp/WeGen
• arXiv Technical Paper：https://arxiv.org/pdf/2503.01115

WeGen's Application Scenarios

• Creative Design: Help designers quickly generate creative concept images and spark inspiration, suitable for fields such as advertising, packaging, and architecture.
• Content Creation: Generate concept images of scenes, characters, or props for industries such as film and television, games, and animation, accelerating the creation process.
• Educational assistance: Generate images related to teaching content to help students understand abstract concepts more intuitively.
• Personalized customization: Generate customized design solutions according to user needs, such as clothing, home decoration, etc.
• Virtual social interaction and entertainment: Generate virtual avatars, scenes, or props to enhance the experience of virtual social interaction and games.