What is Nexus-Gen
Nexus-Gen is an open-source all-in-oneModelScopeimage generation model launched by the team, East China Normal University, and other institutions, supporting image understanding, generation, and editing tasks simultaneously. Nexus-Gen integrates the capabilities of powerful language models and diffusion models, and based on a pre-filled autoregressive strategy, solves the problem of error accumulation in image embeddings in traditional methods. The model achieves image quality and editing capabilities comparable toimage generationmodel, supporting image understanding, generation, and editing tasks simultaneously. Nexus-Gen integrates the capabilities of powerful language models and diffusion models, and based on a pre-filled autoregressive strategy, solves the problem of error accumulation in image embeddings in traditional methods. The model achieves image quality and editing capabilities comparable to GPT-4o , promoting the development of the full-modal model field.
Main Features of Nexus-Gen
- Image Understanding: Analyzes image content, generates descriptive text, and answers questions related to images.
- Image Generation: Generates high-quality images based on text descriptions, supporting the generation of complex scenes and styles.
- Image Editing: Provides a variety of editing functions, including color adjustment, object addition/deletion, style transfer, etc.
Technical Principles of Nexus-Gen
- Architecture Design: Converts input text and images into embedding vectors based on a text Tokenizer and a Vision Encoder, and inputs them into an autoregressive Transformer to generate output text tokens and image embeddings. The image embeddings are aligned to the same feature space as the input using a Vision Projector and decoded into pixel-level images based on a diffusion model (Vision Decoder).
- Pre-filled Autoregressive Strategy: Introduces a pre-filled autoregressive strategy. During training, the model fills image embedding positions with learnable special tokens. During inference, after predicting the image start token (BOI), the model directly pre-fills special tokens, unifying training and inference behavior and avoiding error accumulation.
- Unified Task Representation: Based on a unified task representation format, integrates image understanding, generation, and editing tasks into one framework. The data for all tasks is organized in a similar message format, and the model learns the synergy between different tasks, improving the processing capability of multimodal tasks.
- Two-Stage Alignment Training
- Autoregressive Transformer Training: Based on three-stage training, gradually endows the model with image generation and editing capabilities, and finally uses high-quality data to improve generation quality.
- Diffusion Model Training: Based on single-stage training, adjusts the input condition from text to image embeddings, achieving high-quality image reconstruction.
Nexus-Gen Project Address
- GitHub Repository：https://github.com/modelscope/Nexus-Gen
- HuggingFace Model Library：https://huggingface.co/modelscope/Nexus-Gen
- arXiv Technical Paper：https://arxiv.org/pdf/2504.21356
Application Scenarios of Nexus-Gen
- Creative Design: Quickly generates artworks, illustrations, or concept art, supports style transfer, and improves creative efficiency.
- Content Creation: Generates article illustrations and video materials, and edits images to match content themes.
- Advertising and Marketing: Produces advertising materials that conform to brand style, and quickly generates product display images.
- Education and Learning: Generates images of historical, scientific, or literary scenes to enhance learning intuitiveness.
- Game Development: Quickly generates game scenes, characters, and props, supporting virtual reality (VR) and augmented reality (AR) applications.