Lumina-DiMOO is a fully open-source all-in-one foundation model for multimodal generation and understanding. Its core feature is the use of a fully discrete diffusion modeling architecture that uniformly handles the input and output of multiple modalities such as text and images, which is clearly different from previous unified multimodal models.
In terms of main capabilities, Lumina-DiMOO supports a wide range of multimodal tasks: text-to-image generation (supporting arbitrary resolutions), image-to-image generation (including image editing, subject-driven generation, image inpainting, etc.), and advanced image understanding. It is officially claimed to have achieved state-of-the-art performance on multiple benchmarks, surpassing existing open-source unified multimodal models. In terms of sampling efficiency, compared with previous autoregressive (AR) or hybrid AR-diffusion paradigms, Lumina-DiMOO performs better, and a specially designed Max Logit-based Cache (ML-Cache) scheme can further increase sampling speed by 2 times.
In terms of usage, the project has open-sourced model checkpoints (HuggingFace), inference code for text-to-image and image-to-image generation, inference code for image understanding, training code, fine-tuning code, image inpainting and outpainting code, and benchmark evaluation code, and has added support for Diffusers and ComfyUI. Text-to-image inference supports a ddp script for multi-GPU parallel sampling, and the --use-cache parameter can be added to accelerate sampling through ML-Cache. The trade-off between efficiency and quality can be adjusted through parameters such as cache_ratio, warmup_ratio, and refresh_interval. For fine-tuning, the official sample json files assets/mmu_sample.json and assets/t2i_sample.json are provided as format references. Benchmark evaluation uses OpenCompass's VLMEvalKit, supports 5 benchmarks, and requires configuring an LLM as the judging model, filling in OPENAI_API_KEY and OPENAI_API_BASE in VLMEvalKit/.env.
Applicable scenarios include academic research on multimodal generation and understanding, application development for image generation and editing, and model fine-tuning and evaluation. In terms of limitations, the official description states that text generation is performed in a block-wise manner, unlike image generation which uses a single global decoding step, and its speed is affected by the number of blocks and steps, so the speed improvement for image understanding is not as significant as for image generation. Specific installation steps, performance data, supported hardware, and other information are not fully elaborated in the provided materials, and detailed content is not yet provided.
