DreamLite

DreamLite is the official implementation of the ECCV 2026 paper "DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing", developed by ByteVisionLab.

Purpose DreamLite targets on-device image generation and editing scenarios, aiming to complete text-to-image generation and text-guided image editing on mobile devices with extremely low latency, without relying on cloud processing.

Key Capabilities

• Unified architecture: supports both text-to-image generation and text-guided image editing within a single network architecture.
• Lightweight model: the model size is 0.39B, based on a pruned mobile U-Net backbone.
• Unified multimodal conditioning: unifies multimodal conditions directly in latent space through In-Context Spatial Concatenation.
• Ultra-fast inference: achieves 4-step inference via progressive step distillation, generating or editing 1024×1024 images in about 3 seconds on an iPhone 17 Pro (using 4-bit Qwen-VL and fp16 VAE+UNet), running entirely on-device with zero cloud dependency.
• Two model variants: DreamLite-base (28 steps, high fidelity) and DreamLite-mobile (4 steps, ultra-fast), used to balance visual fidelity and on-device inference latency.
• Ecosystem integration: has been merged into the official Hugging Face Diffusers library (pending the next release), and can be loaded and run directly via diffusers.
• Community extensions: the community has contributed dreamlite-comfyui-lowvram, providing ComfyUI custom nodes and a low-VRAM inference pipeline, capable of running DreamLite-base at 1024×1024 on NVIDIA GPUs with 4 GB VRAM (such as the GTX 1650 Ti), achieved through sequential CPU offload, float32 precision, and a GQA-aware query token attention slicing strategy that preserves the grouped query attention layout, and comes with ComfyUI workflows for DreamLite-base and DreamLite-mobile as well as tiled RealESRGAN upscaling support.

Usage

1. Environment setup: place the modelWeights (DreamLite-base and DreamLite-mobile) into the specified directory structure.
2. Inference via Diffusers: Since the changes have not yet entered the stable version of Diffusers, the latest main branch must be installed from source. The model weights are hosted on the diffusers branch of the Hugging Face repository: carlofkl/DreamLite-base (28 steps, high fidelity) and carlofkl/DreamLite-mobile (4 steps, ultra-fast). Access is currently restricted, so you must first apply via the Access Request Form. After approval, you will receive a Hugging Face access token; use that token to log in locally; from_pretrained(..., revision="diffusers") will automatically download the weights on first use. You can also use the same token to pre-download the weights via the CLI, then load them from a local path: DreamLitePipeline.from_pretrained("models/DreamLite-base", torch_dtype=dtype). Then DreamLite can be loaded and run with a small amount of code.
3. Inference via CLI: The command-line interface provided by the project can be used to generate or edit images.
4. Benchmark evaluation: Complete benchmark evaluation scripts for GenEval and ImgEdit are provided. Before running, you need to configure the local dataset paths in tools/benchmark/infer_geneval.py and tools/benchmark/infer_imgedit.py.
5. Interactive Gradio demo: A Gradio-based web interface is provided. You can try the online demo on Hugging Face Spaces, or deploy and run it on a local machine (GPU/CPU).
6. LoRA fine-tuning: Complete LoRA fine-tuning and inference support is provided, allowing lightweight customization of DreamLite on your own domain datasets. See the dedicated LoRA Fine-Tuning Guide for details.
7. On-device deployment: A complete iOS on-device deployment reference is provided, including model export scripts (CoreML + mlx-vlm 4-bit quantization), modified Swift library files, and iOS application source code.

Use Cases

• Real-time text-to-image generation and image editing on mobile or on-device devices.
• Privacy-sensitive scenarios with no network or where cloud processing must be avoided.
• Lightweight LoRA customization on your own domain data.
• Running via ComfyUI on low-VRAM GPUs.
• Generation and editing benchmark comparisons in academic research.

Limitations

• The model weights are currently under security review, and access must be requested by filling out the Access Request Form.
• The model is limited to non-commercial, research use only; generating, editing, or disseminating pornographic, violent, discriminatory, or other illegal content is strictly prohibited; commercial use and public redistribution of the model weights are strictly prohibited.
• The code is licensed under Apache-2.0, and the model weights are licensed under CC BY-NC 4.0 (see WEIGHTS_LICENSE).
• The Diffusers integration has not yet entered the stable version, so the main branch must be installed from source.
• Some demos may not render natively on GitHub and must be viewed on the project page.

Open Source Plan Release the arXiv paper, release the inference code, release LoRA training, release the model weights on HuggingFace, release the online demo, and provide an on-device deployment reference.

Acknowledgements The project acknowledges the work of SDXL, SnapGen, Qwen, and TAESDXL, under the guidance of Prof. Wangmeng Zuo.