What is Lance
Lance is a lightweight native unified multimodal model open-sourced by ByteDance's Intelligent Creation team, with only 3B activated parameters, supporting the full chain of image and video understanding, generation, and editing tasks within a single framework. The model is trained from scratch using a staged multi-task approach, consuming only 128 A100 GPUs over the entire training cycle. It performs excellently on multiple benchmarks such as GenEval and VBench, follows the Apache-2.0 open-source license, and supports commercial use.
Main Features of Lance
- Image Understanding: Performs semantic parsing, content recognition, and visual question answering on input images.
- Image Generation: Generates high-quality images based on text prompts, supporting complex compositions and attribute binding.
- Image Editing: Supports instruction-level editing such as background replacement, object addition and removal, style transfer, and appearance reshaping.
- Video Understanding: Performs temporal analysis, action recognition, and semantic understanding of video content.
- Video Generation: Generates coherent videos based on text descriptions, supporting character motion and scene construction.
- Video Editing: Enables single-step and compositional video editing, including background transformation, subject replacement, and action modification.
- Multi-turn Consistent Editing: Performs continuous multi-turn editing on the same subject, maintaining identity and style consistency.
Technical Principles of Lance
- Dual-stream Mixture-of-Experts Architecture: While sharing a multimodal sequence representation, allocates independent expert paths for understanding and generation tasks, avoiding interference between heterogeneous objectives.
- Unified Interleaved Sequence Representation: Organizes text tokens, ViT semantic tokens, clean VAE latent tokens, and noisy VAE latent tokens into a unified sequence, supporting understanding, generation, and conditional editing.
- Generalized 3D Causal Attention: Segments the sequence by modality, using causal attention for text tokens and bidirectional attention for visual tokens, uniformly handling multimodal understanding and generation.
- Modality-Aware Positional Encoding: Introduces rotary positional encoding for heterogeneous visual tokens of images and videos, reducing signal interference between different modalities.
- Staged Multi-Task Training: Adopts a progressive recipe of pretraining, continued training, and supervised fine-tuning, achieving multi-task synergy within a limited compute budget.
How to Use Lance
- Environment Preparation: Ensure that a Python environment and at least one CUDA-supported GPU are available locally or in the cloud.
- Clone the Repository: Execute
git clone https://github.com/bytedance/Lance.gitto download the project source code. - Install Dependencies: After entering the project directory, run
pip install -r requirements.txtto install the necessary libraries. - Download Weights: Obtain the Lance pretrained model weights from Hugging Face or the project Release page.
- Run Inference: Load the model according to the official example scripts, and input text or visual prompts for generation, editing, or understanding tasks.
Core Advantages of Lance
- Extremely Lightweight: With only 3B activated parameters, it achieves the best balance between parameter count and performance in a unified model.
- Full-Chain Unification: A single model covers six major tasks of image and video understanding, generation, and editing, without the need to switch between specialized models.
- Low-cost training: Trained from scratch within a budget of 128 A100 GPUs, greatly lowering the barrier to reproduction.
- Business-friendly: Apache-2.0 open-source license, supporting free commercial use, modification, and distribution.
- Leading performance: Outperforms existing open-source unified models on multiple benchmarks such as GenEval, GEdit-Bench, and VBench.
Lance's project address
- Project website：https://lance-project.github.io/
- GitHub repository：https://github.com/bytedance/Lance
- HuggingFace model library：https://huggingface.co/bytedance-research/Lance
- arXiv technical paper：https://arxiv.org/pdf/2605.18678
Comparison of Lance with similar competitors
| Comparison dimension | Lance | TUNA | Show-o2 |
|---|---|---|---|
| Activated parameter count | 3B | 7B | 7B |
| Task coverage | Image/video understanding, generation, editing | Image/video understanding, generation | Image/video understanding, generation |
| Open-source license | Apache-2.0 | Not specified | Apache-2.0 |
| GenEval total score | 0.90 | 0.90 | 0.76 |
| GEdit-Bench average score | 7.30 | 6.52 | Not listed |
| VBench total score | 85.11 | Not listed | Not listed |
| Architecture features | Dual-stream MoE + modality-aware positional encoding | Unified autoregressive architecture | Unified autoregressive architecture |
Lance's application scenarios
-
Intelligent content creation: Provides designers and creators with a one-stop image and video generation and editing tool, accelerating the realization of ideas.
-
Short video production: Supports rapid generation and editing of short video content, lowering the barrier and cost of video production.
-
Advertising and marketing materials: Batch-generates product images, posters, and promotional videos, enabling efficient iteration of visual materials.
-
Visual search enhancement: Based on image and video understanding capabilities, improves the visual retrieval experience of e-commerce and content platforms.
-
Educational content production: Automatically generates teaching illustrations and demonstration videos, enriching multimedia resources for online education and training.