Bernini - ByteDance's Open-Source Unified Video Generation and Editing Framework

What is Bernini

Bernini is ByteDance's open-source unified video generation and editing framework, employing a two-stage decoupled architecture of MLLM semantic planning + DiT visual rendering. A multimodal large model understands instructions and plans semantic sketches, while a Diffusion Transformer completes high-quality video generation. The model supports text/image-to-video generation, environment/viewpoint/motion editing, multi-reference-guided insertion, and other tasks, with inference code and weights fully open-sourced.

Bernini's Main Features

• Text/Image-to-Video Generation: Supports direct video generation from text prompts or reference images, capable of handling single images, multi-element composite images, and multi-angle reference images.
• Video Editing: Supports environment editing, viewpoint editing, focus editing, and motion editing.
• Reference-Guided Editing: Precisely integrates specified objects, textures, or styles into videos through subject reference images, material reference images, or style reference images.
• Image/Video Insertion: Inserts posters or videos into areas such as screens and signs within the frame, following camera movement while maintaining perspective and temporal stability.
• Multi-Element Composition: Combines multiple unrelated reference images into the same video character, or generates continuous panning shots based on keyframes of the same scene.

Bernini's Technical Principles

• Two-Stage Decoupled Architecture: In the first stage, a Multimodal Large Language Model (MLLM) serves as the Semantic Planner, predicting target semantic representations in the ViT embedding space, responsible for understanding text instructions, source videos, and reference materials and planning target content. In the second stage, a Diffusion Transformer (DiT) serves as the Renderer, performing flow matching denoising in the VAE latent space, transforming semantic plans into high-quality video frames. In editing tasks, source video VAE features are also injected to preserve details in non-edited regions.
• Segment-Aware 3D RoPE（SA-3D RoPE）: When multiple reference images, source videos, and target outputs are concatenated into a unified sequence, tokens from different segments may share the same spatiotemporal coordinates. By assigning a segment index to each visual segment and integrating it into rotary position encoding, the model can distinguish tokens from different sources while preserving the original spatiotemporal modeling characteristics.
• Chain-of-Thought Reasoning and Three-Stage Training: The Planner performs chain-of-thought reasoning in latent space, enhancing semantic fidelity in complex editing scenarios. Training is divided into three stages: Planner pre-training, Renderer pre-training, and lightweight joint training of Planner and Renderer, retaining pre-training advantages while maintaining efficiency.

How to Use Bernini

• Clone the Repository: Clone the Bernini project from GitHub to your local machine.
• Install Dependencies: Run pip to install the dependency packages in requirements.txt.
• Download Models: Download the Bernini-R-Diffusers weights via HuggingFace.
• Configure the Environment: Ensure Python 3.11.2 and CUDA 12.4+ are used; Hopper architecture GPUs are recommended.
• Run Generation: Use torchrun to execute the multi-GPU inference script, specifying the configuration file and test cases.
• Configure Prompt Enhancement: Set up an OpenAI-compatible API endpoint to improve generation quality.

Bernini's Core Advantages

• Understand First, Then Generate: The MLLM handles semantic planning, while the DiT handles visual rendering, solving the pain point of traditional video models "not understanding human language".
• Unified Framework: A single architecture covers multiple tasks such as generation, editing, and reference guidance, with no need to switch models.
• Multi-Reference Input: Supports simultaneous input of text, images, and video, using SA-3D RoPE to avoid spatiotemporal confusion.
• Detail Preservation: During editing, source video VAE features are injected to preserve details in non-edited regions and avoid "changing everything with one edit".
• Fully Open Source: Inference code and weights have been released, lowering the barrier to research and development.
• Strong Generalization Ability: It still performs excellently on instructions not explicitly covered in training, such as causal reasoning and action changes.

Bernini Project Address

• Project Official Website：https://bernini-ai.github.io
• GitHub Repository：https://github.com/bytedance/Bernini
• HuggingFace Model Library：https://huggingface.co/ByteDance/Bernini
• arXiv Technical Paper：https://arxiv.org/pdf/2605.22344

Comparison of Bernini with Similar Competitors

Bernini Application Scenarios

• Advertising Creativity: The model can quickly generate product showcase videos, supporting product image insertion and style adjustment.
• E-Commerce Display: Generates highly consistent dynamic showcase videos based on multi-angle reference images of products.
• Film and Television Previsualization: Generates continuous shots through keyframes, assisting virtual walkthroughs and scene construction.
• Secondary Creation: Supports style transfer, weather transformation, and action editing, lowering the barrier to video re-creation.
• Virtual Production: Precisely places posters or videos into screens or signboards in the frame, achieving the fusion of virtual and real.