Bernini

Bernini is a unified video generation and editing framework released and open-sourced by ByteDance, whose core design combines a semantic planner based on a multimodal large language model (MLLM) with a renderer based on a Diffusion Transformer (DiT). The related paper is "Bernini: Latent Semantic Planning for Video Diffusion".

Purpose Bernini targets video generation and video editing tasks. The official statement is that it reaches the first tier of leading closed-source commercial models in video editing; the evaluation data comes from its self-built arena platform, where human annotators perform blind paired editing votes, which are aggregated into Bradley-Terry scores and a pairwise win-rate matrix.

Main capabilities The project provides two model families: Bernini (the full framework, including the semantic planner and renderer) and Bernini-R (the renderer). Both share the same task interfaces: t2i, i2i, t2v, v2v, rv2v, and r2v. The repository states that Bernini-R's 1.3B weights, fine-tuned from Wan2.1-1.3B, perform close to the 14B variant on simple tasks such as style transfer, subtitle or watermark removal, and local editing, but lag behind the 14B variant on more complex tasks such as human generation.

Timeline (from official statements)

• 2026-07-13: Released the training code for Bernini Renderer (Bernini-R); the full training guide is in docs/bernini_r_train.md.
• 2026-06-11: Open-sourced the inference code and model weights for the full Bernini (Bernini) at ByteDance/Bernini-Diffusers.
• 2026-06-09: Open-sourced the 1.3B weights of Bernini Renderer (Bernini-R) at ByteDance/Bernini-R-1.3B-Diffusers.
• 2026-06-01: Open-sourced the inference code and model weights of Bernini Renderer (Bernini-R) at ByteDance/Bernini-R-Diffusers.
• 2026-05-22: Released the paper Bernini: Latent Semantic Planning for Video Diffusion.

How to use

1. Model selection: The repository provides two model families; you must choose one and follow the respective guides for weight download, inference commands, and script execution.
2. Environment requirements: A CUDA GPU is required, with a Hopper GPU (H100/H800/H200) recommended to use FlashAttention-3; other CUDA GPUs can fall back to FlashAttention-2 or PyTorch SDPA. CUDA toolkit 12.6 (matching the pinned torch==2.7.1+cu126; if building FlashAttention-3 yourself, the minimum is 12.3+). The pinned versions in requirements.txt are: torch==2.7.1+cu126, diffusers==0.35.2, accelerate==0.34.2, transformers==4.57.3.
3. Inference installation: Open-VeOmni (Apache-2.0, Python 3.11) is a required dependency; all inference paths import it, including single-GPU.
4. Training installation: Using uv to manage the Python environment is recommended; pyproject.toml declares all training dependencies and automatically routes torch/torchvision to the correct CUDA index. uv sync automatically creates and uses .venv, which can be activated via source .venv/bin/activate, or commands can be run within it using uv run <cmd>. See docs/bernini_r_train.md for the complete training guide.
5. Optional acceleration: FlashAttention-2 works on general CUDA GPUs (including A100/A800), with the installation command pip install flash-attn==2.8.3; FlashAttention-3 is only for Hopper (H100/H800/H200, CUDA ≥ 12.3, PyTorch ≥ 2.4), flash_attn_interface is not on PyPI and must be built from the hopper/ directory of the flash-attention repository under tag v2.8.3, with the command: git clone https://github.com/Dao-AILab/flash-attention.git；cd flash-attention && git checkout v2.8.3; cd hopper && MAX_JOBS=$(nproc) python3 setup.py install --user.
6. Case files: A single run is described by one case file—a small JSON under assets/testcases/ that packages the routing and inputs for one task (task_type, guidance_mode, prompt, source media, output), to avoid long prompts appearing on the command line. Each task has a directory under assets/testcases/ containing one or more examples.
7. Prompt enhancer (strongly recommended): --use_pe enhances prompts via an OpenAI-compatible endpoint and is officially recommended for obtaining the best generation quality. The openai SDK is installed by requirements.txt and requires configuring the endpoint via environment variables.
8. Gradio demo: gradio_demo.py exposes the same pipeline for Bernini and Bernini-R through a Gradio UI; the task type dropdown automatically fills guidance_mode (still user-editable), uploaded media is routed to the matching slot, and results are rendered inline. See each model guide for the startup command. Adding --use_pe in gradio_demo.py (together with the above prompt enhancer environment variables) enables prompt enhancement; the checkbox within the UI is a toggle that switches on a per-request basis on top of this flag.

Applicable scenarios and limitations Applicable scenarios: video generation and video editing, including style transfer, subtitle or watermark removal, local editing, etc.; the official claim is that the editing capability reaches the first tier among closed-source commercial models. Limitations: requires a CUDA GPU, with Hopper architecture recommended; requires CUDA toolkit 12.6 (minimum 12.3+, for building FlashAttention-3); inference depends on Open-VeOmni; the 1.3B variant lags behind the 14B variant on complex tasks (such as human body generation); weight download and per-task inference commands vary by model and must be operated according to the corresponding documentation. All of the above is official description; information on features, charges, prices, platforms, or effects that is not provided is not available at this time.

License and acknowledgments The project uses the Apache License 2.0. It is built on multiple open-source projects, including Wan2.2-T2V-A14B and Qwen2.5-VL-7B-Instruct.