SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI videoDitto

Ditto

Ditto is a scalable framework and dataset project for instruction-based video editing, proposed in the paper "Ditto: Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset" (CVPR'26 Highlight). The core goal of this work is to address the data scarcity problem in the field of instruction-based video editing: it provides a novel data generation pipeline that combines the creative diversity of leading image editors with a context video generator to produce high-quality video editing data. To control cost and quality, the framework adopts an efficient distilled model architecture supplemented by a temporal enhancer, reducing computational overhead while improving temporal consistency; the entire pipeline is agent-driven, automatically constructing diverse instructions and strictly filtering outputs to achieve large-scale quality control. The project team invested over 12,000 GPU days to build the Ditto-1M dataset, containing one million high-fidelity video editing samples, and based on this trained a new instruction-based video editing model, Editto, using a curriculum learning strategy. The repository provides links to the paper, project homepage, model weights, dataset, and Hugging Face online demo, and supports inference and training via DiffSynth-Studio, use via ComfyUI (including a note on the quality trade-off introduced by quantized and distilled models), and also provides denoising enhancement functionality. The project is licensed under CC BY-NC-SA 4.0, and the code is for academic research use only.

Open website

About this tool

Project Positioning

Ditto is an overall framework for "instruction-based video editing," accompanied by the paper "Ditto: Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset," accepted as a Highlight at CVPR'26. This work attempts to address the fundamental challenge in this field—data scarcity. Around this goal, the project simultaneously provides a data generation pipeline, the Ditto-1M dataset, and the model Editto trained on this data.

Main Capabilities and Design

According to the official description, Ditto's data generation pipeline combines the creative diversity of leading image editors with a context video generator to overcome the limited coverage of existing models. To alleviate the trade-off between cost and quality, the framework adopts an efficient distilled model architecture and adds a temporal enhancer, thereby reducing computational overhead and improving temporal consistency. The entire pipeline is agent-driven, used to generate diverse instructions and strictly filter outputs to achieve large-scale quality control. Based on this framework, the project invested over 12,000 GPU days to build Ditto-1M—a dataset containing one million high-fidelity video editing samples; Editto is trained on Ditto-1M using a curriculum learning strategy, and is officially said to exhibit stronger instruction-following ability and to have established a new SOTA in instruction-based video editing.

Usage

The repository provides multiple usage paths:

  • With DiffSynth: Download the base model and project model from Google Drive or Hugging Face, test the local editing model with a single command, or use the provided scripts or run Python directly for general editing; see the HF Dataset for test cases, and inference/example_prompts.txt for reference editing prompts.
  • With ComfyUI: First set up the basic environment according to the ComfyUI installation guide, and it is strongly recommended to install ComfyUI-Manager to manage custom nodes. Then you can choose to use ComfyUI-Manager's "Install Missing Custom Nodes" to automatically install the required nodes, or manually install ComfyUI-WanVideoWrapper, KJNodes for ComfyUI, ComfyUI-VideoHelperSuite. The required weights are downloaded from Kijai/WanVideo_comfy to subfolders of models/, including Wan2_1-T2V-14B_fp8_e4m3fn.safetensors (diffusion_models/), Wan21_CausVid_14B_T2V_lora_rank32_v2.safetensors (loras/, used for inference acceleration), Wan2_1_VAE_bf16.safetensors (vae/wan/), umt5-xxl-enc-bf16.safetensors (text_encoders/); the project's own model is downloaded from Google Drive or Hugging Face to diffusion_models/ (loaded with the VACE Module Select node), and uses the ditto_comfyui_workflow.json workflow in the repository. The official note states that ComfyUI runs faster and has lower computational requirements (a 832×480×73 video requires 11G of VRAM on an A6000 and about 4 minutes), but because quantized and distilled models are used, there may be some quality degradation. If testing sim2real scenarios, you can try prompts similar to "Turn it into the real domain".
  • Training: first download the training CSV files from the csvs directory on Hugging Face, then use the provided train.sh script to train. With DiffSynth-Studio, this codebase supports multi-node training, and DLRover can be considered for training across multiple machines.
  • Denoising enhancement: you first need to install the dependencies for Wan2.2, download the Wan-AI/Wan2.2-T2V-A14B model, write the paths of the videos to be processed into a txt file, and then run the corresponding script.

Update Log and Resources

The version log shows: on 2025/10/29, the Hugging Face online demo was released; on 2025/10/27, denoising enhancement code was added; on 2025/10/22, the CSV that can be directly used for DiffSynth-Studio model training was uploaded, as well as the metadata json for the sim2real settings, and the upload of all videos in the dataset was completed. Resource links include the paper, project homepage, model weights, dataset, and Hugging Face Demo.

Applicable Scenarios and Limitations

This project is aimed at experiments and training in academic research and instruction-based video editing. In terms of licensing, the project uses CC BY-NC-SA 4.0 (Attribution—NonCommercial—ShareAlike 4.0 International License), and the code is for academic research purposes only. The acknowledgments mention that Wan, VACE, and Qwen-Image provide the base models, QwenVL provides visual understanding capabilities, and DiffSynth-Studio serves as the codebase. The contact email is qingyanbai@hotmail.com. Apart from the above, the materials do not currently provide information on specific feature details, charges, prices, or the scope of platform support.

Related tools

Browse category