What is SketchVideo
SketchVideo is a sketch-basedvideo generationand editing framework launched by the University of Chinese Academy of Sciences, the Hong Kong University of Science and Technology, and the Kuaishou Kling team. Draw sketches on keyframes and combine them with text prompts to achieve fine-grained control over the spatial layout and motion of videos. The framework is based on a DiT video generation model and designs an efficient sketch control network, including sketch control blocks and inter-frame attention mechanisms, which can propagate sparse keyframe sketch conditions to all video frames. SketchVideo supports fine-grained editing of real or synthetic videos. Based on a video insertion module and latent fusion technology, it ensures spatial and temporal consistency between new content and the original video, preserving details in unedited regions.
Main Features of SketchVideo
- Video generation: Generate videos based on sketches and text.
- Video editing: Draw sketches on keyframes to modify video content.
- Motion control: Supports motion interpolation and extrapolation.
- Detail preservation: Preserves details in unmodified regions during editing.
- Efficient generation: Memory optimization, quickly generates high-quality videos.
Technical Principles of SketchVideo
- Sketch condition network: Based on the DiT (Diffusion-based Transformer) video generation model, a dedicated sketch condition network is designed. The sketch condition network contains multiple sketch control blocks that predict the residual features of skipped DiT blocks. The sketch control blocks are evenly distributed among the DiT blocks to inject control signals into features at different levels.
- Inter-frame attention mechanism: Based on the inter-frame attention mechanism, the sketch conditions on keyframes are propagated to all video frames. The relationships between the hidden features of all frames and the hidden features of the control frames are computed to achieve spatiotemporal propagation of sketch features.
- Video insertion module: In video editing tasks, a video insertion module is designed to analyze the relationship between the input sketch and the original video. The module generates new content that is spatially and temporally consistent with the original video, ensuring seamless integration of the edited video with the original video.
- Latent fusion technology: During inference, DDIM (Denoising Diffusion Implicit Models) inversion is used to generate the noise latent codes of the input video. These latent codes are replaced in unedited regions to preserve the details of the original video and ensure that the edited video is visually natural and coherent.
- Hybrid training strategy: Based on a hybrid training strategy, training combines image and video data. In the first stage of training, image and video data are used to accelerate convergence and address the problem of limited video data. In the second stage, video data is used to further optimize temporal coherence.
Project Address of SketchVideo
- Project website：http://geometrylearning.com/SketchVideo
- GitHub repository：https://github.com/IGLICT/SketchVideo
- arXiv technical paper：https://arxiv.org/pdf/2503.23284
Application Scenarios of SketchVideo
- Film and advertising: Quickly generate creative videos and special effects previews, optimize the production process, and save time and cost.
- Education and training: Assist in producing instructional videos and training materials to improve teaching effectiveness.
- Game development: Quickly generate level previews and character animations to improve development efficiency.
- Personal creation: Easily create personalized short videos and lower the barrier to creation.
- Architectural Design: Generate dynamic previews of architectural and interior design to enhance client communication.