SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Kaleido - Zhipu AI's Open-Source Multi-Subject Video Generation Framework

Author: AI小集 · Original source: AI工具集

What is Kaleido

Kaleido is Zhipu AI's open-source multi-subjectvideo generationframework, capable of solving the subject consistency and background decoupling problems in multi-subject video generation. The framework, through an innovative data construction pipeline and the R-RoPE (Reference Rotary Positional Encoding) mechanism, effectively separates subject and background information, maintaining subject feature consistency in multi-subject scenarios. Kaleido achieves SOTA results in multi-subject video generation, providing a powerful open-source solution for the video generation field and helping developers achieve high-quality multi-subject video creation.

Kaleido

Kaleido's Main Features

  • Multi-Subject Consistent Generation: The framework can maintain the visual consistency of multiple subjects (such as people, objects, etc.) in videos, and can maintain stable subject features in complex scenes and multi-person interactions.
  • Background Decoupling: The framework can effectively separate subject and background information, avoiding the appearance of irrelevant background details from reference images in generated videos, giving subjects better adaptability in different backgrounds.
  • High-Quality Video Generation: Through optimized data construction and training methods, it generates high-quality, high-fidelity video content, suitable for various application scenarios such as advertising, film and television production, etc.
  • Flexible Conditional Control: The framework supports video generation through multiple reference images and text prompts, providing creators with more flexible creative space.

Kaleido's Technical Principles

  • Data Construction Pipeline: Kaleido adopts an innovative data construction pipeline to solve the background entanglement and subject consistency problems in multi-subject video generation. The pipeline, through cross-pair data synthesis (combining subjects and backgrounds from different instances) and background inpainting techniques, forces the model to decouple subject features during training, avoiding simple copying of background information from reference images. Through multi-stage data augmentation (such as subject localization, segmentation, quality filtering, etc.), it ensures data diversity and high quality, enhancing the model's generation capability in complex scenarios.
  • R-RoPE（Reference Rotary Positional Encoding）: To precisely distinguish different reference images from video Tokens, Kaleido introduces the R-RoPE mechanism. By providing independent rotary positional encoding for the Tokens of reference images, it explicitly establishes boundaries between different subjects and video frames in attention computation. R-RoPE, through spatial displacement, ensures that reference image Tokens occupy unique positions in the model's spatiotemporal embedding space, effectively avoiding confusion of multi-subject features and significantly improving the spatiotemporal consistency of multi-subject generation.
  • Model Architecture and Conditional Injection: Kaleido is based on diffusion models and the Diffusion Transformer architecture, combining multiple reference images with the video generation process through a simple and efficient conditional injection strategy. The model concatenates reference image Tokens with video noise representations along the sequence dimension, while using the R-RoPE mechanism to distinguish image and video information.

Kaleido's Project Address

  • Project Official Website：https://criliasmiller.github.io/Kaleido_Project/
  • GitHub Repository：https://github.com/zai-org/Kaleido
  • HuggingFace Model Library：https://huggingface.co/zai-org/Kaleido-14B-S2V
  • arXiv Technical Paper：https://arxiv.org/pdf/2510.18573

Kaleido's Application Scenarios

  • Film and Television Production: Kaleido can generate high-quality animated videos based on multiple reference images of characters, helping animators quickly generate preliminary animation materials and improving production efficiency.
  • Advertising Creativity: Through multi-subject video generation, Kaleido can provide advertising creators with flexible material generation solutions, quickly generating advertising videos containing multiple products or people.
  • Virtual Try-On: In the e-commerce field, Kaleido can generate videos of users trying on virtual clothing or wearing virtual accessories, enhancing the user experience.
  • Virtual Character Interaction: Generate natural interactive videos for virtual streamers or virtual characters, enabling them to exhibit richer dynamic effects in scenarios such as live streaming and gaming.
  • Education and Training: Used to generate instructional videos, for example, through multi-subject generation to demonstrate experimental processes or multi-person collaboration scenarios, enhancing the intuitiveness and fun of teaching.