What is UnityVideo
UnityVideo is a novel multimodal, multi-task framework launched by the Hong Kong University of Science and Technology in collaboration with the Kuaishou Kling team, Tsinghua University, and othersvideo generationframework that can enhance video generation models' understanding of the physical world by integrating multiple visual modalities (such as segmentation, skeleton, depth, optical flow, etc.) and training paradigms. Through dynamic noise injection and modality-adaptive learning mechanisms, the framework enables bidirectional learning between RGB videos and auxiliary modalities, accelerates model convergence, and significantly enhances zero-shot generalization ability.
Main Features of UnityVideo
-
Multimodal video generation: The framework can generate high-quality RGB videos from text descriptions and perform joint generation with multiple auxiliary modalities (such as depth maps, optical flow, segmentation masks, human skeletons, DensePose, etc.), improving the physical realism and consistency of videos.
-
Controllable video generation: Supports controllable video generation based on multiple modality conditions (such as depth maps, optical flow, etc.), generating video content that meets specific requirements according to the given modality information.
-
Modality estimation: Estimates other auxiliary modality information (such as depth maps, optical flow, segmentation masks, etc.) from RGB videos, enabling multimodal parsing of video content.
-
Zero-shot generalization: Has strong zero-shot generalization ability, can generate high-quality videos on scenes and objects outside the training data, and accurately estimate various modality information.
-
Multi-task joint training: Supports multiple tasks such as video generation, controllable generation, and modality estimation within one framework, improving the model's comprehensive performance and generalization ability through multi-task joint training.
Technical Principles of UnityVideo
- Unified multimodal framework: Uses a unified framework based on diffusion models (such as the Diffusion Transformer DiT) to integrate RGB videos and various auxiliary modality information into a shared feature space. Through a dynamic noise injection strategy, different tasks (such as conditional generation, modality estimation, and joint generation) are randomly selected during training, enabling the model to simultaneously learn the joint distribution of multiple tasks and modalities.
- Modality-adaptive learning: Introduces a Modality-Adaptive Switcher, assigning independent parameter tables (such as AdaLN parameters) to each modality, allowing the model to dynamically adjust network parameters according to different modalities. Uses an In-Context Learner to inject text prompts of modality types (such as "depth map" and "optical flow") so that the model can distinguish different modalities at the semantic level and enhance modality awareness.
- Dynamic noise scheduling: During training, dynamically adjusts the noise injection strategy, applying different noises to RGB and auxiliary modalities according to task types (conditional generation, modality estimation, and joint generation), promoting collaborative learning across tasks. Through a probabilistic task selection mechanism, balances the learning difficulty of different tasks and prevents the model from favoring one task during joint training.
- Curriculum learning strategy: Adopts a staged curriculum learning method, first training pixel-aligned modalities (such as depth and optical flow) on single-person data to establish the foundation of spatial correspondence, then expanding to multi-person data and more modalities, gradually improving the model's understanding of complex scenes.
- Large-scale multimodal dataset: Construct the OpenUni dataset, containing 1.3 million multimodal video sample pairs, covering multiple modalities such as RGB, depth, optical flow, segmentation masks, and skeletons, providing rich data support for model training. Provide the UniBench benchmark test set for evaluating model performance on multimodal video tasks, ensuring the model's generalization ability and accuracy.
UnityVideo's project address
- Project official website：https://jackailab.github.io/Projects/UnityVideo/
- GitHub repository：https://github.com/dvlab-research/UnityVideo
- HuggingFace model library：https://huggingface.co/JackAILab/UnityVideo
- arXiv technical paper：https://arxiv.org/pdf/2512.07831
Application scenarios of UnityVideo
-
Film and television special effects production: The framework can quickly generate realistic special effects videos, such as waterfalls, auroras, liquid flow, etc., providing efficient previews and creative validation for film and television post-production.
-
Virtual reality (VR) and augmented reality (AR): The framework can generate highly realistic virtual scenes and dynamic backgrounds, enhancing users' sense of immersion and interactive experience.
-
Education and training: Generate simulation videos of scientific phenomena based on teaching content, such as physics experiments and biological animations, helping students better understand complex concepts.
-
Game development: The framework can generate dynamic scenes, character actions, and cutscene animations in games, improving the game's visual effects and development efficiency.
-
Advertising and marketing: Quickly generate creative advertising videos based on brand descriptions, for use in social media, television advertising, etc., meeting diverse marketing needs.