OmniWeaving - A video generation framework open-sourced by Tencent Hunyuan in collaboration with Zhejiang University and others

What is OmniWeaving

OmniWeaving is a unifiedvideo generationframework launched by Zhejiang University in collaboration with Tencent Hunyuan and Nanyang Technological University. The framework breaks through the single-task limitation of traditional open-source models, enabling free multimodal composition and reasoning-enhanced generation. It can temporally bind interleaved image-text-video content to generate coherent content, like an "intelligent director" actively inferring complex intent. Through a three-component architecture of MLLM semantic understanding, MMDiT generation, and VAE encoding, and accompanied by the launch of the IntelligentVBench evaluation benchmark, the framework achieves SoTA performance among open-source unified models, providing a strong open-source reference for narrowing the gap between open-source and commercial video generation.

Main features of OmniWeaving

• Unified multimodal generation: Seamlessly integrates text, multiple images, and video inputs within one framework, supporting interleaved free composition to achieve diverse video generation tasks, moving away from the fragmented limitations of traditional single-task models.
• Temporal binding generation: Temporally aligns and binds content from different modalities to generate cross-modal coherent dynamic videos.
• Reasoning-enhanced creation: Through the MLLM's "thinking mode", it actively infers complex, ambiguous user intent, autonomously planning camera language and narrative logic like an experienced director, upgrading from passive rendering to active creation.
• Advanced semantic understanding: Uses a multimodal large language model to map free-form input into a high-level semantic space, combined with a diffusion Transformer to generate refined video content.
• End-to-end video generation: An integrated pipeline from concept understanding to video output, supporting complex creative needs such as character consistency maintenance and style transfer, achieving SoTA-level performance among open-source models on the IntelligentVBench benchmark.

How to use OmniWeaving

• Environment preparation: After cloning the OmniWeaving repository from GitHub, install the project dependencies in requirements.txt and optionally install Flash Attention or SageAttention acceleration libraries as needed to optimize inference performance.
• Model download: Download the HY-OmniWeaving model weight files released by Tencent Hunyuan from the HuggingFace platform to a specified local directory.
• Text-to-video: Run the generation script and specify the task type as t2v, input the text description, set the aspect ratio and output path, and optionally add thinking mode to let the model first reason about the intent before generating the video.
• Image-to-video: Use the i2v task type, provide the path to the first-frame image and the action description text, and the model will generate a continuous dynamic video based on the static image and prompt.
• First-last frame interpolation: Select the interpolation task, pass in two images for the starting frame and ending frame along with a transition description, and the model automatically generates a coherent video filling in the intermediate process.
• Multi-image composition generation: Through the reference2v task, upload one to four reference images (such as characters, props, backgrounds), and combine them with text prompts to achieve video creation with free composition of multiple elements.
• Video editing: Use the editing task to upload a source video and input editing instructions (such as style conversion or object replacement), and the model intelligently modifies the video content based on text guidance.
• Joint image-text-video editing: Uses the tiv2v task to simultaneously input a source video and a reference image, enabling advanced editing that fuses visual elements from the reference image into the dynamic video scene.

OmniWeaving Project Address

• Project Website：https://omniweaving.github.io/
• GitHub Repository：https://github.com/Tencent-Hunyuan/OmniWeaving
• HuggingFace Model Library：https://huggingface.co/tencent/HY-OmniWeaving
• arXiv Technical Paper：https://arxiv.org/pdf/2603.24458

Key Information and Usage Requirements of OmniWeaving

• Project Positioning: A unified video generation framework open-sourced by Tencent Hunyuan in collaboration with Zhejiang University and Nanyang Technological University on April 3, 2026.
• Core Technologies：
  • MLLM+MMDiT+VAE Architecture: A multimodal large language model parses interleaved text-image-video inputs, a diffusion Transformer generates video, and a variational autoencoder compresses visual signals
  • Free-form Composition: Supports temporal binding and flexible combination of text, multiple images (1-4), and video inputs
  • Reasoning-enhanced Generation: Activates the MLLM "thinking mode" to actively infer complex user intent and generate semantically precise video content
• Usage Requirements：
  • Hardware: Multi-GPU environment (official example uses 8 GPUs), supports memory optimization options (Flash Attention/SageAttention).
  • Software: Python environment, based on PyTorch, requires installing HunyuanVideo-1.5 dependencies.

Core Advantages of OmniWeaving

• Unified and Versatile: OmniWeaving integrates six major task categories, including text-to-video, image-to-video, multi-image composition, and video editing, through a single framework, replacing the fragmented solutions that traditionally require multiple specialized models and achieving unified generation across the entire pipeline.
• Free Composition: It breaks through the fixed input format limitations of traditional models, supports interleaved multimodal inputs of 1-4 images, video clips, and text, and can perform temporal binding to understand the spatiotemporal relationships between elements, achieving true fusion generation for complex scenes rather than simple concatenation.
• Reasoning Enhancement: After activating the MLLM thinking mode, the model upgrades from a passive instruction-following "renderer" to an "intelligent director" that actively infers user intent. It can automatically complete cinematic language and narrative logic, significantly lowering the prompt engineering barrier for complex creative descriptions.
• Deep Semantic Injection: It employs a DeepStacking mechanism to extract multi-granularity semantic features from multiple layers of MLLM hidden states and directly injects them into the underlying layers of the generation network, achieving dual guarantees of pixel-level detail control and high-level semantic alignment, and solving the problem of detail loss in multi-subject generation.

Comparison of OmniWeaving with Similar Competitors

Application Scenarios of OmniWeaving

• Film and Television Advertising Creative Production: Directly generate concept preview videos from text to validate storyboards, or freely combine reference images of characters, scenes, and props to generate complete advertisements.
• E-commerce Product Dynamic Display: Upload a product image on a white background and a scene reference image to automatically generate a product usage scenario video, and combine user photos with product videos to generate personalized virtual try-on effects.
• Social Media Content Creation: Transform static photos into dynamic videos with natural facial expressions and movements to bring old photos "to life," and quickly generate looping animations or fun emoji packs based on first-last frame interpolation.
• Game Animation Asset Production: Upload character design images and action descriptions to directly generate character animation clips and accelerate cutscene production, and provide keyframes to let AI automatically fill in the intermediate animation for scene transitions.