OmniShow - A video generation model open-sourced by ByteDance in collaboration with CUHK, HKU, and others

What is OmniShow

OmniShow is a multimodal human-object interaction model open-sourced by ByteDance in collaboration with CUHK, Monash, and HKUVideo generationmodel. As the first end-to-end framework supporting RAP2V (reference image + audio + pose), the model uniformly processes four types of input: text, image, audio, and pose, and a single 12.3B-parameter model can generate 10-second high-quality videos. The model adopts techniques such as gated local context attention to achieve precise audio-video synchronization, reaching multiple SOTA results on the HOIVG-Bench benchmark.

Main features of OmniShow

• Full-modality conditional input: As the first end-to-end framework to fully support RAP2V (Reference+Audio+Pose-to-Video), OmniShow can simultaneously receive four modality inputs: reference images, audio, pose signals, and text descriptions, achieving comprehensive control over human-object interaction videos.
• Unified multi-task generation: By flexibly combining different input conditions, the model can complete multiple video generation tasks within a single architecture, including reference-image-based video generation (R2V), audio-driven digital human videos (RA2V), pose-driven animation (RP2V), and full-modality precise control (RAP2V).
• High-quality long video synthesis: OmniShow natively supports generating continuous videos up to 10 seconds long. While maintaining character appearance consistency, it achieves precise synchronization of lip movements, facial expressions, and body movements with audio, with output reaching industry-level visual quality.
• Object replacement and video remixing: Users can replace objects in a video while preserving the person's movements and poses, or reassemble poses, objects, and person references from different sources, enabling flexible creative video editing and compositing.

Technical principles of OmniShow

• Unified channel condition injection: By concatenating reference images and pose signals along the channel dimension, multimodal conditions are efficiently injected into the pretrained video generation model, avoiding the damage to the base model's pretraining priors caused by traditional modulation methods and ensuring that the original generation quality is maintained while introducing human-object appearance and motion control.
• Gated local context attention: Using masked attention combined with learnable gating vectors, the model can dynamically focus on the associations between audio features and local facial/body regions, achieving precise temporal synchronization of lip movements, facial expressions, and body movements with audio signals, and resolving feature conflicts during multimodal fusion.
• Decoupled-joint training strategy: To address the challenge of heterogeneous and imbalanced data scales for R2V and A2V tasks, dedicated sub-models are trained separately to learn the independent mapping relationships of each modality, then fused through weight interpolation and fine-tuned on joint data, effectively integrating heterogeneous datasets and improving generation consistency under full-modality input.

How to use OmniShow

• Obtain the open-source code: Clone the code repository from the official channel to your local machine, and configure the runtime environment including dependencies according to the README.
• Prepare multimodal input: According to the generation task requirements, organize reference images (person or object appearance), driving audio, pose sequence files, and text descriptions.
• Execute video generation: Call the model API or run the inference script, input the combined multimodal conditions, generate a high-quality human-object interaction video up to 10 seconds long, and save the output results.

Key information and usage requirements of OmniShow

• Project positioning: OmniShow is an industry-grade multimodal human-object interaction video generation model jointly open-sourced by ByteDance, the Chinese University of Hong Kong, Monash University, and the University of Hong Kong, specifically designed for complex human-object interaction scenarios.
• Technical Scale: The model adopts a 12.3B parameter architecture, making it the most parameter-efficient solution among similar models (such as HuMo-17B, Phantom-14B), significantly reducing computational resource requirements while maintaining high performance.
• Core Capabilities: As the first end-to-end unified framework to fully support RAP2V (Reference+Audio+Pose-to-Video), it can simultaneously receive four modality inputs—reference images, audio signals, pose sequences, and text descriptions—for joint reasoning.
• Generation Quality: It natively supports generating continuous videos up to 10 seconds long, achieving precise synchronization of lip movements, facial expressions, and body movements with audio through gated local context attention technology, reaching industry-grade visual standards.
• Performance: In the self-built HOIVG-Bench benchmark, OmniShow achieves SOTA levels on all four tasks—R2V, RA2V, RP2V, and RAP2V—making it the only leading solution that supports all-modality input.

OmniShow's Core Advantages

• All-Modality Unified Architecture: As the industry's first end-to-end framework to fully support RAP2V (reference image+audio+pose→video), OmniShow achieves native unified processing of four conditions—text, image, audio, and pose—without the need to stitch together multiple specialized models to handle complex multimodal input combinations.
• Extreme Parameter Efficiency: With 12.3B parameters, it achieves multiple SOTA results on the HOIVG-Bench benchmark, significantly outperforming larger-parameter models such as HuMo-17B (17B) and Phantom-14B (14B), greatly reducing inference costs while maintaining industry-grade generation quality.
• Single-Model Multi-Task Coverage: Through flexible input combination strategies, a single OmniShow model can handle four types of tasks—R2V (reference image generation), RA2V (audio-driven digital human), RP2V (pose-driven animation), and full RAP2V (all-modality control)—without the need to switch models for different scenarios.
• Precise Audio-Video Synchronization: It adopts an innovative gated local context attention mechanism, achieving a high degree of synchronization of lip movements, facial expressions, and body movements with audio signals through learnable gating vectors and masked attention, reaching a leading level of 8.612 on the Sync-C metric.
• Native Long Video Generation: Unlike extrapolation methods that rely on sliding windows, OmniShow natively supports generating continuous videos up to 10 seconds long in a single pass, with significant advantages in maintaining character appearance, object style, and temporal coherence.

OmniShow's Project Address

• Project Website：https://correr-zhou.github.io/OmniShow/
• GitHub Repository：https://github.com/Correr-Zhou/OmniShow

Comparison of OmniShow with Similar Competitors

Application scenarios of OmniShow

• E-commerce product display: Through the reference image replacement feature, display clothing, accessories, or products can be quickly replaced while keeping the model's movements and poses unchanged, enabling efficient product video presentation.
• Digital human short video production: Based on the RA2V (reference image + audio) capability, inputting a person's photo and voice can generate a lip-synced talking or singing video, suitable for virtual anchors and social media content creation.
• Creative video mixed editing: With multimodal recombination capability, poses, objects, and person references can be extracted from different sources for creative composition, enabling cross-video personalized content editing and secondary creation.
• Interactive entertainment and games: With the pose-driven (RP2V) feature, user motion capture can be converted in real time into animation for game characters or virtual avatars, supporting real-time interactive experiences with full-body pose control.
• Advertising and marketing content generation: Combined with full-modal input (RAP2V), the appearance, dialogue audio, and body movements of brand spokespersons can be precisely controlled to generate promotional video materials with a consistent style in batches.