HOMIE - An Open-Source Digital Human Video Generation Framework from HKUST

What is HOMIE

HOMIE is an open-source digital human video generation framework from the Hong Kong University of Science and Technology, built on the Wan2.1-T2V-14B backbone and integrating Qwen3-VL multimodal large models. The framework can uniformly handle the four elements of digital human + product + Logo + text, precisely solving the four major challenges of human-object interaction, brand logo placement, OCR text fidelity, and multi-view consistency. Model training requires only 5.5K steps / about 10,000 A100 hours, OCR accuracy is improved by 38.7% over the strongest open-source model, and it supports 480P single-GPU to 720P multi-GPU inference, greatly lowering the barrier to video production for scenarios such as e-commerce livestreaming and virtual anchors.

Main Features of HOMIE

• Personalized human-object interaction videos: Supports multiple people and multiple objects appearing in the same video simultaneously, maintaining the appearance consistency of each subject and achieving natural, reasonable interaction actions (such as holding, displaying, and passing products).
• Precise attachment of Logo / abstract concepts: Uses the semantic reasoning capability of a multimodal large model (MLLM) to automatically attach abstract concepts such as brand Logos to the most semantically relevant objects, without the need to explicitly describe positional relationships in the prompt.
• High-fidelity OCR text generation: With OCR reference images, ensures that text on product packaging and labels is clearly readable, not blurry, and not garbled in the generated video, solving the pain point of "text blurring into a mess" in AI video generation.
• Multi-view consistent generation: Given multi-view reference images of the same object, it can maintain consistent appearance and details when the object rotates or moves in the video, suitable for display scenarios such as figurines, 3D models, and dolls.

Technical Principles of HOMIE

• Overall architecture: Using Alibaba Wan2.1-T2V-14B's DiT as the backbone, introducing Qwen3-VL-2B-Thinking as the visual understanding module, and working with the UmT5 text encoder and VAE visual encoder, it forms a three-path parallel generation architecture of "text + multimodal semantics + vision".
• MLLM integration strategy: Without breaking the original text-to-video alignment relationship or incurring costly realignment costs, it injects the semantic reasoning capability of the multimodal large model into the DiT, enabling the model to automatically understand the latent relationships between reference images.
• Global Multimodal Self-Attention Guidance (GMG): In the DiT's self-attention layers, it globally aligns the semantic features extracted by the MLLM with the VAE visual tokens, and through projection, pooling, and affine transformation, enables every video token in the generation process to "see" the global semantic information of the reference images.
• Modality-Reference Embedding (MRE): It assigns modality embeddings to MLLM feature tokens and VAE visual tokens respectively, assigns independent reference embeddings to different reference images, and associates the multi-view/OCR reference image tokens of the same subject, avoiding information confusion.
• Three-stage progressive training: Single subject 480P -> multiple subjects 480P -> high-definition 720P, completing training in only 5.5K steps in total, far lower than the tens of thousands of steps of similar methods.

Follow WeChat and reply "open source" to join theAI Open-Source Project Exchange Group

How to Use HOMIE

• Environment preparation: Clone the HOMIE code repository from GitHub to your local machine and complete the installation of environment dependencies.
• Weight download: Download the officially released pretrained weights from HuggingFace to the specified model directory.
• Material Preparation: Prepare reference images, including multi-subject materials such as portrait photos, product images, Logo images, or OCR text images.
• Prompt Writing: Write text prompts that accurately describe the interaction actions and scenes between people and objects in the target video.
• Run Inference: Run the inference script, and HOMIE will automatically fuse multimodal reference information and generate a personalized video.
• Resolution Selection: A single GPU can directly generate 832×480 resolution videos, while multi-GPU FSDP can scale up to 720P HD output.

HOMIE's Core Advantages

• Unified Framework: A single framework simultaneously handles four major tasks: human-object interaction, Logo branding, OCR text fidelity, and multi-view consistency, without the need to switch between different models for different scenarios.
• Efficient Training: Training can be completed in only 5.5K steps / about 10,000 A100 hours, at a cost far lower than the 30,000–40,000 steps typically required by comparable methods.
• Lossless MLLM Integration: Inject the semantic reasoning capability of multimodal large models into the video generation pipeline without sacrificing the controllability of the text encoder and without incurring expensive realignment costs.
• Leading Quality: OCR accuracy is relatively improved by 38.7%, multi-view consistency is improved by 8.2%, and in a 40-person user study, 64.7% voted the overall quality as first.
• Flexible Inference: Supports inference from 480P on a single GPU to 720P on multiple GPUs, and can output 1280×720 vertical videos, directly suitable for livestream e-commerce and short video production.

HOMIE's Project Addresses

• Project Website：https://yiyangcai.github.io/homie-page.github.io
• GitHub Repository：https://github.com/YIYANGCAI/HOMIE
• HuggingFace Model Library：https://huggingface.co/yychai/homie-r2v-wan2.1
• arXiv Technical Paper：https://arxiv.org/pdf/2607.18217

Comparison of HOMIE with Similar Competitors

HOMIE's Application Scenarios

• E-commerce Livestream Selling: Generate sales videos in which a specified digital human holds a specified product and performs specified actions, with support for 1280×720 vertical output, directly suitable for short video platforms.
• Brand Advertising Production: Automatically attach a brand Logo to semantically related products, without the need to explicitly specify positional relationships in the prompt.
• Virtual Anchors / Digital Humans: Maintain the digital human's identity consistency while naturally interacting with a variety of products.
• Multi-view product showcase: Given multi-view reference images of a figure, 3D model, or doll, generate a rotating showcase video with a consistent appearance across all angles.
• Packaging text fidelity: With OCR reference images, ensure that the text on product packaging and labels is clearly legible in the video, suitable for scenarios such as food, cosmetics, and health supplements that need to display ingredients/instructions.