H3-World - An Action-Controllable World Model Open-Sourced by Tencent and Universities

What is H3-World

H3-World is an action-controllable world model launched by Tencent in collaboration with the National University of Singapore and The Hong Kong Polytechnic University, fine-tuned based on a 33BMiniMax-H3video generator. The model translates keyboard operations into English instructions, injects them into the model through the pretrained text pathway, and uses temporal attention routing to bind each instruction to the corresponding video frame, achieving precise temporal alignment control. H3-World uses only 8,000 game videos and rank-32 LoRA to fine-tune 0.199% of parameters, enabling the model to generate controllable footage in real time according to keyboard input and generalize to unseen scenes, making it an important step toward interactive generated worlds.

H3-World's main features

• Keyboard action-to-video control: Converts keyboard-input character movement and camera operations into corresponding video frame changes, achieving interactive world simulation where "pressing a key is controlling."
• Precise temporal control: Different action instructions can be executed at different time periods within the same video, precisely controlling when actions occur.
• Multi-environment generalization: Starting from training on game footage, it can generalize to diverse visual environments such as neon cities, desert oases, and fantasy cathedrals while keeping scene content stable.
• Combinatorial action generalization: It can combine character/camera action primitives seen during training into action pairs that have never appeared before, achieving combinatorial generalization control.
• Long-horizon generation: Supports longer-sequence continuous controllable video generation, where action instructions can continuously drive the evolution of the footage.

H3-World's technical principles

• Action verbalization representation: Translates each frame's keyboard operations into a short English instruction (such as "strafe left"), letting actions exist in natural language form.
• Reusing the pretrained text pathway: Instructions are injected into the model through MiniMax-H3's original text encoding path, directly reusing the semantic representations learned during video pretraining, so the model naturally understands action meanings without learning action encodings from scratch.
• Temporal attention routing: Through directional attention masks, each instruction is bound to its corresponding video latent frame, limiting the instruction's effect to a predetermined time interval and avoiding control leakage between adjacent actions. This is the key to achieving precise temporal control.
• Lightweight LoRA fine-tuning: Only a rank-32 LoRA is trained on H3's self-attention projections, completing adaptation with 0.199% trainable parameters, 8,000 game videos, and 10,000 optimization steps, greatly reducing the cost of transforming a video generator into a world model.

Follow on WeChat and reply "open source" to jointhe AI Open Source Project Communication Group

How to use H3-World

Method 1: Online experience

• Open the interactive demo Space on Hugging Face (hugging-apps/h3-world-action-demo) to experience keyboard-driven control without downloading the 135GB weights.
• Upload or select an initial frame image and enter a scene description prompt.
• Input control instructions through preset actions or keyboard keys (character movement + camera operations).
• The model generates a controllable video clip of about 5.2 seconds at 832×480 in real time.

Method 2: Local deployment

• Environment preparation: Create a conda environment with Python 3.10 + CUDA 12.8, install PyTorch 2.10 and the dependencies in requirements.txt.
• Get the code: Clone the official H3-World repository, clone DiffSynth-Studio into the specified directory, switch to the specified commit, and apply the official attention patch (this cannot be omitted, otherwise LoRA will be ineffective).
• Download the weights: Download the MiniMax-H3 base weights (about 135GB) and the H3-World LoRA (step-10000.safetensors) from Hugging Face and place them in the corresponding directories of the repository.
• Configure the cache: Run env.sh to point the Hugging Face, Torch, and Triton caches inside the repository.
• Run inference: Provide the first-frame image, static semantic conditions, and a schedule-style action sequence (each segment corresponds to a character/camera instruction).
• Generate the result: The model generates video latents corresponding to the timeline according to the action schedule and decodes them for output, and can be extended autoregressively to longer horizons.

H3-World's core advantages

• Elegant design with zero new modules: Directly reuse the pretrained text pathway, transforming the action control problem into a language understanding problem that the model is already good at.
• Extremely lightweight training cost: Using only 8,000 game videos, 10,000 optimization steps, and rank-32 LoRA fine-tuning of 0.199% of parameters, it unlocks the interactive control capability of a 33B model.
• Time-precise control capability: Through temporal attention routing, it can execute different actions according to a schedule within the same video, solving the pain point that a global prompt cannot precisely control the timing of actions.
• Significantly better than traditional conditioning injection methods: Compared with direct action conditioning methods such as additive-bias and FiLM, the language grounding approach can produce coordinated character + camera motion without destroying the scene content.
• Strong compositional generalization ability: It can combine action primitives seen during training into action pairs that have never appeared before, and generalize to out-of-distribution visual environments.
• Strong versatility: A single model can handle multiple control types such as first/third person, character movement, and camera motion, without the need to train separately for different tasks.

H3-World's project addresses

• Project website：https://danzer1xxxxchan.github.io/H3-World/
• GitHub repository：https://github.com/Danzer1xxxxChan/H3-World
• HuggingFace model library：https://modelscope.cn/models/DANNY621/H3-World
• Technical paper：https://modelscope.cn/papers/2609.01560

Comparison of H3-World with similar competitors

Application scenarios of H3-World

• Game prototyping: Designers can input an initial frame to instantly "play" a virtual scene in real time, rapidly validating level concepts and camera language without first building a complete game engine.
• AI Agent training sandbox: Provides an interactive virtual environment for embodied agents, where the agent explores the world through keyboard actions and receives visual feedback, reducing real-world training costs.
• Film and animation previsualization: Directors can use the keyboard to control virtual camera movements (pan, truck, boom), generating storyboard previews in real time and greatly shortening the pre-production previz cycle.
• Robot manipulation policy validation: Maps movement commands for robotic arms/drones into an action language, testing the temporal plausibility of action sequences in generated simulation environments.
• Immersive content creation: A UGC tool that combines VR/game interaction to let users drive the evolution of personalized virtual worlds with natural key presses, enabling anyone to generate "playable videos".