What GameFactory Is
GameFactory is an innovative framework jointly proposed by the University of Hong Kong and Kuaishou Technology, addressing the challenge of scene generalization in gamevideo generation. The framework is based on a pretrained video diffusion model, combining open-domain video data and a small-scale high-quality game dataset, and through a multi-stage training strategy, achieves action-controllable game video generation.
GameFactory's Main Features
- Scene Generalization Ability: GameFactory can generate diverse game scenes, not limited to a single style or preset scenes. This makes the generated videos more realistic and diverse.
- Action Controllability: Through the action control module, GameFactory can achieve precise control over the actions of characters or objects in the generated videos.
- High-Quality Dataset Support: To support action-controllable video generation, GameFactory adopts the GF-Minecraft dataset. This dataset contains 70 hours of Minecraft gameplay videos, with diverse scenes and detailed action annotations.
- Interactive Video Generation: GameFactory supports the generation of indefinitely long interactive game videos. Users can control changes in the video content by inputting instructions or interaction signals.
GameFactory's Technical Principles
- Pretrained Video Diffusion Model: GameFactory is based on a video diffusion model pretrained on open-domain video data, and can generate diverse game scenes, breaking through the limitations of traditional methods on specific game styles and scenes.
- Three-Stage Training Strategy
- First Stage: Use LoRA (Low-Rank Adaptation) to fine-tune the pretrained model so that it adapts to the target game domain while retaining most of the original parameters. This ensures that the model's generalization ability in the open domain is preserved.
- Second Stage: Freeze the pretrained parameters and LoRA, and focus on training the action control module to avoid entanglement between style and control signals. The goal is to ensure that the generated videos can perform accurate action control according to user input.
- Third Stage: Remove the LoRA weights and retain the action control module parameters, enabling the system to generate controlled game videos in various open-domain scenes, not limited to a specific game style.
- Action Control Module: GameFactory introduces an action control module that supports autoregressive action control and can generate indefinitely long interactive game videos.
- GF-Minecraft Dataset: To support action-controllable video generation, the research team released a high-quality action-annotated video dataset, GF-Minecraft. The dataset contains 70 hours of Minecraft gameplay videos, with diverse scenes and detailed action annotations. The design of the dataset satisfies the following three key requirements:
- Customizable Actions: Supports large-scale, low-cost data collection.
- Unbiased Action Sequences: Ensures the diversity of action combinations and coverage of low-probability events.
- Diverse Scenes: Enhances scene diversity through preset environmental configurations (such as different biomes, weather conditions, and time periods).
GameFactory's Project Address
- Project Official Website：https://vvictoryuki.github.io/gamefactory
- arXiv Technical Paper：https://arxiv.org/pdf/2501.08325
Application scenarios of GameFactory
- Game development: Through a pretrained video diffusion model and an action control module, GameFactory can generate diverse game scenes and interactive video content.
- Game content innovation: GameFactory's open-domain generation capability enables it to break through the limitations of traditional game engines and create entirely new game styles and scenes.
- Autonomous driving: Its action control module and scene generation capability can be used to simulate autonomous driving environments and generate diverse driving scenarios.
- Embodied AI: By generating diverse interactive scenes, it provides support for research on embodied intelligence.