What is Lumine
Lumine is a general-purpose AI agent launched by ByteDance that perceives, reasons, and acts in real time in 3D open-world games. Lumine is based on the Qwen2-VL-7B-Base model and, through a human-like interaction paradigm, integrates perception, thinking, and action. Lumine can perceive the game environment in real time and execute complex tasks, including combat, puzzle solving, interacting with NPCs, and GUI operations. Lumine performs excellently in Genshin Impact, can complete main story quests lasting several hours, and demonstrates cross-game generalization ability in other games. Through multi-stage training, Lumine gradually improves its autonomous thinking and operation abilities, providing a new direction for the future development of embodied intelligence.
Lumine's main features
-
Task execution: Can autonomously complete complex long-cycle tasks, such as completing main storylines lasting several hours in Genshin Impact.
-
Combat ability: Dynamically tracks enemies, shoots accurately, switches characters to perform combo attacks, and efficiently opens chests.
-
Puzzle-solving ability: Supports handling various puzzle challenges, such as collecting Anemoculus and activating elemental monuments.
-
NPC interaction: Can conduct stable conversations with NPCs and complete quests.
-
GUI operations: Performs 2D interface operations like a human player, such as crafting items and using teleport waypoints.
-
Cross-game generalization: Without additional fine-tuning, can complete tasks in other games (such as Honkai: Star Rail and Wuthering Waves).
Lumine's technical principles
-
Perceiving space: Processes one frame of game footage every 200 ms, retains historical reasoning traces, and provides contextual information for decision-making.
-
Hybrid thinking strategy: Reasons in key scenarios (such as sudden environmental changes or plan failures), while directly outputting actions in simple scenarios to improve efficiency.
-
Keyboard and mouse operation modeling: Defines all operations as mouse movements and key sequences, and uses high-quality data for three-stage training:
-
Pre-training: Learns basic visuomotor abilities.
-
Instruction-following training: Associates language instructions with actions.
-
Decision-making reasoning training: Learns to plan and correct autonomously to complete long-cycle tasks.
-
-
Real-time optimization: Through context management and multi-dimensional optimization, reduces latency and ensures real-time interaction.
Lumine's project address
- Project website：https://www.lumine-ai.org/
- arXiv technical paper：https://arxiv.org/pdf/2511.08892
Lumine's application scenarios
- Game development and testing: Used in game automated testing to help developers quickly discover bugs, performance issues, and user experience problems, while also assisting in generating intelligent NPC behavior and quest design, improving game development efficiency.
- Game entertainment: As an AI teammate or opponent, provides players with a more challenging and interesting interactive experience, or assists players in completing complex tasks in single-player games, improving game completion.
- Education and training: Used in virtual training environments to provide task training for students or professionals, or to assist learning in educational games, helping students master knowledge and skills through tasks and challenges.
- Virtual worlds and the metaverse: Interact with users in the metaverse as a virtual character, provide companionship or services, help users generate tasks, storylines, or interactive content in the virtual world, and enrich the user experience.
- Industry and Manufacturing: Execute tasks and optimize processes in industrial simulation environments, help design efficient working modes, and use them in intelligent robot training to improve their autonomous decision-making and operational capabilities.