What is CombatVLA
CombatVLA is an efficient Vision-Language-Action (VLA) model launched by the Future Life Lab team of Taotian Group, designed specifically for combat tasks in 3D action role-playing games (ARPGs). The model is based on a 3B parameter scale and is trained on video-action pairs collected by an action tracker, with data formatted into "Action-of-Thought" (AoT) sequences. The model uses a three-stage progressive learning paradigm, from video level to frame level and then to truncated strategy, achieving efficient inference. CombatVLA surpasses existing models in combat understanding benchmarks, with a 50x increase in inference speed and a task success rate higher than that of human players.
Main Features of CombatVLA
- Efficient Combat Decision-Making: CombatVLA can make combat decisions in real time in complex 3D game environments, including dodging attacks, casting skills, restoring health, etc., with decision-making speed 50 times faster than traditional models.
- Combat Understanding and Reasoning: CombatVLA can assess enemy status, judge enemy attack intentions, and reason out the optimal combat actions, with combat understanding significantly superior to other models.
- Action Command Generation: The model can output specific executable keyboard and mouse operation commands, such as pressing specific keys or performing mouse operations, to control game characters in combat.
- Generalization Ability: Across tasks of different difficulties and different games, CombatVLA demonstrates strong generalization ability and can effectively perform combat tasks in unseen game scenarios.
Technical Principles of CombatVLA
- Action Tracker: Based on an action tracker, it collects human players' game operation data, including keyboard and mouse operations, and synchronously captures game footage to generate video-action pair data.
- Action-of-Thought (AoT) Sequences: The collected data is converted into "Action-of-Thought" (AoT) sequences, with each action accompanied by a detailed explanation to help the model understand the semantics and logic of the action.
- Three-Stage Progressive Learning：
- Stage 1: Video-level AoT fine-tuning, allowing the model to initially understand the combat environment.
- Stage 2: Frame-level AoT fine-tuning, helping the model understand the strict alignment between actions and preceding frames.
- Stage 3: Frame-level truncated AoT fine-tuning, introducing a special token <TRUNC> to truncate output and accelerate inference.
- Adaptive Action Weight Loss: Through action alignment loss and modal contrastive loss, the model's training process is optimized to ensure accurate output of key actions.
- Action Execution Framework: The action commands output by the model are converted into actual keyboard and mouse operations based on the action execution framework, achieving automatic control of game characters.
Project Address of CombatVLA
- Project Official Website：https://combatvla.github.io/
- GitHub Repository：https://github.com/ChenVoid/CombatVLA
- arXiv Technical Paper：https://arxiv.org/pdf/2503.09527
Application Scenarios of CombatVLA
- 3D Action Role-Playing Games (ARPG): Real-time control of game characters in combat, achieving efficient decision-making and action execution, enhancing the gaming experience.
- Game Testing and Optimization: Helps developers test the game's combat system, discover and optimize potential issues.
- Esports Training: Provides intelligent opponents for esports players, helping users practice combat skills and tactics, and improving competitive level.
- Game content creation: Assist game developers in generating battle scenes and storylines, and quickly build complex game levels and quests.
- Robot control: In the real world, extend the technology to robot control, enabling robots to make rapid decisions and execute actions in dynamic environments.