SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsOther AI toolsGamingAgent

GamingAgent

GamingAgent is an open-source project for game agents and model evaluation aimed at large language models (LLMs) and vision-language models (VLMs), and its related paper was accepted by ICLR 2026. The project provides two major functions: the LMGame Bench evaluation benchmark and the Gaming Agent workflow. It can both evaluate the performance of advanced models in standardized interactive game environments under a single-model VLM setup, and deploy and evaluate models through its customized Gaming Agent workflow (gaming harness) to improve game performance. In addition, it provides a computer-use agent (CUA) solution that can run on personal computers and laptops. The project supports multiple model families including OpenAI, Anthropic, Gemini, xAI, and Deepseek, and provides resources such as a leaderboard, papers, a gallery, and an official website.

Open website

About this tool

GamingAgent is maintained by lmgame-org, and its related paper is labeled [ICLR 2026]. Its topic is LLM/VLM game agents and model evaluation through games. The repository also provides entry points to the paper, leaderboard, gallery, and official website.

Main Features

The project contains two core features:

  1. Evaluate frontier models in a diverse suite of video games under a naive single-model VLM setup (without using a game harness).
  2. Deploy and evaluate models using its customized GamingAgent workflow (gaming harness) to improve the models' game performance. In addition, the repository provides a convenient solution for deploying a computer-use agent (CUA) that runs on your PC and laptop to play games.

Supported Models

The project currently supports game agents based on the following models:

  • OpenAI: o4-mini, o3-mini, o3, o1, gpt-4o, gpt-4o-mini
  • Anthropic: claude-4-opus, claude-4-sonnet (including thinking mode), claude-3-7-sonnet (including thinking mode), claude-3-5-haiku, claude-3-5-sonnet
  • Gemini: gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash-thinking-exp, gemini-2.0-pro, gemini-2.0-flash, gemini-1.5-pro
  • xAI: grok-3-mini
  • Deepseek: reasoner (R1), chat (V3) You can check the leaderboard to compare the performance of each model. You need to set API keys in credentials.sh before testing models yourself; the project notes that using high-end models to evaluate or deploy agents may incur relatively high costs.

Game Environments and Setup

The project standardizes game environment interfaces according to Gymnasium. The current evaluation suite includes multiple games based on gym environments, and most games run out of the box without additional setup.

  • Pokemon Red: You need to place the ROM file (pokemon.gb) into gamingagent/configs/custom_06_pokemon_red/rom/; the project uses pyboy as a Game Boy emulator, integrating it into the gymnasium environment.
  • Retro environments: Stable Retro supports providing a standard interface for classic game emulation through Gymnasium. Running classic games implemented by Retro requires legally obtaining the game files and importing them according to the instructions. The current evaluation suite includes Retro environment games such as Super Mario Bros 1985.
  • Additionally integrated Retro environments (no need for retro.import): such as Ace Attorney: Phoenix Wright, which can be enabled simply by placing the ROM file in the specified directory.

Usage

  • Clone the repository and install dependencies.
  • Launch multiple evaluation instances of a model on different games in parallel, or evaluate multiple models in parallel.
  • Evaluate model performance in game agents (supports gaming harness).
  • run.py will launch multiple single_agent_runner.py instances; a single model on a single game can be run with: python3 lmgame-bench/single_agent_runner.py --game_name {game_name} --model_name {model_name} --config_root_dir {path_to_gaming_agent_config} (--harness).
  • Adjust game agent-related configurations in gamingagent/configs/{game_env_dir}/config.yaml; prompts are located in gamingagent/configs/{game_env_dir}/module_prompts.json.
  • The performance analysis in the paper can be reproduced through the lmgame_bench_evaluation colab or evaluation notebook, and the generated cache directory can also be uploaded to the colab workspace to visualize model performance.
  • Based on the agent configuration file and the turn logs in the performance cache, replay videos can be generated for Sokoban, 2048, Tetris, and Candy Crush games (using text representations).
  • Supports computer-use game agents to run frontier models in real time on local PCs and laptops, see computer_use Directory description.

Adding your own games

You can integrate game environments with standardized interfaces according to the instructions for Gymnasium (self-developed game implementations) or Stable Retro (existing classic games). Specifically, implement the observation making and agent action processing methods in gamingagent/envs to connect the game environment with LLM/VLM game agents; and add the agent configuration in configs so that the agent interacts correctly with the environment. More detailed instructions are in gamingagent/envs.

Citation

If this repository is helpful, you can cite it as [ICLR 2026] LLM/VLM gaming agents and model evaluation through games.

Update log

  • [2025/6] Lmgame Bench is officially released.
  • [2025/3] Build game agents and test different models on classic video games.

Related tools

Browse category