MineExplorer - Meituan's open-world minute-level long-horizon task evaluation benchmark

What is MineExplorer

MineExplorer is launched by Meituan's LongCat team and is the first Minecraft-based open-world minute-level long-horizon task evaluation benchmark. MineExplorer contains 813 manually verified instances of 1-4 hops, with tasks containing hidden preconditions; the instructions do not enumerate the dependency graph, and models need to infer subtasks autonomously. MineExplorer proactively strips away game-specific knowledge and only tests general dynamic exploration capabilities. It is accompanied by a five-agent collaborative data synthesis and rule-based milestone automatic evaluation framework, and the code and dataset are now open source.

Main features of MineExplorer

• Minute-level open-world evaluation: Based on the real-time 3D sandbox of Minecraft, each task runs for 1800 environment steps, and the model needs to continuously adjust strategies according to dynamic visuals.
• Hidden-precondition multi-hop tasks: Provides 813 manually verified instances of 1-4 hops; instructions give only the final goal, with hidden precondition subtasks, requiring the model to infer the dependency graph autonomously.
• Knowledge-decoupled evaluation: Proactively strips away Minecraft-specific mechanics and retains only tasks that depend on general world common sense.
• Rule-based automatic evaluation: Converts subtasks into automatically executable milestone checkers, enabling automatic scoring without manual annotation.
• Multi-agent data synthesis: Five agents collaborate to automatically generate tasks, including the full process of task selection, scene design, milestone design, expert review, and validation.

Technical principles of MineExplorer

• Dynamic environment interaction architecture: The evaluation environment is a real-time running physical sandbox, not static screenshot question answering; each time the model executes an action, the world state updates immediately, and it makes continuous decisions based on the latest observations.
• Implicit DAG task modeling: Each composite task is defined as a quadruple τ=(q,s₀,Gτ,Mτ), where Gτ is the task dependency graph. The key design is that the instruction q does not enumerate all nodes in the DAG, and the agent must infer hidden prerequisite tasks from the environment.
• Multi-agent collaborative task creation process: The orchestrator controls five specialized agents (task selector, scene designer, milestone designer, Minecraft expert, validator) to collaborate in a group chat, generating drafts and revisions in two phases, initialization and debate, improving efficiency by about 30% compared with a single agent.
• Knowledge filtering mechanism: Uses an LLM judge to make domain knowledge judgments for each atomic task, distinguishing general world common sense from Minecraft-specific mechanics, and filters out high-difficulty game-exclusive tasks that depend on the latter.
• ReAct capability decomposition framework: Drawing on the ReAct paradigm, it systematically decomposes open-world exploration capability into three major dimensions, perception, reasoning, and action, totaling 14 fine-grained metrics.

Follow WeChat and reply "open source" to join theAI open-source project exchange group

How to use MineExplorer

• Environment preparation: Clone the GitHub repository locally, and ensure Python 3.9+ and the relevant Minecraft runtime environment are installed.
• Start the evaluation environment: Configure and start the Minecraft sandbox server to ensure the evaluation environment can run in real time and receive model action instructions.
• Run the benchmark evaluation: Load the 813 verified instances provided by MineExplorer, connect the multimodal model to be tested to the environment, and the model outputs action sequences based on real-time visuals.
• Automatic Scoring: After the evaluation ends, the system automatically invokes the regularized milestone checker and calculates TSR and MSR scores based on quantitative metrics such as inventory items and coordinate regions.
• Custom Task Generation: Run the multi-agent data synthesis script, configure the number of task hops, and have five collaborating Agents automatically generate new task instances containing hidden preconditions.
• Training and Iteration: Input the generated tasks as training data into the model, and repeatedly execute the exploration-feedback loop in the Minecraft sandbox to improve long-horizon planning capabilities in open worlds.
• View Results: The evaluation report automatically outputs scores for each dimension, supporting stratified comparisons by model and difficulty.

MineExplorer's Core Advantages

• First Minute-Level Open-World Evaluation: Based on a real-time running Minecraft 3D sandbox, the model needs to make continuous decisions during 3 minutes of dynamic interaction.
• Hidden-Prerequisite Multi-Hop Tasks: Instructions only give the final goal and do not enumerate the dependency graph, forcing the model to infer hidden subtasks on its own.
• Knowledge-Decoupled Design: Actively strips away Minecraft-specific mechanics and retains tasks based on general world common sense.
• Efficient Multi-Agent Task Creation: Five Agents collaborate to generate tasks, improving efficiency by about 30% compared with a single agent, and supporting customizable expansion from 1-12 hops.
• Regularized Automatic Evaluation: Converts subtasks into automatically executable milestone checkers, enabling quantifiable scoring without manual annotation, with reproducible results.

MineExplorer's Project Links

• GitHub Repository：https://github.com/meituan-longcat/MineExplorer
• arXiv Technical Paper：https://arxiv.org/pdf/2605.30931

Comparison of MineExplorer with Similar Competitors

MineExplorer's Application Scenarios

• Multimodal Large Model Capability Evaluation: Provides a standardized examination arena for dynamic open-world long-horizon planning and reasoning for top MLLMs such as Claude, GPT, and Gemini.
• Agent Training Environment: As a Minecraft sandbox training ground, it supports models in continuously improving their perception-reasoning-action closed-loop capabilities through ongoing interaction.
• Model Selection Reference: Provides industry with a horizontal comparison leaderboard for open-world exploration capabilities, assisting in model selection and capability boundary assessment.
• Academic Research Platform: Advances cutting-edge research directions such as long-horizon reasoning, implicit task decomposition, visual grounding, and action alignment.
• Embodied Intelligence Pre-research Validation: Provides a low-cost capability validation sandbox for real physical world applications such as robotics and autonomous driving.