What is PlanningBench
PlanningBench is an open-source framework launched by the Tencent Hunyuan team in collaboration with the Gaoling School of Artificial Intelligence at Renmin University of China and other institutions, focusing on the evaluation and training of the planning capabilities of large language models. Starting from real-world scenarios, the framework builds a verifiable data generation system covering six major categories and more than 30 kinds of planning tasks. Through a constraint-driven closed-loop synthesis process, it moves model planning results from appearing reasonable to being truly executable.
Main features of PlanningBench
- Real-world scenario coverage: Covers six major categories, including scheduling, resource allocation, workforce rostering, route scheduling, production operations, and emergency services, with more than 30 specific planning tasks in total.
- Three-tier constraint system: Decomposes planning difficulty into basic constraints, intermediate constraints, and hard constraints, enabling fine-grained control over task difficulty.
- Automatic verification mechanism: Each instance is paired with an executable checklist, supporting evaluation with both Avg-pass and All-pass metrics.
- Closed-loop data generation: Through a closed-loop process in which the Generator generates questions, the Responder attempts to solve them, and the Critic verifies the answers, data difficulty is automatically enhanced.
- Training signal output: Verifiable samples can be directly used for reinforcement learning training, providing the model with stable optimization signals for planning capability.
Technical principles of PlanningBench
- Abstraction of real-world scenarios and construction of a constraint system: PlanningBench has professional annotators with planning experience collaborate with algorithm researchers to abstract reusable problem structures from real-world scenarios such as personal commuting, enterprise production scheduling, and medical emergencies, ultimately forming six major planning task families and more than 30 specific task types. Constraints are divided into three levels: basic constraints (time windows, resource boundaries), intermediate constraints (multi-objective optimization, dependency relationships), and hard constraints (exception recovery, goal conflicts), enabling data generation to combine tasks and constraints within a structured design space.
- Constraint-driven closed-loop synthesis process: The core of the framework is a closed-loop system in which the three roles of Generator, Responder, and Critic collaborate: the Generator generates self-contained planning problems and corresponding verification checklists based on sampled combinations of tasks and constraints; the Responder model attempts to generate solutions; the Critic checks item by item according to the checklist whether the answer satisfies all requirements. If it passes, the difficulty is increased; otherwise, it is retained as a challenging sample, making PlanningBench a dynamic data generation system that can actively seek the boundaries of model capabilities.
- Three-tier difficulty constraint system: PlanningBench decomposes planning difficulty into factors such as task structure, constraint levels, resource tightness, goal conflicts, dependency relationships, and exception handling, and controls the difficulty of generated data by controlling the sampling distribution of the constraint pool. Basic constraints ensure solution completeness and executability; intermediate constraints introduce fairness and load balancing requirements; hard constraints test the model's capabilities in infeasibility identification, minimal-change replanning, and robustness design, making the increase in difficulty truly affect the depth of planning reasoning.
How to use PlanningBench
- Obtain resources: Visit the GitHub repository to clone the code, or download the dataset from HuggingFace.
- Run evaluation: Load the PlanningBench evaluation set, call the model to generate planning solutions, and use the built-in checklist to automatically calculate the Avg-pass and All-pass metrics.
- Generate customized data: Configure task types and constraint levels, run the closed-loop generation process, and produce planning training data for specific difficulty levels and domains as needed.
- Train the model: Use the generated verifiable samples as reward signals to perform reinforcement learning training on the target model, transferring to downstream planning tasks.
Core Advantages of PlanningBench
-
Verifiability: Precisely determine whether a plan is globally executable through an automatic checklist.
-
Controllable Difficulty: Regulate task difficulty based on constraint levels rather than simply increasing prompt length, bringing it closer to real-world planning complexity.
-
Precise Diagnosis: The dual metrics of Avg-pass and All-pass can identify model outputs that are "locally correct but globally unexecutable."
-
Training Transfer: Only 300 samples are needed to improve the model's performance on external planning benchmarks and general instruction-following tasks through GRPO training.
-
Realistic Scenarios: Tasks are derived from real-world scenarios such as commuting, production scheduling, and medical scheduling, avoiding model overfitting on fixed question banks.
Project Address of PlanningBench
- GitHub Repository：https://github.com/Tencent-Hunyuan/PlanningBench
- HuggingFace Model Hub：https://huggingface.co/datasets/tencent/PlanningBench
- arXiv Technical Paper：https://arxiv.org/pdf/2605.20873
Comparison of PlanningBench with Similar Competing Products
| Dimension | PlanningBench | AgentBench |
|---|---|---|
| Core Positioning | An evaluation and training framework focused on theplanning capabilitiesof large models, emphasizing the shift from "looking reasonable" to "actually executable." | A comprehensive evaluation framework for large modelagent capabilities, covering multiple dimensions such as reasoning, decision-making, tool use, and environment interaction. |
| Task Coverage | Covers more than 30 real planning tasks across 6 major categories, including scheduling, resource allocation, workforce scheduling, route dispatching, production operations, and emergency services. | Covers 8 interaction environments, including operating systems, databases, knowledge graphs, digital cards, lateral thinking, home management, online shopping, and web browsing. |
| Verification Mechanism | Each instance is paired with a structured checklist to automatically verify constraint satisfaction item by item, precisely determining whether a plan is globally executable. | Determines task success based on environment execution feedback and result matching, with independent verification logic for different environments. |
| Evaluation Metrics | Uses the dual metrics of Avg-pass (local constraint satisfaction rate) and All-pass (global executability rate) to identify outputs that are "locally correct but globally unexecutable." | Mainly uses Success Rate, calculated separately for the 8 environments to measure the final task completion level. |
| Difficulty Control | Actively regulates the degree of constraint coupling and resource conflict intensity through a three-tier constraint system (basic/medium/hard) and a closed-loop difficulty enhancement mechanism. | Increases difficulty through multi-turn interaction (5-50 turns) and progressively increasing environment complexity, focusing more on interaction depth than constraint density. |
| Environment Interaction | Targets static planning problem solving; the model generates a complete plan under clear inputs without the need for real-time interaction with an external environment. | Requires the model to interact in real time with simulated environments (such as a bash terminal, browser, or database) over multiple turns, dynamically obtaining feedback. |
| Training Support | Native support for training; verifiable data can be directly used as reward signals for reinforcement learning such as GRPO, and can transfer to external benchmarks. | Mainly used for evaluation; the framework itself does not directly provide scalable training data generation capabilities. |
Application Scenarios of PlanningBench
-
Model Capability Diagnosis: Precisely identify the weaknesses of large models in core planning capabilities such as constrained reasoning, resource allocation, and state tracking.
-
Agent Training: Provide verifiable planning training data for AI Agents, improving their reliability in tasks such as scheduling and logistics dispatching.
-
Academic Research: Serve as a benchmark tool for planning capability research, supporting reproducible and scalable model comparison experiments.
-
Enterprise Scheduling Optimization: Generate evaluation data for specific business scenarios (such as nurse scheduling and production scheduling) based on the framework, to assess the feasibility of model deployment.
-
Educational Grouping and Resource Allocation: Use task types such as student grouping and supply distribution to verify the fairness and optimality of models under multiple constraints.