HarnessEval - The evaluation system jointly launched by MirroS, Tsinghua University, Peking University, NVIDIA, and others

What is HarnessEval

HarnessEval is an evaluation system jointly launched by MirroS with Tsinghua University, Peking University, Berkeley, MIT, NVIDIA, and other institutions. HarnessEval turns evaluation itself into an Agent. For each case, the system dynamically plans the evaluation path, selects applicable skills from the Skill Library, decomposes it into verifiable sub-problems, calls tools to collect evidence, and ultimately outputs a traceable evidence tree. HarnessEval breaks through the limitations of traditional static benchmarks, turning evaluation from a black-box score into an interpretable, reproducible intelligent workflow. The first implemented project, HarnessEval-W, has been used for interactive world model evaluation.

Main features of HarnessEval

• Dynamic evaluation planning: Automatically generates a customized evaluation plan based on the context and goals of each case.
• Skill routing: Selects applicable skills on demand from the Skill Library, rather than running all fixed metrics.
• Problem decomposition: Decomposes abstract evaluation problems into measurable, verifiable sub-problems.
• Evidence collection: Calls sub-agents and diagnostic tools to collect visual and logical evidence from different perspectives.
• Evidence verification: The main agent verifies the quality and logical relationships of evidence from each branch to ensure reliable conclusions.
• Evidence Tree output: Outputs a complete evidence tree, recording the test content, tool calls, reasoning chain, and final score.
• Extensible Skill Library: Built in with 9 core skill modules, supporting the dynamic addition of new skills as models evolve.
• Automatic evaluation case construction: Automatically generates the initial world, actions, and verification based on scenario taxonomy to ensure evaluation quality.

Technical principles of HarnessEval

• Four-layer Agentic workflow: HarnessEval adopts a four-layer Plan-Router-Decompose-Verify workflow: the Plan stage understands the case context and generates a customized evaluation plan; the Route stage dynamically selects applicable skills from the Skill Library; the Decompose stage splits high-level problems into sub-problems and delegates them to sub-agents; and the Verify stage has the main agent verify evidence quality and logical relationships to form the final conclusion.
• Layered agent architecture: The system uses a main agent to coordinate planning and verification, with each Skill Agent responsible for specialized evaluation, and each Skill further split into multiple sub-agents to handle specific sub-tasks, forming a hierarchical evaluation execution structure.
• Extensible Skill Library: The framework is built with 9 core skill modules, covering dimensions such as rendering quality, physical plausibility, state change verification, drift analysis, revisit consistency, and off-screen evolution, and supports dynamically adding new skills as model capabilities evolve, enabling continuous expansion of evaluation capabilities.
• Evidence Tree output mechanism: Evaluation results are presented in the form of a hierarchical evidence tree, fully recording the test content, tool calls, visual evidence, and reasoning chain, making evaluation conclusions inspectable, reproducible, and accountable, rather than outputting only a single scalar score.

Follow on WeChat and reply “open source” to join theAI open source project exchange group

How to use HarnessEval

• Clone the repository: Pull the HarnessEval-W codebase from GitHub.
• Create an environment: Create harnesseval-main、harnesseval-metrics and harnesseval-pavrm three conda environments respectively.
• Configure credentials: Copy config/example.env to harnesseval.env and load the environment variables.
• Prepare data: Place the video results generated by the model to be evaluated into the specified directory, and prepare the manifest.json manifest file.
• Run evaluation: Run the harnesseval eval command, specifying the model result path, evaluation plan, and output directory.
• Verify results: Run harnesseval verify run to check whether the evaluation output is complete and whether scores have been generated.
• View report: In the output directory, view summary.json、leaderboard_latest.csv and LEADERBOARD.md to obtain detailed scores and rankings.

Core advantages of HarnessEval

• Dynamic evaluation planning: Automatically generate a customized evaluation plan based on the context of each case, rather than applying a fixed process.
• Intelligent skill routing: Select applicable skills on demand from the Skill Library, avoiding running irrelevant metrics on all cases.
• Explainable evidence chain: Output a complete Evidence Tree, making it possible to trace the tool calls, visual evidence, and reasoning logic behind each score.
• Extensible skill library: The Skill Library supports dynamically adding new skills, continuously expanding evaluation dimensions as model capabilities evolve.
• Hierarchical agent collaboration: The main agent coordinates planning and verification, while sub-agents handle sub-problems by division of labor, enabling fine-grained decomposition of complex evaluation tasks.
• Automatic generation of evaluation data: Automatically construct initial worlds, actions, and verification based on scenario taxonomy, ensuring the quality and diversity of evaluation cases.
• Transparent and auditable evaluation process: Complete reasoning chains and evidence records make evaluation conclusions inspectable, reproducible, and accountable.
• Close to the human evaluation approach: Simulate the evaluation workflow of human experts—understand, plan, gather evidence, reason—rather than mechanical scoring.

Project address of HarnessEval

• Project official website：https://mirros.ai/blog/harnesseval
• GitHub repository：https://github.com/mirros-lab/harnesseval-w

Comparison of HarnessEval with similar competitors

Application scenarios of HarnessEval

• Interactive world model evaluation: Evaluates the state transition correctness, physical plausibility, and world persistence of video generation models under action conditions.
• Embodied intelligence and robot simulation: Evaluates a robot's manipulation capabilities, physical interaction causality, and long-term state consistency in virtual environments.
• Autonomous driving world models: Verifies camera trajectory adherence, object permanence, off-screen evolution, and physical dynamics plausibility in generated scenes.
• Game and virtual world generation: Evaluates scene consistency, NPC behavior causality, revisit consistency, and visual quality in open-world games.
• Physical simulation and scientific computing: Examines adherence to physical laws and causal fidelity in generated videos.
• AI video creation platforms: Provides interpretable quality diagnostics for models such as Seedance, Kling, and Sora, pinpointing specific failure modes (e.g., extra events, object drift, physical violations).