WebCraftBench - An AI Webpage Generation Evaluation Benchmark Jointly Launched by Tencent and Tsinghua University

What is WebCraftBench

WebCraftBench is an AI webpage generation evaluation benchmark jointly launched by Tencent Hunyuan, Tsinghua University, and Peking University. WebCraftBench first automatically instruments the generated application, allowing an agent to actually operate the webpage through Playwright to explore its functionality, using code coverage to guide gap-finding; it then abstracts the interaction records into a state graph, scoring separately across three dimensions: aesthetics, usability, and requirement compliance. The benchmark contains 369 real requirements and 5,088 manually verified acceptance criteria, covering 6,273 applications from 17 frontier models, with an 85.3% consistency rate with human preferences.

WebCraftBench's Main Features

• Real Requirement Evaluation Set: Contains 369 user requirements from real usage traffic, paired with 5,088 manually verified acceptance criteria, covering three types of requirements: functionality, content, and visual.
• Interactive Automatic Exploration: The agent clicks, inputs, and switches pages like a tester through Playwright MCP, proactively discovering functionality and testing boundary cases.
• Code Coverage Guidance: Based on Istanbul automatic instrumentation, it monitors coverage in real time; when insufficient exploration is detected, an auxiliary model suggests untapped code directions.
• State Abstraction and Organization: Deduplicates and normalizes lengthy operation records, merges them into a state transition graph, and distills key runtime evidence for scoring.
• Three-Dimensional Independent Scoring: Separately evaluates aesthetics, usability, and requirement compliance, and synthesizes a standardized composite score.
• Human Preference Validation: Compared against reviewed human votes on an independent validation set, the pairwise judgment consistency rate reaches 85.3%, validating the reliability of the evaluation.

WebCraftBench's Technical Principles

• Automatic Instrumentation to Collect Runtime Evidence: The system injects counters into the application code, supporting frameworks such as static HTML, Vite, Next, CRA, and Astro. After the application starts, each interaction reports code coverage and increments, making "which code has been executed" observable.
• Coverage-Guided Agent Exploration: The exploration agent does not look at the source code or the acceptance checklist, and operates the webpage according to a GUI testing workflow. When three consecutive interactions bring no new coverage, the system asks another model to analyze the not-yet-executed code and converts the suggestions into natural language for the exploration agent to continue trying.
• State Abstraction Compresses Interaction Trajectories It normalizes the page structure, merges identical pages into the same state, and connects operations into a state transition graph. This avoids duplicate screenshots crowding the scoring model's context and makes failure paths clear at a glance.
• Multi-Dimensional Scoring with Exploration and Scoring Separated: During the exploration phase, evidence is collected without looking at the acceptance checklist; only in the scoring phase are criteria checked one by one against retrieved evidence. Aesthetics and usability use a three-round review of "find strengths—pick flaws—judge scoring"; each requirement compliance judgment must cite runtime evidence and is scored by satisfaction ratio. Each of the three dimensions accounts for one third of the composite score.

How to Use WebCraftBench

• Prepare the Generated Application: Have the model under evaluation generate a web application according to the requirements and deploy it in a runnable state.
• Automatic Instrumentation: Run WebCraftBench's instrumentation tool to inject coverage counters into the application code, so that each interaction can report code execution information.
• Interactive Exploration: Launch the exploration agent, operate web pages through Playwright MCP, perform actions such as clicking, typing, and switching pages, proactively discover features and test edge cases.
• Coverage-guided gap detection: When three consecutive interactions produce no new coverage, the system uses an auxiliary model to analyze unexecuted code and generate natural language suggestions for the exploration agent to continue exploring (up to 100 steps).
• Organize runtime evidence: The system automatically abstracts the page structure into states, deduplicates and merges operation records into a state transition graph, and retains screenshots and interaction traces.
• Three-dimensional scoring: The scoring agent separately evaluates aesthetics and usability, and checks requirement compliance item by item against the acceptance criteria; each judgment must cite runtime evidence.
• Output results: Obtain standardized scores in three dimensions and an overall total score, which can be compared with model pool rankings; the larger the score gap, the more credible the result.

Core advantages of WebCraftBench

• Hands-on testing orientation: Have the agent actually operate the web page and score it, rather than only looking at code or screenshots, so it can find issues that static evaluation misses, such as "the entry button is broken, the homepage won't open".
• Coverage-guided exploration: Use code coverage as a "gap-finding clue"; when the testing agent cannot find a feature, the backend model suggests directions, reducing the misclassification of exploration failures as product defects.
• Separation of exploration and scoring: The acceptance checklist is used only in the scoring phase, which preserves requirement constraints while not restricting interactions outside the checklist from being observed, expanding evaluation coverage.
• Independent multi-dimensional scoring: Aesthetics, usability, and requirement compliance are scored separately, preventing a model that is "high in looks but crashes as soon as you use it" from having its weaknesses concealed by a single total score.
• High consistency with human preferences: In 197 reviewed human comparisons, the judgment agreement rate reached 85.3%; when the score gap is large (≥0.75), the agreement rate rises to 98%, indicating high result credibility.
• Strong evaluation stability: After changing the scoring judge model, the ranking correlation coefficient still reached 0.980, and the order of the top four remained unchanged, so the conclusion does not depend on a single judge.

Project address of WebCraftBench

• arXiv technical paper：https://arxiv.org/pdf/2609.15387

Comparison of WebCraftBench with similar competitors

Application scenarios of WebCraftBench

• Model selection for AI programming tools: When a team chooses which large model to use to generate web applications, use WebCraftBench's three-dimensional scores for objective comparison, avoiding decisions based only on demo aesthetics.
• Acceptance testing of AI web page generation products: Integrate WebCraftBench's "hands-on testing process" into the product delivery stage, replacing manual page-by-page verification to automatically discover issues such as invalid entry points and unreachable functions.
• Iterative regression testing of code generation models: After each model upgrade, run the benchmark once, compare changes in aesthetics, usability, and requirement compliance, and quantify whether the new version is a step forward or backward.
• Layered diagnosis of generation quality: Use dimension breakdown and low-coverage clues to locate problems—whether the design is inadequate, the interaction has bugs, or the requirement understanding has gone off track—guiding targeted optimization.