What is Vision2Web
Vision2Web is jointly launched by Tsinghua University and Zhipu AIVisual Website Development Evaluation Benchmark, used to evaluate the end-to-end website-building capabilities of multimodal AI Agents. Vision2Web contains 193 real website tasks, divided into three progressive difficulty tiers: static webpage generation, interactive front-end development, and full-stack website construction. The benchmark adopts an innovative "workflow-based Agent verification" paradigm, combining GUI Agent verification of functional correctness and VLM judgment of visual fidelity, systematically revealing the capability boundaries of current SOTA models in complex long-horizon software development tasks.
Main Features of Vision2Web
- Three-tier progressive capability evaluation: From static webpage generation to interactive front-end development, and then to full-stack website construction, progressively testing the visual understanding and engineering implementation capabilities of AI Agents.
- Dual-dimension automated verification: Verifying functional correctness through GUI Agent, combined with VLM Judge evaluating visual fidelity, achieving objective and reproducible end-to-end testing.
- Support from real-world scenario data: Built on a dataset of 193 real websites, covering four major categories and 16 subcategories, providing 918 prototype images and 1,255 test cases.
- Systematic capability diagnosis: Precisely locating the capability boundaries and failure causes of Agents in cross-modal reasoning, long-horizon planning, complex system construction, and other stages.
How to Use Vision2Web
- Environment preparation: Configure a containerized runtime environment including front-end, back-end, and database dependencies.
- Data loading: Place prototype images, requirement documents, and multimedia resources into the specified working directory.
- Agent integration: Integrate the model under test into Agent frameworks such as OpenHands or Claude Code.
- Task execution: The Agent reads the input and generates code, ultimately outputting a deployable website project.
- Automatic deployment: Run the startup script generated by the Agent to complete website deployment on a fixed port.
- Functional verification: GUI Agent Verifier executes expert-designed test workflows, evaluates interaction correctness, and outputs a Functional Score.
- Visual verification: VLM Judge compares the rendering results with the prototype images, evaluates visual fidelity, and outputs a Visual Score.
- Result analysis: Combine the dual-dimension scores to locate the model's capability weaknesses in specific development stages.
Key Information and Usage Requirements of Vision2Web
- Research background: Jointly launched by Tsinghua University and Zhipu AI, aiming to systematically evaluate the visual website development capabilities of multimodal AI Agents.
- Task scale: Contains 193 real website tasks, divided into three progressive difficulty tiers: static webpage generation, interactive front-end development, and full-stack website construction.
- Data composition: Provides 918 multi-device prototype images and 1,255 test cases, covering four major fields—content, commerce, SaaS, and public services—with a total of 16 subcategories.
- Verification mechanism: Adopts a dual-dimension automated verification paradigm in which GUI Agent verifies functional correctness (Functional Score) and VLM Judge evaluates visual fidelity (Visual Score).
- Environment requirements: A containerized runtime environment must be configured, with complete dependencies such as front-end frameworks, back-end services, and databases pre-installed.
- Framework integration: The model under test needs to be integrated into standardized Agent frameworks such as OpenHands or Claude Code.
Core Advantages of Vision2Web
- Layered and Progressive Task Design: A three-tier architecture from static web pages to full-stack development, which can precisely identify the capability boundaries of Agents at different development stages.
- Data Derived from Real Scenarios: Built from real websites in the C4 validation set, avoiding synthetic data bias and ensuring that evaluation results are close to real-world applications.
- Objective and Reliable Verification Mechanism: The two components of GUI Agent and VLM Judge work together to achieve reproducible automated evaluation of functional correctness and visual fidelity.
- Comprehensive and Diverse Coverage Dimensions: Covers 16 subcategories across four major domains, supporting systematic comparative analysis across models, frameworks, and devices.
- Fine-Grained and In-Depth Diagnostic Capability: Through workflow-style verification nodes, it can clearly trace the specific reasons for Agent failures in areas such as UI understanding, interaction logic, and long-horizon planning.
Project Address of Vision2Web
- arXiv Technical Paper：https://arxiv.org/pdf/2603.26648
Comparison of Vision2Web with Similar Competing Products
| Comparison Dimension | Vision2Web | Design2Code | WebGen Bench |
|---|---|---|---|
| Task Scope | Three progressive tiers: static web pages, interactive frontend, full-stack development | Only static UI to code generation | Text-driven end-to-end development, with insufficient multimodal support |
| Verification Method | Two-component workflow verification with GUI Agent + VLM Judge | Pixel-level comparison and rule scripts | Open-ended text evaluation, lacking structured constraints |
| Reproducibility | High: structured workflow clearly constrains the execution path | Low: layout changes make scripts fragile | Low: open-ended evaluation is difficult to reproduce |
| Data Scale | 193 tasks / 918 prototype images / 1255 test cases / 16 subcategories | Limited scale, with few categories covered | No clear layering, and the number of tasks is relatively small |
| Multi-Device Support | Supports adaptation evaluation for desktop / tablet / mobile | Does not clearly distinguish multi-device scenarios | Does not emphasize multi-device adaptation |
| Capability Diagnosis | Fine-grained: can identify shortcomings in UI understanding / interaction logic / system construction | Coarse: only visual similarity scoring | Coarse: end-to-end overall scoring, difficult to break down |
Application Scenarios of Vision2Web
- Model Capability Evaluation: Provides standardized benchmark testing of visual website development capabilities for multimodal large models such as Claude, Gemini, and GPT.
- Agent Framework Optimization: Compares the performance of different Agent frameworks such as OpenHands and Claude Code, guiding iterative framework upgrades.
- Algorithm R&D Validation: Evaluates the innovative effects of new models on key technologies such as UI understanding, code generation, and long-horizon planning.
- Product Capability Benchmarking: Helps AI website-building products (such as Vibe Coding tools) quantify their own level and clarify the gap with SOTA.
- Education and Training Reference: Serves as a teaching case library for cultivating engineers and researchers in the direction of AI-assisted development.