One-Eval - Peking University's Open-Source Automated Large Model Evaluation Framework

What is One-Eval

One-Eval is an open-source automated Agent-driven large model evaluation framework from the OpenDCAI team at Peking University, built on DataFlow and LangGraph . It focuses on NL2Eval (natural language to evaluation) capabilities. Users only need to describe the evaluation goal in natural language, and the system can automatically complete benchmark recommendation, data download, inference execution, metric matching, scoring, and multi-dimensional report generation, achieving full-chain automation from requirements to reports.

One-Eval's Main Features

• NL2Eval Intelligent Parsing: Users input natural language requirements, and the Agent automatically parses the intent and plans the evaluation path.
• Bench Gallery Benchmark Library: Built-in meta-information management for mainstream evaluation benchmarks such as GSM8K, MATH, MMLU, C-Eval, IFEval, and HumanEval.
• End-to-End Automated Execution: Automatically completes data download, model inference, answer scoring, statistical analysis, and report generation.
• Human-in-the-Loop Intervention: Supports interruption, review, editing, and rerunning at key nodes, allowing real-time adjustment of the evaluation strategy.
• Unified Interface for Heterogeneous Data: Standardizes the format and column mapping of different datasets through the DataFlow engine.

One-Eval's Technical Principles

• Agent Graph Orchestration Architecture: Builds a state machine workflow based on LangGraph, breaking down evaluation into three major stages: NL2Bench → BenchResolve → Metrics & Reporting.
• DataFlow Operator System: Uses the DataFlow engine at the底层 to handle data preparation and streaming computation, achieving unified access and transformation of heterogeneous datasets.
• Local-first + HuggingFace Dual-Source Parsing: Prioritizes loading local preset configurations, and when missing, automatically calls HuggingFace tools to search, download, and structure dataset meta-information.
• Traceable State Management: Persists the execution state of each step, supporting checkpoint recovery, backtracking and rerunning, and failure data analysis.

How to Use One-Eval

• Environment Preparation: Create a virtual environment via Conda or uv and execute pip install -e . to install dependencies.
• Start the Backend: Run uvicorn one_eval.server.app:app --host 0.0.0.0 --port 8000 to start the FastAPI service.
• Start the Frontend: Enter the one-eval-web directory and execute npm install && npm run dev, then visit localhost:5173。
• Configure Parameters: Set the API Key, target model, and HuggingFace Token in the Web interface. After saving, interactive evaluation can begin.
• Initiate Evaluation: Describe requirements in natural language in the input box, such as "evaluate the model's performance on mathematical reasoning tasks". The Agent automatically executes and generates a report.
• Developer Mode: Directly run python -m one_eval.graph.workflow_all "评测目标" to trigger the complete workflow via the command line.

One-Eval's Key Information and Usage Requirements

• Development Team: OpenDCAI Lab at Peking University.
• Open-Source License: Fully open source, with code hosted on GitHub and directly accessible.
• Technology Stack: The backend is based on Python 3.11 and FastAPI, the frontend uses React + Vite, the core workflow relies on LangGraph for graph orchestration, and the underlying data processing is driven by the DataFlow operator system.
• Supported Models: Compatible with any provider that offers the OpenAI standard interface, including OpenRouter, SiliconFlow, Volcano Engine, and self-deployed vLLM, etc.
• Hardware Requirements: Running locally only requires a regular development machine configuration; the computation and token consumption during the actual inference phase depend entirely on the external model API selected by the user.
• Usage Threshold: Before use, a valid API Key must be configured for model calls, and a HuggingFace Token is also required to download and load some evaluation datasets.

Core Advantages of One-Eval

• Zero-Script Evaluation: Say goodbye to manually downloading data and writing configurations; a full-chain evaluation can be initiated in natural language.
• Extensible Architecture: Based on DataFlow operators and LangGraph state management, private datasets and custom metrics can be easily integrated.
• Human-in-the-Loop Closure: Key nodes support human review and intervention, balancing automation and controllability.
• Multi-Dimensional Reports: Automatically generates score statistics, model rankings, failure case analysis, and actionable recommendations.
• Full-Chain Tracing: The evaluation process is fully traceable, and the input, output, and state changes of each node can be audited.

One-Eval Project Address

• GitHub Repository：https://github.com/OpenDCAI/One-Eval
• arXiv Technical Paper：https://arxiv.org/pdf/2603.09821

Comparison of One-Eval with Similar Competing Products

Comparison Dimension	One-Eval	OpenCompass	EleutherAI LM Harness
Development Team	Peking University OpenDCAI Lab	Shanghai AI Laboratory	EleutherAI Community
Core Positioning	Agent-driven, NL2Eval automated evaluation	Mainstream in the Chinese community, configuration-driven evaluation	Long-established overseas, script-based evaluation tool
Usage Method	Natural language description + Web UI interaction + code invocation	YAML configuration file + command-line scripts	Python scripts + CLI command line
Onboarding Threshold	Low; an evaluation can be initiated with one sentence	Medium; configuration files need to be written	Medium; code and scripts need to be written
Human-Machine Collaboration	Supported; key nodes can be interrupted, reviewed, and intervened in	Not supported; purely automated execution	Not supported; purely automated execution
Frontend Interface	Built-in React + Vite visual workflow	Has a results display page, no interactive frontend	No frontend interface
Chinese Ecosystem	Native support for C-Eval, CMMLU, etc.	Very strong; comprehensive coverage of Chinese benchmarks	Relatively weak; Chinese datasets need to be configured manually
Heterogeneous Data Support	Unified access via the DataFlow operator system	Relatively complete	Relatively complete, but configuration complexity is high
Agent/Sandbox Evaluation	Planned (future support for SWE-bench, etc.)	Partially supported	Not supported
Applicable Scenarios	Rapid selection, engineering acceptance, academic experiments	Deep customization, large-scale batch evaluation	English benchmark research, code-level flexible customization

One-Eval Use Cases

• Preliminary model selection screening: Quickly compare the performance of multiple candidate models across dimensions such as mathematics, reasoning, code, and instruction following.
• Private deployment model acceptance: Conduct standardized capability acceptance and regression testing on self-deployed or fine-tuned models.
• Benchmark research: Quickly search and configure evaluation sets suitable for specific task types through the Bench Gallery.
• Academic research: Provide a reproducible and traceable automated evaluation pipeline for paper experiments.
• Agent capability evaluation: Evaluate the performance of LLMs in tool calling, planning, and complex task execution (future support for scenarios such as SWE-bench).