EdgeBench - ByteDance's AI learning capability benchmark framework

What is EdgeBench

EdgeBench is a benchmark framework launched by ByteDance's Seed team, used to evaluate the long-term learning capability of autonomous AI Agents in real-world environments. The framework includes 134 real tasks, covering 6 major domains including scientific computing, software engineering, and combinatorial optimization. Through continuous operation for 12-72 hours, it tracks the Agent's attempt-observe-absorb-improve learning curve, revealing the predictable scaling laws of AI learning from the environment.

Main features of EdgeBench

• Long-term learning evaluation: Simulates an Agent running continuously in a real environment for 12-72 hours, tracking the complete learning curve.
• Multi-domain task coverage: Covers 134 real-world tasks, spanning 6 major categories including science, engineering, and optimization.
• Anti-contamination design: 51 tasks are public, and 83 are held out to prevent benchmark contamination and overfitting.
• Quantification of learning laws: Finds that Agent performance follows a log-sigmoid scaling law, with learning speed roughly doubling every 3 months.
• Human baseline comparison: Provides a human expert completion baseline of an average of 57.2 hours, quantifying the human-machine gap.

Technical principles of EdgeBench

• Environment interaction learning loop: EdgeBench builds a closed-loop evaluation framework of attempt-observe-absorb-improve. The Agent executes actions in a real task environment, receives environment feedback, updates its strategy, and tries again, simulating the gradual learning process of humans in complex tasks.
• Time-segmented performance tracking: Divides long-running operation into multiple stages, continuously recording the Agent's performance scores at each time point, forming quantifiable learning curve data that supports fine-grained analysis of long-term learning dynamics.
• Cross-domain task modeling: Designs a unified evaluation protocol for 6 types of tasks with different cognitive difficulty, from scientific computing to formal mathematics, ensuring that the evaluation framework can cover diverse real-world challenges.

Follow WeChat and reply "open source" to joinAI open-source project exchange group

How to use EdgeBench

• Visit the repository: Go to GitHub and search ByteDance-Seed/EdgeBench to get the leaderboard and public task list.
• Select a task: Choose a target domain task from the 51 public tasks, and learn about its evaluation metrics and environment configuration.
• Deploy the Agent: Connect the AI Agent to be tested to the task environment, and configure a continuous runtime of 12 hours or more.
• Collect data: Record the Agent's performance scores at each time stage, and generate learning curve data.
• Submit for evaluation: Compare and analyze the results with models such as Claude Opus 4.8 and GPT-5.5 on the leaderboard.

Core advantages of EdgeBench

• Real-environment orientation: Based on real-world tasks rather than static question answering, evaluating the Agent's actual working capability.
• Long-term dynamic tracking: Breaks through the limitations of single-inference evaluation, capturing the Agent's continuous improvement trajectory during long-term operation.
• Predictable laws: Finds that the AI learning curve follows a highly predictable log-sigmoid scaling relationship (R² = 0.998).
• Anti-contamination mechanism: 83 held-out tasks effectively prevent models from over-optimizing for the benchmark.
• Frontier model coverage: Top models such as Claude Opus 4.8, GPT-5.5, Gemini 2.5 Pro, o3, and DeepSeek-V4-Pro have been evaluated.

EdgeBench Project Address

• Project Website：https://edge-bench.org/
• GitHub Repository：https://github.com/ByteDance-Seed/EdgeBench
• HuggingFace Model Hub：https://huggingface.co/datasets/ByteDance-Seed/EdgeBench
• Technical Paper：https://edge-bench.org/paper.pdf

Comparison of EdgeBench with Similar Competing Products

Application Scenarios of EdgeBench

• General Intelligence Research: Provides quantitative evaluation criteria for long-term learning capability for general intelligence initiatives such as "Seed Edge".
• Agent Capability Iteration: Helps developers identify Agent bottlenecks in long-duration tasks and guides the direction of model optimization.
• Model Selection Reference: Compares the long-term learning performance of models such as Claude, GPT, and Gemini across domains through leaderboards.
• Human-Machine Capability Benchmarking: Uses the baseline of 57.2 hours for expert humans as a reference to measure progress of AI approaching human-level performance.
• Educational Training Design: Provides a standardized evaluation environment for research on AI autonomous learning and continuous improvement algorithms.