What Is WorkBuddy Bench
WorkBuddy Bench is Tencent's open-source coding agent evaluation suite, covering four real-world scenarios: code, web, office, and security. Tasks are reverse-engineered from real commits, PRs, and business scenarios, and rewritten as colloquial requests to resist data contamination. Its 260 tasks are scored via hidden tests, rule checks, and LLM judgments in isolated sandboxes, with cross-validation under dual frameworks, enabling open-source, reproducible, and auditable end-to-end evaluation.
Main Features of WorkBuddy Bench
- Code: 80 repository-level software engineering tasks, covering 18 categories such as bug fixing, feature development, interface adjustments, and algorithm implementation, with requests initiated by five roles: developers, algorithm engineers, product managers, QA, and operations.
- Web: 70 front-end tasks, covering 7 categories such as page implementation, interaction design, data visualization, visual design, code testing, and document conversion, requiring the delivery of runnable front-end artifacts rather than conversational text.
- Office: 50 multi-file business process tasks, where the agent must handle mixed-format files such as spreadsheets, documents, PDFs, and JSON to complete handoff-type work such as data verification, report generation, and status updates.
- Security: 60 red-team and blue-team security tasks, including vulnerability discovery and exploitation, malware analysis, security operations, and AI agent security assessment, using deterministic scoring scripts with no LLM judge involved.
Technical Principles of WorkBuddy Bench
- Reverse Engineering Construction: Each task is reverse-engineered from real code commits, PRs, CVEs, or business scenarios, and rewritten into short, colloquial role-play requests, making the prompts impossible to recover via search engines.
- Deliberate Information Insufficiency: Requests state only the intent and constraints, omitting details such as target files, exact interfaces, and edge-case handling. The agent must recover context from the workspace itself, testing requirement disambiguation and execution capabilities.
- Post-Turn Evaluation Isolation: During problem solving, the agent can see the task instructions and workspace, while scoring assets (hidden tests, scoring rubrics, etc.) are introduced into the sandbox only after completion, avoiding leakage of scoring information.
- Dual-Framework Cross-Validation: All tasks are run separately under two agent frameworks, CodeBuddy Code and Claude Code, eliminating systematic bias from a single framework.
- Anti-Contamination Mechanism: It relies on the task construction method to close off searchable prompt paths, and combined with dataset version management and regular refreshes, replaces traditional secrecy-based anti-contamination approaches.
Follow on WeChat and reply "Open Source" to join theAI Open-Source Project Communication Group
How to Use WorkBuddy Bench
- Get the Task Set: Visit GitHub to download the standardized task directory, including instruction.md, task.toml, the environment/ environment image, and tests/ scoring assets.
- Run the Evaluation: Connect the agent to a Harbor-style task directory and execute it in an isolated Docker sandbox. The agent reads the natural-language request, explores the workspace, and generates artifacts.
- View Results: After task completion, the scorer runs automatically: the code subset is scored by hidden unit tests, the web subset by a rule + LLM/VLM + agent judge, the office subset by rule checks and an evidence-driven LLM judge, and the security subset by the deterministic scoring.py.
- Independent auditing: Any third party can rerun a single task offline and directly inspect its content, achieving end-to-end reproducibility and auditability.
Core advantages of WorkBuddy Bench
- Realistic work distribution: Task categories, modes, and difficulty all match the internal taxonomy of real usage, and are not constructed from public question banks or tutorial exercises.
- Fully open source and auditable: Task catalogs, environment images, evaluation code, scoring tests, and reference answers are all public, unlike closed vendor benchmarks such as CursorBench.
- Multi-dimensional scoring system: Code uses hidden tests, web uses triple judging by rules + semantics + interaction, office uses rule + evidence-driven LLM judging, and security uses deterministic scripts; the scoring tools for each subset are designed independently.
- Diversity of roles and scenarios: The code subset introduces five requester roles, and the security subset covers the full spectrum of red and blue teams, avoiding evaluation bias caused by a single task type.
- Decoupling of efficiency and capability: The leaderboard reports both token consumption and turn efficiency; GPT-5.5 achieves top-tier scores with the smallest output budget, proving that high scores do not equal high consumption.
Project addresses of WorkBuddy Bench
- Project website：https://workbuddybench.com/
- GitHub repository：https://github.com/Tencent/workbuddy-bench
- HuggingFace model hub：https://huggingface.co/datasets/tencent/workbuddy-bench
- Technical paper：https://workbuddybench.com/report/main.pdf
Comparison of WorkBuddy Bench with similar competitors
| Dimension | WorkBuddy Bench | SWE-bench |
|---|---|---|
| Task source | Reverse-engineered from real commits/PRs/business scenarios | Public GitHub issues |
| Anti-contamination method | Closed search paths at the construction level + version management | Relying on novelty at release time |
| Covered domains | Four domains: Code / Web / Office / Security | Only code defect fixes |
| Degree of open source | Tasks/images/scoring/answers all public | Public task set and tests |
| Request style | Colloquial, role-playing, insufficient information | Detailed technical issue descriptions |
| Evaluation framework | Dual framework (CodeBuddy + Claude Code) | Single framework |
Application scenarios of WorkBuddy Bench
- Benchmark for agent R&D: Provides a standardized capability yardstick for coding agents, front-end AI tools, office automation Agents, and security AI.
- Reference for model selection: The cross-model leaderboard covers mainstream models such as Claude Opus 4.8, GPT-5.5, GLM-5.2, and DeepSeek-V4, assisting enterprise model selection.
- Academic research and comparison: The fully open-source task set and scoring protocol support third-party reproduction, auditing, and improvement, driving iteration of benchmarks in the field.
- Product iteration optimization: Through detailed subset scores and token efficiency data, it identifies an agent's weaknesses in specific areas such as code navigation, front-end state management, and cross-file consistency.
- Security capability evaluation: An independent security subset provides a quantifiable attack-defense testing environment for AI red team and blue team capabilities.