ArenaRL - An Open-Source Open-Domain Comparative Reinforcement Learning Method by Tongyi and Amap

What is ArenaRL

ArenaRL is an open-source comparative reinforcement learning method for open-domain agents, developed jointly by the Tongyi DeepResearch team and Amap. By introducing a tournament mechanism, ArenaRL transforms traditional absolute scoring into intra-group relative ranking, and uses an efficient seeded single-elimination tournament topology to keep computational complexity at a linear level, effectively solving the discrimination collapse problem caused by the lack of standard answers in open-domain tasks. ArenaRL performs excellently in academic benchmarks and has completed deployment validation in real business scenarios of Amap, significantly improving the planning and execution capabilities for complex tasks.

Main Features of ArenaRL

• Optimizing performance on open-domain tasks: Through comparative reinforcement learning, ArenaRL addresses the bottleneck of traditional reinforcement learning in open-domain tasks caused by the lack of standard answers, helping agents find better solutions in complex tasks.
• Improving training efficiency: ArenaRL adopts a seeded single-elimination tournament mechanism, keeping computational complexity at aO(N) linear level, achieving the best balance between training efficiency and effectiveness.
• Enhancing reasoning and planning capabilities: By introducing a process-aware evaluation mechanism, it can evaluate the final result and examine the logical rigor of the chain of thought and the precision of tool calls, improving the agent's reasoning and planning capabilities.
• Supporting diverse application scenarios: It performs excellently in multiple scenarios such as complex travel planning, deep information retrieval, and general writing, demonstrating strong task generalization ability.

Technical Principles of ArenaRL

• From absolute scoring to relative ranking: Traditional reinforcement learning relies on a reward model to assign an absolute scalar score to each generated trajectory, which is prone to discrimination collapse in open-domain tasks. ArenaRL innovatively introduces a relative ranking mechanism, reconstructing reward modeling as an intra-group relative ranking problem through pairwise comparison, avoiding the limitations of absolute scoring.
• Tournament mechanism and seeded single-elimination tournament: ArenaRL adopts a tournament mechanism, having the agent generate a set of candidate solutions for the same instruction to build a mini "arena." Through the topology of a seeded single-elimination tournament, ArenaRL strictly keeps computational complexity at a O(N) linear level while achieving an advantage estimation accuracy that closely approaches that of a full round-robin tournament, achieving the best balance between training efficiency and effectiveness.
• Process-aware evaluation mechanism: ArenaRL can evaluate the quality of the final result and deeply examine the logical rigor of the chain of thought (CoT) as well as the precision of tool calls. This process-aware evaluation mechanism helps agents better weigh multidimensional constraints in complex tasks, improving reasoning and planning capabilities.
• Bidirectional scoring protocol: To eliminate positional bias when a large model serves as a judge, ArenaRL adopts a bidirectional scoring protocol. In each comparison, the system swaps the order of the two candidate solutions for scoring, ensuring the fairness and fine granularity of evaluation results.
• Open-source data and training framework: ArenaRL has open-sourced a complete training framework and full-process evaluation benchmark, including two major benchmark datasets, Open-Travel and Open-DeepResearch. It provides developers with rich training and testing data, lowers the barrier to use, and promotes research and application of open-domain agents.

ArenaRL Project Address

• Project Official Website：https://tongyi-agent.github.io/zh/blog/arenarl/
• GitHub Repository：https://github.com/Alibaba-NLP/qqr
• HuggingFace Model Library：https://huggingface.co/papers/2601.06487
• arXiv technical paper：https://arxiv.org/pdf/2601.06487

Application scenarios of ArenaRL

• Complex travel planning: ArenaRL can generate multiple candidate routes for users and, through relative comparison, select the optimal route that best matches vague requirements (such as fewer people, shade, suitable for pushing a baby stroller, etc.).
• Long-text generation and information retrieval: In long-text generation tasks, ArenaRL can significantly improve the instruction-following ability and usability of generated content, avoiding quality degradation caused by length bias.
• Multi-tool collaborative tasks: ArenaRL evaluates the logic of the chain of thought and the accuracy of tool calls to ensure efficient completion of complex instructions in multi-tool collaborative tasks and avoid unreasonable calls.
• Personalized recommendation: In personalized recommendation scenarios, ArenaRL selects from multiple candidate options the one that best matches the user's vague and personalized needs (such as suitable for a date, with a river-view terrace).
• Open-domain question answering: In open-domain question answering, ArenaRL uses relative comparison to select the most reasonable and useful answer from multiple candidate answers, improving response quality.