AReaL is a large-scale asynchronous reinforcement learning system, positioned as RL bridge infrastructure connecting foundation model training with agent applications. It was initially developed by researchers and engineers from the Institute for Interdisciplinary Information Sciences (IIIS) at Tsinghua University and the AReaL team at Ant Group. Based on a fully asynchronous RL training paradigm, it is optimized for efficiency and scalability in large-scale reasoning and agent model training. Its mission is to enable a broad range of developers and researchers to build AI agents more easily, efficiently, and at lower cost.
In terms of main capabilities, AReaL supports agentic reinforcement learning (Agentic RL) and online RL training. For black-box agent applications, it can be seamlessly customized by simply replacing base_url; fully asynchronous RL training is stable and industry-leading in speed; and it achieves advanced performance on agents in scenarios such as math, coding, search, and customer service. The project is continuously iterated, and multiple versions have been released: v0.1 provides reproducible results for 1.5B and 7B large reasoning models; v0.2 (boba) introduces SGLang support and releases advanced 7B and 32B models for mathematical reasoning; v0.3 (boba²) implements fully asynchronous RL training, achieving a 2.77x speedup compared with synchronous systems while training performance is comparable or better; AReaL-lite is a lightweight version for AI researchers and rapid prototyping, adopting an algorithm-first API design, with 80% less code while maintaining 90% of the performance and core functionality; AReaL 2.0 is a major architectural milestone, refactored into a microservices architecture, including independent training, inference, agent, and weight update services, and releasing the Hermes online RL loop and an end-to-end SWE RL training example. Other recent updates include: KPop (bidirectional binary KL divergence token masking, configured via rejection_sampling.metric=binary_kl) and IcePop (token masking based on importance ratios); integration with Scaffoldings for agentic RL training, decoupling agent execution, reward computation, and trajectory acquisition; support for stable training on Ascend NPU devices; a complete example for training the OpenClaw agent (just replace base_url and api_key); the AReaL-SEA self-evolving data synthesis engine; the ASearcher search agent; and community biweekly meetings, etc.
In terms of usage, one can start by installing the package, and the training script will automatically download the required datasets(openai/gsm8k) and the model (Qwen/Qwen2-1.5B-Instruct). It supports both single-node execution and execution on a Ray cluster (the paths in the YAML file need to be updated to point to shared storage). All RL algorithms can be switched to asynchronous or synchronous versions by setting max_head_offpolicyness=0. AReaL supports multiple training backends and inference backends, and provides a CLI guide, large-scale MoE, and Megatron/PyTorch Archon support. The documentation covers code walkthroughs, running GRPO on GSM8K, improving algorithm performance, best practices for agent workflows, OOM handling, performance profiling, custom datasets, custom agent/RVLR rollout workflows, explanation of asynchronous RL, PPO/GRPO and related algorithms, CLI configuration, metric tracking, allocation modes, rollout workflows, and AI-assisted development, among other content.
Applicable scenarios include: mathematics and reasoning, vision-language models, alignment and infrastructure, programming, search, customer service, and other agent training; rapid prototyping for researchers (AReaL-lite); large-scale inference and agent model training.
Regarding limitations, the official materials do not explicitly state specific limitations, but the project depends on specific training and inference backends, and some features require a Ray cluster or specific hardware (such as Ascend NPU) support. For specific installation commands, code content, and complete configuration parameters, please refer to the official documentation and repository.
