AReaL 2.0 - Ant Group and others' open-source Agent online reinforcement learning infrastructure

What is AReaL 2.0

AReaL 2.0 is an Agent online reinforcement learning infrastructure open-sourced by Ant Group in collaboration with The Hong Kong University of Science and Technology and Tsinghua University, enabling deployed Agents to continuously self-evolve from real interaction trajectories. Through three pillars—the trajectory data protocol, the enterprise-grade data proxy, and the evolution control plane—AReaL 2.0 transforms session streams in production environments into trainable experience, without rewriting the Agent or tearing down business systems. It has joined the PyTorch Foundation and supports Ascend NPU adaptation.

Main features of AReaL 2.0

• Online RL closed loop: Automatically transforms real Agent interaction trajectories into trainable experience, realizing a closed loop from execution to learning.
• Trajectory data collection: Records the complete decision-making process on a per-step basis through the ATDP protocol, including observations, actions, results, and metadata.
• Enterprise-grade data governance: Data Proxy is responsible for interception, desensitization, permission control, and trajectory persistence, ensuring compliance processing is completed before training.
• Evolution control plane: Based on multi-dimensional signals such as trajectory statistics, failure clusters, and user correction rates, determines repair entry points such as model updates, memory writes, or tool adjustments.
• Microservices architecture: Splits training, inference, and weight updates into independently deployable components such as Gateway, Router, and Data Proxy.

Technical principles of AReaL 2.0

• Agent Trajectory Data Protocol (ATDP): A learning-oriented agent trajectory protocol that splits complex tasks into accountable, replayable learning samples. It records the Agent's observations, internal states, action selections, action results, reward feedback, as well as metadata such as model version, tool version, tenant, and cost on a per-step basis, answering key questions such as "which retrieval or which tool call affected the success or failure of the task."
• Agentic Data Proxy: A learning data layer deployed at key boundaries such as models, tools, retrieval systems, and human feedback channels. It is responsible for interception, collection, desensitization, permission control, trajectory persistence, reward collection, and replay management. Governance is completed before data enters the training queue, distinguishing visible fields, training-eligible data, and data used only for debugging and auditing.
• Agent Evolution Control Plane: The agent evolution control plane, turning "whether to update and where to update" into a governable systemic decision. Based on trajectory statistics, user correction rates, tool failure clusters, evaluator scores, cost signals, safety constraints, and distribution drift, it determines whether evolution should occur at the level of model RL, preference optimization, memory writes, or tool routing adjustments, and supports replay evaluation, regression testing, canary release, and versioned tracking.
• Online RL microservice runtime: Transforms the original RL infrastructure into an online system that handles Agent service traffic. Gateway serves as the entry point for requests; Router maintains the binding relationship between sessions and Data Proxy, ensuring continuity of multi-turn interaction context; Data Proxy manages session state and trajectories; Agent-Compute Worker performs inference, sampling, and training computation; Controller is responsible for scheduling, autoscaling, and health checks.

Follow WeChat and reply "open source" to join theAI open-source project community group

How to use AReaL 2.0

• Visit the project homepage: Open the GitHub repository https://github.com/areal-project/AReaL to get the source code and documentation.
• Install the infrastructure: Deploy microservice components of AReaL 2.0 such as Gateway, Router, Data Proxy, and Controller.
• Switch the inference entry point: Point the LLM API call backend (base_url/api_key) of the existing Agent to the Gateway managed by AReaL 2.0.
• Configure trajectory protocol: Enable ATDP to record the Agent's observations, actions, tool calls and results on a step-by-step basis.
• Deploy data proxy: Configure the Data Proxy to complete data interception, desensitization, access control and trajectory persistence.
• Start the training loop: Import the collected real trajectories into the training service, start online RL, and let the Agent continuously learn from interactions.

Core advantages of AReaL 2.0

• Low-intrusion integration: Only switch the LLM inference backend; no need to rewrite the Agent's planning, tool calling, sandbox or memory modules.
• Real trajectory driven: Train directly with real online interaction data, bridging the gap between offline simulation environments and actual behavior.
• Enterprise-grade security governance: Data desensitization, access control and compliance boundaries are completed before training, adapting to multi-tenant and multi-business-line scenarios.
• Interpretable and rollback-capable: Each evolution undergoes replay evaluation, regression testing and versioned tracking, and can be quickly rolled back if problems occur.
• Rich open-source ecosystem: Has joined the PyTorch Foundation, and the community provides Ascend NPU adaptation and LoRA low-compute solutions.

Project address of AReaL 2.0

• Project official website：https://areal-ai.io/en/intro.html
• GitHub repository：https://github.com/areal-project/AReaL
• arXiv technical paper：https://arxiv.org/pdf/2607.01120

Comparison of AReaL 2.0 with similar competing products

Application scenarios of AReaL 2.0