Polar - NVIDIA's Open-Source Agentic Reinforcement Learning Training Framework

What Is Polar

Polar is an open-source agentic reinforcement learning (Agentic RL) training framework released by NVIDIA. Its core innovation is that it can connect existing agent frameworks to RL algorithms such as GRPO for training without modifying their internal code. The framework places a proxy at the LLM API call boundary to capture token-level interaction data and reconstruct training trajectories, enabling Codex CLI、Claude Code, Qwen Code, Pi, and other complex code agent harnesses to directly become trainable RL environments.

Polar's Main Features

• API Proxy Capture: Inserts an API proxy compatible with Anthropic, OpenAI, and Google styles between the agent and the inference server, transparently forwarding requests and recording prompts, sampled tokens, log probabilities, and responses.
• Trajectory Reconstruction: Provides two strategies, per-request and prefix merging, to reconstruct multi-turn model calls into RL trajectories that the trainer can directly consume.
• Asynchronous Serving Architecture: The Rollout Server handles task scheduling and load balancing, while Gateway Nodes handle runtime warm-up, agent execution, trajectory construction, and evaluation, decoupling training from execution.
• Multi-Harness Compatibility: Built-in Claude Code、Codex、Qwen Code、OpenCode、Pi、Gemini CLI and other mainstream code agents for quick adaptation.
• Containerized Runtime: Supports Docker and rootless Apptainer, providing isolated execution environments.

Polar's Technical Principles

• Black-Box Proxy Paradigm: Polar does not rewrite agent harnesses into the env.init()/env.step() interface. Instead, it treats LLM API traffic as the rollout boundary, keeping the harness's native execution logic unchanged.
• Token-Faithful Trajectory Reconstruction: Directly obtains token IDs and log probabilities from the inference backend, avoiding retokenization drift and ensuring that training signals are strictly aligned with the behavior policy.
• Prefix Merging Algorithm: Detects token-prefix relationships among prompts in multi-turn conversations and merges append-only conversation chains into longer training trajectories, reducing the number of trainer updates.
• Asynchronous Staged Execution: Within the Gateway, separates three independent worker pools: INIT (runtime startup), RUN (harness execution), and POSTRUN (trajectory construction and evaluation), and works with a READY buffer to enable runtime warm-up and GPU training in parallel.
• Weight Synchronization Mechanism: Asynchronously synchronizes model weights between the Trainer and the Inference Server. Rollout continues sampling on the old policy, and the trainer performs policy updates after receiving enough trajectories.

How to Use Polar

• Deploy the Polar Service: Start the Rollout Server and Gateway Nodes, and configure the Inference Server (such as SGLang).
• Configure the Harness: Point the model base URL of the target agent (such as Codex CLI) to the Polar Gateway proxy endpoint.
• Write an Adapter: Create a harness adapter (usually only requires configuring environment variables, provider settings, and startup commands).
• Submit a Training Task: Submit a TaskRequest through the Polar API, specifying the harness, runtime, evaluator, and trajectory construction strategy.
• Connect to the Trainer: The training framework (such as Slime, Megatron) receives the trajectory data returned by Polar through callbacks and executes RL algorithm updates such as GRPO.

Polar's Core Advantages

• Zero-Intrusion Integration: No need to modify the source code of existing agent frameworks, lowering the technical barrier to integrating RL training.
• Harness Agnosticism: Compatible with any agent based on LLM APIs, including closed-source binary programs.
• Efficient Resource Utilization: The asynchronous architecture prevents CPU-intensive runtime preparation from blocking GPU training, and prefix merging shortens training time by about 5.39x.
• Token-Level Fidelity: Captures raw tokens directly from the inference backend, avoiding training signal distortion caused by text re-encoding.
• Elastic Scaling: The Rollout-as-a-service design supports large-scale distributed asynchronous RL training.

Polar's Project Address

• GitHub Repository：https://github.com/NVIDIA-NeMo/ProRL-Agent-Server
• arXiv Technical Paper：https://arxiv.org/pdf/2605.24220

Comparison of Polar with Similar Competing Products

Polar's Application Scenarios

• Code Agent Reinforcement Learning: RL fine-tuning of programming assistants such as Codex and Claude Code to improve performance on software engineering benchmarks such as SWE-Bench.
• Multi-Turn Tool-Use Agent Training: Long-horizon agents whose training requires continuous calls to external tools (browsers, databases, APIs).
• Offline SFT Data Generation: Use Polar to batch-generate high-quality training data on a custom harness for supervised fine-tuning.
• Multi-Agent Collaboration Optimization: End-to-end RL training for complex multi-agent systems that include sub-agent orchestration and context compression.
• Closed-Source Agent Evaluation and Improvement: Black-box RL training and capability enhancement for closed-source agent products whose source code is unavailable.