Relax - Xiaohongshu's Open-Source Large Model Reinforcement Learning Training Engine

What is Relax

Relax is a large model reinforcement learning training engine open-sourced by the Xiaohongshu AI Platform team, designed for all-modality and Agentic scenarios. Built on Megatron-LM and SGLang, it adopts a service-oriented fault-tolerant architecture (Ray Serve) and a TransferQueue asynchronous data bus to achieve complete decoupling of the Rollout, Actor, and Critic roles. Relax supports unified training for text, images, audio, and video, and has verified stable convergence of RL across four modalities on Qwen3-Omni-30B. The 16×H800 fully asynchronous mode is 76% faster than Colocate and 20% faster than veRL, features minute-level fault recovery, elastic scaling, and R3 MoE support capabilities, and has been applied to Qwen3 series post-training.

Main Features of Relax

• All-Modality RL Training: Unified support for end-to-end reinforcement learning post-training of text, images, audio, and video, with verified stable convergence on the Qwen3-Omni series.
• Asynchronous Training Architecture: Based on the TransferQueue data bus, it achieves fully asynchronous parallelism of the Rollout, Actor, and Critic roles, eliminating idle GPU waiting.
• Service-Oriented Fault Tolerance: Each RL role is deployed as an independent Ray Serve service, with fault isolation and minute-level automatic recovery capabilities, supporting uninterrupted training.
• Elastic Scaling: Dynamically add or remove Rollout inference engines via HTTP REST API, supporting inference resource scheduling within the same cluster and across federated clusters.
• Distributed Weight Synchronization: The DCS service implements low-latency weight broadcasting over dual channels NCCL/TCP, requiring no fallback to disk Checkpoint for fault recovery.
• Agentic Multi-Turn Training: Native support for multi-turn interaction, tool calling, visual context carry-over, and Loss Masking, adapting to the "execute→observe→decide" closed loop.
• Algorithm Suite Support: Built-in GRPO, GSPO, SAPO, and On-Policy Distillation algorithms, providing rule-based rewards and GenRM generative evaluation.

Technical Principles of Relax

• Service-Oriented Asynchronous Architecture: Encapsulates roles such as Actor, Critic, and Rollout as independent Ray Serve services, performs streaming micro-batch transmission through TransferQueue, achieves decoupling of computing resources and pipeline parallelism, and eliminates the serial waiting bottleneck of the traditional Colocate mode.
• All-Modality Data Pipeline: Built-in Omni Processor uniformly processes image, text, audio, and video data, supports modality-aware parallelism and end-to-end asynchronous pipelines, and through the Field-Level storage mechanism allows different fields of the same sample to be read and written independently, adapting to the multi-stage computing characteristics of RL.
• Fault Tolerance and Elasticity Mechanisms: Adopts a two-level recovery strategy (in-place restart for stateless roles, global recovery for stateful roles), and together with the DCS distributed Checkpoint service achieves direct GPU-GPU weight synchronization, avoiding disk IO overhead, and supports dual-channel NCCL/TCP adaptation to cross-cluster topologies.
• Asynchronous Consistency Control: Flexibly controls On/Off-Policy modes through the max_staleness parameter, StreamingDataLoader supports incremental consumption, and DCS asynchronous weight broadcasting overlaps with training computation, achieving a balance between throughput and policy freshness.

How to Use Relax

• Environment Deployment: Pull the official Docker image (pre-installed with CUDA, PyTorch, Megatron-LM, SGLang, Ray):docker pull relaxrl/relax:latest, and mount the GPU and working directory when starting the container.
• Data and Model Preparation: Use hf download Download the datasets (e.g., dapo-math-17k, multimodal-open-r1-8k-verified) and base models (Qwen3-4B, Qwen3-VL-4B, Qwen3-Omni-30B-A3B) to a local directory.
• Start training: Set the EXP_DIR environment variable to point to the data root directory, and execute the bash script for the corresponding modality:
  • Text tasks (8 GPUs)：bash scripts/training/text/run-qwen3-4B-8xgpu.sh
  • Vision-language tasks (8 GPUs)：bash scripts/training/multimodal/run-qwen3-vl-4B-8xgpu.sh
  • Omni-modal tasks (16 GPUs / dual-node)：bash scripts/entrypoint/spmd-multinode.sh scripts/training/multimodal/run-qwen3-30B-A3B-omni-16xgpu.sh
• Model export: After training is complete, use scripts/tools/convert_torch_dist_to_hf_bridge.py to convert Megatron DCP format weights to HuggingFace format.

Relax's key information and usage requirements

• Open-source entity: Officially open-sourced by Xiaohongshu's AI Platform Team (RedAI Infra) on April 15, 2026.
• Product positioning: A high-performance asynchronous reinforcement learning post-training engine for omni-modal and Agentic scenarios.
• Technical foundation: Built on the Megatron-LM training framework and the SGLang inference engine, adopting a Ray Serve service-oriented architecture.
• Core architecture: A six-layer service-oriented design (Entrypoints→Orchestration→Components→Engine→Backends→Distributed), with TransferQueue implementing an asynchronous data bus and DCS implementing distributed weight synchronization.
• Performance metrics: In 16×H800 fully asynchronous mode, end-to-end speed is 76% faster than the Colocate baseline and 20% faster than veRL; the R3 overhead for MoE models is only 1.9% (versus 32% for veRL).
• Modality support: Native support for unified training across text, images, audio, and video; Qwen3-Omni-30B has been verified to stably converge for 2000+ steps across four modalities.
• Algorithm coverage: Built-in GRPO, GSPO, SAPO, and On-Policy Distillation, supporting rule-based rewards and GenRM generative evaluation.
• Hardware configuration: A single node requires at least 8×H800 (4B model); omni-modal Omni training requires 16×H800 across 2 nodes, with support for elastic scaling and dynamic adjustment of Rollout resources.

Relax's core advantages

• Native omni-modal support: Built-in unified processing pipeline for text, images, audio, and video; it is one of the few open-source frameworks supporting end-to-end post-training for Qwen3-Omni, with verified stable convergence over 2000+ steps.
• Extreme asynchronous performance: TransferQueue fully decouples Rollout/Actor/Critic; 16×H800 fully asynchronous mode is 76% faster than Colocate and 20% faster than veRL.
• Service-oriented fault-tolerant architecture: Independent fault domain design based on Ray Serve, supporting minute-level automatic recovery and elastic scaling; DCS enables second-level weight synchronization without falling back to disk.
• Agentic-native design: Built-in multi-turn interaction, tool calling, visual context carry-over, and loss masking, adapted for "execute→observe→decide" closed-loop training.
• Efficient MoE support: R3 (Rollout Routing Replay) overhead is only 1.9%, achieving near-zero-overhead replay compared with veRL's 34% performance loss.
• Production-grade operations: HealthManager automatic monitoring, Metrics integration with multiple platforms (WandB/TensorBoard), Apprise real-time alerts, and support for dynamic scaling via HTTP API.

Relax's project address

• GitHub repository：https://github.com/redai-infra/Relax
• arXiv technical paper：https://arxiv.org/pdf/2604.11554

Comparison of Relax with similar competing products

Relax Application Scenarios

• Full-modality large model post-training: Unified optimization to support Omni models with text, image, audio, and video inputs (such as Qwen3-Omni), enabling reinforcement of cross-modal understanding and generation capabilities.
• Agentic intelligent agent development: Train intelligent agents with multi-turn tool calling, environment interaction, and autonomous decision-making capabilities, suitable for "execute→observe→decide" closed-loop tasks (such as DeepEyes multi-turn visual reasoning).
• Vision-language task optimization: Perform RL post-training for tasks such as visual question answering (VQA), image captioning, and video understanding (NextQA) to improve multimodal reasoning accuracy.
• Mathematical and Code Reasoning: Enhance large models' capabilities in mathematical problem solving (DAPO-Math), logical reasoning, and code generation through algorithms such as GRPO/GSPO.
• Efficient Training of MoE Models: Support low-cost R3 replay training for MoE architectures such as Qwen3-30B-A3B, reducing routing errors and computational overhead.