SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsUniRL

UniRL

UniRL is a unified reinforcement learning (RL) framework for multimodal models open-sourced by the Tencent Hunyuan team, used for RL post-training of multimodal model families. It applies the post-training loop of "generate samples, score, compute advantages, update the policy, and synchronize the weights back to the rollout worker" to different multimodal models. The framework adopts a layered, composable design: the training entry point loads Hydra example configurations and creates the trainer for the corresponding domain; the RL trainer coordinates generation, scoring, and updates, and connects to pluggable rollout engines, algorithms, model bundles, reward services, and a shared distributed runtime (Ray DevicePool, FSDP, Transfer Queue, LoRA/full-weight synchronization). UniRL also includes algorithms proposed by the team (such as Flow-DPPO, DRPO, CPPO) as well as reference algorithms such as GRPO, DiffusionNFT, DanceGRPO, and MixGRPO, and supports multiple training modes such as SFT, asynchronous AR, diffusion, and agentic. The project is aimed at researchers and engineers and can be used for reinforcement learning training and experimentation on multimodal models (including diffusion models and autoregressive models).

Open website

About this tool

UniRL is a reinforcement learning framework for unified multimodal models, open-sourced by the Tencent Hunyuan team (Tencent-Hunyuan). Its positioning is: to use the same RL post-training loop—generate samples, score, compute advantages, update the policy, and then synchronize the weights back to the rollout worker—to cover multiple multimodal model families.

Purpose UniRL is used for reinforcement learning post-training of multimodal models, and is applicable to different model families such as diffusion models and autoregressive (AR) models. It modularizes the reinforcement learning training process, making it easy to combine different rollout engines, algorithms, model bundles, and reward services.

Main capabilities

  • Unified RL post-training loop: generation, scoring, advantage computation, policy update, weight synchronization.
  • Layered, composable system: the training entry point loads Hydra example configurations and creates the trainer for the corresponding domain; the RL trainer coordinates generation, scoring, and updates.
  • Pluggable components: rollout engines, algorithms, model bundles, reward services, shared distributed runtime.
  • Distributed runtime: Ray DevicePool, FSDP, Transfer Queue (TQ), LoRA/full-weight synchronization.
  • Training modes: SFT (consumes supervision manifests), asynchronous AR and diffusion entry points (overlapping rollout and training on different GPU partitions), and agentic multi-turn tool-use workflows.
  • Algorithm support: algorithms proposed by the team (Flow-DPPO, DRPO, CPPO) each include step-by-step tutorials and runnable examples; reference algorithms such as GRPO, DiffusionNFT, DanceGRPO, and MixGRPO are also integrated.
  • Model support: the model dimension and the algorithm dimension are independent of each other and composable; the same diffusion algorithm can run on diffusion models, and AR algorithms run on AR models. Each model row links to a runnable recipe, with status notes such as runnable, bundle-only, SFT-only, and combined.
  • Reward backends and rollout engine coverage are extensible.

How to use

  • First install the dependencies, see INSTALL.md.
  • Start training through Hydra configurations and entry points; each entry point has a built-in default value, used when --config-name is omitted.
  • examples/README.md provides a complete launch guide, naming conventions, and how to add a recipe; unirl/README.md explains the runtime loop, deployment modes, and module map.
  • Agentic training extends the AR path via train_agentic, supporting multi-turn tool use; before each rollout, the current training weights are synchronized, trajectories at the same level are dispatched concurrently, and scoring and training wait until the complete GRPO group is ready; successful trajectories receive group-normalized advantages that are applied to each assistant turn, while failed trajectories do not participate in updates. For the agent environment, tools, and trajectory contract, see the agent environment guide.
  • For multi-node setups, all entry points, and the complete startup method for mooncake, see the documentation.

Applicable scenarios

  • Reinforcement learning post-training research for multimodal models (including diffusion models and AR models).
  • Experiments that need to combine multiple algorithms and model families and reuse the rollout engine and reward service.
  • Engineering scenarios that require SFT, asynchronous training, diffusion training, or agentic multi-turn tool-calling training.

Limitations and notes

  • The materials do not provide specific data such as pricing, cost, or performance metrics.
  • Some model combinations are bundle-only, SFT-only, or in a combined state, and not all combinations have an independently runnable pipeline; for the complete matrix (packages, each recipe, rollout engine, and the limitations of each model), see unirl/models/README.md.
  • Recent directions include: expanding algorithm coverage for new model families (FLUX.2-Klein, HunyuanVideo 1.0/1.5, Bagel); extending algorithms proposed by the team (Flow-DPPO, DRPO) to more model families; and expanding reward backend and rollout engine coverage.
  • Version history: UniRL v0.2.0 was released in 2026-10, adding 12 model packages, SFT/agentic/asynchronous training modes, and the SGLang 0.5.19 + vLLM-Omni 0.28 rollout stack; the DRPO, Flow-DPPO, and CPPO papers were released in 2026-06.

Contributions and acknowledgements Before submitting a PR, please read the repository conventions in AGENTS.md, run the pre-PR checks for the files you changed, and fill out the PR template; use the issue form for bug reports or feature requests. The framework is built on top of the open-source RL and inference ecosystem, with special thanks to vLLM, SGLang, slime, and verl.

Related tools

Browse category