SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsOther AI toolsADORA

ADORA

ADORA (Advantage Dynamics via Online Rollout Adaptation) is a reinforcement learning framework for dynamically adjusting advantage values based on the model's rollout distribution when training large language models (LLMs) and vision-language models (VLMs). The project is open-sourced on GitHub and provides a blog, code repository, wandb report, and model weights. According to its description, implementing ADORA in the Logic-RL framework reaches a 40 AMC score with only 100 training steps (the original paper reached 39 AMC with 1200 steps), with comparable AIME performance of 8; on the VLM side, using only 2K samples from the Geometry3K dataset and starting from Qwen2.5-VL-7B-Instruct, it achieves 73.5% accuracy on MathVista and maintains continuously growing response length, reportedly setting a new SOTA for DeepSeek-R1-Zero multimodal implementations. ADORA can be integrated into verl or OpenRLHF by modifying only one function, and can be combined with cold-start scenarios and DAPO.

Open website

About this tool

ADORA stands for Advantage Dynamics via Online Rollout Adaptation. It is a framework for reinforcement learning training whose core idea is to dynamically adjust advantage values based on the model's rollout distribution during training, in order to improve the long chain-of-thought (CoT) reasoning and reflection capabilities of large language models (LLMs) and vision-language models (VLMs).

Purpose

ADORA is used in the reinforcement learning training stage and aims to improve the reasoning performance of LLMs and VLMs. The project shows through experiments that it can improve long CoT reasoning and reflection capabilities, and supports implementation within the Logic-RL framework.

Main capabilities and experimental results

  • On the LLM side, ADORA's implementation in the Logic-RL framework reaches 40 AMC with only 100 training steps, whereas the original paper reached 39 AMC using 1200 steps; AIME performance remains comparable at 8. The text states that this result exceeds GRPO's 35 AMC and 6 AIME.
  • On the VLM side, using only 2K samples from the Geometry3K dataset and starting from Qwen2.5-VL-7B-Instruct, it achieves 73.5% accuracy on MathVista, accompanied by consistent response length growth, and is reportedly setting a new SOTA for DeepSeek-R1-Zero multimodal implementations.
  • A comparison of training dynamics shows that on Qwen2.5-VL-7B-Instruct (geometry3k), GRPO exhibits stalled response length growth and KL/policy loss outliers; ADORA instead achieves continuous length expansion and more stable optimization, at the cost of a slight decrease in training reward.
  • All results are presented as pass@1 accuracy.

Usage

  • To reproduce the LLM and VLM experiments in the article, refer to the tutorials in the ADORA and ADORA_VL folders.
  • The experiments were conducted in an environment with 8 A800 GPUs, and each of the two experiments takes about 1.5 days to complete.
  • ADORA can be integrated into verl or OpenRLHF by modifying only one function. It is still necessary to define a method for generating advantage weights from actor rollout results according to the specific training objective.
  • You may also choose to introduce ADORA only during certain stages of RL training.
  • The project description states that ADORA is compatible and independent, and can be seamlessly integrated with cold-start scenarios and the recently proposed DAPO.

Release information

[2025/03/20] Released the blog "Training Reasoning Model with Dynamic Advantage Estimation on Reinforcement Learning", the code repository, the wandb report, and the model weights AdoraRL.

Applicable scenarios and limitations

ADORA is intended for research, training, and reproduction scenarios involving the reinforcement learning reasoning capabilities of LLMs/VLMs, and is especially suitable for developers using Logic-RL, verl, or OpenRLHF. Before use, a corresponding GPU environment is required, and you must define the method for generating advantage weights yourself. Specific installation commands, license agreements, dependency versions, prices, and charges are not yet provided.

Related tools

Browse category