OpenManus-RL is an open-source project jointly led by Ulab-UIUC and MetaGPT, and is an extended version of the original OpenManus initiative. Inspired by the success of reasoning models such as Deepseek-R1 and QwQ-32B in reinforcement learning (RL) tuning, the project explores a new paradigm for RL-based LLM agent tuning and regularly updates exploration directions and results in a livestream-style manner. All progress, including rigorous testing on agent benchmarks such as GAIA, AgentBench, WebShop, and OSWorld as well as the tuned models, will be publicly shared and continuously updated.
In terms of main capabilities, the project proposes an advanced RL-based agent tuning framework designed to significantly improve the reasoning and decision-making abilities of large language models. The framework draws on RAGEN's Reasoning-Interaction Chain Optimization (RICO), and further explores new algorithm structures, diverse reasoning paradigms, fine-grained reward strategies, and rich benchmark environments. The project supports agent environment setup for online RL tuning; connects specialized reasoning models such as deepseek-r1 and QwQ-32B for complex reasoning tasks to collect comprehensive agent trajectories; provides RL fine-tuning methods to customize behavior in agent environments; and evaluates the framework on agent benchmarks such as Webshop, GAIA, OSWorld, and AgentBench.
In terms of reasoning model exploration, the project evaluates multiple advanced reasoning models, each providing unique reasoning capabilities and offering reference for downstream optimization and training strategies.
In terms of alternative rollout strategies, the project experiments with multiple rollout strategies to improve agent planning efficiency and reasoning robustness, including: Tree-of-Thoughts (ToT, tree-structured reasoning paths), Graph-of-Thoughts (GoT, graph-structured representation of complex reasoning dependencies), DFSDT (depth-first search decision tree, optimizing action selection), and Monte Carlo Tree Search (MCTS, probabilistic exploration of reasoning and decision paths). These methods help identify the optimal rollout technique for different reasoning tasks.
In terms of reasoning formats, the project specifically analyzes and compares multiple reasoning output formats, including ReAct (explicitly integrating reasoning and action) and Outcome-based Reasoning (reasoning based on outcomes, optimized toward explicit outcome prediction), in order to determine the most effective reasoning representation for different tasks.
Post-trainingIn terms of strategy, the project studies multiple post-training methods: Supervised Fine-Tuning (SFT), Generalized Reward-based Policy Optimization (GRPO, including format rewards and outcome rewards), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Preference-based Reward Modeling (PRM). The project also uses annotated data to train specialized agent reward models to precisely quantify reward signals and guide trajectory selection during the training and evaluation stages.
In the inference stage, the project implements a trajectory scaling method so that the agent can flexibly adapt to different task complexities and improve robustness and performance in real-world scenarios. The agent has action-space awareness and adopts a systematic exploration strategy to effectively navigate complex action spaces and maximize expected rewards.
In terms of framework integration, the project integrates insights and methods from leading RL tuning frameworks, among which Verl is integrated as a Git submodule and is the main RL framework, providing advanced training capabilities: advanced RL algorithms (PPO, DPO, custom reward modeling), efficient training (optimized for large language model fine-tuning), flexible configuration (easy customization of training parameters), and production readiness (a battle-tested framework from ByteDance).
In terms of datasets, OpenManusRL-Dataset combines agent trajectories from AgentInstruct, Agent-FLAN, and AgentTraj-L (AgentGym), with: the ReAct framework (reasoning-action integration), structured training (separation of format/reasoning learning), anti-hallucination (negative samples + environment grounding), and 6 domains (OS, DB, Web, KG, home, e-commerce). Supported tasks include text-generation (ReAct-style instruction following) and conversational-ai (tool-augmented dialogue).
In terms of usage, the project uses git submodules; after cloning the repository, the submodules need to be initialized and updated. Environment setup requires creating and activating a conda environment and installing the required dependencies; setting up the WebShop evaluation environment requires downloading PDDL and game files as well as the pretrained MskRCNN detector. Data preparation requires downloading the OpenManus-RL dataset from Hugging Face. Training examples and related papers and classsimilar frameworks are listed in the repository. The code and datasets are now available, and the verl submodule has been integrated to enhance RL training capabilities.
In terms of applicable scenarios, the project is suitable for researchers and developers to explore reinforcement learning tuning of LLM agents, agent trajectory data collection, post-training strategies and benchmark evaluation, as well as tool integration and reasoning capability improvement. The community can contribute fine-tuning codebases, tuning datasets, environment setup, and computing resources, and important contributors will be listed as co-authors of the paper.
In terms of limitations, the project is still in a live development process, and some content (such as the training part) is still under active development. Specific features, charges, prices, platforms, effects, or usage steps are not yet provided in the materials.
