What is AReaL-boba
AReaL-boba is an open-source reinforcement learning training framework jointly launched by Ant Technology Research Institute and Tsinghua University. AReaL-boba is an upgraded version of AReaL, which lowers the threshold for reinforcement learning training, allowing users to easily train reasoning models. The framework offers fast training speed, supports a variety of computing resources, and based on innovative optimizations, significantly improves training throughput. Among them, the 7B model performs excellently in mathematical reasoning, setting a new AIME score record. AReaL-boba open-sources training data, scripts, and models. At the 32B model size, using 200 data samples and a cost of 200 US dollars, it replicates QwQ-32B reasoning performance, promoting the democratization of reinforcement learning technology.
Main features of AReaL-boba
- Efficient training: Based on optimization and adaptation of the SGLang inference framework, it significantly improves training throughput and supports training from small-scale to large-scale distributed training.
- Improved reasoning capability: It performs excellently on tasks such as mathematical reasoning, and the 7B model sets a new score record among models of the same size on the AIME benchmark test.
- Low-resource training: Based on innovative data distillation technology, it reproduces the reasoning performance of QwQ-32B using 200 data samples, reducing training costs.
- Fully open source: It provides complete code, datasets, training scripts, and evaluation scripts to ensure reproducibility and make it convenient for developers to use and improve.
Technical principles of AReaL-boba
- Reinforcement learning: It optimizes model behavior based on reward signals and learns optimal policies through interaction with the environment. In language models, reinforcement learning is used to optimize the model's generation capability and perform better on specific tasks.
- SGLang inference framework integration: AReaL-boba is the first open-source training system to fully integrate the SGLang inference framework. SGLang provides efficient inference capability and optimizes computational efficiency during training.
- Engineering optimization: Multiple engineering optimizations are applied to the training process, including parallel computing and video memory management, improving training throughput. Significant training speed improvements are achieved across different model sizes.
- Data distillation technology: Based on an innovative data distillation method, it extracts key information from large amounts of data and streamlines training data.
Project addresses of AReaL-boba
- GitHub repository：https://github.com/inclusionAI/AReaL
- HuggingFace model library：https://huggingface.co/collections/inclusionAI/areal-boba
Application scenarios of AReaL-boba
- Mathematical reasoning and education: Develop intelligent educational tools to help students solve complex mathematical problems.
- Natural language processing tasks: Improve the performance of text generation, question answering systems, machine translation, and more.
- Agent development: Used for training agents in fields such as games and robot control.
- Low-resource model training: Suitable for environments with limited data resources for efficient model training.
- Academic research and community collaboration: As a research tool, it promotes academic exchange and technology sharing.