FIPO - A reinforcement learning algorithm launched by Alibaba Tongyi

What is FIPO

FIPO (Future-KL Influenced Policy Optimization) is a reinforcement learning algorithm launched by the Alibaba Tongyi Lab. Through the Future-KL mechanism, it quantifies the influence of each token on the subsequent reasoning trajectory, achieving token-level precise credit assignment and solving the reasoning bottleneck caused by the "equal sharing of rewards and penalties" in traditional RL. Under pure RL training of a 32B model, it pushes the reasoning length from 4,000 to 10,000+ tokens, increases AIME 2024 accuracy from 50% to 58%, surpasses o1-mini, and breaks through the length stagnation problem in deep reasoning.

Main features of FIPO

• Token-level precise credit assignment: It breaks through the traditional RL "unified settlement at the endpoint" model, and through the Future-KL mechanism identifies and rewards about 2% of the key decision tokens in the reasoning chain, achieving fine-grained optimization.
• Autonomous expansion of reasoning length: It breaks the "length stagnation" bottleneck of traditional methods (~4,000 tokens), drives the model to generate deep thinking chains of 10,000+ tokens, and length is positively correlated with accuracy.
• Intelligent attribution and error correction: It quantifies the causal influence of each token on the subsequent trajectory: positively reinforcing "stable anchors" and reversely suppressing "misleading nodes," reducing the model's Oops Moment of "self-overturning the correct answer."
• Training stability guarantee: Through the triple mechanisms of extreme value filtering, soft decay window, and influence weight clipping, it prevents gradient explosion and ensures the numerical stability of long-chain reasoning training.

How to use FIPO

• Download the code: Obtain the FIPO code from GitHub and configure the runtime environment (based on the VeRL framework, configured similarly to DAPO).
• Prepare data: Use a mathematical reasoning dataset verified with standard answers (such as DAPO-17K); there is no need to prepare long chain-of-thought example data.
• Adjust parameters: Set the soft decay window half-life to 32 steps, limit the influence weight to the range of 1 to 1.2 times, and enable extreme value filtering to ensure training stability.
• Start training: Run the training program. The system will automatically identify key tokens in the reasoning chain and precisely reward and penalize them. During training, the model's answer length will automatically expand from 4,000 words to more than 10,000 words.
• Model inference: Load the trained model to use it. After inputting a math problem, the model will generate a long-chain thinking process with in-depth self-verification and give an answer.

Key information and usage requirements of FIPO

• Precise identification: Locate only 2% of the key decision tokens in the reasoning chain for targeted rewards and penalties.
• Length breakthrough: Under pure RL training, reasoning length goes from 4,000 to 10,000+ tokens, and AIME 2024 accuracy goes from 50% to 58% (surpassing o1-mini).
• No cold start required: Directly use a Base model + mathematical data for training, with no need for long chain-of-thought examples.
• Training framework: VeRL + Ray distributed training (a 32B model requires a multi-GPU A100/H100 cluster).
• Inference environment: A standard PyTorch environment is sufficient to load HuggingFace-format models.

Core advantages of FIPO

• Token-level precise rewards and penalties: Through the Future-KL mechanism, it identifies only 2% of the key decision tokens in the reasoning chain, achieving fine-grained credit assignment and saying goodbye to the equal sharing of rewards and penalties caused by traditional RL "unified settlement at the endpoint."
• Breakthrough in the length bottleneck: Breaking through the 4,000-token "glass ceiling" of traditional methods, it drives the model to autonomously generate 10,000+ token deep reasoning chains, with a strong positive correlation between length and accuracy.
• Pure RL with zero cold start: No long chain-of-thought example data is needed; training directly with a Base model + verifiable reward data can stimulate deep reasoning ability, greatly reducing data preparation costs.
• SOTA performance: Under the 32B-scale pure RL setting, AIME 2024 accuracy reaches 58% (peak), surpassing o1-mini (56%) and DeepSeek-R1-Zero-32B (47%).
• Stable and controllable training: The triple protection mechanism (extreme value filtering + soft decay window + weight clipping) effectively suppresses Future-KL variance and prevents gradient explosion and catastrophic collapse during long-chain training.

FIPO project address

• GitHub repository：https://github.com/qwenpilot/FIPO
• arXiv technical paper：https://arxiv.org/pdf/2603.19835

Comparison of FIPO with similar competitors

FIPO application scenarios

• Mathematical competitions and research assistance: Solves AIME/IMO-level high-difficulty math problems, generates complete derivation processes with self-verification, and assists mathematicians in exploring complex proof ideas.
• Complex code generation and debugging: Handles algorithmic problems requiring multi-step logical decomposition (such as difficult LeetCode problems), analyzes the root causes of code errors through long-chain reasoning, and provides fixes.
• Automated theorem proving: In the fields of formal mathematics and logic, constructs rigorous chains of argument from premises and assumptions to conclusions, and automatically checks the completeness of the proof process.
• Scientific research reasoning: Assists with complex formula derivations and experimental design analysis in fields such as physics and chemistry, integrating multi-dimensional constraints through deep thinking.
• Strategic decision-making and business analysis: Handles business decision problems that require weighing multiple factors (such as investment analysis and supply chain optimization), and generates decision recommendations that include risk verification.