EAPO - Alibaba Tongyi's New Reinforcement Learning Framework

What is EAPO

EAPO is a long-text reasoning reinforcement learning framework launched by Alibaba Tongyi Lab. By introducing an "evidence reward" mechanism, it moves the supervision signal from the final answer down to the evidence extraction process. The framework has been accepted by ACL 2026. On 8 authoritative long-text benchmarks, it enables a 30B model to surpass 120BGPT-OSSand closed-source large models such as Claude-Sonnet-4.

Main Features of EAPO

• Structured Evidence Reasoning: Forces the model to execute a four-step workflow of "task analysis -> evidence extraction -> reasoning execution -> answer generation", using special tokens to separate each step so that intermediate evidence states can be directly supervised.
• Multi-Granularity Process Rewards: Builds a composite reward signal, including format-following rewards, group-relative evidence quality rewards, and outcome accuracy rewards, shifting from sparse outcome supervision to dense process supervision.
• Group-Relative Evidence Evaluation: Samples multiple evidence trajectories for the same question, has the reward model uniformly evaluate them and give integer quality scores from 1 to 5, and normalizes within the group to generate relative rewards, guiding the model to prioritize extracting high-quality evidence.
• Reward-Policy Co-Evolution: Designs an adaptive closed-loop mechanism that filters high-confidence, outcome-consistent high-quality evidence chains generated by the policy model and feeds them back to the reward model for rejection fine-tuning, enabling the evaluation criteria to dynamically evolve as model capabilities improve.
• Long-Text Reasoning Enhancement: Significantly improves performance on 8 authoritative long-text benchmarks such as SEAL and LongBench-V1/V2, enabling a 30B model to surpass 120B closed-source large models such as GPT-OSS and Claude-Sonnet-4 in long-text reasoning.

Technical Principles of EAPO

• Evidence-Augmented Reasoning (EAR) Paradigm: Establishes a structured reasoning paradigm that forces the model to excerpt relevant evidence snippets verbatim from the source text before generating an answer. Special tokens split the process into four steps: "task analysis -> evidence extraction -> reasoning execution -> answer generation", exposing intermediate evidence states so they can be directly supervised, fundamentally solving the hallucination problem of traditional models "guessing the answer correctly but citing it incorrectly".
• Group-Relative Evidence Reward: EAPO shifts the optimization objective of reinforcement learning from "correct outcome" to "correct evidence". During training, the model samples multiple different evidence trajectories for the same question. The reward model simultaneously evaluates these evidence sets, gives utility scores from 1 to 5, and normalizes them within the sampling group into relative rewards in [0,1]. Dense process supervision lets the model understand that "finding the right evidence" is more important than "guessing the right answer", effectively suppressing parametric shortcuts.
• Adaptive Reward-Policy Co-Evolution: As the policy model's capabilities improve, a fixed reward model will gradually become unable to distinguish subtle differences in evidence quality. EAPO designs a self-reinforcing closed loop: through Outcome-Consistent Rejection Fine-Tuning, it selects high-confidence rollout data that is either "high evidence score and correct answer" or "low score and wrong answer", and uses these instances to supervise and fine-tune the reward model. The stronger the policy model, the better the training data it generates; the more accurate the reward model's evaluation, the more it can in turn guide the policy model to extract more precise evidence, achieving dynamic synchronized evolution.
• GRPO-Based Composite Reward Mechanism: EAPO uses Group Relative Policy Optimization as its base algorithm to build a multi-granularity composite reward: format compliance reward (α=0.1) ensures the output conforms to the EAR structure; group relative evidence quality reward (β=0.3) provides process supervision; result accuracy reward (γ=0.6) verifies the final answer. The weighted combination of the three transforms sparse result signals into dense process-oriented guidance.

Key information and usage requirements of EAPO

• R&D team: Alibaba Tongyi Lab (Xin Guan, Zijian Li, Shen Huang, etc.), the paper has been accepted by ACL 2026.
• Base models: Trained on three models: Qwen3-14B (Dense architecture), Qwen3-30B-A3B-Instruct (MoE architecture), and Qwen3-30B-A3B-Thinking.
• Context length: Training and evaluation samples are uniformly limited to within 128K tokens.
• Training data: 4,664 composite samples, covering multi-hop QA (MuSiQue) with 32K-128K padded context and Wikipedia mixed QA (structured + unstructured).
• Reward model: Initialized based on Qwen3-30B-A3B-Thinking, updated every 20 RL steps.
• Reward weights: Format compliance α=0.1, evidence quality β=0.3, result accuracy γ=0.6.
• Core algorithm: Based on GRPO (Group Relative Policy Optimization), introducing group relative evidence reward and co-evolution mechanism.
• Model foundation: Needs to be trained based on Qwen3 series models that support long text, 30B-A3B-Thinking is recommended for the best results.

Core advantages of EAPO

• Process supervision innovation: Breaks the sparse reward bottleneck of long-text RL, and for the first time achieves evidence-level dense process supervision, forcing the model to execute the four-step workflow of "analysis → evidence extraction → reasoning → answering", making every step of reasoning traceable.
• Performance breakthrough: The model trained based on Qwen3-30B-Thinking achieves an average score of 63.1%, surpassing closed-source large models such as 120B GPT-OSS and Claude-Sonnet-4, realizing small models outperforming large models.
• Double reduction in errors: Evidence error rate drops from 17.7% to 13.5%, and reasoning error rate drops from 20.7% to 15.4%. Both types of errors decrease simultaneously without explicit supervision of reasoning steps.
• Leading training efficiency: Compared with the GRPO baseline that relies only on result rewards, EAPO converges significantly faster and has a higher accuracy ceiling, with evidence quality scores consistently leading by a wide margin during training.
• Evaluation criteria evolution: The reward model and policy model continuously iterate and co-evolve through high-confidence rollout data, avoiding fixed evaluation criteria as model capabilities improve.

Project address of EAPO

• Technical paper：https://arxiv.org/pdf/2601.10306

Comparison of EAPO with similar competitors

Application scenarios of EAPO

• AI search and question answering: Addresses the core pain point of AI search engines "retrieving correctly but answering incorrectly," forcing the model to precisely locate and cite supporting evidence within massive retrieval results, and eliminating hallucinated answers.
• Professional domain document analysis: Suitable for scenarios such as law, finance, and healthcare that require strict factual basis, ensuring that every report and every analysis has clear source text and an evidence chain in support.
• Scientific research literature review: Supports cross-validation and comprehensive reasoning across multiple papers, automatically extracts key experimental data and cites it accurately, ensuring that all review conclusions are supported by the literature.
• Enterprise knowledge base question answering: Precisely locates the basis for decisions in ultra-long internal documents, contracts, and manuals, helping employees quickly obtain business answers supported by clear sources.
• Educational tutoring and automatic grading: Used in problem-solving tutoring to require that every step of reasoning cite source text evidence, and in automatic grading to verify whether students' answers are derived from valid information in the problem materials.