AAPO: Enhance the Reasoning Capabilities of LLMs with Advantage Momentum
Jian Xiong, Jingbo Zhou, Jingyong Ye, Dejing Dou · 2025
Read AAPO: Enhance the Reasoning Capabilities of LLMs with Advantage Momentum by Jian Xiong, Jingbo Zhou, Jingyong Ye, Dejing Dou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14264.txt.
Introduction
Jian Xiong and co-authors improve how large language models learn to reason through reinforcement learning. Group-relative methods score each answer against its peers, dropping the need for a separate value model. But when all answers in a group score alike, the learning signal fades to zero and training stalls.
Their method, AAPO, pads the advantage with a margin before it steers the update. The padded signal keeps teaching even when peers tie. Tests on several math reasoning benchmarks put AAPO ahead of the group-relative baseline. The code is public. The fix is small, but it targets a stall that wastes much training compute.
Catalog record
| CREATOR | Jian Xiong, Jingbo Zhou, Jingyong Ye, Dejing Dou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AAPO: Enhance the Reasoning Capabilities of LLMs with Advantage Momentum Jian Xiong, Jingbo Zhou, Jingyong Ye, Dejing Dou Abstract Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models (LLMs), especially in scenarios where supervised fine-tuning (SFT) falls short due to limited chain-of-thought (CoT) data. Among RL-based post-training methods, group relative advantage estimation, as exemplified by Group Relative Policy Optimization (GRPO), has attracted considerable attention for eliminating the dependency on the value model, thereby simplifying training compared to traditional approaches like Proximal Policy Optimization (PPO). However, we observe that exsiting group relative advantage estimation method still suffers from training inefficiencies, particularly when the estimated advantage approaches zero. To address this limitation, we propose Advantage-Augmented Policy Optimization (AAPO), a novel RL algorithm that optimizes the cross-entropy (CE) loss using advantages enhanced through a momentum-based estimation scheme. This approach effectively mitigates the inefficiencies associated with group relative advantage estimation. Experimental results on multiple mathematical reasoning benchmarks demonstrate the superior performance of AAPO. 1 Introduction Reinforcement learning (RL) has emerged as a powerful approach for enhancing the reasoning and decision-making capabilities of large language models (LLMs). While LLMs have demonstrated strong performance in both language understanding and generation tasks [ 3 , 6 , 40 , 49 ] , traditional training strategies such as pre-training and supervised fine-tuning (SFT) [ 30 , 2 , 24 ] often fall short in enabling effective chain-of-thought (CoT) [ 43 ] reasoning for complex decision-making tasks. To address this limitation, recent research has explored RL-based training paradigms, which have shown considerable empirical success in specialized domains such as mathematical reasoning. Models including GPT-o1 [ 26 ] , DeepSeek-R1 [ 14 ] , and QwQ [ 29 ] exemplify this promising direction, demonstrating the potential of RL to substantially improve the reasoning abilities of LLMs. A recent advancement in LLM post-training with RL is the introduction of novel methods for advantage estimation, a technique for quantifying how favorable a specific action is in a given state. In this context, the term advantage measures the relative benefit of an action compared to the average in that state, thereby providing an informative learning signal during training. Traditionally, approaches such as Proximal Policy Optimization (PPO) [ 35 ] , exemplified by InstructGPT [ 27 ] , rely on a value model to estimate the advantage. Although PPO offers stable and reliable performance, the need to maintain a separate value model leads to substantial consumption of GPU resources. In contrast to conventional approaches, the group relative advantage estimation method was originally proposed in Group Relative Policy Optimization (GRPO) [ 36 ] and has since been widely adopted for enhancing reasoning capabilities in LLMs. Group relative advantage estimation method removes the need for value models entirely by evaluating responses relative to the average within a group of sampled responses. This approach significantly reduces GPU memory usage and computational costs while maintaining competitive performance in downstream reasoning tasks. Its effectiveness is further evidenced by the strong performance of modern reasoning models such as DeepSeek-R1 [ 14 ] , which highlights the effectiveness of group relative advantage estimation in balancing computational efficiency and performance robustness. Several extensions have further refined this paradigm. Decoupled Clipping and Dynamic sAmpling Policy Optimization (DAPO) [ 48 ] improves training on long CoT sequences through token-level gradient estimation and relaxed clipping strategies. Dr. GRPO [ 25 ] introduces an unbiased optimization method that enhances token efficiency, while Group Policy Gradient (GPG) [ 8 ] further simplifies the learning process by removing surrogate losses and eliminating the reference model. While these methods vary in implementation specifics, they all share a common foundation in the principle of group relative advantage estimation for effective LLMs post-training. We provide a comprehensive discussion of GRPO and GPG in Section 3 . Although the above approaches have significantly advanced RL in the post-training stage and enhanced the reasoning capabilities of LLMs beyond what SFT can achieve, the practical limitations associated with the group relative advantage estimation method remain unresolved. As shown in the left panel of Figure 1 , when the group relative advantage estimation method [ 36 , 8 , 25 ] is adopted, the advantage may approach when the rewards within a group exhibit low variance, resulting in zero gradient and, consequently, no parameter updates. Conversely, when rewards among responses within the group vary significantly, the resulting advantages can exhibit high variance, potentially leading to unstable or unintended gradient ascent. Both scenarios deviate from the desired optimization trajectories. Figure 1 : Left : An example convergence trajectories of policy optimization with group relative advantage estimation method proposed in GRPO. Red node denotes gradient euqals to zero. Purple arrow denotes gradient ascent. Green arrow denotes gradient descent. Right : An example of ideal convergence trajectories of policy optimization. In this work, to address the challenges of policy optimization from the group relative advantage estimation method mentioned above, we propose a novel RL algorithm Advantage-Augmented Policy Optimization (AAPO), which mitigates the issues demonstrated in Figure 1 by estimating the advantage with advantage momentum. Advantage momentum is defined as the distance between the rewards of responses from the policy model and those of the responses from the reference model. This approach incorporates a reference gradient into the original gradient, thus providing a reliable optimization signal that reflects the overall direction of improvement, even when the advantages approach zero. Experiments on several representative mathematical reasoning benchmarks demonstrate the effectiveness and robustness of AAPO. In summary, our main contributions are as follows: • We delve into the optimization behavior of current RL algorithms adopting the group relative advantage estimation method, in the context of post-training LLMs with RL, with a particular emphasis on potential issues related to advantage estimation during the optimization. • We propose Advantage-Augmented Policy Optimization (AAPO), which mitigates the issues of advantage estimation with advantage momentum by taking a comparison with the reference model. • Numerous experiments have demonstrated that AAPO exhibits better optimization behavior during training and achieves superior performance on representative mathematical reasoning benchmarks. 2 Related Work Large Language Models LLMs such as PaLM 2 [ 1 ] and Llama [ 40 , 41 ] achieved impressive performance in understanding and generation tasks, largely due to scaling laws, high-quality pre-train corpus, and optimized training techniques [ 5 , 27 , 13 , 42 ] . However, these models still struggle with complex reasoning. In contrast, recent LLMs such as GPT-o1 [ 26 ] , DeepSeek-R1 [ 14 ] , and QwQ [ 29 ] show strong reasoning capabilities, particularly in mathematical tasks. These gains are primarily attributed to RL-based post-training, which significantly outperforms traditional SFT. Reinforcement Learning for LLMs Early RL-based post-training methods [ 7 , 37 , 27 , 50 ] mainly relied on human-labeled preference data and reward models to evaluate responses. PPO [ 35 ] estimates state values via a value model for advantage estimation while using a reward model to assign rewards. DPO [ 31 ] uses paired preference data to encourage preferred responses and suppress undesired ones. However, annotating preferences and training reward models are resource intensive. Moreover, the scarcity of explicit reasoning data limits the enhancement of models’ reasoning capabilities