Posterior-GRPO: Rewarding Reasoning Processes in Code Generation
Lishui Fan, Yu Zhang, Mouxiang Chen, Zhongxin Liu · 2025
Read Posterior-GRPO: Rewarding Reasoning Processes in Code Generation by Lishui Fan, Yu Zhang, Mouxiang Chen, Zhongxin Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05170.txt.
Introduction
Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu propose ReCode, a reinforcement learning framework that rewards the quality of a model's reasoning, not just whether its final code runs. Process-level rewards are attractive but hard: there is little fine-grained data on which reasoning traces are better, and a learned reward can be gamed. ReCode answers the first problem by training its reward model on pairs of improved and degraded reasoning, made synthetically.
The second answer is a training rule that gates the reasoning reward with actual execution results, so a trace earns credit only when the code passes. The authors also release a benchmark of reasoning preference pairs for testing reward models. A 7-billion-parameter model trained with ReCode gained 16.1 percent over its base version and matched a much larger commercial model on code tests.
Catalog record
| CREATOR | Lishui Fan, Yu Zhang, Mouxiang Chen, Zhongxin Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05170
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Posterior-GRPO: Rewarding Reasoning Processes in Code Generation Lishui Fan, Yu Zhang, Mouxiang Chen, Zhongxin Liu Abstract Reinforcement learning (RL) has significantly advanced code generation for large language models (LLMs). However, current paradigms rely on outcome-based rewards from test cases, neglecting the quality of the intermediate reasoning process. While supervising the reasoning process directly is a promising direction, it is highly susceptible to reward hacking, where the policy model learns to exploit the reasoning reward signal without improving final outcomes. To address this, we introduce a unified framework that can effectively incorporate the quality of the reasoning process during RL. First, to enable reasoning evaluation, we develop LCB-RB, a benchmark comprising preference pairs of superior and inferior reasoning processes. Second, to accurately score reasoning quality, we introduce an Optimized-Degraded based (OD-based) method for reward model training. This method generates high-quality preference pairs by systematically optimizing and degrading initial reasoning paths along curated dimensions of reasoning quality, such as factual accuracy, logical rigor, and coherence. A 7B parameter reward model with this method achieves state-of-the-art (SOTA) performance on LCB-RB and generalizes well to other benchmarks. Finally, we introduce Posterior-GRPO (P-GRPO), a novel RL method that conditions process-based rewards on task success. By selectively applying rewards to the reasoning processes of only successful outcomes, P-GRPO effectively mitigates reward hacking and aligns the model’s internal reasoning with final code correctness. A 7B parameter model with P-GRPO achieves superior performance across diverse code generation tasks, outperforming outcome-only baselines by 4.5%, achieving comparable performance to GPT-4-Turbo. We further demonstrate the generalizability of our approach by extending it to mathematical tasks. Our models, dataset, and code are publicly available. Introduction Reinforcement learning (RL) has emerged as a transformative post-training paradigm for large language models (LLMs) ( Guo et al. 2025 ; Yang et al. 2025 ) . The recent breakthrough of DeepSeek-R1 ( Guo et al. 2025 ) exemplifies this paradigm shift, achieving remarkable success through techniques like GRPO algorithm ( Shao et al. 2024 ) . This advancement has catalyzed extensive research applying RL post-training to enhance code generation in LLMs ( Abdin et al. 2025 ; Zhao et al. 2025 ; Xu et al. 2025 ) . Despite these advances, existing approaches exclusively rely on outcome signals from generated code—such as test pass rates—while overlooking the reasoning processes that underlie code quality ( Guo et al. 2025 ; Abdin et al. 2025 ; Zeng et al. 2025 ) , which may lead to suboptimal reasoning processes, ultimately compromising the accuracy of the final solutions ( Zhang et al. 2025 ) . Our preliminary investigation reveals a significant correlation between reasoning process quality and solution correctness ( \chi^{2} test ( Pearson 1900 ) , p=9.3\times 10^{-15}\ll 0.001 ; see the correlation analysis in supplementary materials for details). Extensive work has also demonstrated that enabling LLMs to simultaneously generate reasoning processes during solution generation can substantially enhance their capabilities ( Wei et al. 2022 ; Lyu et al. 2023 ) . This gap motivates a fundamental research question: Can we optimize the reasoning processes of policy models to achieve more efficient improvements in their code generation capabilities? A typical RL training pipeline for code generation ( Guo et al. 2025 ) follows an intuitive paradigm: the model generates code solutions, receives feedback based on test case outcomes, and updates its policy accordingly. While widely adopted, directly integrating the reasoning process rewards presents three critical challenges: • Lack of appropriate evaluation benchmarks for assessing reward models’ discrimination capabilities on reasoning processes. Existing benchmarks ( Liu et al. 2025 ; Lambert et al. 2024 ) primarily focus on using solutions rather than reasoning processes, creating misalignment with our objectives. • Absence of reliable reward models specifically designed for reasoning evaluation. Current state-of-the-art (SOTA) reward models ( Liu et al. 2024a ; Yuan et al. 2024 ) , such as Skywork-Reward-V2-Llama-3.1-8B ( Liu et al. 2024a ) , are trained on solutions rather than reasoning processes. While correlation exists between reasoning and code quality, the semantic gap between natural language and code structure ( Wang et al. 2021 ) renders direct application of existing models suboptimal. • Lack of effective RL algorithms that leverage signals from reward models. Research ( Guo et al. 2025 ) has demonstrated that neural reward models may suffer from reward hacking during RL training, particularly in code generation tasks where neural reward model signals are more susceptible to exploitation compared to test case pass rate reward signals. To address these challenges, we propose a unified training framework that enhances code generation capabilities through reasoning-aware RL. Our core insight is to leverage the intrinsic features of reasoning processes to create reliable training signals that guide policy models toward both correct solutions and high-quality reasoning patterns. To address the first challenge, we introduce LCB-RB, a benchmark derived from LiveCodeBench ( Jain et al. 2025 ) composed of 187 preference pairs. Each pair consists of a superior and an inferior reasoning process. We then introduce the Optimized-Degraded-based method, which we refer to as the OD-based method, for reward model training to address the second challenge. Specifically, we employ a powerful LLM to generate initial reasoning processes for problems, then transform these into optimized and degraded versions based on three critical dimensions that determine reasoning quality, including factual accuracy, logical rigor and logical coherence. Training on these inherently contrastive pairs enables our reward model to distinguish between high-quality and low-quality reasoning patterns. To address the last challenge, we propose Posterior-GRPO (P-GRPO), a novel algorithm designed to prevent reward hacking while maximizing training signal quality, based on GRPO. Our approach integrates signals from three complementary sources: the thinking reward, outcome reward (i.e., pass rates), and the format reward. Crucially, we employ a posterior reward assignment strategy, in which reasoning rewards are computed only after correct outcomes (i.e., when all test cases pass), ensuring alignment between reasoning quality and functional correctness. An advantage of P-GRPO is its data utilization efficiency, which enables differentiated rewards when all samples are correct, improving the original GRPO’s limitation where uniform success yields zero advantage values and no gradient information. Specifically, for the reward model, we train 3B and 7B parameter models from Qwen2.5-Coder-Base with OD-based method. The 7B model achieves comparable performance to GPT-4-Turbo on LCB-RB while demonstrating strong generalization on the reasoning subset of RewardBench ( Lambert et al. 2024 ) . And for RL, P-GRPO effectively enhances the code generation capabilities of Qwen2.5-Coder-7B-Instruct with a relative improvement of 13.9% over the base model (50.4% \rightarrow 57.4%) on LiveCodeBench, HumanEval(+), MBPP(+), and BigCodeBench, surpassing the RL with outcome-only rewards baseline by 4.5% and reaching performance comparable to GPT-4-Turbo. Furthermore, when we extend P-GRPO to mathematical tasks, Qwen2.5-Math-7B achieves a relative improvement of 7.3% over outcome-only reward baselines, demonstrating the generalization capability of our approach. In summary, our contributions are threefold. First, we introduce LCB-RB, a benchmark designed to evaluate the ability of reward models to discriminate between different levels of reasoning quality. Second, we identify multi-dimensional reasoning features and propose an OD-based method for training thinking reward models. Finally, we present P-GRPO, a novel RL algorithm that leverages process-based rewards to enhance model reasoning capabilities. Our extensive evaluation across 4 code generation benchmarks and