Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
Shenghua He, Tian Xia, Xuan Zhou, Hui Wei · 2025
Read Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective by Shenghua He, Tian Xia, Xuan Zhou, Hui Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02553.txt.
Introduction
Shenghua He, Tian Xia, Xuan Zhou, and Hui Wei give a math account in 2025 of reward design for large language models. They ask why rewarding only the final answer can be enough when training by reinforcement learning. In practice, each token gets no score of its own, and only the last one carries the reward.
Their new theorem shows that the true policy gradient can be estimated without bias from a response level reward alone. The true gradient is built on unknown token level rewards. That holds whether the zero reward assumption is true or not, for a wide family of methods. A simpler new algorithm follows from the theory.
Catalog record
| CREATOR | Shenghua He, Tian Xia, Xuan Zhou, Hui Wei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02553
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective Shenghua He, Tian Xia, Xuan Zhou, Hui Wei Abstract We study a common challenge in reinforcement learning for large language models (LLMs): the Zero-Reward Assumption , where non-terminal actions (i.e., intermediate token generations) receive zero task-specific immediate reward, while only the final token receives a reward for the entire response. This assumption arises frequently in practice, as precise token-level rewards are often difficult or infeasible to obtain in LLM applications. Recent work has addressed the limitations of this assumption in two major directions. One stream, including methods such as GRPO and ReMax, simplifies the critic network under the belief that zero intermediate rewards render it less influential. Another stream argues that the lack of fine-grained token-level feedback hinders effective credit assignment and attempts to model explicit token-level rewards. In this work, we provide a unifying theoretical perspective. We introduce the Trajectory Policy Gradient Theorem , which shows that the policy gradient based on true, unknown token-level rewards can be unbiasedly estimated using only a response-level reward model, regardless of whether the Zero-Reward Assumption holds or not, for algorithms in the REINFORCE and Actor-Critic families. This result reveals that widely used methods such as PPO, GRPO, ReMax, and RLOO inherently possess the capacity to model token-level reward signals, offering a theoretical justification for response-level reward approaches. Our findings pave the way for more practical, efficient LLM fine-tuning, allowing developers to treat training algorithms as black boxes and focus on improving the response-level reward model with auxiliary sub-models. We also offer a detailed analysis of popular RL and non-RL methods, comparing their theoretical foundations and practical advantages across common LLM tasks. Finally, we propose a new algorithm: T oken- Re inforced P olicy O ptimization ( TRePO , /trep/), a theoretically grounded method that is simpler than PPO, matches GRPO in memory efficiency, and holds promise for broad applicability. * * footnotetext: Equal Contribution. Tian Xia originated the idea and provided the mathematical proof; Shenghua He extended the idea and derived key conclusions; Xuan Zhou provided invaluable feedback on the proof. 1 Introduction With the rapid advancement of large language models (LLMs), reinforcement learning (RL) has become a key post-training technique for enhancing their general instruction-following abilities (e.g., GPT-4o [ 1 ] , Gemini [ 2 ] , Qwen2.5 [ 3 ] , Nova [ 4 ] ) and advanced reasoning skills (e.g., O-1 [ 5 ] , Deepseek-R1 [ 6 ] ). Proximal Policy Optimization (PPO) [ 7 ] , widely used in reinforcement learning from human feedback (RLHF), has been a foundational RL algorithm for post-training LLMs. It follows a general actor-critic architecture, where the policy model is the LLM being trained and the critic model estimates the expected long-term reward of intermediate states to reduce gradient variance during optimization. PPO has demonstrated strong empirical success across various tasks, including summarization [ 8 ] , instruction following [ 9 ] , and dialogue [ 10 ] . Despite its practical effectiveness, PPO requires training both the policy and critic models, even though only the policy is typically used during inference. In practical LLM application scenarios, PPO is typically implemented based on an assumption referred to in this work as the Zero-Reward Assumption : any state-action leading to an intermediate state receives zero immediate reward, while the state-action leading to the terminal state receives the full response-level reward for the entire trajectory. This is primarily because, in practice, it is often difficult, costly, or even infeasible to precisely define an immediate reward for each state-action pair, whereas a response-level reward is much more readily obtainable. To date, many popular prior works have recognized the limitations of the flawed Zero-Reward Assumption and attempted to address it by different ideas, which are roughly categorized into two groups. In the first group, Re-Max [ 11 ] states that “ r(s_{t},a_{t}) is non-zero only when the whole trajectory ends at t=T . This means that RLHF tasks are close to single-stage optimization problems since the rewards of the intermediate stages are 0 ”. Group Relative Policy Optimization (GRPO) [ 12 ] states that-“While in the LLM context, usually only the last token is assigned a reward score by the reward model, which may complicate the training of a value function that is accurate at each token”. Leave-One-Out Reinforce (RLOO) [ 13 ] states that “modeling of partial sequences is unnecessary since rewards from humans are only attributed to full generations, with no true rewards for any intermediary token generate”. These works focus on modeling response-level rewards and eliminate the critic network from PPO, which is typically as large as the policy model. Accordingly, methods falling under the Zero-Reward Assumption are generally categorized as response-level algorithms. However, while these approaches are strong from an engineering standpoint, they often overlook the theoretical shortcomings of the Zero-Reward Assumption . As a result, there is a general lack of theoretical analysis surrounding them. In another group, some works [ 14 , 15 , 16 , 17 ] attempted to address it by approximating instant rewards for intermediate states using a learned response-level reward model in specific application scenarios. Chan et al. [ 15 ] proposed an PPO variant called ABC, which distributes the response-level reward across tokens using attention maps from the reward model itself. Reinforced Token Optimization (RTO) [ 14 ] employs a DPO model that is trained with response-level labels as an implicit token-level reward model to generate token-level rewards for PPO training. More recently, Cui et al. [ 16 ] introduced a framework called Process Reinforcement through Implicit Rewards (PRIME), which learns an online implicit process reward model from outcome labels using a cross-entropy loss during the RL training for LLM mathematical tasks. These approaches typically rely on a second assumption, which we refer to as partial-reward assumption : a token-level reward model can be learned from response-level preference or outcome labels and can provide rich and reliable feedback for partial trajectories . However, the partial-reward assumption is largely heuristic, even though it does provide token-level rewards in form. In addition, RL approaches based on this assumption typically require training a response-level reward model to approximate token-level instant rewards, which greatly limits their practical generality. Though, these works brings improvements over PPO, yet there are still no clear analysis of where the improvement comes from. For example, is it from equipping the new methods with token-level capability, or bringing in additional information actually? In this study, we revisit the fundamentals of RL to re-derive all formulas relevant to RL for LLMs and explore a general RL solution. Our contributions span three main aspects as described below: Trajectory Policy Gradient Theorem: We propose a trajectory policy gradient theorem , which demonstrates in LLMs application scenarios, the Zero-Reward Assumption used in REINFORCE and Actor-Critic RL algorithms, combined with a response-level reward model, can provide an unbiased estimate of any unknown token-level rewards. This theorem offers both a theoretical guarantee and practical guidance for developing RL algorithms across a wide range of LLM applications—where token-level rewards are difficult, costly, or even impossible to define precisely, while response-level rewards are readily available. Such scenarios include