Not only where, But when: Temporal Scheduling for RLVR

Jinghao Zhang, Ruilin Li, Feng Zhao, Jiaqi Wang · 2026

Read Not only where, But when: Temporal Scheduling for RLVR by Jinghao Zhang, Ruilin Li, Feng Zhao, Jiaqi Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25381.txt.

Introduction

Jinghao Zhang and colleagues argue that timing matters as much as place for learning signals in reinforcement learning with verifiable rewards. Current methods weight tokens or select tokens, but their criteria stay fixed throughout training, curbing healthy policy growth. The team adds temporal scheduling: early training stresses tokens tied to policy behaviors, then eases toward general optimization. Simple trajectory percentiles distinguish those behaviors and pair well with the schedule. Standard optimization sacrifices policy entropy when mixing varied behaviors, while temporal scheduling keeps evolution sound and lifts reasoning benchmarks.

Catalog record

CREATORJinghao Zhang, Ruilin Li, Feng Zhao, Jiaqi Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55687
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25381

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not only where, But when: Temporal Scheduling for RLVR Jinghao Zhang, Ruilin Li, Feng Zhao, Jiaqi Wang Abstract Reinforcement learning with verifiable rewards (RLVR) has become a core technique for post-training of Large Language Models (LLMs). While policy optimization is driven by all sampled tokens under a globally broadcast scalar reward, the heterogeneous policy behaviors exhibited along trajectories are largely overlooked without differentiation. Existing works address this by credit allocation, including token-level advantage reweighting, and selective token optimization, however, the allocation criterion are principally stagnant throughout training, limiting resilient policy evolution. In this work, we argue that when learning signals are scheduled can be as important as where they are allocated across tokens, and introduce the temporal dimension that scheduling the credit allocation criteria over the course of RLVR optimization. We find that prioritizing targeted tokens emphasized with specific policy behaviors, and gradually attenuating toward general optimization leads to more stable and efficient learning dynamics. Furthermore, we show that simple trajectory percentiles provide a natural perspective for distinguishing policy behaviors, and works effectively with temporal scheduling. Our analysis reveals that standard optimization substantially sacrifices policy entropy when simultaneously accommodating heterogeneous behaviors, whereas temporal scheduling yields healthier policy evolution dynamics. Experiments across mathematical and general reasoning benchmarks demonstrate consistent improvements, suggesting that temporal scheduling constitutes a promising optimization dimension. Figure 1: Performance and training dynamics on Qwen3-4B model. (a) Temporal scheduling shows stable training dynamics across both existing credit allocation method, e.g., entropy-based advantage reweighting, and the identified trajectory percentiles scheduling, i.e., TP-Schedule, that extends optimization from later to earlier tokens, consistently pushing the optimization plateau to a higher ceiling throughout training. (b) Temporal scheduling consistently outperforms vanilla GRPO and corresponding stagnant credit allocation baselines over representative reasoning benchmarks. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has emerged as a central paradigm for post-training of Large Language Models (LLMs), yielding substantial improvements in reasoning performance across a wide range of tasks ( 6 ; 4 ; 27 ; 24 ) . Unlike supervised fine-tuning, where dense next-token supervision provides explicit learning signals along the trajectory, RLVR typically broadcasts a single scaler reward across the full sequence, engaging all sampled tokens to participate equally in policy optimization. However, the heterogeneous policy behaviors exhibited along the trajectory are largely overlooked, such as reasoning scaffolding and answer convergence, and are optimized roughly without differentiation. To encourage effective optimization, existing works address this by introducing credit allocation, supposed to identify where the learning signals are distributed within the sampled response. Representative approaches include Process Reward Models (PRMs) that provide step-level feedbacks ( 33 ; 30 ) , while the heavy annotation cost and computation burden pose a significant scalability bottlenecks. More recent works instead focus on token-level signals derived from policy proxies for credit allocation, such as token entropy ( 3 ) , likelihood ( 5 ; 29 ) , and KL divergence ( 15 ; 10 ) . These methods typically identify specific policy behaviors, such as logic connection and reflection, with predefined allocation criteria, and reweight token advantages for preference in policy optimization, resulting in improved performance. Consistently, parallel studies further demonstrate that optimizing only a subset of critical tokens can outperform full-token optimization ( 25 ) , reinforcing the view that policy gradients accumulated uniformly from all sampled tokens are suboptimal, and could be more effective through more targeted optimization criteria. Despite progress, existing credit allocation methods are primarily restricted to stagnant allocation criteria throughout training, where once a token-level proxy is applied, the corresponding targeted policy behavior is continuously emphasized during the entire optimization. In view of this, the effectiveness of policy optimization is fundamentally determined by how well the adopted proxy in capturing the informative learning signals, while the behaviors outside the proxy identification could be suppressed for insufficient optimization. We thereby asking, instead of persistently pursuing effective proxies for directing policy optimization, are there other potential optimization dimensions that have been underexplored. From the perspective of training dynamics, we advocate that a single credit allocation criteria should not dominate the entire policy optimization process in practice, which is essentially different from their original design choice within individual sample. Building on this principle, we revisit RLVR optimization from temporal dimension and argue that when learning signals are scheduled throughout training can be as important as where they are allocated across sampled tokens. Our key idea is to schedule the allocation criteria over the course of optimization, which prioritizing targeted tokens that emphasized by corresponding allocation criteria, and gradually attenuating toward general optimization. We show that temporally scheduling the allocation criteria complements existing credit allocation methods, including both advantage reweighting, and sparse token optimization. Furthermore, we identify that simple trajectory percentiles serve as an effective criteria for scheduling policy optimization, supported by its natural perspective in distinguishing heterogeneous policy behaviors. Specifically, we schedule from later tokens for trustworthy continuation, and incrementally incorporate earlier tokens for reliable reasoning scaffolding. In addition, we reveal that the policy entropy is largely sacrificed in standard RLVR optimization for accommodating heterogeneous policy behaviors, whereas temporal scheduling mitigate this effect with healthier optimization dynamics. Experiments across model scales and RLVR algorithms demonstrate consistent improvements, suggesting a promising dimension for RLVR optimization. The contributions of this work can be summarized as follows: (1) We identify the temporal dimension beyond conventional stagnant credit allocation paradigm for RLVR optimization, and propose temporally scheduling the allocation criteria over the course of training, that prioritizing credit allocated tokens, and gradually attenuating toward general optimization. (2) Temporal scheduling complements existing credit allocation methods, including continuous advantage reweighting, and discrete sparse token optimization. We further identify simple trajectory percentiles as a natural perspective in distinguishing policy behaviors, which works effectively with temporal scheduling. (3) Experiments across models scales, RL algorithms, and credit allocation strategies show consistent improvement, and further analysis show that temporal scheduling substantially mitigates the sacrificed policy entropy induced by accommodating heterogeneous policy behaviors in RLVR optimization. Figure 2: Policy behavior analysis and corresponding training dynamics. (a) Histogram of sampled token frequency across different trajectory positions. The x-axis represents the union of all sampled tokens, sorted by TP-Score (Eq. 5 ) to approximate the most likely position of each token along the trajectory. While different trajectory position exhibit substantially different policy behavior with respective token distribution. (b) Entropy dynamics of tokens at different trajectory percentiles. Tokens from different positions exhibit different entropy profiles, while temporal scheduling consistently preserves higher entropy throughout training. (c)-(e) Textual visualization of trajectory positions. 2 Preliminaries Reinforcement learning with verifiable rewards (RLVR) has emerged as a common paradigm for post-training of large language models. Consider a language model parameterized by \pi_{\theta} . Given a prompt x sampled from data distribution \mathcal{D} , the policy generates a response y=(y_{1},\ldots,y_{T}) of length T . A verifier then assigns a global scalar reward r(x,y)\in\mathbb{R} that measures the quality of the sampled response. The RLVR objective is to maximize the expected reward J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta}(\cdot\mid x)}\left[r(x,y)\right]. (1) Group Relative Policy Optimization (GRPO). GRPO ( 21 ) simplfies the Proximal Policy Optimization (PPO) ( 20 ) by eliminating the critic model for advantage estimation. For each prompt, GRPO samples a group of response \{\textbf{y}_{i}\}_{i=1}^{G} and estimates the advantage of each response in a group-relative

Read the complete text