Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training
2026
Read Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17003.txt.
Introduction
Cui, Yang and Zhu propose Learning-Zone Energy, an online selector that focuses post-training on the model's learning frontier. Standard methods spread budgets evenly across prompts, wasting compute on mastered or unreachable samples. Their score fuses a difficulty anchor, outcome uncertainty, and pass-rate momentum into one value tied to gradient updates. A pruner with replay skips solved prompts while rechecking for forgetting. Using only 40 percent of data per step, it matches or beats baselines, with out-of-distribution gains on AIME25 and AMC23 and 36 percent lower FLOPs.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51102 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17003
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training Thanks: Equal contribution. Affiliation: Peng Cui, Boyao Yang 1 1 footnotemark: 1 , Jun Zhu Affiliation: Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center,Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing 100084, China Thanks: Corresponding author. Affiliation: Dept. of Automation, Tsinghua University, Beijing 100084, China [email protected], [email protected], [email protected] Abstract Reinforcement Learning (RL) post-training has emerged as the dominant paradigm for eliciting mathematical reasoning in Large Language Models (LLMs), yet prevailing techniques such as GRPO and DAPO distribute rollout and gradient budgets nearly uniformly across prompts, squandering compute on samples that are already mastered or remain far beyond the model’s current capability. To address this fundamental inefficiency, we propose Learning-Zone Energy ( LZE ), a theoretically grounded, fully online data selection framework that concentrates computation on the model’s active learning frontier . At its core, we define a closed-form Learning-Zone Energy Score that fuses three complementary signals—an initial-difficulty anchor, a normalized outcome-uncertainty term, and a pass-rate momentum—into a single scalar that is provably aligned with the expected magnitude of group-relative policy gradient updates. A forward pruner with replay further reduces wall-clock time cost by skipping rollout generation for persistently solved prompts while periodically checking for forgetting. Evaluated on Qwen-family models (1.5B–8B) across GSM8K, MATH and DAPO-MATH, our method retains only 40% of the training data per step yet matches or surpasses full-data baselines, with especially pronounced out-of-distribution gains on AIME25( + 45.9 % +45.9\% ) and AMC23( + 18.2 % +18.2\% ), alongside an estimated 36% reduction in training FLOPs. Our code is available at https://github.com/Stellaris167/LZE . 1 Introduction Reinforcement Learning (RL) with verifiable rewards has emerged as the dominant paradigm for improving mathematical reasoning in Large Language Models [ 5 , 22 ] . In modern group-based RL pipelines such as GRPO [ 27 , 2 ] and DAPO [ 44 ] , the model generates multiple rollouts for each prompt, aggregates rewards within each group, and performs policy updates accordingly. However, these methods treat prompt groups nearly uniformly during training, even though the groups are highly heterogeneous in their usefulness: some prompts are already solved reliably and therefore contribute little additional learning signal, whereas others remain far beyond the model’s current capability and produce almost no actionable gradient information. Consequently, a considerable fraction of the rollout and optimization budget is spent on prompt groups that are only weakly informative for policy improvement. To mitigate this inefficiency, prior work has explored curriculum-style training, offline filtering, and online rejection-based heuristics. Early work on curriculum learning and self-paced learning showed that training efficiency can benefit from organizing examples by difficulty [ 4 , 14 ] . In the context of LLM post-training, offline filtering and self-training style methods can remove obviously uninformative examples before optimization begins [ 30 , 9 ] . However, such strategies are inherently static and cannot adapt as the policy evolves and the effective difficulty of a prompt changes over time. Online rejection methods are more adaptive, yet they typically apply binary keep-or-drop decisions once a prompt becomes all-correct or all-incorrect [ 8 , 39 ] , making them insensitive to finer-grained differences in informativeness. A more recent online selection method [ 21 ] avoids redundant rollouts by predicting prompt solvability via a hidden Markov model, but requires maintaining a learned generative state model, inevitably incurring additional computational overhead. Therefore, existing approaches are often either static, overly rigid, or expensive. What is still missing is a lightweight online criterion that continuously adapts to the current policy, assigns prompt groups a graded notion of informativeness, and improves training efficiency without relying on heavy auxiliary machinery. To address this gap, we propose Learning-Zone Energy ( LZE ), a simple and fully online data selection framework for RL post-training. Our key idea is to prioritise prompt groups in the model’s current learning zone : groups that are neither already mastered nor completely out of reach, but actively capable of producing meaningful policy updates. Concretely, we define an Energy Score that combines three signals: an initial-difficulty anchor that preserves attention on historically challenging prompts, a normalized uncertainty term based on the current group pass rate that emphasises prompts near the decision boundary, and a momentum term that highlights groups on which the policy is actively changing. Beyond this practical intuition, we provide theoretical analysis for the design: in group-relative RL with Bernoulli rewards, the uncertainty term is aligned with the expected GRPO gradient variance under a standard fixed-baseline approximation (Theorem 1 ), thereby explaining why frontier prompt groups are especially informative. We further show that the momentum term corresponds to the output of a causal high-pass filter on the pass-rate sequence (Proposition 2 ), and that the score as a whole admits an interpretation as a sample-level attention mechanism. To further reduce training cost, we complement the backward scorer with a replay-based forward pruner that skips rollout generation for prompts that have remained solved over consecutive epochs, with periodic replay to detect forgetting. Our contributions are threefold: • A principled online data selection framework. The Learning-Zone Energy Score is a closed-form, per-step criterion that integrates difficulty history, outcome uncertainty, and pass-rate momentum into a single scalar provably aligned with the GRPO gradient variance. As it operates solely on group-level pass-rate statistics, LZE is compatible with any group-based RL post-training algorithm and requires no modification to the underlying optimizer. Retaining only 40% of prompts per training step, LZE matches and even surpasses full-data baselines across all evaluated configurations, with particularly pronounced out-of-distribution gains: up to +45.9\% on AIME 25 and +18.2\% on AMC 23. • Theoretical analysis of gradient informativeness in RL post-training. We prove that the Bernoulli variance 4p(1{-}p) governs the GRPO gradient variance up to a small-step-size approximation (Theorem 1 ), establish an EMA–convolution duality linking the momentum term to causal high-pass filtering of the pass-rate sequence (Proposition 2 ). • Compute efficiency via dual-stage selection. The backward selector and forward pruner together reduce theoretical training FLOPs by an estimated 36%, consistent with the empirical FLOPs measurements across all model and dataset configurations reported in Table 1 . Figure 1 : Overview of the proposed method. The backward scorer computes the Learning-Zone Energy Score for each prompt group at every training step, guiding Top- K selection for policy updates. The forward pruner tracks group pass rates at the epoch level and skips rollout generation for groups that have been stably solved, with replay providing a safety mechanism to detect forgetting. 2 Background We review the group-based RL post-training paradigm, introduce the notation used throughout, and identify the gradient-informativeness property that motivates our approach. RL post-training for mathematical reasoning. Given a prompt dataset \mathcal{D}=\{x_{i}\}_{i=1}^{N} and a policy \pi_{\theta} , RL post-training maximises the expected reward: \max_{\theta}\;\mathbb{E}_{x\sim\mathcal{D},\;y\sim\pi_{\theta}(\cdot|x)}\bigl[r(x,y)\bigr], (1) where r(x,y)\in\{0,1\} is a rule-based binary verifier that checks the correctness of response y for prompt x . This binary reward structure, standard in mathematical reasoning, makes the per-prompt pass rate a natural proxy for the policy’s current capability on that problem. Group Relative Policy Optimization (GRPO). GRPO [ 27 , 2 ] optimizes Eq. ( 1 ) without a learned value network by normalizing rewards within each sampled group. For prompt