Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
Chanuk Lee, Sangwoo Park, Minki Kang, Sung Ju Hwang, [0.5em] KAIST · 2026
Read Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR by Chanuk Lee, Sangwoo Park, Minki Kang, Sung Ju Hwang, [0.5em] KAIST (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15726.txt.
Introduction
Chanuk Lee and three colleagues present NudgeRL, a framework for structured, diversity-driven exploration in RLVR. RLVR improves math reasoning but stalls on exploration: policies only improve on paths already sampled. Strategy Nudging conditions each rollout on a strategy context, without asking its maker to solve the task. Advantage splits into inter- and intra-context terms, plus a correction that moves behaviors to the base policy. On five math benchmarks, NudgeRL with 8 rollouts matches the top GRPO baseline using 32 rollouts, with fewer tokens and less training compute.
Catalog record
| CREATOR | Chanuk Lee, Sangwoo Park, Minki Kang, Sung Ju Hwang, [0.5em] KAIST |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74021 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15726
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR Chanuk Lee, Sangwoo Park, Minki Kang, Sung Ju Hwang, [0.5em] KAIST Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models. However, its effectiveness is fundamentally limited by exploration: the policy can only improve on trajectories it has already sampled. While increasing the number of rollouts alleviates this issue, such brute-force scaling is computationally expensive, and existing approaches that modify the optimization objective provide limited control over what is explored. In this work, we propose NudgeRL , a framework for structured and diversity-driven exploration in RLVR. Our approach introduces Strategy Nudging , which conditions each rollout on lightweight, strategy-level contexts to induce diverse reasoning trajectories without relying on expensive oracle supervision. To effectively learn from such structured exploration, we further propose a unified objective, which decomposes the reward signal into inter- and intra-context components and incorporates a distillation objective to transfer discovered behaviors back to the base policy. Empirically, NudgeRL outperforms standard GRPO with up to 8 × \times larger rollout budgets, while outperforming oracle-guided RL baseline on average across five challenging math benchmarks. These results demonstrate that structured, context-driven exploration can serve as an efficient and scalable alternative to both brute-force rollout scaling and feasibility-oriented methods based on privileged information. Our code is available at https://github.com/tally0818/NudgeRL . * * footnotetext: Equal contribution 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for improving the reasoning capabilities of large language models (LLMs) [ 20 , 7 ] . By leveraging verifiable rewards, methods such as Group-Relative Policy Optimization (GRPO) [ 18 ] enable scalable post-training without requiring dense supervision. This paradigm has been successfully applied across a wide range of domains. Despite its success, RLVR remains fundamentally limited by its ability to explore the space of reasoning trajectories. A natural approach is to scale the number of sampled rollouts, which increases the probability of discovering rare trajectories [ 5 ] . However, such brute-force scaling quickly becomes computationally prohibitive, motivating alternative approaches that improve exploration efficiency. Figure 1 : Concept: Improving exploration diversity through Strategy Nudging. (a) Naive sampling methods (e.g., GRPO) often collapse to a dominant reasoning mode, limiting the exploration of the reasoning space. (b) NudgeRL introduces Strategy Nudging, which appends lightweight strategy to the input, forcing the model to traverse diverse reasoning modes. (c) As a result, Strategy Nudging significantly increases the number of distinct reasoning approaches discovered compared to the baseline, effectively mitigating the exploration bottleneck. Additional details are in Appendix B Recent work has sought to address this limitation by modifying the optimization objective, for example through entropy regularization or decoupled clipping [ 26 , 24 ] . While these methods encourage broader exploration at the distribution level, they provide limited control over what is explored, and often fail to ensure coverage of semantically meaningful reasoning strategies. Another line of work leverages privileged information , such as oracle solutions or intermediate reasoning steps, to improve the feasibility of discovering correct trajectories [ 27 , 16 , 8 , 19 ] . Although effective, these approaches are primarily feasibility-oriented and rely on strong supervision signals that are expensive to obtain and difficult to scale. Moreover, by guiding the policy toward a narrow set of predefined successful trajectories, they may limit exploration diversity and hinder the discovery of alternative reasoning strategies [ 25 , 23 ] . In this work, we address the exploration bottleneck by explicitly structuring the reasoning space in a scalable manner. We propose NudgeRL , a framework that introduces Strategy Nudging during the exploration phase. Instead of relying on expensive oracle data, Strategy Nudging appends lightweight, heuristic text prompts (e.g., specific strategies for math problems or reasoning keywords) to the input. This deliberately forces the model to traverse distinct, diverse reasoning modes that it might otherwise ignore under purely naive sampling. However, learning from such context-conditioned exploration introduces new challenges. Since rollouts are generated under different context-conditioned prompts, the samples are naturally partitioned into multiple distinct groups, where reward variation reflects both the intrinsic trajectory quality and context-specific biases, making standard group-wise advantage estimation unreliable. Furthermore, context forcing creates a mismatch between how trajectories are sampled and how the policy is finally used at inference time. Without intervention, improvements discovered under context-forced exploration may not transfer directly to the base policy. To address these challenges, we further introduce (i) an Inter-Intra group advantage to enable meaningful credit assignment across context-induced groups, and (ii) a distillation-augmented objective that explicitly transfers effective behaviors discovered during context-forced exploration back to the base policy. Our approach enables structured and diversity-driven exploration while remaining fully compatible with standard RLVR pipelines. Empirically, NudgeRL achieves performance surpassing GRPO even when GRPO is given an 8\times larger rollout budget, while outperforming oracle-guided baselines. This demonstrates that scalable, diversity-oriented exploration can serve as an effective alternative to both brute-force rollout scaling and feasibility-driven privileged information. 2 Preliminaries 2.1 Group-Relative Policy Optimization (GRPO) We consider an empirical distribution of prompts x_{0}\in\mathcal{D} . For each prompt x_{0} , a policy \pi_{\theta} generates a group of N rollouts \{y_{i}\}_{i=1}^{N} , where each rollout is sampled as y_{i}\sim\pi_{\theta}(\cdot\mid x_{0}) . Each rollout is evaluated by a verifiable reward function R(x_{0},y_{i})\in\{0,1\} . Unlike standard PPO [ 17 ] , which typically estimates advantages using a learned value function, GRPO [ 18 ] derives advantages from group-wise rewards. For rollouts sampled from the same prompt x_{0} , let r_{i}=R(x_{0},y_{i}) denote the reward of rollout y_{i} . The group-wise advantage is then defined as: \hat{A}_{i}=\frac{r_{i}-\mu}{\sigma+\delta}, (1) where \mu and \sigma are the reward mean and standard deviation within the group, and \delta>0 is used for numerical stability. This yields a relative advantage estimate without training a value function. The policy is then optimized with a PPO-style clipped objective: \mathcal{L}_{\textrm{GRPO}}(\theta)=-\mathbb{E}_{y\sim\pi_{\theta}(\cdot|x_{0})}\left[\min\big(r_{0}\hat{A},\mathrm{clip}(r_{0},1-\epsilon,1+\epsilon)\hat{A}\big)\right],\ \text{where}\ r_{0}=\frac{\pi_{\theta}(y|x_{0})}{\pi_{\textrm{old}}(y|x_{0})}. (2) Thus, GRPO retains PPO’s clipped objective while using group-relative advantages. 2.2 Motivation: From Exploration to Performance Gain To understand why exploration is a fundamental bottleneck in RLVR, we look beyond trajectory-level rewards and examine how the probability mass of generated tokens shifts during training. 5 characterizes the expected one-step performance improvement ( \Delta Q_{\text{pos}} ) in RLVR as: \Delta Q_{\text{pos}}=\frac{\eta}{N}\big[(1-S_{R})Q_{\text{neg}}A_{2}+S_{R}Q_{\text{pos}}B_{2}+S_{R}(Q_{\text{pos}}U_{\text{neg},2}-Q_{\text{neg}}U_{\text{pos},2})\big], (3) where Q_{\text{pos}} and Q_{\text{neg}}=1-Q_{\text{pos}} denote the total probability mass of correct and incorrect tokens, \eta is the learning rate, and N is the number of rollouts. A_{2} and B_{2} are the second moments of sampled correct and incorrect tokens, while U_{\text{pos},2} and U_{\text{neg},2} are those of unsampled correct and incorrect tokens. S_{R}\in[0,1] represents the net reward contribution from sampled tokens. Since S_{R}\in[0,1] , the first two terms in Equation 3 are non-negative and drive learning forward. The third term, however, acts as a potential penalty. Because incorrect tokens typically dominate the probability mass ( Q_{\text{neg}}\gg Q_{\text{pos}} ), a large U_{\text{pos},2} , meaning the model has significant probability mass on correct trajectories that it simply fails to explore , creates a dominant negative force that hinders performance gain. Therefore, the core bottleneck of RLVR lies in the unexplored correct regions. Limitations of rollout scaling. To mitigate this penalty, a naive solution is to increase the rollout size N . 5