SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs

Chanuk Lee, Minki Kang, Sung Ju Hwang · 2026

Read SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs by Chanuk Lee, Minki Kang, Sung Ju Hwang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18864.txt.

Introduction

Chanuk Lee, Minki Kang and Sung Ju Hwang ask why reinforcement learning with verifiable rewards sharpens a language model's best answer but rarely widens its reasoning paths. They trace the limit to reverse-KL regularization, which anchors training to a reference model and suppresses new modes of thought. Dropping or swapping that penalty creates reward hacking or wasted probability. Their SAGE framework instead reshapes the anchor itself through a guide function, and tests on hard mathematics benchmarks show gains in both single-answer accuracy and broader coverage.

Catalog record

CREATORChanuk Lee, Minki Kang, Sung Ju Hwang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71738
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18864

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs Chanuk Lee, Minki Kang, Sung Ju Hwang Abstract Recent studies observe that reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1 on reasoning tasks, yet often fails to yield comparable gains in pass@k, raising the question of whether RLVR genuinely enables large language models to acquire novel reasoning abilities or merely enhances the efficiency of sampling reasoning modes already present in the base model. Prior analyses largely support the latter view, attributing this limitation to structural properties of standard RLVR objectives that result in insufficient exploration pressure. In this work, we argue that a central structural constraint arises from reverse-KL regularization, which stabilizes training but inherently anchors the policy to the reference distribution, thereby suppressing the emergence of alternative reasoning modes. However, we show that neither removing the KL term nor replacing it with forward-KL provides a satisfactory solution, as both disrupt the efficiency–coverage trade-off by either inducing reward hacking or allocating probability mass to off-target regions. To resolve this tension, we propose SAGE , a principled framework that enables controllable empirical support expansion by reshaping the reverse-KL anchor distribution itself through a guide function q ⁡ ( x , y ) q(x,y) , achieving consistent improvements in both pass@1 and pass@k across challenging mathematical reasoning benchmarks. Our code is available at https://github.com/tally0818/SAGE. 1 Introduction Large language models (LLMs) have achieved substantial progress on complex reasoning tasks, driven by advances in supervised post-training and reinforcement learning-based alignment. Among these approaches, reinforcement learning with verifiable rewards (RLVR) has emerged as a practical and effective framework for improving multi-step reasoning in domains such as mathematical problem solving and code generation ( 19 ; 27 ; 38 ) . By exploiting verifiable reward signals, RLVR enables scalable optimization of reasoning trajectories without reliance on human preference data. Despite these successes, recent analyses ( 39 ; 32 ) demonstrate that RLVR predominantly reinforces a narrow subset of a model’s pre-existing reasoning modes. During training, the policy frequently collapses onto a small collection of familiar solution paths, leading to support shrinkage. This mode-sharpening behavior highlights a fundamental need for effective exploration strategies in RLVR. Subsequent work ( 11 ) further indicates that KL-regularized RL can be suboptimal not only in terms of exploration but also exploitation. While 11 propose reward shaping as a partial remedy for the exploitation bias, the design of mechanisms that enable systematic and sustained exploration in RLVR remains an open challenge. Figure 1 : Conceptual illustration of SAGE. Left: Standard reverse-KL RLVR with the original anchor leads to mode collapse, failing to cover valid but low-density regions. Middle: SAGE introduces a multiplicative guide function q that forms a shaped anchor q\cdot\pi_{\text{ref}} to guide exploration, enabling the recovery of underexplored yet reward-compatible reasoning modes. Right: Performance comparison on math reasoning tasks (AIME, AMC23, MATH-500) shows that applying SAGE (blue) consistently outperform baselines in both Pass@1 and Pass@256 metrics across three different RL algorithms with Qwen-2.5-7B-Math model. See Section 5 for more details. To address this exploration challenge, a broad class of heuristic approaches has been proposed, with a dominant line of work focusing on modifying the KL regularization term. Originally introduced to constrain excessive deviation from the reference model, the KL penalty often induces distributional sharpening and suppresses low-support trajectories. Recent studies have explored eliminating the KL penalty altogether ( 38 ; 37 ; 22 ) , replacing reverse KL with forward-KL ( 10 ) , generalizing to alternative f -divergences ( 21 ) , or applying annealed KL schedules ( 3 ) . In contrast, our work stems from a key insight: both discarding the reverse-KL term and replacing it with forward-KL lead to suboptimal exploration dynamics . The reverse-KL divergence—often viewed as suppressing exploration—serves as a critical stabilizing anchor that prevents overfitting to sparse reward signals. While substituting it with forward-KL divergence can formally enlarge the support, it frequently allocates probability mass to reward-irrelevant regions, yielding limited improvements in pass@1. This leads to the following question: Can we retain the stabilizing benefits of reverse-KL while explicitly enabling controlled empirical support expansion? To address this question, we introduce Shaping Anchors for Guided Exploration ( SAGE ), a framework that constructs a guide function q(x,y) to guide the policy toward under-explored or previously inaccessible reasoning modes by reshaping the distribution toward which the KL term anchors the policy . We show that q can be instantiated using lightweight intrinsic signals that naturally arise during LLM reasoning, such as entropy and surprisal. Depending on how these signals are aggregated, SAGE yields several practical variants, including random exploration, token-level exploration, and branch-level exploration. Although these variants differ in implementation, they share a unifying principle: guided exploration is achieved through a modification of the KL anchor. Figure 1 provides a conceptual illustration of how these variants instantiate a shared exploration mechanism. We evaluated multiple instantiations of SAGE on the AIME, AMC23, and MATH-500 datasets and observed consistent improvements in both pass@1 and pass@k across diverse RLVR settings, demonstrating that principled exploration is compatible with reverse-KL-based regularization. Our main contributions are as follows: • We analyze the limitations of existing exploration strategies in reverse-KL RLVR and show that controllable mode expansion can be achieved by leveraging , rather than weakening, the reverse-KL anchor. • We propose SAGE , a principled framework for constructing exploration-enhanced anchor distributions that promote reasoning-mode expansion while preserving stability. • We instantiate SAGE using intrinsic reasoning signals, such as surprisal and entropy, yielding a unified family of practical exploration mechanisms. • We demonstrate through extensive experiments that SAGE-style variants consistently improve reasoning accuracy and exploration capability on mathematical reasoning benchmarks. 2 Related Works RLVR and Mode Collapse. Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective framework for improving multi-step reasoning in large language models, achieving strong empirical performance on tasks such as mathematical problem solving and code generation ( 19 ; 27 ; 38 ) . Despite these successes, a growing body of evidence indicates that RLVR predominantly induces mode sharpening rather than mode discovery . Recent analyses report rapid entropy collapse, support saturation, and even support shrinkage, suggesting that standard RLVR pipelines tend to reinforce a narrow subset of pre-existing reasoning modes ( 39 ; 32 ) . These findings highlight that the core challenge in RLVR lies in exploration. Weakening or Removing the KL Anchor. A prominent line of work attempts to alleviate mode collapse by modifying the KL regularization that anchors the policy to a reference model. Representative approaches include removing the KL penalty ( 38 ; 37 ; 22 ) , replacing reverse-KL with forward-KL ( 10 ) , generalizing to alternative f -divergences ( 21 ) , or applying annealed KL schedules ( 3 ) . While these methods can enlarge the effective support and improve coverage-oriented metrics such as pass@k, weakening the anchoring constraint often destabilizes optimization, leading to reward hacking or degraded pass@1 performance. These trade-offs suggest that simply relaxing the KL anchor does not provide a principled solution to the exploration-exploitation dilemma in RLVR. Exploration via Sampling and Objective Heuristics. Another line of work improves exploration while retaining the standard reverse-KL anchor. These approaches include modifying sampling dynamics (e.g., temperature tuning

Read the complete text