How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors

Yifan Xu, Junren Chen, Yifan Chen · 2026

Read How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors by Yifan Xu, Junren Chen, Yifan Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08817.txt.

Introduction

Yifan Xu, Junren Chen, and Yifan Chen improve how language models learn to reason from checkable rewards. Sparse rewards over long reasoning chains make exploration hard, and training often collapses onto a narrow set of successful paths, raising single try accuracy without widening coverage. Their IMAX framework trains a pool of soft prefixes, small trainable prompts that each tilt the same base model toward a distinct style of reasoning rollout.

An information maximizing reward joins the verifiable reward to push the prefixes toward diverse, task relevant behaviors instead of noisy wandering. The method plugs into existing pipelines unchanged, and across three model sizes it lifts multi try success by clear margins. Reasoning training gains exploration by design at the start, not by accident.

Catalog record

CREATORYifan Xu, Junren Chen, Yifan Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08817

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors Yifan Xu, Junren Chen, Yifan Chen Abstract Reinforcement learning with verifiable rewards (RLVR) recently thrives in large language model (LLM) reasoning tasks. However, the reward sparsity and the long reasoning horizon make effective exploration challenging. In practice, this challenge manifests as the entropy collapse phenomenon, where RLVR improves single-rollout accuracy but fails to expand coverage on successful reasoning trajectories. Passive exploration techniques like entropy regularization tend to dismiss generation quality, resulting in noisy rollouts. In response to this issue, we propose an Information-Maximizing Augmented eXploration (IMAX) framework to train a pool of soft prefixes that reshapes the base model’s prior over reasoning trajectories. Rather than relying on RL to incentivize exploration on top of the base model, each prefix acts as a trainable control knob that induces a distinct rollout distribution from the same backbone model. To encourage discovery of diverse and task-relevant reasoning behaviors, we derive an Information Maximization (InfoMax) reward to complement the verifiable rewards for RL training. IMAX is in general algorithm-agnostic and can be seamlessly integrated into existing RLVR pipelines. Experiment results have shown that across three backbone scales, IMAX consistently improves reasoning performance over standard RLVR, with gains up to 11.60% in Pass@4 and 10.57% in Avg@4. 1 Introduction Large language models (LLMs) have shown notable capabilities in complex reasoning tasks, including mathematical problem solving [ 18 , 14 , 42 , 9 ] , automated theorem proving [ 45 , 40 , 4 ] , multi-step planning [ 46 , 28 ] , etc. These advances in complex reasoning capabilities of LLMs are attributed to large-scale post-training procedures [ 3 , 20 , 41 ] , among which reinforcement learning from verifiable feedback (RLVR) has shown great potential and garnered widespread interest [ 14 , 47 ] . However, despite its empirical success, RLVR faces critical challenges. Prior studies have reported an entropy collapse phenomenon [ 10 , 47 ] , where the entropy of the LLM output distribution drops during training; output diversity is thereby reduced. In particular, some works observe that although the Pass@ 1 accuracy of the LLM significantly improves after RLVR, the Pass@ K performance may deteriorate compared to the base model [ 48 , 7 ] . These phenomena collectively suggest that current RLVR paradigm suffers from limited exploration , which in turn influences reasoning capabilities. Existing approaches for exploration enhancement of RLVR typically intervene through the RL objective, reward design, or sampling procedure, aiming to encourage broader exploration on top of the base model prior [ 13 , 49 , 16 , 29 ] . However, recent studies suggest that RL is strongly shaped by the base model’s distribution [ 39 , 48 , 19 ] . This observation points to a complementary direction: rather than only modifying the RL process itself, one can also steer the model prior that RLVR builds on. In parallel, prompt-based adaptation has shown that learned prompts can effectively steer pretrained models by changing their generation context, thereby eliciting different behaviors from the same underlying model for a single prompt [ 23 , 21 , 25 , 27 ] . This motivates us to shift the optimization target from the LLM parameters to prompt tuning: rather than relying on RL to enhance exploration through samples drawn from the base model prior, we optimize lightweight conditioning variables that directly control the sampling distribution and open different regions of the model’s reasoning space for each prompt. Once learned, these prefixes can serve as reusable control handles for inference or subsequent policy training, producing rollouts that are both more diverse and more likely to contain high-quality reasoning paths. Figure 1 : Overview of the IMAX framework. For a reasoning problem, a frozen LLM is conditioned on a learned soft-prefix pool, where different prefixes induce different reasoning priors and lead to diverse candidate trajectories, such as algebraic, diagrammatic, code-like, or symbolic solutions. Training updates only the prefix priors using RLVR correctness rewards and the InfoMax reward, encouraging prefixes to produce high-quality and non-redundant reasoning trajectories. Therefore, we introduce an Information-Maximizing Augmented eXploration (IMAX) method on soft prompt tuning. The overall framework for IMAX is shown in Figure 1 . We first initialize a pool of soft prompts, each soft prompt acts as a lightweight conditioning handle: when prepended to the same frozen backbone model, it shifts the rollout distribution toward a different region of the model’s reasoning space. We then train these prompts with RLVR so that the induced priors favor correct reasoning, meanwhile adding an intrinsic Information Maximization (InfoMax) reward to prevent the prompt pool from collapsing into redundant ones. Concretely, we maximize the mutual information (MI) between the prompt identity and the generated trajectory. The resulting objective encourages prompts to induce distinct trajectories for different soft prompts, yielding a lightweight method for improving both diversity and quality during inference. And to encourage the soft prompts to explore in semantically-meaningful regions, we initialize soft prompts from textual prompts that encode different reasoning strategies. In summary, our main contributions in this paper are threefold: 1. We introduce a lightweight RLVR paradigm for learning a soft prompt pool that reshapes the prior of a frozen reasoning model without updating the full policy. 2. We propose an InfoMax reward that encourages different prompts to induce distinguishable, task-relevant reasoning trajectories, preventing collapse to redundant prefixes. 3. We conduct extensive experiments on challenging mathematical reasoning benchmarks, verifying that the learned prompt pool improves inference diversity and performance over existing baselines. 2 Related Works We briefly introduce the related topics in this section, and provide a detailed discussion in Appendix A . Reinforcement learning from verifiable rewards. RLVR has become a prominent approach for improving LLM reasoning with verifiable feedback, with progress highlighted by OpenAI o1 [ 18 ] , DeepSeek-R1 [ 14 ] , and Group Relative Policy Optimization (GRPO) from DeepSeekMath [ 30 ] . These advances have sparked growing research on RLVR for LLM reasoning [ 47 , 51 , 34 ] . However, RLVR still faces key challenges, including the reported “entropy collapse” [ 10 , 36 ] and debates over whether it truly improves reasoning or mainly increases confidence within the original distribution, potentially driving models toward determinism rather than better exploration [ 48 , 54 ] . Exploration in RLVR. Exploration remains a central challenge in RLVR. Recent studies show that effective exploration requires preserving informative tokens, not simply increasing randomness [ 10 , 36 , 17 , 12 , 6 ] . Existing methods address this exploration issue by modifying the optimization objective [ 47 , 7 ] , adding exploration or diversity bonuses [ 13 , 49 , 16 ] , or intervening in rollouts through entropy-targeted tokens [ 52 , 1 ] . A concurrent work [ 29 ] also uses mutual information to improve RLVR exploration. However, our method differs fundamentally in both objective and algorithm design: IMAX optimizes in the soft-prompt space instead of the full model parameter space, and we also initialize prefixes from semantically meaningful prompts that encode distinct reasoning strategies. A detailed comparison of the two methods is provided in Appendix A . Soft

Read the complete text