Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization

Jiashu Yao, Heyan Huang, Chuwei Luo, Daiqing Wu, Zeming Liu, Yuhang Guo, Yangyang Kang · 2026

Read Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization by Jiashu Yao, Heyan Huang, Chuwei Luo, Daiqing Wu, Zeming Liu, Yuhang Guo, Yangyang Kang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11510.txt.

Introduction

Yao and four colleagues split a language model's policy into two modes to spur wider exploration without losing accuracy. A normal mode aims at correct answers, while a high-entropy mode favors exploration, and the two learn together under shared parameters. Tests across model sizes show the split beats standard entropy-guided baselines on general and creative tasks. The high-entropy mode yields distinct behavior patterns that supply fresh learning signals.

Catalog record

CREATORJiashu Yao, Heyan Huang, Chuwei Luo, Daiqing Wu, Zeming Liu, Yuhang Guo, Yangyang Kang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44740
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization Jiashu Yao, Heyan Huang, Chuwei Luo, Daiqing Wu, Zeming Liu, Yuhang Guo, Yangyang Kang Abstract To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals. 1 Introduction The application of reinforcement learning (RL) on large language models (LLMs) 24 ; 26 ; 16 has demonstrated significant success, most notably in the emergence of the o1- and R1-like large reasoning models (LRMs) 7 ; 22 ; 12 ; 25 . These LRMs exhibit sophisticated reasoning capacity, particularly in domains requiring complex problem-solving and mathematical proficiency. Building on the success of RL for LLMs, recent research has drawn inspirations from entropy-regularized RL 50 ; 20 ; 8 to develop adapted RL frameworks 3 ; 40 ; 29 , aiming to incentivize high-entropy rollouts for more effective exploration. These methods have proven successful in mitigating entropy collapse during training, ultimately enhancing the final accuracy and reasoning robustness. Figure 1: (a) Entropy-guided RL can leads to accuracy drop when prefering high entropy. (b) Unlike previous work, Policy Split innovatively bifurcates the policy into normal and high-entropy modes. (c) Through dual-mode entropy regularization in Policy Split, two modes obtain a larger KL divergence and exhibit distinct entropy behaviors. However, a fundamental conflict between exploration and exploitation persists in entropy-guided RL for LLMs. Recent studies suggest that performance gains are often strongly correlated with entropy reduction 42 ; 4 ; 6 ; 45 , implying that incentivizing higher entropy may inadvertently degrade accuracy. More directly, recent paper 11 demonstrates that naive entropy regularization can cause outputs to degenerate into incoherent high-entropy sequences. As illustrated in Figure 1 (a), this performance degradation is particularly acute during the continual post-training of low-entropy LRMs. Since these models already possess robust capabilities, forcing higher entropy tends to undermine their established proficiency. To maintain high accuracy while encouraging exploration during the continual post-training of LRMs, we adopt a simple intuition, that we can instantiate two policies within one model. One policy is optimized for accuracy, while the other prioritizes high entropy, and these two policies benefit from each other’s experiences. Specifically, we propose Policy Split, a novel paradigm for entropy-guided LRM reinforcement learning. In Policy Split, the policy is firstly bifurcated into normal mode policy and high-entropy mode policy, where the latter one is identified with an additional high-entropy system prompt. During training, as is shown in Figure 1 (b), the normal mode is trained towards maximized correctness the same as original GRPO-like RL, while the high entropy mode policy additionally adds a high-entropy preference to encourage exploration. Inter-policy information sharing is conducted between two policies, enabling the normal policy to learn from the newly discovered correct rollouts, and encouraging the high entropy policy to explore more while maintaining an acceptable accuracy. Experiments validate that Policy Split manages to train two policies of different entropy behaviors out of one model, as is shown in Figure 1 (c). The policy of normal mode exhibits a conservative entropy change to maintain accuracy, while the policy of high entropy mode aggressively explores more possible solutions, leading to a progressively larger divergence between the two policies. Through comprehensive evaluation across diverse general and creative tasks, we show that Policy Split obtains higher accuracy while significantly revives the entropy through the post-training of a low-entropy LRM, consistently outperforming strong baselines. Furthermore, we conduct a detailed analysis on the dual-mode exploration of Policy Split, verifying its capability to generate distinct and unique responses which achieve higher Best-of-N scores. We also show that such capability exhibits certain generalizability given different unseen prompts. Our contributions are listed as follows. • We introduce Policy Split, a novel RL framework that enables a dual-mode (normal mode and high entropy mode) policy bifurcation and entropy regularization, where two modes pursue different optimization targets while maintaining a collaborative learning synergy. • Through extensive experiments on varying tasks and different model sizes, we show that Policy Split obtains significant accuracy and creativity improvement on strong reasoning models, and successfully revives their entropy. • We conduct a detailed analysis of Policy Split’s dual-mode exploration, demonstrating its capacity to unlock unique responses distinct to the vanilla sampling for training by employing generalizable prompting control. 2 Related Work Entropy-Guided LLM Reinforcement Existing research shows that entropy plays an important role in the reinforcement learning (RL) of LLMs 42 ; 4 . One line of work applies entropy or its proxy by assigning weight on different tokens or sequences, including weighting more on high entropy tokens 35 ; 4 ; 32 ; 33 , weighting more on low entropy tokens 2 ; 47 , or weighting more on medium entropy tokens 46 . Another line of work, inspired by the entropy regularization from RL methods before the era of large language models (LLMs) 50 ; 20 ; 8 , proposes to directly maximize entropy for more effective exploration 3 ; 40 ; 29 , or directly minimize entropy for unsupervised accuracy improvement 6 ; 45 . Policy Split continues the work of entropy maximization, and makes significant methodology innovation by conducting dual-mode policy bifurcation and dual-mode entropy regularization. It applies distinct entropy preferences to two modes accordingly, decoupling the pressure for accuracy from the need for high-entropy exploration. Generation Modes of LLMs Generally, LLMs can switch modes of generation styles 30 ; 1 with its inherent role-playing and personalization capability 21 ; 48 ; 36 or with additional training 27 ; 34 ; 13 . More recently, building on foundation models which support multiple reasoning modes 38 ; 15 ; 43 , adaptive reasoning mode switching has been proposed to guide LRMs to manually or automatically adopt different reasoning efforts 49 ; 44 ; 17 ; 10 ; 31 . Policy Split extends this line of work by training modes differentiated by their policy entropy rather than by linguistic styles or reasoning efforts. Furthermore, Policy Split bifurcates the model not only to enhance inference-time utility but also to assist entropy-guided training process. 3 Method Figure 2: The training framework of Policy Split. We first introduce the dual-mode implementation within one model with prompting, then formulate the dual-mode entropy regularization for policy training, and finally describe the overall algorithm. 3.1 High Entropy Mode Mode switching within one LLM is typically implemented with prompts. For simplicity and generalizability, we add a natural language instruction as the system prompt to identify a high-entropy mode. The system prompt is shown as follows. You are now in the "High Entropy" mode. Your primary directive is to explore a unique thought

Read the complete text