Hint-Guided Diversified Policy Optimization for LLM Reasoning

Zhiyu Cao, Kaixin Wu, Mingjie Zhong, Peifeng Li, Xiaobo Li, Can Ye, Qiaoming Zhu · 2026

Read Hint-Guided Diversified Policy Optimization for LLM Reasoning by Zhiyu Cao, Kaixin Wu, Mingjie Zhong, Peifeng Li, Xiaobo Li, Can Ye, Qiaoming Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03021.txt.

Introduction

Cao and co-authors propose HDPO, a training method that teaches language models to weigh several solutions before reasoning. Reinforcement learning with verifiable rewards improves reasoning, but its rewards check only final correctness. Nothing pushes a model to consider diverse approaches, as human solvers do.

HDPO makes the model list candidate solution outlines as hints first. It then selects the most reliable one and reasons from it. Training has two stages. A cold start teaches the structured propose-select-think format. A hint-guided reinforcement stage then rewards diverse and reliable solutions. Experiments show that HDPO boosts reasoning performance. It also raised the diversity of candidate solutions. Models got better at spotting which solution was reliable. The authors present the method as a step toward deliberative reasoning in language models. It adds an explicit choice point before the chain of thought begins.

Catalog record

CREATORZhiyu Cao, Kaixin Wu, Mingjie Zhong, Peifeng Li, Xiaobo Li, Can Ye, Qiaoming Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03021

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hint-Guided Diversified Policy Optimization for LLM Reasoning Zhiyu Cao, Kaixin Wu, Mingjie Zhong, Peifeng Li, Xiaobo Li, Can Ye, Qiaoming Zhu Abstract Recent developments in Large Language Models (LLMs) have showcased impressive reasoning capabilities, with Reinforcement Learning with Verifiable Rewards (RLVR) being a promising enhancement strategy. However, existing reward mechanisms are constrained to the outcome-level correctness and lack explicit signals to guide the model to consider diverse solutions. In contrast, human problem solving typically involves evaluating multiple potential approaches and selecting the most reliable solution, a cognitive process that current RLVR frameworks do not explicitly incentivize. Inspired by this, we propose Hint-Guided Diversified Policy Optimization (HDPO), allowing the model to first list all potential candidate solution outlines as hints and then select the most reliable one for further reasoning. HDPO comprises two stages of Cold Start for Structured Reasoning and Hint-Guided Diversified Reinforcement Learning to incentivize the model to generate diverse and reliable solutions following the “propose-select-think” trajectory. Experimental results show that HDPO effectively boosts LLM reasoning and enhances the diversity of candidate solutions as well as the LLM’s ability to identify reliable solutions. 1 Introduction Recent research has yielded significant progress in enhancing the reasoning capabilities of Large Language Models (LLMs), exemplified by OpenAI-o1 Jaech et al. 2024 , DeepSeek-R1 Guo et al. 2025 , and Kimi-1.5 Team et al. 2025 , particularly for complex mathematical problem solving. A pivotal development underpinning these improvements is the adoption of Reinforcement Learning with Verifiable Reward (RLVR) Liu et al. 2025b ; Yan et al. 2025 ; Zhang et al. 2025a as a core training paradigm. Within this framework, policy optimization algorithms are widely adopted to fine-tune LLMs, with Group Relative Policy Optimization (GRPO) Shao et al. 2024 serving as a prominent instantiation of this approach. Figure 1: Hit rate of HDPO and GRPO on Olympiad-Bench under different number of attempts. To ensure the credibility of complex reasoning, outcome-level rewards Cobbe et al. 2021 ; Shao et al. 2024 are employed in RLVR to assess the correctness of the entire reasoning trajectory. However, this approach does not take into account the rationality of the process, and its sparse rewards may lead to homogenization of rewards, failing to provide comprehensive supervision. Therefore, recent studies Zhang et al. 2025b ; Zhang et al. 2025a ; Luo et al. 2025b enhance the model’s reasoning ability by providing more detailed feedback, such as feedback in natural language and process supervision. Figure 2: The comparison of HDPO’s “propose-select-think” reasoning process ( left ) and the traditional reasoning process ( right ). More case studies can be found in Appendix K . We noticed that although the above methods have achieved notable success, they provide rewards based exclusively on correctness and thus fail to encourage the exploration of diverse solutions. Consequently, if the model gets trapped in a wrong solution, it may struggle to recover and identify the correct answer, reducing the fault tolerance of reasoning. As shown in Figure 1 , we evaluate the Hit@N accuracy of Qwen3-4B across varying numbers of sampling attempts. The results reveal that GRPO exhibits markedly lower performance when the number of attempts is limited, whereas HDPO consistently achieves superior accuracy even with fewer attempts. This suggests that the GRPO-trained model has insufficiently explored the space of diverse reasoning paths, which hinders its ability to generate correct answers within a limited number of trials. Ideally, the model should be able to explore a broader range of solutions and choose the most reliable solutions for further refinement. The key challenge lies in designing mechanisms that both incentivize the generation of diverse solutions and effectively assess their reliability. To address this issue, we propose Hint-Guided Diversified Policy Optimization (HDPO), a novel approach to enhance the diversity and reliability of LLM reasoning. HDPO is built upon the “propose-select-think” reasoning trajectory, effectively stimulating the model to reason about diverse potential solutions. Specifically, our training methodology instills this capability in two stages: we first employ supervised fine-tuning (SFT) in the cold start stage to establish the model’s proficiency in following the structured reasoning trajectory, followed by Reinforcement Learning (RL) to cultivate the policy model capable of generating diverse and reliable solutions. In the cold start phase, we employ an advanced LLM to construct such structured reasoning trajectories and apply filtering criteria based on correctness and reliability to ensure data quality. In the RL phase, we introduce diversity scheduling strategy and reliability reward to guide the model to actively explore all potential solutions. As illustrated in Figure 2 , the “propose-select-think” framework employed by HDPO offers a significant advantage over the traditional approach. While the conventional paradigm extends the given condition in a brute-force manner and arrives at an incorrect answer, HDPO promotes broader exploration of the solution space by generating multiple candidate solutions prior to engaging in detailed reasoning. This preliminary exploration enables the model to identify underlying symmetries, a critical insight that contributes to the correct answer. Crucially, HDPO advances existing paradigms through policy internalization and joint optimization . Unlike inference-time search methods Wang et al. 2023 ; Yao et al. 2023 that incur high latency via repeated sampling, HDPO internalizes the explore-then-exploit mechanism into the policy parameters via RLVR, enabling zero-overhead, single-pass reasoning. Furthermore, departing from accuracy-centric RLVR, it directly mitigates solution homogenization and unreliable selection by jointly optimizing a scheduled diversity reward and a confidence-based reliability reward. To summarize, our contributions are as follows: • We introduced HDPO to enhance LLM reasoning, a novel framework based on the “propose-select-think” structured reasoning process. • The proposed diverse scheduling strategy and reliability reward enable LLMs to actively explore diverse and reliable solutions. • Our extensive experimental results demonstrate that HDPO greatly boosts the model’s reasoning ability. 2 Related Work Chain-of-Thought for LLM Reasoning. Chain-of-thought Wei et al. 2022 reasoning has shown promising progress in recent years. Building on this idea, tree-of-thought (ToT) Yao et al. 2023 generalizes reasoning into structured sequences of thoughts and determines the next action through self-evaluation. Zhou et al. 2023 proposes least-to-most prompting by decomposing problems, enabling language models to solve problems that are more difficult than those in the prompts. Graph of Thoughts (GoT) Besta et al. 2024 models LLM reasoning as an arbitrary graph, where thoughts were represented as vertices and dependencies between thoughts are represented as edges. Unlike ToT/GoT, which require repeated sampling and heuristic scoring at inference, HDPO internalizes the propose-select-think paradigm into the policy weights via reward-shaped RL, yielding zero-overhead inference. Through structured reinforcement learning, the model learns to generate diverse candidate hints and autonomously select the most reliable trajectory in a single forward pass. Reinforcement Learning for LLM Reasoning. Recent advancements in reinforcement learning (RL) Shao et al. 2024 ; Yue et al. 2025 ; Liu et al. 2025a have made significant progress in enhancing complex reasoning, shifting from surface-level response generation to sophisticated problem-solving. Ouyang et al. 2022 utilize preference data to train reward models and apply Proximal Policy Optimization (PPO) Schulman et al. 2017 to perform reinforcement learning on LLMs for alignment with human preferences. Subsequently, numerous methods have employed reinforcement learning to align LLMs with human preferences Lee et al. 2024 ; Ryu et al. 2024 ; Xie et al. 2024 . DeepSeekMath

Read the complete text