Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, Yatao Bian · 2025
Read Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization by Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, Yatao Bian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05812.txt.
Introduction
Qingyang Zhang and four co-authors show in 2025 that a language model can sharpen its reasoning with no labeled training data at all. Standard routes need marked solution steps, checked answers, or a trained scoring model, all of which cost time and money. Their method needs only raw questions.
The method, called EMPO in the work, steadily lowers the model's doubt about its own answers in a hidden meaning space. That pressure alone lifted a maths model's score from near thirty-one percent to near forty-eight percent on test sets. A general model also gained on a broad knowledge test. The team adds early tests and notes on why the trick works.
Catalog record
| CREATOR | Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, Yatao Bian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05812
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, Yatao Bian Abstract Existing methods to enhance the reasoning capability of large language models predominantly rely on supervised fine-tuning (SFT) followed by reinforcement learning (RL) on reasoning-specific data. These approaches critically depend on external supervisions–such as labeled reasoning traces, verified golden answers, or pre-trained reward models. In this work, we propose Entropy Minimized Policy Optimization ( EMPO ), which makes an early attempt at fully unsupervised LLM reasoning incentivization. By continuously minimizing the predictive entropy of LLMs on unlabeled questions in a latent semantic space, EMPO achieves competitive performance compared to supervised counterparts on both mathematical and free-form natural reasoning tasks. Specifically, without any supervised signals, EMPO boosts the accuracy of Qwen2.5-Math-7B Base from 30.7% to 48.1% on mathematical benchmarks and improves the accuracy of Qwen2.5-7B Base from 32.1% to 50.1% on MMLU-Pro. Primary experiments and analysis are also provided to interpret the effectiveness of EMPO . Code is available at this url . 1 Introduction Figure 1: Improvement of the proposed method on Qwen2.5-7B and Qwen2.5-7B-Math model. Large language models (LLMs) have demonstrated exceptional potential in challenging tasks such as mathematical reasoning [ 1 , 2 , 3 ] and code generation [ 4 ] . A prevailing paradigm for training reasoning LLMs involves firstly performing supervised fine-tuning (SFT) and then reinforcement learning (RL), or iterative combinations of both, applied to reasoning-specific datasets after pretraining [ 5 ] . Unfortunately, these methods typically depend on large-scale reasoning datasets with various forms of supervised information, such as human-labeled reasoning traces, verified golden answers, or an additional pre-trained reward model. As a consequence, endowing LLMs with powerful reasoning capability through human experts is becoming increasingly time-consuming and costly, which greatly limits the scalability and broader adoption of reasoning models. To mitigate this, previous work employs self-consistency to construct pseudo data and deploy supervised finetuning for better performance [ 6 ] . However, the performance improvement is limited and under risks of model collapse [ 7 ] . Recent advancements, such as the pioneering work PFPO [ 8 ] , frame the labeling of solutions as evaluation against test cases and then leverage self-consistency to generate pseudo test cases. Despite the promising results, the proposed method still necessitates supervision from instruction finetuning data and supervision signals from the frontier LLMs to initialize the RL process. Another more recent work [ 9 ] introduces a two-stage framework to construct self-rewarding reasoning models using self-generated data followed by RL. Despite the superior performance, the proposed method relies on a ground-truth verifier to obtain self-correction reasoning traces by rejection sampling. These approaches inspire our exploration of a critical open question: How can we incentivize LLM reasoning capacities in a fully unsupervised manner? Figure 2: Overview of the proposed method. (a) Previous method like PPO [ 10 ] or GRPO [ 11 ] typically relies on external supervised signals, e.g., a pretrained reward model or golden answers. (b) The proposed Entropy Minimized Policy Optimization ( EMPO ) samples a set of responses from the current policy model, and then builds semantic clusters according to their equivalence. By continuously minimizing the entropy at a meaning level, our method achieves competitive benchmark performance without any external supervision, i.e., rule-based reward, pre-defined test cases or an pre-trained reward model. Recent advanced DeepSeek-R1-Zero [ 12 ] demonstrates robust reasoning capabilities without dependency on SFT data. By directly initiating RL from the base model, DeepSeek-R1-Zero autonomously evolves sophisticated reasoning behaviors such as reflection and self-critic by exploring the reward signals provided by rule-based rewards. i.e., verified golden answers or an additional pre-trained reward model. Inspired by the success of DeepSeek-R1-Zero, our motivation is to devise a fully unsupervised approach for powerful reasoning capability. Specifically, we propose a novel reinforcement learning algorithm termed as Entropy Minimized Policy Optimization ( EMPO ), which incentivizes the reasoning capability of LLMs in a fully unsupervised manner by minimizing their predictive entropy in a latent semantic space. This method optimizes the model to favor reasoning traces yielding consistent answers, enhancing output reliability. The semantic entropy objective we propose to minimize is a well-established measurement of LLMs’ uncertainty, which extends beyond mathematical reasoning to free-form question-answering tasks. We further introduce entropy thresholding to filter unreliable reasoning traces, stabilizing the unsupervised training process. Experiments on various tasks including mathematical reasoning and free-form natural reasoning are conducted to validate the proposed method. Our contributions are summarized as follows: • We propose an effective and principled strategy called Entropy-Minimized Policy Optimization ( EMPO ) for incentivizing the reasoning capabilities of LLMs in a fully unsupervised manner. • We establish semantic entropy as a potent intrinsic reward signal for guiding LLM reasoning. Our analysis confirms a strong negative correlation between semantic entropy and model accuracy, validating its efficacy as a robust, unsupervised optimization objective that drives models towards generating more consistent and reliable outputs. • Experiments on both math reasoning tasks with deterministic golden answers and free-form natural reasoning tasks are conducted to validate the efficacy and versatility of EMPO . Additionally, we provide critical insights into EMPO ’s mechanism, demonstrating that its effectiveness stems from an enhanced ability to consistently select and prioritize strong, pre-existing reasoning pathways learned during pre-training, rather than instilling fundamentally new reasoning skills. This underscores EMPO ’s strength in efficiently eliciting and refining latent capabilities within base models. 2 Related Work Self-Supervised and Semi-Supervised Reasoning. To address the dependency on labeled data, several self-supervised and unsupervised methods have emerged. Huang et al. [ 6 ] propose a self-improvement framework where LLMs generate high-confidence answers using Chain-of-Thought (CoT) prompting and self-consistency, subsequently fine-tuning on these pseudo-labels. However, the performance gains are often limited, and there is a risk of model collapse, as noted in [ 7 ] . Recently, Patel et al. [ 13 ] apply self-improvement to web navigation tasks in WebArena, fine-tuning on synthetic data generated by the model itself. Li et al. [ 14 ] enhance long-context reasoning via SeaLong, sampling multiple outputs and optimizing with Minimum Bayes Risk. These methods, while reducing reliance on external labels, still involve supervised fine-tuning steps, contrasting with EMPO ’s fully unsupervised RL approach. A concurrent work, i.e., test-time reinforcement learning (TTRL) [ 15 ] directly obtains pseudo label by major voting and then conducts RL on test prompts at inference time, whereas our EMPO strictly maintains the separation between training and testing phases for ensuring that the model remains unexposed to any test prompts during training. Furthermore, while TTRL is currently limited to mathematical tasks, our approach is applicable to more general free-form reasoning tasks. Self-Rewarding and RL-based Reasoning. RL has become a prominent technique for enhancing LLM reasoning, often leveraging external or self-generated rewards. Yuan et al. [ 16 ] propose using the LLM itself via LLM-as-a-Judge prompting to provide rewards during training, reducing reliance on human feedback. Similarly, Xiong et al. [ 9 ] propose a two-stage self-rewarding framework for mathematical reasoning, generating data and applying RL with a ground-truth verifier for self-correction, achieving superior performance but requiring supervised signals. Jiao et al. [ 8 ] frame solution labeling as evaluation against test cases, yet