The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, Hao Peng Abstract Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs. We show that this simple objective alone, without any labeled data, can substantially improve large language models’ (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy similarly to instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL : reinforcement learning with negative entropy as the only reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce entropy without any training data or parameter updates. On Qwen-7B, EM-RL , without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO [ 68 ] and RLOO [ 1 ] that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark [ 78 ] , while being 3x more efficient than self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates. 1 1 1 Code: https://github.com/shivamag125/EM_PT 1 Introduction Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs without any labeled data or external supervision [ 23 , 60 , 25 ] . It operates on a key assumption and a simple intuition: if a model is reasonably capable, it is more likely to be correct when it is confident [ 56 ] . Modern large language models (LLMs) are pretrained on massive data and often undergo a dedicated annealing phase to enhance downstream task performance [ 59 , 3 , 92 , 77 ] . It should be, therefore, fairly reasonable to treat them as capable models even before any supervised finetuning (SFT) or reinforcement learning (RL) [ 90 ] . This naturally invites the questions: Can entropy minimization alone improve their performance? And how far can they go without any labeled data? This work answers both: yes, and surprisingly far, at least for reasoning tasks. We present a systematic exploration of post-pretraining adaptation and inference-time scaling of LLMs, using entropy minimization alone without any labeled data. We propose three novel methods, each aligned with an established post-pretraining stage. (1) Unsupervised finetuning by directly minimizing token-level entropy ( EM-FT ) mirrors SFT and minimizes a token level loss, on unlabeled outputs sampled from the model conditioning on the input prompts [ 46 ] . We find that EM-FT achieves surprisingly strong performance on math and coding tasks, and can even outperform labeled GRPO and RLOO on LeetCode [ 26 ] (coding) and Minerva [ 42 ] (math). (2) Reinforcement learning with a negative entropy reward ( EM-RL ) uses a reward signal based solely on entropy: the negative sum of token-level entropy across a rollout, adjusted by a constant baseline. This is analogous to the REINFORCE algorithm [ 76 , 1 ] but with entropy as the only supervision without any labeled data. We find that without any labeled data EM-RL can achieve competitive performance to RLOO and GRPO on most math and coding tasks while outperforming it on LeetCode, Minerva and AMC (math) [ 43 ] . (3) Inference-time scaling through entropy minimization (EM-INF) optimizes the logits during each decoding step to reduce the entropy of the LLM’s distribution without any parameter update. We find that EM-INF works best in complex tasks with high uncertainty (e.g. AIME math [ 43 ] , UGPhysics [ 88 ] and SciCode [ 78 ] ). We observe that Qwen 32B [ 77 ] can outperform frontier models like GPT-4o on Scicode [ 78 ] and is 3x more efficient than inference scaling through self-consistency and sequential refinement. While the results are promising, they must be interpreted with care (§ 5 ). First, EM is most effective when model confidence correlates with correctness, as in the experiments above. It is less suited for tasks like aligning with human values [ 35 ] , where confidence alone is not a reliable proxy for quality. Through experiments on individualistic value reasoning [ 35 ] , we observe that Qwen-2.5 struggles on the task without finetuning on labeled data, and further EM leads to no improvements. Second, the effectiveness of EM hinges on the assumption that the pretrained model is already capable in the tasks of interest. We observe that the improvements from EM-FT and EM-RL on Llama-3.1-8B are less substantial on math reasoning tasks compared to those on Qwen-2.5. This may be due to Llama-3.1-8B is less capable on these reasoning tasks. The success and limitations of EM highlight the importance of the capabilities of the pretrained models, which is sometimes underappreciated, at least for reasoning tasks. We call for the inclusion of EM as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better separate the contribution of algorithmic innovations from the intrinsic capabilities of the model itself. Besides, as we show, an approach that performs surprisingly well on certain models and tasks can fail in trivial ways on others. Broad evaluation across diverse pretrained models and tasks can help ensure that the conclusions generalize and are not artifacts of a specific model or dataset. 2 Background and Related Work 2.1 Related Work Entropy-regularized RL augments the expected reward objective with a regularization term aiming to increase entropy [ 98 ] , in order to promote exploration by adding uncertainty into the actions [ 64 , 33 , 27 , 18 , 41 ] . Entropy-regularized RL usually learns a policy from scratch in grid world situations (e.g. robot simulations) where adding perturbations can lead to exploring various paths to get to the goal. In contrast, we investigate if reducing the uncertainty of the policy can improve a pre-trained language models capabilities, with the hypothesis that they already acquire some capabilities for the tasks of interest during pretraining. Instead of learning from scratch, we finetune LLMs to reinforce confident outputs. Therefore, we draw inspiration from entropy-regularization in RL to design reward function ( EM-RL ) that minimize the entropy of the model encouraging it to commit to a fewer more confident paths, instead of prompting exploration like in previous works. Entropy minimization (EM) has been extensively used as regularization in semi-supervised learning [ 23 , 40 , 8 , 51 ] , domain adaptation [ 60 , 62 , 80 ] , and few-shot learning [ 25 , 39 ] . [ 38 , 20 ] apply EM for reinforcement learning with semi-supervised scenarios, where entropy is used to estimate the rewards on unlabeled data. [ 81 , 94 , 56 ] use EM for unsupervised test-time adaptation, where pre-trained classifiers are adapted to new data at test time without using any labels. Despite its well-studied history [ 53 , 52 , 48 , 15 , 61 , 71 ] , EM has rarely been explored as a standalone training objective for LLMs. Our key hypothesis is that EM can reinforce LLMs’ strong capabilities acquired during pretraining. Instead of applying entropy minimization for test time-domain adaptation or as a regularizer, we study if it can be used for LLM finetuning and inference time scaling. Unsupervised and few-shot RL for LLMs has raised the possibility of self-improvement [ 85 , 5 , 54 ] . Here a model learns via self-training in which it critiques its own generations without external feedback [ 22 , 47 , 57 , 93 ] or refines generated text [ 11 , 96 , 70 ] . Building on self-improvement, self-verification [ 45 ] has been applied for RL tuning [ 95 , 84 , 99 ] , concurrently. [ 32 ] uses the most frequent answer as reward (majority voting) for finetuning, since optimizing the likelihood of most consistent answer can reinforce pretraining priors. [ 99 ] applies this idea to unsupervised RL tuning at test time by optimizing on test dataset as a whole. We work on the same intuition of reinforcing model confidence, however, we study post-training where EM-FT and EM-RL are used for finetuning LLMs on large general math and coding data. Further, consistent answers and output extraction is not always possible in complex tasks like scientific coding [ 2 , 10 , 82 , 4 , 73 ] . On the other hand, EM-FT and EM-RL can be applied on tasks like code generation where output extraction is non-trivial. Unlike [ 99 ] which updates the model parameters on each test set individually, inference scaling via EM-INF optimizes the models’ entropy without updating parameters and does not need task-specific optimization. EM-INF can be deployed and served like regular text decoding [ 30 , 58 , 19 ] requiring neither assumption on task nor model optimization. [ 95 ] proposes to estimate rewards by computing the frequency of extracted answers for a prompt. Here, the model is incentivized to prefer certain answers, which makes the answer distribution more concentrated. While it reduces the entropy, it is not directly enforced and emerges from learning to repeat most frequent solutions. EM-FT and EM-RL target uncertainty in models’ token level distribution, whereas the frequency-based approaches target alignment with desired outputs where entropy reduction is not an inherent outcome and task-dependent. [ 84 ] investigates RL from few-shot output verified labeled data and observe that direct entropy maximization as a standalone objective can be used for finetuning to a small extent. They find that carefully selecting a few output-verifiable problems along with entropy maximization as a regularizer can lead to improvements. We find that only negative entropy rewards (entropy minimization) are sufficient for RL with similar effects without requiring any data selection. Standalone entropy maximization does not converge and eventually collapses [ 84 ] , but EM-RL and EM-FT via EM converge and can be used to train on large datasets without output verification. 2.2 Preliminaries Problem setup: Let \pi_{\theta} denote an autoregressive LLM policy: \pi_{\theta}({\bm{y}}|{\bm{x}})=\prod_{i}\pi_{\theta}(y_{i}|{\bm{x}},{\bm{y}}_{0 such that \mathcal{H}(\sigma({\bm{z}}_{t}/\tau))\leq\max(\alpha\mathcal{H}(\sigma({\bm{z}}_{t})),\delta) , where \alpha\in(0,1) controls the fraction of entropy to reduce. We find \alpha\in(0.4,0.6) works the best. Proposition 1 . (Informal) Both adaptive temperature scaling and logit optimization reduce the entropy of a model’s output distribution. However, in high-uncertainty settings (i.e., when the model’s predictive distribution has high entropy), logit optimization can change the order of non-top logits. In contrast, temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model’s behavior when it is highly confident (Appendix A ). Figure 1: Performance comparison of inference time scaling with Adaptive Temp and EM-INF with API models and inference time scaling methods on Scicode. Sub and main indicate subproblem and main problem mode in Scicode, whereas w/BG refer to with background knowledge in the prompts. Bold , Italics indicate the best and second best performance over all methods. Sub w/ BG Main w/ BG GPT-4o 35.4 9.2 GPT-4-Turbo-2024-04-09 33.7 0 9.2 Claude3.5-Sonnet 35.4 12.3 Claude3-Opus 26.7 0 4.7 Claude3-Sonnet 25.7 0 4.7 Gemini 1.5 Pro 30.6 0 7.7 Qwen2.5-7b-Instruct 11.5 0 0.0 Greedy decoding ( t=0 ) 13.4 0 3.0 Iterative-refinement ( n=2 ) 12.1 0 0.0 Adaptive Temp 13.2 0 0.0 EM-INF 16.7 0 1.5 Qwen2.5-32b-Instruct 25.4 0 4.6 Greedy decoding ( t=0 ) 25.4 0 4.6 Iterative-refinement ( n=2 ) 24.7 0 7.6 Adaptive Temp 28.8 0 7.6 EM-INF 27.1 10.7 EM-INF for scientific coding: We apply EM-INF to scientific coding in Table 1 . SciCode is a challenging code generation benchmark requiring significant domain knowledge and reasoning capabilities where even frontier models such as GPT4o struggle [ 78 ] . We observe that test time entropy minimization ( EM-INF , adaptive temperature) can improve performance of the base models consistently on easier subproblems and eventually on the more difficult main problem by 2.5\% . We note that logit optimization (EM-INF) is better than adaptive temperature scaling on SciCode by 3\% . Owing to the difficulty of SciCode, the models’ output distribution has large entropy. Under this scenario, EM-INF can change the relative order of the logits which likely leads to better reasoning chains. Whereas, adaptive temperature scaling will never change the relative ordering of the logits on each generation step – it only makes the distribution peakier (Proposition 1 ). Next, we observe that scaling through iterative-refinement [ 47 ] leads to some improvements over the base model, however, it is bottlenecked by the max context length of the model making it harder to refine long chain of thoughts. Whereas, EM-INF does not require refinement and maintains the same computational complexity as the base model when used with regular decoding. 0 10 20 30 40 40 42 44 46 48 50 52 Time Consumed (hours) Accuracy (%) Self-consistency EM-INF Adap. Temp. Figure 2 : Computational efficiency of EM-INF compared to self-consistency with N\!=\!4,8,16 on AMC. We note that EM-INF with Qwen2.5-32B-Instruct can outperform several frontier models on the main problem without requiring refinement or editing model parameters by 3\% and over the base model by 6\% . Building on these observations, we put forward EM-INF as a practical, compute efficient inference time scaling method that does not require any supervision and makes no assumption on the nature of the problem. EM-INF for math and coding: Table 3 compares the performance of EM-INF for test time scaling. We observe that minimizing the entropy of the model at test time without updating the parameters (EM-INF, adaptive temperature) improves the performance of the base model for almost all tasks and model classes by 3\% on average, indicating that EM is an useful method for test time scaling. We also note that EM can be applied to all tasks (math, coding and ugphysics) without assumptions such as output extraction in self-consistency or output scoring. Next, we note that EM competitively performs against self-consistency while outperforming it on tasks requiring significant reasoning capabilities, e.g. AIME, and UGPysics by 4\% and 3.6\% , respectively. Unlike self-consistency which requires sampling multiple trajectories ( N>1 ), EM-INF requires only one trajectory leading to lower computational requirements. These observations collectively indicate that pretraining performance can be improved via EM, and making the model more deterministic at inference time via unsupervised EM in-order to prefer fewer chain of thought options can lead to better performance. Computational efficiency: Figure 2 compares the compute requirements of EM-INF with inference scaling baselines. We observe that EM-INF can achieve better performance while using a third of time. This observation suggests that EM-INF is a practical inference scaling method in compute bound scenarios, for online adaption without updating model parameters [ 81 , 9 ] . EM-INF can also be combined with self-consistency and sequential refinement to scale up (Appendix § C.1 ). 5 Limitations: Dependence on Base Model Capability and Confidence Table 4: The limitations of EM on tasks and models where model confidences do not help. Left shows performance of Llama-3.1-8B on math tasks trained with the setup in Sec. 3.3 . Right compares the performance of Qwen-2.5-7b instruct on individualistic value alignment. Math AMC AIME Minerva Olymp . Llama-3.1-8B-Instruct 40.6 18.1 1.0 22.4 15.7 W/ RLOO 52.0 25.3 3.3 26.8 20.7 EM-FT (N=1) 42.2 21.7 3.3 18.8 16.4 EM-RL-token (N=4) 41.0 18.1 5.6 19.1 15.6 EM-RL-sequence (N=4) 41.6 18.1 7.8 18.4 16.6 IndVal Qwen2.5-7B-Instruct 65.9 EM-FT (N=1) 66.0 EM-RL-token (N=4) 66.0 EM-RL-sequence (N=4) 66.3 To contextualize the applicability of EM, we probe its performance by 1) training on tasks with different bias as compared to pretraining; and 2) finetuning models lacking target reasoning behaviors. Table 4 (right) evaluates EM on individualistic value reasoning ( IndVal ) [ 35 ] . Individualistic value reasoning is a difficult alignment task that evaluates LLMs’ ability to reason about an individual’s value preferences in new situations. Given known statements about a certain individual, the task is to predict if the user would or would not associate with a new situation (measured using accuracy). We observe that Qwen-2.5-7B struggles with the task, likely because LLMs’ are biased to coarse values acquired from pretraining which may not generalize to a large pluralistic audience [ 97 , 35 ] . As expected, we note that EM is not effective in this task, likely because model confidences are incorrectly biased to training priors that do not benefit the target task, as also observed in [ 99 , 31 ] . Furthermore, Table 4 (left) compares entropy minimization ( EM-FT , EM-RL ) with supervised RL (RLOO) using the same training data, setup and evaluation as Sec. 3.3 with Llama-3.1-8b-Instruct as the base model [ 24 ] . We observe that while supervised RL can improve model performance, improvements from entropy minimization is less effective as compared to Qwen-2.5 and can sometimes hurt performance on math reasoning. We tie this observation to [ 21 ] who show that Llama models may lack reasoning behaviors initially as compared to Qwen models, as also observed in [ 84 ] . The lack of certain reasoning behaviors for math in the initial policy possibly leads to unreliable model confidences and eventually inaccurate entropy rewards [ 12 ] . This observation suggests that success of entropy minimization hinges on the assumption that the pre-trained model’s initial reasoning behaviors are critical for the capacity of improvement. Overall, the success of EM-FT and EM-RL must be interpreted with care, as it can be less effective depending on the task and base model choice. 6 Conclusion We show that entropy minimization through unsupervised finetuning (EM-FT), RL with standalone negative entropy as reward (EM-RL) and inference time scaling through logit optimization (EM-INF) can improve LLMs on complex math, physics, and coding tasks without using any labeled data. We observe that through EM-FT and EM-RL, Qwen-2.5-7B can achieve comparable or better performance than RL methods like GRPO and RLOO. Through test time logit optimization, EM-INF enables Qwen-2.5-32B to outperform proprietary models like GPT4o on complex scientific coding without training or updating model parameters. Our findings suggest that many pretrained LLMs already possess strong reasoning ability which can be enhanced using EM alone. While successful, EM is less beneficial on tasks and models with different inductive biases as pretraining. We call for the inclusion of EM-FT, EM-RL and EM-INF as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better attribute the source of improvements in them. Acknowledgments and Disclosure of Funding This research used the DeltaAI advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2320345) and the State of Illinois. DeltaAI is a joint effort of the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications. This work was also in part supported by research awards from Apple and the Allen Institute for AI. Research was also supported in part by US DARPA INCAS Program No. HR0011-21-C0165 and BRIES Program No. HR0011-24-3-0325, National Science Foundation IIS-19-56151, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 2118329. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Government. Appendix A Proof for proposition 1 (Informal) Both adaptive temperature scaling and logit optimization reduce the entropy of a model’s output distribution. However, in high-uncertainty settings (i.e., when the model’s predictive distribution has high entropy), logit optimization can change the order of non-top logits. In contrast, temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model’s behavior when it is highly confident. Part 1: \forall{\bm{z}}\in\mathbb{R}^{|\mathcal{V}|} let p({\bm{z}})=\text{softmax}({\bm{z}}) . Let \mathcal{H}(p) denote the entropy of p ; \mathcal{H}=-\sum_{i=1}^{|\mathcal{V}|}p_{i}\log p_{i} . \exists{\bm{z}} , \eta>0 and a,b\in[1,\dots,|V|] , such that {\bm{z}}_{a}<{\bm{z}}_{b} and {\bm{y}}_{a}>{\bm{y}}_{b} where {\bm{y}}_{i}=z_{i}+\eta p_{i}(\log p_{i}+\mathcal{H}) , i=1,\dots,|\mathcal{V}| . Proof: Consider {\bm{y}}_{a}-{\bm{y}}_{b}={\bm{z}}_{a}-{\bm{z}}_{b}+\eta(g_{a}-g_{b}) where g_{i}=p_{i}(\log p_{i}+\mathcal{H}) . g is the gradient of entropy \mathcal{H} w.r.t. {\bm{z}} . We now show that \exists{\bm{z}},a,b such that z_{a}g_{b} . By Mean-Value Theorem \exists c\in(p_{a},p_{b}) such that g_{a}-g_{b}=(p_{a}-p_{b})(\log c+\mathcal{H}+1) . When p_{a}-p_{b}0 (6) Thus, we can set an arbitrary large \eta such that, z_{a}-z_{b}+\eta(g_{a}-g_{b})>0 (7) Part 2: b cannot be \arg\max_{i}z_{i} . Proof: Let j=\arg\max_{i}{\bm{z}}_{i} , i.e. the index of the maximum logit. Since p_{j}=\max_{i}p_{i} we can bound it using the inequality: \mathcal{H}(p)=-\sum_{i}p_{i}\log p_{i}\geq-\log p_{j}\quad\Rightarrow\quad p_{j}\geq e^{-\mathcal{H}(p)}\geq e^{-(\mathcal{H}(p)+1)} (8) We now analyze how g_{i} behaves as a function of p_{i} . Let g(p_{i})=p_{i}(\log p_{i}+\mathcal{H}) and g^{\prime}(p_{i})=\log p_{i}+\mathcal{H}+1 . Thus, g(p_{i}) is monotonically increasing whenever g^{\prime}(p_{i})>0 , i.e. when, p_{i}>e^{-(\mathcal{H}+1)} (9) So no lower-probability index i can have g_{i}>g_{j} , i.e., the entropy gradient cannot reduce {\bm{y}}_{j} relative to others. Therefore, b\neq j , leading to the following behavior \displaystyle 00 ; Maximum iterations M ; Tolerance \epsilon_{tol}>0 ; Target entropy reduction ratio \alpha\in(0,1) ; Target entropy threshold \delta>0 . 2: Output: Scaled logits z_{t}^{\prime} . 3: function ComputeAdaptiveTemperature ( z_{t},\tau_{max\_init},\tau_{min\_init},M,\epsilon_{tol},\alpha,\delta ) 4: P_{initial}\leftarrow\text{Softmax}(z_{t}) 5: H_{initial}\leftarrow\text{Entropy}(P_{initial}) 6: H_{target}\leftarrow\max(\delta,\alpha\cdot H_{initial}) 7: \tau_{final}\leftarrow 1.0 8: if H_{initial}>\delta then 9: \tau_{low}\leftarrow\tau_{min\_init} 10: \tau_{high}\leftarrow\tau_{max\_init} 11: for iteration\leftarrow 1 to M do 12: \tau_{mid}\leftarrow(\tau_{high}+\tau_{low})/2 13: P_{current}\leftarrow\text{Softmax}(z_{t}/\tau_{mid}) 14: H_{current}\leftarrow\text{Entropy}(P_{current}) 15: if |H_{current}-H_{target}|<\epsilon_{tol} then 16: \tau_{final}\leftarrow\tau_{mid} 17: break 18: else if H_{current}1 benefits direct entropy minimization in EM-FT. Bold indicates best performing setting while using the EM-FT variations. The training setup is described in Sec. 3.3 . Math Coding Math AMC AIME Minerva Olymp. Avg. LeetC LiveC Avg. Qwen2.5-7b 43.8 31.3 15.6 14.7 19.0 24.9 26.1 18.4 22.3 Trained using 60K labeled prompts w/ SFT 48.2 30.2 10.0 17.6 22.4 25.7 18.3 18.3 18.3 w/ RLOO 73.0 57.8 23.3 31.2 34.2 43.9 28.3 26.7 27.5 w/ GRPO 71.8 56.6 21.1 25.0 35.9 42.1 25.0 25.8 25.4 Our unsupervised methods trained on 60K unlabeled prompts EM-FT w KL ( N=4 ) 66.4 53.0 21.1 33.5 34.5 41.7 26.7 16.8 21.8 EM-FT w KL ( N=1 ) 65.6 54.2 16.7 26.8 34.8 39.6 28.3 13.7 21.0 EM-FT w/o KL ( N=1 ) 67.2 51.8 14.4 33.1 34.4 40.2 28.3 17.2 22.8 Appendix D Experimental Setup D.1 Evaluation Benchmarks D.1.1 Math Reasoning We adopted the data and evaluation pipeline from PRIME-RL codebase [ 13 ] for all math reasoning evaluations. The final answer of LM is the string inside the last \backslash boxed\{\} . All math benchmarks is evaluated on Pass@1 accuracy. Here, we will introduce each math benchmarks in detail. MATH 500 MATH 500 contains 500 prompts sampled from the 5,000 test problems in the original MATH dataset [ 29 ] . The test set of MATH comprises challenging competition-style math problems drawn from sources such as the AMC and AIME exams. These problems span seven subjects—Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus—and are labeled with difficulty levels from 1 (easiest) to 5 (hardest). Each test problem includes a final boxed answer and a full step-by-step solution in LaTeX, enabling exact match evaluation and providing interpretable reasoning paths for model analysis. AMC We use the validation set from AI-MO’s AMC dataset [ 43 ] , which contains 83 problems extracted from AMC 12 (2022 and 2023). The AMC dataset comprises math competition problems from the American Mathematics Competitions (AMC), covering a broad range of topics including arithmetic, geometry, combinatorics, and algebra. It is designed to evaluate a language model’s capacity for mathematical reasoning, abstraction, symbolic manipulation, and logical deduction. AIME We use the validation set from AI-MO’s AIME dataset [ 43 ] , which includes 90 problems drawn from AIME 22, AIME 23, and AIME 24. The AIME (American Invitational Mathematics Examination) dataset features high-difficulty problems intended for top-performing students from the AMC competitions. These problems typically require advanced mathematical reasoning across topics such as algebraic manipulation, number theory, combinatorics, and geometry. Each problem involves multiple intermediate steps and provides minimal guidance, making the dataset well-suited for evaluating a language model’s ability to perform multi-step symbolic reasoning and maintain logical coherence throughout extended problem-solving chains. Minerva Math Minerva Math refers to the OCWCourses evaluation dataset introduced in the Minerva paper [ 42 ] . This dataset comprises 272 undergraduate-level STEM problems sourced from MIT OpenCourseWare (OCW), covering topics such as solid-state chemistry, information theory and entropy, differential equations, and special relativity. Each problem is self-contained and accompanied by a clearly defined final answer—either numeric (191 problems) or symbolic (81 problems)—enabling automatic evaluation. The dataset is designed to assess a language model’s ability to perform multi-step scientific reasoning, including the application of domain-specific knowledge, symbolic manipulation, and precise quantitative computation. OlympiadBench OlympiadBench [ 28 ] is a large-scale bilingual multimodal benchmark comprising 8,476 Olympiad-level mathematics and physics problems sourced from international competitions such as the IMO and IPhO, as well as elite Chinese contests and Gaokao mock exams. The dataset evaluates LLMs on advanced mathematical and physical reasoning tasks, including multi-step logical deduction, symbolic manipulation, equation solving, theorem proving, and visual-spatial understanding from diagrams. Its high difficulty, multimodal diversity, and expert-level content make it a rigorous benchmark for assessing scientific reasoning capabilities. In this work, we use the open-ended, English, text-only subset of math competition problems as our evaluation set, which contains a total of 674 problems. D.1.2 Coding LeetCode LeetCode benchmark is introduced in the technical report of DeepSeek-Coder [ 26 ] . This benchmark consists of 180 recent competition-level programming problems from LeetCode, collected between July 2023 and January 2024. Each problem is paired with 100 test cases to ensure comprehensive coverage, and prompts are constructed in a standardized format to simulate realistic coding challenges. This dataset evaluates a language model’s ability to perform code generation in competitive programming settings, testing a wide range of capabilities such as problem comprehension, algorithm design, logic reasoning, and implementation precision. The inclusion of varying difficulty levels (Easy, Medium, Hard) allows for a thorough assessment of a model’s generalization and problem-solving skills under pressure. During evaluation, the python code will be extracted from model’s response and run the test cases. Performance is reported using the Pass@1 metric, indicating the proportion of problems for which the model’s first generated solution passes all test cases. LiveCodeBench LiveCodeBench [ 34 ] is a large-scale, contamination-free benchmark designed to holistically evaluate the code-related capabilities of large language models. It comprises 612 recent coding problems collected from competitive programming platforms such as LeetCode, AtCoder, and Codeforces between May 2023 and August 2024. Unlike traditional benchmarks that focus solely on code generation, LiveCodeBench assesses four key skills: (1) code generation from natural language, (2) self-repair using error feedback, (3) code execution via output prediction, and (4) test output prediction without implementation. This comprehensive setup evaluates not only functional correctness but also a model’s ability to debug, reason about program behavior, and construct tests. Its time-segmented evaluation protocol further supports rigorous detection of training data contamination. In this work, we adopt the V2 release, which includes 511 problems released between May 2023 and May 2024. We evaluate methods on the code generation ability. The model is given a natural language problem description and tasked with generating a correct program. Correctness is evaluated by passing all test cases. The Pass@1 metric is used, measuring the fraction of problems where the first generated solution passes all tests. D.1.3 Scientific Reasoning UGPhysics UGPhysics [ 88 ] is a large-scale bilingual benchmark designed to evaluate undergraduate-level physics reasoning in large language models. It comprises 5,520 problems (11,040 including both English and Chinese versions) spanning 13 physics subjects and 59 topics, featuring diverse answer formats such as numerical values, symbolic expressions, equations, and intervals. Each problem is annotated with one of four reasoning skill types: Knowledge Recall, Laws Application, Math Derivation, and Practical Application. Unlike prior benchmarks that primarily target multiple-choice or high-school-level content, UGPhysics emphasizes deeper problem-solving and derivation skills, requiring LLMs to perform symbolic reasoning, apply physical laws, and handle real-world quantities. This dataset is designed to assess LLMs’ ability to carry out multi-step physics reasoning beyond simple mathematical manipulation. In this work, we evaluate on the Semiconductor Physics subset, which consists of 372 problems. During evaluation, models answer physics problems using prompts tailored to different answer types for improved extraction and judgment. A rule-based correctness assessment then extracts answers from the model’s output and reports accuracy. SciCode SciCode [ 78 ] is a challenging, scientist-curated benchmark designed to evaluate large language models (LLMs) on real-world scientific coding tasks across 16 subfields in mathematics, physics, chemistry, biology, and materials science. It comprises 80 complex main problems, each decomposed into multiple subproblems (338 in total), requiring models to integrate domain-specific scientific knowledge, reasoning, and Python programming. Each problem includes gold-standard solutions and test cases, with optional expert-authored background context. SciCode assesses LLMs’ abilities in multi-step problem decomposition, code synthesis from scientific instructions, and integration of partial solutions. By emphasizing scientific reasoning and simulating real-world workflows, it provides a rigorous benchmark for evaluating LLMs as scientific assistants. In our paper, we evaluate methods on the test set, which contains 65 main problems and 288 subproblems in total. Models generate Python functions for each subproblem and integrate them into a complete solution. Evaluation is fully automatic, using gold test cases for both subproblems and main problems. The evaluation is conducted with and without scientific background information, as well as using gold or generated prior solutions, to measure realistic performance. Success requires all subproblems and the main solution to be correct. Final accuracy is reported as the percentage of problems achieving success. D.1.4 Individualistic Value Reasoning IndVal The IndVal dataset [ 36 ] is derived from the World Values Survey (WVS) and consists of natural language statements expressing individual human values and preferences from over 93,000 participants worldwide. Each participant responds to 253 questions grouped into 13 distinct categories, including Social Values, Economic Values, and Science & Technology, among others. The dataset standardizes these diverse survey responses into over 900 value-expressing statements per individual, enabling evaluation of language models’ ability to reason about individualistic human values in a bottom-up, granular manner. This dataset is particularly valuable as it challenges models to understand and predict nuanced personal value judgments beyond demographic stereotypes. Consequently, it reveals limitations in current language models’ capacity for individual value reasoning and promotes more equitable, personalized AI alignment that respects individual diversity. In our work, we randomly sampled 200 individuals excluded from training, uniformly distributed across all continents. For each individual, we randomly selected 4 statements per question category to form an individualistic value profile consisting of 52 known statements. Then, for each question category, we selected 2 statements not included in the known profile and asked the language model to select the most appropriate statement from the given candidates based on the individual profile. This process resulted in an evaluation dataset containing 5,400 prompts. During evaluation, the selected statement is extracted from the model generation and compared with the ground truth. Final performance is reported as accuracy. D.2 Training Data Math Reasoning We randomly sampled 35K prompts from Numina math training set [ 43 ] as our math training prompts. The NuminaMath dataset contains 860,000 problems ranging from high-school to advanced competition levels, including Olympiad, AMC, AIME, and Chinese K-12 math problems. Coding We randomly sampled 25K prompts from Eurus-2-RL training dataset’s coding split [ 13 ] as our coding training prompts. The Eurus-2-RL training dataset is a high-quality reinforcement learning dataset consisting of about 457,000 math problems and 27,000 coding problems with outcome verifiers such as LaTeX answers for math and test cases for coding. The math problems are sourced from the NuminaMath-CoT dataset, which covers a wide range from Chinese high school math to International Mathematical Olympiad-level questions. The dataset is carefully cleaned, filtered, and reformatted into a direct question-answer format to ensure high quality and solvability. It is especially useful for training because it provides diverse, verified, and challenging problems covering different level of difficulties and tasks. Individualistic Value Reasoning The training dataset is derived from the IndVal dataset [ 36 ] . We sampled 900 individuals uniformly across seven continents. For each individual, we randomly selected 4 statements per question category to form an individualistic value profile comprising 52 known statements. Then, we sampled 6 questions per question category as queries, resulting in a total of 70,200 training prompts. D.3 Models Qwen2.5 Family For the entropy minimization for post-training experiments, we use two models derived from the Qwen2.5 family: Qwen2.5-Math-7B [ 89 ] and Eurus-2-7B-SFT [ 13 ] . Qwen2.5-Math-7B is selected as the base model for math reasoning tasks. We apply various post-training methods using the math training dataset and evaluate their performance on math reasoning benchmarks. This model serves as an excellent starting point for our experiments due to its strong capability in tackling complex mathematical reasoning problems. Eurus-2-7B-SFT is a SFT checkpoint obtained by further fine-tuning Qwen2.5-Math-7B on the Eurus-2-SFT-Data from PRIME-RL [ 13 ] , which is an action-centric chain-of-thought reasoning dataset. Since Eurus-2-7B-SFT further enhances the reasoning abilities of Qwen2.5-Math-7B, we select it for coding experiments, applying different post-training methods with the coding training dataset and evaluating on coding benchmarks. For entropy minimization during inference time experiments, we select Qwen2.5-Math-7B-Instruct and Qwen2.5-7B-Instruct as the base models for math reasoning, coding, and scientific reasoning tasks because of their inherent instruction-following and reasoning abilities. Specifically, for SciCode, we include a more powerful model, Qwen2.5-32B-Instruct, due to its superior capability in handling complex problems, allowing us to evaluate the performance of EM-INF on stronger models. Llama-3.1 Family Llama-3.1 [ 24 ] is the second model family included in this work to ensure the generalizability of our methods across different models. For both the entropy minimization post-training and inference-time experiments, we select Llama-3.1-8B-Instruct as the base model. D.4 FLOPs Computation Following [ 63 , 37 ] , we estimate the training FLOPs as 6PD and inference FLOPs as 2PD where P is the number of parameters in the model and D is the number of tokens. We use early stopping based on the validation set accuracy for all methods, which significantly changes the FLOPs utilized. • EM-FT: On each training step, we sample one trajectory from the model costing 2Pn where n is the sequence length followed by training costing 6Pn . The total cost per step is thus 8Pn . We set sequence length to 4096 , batch size to 512 and the training early stops at 40 gradient steps costing 1.01\times 10^{17} FLOPs for a 7B parameter model. • EM-RL, GRPO, RLOO: None of the methods use a reward model. We use GRPO and RLOO with output verification only. We sample N=4 trajectories costing 8Pn , then we make a forward pass on the reference model costing another 8Pn followed by training costing 24Pn . The total training cost per step is thus 40Pn . We set sequence length to 4096 , batch size to 512 and the training early stops at 100 gradient steps costing 13\times 10^{17} FLOPs for a 7B parameter model.