Optimizing Temperature for Language Models with Multi-Sample Inference
Weihua Du, Yiming Yang, Sean Welleck · 2025
Read Optimizing Temperature for Language Models with Multi-Sample Inference by Weihua Du, Yiming Yang, Sean Welleck (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05234.txt.
Introduction
Weihua Du, Yiming Yang, and Sean Welleck tune a key dial for language models in this 2025 paper. The dial is temperature, which sets how random the model's word choices are. Many systems take several samples and vote or pick the best, and the dial shapes how well that works.
Fixed settings are crude, and tuning by hand needs labeled test data that is often scarce. The authors study how the best setting shifts across models and tasks. They seek a way to find a near-best value with no task labels. Their study maps the dial's role in multi-sample methods.
Catalog record
| CREATOR | Weihua Du, Yiming Yang, Sean Welleck |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05234
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Temperature for Language Models with Multi-Sample Inference Weihua Du, Yiming Yang, Sean Welleck Abstract Multi-sample aggregation strategies, such as majority voting and best-of-N sampling, are widely used in contemporary large language models (LLMs) to enhance predictive accuracy across various tasks. A key challenge in this process is temperature selection, which significantly impacts model performance. Existing approaches either rely on a fixed default temperature or require labeled validation data for tuning, which are often scarce and difficult to obtain. This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different LLMs using multi-sample aggregation strategies, without relying on task-specific validation data. We provide a comprehensive analysis of temperature’s role in performance optimization, considering variations in model architectures, datasets, task types, model sizes, and predictive accuracy. Furthermore, we propose a novel entropy-based metric for automated temperature optimization, which consistently outperforms fixed-temperature baselines. Additionally, we incorporate a stochastic process model to enhance interpretability, offering deeper insights into the relationship between temperature and model performance. Figure 1: (a) The entropy turning point (EntP) is defined as the point where the log-scale of token-level entropy shifts from concave to convex. (b) The accuracy tested in EntP is highly correlated with the best accuracy from grid search. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across various domains, including question answering ( Kamalloo et al. 2023 ) , intelligent agents ( Wang et al. 2024b ; Zhang et al. 2023 ) , scientific discovery ( Ma et al. 2024 ; Romera-Paredes et al. 2024 ) , and mathematical reasoning ( Ahn et al. 2024 ; Sun et al. 2024 ; Lin et al. 2024 ; Wu et al. 2024 ) . A fundamental research question in generative models is how to effectively sample solutions from a learned distribution and perform inference-time reasoning. Recently, multi-sample aggregation strategies have gained increasing attention. These strategies involve generating multiple solutions from the underlying distribution and aggregating them into a final prediction ( Wei et al. 2022 ; Yao et al. 2024 ) . Common aggregation techniques, such as majority voting, weighted majority voting, and best-of-N selection, have demonstrated significant performance improvements in benchmark evaluations of LLMs ( Welleck et al. 2024 ; Wang et al. 2024a ) . Despite the promising success of multi-sample aggregation strategies, there remains a lack of deep understanding regarding how to optimize the sampling process to enhance LLM performance under different conditions, including variations in training datasets, task types, and model sizes. A crucial open question is how to tune temperature, a key hyperparameter that controls the smoothness of the system-learned distribution. Intuitively, increasing the temperature leads to a smoother distribution, enhancing the diversity of sampled outputs. However, excessively high temperatures can introduce many low-quality samples, making aggregation more challenging ( Holtzman et al. 2019 ; Renze & Guven 2024 ) . Conversely, lowering the temperature results in a highly concentrated distribution, reducing diversity and potentially omitting high-quality samples. Striking the right balance between over-sampling and under-sampling is therefore essential for optimizing LLM performance. A common practice in prior evaluations is to use the same temperature across all methods despite variations in training datasets, task types, model sizes, and aggregation strategies. This is clearly suboptimal. An alternative approach is to empirically tune the temperature using labeled validation data for each task, dataset, model size, and aggregation strategy ( Zhang et al. 2024a ; Dhuliawala et al. 2024 ) . However, such a process is tedious and time-consuming and heavily dependent on the availability of labeled validation data, limiting its applicability when such data are scarce. In this paper, we present the first systematic investigation of how temperature affects LLM performance under multi-sample aggregation strategies across various conditions. Furthermore, we propose a principled algorithmic solution for automated temperature optimization without requiring labeled validation data. Our key idea is as follows: 1. We use the confidence score of each model as a self-assessment measure. 2. If this self-assessment measure is highly correlated with model accuracy on test data, it can serve as a surrogate metric for tuning temperature in the absence of labeled validation data. A surprising finding from our temperature tuning experiments is the discovery of a phenomenon we term the entropy turning point (EntP) in the self-assessed performance curve. As illustrated in Figure 1 (a), the token-level entropy (y-axis) of an LLM varies with temperature values (x-axis), shown by the blue curve, while its log-scale representation appears as the red curve. Notably, there is a transition point (EntP) where the red curve shifts from concave to convex. Figure 1 (b) shows that the accuracy scores at EntP for a set of LLMs are strongly correlated with their highest accuracy scores obtained through grid-based temperature tuning. This finding supports our intuition that EntP can be leveraged to automatically determine the optimal temperature for each LLM using multi-sample aggregation strategies. We introduce TURN , our proposed approach for automated temperature optimization. Through extensive experiments, TURN has demonstrated strong generalizability across diverse tasks (e.g., mathematical problem-solving, code generation), model sizes, and aggregation strategies (e.g., majority voting, best-of-N). It consistently outperforms baseline methods using a fixed temperature, yielding significant performance improvements. Additionally, our approach enhances the interpretability of temperature’s role in model performance by analyzing EntP. Moreover, our analysis explores how the optimal temperature is influenced by the divergence or similarity between model training and tasks (Section 3 ). In summary, TURN provides a novel, efficient, and principled method for optimizing temperature in LLM inference with multi-sample aggregation. It eliminates the need for labeled validation data and significantly improves performance across a wide range of applications. Figure 2: (a) Accuracy Heatmap. Performance of Mistral-7B-Instruct-v0.3 under majority voting across different temperatures. The best temperature for each sampling size is highlighted in bold white, and the optimal temperature range is shaded white. The green line shows the temperature predicted by our method. (b) Midpoint of Optimal Temperature Range vs. Number of Samples. The optimal temperature range varies by model; those with training data more closely matching the task tend to favor higher temperatures. 2 Preliminary & Related Work Before moving to our main contributions, we first review how language models typically generate samples and an introduction to multi-sample aggregation strategies. Language Model Sampling Language models typically generate outputs for generative tasks by autoregressively sampling from the conditional probability distribution over the next token, given both the input context and previously generated tokens. Formally, for an input X and an output sequence Y=(y_{1},y_{2},\dots,y_{N}) , the probability of producing Y is given by: \displaystyle P(Y\mid X)\;=\;\prod_{i=1}^{N}P\bigl(y_{i}\;\bigm|\;y_{<i},\,X\bigr). (1) To compute the probability distribution, the model obtains a set of logits z_{i} and then divides them by a temperature hyperparameter T before applying the softmax function and a regularization function \mathcal{F} : \displaystyle P(y_{i}\mid y_{<i},X)\;=\;\mathcal{F}\left(\operatorname{softmax}\Bigl(\frac{z_{i}}{T}\Bigr)\right), (2) where z_{i} is the logit corresponding to token y_{i} . The temperature T controls how peaked or flat the resulting probability distribution will be. The regularization function \mathcal{F} is used to reschedule the sampling process (e.g., Top- k ( Kool et al. 2019 ) , Top- p ( Holtzman et al. 2019 ) , Min- p ( Nguyen et al. 2024 ) and Locally Typical Sampling ( Meister et al. 2023 ) ). Multi-Sample