Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation

Erfan Baghaei Potraghloo † \dagger, Seyedarmin Azizi † \dagger, Souvik Kundu, Massoud Pedram · 2025

Read Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation by Erfan Baghaei Potraghloo † \dagger, Seyedarmin Azizi † \dagger, Souvik Kundu, Massoud Pedram (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02510.txt.

Introduction

Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram wrote a 2025 natural language processing paper on sampling text from language models without losing coherence. Truncated sampling methods cut off unlikely words, but they use the model's confidence crudely.

The authors frame the creativity-coherence balance as an entropy-constrained optimisation problem and prove it equals a mass maximisation problem that is NP-hard to solve exactly. Their Top-H decoding is a fast greedy method for that problem. It uses the full probability distribution rather than a single top token. On creative writing tests, Top-H beat the leading min-p sampler by a wide margin. It also held up on question-answering sets, and judge-model checks confirmed coherent output even at high randomness settings.

Catalog record

CREATORErfan Baghaei Potraghloo † \dagger, Seyedarmin Azizi † \dagger, Souvik Kundu, Massoud Pedram
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation Erfan Baghaei Potraghloo † \dagger, Seyedarmin Azizi † \dagger, Souvik Kundu, Massoud Pedram Abstract Large language models (LLMs), despite their impressive performance across a wide range of tasks, often struggle to balance two competing objectives in open-ended text generation: fostering diversity and creativity while preserving logical coherence. Existing truncated sampling techniques, including temperature scaling, top- p p (nucleus) sampling, and min- p p sampling, aim to manage this trade-off. However, they exhibit limitations, particularly in the effective incorporation of the confidence of the model into the corresponding sampling strategy. For example, min- p p sampling relies on a single top token as a heuristic for confidence, eventually underutilizing the information of the probability distribution. Toward effective incorporation of the confidence of the model, in this paper, we present top-H decoding. We first establish the theoretical foundation of the interplay between creativity and coherence in truncated sampling by formulating an entropy-constrained minimum divergence problem. We then prove this minimization problem to be equivalent to an entropy-constrained mass maximization (ECMM) problem, which is NP-hard. Finally, we present top-H decoding, a computationally efficient greedy algorithm to solve the ECMM problem. Extensive empirical evaluations demonstrate that top-H outperforms the state-of-the-art (SoTA) alternative of min- p p sampling by up to 25.63 % \mathbf{25.63}\% on creative writing benchmarks, while maintaining robustness on question-answering datasets such as GPQA, GSM8K, and MT-Bench. Additionally, an LLM-as-judge evaluation confirms that top-H indeed produces coherent outputs even at higher temperatures, where creativity is especially critical. In summary, top-H advances SoTA in open-ended text generation and can be easily integrated into creative writing applications. The code is available at https://github.com/ErfanBaghaei/Top-H-Decoding . 1 Introduction Large language models (LLMs) have exhibited impressive abilities in open-ended generation tasks, including creative writing and multi-turn dialogue ( Lee et al. 2022 ) . However, these models often need to deal with the challenge of balancing creativity and coherence , accepting less likely and more imaginative token choices while avoiding incoherent or nonsensical output. This trade-off is complex, as indiscriminate broadening of the sampling pool can lead to fragmented or disjointed text ( Holtzman et al. 2019 ) . To navigate this balance, various sampling strategies have emerged, including temperature scaling ( Ackley et al. 1985 ) , top- k ( Fan et al. 2018 ) , top- p (nucleus) ( Holtzman et al. 2019 ) , \eta ( Hewitt et al. 2022 ) , and min- p sampling ( Nguyen et al. 2024 ) . They generally apply heuristics to control diversity and risk. Specifically, min- p sampling ( Nguyen et al. 2024 ) stands out for its dynamic truncation of low-probability tokens using a threshold tied to the probability of the top token. Although this method performs well at high temperatures ( T ), its exclusive reliance on the maximum probability token to estimate confidence disregards the potential distribution of the probability mass over the remaining vocabulary. As a result, min- p remains vulnerable to over-truncation in sparse (low-entropy) distributions and under-truncation in dense (high-entropy) distributions. The above limitation motivates the need for a more methodical confidence-aware sampling framework that accounts for the overall shape of the distribution, rather than only its peak. In addition, the proliferation of heuristic methods highlights a deeper issue, namely the lack of a theory-based foundation to analyze the interplay between creativity and coherence in autoregressive generation . Our Contributions. Towards effective incorporation of the confidence of the model, in this work, we present top-H decoding. In particular, top-H maintains the creativity and coherence balance guided by bounded entropy in text generation. Unlike most of the earlier approaches that rely on a fixed threshold, top-H dynamically selects a subset of tokens such that the resulting truncated distribution over the selected subset has an upper-bounded uncertainty, while having minimal divergence from the original distribution predicted by the model. To formally ground top-H, we first introduce a constrained optimization problem that characterizes the trade-off between creativity and coherence in language generation, namely, entropy-constrained minimum divergence (ECMD). We show that this minimization is equivalent to an entropy-constrained mass maximization (ECMM) problem. We then prove that ECMM is NP-hard. Thus, in top-H, we offer a greedy solution that is both efficient and practically effective in approximating the solution of the ECMM while bounding the entropy of the selected distribution. During autoregressive generation, as the token distribution evolves at each step, top-H adjusts its entropy threshold based on the entropy of the token distribution, thereby dynamically adapting to the model’s varying confidence over time. We validate the effectiveness of top-H through extensive experiments in a diverse set of tasks, including creative writing (Alpaca-Eval ( Li et al. 2023 ) and MT-Bench ( Zheng et al. 2023 ) ), reasoning (GSM8k ( Cobbe et al. 2021 ) and GPQA ( Rein et al. 2024 ) ), and human-aligned evaluations using LLM as a judge framework. Specifically, top-H consistently outperforms existing sampling methods in accuracy while maintaining a robust balance between expressiveness and fluency. For example, compared to min- p , top-H demonstrates an accuracy improvement of up to \mathbf{25.63}\% . 2 Related Work 2.1 Stochastic Sampling Strategies for Autoregressive Models Temperature scaling ( Ackley et al. 1985 ) multiplies the logits by a scalar, encouraging the exploration of less likely tokens. However, it can get too indiscriminate at high T s, generating incoherent or contradictory texts. Top- k ( Fan et al. 2018 ) includes only the k highest probability tokens. Although simple, this hard cutoff is insensitive to context, sometimes excluding large swaths of moderately plausible tokens. Top- p (nucleus) sampling ( Holtzman et al. 2019 ) chooses the smallest subset of tokens whose cumulative probability exceeds p . This alleviates some of the rigidity of top- k . Unfortunately, at high T , the distribution can be so flat that the top- p may inadvertently include very low-probability tokens, harming coherence. This incoherence in top- p sampling is demonstrated in the experimental results Table 4 , where the coherence score on text drops significantly at higher T . Min- p ( Nguyen et al. 2024 ) sampling dynamically scales a base probability score threshold p_{\mathrm{base}} by the probability of the top-1 token. This effectively restricts the sample space more aggressively when the model is confident. Min- p has been shown to outperform top- p in tasks requiring both diversity and correctness at higher temperatures. However, its reliance on only the highest-probability token can overlook broader features of the distribution. Two different probability mass functions might share a top-1 token probability; however, they differ widely in their overall confidence. 2.2 Entropy-Based Sampling Strategies for Autoregressive Models Several methods attempt to exploit entropy or related uncertainty measures when sampling. \eta -sampling ( Hewitt et al. 2022 ) dynamically adjusts the sampling threshold based on the entropy of the distribution of the next token. However, this method often requires carefully tuned hyperparameters and can introduce significant runtime overhead at higher T s. Mirostat ( Basu et al. 2020 ) aims to maintain a target perplexity (related to entropy) via feedback control. While it can yield steady perplexities, it adds complexity to parameter tuning and

Read the complete text