Semantic-guided Diverse Decoding for Large Language Model

Weijie Shi, Yue Cui, Yaguang Wu, Jingzhi Fang, Shibo Zhang, Mengze Li, Sirui Han, Jia Zhu, Jiajie Xu, Xiaofang Zhou · 2025

Read Semantic-guided Diverse Decoding for Large Language Model by Weijie Shi, Yue Cui, Yaguang Wu, Jingzhi Fang, Shibo Zhang, Mengze Li, Sirui Han, Jia Zhu, Jiajie Xu, Xiaofang Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23601.txt.

Introduction

Weijie Shi, Yue Cui, and eight colleagues make language models produce answers that differ in meaning, not just in wording, in a paper published in two thousand twenty-five. Many tasks want several genuinely distinct responses: picking the best of many tries, group-based training, and synthesizing varied training data all depend on real diversity.

Temperature sampling shuffles word choice, and diverse beam search penalizes repeated phrases, yet the outputs often say the same thing. The team's method, SemDiD, works directly in embedding space, where meaning lives. Three mechanisms cooperate: guidance that pushes new answers in fresh directions, repulsion that keeps answer groups apart, and a debiased scoring of positions. Adaptive gains and constraint optimization balance quality against spread. The goal is a set of answers that are each good and truly different. Semantic variety, the authors argue, is the bottleneck for several modern training recipes.

Catalog record

CREATORWeijie Shi, Yue Cui, Yaguang Wu, Jingzhi Fang, Shibo Zhang, Mengze Li, Sirui Han, Jia Zhu, Jiajie Xu, Xiaofang Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE124 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic-guided Diverse Decoding for Large Language Model Weijie Shi, Yue Cui, Yaguang Wu, Jingzhi Fang, Shibo Zhang, Mengze Li, Sirui Han, Jia Zhu, Jiajie Xu, Xiaofang Zhou Abstract Diverse decoding of large language models is crucial for applications requiring multiple semantically distinct responses, yet existing methods primarily achieve lexical rather than semantic diversity. This limitation significantly constrains Best-of-N strategies, group-based reinforcement learning, and data synthesis. While temperature sampling and diverse beam search modify token distributions or apply n-gram penalties, they fail to ensure meaningful semantic differentiation. We introduce Semantic-guided Diverse Decoding (SemDiD), operating directly in embedding space that balances quality with diversity through three complementary mechanisms: orthogonal directional guidance, dynamic inter-group repulsion, and position-debiased probability assessment. SemDiD harmonizes these competing objectives using adaptive gain functions and constraint optimization, ensuring both quality thresholds and maximal semantic differentiation. Experiments show SemDiD consistently outperforms existing methods, improving Best-of-N coverage by 1.4-5.2% across diverse tasks and accelerating RLHF training convergence by 15% while increasing accuracy by up to 2.1%. 1 Introduction Diverse decoding of large language models (LLMs) aims to produce multiple semantically diverse responses for the same query. High-quality, creative, and non-repetitive multiple responses play a crucial role in numerous fields. In best-of-N strategies, multiple-answer candidates coupled with a verification mechanism (like self-consistency) enable smaller models to surpass larger ones under equivalent computational budgets wang2022self ; snell2024scaling ; cheng2024relic . In group-based reinforcement learning from human feedback (RLHF) algorithms like Iterative-RLHF dong2024rlhf ; pang2024iterative , RLOO ahmadian2024back , and GRPO shao2024deepseekmath , diverse decoding facilitates self-improvement through reward-based evaluation of answer groups. Additionally, in seed-based data synthesis, diverse decoding serves as a fundamental component for generating varied training data, thereby enhancing the robustness and generalization capabilities wang2022self-instruct ; xu2023wizardlm ; xu2024magpie . Current methods like temperature sampling renze2024effect , nucleus sampling holtzman2019curious , diverse beam search vijayakumar2018diverse ; li2016simple , and code-book vilnis2023arithmetic create lexical variations, but they often produce semantically similar responses. To ensure semantic diversity among multiple-answer candidates, an ideal approach is to generate numerous candidates, embedding and clustering them semantically, and then selecting high-probability representatives from different clusters. Unfortunately, this is computationally prohibitive in practice, and calls for resource-efficient methods to achieve semantic diversity during the decoding process. We propose Semantic-guided Diverse Decoding (SemDiD), a decoding algorithm that generates multiple ( k ) semantically diverse and high-quality answers through three key mechanisms: directional guidance, inter-group repulsion, and probability preference. Directional guidance steers each decoding trajectory toward distinct semantic regions, inter-group repulsion maintains semantic distances between groups, and probability preference prioritizes tokens with higher likelihood to ensure quality. SemDiD runs k groups of beam searches simultaneously, with the first group employing greedy decoding to establish a quality baseline. Despite the conceptual simplicity, effective implementation presents several challenges: Diminishing Returns in Probability-Based Quality Assessment: While probability offers a computationally efficient quality metric, it has significant limitations. Extremely low probabilities correlate with poor outputs, but quality improvements plateau as probabilities increase beyond moderate levels. Rather than aggressively maximizing probability, SemDiD treats probability as a lower bound using a harmonic gain function that adaptively prioritizes quality or diversity based on their relative performance. This ensures both metrics remain above acceptable thresholds while optimizing for overall effectiveness. Position and Length Bias in Probability Estimation: As sequences progress, tokens appearing later typically receive higher probabilities due to increasing contextual certainty, resulting in distorted quality assessment. To address this, SemDiD employs position-based regularization, applying diminishing weights to tokens in later sequence positions while imposing a threshold to limit the influence of excessively high-confidence tokens. Balancing Competing Forces: To simultaneously maintain quality and maximize diversity, SemDiD integrates harmonic mean theory with \epsilon -constraint optimization to guarantee that each group’s quality score remains above the greedy baseline threshold while pursuing semantic diversity. Within the diversity dimension, SemDiD implements stage-aware transition between directional guidance and inter-group repulsion. During early exploration stages, partial answer embeddings are closely in semantic space, so directional guidance predominates to establish initial diversity. As decoding progresses, weighting gradually shifts toward inter-group repulsion, recognizing that predetermined directional guidance may become suboptimal and potentially limit model precision. We evaluate SemDiD on multiple benchmarks to test its effectiveness. For Best-of-N, our approach enhances performance across various tasks, with improvements ranging from 1.4% on ARC-Challenge to 5.2% on MMLU-Pro+ with 25 samples. In the RLHF framework, SemDiD enables more efficient exploration of the solution space through the Iterative-RLHF, GRPO, and RLOO algorithms, reaching target performance levels more quickly during training and achieving 1.8-2.1% higher accuracy compared to the best baseline methods. 2 Related Work Decoding algorithms determine how tokens are selected during language model generation. Diverse decoding methods extend this concept by producing multiple distinct outputs for a single prompt, and can be categorized into parallel sampling approaches and diverse beam search variants. Parallel samplings generate tokens independently across different decoding paths. Methods such as temperature sampling, top-k sampling fan2018hierarchical , nucleus sampling holtzman2019curious , and typical decoding meister2023locally modify the conditional token distribution to increase randomness, but provide no guarantees against duplicate outputs. Arithmetic sampling vilnis2023arithmetic uses an implicit arithmetic code-book to ensure character diversity. To prevent sampling tokens with excessively low probabilities that might compromise quality, truncation sampling hewitt2022truncation and epsilon sampling freitag2023epsilon introduce cutoff mechanisms based on probability thresholds and distribution shape. Diverse beam search variants li2016simple ; vijayakumar2018diverse explicitly model interactions between candidate sequences to prevent redundancy, introducing diversity penalties between beam groups like Hamming, Cumulative, and n-gram penalties. Determinantal beam search meister2021determinantal formulates decoding as a series of subdeterminant maximization problems based on determinantal point processes (DPPs), inherently encoding intra-set interactions to promote n-gram coverage. Stochastic beam search with Gumbel-Top-k trick kool2019stochastic and Conditional Poisson meister2021conditional enable both exploration and principled sampling without replacement from sequence models. While these interaction-based methods improve lexical diversity, they primarily operate at the token or n-gram level rather than in semantic space. Our proposed SemDiD advances beyond these approaches by explicitly guiding the decoding process in embedding-based semantic space. Furthermore, to ensure the ease of deployment, our design principle aims to be usable out-of-the-box without any training. Figure 1: Comparison between ideal semantic decoding and SemDiD. While generating numerous samples followed by embedding and clustering would optimally identify diverse representatives, this is computationally intractable in practice. Our proposed SemDiD achieves similar diversity more efficiently by directly guiding decoding in semantic space through orthogonal directions and inter-group repulsion, while maintaining quality with debiased probability assessment. 3 Semantic-guided Diverse Decoding Let q denote a query input and M a pretrained language model with a conditional distribution p_{M}(y|q) over possible responses y\in\mathcal{Y} . Our Semantic-guided Diverse Decoding (SemDiD) aims to generate k responses \{y_{1},y_{2},\ldots,y_{k}\} that exhibit both high quality and semantic diversity. SemDiD Architecture. As shown in Figure 1 , SemDiD extends the group beam search framework using k groups and selecting one representative output from each group. Candidates are evaluated on both quality using debiased sequence probability as a lower bound and diversity by assessing embedding space representations through exploration direction and inter-group distance. The first group uses greedy decoding to establish a quality baseline and reference direction. 3.1 Efficient Quality Assessment with Debiased Probability Quality evaluation is crucial during generation. Although dialogue-based LLMs could be used, applying such evaluations to

Read the complete text