Inferring from Logits: Exploring Best Practices for Decoding-Free Generative Candidate Selection
Mingyu Derek Ma, Yanna Ding, Zijie Huang, Jianxi Gao, Yizhou Sun, Wei Wang · 2025
Read Inferring from Logits: Exploring Best Practices for Decoding-Free Generative Candidate Selection by Mingyu Derek Ma, Yanna Ding, Zijie Huang, Jianxi Gao, Yizhou Sun, Wei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17338.txt.
Introduction
Mingyu Derek Ma, Yanna Ding, Zijie Huang, Jianxi Gao, Yizhou Sun, and Wei Wang compare shortcut ways to pick the best answer from a language model, in work from 2025. Models normally build answers token by token, which is slow and breaks the flow of learning signals. Shortcuts instead read early output scores to rate each choice at once.
The team tests a broad set of these shortcuts on nine tasks in all. Five are quiz tasks with few choices, and four are medical choice tasks with huge pools, some over ten thousand options. Many model types and sizes are covered. The results sort which shortcuts work best where, and the authors draw lessons for future model design.
Catalog record
| CREATOR | Mingyu Derek Ma, Yanna Ding, Zijie Huang, Jianxi Gao, Yizhou Sun, Wei Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17338
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inferring from Logits: Exploring Best Practices for Decoding-Free Generative Candidate Selection Mingyu Derek Ma, Yanna Ding, Zijie Huang, Jianxi Gao, Yizhou Sun, Wei Wang Abstract Generative Language Models rely on autoregressive decoding to produce the output sequence token by token. Many tasks such as preference optimization, require the model to produce task-level output consisting of multiple tokens directly by selecting candidates from a pool as predictions. Determining a task-level prediction from candidates using the ordinary token-level decoding mechanism is constrained by time-consuming decoding and interrupted gradients by discrete token selection. Existing works have been using decoding-free candidate selection methods to obtain candidate probability from initial output logits over vocabulary. Though these estimation methods are widely used, they are not systematically evaluated, especially on end tasks. We introduce an evaluation of a comprehensive collection of decoding-free candidate selection approaches on a comprehensive set of tasks, including five multiple-choice QA tasks with a small candidate pool and four clinical decision tasks with a massive amount of candidates, some with 10k+ options. We evaluate the estimation methods paired with a wide spectrum of foundation LMs covering different architectures, sizes and training paradigms. The results and insights from our analysis inform the future model design. 1 Introduction Generative LMs respond to queries by generating tokens to form an output sequence and optimize themselves by learning to generate the correct tokens ( Lewis et al. 2020 ) . The simplicity of token-level inference and optimization compromises its performance on end tasks, as there is a gap between the token-level paradigm and sequence-level task results and learning signals ( Longpre et al. 2023 ; Ethayarajh et al. 2024 ) . Some tasks use generative LM to select the answer(s) from a given pool of options where each candidate answer is a natural language sequence. For example, multiple-choice QA considers answer options as the candidate pool ( Khashabi et al. 2020 ) ; the large collection of labels are candidate answers for extreme label classification tasks ( Amigo and Delgado 2022 ) ; and disease ontology forms the candidate space for diagnosis tasks ( Singhal et al. 2023 ) . The typical practice is to decode a complete output sequence and then match it with candidates ( Mishra et al. 2022 ) . However, selecting candidates using full decoding not only cuts off the gradient flow and disables direct optimization on decoded results but also limits the output bandwidth due to time-consuming discrete decoding. Figure 1: Decoding-free generative candidate selection produces results without training specialized parameters, building index, or token-by-token decoding. Existing works perform candidate selection without decoding for outcome-level optimization or efficient parallel predictions ( Rafailov et al. 2023 ; Ma et al. 2025 ) . For example, Ma et al. 2023b calculate averaged logits of MCQA options to select an answer without decoding; Xu et al. 2023a estimate the NLI result using logits of a single token. Though these decoding-free candidate selection practices are widely used, there is no formal definition or clear investigation of the properties of each method. There is also no consensus about the guiding principles for deploying those methods under various tasks and data scenarios with diverse numbers, lengths, and complexity of candidate sequences. In this work, we formally define the decoding-free generative candidate selection task, and conduct the first systematic evaluation of typical practices on downstream tasks, reflecting the ultimate influence of the selection methods compared with conducting full decoding. Our systematic evaluation covers an extensive collection of candidate selection methods , including five decoding-free approaches to calculating candidate probability distribution from token logits, as well as encoding-only dense retrieval method and full decoding approach. The effects of deploying various methods are evaluated with a comprehensive set of downstream testbeds widely used for LLM evaluation. The first type of testbed includes five multiple-choice QA tasks with broad target capabilities and candidate diversity, reflecting the candidate selection capabilities while the candidate pool is limited. We further increase the difficulty and examine the performance on tasks with massive numbers of candidates on expert-curated large ontology with 10k+ options for making diagnoses, procedure decisions, ordering lab tests, and prescribing medications. We further dive into the characteristic shifts of candidate selection methods while using a wide spectrum of foundational LMs . The base models are diverse in terms of architectures (decoder-only or encoder-decoder), sizes (spanning from 137M to 11B), and training methods (pre-trained or instruction-tuned). The evaluation provides insights into the properties of decoding-free candidate selection methods. The performance of the token-logits-based candidate representation is highly dependent on the properties of the pretrained LM, dataset domain difficulty, and candidate space diversity. Pure estimation methods can outperform non-instruction-tuned models due to the challenges faced by weak base models in handling certain question formats during decoding. In this case, estimation methods offer a more straightforward means of exhibiting knowledge through token logits. The insights derived from our evaluation enable more informed and confident design for future estimation methods. We empirically demonstrate that the logits of the first output step are most informative; using selective tokens for estimations compromises the performance and scaling properties of various model sizes. 2 Problem Formulation Figure 2: Demonstration of generative candidate selection methods. (a) Using full decoding for tasks with answer candidates by generating output sequences token-by-token. The task is to identify diagnoses given the patient’s medical record. (b) Candidate space, consisting of coded diagnoses. (c) Using decoding-free generative candidate selection method for the diagnoses task based on prior-decoding logits over vocabulary. 2.1 Decoding Paradigm of Generative LMs The ordinary sequence-to-sequence formulation of generative LM takes the input sequence seq_{in}=t_{1}^{in},\ldots,t_{|seq_{in}|}^{in} and is expected to generate an output sequence \hat{seq_{out}}=\hat{t_{1}^{out}},\ldots,\hat{t_{|seq_{out}|}^{out}} . The output sequence generation involves encoding the input sequence to contextual vector representation ( i . e . , output of the final transformer block), and decoding the outputs following \hat{seq_{out}}=f_{\operatorname{full-decode}}(f_{\operatorname{encode}}(seq_{in})). (1) During inference, the decoding function f_{\operatorname{full-decode}} involves |seq_{out}| discrete decoding steps, in which each step produces one output token. For (k+1)-th step of decoding, which is conditioned on both the input sequence and k generated tokens, the model produces logits \mathbf{z}_{k}\in\mathbb{R}^{\lvert V\rvert} over the vocabulary V after passing output of encoding through unembedding matrix, and then obtain a probability distribution over the possible next token ( w\in V ) in the output sequence P\left(w\mid t_{1:|seq_{in}|}^{in},\hat{t}_{1:k}^{out}\right)=\operatorname{softmax}(\mathbf{z}) . Then a discrete token at this autoregressive decoding step is produced by Equation 2 . \hat{t}_{k+1}^{out}=\operatorname{argmax}_{w\in V}P\left(w\mid t_{1:|seq_{in}|}^{in},\hat{t}_{1:k}^{out}\right) (2) 2.2 Candidate Selection with Answer Pools When performing tasks using generative LMs, we include task instruction and query in the input sequence seq_{in} and expect the derived answer of the query \hat{ans} from the generated output sequence \hat{seq_{out}} can match the ground-truth answer ans . Some tasks expect open-ended free generation where the final answer is the generated output ( \hat{ans}=\hat{seq_{out}} ), such as translation, creative story generation and dialogue conversation ( Wang et al. 2023 ; Ma et al. 2023a ) . However, many tasks have an existing answer candidate pool, and the output sequence needs to find a matched candidate as the final prediction. We notate the candidate pool as C and a candidate as c\in\{c_{1},c_{2},\ldots,c_{|C|}\} where |C| is the total number of candidate options. Each candidate is a natural language sequence.