Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Prefilling Attack

Silvia Cappelletti, Tobia Poppi, Samuele Poppi, Zheng-Xin Yong, Diego Garcia-Olano, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara · 2025

Read Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Prefilling Attack by Silvia Cappelletti, Tobia Poppi, Samuele Poppi, Zheng-Xin Yong, Diego Garcia-Olano, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15323.txt.

Introduction

Grading a language model by its first answer token is fast, but the token can mislead the grade. Silvia Cappelletti and co-authors fix that fragility with a prompt trick in a 2025 paper. Their trick, called prefilling, starts the model's reply with a set phrase like the correct option is. The phrase steers the model to give a clean option token at once. No model weights are changed.

With prefilling, first-token grading gains in accuracy, calibration, and steadiness across many models. It often matches the slower full-answer grading that needs extra sorting tools. Two failure modes are named: tokens aimed at the wrong thing, and answer tokens buried in filler talk. The method began in safety work and is here turned to measurement.

Catalog record

CREATORSilvia Cappelletti, Tobia Poppi, Samuele Poppi, Zheng-Xin Yong, Diego Garcia-Olano, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15323

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Prefilling Attack Silvia Cappelletti, Tobia Poppi, Samuele Poppi, Zheng-Xin Yong, Diego Garcia-Olano, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara Abstract Large Language Models (LLMs) are increasingly evaluated on multiple-choice question answering (MCQA) tasks using First-Token Probability (FTP), which selects the answer option whose initial token has the highest likelihood. While efficient, FTP can be fragile: models may assign high probability to unrelated tokens ( misalignment ) or use a valid token merely as part of a generic preamble rather than as a clear answer choice ( misinterpretation ), undermining the reliability of symbolic evaluation. We propose a simple solution: the prefilling attack , a structured natural-language prefix ( e.g. , ‘ The correct option is: ’) prepended to the model output. Originally explored in AI safety, we repurpose prefilling to steer the model to respond with a clean, valid option, without modifying its parameters. Empirically, the FTP with prefilling strategy substantially improves accuracy, calibration, and output consistency across a broad set of LLMs and MCQA benchmarks. It outperforms standard FTP and often matches the performance of open-ended generation approaches that require full decoding and external classifiers, while being significantly more efficient. Our analysis suggests that prefilling is a simple, robust, and low-cost method to enhance the reliability of FTP-based evaluation in multiple-choice settings. 1 Introduction Figure 1: We show that a simple prefilling attack template, which directs an LLM’s first generated token to a valid option for MCQA, substantially improves the standard first-token probability approach. Large Language Models (LLMs) are increasingly deployed as general-purpose reasoning systems ( Plaat et al. 2024 ; Kojima et al. 2022 ; Huang and Chang 2023 ) : they are expected not only to generate fluent text, but also to make decisions ( Liu et al. 2024 ; Lyu et al. 2025 ) , answer questions ( Liang et al. 2022 ; Kamalloo et al. 2023 ) , and demonstrate understanding across a wide range of domains ( Wei et al. 2022 ; Naveed et al. 2023 ) . A common way to evaluate these capabilities is through question-answering tasks, where the model is required to process a question and return a correct and relevant answer ( Liang et al. 2022 ; Kamalloo et al. 2023 ; Chang et al. 2024 ; Khashabi et al. 2020 ) . Among the many QA formats, one of the most widely adopted is multiple-choice question answering (MCQA), in which the model selects from a fixed set of answer candidates, typically labeled A through D ( Hendrycks et al. 2021 ; Lai et al. 2017 ; Clark et al. 2018 ; Richardson et al. 2013 ; Li et al. 2024 ) . MCQA benchmarks such as MMLU ( Hendrycks et al. 2021 ) are widely used to evaluate the general knowledge, reasoning ability, and decision-making skills of a model across diverse subjects. Models are typically prompted with a question and a set of answer options, and evaluation is done by checking the model choice across the given, valid options. However, there are various ways with differing tradeoffs to extract the model choice. Some approaches decode free-form generations into valid options either by using a separate language model ( e.g. , GPT-3.5-Turbo Ouyang et al. 2022 ) or by training a dedicated classifier ( Wang et al. 2024a ) . Another approach that does not require an additional model for extraction is to let the model generate one single token, and compare its probability across the set of valid option tokens ( e.g. , ‘ A ’, ‘ B ’, ‘ C ’, ‘ D ’). This method, known as First-Token Probability (FTP) Hendrycks et al. 2021 ; Santurkar et al. 2023 , scores each candidate answer by computing the likelihood of it being generated as the first token and selecting the most probable one, without requiring full autoregressive decoding. It is efficient and model-agnostic, but relies on the assumption that the model is ready to commit to an answer immediately – an assumption that might break down in practice. In particular, models frequently diverge from the expected output structure, producing full sentences ( e.g. , ‘ I believe the answer is A ’), or ambiguous strings that are hard to interpret ( i.e. , first-token misalignment ). Worse still, we find that even when the first token is a valid label, it may serve a purely grammatical role that does not reflect the model intended answer ( i.e. , first-token misinterpretation ). For instance, ‘ A ’ could begin ‘ A possible answer could be C ’. These ambiguities make first-token evaluation noisy and potentially misleading ( Figure 1 ), eventually distorting accuracy metrics. In this work, we introduce a simple, yet effective, solution to these two problems by using the prefilling attack Tang 2024 ; Andriushchenko et al. 2025 to guide the LLM towards generating a valid option as the first token. Specifically, we draw on the underlying principle of biasing model behavior through structured prefilling and show that by prepending a short and benign phrase (such as ‘ The correct option is: ’) to the model generation, it is possible to effectively steer the model toward generating a valid multiple-choice response. We show that our approach leads to consistent improvements across different LLMs and benchmarks ( e.g. , up to + 40 % on the Gemma-2-9B model ( Team et al. 2024 ) , when tested on the MMLU benchmark). Despite its simplicity, this prefilling strategy substantially improves the reliability of FTP-based evaluation by reducing both the first-token misalignment and first-token misinterpretation issues. To further validate this, we conduct controlled experiments where the model selects its first token from the full vocabulary , rather than a filtered set as in standard FTP, allowing us to observe whether the top-ranked token aligns with the intended answer. Prefilling consistently steers the model toward ranking the correct option highest, substantially reducing first-token misalignment – for instance, boosting Llama-3.1-8B ’s accuracy on SciQ ( Welbl et al. 2017 ) from 2.2% to 96.9%. Compared to stronger baselines ( e.g. , open-ended generation with GPT-based answer extraction), prefilling yields similarly aligned answers while being more efficient, as it avoids relying on an additional model for evaluation. For example, with Phi-4-14B , the alignment gap with this baseline drops from 7.6% to 1.6%, on the OpenBookQA dataset ( Mihaylov et al. 2018 ) . To better understand the broader impact of prefilling beyond accuracy, we also examine its effect on model calibration ( i.e. , the ability to reflect uncertainty in predictions). For this purpose, we also evaluate whether prefilling improves the model confidence estimates. We find that prefilling consistently improves output calibration, lowering the expected calibration error across all tested models. Overall, our analysis demonstrates the effectiveness of the prefilling attack as a low-cost and easily applicable intervention for improving the reliability of LLM outputs in multiple-choice settings. 2 Related Work LLMs as Classifiers. Large Language Models (LLMs) have been increasingly used as zero-shot and few-shot classifiers by leveraging their in-context learning abilities Brown et al. 2020 ; Chowdhery et al. 2023 ; Achiam et al. 2023 . In multiple-choice question answering (MCQA), LLMs

Read the complete text