Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models

Ashwath Vaithinathan Aravindan, Mayank Kejriwal · 2026

Read Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models by Ashwath Vaithinathan Aravindan, Mayank Kejriwal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25394.txt.

Introduction

Ashwath Vaithinathan Aravindan and Mayank Kejriwal propose a simple prompting trick that helps small language models admit uncertainty instead of answering wrongly with confidence. Their Second Guess method adds an 'I don't know' option and watches answer stability. Models that truly know an answer pick it consistently, while uncertain models waver once abstention is offered. Tested on four small models and four benchmarks, the method achieves the best composite risk improvement, about 11 percent. It keeps an 8 percent improvement on fine-tuned models where entropy methods degrade, helping weak models most.

Catalog record

CREATORAshwath Vaithinathan Aravindan, Mayank Kejriwal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35983
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25394

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models Ashwath Vaithinathan Aravindan, Mayank Kejriwal Abstract Large language models often generate confident but incorrect answers rather than abstaining when uncertain. This problem is particularly acute for small language models (SLMs), where computational constraints and autonomous operation amplify the need for reliable uncertainty detection. We propose Second Guess , a lightweight, parameter-free prompting technique for abstention in multiple-choice question answering (MCQA) that is well-suited for SLMs. Our key empirical insight is that models which truly know an answer will select it consistently, while uncertain models exhibit unstable behavior when an “I don’t know” option is added. Evaluated on four open models (2B-8B parameters) and four benchmarks, Second Guess achieves the highest composite risk improvement of 10.81%. Notably, it maintains an 8% composite risk improvement on fine-tuned models where entropy-based methods degrade, and improves most for lower-performing models. All code and results required to reproduce this work is available in https://github.com/Mystic-Slice/second-guess Figure 1: Overview of the Second Guess technique. The model is prompted twice: once with the original options and once with an “I don’t know” option added. If the model’s answer changes between the two prompts, it is classified as an abstention; otherwise, the original answer is retained. 1 Introduction While large language models (LLMs) often generate confident, but incorrect, answers rather than acknowledging uncertainty and abstain, the challenge is even more acute for small language models (SLMs), denoted here as transformer-based generative models with fewer than 10B parameters. SLMs are important in many contexts, including deployment on edge and mobile devices 14 , where computational constraints can amplify the importance of reliable uncertainty detection, and autonomous operation precedes the availability of human oversight 19 . Recent work addresses abstention through diverse methodological approaches, each with distinct constraints. Training-based methods ( 27 ; 11 ) leverage “I don’t know” labels or dedicated tokens during fine-tuning, though they require direct access to model weights. Prompt-based methods ( 28 ; 9 ; 16 ) attempt to elicit uncertainty through explicit queries, but their effectiveness depends on the assumption that models accurately self-report their confidence. Sampling-based methods ( 3 ; 21 ) achieve robustness through consensus across multiple model outputs, at the cost of substantial computational overhead. In this paper, we propose Second Guess , a novel parameter-free prompting technique that operates under the key insight that models exhibiting stable knowledge will provide consistent answers regardless of available options (Figure 1 ). The method prompts each model twice per question: once with the standard choice set and again with an “I don’t know” option added. We treat answer discrepancies between these two prompts as an abstention signal, while unchanged responses are retained as confident predictions. This approach demands only two forward passes per question, making it applicable even to black-box SLMs without access to (internal) model parameters or uncertainty scores. We evaluate Second Guess on four SLMs (2B–8B parameters) and four MCQ benchmarks, and contribute: (i) a parameter-free abstention technique with only 2 queries and no access to internal state of the model; (ii) comprehensive evaluation showing Second Guess achieves highest composite risk improvements while maintaining robustness to fine-tuning; and (iii) empirical insights demonstrating that improvements stem from detecting unstable answer switching and showing the method is best suited for challenging tasks and lower-performing models. 2 Related Work Abstention, the ability to decline answering a question when uncertain, is critical for deployed language models, as incorrect predictions often cause greater harm than no prediction at all in safety critical domains like healthcare. Existing approaches include training-based methods that require weight access and retraining ( 27 ; 11 ) and prompt-based methods that rely on self-reported uncertainty ( 28 ; 9 ) . Consistency-based approaches measure behavioral stability across perturbations ( 3 ) , but existing work uses unstructured stochastic sampling requiring many forward passes. While these approaches are well established, achieving reliable abstention in LLMs remains a challenge ( 13 ; 24 ) . Small language models (SLMs), typically fewer than 10 billion parameters, increasingly deployed on edge and mobile devices where latency and privacy constraints preclude API calls ( 14 ; 20 ) . Calibration generally improves with model scale ( 10 ; 5 ) : smaller models exhibit greater overconfidence and susceptibility to perturbations, thereby more in need of reliability harness around them ( 1 ) . Crucially, SLMs are often deployed autonomously in embedded systems where incorrect answers carry direct consequences, yet sampling-based consistency methods ( 3 ) are computationally prohibitive. 3 Methodology 3.1 Second Guess Consider a question Q and its four corresponding answer choices \mathcal{C}=\{c_{1},c_{2},c_{3},c_{4}\} presented in a zero-shot prompt 1 1 1 We reproduce a standard multiple-choice prompt template in Figure 3 in Appendix A , demonstrating the structured format used in our experiments. . In the first stage, the model’s response R_{1} is extracted from the original prompt. In the second stage, an abstention option "I don’t know" is inserted at a random position to \mathcal{C} as the fifth option, and the model’s response R_{2} to the augmented prompt is extracted. If the model’s answer remains consistent between stages ( R_{1}=R_{2} ), the original answer choice is retained. Conversely, if the response changes ( R_{1}\neq R_{2} ), regardless of what it changes to, the model’s output is classified as an abstention response "I don’t know". Formally, the final model answer R is derived as follows: R=\begin{cases}R_{1}&\text{if }R_{1}=R_{2}\\ \text{``I don't know''}&\text{if }R_{1}\neq R_{2}\end{cases} (1) 3.2 Experimental Setup 3.2.1 Models and Datasets We evaluate four instruction-tuned language models: Mistral-7B-Instruct-v0.3 8 , Llama-3.1-8B-Instruct 6 , IBM Granite-3.3-2B-Instruct 2 2 2 https://huggingface.co/ibm-granite/granite-3.3-2b-instruct , and Qwen3-4B-Instruct-2507 25 . More information about the models used is provided in Appendix A (Table 2 ). We evaluate model behavior across four multiple-choice question-answering datasets: CommonsenseQA 18 , QASC 12 , MMLU-Pro 22 , and SuperGPQA 4 . For each model and dataset, we sample 100 random questions and normalize them to contain exactly four options per question. A detailed summary of the datasets is provided in Appendix A (Table 3 ). 3.2.2 Baselines Because our approach is lightweight and requires only model outputs, we select baselines that are similarly lightweight and practical. Therefore complex and sampling based approaches like proposed by 3 are not evaluated against. The baselines evaluated are: 1. Original Prompt only : The standard multiple-choice options \mathcal{C}=\{c_{1},c_{2},c_{3},c_{4}\} without augmentation. The options are shuffled randomly to avoid bias due to ordering. 2. Augmented Prompt only : A single-stage variant where the abstention option “I don’t know” is added at a random position to the original options from the outset, forming \mathcal{C}^{\prime}=\{c_{1},c_{2},c_{3},c_{4},\text{``I don't know''}\} . This baseline directly exposes the model to the possibility of abstaining in a single pass. 3. Self-Evaluation 9 : A two-step prompting strategy adapted for abstention. In the first stage, the model generates an answer; in the second stage, it is presented with this answer and asked to verify its correctness, abstaining when it doesn’t accept the proposed answer. 4. Entropy-based Uncertainty Estimation and Thresholding 15 : A post-hoc baseline where responses with output token entropy exceeding the mean plus one standard deviation of the entropy distribution across the entire dataset are automatically converted to abstentions;

Read the complete text