UAT-Lite : Inference-Time Uncertainty-Aware Attention for Pretrained Transformers
Elias Hossain, Shubhashis Roy Dipta, Subash Neupane, Rajib Rana, Ravid Shwartz-Ziv, Ivan Garibay, Niloofar Yousefi · 2026
Read UAT-Lite : Inference-Time Uncertainty-Aware Attention for Pretrained Transformers by Elias Hossain, Shubhashis Roy Dipta, Subash Neupane, Rajib Rana, Ravid Shwartz-Ziv, Ivan Garibay, Niloofar Yousefi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02952.txt.
Introduction
Transformers that know when they are unsure are the goal of Elias Hossain and 6 co-authors in a 2026 paper. Language models are often overconfident, assigning high certainty to wrong answers. Adjusting output probabilities after the fact leaves the internal computation unchanged, while ensembles cost much more training and storage.
Their framework, UAT-LITE, injects uncertainty directly into self-attention at inference time. Monte Carlo dropout creates varied forward passes, and the spread estimates token-level uncertainty. That signal then modulates attention without changing pretrained weights. A layer-wise analysis tracks how uncertainty builds with depth. Across question answering and classification benchmarks, calibration error fell by about a fifth while accuracy held.
Catalog record
| CREATOR | Elias Hossain, Shubhashis Roy Dipta, Subash Neupane, Rajib Rana, Ravid Shwartz-Ziv, Ivan Garibay, Niloofar Yousefi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UAT-Lite : Inference-Time Uncertainty-Aware Attention for Pretrained Transformers Elias Hossain, Shubhashis Roy Dipta, Subash Neupane, Rajib Rana, Ravid Shwartz-Ziv, Ivan Garibay, Niloofar Yousefi Abstract Neural NLP models are often miscalibrated, assigning high confidence to incorrect predictions, which undermines selective prediction and high-stakes deployment. Post-hoc calibration methods adjust output probabilities but leave internal computation unchanged, while ensemble and Bayesian approaches improve uncertainty at substantial training or storage cost. We propose UAT-Lite , an inference-time framework that makes self-attention uncertainty-aware using approximate Bayesian inference via Monte Carlo dropout in pretrained transformer classifiers. Token-level epistemic uncertainty is estimated from stochastic forward passes and used to modulate self-attention during contextualization, without modifying pretrained weights or training objectives. We additionally introduce a layer-wise variance decomposition to diagnose how predictive uncertainty accumulates across transformer depth. Across the SQuAD 2.0 answerability, MNLI, and SST-2, UAT-Lite reduces Expected Calibration Error by approximately 20% on average relative to a fine-tuned BERT-base baseline while preserving task accuracy, and improves selective prediction and robustness under distribution shift. 1 1 1 github.com/eliashossain001/uq_decomposition 1 Introduction Figure 1: Left, A: Standard inference: deterministic forward pass without uncertainty. Center, B: Output-level uncertainty estimation via stochastic inference or post-hoc calibration. Right, C: UAT-Lite : uncertainty-aware inference in which epistemic uncertainty modulates attention with diagnostic insight. Pretrained transformer-based language models have achieved state-of-the-art performance across a wide range of NLP tasks, from question answering to clinical decision support Devlin et al. 2019 ; Brown et al. 2020 . Despite these advances, modern transformers exhibit systematic miscalibration , frequently assigning high confidence to incorrect predictions Desai and Durrett 2020 . This misalignment between confidence and correctness is particularly problematic for selective prediction and related high-stakes applications. fig. 1 summarizes existing approaches to uncertainty estimation. Standard transformer inference is deterministic and does not represent uncertainty during internal computation ( fig. 1 A). Post-hoc calibration methods, such as temperature scaling, adjust confidence only after representations are formed, leaving internal reasoning unchanged ( fig. 1 B; Guo et al. 2017 ). Stochastic approaches, including Monte Carlo (MC) dropout and deep ensembles ( Gal and Ghahramani 2016 ; Lakshminarayanan et al. 2017 ) , improve uncertainty estimates but largely treat uncertainty as an output-level signal, without influencing attention. This limitation reflects a broader gap: transformers are trained to optimize predictive accuracy rather than confidence reliability Guo et al. 2017 . While post-hoc methods are lightweight and effective in-domain, they do not alter token interactions. Bayesian approaches integrate uncertainty more deeply but typically require architectural modification or specialized training, limiting compatibility with pretrained models. This raises the following question: can epistemic uncertainty shape a transformer’s internal attention at inference time, without retraining or modifying pretrained weights? We argue that effective uncertainty quantification requires uncertainty to actively shape attention , rather than merely annotate final predictions. For ambiguous inputs, a calibrated model should hedge internal reasoning instead of propagating early attention decisions deterministically. Existing stochastic inference methods expose uncertainty only at the output level and do not use it to modulate internal attention patterns ( fig. 1 C). Motivated by this observation, we propose UAT-Lite , an inference-time framework for encoder-based pretrained transformer classifiers. UAT-Lite estimates token-level epistemic uncertainty via Monte Carlo sampling and uses it to downweight attention contributions from unstable tokens, without modifying pretrained weights or training objectives. Our main contributions are: • Uncertainty-Weighted Attention. An inference-time mechanism that modulates self-attention using token-level epistemic uncertainty estimated via Monte Carlo sampling. • Layer-Wise Uncertainty Attribution. A diagnostic variance decomposition that attributes predictive uncertainty across transformer depth. • Comprehensive Evaluation. Empirical evaluation across multiple NLP benchmarks demonstrating improved calibration, selective prediction, and robustness under distribution shift without degrading task performance. UAT-Lite introduces no additional trainable parameters and operates entirely at inference time, requiring only a small number of stochastic forward passes. Post-hoc calibration methods such as temperature scaling are complementary rather than competing. The framework trades modest inference-time cost for improved reliability while avoiding the training and storage overhead of ensemble-based approaches. Our study focuses on encoder-based transformers; extending uncertainty-aware attention to decoder-only generative models is left for future work. 2 Preliminaries Bayesian inference and uncertainty. Bayesian analysis incorporates parameter uncertainty by averaging predictions over probable parameter values inferred from the training data. This perspective distinguishes epistemic uncertainty , which arises from limited data or insufficient coverage and reflects uncertainty about the model itself, from aleatoric uncertainty , which represents irreducible noise in the observations. In large neural models, epistemic uncertainty becomes particularly significant in the presence of semantic ambiguity or distributional shifts, where confident yet incorrect predictions frequently occur. Monte Carlo (MC) Dropout. We briefly recall MC Dropout as an approximate Bayesian inference technique used throughout this work. MC dropout retains dropout layers at inference time and performs multiple stochastic forward passes, yielding a predictive distribution whose variance estimates epistemic uncertainty and whose mean defines the final prediction. Conventional MC dropout treats uncertainty solely at the output level and does not influence internal representations or attention mechanisms, motivating methods that integrate uncertainty directly into the inference process. Token-level uncertainty as a proxy signal. While epistemic uncertainty is often measured at the output level, internal transformer computation is structured around token-level representations and attention interactions. Consequently, uncertainty estimates derived from stochastic embedding representations can serve as a lightweight proxy for identifying unstable or ambiguous token contributions. This perspective motivates the use of token-level uncertainty as a control signal for modulating attention during inference, without requiring architectural changes or retraining. 3 Related Work Uncertainty estimation and calibration for neural networks have been widely studied, with post-hoc methods such as temperature scaling and Platt scaling ( Guo et al. 2017 ; Platt et al. 1999 ) providing simple and effective confidence correction without altering model internals. Ensemble-based approaches, including Deep Ensembles ( Lakshminarayanan et al. 2017 ) , yield strong calibration but incur substantial training and storage overhead. Bayesian neural networks, stochastic inference methods, and recent Bayesian Transformer variants ( Blundell et al. 2015 ; Gal and Ghahramani 2016 ; Ritter et al. 2018 ; Fan et al. 2020 ; Zhang et al. 2021 ; Chen and Li 2023 ) offer principled uncertainty modeling but typically require architectural modification, variational training, or task-specific retraining. In contrast, UAT-Lite integrates epistemic uncertainty directly into self-attention at inference time, preserving pretrained weights and standard training pipelines while enabling uncertainty-aware internal computation. A comprehensive discussion of related work and detailed comparisons is provided in appendix G . 4 UAT-Lite UAT-Lite is an uncertainty-aware extension of pretrained transformer classifiers designed to improve confidence calibration at inference time without modifying model architecture, training objectives, or learned parameters. The framework integrates approximate Bayesian uncertainty estimation into internal attention computation, allowing epistemic uncertainty to actively influence contextualization while preserving the base transformer’s parameterization. In addition, UAT-Lite introduces a diagnostic layer-wise variance decomposition that characterizes how predictive uncertainty evolves across transformer depth, enabling post-hoc analysis without affecting the forward pass. Final predictions are produced via stochastic inference with confidence-aware decision shaping. Overall, UAT-Lite preserves the training cost and parameter count of the underlying encoder, incurring only modest inference-time overhead. Figure 2: Overview of UAT-Lite . Token-level epistemic uncertainty is estimated via Monte Carlo dropout at the embedding layer and used to modulate self-attention within a