Uncovering Uncertainty in Transformer Inference
Greyson Brothers, Willa Mannering, Amber Tien, John Winder · 2024
Read Uncovering Uncertainty in Transformer Inference by Greyson Brothers, Willa Mannering, Amber Tien, John Winder (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05768.txt.
Introduction
Greyson Brothers, Willa Mannering, Amber Tien, and John Winder look inside transformer language models as they pick each next word. Their 2024 paper tests an idea called the Iterative Inference Hypothesis. The idea says the model's inner signals are refined step by step, like drafts moving toward a final answer.
Tracking word signals through the model, they found the inner draft steadily moves closer to the final word. How fast it settles also tracks how unsure the model is. Slow settling tends to mark wrong picks; fast settling marks right ones. They built a check based on this pattern and tested it on a set of idioms. The check sorted right from wrong outputs well.
Catalog record
| CREATOR | Greyson Brothers, Willa Mannering, Amber Tien, John Winder |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05768
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncovering Uncertainty in Transformer Inference Greyson Brothers, Willa Mannering, Amber Tien, John Winder Abstract We explore the Iterative Inference Hypothesis (IIH) within the context of transformer-based language models, aiming to understand how a model’s latent representations are progressively refined and whether observable differences are present between correct and incorrect generations. Our findings provide empirical support for the IIH, showing that the n t h n^{th} token embedding in the residual stream follows a trajectory of decreasing loss. Additionally, we observe that the rate at which residual embeddings converge to a stable output representation reflects uncertainty in the token generation process. Finally, we introduce a method utilizing cross-entropy to detect this uncertainty and demonstrate its potential to distinguish between correct and incorrect token generations on a dataset of idioms. 1 Introduction and Related Work Transformer-based architectures [ 13 ] currently dominate artificial intelligence applications and serve as the underlying architecture for most Large Language Models (LLMs). While LLMs show impressive emergent abilities, these models exhibit limitations such as hallucinations and biased outputs which pose significant societal challenges [ 10 ] . Inaccurate outputs can mislead users, while malicious actors can exploit AI models to create deceptive images, videos, and text to represent fictional occurrences as truth. Mitigating harms caused by model misuse, biased outputs, or misalignment with human values is a primary motivation behind research and policy decisions related to AI interpretability [ 2 ] . In this work we investigate a novel method for detecting uncertainty during the token generation process of transformer-based language models. One framework that has emerged for understanding the feed-forward behavior of residual models, such as the transformer, is the Iterative Inference Hypothesis (IIH) [ 1 , 4 , 8 ] . This hypothesis posits that predictions are formed in the residual stream, and that each block in a residual architecture incrementally updates these predictions in a direction of decreasing loss [ 7 ] . A related line of research on in-context learning has suggested that transformers trained on autoregressive tasks are closely related to formulations of iterative optimization algorithms, namely gradient descent [ 14 ] . We combine these two threads of research, framing transformer inference as an optimization process that iteratively updates the n^{th} input embedding (i.e. the last word in the input sequence) to converge toward the most likely next-token embedding given the context and model weights. We propose methods for evaluating how input embeddings evolve towards output embeddings and find preliminary evidence of observable differences between correct and incorrect outputs, suggesting that these metrics could serve as useful indicators of a model’s output certainty. Our contributions are as follows: 1. We find preliminary evidence that the n^{th} token embedding in the residual stream follows a path of decreasing loss in token embedding space, supporting the IIH. 2. We propose a novel method for detection of uncertainty during the token generation process and find preliminary evidence that this metric reflects observable differences in correct and incorrect generations. 2 Methods Here we define methods that offer insight into how representations in the residual stream evolve during inference. The transformer architecture, excluding token embedding and unembedding, can be succinctly represented by the following recurrence relation: \mathbf{r}_{i+1}=\mathbf{r}_{i}+L_{i+1}(\mathbf{r}_{i}), where \mathbf{r}_{i}=[e_{1}^{i},...,e_{n}^{i}] represents the set of token embeddings in the residual stream after an update from layer L_{i} . Each layer contains attention and feed-forward sublayers. With \mathbf{r}_{0} as the set of input token embeddings plus positional encodings, the residual stream is the sequence (\mathbf{r}_{0},\mathbf{r}_{1},...,\mathbf{r}_{k}) for a model with k layers. In this work we are particularly interested in tracking the evolution of the embedding of the n^{th} input token, e_{n}^{i} , as shown in red in Figure 1 . Its residual representation after the final layer update, e_{n}^{k} , is used to predict the next token in an autoregressive framework. (a) Figure 1: The transformer as a recurrence relation, iteratively refining a prediction for the next token. In the convolutional architecture of the original residual network (ResNet), the residual stream contained shortcut connections that transformed the basis of latent space in order to reduce its dimensionality at regular intervals [ 5 ] . However, in the transformer, the residual stream has an entirely linear structure that preserves the basis of the embedding space [ 3 ] . Each layer adds an output to each residual embedding, corresponding to translations in token embedding space. Thus the residual stream can be viewed as a path through token embedding space, and each point along the path can be mapped back to a distribution over tokens via methods like logit lens [ 9 ] . To get the intermediate distribution predicted by the residual stream after layer L_{i} , we pass e_{n}^{i} through the output layer norm and then to the model’s output head to obtain logits over the vocabulary. We refer to these logits as residual predictions. This framework forms the basis of our methods and analysis. 2.1 Residual Cross-Entropy We present a method by which cross-entropy can be used to measure the evolution of residual predictions during token generation. The IIH posits that the output of each layer updates the residual prediction in a direction of decreasing loss. We thus examine per-layer changes in cross-entropy, the loss function used to train transformers on autoregressive language tasks [ 6 ] , including the specific model we examine here (GPT-2 XL) [ 11 ] . Cross-entropy is a measure of dissimilarity, requiring a candidate and a target distribution. We use the residual predictions described above as the candidate, and for the target distribution we examine two choices: (1) the one-hot distribution of the token sampled deterministically by taking the argmax the model’s predicted probabilities, denoted \hat{y} , and (2) the one-hot distribution of the ground truth next token given by the dataset, denoted y . These measures are equivalent to the negative log likelihood of the sampled and target tokens respectively, as elaborated in A.1 . When the sampled output is correct, or \hat{y}=y , these two measures are the same. Examining how layer updates affect cross-entropy with respect to the training objective y is necessary for our evaluation of the IIH, but the ground truth next token may be unavailable or ambiguous at inference time, making \hat{y} more practical for analyzing the residual stream in an online setting. Some existing works have examined Kullback-Leibler divergence between residual predictions and the final logits output by the model [ 9 , 8 ] , rather than \hat{y} as we do here. Our approach implicitly penalizes generations that have high output entropy, which is advantageous for our study of uncertainty during generation. We provide a comparison between the two in A.3 . 2.2 Model To investigate how representations evolve in the residual stream we examine GPT-2 XL [ 11 ] , which has 48 layers and 1.5 billion parameters. We chose this model as it has low compute requirements while still being representative of frontier model architectures. In addition, it has widely used open source implementations and official weights made publicly available by OpenAI on HuggingFace 1 1 1 Official weights for GPT-2 XL hosted on Hugging Face: https://huggingface.co/openai-community/gpt2-xl . 2.3 Inference Data For our preliminary study of the IIH and model uncertainty,