At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization
Praneet Suresh, Jack Stanley, Sonia Joseph, Luca Scimeca, Danilo Bzdok · 2026
Read At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization by Praneet Suresh, Jack Stanley, Sonia Joseph, Luca Scimeca, Danilo Bzdok (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26396.txt.
Introduction
Praneet Suresh and colleagues use sparse autoencoders, tools that unpack a transformer's internal activity into readable concepts, to map where the model's understanding breaks down. Inputs unlike the training data, from typos to jailbreak prompts, are the test cases.
Such out-of-distribution inputs push the model to activate a larger number of spurious concepts internally. That signal becomes a meter for how far an input has drifted, measured at inference time inside the model's own computation. The team turns it into a fine-tuning strategy that improves robustness, aiming at safer deployment in science, business, and government.
Catalog record
| CREATOR | Praneet Suresh, Jack Stanley, Sonia Joseph, Luca Scimeca, Danilo Bzdok |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26396
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization Praneet Suresh, Jack Stanley, Sonia Joseph, Luca Scimeca, Danilo Bzdok Abstract Pre-trained transformers have demonstrated remarkable generalization abilities, at times extending beyond the scope of their training data. Yet, real-world deployments often face unexpected or adversarial data that diverges from training data distributions. Without explicit mechanisms for handling such shifts, model reliability and safety degrade, urging more disciplined study of out-of-distribution (OOD) settings for transformers. By systematic experiments, we present a mechanistic framework for delineating the precise contours of transformer model robustness. We find that OOD inputs, including subtle typos and jailbreak prompts, drive language models to operate on an increased number of spurious concepts in their internals. We leverage this device to quantify and understand the degree of distributional shift in prompts, enabling a mechanistically grounded fine-tuning strategy to increase the robustness of LLMs. Expanding the very notion of OOD from input data to a model’s private computational processes—a new transformer diagnostic at inference time—is a critical step toward making AI systems safe for deployment across science, business, and government. 1 Introduction The assumption that training and test data are identically distributed underpins most machine learning theory and practice ( 5 ) . Yet, this assumption is rarely satisfied outside controlled research environments ( 50 ) . Large language models (LLMs), despite their scale and versatility, are not immune to this generalization challenge. This model class often displays erratic and brittle failure modes under distribution shift ( 41 ; 25 ) . Compounding this issue, the scale of pre-training and the effects of post-training optimization can obscure the specific limitations of transformer models ( 45 ; 23 ) . Systematically identifying such lapses would boost trust in the deployment of LLMs in safety-critical environments. A promising way forward may be to explore how LLMs represent knowledge internally. According to the linear representation hypothesis, LLMs approximate human understandable abstractions as linear directions in its learned activation space using an overcomplete basis of non-orthogonal directions ( 47 ; 12 ) . Sparse autoencoders (SAEs) build directly upon this theoretical framework to disentangle human understandable concepts from the intermediate representations of the transformer ( 10 ; 7 ; 57 ; 16 ) . Such principled re-interpretation of otherwise opaque transformer internals offers significant promise in improving interpretability and inference-time auditing of model reliability. To this end, we recast SAEs as a microscope trained on the boundaries of a subject LLM’s internal representation space. In particular, our core contributions show that: 1. LLMs infer spurious concepts when encountering input data points that raise OOD exceptions. 2. Minor distribution shifts in input prompts, detectable by SAEs, can lead to drops in LLM performance on established performance benchmarks. 3. SAE-derived indicators provide a sharp lens into per-sample distribution shift within LLMs, allowing the manifold-informed selection of samples for improved fine-tuning performance. 4. SAEs flag successful jailbreak attempts as OOD exceptions, which we counter by aligning their vulnerability-sensitive directions in representation space, safeguarding LLMs against such attacks. 2 Related work 2.1 Out-of-distribution generalization The capacity of neural networks to generalize beyond their training distribution has been extensively investigated ( 63 ; 54 ; 2 ; 40 ) . A key aspect of such generalization is robustness to encountering “out-of-distribution” (OOD) settings, which has motivated a range of methods for detecting distributional shift. ( 21 ) introduce “maximum soft probability”, noting that OOD samples have lower maximum softmax probabilities than in-distribution samples. ( 35 ) model learned representations as class-conditional Gaussians, using Mahalanobis distance to detect distribution shift. ( 22 ) leverages large auxiliary datasets of outliers to improve detection of distribution shift, while ( 39 ) improves upon softmax-based scores with a more unified energy function designed for the same purpose. With LLMs, this broad transferability manifests itself in the form of impressive zero-shot, few-shot, and in-context learning capabilities ( 51 ; 8 ) ( 30 ) ( 60 ) . Despite internet-scale pre-training, even frontier AI systems are known to exhibit sensitivity to prompt phrasing, engage in faulty reasoning, and confabulate details ( 29 ; 56 ) . There has been some recent work exploring LLM fragility in the face of unstructured inputs ( 56 ; 15 ) , jailbreaks ( 66 ; 59 ; 55 ; 62 ) , and novel shifts in context ( 18 ) . In contrast to model-naive approaches, our approach surveys a continuous in-distribution to out-of-distribution transition in the LLM’s latent manifold. This novel paradigm allows us to chart the limits of LLM generalization that are otherwise obfuscated by large, heterogeneous pre-training datasets. 2.2 Transformer representations The linear representation hypothesis asserts that transformer embedding spaces contain linearly composable elements that can be unravelled and examined using simple linear transformations ( 47 ; 12 ) . Building upon this principle, sparse autoencoders (SAEs) have emerged as powerful tools for decomposing dense transformer activations into an overcomplete set of interpretable linear components ( 10 ; 7 ; 57 ; 16 ) . Similar approaches have been extended to vision transformers with comparable success ( 28 ; 27 ) . SAEs have been shown to surface highly interpretable and even directable concepts from transformer representations ( 46 ) ( 38 ) . Recent work by ( 43 ) and ( 13 ) aims to more systematically characterize these linear feature manifolds with the assistance of SAEs, and ( 14 ) attempts to explore the origins and utility of SAE reconstruction error in its own right. In our study, we uniquely exploit these linear directions in transformer representation space to effectively differentiate between in-distribution and out-of-distribution samples. This reveals where in semantic concept space the model replaces compositional features with spurious features. Further, we show that the careful excision of these directions allows us to reinforce an LLM against harmful adversarial inputs without sacrificing its general reasoning abilities. 3 Methods 3.1 Rationale SAEs have become a go-to solution to mirror LLM internals. This model class opens up new paths for insight into the mechanisms behind concept representations, circuits, and steerable outputs ( 1 ) . Building on these practical successes, we here repurpose SAEs as a surrogate model for laying out a subject LLM’s spectrum of in-distribution versus off-distribution internal processing streams. If we assume that the SAE learns a useful approximation of the transformer representation space, it is likely that unexpected and OOD inputs will result in high reconstruction error, a large number of concepts required to represent them, or both. This setup allows us to flag OOD events on the fly as a transformer is processing an input, before the model even starts to form a response. We thus provide a device that extends the notion of in-versus-out-of-distribution from mere data points to the complex processing operations private to LLM internals. If these off-distribution events are correctly tracked, this “inside knowledge” should enable surgical corrective procedures on the LLM, which we showcase in important AI safety use cases like jailbreaking. 3.2 Out-of-distribution inputs It can be challenging to define true “out-of-distribution” datasets for massively pre-trained LLMs ( 6 ; 37 ) . Therefore, we first construct a toy setting where we can more cleanly evaluate our hypotheses. We