Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning
Brett Barkley, Preston Culbertson, David Fridovich-Keil · 2026
Read Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning by Brett Barkley, Preston Culbertson, David Fridovich-Keil (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05638.txt.
Introduction
A classifier that cannot say "this is unfamiliar" will fail silently on shifted data. Brett Barkley, Preston Culbertson, and David Fridovich-Keil revisit whether detecting out-of-distribution inputs really needs labeled training or fine-tuning. Their claim: modern pretrained models already carry the needed geometry in their frozen representations.
Across 59 backbone-and-task pairings in vision and language, they test two label-free detectors. One fits a global Mahalanobis estimator on unlabeled representations; the other, ReSCOPED, estimates typicality with a light diffusion model. Both performed strongly without any fine-tuning, reframing the problem as one of reading representations well.
Catalog record
| CREATOR | Brett Barkley, Preston Culbertson, David Fridovich-Keil |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05638
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning Brett Barkley, Preston Culbertson, David Fridovich-Keil Abstract Models trained with deep learning often fail to signal when inputs fall outside their training data manifold, leading to unreliable predictions under distribution shift. Prior work suggests that effective out-of-distribution (OOD) detection often requires class-conditional modeling or specialized models obtained through supervised fine-tuning. We revisit this assumption in modern pretrained models and show that their frozen representations already encode sufficient geometric structure for accurate label-free OOD detection. Across 59 backbone–task pairings spanning vision and language, we compare two complementary label-free detectors: a global Mahalanobis estimator fit on unlabeled latent representations, and ReSCOPED, a lightweight, diffusion-based typicality estimator operating on the same features at a local level. Despite their different detection mechanisms, representation scaling reveals a consistent regime-dependent pattern: both local and global detectors’ absolute performance improves with better representation quality, and performance gaps between the two detectors disappear across both language and vision tasks as representations scale. These results suggest that label-free OOD detection depends strongly on the geometry exposed by frozen pretrained backbones, reducing the importance of detector choice as backbone scale increases and enabling efficient deployment directly on frozen models. Figure 1 : AUROC for Mahalanobis and ReSCOPED across DINO generation, DINOv3 model scale (S, B, L), and Qwen3 model scale (0.6, 1.7, 4.0). For DINO, AUROC is averaged across the 9 tasks in Table 3 ; for Qwen3, AUROC is averaged across the tasks in Table 2 . Across both vision and language, scaled frozen backbones yield high-performing label-free OOD detectors while also reducing the gap between global Mahalanobis and local score-curvature detectors. In weaker frozen BERT representations, rankings are dataset-dependent: ReSCOPED is stronger on CLINC150, while Mahalanobis is stronger on SST. 1 Introduction Whether utilizing a Vision Transformer (ViT) 9 for image classification or a Large Language Model (LLM) for natural language processing, current architectures generate high confidence predictions even when presented with inputs that fall outside their training manifold. Unsupervised detection strategies are particularly valuable because they rely only on unlabeled in-distribution data and therefore remain task agnostic ( 24 ; 13 ; 16 ) . However, traditional out-of-distribution (OOD) methods face persistent challenges: likelihood-based estimators can assign higher probabilities to OOD samples than to training data ( 30 ; 5 ) , while reconstruction-based methods are sensitive to bottleneck tuning ( 1 ; 34 ) . In this work, we provide empirical evidence that frozen representation structure governs not only label-free OOD detection performance, but also how much detector choice matters. Here, representation structure refers to how readily OOD-relevant features are exposed to global and local detectors as backbone families and model scales change. A common perspective across modalities is that strong OOD detection often requires either class labels for in-distribution (ID) data or adaptation of the learned representation. For example, 22 model feature representations using class-conditional Gaussian distributions, an assumption that persists in later extensions ( 23 ; 29 ; 43 ) . Similarly, prior work has shown that near-OOD detection remains challenging when using frozen representations, and that supervised fine-tuning can substantially improve performance ( 11 ; 45 ) . At the same time, OOD detection methods have been shown to exhibit inconsistent performance across datasets, with no single detector consistently outperforming others ( 41 ) . Together, these results suggest that OOD performance depends not only on the detector applied to a representation, but also on the quality and geometry of the representation itself. This motivates the fully unsupervised setting, where only unlabeled in-distribution data is available. While labels and fine-tuning can improve OOD detection, they limit deployment alongside frozen foundation models: class labels are often unavailable, especially in language settings where models are trained on large-scale unlabeled corpora and deployed on open-ended inputs, and fine-tuning requires maintaining task-adapted model variants. Recent work such as DiffPath has focused on making label-free OOD detection more general-purpose by using a single pretrained diffusion model across multiple tasks ( 16 ) . Recent work has also observed that DINOv2 ( 33 ) and other foundation-model representations can support competitive OOD detection with simple scores such as nearest neighbors ( 47 ; 48 ) . However, these observations do not establish whether this behavior generalizes across fundamentally different detectors and modalities, nor whether the inconsistencies observed in prior OOD detectors persist as representation quality improves. We therefore ask whether the frozen model already used for the downstream task encodes sufficient representation geometry for label-free OOD detection, avoiding the need for labels or specialized models obtained through supervised fine-tuning. We study this question using two deliberately different OOD detectors. First, a label-free global Mahalanobis distance-based scheme which models the frozen representation distribution using only unlabeled in-distribution latents, removing the class-conditioning used in prior Mahalanobis-based methods ( 22 ; 23 ; 29 ; 43 ) . The second is a typicality-based method, ReSCOPED, that builds upon SCOPED ( 3 ) and evaluates local representation geometry using diffusion-based score and curvature information. In particular, diffusion-based detectors can be viewed as multi-scale filters that selectively expose structure at different granularities, allowing them to recover OOD signal when it is not globally accessible. These methods differ substantially in complexity but test the same underlying hypothesis: that modern frozen representations already contain enough OOD-relevant structure for high-performing label-free detection, whether measured through global covariance structure or local score-curvature information. We validate this representation-centered view across 59 backbone-task pairings sourced from the Bonsai repository ( 12 ) and HuggingFace Transformers ( 44 ) . As shown in Figure 1 , stronger representations improve OOD detection as measured by the area under the receiver operating characteristic (AUROC) curve across both vision and language while also reducing the importance of detector choice. In weaker and moderately strong representations, global Mahalanobis and local score-curvature methods can trade places across datasets and backbones. However, as representation quality improves, the performance gap between methods becomes negligible, indicating convergence between global and local views of OOD-relevant geometry. This regime-dependent pattern appears in both vision and language: detector choice matters more in weaker or less structured representations, while in strong modern backbones the performance gap between simple global and expressive local methods is small and both achieve strong performance. Because both detectors operate directly on latents from frozen models already used downstream, this representation-centered view also enables efficient deployment. For example, ReSCOPED can act as a lightweight LLM gatekeeper prior to decoding, adding only 5.4 ms of latency for Qwen3-4B ( 46 ) . Our ablations further show that the highest performing score-curvature detection occurs across a broad and consistent diffusion-noise plateau, suggesting that OOD-relevant structure is exposed at stable scales across Transformer ( 42 ) backbones. Key Contributions: 1. We show that modern pretrained vision and language representations enable strong label-free OOD detection without fine-tuning or class labels, using ReSCOPED and Mahalanobis detectors to compare local/global views of representation geometry. 2. We show that backbone family and scale shape detector choice: in earlier or smaller backbones, no method is universally best, while DINOv3 and Qwen3 scaling make the gap between global and local OOD detectors negligible. 3. We identify a consistent denoising-scale plateau across vision and language Transformers, suggesting OOD-relevant structure becomes