ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora
Nikolas Adaloglou, Diana Petrusheva, Mohamed Asker, Felix Michels, Markus Kollmann · 2025
Read ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora by Nikolas Adaloglou, Diana Petrusheva, Mohamed Asker, Felix Michels, Markus Kollmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07068.txt.
Introduction
Nikolas Adaloglou and four co-authors remove a hidden dependency from out-of-distribution detection, the task of flagging images that do not belong to any known class. Leading methods built on vision-language models like CLIP need the names of the in-distribution classes in advance, yet those labels can be unavailable, unreliable, or outdated after deployment shifts. ClusterMine instead mines positive concepts from large text corpora that anyone can obtain.
The method combines two consistency checks: visual-only consistency, grouping image samples by clustering, and zero-shot image-text consistency against candidate concepts. It is the first approach to reach state-of-the-art detection performance with no access to positive labels at all. Tests across many CLIP models show it scales and stays robust when the in-distribution data itself shifts.
Catalog record
| CREATOR | Nikolas Adaloglou, Diana Petrusheva, Mohamed Asker, Felix Michels, Markus Kollmann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07068
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora Thanks: Corresponding author. Nikolas Adaloglou, Diana Petrusheva, Mohamed Asker, Felix Michels, Markus Kollmann Abstract Large-scale visual out-of-distribution (OOD) detection has witnessed remarkable progress by leveraging vision-language models such as CLIP. However, a significant limitation of current methods is their reliance on a pre-defined set of in-distribution (ID) ground-truth label names (positives). These fixed label names can be unavailable, unreliable at scale, or become less relevant due to in-distribution shifts after deployment. Towards truly unsupervised OOD detection, we utilize widely available text corpora for positive label mining, bypassing the need for positives. In this paper, we utilize widely available text corpora for positive label mining under a general concept mining paradigm. Within this framework, we propose ClusterMine, a novel positive label mining method. ClusterMine is the first method to achieve state-of-the-art OOD detection performance without access to positive labels. It extracts positive concepts from a large text corpus by combining visual-only sample consistency (via clustering) and zero-shot image-text consistency. Our experimental study reveals that ClusterMine is scalable across a plethora of CLIP models and achieves state-of-the-art robustness to covariate in-distribution shifts. The code is availiable at https://github.com/HHU-MMBS/clustermine_wacv_official . 1 Introduction Given a set of training images comprising the in-distribution (ID), visual out-of-distribution (OOD) detection aims to identify images sampled from a shifted distribution while having access only to the ID [ 70 , 1 ] . For each ID, there exists a predefined set of semantic categories \mathcal{Y}_{\text{real}} that can be assigned to each sample [ 72 ] . Recently, distribution shifts have been categorized into semantic shifts ( \mathcal{Y}_{\text{OOD}}\cap\mathcal{Y}_{\text{real}}=\varnothing ) and non-semantic shifts, also known as covariate shifts. A covariate-shifted ID pair (x,y) preserves its label y\in\mathcal{Y}_{\text{real}} while x undergoes a distribution shift [ 6 , 73 ] . In the context of natural image recognition, examples of covariate shifts can arise from camera variations, background changes, or style shifts [ 23 , 21 , 62 ] . Motivation. In practice, the ID images exhibit varying levels of class specificity and overlapping concepts [ 54 , 30 , 44 , 5 , 4 ] . While \mathcal{Y}_{\text{GT}} often being a good proxy of \mathcal{Y}_{\text{real}} in small data regimes, GT label names at scale can be incomplete, unreliable, or arbitrarily defined. Deriving a comprehensive set of visual concepts that fully characterizes the ID is challenging [ 61 ] . For instance, automated curation methods, such as hashtags [ 55 , 42 ] , are frequently underdescriptive or inconsistent with natural language [ 7 , 67 ] . Fixed label names can bottleneck performance when class semantics shift due to in-distribution drifts, a common occurrence after deployment. To overcome these challenges, our work proposes to extract ID concepts directly from large text corpora that are aligned with the ID images. Figure 1 : An overview of the label mining framework for OOD detection using CLIP . Given a text corpus \mathcal{Y}_{\text{corpus}} and its feature representation \mathcal{Z}_{\text{corpus}} , ClusterMine and PosMine aim to extract in-distribution-related class names \mathcal{Y}_{\text{pos}} in the shared vision-language space of CLIP. \mathcal{Z}_{\text{neg}} can be either realized as the non-overlapping elements of \mathcal{Y}_{\text{pos}} and \mathcal{Y}_{\text{corpus}} , or as the most dissimilar text representations from \mathcal{Z}_{\text{pos}} (negative label mining). The OOD detection score is S(x) . Best viewed in color. Detecting covariate shifts. An ideal OOD detector should be sensitive to semantic shifts and, at the same time, robust to covariate shifts [ 68 ] . Misclassified covariate-shifted samples can undermine the generalization capability of an ID classifier. When these samples are flagged as OOD, they render the classifier’s predictions unreliable [ 72 ] . Yang et al. [ 70 ] have recently demonstrated that existing supervised OOD detection algorithms are susceptible to covariate shifts. To date, the OOD robustness to covariate shifts has not been sufficiently explored [ 67 , 17 , 71 ] . Scalability and vision-language (VL) models. In parallel, VL models have been established for large-scale OOD detection benchmarks [ 49 , 20 , 46 , 16 , 1 ] . However, the majority of literature comparisons [ 45 , 46 , 74 , 65 ] are conducted with small-sized models relative to the pretraining dataset, such as CLIP ViT-B [ 14 , 49 ] . Smaller-sized models bottleneck performance when other scaling factors remain constant [ 35 , 12 ] . This raises questions whether the recently developed methods are scalable with respect to model size. Plain image-text similarities have shown non-optimal performance as the model size increases [ 1 ] . Last but not least, current VL methods for OOD detection assume access to \mathcal{Y}_{\text{GT}} , which simplifies the task at hand. Questionable benchmarks. Another point of criticism concerns the adopted large-scale OOD benchmarks. Huang et al. [ 31 ] introduced a quadruplet of OOD benchmarks for ImageNet with unknown levels of semantic overlap and limited diversity. This benchmark suite remains widely adopted to date [ 34 , 46 , 65 ] , despite recent studies raising concerns about its validity [ 70 ] . For instance, Bitterwolf et al. [ 10 ] estimate a semantic overlap of almost 60% in the Places OOD benchmark compared to ImageNet [ 13 ] . Several newer datasets have been carefully designed to minimize semantic overlap with the ID [ 10 ] . The above factors - reliance of VL models on ID label names, covariate shifts, scalability, benchmarks - OOD detection methods may have begun to overfit the idiosyncrasies of certain models and benchmarks, mirroring concurrent trends observed in image classification [ 8 ] . In other words, it is unclear how robust the recent advancements in large-scale OOD detection using CLIP models are. In this paper, we present a general framework for extracting relevant concepts from a large text corpus that encompasses existing methods. Moreover, we develop the first positive label name mining method, ClusterMine, which extracts high-quality ID-related concepts from the corpus. A systematic large-scale OOD benchmarking reveals that ClusterMine i) scales with model size and performance, ii) achieves state-of-the-art OOD detection AUROC on most benchmarks without access to ground truth ID label names , and iii) is robust against most ID shifts on ImageNet, iv) outperforms state-of-the-art approaches in near-OOD benchmarks, such as AdaNeg [ 74 ] that impose additional requirements (i.e. sequential access to OOD samples). 2 Related work Supervised visual OOD detection methods. Supervised OOD detection involves training or fine-tuning a classifier on labeled ID samples [ 39 , 33 , 63 , 36 ] . Posthoc detectors then typically compute an OOD score such as the maximum softmax probability (MSP) [ 24 ] . Alternative training-time modifications include regularizations by incorporating OOD data [ 40 , 66 , 39 ] , auxiliary samples [ 25 ] , or synthetic outliers [ 15 ] . However, when benchmarked rigorously and at scale, Yang et al. [ 70 ] demonstrate that no OOD detection score [ 40 , 29 , 75 , 57 , 56 ] consistently outperforms MSP when applied to a supervised classifier, which has shifted the focus to VL models. OOD detection using CLIP. Fort et al. [ 18 ] utilize CLIP by combining ID and OOD candidate