CLIP-Free, Label Free, Unsupervised Concept Bottleneck Models
Fawaz Sammani, Jonas Fischer, Nikos Deligiannis · 2025
Read CLIP-Free, Label Free, Unsupervised Concept Bottleneck Models by Fawaz Sammani, Jonas Fischer, Nikos Deligiannis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10981.txt.
Introduction
Fawaz Sammani, Jonas Fischer, and Nikos Deligiannis turn any trained image classifier into an explainable concept model with no labels, in work from twenty twenty-five. Concept models route a call through human readable ideas, which aids trust. Past versions need a big vision language model or costly hand labels for those ideas.
Their method lines up the classifier's output with a text based twin built from class names. The match keeps the original accuracy while it extracts the concepts. The final concept to class map is derived without supervision. Tests on more than forty classifiers set a new state of the art, beating even supervised rivals. The method also writes image captions at top level without the usual backbone model.
Catalog record
| CREATOR | Fawaz Sammani, Jonas Fischer, Nikos Deligiannis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLIP-Free, Label Free, Unsupervised Concept Bottleneck Models Fawaz Sammani, Jonas Fischer, Nikos Deligiannis Abstract Concept Bottleneck Models (CBMs) map dense feature representations into human-interpretable concepts which are then combined linearly to make a prediction. However, modern CBMs rely on the CLIP model to obtain image-concept annotations, and it remains unclear how to design CBMs without the CLIP bottleneck. Methods that do not use CLIP instead require manual, labor intensive annotation to associate feature representations with concepts. Furthermore, all CBMs necessitate training a linear classifier to map the extracted concepts to class labels. In this work, we lift all three limitations simultaneously by proposing a method that converts any frozen visual classifier into a CBM without requiring image-concept labels (label-free), without relying on the CLIP model (CLIP-free), and by deriving the linear classifier in an unsupervised manner. Our method is formulated by aligning the original classifier’s distribution (over discrete class indices) with its corresponding vision-language counterpart distribution derived from textual class names, while preserving the classifier’s performance. The approach requires no ground-truth image–class annotations, and is highly data-efficient and preserves the classifiers reasoning process. Applied and tested on over 40 visual classifiers, our resulting unsupervised, label-free and CLIP-free CBM ( U-F 2 -CBM ) sets a new state of the art, surpassing even supervised CLIP-based CBMs. We also show that our method can be used for zero-shot image captioning, outperforming existing methods based on CLIP, and achieving state-of-art. 1 Introduction Visual classifiers predict a class as a linear combination of dense, high - dimensional visual feature vectors that are difficult to interpret by humans. Concept Bottleneck Models (CBMs) [ 20 ] address this challenge by mapping these feature vectors into a set of human-interpretable concepts, each associated with an activation score (referred to as a concept activation). Predictions are then made as a linear combination of these concept activations. Initial CBMs required image–concept annotations to train the bottleneck layer that maps dense features to concepts. Modern CBMs [ 36 , 60 , 5 ] overcome this limitation by leveraging the CLIP model [ 43 ] or vision–language grounding models [ 48 ] to provide image-concept annotations. Since CLIP models map image and text into a shared embedding space, these approaches can query images against an entire pool of predefined textual concepts and use cosine similarity scores to find a matching annotation. These approaches are commonly referred to as label-free CBMs . In many real-world settings, a high-performing, task-specific legacy model already exists and often achieves strong results on the target task [ 63 ] . When CLIP is used to generate image-concept annotations, the resulting CBM is anchored to CLIP’s embedding space; the legacy model must be interpreted through CLIP’s notion of similarity, rather than through its own learned representation. In this sense, the approach is not fundamentally different from replacing the legacy model with a CLIP-based CBM. Furthermore, this dependence can transfer CLIP’s biases and behavior into the legacy model (e.g., importing the CLIP typographic bias [ 11 ] into a DINO model). A natural question that then arises is: how to develop a label-free CBM for such legacy specialist models without the CLIP constraint? Retraining such a specialist on a large image–text corpus following the CLIP approach is impractical, in terms of computational cost and need of a huge amount of image-text data. Furthermore, obtaining ground-truth image-concept annotations through manual human-labor is time consuming and expensive. Finally, retraining this legacy model further alters its original decision-making process and distribution, which is typically not desired. In this work, we lift these limitations by first proposing a method dubbed as TextUnlock, which aligns a distribution of a frozen visual classifier to its corresponding vision-language counterpart, without relying on CLIP. TextUnlock has four important properties: First, it is efficient ; it is inexpensive to train and can be performed on any standard hardware, regardless of the size of the original classifier. The number of data points is also significantly reduced compared to CLIP-based approaches. Secondly, it is label-free , no labels are required to achieve this formulation. Thirdly, TextUnlock is trained to preserve the original distribution and reasoning process of the classifier, and does not compromise the classifier’s original performance (average of 0.2 points drop in accuracy). Finally, our method is applicable to any vision architecture, whether convolutional-based, transformer-based or hybrid. After the original classifier’s distribution is aligned to its vision-language counterpart distribution with TextUnlock, we simply query the transformed classifier’s image features against a set of predefined text concepts to obtain concept activations for our CBM, and then derive the concept-to-class classifier directly from the transformed classifier’s text source, without requiring any additional training and operating in an unsupervised manner (i.e., we do not train a linear probe to associate the concept activations to class labels). Figure 1 : Overview of our proposed TextUnlock. (a) The process of training the MLP mapping between vision and text space. (b) The process of inference with the adapted visual classifier. The text encoder acts as weight generator for a linear classifier. indicates that the module is frozen, while indicates trainable. In summary, our contributions are as follows: (i) We propose a method to convert any frozen visual classifier into a CBM without relying on the CLIP constraint (CLIP-free), neither on annotated image-concept data (label-free), and for the first time, also deriving the concept-to-class classifier in a fully unsupervised manner. (ii) We demonstrate the effectiveness of our method with 40 different architectures, along with extensive ablation studies and intervention results, and further show how our method can also be used to perform zero-shot image captioning. (iii) We set new state-of-the-art results on CBMs, outperforming existing works including supervised CLIP-based CBMs, despite being trained only on ImageNet-1K. 2 Related Work Concept Bottleneck Models (CBMs) are inherently interpretable models that predict by first mapping inputs to human-understandable concepts and then combining those concepts linearly to make the final decision, originally requiring concept annotations for each image [ 36 ] . Recently, Label-Free CBMs (LF-CBMs) have been introduced, which leverage CLIP to provide image-concept annotations. CLIP is used to generate ground-truth image–concept similarity scores, which serve as supervision for training the bottleneck layer of a downstream classifier. Several further works [ 60 , 5 ] adopted this pipeline. Another work uses vision-language grounding models [ 48 ] to obtain image-concept annotations. Another line of work [ 59 , 39 , ao2024DiscoverthenNameTC , 19 ] builds CBMs directly on top of CLIP, making them specifically tailored to CLIP-only models and thereby eliminating the need for a separate bottleneck training stage. In these methods, image features are queried against a predefined set of concepts in the CLIP embedding space, and the resulting cosine similarities are used directly as concept activations. As a result, all the above methods either (1) are limited to CLIP models, or (2) depend on CLIP or vision–language models to obtain the image-concept annotations. On the other hand, we propose a fully CLIP-free CBM; our method does not rely on any external vision-language model to generate the image-concept annotations, nor on human annotation, and can be applied to any legacy model without the CLIP constraint. Finally, all CBMs works to date require training a linear probe to