Pruning the Paradox: How CLIP’s Most Informative Heads Enhance Performance While Amplifying Bias
Avinash Madasu, Vadudev Lal, Phillip Howard · 2025
Read Pruning the Paradox: How CLIP’s Most Informative Heads Enhance Performance While Amplifying Bias by Avinash Madasu, Vadudev Lal, Phillip Howard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11103.txt.
Introduction
Avinash Madasu, Vasudev Lal, and Phillip Howard probed the attention heads inside CLIP, the famous image-text model, in a paper published in 2025. Each head attends to parts of a scene, but which heads hold which ideas was unclear. Their score checks how steadily a head's attention fits one concept across many tests. Soft pruning then removes heads to see what breaks.
High-score heads proved key: cutting them hurt far more than cutting heads chosen at random. They also drive new scene detection, concept tasks, and video readings. But the same heads learn false links that amplify social bias, the team proves. The score thus shows a clash at the core of the model: its best heads are its most biased. The authors present it as a lens for safer builds.
Catalog record
| CREATOR | Avinash Madasu, Vadudev Lal, Phillip Howard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11103
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pruning the Paradox: How CLIP’s Most Informative Heads Enhance Performance While Amplifying Bias Avinash Madasu, Vadudev Lal, Phillip Howard Abstract CLIP is one of the most popular foundational models and is heavily used for many vision-language tasks. However, little is known about the inner workings of CLIP. While recent work has proposed decomposition-based interpretability methods for identifying textual descriptions of attention heads in CLIP, the implications of conceptual consistency in these text labels on interpretability and model performance has not been explored. To bridge this gap, we study the conceptual consistency of text descriptions for attention heads in CLIP-like models. We conduct extensive experiments on six different models from OpenAI and OpenCLIP which vary by size, type of pre-training data and patch size. We propose Concept Consistency Score (CCS), a novel interpretability metric that measures how consistently individual attention heads in CLIP models align with specific concepts. To assign concept labels to heads, we use in-context learning with ChatGPT, guided by a few manually-curated examples, and validate these labels using an LLM-as-a-judge approach. Our soft-pruning experiments reveal that high CCS heads are critical for preserving model performance, as pruning them leads to a significantly larger performance drop than pruning random or low CCS heads. Notably, we find that high CCS heads capture essential concepts and play a key role in out-of-domain detection, concept-specific reasoning, and video-language understanding. Moreover, we prove that high CCS heads learn spurious correlations amplifying social biases. These results position CCS as a powerful interpretability metric exposing the paradox of performance and social biases in CLIP models. 1 Introduction Large-scale vision-language (VL) models such as CLIP ( Radford et al. 2021 ) have significantly advanced state-of-the-art performance in vision tasks in recent years. Consequently, CLIP has been extensively used as a foundational model for downstream tasks such as video retrieval, image generation, and segmentation ( Luo et al. 2022 ; Liu et al. 2024 ; Brooks et al. 2023 ; Esser et al. 2024 ; Kirillov et al. 2023 ) . This has enabled the construction of compositional models combining CLIP with other foundation models, thereby increasing the functionality of CLIP while also adding complexity to the overall model structure. However, as these models gain prominence in real-world applications, their embedded social biases ( Howard et al. 2024 ; Hall et al. 2023 ; Seth et al. 2023 ) have emerged as a critical concern with potentially harmful downstream consequences. Despite the growing body of work documenting these biases, a fundamental question remains unanswered: what mechanisms within these models’ architectures drive both their impressive capabilities and problematic shortcomings? Recent interpretability advances ( Gandelsman et al. ) have made initial progress by decomposing CLIP’s image representations into contributions from individual attention heads, identifying text sequences that characterize different heads’ semantic roles. However, this approach provides only a partial view into CLIP’s inner workings, leaving a critical missing piece: systematic understanding of the visual concepts encoded at the attention head level—and how these concepts underpin both the model’s strengths and its social failures. Our work addresses this critical gap through a novel interpretability framework we call “conceptual consistency”. This framework systematically analyzes which visual concepts are learned by individual attention heads and how consistently these concepts are processed throughout the model’s architecture. First, we identify interpretable structures within the individual heads of the last four layers of the model using a set of text descriptions. To accomplish this, we employ the TextSpan algorithm ( Gandelsman et al. ) , which helps us find the most appropriate text descriptions for each head. After identifying these text descriptions, we assign labels to each head representing the common property shared by the descriptions. This labeling process is carried out using in-context learning with ChatGPT. We begin by manually labeling five pairs of text descriptions and their corresponding concept labels, which serve as examples. These examples are then used to prompt ChatGPT to assign labels for the remaining heads. Leveraging the resulting text descriptions and concept labels of attention heads, we introduce the Concept Consistency Score (CCS), a new interpretability metric that quantifies how strongly individual attention heads in CLIP models align with specific concepts. Using GPT-4o, Gemini and Claude as automatic judges, we compute CCS for each head and classify them into high, moderate, and low categories based on defined thresholds. A key contribution of our work is our targeted soft-pruning experiments which show that heads with high CCS are essential for maintaining model performance; pruning these heads causes a significantly larger performance drop compared to pruning any other heads. We also show that high CCS heads are not only crucial for general vision-language tasks but are especially important for out-of-domain detection and targetted concept-specific reasoning. Additionally, our experiments in video retrieval highlight that high CCS heads are equally vital for temporal and cross-modal understanding. Moreover, we demonstrate that high CCS heads often encode spurious correlations, contributing to social biases in CLIP models. Selective pruning of these heads can reduce such biases without the need for fine-tuning. Together, these results expose a fundamental paradox: while high-CCS heads are indispensable for strong model performance, they are simultaneously key contributors to undesirable biases. Figure 1 : Figure shows the steps of computing Concept Consistency Score for each head. High CCS (CCS=5) Moderate CCS (CCS=3) Low CCS (CCS\leq 1) L23.H11 (“People”) L23.H0 (“Material”) L21.H6 (“Professions”) Playful siblings Intrica wood carvingte Photo taken in the Italian pizzerias A photo of a young person Nighttime illumination thrilling motorsport race Image with three people Image with woven fabric design Urban street fashion A photo of a woman Image with shattered glass reflections An image of a Animal Trainer A photo of a man A photo of food A leg L22.H10 (“Animals”) L11.H0 (“Locations”) L10.H6 (“Body parts”) Image showing prairie grouse Photo taken in Monument Valley A leg Image with a donkey Majestic animal colorful procession Image with a penguin An image of Andorra Contemplative monochrome portrait Image with leopard print patterns An image of Fiji Graceful wings in motion detailed reptile close-up Image showing prairie grouse Inviting reading nook L23.H5 (“Nature”) L11.H11 (“Letters”) L9.H2 (“Textures”) Intertwined tree branches A photo with the letter J Photo of a furry animal Flowing water bodies A photo with the letter K Closeup of textured synthetic fabric A meadow A swirling eddy Eclectic street scenes A smoky plume A photo with the letter C Serene beach sunset Blossoming springtime blooms awe-inspiring sky Minimalist white backdrop Table 1 : Examples of high, moderate and low CCS heads. 2 Related Work Early research on interpretability primarily concentrated on convolutional neural networks (CNNs) due to their intricate and opaque decision-making processes ( Zeiler & Fergus 2014 ; Selvaraju et al. 2017 ; Simonyan et al. 2014 ; Fong & Vedaldi 2017 ; Hendricks et al. 2016 ) . More recently, the interpretability of Vision Transformers (ViT) has garnered significant attention as these models, unlike CNNs, rely on self-attention mechanisms rather than convolutions. Researchers have focused on task-specific analyses in areas such as image classification, captioning, and object detection to understand how ViTs process and interpret visual information ( Dong et al. 2022 ; Elguendouze et al. 2023 ; Mannix & Bondell 2024 ; Xue et