Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers.
Jakub Grzywaczewski, Dawid Płudowski, Przemysław Biecek · 2026
Read Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers. by Jakub Grzywaczewski, Dawid Płudowski, Przemysław Biecek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14893.txt.
Introduction
Grzywaczewski and team study the covariance spectrum of contrastively trained vision-language models. Shared latent spaces hold non-semantic multi-modal noise with meaning. Spectral split divides the latent space into a signal part and a shared noise subspace. The noise shape shows strong subgroup invariance across data subsets. Pruning shared noise dimensions is claimed to be mainly harmless, preserving or improving downstream scores, hinting much latent shape reflects design-level noise rather than task meaning.
Catalog record
| CREATOR | Jakub Grzywaczewski, Dawid Płudowski, Przemysław Biecek |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20736 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14893
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers. Jakub Grzywaczewski, Dawid Płudowski, Przemysław Biecek Abstract Contrastively pre-trained Vision-Language Models (VLMs) serve as powerful feature extractors. Yet, their shared latent spaces are prone to structural anomalies and act as repositories for non-semantic, multi-modal noise. To address this phenomenon, we employ spectral decomposition of covariance matrices to decompose the VLM latent space into a multi-modal semantic signal component and a shared noise subspace. We observe that this noise geometry exhibits strong subgroup invariance across distinct data subsets. Crucially, pruning these shared noise dimensions is mainly harmless, preserving or actively improving downstream task performance. By isolating true semantic signals from artifactual noise, this work provides new mechanistic insights into the representational structure of modern VLMs, suggesting that a substantial fraction of their latent geometry is governed by shared, architecture-level noise rather than task-relevant semantics alone. † † affiliationnotice: Equal contribution 1 Introduction Contrastively pre-trained Vision-Language Models (VLMs) ( 17 ; 3 ; 23 ; 31 ) are deployed globally as feature extractors across a wide variety of tasks from zero-shot classification ( 23 ) and cross-modal retrieval ( 10 ) to text-based segmentation ( 29 ) and visual question answering ( 26 ) . However, the widespread reliance by both researchers and industry on the rich context provided by these shared latent spaces exposes an intriguing point of failure: these representations often begin to encode non-semantic, uninformative noise originating from both modalities. Figure 1 : The eigenspectra of sample covariance matrices, computed from CLIP ViT-L/14 embeddings for two unrelated ImageNet classes, are highly similar (shown here as the \log_{10} of the eigenvalues). Despite no apparent correlation nor common samples, their identified class-specific noise directions exhibit strong overlap, achieving mSCSA of 91\% (Equation 3 ). To address this vulnerability, extensive research has sought to map the geometry of these latent spaces ( 15 ; 1 ; 11 ) , uncovering structural phenomena such as the linear representation hypothesis ( 22 ) , the hierarchical nature of concepts ( 21 ) , and the modality gap ( 12 ; 4 ; 7 ; 30 ; 16 ) . The urgency of this geometrical analysis is underscored by a broader push for the mechanistic decomposition of neural networks ( 25 ) , in this case isolating true semantic signals from artifactual noise. A critical prior work context for this challenge is dimensional collapse ( 9 ) , a phenomenon initially observed in single-modal contrastively-learned models like ResNet-50 ( 8 ) . While contrastive learning prevents the entire latent space from collapsing, it still allows individual dimensions to collapse. In this short paper, we demonstrate that dimensional collapse persists in modern VLMs such as CLIP and SigLIP, albeit in an altered form. Rather than collapsing entirely, these dimensions become repositories for the model to encode irrelevant data. Following 9 , we analyze these dimensions through the spectral decomposition of covariance matrices computed over large-scale datasets ( 5 ; 24 ) of different modalities. The primary contribution lies in formally decomposing the VLM latent space into the shared noise subspace and multi-modal signal . Specifically, we observe a striking alignment of subspaces within the noise component, which constitutes the lower bulk of the covariance eigenvalues ( 14 ) . As another contribution, we demonstrate that pruning these shared noise dimensions is mainly harmless; it does not degrade downstream performance and, in some instances, improves it. Finally, we show that this geometry exhibits strong subgroup invariance. We observe that the eigenspectra of covariance matrices computed over distinct ImageNet ( 5 ) classes remain highly similar, as highlighted in Figure 1 , and that the eigenvectors corresponding to the smallest eigenvalues consistently align with our identified noise dimensions, regardless of the chosen data subset. 2 Related Work The latent space of transformer-based ( 28 ) VLMs, such as the contrastively trained CLIP ( 23 ) and SigLIP ( 31 ; 27 ) models, encodes concepts linearly ( 22 ) and hierarchically ( 21 ) , yet remains subject to structural anomalies ( 15 ; 1 ; 11 ) . For instance, representations often suffer from the cone effect, where projected data is confined to a narrow cone of the embedding space, as initially observed in BERT ( 6 ) , as well as the modality gap ( 12 ) , wherein image and text embeddings remain strictly separated throughout the entire process of pre-training. Subsequent studies ( 4 ; 7 ; 30 ) have linked this modality gap directly to the foundational contrastive alignment objective ( 17 ; 3 ) and to underlying dimensional collapse ( 9 ) . To refine these geometries, recent literature has explored targeted subspace interventions. For example, 32 isolate the subspace spanned by a subset of class-name text representations to prune uninformative features. Similarly, IsoCLIP ( 13 ) filters CLIP dimensions by isolating the isotropic region of the intra-modal singular values, while 20 utilize the Schur complement of the empirical covariance matrix to enhance the diversity of prompt-guided generation. Furthermore, 2 suggest using CLIP as a surrogate for likelihood via whitening (computing the inverse square root of the covariance matrix). This establishes a Mahalanobis distance metric where samples projecting onto the smaller eigenvalues exhibit a higher norm, thereby corresponding to lower likelihood. Our work mechanistically extends this concept by demonstrating that these lower eigenvalues explicitly encode the shared noise dimensions. Table 1 : Effects of projecting the embeddings of different vision backbones on downstream tasks: ImageNet zero-shot image classification and LAION-2B image–text alignment, where classification performance is measured by Top-5 accuracy and alignment is quantified by the change in average cosine similarity. Dimension details ImageNet Zero-shot (Top5 Accuracy) LAION-2B (CosSim) Backbone Latent size Noise size Original Noise-Free Random Noise-Free dimensions dimensions dimensions dimensions ( \times 10^{2} ) CLIP ViT-B/16 512 21 (4%) 88.5 88.5 87.9_{\pm 0.09} +0.090_{\pm 0.070} CLIP ViT-B/32 512 14 (3%) 85.2 85.2 84.9_{\pm 0.04} +0.030_{\pm 0.040} CLIP ViT-L/14 768 164 (21%) 91.2 91.1 89.3_{\pm 0.07} +0.090_{\pm 0.070} SigLIP ViT-B/16 768 118 (15%) 92.7 92.6 91.9_{\pm 0.05} +1.600_{\pm 0.310} SigLIP2 ViT-B/16 768 185 (24%) 88.0 88.0 85.2_{\pm 0.22} +1.590_{\pm 0.380} SigLIP2 ViT-L/16 1024 405 (40%) 89.0 88.4 83.9_{\pm 0.56} +1.000_{\pm 0.280} Figure 2 : Progression of the \log_{10} of eigenvalues of the average covariance matrix ( Equation 2 ) for all considered VLM backbones. All evaluated models exhibit a sharp dip in eigenvalues immediately after the shared noise threshold. 3 Methodology Following 9 , we begin by examining the empirical covariance matrices of the model embeddings. Let z_{i}\in\mathbb{R}^{d} represent the model’s embedding of an input x_{i} (either an image or text tokens). The empirical covariance matrix \Sigma is computed over a dataset of n samples as: \Sigma=\frac{1}{n-1}\sum^{n}_{i=1}(z_{i}-\bar{z})(z_{i}-\bar{z})^{T}, (1) where \bar{z} denotes the empirical mean of the embeddings. We compute these covariance matrices separately for each modality to obtain \Sigma_{\text{text}} and \Sigma_{\text{img}} . Unless stated otherwise, all covariance matrices are scaled to have a trace of 1.0 to allow for direct comparison. The principal directions of the embedding space are then obtained via spectral decomposition. For instance, the image covariance matrix is decomposed as \Sigma_{\text{img}}=U_{\text{img}}\Lambda_{\text{img}}U_{\text{img}}^{T} , where U_{\text{img}} is an orthonormal matrix whose columns correspond