Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation

Kabir Thayani · 2026

Read Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation by Kabir Thayani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06698.txt.

Introduction

Kabir Thayani probes a flaw in teaching small image nets from a large vision teacher: their learned codes collapse to few dimensions. A contrastive add-on widens the code, yet downstream scores fall sharply, since widening dilutes class cues and adds only weak structure elsewhere. A label-aware variant behaves better, and sweeping its heat setting lifts rank and scores together. Rank alone, he argues, is a poor gauge of code quality.

Catalog record

CREATORKabir Thayani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10048
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation Kabir Thayani Abstract Knowledge distillation between asymmetric architectures often induces severe geometric constraints on the learned representation space. In this work, we investigate the dimensional collapse phenomenon when distilling global Vision Transformers into strictly capacity-constrained, local-receptive-field CNNs (0.5M to 8.0M parameters). By employing strictly centered Singular Value Decomposition (SVD) and Variance-based Shannon Entropy Effective Rank, we first demonstrate a capacity-agnostic phase transition on CIFAR-10: while a CLIP ViT-B/32 Teacher exhibits an Effective Rank of 88.68, all Student models trained with standard cosine distillation experience severe collapse to an intrinsic Effective Rank of ∼ \sim 16. We prove via architectural ablation that this collapse is objective-induced. To reverse this, we integrate an auxiliary contrastive objective (InfoNCE), which actively repels representations to maximize uniformity, expanding the student’s intrinsic manifold by 2.4 × \times (to ∼ \sim 38 effective dimensions). Crucially, by generalizing our experiments to a purely self-supervised teacher (DINOv2) on a high-density dataset (CIFAR-100), we reveal that cosine collapse is an artifact of inherited anisotropy. DINOv2’s uniform geometry prevents collapse even under baseline distillation (achieving ∼ \sim 54 dimensions). However, contrastive expansion remains a universal requirement to reach the CNN’s topological capacity limit ( ∼ \sim 82 dimensions). Finally, by probing these expanded manifolds under high-frequency Gaussian noise, we reveal a critical capacity-density trade-off: extreme overparameterization within a fixed manifold induces catastrophic brittleness, while capacity-constrained models act as optimal low-pass semantic filters, successfully recovering inherent noise immunity. I Introduction The deployment of state-of-the-art foundational vision models is heavily bottlenecked by their massive parameter counts. Knowledge Distillation [ 2 ] is the standard paradigm for compressing these models for edge deployment. However, transferring knowledge from a global-receptive-field Vision Transformer (ViT) into a strictly local-receptive-field Convolutional Neural Network (CNN) [ 1 ] creates a severe asymmetric bottleneck. Previous studies have shown that embedding spaces in deep neural networks often suffer from intrinsic anisotropy and dimensional collapse [ 3 ] . However, standard spectral measurements frequently fail to center the data, inadvertently measuring the distance from the origin to the embedding cluster rather than the true structural variance. In this study, we enforce rigorous mathematical constraints to observe the true spectral geometry of asymmetric cross-modal transfer. Our contributions are four-fold: 1. We empirically demonstrate true dimensional collapse under standard cosine distillation (from CLIP to CNN), proving that students scale from 0.5M to 8.0M parameters yet collapse to an Effective Rank of \sim 16. 2. We isolate the cause of this collapse via architectural ablation, demonstrating that the cosine objective itself acts as a rigid geometric restrictor. 3. We successfully reverse dimensional collapse by integrating an auxiliary InfoNCE contrastive objective, expanding the Effective Rank to \sim 38 on CIFAR-10. 4. We demonstrate that this geometric behavior is universal by scaling to DINOv2 and CIFAR-100. We prove that cosine distillation inherits the Teacher’s anisotropy, and reveal that navigating high-dimensional manifolds requires strict parameter moderation to preserve high-frequency noise robustness. II Methodology II-A Architectures and Distillation We designed a custom Scalable CNN Student architecture with a width expansion factor, yielding three variants: Student-S (0.5M), Student-M (2.0M), and Student-L (8.0M). We utilized two fundamentally different Teacher networks to test geometric inheritance: CLIP ViT-B/32 (trained via multi-modal contrastive language-image pretraining) and DINOv2 ViT-Base (trained via pure vision self-supervision). The baseline models were trained using a strict cosine distance distillation objective: \mathcal{L}_{cos}=\frac{1}{N}\sum_{i=1}^{N}\left(1-\frac{z_{s}^{(i)}\cdot z_{t}^{(i)}}{\|z_{s}^{(i)}\|\|z_{t}^{(i)}\|}\right) (1) To counter dimensional collapse, we introduce an auxiliary InfoNCE loss across augmented student views to encourage manifold expansion: \mathcal{L}_{NCE}=-\log\frac{\exp(\text{sim}(z_{s1},z_{s2})/\tau)}{\sum_{k}\exp(\text{sim}(z_{s1},z_{sk})/\tau)} (2) where \tau is a temperature hyperparameter scaling the distribution of similarities. II-B Rigorous Spectral Evaluation To ensure geometric consistency, all embedding matrices were strictly centered prior to extracting the Singular Value Decomposition (SVD): Z_{c}=Z-\mu_{Z} . We calculated the Shannon Entropy Effective Rank using the normalized squared singular values ( \sigma^{2} ): p_{i}=\frac{\sigma_{i}^{2}}{\sum\sigma_{j}^{2}},\quad ER=\exp\left(-\sum p_{i}\ln p_{i}\right) (3) III Results and Analysis (Phase 1: CLIP) III-A Capacity-Agnostic Dimensional Collapse A primary objective of this study was to determine if capacity constraints induce dimensional collapse. We first distilled CLIP into our CNN students on CIFAR-10. The empirical data revealed a severe, capacity-agnostic phase transition. While the Teacher model exhibited an Effective Rank of 88.68, all models collapsed to an Effective Rank of \sim 16 (Table I ), proving that standard distillation enforces a rigid information bottleneck. TABLE I: Baseline Scaling Metrics (CLIP / CIFAR-10) Model Params Effective Rank Teacher-CLIP 500M 88.68 Student-S \sim 0.5M 15.91 Student-M \sim 2.0M 16.62 Student-L \sim 8.0M 16.66 III-B Objective vs. Architecture Ablation To determine if this \sim 16-dimensional ceiling was an unavoidable consequence of local CNN receptive fields, we performed an architectural ablation. We trained a strictly local CNN, a semi-global Hybrid, and a global Tiny ViT student. The architectural changes failed to expand the subspace, with the CNN achieving 15.90, the Hybrid 14.54, and the Tiny ViT collapsing to 8.70. This confirms that dimensional collapse is a geometric bottleneck explicitly enforced by the objective function. III-C Breaking the Bottleneck via Contrastive Expansion Recognizing that cosine distillation transfers alignment but fails to transfer spatial geometry, we added the auxiliary InfoNCE loss. TABLE II: Contrastive Expansion (InfoNCE / CIFAR-10) Model Params Effective Rank Student-S \sim 0.5M 32.80 Student-M \sim 2.0M 38.41 Student-L \sim 8.0M 38.30 The addition of contrastive learning successfully shattered the geometric bottleneck (Table II ), expanding the Effective Rank from \sim 16 to \sim 38. To visually verify this geometric recovery, we analyzed the Singular Value Spectrum (Figure 1 ). The InfoNCE student tracks the heavy-tailed geometric shape of the Teacher’s spectrum significantly further before reaching its architectural capacity limit. Fig. 1: Singular Value Spectrum (Log Scale). The InfoNCE student (green) successfully tracks the Teacher’s geometric shape (black). IV Generalization and Scaling (Phase 2: DINOv2) IV-A Inherited Anisotropy vs. Universal Expansion To prove these geometric properties are universal to distillation, we replaced the multi-modal CLIP teacher with a purely vision-based, self-supervised DINOv2 Teacher, and evaluated on the high-density CIFAR-100 dataset. Fig. 2: Effective Rank Expansion (DINOv2 Teacher on CIFAR-100). As shown in Figure 2 , the pure cosine baseline achieved 53.78 dimensions. This reveals that dimensional collapse is not inherently caused by the cosine distance, but is an artifact of inherited anisotropy . Because DINOv2 enforces spatial uniformity natively, the cosine student inherits a wider space than it did from CLIP’s anisotropic ”cone.” However, applying InfoNCE is still universally required to force the network to its physical topological limit, expanding the manifold massively from 53.78 up to \sim 82 dimensions. IV-B Capacity Saturation and the Robustness Sweet Spot Having expanded the student manifold to \sim 82 dimensions on CIFAR-100, we evaluated the models under high-frequency Gaussian noise to test the “Semantic Filter” hypothesis at scale. Fig. 3: High-Frequency Noise Robustness (CIFAR-100). As illustrated in Figure 3 , the expanded manifold revealed a critical parameter-density trade-off. While the 2.0M and 8.0M students achieved identical Effective Ranks ( \sim 82), their robustness varied drastically. The 8.0M Student-L suffered catastrophic failure under extreme \sigma=0.2 noise. Due to the extreme semantic density of 100 classes, the larger models utilize their parameters to perfectly memorize the clean boundaries, rendering them highly brittle. Conversely, the heavily constrained 0.5M InfoNCE model acts as an extreme low-pass filter. While

Read the complete text