Matched-Learning-Rate Analysis of Attention Drift and Transfer Retention in Fine-Tuned CLIP
Ruize Xia · 2026
Read Matched-Learning-Rate Analysis of Attention Drift and Transfer Retention in Fine-Tuned CLIP by Ruize Xia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16410.txt.
Introduction
Xia untangles a confounded comparison in adapting CLIP, where full fine-tuning and LoRA are usually run at different learning rates. Using eighty matched runs across two image datasets, four shared learning rates and five seeds, he tracks how attention patterns drift and how much zero-shot transfer survives. Learning rate strongly shaped structural change, and LoRA preserved far more transfer than full fine-tuning at the same rates. Attention drift, Xia concludes, best serves as a descriptive gauge of preservation, not a causal explanation of transfer.
Catalog record
| CREATOR | Ruize Xia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32915 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16410
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Matched-Learning-Rate Analysis of Attention Drift and Transfer Retention in Fine-Tuned CLIP Ruize Xia Abstract CLIP adaptation can improve in-domain accuracy while degrading out-of-domain transfer, but comparisons between Full Fine-Tuning (Full FT) and LoRA are often confounded by different learning-rate conventions. We study how adaptation method and optimization scale jointly shape attention drift and transfer retention in CLIP using a controlled matched-learning-rate comparison of Full FT and LoRA. The completed matrix contains 80 runs on CLIP ViT-B/32 across EuroSAT and Oxford-IIIT Pets, spanning four shared learning rates ( 10 − 6 10^{-6} , 5 × 10 − 6 5{\times}10^{-6} , 10 − 5 10^{-5} , 5 × 10 − 5 5{\times}10^{-5} ) and five seeds, and evaluates attention-drift metrics, best validation accuracy, and adapter-aware CIFAR-100 zero-shot accuracy. Learning rate strongly modulates structural change: on EuroSAT, Full FT moves from mild entropy broadening at 10 − 6 10^{-6} to marked contraction at 5 × 10 − 5 5{\times}10^{-5} , whereas LoRA remains entropy-positive across the full matched grid. At matched learning rates, LoRA preserves substantially more zero-shot transfer than Full FT, averaging 45.13 % 45.13\% versus 11.28 % 11.28\% CIFAR-100 accuracy on EuroSAT and 58.01 % 58.01\% versus 8.54 % 8.54\% on Pets. Oxford-IIIT Pets also reveals a regime effect: low-learning-rate LoRA underfits in-domain, so method-only averages can obscure when LoRA becomes competitive. Supporting rollout, patch-to-patch, and CKA analyses are directionally consistent with the controlled matrix. Overall, matched-learning-rate evaluation materially changes the interpretation of Full FT versus LoRA, and attention drift is most useful as a descriptive diagnostic of representation preservation rather than a causal explanation of transfer behavior. Preprint prepared for arXiv. Companion code and analysis assets are available at https://github.com/xiaruize0911/Attention_Collapse_in_CLIP_Fine-tuning_repo . 1 Introduction Contrastive vision–language pretraining makes CLIP a strong starting point for downstream adaptation, but adaptation can trade in-domain gains against the transfer behavior that makes CLIP attractive in the first place [ 13 ] . For practitioners, the question is therefore not only whether fine-tuning improves task accuracy, but also how much of the pretrained model’s out-of-domain capability survives the adaptation process. Prior work shows that full fine-tuning can distort pretrained representations and degrade robustness under shift, while parameter-efficient methods such as LoRA often preserve more of the original model [ 9 , 15 , 5 ] . Attention maps provide one descriptive lens on this trade-off in Vision Transformers, where spatial information routing is mediated by self-attention across tokens [ 2 ] . We do not treat attention as a causal explanation of model predictions [ 6 , 14 ] . Instead, we use attention drift —changes in CLS-to-patch attention support relative to the pretrained encoder—as a compact measurement of structural change during adaptation. A recurring difficulty in comparisons between Full FT and LoRA is that the two methods are often trained under different learning-rate conventions. When optimization scales differ, apparent method effects mix with update-magnitude effects. The central question of this paper is therefore straightforward: under matched optimization scales, how do Full FT and LoRA differ in attention drift, in-domain performance, and out-of-domain transfer retention? We answer this question with a completed 80-run matched-learning-rate study of CLIP ViT-B/32 across EuroSAT and Oxford-IIIT Pets. Full FT and LoRA share the same four-learning-rate grid and five random seeds, and we evaluate attention-drift metrics, best validation accuracy, and adapter-aware CIFAR-100 zero-shot accuracy. Rollout, patch-to-patch, CKA, regularization, backbone, and rank-ablation analyses are included as supporting evidence that broadens the empirical picture without displacing the main controlled matrix. The contributions of this paper are fourfold: • We present a completed 80-run matched-learning-rate comparison of Full FT and LoRA on CLIP ViT-B/32 across EuroSAT and Oxford-IIIT Pets. • We show that learning rate strongly modulates attention drift, and that method comparisons change materially when optimization scale is controlled. • We find that LoRA generally preserves substantially more zero-shot transfer than Full FT at matched learning rates. • We show that broader retained attention support is positively associated with transfer retention, while treating attention as a descriptive diagnostic rather than a stand-alone causal explanation. 2 Related Work 2.1 CLIP Adaptation CLIP established large-scale image–text pretraining as a strong basis for transfer learning across diverse downstream tasks [ 13 ] . A substantial subsequent literature then examined what happens when zero-shot models are adapted rather than used directly. In particular, 9 showed that full fine-tuning can distort pretrained features and underperform under distribution shift, while 15 argued that robust adaptation of zero-shot models requires more care than simply optimizing for in-domain accuracy. These studies motivate the central question of the present paper: if downstream adaptation changes transfer behavior, what measurable internal structural changes accompany that shift? Our work differs from this prior line in emphasis. The studies above focus primarily on predictive robustness and representation quality at the output level. We instead examine a structural correlate inside the visual encoder, namely the drift of attention heatmaps relative to the pretrained baseline. The paper therefore complements, rather than replaces, robustness-oriented analyses of CLIP adaptation. 2.2 Attention in Vision Transformers The visual backbone of CLIP follows the Vision Transformer formulation of 2 , in which token interactions are mediated by multi-head self-attention. That architecture makes it natural to study how adaptation changes information routing across depth. Prior work on attention flow, especially attention rollout [ 1 ] , provides one concrete mechanism for summarizing layerwise attention patterns beyond any single matrix. This broader literature supports the idea that attention can be analyzed as a structural property of the model even when it is not interpreted as a complete explanation of prediction behavior. At the same time, the interpretability status of attention remains contested. 6 argued that attention weights should not automatically be equated with explanations, whereas 14 showed that attention can still support informative analysis when its role is framed more carefully. We adopt that more conservative position. In this paper, attention heatmaps are treated as descriptive measurements of how downstream optimization reshapes the visual encoder, not as stand-alone causal explanations of model decisions. 2.3 Parameter-Efficient Adaptation LoRA introduced a low-rank parameterization of downstream updates that greatly reduces the number of trainable parameters while preserving much of the base model [ 5 ] . Although LoRA was originally developed and evaluated in language modeling settings, the broader idea of parameter-efficient adaptation has clear relevance for vision–language models: if the update space is constrained, one may expect the adapted model to remain closer to the pretrained representation geometry. This intuition is especially relevant for CLIP, where preserving pretrained transfer behavior is often as important as maximizing in-domain accuracy. The present paper builds on that motivation but introduces a stronger experimental control than most method-level comparisons. Rather than assuming that any observed Full-FT-versus-LoRA gap is attributable to the adaptation method alone, we explicitly hold the learning-rate grid fixed. This control is important because optimization scale can itself induce substantial representational and structural drift. 2.4 Study Positioning Taken together, the prior literature leaves an important gap. CLIP adaptation work shows that robustness can degrade after fine-tuning, transformer-analysis work provides tools for studying internal attention structure, and LoRA offers a mechanism for constraining adaptation. What is less well established is how these pieces fit together under a controlled comparison that