𝑨 𝟐 \boldsymbol{A^{2}} : Smaller Self-Supervised ViTs Localize Better than Larger Ones

Sreehari Rammohan a, Huy Ha b, Carl Vondrick a Β· 2026

Read 𝑨 𝟐 \boldsymbol{A^{2}} : Smaller Self-Supervised ViTs Localize Better than Larger Ones by Sreehari Rammohan a, Huy Ha b, Carl Vondrick a (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03148.txt.

Introduction

Rammohan, Ha, and Vondrick report a surprise about vision transformers trained without labels. Good image classification needs the main object found and background clutter ignored. The team finds that small self-supervised ViTs locate the foreground better than large ones do. Large models still win on rich features per patch, so both traits are wanted at once.

Their method, called A2, splits the two jobs. A small model says where to look. A large model says what is there. The system crops around the small model's attention peaks, then embeds the crops with the large model. It uses only pretrained features, needs no group labels, and trains nothing per dataset. Across five benchmarks, A2 matched loss-level methods that use the same backbone. It beat end-to-end attention training when the test data shifted more. Decoupling the two roles gets both gains at once.

Catalog record

CREATORSreehari Rammohan a, Huy Ha b, Carl Vondrick a
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03148

Browse the full library Β· About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

𝑨 𝟐 \boldsymbol{A^{2}} : Smaller Self-Supervised ViTs Localize Better than Larger Ones Sreehari Rammohan a, Huy Ha b, Carl Vondrick a Abstract Robust visual classification often depends on localizing the main foreground objects in an image while ignoring contextual distractors. Surprisingly, we find that the attention maps of smaller self-supervised ViTs localize foreground objects better than those of larger ViTs. However, we still need large ViTs, because they extract richer representations from each patch. To get the best of both worlds, good localization and rich representations, we propose A 2 A^{2} , a simple method that leverages this inverse scaling finding by decoupling where to look (a small attention model) from what to extract (a large embedding model): we crop around the attention peaks of a small model and embed the crops with a larger model. A 2 A^{2} uses entirely pretrained features, requires no group labels, and does not require per-dataset attention or backbone training. Across 5 5 benchmarks, A 2 A^{2} is competitive with backbone-matched loss-level methods like DFR, and outperforms end-to-end attention training under stronger distribution shifts. Figure 1 : Robust Recognition through Exclusion. Images contain spurious correlations that often mislead classifiers, e.g. a man holding a rein is usually associated with horses. A^{2} removes spatially spurious correlations by automatically selecting crops based on a pre-trained model’s attention map. While the original model is misled (left), our A^{2} trained classifier explicitly attends to only the animal body and correctly predicts β€œcow” (right). 1 Introduction Large vision models can approximate a wide range of functions given sufficient training data. However, this flexibility also means the model will exploit all biases and shortcuts [ 12 ] in the dataset to minimize the training loss. For example, Figure 1 (left) shows a linear classifier trained on ViT [ 11 ] embeddings that falsely predicts that the animal is a horse, when it is actually a cow. Spurious correlations, such as the co-occurrence of horses and people, are pervasive in visual data. Despite progress in curating benchmarks and developing debiasing algorithms, recognition in the presence of such distractors remains a longstanding challenge for computer vision. Existing methods have tried to correct for these biases by altering the loss function or augmenting the training data. These methods use group labels [ 27 , 16 ] , multiple training passes [ 19 , 36 ] , end-to-end training of attention [ 2 ] , an LLM to identify spurious features [ 1 ] , or prompts specifying what to segment [ 26 , 3 ] . Most of these methods try to correct for the spurious correlation after it has already entered the model, or require external prompts or supervision specifying the bias. A natural question is whether a pretrained attention model can be used to filter out these spurious cues on their own. In many cases, these tasks can simply be solved by isolating the primary foreground object, something that pretrained vision transformers are already good at doing. We find a surprising result: the attention maps in smaller self-supervised ViTs are better at localizing the foreground object than the attention maps in larger ViTs. This inverse scaling trend runs contrary to the common belief that larger parameter-count models are always better [ 35 , 31 , 9 ] . We design A^{2} to leverage this finding by decoupling where to look (the attention model) from what to extract (the embedding model): one model attends to another model’s attention, hence Attending on Attention . While pretrained attention might not always be enough to localize the features of interest, for many real-world applications (e.g. security camera footage, species identification, medical imaging) the foreground objects are the most task-relevant. This property is also baked into common distribution shift benchmarks (Waterbirds, Spawrious, MetaShift). Contributions . We make two contributions: 1) we find that smaller pretrained self-supervised ViTs localize foreground objects substantially better than larger ones, and that localization quality tracks downstream task performance, and 2) we leverage this finding to propose A^{2} , a simple method that decouples where to look (the attention model) from what to extract (the embedding model) using only frozen pretrained features, outperforming purpose-built methods for learning under distribution shift. 2 Attention in Smaller Models Localizes Better Figure 2 : Do Smaller ViTs Localize Better? We show the proportion of attention mass that is inside ground-truth bounding boxes versus model size across six pretraining families, on Waterbirds (left) and ImageNet Val (right). All five self-supervised families (DINOv1, DINOv2, DINOv3, MAE, iBOT) slope downward as model size grows; the contrastive image-text family (OpenCLIP, dashed) does not. To directly test whether smaller self-supervised ViTs [ 4 ] localize better than larger ones, we use ImageNet [ 10 ] ( 1000 object classes, N{=}50{,}000 ) and Waterbirds ( 2 classes, N{=}5{,}794 ) where annotated bounding boxes are available. We measure the proportion of attention mass captured within the bounding box (taking the union if there are multiple) and also report the hit rate (whether or not the highest attention map value falls within a bounding box). Figure 2 shows that the trend appears across all five self-supervised families we evaluate. The original DINO [ 4 ] shows it at its smallest scale: ViT-S/16 puts 68.9\% of its attention mass inside Waterbirds GT bboxes vs. 64.9\% for ViT-B/16. For DINOv2 [ 24 ] , there is a monotonic decrease in attention mass and hit rate as the ViT gets larger. For ImageNet Val, ViT-S puts 74.6\% of its mass inside the ground truth object box vs. 67.6\% for ViT-G. The same pattern persists at much larger scale with DINOv3 [ 29 ] : on Waterbirds, ViT-S/16 ( 21 M) puts 76.7\% of its mass inside GT bboxes vs. only 57.7\% for ViT-7B/16 ( 6.7 B), a 19 -point drop while scaling 320\times in parameters. Although not monotonic, we observe a similar trend in another self-supervised ViT trained with reconstruction loss, MAE [ 14 ] , with the model best at localization being the smallest, ViT-B/16 (For Waterbirds, ViT-B/16 puts 47.4\% of its mass inside the ground truth object box vs. 37.8\% for ViT-H/14). The same monotonic decrease in attention mass holds for iBOT [ 37 ] , a fourth self-supervised family combining masked image modeling with self-distillation: ViT-S/16 puts 69.6\% of its mass inside Waterbirds GT bboxes vs. 55.7\% for ViT-L/16. In contrastive language-image pretrained ViTs (OpenCLIP) the trend does not hold: ViT-B/16 is both the smallest and the worst at localization with a particularly low hit rate of 0.166 compared with ViT-L/14 ( 0.815 ). Appendix Table 3 reports precise values for all metrics. 3 Attending on Attention Figure 3 : Overview of \boldsymbol{A^{2}} . Given an input image, x , we use an attention model to select crops, \mathrm{Crop}_{i}(x;\;\mathrm{Attn}(x)) , which are resized and embedded using a (possibly different) embedding model, \mathcal{T} . The concatenated embeddings [\,\mathcal{T}(c_{1}),\;\ldots,\;\mathcal{T}(c_{k})\,] are passed to a classifier f to predict \hat{y} . Motivated by this finding, we propose Attending on Attention ( A^{2} ), a method where a larger model attends to the attention of a smaller model. A^{2} has two variants: A^{2}_{\text{ZS}} , which is fully zero-shot, and A^{2}_{\text{LR}} which fits a logistic regression head using crop embeddings

Read the complete text