Visual Sparse Steering: Improving Zero-shot Image Classification with Sparsity Guided Steering Vectors
Gerasimos Chatzoudis, Zhuowei Li, Gemma E. Moran, Hao Wang, Dimitris N. Metaxas, [4pt] Department of Computer Science, Rutgers University · 2025
Read Visual Sparse Steering: Improving Zero-shot Image Classification with Sparsity Guided Steering Vectors by Gerasimos Chatzoudis, Zhuowei Li, Gemma E. Moran, Hao Wang, Dimitris N. Metaxas, [4pt] Department of Computer Science, Rutgers University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01247.txt.
Introduction
The authors study whether sparse features can steer vision language models in a 2025 paper. Sparse autoencoders are often used to interpret such models after the fact. The team ask whether the same features can actively improve image classification without labels, by nudging a frozen model at test time.
Their method trains a sparse autoencoder on unlabeled image features, then boosts each input active features to build a steering shift. A reliability gate falls back to the base model when reconstruction is poor. Across nine data sets, zero shot accuracy rose by up to several points with tiny extra compute.
Catalog record
| CREATOR | Gerasimos Chatzoudis, Zhuowei Li, Gemma E. Moran, Hao Wang, Dimitris N. Metaxas, [4pt] Department of Computer Science, Rutgers University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual Sparse Steering: Improving Zero-shot Image Classification with Sparsity Guided Steering Vectors Gerasimos Chatzoudis, Zhuowei Li, Gemma E. Moran, Hao Wang, Dimitris N. Metaxas, [4pt] Department of Computer Science, Rutgers University Abstract Steering vision foundation models at inference time without retraining or access to large labeled datasets is a desirable yet challenging objective, particularly in dynamic or resource-constrained settings. In this paper, we introduce Visual Sparse Steering (VS2), a lightweight, test-time method that guides vision models using steering vectors derived from sparse features learned by top- k k Sparse Autoencoders without requiring contrastive data. Specifically, VS2 surpasses zero-shot CLIP by 4.12% on CIFAR-100, 1.08% on CUB-200, and 1.84% on Tiny-ImageNet. We further propose VS2 ++ , a retrieval-augmented variant that selectively amplifies relevant sparse features using pseudo-labeled neighbors at inference time. With oracle positive/negative sets, VS2 ++ achieves absolute top-1 gains over CLIP zero-shot of up to 21.44% on CIFAR-100, 7.08% on CUB-200, and 20.47% on Tiny-ImageNet. Interestingly, VS2 and VS2 ++ raise per-class accuracy by up to 25% and 38%, respectively, showing that sparse steering benefits specific classes by disambiguating visually or taxonomically proximate categories rather than providing a uniform boost. Finally, to better align the sparse features learned through the SAE reconstruction task with those relevant for downstream performance, we propose Prototype-Aligned Sparse Steering (PASS). By incorporating a prototype-alignment loss during SAE training, using labels only during training while remaining fully test-time unsupervised, PASS consistently, though modestly, outperforms VS2, achieving a 6.12% gain over VS2 only on CIFAR-100 with ViT-B/32. 1 Introduction Foundation Models (FMs) have exhibited strong generalization capabilities across a wide range of domains, from classification tasks to open-ended generation clip ; sam ; llama2 ; llava ; gpt4 . Despite their impressive performance, these models largely function as black boxes, restricting their controllability, and reliability in practice. The standard approach to edit a model’s behavior involves first collecting a labeled dataset resembling the target objective, followed by a supervised fine-tuning (SFT) procedure upon the holistic zhang2023instruction ; dong2023abilities or partial ( e.g . , PEFT lester2021power ; lora ; ia3 ) parameter space. Nevertheless, this procedure is cumbersome and can become prohibitively expensive given the model scale. Consequently, Steering Vector (SV) methods turner2023activation ; zou2023representation ; liu2024incontext , as an alternative, have gained popularity in the Large Language Model (LLM) domain as they can effectively modulate the model’s generation at inference-time in a training-free manner. While SVs have shown promise in LLMs, their functionality in the vision domain remains unclear. In this work, we investigate the functionality of steering vectors in vision foundation models (VFMs). In particular, we use zero-shot classification tasks as a testbed to explore how SVs can be leveraged to upweight relevant visual features. Conventional SVs operate by first constructing a directional vector within the latent space to indicate a desired change of the functionality. It then linearly combines the directional vector with the original latent embeddings to craft output embeddings. In LLMs, SVs rely on two symbolic anchors that are inherently interpretable to determine the functional direction of the SV. For example, a happiness SV can be extracted using the latent embedding arithmetic corresponding to Happy - Sad . The formulation of SVs in LLM poses two critical challenges in vision domain. Given an image at inference-time: (1) Unlike symbolic language tokens, there exists tremendous redundancy in visual representations; consequently, it is unclear what features should be steered ; (2) Without inherently interpretable anchors ( e.g . , happy, sad), towards which direction to steer is not obvious. (a) VS2: Visual Sparse Steering w/o Contrastive Data (b) VS2 ++ : Selective Visual Sparse Steering Figure 1 : Overview of VS2 and VS2 ++ . At inference, VS2 finds sparse latent concepts, and steers the original embedding towards the direction of amplifying those sparse features. When additional caching data is available, VS2 selectively enhances certain features while suppressing others in a contrastive manner. To address the aforementioned challenges, we propose Visual Sparse Steering (VS2). Drawing inspiration from the interpretability provided by sparse auto-encoders (SAEs) hugofry2024multimodal ; Daujotas2024 ; bhalla2024interpreting ; stevens2025sparse , we hypothesize that the sparse features in a SAE’s latent space can capture key aspects of a visual embedding. VS2 upweights these features equally to construct a steering vector for classification. This equal upweighting of features is imprecise in that we expect different features to be more relevant than others for classification tasks. Despite this, we show that the simple equal upweighting of features still improves zero-shot classification. To construct a more precise steering vector, we propose VS2 ++ , which leverages additional unlabeled data to construct steering vectors for classification. For a given target image, VS2 ++ retrieves the Top-N most similar visual embeddings, and constructs positive and negative groups according to generated pseudo labels 1 1 1 In practice, pseudo labels can be gathered using either unsupervised clustering methods or zero-shot classification results . The VS2 ++ SV is then constructed via subtracting the average SAE features of the negative group from the positive group. Intuitively, VS2 ++ upweights the SAE features most relevant for distinguishing the positive and negative groups. In summary, VS2 and VS2 ++ aim to enhance downstream classification performance while retaining the interpretability of SAEs. To further align the sparse features learned through the SAE reconstruction task with those important for downstream performance, we propose Prototype-Aligned Sparse Steering (PASS). VS2 consistently outperforms all baseline methods across all datasets and ViT backbones, surpassing zero-shot CLIP radford2021learning by 3.45% and 4.12% on CIFAR-100, 0.93% and 1.08% on CUB-200, and 1.50% and 1.84% on Tiny-ImageNet with ViT-B/32 and ViT-B/16 backbones, respectively, without any external data or supervision. When using external unlabeled image data, VS2 ++ achieves absolute top-1 gains over CLIP zero-shot of up to 21.44% on CIFAR-100, 7.08% on CUB-200, and 20.47% on Tiny-ImageNet. Finally, by adding a lightweight prototype-alignment loss during SAE training (labels used only in training, still test-time unsupervised) PASS consistently, though slightly, outperforms VS2, offering a marginal 6.12% gain over VS2 only on CIFAR-100 with ViT-B/32. Our main contributions are: • We investigate the functionality of steering vectors in vision domain, and propose VS2 that leverages sparse feature learning to construct steering vectors for zero-shot classification. • We further propose VS2 ++ for scenarios where additional unlabeled data is available to selectively amplify important concepts. In this case, we use VS2 on pseudo-positive and -negative data to upweight relevant features for classification. • We propose Prototype-Aligned Sparse Steering (PASS), a lightweight method that introduces a prototype-alignment loss during SAE training using class labels only during training while remaining fully test-time unsupervised. • We empirically demonstrate that VS2, VS2 ++ , and PASS can leverage SAEs to substantially improve zero-shot classification performance, outperforming baseline methods by a large margin under their corresponding regimes, while also retaining interpretability. 2 Related Work Mechanistic Interpretability and Sparse Autoencoders. Several traditional approaches exist for interpretability in vision models, including feature visualization simonyan2014visualising ; zeiler2014visualizing ; olah2017feature and network dissection bau2017network ; oikarinen2022clip . Mechanistic interpretability seeks to systematically analyze and understand neural networks elhage2021mathematical ; olah2020zoom , but it faces challenges due to polysemantic neurons i.e. units that activate in response to multiple, seemingly unrelated inputs elhage2022toy