Taking Shortcuts for Categorical VQA Using Super Neurons

Pierre Musacchio and Jaeyi Jeong and Dahun Kim and Jaesik Park · 2026

Read Taking Shortcuts for Categorical VQA Using Super Neurons by Pierre Musacchio and Jaeyi Jeong and Dahun Kim and Jaesik Park (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10781.txt.

Introduction

Pierre Musacchio and co-authors find that single scalar activations inside a vision language model can act as accurate classifiers, a shortcut they call super neurons. Sparse attention vectors had shown a few good heads can replace fine-tuning for a task; probing raw activations widens that search hugely. Enough super neurons appear in shallow layers to allow extreme early exiting, from the first layer at the first generated token. On diverse visually grounded tasks, the probed classifiers beat the original network while running several times faster.

Catalog record

CREATORPierre Musacchio and Jaeyi Jeong and Dahun Kim and Jaesik Park
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52524
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Taking Shortcuts for Categorical VQA Using Super Neurons Pierre Musacchio and Jaeyi Jeong and Dahun Kim and Jaesik Park Abstract Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of vision-language models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model’s prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10 × \times . 1 Introduction Figure 1 : Overview of our approach. Our training-free scheme uncovers Super Neurons (SNs) via probing data. They robustly outperform the base model on a variety of categorical VQA datasets. As a byproduct, they enable extreme early exiting at the first layer of the LLM on the first generated token. Colored boxes in the LLM represent SNs for their data types. Vision-language model (VLM) are frontier models extending the generative capabilities of large language model (LLM) via visual grounding [ 23 , 25 , 1 , 6 ] . Usually consisting of billions of parameters, these models retain extensive knowledge from internet-scale pretraining [ 4 , 39 , 8 , 30 , 33 ] . Although remarkably effective, their complexity hinders attempts to understand how they operate at their core. Current research on VLM explainability and efficiency improvement mainly focuses on what could be called macro -level representations. That is, multidimensional representations that are learned through aggregating information from interactions of the tokens in the model. The most famous examples lie in linear probing [ 35 , 42 ] or attention map extraction [ 17 , 27 ] . However, thanks to the over-parameterization of current state of the art networks, we hypothesize that models accumulate such a tremendous amount of information over training that their individual activation scalars are sufficient to provide accurate answers to specific questions. We term these micro -level representations. Thus, we repurpose the neuron activations of the model into predictions via a simple training-free strategy inspired by [ 27 ] . Analogously, we gather a probing dataset and perform an end-to-end VLM inference on it. During the process, we store activations from the large language model (LLM) of the VLM. However, instead of clustering attention heads, we directly convert the raw activations into classification predictions by thresholding them. We observe that this simple conversion scheme is enough for a subset of neurons to achieve high scores on conventional categorical visual question answer metrics for a wide diversity of datasets. We subsequently deem them Super Neurons (SNs). Surprisingly, SNs obtain even better performance than the models themselves on a diverse suite of unseen categorical VQA validation benchmarks. Since there are more raw activations than attention heads in the network (cf. Fig. 2(b) ), there are more chances to find SNs that have desirable properties, such as better performance and robustness. Specifically, we discover that some SNs located in shallower layers of the model preserve great performance even while the first token is being generated. This allows us to perform extreme early exit i.e . interrupt inference on the first layer of the LLM during the generation of the first token. Our contributions are summarized as follows: • We shift the analysis from macro -level representations (akin to attention vectors) to micro -level ones (scalar activations). By doing so, we present a training-free approach that identifies high-scoring neurons in the LLM of the VLMs, • We comprehensively benchmark the probed neurons and find that they can serve as strong categorical classifiers, outperforming the base models themselves on a diverse suite of VQA benchmarks. We therefore call them Super Neurons , • We thoroughly investigate SNs (discriminative power, location in the model, quantity, robustness) and introduce the agreement rate metric that quantifies the divergence between SNs predictions and model predictions, • As a byproduct, SNs enable extreme early exit at inference time, providing a speedup of up to 5.10\times while maintaining model-level performance. 2 Related work Efficient VLMs. A conventional approach to turn large VLMs to efficient models is to prune them at the parameter level, either by distillation [ 40 ] or by training a policy to search which weights to remove [ 22 ] . Pruning can also occur at the token level, usually via token similarity approaches [ 41 , 14 , 43 , 5 ] , by estimating visual contribution [ 24 ] , or using scale-down approaches [ 25 ] . If the final objective is to improve performance, training a robust visual encoder is also a viable solution [ 37 , 9 ] . Some approaches diverge by considering early exit from the model, either in a supervised setting [ 2 ] or in a training-free manner by estimating layer-wise similarities [ 36 ] . Recently, task vectors [ 11 ] have been leveraged in VLMs in the form of sparse attention vectors to enable training-free improvement of VLMs in classification tasks [ 27 ] . Single modality convnets and LSTMs can rely on some of their weights for accurate prediction [ 19 , 32 ] , but this remains to be shown for transformers, specifically when processing multimodal tokens. (a) Architecture comparison between SAVs and SNs. (b) Search space comparison. Method Search target Search space size SAVs [ 27 ] Attention vectors Layers \times Heads =32\times 32=1024 SNs (Ours) Activation scalars Layers \times Dim. =32\times 4096=\mathbf{131,072} Figure 2 : Comparison between SAVs and ours. We show architectural divergences in Fig. 2(a) and resulting search spaces of the two approaches for LLaVA-v1.5-7b in Fig. 2(b) . While inspired by [ 27 ] , we elect neural activations rather than clustering attention heads, shifting the representation of interest from a macro- to a micro-level (cf. Fig. 2(a) ). This shift continues to hold properties reported in [ 19 , 32 , 27 ] ( e.g . better performance than the model itself), while being robust to prompt variations and distribution shifts, enabling inference on diverse VQA tasks and extreme early stopping. Although we solely focus on categorical VQA tasks, we propose a training-free approach that identifies expert neural activations and establishes a set of SNs that solves the task accurately and robustly, without relying on token similarity or altering model weights. After discovery, we substantially improve the runtime performance of VLM using extreme early exit, as early as the first layer. Explainable VLMs. Model explainability is a fundamental challenge for the deployment of VLMs in the real world. Since VLMs are

Read the complete text