From Colors to Classes: Emergence of Concepts in Vision Transformers

Teresa Dorszewski [email protected], Lenka Tětková [email protected], Robert Jenssen [email protected], Lars Kai Hansen [email protected], Kristoffer Knutsen Wickstrøm [email protected] 1 Department of Applied Mathematics and Computer Science, Technical University of Denmark2 Department of Physics an · 2025

Read From Colors to Classes: Emergence of Concepts in Vision Transformers by Teresa Dorszewski [email protected], Lenka Tětková [email protected], Robert Jenssen [email protected], Lars Kai Hansen [email protected], Kristoffer Knutsen Wickstrøm [email protected] 1 Department of Applied Mathematics and Computer Science, Technical University of Denmark2 Department of Physics an (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24071.txt.

Introduction

Teresa Dorszewski and colleagues trace how concepts emerge inside vision transformers, layer by layer. Vision transformers are widely used in computer vision, but how they process information across layers is understudied. Convolutional networks are known to extract features of increasing complexity, a fact that matters for transfer learning and domain adaptation.

Using neuron labeling, the team analyzed the concepts encoded in state-of-the-art vision transformers. Early layers mainly encode basic features such as colors and textures. Later layers represent more specific classes, including objects and animals, and each layer holds more concepts as complexity grows. Pretraining strategies also change the quantity and category of encoded concepts, and fine-tuning for downstream tasks reshapes them further. The findings give practitioners a clearer map of what these models represent, and where in the network each kind of concept lives.

Catalog record

CREATORTeresa Dorszewski [email protected], Lenka Tětková [email protected], Robert Jenssen [email protected], Lars Kai Hansen [email protected], Kristoffer Knutsen Wickstrøm [email protected] 1 Department of Applied Mathematics and Computer Science, Technical University of Denmark2 Department of Physics an
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-24071

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Colors to Classes: Emergence of Concepts in Vision Transformers Teresa Dorszewski [email protected], Lenka Tětková [email protected], Robert Jenssen [email protected], Lars Kai Hansen [email protected], Kristoffer Knutsen Wickstrøm [email protected] 1 Department of Applied Mathematics and Computer Science, Technical University of Denmark2 Department of Physics and Technology, UiT The Arctic University of Norway3 Pioneer Centre for AI, University of Copenhagen, Denmark4 Norwegian Computing Center, Oslo, Norway Abstract Vision Transformers (ViTs) are increasingly utilized in various computer vision tasks due to their powerful representation capabilities. However, it remains understudied how ViTs process information layer by layer. Numerous studies have shown that convolutional neural networks (CNNs) extract features of increasing complexity throughout their layers, which is crucial for tasks like domain adaptation and transfer learning. ViTs, lacking the same inductive biases as CNNs, can potentially learn global dependencies from the first layers due to their attention mechanisms. Given the increasing importance of ViTs in computer vision, there is a need to improve the layer-wise understanding of ViTs. In this work, we present a novel, layer-wise analysis of concepts encoded in state-of-the-art ViTs using neuron labeling. Our findings reveal that ViTs encode concepts with increasing complexity throughout the network. Early layers primarily encode basic features such as colors and textures, while later layers represent more specific classes, including objects and animals. As the complexity of encoded concepts increases, the number of concepts represented in each layer also rises, reflecting a more diverse and specific set of features. Additionally, different pretraining strategies influence the quantity and category of encoded concepts, with finetuning to specific downstream tasks generally reducing the number of encoded concepts and shifting the concepts to more relevant categories. K eywords Concepts ⋅ \cdot Vision Transformers ⋅ \cdot Explainability. † † footnotetext: Preprint. Accepted at The 3rd World Conference on eXplainable Artificial Intelligence. 1 Introduction Vision Transformers (ViTs) [ 5 ] are becoming increasingly important in the field of computer vision [ 9 ] , but the understanding of their feature extraction process is still in its infancy. This understanding is crucial, as the myriad of learned features form the basis of the network’s decision process and can reveal biases [ 8 ] and shortcuts [ 13 ] , providing a deeper understanding of the learning process as a whole [ 22 ] . One path towards improving this understanding is to investigate where semantic concepts are learned throughout the network. Several such studies exist for convolutional neural networks (CNNs) (e.g. [ 2 , 3 , 36 ] ), but very few studies exist for ViTs (e.g. [ 30 ] ) or are mainly focused on the output layer and do not analyze what concepts are learned in intermediate layers (e.g. [ 3 , 20 ] ). In CNNs, a hierarchical way of processing information is given by the architecture and the constrained receptive fields, which leads to early layers being focused on colors, patterns, and edges while only later layers are able to process complex information like objects and scenes [ 3 ] . In ViTs, however, this hierarchical information processing is not given by the architectural constraints, as the self-attention layers act globally and can attend to the full image from layer one [ 5 ] . ViTs could therefore in principle learn complex features already in early layers. While it has been shown that ViTs process information differently and develop more uniform representations compared to CNNs [ 26 ] , it has not been investigated where ViTs start processing complex information and if they focus on colors and patterns in earlier layers just as CNNs do. Therefore, there is a need for a deeper analysis to understand the layer-wise learning process of ViTs and we propose using concept-based explainability to gain these insights. In this work, we propose a comprehensive analysis of the layer-wise learning process in ViTs using neuron labeling [ 20 ] . We aim to identify and understand the semantic concepts learned at different layers of the network, providing insights into the hierarchical feature extraction process. We use the CLIP-dissect method introduced by Oikarinen and Weng [ 20 ] , which provides a fast and reliable method for neuron labeling, due to its flexibility, computational speed, and high quality labels (e.g. shown in a recent quantitative study [ 16 ] ). Most of these methods showcase neuron labeling in CNNs, such as ResNets [ 11 ] , and have only limited or no experiments on transformer-based models, which are becoming increasingly popular in the computer vision domain [ 10 , 24 ] . The previous works also lack a thorough quantitative analysis of how the concepts develop across layers, and where what kind of information is processed. In this paper, we aim to close this gap by providing a deep analysis of encoded concepts through neuron labeling in ViTs. We find that, despite not being locally constrained, ViTs tend to learn simpler concepts like colors and patterns early on and more complex concepts like objects and natural elements in late layers. Figure 1 shows how different concept categories develop across layers in ViTs. Our study highlights the hierarchical feature extraction in ViTs and provides the first comprehensive layer-wise concept analysis of ViTs. Our main findings include: • ViTs encode more universal concepts, like colors and textures, in early layers and more specialized concepts, like objects and natural elements, and a greater amount of different concepts in later layers. • Neurons in early layers mostly activate on simpler images while neurons in late layers react on more complex images. • Finetuning to specific downstream tasks reduces the number of encoded concepts and can lead to a concept shift to task-relevant concepts while forgetting irrelevant concepts. Figure 1: We analyze how different concepts develop across the layers of vision transformers. Early layers tend to process simpler concepts and images while late layers focus on more complex and diverse concepts. 2 Related Work Here, we present an overview of investigations from prior works that align the most closely to our work. Layer-wise concept analysis in CNNs Numerous studies have investigated the complexity of concepts learned in different layers of the network. An early work by Yosinski et al. [ 35 ] demonstrated empirically that earlier layers learn to extract more general and low-level features while later layers extract more complex and task-specific features. A later work by Bau et al. [ 3 ] modeled concepts explicitly through the Network Dissection framework and found that earlier layers extract concepts related to color and textures while later layers extract concepts related to objects. A recent study by Fel et al. [ 28 ] used information-theoretic estimators to quantify the complexity of concepts learned throughout the layers of CNNs, which also found that CNNs encode simple concepts in early layers and build more complex concepts in late layers, with simple concepts flowing through residual connection to stay available throughout the network. Layer-wise concept analysis in ViTs To the best of our knowledge, very little work has been conducted in layer-wise concept analysis in ViTs. Recently, Vielhaben et al. [ 30 ] looked into concept-based alignment in ViTs and revealed that early and middle layers encode fewer concepts, that are not well aligned with WordNet categories,

Read the complete text