Convolutional Set Transformer
Federico Chinello, Giacomo Boracchi · 2025
Read Convolutional Set Transformer by Federico Chinello, Giacomo Boracchi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22889.txt.
Introduction
Sets of images can share a theme while each image looks quite distinct. Federico Chinello and Giacomo Boracchi design the Convolutional Set Transformer, or CST, for such sets. Older set networks take plain vectors, so a separate image model must first turn each image into numbers. CST works on the raw image tensors instead, learning traits and cross-image links at the same time.
That joint design beat older tools on set classification and set anomaly detection. CST also works with Grad-CAM, a tool that shows which image parts drove a choice. The pair pre-trained CST on ImageNet and adapted it to new tasks. They release the trained model, called CST-15, for other researchers.
Catalog record
| CREATOR | Federico Chinello, Giacomo Boracchi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22889
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Convolutional Set Transformer Federico Chinello, Giacomo Boracchi Abstract We introduce the Convolutional Set Transformer (CST), a novel neural architecture designed to process image sets of arbitrary cardinality that are visually heterogeneous yet share high-level semantics—such as a common category, scene, or concept. Existing set-input networks, e.g., Deep Sets and Set Transformer, are limited to vector inputs and cannot directly handle 3D image tensors. As a result, they must be cascaded with a feature extractor, typically a CNN, which encodes images into embeddings before the set-input network can model inter-image relationships. In contrast, CST operates directly on 3D image tensors, performing feature extraction and contextual modeling simultaneously , thereby enabling synergies between the two processes. This design yields superior performance in tasks such as Set Classification and Set Anomaly Detection and further provides native compatibility with CNN explainability methods such as Grad-CAM, unlike competing approaches that remain opaque. Finally, we show that CSTs can be pre-trained on large-scale datasets and subsequently adapted to new domains and tasks through standard Transfer Learning schemes. To support further research, we release CST-15, a CST backbone pre-trained on ImageNet ( link ). 1 Introduction Figure 1: Deep Sets (DS) and Set Transformer (ST) pipelines compared to the CST architecture (ours). In DS/ST pipelines, a shared CNN is applied independently to each image in the input set, yielding a set of uncontextualized activation volumes whose spatial dimensions are subsequently collapsed via Global Average Pooling, resulting in a set of uncontextualized latent vectors . These vectors are then fed to the DS/ST module, which performs contextualization and outputs a set of contextualized latent vectors for downstream processing. In contrast, CST (ours) processes the set of images through a stack of SetConv2D blocks, simultaneously performing feature extraction and context modeling, and yielding contextualized activation volumes that retain spatial dimensions (H^{\prime}\times W^{\prime}\times C^{\prime}) and can be directly used in downstream tasks. Images are ubiquitous and often occur in sets. For example, images on a web page or in user-generated social media content are typically related to a common concept and can therefore be processed together to enhance automatic visual recognition. In medicine, jointly analyzing multiple images from the same patient can improve diagnostic accuracy, such as in melanoma detection [ 1 ] . Similarly, [ 2 ] demonstrates how sets of room images can be used for fine-grained hotel classification, a task that supports efforts to combat human trafficking. In disaster response, collections of images depicting the same building from different viewpoints provide valuable input for automated damage assessment models [ 3 ] . Overall, processing images as sets is advantageous because it enables visual recognition tasks to exploit both the complementary information carried by individual images and the semantic relationships that link them. Figure 2: SetConv2D layer . SetConv2D takes a set of volumes as input and processes each of these by a shared convolutional layer. A Global Average Pooling (GAP) is applied to each volume to yield a set of latent vectors that is fed to a Multi-Head Self-Attention (MHSA) unit. The output of MHSA is used as context-aware bias vectors that are summed to the volumes fed to the GAP, via a residual connection. After applying these biases, the volumes are passed through a non-linear activation function. Unlike matrices or tensors, sets have no fixed cardinality and lack an inherent ordering of their elements. Convolutional Neural Networks (CNNs) have achieved remarkable success across a wide range of computer vision tasks, but they cannot natively process sets or exploit the contextual relationships among images within a set. By contrast, set-input neural networks such as Deep Sets [ 4 ] and Set Transformer [ 5 ] are explicitly designed to handle set-structured data. However, these models operate only on sets of vectors and cannot directly process sets of 3D tensors such as images. As a result, each image in a set must first be converted into an embedding vector by a feature extractor—typically a CNN—and only then passed to the set-input network, as illustrated in Figure 1 . In this cascaded pipeline, the feature extractor processes each image independently and identically, without leveraging any contextual information, while the set-input network subsequently applies set-specific operations to capture inter-image relationships. While end-to-end training of the CNN and the Deep Sets or Set Transformer is possible, feature extraction and context modeling remain strictly sequential. As a result, the two processes cannot develop true synergies: convolutional layers enforce locality and translation-invariance priors during feature extraction, but these inductive biases cannot interact directly with the set-specific operations that capture semantic relationships among images, since the latter only occur once feature extraction has been completed. This cascaded design also limits explainability: saliency methods such as Grad-CAM [ 6 ] can only be applied to the convolutional layers, which precede any set-specific operation. Figure 3: Unlike Set Transformer (ST), CST is transparent and explainable . The Set Anomaly Detection task aims to identify anomalous images within a set. The figure shows two image sets derived from the CelebA dataset [ 7 ] . In each set, a majority of normal images share two attributes ( wearing hat and smiling in the first, no beard and attractive in the second), while a minority lack these attributes and are thus anomalous. After training a CST and an ST on CelebA for Set Anomaly Detection, we evaluate the explainability of their predictions by overlaying Grad-CAMs on anomalous images. CST explanations correctly highlight the anomalous regions, whereas ST explanations fail to provide meaningful insights. For further details, see Section 6.2 . To address these limitations, we introduce the Convolutional Set Transformer (CST), a general framework for deep learning architectures specifically designed for sets of images. Its fundamental building block is SetConv2D (see Figure 2 ), a permutation-equivariant set-to-set layer that takes as input a set of 3D tensors—such as images or activation volumes—and returns a corresponding set of activation volumes, each primarily associated with one input image. The key distinction between Conv2D and SetConv2D is that, in the latter, biases are dynamically adjusted at inference time to incorporate contextual information extracted from the entire image set. These dynamic biases are computed by a Multi-Head Self-Attention (MHSA) [ 8 ] unit integrated into each SetConv2D layer. CST networks are obtained by stacking multiple SetConv2D layers and, unlike Deep Sets and Set Transformers, can exploit contextual information from the set during feature extraction (Figure 1 ). In particular, dynamic biases in the early SetConv2D layers model relationships between images by analyzing low-level patterns, while in deeper layers they capture increasingly complex semantic connections. Our experiments show that CST consistently outperforms Deep Sets and Set Transformer across multiple tasks, including Set Classification and Set Anomaly Detection. Moreover, unlike Deep Sets and Set Transformer, CST is natively compatible with explainability methods for CNNs, such as Grad-CAM [ 6 ] . In particular, we demonstrate that CST enables meaningful Grad-CAM visualizations in Set Anomaly Detection, while Set Transformer fails to provide reliable explanations (Figure 3 ). Finally, whereas prior set learning literature, such as [ 4 , 5 ] , has mainly relied on training models from scratch on small, low-resolution datasets, we train a CST on a large-scale benchmark (ImageNet [ 9 ]