MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets
Bowei Zhang, Yi Zhang · 2025
Read MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets by Bowei Zhang, Yi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-06040.txt.
Introduction
Bowei Zhang and Yi Zhang published a research paper in 2025 on a shrunken vision transformer that learns well from tiny image sets. Transformers shine on huge data but lag behind classic networks when data are scarce. Their design reads the image at several scales in every layer and trims the attention step to save work.
A wavelet-based filter pulls fine detail from the image early on. A light attention head cuts the token count, and a local feature block replaces fixed position codes. On a hundred-class photo set, the model scored well with only fourteen million parameters and no pretraining on big data. The authors aim it at real jobs where data stay small.
Catalog record
| CREATOR | Bowei Zhang, Yi Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-06040
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets Bowei Zhang, Yi Zhang Abstract Vision Transformer (ViT) has demonstrated significant potential in various vision tasks due to its strong ability in modelling long-range dependencies. However, such success is largely fueled by training on massive samples. In real applications, the large-scale datasets are not always available, and ViT performs worse than Convolutional Neural Networks (CNNs) if it is only trained on small scale dataset (called tiny dataset), since it requires large amount of training data to ensure its representational capacity. In this paper, a small-size ViT architecture with multi-scale self-attention mechanism and convolution blocks is presented (dubbed MSCViT) to model different scales of attention at each layer. Firstly, we introduced wavelet convolution, which selectively combines the high-frequency components obtained by frequency division with our convolution channel to extract local features. Then, a lightweight multi-head attention module is developed to reduce the number of tokens and computational costs. Finally, the positional encoding (PE) in the backbone is replaced by a local feature extraction module. Compared with the original ViT, it is parameter-efficient and is particularly suitable for tiny datasets. Extensive experiments have been conducted on tiny datasets, in which our model achieves an accuracy of 84.68% on CIFAR-100 with 14.0M parameters and 2.5 GFLOPs, without pre-training on large datasets. keywords Vision Transformer (ViT) ,Convolutional Neural Networks (CNNs) ,Multi-scale self-attention ,Tiny datasets † † corresponding: Corresponding author. 1 Introduction CNNs dominated computer vision field in early years. Recently, Transformer begin to prevail, especially ViT [ 1 ] excels in modelling and capturing long-range dependency between tokens. However, without sufficient training data, earlier attention layers have limited learning abilities for local information. For instance, the original ViT was pre-trained on large scale dataset JFT-300M and was fine-tuned on ImageNet-1K. However, if it was only trained on ImageNet-1K, it is inferior to CNNs. In depth exploration of the above phenomenon reveals the following reasons: Firstly, the original ViT lacks inductive bias. Although Pre-training on large-scale datasets compensates for this deficiency, helping ViT to learn stronger representations. In real applications, large-scale datasets are not always accessible, while pre-training followed by fine-tuning is also less desirable and unattainable. In this light, most of the previous methods modified ViT into hierarchical structures and combined convolutional computations within these structures. As a result, such hybrid structures behave more like CNNs, which are more competitive on large-sized datasets (e.g. ImageNet-1K), but there is still a performance gap on tiny datasets. Secondly, CNN has a natural advantage in learning local features.When dealing with sparse training data, the spatial correlation in the data is often insufficient, while CNN can capture local features, learn local information at lower levels, and integrate these local information at higher levels to obtain global information. On the contrary, according to [ 2 ] , without sufficient data, ViT has poor learning abilities for local information in earlier layers. Although ViT has a unified representation and contains more global information in different layers, it exhibits higher similarity across different layers, that means ViT aggregates more global attention in earlier self-attention layers, ignoring local attention (i.e. without sufficient training data, ViT would not learn to attend locally in earlier layers). In this paper, we aim to create a hybrid model that outperforms both CNN and Transformer on tiny dataset. During the attention computation stage, we extract fine-grained features and integrate coarsegrained features by stacking different-sized convolutional blocks in different attention heads. Additionally, we select deep convolution to merge attention tokens through experiments. Furthermore, to further enhance feature representation, we partially convolve input tokens, selecting redundant attention channels for local convolution calculation. This enables the fusion of local information extracted by convolution when interacting between different attention channels. Moreover, the traditional PE has been replaced by a local information encoding block, which not only provides inductive bias in an implicit way for the entire network structure but is also more beneficial to training on tiny datasets. Figure 1: Performance of MSCViT on CIFAR-10 and CIFAR-100. MSCViT performs better than some models with similar structures. Our model has been tested on popular tiny datasets, including CIFAR-10 [ 3 ] and CIFAR-100 [ 3 ] . We also select datasets with extremely imbalanced training samples per class, such as Flowers102 [ 4 ] , and datasets with fewer classes, such as Chaoyang [ 5 ] . Our results on each dataset are trained from scratch rather than transferring the pre-trained model. We develop 3 models of different scales, namely -tiny, -xs, and -s(small). As shown in Fig. 1 , on CIFAR-100, our xs model achieves top-1 accuracy of 83.44% with 7.8M parameters and 1.0 GFLOPs of computation. Meanwhile, our small model achieves 84.68% top-1 accuracy with 14.0M parameters and 2.5 GFLOPs of computation. Compared with other models with similar scales, our model is competitive without pre-training on large datasets. In summary, the contribution of this paper can be summarized as follows: • A Local Feature Extraction (LFE) module has been proposed, which is used to capture local information of the intermediate features and replace the PE in the original ViT network. • We study the role of deep convolution in lightweight token merging for attention computation. Based on this, a Lightweight Multi-scale Self-Attention (LMSSA) has been proposed to improve the model’s fine-grained feature extraction ability. • A Convolutional Feature Fusion (CFF) module has been presented to enhance the local feature extraction capability of input channels, which obtains object shape by using the combination of wavelet and conventional convolutions. It adjusts the relationship between convolution locality and attention globality between different channels in the channel dimension, improving the performance of hybrid structures on tiny datasets. 2 Related works 2.1 Vision Transformer (ViT) Albeit success in NLP, the self-attention mechanism in the original ViT often overlooks the detailed local features. To address this issue, DeiT [ 6 ] used distillation tokens to transfer CNN-based features into ViT. T2TViT [ 7 ] introduced tokenization modules to consider neighboring pixels, recursively rearranging images into tokens. DETR [ 8 , 9 ] input locally extracted features from CNNs into a Transformer encoder to model the global relationships between features in a serial way. CrossViT [ 10 ] processed patches of different sizes using a dual-branch Transformer, while LocalViT [ 11 ] integrated deep convolutions into ViT to improve the local continuity of features. The above works discussed the problem of insufficient feature learning, and proposed different solutions to it. Encouraged by them, we also focus on enhancing local feature extraction ability of ViT on tiny datasets. To accomplish dense prediction tasks such as object detection and semantic segmentation, some methods [ 12 , 13 , 14 , 15 ] introduced pyramid structures from CNNs to ViT backbones. PVT [ 12 ] introduced a pyramid structure into ViT, generating multi-scale feature maps for various pixel-level dense prediction tasks. Swin Transformer [ 13 ] replaced fixed-size positional embeddings with relative positional biases and limits self-attention within shifting windows. Twins [ 16 ] combined local and global attention mechanisms to obtain stronger feature representations. As can be seen, multi-scale feature extraction play critical role in dense prediction tasks. In order to improve the prediction accuracy of the