CNN-ViT Fusion with Adaptive Attention Gate for Brain Tumor MRI Classification: A Hybrid Deep Learning Model
Syed Ibad Hasnain, Hafiza Syeda Yusra Tirmizi, Muhammad Faris, Rabail Khowaja, Hafsa Israr · 2026
Read CNN-ViT Fusion with Adaptive Attention Gate for Brain Tumor MRI Classification: A Hybrid Deep Learning Model by Syed Ibad Hasnain, Hafiza Syeda Yusra Tirmizi, Muhammad Faris, Rabail Khowaja, Hafsa Israr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23137.txt.
Introduction
Hasnain and four colleagues fuse a SqueezeNet-style CNN branch with a MobileViT-style transformer branch to classify brain tumors in MRI scans. An Adaptive Attention Gate weights each branch per sample and per feature, so the merge adapts to each scan. On the Brain Tumor MRI Dataset, the hybrid reaches 97.60 test accuracy and beats single CNN and ViT baselines plus rival fusion methods. The result backs dynamic feature weighting as an effective route for medical image classification.
Catalog record
| CREATOR | Syed Ibad Hasnain, Hafiza Syeda Yusra Tirmizi, Muhammad Faris, Rabail Khowaja, Hafsa Israr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20425 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CNN-ViT Fusion with Adaptive Attention Gate for Brain Tumor MRI Classification: A Hybrid Deep Learning Model Syed Ibad Hasnain, Hafiza Syeda Yusra Tirmizi, Muhammad Faris, Rabail Khowaja, Hafsa Israr Abstract The problem of early detecting and classifying brain tumors using Magnetic Resonance Imaging (MRI) images is highly important but difficult to extract in medical images. Convolutional Neural Networks (CNNs) are good at capturing both local texture and spatial information whereas Vision Transformers (ViTs) are good at capturing long-range global dependencies. We propose a new hybrid architecture that combines a SqueezeNet-style CNN branch with a MobileViT-style global transformer branch, through an Adaptive Attention Gate mechanism, in this paper. The gate learns dynamically per-sample, per-feature weights to weight the contribution of each branch, allowing context-sensitive merging of local and global representations. The proposed model has a test accuracy of 97.60, a precision of 97.30, a recall of 97.50, an F1-score of 97.40, and a macro-average area under the curve (AUC) of 0.9946 with a trained and evaluated on the Brain Tumor MRI Dataset (Kaggle). These scores are higher than single CNN and ViT baselines, and current competitive fusion methods, showing that dynamic feature weighting is an effective way to classify medical images. Keywords Brain Tumor Classification \cdot MRI \cdot CNN \cdot Vision Transformer \cdot Feature Fusion \cdot Adaptive Attention Gate \cdot Deep Learning \cdot Medical Imaging. 1 Introduction Brain tumors are one of the most threatening and life-threatening types of malignancies as the World Health Organization notes that every year hundreds of thousands of new cases are diagnosed all over the world [ 1 ] . Correct and prompt identification of tumor type based on MRI is a requirement to identifying optimal treatment regimens such as surgical planning, radiation therapy, and chemotherapy regimens. Radiologists can be required to interpret images manually although this is time consuming, prone to inter-observer variability, and stretched by increasing diagnostic workload [ 2 ] . Deep learning has radically changed the medical image analysis scene. Convolutional Neural Networks, introduced and improved by VGG, ResNet, EfficientNet and SqueezeNet, have shown good results on both visual recognition challenges and clinical imaging. Their hierarchical nature of feature extraction, which uses local receptive fields and translation equivariance, is highly ideal in the extraction of tumor texture, boundary and morphology features in MRI slices. With the release of the Vision Transformer (ViTs) [ 3 ] , computer vision have entered a new paradigm: images are modeled as a sequence of patch tokens processed by the self-attention multi-heads. ViTs are also good at describing long range spatial dependencies that CNNs struggle to model due to fixed and local receptive fields. Nevertheless, ViTs usually require large-scale pre-training datasets to achieve their performance limit, and can be less effective than CNNs on small or medium medical imaging datasets when trained by default. MobileViT, an architecture developed [ 4 ] , solves this shortcoming by integrating MobileNet-style efficient convolutions with lightweight transformer blocks with the ability to reach competitive accuracy with dramatically fewer parameters[4]. This hybrid architecture is especially attractive to be deployed in clinical settings with resource constraints. Although CNNs and ViTs are complementary, feature concatenation or averaging do not fully take advantage of the complementary nature. Various input samples might need various configurations of local and global context [ 5 ] . An example is that a small, circumscribed pituitary adenoma can be better reliably categorized based on local textural information, but diffuse glioma infiltration may require that the transformer branch capture a wider area [ 6 ] . Motivated by our previous studies [ 7 , 8 , 9 ] , this study proposed an Adaptive Attention Gate (AAG) fusion strategy that learns a feature-dimension-wise soft gate \alpha\in(0,1)^{F} from the concatenation of both branch embeddings. The fused representation is computed as \text{fused}=\alpha\odot f_{\text{CNN}}+(1-\alpha)\odot f_{\text{ViT}} , where \odot denotes element-wise multiplication. This formulation allows the model to dynamically weight each branch’s contribution at the feature level and per-sample basis. The paper has the following contributions: •It introduce a new hybrid architecture with a SqueezeNet-like CNN branch, a MobileViT-like transformer branch, to classify brain tumors based on MRI. •It presents an Adaptive Attention Gate system that learns dynamically to strike a balance between local CNN and global ViT features on a per-sample and per-feature-dimension basis. •It shows that our model is 97.60% accurate on the Brain Tumor MRI Dataset, with higher accuracy than competitive baselines and previous fusion methods. •This research also proposed analysis of the values of gate activation, which can be interpreted as understanding the contribution of CNN or ViT to the model when applied to various tumor classes. 2 Related Work 2.1 CNN-Based Brain Tumor Classification Since the start of deep learning, CNN architectures have been the most popular method to use in brain tumor MRI classification. The study by Cheng et al. (2015) used AlexNet based features with data augmentation, and in comparison, significant benefits compared to hand crafted feature baselines were realized. Sultani et al. and their peers used VGG-16 and ResNet-50 on the multi-class tumor classification whereby ResNet-50 scored about 94.2 percent in similar dataset [ 10 ] . Variants of EfficientNet and specifically EfficientNetB3 scaled up accuracy to approximately 96.1% with the addition of scaling up network width, depth, and resolution using compounds [ 11 ] . Although originally created with parameter efficiency (less than 1MB) in mind, SqueezeNet established that parameter-efficient architectures of fire modules could be competitive in terms of accuracy with much lower parameters, which motivated the use of lightweight feature extractors [ 12 ] . 2.2 Vision Transformers in Medical Imaging Following the success of ViT for natural image classification, researchers rapidly adapted transformer architectures to medical imaging. ViT-B/16 achieved approximately 95.3% accuracy on brain tumor classification tasks [ 13 ] , demonstrating that global self-attention is beneficial for capturing anatomical spatial relationships. However, standard ViTs require large pre-training corpora; when trained from scratch on small medical datasets, performance degrades significantly. TransUNet [ 14 ] applied transformers to medical image segmentation, inspiring hybrid approaches. MedT [ 15 ] proposed gated axial attention for medical image segmentation, demonstrating the potential of attention mechanisms adapted specifically for medical contexts. 2.3 Hybrid CNN-Transformer Architectures Seizing the complementary aspect of local and global feature extraction, a number of studies have come up with hybrid architectures. One study[16] proposed MobileViT that is an inverted residual auto-superimposition of transformer blocks to support effective inference, hitting about 95.8% on brain tumor benchmarks [ 4 ] . The contentious CNN-Transformer fusion model by a research study [ 17 ] combined the ResNet feature and ViT features, but it learns a linear weighting to reach the accuracy of 96.4% [ 18 ] . Liu et al. (2023) suggested the network of two branches whose cross-attention is between CNN and ViT branches, and Naseem et al. (2022) implemented the ensemble averaging [ 19 ] . These writings encourage more advanced, vigorous mixture techniques. The Adaptive Attention Gate that we propose is differentiated by research on a different basis, where the computation of a feature-dimension-wise sigmoid gate to learn per-sample-weighting is performed instead of employing fixed linear weights or straightforwardly concatenating features. 2.4 Attention-Based Feature Fusion Attention-based fusion has been explored in multi-modal learning and