CNN-ViT Hybrid for Pneumonia Detection: Theory and Empiric on Limited Data without Pretraining

Prashant Singh Basnet, Roshan Chitrakar · 2025

Read CNN-ViT Hybrid for Pneumonia Detection: Theory and Empiric on Limited Data without Pretraining by Prashant Singh Basnet, Roshan Chitrakar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08586.txt.

Introduction

Prashant Singh Basnet and 1 co-author test a hybrid image model for finding pneumonia in this 2025 paper. The model joins a CNN, which scans local patterns, with a vision transformer, which looks at wider structure. Training uses limited data from scratch, with no pretraining.

Experiments vary how much data is used and whether sick and healthy cases are balanced. The hybrid reaches strong recall and steady scores, and it beats each part alone on uneven data. Training time stays close to transformer models. The work matters where medical image sets are small and uneven.

Catalog record

CREATORPrashant Singh Basnet, Roshan Chitrakar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CNN-ViT Hybrid for Pneumonia Detection: Theory and Empiric on Limited Data without Pretraining Prashant Singh Basnet, Roshan Chitrakar Abstract This research explored the hybridization of CNN and ViT within a training dataset of limited size; and, introducing a distinct class imbalance. The training was made from scratch with a mere focus on theoretically and experimentally exploring the architectural strengths of the proposed hybrid model. Experiments were conducted across varied data fractions with balanced and imbalanced training datasets. Comparatively, the hybrid model, complementing the strengths of CNN and ViT, achieved the highest recall of 0.9443 ( 50 % data fraction in balanced) and consistency in F1 score around ( 0.85 ) suggesting reliability in diagnosis. Additionally, the model was successful in outperforming CNN and ViT in imbalanced datasets. Despite its complex architecture, it required comparable training time to the transformers in all data fractions. I INTRODUCTION Pneumonia is a severe respiratory infection that inflates the air sacs in one or both lungs. It is diagnosed by observing the swellings on lung tissue [ 1 ] . The role of medical image analysis in early disease diagnosis and treatment is profound. With the advancement in technologies, deep Learning techniques, such as Convolutional Neural Networks (CNNs) have demonstrated impressive performance, outperforming statistical methods like MSE, SSIM, PCC, MI, and SVM. However, CNN is limited in capturing local features and falls short when the consideration of global dependencies is a must, take medical image analysis [ 2 ] . The recent breakthrough in Vision Transformer, a self-attention mechanism, has promising performance in capturing the complexities of the images [ 3 ] . However, it requires a large volume of training images to achieve satisfactory performance with a higher computational demand, which is not always possible in practical implications. Particularly in medical science, it the availability of accurate set of labeled and quality data remains a challenge, hence questioning the direct application of ViTs [ 4 ] . Although hybridization techniques explored in contemporary studies have achieved satisfactory performance, they rely on a pretrained model and lack findings under data-constrained settings where transfer learning is inefficient. To address this gap of ViT, this research proposes a novel CNN-ViT hybrid model, trained from scratch with data constraints. The study theoretically and empirically compares the hybrid with standalone CNN and ViT in varied scenarios to demonstrate its performance when the training dataset is limited and with high complexity. Furthermore, it evaluates the performance when the models are introduced with class imbalance. Vision Transformers, though capable of excelling in determining global dependencies, do not excel when the training dataset is limited. Meanwhile, the transformer architecture demands higher training resources, raising concerns in real-world applications because the attention mechanism demands quadratic complexity. Therefore, the study is motivated to reduce the variance of ViT by integrating CNN; with no dependency on pretrained models. The primary objective is to explore a noble hybrid approach (CNN-VIT) that combines the salient features of CNN and ViT with a minimized generalization error when the training size is limited, and achieve a satisfactory benchmark without relying on pretrained models. I-A Classical Statistical and Machine Learning Approaches Mean Squared Error(MSE) , a statistical tool with efficiency in resource utilization, had limitations in detecting structural differences, thus struggled to differentiate the visually alike images [ 5 ] . Structural Similarity Index (SSIM) overcame MSE’s gap by structurally differentiating the images [ 6 ] , yet with inaccuracies in comparing diverse color formats, as it considered image distortions to follow a Gaussian Distribution consistently [ 5 ] . Pearson Correlation Coefficient (PCC) emerged as a popular choice in pattern identification by establishing a linear relationship between images [ 7 ] ; also with an easy interpretation by providing output in the range between -1 to 1 [ 8 ] . However, the inconsistency in the linearity of correlation challenges the performance in complex data, where it falls short in capturing the relationship between variables. [ 9 ] noted the significance of Mutual Information (MI) in capturing complex information of the images, such as brightness, contrast, and scale, whereas [ 5 ] mentioned its advantage in capturing nonlinear correlation between images while highlighting its limitation in feature capturing and computational efficiency. Support Vector Machine (SVM) , a supervised machine learning model, classifies data by drawing a hyperplane separating two classes [ 10 ] , [ 11 ] . It maximizes the margin, the distance between the hyperplane and the closest data points of each class. To improve its efficiency in complex tasks, techniques like ensemble in multi-class classification, feature overlapping in image annotation, and integrating visual and textual features were explored, yet with limited empirical evidence [ 12 ] . Despite the efforts, the inefficiency remained relevant in large or imbalanced datasets due to the direct proportion of training time to integrations in LIBSVM [ 13 ] . Although its performance in DICON datasets was above 99%, it sharply declined with JPEG medical images [ 14 ] . I-B Deep Learning Techniques Convolutional Neural Network (CNN) addressed the challenges such as semantic gap, feature overlapping, image segmentation accuracy, integration of visual and textual features, evaluation challenge, relevant in CBIR. Visualization techniques given by [ 15 ] bridged semantic gaps between low-level image features and high-level semantic concepts. Additionally, CNN addressed the gap of SVM in JPEG radiographical images by achieving an accuracy of 100% in all instances; while SVM was limited to 87.5 % in CT, 95.8 % in MRI, and 100 % in X-Ray respectively [ 14 ] [ 16 ] . Additionally, [ 17 ] achieved a top 5 error rate of 18.2 % and an average of 16.4 % in 5 similar predictions on a deep convolutional network (60 million parameters) trained on 1.2 million high-resolution images across 1000 classes with a 1000 SoftMax output given by 3 connected layers. However, [ 2 ] noted CNN’s inefficiency in capturing long-range dependencies and relevancy overfitting with denser networks. Vision Transformers (ViT’s) flexible architecture captures global dependencies between input and output, hence providing an effective learning mechanism between distant sequences. Flexibility in parallelization; added advantage to achieve scalability in depth compared to the convolutional layers [ 2 ] ; the division of images into smaller patches contributes to effectively capturing global information; comprehensively contributes to outweigh CNN for detail-centric image recognition tasks. The transformer-based technique does not rely on convolution and pooling operations because it treats image classification as a sequence of processing tasks. It breaks down an image into multiple patches, which are discrete and individually meaningful. Subsequently, visual features are extracted from each of the patches by processing them through a series of attention mechanisms. These attention mechanisms capture the global and local contextual relationships, which outshine CNN in capturing complex information. [ 3 ] explored the application of ViT, initially designed for textual tasks, in image classification. An accuracy of 88.5 % on a larger dataset ( 14M – 300 M images, ImageNet); 90.72 % on ImageNet-RealL (diverse datasets); and 94.55 % on CIFAR-100 (extensive category) was achieved, signifying its impressive performance with a larger scale of training data and with images having natural and adverse disturbances [ 4 ] . At the same time,

Read the complete text