Optimizing multi-label image annotation: a hybrid CNN-Transformer deep learning approach Authors: Chitrapriya Ningthoujam, Tejbanta Singh Chingtham, Dhruba Ningombam Published: 2026 ABSTRACT AbstractMulti-label image annotation remains a challenging task due to highly imbalanced label distributions, co-occurring semantic concepts, and the presence of rare or noisy annotations. To address these challenges, we present ReSwinNet, a hybrid CNN–Transformer framework that integrates ResNet-50 for fine-grained spatial feature extraction with the Swin Transformer for capturing long-range dependencies through hierarchical self-attention. A token projection mechanism maps multi-scale CNN features directly into Swin-compatible representations, bypassing standard patch embedding to preserve fine-grained spatial context that conventional patch partitioning discards. A frequency-based label pruning strategy further reduces annotation noise and class imbalance by retaining only semantically stable labels, significantly improving training convergence and predictive focus. Experiments are conducted on two benchmark datasets under multiple label settings to ensure transparent and fair evaluation. Under the full 374-label space on Corel-5K, ReSwinNet achieves a macro F1 of 0.491, surpassing 7 of 10 compared state-of-the-art methods and recording the highest recall among all compared models. Under the leakage-free 65-label pruned setting, ReSwinNet achieves a macro F1 of 0.701 and mAP of 0.742, outperforming all compared methods under identical experimental conditions. On IAPRTC-12, ReSwinNet achieves a macro F1 of 0.596 and mAP of 0.629 under the 198-label pruned setting, representing an improvement of over 14 percentage points in mAP relative to the strongest baseline. These results confirm the effectiveness of spatially-aware token fusion combined with frequency-based label filtering for large-scale multi-label image annotation.Keywords: Multi-label, Optimization, Annotation, Imbalance, Thresholding, Swin, Hybrid, ResNet, CNN, Label pruningSubject terms: Computational biology and bioinformatics, Engineering, Mathematics and computing Introduction Multi-label image annotation is a fundamental problem in computer vision, where a single image can be associated with multiple semantic categories simultaneously. This task plays a vital role in applications such as image retrieval, scene understanding, content moderation, and medical diagnostics. Accurate annotation not only improves multimedia retrieval systems but also ensures reliable performance across diverse real-world scenarios. Unlike single-label classification, multi-label annotation introduces additional complexity due to the high dimensionality of the label space and dependencies among labels. A key challenge in this domain is label imbalance, where frequent labels dominate the training process while rare labels are underrepresented. This often leads to biased learning, overfitting to dominant categories, and degraded performance on infrequent labels1,2. Su et al.3developed a Visual Contextual Relationship Augmented Transformer (VCRAT) for image captioning, where self-attention-based contextual feature enhancement enhanced the representation of spatial and semantic correlations in images . Recent approaches attempt to address this imbalance through techniques such as data augmentation and resampling4, but these solutions either increase computational costs or fail to generalize across severely imbalanced datasets. This creates a significant barrier to deploying annotation systems in practical environments. Deep convolutional neural networks (CNNs), including ResNet5, are widely adopted for robust spatial feature extraction in annotation tasks6,7. More recently, Vision Transformers (ViTs) and hierarchical variants like the Swin Transformer8 capture long-range dependencies and fine-grained semantic relationships. However, balancing label distributions efficiently remains an open challenge. To address these challenges, we propose the ResNet-Swin Fusion Network (ReSwinNet) shown in Fig. 1 , a unified deep learning framework that integrates threshold-based label pruning with a hybrid CNN–Transformer architecture. The key contributions of this work are as follows: A frequency-based pruning strategy that retains only labels whose occurrence frequency meets or exceeds the global mean, effectively reducing noise and mitigating both intra- and inter-label imbalance without heuristic thresholds. A frequency-based pruning strategy that retains only labels whose occurrence frequency meets or exceeds the global mean, effectively reducing noise and mitigating both intra- and inter-label imbalance without heuristic thresholds. A frequency-normalized image sampling mechanism that constructs a balanced training subset, ensuring approximately uniform representation across the pruned label space. A frequency-normalized image sampling mechanism that constructs a balanced training subset, ensuring approximately uniform representation across the pruned label space. A dual-branch model that combines ResNet-50 for spatial feature extraction with intermediate-layer fusion into a Swin Transformer backbone, enabling the integration of both low-level spatial and high-level contextual representations. A dual-branch model that combines ResNet-50 for spatial feature extraction with intermediate-layer fusion into a Swin Transformer backbone, enabling the integration of both low-level spatial and high-level contextual representations. The entire pipeline from label pruning and balanced sampling to feature extraction and semantic annotation is encapsulated in a unified architecture that maintains consistency through frequency-based selection and deep feature fusion. The entire pipeline from label pruning and balanced sampling to feature extraction and semantic annotation is encapsulated in a unified architecture that maintains consistency through frequency-based selection and deep feature fusion. Outline of the proposed ResSwin Net model. The paper is structured as follows: Section “Related work” reviews of related work in multi-label annotation. Section “Proposed methodology” presents the proposed ReSSwin Net framework. Section “Experimental setup” describes the experimental setup. Section “Results and discussion” discusses results and comparative analysis. Finally, Section “Conclusion” concludes with key insights and future research directions. Related work Studies on automatic image tagging have evolved from initial non-deep techniques to sophisticated deep learning models. The background research can be generally divided into non-deep techniques, threshold-centric methods, deep learning strategies, and hybrid models. Non-Deep Methods Early approaches employed metric learning and dictionary-based embeddings. For example, 2pKNN6 integrates metric learning with multi-label classification, while SLED7 introduces a semantic label-embedding dictionary to capture shared label combinations. Collaborative frameworks9 and sparse reconstruction methods such as 2PKNN-GSR10 further enhanced robustness against noisy or incomplete labels. Threshold-Based Annotation Thresholding strategies have been widely adopted. CNN-AT11 proposed instance-specific adaptive thresholds within a ConvNet-driven binary relevance framework. Extensions in12,13 used neural ranking and nearest-neighbor reasoning to improve prediction. Earlier non-parametric methods relied on density estimation in the word–image space, but lacked scalability. Deep Learning Approaches The adoption of CNNs led to major advances in feature learning. CNN-based architectures14,15, pre-trained models16, and hybrid CNN-RNN frameworks17 captured both label dependencies and image–label relevance. Post-processing refinements, such as label correction18,19, improved robustness. Descriptor-driven enhancements included dominant color descriptors20, edge-weighted descriptors21, and VGG-16 retrieval-based models22. Evolutionary methods such as MLFPSO23 optimized feature selection but incurred high computational costs.EAPT24 shows that pyramid-based attention can greatly enhance efficiency and precision in image processing tasks, achieving a balance between global context modeling and local detail retention.25 addresses how multi-label classifications can be achieved using clustering the space of labels to integrate global and local strategies. Recent Advances and Hybrid Architectures Recent work has emphasized advanced architectures and hybrid frameworks. DMCKN26 incorporated geometric context via multi-order patch connections. A dynamic thresholding method based on Matthews Correlation Coefficient (MCC) was introduced in27 to mitigate label dominance. Semi-supervised frameworks, such as adaptive weighted fusion with co-training28, enhanced robustness under partially labeled data. Similar to Zhang and Tian’s mutual-attention Transformer for anomaly detection29, which demonstrated that attention mechanisms can effectively fuse local and global features, our work also leverages Transformer architectures (Swin) to capture long-range dependencies for improved multi-label annotation, though applied to the challenge of label imbalance rather than anomaly detection. Hybrid CNN–Transformer models have gained attention for combining local and global feature representations. For example, ResNet–SLT–Word2Vec integration5 addressed the semantic gap, while Swin Transformer models8 have been adapted for multi-label annotation, including domain-specific tasks such as content-level annotation of construction imagery30. A more recent contribution, DMVMLC-VT31, addresses the problem of incomplete multi-view multi-label classification by combining autoencoder-based view representation with a completion module for missing views. It further employs contrastive fusion, pseudo-label enhancement, and a label balancing strategy to mitigate the effects of missing or skewed annotations. Graph-based approaches have also been investigated to capture label correlations. A recent research on multi-label semantic sharing in image-to-text retrieval32 utilizes graph convolutional networks (GCNs) to represent semantic connections between labels, consequently improving cross-modal retrieval efficiency. Although these methods effectively model label dependencies, they struggle to capture long-range visual context and may not directly resolve imbalance problems. In contrast, our proposed ReSwinNet utilizes CNN Transformer integration to seize both local and global visual characteristics, while incorporating label pruning to reduce the impact of infrequent or noisy labels.33introduced a unified learning framework that combines low-level and high-level attributes for multi-label identification of ships in intricate environments, showing that merging hierarchical representations improves distinguishing capability in various visual scenarios. Hybrid frameworks have also investigated feature fusion. For instance, a recent investigation into solid waste categorization34 utilizes multi-layer feature integration through the integration of features from EfficientNetB0 and InceptionV3 employing a HyperColumn method, fully connected layers and an enhancer classifier. Although these fusion techniques improve classification precision by utilizing complementary attributes,they are still restricted to CNN backbones and fixed concatenation methods While effective in handling view incompleteness and class imbalance, DMVMLC-VT primarily focuses on multi-view data integration rather than hybrid CNN–Transformer architectures. In contrast, the proposed ReSwinNet integrates statistical label pruning with CNN–Transformer fusion, explicitly addressing both label imbalance and feature integration within an end-to-end framework. Table 1 presents a comparison of existing annotation models with the proposed ReSwinNet. It highlights differences in annotation strategies, thresholding methods, and handling of label imbalance. ReSwinNet stands out by using dynamic thresholding and a hybrid CNN-Transformer fusion to address imbalance more effectively than traditional approaches. Table 1Comparison of existing multi-label annotation models with the proposed ReSwinNet.ModelAnnotation approachThresholding methodHandles label imbalanceCNN-AT11ConvNet + BR frameworkAdaptive (instance-wise)Limited via resamplingDMCKN26Context-aware kernel networkStatic/global thresholdContext-aware weightingCNN-RNN17CNN + RNN label generationSequence-based thresholdsManual pruningDMVMLC-VT31Multi-view, autoencoder-based with view translation and pseudo-labelspseudo-labelingBalancing via pseudo-label enhancement and label distribution controlReSwinNet (proposed)CNN–Transformer hybrid with feature fusionDynamic, label-specificFrequency-based pruning and balanced sampling Comparison of existing multi-label annotation models with the proposed ReSwinNet. Proposed methodology The proposed model addresses the challenges of multi-label image annotation by initially reducing the label space complexity and subsequently employing a hybrid deep learning framework for enhanced classification performance as shown in Fig. 2. Flow of the proposed ResSwin Net model. The methodology is structured into three core stages: (1) Label Pruning, where infrequent or redundant labels are eliminated to simplify the output space (2) Feature Extraction using ResNet-50, which captures high-level semantic features from the input images and (3) Annotation via Swin Transformer, which exploits hierarchical self-attention to model inter-label dependencies and produce the final multi-label predictions. Label pruning The infrequency of specific labels in a dataset can lead to several challenges in machine learning models. Models tend to overlearn frequent labels while neglecting rare ones, resulting in imbalanced learning and potential overfitting. This causes poor generalization on new images, particularly for sparse labels, leading to degraded precision and recall for these labels. As noted in prior works, this imbalance causes models to overfit on prevalent classes while neglecting infrequent ones, severely degrading recall and precision for underrepresented categories. To ensure the model learns from statistically significant patterns rather than noise, we employ a frequency-based pruning and balanced sampling strategy, as formalized in Algorithm 1. Rather than using a fixed threshold alone, we incorporate a tolerance margin around the average label frequency . Specifically, a label with frequency is retained if it satisfies the condition: Equation (1) ensures that labels whose occurrence count is close to or above the average are preserved, while those far below the average are pruned. The concern regarding potential sampling before splitting is also addressed by the proposed methodology that follows a strict sequential pipeline that enforces dataset isolation at every stage. First, the complete dataset is partitioned into disjoint training and testing splits, denoted as and , prior to any label pruning or sampling operations. All subsequent steps are conditioned on this pre-split isolation. Specifically, label frequency statistics and the corresponding retained label set are computed exclusively from the training split. Using this train-derived label set, the test split is only filtered to remove pruned labels , resulting in , without performing any instance-level sampling. Balanced sampling is then carried out solely within the training partition , where samples are drawn only from subsets corresponding to each retained label. Although multi-label instances may be selected across multiple labels, duplicate samples are removed prior to training, ensuring that each image appears at most once in the final training subset . Equation 2 rigorously guarantees the absence of data leakage by enforcing an explicit identity-level verification, formalized as where denotes the set of unique instance identifiers, confirming a zero-overlap (0.0%) between training and testing samples. Label pruning and balanced sampling. Feature extraction using ResNet-50 Following label pruning, the next stage in the proposed pipeline is feature extraction using the ResNet-50 architecture shown in Fig. 3. ResNet-50 is a widely used deep convolutional neural network consisting of 50 layers and residual connections, which mitigate the vanishing gradient problem common in very deep networks. It is particularly effective at capturing hierarchical visual representations, making it well-suited for feature extraction tasks in computer vision. Architecture of feature extraction and annotation using proposed ResSwin Net model. Given an input image , ResNet-50 processes the image through its convolutional and residual blocks, producing a high-dimensional feature tensor , where (h, w) is the reduced spatial resolution after several downsampling stages. To integrate these features with the subsequent Swin Transformer module, a dimensionality reduction step is applied using a convolutional layer. This projection maps the feature tensor into a lower-dimensional space of size suitable for tokenization. The transformation is mathematically represented as: This operation not only reduces the computational complexity of the Swin Transformer but also preserves the essential spatial features extracted by ResNet-50. The use of pretrained ImageNet weights for ResNet-50 enables transfer learning, allowing the model to converge faster and perform better on relatively small datasets, such as ours with 5000 samples. An overview of this process is illustrated in Fig. 2, where the ResNet block receives pruned images and outputs a projected feature map passed into the Swin Transformer for annotation. Annotation with Swin Transformer In the proposed framework in Fig. 3, annotation is performed using the Swin Transformer after feature extraction and fusion from the ResNet-50 backbone. Given an input image , the initial layers of ResNet-50 extract hierarchical features across different levels. The output features from three stages are as follows: Each of these outputs is projected to lower-dimensional spaces using convolutional layers: These three feature maps are concatenated along the channel dimension and compressed: To obtain a compact annotation embedding, global average pooling is applied across the spatial dimensions, resulting in a final vector . This vector is normalized and then passed through a linear classification head to produce logits , where denotes the number of selected valid labels after pruning. The Swin Transformer expects input in the format , so we permute the tensor: This is passed to the Swin Transformer, which applies window-based self-attention and hierarchical merging to extract semantic embeddings: A global average pooling operation is applied across the spatial dimensions of the Swin Transformer output to obtain a compact annotation embedding , where is the batch size. This embedding captures the aggregated contextual information from the spatial features. The resulting vector is then normalized and passed through a linear classification head to produce logits , where represents the number of selected valid labels after pruning. Binary Cross-Entropy (BCE) loss function was employed which is suitable for independent label predictions. The BCE loss for a single instance is defined as: where denotes the ground truth label for class , and is the predicted probability. The loss is averaged across all samples in the batch to form the final training objective. This loss encourages the model to independently maximize the likelihood of each ground truth label while penalizing incorrect predictions for others. Finally, a classification head consisting of LayerNorm and a fully connected layer is used to generate logits for each label: where is the number of pruned labels. To obtain label probabilities: Here, denotes the sigmoid activation function applied element-wise. This final output serves as the predicted likelihood for each label being present in the input image. Experimental setup Datasets The model performance was evaluated using two benchmark datasets: Corel5K and IAPRTC-12. Table 2 provides a detailed description of both datasets after preprocessing, highlighting their key characteristics. Corel5K: The Corel5K dataset35,36 contains 50 categories comprising a total of 5000 images. Each category includes 100 images with resolutions of either or pixels. Corel5K is a multi-label dataset, meaning that each image can be annotated with multiple labels. In total, the dataset includes 374 unique labels with an average of 3.4 labels per image. Corel5K: The Corel5K dataset35,36 contains 50 categories comprising a total of 5000 images. Each category includes 100 images with resolutions of either or pixels. Corel5K is a multi-label dataset, meaning that each image can be annotated with multiple labels. In total, the dataset includes 374 unique labels with an average of 3.4 labels per image. IAPRTC-12: The IAPRTC-12 dataset37,38 is a benchmark collection designed for research in image annotation and scene classification. It consists of 17,585 natural images covering a wide variety of indoor and outdoor scenes, objects, and events. Each image is annotated with a set of 80 semantic concepts, including objects (e.g., car, building), scene types (e.g., beach, urban), and abstract attributes (e.g., sunny, cloudy), with an average of 5.7 labels per image. IAPRTC-12: The IAPRTC-12 dataset37,38 is a benchmark collection designed for research in image annotation and scene classification. It consists of 17,585 natural images covering a wide variety of indoor and outdoor scenes, objects, and events. Each image is annotated with a set of 80 semantic concepts, including objects (e.g., car, building), scene types (e.g., beach, urban), and abstract attributes (e.g., sunny, cloudy), with an average of 5.7 labels per image. Table 2Dataset statistics for Corel-5k and IAPRTC-12 after applying frequency-based label pruning and leakage-free preprocessing. For IAPRTC-12, Version 2 retains more labels than Version 1 due to a relaxed training-only pruning threshold, whereas for Corel-5k, Version 2 retains fewer labels due to stricter leakage correction.DatasetConfiguration#Labels#ImagesTrainValTestCorel-5kVersion 1 (Frequency-based pruning)842,9041,978436436Version 2 (Leakage-free strategy)654,1673,147521499IAPRTC-12Version 1 (Frequency-based pruning)6212,30925,1233,1223,132Version 2 (Leakage-free strategy)19817,58514,1201,7151,750 Dataset statistics for Corel-5k and IAPRTC-12 after applying frequency-based label pruning and leakage-free preprocessing. For IAPRTC-12, Version 2 retains more labels than Version 1 due to a relaxed training-only pruning threshold, whereas for Corel-5k, Version 2 retains fewer labels due to stricter leakage correction. Data preprocessing For both datasets, frequency-based label pruning was applied to remove infrequent labels and reduce noise. For Corel-5K, the initial frequency-based pruning (Version 1) reduced the label vocabulary from 374 to 84 labels by retaining only those labels whose occurrence frequency met or exceeded the training set mean. A further leakage-free strategy was subsequently applied, computing frequency statistics exclusively on the training split, which reduced the retained label set from 84 to 65 labels (Version 2). For IAPRTC-12, the initial frequency-based pruning (Version 1) reduced the label vocabulary from 291 to 62 labels, retaining only those labels whose occurrence frequency met or exceeded the dataset mean of 281.94. Version 2 adopts a leakage-free strategy with a relaxed pruning threshold computed exclusively on the training set, retaining 198 labels to include categories previously discarded under the stricter Version 1 criterion, thereby enabling evaluation across a broader and more representative label space. It is noted that the direction of change between Version 1 and Version 2 differs between the two datasets: for Corel-5K, the retained label count decreases (84 to 65) due to leakage correction, whereas for IAPRTC-12 it increases (62 to 198) due to a relaxed threshold applied on training data only To address class imbalance, images were subsampled such that the number of images per retained label approximated the computed average label frequency, ensuring balanced representation across classes and reducing the dominance of high-frequency labels. Specifically, the number of images sampled per label was proportional to its frequency in the dataset, subject to a minimum of 20 and a maximum of 150 samples per class, with a baseline target of 50 samples. This resulted in 3,668 unique training images and 499 test images for the Corel-5K Version 2 configuration, ensuring balanced class representation while preventing high-frequency labels from dominating the training distribution. For IAPRTC-12, the class distribution exhibited severe imbalance, with label frequencies ranging from as few as 1 to as many as 17 images per label. To accommodate this disparity, the sampling bounds were adjusted to a minimum of 50, a maximum of 800, and a baseline of 150 samples per class. This wider range allowed sufficient representation of rare labels while preventing dominant classes from overwhelming the training distribution, ensuring a more balanced learning signal across the 198 retained labels under Version 2. Further, to prevent data leakage, the dataset was first divided into training, validation, and test sets prior to any augmentation, as described in Algorithm 1. Each image was assigned exclusively to a single split, ensuring that no overlap existed across the partitions. Data augmentation was subsequently applied only to the training set. Consequently, no augmented variant of a training image could appear in either the validation or test sets. This strict separation ensures that the evaluation metrics accurately reflect the model’s generalisation capability on previously unseen data. Threshold optimization In the prediction stage, we adopted an adaptive thresholding strategy. Instead of applying a fixed threshold to the sigmoid output for all labels, we computed optimal thresholds for each label individually using validation data. These thresholds were selected to maximise the F1-score and the precision-recall balance, making the model more sensitive to both common and rare labels. Model outputs are produced as per-label probability scores via a sigmoid activation. An optimal decision threshold iis determined empirically on the validation set by maximising the macro F1-score over candidate thresholds in the range [0.10, 0.95] at increments of 0.05, yielding 50 candidate values. The selected threshold is formally defined as shown in : This threshold is saved at the best validation epoch and applied consistently during final evaluation on the test set, ensuring that the decision boundary is calibrated to the model’s output distribution rather than relying on an arbitrary fixed value. The test set is never accessed during threshold selection, ensuring strict separation between model selection and evaluation protocols. A post-hoc per-label threshold sweep, also conducted over 50 equally spaced steps in [0.10, 0.95], is additionally reported as a diagnostic analysis to characterize per-class decision boundaries. This per-label analysis does not influence any primary reported result, which uses exclusively the validation-derived global threshold . All results are reported using macro-averaged Precision, Recall, and F1-score Evaluation metrics For evaluating the proposed multi-label image annotation model, we employ six key metrics tailored to multi-label annotation: Average Precision (), Label-wise Recall (), Label-wise Precision (), mean Average Precision (), F1 Score (), Hamming Loss (), and ROC-AUC (). The Precision for label measures the fraction of predicted positives that are correct: where and denote the true positives and false positives for label , respectively. The Recall for label measures the fraction of actual positives that are correctly identified: where denotes the false negatives for label . The F1 Score for label is the harmonic mean of precision and recall, balancing both objectives: The Average Precision for label is the area under the Precision-Recall curve, computed as a weighted sum over all decision thresholds: where and denote precision and recall at the decision threshold. All per-label metrics are aggregated using macro-averaging, which computes the metric independently for each label and takes the unweighted mean, ensuring equal treatment of all labels regardless of their frequency: The Hamming Loss measures the fraction of label slots predicted incorrectly across all images and labels: where is the ground-truth label and is the predicted label for image and label . A lower Hamming Loss indicates better performance. The ROC-AUC for label measures the probability that the model ranks a randomly chosen positive instance above a randomly chosen negative instance, computed as the area under the Receiver Operating Characteristic curve: where and denote the true positive rate and false positive rate for label , respectively. The macro-averaged ROC-AUC is . These metrics are computed per label due to the imbalanced nature of multi-label datasets, where frequent labels can dominate performance metrics, then aggregated via macro-averaging to assess overall model performance. Threshold-sensitive metrics (, , , ) use an adaptive threshold optimised on the validation set, while ranking-based metrics (, ) operate directly on predicted probabilities and are threshold-free. Results and discussion Experimental results on Corel5K dataset The proposed ReSwinNet was evaluated on the Corel5K dataset, a widely used benchmark for multi-label image annotation. The architecture processes each input image resized to through ResNet-50, extracting deep spatial features and producing an output feature map of size . The experimental pipeline underwent a two-stage label space refinement to ensure rigorous and leakage-free evaluation, the results of which are summarised in Table 3. In the initial phase (Version 1), label pruning based on frequency thresholding was applied to the merged train and test pool, retaining 84 labels. However, this setting introduced a label selection leakage issue, as label frequencies were computed across both splits rather than exclusively on the training set. Consequently, Version 1 produced inflated performance metrics (P=0.98, R=0.94, F1=0.96, mAP=0.9249), which do not reflect true generalisation ability, as illustrated in Fig. 4. In the corrected leakage-free pipeline (Version 2), label frequencies were computed exclusively on the training set, reducing the retained label vocabulary to 65 labels. This correction resulted in a notable reduction in reported performance (F1=0.7008, mAP=0.7415), which more accurately reflects the model’s ability to generalise to unseen data. The substantial drop between Version 1 and Version 2 confirms that the earlier results were overly optimistic due to leakage. Image-level frequency-proportional sampling was further applied to ensure approximately equal representation per label, addressing the dataset’s inherent class imbalance. All subsequent experiments on Corel-5K are conducted exclusively under the leakage-free Version 2 setting. Table 3Performance comparison of label pruning strategies on the proposed ReSwinNet model (Corel5K dataset).ConfigurationPrecision (P)Recall(R)F1-score(F1)mAPVersion 1 (Frequency-based pruning)0.980.940.960.9249Version 2 (Leakage-free strategy)0.73410.71180.70080.7415 Produced under label selection leakage; retained for transparency only. Performance comparison of label pruning strategies on the proposed ReSwinNet model (Corel5K dataset). Produced under label selection leakage; retained for transparency only. Training and validation curves of ReSwinNet on the Corel5K dataset under the 84-label setting, showing the progression of loss, mAP, and F1-score during the training process. While in the corrected leakage-free pipeline (Version 2), label frequencies were computed exclusively on the training set, reducing the retained label vocabulary to 65 labels. This correction resulted in a notable reduction in reported performance (F1-score: 0.70, mAP: 0.7415), which more accurately reflects the model’s ability to generalize to unseen data. The substantial drop between Version 1 and Version 2 confirms that the earlier results were overly optimistic due to leakage. Image-level frequency-proportional sampling was further applied to ensure approximately equal representation per label, addressing the dataset’s inherent class imbalance. All subsequent experiments on Corel5K are conducted under the leakage-free Version 2 setting. The ranking and global evaluation metrics further demonstrate the effectiveness of the proposed model under the 65-label pruned setting. . It is shown that a Coverage Error of 8.75 indicates that, on average, fewer than nine labels are required to cover all relevant labels for an image, reflecting effective label ranking. The Ranking Average Precision of 0.7516 and the low Ranking Loss of 0.0474 indicate that relevant labels are generally ranked higher than irrelevant ones. Moreover, the Exact Match Ratio of 0.2525 and the low Hamming Loss of 0.0235 confirm the reliability of the model in predicting accurate label assignments. The per-label threshold sweep is visualized across all 65 retained labels, with the optimal threshold marked per label.Threshold selection was performed independently per label on post-hoc test predictions after the model was fully trained and fixed as illustarted in Fig. 5. It is observed that optimal thresholds vary considerably across labels for example, train (Best=0.95), ocean (Best=0.93), garden (Best=0.91) versus leaf (Best=0.18), scotland (Best=0.25), tree (Best=0.23) demonstrating that a single global threshold would be suboptimal and that per-label calibration is justified. This analysis is performed as a post-hoc diagnostic on test predictions and does not influence any primary reported result, which uses exclusively the validation-derived global threshold . Sample per-label threshold analysis showing the variation of F1-score with respect to the decision threshold. The dashed vertical line indicates the threshold value that maximizes the F1-score for each label on Version 2 pruning. The corresponding per-label evaluation metrics are summarized in Fig. 6, reporting per-class Precision (Mean=0.734), Recall (Mean=0.712), F1 score (Mean=0.701) and Average Precision (Mean=0.682). These results indicate consistent performance across all 65 retained labels, with considerable per-label variation reflecting the inherent difficulty differences across semantic categories. Test sample per-class evaluation metrics on for all 65 retained labels showing precision on Corel5k dataset. We employ Precision@4 as a ranking metric since the average number of labels per image in Corel5K is approximately four. This metric evaluates how well the model ranks the most relevant labels in a realistic multi-label annotation scenario. Fig. 7 presents qualitative predictions on three representative test images. In the first image, ground truth labels coral, ocean and people are correctly predicted with high confidence alongside the plausible label reefs, yielding Precision@4=0.75. The second image includes ground truth labels mountain, scotland and water, all accurately predicted alongside the contextually relevant label river (Precision@4=0.75). The third image achieves a perfect match across all four ground truth labels mountain, valley, grass and road (Precision@4=1.00). The average Precision@4 across these three samples is 0.83, demonstrating the model’s ability to generalise across both local spatial features and global scene semantics in multi-label annotation settings. Further Fig. 8 show random test images from the Corel5K dataset under the 65-label leakage-free setting. Qualitative results on the Corel5K dataset (84 labels) using ReSwinNet model. Ground truth and predicted labels are shown below each image, with predictions ranked by confidence score. Green labels indicate correct predictions and red labels indicate incorrect predictions. Results are reported using Precision@4, evaluating the top-4 predicted labels per image. Qualitative top-4 predictions of ReSwinNet on Corel5K (65-label leakage-free setting). Experimental results on IAPRTC dataset The proposed ReSwinNet model was evaluated on IAPRTC-12 dataset with the following statistics of dataset as shown in Table 4. The training dynamics of the proposed model on the pruned IAPRTC-12 dataset are evaluated through loss, accuracy, mean average precision (mAP), and F1 score. As shown in Fig. 9, while the model converges strongly on the training data, a consistent gap between training and test performance across all metrics indicates significant overfitting, primarily attributed to the constrained size of the pruned dataset. Table 4Label frequency analysis and pruning statistics for the IAPRTC-12 dataset.StatisticValueTotal number of original labels291Average label frequency281.94Labels retained after pruning ( average frequency)62 Label frequency analysis and pruning statistics for the IAPRTC-12 dataset. Training and test performance curves for the proposed ResSwin model on IAPRTC dataset on 62 pruned label set. The plots show (top left) loss, (top right) accuracy, (bottom left) mean average precision (mAP), and (bottom right) F1-score. Solid blue lines indicate training metrics, while dashed red lines represent corresponding test metrics. Training loss declines steeply to near zero, whereas test loss remains flat throughout, with the two curves diverging continuously beyond the early epochs, suggesting the model transitions from learning generalizable representations to memorizing training-specific patterns. A similar trend is observed in accuracy, where the test curve stagnates despite continued improvement in training accuracy. The most pronounced divergence is observed in mAP, where training mAP reaches approximately 0.97 while test mAP plateaus near 0.80, a gap of 17 percentage points. The F1 score reflects a comparable disparity of 11 percentage points, confirming that predictions optimised on the training distribution do not transfer effectively to unseen samples. These results indicate that the primary bottleneck is data quantity rather than model architecture or loss design, and motivate the use of the broader Version 2 label setting for a more representative evaluation (Table 5). Table 5Empirical performance of the ReSwinNet model on the IAPRTC-12 dataset in terms of Precision, Recall, F1 Score and mAP . ModelPrecisionRecallF1 ScoremAPVersion 1 (62 labels)0.98560.850.88000.7885Version 2 (198 labels)0.61380.57950.59610.6293 Empirical performance of the ReSwinNet model on the IAPRTC-12 dataset in terms of Precision, Recall, F1 Score and mAP . Version 2, trained on 198 labels, exhibits a notable decline across all metrics, with Precision dropping to 0.6138, Recall to 0.5795, F1 score to 0.5961, and mAP to 0.6293. This degradation is consistent with increased label space complexity, where a larger proportion of rare and semantically ambiguous labels introduces greater class imbalance, making it harder for the model to learn reliable decision boundaries. The trade-off between label coverage and classification reliability is therefore a central consideration in multi-label recognition, and motivates further investigation into adaptive pruning thresholds and label-aware loss weighting strategies. All threshold selection is performed to avoid any test-set bias,exclusively on the validation set by maximizing macro F1 over candidate thresholds in [0.10, 0.95], and the resulting threshold is applied without modification to the test set for final evaluation. The per-label threshold sweep reported in Fig. 10b is presented purely as a post-hoc diagnostic and is not used in any reported metric, ensuring strict separation between model selection and evaluation protocols. Threshold analysis of ResSwinNet: precision, recall, and F1 across varying decision thresholds on the IAPRTC-12 dataset. Complementing these quantitative findings, Fig. 11 illustrates sample predictions at different precision levels. At Prec@6 (Fig. 11a), the model successfully identifies the most salient and high-confidence labels with minimal noise. At Prec@3 (Fig. 11b), additional contextual elements are captured, enhancing the descriptive richness of annotations. Finally, at Prec@5 (Fig. 11c), the model predicts even finer scene-level attributes, with only minor noise introduced. Together, these results confirm that ResSwin Net not only outperforms existing architectures quantitatively but also provides semantically coherent and context-aware annotations, making it well-suited for multi-label image annotation tasks. Test sample predicted images on IAPRTC-12 dataset on version 1: (a) Precision@6, (b) Precision@3, and (c) Precision@5. While Fig. 12 illustrate the top 4 predicted or annotated labels when implemented on the Version 2 prune strategy illustrated in Fig. 9 compares the performance of baseline models and the proposed ReSwinNet (Version 2) on the IAPRTC-12 dataset using macro-level evaluation metrics. The results show that ReSwinNet significantly outperforms the baseline models, achieving the highest macro precision, recall, F1-score, and mAP, while also maintaining a lower Hamming Loss, demonstrating its effectiveness for multi-label image annotation. Test sample predicted images on IAPRTC-12 dataset on version 2. Table 9Comparison of baseline models using macro-level evaluation metrics on the IAPRTC-12 dataset under the Version 2 label setting.ModelPRF1mAPThresholdH LossResNet-500.30230.36070.31420.34980.500.0164Swin-Tiny0.27330.43050.30880.41550.500.0189CNN–Swin Hybrid0.38550.41950.36680.31470.60–ReSwinNet (version 2)0.61380.57950.59610.62000.700.0123 Comparison of baseline models using macro-level evaluation metrics on the IAPRTC-12 dataset under the Version 2 label setting. The top 10 classes (e.g., butterfly, camel, coral) achieve perfect discrimination with , suggesting these are visually distinctive categories with sufficient training support, enabling the model to rank all true positives above all negatives without error shown in Fig. 13. In contrast, the bottom 10 classes span , with semantically ambiguous or visually overlapping categories such as semaphore (0.617), man-made-other (0.662), and ground-vehicles (0.721) exhibiting the weakest separability, likely due to label co-occurrence, low positive frequency, or insufficient visual distinctiveness relative to the fused ResNet–Swin feature space . ROC curves for the top 10 and bottom 10 classes ranked by AUC, illustrating the per-class discriminative capacity of the ReSwinNet fusion model on the IAPRTC-12. The model demonstrates strong ranking capability, achieving a Ranking Average Precision of 0.7119, indicating that relevant labels are consistently ranked above irrelevant ones across the 198-class label space. The Ranking Loss of 0.0203 confirms that label order inversions are rare shown in Fig. 14. The Coverage Error of 15.62 out of 198 total labels further validates that all ground-truth labels are recoverable within the top of the ranked list, reflecting well-calibrated and reliable multi-label predictions despite the large and fine-grained label space. Ranking metrics evaluated on the test set with 198 labels), reporting mean Average Precision (mAP), Ranking Average Precision, Ranking Loss, and Coverage Error. Ablation study Table 6 summarizes the architectural designs and fusion strategies of the five model variants considered in this study. Shallow and naively fused hybrids underperformed due to inadequate depth and feature misalignment, while the ResNet-only baseline proved competitive through multi-scale spatial fusion alone. The proposed ReSwinNet (49.5M) achieves the best design by projecting ResNet spatial features directly into Swin-compatible tokens, bypassing patch embedding to retain multi-scale context within the global attention framework. All variants shared a unified training configuration (Table 7): AdamW (), CosineAnnealingLR, early stopping (patience), and asymmetric/focal loss functions to handle severe label imbalance across both datasets. Table 6Ablation study highlighting architectural designs and fusion strategies of different model variants.Model variantParams (M)#Conv layersFilter/poolingClassifier headFusion strategyCNN–Swin Hybrid28.5433 conv, MaxPoolSwin FCCNN features reshaped to Swin tokensResNet 5023.710 (conv1–layer3)11 conv, Bilinear UpsampleAdaptiveAvgPool + FC (128C)Multi-scale conv feature fusionResNet–50 + Swin (Naive)49.35077 + 33 conv, AvgPoolSwin FCDirect concatenation, no alignment/balancingResNet–50 + Swin (balanced)49.35077 + 33 conv, AvgPoolSwin FCLabel & image balancing + feature projection ReSwinNet49.55077 conv, AvgPool, Linear ProjectionSwin FCResNet outputs projected to Swin-compatible tokens, patch embedding bypassed Ablation study highlighting architectural designs and fusion strategies of different model variants. Table 7Training configuration. Hyperparameters used across all model variants in the ablation study for both Corel-5k and IAPRTC-12 datasets.SettingValueEpochs100 (early stopping, patience = 7)OptimizerAdamW (lr = 110-4, weight_decay = 110-4)SchedulerCosineAnnealingLR ( = 100)Batch size16 (Corel-5k) 32 (IAPRTC-12)Train/Val splitMultilabel stratified (6:1 ratio)Gradient clippingmax_norm = 1.0Loss functionBCEwithLogitsLoss: pos_weight capped at [1.0, 10.0]AsymmetricLoss (ASL): =4, =0, clip=0.05, smoothing=0.1, class_weights appliedFocal BCE: =2.0, label smoothing=0.05, pos_weight capped at 10.0Image size224 224AugmentationRandom horizontal flip, rotation (±10), colour jitter Training configuration. Hyperparameters used across all model variants in the ablation study for both Corel-5k and IAPRTC-12 datasets. We compared the proposed ReSwinNet model with five architectural variants under the pruned-label leakage-free setting (Version 2 strategy), as presented in Table 8. In addition, baseline models such as ResNet-50 were evaluated to analyse the effectiveness of the proposed architecture. The model leverages multi-scale intermediate features extracted from ResNet, which are projected using convolution layers and subsequently fused prior to classification. Despite having relatively fewer parameters, this approach demonstrated strong performance, highlighting the effectiveness of spatial feature fusion in improving representation learning. ReSwinNet achieves the highest Macro Precision (0.734) and Macro F1 (0.701), confirming the effectiveness of the proposed ResNet-Swin token projection fusion. While Swin-Tiny achieves higher recall (0.781), it does so at the cost of substantially lower precision (0.623), collapsing to the minimum threshold of 0.10 and reflecting a tendency to over-predict label. Table 8Architectural ablation on the Corel-5k dataset (pruned 65-label leakage-free setting). All P, R, F1, and mAP values are macro-averaged. Best results per column in bold.ModelPRF1mAPThresholdHlossResNet-340.6990.6370.6500.6290.100.0240ResNet-500.6440.6830.6510.6950.350.0261ResNet-1010.5690.6060.6480.6510.100.0224Swin-Tiny0.6230.7810.6840.7500.100.0274CNN-Swin Hybrid0.6970.7090.6920.7040.250.0220ReSwinNet (Version 2)0.7340.7120.7010.7420.500.0235 Architectural ablation on the Corel-5k dataset (pruned 65-label leakage-free setting). All P, R, F1, and mAP values are macro-averaged. Best results per column in bold. It is observed that the proposed model converges to the highest optimal threshold (0.50), indicating well-calibrated prediction confidence where the model assigns meaningfully higher probabilities to true positive labels relative to negatives. Swin-Tiny marginally outperforms ResNet-50 in recall and mAP, yet both struggle with macro precision below 0.40, reflecting the difficulty of the large label space. The CNN–Swin Hybrid offers modest gains in precision and recall but fails to improve mAP, suggesting that naive feature fusion without alignment is insufficient. ReSwinNet (v2) decisively outperforms all baselines, achieving an F1 of 0.5961 and mAP of 0.6200 nearly double that of ResNet-50 while maintaining the lowest Hamming Loss (0.0123), confirming that token-projected multi-scale fusion is the most effective strategy for large-scale multi-label image annotation. Further , a consistent dataset-dependent pattern in loss function performance is shown in Table 10. On the full label settings (374 labels for Corel-5k and 291 labels for IAPRTC-12), BCE with pos_weight outperforms ASL across most metrics, as explicit frequency-based weighting better handles the severe long-tail label distribution. Conversely, ASL dominates on the pruned Corel-5k setting (65 labels), where the more balanced label distribution allows ASL’s asymmetric negative down-weighting to operate more effectively. On the IAPRTC-12 198-label setting, ASL again outperforms BCE, particularly in precision (0.614 vs. 0.376), suggesting that moderate pruning still retains sufficient label balance to favour asymmetric loss formulations. Table 10Loss function comparison on ReSwinNet across datasets and label settings. All P, R, F1, and mAP values are macro-averaged.DatasetLabelsASLBCEPRF1mAPPRF1mAPCorel-5k3740.4780.4990.4550.4520.4770.6140.4910.478650.7340.7120.7010.7420.6830.7180.6830.669IAPRTC-122910.4760.4900.4630.4520.4680.5030.4710.5171980.61380.57950.59610.62000.37130.57560.44010.5165 Loss function comparison on ReSwinNet across datasets and label settings. All P, R, F1, and mAP values are macro-averaged. Table 8 presents the quantitative comparison across all architectural variants under identical experimental conditions. ReSwinNet achieves the highest Macro Precision (0.734) and Macro F1 (0.701) among all compared variants. While Swin-Tiny achieves higher recall (0.781) and mAP (0.750), it does so at the cost of substantially lower precision (0.623), reflecting a tendency to over-predict labels at its collapsed threshold of 0.10. ReSwinNet’s higher optimal threshold (0.5) indicates greater prediction confidence and more calibrated output probabilities, achieving a superior precision-recall balance that is particularly desirable in annotation quality-sensitive settings. The consistent improvement over all pure CNN variants confirms that the ResNet-Swin token projection fusion contributes meaningfully beyond what either backbone achieves independently. For the Corel-5k dataset, Table 11 analyses the impact of label frequency on model performance across the 65 retained labels. Mean F1 remains stable across high, medium, and low frequency tiers (0.701, 0.703, and 0.700 respectively), indicating that ResSwinNet does not disproportionately underperform on rare labels. A precision-recall trade-off is observed as support decreases, with precision increasing from 0.664 to 0.750 and recall declining from 0.756 to 0.707. Mean AP shows the clearest degradation across tiers (0.696 0.670), reflecting uncertainty in probability ranking under sparse training evidence. Notably, visually distinctive low-support labels such as fox and coral achieve F1 0.947, whereas visually ambiguous labels such as branch and horizon exhibit F1 0.29, confirming that visual ambiguity rather than sparsity alone drives performance degradation on rare labels. Table 11Impact of label frequency on model performance on the Corel-5K dataset (65-label leakage-free setting).Frequency tierNo. labelsSupport RangeMean PMean RMean F1Mean APHigh (50 samples)651–1160.6640.7560.7010.696Medium (15–49 samples)2015–380.7250.7080.7030.700Low (<15 samples)392–130.7500.7070.7000.670 Impact of label frequency on model performance on the Corel-5K dataset (65-label leakage-free setting). Comparative analysis with existing methods The proposed ReSwinNet was experimented on the Corel5K dataset under the standard full label space of 374 labels and compared against some established multi-label image annotation methods as presented in Table 12. All methods are evaluated using macro-averaged precision, recall, and F1 score to ensure consistency with prior work. Among the compared methods, CNN-THOP39 achieves the highest F1 score of 0.553, followed by CNN-MCC28 with F1=0.520 and SSL-AWF40 with F1=0.495. Earlier methods such as CNN-AT11 and AHL41 report comparatively lower performance, with F1 scores of 0.343 and 0.340 respectively, reflecting the substantial progress made by subsequent architectures. The proposed ReSwinNet achieves a macro F1 of 0.491, surpassing some of the compared methods and positioning competitively within the mid-to-upper tier of the benchmark. Notably, ReSwinNet achieves the highest recall among all compared methods at 0.614, exceeding even the top-performing CNN-THOP39 (R=0.583) by a margin of 0.031. Table 12Comparison with existing methods on the Corel5K dataset under the standard full label space (374 labels). All results use macro-averaging.ModelMacro PMacro RMacro F1CNN-AT110.3170.3730.343AHL410.3100.3800.340CCN-R420.3200.4130.372Nearest Neighbor CNN420.4100.3700.390DMCKN270.4090.4750.439CNN-2L430.4200.3800.440PRM Deep190.4530.5170.483CNN-MCC280.4840.5630.520SSL-AWF400.5100.4800.495CNN-THOP390.5270.5830.553ReSwinNet (proposed)0.4770.6140.491Significant values are in bold. Comparison with existing methods on the Corel5K dataset under the standard full label space (374 labels). All results use macro-averaging. Significant values are in bold. This elevated recall reflects the effectiveness of the frequency-weighted BCE loss, which assigns higher penalty to missed predictions on less frequent labels, encouraging the model to recover a broader range of true positive annotations across the full 374-label space. While the proposed model does not surpass the top three methods in terms of F1, it achieves this without task-specific attention mechanisms, graph-based label dependencies, or specialised loss formulations beyond standard weighted binary cross-entropy. The competitive performance under the full label setting confirms that the proposed ResNet-Swin fusion architecture generalises effectively to the standard benchmark, and that the results reported under the pruned label setting in Table 13 reflect genuine architectural capability rather than an using label space reduction. Table 13Performance comparison of the proposed ResWin-Net (Pruned) against state-of-the-art models on the Corel5K dataset under the version 2 65-label leakage-free pruned setting. Best results per column are highlighted in bold.ModelMacroWeightedmAPPRF1PRF1CNN-AT0.66200.68380.65720.69260.68010.67720.7166SSL-AWL0.51320.53660.50430.53770.60030.55100.5623DMCKN0.03901.00000.07190.09071.00000.15740.1479CNN-MCC0.63950.74050.67720.67370.75830.70780.7332CNN-THOP0.63270.71140.65970.67080.72590.68840.7189ResSwin-Net (Version 2)0.73410.71180.70080.72740.74010.71830.7415 Performance comparison of the proposed ResWin-Net (Pruned) against state-of-the-art models on the Corel5K dataset under the version 2 65-label leakage-free pruned setting. Best results per column are highlighted in bold. When the proposed model was trained using the Version 2 label pruning strategy , ReSwinNet achieves the best performance across all evaluation metrics, obtaining a macro-F1 of 0.7008, weighted-F1 of 0.7183, and mAP of 0.7415. These results demonstrate the effectiveness of the proposed pruning strategy in preserving discriminative capacity while reducing model complexity. Among the convolutional baselines, CNN-AT11 ranks second with an mAP of 0.7166 and macro-F1 of 0.6572, while CNN-MCC28 achieves the highest recall (macro: 0.7405; weighted: 0.7583) at the expense of lower precision. CNN-THOP39 provides a more balanced but slightly lower performance (macro-F1 = 0.6597; mAP = 0.7189). Compared with the strongest CNN baseline (CNN-MCC28), our proposed ReSwinNet improves macro-F1 by 2.36%, weighted-F1 by 1.05%, and mAP by 0.83%, highlighting the advantage of the hybrid residual–window attention architecture. In contrast, DMCKN27 exhibited degenerate behaviour under the pruned label setting, achieving R=1.000 with P=0.039 (F1=0.072), consistent with a model predicting all labels positive for every image. This likely reflects a threshold calibration failure when the evaluation label space is reduced from 374 labels. While SSL-AWL40 delivers moderate performance (macro-F1 = 0.5043; mAP = 0.5623), outperforming only DMCKN27. Overall, ResWinNet (Version 2) achieves the most balanced precision–recall trade-off (macro-P = 0.7341, macro-R = 0.7118), making it the most reliable model for multi-label image annotation under the high-frequency label setting of Corel5K. Fig. 15 shows the optimal classification threshold per model. ResWin-Net (Pruned) achieves the highest threshold of 0.50, indicating well-calibrated posterior probabilities, while baseline models require aggressive threshold reduction (0.10–0.40)11,27,28,39,40 to achieve acceptable performance. Optimal classification threshold for each model on the Corel5K dataset. The comparison presented in Table 14 highlights ReSwinNet against existing methods on the IAPRTC-12 dataset under the full 291-label space. Early approaches such as SEM44 and CCA-KNN42 show limited performance (F1 0.40), constrained by shallow feature representations. Subsequent models including SSL-AWF40 (F1 0.475) and CNN-MCC28 (F1 0.537) improved through deeper architectures, yet consistently struggled to balance precision and recall. Matrix factorisation-based methods such as MVG-NMF18 and MvNMF-DK45 remain below 0.50 in F1, reflecting their limited capacity for high-dimensional visual feature modelling. The proposed ReSwinNet achieves a precision of 0.468, recall of 0.503, and F1 of 0.471, offering the strongest recall among all compared methods. While CNN-MCC28 reports a higher F1 (0.537), it does so at the cost of lower recall (0.515), highlighting a precision-recall trade-off that ReSwinNet better balances across the large and imbalanced label space. Table 14Comparison with existing methods on the IAPRTC-12 under the standard full label space.MethodPrecisionRecallF1 ScoreSEM440.41000.39000.4000SSL-AWF400.53000.43000.4750CNN-MCC280.56200.51500.5370MVG-NMF180.48800.43600.3950PRM Deep190.49200.44300.4020MvNMF-DK450.49000.38500.4330MvNMF-SK450.41500.29200.3430CCA-KNN420.45000.38000.4000AHL410.45830.38800.3369ReSwinNet (proposed)0.46800.50300.4710 Comparison with existing methods on the IAPRTC-12 under the standard full label space. Excluding the degenerate DMCKN baseline (recall 1.0, F1 0.057), prior methods plateau around a macro F1 of 0.50–0.52, with CNN-MCC achieving the best among baselines (macro F1 0.5197, mAP 0.5146) shown in Table15. ReSwinNet surpasses all competitors across macro precision, F1, and mAP, achieving 0.6138, 0.5961, and 0.6293 respectively an improvement of over 14 points in mAP relative to the strongest baseline. The weighted recall of 0.5795 remains the only metric where CNN-MCC’s higher recall (0.6473) reflects a precision–recall trade-off, suggesting that ReSwinNet prioritises accurate label prediction over exhaustive retrieval, which is desirable in large and imbalanced label spaces Table 15Performance comparison of the proposed ReSwinNet against state-of-the-art models on the IAPRTC-12 dataset under the pruned label setting (198 labels). Best results per column are highlighted in bold.ModelMacroWeightedmAPPRF1PRF1CNN-AT0.50910.51200.50350.57860.62250.59640.5074PRM-Deep0.52250.49030.49720.58360.59980.58790.4977SSL-AWF0.48720.52100.49680.56310.63140.59180.4985DMCKN0.03101.00000.05700.09241.00000.16080.0517CNN-MCC0.50030.55850.51970.56930.64730.60260.5146CNN-THOP0.50870.51840.50530.57760.62130.59490.5119ReSwinNet (Proposed)0.61380.57950.59610.62030.57950.59030.6293 Performance comparison of the proposed ReSwinNet against state-of-the-art models on the IAPRTC-12 dataset under the pruned label setting (198 labels). Best results per column are highlighted in bold. Conclusion In this paper, we proposed ReSwinNet, a hybrid deep learning architecture that integrates convolutional and transformer-based mechanisms for multi-label image annotation. By combining ResNet-50 for local spatial feature extraction with the Swin Transformer for capturing global contextual relationships, the model effectively bridges the strengths of both paradigms. A key contribution is the token projection mechanism that maps CNN features to transformer-compatible inputs for seamless fusion. Additionally, a label pruning strategy is employed as a pre-processing step to eliminate infrequent and noisy labels, thereby reducing annotation ambiguity and enhancing training stability. This not only improves model convergence but also boosts overall predictive performance by focusing on more semantically meaningful and frequent label categories. Comprehensive experiments on the Corel-5K and IAPRTC-12 benchmarks demonstrate that ReSwinNet consistently outperforms existing state-of-the-art approaches across all evaluation metrics. Under the full 374-label space, ReSwinNet achieves a macro F1 of 0.491, surpassing 7 of 10 compared methods and recording the highest recall among all compared models. Under the leakage-free 65-label pruned setting, ReSwinNet achieves a macro F1 of 0.701 and mAP of 0.742, outperforming all compared methods under identical experimental conditions. On IAPRTC-12, ReSwinNet achieves over 14% higher mAP relative to the strongest baseline under the pruned 198-label setting, highlighting its robustness across datasets of varying scale and label complexity. The results affirm that integrating CNNs with hierarchical self-attention offers a powerful solution for complex multi-label image annotation tasks, particularly under frequency-filtered label settings where annotation noise is reduced and discriminative learning is more tractable. Future work will focus on hierarchical label taxonomy modelling to better capture semantic dependencies among co-occurring annotations, alongside exploring self-supervised pretraining for low-resource and open-vocabulary annotation settings. Author contributions Chitrapriya conceived the research idea, developed the theoretical framework, performed the computational analysis, and drafted the manuscript. Dr. Tejbanta validated the analytical methods and supervised the study. Dr. Dhruba Ningombam has provided the final revisions of the manuscripts .All authors contributed to the interpretation of results and approved the final version of the manuscript. Funding No funding was received for conducting this study. Data availability The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.