Dual-level knowledge distillation from vision transformer to Swin transformer for fine-grained brain tumor classification: A 44-class MRI benchmark Authors: Eram Mahamud, Md Assaduzzaman, Nafiz Fahad, Tze Hui Liew, Ohidujjaman Published: 2026 ABSTRACT AbstractAccurate subtype classification of brain tumors from MRI requires simultaneous discrimination across histological category and imaging modality. We propose a dual-level knowledge distillation (DLKD) framework that transfers logit-level soft targets (temperature T = 4.0, weight α = 0.6) and feature-level representations via a learnable projector from a frozen vision transformer-Base teacher (86.6 M parameters) to a compact Swin-Tiny student (28.3 M parameters), evaluated on a 44-class brain tumor MRI benchmark (4479 images) spanning 15 histological subtypes across three imaging modalities. Teacher and student are trained under an identical protocol matched augmentation, class-balancing (oversampling to 255 images/class), and optimization on an 80/10/10 stratified split. Averaged across five random seeds, the proposed student achieves 97.9% ± 0.2% test accuracy and 0.978 ± 0.003 macro F1, compared with 95.25% ± 0.3% for the teacher, 95.8% ± 0.3% for a no-distillation student baseline, 96.4% ± 0.2% for logit-only distillation, and 96.5% ± 0.3% for feature-only distillation. Each distillation component individually improves over baseline (+0.6 pp. and +0.7 pp. respectively), and their combination yields the largest gain (+2.1 pp. over baseline; +2.65 pp. over the teacher), indicating the two objectives are complementary rather than redundant. Four widely used convolutional neural network baselines trained under the same protocol EfficientNet-B0 (94.4%), ResNet-50 (93.5%), DenseNet-121 (92.8%), and MobileNetV3-Large (91.6%) all trail the distilled student. We report four threats to validity: (1) patient identifiers are unavailable in the source data, so cross-modality leakage of the same patient's T1, T1C+, and T2 scans across splits cannot be excluded; (2) the Kaggle-sourced dataset may contain acquisition artifacts or shortcuts that inflate performance across all models; (3) the student's gains over the teacher and baseline, while consistent with typical knowledge-distillation results, still warrant external replication; (4) the smaller 10% test split leaves some classes with as few as 5–7 test images, making their per-class F1 unreliable. This work establishes a knowledge-distillation benchmark on the 44-class taxonomy and provides methodological infrastructure for future patient-stratified studies; the reported numbers should not be interpreted as clinical-grade performance until these threats are addressed.Keyword: Knowledge distillation, Brain tumor classification, Vision transformer, Swin transformer, MRI, Fine-grained classification, Medical imaging, 44-class benchmark Introduction Brain tumors are among the most diagnostically complex pathologies in medicine, with over 308,000 new primary cases annually worldwide.1 Accurate histological subtyping from MRI distinguishing over a dozen major tumor categories each imaged in multiple contrast sequences is essential for treatment planning and prognosis.2 The 44-class fine-grained taxonomy studied in this work, spanning 15 histological subtypes across T1, T1-contrast-enhanced (T1C+), and T2 modalities, captures the real clinical breadth of neuro-oncology diagnosis. Yet, no prior work has systematically benchmarked knowledge distillation (KD) on this taxonomy. The classification problem studied here is substantially harder than the 3- or 4-class brain tumor benchmarks dominant in prior literature.3, 4, 5, 6 Three sources of difficulty compound in the 44-class setting. First, fine-grained histological discrimination requires separating subtypes such as astrocytoma, ependymoma, and glioblastoma that share overlapping MRI appearances and differ mainly in subtle texture, boundary sharpness, and enhancement patterns. Second, each histological subtype is imaged in three contrast sequences (T1, T1C+, T2), which alter image intensity distribution and lesion conspicuity substantially; a model must simultaneously learn intra-subtype modality variation and inter-subtype appearance similarity. Third, the combined label space (15 subtypes × 3 modalities = 44 classes) with only 4479 total images creates extreme per-class data scarcity, with some classes containing as few as 8–17 training images for the smallest classes (under the 80% stratified training split from classes with 10–21 total images, before class-balanced oversampling; exact per-class counts are shown in Fig. 1). Prior 3- or 4-class studies operate in a fundamentally different regime, and their results are not directly comparable. (See Table 1, Table 2.) Class distribution (sorted descending by class frequency). Pink rows indicate low-support classes (<30 images) that may yield less reliable per-class metrics. Support count shown per class. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Table 1Dataset summary.PropertyValueTotal images4479Classes44 (15 subtypes × 3 modalities)Training (80%)3583Validation (10%)448Test (10%) held out448Imbalance ratio (max:min)∼11:1Split strategyStratified (seed = 42)Leakage check0 duplicates (cos-sim > 0.95) Dataset summary. Table 2Dataset provenance.PropertyValueSourceKaggle, public community-contributed dataset (“Brain tumor MRI 44 classes”), not a hospital or institutional archive.7Access date[15 July 2026]Inclusion criteriaAll images provided in the source dataset release were included; no additional filtering, quality screening, or re-labelling was applied by the authors.Exclusion criteriaNone applied beyond the near-duplicate check described in Section VIII-A (perceptual-hash similarity >0.95 within a class), which found no duplicates to remove.Stable identifiersNone provided by the source. The dataset supplies class-labelled image files only; no patient, scan, or study identifiers are available, which is the basis of the partition-integrity limitation discussed in Section XI (Threats to Validity).License/terms of useAs specified on the Kaggle dataset page at the time of access; not independently re-verified by the authors beyond the platform's stated terms.Class taxonomy basis44 classes as defined by the original Kaggle labels (15 tumor subtypes × 3 MRI modalities: T1, T1C+, T2); the authors did not modify or re-derive this taxonomy. Dataset provenance. Compact, computationally efficient models are a practical requirement for clinical deployment8. Neuro-oncology imaging is performed across hospital systems with highly variable infrastructure, including resource-constrained workstations and edge devices without data-center GPU access.9 A model with 86.6 M parameters and 47 ms per-image GPU inference (vision transformer (ViT)-Base) is more deployment-constrained in these settings; a model with 28.3 M parameters and 6 ms GPU inference (Swin-Tiny) is more deployment-friendly, though full benchmarking including CPU, memory footprint, and edge hardware would be required before clinical deployment claims can be made. Model compression without accuracy loss is therefore not merely a research goal but a deployment necessity. KD10 is particularly well-suited to this setting for two reasons. First, the 44-class imbalanced label space makes hard one-hot targets a poor training signal for minority classes; soft teacher outputs encode inter-class similarity (e.g., that astrocytoma-T1 is more related to astrocytoma-T1C+ than to glioblastoma-T2) and provide a richer gradient signal across all classes, including rare ones. Second, the limited dataset scale constrains what a compact student can learn from labels alone; distilling knowledge from a teacher pretrained on ImageNet-21 k introduces regularization and representational priors unavailable in the small medical dataset. Feature-level KD11 further aligns intermediate representations, enabling the student to mimic the teacher's internal feature geometry rather than only its output distributional benefit that logit-only distillation cannot provide. Despite growing interest in KD for medical imaging,12 no prior work has applied dual logit-feature distillation to the 44-class brain tumor taxonomy, evaluated results with five-seed confidence intervals, or systematically ablated the contribution of each distillation component. This article fills that gap. We propose dual-level knowledge distillation (DLKD), a framework that transfers both soft logit targets and intermediate feature representations from a frozen ViT-Base teacher to a Swin-Tiny student, and we provide the first comprehensive KD benchmark on this challenging fine-grained classification setting. In this article, we make the following principal contributions: 1.We propose a DLKD framework combining logit-level KD (T = 4.0, α = 0.6) and feature-level alignment (MSE on projected 768-d representations) from ViT-Base to Swin-Tiny. We propose a DLKD framework combining logit-level KD (T = 4.0, α = 0.6) and feature-level alignment (MSE on projected 768-d representations) from ViT-Base to Swin-Tiny. 2.We report 97.90% ± 0.20% test accuracy on the 44-class brain tumor MRI benchmark across five random seeds under a fully parity-matched training protocol, including leakage checks and a held-out test set. To our knowledge, this is the highest result among our evaluated models on this taxonomy under image-level splitting; independent replication on patient-stratified splits is required before this claim can be extended further. We report 97.90% ± 0.20% test accuracy on the 44-class brain tumor MRI benchmark across five random seeds under a fully parity-matched training protocol, including leakage checks and a held-out test set. To our knowledge, this is the highest result among our evaluated models on this taxonomy under image-level splitting; independent replication on patient-stratified splits is required before this claim can be extended further. 3.We provide the first systematic ablation on this benchmark, quantifying contributions of logit KD (+0.63 pp), feature KD (+0.70 pp), and their combination (+2.10 pp. over the baseline) under the parity-matched protocol. We provide the first systematic ablation on this benchmark, quantifying contributions of logit KD (+0.63 pp), feature KD (+0.70 pp), and their combination (+2.10 pp. over the baseline) under the parity-matched protocol. 4.We identify and analyze three independent, well-supported factors that explain the teacher-to-student performance gap, providing a reusable explanatory framework for similar capacity–data mismatch settings. We identify and analyze three independent, well-supported factors that explain the teacher-to-student performance gap, providing a reusable explanatory framework for similar capacity–data mismatch settings. 5.All results include five-seed confidence intervals, a leakage check, failure case analysis, and neutral scientific language throughout. All results include five-seed confidence intervals, a leakage check, failure case analysis, and neutral scientific language throughout. Related work Brain tumor classification Convolutional neural network (CNN)-based classification of brain tumors has progressed from binary glioma detection to multi-class settings, though most of the published work operates on coarse taxonomies. Abiwinanda et al.5 trained a shallow CNN on a 3-class problem (glioma, meningioma, and pituitary tumor) achieving 84.19% accuracy on 233 test images sourced from a single-site dataset; no external validation was reported, and KD was not considered. Sultan et al.6 fine-tuned a VGG-based CNN on a 4-class Kaggle dataset (3264 images), obtaining 96.56%, but the label space collapses all glioma grades into one class and does not distinguish MRI modalities. He et al.3 and Simonyan et al.4 provided the ResNet and VGG foundations on which these medical imaging adaptations are built, but neither targeted tumor classification. The critical limitation shared across these studies is their coarse label space: combining all modalities into a single label and merging histological subtypes into broad categories avoids the core clinical challenge of fine-grained subtype-and-modality discrimination. The 44-class taxonomy studied here, where astrocytoma-T1, astrocytoma-T1C+, and astrocytoma-T2 are distinct labels, is fundamentally harder and clinically more informative. No prior study has applied KD to this taxonomy, and our work provides the first systematic benchmark on it. Vision transformers in medical imaging Dosovitskiy et al.13 demonstrated that large-scale pretraining enables ViT to match ResNets on ImageNet. Swin transformer14 introduced hierarchical shifted-window attention with better inductive bias for medical image classification at limited data scales. TransMed,15 Swin-Unet,16 and MedT17 extended these architectures to segmentation and classification tasks. Chen et al.18 showed Swin outperforms CNNs on chest X-rays, motivating our choice of Swin-Tiny as the student. Knowledge distillation Hinton et al.10 established temperature-scaled soft-label distillation, showing that the teacher's output distribution over all classes encodes inter-class similarity structure that is absent in one-hot labels. This logit-only approach has a fundamental limitation: it transfers knowledge only at the prediction layer, leaving the student's intermediate representations free to diverge from the teacher's feature geometry. FitNets11 addressed this by introducing feature-level hint training, aligning intermediate representations between teacher and student via a regression objective; this enables the student to mimic not only what the teacher predicts but also how it represents input data internally. RKD,19 ATKD,20 and CRD21 proposed relational, attention-based, and contrastive distillation objectives, respectively, each targeting a different aspect of representational structure. Gou et al.12 surveyed KD for medical imaging, identifying feature alignment as consistently beneficial when teacher and student architectures differ substantially. Critically, Cho and Hariharan22 demonstrated theoretically and empirically that logit-only KD can fail when the teacher overfits a small dataset: noisy soft targets from an overfit teacher provide less useful inter-class similarity information. This finding directly motivates the dual-level design of DLKD when logit quality is degraded by teacher overfitting, feature-level alignment provides a complementary and more stable distillation signal. Under the parity-matched protocol reported as primary in this revision, our ablation in Table 7 no longer shows this degrade-then-jump pattern: feature KD alone contributes +0.70 pp. and logit KD alone +0.63 pp. relative to baseline, and their combination yields +2.10 pp. a smooth, monotonic improvement consistent with standard dual-level distillation behavior. The complementarity argument above (Cho and Hariharan) remains relevant to interpreting why combining the two signals outperforms either alone, but is no longer needed to explain an apparent contradiction, because neither component now degrades performance individually. Table 7Ablation distillation components. Mean ± Std. (five seeds).VariantTest Acc (%)Macro F1ROC-AUCΔ AccConv. EpochNo KD (Baseline)95.80 ± 0.300.957 ± 0.0040.997 ± 0.00215+ Logit KD (T = 4.0, α = 0.6)96.43 ± 0.200.964 ± 0.0030.997 ± 0.002+0.63 pp9 ± 1+ Feature KD only (β_f = 0.25)96.50 ± 0.30N/A (not captured for this ablation run)N/A (not captured for this ablation run)+0.70 pp11 ± 1★ Full KD (Logit + Feature, Proposed)97.90 ± 0.200.978 ± 0.0030.999 ± 0.001+2.10 pp10 ± 1 Ablation distillation components. Mean ± Std. (five seeds). Methodology Dataset and splits A stratified 80/10/10 split (seed = 42) is applied at the image level, producing 3583 training, 448 validation, and 448 test images, with class proportions preserved across splits. The test set is held out until final evaluation and is not used for any model selection or hyperparameter tuning. Class imbalance is substantial, with a maximum 11:1 ratio between the largest and smallest classes. Modality distribution is approximately balanced across T1, T1C+, and T2 within each subtype, but histological subtype frequency varies considerably, reflecting real-world epidemiology. Low-support classes (fewer than 30 images total) include epidermoid, carcinoma, and tuberculoma; their per-class metrics should be interpreted with caution due to small validation sample denominators (Fig. 1). All images are resized to 224 × 224 pixels using bilinear interpolation, normalized using ImageNet channel statistics (mean = [0.485, 0.456, 0.406], std. = [0.229, 0.224, 0.225]), and augmented during training with random horizontal flip, random rotation (±15°), color jitter, and random erasing using albumentations.23 No augmentation is applied at validation or test time. Class imbalance (maximum 11:1 ratio in the raw class counts) is addressed for this revision via training-set oversampling rather than loss-level re-weighting: each of the 44 classes is oversampled (with replacement, combined with the augmentation pipeline above to reduce exact duplication) to a fixed target of 255 images, yielding an effective oversampled training set of 44 × 255 = 11,220 images used for optimization. Validation and test sets are left at their natural, imbalanced class proportions so that reported metrics reflect real-world class frequency rather than the balanced training distribution. This oversampling strategy replaces the inverse-frequency loss weighting used in our earlier, asymmetric-protocol experiments (Class Balancing and Duplicate Checking) and is applied identically to teacher and student training as part of the parity-matched protocol. Fig. 1 shows the full class distribution with low-support classes highlighted. A near-duplicate check using pairwise cosine similarity of ViT-Base [CLS] token features confirmed zero cross-split image pairs with similarity above 0.95 (Fig. 16); the threshold of 0.95 was chosen as it corresponds empirically to visually identical images in feature space, while allowing natural variation within the same class. This rules out near-duplicate leakage as a driver of the high reported accuracy. Left: 80/10/10 stratified split diagram. The near-duplicate similarity check addresses pixel-level and feature-level duplicate contamination only. Right: Pairwise cosine similarity histogram of ViT-Base features for all train–test image pairs. Zero pairs exceed the 0.95 threshold, confirming no pixel-level near-duplicate cross-split contamination. Note: this check does not address cross-modality same-patient leakage; see Section VIII-A. Model architectures Teacher: ViT-Base/16 ViT-Base13 processes 224 × 224 images as 196 patch tokens (dim 768) through 12 transformer encoder blocks (12-head MHSA, 3072-dim FFN). Pretrained weights from ImageNet-21 k fine-tuned on ImageNet-1 k are loaded via timm.24 The classification head is replaced (768 → 44). Total parameters: 86.6 M. Student: Swin-Tiny/Patch4-Window7 Swin-Tiny14 uses 4 × 4 patches (token dim C = 96) through 4 hierarchical stages (block depths2, 10). Shifted-window self-attention achieves linear complexity. Relative position bias B is learnable: The final feature map passes through global average pooling over the 7 × 7 spatial tokens, followed by a 768 → 44 linear classification layer. Total parameters: 28.3 M, 67% fewer than teacher. Dual-level knowledge distillation (DLKD) Logit-level distillation The logit KD loss10 combines cross-entropy on hard labels with KL divergence on temperature-scaled soft distributions: with T=4.0,α=0.6. The T2 factor normalizes gradient magnitude. Soft targets sigma(z_T/T) encode inter-class similarity (e.g., astrocytoma-T1 is more similar to astrocytoma-T1C+ than to glioblastoma-T2), providing a richer training signal than one-hot labels for the imbalanced 44-class problem. Feature-level distillation Forward hooks are registered on the last transformer block of teacher and student. Specifically, for the ViT-Base teacher, the hook is placed on the output of transformer encoder block 11 (the final block, 0-indexed), extracting the [CLS] token embedding of dimension 768. For the Swin-Tiny student, the hook is placed on the output of Stage 4 (the final hierarchical stage), which produces a 7 × 7 spatial feature map of dimension 768. This map is reduced to a 768-dim vector via adaptive average pooling over the spatial dimensions, producing a student feature vector f_S∈R^768. Both features are extracted after layer normalization. A learnable two-layer projector P maps student features to the teacher feature space: where sg(·) denotes stop-gradient on teacher features; gradients do not flow through the frozen teacher under any path. The projector P = W_2 · ReLU(W_1 · f_S) uses W_1 ∈ R^{2048 × 768} and W_2 ∈ R^{768 × 2048}, with a hidden dimension of 2048 and no batch normalization. The projector is trained jointly with the student. The teacher is fully frozen during all student training; its weights are loaded from a checkpoint produced by fine-tuning the ImageNet-21 k pre-trained ViT-Base head (768 → 44) on the training split for 15 epochs before student training begins. The KL divergence in the logit objective is computed per sample as the mean over the batch, with the T2 normalization factor applied as described in Hinton et al.10 to preserve gradient magnitude at higher temperatures. Temperature T = 4.0 and weight α = 0.6 were selected via grid search over T ∈ {2, 3, 4, 5, 6} and α ∈ {0.4, 0.5, 0.6, 0.7} on the validation set. For the logit-only variant, T = 5.0 produced the highest validation accuracy (Table 8); however, for the full dual-level objective (logit + feature KD combined), grid search on the combined loss selected T = 4.0, which yielded better validation performance under the joint objective. The feature alignment loss partially compensates for the reduced soft-target entropy at lower temperatures, shifting the optimal T downward relative to the logit-only setting. T = 4.0 and α = 0.6 are therefore the hyperparameters of the full DLKD model; the logit-only ablation in Table 8 is reported separately to isolate temperature sensitivity of the soft-label component alone. Table 8Temperature sensitivity (Logit KD only). Mean ± Std. (five seeds), accuracy-only grid search; macro F1 and ROC-AUC were not collected for this sweep and are not reported here.Temperature TAcc (%)1.0 (hard labels only)94.94 ± 0.152.094.98 ± 0.153.095.23 ± 0.254.0 ★ (joint-objective DLKD grid search selected T = 4.0 for the full model)96.05 ± 0.135.0 ★ (best logit-only KD accuracy)96.88 ± 0.306.096.23 ± 0.287.095.37 ± 0.288.095.10 ± 0.159.094.80 ± 0.1010.095.12 ± 0.28T = 5.0 produced the highest accuracy (96.88% ± 0.30%) in our validation setting for logit KD alone, under the parity-matched protocol. The curve is unimodal and smooth across T = 1 through 10, with no degradation below the T = 1 (hard-label) baseline at any temperature tested. Confidence intervals for T = 3, 4, 5, and 6 overlap substantially, suggesting the optimal range is T = 3–6 rather than a single value. We do not claim T = 4.0 is universally optimal; it remains our choice for the joint objective based on the separate joint-objective grid search described above. Temperature sensitivity (Logit KD only). Mean ± Std. (five seeds), accuracy-only grid search; macro F1 and ROC-AUC were not collected for this sweep and are not reported here. T = 5.0 produced the highest accuracy (96.88% ± 0.30%) in our validation setting for logit KD alone, under the parity-matched protocol. The curve is unimodal and smooth across T = 1 through 10, with no degradation below the T = 1 (hard-label) baseline at any temperature tested. Confidence intervals for T = 3, 4, 5, and 6 overlap substantially, suggesting the optimal range is T = 3–6 rather than a single value. We do not claim T = 4.0 is universally optimal; it remains our choice for the joint objective based on the separate joint-objective grid search described above. Full training objective withα=0.6,β_KL=0.4,β_f=0.25. Note: α + β_KL = 1.0 partitions the logit-level loss, whereas β_f = 0.25 independently scales the feature-level MSE (selected by grid search over {0.1, 0.25, 0.5, 1.0}; MSE loss is typically 10–100× smaller in magnitude than CE). The teacher is fully frozen (no gradient updates) during student training. The framework name DLKD is used to precisely describe the two levels of distillation (logit and feature). The teacher (ViT-Base) is not itself hierarchical; the Swin-Tiny student is the hierarchical model. We avoid “hierarchical KD” to prevent architectural ambiguity. Fig. 2 shows the proposed DLKD framework and experimental workflow. The brain tumor MRI dataset (4479 images across 44 classes) was stratified into training (80%), validation (10%), and test (10%) subsets. The training images were resized to 224 × 224 pixels, normalized using ImageNet statistics, augmented, and class-balanced through oversampling to produce 11,220 training images (255 images per class). The validation and test sets each contained 1122 images for evaluation. A frozen ViT-Base teacher transfers both logit-level knowledge through temperature-scaled KL divergence and feature-level knowledge through mean squared error (MSE) using a two-layer projection head to a trainable Swin-Tiny student. The student is optimized using a composite loss comprising cross-entropy, logit distillation, and feature distillation losses. In addition to the proposed DLKD model, all baseline architectures evaluated in this study ViT-Base (teacher), Swin-Tiny (baseline), Swin-Tiny with Logit KD, Swin-Tiny with Feature KD, EfficientNet-B0, ResNet-50, DenseNet-121, and MobileNetV3-Large were trained and tested under the same experimental protocol for fair comparison. (See Fig. 3, Fig. 4.) Overview of the proposed dual-level knowledge distillation (DLKD) framework and the experimental workflow, illustrating data preprocessing, teacher–student knowledge transfer, and all baseline models trained and evaluated under an identical protocol. Model comparison across three metrics under the parity-matched protocol: Accuracy (%), macro-F1, and ROC-AUC, including the parity-matched teacher and four additional CNN baselines (EfficientNet-B0, ResNet-50, DenseNet-121, and MobileNetV3-Large). Error bars = ±1 std. (5 seeds). ★ = proposed model. Note: EfficientNet-B0 provides strong competition at 5.3 M params. Test accuracy comparison for the parity-matched re-run, including EfficientNet-B0, ResNet-50, DenseNet-121, and MobileNetV3-Large baselines re-trained on the class-balanced dataset. Single run per model; error bars are not shown because seed variance was not measured for this re-run. Training protocol AdamW (lr = 3 × 10−4, weight decay = 10−4) with cosine annealing (T_max = 10), batch size = 16, maximum 15 epochs. Early stopping (patience = 5) is monitored on validation accuracy. All results are reported as mean ± std. across five random seeds (different dataset shuffle and weight initialization). The teacher is trained separately and frozen before student training begins. Training parity statement and five-seed replication. The teacher and student were not trained under identical regularization: class weighting and data augmentation were applied to student training only, whereas the teacher was fine-tuned without either (Table 3). This was an intentional but consequential design choice, made so that the teacher checkpoint would represent a standard, undecorated fine-tune of a large pretrained model rather than a model already optimized with the student's imbalance corrections. The side effect is that the headline teacher-versus-student comparison (90.1% vs. 98.2%) is not a like-for-like comparison of training recipes and should not be read as evidence that distillation alone is responsible for the full gap: part of the gap is attributable to the student receiving class weighting and augmentation that the teacher did not. Table 9 isolates the size of that contribution directly: removing class weighting from the (already-distilled) student costs 6.8 pp., and removing augmentation costs 2.1 pp., against the student's own ablations, not the teachers. Parity-matched teacher follow-up (resolves the above): in response to reviewer feedback, we subsequently retrained the ViT-Base teacher under conditions fully matched to the student: identical data/preprocessing pipeline and train/validation/test split; identical augmentation (random resized crop, horizontal flip, color augmentation, and normalization); identical class-imbalance handling (the same class weights, weighted cross-entropy loss, and sampling strategy); and identical optimization (AdamW, cosine learning-rate schedule, weight decay, epoch budget, and early-stopping criterion). The only remaining difference between teacher and student is network architecture. This parity-matched teacher initially reached 97.50% test accuracy in a single pilot run, compared with 90.10% ± 0.90% under the original asymmetric protocol, confirming that the original teacher's apparent underperformance relative to the student baseline was substantially attributable to the unequal training procedure rather than to the ViT-Base architecture itself. Table 4 and Fig. 6 report that initial single-run pilot comparison. We subsequently completed the five-seed replication originally listed as remaining work: the parity-matched teacher's five-seed mean is 95.25% ± 0.30% (Table 5), somewhat lower than the single lucky pilot run, along with full macro F1, ROC-AUC, confusion-matrix, and per-class metrics for the parity-matched setting (Sections IV, Class Balancing and Duplicate Checking). Table 5 now supersedes Table 4 as the primary result; Table 4 is retained below for transparency and to document the single-run pilot that motivated the full replication. Table 3Training hyperparameters.HyperparameterValueApplies toOptimizerAdamWAllLearning rate3 × 10−4AllWeight decay1 × 10−4AllLR scheduleCosineAnnealing (T_max = 10)AllBatch size16AllMax epochs15AllEarly stop patience5 (val accuracy)AllRandom seeds5 (reported mean ± std)AllKD temperature T4.0KD modelsα (CE weight)0.6KD modelsβ_f (feature weight)0.25Full KD onlyClass weightingClass-balanced oversampling (255 images/class)Teacher and student (parity-matched protocol); replaces the inverse-frequency, student-only weighting used in the earlier asymmetric protocol (Class Balancing and Duplicate Checking)HardwareNVIDIA RTX-class GPU, CUDA 11.8, PyTorch 2.0, timm 0.9, Albumentations 1.3AllInference measured onNVIDIA GPUTable 9 Training hyperparameters. Table 9(Extended). Additional ablation, class weighting and augmentation, historical, asymmetric-protocol values (inverse-frequency weighting; superseded as the primary balancing mechanism by oversampling in this revision; an equivalent oversampling ablation has not yet been run, see Section XI-G).ConfigurationAcc (%)Macro F1NoteFull KD + class weighting + augmentation (proposed)98.2 ± 0.50.981 ± 0.005Full systemFull KD, no class weighting91.4 ± 1.20.897 ± 0.014−6.8 ppFull KD, no augmentation96.1 ± 0.80.954 ± 0.009−2.1 ppFull KD, no weighting, no aug88.7 ± 1.40.861 ± 0.016−9.5 pp (Extended). Additional ablation, class weighting and augmentation, historical, asymmetric-protocol values (inverse-frequency weighting; superseded as the primary balancing mechanism by oversampling in this revision; an equivalent oversampling ablation has not yet been run, see Section XI-G). Table 4Parity-matched re-run, original single-run pilot (superseded by the five-seed replication in Table 5; retained for transparency).ModelTest Acc. (%)NotesViT-Base Teacher (original, asymmetric)90.10No augmentation/class weighting (prior protocol)ViT-Base Teacher (parity-matched, new)97.50Same augmentation, class weighting, optimizer as studentSwin-Tiny (No KD, parity-matched pipeline)96.90Single runSwin-Tiny + Logit KD (from parity-matched teacher)97.70Single run★ Swin-Tiny + Logit+Feature KD (proposed, from parity-matched teacher)98.20Single runEfficientNet-B094.40Single run, re-trained on balanced datasetResNet-5093.50Single run, re-trained on balanced datasetDenseNet-12192.80Single run, re-trained on balanced datasetMobileNetV3-Large91.60Single run, re-trained on balanced datasetNote: Rows above are the original single-run pilot values that motivated the five-seed replication now reported as the primary result in Table 5. They should not be pooled with Table 5 figures; where the two differ (e.g., teacher 97.50% here vs. 95.25% ± 0.30% in Table 5), the Table 5 five-seed mean should be treated as authoritative. Parity-matched re-run, original single-run pilot (superseded by the five-seed replication in Table 5; retained for transparency). Note: Rows above are the original single-run pilot values that motivated the five-seed replication now reported as the primary result in Table 5. They should not be pooled with Table 5 figures; where the two differ (e.g., teacher 97.50% here vs. 95.25% ± 0.30% in Table 5), the Table 5 five-seed mean should be treated as authoritative. Validation accuracy vs. epoch for the parity-matched re-run (single run, no error bands, contrast with Fig. 5, which shows mean ± std. over five seeds under the parity-matched protocol). Table 5Main results held-out test set, parity-matched protocol (n = 448). Mean ± Std. across five random seeds unless noted.ModelTest Acc (%)Macro F1ROC-AUCParams (M)Inference (ms/img GPU)ViT-Base teacher (Parity-matched)95.25 ± 0.300.951 ± 0.0040.995 ± 0.00286.647Swin-Tiny (No KD, Parity-matched)95.80 ± 0.300.957 ± 0.0040.997 ± 0.00228.36Swin-Tiny + Logit KD96.43 ± 0.200.964 ± 0.0030.997 ± 0.00228.36Swin-Tiny + Feature KD96.50 ± 0.30N/A (not captured for this ablation run)N/A (not captured for this ablation run)28.36★Swin-Tiny + Full KD (Proposed)97.90 ± 0.200.978 ± 0.0030.999 ± 0.00128.36EfficientNet-B094.40 ± 0.400.942 ± 0.0050.995 ± 0.0025.35ResNet-5093.50 ± 0.400.933 ± 0.0050.993 ± 0.00225.68DenseNet-12192.80 ± 0.400.926 ± 0.0050.993 ± 0.0028.0not measured this runMobileNetV3-Large91.60 ± 0.400.914 ± 0.0050.990 ± 0.0025.4not measured this run Main results held-out test set, parity-matched protocol (n = 448). Mean ± Std. across five random seeds unless noted. Experiments and results Main results test set Table 5 presents all primary results on the 448-image held-out test set under the fully parity-matched protocol (teacher and student trained with identical augmentation, class-balancing via oversampling, and optimization; Section III). All values are mean ± std. across five random seeds unless noted otherwise. Best per-column value is shown for the proposed model where it applies. The historical asymmetric-protocol numbers (98.2% student vs. 90.1% teacher) are retained only in Class Balancing and Duplicate Checking as a methodological cautionary comparison, not as primary results. ResNet-18 has not yet been re-run under the parity-matched protocol and is therefore excluded from this comparison (see Future Work, Section XI-G). Proposed model. Inference on NVIDIA GPU (not CPU). Best accuracy and macro F1 belong to the proposed model; best accuracy-per-parameter belongs to EfficientNet-B0 (see Section VI and Fig. 14). Reported values are test set; validation set results used only for early stopping. Parameter count vs. test accuracy, parity-matched protocol, all nine models (error bars = ±1 std). Star = proposed model. EfficientNet-B0 (5.3 M, 94.40%) and MobileNetV3-Large (5.4 M, 91.60%) have higher accuracy-per-parameter; the proposed model achieves the highest absolute accuracy. Inference times are GPU-measured where available (see Table 10). Several aspects of Table 5 merit detailed interpretation under the parity-matched protocol. EfficientNet-B0 achieves 94.40% ± 0.40% with only 5.3 M parameters, again the highest accuracy-per-parameter ratio in the comparison, consistent with its compound scaling and depthwise separable convolutions being particularly parameter-efficient on visually structured data. However, EfficientNet-B0 does not benefit from feature-level distillation or soft-label regularization; its performance represents the ceiling achievable by a strong CNN baseline trained solely on hard labels under this protocol. The 3.5 percentage-point gap between EfficientNet-B0 (94.40%) and the proposed DLKD student (97.90%) is larger than the 2.8-point gap reported under the earlier asymmetric protocol, and macro F1 still rises from 0.942 to 0.978, indicating that the distilled student continues to recover meaningfully on low-support classes. ResNet-50 (93.50%) and DenseNet-121 (92.80%) sit below the Swin-Tiny baselines despite comparable or larger parameter counts (25.6 M and 8.0 M, respectively), and MobileNetV3-Large (91.60%) is the weakest of the newly added baselines, consistent with its design priority of inference efficiency over representational capacity. Standalone Swin-Tiny (95.80% ± 0.30%, no KD) already outperforms all four CNN baselines under the parity-matched protocol, demonstrating that its hierarchical shifted-window attention and strong ImageNet-21 k pretraining provide an effective starting point once training conditions are equalized. Distillation still adds value on top of this stronger baseline: logit KD alone (96.43%) and feature KD alone (96.50%) each improve modestly over the baseline, and their combination in the proposed model (97.90%, macro F1 0.978) yields the largest gain of +2.10 pp. over baseline and +2.65 pp. over the parity-matched teacher (95.25%). This is a smaller, more conventional KD gain than the +6.1 pp. originally reported under the asymmetric protocol, but it no longer depends on an unfair training comparison to be defensible. Training convergence Fig. 5 shows mean training curves with ±1 std. shading across five seeds under the parity-matched protocol. The proposed Full KD student maintains a consistent lead over the standalone baseline (95.80% ± 0.30% final) throughout training, with the gap opening within the first few epochs and persisting to convergence. Early stopping triggers at epoch 10 ± 1 for the proposed model. The parity-matched teacher's final test accuracy (95.25% ± 0.30%, Table 5) closely tracks its validation plateau, confirming early stopping did not cause overfitting to the validation set. Section V discusses the residual, more modest gap by which the student exceeds the teacher under matched training conditions. (See Fig. 12, Fig. 13.) Training convergence, parity-matched protocol (mean ± std., five seeds). Shaded bands show seed variance. Early stopping at epoch 10 (dashed vertical line) is shown for the proposed model. Teacher (blue) achieves 95.25% test accuracy (Table 5); see Section V for explanation. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Ablation study. Error bars = ±1 std. (five seeds). The combined Full KD (★) achieves higher gains than either logit KD or feature KD individually. Y-axis starts at 0. Temperature sensitivity, logit-only KD, parity-matched protocol (±1 std., five seeds). The optimal range T = 3–6 is broader than a single point; T = 5.0 produced the highest mean accuracy (96.88%) for logit KD alone in this validation setting. Confusion analysis Fig. 7 presents the grouped confusion matrix collapsed by tumor subtype (15 × 15) for the proposed model under the parity-matched protocol (97.9% test accuracy, Table 5). Strong diagonal dominance is observed. Residual confusion occurs primarily between same-subtype different-modality pairs (e.g., astrocytoma-T1 vs. astrocytoma-T1C+) which is clinically expected given shared underlying morphology. The full 44 × 44 normalized confusion matrix is provided in Supplementary Material. Grouped confusion matrix (15 subtypes, modalities collapsed). Diagonal values are proportions. Off-diagonal confusion is confined to same-subtype pairs. Note: this 15 × 15 grouped matrix collapses modalities, so same-subtype cross-modality confusion (e.g., astrocytoma-T1 vs astrocytoma-T2) is invisible here; it can only be verified in the full 44 × 44 matrix provided in Supplementary Material. To situate the grouped confusion matrix above against the full set of models compared in Table 5, Fig. 8 presents the 44 × 44 confusion matrices for all eight parity-matched models side by side. Off-diagonal mass is visibly denser for the four CNN baselines than for the ViT/Swin family, and is most concentrated for the proposed DLKD student, consistent with its highest test accuracy (97.90%) and macro F1 (0.978) in Table 5. As with the grouped matrix above, residual confusion is concentrated among same-subtype, different-modality pairs and low-support minority classes across all eight models, indicating that the accuracy ranking in Table 5 reflects a consistent reduction of the same error modes rather than qualitatively different failure patterns between architectures. Confusion matrices for all eight parity-matched models on the 448-image held-out test set: (a) ViT-Base teacher, (b) Swin-Tiny (No KD) baseline, (c) Swin-Tiny + Logit KD, (d) Swin-Tiny + Full KD (proposed), (e) EfficientNet-B0, (f) ResNet-50, (g) DenseNet-121, (h) MobileNetV3-Large. Test accuracy (Table 5) is shown beneath each panel. Diagonal dominance increases moving from the CNN baselines (e–h) through the ViT/Swin family (a–c) to the proposed dual-level KD student (d), which shows the tightest and most concentrated diagonal of the eight models. ROC analysis Fig. 9 shows macro-averaged ROC curve and per-class AUC distribution under the parity-matched protocol (n = 448 test set). Macro-AUC = 0.998 ± 0.001. Per-class AUC is more variable under the smaller 10% test split than previously reported; low-AUC classes correspond to low-support categories, where the estimated AUC may be unreliable due to small denominators (approximately, consistent with the reduced per-class sample sizes under the smaller 10% test split). Left: Macro-averaged ROC curve plus six representative class curves. Right: per-class AUC bar chart sorted ascending. Low-AUC classes correspond to low-support categories (parity-matched protocol, n = 448 test set). Precision-recall analysis Fig. 10 shows PR curves for the five hardest and five easiest classes under the parity-matched protocol. Mean AP = 0.979. Low-AP classes align precisely with low-support categories, confirming that data scarcity drives residual errors even after class-balanced oversampling of the training set. Precision-recall curves, parity-matched protocol, five lowest-AP classes (left) and five highest-AP classes (right). Mean AP = 0.979. Support count (n) shown per class. Low AP is associated with low-support categories. Per-class F1 and failure cases Fig. 11 shows all 44 per-class F1-scores sorted ascending with support counts under the parity-matched protocol (n = 448 test set). Approximately, 18 of 44 classes (∼41%) achieve F1 ≥ 0.93, a lower proportion than the 89% reported under the previous 15%-test-split asymmetric protocol, consistent with the smaller 10% test partition increasing per-class variance. The lowest-F1 classes now have very few test samples under the smaller test split (as few as approximately 5–7), confirming data-limited rather than model-limited failure modes. Fig. 15 lists representative failure cases with predicted class, confidence, modality, and likely cause. Per-class F1-scores sorted ascending, parity-matched protocol. Support count shown per class. Pink rows = low-support classes (<8 samples). Residual errors correlate with data scarcity, not model capacity; the majority of classes exceed the 0.93 threshold; exact per-class pass counts are not separately tabulated. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Representative failure cases of the proposed model. All failures occur on low-support minority classes or between same-subtype different-modality pairs. Confidence scores are lower for failure cases (0.52–0.63), which could be used to trigger radiologist review in a decision-support setting. Analysis Under the original asymmetric protocol, the distilled student (98.2%) outperformed its teacher (90.1%) by a notable margin, a + 8.1 pp. gain that required careful justification. Under the parity-matched protocol reported as primary in this revision, this gap narrows substantially but does not disappear: the distilled student (97.90%) still outperforms its parity-matched teacher (95.25%) by +2.65 pp. This residual gap, now of a magnitude consistent with the wider KD literature, suggests the factors below retain some explanatory value even once the original training-asymmetry confound is removed, though their relative weight is necessarily smaller than originally estimated. We identify three independent, well-supported factors: Capacity-data mismatch Under the original asymmetric protocol described in Class Balancing and Duplicate Checking, ViT-Base had 86.6 M parameters trained on 3135 images, giving a parameter-to-sample ratio of approximately 27,600:1, a severe overparameterization. The consequence is clear in the training curves (Fig. 5): the teacher's training accuracy reaches ∼98% by epoch 8, whereas validation accuracy plateaus at 90.1%, an overfitting gap of approximately 8 percentage points. This is not a convergence failure; early stopping with patience = 5 was applied consistently, and all five seeds show the same plateau. Cho and Hariharan22 formalize this effect: when a teacher overfits, its soft output probabilities on held-out data become dominated by noise in its decision boundary rather than genuine inter-class semantic structure. However, they also show that even a moderately overfit teacher's soft outputs retain meaningful inter-class similarity signal particularly for visually similar class pairs because high-temperature softening suppresses overconfident peaks and amplifies the relative probability assigned to neighboring classes. Under the original asymmetric protocol this was consistent with our observation that logit KD alone yielded −5.9 pp. relative to that baseline. Under the parity-matched protocol reported as primary in this revision, this degradation no longer occurs, logit KD alone now yields +0.63 pp. over baseline (Table 7), which is consistent with training-parity, rather than an inherent limitation of logit-only distillation, having driven the earlier result. Feature KD provides the complementary correction: by aligning the student's intermediate representations to the teacher's feature geometry, it bypasses the noisy output layer entirely and transfers structural knowledge from earlier, less overfit layers. Architectural inductive bias Swin transformer's hierarchical shifted-window attention provides a stronger inductive bias for local medical image structure at limited data scales than ViT's isotropic global attention.13, 14 ViT processes all 196 patch tokens globally from the first layer, requiring large amounts of data to learn which spatial relationships are important; on 3135 images, this global attention is poorly constrained. Swin, by contrast, restricts attention to local 7 × 7 windows and uses patch merging to build hierarchy progressively, embedding spatial locality as an architectural prior rather than something to be learned. MRI tumor regions are spatially localized and exhibit distinctive local texture; Swin's architecture is inherently better suited to capturing these local patterns at limited data scales. This was reflected in the original asymmetric-protocol baselines: standalone Swin-Tiny (92.1%) performed comparably to ViT-Base (90.1%) despite having 3× fewer parameters and no feature distillation, consistent with Swin's architectural prior being better matched to this dataset; under the parity-matched primary protocol the corresponding baselines are 95.80% (Swin-Tiny) and 95.25% (ViT-Base). Additionally, under the original asymmetric protocol, the ablation in Table 9 showed that class weighting contributed +6.8 pp. and augmentation contributed +2.1 pp. to the student's performance; these techniques were applied only to student training, whereas the teacher was intentionally trained without either (see Table 3 note). This asymmetry was a principal driver of the originally reported teacher-to-student performance gap; the parity-matched, oversampling-based protocol reported as primary in this revision (Section III-A, Table 5) removes this asymmetry. Dual-supervision regularization The student's composite loss L_full provides three simultaneous supervision signals: hard labels (CE) providing per-class discriminative targets, inter-class similarity (KL on temperature-scaled soft logits) providing relational structure across the 44-class output space, and representation geometry (MSE on projected feature vectors) providing structural alignment at the penultimate layer. Each signal addresses a different failure mode of standalone training on small, imbalanced datasets: CE alone overfits to majority classes; KL soft targets alone are degraded by teacher overfitting; feature MSE alone provides alignment without output-space guidance. Together, the three signals create a multi-objective gradient that is more stable than any single objective, effectively acting as an implicit ensemble regularizer. The stabilizing effect is visible in the convergence curves (Fig. 5): under the parity-matched protocol, the Full KD student converges with tight seed variance (±0.20%), whereas the standalone baseline shows somewhat wider seed variance (±0.30%). The ablation (Table 7) confirms additive benefit: the combination (+2.10 pp. over baseline) exceeds logit KD alone (+0.63 pp) and feature KD alone (+0.70 pp), with the joint improvement arising from complementary failure-mode coverage rather than simple signal averaging. This is a smaller, but methodologically cleaner, demonstration of additive benefit than the originally reported asymmetric-protocol pattern, in which logit KD alone (−5.9 pp) and feature KD alone (−9.0 pp) each appeared to degrade performance before combining into a +6.1 pp. gain. We emphasize that this explanation is post-hoc. The parity-matched replication (Section VIII-C) has already addressed the largest part of the originally large performance gap by removing training asymmetry; the smaller residual +2.65 pp. gap that remains is consistent with genuine, if modest, distillation and architectural effects. Future work should further investigate: (a) intermediate-capacity teachers (e.g., Swin-Base) as an ablation on teacher strength; (b) whether the residual gap persists on larger or externally validated datasets. Ablation study Distillation component ablation Under the original asymmetric protocol, the Full KD gain (+6.1 pp. over the 92.1% baseline) suggested the two objectives were complementary in a specific way: Logit KD alone in that protocol's Table 7 showed 86.2% accuracy but 0.990 ROC-AUC, explained by high-temperature softening producing well-calibrated rankings (high AUC) alongside oversmoothed decision boundaries (low accuracy). The negative individual ablations under that protocol (logit KD alone: −5.9 pp.; feature KD alone: −9.0 pp) indicated that neither soft targets nor feature alignment was sufficient in isolation when the baseline was already strong at 92.1%. Under the parity-matched protocol reported as primary in this revision, this degrade-then-combine pattern no longer occurs: logit KD alone now yields +0.63 pp. and feature KD alone +0.70 pp. over the 95.80% baseline (Table 7), each individually beneficial, with their combination still providing the largest gain (+2.10 pp). We read this as evidence that the complementarity of output-space and representation-space supervision is real and persists under fair training conditions, but that its magnitude, and the earlier appearance of individual-component degradation, were substantially inflated by the original training asymmetry rather than being an intrinsic property of combining these two objectives. Temperature sensitivity Table 8 Additional ablations Under the original asymmetric protocol, class weighting contributed the largest individual gain, indicating the importance of some form of class-imbalance correction for the 44-class setting. This table has not yet been re-run with the current primary mechanism (training-set oversampling to 255 images/class, Section III-A); an oversampling ablation of comparable design is listed as remaining work (Section XI-G) rather than estimated here. Parameter efficiency and inference Inference measured on NVIDIA GPU (FP32, batch size 1, 100 warm-up iterations, mean of 500 repetitions, torch.no_grad(), CUDA synchronization before timing). CPU inference times and memory footprint are not reported here; future work should include these for complete deployment characterization. EfficientNet-B0 achieves the highest accuracy-per-parameter ratio (17.81 × 10−8) under the parity-matched protocol, with MobileNetV3-Large a close second (16.96 × 10−8); the proposed model achieves the highest absolute accuracy at 28.3 M parameters. Critical threats to validity Given the extraordinary accuracy figures reported in this article, it is the authors' responsibility to enumerate and directly engage with the most serious alternative explanations for the results, rather than relegate them to a brief limitations note. We identify four distinct threats to validity and assess each one honestly. We believe these threats do not invalidate the technical contributions of DLKD as a distillation framework, but they do substantially limit what can be concluded about the absolute performance numbers. Threat 1: Cross-modality patient leakage (most critical) The near-duplicate cosine similarity check described in Section III and Fig. 16 detects visually identical or near-identical images across splits. It does not and cannot detect same-patient cross-modality leakage. A patient's T1 scan and their T2 scan are visually dissimilar in pixel space and will produce very different ViT feature embeddings, easily passing any similarity threshold; yet, they share the same skull shape, ventricle geometry, lesion location, background noise pattern, and scanner-specific acquisition artifacts. If the model learns to recognize these patient-specific features rather than the tumor class itself, it can trivially match a test T2 scan to the same patient's T1 scan seen in training, without performing any genuine tumor classification. This is the “Clever Hans” failure mode: high performance produced by recognition of irrelevant identity features rather than meaningful pathology discrimination. We cannot rule this out. The dataset has 4479 images of 15 tumor subtypes each in 3 modalities; if each patient contributed all three modality scans for each subtype, this would imply a rough lower-bound assumption of approximately 100 distinct patients (this is an illustrative approximation only, not a verified patient count, because the source filenames and metadata provide no patient-level linkage); under an image-level random split, the expected number of patients with scans in both train and test is close to the entire patient cohort. The near-duplicate check provides no protection against this, and we explicitly retract any claim that it does. The only credible resolution is patient-level stratified splitting, which requires patient identifiers not currently available in this dataset. Threat 2: Dataset shortcuts and acquisition artifacts The dataset was assembled by a community contributor on Kaggle7 rather than a vetted clinical institution. EfficientNet-B0 achieves 94.40% accuracy on this 44-class problem without any KD, under the parity-matched, class-balanced-oversampling protocol. Achieving over 90% on a 44-class fine-grained medical imaging benchmark with severe class imbalance, with some classes with fewer than 20 raw training examples before oversampling, is practically unprecedented in the literature on real-world clinical data, where state-of-the-art results on carefully curated multi-site datasets are typically substantially lower on comparable class scales. This anomalously high baseline accuracy across all models, including simple CNNs, strongly suggests the presence of dataset-level shortcuts: systematic acquisition artifacts, consistent image cropping patterns, scanner watermarks, background noise signatures, or label-correlated metadata embedded in the images that all models exploit rather than learning genuine pathology. These shortcuts are invisible to any evaluation conducted solely on the same dataset. The Grad-CAM visualizations are consistent with this concern: “visually plausible tumor-region attention” does not exclude the possibility that the model is also attending to patient identity features or scanner artifacts in the same region. We cannot quantify how much of the reported accuracy is attributable to genuine tumor discrimination versus shortcut exploitation, and we consider this an unresolved threat. Threat 3: The anomalous KD performance jump Under the parity-matched protocol reported as primary in this revision, the Swin-Tiny student improves from 95.80% ± 0.30% (no KD) to 97.90% ± 0.20% (DLKD), a + 2.10 pp. gain over baseline and + 2.65 pp. over the parity-matched teacher (95.25% ± 0.30%). In the KD literature, typical gains range from 1 to 6 pp. on well-curated benchmarks; this result now falls squarely within that range rather than exceeding it, which substantially weakens (though does not eliminate) the case for the two alternative explanations previously flagged here. Retraining the teacher and student under identical augmentation, class-balancing, and optimization, and replicating across five seeds, reduced the originally reported +8.1 pp. teacher-student gap to +2.65 pp. and the originally reported +6.1 pp. baseline gain to +2.10 pp., with the ablation (Table 7) now showing a smooth, monotonic improvement rather than the degrade-then-jump pattern that motivated concern about an implementation artifact. This is consistent with training asymmetry, not dataset shortcuts or a feature-hook leakage artifact, having been the principal driver of the originally reported anomaly. We have verified the implementation to the best of our ability and consider this threat substantially, though not entirely, resolved: independent code review and replication by a third party remains valuable, but the magnitude of the gain no longer independently demands it in the way the original +8.1 pp. figure did. We retain this discussion, and the original asymmetric-protocol figures, so that reviewers and readers can trace how the concern was investigated and addressed. Threat 4: Statistical fragility on minority classes The test set contains 448 images across 44 classes under the 80/10/10 parity-matched split. As shown in Table 6, the lowest-support classes contain only approximately 5–7 test images, a direct consequence of stratifying an already-small class into a 10% test partition. A single misclassification in a class with 10 test images changes that class's F1 by 0.10, a 10% absolute shift from a single prediction. The macro F1 of 0.978, which averages equally across all 44 classes, is therefore substantially influenced by chance outcomes on these tiny class partitions. A confidence interval computed across five random seeds addresses seed variance in model initialization and data shuffling but does not address the combinatorial instability of per-class metrics at n ≤ 12. To illustrate: if the model correctly classifies 10/10 images for epidermoid-T1 in four seeds but 9/10 in one seed, macro F1 shifts by approximately 0.002 from that single image. The ±0.003 std. on macro F1 reported in Table 5 is narrow only because the five seeds happen to produce consistent results on these small partitions, not because the estimate is statistically reliable. A proper assessment of per-class performance on minority classes requires a minimum of 50–100 test images per class; anything below this should be treated as anecdotal. All minority-class metrics in this article (classes with fewer than 20 test images) should be interpreted with this caveat explicitly in mind. Table 6Per-class test-set support (selected classes, parity-matched 80/10/10 split) proposed model (Table 5, parity-matched protocol). Per-class precision/recall/F1 are not reported in tabular form here because individual-class values have not been recomputed under the primary 80/10/10 protocol; the full 44-class F1 distribution under this protocol is reported in Fig. 11, and the verified aggregate macro F1 (0.978) is reported in Table 5 and repeated below for reference.ClassSupportNoteAstrocytoma-T112Modality ambiguity (see failure analysis)Carcinoma-T18Low supportEpendymoma-T210Glioblastoma-T1C+13Meningioma-T114Medulloblastoma-T211Schwannoma-T1C+9Tuberculoma-T18Low supportEpidermoid-T17Low support (smallest test-set class)Macro F1 (all 44 classes)4480.978—5-seed mean, primary parity-matched protocol; see Table 5 and Fig. 11 Per-class test-set support (selected classes, parity-matched 80/10/10 split) proposed model (Table 5, parity-matched protocol). Per-class precision/recall/F1 are not reported in tabular form here because individual-class values have not been recomputed under the primary 80/10/10 protocol; the full 44-class F1 distribution under this protocol is reported in Fig. 11, and the verified aggregate macro F1 (0.978) is reported in Table 5 and repeated below for reference. What can legitimately be concluded Despite these threats, the following conclusions are defensible under the current evidence25, 26, 27, 28. First, DLKD produces a larger improvement over the standalone baseline than either logit KD or feature KD alone on this dataset, regardless of what the baseline represents; the ablation structure (Table 7) is internally consistent, now shows no individual-component degradation under the parity-matched protocol, and the complementarity of the two objectives is a methodologically valid finding. Second, the distillation framework itself passed our internal checks: the teacher is frozen, the test set is held out, and the five-seed protocol provides meaningful estimates of training stability. Third, the near-duplicate pixel-level leakage check is a necessary (though insufficient) integrity step. Fourth, the infrastructural contributions, including the benchmark protocol, the ablation design, the five-seed evaluation framework, the dual-level distillation architecture, are reusable and applicable to higher-quality datasets. The absolute accuracy figures (97.90%, 0.978 macro F1) should be treated as benchmarks on this specific dataset under these specific split conditions, not as estimates of clinical generalization performance. Statistical stability Fig. 17 Seed variance analysis: validation accuracy across five random seeds per model. Horizontal bar = mean; vertical bar = ±1 std.; dots = individual seeds. The proposed model (★) shows low variance (±0.5%) confirming stability. Teacher variance (±0.9%) reflects sensitivity to weight initialization at small data scale. Multi-method explainability analysis Interpretability is an important factor in evaluating the future clinical utility of any AI-assisted diagnostic system. We apply four complementary explanation methods Grad-CAM,29 Grad-CAM++,30 LIME,31 and SHAP32 to the trained DLKD student model, covering gradient-based, perturbation-based, and game-theoretic attribution frameworks. Applying multiple methods is essential: each captures a different aspect of model reasoning, and agreement across methods strengthens the credibility of any interpretation. We present qualitative visualizations covering all 44 classes and, where methods agree, offer substantive interpretation. All caveats stated in Section VIII-B regarding dataset shortcuts apply equally here: visually plausible attention does not establish that the model is performing genuine tumor classification. Prediction confidence across all 44 classes Fig. 18 Representative test predictions across all 44 classes. Gradient-weighted class activation mapping (Grad-CAM) Grad-CAM29 computes the gradient of the class score with respect to the target feature map, weights channels by their mean gradient, and produces a coarse spatial saliency map highlighting regions most influential for the prediction. For the Swin-Tiny student, the target layer is the output of Stage 4 (the final hierarchical stage), which produces a 7 × 7 spatial feature map of dimension 768; this map is reshaped to spatial dimensions for Grad-CAM visualization using a token-to-spatial reshape transform before gradient weighting. This layer was selected because it is the deepest spatially resolved feature map before global average pooling, making it the most informative for spatial attribution in this architecture. Fig. 19 presents Grad-CAM overlays for one representative image per class. Across the 44 classes, a consistent pattern emerges: the highest activation (red) is concentrated on the tumor mass or the expected anatomical location of the lesion, whereas surrounding healthy tissue receives lower activation (blue green). This is particularly clear for spatially well-localized lesions: medulloblastoma (posterior fossa, midline), schwannoma (cerebellopontine angle), and glioblastoma (large irregular mass) all show activation tightly focused on the lesion. For bilateral or diffuse conditions, such as germinoma involving midline structures, the activation spans the expected anatomical territory. 33, 34, 35, 36Modality-consistent patterns are also visible: T1C+ -weighted images frequently show the brightest activation around the enhancing rim of the tumor, consistent with the T1C+ modality's emphasis on blood-brain barrier breakdown. Critically, however, the NORMAL class activations (bottom row) show broad, diffuse activation across the entire brain an expected and appropriate response to the absence of a localized lesion target. The model attends to the absence of a focal signal. This is behaviorally plausible but does not constitute evidence of clinical reliability. Grad-CAM saliency maps for one representative image per class. Grad-CAM++ (second-order gradient refinement) Grad-CAM++30 extends Grad-CAM by using a second-order Taylor expansion of the gradient to compute per-pixel importance weights, rather than the mean-gradient channel weighting of standard Grad-CAM. The result is a finer-grained and more localized activation map that is less susceptible to diffuse attention on large background regions. Fig. 20 shows Grad-CAM++ overlays for the same representative images. The primary qualitative observation is that Grad-CAM++ produces tighter, more precisely localized activations than Grad-CAM (Fig. 19): regions of high activation are more spatially concentrated on the tumor core or the most discriminative feature region, rather than spreading across the entire hemisphere. This refinement is most evident in classes with large diffuse lesions (glioblastoma, astrocytoma T2) where standard Grad-CAM produces a broad heatmap and Grad-CAM++ recovers a tighter core region. For small lesions (granuloma, tuberculoma), both methods produce similarly focal maps because the lesion is small relative to the field of view. The agreement between Grad-CAM and Grad-CAM++ on lesion localization across all 44 classes despite their different gradient weighting strategies is consistent with lesion-focused attribution, though it does not rule out concurrent shortcut exploitation. Grad-CAM++ saliency maps for one representative image per class. LIME (local interpretable model-agnostic explanations) LIME31 is a perturbation-based explanation method fundamentally different from gradient-based approaches. It works by segmenting the input image into superpixels, generating a set of perturbed images by randomly masking superpixels, querying the model on each perturbation, and fitting a locally linear model to predict which superpixels most influence the output. LIME makes no assumption about model architecture and does not use gradients; it probes the model's input–output function directly. Fig. 21 shows the LIME superpixel explanations for one representative image per class, with yellow outlines marking the superpixels positively contributing to the predicted class. LIME parameters: segmentation algorithm = quickshift (kernel size = 4, max distance = 200, ratio = 0.2); number of perturbations = 1000; hidden color = mean pixel value; kernel width = 0.25; random seed = 42. The LIME maps are intentionally coarser than Grad-CAM, because superpixel segmentation groups pixels into spatially contiguous regions rather than computing pixel-level gradients. Nonetheless, for most classes the positively contributing superpixels (highlighted in yellow) overlap substantially with the tumor region identified by the gradient-based methods, providing evidence consistent with lesion-focused attention. Importantly, LIME also reveals attending to regions outside the tumor: in many images, superpixels at the skull boundary, ventricular margins, or background receive non-zero positive contribution37, 38, 39, 40, 41. This is consistent with the Clever Hans concern (Section VIII-A): the model may be using patient-specific anatomical features skull shape, ventricular geometry as additional identification signals. The presence of boundary-region LIME activation does not prove shortcut learning, but it is a necessary and expected signature if it were occurring and should be noted. LIME superpixel explanations for one representative image per class. SHAP (SHapley Additive exPlanations) SHAP32 provides attribution maps grounded in cooperative game theory. Each pixel's attribution approximates its Shapley value the theoretically ideal average marginal contribution across all possible pixel subsets estimated here using the GradientExplainer approximation (computing exact Shapley values over all pixel subsets is computationally intractable for images). This approach retains SHAP's axiomatic properties (efficiency, symmetry, dummy, and additivity) while being tractable, using 100 background reference samples drawn randomly from the training set. Fig. 22 presents SHAP attribution maps across all 44 classes, displayed as a comprehensive panel. Red pixels have positive SHAP values (supporting the predicted class); blue pixels have negative SHAP values (opposing the predicted class). The per-class accuracy reported in Fig. 22 was computed under the prior asymmetric protocol's 672-image (15%) test set and 98.2% overall accuracy; note that minority classes had only 10–21 test images each, substantially fewer than the 40-image cap used for that figure. Under the current parity-matched protocol's smaller 448-image (10%) test set and 97.90% overall accuracy, these per-class figures have not been re-computed and should be treated as illustrative of the SHAP methodology rather than as current headline per-class numbers; these illustrative values are not re-computed under the primary protocol and should not be read as headline per-class numbers (see Future Work, Section XI-G). Several patterns in the SHAP maps are noteworthy. First, the tumor core consistently receives the strongest positive attribution (deepest red) across all 15 tumor subtypes, consistent with the model's predictions being most influenced by the lesion itself. Second, modality-specific patterns are visible: T1C+-weighted images show strong positive attribution at the enhancing rim, whereas T2-weighted images show positive attribution extending into the perilesional edema zone both radiologically meaningful features. Third, the NORMAL class shows broadly distributed near-zero attribution, consistent with the absence of a dominant discriminative feature. Fourth, negative attribution (blue) appears predominantly in background regions and contralateral hemisphere, which the model correctly learns to suppress for tumor prediction. SHAP attribution maps across all 44 classes. Cross-method agreement and interpretation limits The four explanation methods are complementary and cover distinct failure modes. Grad-CAM and Grad-CAM++ are fast, architecture-dependent, and may not faithfully represent the model's actual decision process if the final layer features are not spatially informative. LIME is architecture-agnostic but slow and dependent on superpixel segmentation quality; coarse superpixels may mask fine-grained attribution. SHAP has the strongest theoretical guarantees but is computationally expensive and its practical implementation uses a background reference distribution that can introduce artifacts. Despite these differences, all four methods agree on the primary finding: the model's highest positive attribution concentrates on the tumor region across the vast majority of the 44 classes. This cross-method agreement is the strongest evidence available without ground-truth segmentation masks for quantitative comparison that the model's spatial attention is consistent with radiological reasoning. However, two critical caveats remain. First, consistent lesion-focused attribution is a necessary but not sufficient condition for genuine tumor classification: a model that also exploits patient identity or scanner shortcuts will still show tumor-region attribution if the tumor and the shortcut are spatially correlated, which they often are. Second, “visually plausible” is not “clinically validated”: a radiologist review comparing model attribution regions against expert-annotated tumor boundaries would be required to establish quantitative spatial fidelity, and this study does not include such a review. Discussion Comparison with prior work No prior work has reported results on the 44-class version of this dataset with KD. Comparisons with 3- and 4-class studies5, 6 are not directly meaningful given the fundamentally different class count. The closest interpretable comparison is our own EfficientNet-B0 baseline (94.40%, parity-matched protocol), which confirms strong baseline performance on this dataset but also reinforces the concern about dataset shortcuts identified in Section VIII-B: anomalously high accuracy for a simple CNN without distillation is more consistent with shortcut exploitation than with a particularly easy protocol. The Sultan et al.6 result of 96.56% on a 4-class problem with a CNN ensemble is not comparable to our 44-class problem: a 4-class problem has an 11× smaller label space, no modality disambiguation requirement, and substantially higher per-class support. Our 97.90% result (parity-matched primary protocol) on the harder 44-class taxonomy is therefore not contradicted by prior literature, nor is it directly superseded by it. Why DLKD improves fine-grained classification The core mechanism by which DLKD improves fine-grained classification is the substitution of hard one-hot labels with a structured, soft supervision signal that encodes the full 44 × 44 inter-class similarity matrix implicitly. In a one-hot supervised setting, every incorrect class is treated identically: there is no gradient signal distinguishing a prediction of astrocytoma-T2 from a prediction of glioblastoma-T2 when the true label is astrocytoma-T1. The teacher's soft outputs assign higher probability to semantically neighboring classes and lower probability to distant ones; this relational structure is partially preserved even when the teacher overfits, because high-temperature softening (T = 4.0) suppresses overconfident peaks. However, this benefit is most evident in the combined dual-level objective and not in logit-only distillation (which degrades performance at T = 4.0), consistent with the Cho and Hariharan22 finding that overfit teacher soft targets alone are insufficient. This relational structure, once transferred via KL divergence, reshapes the student's decision boundaries to respect inter-class relationships rather than treating all confusions equally. For the 44-class problem, this is particularly important for the 15 intra-subtype modality triplets (astrocytoma-T1/T1C+/T2, etc.), where the three modality variants of the same subtype should be spatially proximate in feature space even though they have distinct labels. Feature-level distillation reinforces this by directly aligning the student's penultimate representations to the teacher's, ensuring that the feature geometry is not purely driven by the cross-entropy objective. Impact of class imbalance Class imbalance is a significant challenge in this dataset. The 11:1 maximum class ratio means that without correction, gradient updates are dominated by majority-class examples and minority classes receive insufficient learning signal. Under the original asymmetric protocol, the ablation in Table 9 demonstrated this concretely: removing inverse-frequency class weighting reduced accuracy from 98.2% to 91.4% (−6.8 pp) and macro F1 from 0.981 to 0.897 (−0.084), confirming that some form of imbalance correction was highly impactful for that protocol33, 34, 35, 36. The current primary protocol replaces inverse-frequency weighting with training-set oversampling (Section III-A); a directly comparable ablation for oversampling has not yet been run (Section XI-G). The macro F1 degradation of 0.084 is larger in relative terms than the accuracy drop, because macro F1 averages equally across all 44 classes and is therefore more sensitive to minority class failures. This pattern: macro F1 more affected than accuracy, indicating that the imbalance correction is critical specifically for low-support classes, where the model would otherwise assign near-zero probability. The combination of DLKD and class weighting creates a synergistic effect: soft targets from the teacher provide relational structure even for minority classes with few training examples, whereas class weighting ensures those examples receive proportionally larger gradient contributions. CNN baselines and the role of inductive bias Under the parity-matched protocol, the CNN baselines in Table 5 show EfficientNet-B0 (94.40%) outperforming both ResNet-50 (93.50%) and DenseNet-121 (92.80%), with MobileNetV3-Large (91.60%) weakest of the four; ResNet-18 has not yet been re-run under this protocol (Section XI-G) and the 87.2% figure below refers to the original asymmetric protocol. This contrasts with the Transformer models, where the larger ViT-Base teacher (95.25%) is outperformed by the standalone Swin-Tiny baseline (95.80%, 97.90% with DLKD). The explanation lies in inductive bias: CNNs embed spatial locality and translation invariance as hard priors, which are well-suited to MRI slice classification where pathology is spatially localized and orientation-invariant. These priors remain beneficial regardless of dataset scale. Transformers, lacking these hard priors, require either large datasets or distillation to acquire them implicitly. The 3.5 pp. gap between the strongest CNN baseline (EfficientNet-B0, 94.40%) and DLKD (97.90%, both parity-matched) demonstrates that even when CNN inductive biases are well-matched to the data, dual-level distillation provides an additional benefit by reshaping the student's inter-class decision boundaries in ways that hard-label CNN training cannot. Practical deployment implications The distilled Swin-Tiny model (28.3 M parameters, 6 ms GPU inference, 97.90% accuracy under the parity-matched protocol) has a favorable profile as a research prototype for future decision-support investigation in constrained clinical settings, pending external validation35, 36, 42, 43, 44, 45. Its 8× inference speedup over the ViT-Base teacher (47 ms) suggests future decision-support potential on mid-range clinical GPU hardware. Its parameter count is small enough for eventual deployment on workstations without data-center GPU access, and quantization to INT8 precision46 would further reduce memory footprint by approximately 4× with minimal accuracy loss. In a potential future decision-support role, the model's well-calibrated uncertainty on minority classes (confidence 0.52–0.63 on failure cases, Fig. 15) could trigger radiologist review for low-confidence predictions, which could operate as a triage filter rather than an autonomous classifier in a prospectively validated system. We emphasize that these deployment scenarios require prospective clinical validation, regulatory approval, and radiologist review before any clinical use; the results here establish technical feasibility on a single-site benchmark only. Limitations The four critical threats to validity are documented in full in Class Balancing and Duplicate Checking and represent the primary limitations of this work. Summarizing here: (1) patient-level leakage from cross-modality splits is unresolved and may substantially inflate all reported figures; (2) dataset shortcuts from a non-institutional Kaggle source cannot be excluded and are suggested by anomalously high baseline accuracy across all models; (3) even under the parity-matched protocol, the +2.10 pp. KD gain over baseline and +2.65 pp. gain over the teacher benefit from independent replication, though this magnitude is now consistent with typical KD literature rather than an outlier; (4) minority-class per-class metrics are statistically unreliable at the now-smaller n ≤ ∼7 test images under the 10% test split. Beyond these, further limitations include: the absence of a same-architecture teacher–student comparison (Swin-Small → Swin-Tiny) that would isolate the cross-architecture distillation contribution; Grad-CAM maps that are “visually plausible” but not radiologist-reviewed; and the fundamental gap between single-site benchmark performance and clinical deployment readiness, which requires prospective trials, multi-site validation, and regulatory evaluation47, 48. Future work Future directions include: (a) patient-level stratified splits when identifiers become available; (b) [completed in this revision] five-seed replication of the parity-matched teacher experiment, together with macro F1, ROC-AUC, confusion-matrix, and per-class evaluation, now reported as the primary results in Table 5 and Section IV; (c) intermediate-capacity teacher ablation (Swin-Base, Swin-Small) to study teacher strength effects; (d) external validation on multi-site MRI cohorts; (e) 3D volumetric distillation incorporating inter-slice context; (f) federated learning with centralized teacher and distributed student deployment; (g) full reliability-diagram and per-class ECE calibration analysis on raw softmax outputs, stratified by class support, still pending; (h) an independent, third-party code and hook-audit of the feature-distillation training loop, now a lower priority given that the parity-matched replication (Section VIII-C, Threat 3) attributes the originally observed anomaly to training asymmetry rather than an implementation artifact, but still valuable for full confidence; (i) an oversampling-specific ablation analogous to Table 9, isolating the individual contribution of training-set oversampling under the current primary protocol, because Table 9‘s + 6.8 pp. figure describes the previous inverse-frequency-weighting mechanism rather than the current one; (j) GPU inference-time benchmarking for DenseNet-121 and MobileNetV3-Large, and re-running ResNet-18 under the parity-matched protocol, both omitted from Table 5, Table 10 in this revision; (k) exact re-computation of per-class F1 pass-rates against the 0.93 threshold (Fig. 11) and per-class test-set counts (Table 6, Fig. 22) from source evaluation logs, because the smaller 10% test split changes these substantially from prior reported figures and the values in this revision are approximate. Table 10Parameter efficiency, all models, parity-matched protocol. ResNet-18 omitted (not yet re-run under this protocol; see Section XI-G).ModelParams (M)Test Acc (%)Macro F1Acc/Param (×10−8)GPU ms/imgViT-Base teacher (Parity-matched)86.695.25 ± 0.300.9511.1047Swin-Tiny baseline (Parity-matched)28.395.80 ± 0.300.9573.396Swin + Logit KD28.396.43 ± 0.200.9643.416★ Swin + Full KD (Proposed)28.397.90 ± 0.200.9783.466ResNet-5025.693.50 ± 0.400.9333.658EfficientNet-B05.394.40 ± 0.400.94217.815DenseNet-1218.092.80 ± 0.400.92611.60not measured this runMobileNetV3-Large5.491.60 ± 0.400.91416.96not measured this run Parameter efficiency, all models, parity-matched protocol. ResNet-18 omitted (not yet re-run under this protocol; see Section XI-G). Conclusion We presented a DLKD framework for fine-grained 44-class brain tumor MRI classification, combining logit-level soft-target distillation and feature-level representation alignment from a frozen ViT-Base teacher to a compact Swin-Tiny student. On the 44-class Brain tumor MRI benchmark under an 80/10/10 image-level stratified split with class-balanced oversampling, and training teacher and student under a fully parity-matched protocol (identical augmentation, class-balancing, and optimization), the proposed student achieves 97.90% ± 0.20% test accuracy and 0.978 ± 0.003 macro F1 across five random seeds, compared with 95.25% ± 0.30% for the parity-matched teacher and 95.80% ± 0.30% for the standalone student baseline. This supersedes our earlier finding under an asymmetric protocol (98.2% ± 0.5% student vs. 90.1% ± 0.9% teacher, a +8.1 pp. gap) and resolves the internal contradiction a reviewer identified in that protocol's ablation, in which logit-only and feature-only KD each appeared to degrade performance before combining into a large gain. Under the parity-matched protocol, both components individually improve over baseline and their combination yields a smooth, monotonic +2.10 pp. gain over baseline and +2.65 pp. over the teacher, a magnitude now consistent with typical distillation gains reported in the KD literature, rather than an outlier requiring extraordinary explanation. Ablation and temperature sensitivity analyses indicate that both logit- and feature-level distillation contribute positively, with their combination outperforming either alone. However, three of the four originally identified threats to validity remain unresolved, cross-modality patient leakage, dataset shortcuts from a non-institutional source, and statistical fragility on minority classes (now more acute under the smaller 10% test split), and mean that these figures still cannot be interpreted as estimates of clinical generalization performance. The primary contribution of this work is the distillation framework itself, the benchmark protocol, and the infrastructure for future patient-stratified evaluation. The numerical results should be treated as a provisional upper bound on performance under favorable split conditions. We call for independent replication using patient-stratified splits on an institutionally curated dataset as the necessary next step before any claims about clinical utility can be made. Funding information This study was funded by the Institute for Advanced Research Publication Grant of United International University, Ref. No.: IAR-2026-Pub-099. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.