Side-level versus patient-level evaluation in four-view mammography classification: a comprehensive benchmark on the CMMD dataset Authors: Mohamed A Mabrok Published: 2026 ABSTRACT AbstractBackgroundDeep learning for mammographic image classification yields impressive performance metrics, but inconsistent evaluation methodologies—specifically whether results are reported at the independent side level or the bilateral patient level—make cross-study comparisons unreliable. The aim of this study was to quantify, on a single dataset and with a uniform training recipe, how much of the reported performance is determined by evaluation granularity rather than by model architecture.Materials & methodsWe benchmarked six backbone architectures (ResNet-18, ResNet-50, EfficientNet-B0, DenseNet-121, ConvNeXt-Tiny, ViT-B/16) crossed with three multi-view fusion strategies (concatenation, bilateral asymmetry, cross-view spatial attention) on the biopsy-confirmed Chinese Mammography Database (CMMD; 706 four-view patients), using five-fold patient-level stratified cross-validation. Sixteen configurations completed training for both binary malignancy diagnosis and five-class BI-RADS assessment. We report side-level and patient-level metrics; statistical analyses include 5-fold Wilcoxon signed-rank tests, DeLong’s paired AUC test on pooled per-case scores, and bootstrap 95% confidence intervals.ResultsSide-level AUC exceeded patient-level AUC by an average of 17.5 percentage points (range 12.7–22.4), an effect that dwarfs the absolute differences observed between CNN backbones (<3 AUC points). DeLong tests resolved approximately half of all CNN-vs-CNN pairwise comparisons at p<0.05 despite small effect sizes, whereas ViT-B/16 underperformed every CNN variant by 8–10% AUC despite having 6–10× more parameters. Patient-level multi-class BI-RADS evaluation under the standard probability-averaging aggregation rule returned a degenerate macro-AUC of exactly 0.000—a property of the metric/aggregation pair, not of the models—and three concrete alternative aggregators are proposed. The extreme patient-level malignancy prevalence intrinsic to this biopsy-confirmed cohort (96.2%) rendered all models unable to identify non-malignant patients at the patient level.ConclusionReporting methodology, evaluation granularity, and dataset composition are compounding confounds in mammography classification research. Absolute performance numbers reported on CMMD should not be extrapolated to population screening settings, where prevalence is several orders of magnitude lower; studies should report both side-level and patient-level metrics with mutually consistent label/aggregation rules, and characterise performance using confidence intervals or paired statistical tests rather than fold-level Wilcoxon alone.Keywords: Bilateral asymmetry, benchmarking, class imbalance, CMMD, deep learning, evaluation methodology, mammography, multi-view classification Introduction Computer-aided detection and diagnosis (CAD) systems for the analysis of mammographic images have advanced rapidly with the adoption of deep learning [1,2]. Recent studies report area under the receiver operating characteristic curve (AUC) values exceeding 0.95 for malignancy detection, sometimes approaching or surpassing expert radiologists. However, these impressive numbers often obscure fundamental inconsistencies in how performance is measured and reported. Beyond reporting consistency, recent reviews have identified broader implementation challenges—reproducibility, evidentiary standards, technological maturity, user trust, and ethical and legal frameworks—that must be resolved before AI is safely adopted in breast cancer screening programs [3]. A critical but underappreciated source of variability lies in the evaluation granularity. In four-view mammographic acquisition, each patient has four images: craniocaudal (CC) and mediolateral oblique (MLO) views of both the left and right breasts. This four-view acquisition creates a natural ambiguity in evaluation: should the model be assessed at the side level (treating left and right breast predictions as independent samples, effectively doubling the test set) or at the patient level (aggregating the per-side outputs into a single per-patient prediction)? Side-level evaluation is more common in the literature [4,5] because it is simpler to implement and produces larger effective sample sizes. However, it can artificially inflate performance metrics because symmetric patients (those with the same pathology on both sides) contribute two correlated predictions rather than one independent assessment. For asymmetric patients, the model receives credit for correctly classifying the easier side even when it fails on the harder one. In this paper, we present a systematic benchmark designed to quantify empirically how much evaluation granularity matters, and to contextualize this effect against the impact of model architecture choices. This study makes several contributions. We provide a comprehensive evaluation of six backbone architectures and three multi-view fusion strategies (16 configurations total) using five-fold stratified cross-validation on the biopsy-confirmed CMMD dataset; to our knowledge this is among the broadest cross-architecture, cross-fusion comparisons reported on CMMD. We quantitatively demonstrate that side-level evaluation exceeds patient-level AUC by an average of 17.5 percentage points—a gap far larger than any architectural difference observed on this dataset—and show that absolute CNN-vs-CNN AUC differences are small (<3 points), and the conservative 5-fold Wilcoxon test cannot resolve them, although a higher-power DeLong paired test on pooled per-case scores does identify some significant CNN-vs-CNN differences. We further demonstrate that the standard probability-averaging aggregation rule for patient-level multi-class BI-RADS evaluation is mathematically incompatible with the prevailing label definition under bilateral asymmetry, and we propose concrete decision-level alternatives, and that ViT-B/16 substantially underperforms all CNN variants, consistent with the established observation that vision transformers require larger training sets for medical imaging tasks. Finally, our per-case error analysis reveals that misclassification aligns with clinical severity (BI-RADS category), that inter-model error overlap is higher among CNNs than between CNNs and ViT, and that the extreme patient-level class imbalance intrinsic to this enriched diagnostic cohort (96.2% malignant) renders all models unable to identify benign patients at the patient level—an additional reporting confound masked by seemingly balanced side-level distributions. We emphasize that absolute performance numbers reported here pertain to a biopsy-confirmed enrichment cohort and should not be extrapolated to population screening settings, where prevalence is several orders of magnitude lower. Related work Deep learning for mammography Deep learning approaches to mammographic analysis have evolved substantially over the past decade [6]. Early work focused on single-view classification using standard convolutional architectures [7], treating each mammographic image independently. The transition to multi-view architectures, which jointly process all available views for a given patient, marked a significant advancement. Wu et al. [4] demonstrated that multi-view models outperform single-view baselines by leveraging complementary information between CC and MLO projections, establishing a paradigm that subsequent work has largely followed. The GMIC model [8] introduced globally-aware attention mechanisms that enable the network to focus on clinically relevant regions, improving both performance and interpretability. More recently, transformer-based architectures have been explored for mammographic analysis [9], though their effectiveness on small medical datasets remains the subject of ongoing debate due to their need for large-scale pretraining data. Despite this rapid architectural progress, comparatively little attention has been paid to ensuring that the evaluation methodologies used to measure progress are themselves consistent and reliable. Multi-view fusion strategies Fusion of multiple mammographic views has been approached through various strategies. Simple concatenation of view-level features remains a strong baseline [4]. Bilateral asymmetry modeling, inspired by clinical practice in which radiologists compare left and right breasts, has been explored by several groups [10]. Cross-view attention mechanisms that allow views to exchange information during feature extraction have shown promise in recent work [11]. Evaluation methodology in medical imaging The importance of consistent evaluation in medical imaging has been highlighted by several meta-analyses [12,13]. However, specific analysis of how evaluation granularity (side-level vs. patient-level) affects reported performance in mammography has received limited attention. This gap motivates the systematic benchmark presented in this work. The CMMD dataset The Chinese Mammography Database (CMMD) [14] is a publicly available dataset containing mammographic images with both diagnosis labels (biopsy-confirmed benign or normal contralateral vs. biopsy-confirmed malignant) and BI-RADS assessments. Unlike proprietary datasets used in many studies, CMMD enables reproducible research, making it well-suited for benchmarking purposes. We emphasize, however, that CMMD is a biopsy-confirmed diagnostic enrichment cohort rather than a population screening cohort: cases are referred for biopsy following an imaging finding, so the dataset is heavily enriched for malignancy (96.2% at the patient level in the four-view subset analyzed here, see Section 3.1). This composition is appropriate for cross-model benchmarking—all configurations face the same prior—but absolute performance numbers reported on CMMD do not transfer to population-screening settings, where prevalence is on the order of 0.5%. Methodology An overview of the full evaluation pipeline—from raw four-view inputs through preprocessing, fold assignment, the shared backbone, the three fusion strategies, the dual-head classifier, and the side-level and patient-level metrics—is shown in Figure 1. Each stage is described in detail in the corresponding subsection below. Overview of the evaluation pipeline. Inputs (blue) flow through preprocessing and fold assignment (orange), the shared backbone, one of three fusion strategies, and the shared dual-head classifier; outputs are summarized at both the side and patient level (green). All four views of a patient share a fold by construction. Dataset and preprocessing We use the CMMD dataset, restricting to patients with all four views available (L_CC, L_MLO, R_CC, R_MLO) to ensure consistent multi-view evaluation. This restriction yields 706 patients from the original ∼1,360 patients in the dataset. Each patient has independent labels for the left and right breast, enabling both side-level and patient-level evaluation. Throughout this paper, we use the term benign to refer to biopsy-confirmed benign pathology, normal contralateral to refer to an unaffected side at BI-RADS 0 belonging to a patient whose other side is biopsied, and nonmalignant when both categories are referenced collectively. We avoid the compound expression “benign/normal” that appeared in the original draft. We evaluate two tasks: binary diagnosis, which distinguishes nonmalignant (class 0) from malignant (class 1), and BI-RADS assessment, a five-class classification problem (BI-RADS 1–5, mapped to classes 0–4). Class distribution The four-view subset exhibits severe class imbalance. At the patient level, 679 of 706 patients (96.2%) have at least one malignant side, while only 27 patients (3.8%) are bilaterally benign. Most patients are unilaterally affected: 352 have malignancy on the left only, 324 on the right only, 3 are bilaterally malignant, and 27 are bilaterally benign. At the side level (1,412 breasts), 682 (48.3%) are malignant and 730 (51.7%) are benign—a near-balanced distribution that contrasts sharply with the patient-level imbalance. For BI-RADS, the side-level distribution spans five classes: 683 sides are BI-RADS 0 (normal contralateral), 16 are BI-RADS 1, 86 are BI-RADS 2, 225 are BI-RADS 3, and 402 are BI-RADS 4. Table 1 presents the cross-tabulation of diagnosis and BI-RADS labels, confirming the expected clinical alignment: all malignant sides carry BI-RADS ≥2, while benign sides concentrate at BI-RADS 0–2. Table 1.Side-level cross-tabulation of diagnosis labels (rows) and BI-RADS categories (columns). BR-0BR-1BR-2BR-3BR-4TotalBenign683162650730Malignant0060220402682Total68316862254021412Each cell shows the number of breast sides. Side-level cross-tabulation of diagnosis labels (rows) and BI-RADS categories (columns). Each cell shows the number of breast sides. This asymmetry between side-level and patient-level class balance has important implications for interpreting model performance, which we analyze in detail in Section 4.9. Preprocessing Following the established protocol of [14], each mammogram (stored as a NumPy array) is normalized to [0, 1] on a per-image basis (each image is rescaled by its own min and max), resized to 384×384 pixels as a tensor, and replicated to three channels. No ImageNet normalization is applied, as the grayscale medical images occupy a different intensity distribution from natural images. Data augmentation consists of random affine transformations (rotation ±10°, translation ±5%, scale 0.95–1.05) and random horizontal flipping. Three clarifications regarding this protocol are warranted. (i) Train-only fitting: the preprocessing pipeline does not estimate any dataset-wide statistics. Per-image min–max normalization, deterministic resizing, and channel replication are all fit-free; consequently, no statistic crosses the train/validation/test boundary, and the cross-validation procedure cannot leak preprocessing-derived information. (ii) Resolution choice: the 384×384 input was selected because it is natively supported by all six pretrained backbones in our comparison and therefore enables a like-for-like architectural evaluation; it is also a common operating point for medical imaging benchmarks. We acknowledge that this resolution may attenuate features at the scale of microcalcifications, which can be small relative to the original mammogram, and that higher-resolution or multi-scale pipelines may yield improved absolute performance. However, the central object of this paper—the side-vs-patient evaluation gap—is a property of the evaluation protocol and is not expected to depend on the input resolution; we discuss this further in Section 5.7. (iii) Horizontal flipping: the random horizontal flip was applied independently to each of the four views (the default behavior of torchvision.transforms.RandomHorizontalFlip). This implementation does not preserve the left/right correspondence between paired views and is therefore suboptimal for explicit bilateral asymmetry modeling. Importantly, because the bilateral fusion subtracts the (averaged) left feature from the (averaged) right feature, independent flipping injects noise into the asymmetry vector and would tend to attenuate, not inflate, any benefit attributable to bilateral fusion. We therefore expect the small but consistent advantage we report for bilateral fusion (Section 4.5) to be a conservative estimate; we recommend that future work apply paired (per-patient) flipping or omit the flip entirely. Backbone architectures We evaluate six backbone architectures spanning different design families. On the CNN side, we include ResNet-18 (512-d, 11.2 M parameters) and ResNet-50 (2048-d, 23.5 M parameters) as residual network baselines, EfficientNet-B0 (1280-d, 4.0 M parameters) as a compound-scaled efficient architecture, DenseNet-121 (1024-d, 7.0 M parameters) with its dense connectivity pattern, and ConvNeXt-Tiny (768-d, 28.6 M parameters) as a modernized ConvNet design. To contrast with these CNN-based models, we also evaluate ViT-B/16 (768-d, 86.6 M parameters), a Vision Transformer that serves as a representative of attention-based architectures. All backbones are initialized with ImageNet-pretrained weights. A single shared backbone processes all four views through weight sharing, which is standard practice for multi-view mammography models. Fusion strategies We evaluate three fusion strategies applied identically across all backbones. Concatenation (Concat) The simplest baseline: four view features are concatenated into a single vector of dimension 4×d, where d is the backbone’s feature dimension after global average pooling (GAP). Bilateral asymmetry fusion Inspired by clinical practice, we compute explicit bilateral asymmetry features: The fused representation concatenates all four individual view features with the ipsilateral averages and the asymmetry vector: [fL_CC,fL_MLO,fR_CC,fR_MLO,fleft,fright,fasym], yielding dimension 7d. Cross-view spatial attention We inject multi-head cross-attention between views at the penultimate convolutional layer, before global average pooling. Each view’s spatial feature tokens attend to all other views’ tokens: where fv denotes the spatial tokens of view v, fv is the concatenation of all other views’ tokens, and MHA denotes multi-head attention. The attended features then pass through the backbone’s final convolutional block before GAP and bilateral fusion. This strategy applies only to CNN backbones where intermediate spatial features are accessible; for ViT-B/16, it defaults to bilateral fusion (as ViT lacks standard spatial feature maps at intermediate layers). Classification head All configurations share the same classification head: the fused feature vector passes through a two-layer MLP (fused_dim→512→256 with ReLU and dropout), followed by separate linear heads for left and right breast predictions. The training loss is the sum of cross-entropy losses for both sides: Training protocol We use an identical training recipe for all configurations to ensure fair comparison: AdamW optimizer with learning rate 10−4 and weight decay 0.1, cosine annealing learning rate schedule, gradient clipping at 1.0, batch size 8, maximum 50 epochs with early stopping (patience 15 on validation AUC). No label smoothing, mixup, or other regularization techniques are used. All hyperparameters were fixed in advance based on standard practice for fine-tuning ImageNet-pretrained backbones on small medical-imaging datasets; no cross-validation-based hyperparameter search was performed, so cross-validation is used purely to estimate generalization performance and not to tune the training recipe. Evaluation protocol Cross-validation We use five-fold stratified cross-validation, stratifying by pathology pattern (the tuple of left and right labels). Within each fold, the training set is further split 80/20 for training and validation (used for early stopping). The same fold assignments are used across all configurations. The cross-validation operates on patient-level samples (each sample bundles all four views of a single patient), so all four views of a given patient are guaranteed by construction to share a fold; the 80/20 train/validation split within each fold is likewise performed at the patient level. There is therefore no cross-fold or train/validation leakage of either views or patients. Side-level metrics Left and right breast predictions are pooled as 2 N independent samples, where N is the number of patients. We report AUC (binary or macro-averaged one-vs-rest for multi-class), accuracy, F1-score, sensitivity, and specificity. Patient-level metrics The patient-level label is defined under any-malignancy semantics, ypatient=max(yleft,yright), which corresponds to the clinical question “does this patient have malignancy on at least one side?”. For binary diagnosis, the patient-level malignancy score is ppatient=max(pleft,pright) (the OR-semantics aggregator that matches the label), and the patient-level prediction at threshold τ=0.5 is y^patient=1[ppatient>τ]. Patient-level AUC, accuracy, F1, sensitivity and specificity are then computed against ypatient in the standard way. We note that the original draft of this manuscript reported a different patient-level accuracy, namely 1[both sides classified correctly], which is a strict bilateral exact-match criterion rather than a patient-level criterion and is moreover inconsistent with the patient-level AUC definition. Table 3 now reports the corrected metrics; the AUC column is unchanged because it was already correctly computed against ypatient. For BI-RADS, the prevailing convention in the literature is to obtain patient-level probabilities by averaging left and right probability vectors and to take the patient label as the maximum of both sides. We deliberately retain this prevailing rule in our main analysis to expose its pathological behavior under bilateral asymmetry (Section 4.8.1) and recommend alternatives in Section 5.5. Table 3.Binary diagnosis results at patient level (5-fold CV, mean ± std).BackboneFusionAUCACCF1Sens.Spec.ResNet-18Concat.812 ± .044.939.967.969.180ResNet-18Bilateral.761 ± .126.932.964.968.040ResNet-18Crossview.792 ± .084.941.969.973.113ResNet-50Concat.816 ± .068.915.956.944.173ResNet-50Bilateral.796 ± .062.919.957.951.100ResNet-50Crossview.795 ± .082.938.967.969.160EffNet-B0Concat.799 ± .099.933.965.968.080EffNet-B0Bilateral.842 ± .077.936.967.969.113DenseNet-121Concat.749 ± .100.958.978.991.107DenseNet-121Bilateral.805 ± .045.938.968.972.080DenseNet-121Crossview.766 ± .096.936.967.969.100ConvNeXt-TConcat.751 ± .071.905.950.932.220ConvNeXt-TBilateral.773 ± .135.932.964.965.107ViT-B/16Concat.670 ± .099.928.961.962.073ViT-B/16Bilateral.692 ± .100.941.969.972.147ViT-B/16Crossview.707 ± .069.919.957.954.033AUC and side-level metrics are unchanged from the previous version. ACC, F1, sensitivity, and specificity were recomputed from the saved per-case predictions using y^patient=1[max(pL,pR)>0.5] against ypatient=max(yL,yR) . Binary diagnosis results at patient level (5-fold CV, mean ± std). AUC and side-level metrics are unchanged from the previous version. ACC, F1, sensitivity, and specificity were recomputed from the saved per-case predictions using y^patient=1[max(pL,pR)>0.5] against ypatient=max(yL,yR) . Statistical testing We use the Wilcoxon signed-rank test on per-fold AUC values for pairwise model comparison. With five folds, the minimum achievable p-value is 0.0625; the Wilcoxon test on five paired observations is therefore conservative by construction. To complement this, we also report DeLong’s paired AUC test [15] on the pooled per-case scores within each task. DeLong does not suffer from the 5-fold p-value floor and is the standard test for paired AUC comparison; however, its independence assumption is mildly violated by within-patient correlation (left/right sides). We therefore present both tests and view their conjunction as more informative than either in isolation. We additionally report bootstrap 95% confidence intervals on AUC computed by resampling patients within each fold (B=2,000, percentile interval) to characterize the magnitude—rather than only the significance—of pairwise differences (Supplementary Table S1). Experimental results Configuration completion Of the planned 21 configurations (7 backbones × 3 fusions), 16 completed successfully for both tasks. Cross-view attention failed for EfficientNet-B0 and ConvNeXt-Tiny due to architectural incompatibilities in extracting intermediate spatial features, and Swin-Tiny failed entirely due to feature extraction issues. This itself highlights a practical finding: cross-view spatial attention is primarily applicable to ResNet-family architectures where clean layer-level feature extraction is straightforward. Diagnosis: side-level results Table 2 presents side-level results for binary malignancy diagnosis. The top-performing configuration is ConvNeXt-Tiny with concatenation fusion (AUC 0.975±0.008), followed closely by ResNet-50 with bilateral fusion (AUC 0.973±0.007). All CNN-based configurations achieve side-level AUC between 0.950 and 0.975, a remarkably narrow range given the diversity of architectures evaluated. Table 2.Binary diagnosis results at side level (5-fold CV, mean ± std).BackboneFusionAUCACCF1Sens.Spec.ResNet-18Concat.957 ± .012.901 ± .023.901 ± .023.896 ± .024.905 ± .026ResNet-18Bilateral.963 ± .011.893 ± .020.893 ± .020.890 ± .023.896 ± .018ResNet-18Crossview.960 ± .011.897 ± .013.897 ± .013.896 ± .019.899 ± .013ResNet-50Concat.967 ± .015.900 ± .035.900 ± .035.883 ± .038.916 ± .032ResNet-50Bilateral.973 ± .007.914 ± .017.913 ± .017.902 ± .029.925 ± .013ResNet-50Crossview.958 ± .010.887 ± .025.887 ± .025.884 ± .027.889 ± .026EffNet-B0Concat.962 ± .015.895 ± .020.895 ± .021.893 ± .028.897 ± .015EffNet-B0Bilateral.969 ± .012.905 ± .025.905 ± .025.902 ± .029.908 ± .023DenseNet-121Concat.950 ± .014.876 ± .024.876 ± .024.883 ± .024.870 ± .024DenseNet-121Bilateral.964 ± .013.904 ± .020.904 ± .020.905 ± .030.904 ± .012DenseNet-121Crossview.963 ± .016.911 ± .024.911 ± .024.908 ± .025.914 ± .027ConvNeXt-TConcat .975 ± .008 .915 ± .009 .915 ± .009 .891 ± .014 .937 ± .011 ConvNeXt-TBilateral.950 ± .059.891 ± .062.891 ± .063.886 ± .068.896 ± .059ViT-B/16Concat.879 ± .030.804 ± .028.804 ± .028.795 ± .029.812 ± .027ViT-B/16Bilateral.880 ± .049.783 ± .058.782 ± .058.776 ± .059.789 ± .056ViT-B/16Crossview.864 ± .042.776 ± .040.776 ± .041.763 ± .050.789 ± .032Best result per metric in bold. Binary diagnosis results at side level (5-fold CV, mean ± std). Best result per metric in bold. ViT-B/16 is a clear outlier, achieving only AUC 0.864–0.880 across fusion strategies—approximately 8–10% lower than its CNN counterparts despite having 87–89 M parameters (6–10× more than the CNNs). This result confirms that vision transformers require substantially larger training sets to compete with CNNs in medical imaging settings. Diagnosis: patient-level results Table 3 presents the same configurations evaluated at the patient level under the corrected definitions of Section 3.6.3 (ypatient=max(yL,yR), y^patient=1[max(pL,pR)>0.5]). Compared with the original draft, the AUC column is unchanged (it was already computed correctly), whereas the ACC, F1, sensitivity, and specificity columns have been re-derived from the saved per-case predictions and reflect any-malignancy patient-level classification rather than the bilateral exact-match criterion. Several important observations emerge. First, patient-level AUC drops dramatically relative to side-level AUC: the best configuration (EfficientNet-B0 bilateral, AUC 0.842) would rank last among the CNN side-level results. Second, patient-level specificity is near zero for most configurations (mean 0.04–0.22), indicating that the models almost never correctly identify patients who are nonmalignant on both sides—they default to predicting malignancy on at least one side. Third, patient-level sensitivity remains extremely high (>0.93), revealing a strong bias toward positive predictions; this bias—combined with the 96.2% malignancy prevalence—is what produces the high corrected accuracy values, and it should be interpreted in light of the prevalence-shift discussion in Section 5.6. The side-level vs. patient-level gap Figure 2 presents the central finding of this paper. Every model configuration lies well below the y=x diagonal, demonstrating a systematic and substantial gap between side-level and patient-level AUC. Side-level AUC vs. patient-level AUC for all configurations (diagnosis task). All points fall well below the y=x line, demonstrating systematic inflation of side-level metrics. Error bars show ±1 std across 5 folds. Table 4 quantifies this gap. The average AUC difference from side-level to patient-level evaluation is +0.175 (17.5 percentage points). The gap ranges from +0.127 (EfficientNet-B0 bilateral) to +0.224 (ConvNeXt-Tiny concat). For accuracy, the original draft reported a gap of +3.6% under the (now-corrected) bilateral exact-match criterion; under the corrected any-malignancy criterion (Table 3), patient-level accuracy is actually higher than side-level accuracy on average (∼0.93 vs. ∼0.88), because in this enriched cohort the trivial “always-malignant” classifier already attains ∼0.96 accuracy at the patient level. This crossover further reinforces the message that single-number accuracy comparisons across granularities are uninformative without prevalence context. Critically, the magnitude of this evaluation gap (in AUC terms) exceeds the entire range of performance differences between CNN architectures (which spans only ∼2.5% at side level). Table 4.Side-level vs. patient-level AUC gap for binary diagnosis.BackboneFusionSidePatient Δ EffNet-B0Bilateral.969.842+.127RN-18Concat.957.812+.145RN-50Concat.967.816+.151RN-50Crossview.958.795+.163DN-121Bilateral.964.805+.159EffNet-B0Concat.962.799+.162RN-18Crossview.960.792+.168RN-50Bilateral.973.796+.176CNX-TBilateral.950.773+.177ViT-B/16Bilateral.880.692+.188DN-121Crossview.963.766+.197DN-121Concat.950.749+.201RN-18Bilateral.963.761+.202ViT-B/16Concat.879.670+.209CNX-TConcat.975.751+.224ViT-B/16Crossview.864.707+.156Average.946.770+.175 Δ = side − patient. Side-level vs. patient-level AUC gap for binary diagnosis. Δ = side − patient. To put this in perspective: a researcher comparing a study that reports side-level AUC of 0.975 with one that reports patient-level AUC of 0.842 might conclude that the first system is substantially superior, when in fact both numbers could come from the same model evaluated at different granularities. This observation has direct implications for systematic reviews and meta-analyses in mammographic image classification, where heterogeneous evaluation protocols make it difficult to draw reliable conclusions about the state of the art. Two caveats are warranted in the interpretation of Δ. First, side-level and patient-level AUC are not two views of a single estimand: they are computed against different label vectors (yside for 2 N pooled sides versus ypatient for N patients), they involve different class priors (∼48% vs. 96.2% malignant), and they use different sample units. The 17.5-point gap should therefore be read as the empirical performance gap a researcher would observe by switching reporting protocols on the same trained model, rather than as a bias measure of either AUC individually; both estimands are valid AUCs with their own scientific interpretations. Second, the absolute magnitude of Δ depends on dataset prevalence: on a more class-balanced dataset the side-vs-patient gap would be smaller. The qualitative finding—that side-level and patient-level evaluation give materially different headline numbers, and that the choice should be reported—is, however, mechanism-driven and is expected to generalize. Backbone and fusion comparison Figure 3 shows side-level AUC grouped by backbone and fusion strategy. Averaged across all backbones, the three fusion strategies perform nearly identically: concat (0.948), bilateral (0.950), and crossview (0.936). The bilateral asymmetry approach provides a marginal benefit (+0.2% over concat), while cross-view attention slightly underperforms, possibly due to the additional parameters without sufficient training data. Side-level (left) and patient-level (right) AUC by backbone and fusion strategy for binary diagnosis. CNN backbones cluster tightly at side level; patient-level reveals greater variance. Averaged across fusion strategies, backbone performance ranks as: ResNet-50 (0.966) > EfficientNet-B0 (0.965) > ConvNeXt-Tiny (0.962) > ResNet-18 (0.960) > DenseNet-121 (0.959) ≫ ViT-B/16 (0.874). The top five CNNs are within 0.7% of each other, while ViT-B/16 trails by approximately 9%. Cross-fold stability Figure 4 shows the distribution of AUC across five folds for each configuration. Several configurations exhibit substantial fold-level instability: ConvNeXt-Tiny bilateral drops from AUC 0.989 (fold 1) to 0.833 (fold 4) at side level. This ∼15% swing from a single fold assignment underscores the unreliability of single train/test split evaluations and the necessity of cross-validation for trustworthy comparison. Distribution of AUC across 5 folds. Left: side-level, right: patient-level. Note the wider spread at patient level and the catastrophic fold for ConvNeXt-Tiny bilateral. Statistical significance No pairwise comparison between any two configurations reaches statistical significance at p<0.05 using the Wilcoxon signed-rank test on per-fold AUCs. This is primarily a power limitation (with 5 folds, the minimum achievable p-value is 0.0625). The closest comparisons to significance (all at p=0.0625) are consistently CNN vs. ViT-B/16 pairs, confirming that ViT is the only meaningfully different architecture at the resolution provided by this conservative test. To overcome the 0.0625 floor and to reduce ambiguity, we additionally computed DeLong’s paired AUC test on the pooled side-level per-case scores (N=1,412 sides) for every pair of configurations on the diagnosis task. This higher-power analysis paints a more nuanced picture: 77 of 120 pairs reach p<0.05, and 56 of 120 reach p<0.001. All 39 CNN-vs-ViT pairs are highly significant (median p<10−12), confirming that ViT is meaningfully separated from the CNN cluster. Among the 78 CNN-vs-CNN pairs, 38 (49%) reach p<0.05 and the median p-value is 0.062. The most strongly significant CNN-vs-CNN comparisons involve DenseNet-121 concat (the weakest CNN at AUC 0.942 on pooled sides) vs. stronger CNNs such as ResNet-50 bilateral and ConvNeXt-T concat (AUC differences of 0.022–0.030; p as low as 4.6×10−10); whereas adjacent-performing pairs such as ResNet-18 concat vs. ResNet-50 concat are not separated (p≳0.6). The qualitative conclusion is therefore unchanged in substance but should be stated more precisely than in the original draft: the absolute magnitude of CNN-vs-CNN AUC differences is small (≤0.030) compared with the side-vs-patient gap (0.175), and adjacent-performing CNNs are statistically indistinguishable; however, well-separated CNNs do reach formal significance under DeLong, and the original blanket statement that “no CNN-vs-CNN comparison reaches p<0.05“ was a property of the conservative 5-fold Wilcoxon test rather than of the underlying performance difference. Bootstrap 95% CIs on AUC for every configuration are reported in Supplementary Table S1; the side-level CIs are tight (typical width 0.014–0.020) while the patient-level CIs are wider (typical width 0.10–0.18), which is itself useful information about the reliability of patient-level estimates on this dataset size. BI-RADS assessment Supplementary Table S2 presents BI-RADS results at side level. Performance is substantially lower than binary diagnosis, with the best configuration (DenseNet-121 crossview) achieving AUC 0.846±0.015. This reflects the difficulty of five-class classification with severe class imbalance. Probability averaging is invalid for patient-level multi-class BI-RADS evaluation When the prevailing label/aggregation rule (label max(yL,yR), score = mean of per-side probability vectors) is applied to BI-RADS, every configuration we evaluated returns a patient-level macro-AUC of exactly 0.000. An AUC of exactly zero across all models is a red flag and warrants explicit explanation: this number is not a model-performance result but a demonstration that the chosen aggregation rule is incompatible with the chosen label definition. Investigation reveals a compounding of two effects. First, 98.9% of patients have different BI-RADS scores on their left and right breasts, with the dominant patterns being (BI-RADS 5, BI-RADS 1) and vice versa. Second, probability averaging destroys the discriminative signal. When a patient has left = BI-RADS 1 (normal) and right = BI-RADS 5 (malignant), the model correctly predicts each side individually. However, averaging the probability vectors (e.g., [0.85, 0.02, 0.03, 0.05, 0.05] and [0.05, 0.02, 0.03, 0.10, 0.80]) yields [0.45, 0.02, 0.03, 0.075, 0.425], which peaks at class 0 (BI-RADS 1), while the patient label max(0,4)=4 never equals 0. Consequently, 99.6% of patient-level predictions are class 0, whereas class 0 never appears in the ground truth (since max(left,right) is always ≥1 for patients with at least one abnormal side). Probability averaging is fundamentally incompatible with patient-level multi-class evaluation when bilateral asymmetry is prevalent; the 0.000 AUC is therefore best read as a sanity-check failure of the metric/aggregation pair, not as a measurement of model capability. This result is not merely an artifact of the CMMD dataset; it generalizes to any bilateral medical imaging scenario where (a) the two sides can have different multi-class labels, (b) patient-level labels are defined as the maximum of both sides, and (c) patient-level probabilities are obtained by averaging per-side probability vectors. The mathematical mechanism is straightforward: averaging two peaked distributions over different classes produces a flat distribution that peaks at neither class. We therefore do not interpret the 0.000 figures as a model-comparison result and we advise against using the (label-max, score-mean) rule. Concrete alternatives are developed in Section 5.5. Per-case error analysis To understand whether different models fail on the same cases and how errors relate to clinical severity, we analyze per-case predictions across all 16 configurations. Inter-model error consistency Of 706 patients, 356 (50.4%) are correctly classified by all 16 model configurations, while only 7 (1.0%) are misclassified by every model. The remaining 343 patients (48.6%) are correctly classified by some models but not others, revealing substantial model-specific variation in error patterns. Supplementary Table S3 summarizes the distribution. The pairwise error overlap, measured by Jaccard similarity on misclassified patient sets, reveals an important architectural distinction. CNN-to-CNN Jaccard similarity averages 0.42 (range 0.31–0.55), while CNN-to-ViT similarity averages only 0.29 (range 0.21–0.39). This confirms that CNNs make systematically similar errors to each other, while ViT-B/16 fails on a partially different patient population—a finding relevant for potential ensemble strategies. Error rates by BI-RADS category Side-level error rates show a clear gradient aligned with clinical severity (Supplementary Table S4). For malignant sides, error rates decrease monotonically with increasing BI-RADS: 22.9% for BI-RADS 2, 17.5% for BI-RADS 3, and 8.7% for BI-RADS 4. This is clinically intuitive—higher BI-RADS lesions have more prominent imaging features. For benign sides, the pattern is reversed: benign sides with BI-RADS 0 (normal contralateral) have a low error rate (9.3%), but the rare benign sides with elevated BI-RADS scores (1–3) are misclassified 38–48% of the time. These represent clinically ambiguous cases where benign tissue presents with suspicious imaging features. The benign patient problem The most concerning finding from the per-case analysis is that no benign patient is correctly classified by all models. The 27 bilaterally benign patients have a mean error rate of 88.4% across all configurations (i.e., on average, 14.1 out of 16 models misclassify each benign patient). Four benign patients are misclassified by every single model. By contrast, malignant patients have a mean error rate of only 12.9%, with 52.4% of malignant patients correctly classified by all 16 models. Patient-level error rates stratified by both diagnosis and BI-RADS (Supplementary Table S5) reveal the compounding effect: benign patients with low BI-RADS scores are misclassified by nearly all models (≥12/16 on average), while malignant patients with high BI-RADS are relatively easy (≤1.4/16 wrong on average). Cross-task error alignment For each model configuration, we examine whether patients misclassified on the diagnosis task are also misclassified on the BI-RADS task. The overall agreement rate is 60.3%: when diagnosis is wrong, BI-RADS is also wrong 84.9% of the time (1,509 of 1,779 cases). However, when diagnosis is correct, BI-RADS is still wrong 44.3% of the time (4,211 of 9,517 cases). This asymmetry confirms that BI-RADS classification is a strictly harder task that fails on a superset of the cases where diagnosis fails, plus many additional cases where the binary decision is correct but the fine-grained severity assessment is not. Discussion Evaluation granularity as a confound Our central finding—that the side-vs-patient evaluation gap (17.5% AUC on CMMD) dwarfs all architectural differences (<3% among CNNs in absolute AUC)—has important implications for the mammography classification literature. A study reporting side-level AUC of 0.975 and another reporting patient-level AUC of 0.842 may in fact be evaluating nearly identical models. Without standardized evaluation protocols, cross-study comparison is unreliable. The inflation mechanism involves three distinct factors. First, side-level evaluation pools left and right predictions as independent samples, but they share the same patient context and model weights; the statistical independence assumption underlying AUC computation is therefore violated. Second, for symmetric patients (same pathology on both sides), both predictions are effectively the same case counted twice, inflating apparent sample size without adding information. Third, for asymmetric patients, the model receives credit for correctly classifying the easier side even when it fails on the harder one—a partial success that is rewarded at side level but penalized at patient level. This finding has practical consequences beyond academic benchmarking. Regulatory submissions for CAD systems, clinical validation studies, and purchasing decisions by healthcare institutions all rely on reported performance metrics. If the evaluation granularity is not explicitly specified, stakeholders may draw misleading conclusions about system capabilities. We therefore recommend that all future mammography classification studies report both side-level and patient-level metrics, and explicitly state which granularity is used for primary comparison. It is worth distinguishing here between findings directly supported by our experiments on CMMD and broader generalizations that follow as conjectures. (i) Directly supported on CMMD: the absolute 17.5-point side-vs-patient AUC gap; the empirical ranking of the 16 configurations; the fact that the 5-fold Wilcoxon test resolves no CNN-vs-CNN pair while DeLong on pooled scores resolves about half (median p≈0.06); and the bootstrap CI widths reported in Supplementary Table S1. (ii) Mechanism-driven and therefore portable: the symmetric-patient double-counting effect, the asymmetric-patient partial-credit effect, and the BI-RADS averaging-incompatibility argument of Section 4.8.1 all follow from the structure of the metric and label rather than from properties of CMMD; we expect them to apply, with possibly different magnitudes, to any bilateral medical-imaging task with comparable label structure. (iii) Conjectural extensions: the magnitude of the gap is expected to shrink on more class-balanced datasets, and the relative ranking of architectures may change with substantially more training data—these statements are reasonable expectations but cannot be verified from CMMD alone. The diminishing returns of architectural complexity Our results support a growing body of evidence that architectural complexity provides diminishing returns on small medical imaging datasets. The simplest CNN backbone (ResNet-18, 11.2 M parameters) achieves AUC within 0.7% of the most complex CNN (ConvNeXt-Tiny, 28.6 M parameters) at side level. EfficientNet-B0 achieves the best patient-level AUC with only 4.0 M backbone parameters. This suggests that, for datasets of this scale, the primary bottleneck is the quantity and quality of training data, not the representational capacity of the model. The dramatic underperformance of ViT-B/16 (AUC ∼0.87 vs. ∼0.96 for CNNs) is consistent with the established finding that vision transformers require large-scale datasets to learn effective representations [16]. With only ∼560 training patients per fold, the inductive biases of CNNs (translation equivariance, local connectivity) provide a critical advantage that cannot be overcome by the transformer’s greater theoretical flexibility. Notably, our per-case error analysis reveals that ViT-B/16 not only performs worse overall but also fails on a partially different patient population than CNNs (Jaccard similarity 0.29 vs. 0.42 for CNN-CNN pairs). This complementarity suggests that CNN-ViT ensembles could be beneficial if sufficient data were available for the transformer component to achieve competitive individual performance. From a practical standpoint, these findings suggest that researchers working with small medical imaging datasets (<1,000 patients) should prioritize data collection and curation over architectural innovation. A well-tuned ResNet-18 with careful data preprocessing may outperform a state-of-the-art architecture trained on the same limited data. Bilateral asymmetry: marginal but consistent The bilateral fusion strategy provides a small but consistent benefit: averaged across backbones, it improves side-level AUC by 0.2% over concatenation and patient-level AUC by a more meaningful margin (e.g., EfficientNet-B0 bilateral achieves the highest patient-level AUC at 0.842). The clinical motivation—that radiologists diagnose by comparing bilateral symmetry—appears to translate into a modest computational advantage, primarily at the patient level where bilateral relationships matter most. Cross-view attention: limited benefit Cross-view spatial attention, despite its computational elegance, provides no consistent improvement over simpler fusion strategies. On this dataset, the additional parameters and architectural complexity may introduce optimization difficulties that offset any representational advantage. Moreover, the approach is limited to architectures with accessible intermediate spatial features, failing for EfficientNet and ConvNeXt variants. Valid alternatives for patient-level multi-class aggregation The patient-level BI-RADS result of Section 4.8.1 reveals a methodological trap: probability-averaging across laterally asymmetric predictions can produce structurally degenerate metric values. We therefore propose three concrete alternative aggregation strategies, ordered from drop-in fixes that require no retraining to architectural changes that require a new model. Throughout, pL,pR∈ΔC−1 denote per-side probability vectors over C BI-RADS classes, yL,yR the per-side labels, and ypatient=max(yL,yR) the prevailing patient-level label. Decision-level max-class (drop-in) Take the per-side argmax and aggregate at the label space: Patient-level performance is then evaluated as a C-class classification problem with predictions c^patient against ypatient, using accuracy, macro-F1, and per-class confusion matrices. AUC is sidestepped (or computed in a one-vs-rest fashion on a synthesized score, see (B)). This rule is the simplest fix because it requires no retraining and operates only on saved per-side outputs. It is the patient-level analogue of the binary max(pL,pR) rule we use for diagnosis and is consistent with the any-malignancy interpretation of ypatient. Per-class max-pooling of probabilities (drop-in) Replace probability averaging with element-wise maximum: The resulting vector is generally unnormalized (and often super-stochastic), but it preserves the discriminative signal of whichever side is most confident in each class, which is precisely the semantics that a max-of-both-sides label requires. After re-normalising (divide by ∑cppatient[c]), one can compute one-vs-rest AUC against ypatient in the standard way. This is also a drop-in fix that requires only saved per-side outputs. Direct patient-level training Treat ypatient=max(yL,yR) as the optimization target and add a dedicated patient-level head whose loss is CE(y^patient,ypatient) in addition to (or in place of) the per-side cross-entropies. This is the cleanest formulation because the metric and the loss are aligned, but it requires retraining. For ordinal targets such as BI-RADS, an ordinal-regression head (cumulative-link or proportional-odds parametrisation) over the patient-level label can further exploit the ordering of severity classes; we view this as the most promising direction for future work. In all three cases, side-level metrics should continue to be reported alongside patient-level metrics, so that the contribution of the aggregation rule itself can be isolated from the discriminative ability of the underlying model. Class imbalance and clinical interpretability: a hidden confound The CMMD four-view subset contains 679 malignant patients (96.2%) versus only 27 nonmalignant patients (3.8%). This extreme patient-level imbalance is intrinsic to the design of CMMD as a biopsy-confirmed enrichment cohort, not a screening cohort, and has several important consequences for interpreting our results. First, the high AUC values (>0.95) at side level are partly sustained by the imbalance. The side-level distribution is near-balanced (48.3% malignant, 51.7% nonmalignant), but this apparent balance is misleading: the 683 normal-contralateral sides at BI-RADS 0 belong to patients whose other side is malignant. Models may learn to exploit correlations between bilateral views rather than truly distinguishing benign from malignant tissue. Second, the near-zero patient-level specificity (0.04–0.22 across all configurations) reveals that models have effectively learned to predict malignancy for every patient. With 96.2% of patients being malignant, this strategy yields 96% patient-level accuracy without any discriminative ability for the nonmalignant class. Our per-case analysis confirms this: all 27 nonmalignant patients are misclassified by most models (mean error rate 88.4%), while malignant patients have a mean error rate of only 12.9%. Third, and perhaps most importantly, this imbalance represents another dimension of the “how you measure” problem central to our paper. A study reporting side-level AUC of 0.97 appears to demonstrate strong performance, but the model cannot identify nonmalignant cases at the patient level. The contrast between the near-balanced side-level distribution and the extremely imbalanced patient-level distribution creates an additional layer of reporting ambiguity that compounds with the evaluation granularity effect. We emphasize that our relative comparisons between architectures remain valid: all models face identical class distributions, and the ranking of configurations is unaffected by prevalence. However, the absolute performance numbers should not be interpreted as evidence of clinical utility without careful consideration of the deployment prevalence, which in screening populations would be far lower than the 96.2% malignancy rate in this dataset. To make the prevalence-shift implications concrete, consider a model with sensitivity 0.97 and specificity 0.10 on CMMD (a representative pattern in Table 3). Under the CMMD prior of 96.2% malignancy, the positive predictive value (PPV) is PPV=(0.97·0.962)/(0.97·0.962+0.90·0.038)=0.965, which trivially matches prevalence. Under a screening prior of 0.5% malignancy, the same operating point yields PPV=(0.97·0.005)/(0.97·0.005+0.90·0.995)=0.0054—i.e., only 0.5% of model-flagged patients would actually have malignancy. This is not a flaw of the model so much as a consequence of evaluating at a non-representative prior. The corollary is that AUC and sensitivity/specificity reported on CMMD characterize discriminative ability, not deployment value, and should not be cited as evidence that an AI system is ready for screening use; clinical validation under realistic prevalence is required, in line with the implementation challenges enumerated by Goh et al. [3] and the broader scoping review of Chia et al. [6]. Per-class metrics (sensitivity for the nonmalignant class, in addition to overall AUC) are essential when reporting on enriched cohorts. Limitations Our study has several limitations. First, the CMMD dataset is relatively small (∼700 four-view patients) and severely class-imbalanced at the patient level (96.2% malignant) as a consequence of being a biopsy-confirmed enrichment cohort rather than a screening cohort, which constrains both the statistical power of pairwise comparisons and the evaluation of specificity. Results may not generalize to larger or more balanced datasets where architectural differences could become significant and models could learn to reliably identify nonmalignant cases. Second, we use only ImageNet pretraining; domain-specific pretraining (e.g., on large mammography corpora) might change the backbone ranking. Third, our cross-view attention module uses a fixed architecture; task-specific attention designs might yield better results. Fourth, the five-fold CV protocol limits the minimum achievable p-value of the Wilcoxon signed-rank test to 0.0625; we have partly addressed this in the revised manuscript by adding DeLong’s paired AUC test and bootstrap CIs (Section 3.6.4, Supplementary Table S1), but a future replication should adopt ≥10-fold CV or repeated CV to obtain stronger fold-level p-values. Fifth, the extreme patient-level class imbalance means that no model in our benchmark achieves meaningful nonmalignant-class sensitivity, limiting the clinical conclusions that can be drawn from absolute performance metrics. Sixth, all images were resized to 384×384 to enable a fair like-for-like comparison across pretrained backbones, but this resolution may attenuate features at the scale of microcalcifications. We expect higher-resolution or multi-scale pipelines to improve absolute performance, although the side-vs-patient evaluation gap that is the central object of this paper is a property of the protocol and is not expected to depend on resolution. Seventh, the random horizontal flip in our augmentation pipeline was applied independently per view (rather than coherently per patient), which disrupts the left/right correspondence used by the bilateral fusion arm; this would tend to attenuate the bilateral-fusion benefit we report, so any advantage attributable to bilateral fusion in this benchmark is conservative. We recommend coherent flipping (or no flipping) in future work. Eighth, we have not benchmarked architectures with weakly supervised localization, multi-scale processing, or patch-based pipelines that are increasingly common in modern mammography AI; while these may change the absolute performance ceiling, none of them is expected to alter the side-vs-patient gap, the BI-RADS averaging incompatibility, or the prevalence-shift caveat developed in Section 5.6. Conclusion We have presented a systematic benchmark of deep learning models for four-view mammography classification, evaluating six backbone architectures, three multi-view fusion strategies, and two evaluation granularities on the CMMD dataset. Through 16 model configurations evaluated with five-fold stratified cross-validation on both binary diagnosis and five-class BI-RADS assessment tasks, we have identified several findings with significant implications for the field. Summary of findings The most consequential finding is that evaluation methodology matters more than model choice on this dataset. Side-level AUC exceeds patient-level AUC by 17.5% on average—a gap that dwarfs all architectural differences among CNNs (<3%) and represents the single largest source of variability in our benchmark, yet it reflects a reporting choice rather than a model property. At the architectural level, absolute CNN-vs-CNN AUC differences are small (≤0.030 on pooled per-case scores; the conservative 5-fold Wilcoxon test resolves none of them, while higher-power DeLong paired tests resolve approximately half), and the top five CNNs fall within 0.7% AUC of each other at side level. Meanwhile, ViT-B/16 substantially underperforms all CNN variants by approximately 9% AUC, consistent with the established observation that vision transformers require larger training sets than are typically available in medical imaging. The per-case analysis further reveals that ViT fails on a partially different patient population than CNNs (Jaccard overlap 0.29 vs. 0.42 for CNN-CNN pairs), suggesting architectural complementarity despite inferior individual performance. On the multi-class evaluation front, the prevailing patient-level aggregation rule for BI-RADS—probability averaging combined with the max-of-sides label—is mathematically incompatible with bilaterally asymmetric ground truth, and we therefore advise against using it as a basis for performance comparison; we propose three concrete alternatives in Section 5.5. Two additional methodological concerns emerged from our analysis. Cross-validation proves essential, as single-fold AUC can vary by up to 15% for the same model configuration, meaning that studies relying on single train/test splits cannot provide reliable performance estimates. Furthermore, class imbalance creates a hidden confound: the four-view subset is 96.2% malignant at the patient level but near-balanced at the side level (48.3% malignant), yielding high side-level AUC (>0.95) despite effectively zero ability to identify nonmalignant patients (patient-level specificity <0.22 for all configurations). Per-case error analysis reveals that misclassification aligns with clinical severity—BI-RADS 4 malignant sides have only 8.7% error rate, while low-BI-RADS nonmalignant sides have 38–48% error rate—and no bilaterally nonmalignant patient is correctly classified by all 16 models. Recommendations for the community Based on these findings, we propose several concrete recommendations for mammography classification research. First and foremost, all studies should report both side-level and patient-level metrics, clearly distinguishing between the two and identifying which is used for primary comparison; the patient-level metric should use a label/aggregation rule pair that is consistent with the clinical question being answered (Section 3.6.3). Single train/test splits are insufficient for reliable performance estimation on datasets of this scale, so a minimum of five-fold cross-validation should become standard practice, and AUC differences should be characterized by confidence intervals or DeLong-style paired tests rather than fold-level Wilcoxon alone. Researchers should also characterize class distributions at each evaluation level, since the contrast between side-level and patient-level class balance (51.7% vs. 3.8% nonmalignant in our study) can create misleading impressions of model capability. For multi-class bilateral tasks, decision-level aggregation or per-class max-pooling should replace probability-level averaging (see Section 5.5 for explicit formulas). Finally, specificity should be reported alongside AUC, because in highly imbalanced settings, overall AUC can mask a complete inability to identify the minority class—per-class performance metrics are essential for clinical interpretability, and absolute performance numbers reported on enriched cohorts such as CMMD should not be cited as evidence of clinical utility under screening prevalence (Section 5.6). Future work Several directions emerge from this work. First, evaluating these findings on larger and more balanced mammography datasets (e.g., VinDr-Mammo, DDSM) would determine whether architectural differences become significant at larger scale. Second, investigating class-imbalance mitigation strategies (oversampling, cost-sensitive learning, focal loss) could improve nonmalignant-class sensitivity. Third, the complementary error patterns between CNNs and ViT suggest that heterogeneous ensembles deserve investigation once sufficient data is available. Finally, developing standardized evaluation protocols—potentially as a community-agreed benchmark suite—would directly address the reporting inconsistencies identified in this work. We release our complete benchmark code and per-fold, per-case predictions to support reproducibility and to enable the community to build upon these findings. Supplementary Material supplementary.docx Acknowledgments Open Access funding provided by the Qatar National Library. The author acknowledges the use of Claude (Anthropic) for assistance with manuscript formatting and language checking. The author contributions reported in this manuscript follow the Contributor Roles Taxonomy (CRediT). M.A. Mabrok: conceptualization, methodology, software, formal analysis, investigation, data curation, writing – original draft, writing – review and editing, visualization, project administration, funding acquisition. Funding Statement This work was supported by the International Research Collaboration Co-Fund (IRCC) between Qatar University and the University of Hyogo (Grant No. IRCC-2025-633). Disclosure statement The author reports no conflicts of interest. Data availability statement The CMMD dataset is publicly available through The Cancer Imaging Archive. The complete benchmark code (benchmark_mammography.py), the post-hoc reanalysis script used in this revision (bootstrap_and_delong.py), and the per-fold and per-case predictions for all 16 configurations on both tasks are publicly available at https://github.com/Minds-R-Lab/side-vs-patient-mammography under the MIT license, with a README documenting how to reproduce every table, figure, and statistical test in this manuscript. A Zenodo DOI for archival citation will be minted prior to camera-ready publication. References Associated Data Supplementary Materials supplementary.docx Data Availability Statement The CMMD dataset is publicly available through The Cancer Imaging Archive. The complete benchmark code (benchmark_mammography.py), the post-hoc reanalysis script used in this revision (bootstrap_and_delong.py), and the per-fold and per-case predictions for all 16 configurations on both tasks are publicly available at https://github.com/Minds-R-Lab/side-vs-patient-mammography under the MIT license, with a README documenting how to reproduce every table, figure, and statistical test in this manuscript. A Zenodo DOI for archival citation will be minted prior to camera-ready publication.