Oral squamous cell carcinoma diagnosis: an interpretable deep learning approach using multi-scale pathological images Authors: Yuchen Cui, Xiaoli Gao, Wei Guo, Ying Jin, Yu Liu, Yixuan Qin, Zuomin Wang Published: 2026 ABSTRACT AbstractBackgroundDiagnosis of oral squamous cell carcinoma (OSCC) currently relies heavily on the subjective experience of pathologists. This process is time-consuming and may have diagnostic inconsistencies due to subjective factors. This study aims to develop a novel deep learning model for accurate and interpretable classification of OSCC pathological images across different magnifications, while effectively addressing multi-center data heterogeneity and the difficulty in recognizing minority class samples.MethodsThis study proposes a Pathology-prior Guided Multi-magnification Adaptive Fusion Network (PGMA-Net). The model incorporates pathological prior maps to direct attention toward crucial pathological features, and employs Pathology-Guided Adaptive Fusion Module (PGAFM) for multi-dimensional feature learning. Cross-Magnification Feature Alignment Regularization (CMFAR) is utilized to recognize images across magnifications, while Dynamic Weighted Class Balance (DWCB) improves learning from minority-class samples. Additionally, model integrates domain adaptation techniques to address multi-center data discrepancies. PGMA-Net was trained and validated on two different datasets, and compared with mainstream models. We verified the role of each core module through ablation experiments and visualized the model for interpretation.ResultsOn the internal test set, PGMA-Net achieved good classification performance with an Area Under the receiver operating characteristic Curve (AUC) of 0.9740 and an accuracy of 0.9268, which was significantly superior to other comparative models. On the independent external test set, the model achieved an AUC of 0.9344 after domain adaptation, demonstrating robust cross-center generalization. Visualization results further demonstrated high alignment between the model’s decision regions and pathologists’ diagnostic bases.ConclusionsThe PGMA-Net model achieves efficient and accurate classification of OSCC pathological images from different magnifications and multi-center sources. The model exhibits good generalization and interpretability, providing effective assistance to pathologists in OSCC diagnosis.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1186/s12903-026-08945-9.Keywords: Oral squamous cell carcinoma, Deep learning, Histopathological image, Prior knowledge, Multi-magnification, Multi-center Introduction Oral cancer is the most common malignant tumor in the maxillofacial region and ranks sixth in global prevalence. Its main type is oral squamous cell carcinoma (OSCC), accounting for more than 90% of all cases [1]. In 2022, a total of 389,846 new cases of oral cancer were recorded globally, resulting in 188,438 deaths [2]. Major risk factors include long-term smoking [3], alcohol consumption [4], betel quid chewing [5], and human papillomavirus infection [6], while poor oral hygiene [7] and certain genetic factors [8] may also increase susceptibility. This malignancy typically originates from the epithelial cells of the oral mucosa, most frequently occurring on the lateral borders of the tongue, followed by the floor of the mouth, soft palate, gingiva, buccal mucosa, and hard palate [9]. Early clinical signs are often subtle, presenting as leukoplakia, erythroplakia, or non-healing ulcers. In advanced stages, the tumor may form conspicuous masses, cause facial deformation, lead to cervical lymph node metastasis, and severely impair speech, mastication and swallowing functions [10]. The prognosis of OSCC is highly stage-dependent, with five-year survival rates ranging from 76% to 94% for early-stage cases, compared to only 20% to 40% for those diagnosed at advanced stages [11]. Due to its insidious onset and atypical early symptoms, many patients are diagnosed at intermediate or late stages, resulting in reduced treatment efficacy and poorer survival outcomes [12]. Thus, accurate and early diagnosis is crucial for improving patient survival [13]. Currently, histopathological examination is the gold standard for the diagnosis of OSCC [14]. However, this approach has several limitations [15]. First, the human eye may overlook subtle morphological alterations within complex tissue architectures, potentially leading to missed diagnostic information. Second, variations in interpretation may occur between different pathologists or even by the same pathologist at different times, affecting diagnostic consistency and reproducibility [16]. Moreover, histopathological diagnosis is often time-consuming, particularly with large sample volumes, which may result in diagnostic delays that impede timely clinical decision-making [17]. With the rapid advancement of artificial intelligence, there is growing potential for automated analysis in histopathology [18–21]. Deep learning models emulate the hierarchical structure of neural networks in the human brain, enabling automatic discriminative feature extraction from large datasets and facilitating complex pattern recognition and classification tasks [22]. These technologies enhance the accuracy and efficiency of pathological image analysis, promote diagnostic standardization, reduce the burden of manual interpretation, and provide more reliable objective support for early diagnosis and precision treatment of OSCC [23]. In recent years, researchers have explored issues such as model optimization, feature fusion, and sample adaptability, driving the advancement of intelligent diagnosis of OSCC. In terms of model architecture innovation, multi-module collaborative design has become a pivotal approach for enhancing performance. Pandit et al. [24] proposed a hybrid deep learning model integrating Gaussian filtering, MCD-BIRCH segmentation, and CAA-Bi-LSTM+LinkNet, achieving an F1-score of 0.918 using 70% of the data for training. Shah et al. [25] designed OCANet with a dual attention mechanism and achieved a classification accuracy (ACC) of 0.909. Regarding feature fusion, several studies have combined features extracted by convolutional neural networks (CNNs) with features extracted by traditional image processing algorithms to improve model performance. Ahmad et al. [26] fused CNN-based and conventional features, using an SVM classifier to achieve an Area Under the receiver operating characteristic Curve (AUC) of 0.968 in early OSCC diagnosis. Haq et al. [27] integrated Gabor texture features with features from ResNet50 and employed a CatBoost classifier, achieving an AUC of 0.949 in OSCC histopathological image analysis. To address data and generalization challenges, multiple tailored solutions have been proposed. Soni et al. [28] introduced a dual-attention mechanism to enhance EfficientNetB0, achieving an ACC of 0.911 in benign-malignant classification. Rahman et al. [29] applied transfer learning based on AlexNet for OSCC biopsy image classification, achieving a test ACC of 0.901. Guo et al. [30] proposed a dual-feature extraction prototypical network with a custom loss function, enabling efficient OSCC diagnosis with limited samples at an ACC of 0.860. These studies have laid an important technical foundation for intelligent OSCC diagnosis. However, existing research still faces several critical challenges. On one hand, data-driven feature extraction fails to effectively incorporate pathological knowledge, particularly under the guidance of key information such as nuclear boundaries and tissue structures. This leads to attention drift in regions with staining variations or blurred boundaries, making it difficult to accurately capture critical pathological features. On the other hand, existing models often analyze images at a single magnification, while pathological features vary across different magnifications. Current methods fail to effectively integrate such multi-scale information, making it difficult to accurately recognize images at different magnifications. Furthermore, existing approaches lack adequate adaptability to class imbalance and domain shift issues, causing minority class samples to be underfitted and resulting in significant performance degradation on external datasets. This limits their effectiveness in adapting to diverse data sources and imaging conditions. To overcome these problems, our research proposes a novel deep learning model, the Pathology-prior Guided Multi-magnification Adaptive Fusion Network (PGMA-Net). It enhances the diagnostic performance for OSCC and cross-center generalization ability by incorporating prior knowledge, integrating multi-scale pathological features, and combining domain adaptation mechanisms. Materials and methods Dataset processing Dataset description This study utilizes two publicly available datasets of oral histopathological images, both sourced from the Mendeley data platform. The training data were obtained from the OSCC histopathology image dataset provided by Rahman et al. [31]. Hematoxylin and Eosin (H&E) stained tissue slides from 230 patients were utilized in this study. These slides have been collected, prepared, and cataloged by medical experts and were imaged with a Leica ICC50 HD microscope. This dataset consists of 1,224 oral histopathology images, including samples at both 100× and 400× magnifications. At 100× magnification, the dataset includes 89 normal and 439 OSCC images; at 400× magnification, it includes 201 normal and 495 OSCC images, respectively. The distribution of images within the dataset is summarized in Table 1, and representative examples of both normal and OSCC tissue are shown in Fig. 1. Table 1Number of images in the training dataset at different magnificationsMagnificationNormalOSCCTotal100×89439528400×201495696Total2909341224 Number of images in the training dataset at different magnifications Representative histopathological images from the training dataset. a, b Normal oral tissue (100×magnification); c, d Normal oral tissue (400×magnification); e, f OSCC tissue (100×magnification); g, h OSCC tissue (400×magnification) The external test data were derived from clinically confirmed samples obtained from the archives of the Department of Oral and Maxillofacial Pathology at the Federal University of Uberlândia, from patients diagnosed with OSCC [32]. All images were captured at 20× magnification using an Aperio AT2 slide scanner and were randomly cropped from whole-slide images (WSIs). The dataset comprises 1,020 images of size 640 × 640 pixels, including two categories: 451 images containing OSCC tumor regions and 569 images of normal tissue without tumor regions. Each patch has been manually annotated by certified pathologists to segment tumor areas, followed by rigorous validation. Representative examples of both normal and OSCC tissues are shown in Fig. 2. Representative histopathological images from the external test dataset. a, b Normal oral tissue (20× magnification); c, d OSCC tissue (20× magnification) Dataset splitting A stratified sampling strategy was employed for training dataset splitting. Using “category-magnification” as a joint label, the dataset was divided into training (856 images), validation (245 images), and internal test (123 images) sets at a ratio of 7:2:1, with a fixed random seed. This approach ensured that the four subgroups (Normal-100×, OSCC-100×, Normal-400×, and OSCC-400×) maintained consistent distribution proportions with the original dataset across all subsets. Data augmentation During the data preprocessing stage, to improve the model’s generalization ability and mitigate potential overfitting caused by limited data, multiple data augmentation techniques were applied. These included random rotation (ranging from − 45° to + 45°), horizontal and vertical flipping (each at 40% probability), linear adjustments of brightness and contrast (from 80% to 120%), shifts in hue and saturation (± 20 units), addition of Gaussian noise (with standard deviation σ ∈ [0, 12.75]) and Gaussian blur (σ ∈ [0.5, 1.0] pixels), and CoarseDropout for regional occlusion (dropout probability 3%–15%, region size 2%–5%). All transformations were applied in random order, maximizing diversity while preserving semantic validity. To relieve class imbalance, each normal tissue image in the training set was augmented to generate 6 additional samples (7 in total), while each OSCC tissue image was augmented to generate 4 additional samples (5 in total). After augmentation, the training set contained 1,421 normal tissue images and 3,265 OSCC images. No additional augmentation was applied to the validation and test sets, which retained only the original images and their corresponding prior maps. PGMA-Net architecture This study proposes a novel deep learning model, PGMA-Net, for the diagnosis of OSCC pathological images. The model utilizes a pretrained ResNet50 as its backbone [33], preserving its feature extraction layers to capture multi-scale semantic information from the images. In addition, PGMA-Net innovatively integrates several components: a prior graph generation mechanism, the Pathology-Guided Adaptive Fusion Module (PGAFM), Cross-Magnification Feature Alignment Regularization (CMFAR), and the Dynamic Weighted Class Balance (DWCB) loss strategy. The overall architecture of PGMA-Net is shown in Fig. 3. Architecture diagram of the PGMA-Net To improve the model’s focus on critical regions in pathological images, we introduce a prior map generation mechanism. Specifically, the original color image is converted to grayscale, and Sobel operators are applied to compute gradient magnitudes along the x and y directions. The results are fused and normalized to the 0–255 range, generating a single-channel prior map that emphasizes cell boundaries and texture features, as shown in Fig. 4. This prior map is incorporated as an auxiliary input alongside the original image, guiding the model toward potential lesion areas and thereby enhancing classification robustness and accuracy. Prior maps generated from histopathological images. a-d Original H&E-stained images: a Normal (100×magnification); b Normal (400×magnification); c OSCC (100×magnification); d OSCC (400×magnification). e-h Corresponding prior maps: e from a; f from b; g from c; h from d For feature fusion, to address the multi-level diagnostic requirements of OSCC histopathological images, we propose the PGAFM, which integrates channel and spatial attention mechanisms. The channel attention branch utilizes multi-scale pooling (global, regional, and detail) to effectively integrate multi-level information, ranging from global tissue structures to cellular details. A multilayer perceptron (MLP) is then used to adaptively assign channel weights. Specifically, adaptive average pooling is used for the global receptive field, a 3 × 3 average pooling window is applied to capture regional pathological structures, and a 2 × 2 average pooling window is adopted to extract details. The spatial attention branch aggregates average and max pooling across channels, fuses them via convolution to generate a spatial attention map, and then multiplies it with pathological priors to highlight key lesion regions. The two branches are fused through a learnable weight parameter, followed by convolution, batch normalization, and ReLU activation for refined feature representation. By explicitly embedding pathology-specific priors, the PGAFM strengthens the model’s capability to recognize complex pathological structures. In the analysis of histopathological images, the heterogeneity in feature distribution caused by different magnifications severely limits the model’s generalization performance across multi-center and multi-protocol data. To address this, we propose the CMFAR method. This method introduces a feature space alignment constraint, encouraging the model to learn essential feature representations that are invariant to changes in magnification. Specifically, the 2048-dimensional pooled features are mapped to a 128-dimensional projection space, where we compute a mean feature vector for each class at each magnification. The Mean Squared Error (MSE) between the mean vectors of the same class from different magnifications is then minimized as a regularization loss term. This loss is jointly optimized with the primary classification loss, significantly improving the model’s generalization ability across different magnifications while maintaining classification performance. To address the class imbalance issue commonly found in OSCC datasets, we propose the DWCB loss, which adaptively adjusts class weights according to class-wise recognition performance, thereby enhancing the model’s classificationand generalization ability on imbalanced data. The DWCB loss is based on cross-entropy loss, and at the end of each training epoch, new weights are computed based on the recall rate for each class on the training set. Classes with lower recall rates are assigned higher weights, alleviating the data imbalance issue. To ensure stable weight updates, a momentum mechanism is used for smoothing the weight adjustments, preventing fluctuations in weights during training. This approach enables adaptive adjustment without the need for pre-computing class distributions. The DWCB loss is combined with the CMFAR loss to form the total loss function, effectively balancing classification performance and feature representation alignment. Model training and validation Model training Our model was developed using the PyTorch framework and trained on an NVIDIA GPU. A batch size of 8 was utilized for the training process. We adopted the Adam optimizer, setting both the initial learning rate and weight decay to 1×10⁻⁵. Furthermore, a StepLR scheduler was implemented to modify the learning rate, which was reduced by half every 10 epochs. To mitigate overfitting, an early stopping mechanism was applied with a patience of 10 and a minimum improvement threshold of 0.001. Model evaluation A comprehensive evaluation of the model’s performance was evaluated using both quantitative and qualitative metrics. We employed five key metrics to evaluate its classification performance: AUC, accuracy, precision, recall, and F1-score. Additionally, Receiver Operating Characteristic (ROC) curves, calibration curves, precision-recall curves (PR), decision curves analysis (DCA) and confusion matrices were generated to visually illustrate the model’s performance across varying decision thresholds and its ability to differentiate between classes. Ablation study To validate the effectiveness of each key module in PGMA-Net, a systematic ablation study was conducted. By removing specific modules while keeping the same input preprocessing and training hyperparameters, we compared the performance of various model variants across multiple performance metrics, thereby quantitatively assessing the role of individual modules in the final performance. Five model variants were compared in the experiments: Complete Model: All proposed modules are enabled. Complete Model: All proposed modules are enabled. No PGAFM: The PGAFM is replaced by a standard convolutional block to evaluate its role in feature enhancement. No PGAFM: The PGAFM is replaced by a standard convolutional block to evaluate its role in feature enhancement. No CMFAR: The CMFAR loss is removed to examine its effect on feature consistency across magnifications. No CMFAR: The CMFAR loss is removed to examine its effect on feature consistency across magnifications. No DWCB: The DWCB is replaced by the standard cross-entropy loss to verify its efficacy in mitigating class imbalance. No DWCB: The DWCB is replaced by the standard cross-entropy loss to verify its efficacy in mitigating class imbalance. ResNet50: All proposed modules (PGAFM, CMFAR, DWCB) are removed, resulting in a standard ResNet50 to quantify the overall performance improvement of PGMA-Net. ResNet50: All proposed modules (PGAFM, CMFAR, DWCB) are removed, resulting in a standard ResNet50 to quantify the overall performance improvement of PGMA-Net. External testing Guided by clinical requirements, we prespecified a recall of > 0.90 for OSCC on the internal validation set. Using this as the criterion, we maximized specificity while ensuring clinically high recall to select the optimal classification threshold. This fixed threshold was then applied to the external test set. To ensure objectivity and impartiality in model evaluation, we employed a domain adaptation strategy. Specifically, 500 samples were randomly selected from the independent external dataset, with 400 used to construct the target-domain fine-tuning set and 100 reserved as a strictly isolated external test set. To address domain shift across centers, we employed a parameter-efficient fine-tuning method: the backbone weights of the pretrained model were frozen, and the scaling (γ) and shift (β) parameters within the batch normalization (BN) layers were updated to align with the target domain distribution. By updating only a small number of parameters, this approach significantly reduced computational and memory requirements during fine-tuning and improved training efficiency. The external test set was excluded from all training and adaptation procedures and used solely for evaluating the model’s generalization ability on unseen data, thereby ensuring reliable and unbiased assessment. During testing, we applied a test-time augmentation (TTA) strategy to improve the model’s robustness to image distortions and color variations. For each input image, four augmented versions were generated, including horizontal flipping, vertical flipping, color jittering (brightness and contrast adjustment factor of 1.1), and the original image. The model produced independent predictions for each version, and the final output was obtained by averaging the probabilities. To evaluate the effectiveness of the domain adaptation methods, we additionally assessed two other domain shift mitigation strategies: one was color normalization alone without fine-tuning the BN layers, and the other was BN fine-tuning alone without TTA. Model visualization and interpretation To enhance the interpretability of model decisions, we employed two independent visualization techniques: Uniform Manifold Approximation and Projection (UMAP) for feature dimensionality reduction, and Gradient-weighted Class Activation Mapping (Grad-CAM) for class activation heatmaps. For UMAP visualization, we used the projection module to compress the extracted features into 128-dimensional vectors, which served as the feature matrix for UMAP analysis. These 128-dimensional features were then input into UMAP and reduced to two-dimensional space, allowing intuitive visualization of the clustering and distribution patterns of normal and OSCC samples. For Grad-CAM visualization, we selected the last convolutional layer of the PGAFM module as the target layer. The gradient of the target class with respect to this layer’s feature maps was computed to obtain importance weights for each channel, which were then fused to generate the final spatial heatmap. Results Comparison with multiple models To comprehensively evaluate the performance of PGMA-Net, we compared it with several representative deep learning models, including AlexNet, ResNet18, DenseNet121, EfficientNet-B0, and Vision Transformer (ViT-B/16). In internal testing, the model demonstrated the best classification performance, with an AUC of 0.9740, an accuracy of 0.9268, a precision of 0.8898, a recall of 0.9164, and an F1-Score of 0.9019, significantly outperforming the other comparative models. Although AlexNet and ViT-B/16 also showed good performance with AUC values of 0.9347 and 0.8991, respectively, their other metrics were lower than those of PGMA-Net. ResNet18 and EfficientNet-B0 performed moderately, with AUC values of 0.8672 and 0.8250, respectively. DenseNet121 showed the weakest performance, with an AUC of 0.7704. These results demonstrate that PGMA-Net achieved the best and most stable classification performance in distinguishing between normal tissue and OSCC. Detailed performance metrics of each model on the internal test set are shown in Table 2, and the ROC curves are shown in Fig. 5(a), where the curve of PGMA-Net is closest to the top-left corner, further validating its superior classification capability. PGMA-Net also achieved the best overall calibration performance with a Brier Score of 0.0509, and the calibration curves are shown in Fig. 5(b). Additionally, the precision-recall curves in Fig. 5(c) show PGMA-Net achieving the highest AUPRC of 0.9891, indicating its optimal precision-recall balance, especially in imbalanced datasets. The decision curve analysis in Fig. 5(d) shows PGMA-Net yielding the highest net benefit across threshold probabilities, confirming its great clinical utility. Table 2Model performance metrics on the internal test setModelAUCAccuracyPrecisionRecallF1-ScorePGMA-Net0.97400.92680.88980.91640.9019ResNet180.86720.82110.75210.76380.7575EfficientNet-B00.82500.79670.71590.68820.6994AlexNet0.93470.87800.83290.82480.8287DenseNet1210.77040.74800.65750.66820.6621ViT-B/160.89910.84550.78420.80360.7929 Model performance metrics on the internal test set Performance of different models on the internal test set. a ROC curves; b calibration curves; c precision-recall curves; and d DCA curves The confusion matrices of different models on the internal test set are shown in Fig. 6. Compared with AlexNet, DenseNet121, EfficientNet-B0, ResNet18, and ViT-B/16, PGMA-Net exhibits fewer misclassifications, with its predictions aligning with the true labels in the vast majority of cases. This indicates that the model achieves higher classification accuracy for OSCC and normal tissue and represents the best model among those evaluated. Confusion matrices for different models on the internal test set Ablation study The results of the ablation study are shown in Table 3. The complete model significantly outperforms the ablation variants and the ResNet50 baseline model on all metrics, achieving an AUC of 0.9802 and an accuracy of 0.9431 on the validation set. Removal of any single module, such as PGAFM, CMFAR, or DWCB, resulted in varying degrees of performance degradation. Among them, the removal of the CMFAR module led to the most significant decline, with AUC decreasing by 5.71% points and accuracy decreasing by 13.9% points, highlighting the crucial role of this module in feature fusion and information integration. The experiments validate that each module contributes positively to improving model performance, and the complete model achieves optimal classification performance through the effective integration of all module strengths. Furthermore, the ROC curves also demonstrate that removal of any module leads to a performance decline, as shown in Fig. 7. Table 3Performance comparison of ablated modelsConfigAUCAccuracyPrecisionRecallF1-ScorePGMA-Net0.98020.94310.91090.93890.9237No-PGAFM0.94920.86940.81660.83120.8234No-CMFAR0.92310.80410.75430.83000.7694No-DWCB0.92700.84080.78020.82440.7968ResNet500.88420.82450.75770.77800.7666 Performance comparison of ablated models ROC curves of ablated models External testing As shown in Table 4, without domain adaptation, PGMA-Net exhibited weak overall performance on the external test set, with an AUC of 0.7095 and an accuracy of 0.6373, indicating significant domain shift issues. After applying the domain adaptation strategy, which involves freezing the backbone network and training only the BN layers, the model’s performance improved significantly. The AUC improved to 0.9344, and the accuracy improved to 0.8762. This demonstrates that the strategy not only enhanced the model’s classification and recognition capabilities but also improved the balance between positive and negative sample classification. The ROC curve in Fig. 8(a) further validates the effectiveness of the proposed method, while enhanced probability calibration in Fig. 8(b), an AUPRC of 0.9459 in Fig. 8(c), and greater net benefit across a broader threshold range in Fig. 8(d) collectively underscore PGMA-Net’s improved predictive reliability and clinical utility on OSCC datasets after domain adaptation. Table 4Performance comparison before and after domain adaptation on external test setModelAUCAccuracyPrecisionRecallF1-ScorePre-domain adaptation0.70950.63730.65360.65670.6369Post-domain adaptation0.93440.87620.90480.89070.8977 Performance comparison before and after domain adaptation on external test set Performance of PGMA-Net on the external test set before and after domain adaptation. a ROC curves; b calibration curves; c precision-recall curves; and d DCA curves We evaluated the model efficacy when using color normalization alone as a domain shift mitigation strategy. After applying color normalization alone, PGMA-Net achieved an AUC of 0.7294 and an accuracy of 0.6952 on the external test set. Compared to the performance without domain adaptation, color normalization partially alleviated the domain shift problem, albeit with limited performance improvement. Detailed performance of the model is provided in Supplementary Tables 1 and Supplementary Fig. 1. In addition, we evaluated the performance of the model using BN fine-tuning alone without TTA as a domain adaptation strategy. The model after domain adaptation without TTA achieved an AUC of 0.9261 and an accuracy of 0.8667 on the external test set. Compared with the results using TTA, the model without TTA showed a slightly decreased performance. Detailed performance of the model is presented in Supplementary Tables 2 and Supplementary Fig. 2. This indicates that the model maintained good external test performance without TTA, demonstrating that its generalization ability was largely driven by the proposed PGMA-Net architecture and BN fine-tuning-based domain adaptation. TTA served only as an inference-stage robustness enhancement strategy, improving stability against input perturbations by reducing prediction variance without using additional training data or updating model parameters. Visualization results Evaluation of pathological feature separation with UMAP Figure 9(a) and (b) show the UMAP visualizations of the model in the parameter initialization state and after training completion. In the initialization state, Normal and OSCC samples exhibit substantial overlap in the feature space, with mixed cross-category feature points and blurred boundaries, making effective separation difficult. After training, however, the feature points of the two classes become more dispersed and form distinct clusters, with much clearer inter-class boundaries and improved separation. These results demonstrate that the trained model is able to capture the essential differences between the two classes more effectively, thus substantially enhancing the discriminative power of the learned features. Visualization of feature distributions using UMAP. a untrained and b trained PGMA-Net models OSCC suspected region detection based on Grad-CAM As shown in Fig. 10, this study uses Grad-CAM technology to visualize the detection of OSCC lesions. By extracting gradient information from the last convolutional layer of the deep neural network, class activation heatmaps are generated, clearly identifying tissue regions closely associated with tumor diagnosis. The high-activation areas in the heatmap (indicated in warm colors such as red and yellow) closely match the malignant lesion regions annotated by pathologists (indicated in blue). The results visually validate the effectiveness of Grad-CAM in identifying OSCC lesions, demonstrating that the model accurately focuses on diagnostically critical regions, with its attention showing high consistency with areas emphasized by pathologists. Sample images from the external dataset. a, e Original images of OSCC; b, f Grad-CAM attention heatmaps; c, g Overlay of model-predicted regions on the original images; d, h Pathologist annotations with tumor regions highlighted in blue Furthermore, we present the heatmaps of true negative, false positive, and false negative samples in Fig. 11. Through systematic analysis of the Grad-CAM heatmaps, we examined the heatmaps of misclassified samples. The results indicate that inflammatory cell nests formed by dense aggregations of lymphocytes and plasma cells, characterized by hyperchromatic and densely packed nuclei, may increase confusion, occasionally leading the model to misclassify inflammatory regions as OSCC. In false negative samples, sparsely distributed tumor cells scattered as single cells or small clusters are visually less prominent. As the model tends to focus on dense, well-defined cell clusters while exhibiting insufficient sensitivity to sparse signals, small tumor regions may sometimes be overlooked. Sample images from the external dataset. a-c True negative sample: a Original image; b Grad-CAM attention heatmap; c Overlay of model-predicted regions on the original images. d-f False negative sample: d Original image; e Grad-CAM attention heatmap; f Overlay of model-predicted regions on the original images. g-i False positive sample: g Original image; h Grad-CAM attention heatmap; i Overlay of model-predicted regions on the original images Discussion This study proposes a novel deep learning model, PGMA-Net, for the automated recognition and classification of OSCC histopathological images. By integrating computer vision techniques with pathological prior knowledge, the model innovatively introduces several key technologies, including a pathology-prior guided mechanism, multi-scale feature fusion, cross-magnification feature alignment, and dynamic class balancing, to construct a high-performance diagnostic system for OSCC. The results demonstrate that PGMA-Net not only exhibits excellent classification accuracy and generalization ability but also offers good interpretability and clinical applicability. It provides an effective auxiliary tool for the precise diagnosis of OSCC and holds high clinical value. The effectiveness of PGMA-Net stems from several core modules designed specifically for OSCC pathological images. In histopathological images, OSCC regions often exhibit complex morphological features and spatial distribution patterns, making it difficult for conventional methods to accurately capture key pathological changes such as nuclear atypia and disordered tissue architecture [34, 35]. PGMA-Net addresses this by introducing a prior map generation mechanism based on the Sobel operator, which effectively simulates the diagnostic logic of pathologists. This mechanism highlights critical pathological features, such as cell boundaries and tissue structural textures, by computing gradient magnitude, generating a prior map that clearly represent morphology and regional structures [36–38]. Unlike traditional models that rely solely on original images for feature extraction, PGMA-Net processes the prior map in parallel with the original image. This parallel processing strategy progressively guides the network to focus on regions highly relevant to pathological diagnosis, effectively bridging the gap between data-driven features and pathological prior knowledge. This mechanism enhances robustness to staining variations and blurred boundaries, reduces attention drift, and enables the model to maintain precise feature perception even in highly heterogeneous OSCC images, making the overall diagnostic process more aligned with clinical pathological practice. The PGAFM integrates channel attention, spatial attention, and the pathological prior map to enhance the model’s perception and recognition of complex pathological structures. It enables the network to simultaneously capture global tissue architecture, meso-scale regional patterns, and fine details. Further analysis of the convolutional weights in the spatial attention module showed that the relative contributions of the average-pooling and max-pooling branches were 46.63% and 53.37%, respectively, suggesting a slight preference for max-pooled responses when generating the spatial attention map. This indicates that, while maintaining stable global spatial responses, the model may tend to place slightly greater emphasis on locally salient features for highlighting key lesion regions. In addition to structural feature analysis, magnification differences present a significant challenge in tissue pathology image analysis [39]. In pathological diagnosis, images at different magnifications provide complementary diagnostic information. Low magnification images are helpful for observing the overall structure of tissues and the macroscopic patterns of lesions, while high magnification images are used to examine fine cellular morphological features [40]. However, most existing models are typically trained and optimized at a fixed magnification, with the learned feature representations coupled to that specific magnification. When confronted with unseen magnifications, the model’s performance often degrades. This occurs because the feature distribution varies across magnifications, leading to shifts in texture and detail. This discrepancy highlights the model’s limited generalization ability. To address this core issue, the CMFAR module helps the model to study the essential characteristics that are not affected by magnification. This enables the model to accurately recognize tissue pathology slides at different magnifications, significantly enhancing its generalization ability in diverse data environments. To address the class imbalance commonly observed in the OSCC dataset, this study proposes a DWCB loss strategy that adopts a performance feedback-based adaptive adjustment mechanism. Instead of relying solely on sample distribution, the strategy emphasizes hard-to-classify and minority-class samples. By dynamically adjusting loss weights during training, it significantly enhances the sensitivity to clinically critical cases, providing an effective technical approach for the screening of OSCC. When the model is applied to new clinical scenarios, different imaging devices, or cross-center data, domain shift may occur. Therefore, in practical applications, appropriate domain adaptation should be performed based on the distribution of the target data. We evaluated the generalization ability of the model using external data, and the domain adaptation framework effectively mitigated the domain shift problem in cross-center applications. Due to differences across medical centers in slide preparation procedures, staining techniques, imaging equipment, and scanning systems, pathological images often exhibit inter-domain distribution disparities, such as heterogeneity in color, texture, and noise levels, which limit the clinical deployment of models [41, 42]. Although color normalization can adjust low-level pixel distributions of input images, such as brightness, contrast, or staining intensity, it has limited effect on high-level semantic features, including tissue texture, morphological structure, and contextual information. Therefore, in cross-center applications, color normalization alone may be insufficient to effectively mitigate domain shift. In contrast, fine-tuning the batch normalization (BN) layers adapts input features by adjusting the statistical distributions (running mean and standard deviation) of convolutional features along with learnable scaling and shifting parameters, thereby more directly improving model performance on external data. Experimental results demonstrate that BN fine-tuning outperforms color normalization alone in enhancing model generalization and classification accuracy. BN layers are specifically designed to characterize feature distributions by normalizing with stored running statistics and applying learnable affine parameters (γ and β) to scale and shift the normalized activations. When domain shift occurs, fine-tuning only the BN parameters provides a direct and efficient strategy to calibrate the convolutional feature distribution to the target domain, thereby improving model performance on data from new institutions. This approach not only substantially reduces computational costs but also enables the model to rapidly adapt to the imaging characteristics of different institutions. The results demonstrate that the model maintains robust classification performance and generalization ability on external data, showing its robustness across different magnifications and whole-slide images. This method prevents overfitting while preserving the backbone representational capacity of the original model, significantly improving its compatibility in multi-center environments. Thus, it provides a feasible pathway for enhancing the efficiency and consistency of pathological diagnosis, as well as a generalized solution for cross-center and cross-device pathological AI applications. The performance advantages and innovative features exhibited by PGMA-Net highlight its potential in the clinical diagnosis of OSCC. The model can serve as a rapid screening tool to detect suspicious lesion regions, thereby improving diagnostic efficiency, or as a second reader to provide objective and reproducible results, helping reduce inter-observer variability caused by subjective factors. Moreover, in resource-limited settings, the model can deliver expert-level diagnostic support, alleviating the shortage of professional pathologists. This study has several limitations that should be acknowledged. First, the current training dataset remains limited, particularly in terms of rare pathological subtypes. We aim to incorporate more cases covering different stages of disease progression and rare pathological types to further improve the model’s ability to recognize OSCC. Second, all experiments in this study were conducted based on publicly available datasets, and no dataset specifically collected for this study was used. Therefore, the generalization ability of the model across diverse clinical application scenarios requires further validation using more clinical data. Third, although the model performs well overall, its recall on the external test set still needs to be improved. Considering that OSCC is a malignant tumor and missed diagnosis may delay clinical treatment, improving recall will help enhance the model’s applicability in clinical practice. Future research could further improve recall by optimizing threshold selection, introducing more domain adaptation techniques, or incorporating more hard examples. Conclusion This study successfully developed a novel deep learning model, PGMA-Net, for the classification of OSCC histopathology images. The model effectively emulates the diagnostic logic of pathologists by focusing on key lesion regions and integrating multi-dimensional features. It addresses the issue of inaccurate recognition of minority class samples adapts to differences in image magnification and scanning devices, improving classification accuracy and robustness. Validated on both internal and multi-center external datasets, PGMA-Net demonstrated not only good diagnostic performance but also robust generalization ability and clinical utility. As a reliable and efficient solution for computer-aided diagnosis of OSCC, PGMA-Net holds promise for assisting pathologists in improving diagnostic consistency and workflow efficiency in clinical practice, indicating potential for translational application. Supplementary Information Supplementary Material 1: Supplementary Table 1 Performance of PGMA-Net with color normalization on the external test set. Supplementary Table 2 Performance of PGMA-Net after domain adaptation without TTA on the external test set. Supplementary Fig. 1 Performance of PGMA-Net with color normalization on the external test set. Supplementary Fig. 2 Performance of PGMA-Net after domain adaptation without TTA on the external test set. Abbreviations Oral squamous cell carcinoma Pathology-prior guided multi-magnification adaptive fusion network Pathology-guided adaptive fusion module Cross-magnification feature alignment regularization Dynamic weighted class balance Multilayer perceptron Mean squared error Area under the receiver operating characteristic curve Classification accuracy Receiver operating characteristic Precision-recall curves Decision curves analysis Batch normalization Hematoxylin and eosin Whole-slide images Uniform manifold approximation and projection Gradient-weighted class activation mapping Authors' contributions YC and ZW conceptualized the study and designed the methodology. YC, XG, YL and ZW participated in the data curation and formal analysis. YC, XG, WG, YJ, YL and ZW performed the software implementation, validation, and visualization. YC, XG, YJ, YL, YQ and ZW contributed to the writing - review & editing. Supervision and resources were in charge of ZW. The original manuscript draft was written by YC. All authors reviewed, edited, and approved the final manuscript. Funding This study was funded by Natural Science Foundation of Beijing Municipality [L232104]. Data availability The datasets supporting the conclusions of this article are available in the Mendeley Data repository via the following links: https://data.mendeley.com/datasets/9bsc36jyrt/1 and https://data.mendeley.com/datasets/ftmp4cvtmb/1. Declarations Ethics approval and consent to participate This study was conducted in accordance with the principles of the Declaration of Helsinki. Ethics approvals for the datasets were granted by the Research Ethics Committee of the Federal University of Uberlândia, Brazil under approval number CAAE 15188713.9.0000.5152, and by the Ethical Committee of the Institute of Advanced Study in Science and Technology, Guwahati, India with reference number IEC (HS)/IASST/1082/2014-15/2. Consent for publication The data used are from CC BY 4.0 licensed public datasets that permits publication with attribution. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Supplementary Materials Supplementary Material 1: Supplementary Table 1 Performance of PGMA-Net with color normalization on the external test set. Supplementary Table 2 Performance of PGMA-Net after domain adaptation without TTA on the external test set. Supplementary Fig. 1 Performance of PGMA-Net with color normalization on the external test set. Supplementary Fig. 2 Performance of PGMA-Net after domain adaptation without TTA on the external test set. Data Availability Statement The datasets supporting the conclusions of this article are available in the Mendeley Data repository via the following links: https://data.mendeley.com/datasets/9bsc36jyrt/1 and https://data.mendeley.com/datasets/ftmp4cvtmb/1.