Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading
Shuo Tong, Shangde Gao, Ke Liu, Zihang Huang, Hongxia Xu, Haochao Ying✉, Jian Wu · 2025
Read Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading by Shuo Tong, Shangde Gao, Ke Liu, Zihang Huang, Hongxia Xu, Haochao Ying✉, Jian Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00592.txt.
Introduction
Shuo Tong, Shangde Gao, and their co-authors present a method for grading disease from images when some grades are far rarer than others. Their 2025 paper notes that data imbalance and shifts between clinics bias models, which can block clinical use. Their framework distills knowledge from several expert models into one student model, with a focus on uncertainty.
The method separates task general and task specific features, aligning them in shallow and compact ways. At the output stage, each expert's influence is weighted by how unsure it is, so shaky experts count less. The design also copes with experts built from different architectures and with gaps between source and target data. Tests on imbalanced grading tasks showed more reliable results than prior distillation methods.
Catalog record
| CREATOR | Shuo Tong, Shangde Gao, Ke Liu, Zihang Huang, Hongxia Xu, Haochao Ying✉, Jian Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading Shuo Tong, Shangde Gao, Ke Liu, Zihang Huang, Hongxia Xu, Haochao Ying✉, Jian Wu Abstract Automatic disease image grading is a significant application of artificial intelligence for healthcare, enabling faster and more accurate patient assessments. However, domain shifts, which are exacerbated by data imbalance, introduce bias into the model, posing deployment difficulties in clinical applications. To address the problem, we propose a novel U ncertainty-aware M ulti-experts K nowledge D istillation (UMKD) framework to transfer knowledge from multiple expert models to a single student model. Specifically, to extract discriminative features, UMKD decouples task-agnostic and task-specific features with shallow and compact feature alignment in the feature space. At the output space, an uncertainty-aware decoupled distillation (UDD) mechanism dynamically adjusts knowledge transfer weights based on expert model uncertainties, ensuring robust and reliable distillation. Additionally, UMKD also tackles the problems of model architecture heterogeneity and distribution discrepancies between source and target domains, which are inadequately tackled by previous KD approaches. Extensive experiments on histology prostate grading ( SICAPv2 ) and fundus image grading ( APTOS ) demonstrate that UMKD achieves a new state-of-the-art in both source-imbalanced and target-imbalanced scenarios, offering a robust and practical solution for real-world disease image grading. † † footnotetext: *: Equal-contribution authors. ✉: Corresponding author. 1 Introduction Image-driven disease grading systems are pivotal for enhancing clinical decision-making efficiency [ 14 , 22 ] , especially for diabetic retinopathy (DR) and prostate cancer. Early-stage precise grading significantly improves patient prognosis ( e.g. , timely intervention reduces blindness risk by 90% in DR patients) [ 17 , 1 ] . However, traditional grading is largely limited by challenges such as differences in subjective expert judgment and difficulty in identifying subtle pathologic features. Clinical evidence reports a 40% inter-observer variability in Gleason scoring and a misdiagnosis rate of more than 25% for early DR microaneurysms. Recently, medical efficiency has been enhanced by the rapid development of artificial intelligence-based automatic disease grading systems [ 2 , 6 ] . For DR grading, Dai et al. [ 4 ] developed DeepDR to detect early-to-late stages of diabetic retinopathy. Wang et al. [ 20 ] reformulated DR grading as sequence prediction, resolving ambiguous boundary issues effectively. On the other hand, for prostate cancer grading, Morpho-Grader [ 3 ] disentangles glandular morphology from stromal textures. BayeSeg [ 8 ] employs variational inference to separate structural invariants from texture variations, significantly improving model robustness. Despite progress in disease grading methods [ 15 , 1 ] , their deployment in the clinical applications is still limited by domain shifts. Especially, imbalanced data exacerbate domain shifts (differences between source and target domain distributions). As demonstrated in the SICAPv2 dataset (Fig. 1 left), stage III prostate cancer samples constitute 8% of the cohort, leading models overfit to the majority class. This imbalance is equally pronounced in DR grading, where scarce early-stage lesions reduce microaneurysm detection sensitivity by over 30% [ 12 ] . Multi-expert knowledge distillation (MKD) [ 5 , 9 , 7 ] , a technique that improves the generalization of the student for minority class samples by transferring expert model knowledge. Due to its robustness to domain shifts, MKD has been applied to address class imbalance in natural images [ 21 , 13 ] , but its study in disease image grading remains underexplored. In this paper, we propose a novel U ncertainty-aware M ulti-experts K nowledge D istillation (UMKD) framework to tackle the problem of class imbalance. Specifically, to decouple the structural and semantic information of image representation, we design two feature alignment mechanisms: shallow feature alignment (SFA) and compact feature alignment (CFA). SFA generalizes the alignment between expert and student features by multi-scale low-pass filtering, thereby preserving structural information (task-agnostic features) of disease images. CFA maps the features of the expert and student models to a common spherical space, allowing the student model to learn grading-related feature knowledge from each expert. We design an uncertainty-aware decoupled distillation (UDD) mechanism at the output space, automatically detecting uncertainties in the expert model caused by class imbalance. Via uncertainty metrics, the student model dynamically adjusts knowledge transfer weights, reducing bias propagation and ensuring a more robust and reliable knowledge transfer process. Experimental results show that our method significantly outperforms existing multi-expert distillation approaches in fundus and prostate disease image grading tasks. Especially for imbalanced class and heterogeneous models, UMKD achieves much more reliable knowledge transfer as shown in Fig. 1 right. Figure 1 : Domain shifts between source and target data (left) and the performance of methods (right) for sources-imbalanced and target-imbalanced KD tasks. 2 Method Model of the uncertainty-aware multi-expert knowledge distillation. We aim to distill the knowledge from multi-expert models into a target student model for imbalanced disease grading tasks. As shown in Fig. 2 , our framework includes shallow feature alignment (SFA), compact feature alignment (CFA), and uncertainty-aware decoupled distillation (UDD). First, we will introduce the feature alignment loss, which will be reused by the SFA and CFA modules. Next, a detailed description of each module will be provided. Maximum Mean Discrepancy and Reconstruction Loss. To measure the distribution differences between the student’s features and those of each expert for feature alignment in the feature space, we employ Maximum Mean Discrepancy (MMD) [ 5 ] . The MMD distance is calculated as follows: \small\mathcal{L}_{\text{MMD}}=\frac{1}{B}\sum_{t=1}^{N}\left\|\sum_{i=1}^{B}\phi\left(\hat{F}_{T_{t}}^{i}\right)-\sum_{j=1}^{B}\phi\left(\hat{F}_{S}^{j}\right)\right\|_{2}^{2}, (1) where \phi is an explicit mapping function, \hat{F}_{T_{t}} and \hat{F}_{S} represent the features of expert T_{t} and student S after projection, respectively, B is the batch size, and N is the number of experts. Meanwhile, to ensure that the expert models remain unchanged due to privacy constraints, the reconstruction loss \mathcal{L}_{\text{MSE}} is used to measure the changes in the expert models before and after feature alignment: \small\mathcal{L}_{\text{MSE}}=\sum_{t=1}^{N}\left\|F_{T_{t}}-\hat{F}_{T_{t}}\right\|_{2}^{2}, (2) where F_{T_{t}} represents the original features of the expert model T_{t} before alignment, and \hat{F}_{T_{t}} denotes the decoded features of the expert model T_{t} after alignment. By aggregating the alignment loss and the reconstruction loss, the total loss for feature alignment can be expressed as: \mathcal{L}_{\text{FA}}=\mathcal{L}_{\text{MMD}}+\mathcal{L}_{\text{MSE}}. (3) Figure 2 : Model of uncertainty-aware multi-expert knowledge distillation. Shallow Feature Alignment. SFA preserves the main structural information of given images while removing noise and high-frequency details, ensuring consistency in generalized feature learning between expert models and student models [ 9 ] . Specifically, given shallow-layer feature representations between heterogeneous expert-student models, we propose to align features with multi-scale low-pass filtering from a frequency domain. For each expert feature F_{T_{t}} , we adopt traditional average pooling as the low-pass filter and construct multi-scale filters by adjusting different kernel sizes and strides across multiple groups to accommodate different cutoff frequencies. For the m -th group, the frequency-domain features \hat{F}_{T_{t}} via multi-scale low-pass filter (msLF) can be expressed as: \small\hat{F}_{T_{t}}=\operatorname{msLF}(F_{T_{t}})=\Phi(\operatorname{AvgPool}_{k_{m}\times k_{m}}(F_{T_{t}})), (4) where \operatorname{AvgPool}_{k_{m}\times k_{m}} denotes the average pooling function with a kernel size of k_{m}\times k_{m} , and \Phi(\cdot) represents the bilinear interpolation operation. For the student feature F_{S} , we design a learnable low-pass filter, which consists of a multi-scale low-pass filter, a convolutional downsampling module, and a depthwise separable convolution (DSConv). The learnable student feature \hat{F}_{S} in frequency domain is expressed as: \hat{F}_{S}=\operatorname{Conv}_{3\times 3}(\operatorname{Concat}[\operatorname{DownSample}_{s\times s}(F_{S}),\operatorname{msLF}(F_{S})]), (5) where \text{DownSample}_{s\times s} , Concat and \text{Conv}_{3\times 3} indicates the convolutional