Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation
Shutong Duan, Jingyun Yang, Yang Tan, Guoqing Zhang, Yang Li 2 , Xiao-Ping Zhang · 2025
Read Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation by Shutong Duan, Jingyun Yang, Yang Tan, Guoqing Zhang, Yang Li 2 , Xiao-Ping Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02340.txt.
Introduction
Shutong Duan and co-authors published a computing paper in 2025 on stopping transfer learning from hurting medical image segmentation. When a model trained on one task is reused on another, some image regions can get worse, a harm called negative transfer. Past fixes target sorting or regression tasks and treat the image as uniform. The team instead maps risk pixel by pixel.
Their map scores how hard each pixel is to transfer and how likely harm is there. During fine-tuning, a weighted loss points the model at the risky regions, scaled to handle class imbalance. Tests on brain datasets gained several points over baselines and avoided negative transfer. A few-shot test also held up.
Catalog record
| CREATOR | Shutong Duan, Jingyun Yang, Yang Tan, Guoqing Zhang, Yang Li 2 , Xiao-Ping Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-02340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation Thanks: 2 Corresponding author. Shutong Duan, Jingyun Yang, Yang Tan, Guoqing Zhang, Yang Li 2 , Xiao-Ping Zhang Abstract How to mitigate negative transfer in transfer learning is a long-standing and challenging issue, especially in the application of medical image segmentation. Existing methods for reducing negative transfer focuses on classification or regression tasks, ignoring the non-uniform negative transfer risk in different image regions. In this work, we propose a simple yet effective weighted fine-tuning method that directs the model’s attention towards regions with significant transfer risk for medical semantic segmentation. Specifically, we compute a transferability-guided transfer risk map to quantify the transfer hardness for each pixel and the potential risks of negative transfer. During the fine-tuning phase, we introduce a map-weighted loss function, normalized with image foreground size to counter class imbalance. Extensive experiments on brain segmentation datasets show our method significantly improves the target task performance, with gains of 4.37% on FeTS2021 and 1.81% on iSeg-2019, avoiding negative transfer across modalities and tasks. Meanwhile, a 2.9% gain under a few-shot scenario validates the robustness of our approach. I Introduction For medical segmentation tasks, acquiring ground-truth labels is challenging because detailed annotation of large 3D images is both time-consuming and requires expert input [ 1 , 2 ] . To compensate the lack of labeled training samples, transfer learning has become a common approach for medical semantic segmentation [ 3 , 4 ] , leveraging the knowledge from a high resource domain to improve the performance on low-resource tasks [ 5 ] . However, the effectiveness of transfer learning is not guaranteed: low task and domain correlation can impair the target domain performance. This phenomenon is known as negative transfer [ 6 ] . Due to the inherent heterogeneity in medical images, including differences in imaging modalities, contrast variations, and patient-specific anatomy [ 3 , 7 ] , negative transfer is common in medical image segmentation [ 8 ] . Most existing works address negative transfer by selecting more relevant source domain [ 9 , 10 ] , reweighting the source samples [ 11 , 6 ] , and aligning distributions in the feature space [ 12 ] . The principle idea in these works lies in the notion of domain similarity [ 13 , 14 , 15 ] or some transferability metric across different tasks [ 16 , 17 , 18 , 19 , 20 ] to quantify transfer performance. As these approaches have originated from a classification or regression context, they ignore the complex output structures in image-output tasks such as semantic segmentation. In segmentation, the risk of negative transfer of a given model does not uniformly distribute over the whole image. For example, some textures such as the white matter or the grey matter in the brain are shared across multiple domains, while some are unique to a specific domains such as the cortical protrusions and depressions across individual brains. This observation necessitates the need for a fine-grained transfer learning approach that could adapt to variable negative transfer risk at different image regions. Recently, a few natural image segmentation works have incorporated the category-level and pixel-level transferability into loss function by propagating the global category-wise transferability calculated by entropy criterion [ 21 ] , or weighting the loss function by transferability score during fine-tuning phase [ 22 ] . However, they don’t work effectively on medical segmentation tasks due to fewer semantic categories, tiny segmentation foreground, and severed class imbalance of medical images, compared to natural images [ 23 ] . To solve the aforementioned challenges, we propose a simple yet effective weighted fine-tuning approach that directs the model’s attention towards regions with significant transfer risk, tailored to the medical semantic segmentation problem (Fig. 1 ). Specifically, we introduce a pixel-level transfer risk map , which quantifies the transfer hardness for each pixel and the potential risks of negative transfer associated with them. Here we adopt LEEP (Log Expected Empirical Prediction) [ 18 ] metric as our transferability method for its high computational efficiency, simplicity, and superior performance [ 22 ] . And to alleviate the adverse effects of class imbalance during the fine-tuning phase, we calculate loss values across all pixels but average them exclusively over the foreground pixels. Such a method effectively reduces the impact that a large amount of well-learned background pixels might have on biasing the loss value, which in turn expedites the refinement of model parameters. Fig. 1 : Illustration of the proposed transfer risk map guided weighted fine-tuning framework. The strategy consists of: 1) a transferability-based pixel-level transfer risk map, which quantifies the potential risks of negative transfer for each pixel, 2) a transfer risk map weighted fine-tuning process, which allows the model’s attention towards regions with significant transfer hardness. Extensive experiments on brain tumor and brain matter segmentation datasets demonstrate that our proposed fine-tuning method achieves significantly enhanced performance when transferring knowledge between distinct modalities and tasks, with a 4.37% gain in brain tumor segmentation dataset FeTS 2021 and a 1.81% gain in brain matter segmentation dataset iSeg-2019, indicating that it indeed avoids negative transfer from diverse modalities and tasks while learning beneficial knowledge for segmentation across multiple modalities and tasks. In the few-shot scenario, our method also improved the baseline by 2.9% on average, validating the robustness of our approach under different sample sizes. II Methodology II-A Problem Definition In a typical source free transfer learning setting, We transfer the domain knowledge with only a pre-trained source model \theta_{S} and without access to source data. Specifically, we are given a pre-trained model \theta_{S} corresponding to a source task S with \mathcal{N}_{S} labeled samples \{x_{s}^{i},y_{s}^{i}\}_{i=1}^{\mathcal{N}_{S}} where x_{s}^{i}\in\mathcal{X}_{S},y_{s}^{i}\in\mathcal{Y}_{S} , and also a target task T with \mathcal{N}_{T} labeled samples \{x_{t}^{i},y_{t}^{i}\}_{i=1}^{\mathcal{N}_{T}} where x_{t}^{i}\in\mathcal{X}_{T},y_{t}^{i}\in\mathcal{Y}_{T} . \mathcal{X}_{S} , \mathcal{X}_{T} represent the input image and \mathcal{Y}_{S} , \mathcal{Y}_{T} represent their corresponding labels. The problem goal is to transfer the knowledge learned from the source to target that mitigate the negative transfer and improve the performance within the target domain. Transferability reveals how easy it is to transfer knowledge learned from a source task to a target task. We can obtain the transferability from the source task to the target task by using the testing data of the target task (\mathcal{X}_{T},\mathcal{Y}_{T}) to calculate the expected log-likelihood of the source pretrained model \theta_{S} : Trs(\theta_{S},\mathcal{X}_{T},\mathcal{Y}_{T})=\underset{x_{i},y_{i}\in\mathcal{X}_{T}\times\mathcal{Y}_{T}}{\mathbb{E}}[logP(y_{i}\mid x_{i};\theta_{S})] (1) II-B Transferability Estimation on Segmentation Task LEEP score is a more general metric for classification task, which is defined by the average log-likelihood of the expected empirical predictor, which predicts the dummy label distributions for target data in source label space and then computes the empirical conditional distribution of target label given the dummy source label. We select LEEP for our transferability estimation metric due to its high computational efficiency, simplicity and superior performance. We define the LEEP method as follows: \operatorname{LEEP}(\theta_{S},\mathcal{D})=\frac{1}{n}\displaystyle\sum_{i=1}^{n}\log(\displaystyle\sum_{z\in\mathcal{Z}}\hat{P}(y_{i}\mid z)\theta_{S}(x_{i})) (2) where \theta_{S} and \mathcal{D}=\{(x_{1},y_{1}),\cdots,(x_{n},y_{n})\} denote the source model and target dataset, respectively. \mathcal{Z} denotes the dummy source label, \hat{P}(\cdot|\cdot) denotes the empirical conditional distribution. Then we adapt the LEEP score to work on medical segmentation task. The global feature vector extracted by source pretrained model can be precisely decomposed into distinct pixel-wise feature, each indexed according