Region-Normalized DPO for Medical Image Segmentation under Noisy Judges

Hamza Kalisch, Constantin Seibold, Jens Kleesiek, Ken Herrmann, Frederic Jonske · 2026

Read Region-Normalized DPO for Medical Image Segmentation under Noisy Judges by Hamza Kalisch, Constantin Seibold, Jens Kleesiek, Ken Herrmann, Frederic Jonske (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23222.txt.

Introduction

Hamza Kalisch and four co-authors train medical image segmentation without costly pixel-level labels. Dense annotations are the gold standard, but they are expensive and limit scale. Deployed systems already emit cheap quality signals, like model agreement and uncertainty scores.

Their region-normalized preference method learns from those noisy judges instead of ground truth. Normalizing by region keeps large and small structures treated fairly. Training from existing quality signals could scale segmentation to hospitals without annotation teams. Model agreement, uncertainty measures, and learned mask-quality scores are the inexpensive signals the method learns from instead of annotations.

Catalog record

CREATORHamza Kalisch, Constantin Seibold, Jens Kleesiek, Ken Herrmann, Frederic Jonske
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-23222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Region-Normalized DPO for Medical Image Segmentation under Noisy Judges Hamza Kalisch, Constantin Seibold, Jens Kleesiek, Ken Herrmann, Frederic Jonske Abstract While dense pixel-wise annotations remain the gold standard for medical image segmentation, they are costly to obtain and limit scalability. In contrast, many deployed systems already produce inexpensive automatic quality-control (QC) signals like model agreement, uncertainty measures, or learned mask-quality scores which can be used for further model training without additional ground-truth annotation. However, these signals can be noisy and biased, making preference-based fine-tuning susceptible to harmful updates. We study Direct Preference Optimization (DPO) for segmentation from such noisy judges using proposals generated by a supervised base segmenter trained on a small labeled set. We find that outcomes depend strongly on how preference pairs are mined: selecting the judge’s top-ranked proposal can improve peak performance when the judge is reliable, but can amplify harmful errors under weaker judges. We propose Region-Normalized DPO (RN-DPO), a segmentation-aware objective which normalizes preference updates by the size of the disagreement region between masks, reducing the leverage of harmful comparisons and improving optimization stability. Across two medical datasets and multiple regimes, RN-DPO improves sustained performance and stabilizes preference-based fine-tuning, outperforming standard DPO and strong baselines without requiring additional pixel annotations. 1 Introduction Medical image segmentation is essential for the development of automated clinical pipelines, supporting tasks such as large scale screening processes ( 5 ) , treatment planning ( 22 ) , and longitudinal monitoring ( 9 ) . When trained with dense annotations segmentation methods achieve strong performance ( 2 ) but for novel tasks such masks are expensive to obtain ( 32 ) . To bypass annotation shortcomings alternative annotation settings have been investigated ( 18 ) . A widely used data-collection strategy for NLP settings is preference-based feedback, where users are presented multiple candidate outputs and select the one they prefer most. These pairwise comparisons can then be leveraged to align models, for instance via RLHF ( 26 ; 17 ) or more directly through Direct Preference Optimization (DPO), which provides a simple and effective training objective ( 19 ) . When translating such a setting to medical imaging, a clinician would compare two candidate masks and indicate which one is better, when producing pixel-level annotation is infeasible. Initial results ( 14 ; 16 ) suggest that comparative supervision via DPO improve medical segmentation without requiring additional dense labels. However, despite growing interest in DPO for segmentation, existing studies often rely on preference signals that are effectively derived from ground truth ( 36 ; 14 ; 16 ) , which bypassess the challenge of realistic imperfect feedback. In practice, comparative feedback in medical pipelines can arise from quality control (QC) signals, ranging from lightweight human review and heuristic plausibility checks to automated segmentation quality estimation and failure detection models ( 39 ; 25 ; 29 ) . Such judges can be instantiated in many forms, including ensemble-agreement based checks ( 11 ) or consistency signals under test-time perturbations ( 31 ) . Beyond evaluation, QC signals have also been used to improve segmentation models, for instance by selecting reliable outputs in ensemble-based pipelines ( 11 ) , filtering pseudo-labels, or prioritizing samples for annotation in active learning ( 23 ) . Importantly, QC is not perfectly reliable: benchmarking studies show that segmentation QC and failure detection remain challenging and can degrade under distribution shift ( 39 ) , while learned QC models exhibit non-negligible error rates in metric estimation and error detection ( 25 ) . This motivates the central question of our work: how can we make preference-based fine-tuning for medical image segmentation stable and effective under noisy feedback? To tackle this question, we study preference-based fine-tuning for medical segmentation under controlled noisy judges. Concretely, we instantiate a model-based QC proxy by training an auxiliary segmenter on a separate labeled split. This design yields a realistic source of structured model bias while allowing systematic variation of judge reliability. Across two benchmarks, preference fine-tuning is evaluated under different combinations of base strength, judge strength, and pair-mining strategies, revealing that vanilla DPO can be fragile under imperfect rankings and suffer performance degradation from high-leverage misranked comparisons. To mitigate this, we introduce Region-Normalized DPO (RN-DPO), which normalizes preference updates by the disagreement region between candidate masks, improving robustness and performance across regimes. Our contributions are: (i) a systematic empirical study of DPO for medical image segmentation under noisy comparative feedback, (ii) RN-DPO, a segmentation-aware objective that stabilizes preference fine-tuning under judge noise, and (iii) consistent improvements over vanilla DPO and other strong baselines across different judge strengths, including strong gains under an oracle judge. 2 Related Work Direct Preference Optimization and its variants. Direct Preference Optimization (DPO) ( 19 ) casts preference learning as a simple binary classification objective derived from a KL-regularized reward-maximization view, directly optimizing a policy against a fixed reference model and thereby avoiding explicit reward modeling and online RL. A number of extensions refine DPO-style training via alternative parameterizations ( 28 ) , offsets ( 1 ) , or reference-free formulations ( 15 ; 12 ) . Token-level variants such as TDPO ( 38 ) further increase granularity by applying preference constraints within sequences, and closely related objectives like IPO ( 3 ) emphasize stability through alternative regularization choices. Preference optimization for vision and segmentation. Preference optimization has also been adapted to vision-related tasks, for example to align text-to-image diffusion models ( 30 ) and to improve vision-language model faithfulness ( 37 ) . In medical image segmentation, ( 14 ) apply preference optimization to further fine-tune a SAM-based model ( 13 ) using comparative supervision over candidate masks generated by thresholding the output probabilities of the encoder’s prediction. Similarly, SAMPO ( 36 ) applies DPO to promptable vision foundation models by mining preference pairs from candidate masks generated under varied sparse prompts. MAPO ( 16 ) extends this to arbitrary segmentation backbones via a model-agnostic framework and utilizes dropout to sample more diverse candidates. While these works demonstrate the feasibility of learning from comparative signals for the segmentation task, they rely on ground-truth masks to define preferences. With only limited rating-noise ablations reported in ( 14 ) , robustness under noisy mined preferences in segmentation remains largely unexplored. Robustness under noisy preferences. Preference data can be noisy in various ways (e.g., flipped comparisons, near-ties, judge-specific biases), which can lead to performance degradation and instability ( 6 ) . rDPO ( 6 ) provides a theoretically grounded robust loss for DPO under random label flips, while Dr. DPO ( 34 ) adopts a distributionally robust view to handle unreliable pairs. Complementary approaches improve robustness via adaptive reweighting and calibration, e.g., \beta -DPO ( 35 ) , which dynamically adjusts the DPO temperature to reduce sensitivity to noisy comparisons. In contrast, \alpha -DPO ( 33 ) and \gamma -PO ( 27 ) introduce margin-based formulations, using adaptive or pair-specific target margins to down-weight ambiguous preference pairs. Beyond objective design, robustness can be influenced by how preference pairs are selected or scheduled, e.g. via a curriculum sampling of easier vs harder pairs ( 8 ) . Figure 1 : Main pipeline. We fine-tune a base segmenter \pi_{\theta} on unlabeled images using comparative

Read the complete text