Relation U-Net

Sheng He, Rina Bao, P. Ellen Grant, Yangming Ou · 2025

Read Relation U-Net by Sheng He, Rina Bao, P. Ellen Grant, Yangming Ou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09101.txt.

Introduction

Sheng He and co-authors present a medical image network in 2025 that reports its own confidence. Clinics need not just an outline of organs or lesions but also a sense of when the outline may be wrong. Their network takes several input images and outputs several maps plus the relations among them.

Gaps among the relation maps yield a confidence score for a test image, with no true answer needed. Tests on four public data sets show better accuracy than the standard network it builds on. The confidence score also rises and falls in step with true accuracy on test images. The authors aim the tool at safer clinical reading of scans.

Catalog record

CREATORSheng He, Rina Bao, P. Ellen Grant, Yangming Ou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09101

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Relation U-Net Sheng He, Rina Bao, P. Ellen Grant, Yangming Ou Abstract Towards clinical interpretations, this paper presents a new “output-with-confidence” segmentation neural network with multiple input images and multiple output segmentation maps and their pairwise relations. A confidence score of the test image without ground-truth can be estimated from the difference among the estimated relation maps. We evaluate the method based on the widely used vanilla U-Net for segmentation and our new model is named Relation U-Net which can output segmentation maps of the input images as well as an estimated confidence score of the test image without ground-truth. Experimental results on four public datasets show that Relation U-Net can not only provide better accuracy than vanilla U-Net but also estimate a confidence score which is linearly correlated to the segmentation accuracy on test images. † † address: Boston Children’s Hospital, Harvard Medical School 1 Introduction Many deep learning-based methods for medical image segmentation [ 1 ] have been proposed to improve the accuracy. Most deep learning models are “output-only” models, which only output a single segmentation map given the input image. In general, the average accuracy on the evaluation dataset is reported. However, in practice the average accuracy on the evaluation dataset does not mean that every test image has the same or similar accuracy, especially the test images sampled from the clinical practice. Some test images are easy samples with high quality while some test images are difficult samples with smooth boundaries, poor contrast or small size of the target regions with high uncertainty [ 2 ] . Thus, deploying the trained model in clinical practice requires an interactive interpretation or post-correction of the segmentation outputs to automatically identify a subset of the difficult testing samples for experts inspection [ 3 ] . There is an unmet need to evaluate segmentation accuracies and even to reject failed segmentation when, in real-world applications, the ground-truth is often unavailable. To address this problem, an “output-with-confidence” model is required in real-world applications. It needs to automatically rank the test samples without ground-truth by the difficulty or the confidence score of segmentation on each sample image. Based on the rank list, end users can select the more challenging testing samples for further inspection [ 3 ] , which can save the human annotation and auditing time. Figure 1: Illustration of the proposed Relation U-net. In this paper, we propose an example of the “output-with-confidence” model based on the popular U-Net for segmentation, named Relation U-Net (Fig. 1 ). It is a generalization of the vanilla U-Net and Fig. 2 shows the main difference between them. The proposed Relation U-Net receives two input images and outputs four segmentation maps including the segmentation of each input image and their corresponding relations. During testing, the two input images can be sampled from the same input image and the discrepancy of the estimated relation segmentation maps is correlated to the segmentation accuracy of each test images. Thus, the discrepancy of the estimated relation segmentation maps can be considered as a confidence score which can be used to rank the test images for the end users to inspect and review the difficult samples. Figure 2: The comparison of the vanilla U-Net (a) and the proposed Relation U-Net (b). (Zoom in for a better visualization) 2 Method 2.1 Relation U-Net Fig. 1 shows the framework of the Relation U-Net. Given two different input images x_{i} , i=1,2 their corresponding binarized segmentation maps s_{i} in the training set, we use logical operations to define the relations based on segmentation maps s_{i} , similar to studies [ 4 ] . Two relations are considered in this paper, including the Possible (union) r^{p}=s_{1}\vee s_{2} and Consensus (intersection) r^{c}=s_{1}\wedge s_{2} relations between the segmentation maps s_{1} and s_{2} of two different images x_{1} and x_{2} . The Possible relation map captures the pixels/voxels that have been segmented as belonging to the target in at least one image. The Consensus relation map represents the pixels/voxels that have been segmented as belonging to the target object in both images. 2.2 The structure of neural network In this paper, we use the multiple-input multiple-output (MIMO) configuration [ 4 , 5 ] to build Relation U-Net. We use the popular U-Net [ 1 ] as the baseline which only requires two changes. At the input layer, the two input images are concatenated. Since we need four outputs (see Fig. 2 ), at the last layer, four different predictions are obtained by adding four different classifiers. The Relation U-Net takes two inputs and returns four outputs including the estimated segmentation s_{i} for the input images and their corresponding relations r^{p} and r^{c} . During training, the two inputs are sampled independently from the training set and the four different classifiers for segmentation are trained based on the ground-truth of corresponding labels. Thus, the Relation U-Net does not only learn the information for each input image, but also learns the relations among the pairs of samples. At test time, the multiple input images can be sampled from different images or the same input image is repeated two times to feed into the Relation U-Net, which can make four independent segmentation maps. 2.3 Estimating of Confidence Score Based on the definition , the Possible r^{p} and Consensus r^{c} relations are reflexive [ 4 ] , indicating that when the pair of input images x_{1} , x_{2} are from the same sample x_{1}=x_{2} , the output of the relations r^{p} and r^{c} is the same to the segmentation of the input: r^{p}=s^{c}=s_{1}=s_{2} . However, since we train the Relation U-Net with two different input images independently sampled from the training set and the four outputs are predicted independently. Thus, the estimations of the Possible \hat{r}^{p} and Consensus \hat{r}^{c} are not exactly the same given the same input since they are independently predicted by different sub-networks of the Relation U-Net. We compute the discrepancy between the estimation of \hat{r}^{p} and \hat{r}^{c} by: \mathcal{C}=\mathcal{D}\big(\hat{r}^{p},\hat{r}^{c}\big) . where \mathcal{D} is a distance metric to measure the difference between two estimated segmentation maps of \hat{r}^{p} and \hat{r}^{c} . In this paper, we use the Dice coefficient as the distance metric to measure the difficulty to segment each image, defined as: \mathcal{C}=2|\hat{r}^{p}\cap\hat{r}^{c}|/(\hat{r}^{p}+\hat{r}^{c}\big) . where \mathcal{C} measures the consistency between the segmentation maps between \hat{r}^{p} and \hat{r}^{c} . Fig. 3 shows several images with the segmentation maps of the relations \hat{r}^{p} and \hat{r}^{c} . For difficult samples, the segmentation maps of these two relations are different because they are independently predicted by different sub-networks within the Relation U-Net, yielding a low \mathcal{C} . However, for easy samples, the segmentation maps of these two relations are almost same and the corresponding \mathcal{C} values are also high. Thus, the value of \mathcal{C} can be used to measure the difficulty of the test images which is correlated to the accuracy of the segmentation result. In this paper, we name the value of \mathcal{C} as the confidence score which can rank test images by difficulty without ground-truth. Figure 3: Examples of images on the ISIC2018 dataset with relation segmentation maps: \hat{r}^{p} (green contours) and \hat{r}^{c} (red contours). The \hat{r}^{p} and \hat{r}^{c} are the same on easy samples while are different

Read the complete text