Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation

Xin Zhang, Dongdong Meng, Sheng Li · 2025

Read Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation by Xin Zhang, Dongdong Meng, Sheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05897.txt.

Introduction

Xin Zhang, Dongdong Meng, and Sheng Li sharpen a transformer segmenter for the small organs that medical scans keep fumbling. Outlining organs guides radiation therapy and surgery, but compact structures get smeared by global attention models. Their fix stays inside the Mask2Former design rather than replacing it.

An offset adjustment nudges the attention's sampling points to stay within the same organ, gathering compact foreground cleanly. A coarse feature map supplies rough organ locations, and a small auxiliary head speeds training with an overlap loss. On two public head, neck, and cancer data sets the model set the pace, with the clearest gains on mid-sized and small organs.

Catalog record

CREATORXin Zhang, Dongdong Meng, Sheng Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation Xin Zhang, Dongdong Meng, Sheng Li Abstract Medical segmentation plays an important role in clinical applications like radiation therapy and surgical guidance, but acquiring clinically acceptable results is difficult. In recent years, progress has been witnessed with the success of utilizing transformer-like models, such as combining the attention mechanism with CNN. In particular, transformer-based segmentation models can extract global information more effectively, compensating for the drawbacks of CNN modules that focus on local features. However, utilizing transformer architecture is not easy, because training transformer-based models can be resource-demanding. Moreover, due to the distinct characteristics in the medical field, especially when encountering mid-sized and small organs with compact regions, their results often seem unsatisfactory. For example, using ViT to segment medical images directly only gives a DSC of less than 50%, which is far lower than the clinically acceptable score of 80%. In this paper, we used Mask2Former with deformable attention to reduce computation and proposed offset adjustment strategies to encourage sampling points within the same organs during attention weights computation, thereby integrating compact foreground information better. Additionally, we utilized the 4th feature map in Mask2Former to provide a coarse location of organs, and employed an FCN-based auxiliary head to help train Mask2Former more quickly using Dice loss. We show that our model achieves SOTA (State-of-the-Art) performance on the HaNSeg and SegRap2023 datasets, especially on mid-sized and small organs.Our code is available at link . 1 Introduction Medical segmentation plays an important role in clinical applications, including surgical guidance and radiation therapy [ 4 , 9 ] . To achieve clinically acceptable results, segmentation must consider the problem of how to combine local information and global information; Convolutional Neural Networks (CNNs) can integrate local information, but less effective on global information compared to transformer architectures [ 17 ] , but how to reduce the heavy computational burden and data demand is important. Deformable DETR [ 24 ] , featuring its deformable attention computation aimed at reducing the burden of training by querying sampled points instead of all pixels, provides insights for leveraing transformers. But utitlizing them directly is not enough. As depicted in Fig. 1, using transformer-based models directly would be accuracy-limited, especially for small organs as depicted in Fig. 2, which appear less frequently. Therefore, to utilize mask-attention-transformer-based models for clinical usage, enhancing the segmentation accuracy of small organs is vital to achieving better medical segmentations. In this paper, we used a deformable-attention-based transformer, Mask2Former, directly as our framework. Based on the sampling policy in attention computing to get points to be queried (computing offsets to the querying point and adding them onto the coordinates to get sampled points), we complicated the process to better segment mid-sized and small organs. In fact, we encouraged points to query nearby points that are suspected to be within the same organ, since the regions of the foreground are often small and compact. Based on this mechanism, we contrasted three strategies of offset adjustment to help enhance accuracy on relatively small organs. This process is integrated into any transformer that uses Deformable attention to compute attention weights, allowing any lightweight transformer to apply it. Besides, we merged the unused largest feature maps into the training process and added an FCN-based branch to help obtain coarse localization information and enhance contrast to the background. Our contributions can be concluded into three parts as follows: 1. We modified Mask2Former with three strategies to control offset computing, thereby enhancing the model’s general ability for small object segmentation. This can be applied to any lightweight transformer that utilizes deformable attention. 2. We adopted a feature fusion strategy to utilize the full information extracted inside Mask2Former itself. Since these feature maps already exist, we do not add the burden of extracting more features, which better combines global information for segmentation. 3. We added an FCN-based branch to help obtain coarse localization information and enhance contrast to the background, thus helping the model quickly find the location of organs and focus on better margin improvement. This alleviates the problem of vast background disturbance in typical medical screens. 1.1 Related work In past years, methods based on UNet [ 14 ] have achieved significant progress with the advantage of CNNs. It can integrate local information, but less effective on global information compared to transformer architectures [ 17 ] . However, due to the heavy computational burden and data demand, transformer-relevant methods [ 2 , 11 ] are often adopted by adding attention mechanism alone, such as TransUnet [ 2 ] , UNETR [ 8 ] and CSAF-CNN [ 11 ] . Transformer architecture as based model is not as popular. For example, in the SegRap2023 challenge, to segment head and neck organs, the top-10 methods are all based on nnUNet [ 12 ] . However, DETR [ 1 ] and Deformable DETR [ 24 ] inspire researchers to use lightweight deformable attention to reduce resource cost while keeping accuracy, and many trials are based on them, like SAM-DETR [ 22 ] , MaskFormer [ 5 ] , and Mask2Former. There have been many trials of adopting Mask2Former to segment medical images [ 20 , 21 ] , and enhancing accuracy with a CNN auxiliary branch is effective even without a complicated architecture. Besides, Takikawa [ 15 ] proposed Gated-SCNN to add an auxiliary branch to learn the marginal information of objects. Figure 1: Results on Different Models for the HaN-Seg Dataset and Visualization of Selected Results. 2 Methods 2.1 Overview An overview of our framework is presented in Fig. 2. We first use a standard backbone, such as ResNet-50, to extract feature maps and use them as inputs to the decoder as well as one FCN auxiliary head. Then, during the computation of attention weights, we apply adjustment strategies when acquiring the offset so that it is more efficient on mid-sized and small organs. The fusion strategy is performed in the next process. In Fig. 2, only one fusion strategy adopted in our model, the late fusion, is depicted. During the attention interaction with feature maps of four sizes, the fourth one, which is the biggest, is used for computing coarse embeddings (i.e., memories) and the weights of the attention layer. Then, other feature maps containing more local information are utilized to obtain better memories. The two are integrated with a residual connection. Finally, with outputs from the decoder, we use another auxiliary head for quick coarse location segmentation against the background to help accelerate training. The mask and class losses are then computed for loss backward. The details of the three kinds of fusion strategies, offset adjustment strategies, and the auxiliary branch will be discussed in the following sections. Figure 2: Overview of Our Offset-Adjusted Mask2Former with Feature Fusion, Background-Location Sensitive Auxiliary Branch, and Three Offset Adjustment Strategies. 2.2 Offset Adjustment Methods for Middle and Small Organ Segmentation Improvement As mentioned above, Mask2Former [ 5 ] utilizes a simple linear layer to compute the offset for each input point. The offset is used to acquire candidate points to be queried by the original point, making the attention weights more efficient. However in medical images, the background often occupies a large proportion, while target

Read the complete text