No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top- K Patch Sampling
Young Seok Jeon, Hongfei Yang, Huazhu Fu, Yeshe Kway, Mengling Feng · 2025
Read No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top- K Patch Sampling by Young Seok Jeon, Hongfei Yang, Huazhu Fu, Yeshe Kway, Mengling Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10814.txt.
Introduction
Young Seok Jeon and co-authors speed up 3D medical image outlining by dropping the sliding window method. Three dimensional models beat flat ones on CT and MRI scans, since they use links between slices. But patch based work eats memory and slows the reading of new scans.
Their 2025 framework, NMSW, picks only the top scoring patches through a trainable top-k step, which cuts wasted work. When patch calls fall short, broad global guesses refine the result. Across three tasks and three backbones, accuracy stayed on par with the old method. Compute cost fell by 91 percent, and run times dropped about nine to eleven fold.
Catalog record
| CREATOR | Young Seok Jeon, Hongfei Yang, Huazhu Fu, Yeshe Kway, Mengling Feng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10814
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top- K Patch Sampling Young Seok Jeon, Hongfei Yang, Huazhu Fu, Yeshe Kway, Mengling Feng Abstract 3D models surpass 2D models in CT/MRI segmentation by effectively capturing inter-slice relationships. However, the added depth dimension substantially increases memory consumption. While patch-based training alleviates memory constraints, it significantly slows down the inference speed due to the sliding window (SW) approach. We propose No-More-Sliding-Window (NMSW), a novel end-to-end trainable framework that enhances the efficiency of generic 3D segmentation backbone during an inference step by eliminating the need for SW. NMSW employs a differentiable Top-k module to selectively sample only the most relevant patches, thereby minimizing redundant computations. When patch-level predictions are insufficient, the framework intelligently leverages coarse global predictions to refine results. Evaluated across 3 tasks using 3 segmentation backbones, NMSW achieves competitive accuracy compared to SW inference while significantly reducing computational complexity by 91% (88.0 to 8.00 TMACs). Moreover, it delivers a 9.1× faster inference on the H100 GPU (99.0 to 8.3 sec) and a 11.1× faster inference on the Xeon Gold CPU (2110 to 189 sec). NMSW is model-agnostic, further boosting efficiency when integrated with any existing efficient segmentation backbones. The code is available: https://github.com/Youngseok0001/open_nmsw . K eywords Deep Learning \cdot 3D Medical Image Segmentation \cdot Differentiable Top- K Sampling \cdot Gumbel-Softmax Trick \cdot Efficient Infernce \cdot Sliding Window Inference. 1 Introduction 1.1 2D vs 3D model for 3D Medical Image Segmentation Volumetric (3D) segmentation is a key challenge in medical imaging, with numerous applications such as tumor detection [ 40 ] , volume estimation [ 1 ] , and controlled medical image generation [ 2 ] . Both 2D and 3D UNet-like encoder-decoder models can be applied to solve 3D segmentation tasks, though they come with distinct advantages and limitations. For 2D models, the 3D input is destructured into a batch of 2D slices, typically along the axial direction, with predictions made independently for each slice. While 2D models are computationally efficient, they fail to capture inter-slice relationships, potentially reducing the accuracy of the final whole-volume prediction. In contrast, 3D models analyze the complete volumetric input without structural decomposition. By distributing model weights along the axial plane, the models capture inter-slice relationships, resulting in improved accuracy compared to 2D counterparts, as shown in prior studies [ 4 , 3 ] . However, this structural fidelity comes at the expense of significantly higher computational complexity, limiting their practicality in resource-constrained settings. Figure 1: NMSW provides an efficient alternative to the computationally intensive Sliding Window inference method, which uniformly allocates resources across all patches. Instead, NMSW first generates a coarse whole-volume prediction and then employs a differentiable patch sampling module to automatically select a small subset of patches that require finer segmentation. Table 1: Inference speed of SW across different backbones and processing units for an input of spatial size 480\times 480\times 480 with 20 output classes, using a patch size of 128\times 128\times 128 and an overlapping ratio of 0.5. Backbone H100 (s) A100 (s) Xeon Gold (s) UNet 4.58 7.93 135 Swin-unetr 24.4 43.4 2110 MedNext 32.2 53.2 1720 1.2 Computational Ineffciency of 3D Segmentation Models Increased computational demand and memory requirements present a significant bottleneck for all 3D segmentation models, regardless of their underlying backbone—whether Conv-net [ 5 ] , Self-attention [ 6 ] , or State-Space models [ 7 ] . Using ConvNet as an example, increasing the convolutional kernel dimension from k\times k to k\times k\times k amplifies the number of multiplications and additions required to produce each output pixel/voxel by a factor of k . More critically, the memory required to store intermediate tensor outputs during backpropagation also increases, as it now includes an additional depth dimension. This challenge becomes particularly pronounced when using parallelizable matrix multiplication-based convolution algorithms, such as im2col [ 8 ] or Winograd convolutions [ 9 ] . Transformer models are no exception to the challenges of increased memory usage and computational costs. While their architecture remains largely consistent when transitioning from 2D to 3D, the addition of tokens in the axial dimension increases the size of the self-attention matrix quadratically. For example, if a token is worth a patch of size 16\times 16\times 16 (as suggested by [ 10 ] ), a CT scan of size 512\times 512\times 512 yields 32,768 tokens ( 32\times 32\times 32 ), resulting in a self-attention matrix of size 32,768\times 32,768 . 1.3 Sliding Window to Reduce 3D Segmentation Model’s Memory Patch-based training, coupled with Sliding-Window (SW) inference, is currrently the predominant method to address the substantial memory requirements of 3D models. Rather than processing the entire volume in a single step, patch-based training randomly samples a set of patches that are significantly smaller than the actual volume. To generate the final whole-volume prediction using the patch-trained model, SW, as shown in Figure 1 , is employed. SW sequentially predicts patches sampled at uniform intervals with some degree of overlap (typically 25–50% [ 12 ] ). The overlapping patch predictions are then aggregated into the final whole-volume prediction using an appropriate post-processing step, such as weighting each patch with a discretized Gaussian distribution to minimize edge artifacts. 1.4 Inefficiency and Potential Prediction Bias in SW Inference Although SW is memory-efficient with constrained input patch size, it has significant drawbacks, including slower inference speed and increased computational demands. Because the patch-based model is not trained to rank patch importance, SW adopts a conservative approach, segmenting all patches with equal computational resources, even those that clearly do not contain objects of interest or the object is simple enough for a less complex model to handle equally well. A typical ensemble UNet model [ 12 ] takes nearly a minute to perform a full SW on a whole-volume size of 512\times 512\times 458 using an NVIDIA GeForce RTX 3090 GPU [ 13 ] . In addition, healthcare professionals often use open-source visualization tools such as 3D Slicer [ 47 ] , which rely on on-device processors for model inference. In such settings, inference speed drops significantly, exceeding 10 minutes (Table 2 ). SW is not only slow during inference, but could potentially lead to poorer performance due to small patch size. One unique aspect of medical image segmentation, compared to generic segmentation tasks on natural images, is the need for global features, such as location, which greatly help in distinguishing objects that may appear similar at a local level. Patch-trained models, however, may miss these global aspects. The most common approach to address this inherent limitation in patch-based models is to train another model that segments using a low-resolution input [ 12 ] . However, this adds additional complexity to a model that is already computationally expensive. SW is not only slow during inference, but could potentially lead to poorer performance due to a small patch size. One unique aspect of medical image segmentation, compared to generic segmentation tasks on natural images, is the need for global features, such as location, which greatly help in distinguishing objects that may appear similar at a local level. Patch-trained models, however, may miss these global aspects. The most common approach to address this inherent limitation in patch-based models is to train