Investigating the feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Images
Saikat Roy, Mahmoud Mostapha, Radu Miron, Matt Holbrook, Mariappan Nadar · 2025
Read Investigating the feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Images by Saikat Roy, Mahmoud Mostapha, Radu Miron, Matt Holbrook, Mariappan Nadar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15309.txt.
Introduction
Saikat Roy and four co-authors present a study from twenty twenty-five on using patches, not whole scans, when a diffusion model cleans up medical images. Diffusion priors can restore scans or boost their detail. Whole image runs cost much time and memory. Patch runs promise a leaner path, but their worth was still an open ask.
The team trains and tests priors on MRI scans in patch form. Small tweaks keep patch seams from showing as flaws. They weigh score, speed, and memory use against whole image runs. They also test if a whole image trained prior can run on patches as is. Tests span many plug-in methods, tasks, and data sets.
Catalog record
| CREATOR | Saikat Roy, Mahmoud Mostapha, Radu Miron, Matt Holbrook, Mariappan Nadar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15309
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating the feasibility of Patch-based Inference for Generalized Diffusion Priors in Inverse Problems for Medical Images Saikat Roy, Mahmoud Mostapha, Radu Miron, Matt Holbrook, Mariappan Nadar Abstract Plug-and-play approaches to solving inverse problems such as restoration and super-resolution have recently benefited from Diffusion-based generative priors for natural as well as medical images. However, solutions often use the standard albeit computationally intensive route of training and inferring with the whole image on the diffusion prior. While patch-based approaches to evaluating diffusion priors in plug-and-play methods have received some interest, they remain an open area of study. In this work, we explore the feasibility of the usage of patches for training and inference of a diffusion prior on MRI images. We explore the minor adaptation necessary for artifact avoidance, the performance and the efficiency of memory usage of patch-based methods as well as the adaptability of whole image training to patch-based evaluation – evaluating across multiple plug-and-play methods, tasks and datasets. † † address: ∗ Siemens Healthineers, Princeton, NJ, USA † Siemens Industry Software, Romania (Advanta) 1 Introduction † † Disclaimer: The concepts and information presented in this paper/presentation are based on research results that are not commercially available. Future commercial availability cannot be guaranteed. Image restoration problems involve the recovery of clean image x from its noisy measurement y=Hx+n , where H is a degradation matrix and n is additive Gaussian noise of standard deviation \sigma . This can be reformulated as a solution \hat{x} for the following optimization problem: \hat{x}=\argmax_{x}\frac{1}{2}||y-Hx||^{2}+\lambda\phi(x) (1) where \frac{1}{2}||y-Hx||^{2} is a data-fidelity term, \lambda is the trade-off parameter and \phi(x) is the data prior. In [ 1 ] , it was demonstrated that while Eq. 1 could be decoupled and solved iteratively as in [ 2 ] , \phi(x) could be reformulated as an image Denoiser . Diffusion models have since been adopted as fast denoising priors in image restoration tasks [ 3 , 4 ] . While image restoration methods [ 5 , 6 ] themselves are usually computationally independent of the type of inference in the diffusion priors, the later have primarily favored whole image inference. In contrast, patch-based inference in Diffusion models usually involve inference over sub-regions (or patches) of the input image, along with some form of global aggregation. However, numerous methods focus on generation while requiring specialized architectures [ 7 , 8 , 9 ] . Fewer still deal specifically with plug-and-play approaches to inverse problems or specifically radiological data such as Magnetic Resonance Imaging (MRI). While [ 10 , 11 ] has explored the benefits of patch-based inference in CT abdominal images with positional embeddings, this area remains underexplored. In this work, we demonstrate using a generalized Diffusion-based prior trained over a large MRI dataset of multiple anatomies that: a) Patch-wise trained models can offer comparable performance whole image training, b) Patch-based inference as in [ 10 ] can be generically used regardless of training scheme or plug-and-play method, c) Patch-based training renders prior models more pliable to patch-based plug-and-play inference, d) We highlight expected memory benefits of using patches but also that these benefits plateau on continuously reducing patch sizes. Overall, we offer insights for usage and feasibility of patch-based training and inference in plug-and-play techniques for medical image restoration. 2 Method 2.1 Training of a Single Generalized Diffusion Prior In contrast to prior work [ 10 , 12 , 13 , 14 ] , we use a single diffusion prior p_{\theta}(x)\thicksim\phi(x) parameterized by \theta , trained on a large and diverse dataset of MRI images. Our prior is trained on a diverse data distribution populated by a large collection of approximately 289,000 MR images including brain, knee, prostate and other body regions from 1.5T, 3T scanners etc. The advantage of training a diverse prior instead of ones specific to certain anatomies is that a single prior can be used across inverse problems in multiple anatomical regions (for example, knees, brain etc.). The evaluation data used for validating this and the preparation of this article, was obtained from the NYU fastMRI Initiative database [ 15 ] , which is further discussed in Section 3 . 2.2 Shifted-Grid Inference We adopt the patch-based inference scheme (termed Shifted-Grid in this work), introduced by Patch-based position-aware Diffusion Inverse Solver (PaDIS) [ 10 , 11 ] . The use of a shifting grid allows the smoothing of grid-based artifacts, which would otherwise be visible if patches are naively stitched together. However, unlike [ 10 ] , we observe that the shifted grid inference can be independently applied to multiple inverse solvers such as DPS [ 5 ] or DiffPIR [ 6 ] . This is regardless of the use of positional embeddings as in [ 10 ] or adherence to a prior trained on a specific anatomical region. 2.3 Avoidance of Foreground-to-Background Boundary Transition Artifacts Figure 1 : Foreground-to-Background Boundary Transistion Artifacts during patchwise inference. These artifacts are seen on the left edge of the image in the case of zero-padding (left) while being absent for reflection padding (right). The usage of a single diverse prior trained on a large dataset with multiple anatomical regions transitions our prior, \phi(x) , away from a consistent idea of location . The proximal solution in image restoration techniques such as DiffPIR or DPS struggles particularly when foreground transitions to background, leading to horizontal or vertical lines as artifacts near generated image boundaries. This is accentuated in subsequent steps by the denoising prior. However, zero-padding is a standard part of shifted grid inference ( Section 2.2 ) and necessarily generates such foreground-background transitions. One simple solution to avoid such artifacts is to simply avoid boundary discontinuities by switching to reflection-padding . This results in images free of transition artifacts as illustrated in Fig. 1 . Knee MRI Dataset Denoising Inference Training Image Size Solver Image Size \mathbf{256\times 256} \mathbf{128\times 128} PSNR \uparrow LPIPS \downarrow PSNR \uparrow LPIPS \downarrow DPS 256\times 256 27.88_{\pm 1.09} 0.19_{\pm 0.02} 27.93_{\pm 1.11} 0.20_{\pm 0.02} 128\times 128 27.77_{\pm 1.07} 0.19_{\pm 0.03} 27.86_{\pm 1.05} 0.19_{\pm 0.02} 64\times 64 27.67_{\pm 1.05} 0.19_{\pm 0.03} 27.85_{\pm 1.09} 0.20_{\pm 0.02} DiffPIR 256\times 256 27.85_{\pm 1.12} 0.19_{\pm 0.03} 27.75_{\pm 1.08} 0.18_{\pm 0.02} 128\times 128 27.80_{\pm 1.05} 0.19_{\pm 0.02} 27.76_{\pm 1.07} 0.18_{\pm 0.02} 64\times 64 27.68_{\pm 1.05} 0.19_{\pm 0.02} 27.71_{\pm 1.11} 0.19_{\pm 0.02} Brain MRI Dataset Denoising Inference Training Image Size Solver Image Size \mathbf{256\times 256} \mathbf{128\times 128} PSNR \uparrow LPIPS \downarrow PSNR \uparrow LPIPS \downarrow DPS 256\times 256 28.72_{\pm 1.19} 0.18_{\pm 0.03} 28.64_{\pm 1.23} 0.17_{\pm 0.03} 128\times 128 28.57_{\pm 1.19} 0.18_{\pm 0.03} 28.64_{\pm 1.27} 0.16_{\pm 0.03} 64\times 64 28.35_{\pm 1.16} 0.20_{\pm 0.04} 28.64_{\pm 1.29} 0.17_{\pm 0.03} DiffPIR 256\times 256 28.75_{\pm 1.24} 0.20_{\pm 0.04} 28.52_{\pm 1.29} 0.16_{\pm 0.03} 128\times 128 28.63_{\pm 1.17} 0.18_{\pm 0.03} 28.48_{\pm 1.26} 0.16_{\pm 0.03} 64\times 64 28.46_{\pm 1.19} 0.19_{\pm 0.04} 28.47_{\pm 1.27} 0.16_{\pm 0.03} Super-Resolution ( \times 2 ) Solver Image Size \mathbf{256\times 256} \mathbf{128\times 128} PSNR \uparrow LPIPS \downarrow PSNR \uparrow LPIPS \downarrow DPS 256\times 256 27.87_{\pm 1.75} 0.25_{\pm 0.03} 27.87_{\pm 1.78} 0.26_{\pm 0.03} 128\times 128 27.72_{\pm 1.71} 0.26_{\pm 0.03} 27.75_{\pm 1.71} 0.27_{\pm 0.03} 64\times 64 27.66_{\pm 1.71} 0.26_{\pm 0.03} 27.75_{\pm 1.72} 0.27_{\pm 0.03} DiffPIR 256\times 256 27.86_{\pm 1.77} 0.25_{\pm 0.03} 27.84_{\pm 1.77} 0.27_{\pm 0.04} 128\times 128 27.76_{\pm 1.71} 0.26_{\pm 0.03} 27.77_{\pm 1.72} 0.27_{\pm 0.03}