Local Patches Meet Global Context: Scalable 3D Diffusion Priors for Computed Tomography Reconstruction
Taewon Yang, Jason Hu, Jeffrey A. Fessler, Liyue Shen · 2025
Read Local Patches Meet Global Context: Scalable 3D Diffusion Priors for Computed Tomography Reconstruction by Taewon Yang, Jason Hu, Jeffrey A. Fessler, Liyue Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18161.txt.
Introduction
Taewon Yang and coauthors at the University of Michigan bring full 3D diffusion models to CT reconstruction. Diffusion models make strong image priors for rebuilding scans from limited data, but training one on whole 3D volumes needs more computing power and data than most labs have.
Their model learns from 3D patches instead of whole volumes, which keeps training light enough for limited data. Each patch is learned together with a downsampled copy of the full volume, so local detail stays tied to global anatomy, and position codes tell the model where each patch sits. Across several datasets the method beats prior state of the art systems in both quality and speed. It rebuilds a 512 by 512 by 256 volume in about 20 minutes.
Catalog record
| CREATOR | Taewon Yang, Jason Hu, Jeffrey A. Fessler, Liyue Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-18161
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Local Patches Meet Global Context: Scalable 3D Diffusion Priors for Computed Tomography Reconstruction Taewon Yang, Jason Hu, Jeffrey A. Fessler, Liyue Shen Abstract Diffusion models learn strong image priors that can be leveraged to solve inverse problems like medical image reconstruction. However, for real-world applications such as 3D Computed Tomography (CT) imaging, directly training diffusion models on 3D data presents significant challenges due to the high computational demands of extensive GPU resources and large-scale datasets. Existing works mostly reuse 2D diffusion priors to address 3D inverse problems, but fail to fully realize and leverage the generative capacity of diffusion models for high-dimensional data. In this study, we propose a novel 3D patch-based diffusion model that can learn a fully 3D diffusion prior from limited data, enabling scalable generation of high-resolution 3D images. Our core idea is to learn the prior of 3D patches to achieve scalable efficiency, while coupling local and global information to guarantee high-quality 3D image generation, by modeling the joint distribution of position-aware 3D local patches and downsampled 3D volume as global context. Our approach not only enables high-quality 3D generation, but also offers an unprecedentedly efficient and accurate solution to high-resolution 3D inverse problems. Experiments on 3D CT reconstruction across multiple datasets show that our method outperforms state-of-the-art methods in both performance and efficiency, notably achieving high-resolution 3D reconstruction of 512 × 512 × 256 512\times 512\times 256 ( ∼ \sim 20 mins). 1 Introduction Figure 1 : Training the 3D patch diffusion model. Noisy 3D patch \bm{G}_{c}\bm{x}_{t} (local details), downsampled volume \bm{D}\bm{x}_{t} (global context), and positional encoding are concatenated as inputs to the denoiser network D_{\theta} in each training iteration, to predict the noise \bm{\epsilon}_{\theta} in the input noisy 3D patch. Positional encoding consists of voxel-based coordinates in x,y,z that are normalized to [-1,1] . Diffusion models have emerged as a powerful class of generative model for image generation 35 ; 28 ; 30 ; 31 , modeling the underlying image distribution p(\bm{x}) through learned denoising processes across progressively varied noise levels. The trained network approximates the score function of image distribution \bm{s}(\bm{x})=\nabla\log p(\bm{x}) . At inference time, starting from the initialization of Gaussian noise image, the noise signal is iteratively removed using the trained denoiser network, ultimately resulting in clean image samples drawn from the learned distribution prior p(\bm{x}) . Inverse problems can be formulated as \bm{y}=\mathcal{A}(\bm{x})+\bm{n} , where \bm{y} is a known measurement, \mathcal{A} is the forward operator, \bm{x} is the unknown image to be recovered, and \bm{n} is random noise. Specifically, computed tomography (CT) reconstruction can also be formulated as an inverse problem that aims to reconstruct a 3D volume \bm{x} from the measurement \bm{y} , containing projection views of \bm{x} acquired from different angles 15 . Furthermore, to minimize patient radiation exposure, it is desirable to reduce the number of acquired projection views, typically on the order of 100-1000 views, to substantially fewer 33 ; 9 , which is called sparse-view CT reconstruction . In this case, the inverse problem becomes ill-posed, as there are many potential images \bm{x} that could correspond to a measurement \bm{y} . Hence, it is necessary to impose prior knowledge on \bm{x} to obtain a reasonable reconstruction, such as 21 ; 22 in previous works. Given that diffusion models can learn powerful generative priors over image data, it is natural to leverage these priors to address sparse-view CT reconstruction problem. Previous works 7 ; 20 ; 38 ; 24 ; 11 have proposed various approaches leveraging diffusion models to solve different inverse problems, including 2D CT reconstruction. These approaches generally consist of, at inference time, interweaving the original diffusion model steps with data fidelity enforcement steps that push the image to be consistent with the measurements. However, challenges emerge when directly applying these existing approaches to real-world sparse-view CT task, where the images are inherently three-dimensional volumes. A clinically standard CT image have the full size of 512\times 512\times 512 , making it almost infeasible to fit the entire image into the network training due to memory constraints. Furthermore, training diffusion models typically requires large-scale datasets 44 , often comprising hundreds of thousands of images such as ImageNet 32 or LSUN 43 . For CT reconstruction, there is a lack of high-quality volumetric CT data in such scale. Finally, training time of diffusion models on 3D images would be prohibitively expensive. Thus, the high computational costs in memory, time, and data pose significant challenges to directly train diffusion models on 3D CT images. To address these challenges, prior works have proposed leveraging 2D image priors, either by combining them with hand-crafted external regularization terms 9 or by training an additional 2D image prior along a perpendicular plane 23 . However, these approaches tend to produce blurry artifacts because of sensitivity to the hand-crafted regularization term, or require long inference times due to the use of two perpendicular 2D CT priors. Recently, 33 introduced a new solution to learn the joint distribution over a small set of 2D slices simultaneously, but the underlying network remains a 2D architecture with multiple channels, requesting significant reconstruction time comparable to that of 8 . However, these methods do not fully realize and exploit the generative capacity of diffusion models for high-resolution 3D volumetric images. To overcome these limitations, we propose a novel approach that learns scalable 3D diffusion priors by representing 3D volumes as position-condition local patches . To ensure high-quality 3D image generation, our study emphasizes the importance of incorporating global context. Specifically, by jointly modeling the distribution of local patches alongside a downsampled version of the entire volume, our method effectively unifies global contextual information with local image statistics, to obtain improved global-aware patch priors . Beyond image generation, the resulting fully 3D prior demonstrates superior performance in large-scale inverse problem solving, such as high-resolution 3D CT reconstruction. Overall, our contributions are three-fold. • We propose a novel 3D patch-based diffusion model that learns a fully 3D diffusion prior, enabling the scalable generation of high-resolution 3D images, and offering an unprecedentedly efficient and accurate solution to address high-resolution 3D inverse problems. • Our key technical contribution is a framework that models the joint distribution of position-aware 3D local patches and a downsampled 3D volume, enabling scalable learning of a global-contextual 3D patch-based prior by effectively integrating local and global information to achieve high-quality 3D image generation. • Extensive experiments on different datasets and different-sized images show that our proposed method achieves SOTA performance for 3D sparse-view CT reconstruction in various settings. 2 Background and Related Works 2.1 Diffusion Models Diffusion models define a forward stochastic differential equation (SDE) that progressively perturbs a clean sample with Gaussian noise 35 . For t\in[0,T] , let \bm{x}(t)\in\mathbb{R}^{d} evolve as \mathrm{d}\bm{x}=-\frac{\beta(t)}{2}\bm{x}\,\mathrm{d}t+\sqrt{\beta(t)}\,\mathrm{d}\bm{w}, (1) where \beta(t) specifies the variance schedule. Under this process, \bm{x}(0) follows the data distribution, while \bm{x}(T) approaches a standard Gaussian. To synthesize new samples, one solves the corresponding reverse-time SDE 1 : \mathrm{d}\bm{x}=\left(-\frac{\beta(t)}{2}-\beta(t)\nabla_{\bm{x}_{t}}\log p_{t}(\bm{x}_{t})\right)\mathrm{d}t+\sqrt{\beta(t)}\,\mathrm{d}\overline{\bm{w}}, (2) where \nabla_{\bm{x}_{t}}\log p_{t}(\bm{x}_{t}) is the score function of the noisy data distribution at time t . By training a neural network to approximate this score function, one can iteratively integrate the reverse SDE starting from Gaussian noise to produce realistic samples from the learned data