Memory-Efficient 3D High-Resolution Medical Image Synthesis Using CRF-Guided GANs
Mahshid Shiri, Alessandro Bruno, Daniele Loiacono · 2025
Read Memory-Efficient 3D High-Resolution Medical Image Synthesis Using CRF-Guided GANs by Mahshid Shiri, Alessandro Bruno, Daniele Loiacono (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10899.txt.
Introduction
Mahshid Shiri and co-authors fake high detail 3D medical scans without hogging computer memory. Rival image generators learn by contest, and they shine at made up imagery. In three dimensions, though, graphics memory caps the scan size, or the output breaks into patches. Their design splits the generator in two to dodge the cap.
A middle sketch of the scan is first passed through a field model that ties nearby points together. The second half then paints one sub volume at a time from that sketch. At run time, the full scan is built whole, with no patch seams. On lung and brain scans, it beat top rivals while using less memory.
Catalog record
| CREATOR | Mahshid Shiri, Alessandro Bruno, Daniele Loiacono |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10899
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memory-Efficient 3D High-Resolution Medical Image Synthesis Using CRF-Guided GANs Mahshid Shiri, Alessandro Bruno, Daniele Loiacono Abstract Generative Adversarial Networks (GANs) have many potential medical imaging applications. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models cannot scale to high-resolution or are susceptible to patchy artifacts. In this work, we propose an end-to-end novel GAN architecture that uses Conditional Random field(CRF) to model dependencies so that it can generate consistent 3D medical Images without exploiting memory. To achieve this purpose, the generator is divided into two parts during training, the first part produces an intermediate representation and CRF is applied to this intermediate representation to capture correlations. The second part of the generator produces a random sub-volume of image using a subset of the intermediate representation. This structure has two advantages: first, the correlations are modeled by using the features that the generator is trying to optimize. Second, the generator can generate full high-resolution images during inference. Experiments on Lung CTs and Brain MRIs show that our architecture outperforms state-of-the-art while it has lower memory usage and less complexity. 1 Introduction Generative models have gained considerable popularity among researchers for a variety of tasks, including data augmentation [ 4 ] . Generative Adversarial Networks (GANs) [ 7 ] have made remarkable strides in the biomedical field. These advancements are largely due to recent improvements in computational power and the availability of extensive medical datasets. 3D GAN models were proposed for reducing noise in low-dose CT scans [ 22 ] , enhancing the quality of CT images [ 16 ] , generating realistic 3D brain MRI images [ 13 ] , and creating tumor masks for segmentation [ 5 ] . However, these methods have a limitation in the generated image resolution, which is usually 128^{3} or smaller, because of limited memory during training [ 23 ] . Memory-efficient GANs aim to balance memory efficiency and generative capability, ensuring that GANs can be trained effectively even with limited computational resources. To this aim, the experimental campaigns by Lei et al. [ 19 ] and Yu et al. [ 27 ] focused on producing new images slice by slice or patch by patch. However, as these methods generate patches and slices independently, the related boundaries may exhibit artefacts. Also, Uzunova et al. [23] use two GANs to generate images, with the first network releasing a lower-resolution version of the image and the second producing higher-resolution patches conditioned on the first GAN. This approach is still patch-based and lacks a comprehensive understanding of the entire image structure. Furthermore, it does not support end-to-end training, making it impractical to use encoders in this context. Also, Sun et al. [ 23 ] proposed an end-to-end hierachichal GAN architecture (HA-GAN), capable of generating high-resolution 3D images at a resolution of 256^{3} . The HA-GAN architecture consists of two interconnected GANs: low-resolution GAN that produces a low-resolution version of the 3D image, capturing essential global structure with reduced computational and memory requirements, and high-resolution GAN that generates high-resolution patches for a randomly selected sub-volume of the image. The high-resolution generator shares its initial layers with the low-resolution generator, ensuring consistency and efficient use of memory. During inference, HA-GAN is capable of directly generating full high-resolution 3D images without the need for additional post-processing. HA-GAN demonstrates superior performance compared to state-of-the-art models in generating high-resolution 3D images. This work aims to address the limitations of existing 3D GAN models by presenting a novel architecture for high-resolution 3D medical image synthesis that combines Conditional Random Fields (CRF) with GANs. The main contribution of this research is the development of a more memory-efficient and high-performing architecture. This study aims to advance the quality and applicability of synthetic medical images by tackling key challenges such as consistency, memory efficiency, model complexity, and performance. 2 Conditional Random Field(CRF) Conditional Random Field(CRF) is a type of probabilistic graphical model used in machine learning. It tackles dependencies between the output variables, taking into account the data’s sequential or structural nature. CRFs˝ allow capturing correlations between image patches. Let’s denote the embedding of patches as x=\{x_{i}\}_{i=1}^{N} , where N represents the number of patches. Each patch i is associated with a random variable y=\{y_{i}\}_{i=1}^{N} , denoting the binary label of each patch. The conditional distribution P(Y|x) can be formulated as a CRF with a Gibbs distribution, as depicted in Equation 1 . P(Y=y\mid x)=\frac{1}{Z(x)}\exp\left(-E(y,x)\right) (1) Here, E(y,x) represents the energy function, quantifying the cost of Y assuming a specific configuration y given x . Z(x) stands for the partition function ensuring that P(Y=y|x) is a legitimate probability distribution. In the context of a fully-connected pairwise CRF [ 15 ] , the energy function can be expressed as Equation 2 . E(y,x)=\sum_{i}u(y_{i})+\sum_{i<j}p(y_{i},y_{j}) (2) \sum_{i} and \sum_{i<j} indicate summations over all patches and pairwise combinations of patches, respectively. u(y_{i}) signifies the unary potential associated with patch i that measures the cost of patch i taking the label y_{i} given the patch embedding x_{i} , while p(y_{i},y_{j}) represents the pairwise potential between patches i and j that measures the cost of jointly assigning patch i,j with labels y_{i},y_{j} given the patch embeddings x_{i},x_{j} respectively. Pairwise potential p(y_{i},y_{j}) models spatial correlations between neighboring patches, and would encourage low cost for assigning y_{i},y_{j} with the same label if x_{i},x_{j} are similar [ 25 ] . Li et al. [ 20 ] incorporate CRF on top of a feature extractor to refine segmentation results and ensure spatial coherence within the image. Zhong [ 28 ] explores the combined use of GANs and CRF for hyper-spectral image classification. Zhong [ 28 ] employed CRFs as a post processing layer to refine the classification results obtained from GAN-generated hyper-spectral images. 3 Method In this section, we explain the underlying idea behind the proposed method and the overall architecture. 3.1 Idea Behind Proposed Architecture When training a GAN, the generator learns different levels of detail across its layers. The initial layers focus on broad, coarse features, while the later layers refine the finer details. The generator starts with a noise vector, progressively adding information with each layer, until the final output is a complete image. We can split the generator into two parts: the first part generates an embedding that captures the overall structure of the image, and the second part refines this into the full image. The point in the network where we divide the network is critical because the earlier layers are too abstract, and the later layers are too specific. A good division is usually somewhere in the middle, where the output is detailed enough to be informative. From another perspective, the output of the first layers can be seen as the embedding of the images, which encapsulates the knowledge of the global structure. A common approach in the literature [ 23 , 24 ] to provide more feedback on the first layers regarding the consistency in the final images is to use another GAN. In this work, we capture structures and dependencies by applying CRFs to the embeddings rather than training an additional GAN. CRFs are lightweight discriminative models that capture correlations between patches without producing extra