Pixels Under Pressure: Exploring Fine-Tuning Paradigms for Foundation Models in High-Resolution Medical Imaging

Zahra TehraniNasab, Amar Kumar, Tal Arbel · 2025

Read Pixels Under Pressure: Exploring Fine-Tuning Paradigms for Foundation Models in High-Resolution Medical Imaging by Zahra TehraniNasab, Amar Kumar, Tal Arbel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14931.txt.

Introduction

Zahra TehraniNasab, Amar Kumar, and Tal Arbel study fine tuning of image generators for medical imaging in a 2025 paper. Large diffusion models make striking pictures, yet most work stops at low resolution. Clinics need high resolution output that matches their data, and tuning the big models is the bridge.

The team compares full tuning with light weight tuning methods at five hundred twelve pixel square size. Scores cover likeness to real images, variety, and match to the text prompt. Generated images then serve as training data for a classifier tested on real scans, a tough low data setting. Some tuning routes lifted both picture quality and classifier results. The study gives labs a map for adapting generators responsibly.

Catalog record

CREATORZahra TehraniNasab, Amar Kumar, Tal Arbel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14931

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pixels Under Pressure: Exploring Fine-Tuning Paradigms for Foundation Models in High-Resolution Medical Imaging Zahra TehraniNasab, Amar Kumar, Tal Arbel Abstract Advancements in diffusion-based foundation models have improved text-to-image generation, yet most efforts have been limited to low-resolution settings. As high-resolution image synthesis becomes increasingly essential for various applications, particularly in medical imaging domains, fine-tuning emerges as a crucial mechanism for adapting these powerful pre-trained models to task-specific requirements and data distributions. In this work, we present a systematic study, examining the impact of various fine-tuning techniques on image generation quality when scaling to high resolutions ( 512 × 512 512\times 512 pixels). We benchmark a diverse set of fine-tuning methods, including full fine-tuning strategies and parameter-efficient fine-tuning (PEFT). We dissect how different fine-tuning methods influence key quality metrics, including Fréchet Inception Distance (FID), Vendi score, and prompt-image alignment. We also evaluate the utility of generated images in a downstream classification task under data-scarce conditions, demonstrating that specific fine-tuning strategies improve both generation fidelity and downstream performance when synthetic images are used for classifier training and evaluation on real images. Our code is accessible through the project website 1 1 1 https://tehraninasab.github.io/PixelUPressure/ . 1 Introduction Text-to-Image Foundation models have demonstrated remarkable success across various computer vision tasks, consistently achieving strong performance on standard benchmarks [ 12 , 4 ] . Trained on massive corpora of natural images, these models acquire visual representations, such as textures, shapes, and complex spatial patterns, that often transfer effectively to medical imaging domains. This transferability is particularly valuable in clinical settings, where the availability of labeled or paired text-image medical data is scarce. Given the challenges of training large-scale models from scratch with small, specialized datasets, fine-tuning has emerged as a practical and effective strategy for adapting foundation models to medical applications [ 23 , 14 , 18 ] . Fine-tuning leverages the rich visual priors learned during pretraining, enabling models to be adapted to domain-specific tasks through targeted updates, rather than full retraining [ 25 , 1 ] . In this work, we focus on Stable Diffusion [ 17 ] v1.5, a prominent latent diffusion model pre-trained on large-scale natural image-text pairs. While its latent-space architecture enables more efficient high-resolution synthesis compared to pixel-space alternatives, fine-tuning Stable Diffusion on high-resolution medical images still poses significant computational challenges. Scaling to high resolution rapidly increases memory and compute costs, especially for attention-based architectures, posing significant barriers to deployment [ 25 ] . These challenges introduce a trade-off between model capacity and computational feasibility, often leading to compromises in generative quality or diagnostic reliability. Addressing this tension is essential for enabling the practical and scalable use of text-to-image foundation models in real-world medical imaging workflows [ 5 ] . Recent developments in parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) [ 8 ] , DoRA (Decoupled Rank Adaptation) [ 15 ] , BitFit [ 24 ] and Diffusion-specific PEFT (DiffFit) [ 22 ] , offer promising solutions to these computational scaling challenges by selectively updating model components rather than the entire parameter space [ 5 ] . Adapter modules [ 13 , 2 ] offer an alternative approach by inserting small, trainable layers between frozen, pre-trained components, enabling domain-specific adaptation without modifying the original model weights. Despite these methodological advances and their demonstrated effectiveness on standard resolution tasks, the performance characteristics and scaling behaviour of these parameter-efficient approaches in high-resolution medical imaging contexts ( 512\times 512 ) remain largely unexplored, particularly regarding their ability to preserve critical diagnostic information while maintaining computational efficiency as input dimensions increase substantially. Previous works by Dutt et al. [ 5 ] have analyzed the effect of different strategies to fine-tune at 224\times 224 resolution but did not focus on the effect of fine-tuning on image generation quality at higher resolution. Recent work by Davila et al. [ 3 ] have compared fine-tuning strategies for image classification in medical imaging, but these images are also at a low resolution of 320\times 320 pixels. This paper presents a comprehensive study of fine-tuning paradigms for high-resolution image generation using Stable Diffusion, a pre-trained diffusion-based text-to-image foundation model. The key contributions of our work are as follows: • A systematic comparison of fine-tuning strategies—including full fine-tuning and parameter-efficient approaches such as LoRA [ 8 ] , DoRA [ 15 ] , BitFit [ 24 ] , and DiffFit [ 22 ] —for 512\times 512 image synthesis. • An in-depth analysis of how these fine-tuning strategies affect generation quality, including visual fidelity (measured by Fr’echet Inception Distance, FID [ 7 ] ), diversity (via Vendi Score [ 6 ] ), and prompt-image consistency (evaluated using a classifier-based metric). • A downstream evaluation where classifiers are trained on synthetic images and tested on real data, assessing the utility of generated images for real-world diagnostic tasks. 2 Methodology Figure 1: Overview of our architecture. Stable Diffusion v1.5 model is adapted for high-resolution chest X-ray generation using different fine-tuning strategies. The PEFT module shown within the U-Net is only used in parameter-efficient fine-tuning configurations (e.g., LoRA, DoRA, BitFit); in full fine-tuning settings, the U-Net (and optionally the VAE and text encoder) are directly fine-tuned without PEFT modules. 2.1 Fine-tuning Strategies We evaluate diverse fine-tuning configurations (Figure 1 ) for high-resolution image generation through an extensive set of experiments covering combinations of VAE, U-Net, and text encoder using full and parameter-efficient fine-tuning (PEFT); see Table 1 . Following [ 11 , 21 ] , we generate image-text pairs from tabular data using the template - Chest X-ray of a subject with [disease(s) 2 2 2 The diseases include: No Finding , Enlarged Cardiomediastinum , Cardiomegaly , Lung Opacity , Lung Lesion , Edema , Consolidation , Pneumonia , Atelectasis , Pneumothorax , Pleural Effusion , Pleural Other , Fracture , and Support Devices . ] . Table 1: Summary of fine-tuning strategies. ✗: Frozen, ✓: Trainable #Model VAE Text Encoder U-Net Description / Trainable A. Full Fine-Tuning Strategies 1 ✗ ✗ ✓ U-Net only 2 ✗ ✓ ✗ Text Encoder only 3 ✓ ✗ ✗ VAE only 4 ✗ ✓ ✓ Text encoder + U-Net 5 ✓ ✗ ✓ VAE + U-Net 6 ✓ ✓ ✗ VAE + Text Encoder 7 ✓ ✓ ✓ U-Net + VAE + Text Encoder B. Parameter-Efficient Fine-Tuning on U-Net 8 ✗ ✗ ✓ LoRA [ 8 ] 9 ✗ ✗ ✓ DoRA [ 15 ] 10 ✗ ✗ ✓ BitFit: Only bias terms updated [ 24 ] 11 ✗ ✗ ✓ DiffFit: Diffusion-specific method [ 22 ] Full Component Fine-Tuning For the full component fine-tuning experiments (Models 1–7), we explored the effects of selectively training different combinations of the VAE, Text Encoder, and U-Net modules. This approach allowed us to isolate the contribution of each component to the overall performance of the diffusion model. Some models focused on fine-tuning a single component while freezing the others to assess its standalone impact. For example, Model 1 fine-tuned only the U-Net, while Models 2 and 3 focused solely on the Text Encoder and VAE, respectively. Others involved combinations of two components (Models 4–6) or all three (Model 7). Parameter Efficient Fine-Tuning To improve efficiency and deployment flexibility, we explored four prominent parameter-efficient fine-tuning strategies: • Low-Rank

Read the complete text