Hybrid Diffusion Model for Breast Ultrasound Image Augmentation

Farhan Fuad Abir, Sanjeda Sara Jennifer, Niloofar Yousefi, Laura J. Brattain · 2026

Read Hybrid Diffusion Model for Breast Ultrasound Image Augmentation by Farhan Fuad Abir, Sanjeda Sara Jennifer, Niloofar Yousefi, Laura J. Brattain (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26834.txt.

Introduction

Abir and co-authors propose a hybrid diffusion method to expand breast ultrasound training data. Their pipeline mixes text-to-image generation with image-to-image refinement. It also fine-tunes the model with low-rank adaptation and textual inversion. The goal is realistic images that keep true ultrasound texture. On the open BUSI dataset, the FID score improved from 45.97 to 33.29. Downstream classification performance stayed comparable to the baseline approach.

Catalog record

CREATORFarhan Fuad Abir, Sanjeda Sara Jennifer, Niloofar Yousefi, Laura J. Brattain
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12325
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26834

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hybrid Diffusion Model for Breast Ultrasound Image Augmentation Farhan Fuad Abir, Sanjeda Sara Jennifer, Niloofar Yousefi, Laura J. Brattain Abstract We propose a hybrid diffusion-based augmentation framework to overcome the critical challenge of ultrasound data augmentation in breast ultrasound (BUS) datasets. Unlike conventional diffusion-based augmentations, our approach improves visual fidelity and preserves ultrasound texture by combining text-to-image generation with image-to-image (img2img) refinement, as well as fine-tuning with low-rank adaptation (LoRA) and textual inversion (TI). Our method generated realistic, class-consistent images on an open-source Kaggle breast ultrasound image dataset (BUSI). Compared to the Stable Diffusion v1.5 baseline, incorporating TI and img2img refinement reduced the Fréchet Inception Distance (FID) from 45.97 to 33.29, demonstrating a substantial gain in fidelity while maintaining comparable downstream classification performance. Overall, the proposed framework effectively mitigates the low-fidelity limitations of synthetic ultrasound images and enhances the quality of augmentation for robust diagnostic modeling. † † address: 1 Electrical and Computer Engineering, University of Central Florida 2 Department of Medicine, University of Central Florida College of Medicine 3 Department of Computer Science, University of Central Florida 4 Industrial Engineering and Management Systems, University of Central Florida 1 Introduction Ultrasound is a key medical imaging modality for breast cancer diagnosis due to its affordability, non-invasive nature, and real-time imaging capabilities [ 1 ] . However, deep learning–based ultrasound interpretation faces persistent challenges due to limited and imbalanced datasets that constrain model generalizability across lesion types. Traditional augmentation techniques, such as flipping, rotation, and intensity variations, offer only limited diversity [ 2 ] . Early generative approaches using Generative Adversarial Networks (GANs) often produce artifacts and fail to replicate the speckle noise and subtle tissue textures critical for diagnosis [ 3 ] . Diffusion models progressively denoise latent representations, yielding structural detail essential for ultrasound synthesis. These models have been widely used for denoising, despeckling [ 4 , 5 ] and text-conditioned generation of echocardiograms or breast ultrasound (BUS) images [ 6 , 7 ] . These methods showed promising results of using diffusion-based image generation for BUS. However, existing diffusion-based BUS generation tends to produce overly smooth images that lack the characteristic speckle noise and structural complexity of real BUS images. To address the research gap, we propose a hybrid diffusion augmentation approach to improve the fidelity of ultrasound synthesis and the severe class imbalance in breast ultrasound datasets. Our framework integrates semantic conditioning with text-to-image (text2img) generation and structural refinement with image-to-image (img2img) diffusion [ 8 ] . First, we implemented Stable Diffusion v1.5 (SD1.5) [ 8 ] , fine-tuned with low-rank adaptation (LoRA) [ 9 ] and enhanced with textual inversion (TI) [ 10 ] to embed domain-specific ultrasound concepts into the model’s latent space. Then, we use img2img for retaining fine-grained BUS structures. This hybrid strategy retains ultrasound texture, tissue heterogeneity, and lesion boundaries more faithfully than other diffusion-based approaches. We used the proposed framework to generate synthetic samples for underrepresented classes, addressing data imbalance in the open-source Kaggle breast ultrasound image (BUSI) dataset [ 11 ] . By producing class-consistent and anatomically coherent ultrasound images, the method improves downstream classification performance. 2 Methods Figure 1: Overview of the proposed hybrid diffusion-based image generation framework for breast ultrasound. The method consists of three main stages. (a) The preprocessing stage converts the labels into descriptive prompts. (b) LoRA finetuning and Token Generation adapts the Stable Diffusion v1.5 using LoRA for image–prompt alignment and Textual Inversion for learning domain-specific <ultrasound> token. (c) In the final workflow, the prompts and learned token <ultrasound> are passed through the finetuned text-to-image (text2img) model to generate synthetic images, which are further refined using an image-to-image (img2img) stage with LoRA weights, yielding the final synthetic ultrasound images. 2.1 Dataset Description We used the Kaggle open-source Breast Ultrasound Images (BUSI) dataset consisting of 780 images categorized as benign (437), malignant (210), and normal (133) [ 11 ] . All images were resized and normalized. The train-validation split (80-20) had 623 training images (349 benign, 168 malignant, 106 normal) and 157 validation images (88 benign, 42 malignant, 27 normal), with benign being the majority class. 2.2 Hybrid Diffusion Model Fig. 1 illustrates the three-stage model workflow: prompt generation, model fine-tuning, and synthetic image generation. Prompt Generation. We derived class-specific prompts from BUSI labels (normal, benign, malignant) to guide the generation process. These prompts were mapped to radiology-style descriptions such as “ultrasound image of a benign (or malignant) breast lesion.” For the normal class, the prompt was: “ultrasound image of normal breast tissue.” Model Fine-Tuning. Firstly, we used low-rank adaptation (LoRA) to fine-tune the Stable Diffusion model with the images and associated text prompts. This facilitated domain-specific feature learning while avoiding full-scale model pretraining. Then we introduced a custom token <ultrasound> using TI to enhance the model’s understanding of ultrasound-specific prompts. TI learned a new embedding that captured characteristic texture and speckle patterns from representative BUSI images. This embedding was incorporated into all prompts to strengthen domain alignment and ensure consistent synthesis across BUS classes. Synthetic Image Generation. At first, the TI embedding was imported alongside the LoRA adapters. Then we generated the synthetic BUS images from the finetuned text2img SD1.5 model. Additionally we applied Stable Diffusion’s img2img model to enhance the fidelity of the generated outputs. This refinement process was performed with an empirically chosen low denoising strength of 0.3. Afterward, we added synthetic images to each class of the training set, yielding 350 samples per class to match the majority class, while the validation set remained unchanged. 2.3 Evaluation Metrics We used accuracy, F1-score, Area Under the Receiver Operating Characteristic Curve (AUC-ROC), positive predictive value (PPV), and recall to evaluate classification performance. Moreover, we used the Fréchet Inception Distance (FID), which quantifies the similarity between real and synthetic image distributions. FID scores were computed using Inception v3 features on 780 real and synthetic images. Table 1: Impact of each component in the hybrid diffusion framework on classification and image quality metrics. Components Accuracy \uparrow F1-Score \uparrow AUC-ROC \uparrow PPV \uparrow FID \downarrow Baseline (Original Images) 0.904 0.887 0.979 0.890 - Original + SD1.5 0.917 0.905 0.986 0.901 45.97 Original + SD1.5 + img2img 0.898 0.879 0.978 0.878 38.34 Original + SD1.5 + TI 0.924 0.912 0.980 0.906 45.66 Original + SD1.5 + TI + img2img 0.905 0.884 0.975 0.89 37.18 Figure 2: Comparison of real ultrasound images and synthetic variants generated by 4 different approaches based on SD1.5. Rows correspond to breast lesion categories: benign (top), malignant (middle), and normal (bottom). Columns show (from left to right): real images, baseline SD1.5 generations, SD1.5 with img2img refinement, SD1.5 with TI, and SD1.5 combined with TI and img2img. The img2img refinement increases the fidelity by improving ultrasound textures. 3 Experiments and Results 3.1 Experimental Setup We conducted five experiments using original ultrasound images as the baseline. ResNet18 was used to classify the three classes. (1) Baseline (Original): trained only on original ultrasound images; (2) Original + SD1.5: augmented with synthetic samples generated by SD1.5; (3) Original + SD1.5 + img2img: adding img2img translation for style adaptation; (4) Original + SD1.5 + TI: leveraging textual inversion for domain-specific embedding alignment; and (5) Original + SD1.5 + TI + img2img: combining all components. ResNet18 was trained with the

Read the complete text