From Transthoracic to Transesophageal: Cross-Modality Generation using LoRA Diffusion
Emmanuel Oladokun, Yuxuan Ou, Anna Novikova, Daria Kulikova, Sarina Thomas, Jurica Šprem, Vicente Grau · 2025
Read From Transthoracic to Transesophageal: Cross-Modality Generation using LoRA Diffusion by Emmanuel Oladokun, Yuxuan Ou, Anna Novikova, Daria Kulikova, Sarina Thomas, Jurica Šprem, Vicente Grau (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13077.txt.
Introduction
Emmanuel Oladokun and co-authors teach an image generator trained on chest ultrasound of the heart to produce the throat-based kind instead, in a 2025 paper. Transesophageal images guide high-stakes care but are scarce, starving learning systems. The team adapted a mask-guided diffusion model with low-rank adapters as small as a hundred thousand parameters. A light extra layer translates new mask formats into ones the model already understands.
Tuning only part of the network sufficed for faithful synthesis. Mixing fewer than 200 real frames of the rare view with synthetic ones improved a segmentation task, especially on underrepresented right-heart structures. The recipe targets other data-poor medical imaging problems.
Catalog record
| CREATOR | Emmanuel Oladokun, Yuxuan Ou, Anna Novikova, Daria Kulikova, Sarina Thomas, Jurica Šprem, Vicente Grau |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Transthoracic to Transesophageal: Cross-Modality Generation using LoRA Diffusion Emmanuel Oladokun, Yuxuan Ou, Anna Novikova, Daria Kulikova, Sarina Thomas, Jurica Šprem, Vicente Grau Abstract Deep diffusion models excel at realistic image synthesis but demand large training sets—an obstacle in data-scarce domains like transesophageal echocardiography (TEE). While synthetic augmentation has boosted performance in transthoracic echo (TTE), TEE remains critically underrepresented, limiting the reach of deep learning in this high-impact modality. We address this gap by adapting a TTE-trained, mask-conditioned diffusion backbone to TEE with only a limited number of new cases and adapters as small as 10 5 10^{5} parameters. Our pipeline combines Low-Rank Adaptation with MaskR 2 , a lightweight remapping layer that aligns novel mask formats with the pretrained model’s conditioning channels. This design lets users adapt models to new datasets with a different set of anatomical structures to the base model’s original set. Through a targeted adaptation strategy, we find that adapting only MLP layers suffices for high-fidelity TEE synthesis. Finally, mixing less than 200 real TEE frames with our synthetic echoes improves the dice score on a multiclass segmentation task, particularly boosting performance on underrepresented right-heart structures. Our results demonstrate that (1) semantically controlled TEE images can be generated with low overhead, (2) MaskR 2 effectively transforms unseen mask formats into compatible formats without damaging downstream task performance, and (3) our method generates images that are effective for improving performance on a downstream task of multiclass segmentation. 1 Introduction Echocardiography (Echo) plays a pivotal role in cardiovascular care, providing a fundamental tool to evaluate and manage cardiac diseases [ 15 ] . In literature, the standard usage of ’echocardiogram’ often refers to transthoracic echocardiography (TTE) specifically, which captures images from outside the subject’s chest. Transesophageal Echocardiography (TEE) involves inserting a specialised probe with an ultrasound transducer into the esophagus. This allows for clearer and more precise imaging, as the esophagus is located close to the upper chambers of the heart and the probe is not occluded by the sternum and ribs [ 2 ] . TEE is used less frequently compared to TTE due to the more complex and more invasive setup, but is especially beneficial for the diagnosis of valvular diseases and guiding minimally invasive heart surgery such as the insertion of the mitral or tricuspid clip or closure of the left atrial appendage. Although there is much research on TTE image analysis [ 16 , 10 , 20 ] , TEE is less researched and lacks public resources. To address data scarcity, many resort to data augmentation, which has been shown to aid in the training of rigorous models with limited data [ 18 ] . Common augmentation methods such as standard geometric transformations or contrast adjustments, have limited use in echo. Moreover, geometric transforms could easily generate physiologically impossible images. Consequently, some works have resorted to training generative models to source augmented training data [ 16 , 10 , 20 ] . Generative models have significantly shaped medical image analysis and generation in recent years starting with Variational Autoencoders (VAE) [ 7 ] . VAEs provide a probabilistic framework for learning latent representations but are prone to blurry results which is a significant drawback for echocardiography. VAEs were followed by Generative Adversarial Networks (GAN) [ 1 ] that consist of a generator part, producing images, and a discriminator part verifying that the images look realistic. However, GANs are prone to mode collapse and training instabilities. In recent years, there has been a paradigm shift towards diffusion models which were first introduced by Sohl-Dickstein et al. [ 19 ] . Diffusion models have emerged as a powerful class of generative models, demonstrating state-of-the-art performance in image synthesis and various data-generation tasks. These models are based on a two-step process: a forward diffusion process, where noise is gradually added to the data over multiple steps, and a reverse denoising process, where a trained model gradually removes noise to reconstruct the original data. Diffusion models are highly effective for applications such as text-to-image generation and are established as a strong choice not only in the natural, but also in the medical image domain [ 22 , 24 ] for high-fidelity image generation. Recent literature has attempted TTE video synthesis using several public datasets and large models. Reynaud et al. [ 16 ] trained a text-to-video diffusion model to generate TTE videos with a user-specified ejection fraction. Nguyen et al. [ 10 ] synthesised TTE videos with a training-free approach using a 3D UNet. These achievements are made possible due to the availability of high-quality, rich, public TTE datasets [ 12 , 8 , 9 ] . Other research such as [ 20 ] has used diffusion models to generate key frames from TTE semantic maps. However, these approaches are limited to TTE. For TEE, [ 11 ] generated synthetic TEE images of key frames using a CycleGAN [ 25 ] and Contrastive Unpaired Translation method [ 13 ] . TEE is significantly underrepresented in the literature and there are very few resources (i.e. public datasets) available to enable this to change. This paper aims to tackle this underrepresentation with the following contributions: 1. We present a LoRA-based method for efficient training on limited echocardiography data. This enables generation of realistic synthetic TEE images that strongly respect image conditioning, helping address data scarcity in this domain. 2. We propose a targeted adaptation strategy that reveals the functional importance of different layers within a diffusion model, identifying which layers are most critical for adapting to echo data. 3. We introduce a mask adaptation scheme that transforms new semantic maps to match the expected input of pretrained base models. Despite this transformation, our synthetic data significantly boosts downstream multiclass segmentation performance, particularly for underrepresented classes. 2 Methods Diffusion Models In a typical diffusion model, there are two processes: forward and reverse. The forward process gradually adds Gaussian noise to a data sample {\bm{x_{0}}} such that at time t the sample \bm{x_{t}} has the following distribution: q(\bm{x_{t}}\mid\bm{x_{0}})=\mathcal{N}\left(\bm{x_{t}};\mu_{t}\bm{x_{0}},\,\sigma_{t}^{2}\mathbf{I}\right). (1) where \mu_{t} and \sigma_{t} are the mean and variance at time t . If t is chosen to be large enough, the image becomes indistinguishable from random noise. For the reverse process, a denoiser is trained to iteratively remove the added noise. In this work, we make use of the Elucidated Diffusion Model (EDM) [ 6 ] . EDM presents popular variants of diffusion models—such as variance-preserving, variance-exploding, and DDIM—within a unified framework that highlights key design choices contributing to generative performance. Karras et al. [ 6 ] identify two major sources of error in the reverse step: inaccurate denoising by the neural network, and the discrete solver steps that follow incorrect trajectories during sampling. To mitigate these issues, they use a second-order Heun solver for the reverse step and propose a range of conditioning strategies. Given a dataset with variance \sigma^{2}_{\text{data}} and a noise schedule with variance \sigma^{2}=\sigma^{2}(t) , EDM introduces the following preconditioning steps: scale the network input by c_{\text{in}}=\frac{1}{\sqrt{\sigma^{2}_{\text{data}}+\sigma^{2}}} ; scale the skip connections by c_{\text{skip}}=\frac{\sigma^{2}}{\sigma^{2}_{\text{data}}+\sigma^{2}} ; condition the network on noise using c_{\text{noise}}=\frac{1}{4}\ln(\sigma) ; and scale the output by c_{\text{out}}=\frac{\sigma\cdot\sigma_{\text{data}}}{\sqrt{\sigma^{2}_{\text{data}}+\sigma^{2}}} . The resulting EDM loss is: \mathds{E}_{\sigma,x,n}\|\underbrace{F_{\theta}\big(c_{\text{in}}(\sigma)\cdot\bm{\tilde{x}};c_{\text{noise}}(\sigma)\big)}_{\text{network output}}-\underbrace{\frac{1}{c_{\text{out}}(\sigma)}\left(y-c_{\text{skip}}(\sigma)\cdot\tilde{\bm{x}}\right)}_{\text{effective training target}}\|_{2}^{2} (2) where \bm{\tilde{x}}=\bm{x+n} , \bm{n}\sim\mathcal{N}(\bm{0},\sigma^{2}\bm{I}) , and F_{\theta} is