Language-Guided Trajectory Traversal in Disentangled Stable Diffusion Latent Space for Factorized Medical Image Generation

Zahra TehraniNasabMcGill UniversityMILA-Quebec AI Institute [email protected], Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected] · 2025

Read Language-Guided Trajectory Traversal in Disentangled Stable Diffusion Latent Space for Factorized Medical Image Generation by Zahra TehraniNasabMcGill UniversityMILA-Quebec AI Institute [email protected], Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23623.txt.

Introduction

Zahra TehraniNasab, Amar Kumar, and Tal Arbel steer a text-to-image model through its own latent space to edit medical images factor by factor, in a 2025 paper. Diffusion models make striking pictures from words, but separating one trait, anatomy say, from another, disease signs, is unproven in medicine. The team fine-tuned vision-language models on chest X-rays and skin photos.

The tuned models had quietly learned to factor the key traits apart. The authors built paths through latent space that slide one trait while holding the rest, enabling clean interpolations. Such control supports explaining disease and probing cause in synthetic cohorts. Extensive tests on both datasets back the claim that language guidance does the disentangling.

Catalog record

CREATORZahra TehraniNasabMcGill UniversityMILA-Quebec AI Institute [email protected], Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23623

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Language-Guided Trajectory Traversal in Disentangled Stable Diffusion Latent Space for Factorized Medical Image Generation Zahra TehraniNasabMcGill UniversityMILA-Quebec AI Institute [email protected], Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected] Abstract Text-to-image diffusion models have demonstrated a remarkable ability to generate photorealistic images from natural language prompts. These high-resolution, language-guided synthesized images are essential for the explainability of disease or exploring causal relationships. However, their potential for disentangling and controlling latent factors of variation in specialized domains like medical imaging remains under-explored. In this work, we present the first investigation of the power of pre-trained vision-language foundation models, once fine-tuned on medical image datasets, to perform latent disentanglement for factorized medical image generation and interpolation. Through extensive experiments on chest X-ray and skin datasets, we illustrate that fine-tuned, language-guided Stable Diffusion inherently learns to factorize key attributes for image generation, such as the patient’s anatomical structures or disease diagnostic features. We devise a framework to identify, isolate, and manipulate key attributes through latent space trajectory traversal of generative models, facilitating precise control over medical image synthesis. 1 1 footnotetext: Equal contribution. 1 Introduction Deep learning models for medical imaging have shown state-of-the-art performance across several tasks such as disease classification, image segmentation, drug discovery and high-resolution image synthesis. However, these models often struggle to generalize well to new, unseen data due to domain shifts Quiñonero-Candela et al. 2022 or entanglement in image features Meng et al. 2020 . For example, in medical imaging, multiple factors, such as disease pathology and imaging modality, are often intertwined, making it difficult for the model to isolate the relevant features for the specified task. This entanglement can lead to the model relying on spurious correlations, which may not be present in new data, thus hindering its ability to generalize. Thus, disentanglement aims to separate the various factors of variation in the data, allowing the model to learn more robust and interpretable representations. In computer vision, disentanglement models Higgins et al. 2017 ; Kim and Mnih 2018 ; Chen et al. 2018 have been shown to help increase generalization Long et al. 2015 ; Tzeng et al. 2017 and explainability Soelistyo and Lowe 2024 by isolating task-relevant features from confounding factors. This enables the model to adapt to new data distributions. In addition, disentanglement improves model explainability by separating the factors of variation and providing a clearer understanding of how different features contribute to the final prediction. Figure 1: Traversal along the latent trajectories of Stable Diffusion using language guidance. Given an initial chest X-ray projected onto latent space (start point), traversal along the trajectory is performed via language guidance. Sampling along the trajectory results in only a single attribute (e.g. “medical devices", “pleural effusion") being altered from the start point (“neutral"), while the patient identity is maintained. Vision-language foundation models have emerged as a powerful approach for learning disentangled representations, offering several advantages over traditional generative architectures due, in large part, to the enormous dataset size they have been trained on. Earlier approaches designed to disentangle latent factors relied on specialized generative architectures such as Variational Auto Encoders (VAE) Leeb et al. 2020 , which don’t produce high-resolution images required for medical imaging, or Generative Adversarial Networks (GAN) Chen et al. 2016 , which are notoriously hard to train. Normalizing Flows Pawlowski et al. 2020 have been adapted to address disentanglement in simpler contexts but are prohibitively computationally expensive. Other challenges faced when adapting traditional models include their difficulties in training them end-to-end Tang et al. 2021 ; Zhou et al. 2022 , and their requirement for specialized architectural components for conditioning such as AdaIN Huang and Belongie 2017 , FiLM Perez et al. 2018 , SPADE Park et al. 2019 . Furthermore, models often require specific heuristic to permit traversals in latent space Härkönen et al. 2020 ; Shen et al. 2020a ; Shen et al. 2020b ; Shen and Zhou 2021 . In contrast, vision-language foundation models offer more efficient and scalable solutions. The power of these foundation models to disentangle latent representations, enabling targeted image modifications while preserving semantic content and other attributes, has been explored in the natural imaging domain Wu et al. 2023a ; Wu et al. 2023b . However, this domain remains under-explored in the context of medical images, where complex entanglement is common and challenging to address. In this paper, we investigate the disentanglement capabilities of Stable Diffusion Rombach et al. 2022 , fine-tuned on medical images, and propose the first method to traverse latent space based on language guidance and see the factorizing effect on the resulting images. Language guidance is seen to be effective at identifying attribute-specific trajectories in the latent space (see Figure 1 ). Additionally, interpolations between samples permit continuous trajectories that can be sampled. Experiments illustrate that the samples exhibit the same disentangled properties, that is, the attribute of interest remains the same in the image throughout the trajectory, becoming more prevalent proportional to the distance from the starting point. We propose a new metric, Classifier Flip Rate along a Trajectory (CFRT), to validate the presence of the desired disentanglement. 2 Methodology Our method for building disentangled representations follows a two-stage process: (i) fine-tune the pre-trained Stable Diffusion on the relevant dataset (in this paper, CheXpert or ISIC); (ii) denoising diffusion implicit model (DDIM) Song et al. 2020 conditioned on the text embedding to synthesize images along the trajectory (see Figure 2 ). Figure 2: Reverse Diffusion for synthesizing disentangled images from text prompts using fine-tuned Stable Diffusion. The reverse diffusion process takes noisy latents, z_{T} , as input. A U-Net architecture generates the denoised latent, z_{0} . During the denoising process, the text embeddings, e_{t} ( t\in{0,..,T} ), from the pre-trained CLIP encoder are added to the latent via cross-attention modules Rombach et al. 2022 . Finally, the de-noised latent is passed through the decoder to create the synthesised image. Note that the text embeddings can be replaced at some intermediate timestep t during the reverse diffusion. 2.1 Text-to-Image Generation using fine-tuned Stable Diffusion Given a text embedding, e , the objective of the text-conditioned fine-tuned DDIM is to generate an image, X . The forward diffusion process starts at the timestep t=0 and noises the image X_{0} till timestep t=T resulting in a set of noisy images, X_{1:T} . The denoising process of the DDIM attempts to denoise from X_{T} to the original image X_{0} where a noisy image at each step is given by X_{t-1} : X_{t-1}=\gamma_{t_{0}}X_{t}+\gamma_{t_{1}}\epsilon_{0}\left(X_{t},t,e_{t}\right). (1) Here, \gamma_{t_{0}} and \gamma_{t_{1}} are computed as \gamma_{t_{0}}=\sqrt{\frac{\alpha_{t-1}}{\alpha_{t}}} and \gamma_{t_{1}}=\sqrt{1-\alpha_{t-1}}-\sqrt{\frac{\alpha_{t-1}}{\alpha_{t}}-\alpha_{t-1}} respectively with \alpha_{1:T} as the hyperparameters for the diffusion process. In the usual setting of text-to-image generation, a single text prompt can generate a single image. In this work, image generation depends on the diffusion timestep, t , to explore the properties of the latent representation. Thus, the image generation process, X_{0} , Wu et al. 2023a can be described as in Eq 2 , where \mathbf{g} is the generator: X_{0}=\mathbf{g}(X_{T},e_{1:T}). (2) Fine-tuning Stable Diffusion: Similar to Kumar et al. 2025 , CheXpert’s binary labels are converted into text captions to fine-tune Stable Diffusion v1.5 Rombach et al. 2022 , which utilizes CLIP language embeddings rather than numerical labels

Read the complete text