Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imaging
Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected], Anita KrizMcGill UniversityMILA-Quebec AI Institute [email protected] Barak PertzovMcMaster University [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected] · 2025
Read Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imaging by Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected], Anita KrizMcGill UniversityMILA-Quebec AI Institute [email protected] Barak PertzovMcMaster University [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23618.txt.
Introduction
Amar Kumar and co-authors ask whether vision-language models can expose hidden patterns in medical image data, in this 2025 paper. Such models link images and text and now guide image editing. The team fine-tuned one on chest X-rays and tested how well it edits images along chosen traits. They compared it with causal model methods that need fine labels.
The tuned model produced sharp, precise edits and scored better on many measures. It also surfaced links between image traits that coarse records had hidden before. At the same time, the edits were not always accurate. The model picked up bias and false links in the data. The authors say these models can reveal data traits, but their edits need careful checks before use.
Catalog record
| CREATOR | Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected], Anita KrizMcGill UniversityMILA-Quebec AI Institute [email protected] Barak PertzovMcMaster University [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23618
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imaging Amar KumarMcGill UniversityMILA-Quebec AI Institute [email protected], Anita KrizMcGill UniversityMILA-Quebec AI Institute [email protected] Barak PertzovMcMaster University [email protected] Tal ArbelMcGill UniversityMILA-Quebec AI Institute [email protected] Abstract Vision-language foundation models (VLMs) have shown impressive performance in guiding image generation through text, with emerging applications in medical imaging. In this work, we are the first to investigate the question: ‘Can fine-tuned foundation models help identify critical, and possibly unknown, data properties?’ By evaluating our proposed method on a chest x-ray dataset, we show that these models can generate high-resolution, precisely edited images compared to methods that rely on Structural Causal Models (SCMs) according to numerous metrics. For the first time, we demonstrate that fine-tuned VLMs can reveal hidden data relationships that were previously obscured due to available metadata granularity and model capacity limitations. Our experiments demonstrate both the potential of these models to reveal underlying dataset properties while also exposing the limitations of fine-tuned VLMs for accurate image editing and susceptibility to biases and spurious correlations. 1 1 footnotetext: Equal contribution. 1 Introduction Dataset biases present a significant challenge in the development of trustworthy machine learning models in healthcare applications. These biases can manifest as spurious correlations – misleading patterns between attributes of the data – which cause the model to learn incorrect associations that do not reflect true clinical relationships. For instance, algorithms trained on chest X-ray learned to classify pneumothorax based on the presence of a chest drain, which is inserted after pneumothorax diagnosis to treat it, and thus does not reflect the true pathological disease markers Banerjee et al. 2023 . Similarly, dermatological diagnostic models demonstrated significantly decreased accuracy when analyzing underrepresented skin types, reducing diagnostic reliability Kleinberg et al. 2022 . These learned biases can lead to potentially harmful clinical outcomes when models are deployed in real-world healthcare settings, where they will fail to generalize beyond the underlying distributions of their training data Yoon et al. 2023 . A particularly concerning aspect of these biases is that they may be unobserved within the metadata, making them difficult to identify through standard statistical analyses. By developing methodologies to identify and mitigate dataset bias, we can address the critical need for robust and reliable AI tools prior to their deployment in clinical settings. Causal-based approaches aim to answer causal queries and consequentially overcome challenges with spurious correlations by explicitly enforcing attribute relationships within a causal framework. These methods incorporate causal principles by embedding Structural Causal Models (SCMs) into the generation process Pawlowski et al. 2020 ; Glocker et al. 2023 ; Xia et al. 2021 and adhering to Pearl’s Causal Hierarchy Theorem (CHT) Pearl 2000 ; Pearl 2009 to generate results. However, many SCM-based methods still require external classifiers to function effectively Pawlowski et al. 2020 ; Pan and Bareinboim 2024 . Moreover, despite their theoretical rigor, the assumptions these methods require for theoretical validity often become unrealistic in real-world applications. In particular, their constrained ability to generate high-resolution images results in model outputs that may not capture underlying dataset biases due to limited editing precision. Foundation models, trained on vast and diverse datasets, have demonstrated remarkable generative capabilities in both computer vision Yuan et al. 2021 and medical imaging Azad et al. 2023 ; Xu et al. 2023 ; Liu et al. 2023 . Two significant breakthroughs are particularly relevant: (i) enhanced image generation capabilities that enables the generation of unprecedented high-resolution images Kumar et al. 2025 , and (ii) precise targeted editing of specific image attributes Lang et al. 2021 . Despite these capabilities, important questions remain about what (and how) these models learn (e.g. the field of mechanistic interpretability) and how the underlying data distribution impacts them. In this work, we explore the strengths and limitations of fine-tuned VLMs in generating high-resolution, realistic medical images while analyzing their dependence on, and ability to reveal, underlying dataset properties. We first demonstrate that a non-SCM VLM model can generate high-resolution, realistic images faithful to the original (factual) images when compared to an SCM-based model. We quantitatively evaluate both methods using perceptual similarity to the factual image, pixel-wise differences that measure fidelity to expected transformations, and effectiveness ensuring counterfactuals induce the intended change while preserving unrelated attributes. Beyond demonstrating high-quality image editing with fine-tuned VLMs, we further analyze how dataset properties influence their learning process. By leveraging these high-capacity models for precise image editing, we: (i) Reveal Dataset Properties that may not be explicitly captured in metadata labels, and thus not detectable through traditional statistical analyses, and (ii) Demonstrate Failure Modes that show how these models deviate from the true data-generating process, revealing their reliance on spurious correlations and dataset-specific artifacts Gupta et al. 2024 . Figure 1: Comparison of counterfactual generation in the SOTA structural causal model (SCM) and a foundation vision-language model. Inference involves generating images with increased age, which may have downstream effects on disease. Boxes highlight changes related to disease state, while boxes indicate changes associated with age (e.g., decreased bone and tissue density). Given that the SCM model is built upon an HVAE and cannot generate high-resolution images, it is limited in its ability to capture plausible counterfactual generation at finer detail levels. 2 Methodology 2.1 Background Structural Causal Models & Counterfactual Generation Our work investigates how fine-tuned vision-language foundation models can generate high-resolution, faithful counterfactual images. We discuss counterfactual generation using Structural Causal Models (SCMs) to provide context. These SCMs enable counterfactual reasoning for answering questions such as: What would image X look like if attribute A_{i}=a had instead been A_{i}=a^{\prime} ? This is achieved by intervening on specific variables while preserving the underlying causal mechanisms. According to Pearl’s Causal Hierarchy Theorem (CHT), counterfactual generation follows three key steps: (i) Abduction: The structural equations are inverted to infer the exogenous noise variables U that explain the observed image and attributes;(ii) Action: An intervention is performed on attribute A , changing its value from A=a to A=a^{\prime} , represented using the do-operator as do(A=a^{\prime}) ; (iii) Prediction: The modified attribute A=a^{\prime} is propagated through the structural equations, generating a new counterfactual image X^{a^{\prime}}=f(A=a^{\prime},U) , while preserving U to maintain consistency with the original latent factors. Stable Diffusion Stable Diffusion Rombach et al. 2022 , our chosen generative model for counterfactual generation, is comprised of four components: (i) an Image encoder that transforms input images into low-dimensional latent representations capturing essential features; (ii) a CLIP text encoder that provides text conditioning to guide generation and ensure alignment between synthesized images and text prompts; (iii) a U-net denoiser that forms the core of the reverse diffusion process, receiving latent representations and iteratively removing Gaussian noise across multiple timesteps by predicting and eliminating noise from the latent representation. In particular, Denoising Diffusion Implicit Models (DDIM) allows for a more efficient and deterministic sampling process by eliminating the need for all timesteps in the reverse diffusion process, resulting in faster generation with high-quality outputs; and (iv) an Image decoder that converts denoised latents back to the original image space, producing high-resolution output images. By utilizing a Stable Diffusion model pre-trained on extensively large datasets, we can leverage its learned representations. Fine-tuning the model on our dataset then allows us to