Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models
Felix Nützel 1, Mischa Dombrowski 1, Bernhard Kainz 1,2 · 2025
Read Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models by Felix Nützel 1, Mischa Dombrowski 1, Bernhard Kainz 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12236.txt.
Introduction
Felix Nützel and co-authors ground medical phrases in images with generative models in a 2025 paper. Phrase grounding links report words to the image spots they describe. It could pinpoint disease in scans from text alone. Contrastive models lead the field so far.
Diffusion image generators, the team show, ground phrases better with no task training. Their attention maps localize the named findings. Swapping in a chest-specific language model, kept frozen, doubles overlap scores over prior bests. A post-step, bimodal bias merging, aligns text and image leanings to sharpen the maps further. Generative models may be the stronger base for explainable medical imaging.
Catalog record
| CREATOR | Felix Nützel 1, Mischa Dombrowski 1, Bernhard Kainz 1,2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12236
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models Felix Nützel 1, Mischa Dombrowski 1, Bernhard Kainz 1,2 Abstract Phrase grounding, i.e. , mapping natural language phrases to specific image regions, holds significant potential for disease localization in medical imaging through clinical reports. While current state-of-the-art methods rely on discriminative, self-supervised contrastive models, we demonstrate that generative text-to-image diffusion models, leveraging cross-attention maps, can achieve superior zero-shot phrase grounding performance. Contrary to prior assumptions, we show that fine-tuning diffusion models with a frozen, domain-specific language model, such as CXR-BERT, substantially outperforms domain-agnostic counterparts. This setup achieves remarkable improvements, with mIoU scores doubling those of current discriminative methods. These findings highlight the underexplored potential of generative models for phrase grounding tasks. To further enhance performance, we introduce Bimodal Bias Merging (BBM), a novel post-processing technique that aligns text and image biases to identify regions of high certainty. BBM refines cross-attention maps, achieving even greater localization accuracy. Our results establish generative approaches as a more effective paradigm for phrase grounding in the medical imaging domain, paving the way for more robust and interpretable applications in clinical practice. The source code and model weights are available at https://github.com/Felix-012/generate_to_ground . \midlauthor 1 Friedrich-Alexander-Universität Erlangen-Nürnberg, DE 2 Imperial College London, UK † † volume: – 036 † † year: 2025 † † workshop: Full Paper – MIDL 2025 † † editors: Accepted for publication at MIDL 2025 keywords Phrase Grounding, Visual Grounding, Stable Diffusion, Latent Diffusion Models, Cross-Attention, Chest X-Rays 1 Introduction Phrase grounding refers to the ability of a model to map textual tokens to regions in an image. Unlike typical object detection or segmentation tasks, phrase grounding usually takes natural language as input, such as medical reports, instead of relying on a predefined set of categories. Thus, phrase grounding can be seen as a generalization of object detection. In the medical domain, phrase grounding can be used to localize anomalies in images based on textual descriptions provided by experts Bhalodia et al. 2021 . This is attractive, since it works without explicit labels, which are rare and expensive for medical data. By inspecting phrase grounding performance, it is possible to infer which phrases or regions influenced the decision of the system, assess whether the model made proper use of all available modalities, and if the modalities were aligned properly without confusion Parcalabescu and Frank 2020 . These properties are vital for interpretability, which is a necessary requirement for models to be used in the medical field Chen et al. 2023 . In addition, without interpretability, models could introduce harmful biases without explanations, which is especially critical for high-risk decision-making Hakkoum et al. 2022 . Existing discriminative medical phrase grounding approaches can be roughly categorized into two groups: supervised and self-supervised with contrastive learning Boecking et al. 2022 ; Gupta et al. 2020 ; Zhang et al. 2022 . An important supervised approach is MedRPG Chen et al. 2023 , which uses ground-truth bounding boxes of radiographs to formulate a contrastive loss based on the features of bounding boxes, as well as the joint attention of bounding boxes, the class token and an additional learnable token. Another relevant branch of medical phrase grounding methods are those that work with 3D medical data, such as the paper by Ichinose et al. 2023 , which addresses the unique issues of phrase grounding in CT scans. They suggest using a pre-trained segmentation model that labels the anatomic structures visible in the scan and introduce a module to structure the corresponding medical reports. However, such supervised methods require ground-truth bounding boxes or annotators, which are difficult to obtain, especially in the medical domain. Self-supervised methods do not require explicit labels but do not always lead to the desired result. Discriminative methods would usually evaluate their phrase grounding performance by computing the cosine similarity between the text embeddings and the corresponding image embeddings. However, it has recently been shown that phrase grounding tasks can also be solved using generative models in an unsupervised context Dombrowski et al. 2024 . Specifically, text-to-image Latent Diffusion Models (LDMs) are useful, due to their use of cross-attention to combine the two modalities, as well as their ability to produce high-quality images. Text-to-image LDMs are trained to generate images from a dataset while receiving additional text conditioning from the corresponding text inputs Dombrowski et al. 2023 ; Vilouras et al. 2024 . Instead of using cosine similarity, the phrase grounding capabilities of LDMs are easier to evaluate by using their cross-attention layers. Earlier, Dombrowski et al. 2024 showed that using a frozen text encoder improves the phrase grounding capabilities of an LDM. So far, the self-supervised approach by Boecking et al. 2022 achieved the highest phrase grounding performance on Chest X-ray (CXR) data by fine-tuning a Large Language Model (LLM) pre-trained on the biomedical domain on CXR reports. The resulting LLM is known as CXR-BERT. This model is jointly trained with an image encoder, in a framework called BioViL. In this work, we leverage these and use CXR-BERT as a frozen text encoder that conditions the U-Net in an LDM. Consequently, we inject the learned embeddings of CXR reports from CXR-BERT, while additionally fine-tuning the U-Net on corresponding CXR images. CXR-BERT and the LDM support each other in a bidirectional manner: the LDM, having a generative architecture, is able to leverage the full phrase grounding potential of the text embeddings compared to the simple CNN that is used in BioViL. Additionally, the powerful text embeddings learned by CXR-BERT provide the necessary conditioning to the LDM that enables the model to learn a well-grounded multimodal representation. As a result, the contributions of our work include the following: • We demonstrate that a multimodal text encoder with domain-specific knowledge can vastly improve phrase grounding capabilities of an LDM. • We show that generative approaches can yield far better phrase grounding results than traditional discriminative approaches by nearly doubling conventual performance metrics such as mIoU. • We discuss a novel post-processing method that can boost the phrase grounding capabilities of phrase grounding frameworks. Figure 1: During fine-tuning, radiological text reports are extracted (a) . These reports are fed into CXR-BERT with frozen parameters (b) . The resulting text embeddings are used to condition the U-Net in the LDM, by injecting the embeddings into each cross-attention layer (represented as gray bars) in the U-Net. The LDM learns to generate images by getting noisy radiology images corresponding to the reports and a timestep as input (c) . During evaluation, noisy ground-truth images are repeatedly fed into the LDM to extract the corresponding cross-attention. These maps are processed based on relevant tokens and to get them into the correct dimensionality (d) . After a processing step, we obtain an activation map and its corresponding binary mask (e) . For BBM, we need to extract the image bias (f) and generate the text bias (g) , merge them and combine them with our original activation map (h) . 2 Method 2.1 Exchanging Text Encoders in Stable Diffusion Our approach is based on fine-tuning Stable Diffusion (SD) Rombach et al. 2022 , a popular pre-trained text-to-image LDM. We only train the underlying U-Net of SD while keeping the