Structure causal models and LLMs integration in medical visual question answering
Zibo Xu, Qiang Li, Weizhi Nie*, Weijie Wang, Anan Liu · 2025
Read Structure causal models and LLMs integration in medical visual question answering by Zibo Xu, Qiang Li, Weizhi Nie*, Weijie Wang, Anan Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02703.txt.
Introduction
Zibo Xu and co-authors bring causal reasoning to medical image question answering in a 2025 paper. Doctors ask questions about scans, and models answer. Hidden confounders warp the link between image and question. That bias leads models to answers that sound right but reason wrong.
Their framework draws the image-question interaction as a causal graph. Questions shape which visual features matter, explicitly in the graph. Mutual information hunts down spurious correlations during training. A front-door resampling method then cancels the confounding effect. Cleaner causal answers could make medical AI safer to trust.
Catalog record
| CREATOR | Zibo Xu, Qiang Li, Weizhi Nie*, Weijie Wang, Anan Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02703
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structure causal models and LLMs integration in medical visual question answering Zibo Xu, Qiang Li, Weizhi Nie*, Weijie Wang, Anan Liu Abstract Medical Visual Question Answering (MedVQA) aims to answer medical questions according to medical images. However, the complexity of medical data leads to confounders that are difficult to observe, so bias between images and questions is inevitable. Such cross-modal bias makes it challenging to infer medically meaningful answers. In this work, we propose a causal inference framework for the MedVQA task, which effectively eliminates the relative confounding effect between the image and the question to ensure the precision of the question-answering (QA) session. We are the first to introduce a novel causal graph structure that represents the interaction between visual and textual elements, explicitly capturing how different questions influence visual features. During optimization, we apply the mutual information to discover spurious correlations and propose a multi-variable resampling front-door adjustment method to eliminate the relative confounding effect, which aims to align features based on their true causal relevance to the question-answering task. In addition, we also introduce a prompt strategy that combines multiple prompt forms to improve the model’s ability to understand complex medical data and answer accurately. Extensive experiments on three MedVQA datasets demonstrate that 1) our method significantly improves the accuracy of MedVQA, and 2) our method achieves true causal correlations in the face of complex medical data. I Introduction Medical Visual Question Answering (MedVQA) is an important branch of the Visual Question Answering (VQA) task [ 1 , 2 , 3 ] in the medical field. Its main goal is to accurately answer questions according to medical images, which improves medical diagnosis and treatment efficiency. The rise of this field highlights the potential of artificial intelligence in medical diagnosis [ 4 , 5 , 6 ] , providing new possibilities for improving the interpretation of multi-modal medical data and clinical decision-making. Besides, large language models (LLMs) have shown great potential in addressing the MedVQA task by enhancing the understanding of complex medical questions and generating accurate answers [ 7 , 8 , 9 ] . Fig. 1: Illustration of how the confounder C affects model inference. The imbalanced training set leads to data bias, which can manifest in various ways, such as (a) difficulty in learning pathology, (b) statistical association, (c) relativity of confounders, and (d) co-occurrence of multiple pathologies, among others. These biases introduce confounders during inference, creating misleading causal paths. As a result, the model relies on spurious correlations rather than true causal relationships, leading to incorrect predictions. Common VQA tasks [ 2 , 1 ] are designed to accurately respond to multi-modal inputs comprising images and queries, often focusing on natural images with diverse visual content. These tasks benefit from large-scale datasets and standardized benchmarks that allow models to learn generalizable patterns and associations effectively. However, MedVQA introduces unique challenges compared to common VQA tasks, including the complexity of medical images, which contain specialized and intricate details that require domain-specific knowledge to interpret [ 10 , 11 ] . Moreover, MedVQA is particularly prone to confounding effects, such as spurious correlations between visual and textual features, which can mislead models and undermine their reliability in clinical applications. Recent MedVQA methods [ 12 , 9 , 8 ] generally rely on large amounts of medical data for pre-training to build visual encoders or language models with medical knowledge. Some of these approaches [ 12 , 8 ] employ pre-trained encoders to build unique integrated models or introduce a projection module [ 9 ] to ensure alignment between medical images and questions. Although these methods have made remarkable achievements in improving the accuracy of MedVQA, they rely mainly on statistical associations and ignore the potential impact of spurious causation [ 13 ] on the inference process. In the MedVQA field, it’s a challenge to understand the rationale behind the answer [ 9 ] . Models may fail to identify confounders and spurious associations without considering the effects of multi-modal data bias [ 14 , 15 , 16 ] . In this context, excessive reliance on statistical associations may lead to wrong decisions. When the model answers different medical questions, confounding factors often involve different areas in the image. It means that confounders corresponding to one question may be the real reason for another question, and this relativity is a difficult challenge in MedVQA tasks. In Fig. 1 , the medical training set is highly imbalanced, which introduces data bias and affects model inference. This data bias manifests in multiple ways: (a) difficulty in learning pathology for diseases with fewer samples, (b) statistical associations where certain regions are more frequently linked to specific diseases, (c) relative confounding, where the causal region for one question might act as a confounder for another, and (d) co-occurrence of multiple pathologies, influencing the model’s learning preferences. During training, these biases introduce confounders into the causal path, leading to spurious learning pathways and ultimately affecting model predictions. For instance, when answering whether there is infiltration in the lower right lung, the model provides the correct answer. This is because the question provides sufficient information about both the pathology and location, and due to the larger number of infiltration samples in the training set, the model has learned the relevant features more effectively. However, when the model is asked about the location of pleural effusion, the data bias becomes more evident. The training set contains significantly more cases of infiltration in the lower right lung compared to pleural effusion, and both pathologies share similar visual features. Due to the training data imbalance, the model is exposed to fewer pleural effusion cases, making it harder to learn its distinguishing features. As a result, during inference, the model relies more on frequently seen infiltration patterns, leading to incorrect predictions based on spurious correlations rather than true causal relationships. The correct use of LLMs in answer generation is also a challenge [ 3 ] . When only a single question is provided as textual input, LLMs tend to produce a large number of irrelevant or non-answering statements as answers. This can lead to non-standard answers and a lack of practical reference value. Above all, the issues in the MedVQA task can be summarized as the following two challenges: 1) How to discover medical multi-modal data bias and eliminate its effects? The complexity of medical data leads to biases that are difficult to observe and understand [ 17 , 18 , 19 , 20 ] . Medical images often contain intricate details that may introduce spurious associations when paired with questions, leading to confounded inference results [ 13 , 21 ] . These biases are not only challenging to detect but are also variable, with different questions corresponding to different biases within the same medical image. This variability increases the difficulty for models to accurately interpret inputs and deduce correct answers. The presence of confounders can cause the model to focus on irrelevant or misleading associations, ultimately compromising the reliability of the diagnosis or treatment suggestions provided. 2) How can large language models learn to generate answers with practical reference value? Large Language Models (LLMs) have been introduced in MedVQA due to their exceptional capabilities in understanding