Computed Tomography Visual Question Answering with Cross-modal Feature Graphing

Yuanhe Tian, Chen Su, Junwen Duan, Yan Song · 2025

Read Computed Tomography Visual Question Answering with Cross-modal Feature Graphing by Yuanhe Tian, Chen Su, Junwen Duan, Yan Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04333.txt.

Introduction

Yuanhe Tian, Chen Su, Junwen Duan, and Yan Song build a system that answers doctors' questions about CT scans. Tools of this kind usually encode the images and the question separately, then merge them. That loses how slices of a scan connect to each other in three dimensions, and to the words of the question.

Their design weaves slices and question words into one graph, where each slice and each word is a node. A graph network then passes information along the links, gathering the context that matters for the question. The blended features steer a large language model as it writes the answer. On a standard medical benchmark, the system beat earlier approaches across the board, which the authors credit to the graph's richer reasoning.

Catalog record

CREATORYuanhe Tian, Chen Su, Junwen Duan, Yan Song
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Computed Tomography Visual Question Answering with Cross-modal Feature Graphing The code is available at https://github.com/synlp/A-GCN-CTVQA . Yuanhe Tian, Chen Su, Junwen Duan, Yan Song Abstract Visual question answering (VQA) in medical imaging aims to support clinical diagnosis by automatically interpreting complex imaging data in response to natural language queries. Existing studies typically rely on distinct visual and textual encoders to independently extract features from medical images and clinical questions, which are subsequently combined to generate answers. Specifically, in computed tomography (CT), such approaches are similar to the conventional practices in medical image analysis. However, these approaches pay less attention to the spatial continuity and inter-slice correlations in the volumetric CT data, leading to fragmented and imprecise responses. In this paper, we propose a novel large language model (LLM)-based framework enhanced by a graph representation of salient features. Different from conventional multimodal encoding strategies, our approach constructs a cross-modal graph integrating both visual and textual features, treating individual CT slices and question tokens as nodes within the graph. We further leverage an attentive graph convolutional network to dynamically fuse information within this structure. The resulting aggregated graph features then serve as a soft prompt to guide a large language model in generating accurate answers. Extensive experiments on the M3D-VQA benchmark demonstrate that our approach consistently outperforms baselines across multiple evaluation metrics, offering more robust reasoning capabilities. I Introduction Visual question answering (VQA) attracts significant attention in both academic and industrial communities, owing to its capacity to integrate computer vision and natural language processing for knowledge inference. Particularly, in the context of medical imaging, VQA holds immense potential for assisting clinical diagnosis and decision-making. Especially for computed tomography (CT), its multi-slice nature offers VQA greater value since processing many images at a time costs physicians much labor and often causes problems such as inaccurate interpretation of the input or false answers. As illustrated in Figure 1 , CT scans provide three-dimensional information by stacking multiple slices, providing rich anatomical and pathological details, which significantly differs from typical conventional images (e.g., X-ray radiographs). Therefore, CT VQA requires accurate analysis of multiple CT slices rather than one or a few images, so that raises substantial challenges that the VQA model needs to handle voluminous data, effectively leverage cross-slice correlations, and interpret the spatial complexity across slices to accurately answer clinically relevant questions. Fig. 1 : Comparison between CT slices and X-ray radiographs, where a CT volume generally contains much more images than an X-ray radiograph case. Fig. 2 : An overview of our approach for CT VQA. The bottom part illustrates the input CT slices with the question, the vision and text encoder, the cross-modal feature graphing, and the LLM to generate the answer. The top part presents the graph built over the CT slices and question tokens, as well as the attention mechanism used in the cross-modal feature graphing for distinguishing important slices for answering the question. Existing medical VQA approaches largely build upon general-domain versatile architectures by combining visual and textual encoders. In doing so, a deep vision encoder extracts features from medical images, while a text encoder processes the clinical query; these features are then fused via attention mechanisms or joint embedding strategies to generate answers [ 2 , 4 ] . Based on the conventional structures, researchers further improve medical VQA by leveraging transfer learning, multi-modal pretraining, and meta-learning, as well as incorporating external domain knowledge through medical ontologies and knowledge graphs [ 27 , 1 , 26 ] . While these studies achieve impressive results on normal medical images, directly applying them to CT volumes introduces significant challenges owing to the fact that a CT scan is composed of multiple adjacent image slices. Therefore, conventional approaches are limited in their model structure to treat the volume as a set of unrelated 2D images, so that fail to capture the continuity and inherent spatial relationships among consecutive slices [ 20 ] . This lack of integration across slices weakens the model’s ability to perform robust spatial and quantitative reasoning, which causes the generated answers to be clinically superficial or inaccurate [ 30 ] . Moreover, existing studies pay less attention to identifying the important slices that are relevant to the key content in the question, which makes it hard for the model to locate essential information that is relevant to the question. In this paper, we propose an LLM-based approach for CT VQA with cross-modal feature graphing. Specifically, we utilize a vision encoder and a text encoder to capture the visual and textual representations of the CT slices and the text tokens, then build a graph based on the representations using the information across modalities with each slice or token serving as a node. To facilitate graph encoding, we not only model the connections between every two nodes but also weight them accordingly to assess their contributions in predicting the correct answer, especially the images associated with the key nodes in the query. This design allows the model to effectively identify CT slices that are relevant to the question, reducing noise from irrelevant slices. Finally, the encoded representation from the graph of CT slices and key information in the question is passed to an LLM for answer generation. Extensive experiments on a benchmark CT VQA dataset, namely, M3D-VQA, demonstrate the effectiveness of our approach, which outperforms strong baselines and existing studies. II The Approach Our approach for CT VQA enhances multimodal LLM by cross-modal feature graphing, which is modeled by graph convolutional networks (GCN), with the overall architecture presented in Figure 2 . Specifically, given the input CT volume \mathcal{V}=\mathcal{S}_{1}\cdots\mathcal{S}_{N} that is the combination of N CT slices (the n -th CT slice is denoted as \mathcal{S}_{n} ) and the question \mathcal{Q}=q_{1}\cdots q_{M} with M tokens (the m -th token is denoted as q_{m} ), our approach firstly perform a multimodal content encoding process ( f_{e} ) to compute their embeddings. Next, the graph is constructed, where each node represents either CT slices or question tokens, and the representation of the graph structure is derived from the embedding of CT slices and question tokens. Subsequently, we apply an attentive graph convolutional network (A-GCN, denoted as f_{AGCN} ) to encode the graph, weighing different nodes to identify the most relevant information for the QA task. Finally, the output of the graph structure decoding is used as a soft prompt to instruct the LLM ( f_{LLM} ) in generating the answer \widehat{\mathcal{A}} . Therefore, the overall process of our approach is formulated as \widehat{\mathcal{A}}=f_{LLM}(f_{AGCN}(f_{e}(\mathcal{V},\mathcal{Q})),\mathcal{Q}) (1) In the following text, we firstly present the multimodal content encoding process, then the cross-modal feature graph modeling, and finally the LLM decoding process for answer generation. II-A Multimodal Content Encoding The multimodal content encoding process aims to process the CT slices \mathcal{S}_{1}\cdots\mathcal{S}_{N} and the question \mathcal{Q} into their representations so as to facilitate the following processing. In doing so, we utilize a vision encoder f_{v} (e.g., a pre-trained vision Transformer [ 7 ] ) and a text encoder f_{t} (e.g., BERT [ 10 ] ) to model the CT slices and question, respectively. For the vision encoding process, for each CT slice \mathcal{S}_{n} , we firstly split it into non-overlapping patches p_{n,1}\cdots p_{n,U} and use a

Read the complete text