CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging
Pooja Singh * , Siddhant Ujjain * , Tapan Kumar Gandhi, and Sandeep Kumar · 2025
Read CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging by Pooja Singh * , Siddhant Ujjain * , Tapan Kumar Gandhi, and Sandeep Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11034.txt.
Introduction
Pooja Singh, Siddhant Ujjain, and their colleagues introduce CrossMed, a test set for whether medical image models can mix known parts in new ways. A model may know a scan type, a body part, and a task, yet still fail when those three appear in a fresh mix. CrossMed checks that skill in a strict, structured way.
The set rebuilds four public collections, covering X-ray, MRI, and CT, into over twenty thousand choice questions. Two open models were tested on mixes close to their training and on mixes with no overlap at all. Scores fell sharply on the fresh mixes, which shows how hard this kind of broad skill still is.
Catalog record
| CREATOR | Pooja Singh * , Siddhant Ujjain * , Tapan Kumar Gandhi, and Sandeep Kumar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11034
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging Pooja Singh * , Siddhant Ujjain * , Tapan Kumar Gandhi, and Sandeep Kumar Abstract Recent advances in multimodal large language models (MLLMs) has enabled unified processing of visual and textual inputs, with promising implications for general-purpose medical AI. However, their ability to generalize compositionally across unseen combinations of imaging modality, anatomy, and task type remains underexplored. We introduce CrossMed , a benchmark designed to evaluate compositional generalization (CG) in medical MLLMs using a structured Modality–Anatomy–Task (MAT) schema. CrossMed reformulates four public datasets, CheXpert (X-ray classification), SIIM-ACR (X-ray segmentation), BraTS 2020 (MRI classification and segmentation), and MosMedData (CT classification) into a unified visual question answering (VQA) format, resulting in 20,200 multi-choice QA instances. We evaluate two open-source MLLMs, LLaVA-Vicuna-7B and Qwen2-VL-7B , on both Related and Unrelated MAT splits, as well as a zero-overlap setting where test triplets share no Modality, Anatomy, or Task with training data. Models trained on Related splits achieve 83.2% classification accuracy and 0.75 segmentation cIoU, while performance drops significantly under Unrelated and zero-overlap conditions, validating the benchmark’s difficulty. Furthermore, we show cross-task transfer where segmentation performance improves by +7% cIoU even when trained using classification-only data. Traditional models (ResNet-50, U-Net) benefit modestly, confirming MAT’s broad utility, while MLLMs uniquely excel at CG. CrossMed provides a rigorous testbed for evaluating zero-shot, cross-task, and modality-agnostic generalization in medical vision-language models. * * footnotetext: These authors contributed equally to this work I Introduction Fig. 1: Classification accuracy across compositional generalization conditions: Related , Unrelated , with holding individual MAT factors (w/o Modality, w/o Anatomy, w/o Task), and All Data multi-task upper bound. Medical imaging plays an essential role in modern healthcare, facilitating diagnosis, monitoring, and treatment of various diseases. Recent advances in deep learning have substantially improved performance in medical tasks, including classification [ 7 ] , detection [ 19 ] , and segmentation [ 11 ] . However, these approaches typically depend heavily on large-scale annotated datasets and task-specific architectures, limiting their ability to scale and generalize across diverse clinical tasks and imaging modalities. Traditional single-task models, although effective within narrow scopes, struggle with knowledge transfer, especially in scenarios involving rare or emerging medical conditions [ 23 ] . While multi-task learning approaches have demonstrated promise for enhancing flexibility and generalization, the fundamental mechanisms enabling this cross-task performance remain inadequately understood [ 18 ] . Recently, multimodal large language models (MLLMs), which combine visual and textual reasoning capabilities, have emerged as powerful tools showing promising results across a range of biomedical applications [ 12 , 5 ] . Nonetheless, a clear understanding of how these models generalize across heterogeneous medical datasets remains elusive. We hypothesize that compositional generalization (CG), the ability of models to recombine learned elements to address novel tasks and scenarios, plays a crucial role [ 8 ] . For example, in Fig. 2 , a model trained on “white rabbit” and “black piglet” should be able to infer “black rabbit” even if it has never seen this combination before. In medical imaging, each sample can be broken down into a Modality–Anatomy–Task (MAT) triplet, such as ( X-ray , Chest , Classification ) or ( MRI , Brain , Segmentation ). CG suggests that a model trained on certain combinations should be able to generalize to unobserved ones, like ( MRI , Chest , Classification ). Fig. 2: Illustration of compositional generalization under Related vs. Unrelated training splits. Top: A model trained on White Rabbit and Black Piglet images generalizes to an unseen Black Rabbit by recombining color and object attributes. Bottom: Training on MRI Spine and CT Lung enables correct interpretation of a novel CT Spine image, demonstrating generalization through shared modality–anatomy factors. To systematically explore this hypothesis, we introduce CrossMed, a comprehensive benchmark explicitly designed to evaluate CG in medical vision-language models through a standardized visual question answering (VQA) paradigm. CrossMed contains 20,200 MAT-labeled examples spanning multiple imaging modalities (X-ray, MRI, CT), rigorously organized into Related and Unrelated splits to precisely isolate compositional effects. Further, we implement a strict zero-overlap evaluation to robustly assess true zero-shot generalization. Our experiments, conducted with two state-of-the-art MLLMs (LLaVA-Vicuna-7B and Qwen2-VL-7B), reveal significant performance improvements under compositional training conditions, confirming the effectiveness and scalability of the CrossMed benchmark. Key Contributions: • A unified, multi-task VQA benchmark encompassing a diverse set of medical imaging modalities, anatomies, and tasks. • A rigorous zero-overlap evaluation protocol designed to provide a reliable measure of compositional generalization. • Empirical evidence demonstrating robust generalization capabilities of MLLMs to novel MAT combinations. • CrossMed supports both task-level and modality-level transfer, and enables consistent evaluation across architectures via a unified format. II Related Work Multimodal large language models (MLLMs) have catalyzed major advances in visual-language reasoning, enabling capabilities such as visual question answering (VQA), captioning, and instruction-following [ 9 , 1 , 22 ] . These models typically combine powerful pre-trained language backbones with vision encoders, resulting in remarkable zero-shot and few-shot generalization in natural image domains. However, their extension to the medical domain remains nascent [ 10 ] , with most work limited to narrow tasks like report generation or classification in specific imaging modalities. Traditional medical AI systems, largely based on convolutional or transformer architectures, are effective but siloed often built for task-specific deployments such as disease classification or lesion segmentation [ 16 , 15 ] . While some success has been achieved through transfer learning and self-supervision, these models often fail to generalize across tasks or modalities due to the lack of shared structural representations [ 6 , 4 ] . Recent adaptations of MLLMs to medicine have explored integrating language modeling with visual features for structured outputs. HuatuoGPT-Vision [ 5 ] , PMC-VQA [ 21 ] , and MedFlamingo [ 13 ] represent promising directions by aligning biomedical images with instruction-tuned language models. However, these approaches are primarily evaluated on narrow datasets and do not systematically examine generalization under novel task configurations. The concept of compositional generalization (CG), a model’s ability to recombine familiar components to solve unfamiliar tasks has gained traction in both NLP and computer vision. In the medical domain, Med-MAT [ 3 ] marks a notable contribution by curating a suite of 106 public datasets and constructing Modality–Anatomy–Task (MAT) triplets for classification and detection tasks. It evaluates zero-shot recombination performance and provides a strong baseline for CG in medical MLLMs. However, Med-MAT does not integrate tasks into a single inference format, nor does it address segmentation or provide theoretical bounds on CG. In contrast, our CrossMed benchmark unifies medical imaging tasks spanning classification and segmentation into a single VQA-based interface. Unlike Med-MAT, CrossMed provides a principled factorization of CG using MAT triplets and introduces rigorous Related, Unrelated, and zero-overlap evaluation splits. We extend the scope of generalization to include continuous mask decoding and task transfer (e.g., classification-only training with segmentation evaluation). CrossMed is also empirically validated on multiple MLLMs and compared with traditional baselines (e.g., ResNet, U-Net), underscoring its clinical and methodological robustness. Our work bridges a critical gap by offering both a unified evaluation protocol and empirical analysis of CG effects across diverse architectures and imaging modalities, moving toward the goal of generalist medical AI systems.