XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography
Haozhe Luo, Shelley Zixin Shu, Ziyu Zhou, Sebastian Otálora, Mauricio Reyes · 2025
Read XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography by Haozhe Luo, Shelley Zixin Shu, Ziyu Zhou, Sebastian Otálora, Mauricio Reyes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19599.txt.
Introduction
Haozhe Luo and four co-authors introduce XBench, a benchmark that tests whether vision-language models actually look at the right place in a chest X-ray. Such models answer questions about medical images, but their grounding, the match between words and image regions, is rarely measured. Reliable grounding matters before any clinical use.
The benchmark evaluates seven CLIP-style models, generating explanation maps from cross-attention and similarity methods and comparing them with radiologist annotations across diseases. All models localized large, clear findings reasonably well but degraded on small or diffuse lesions. Models pretrained on chest X-rays aligned better than general-domain ones, and recognition skill and grounding skill were strongly linked.
Catalog record
| CREATOR | Haozhe Luo, Shelley Zixin Shu, Ziyu Zhou, Sebastian Otálora, Mauricio Reyes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography Haozhe Luo, Shelley Zixin Shu, Ziyu Zhou, Sebastian Otálora, Mauricio Reyes Abstract † † address: 1 ARTORG Center for Biomedical Engineering Research, University of Bern, Switzerland 2 Shanghai Jiao Tong University, China 3 Kaiko.AI, Switzerland 4 Dept. of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Switzerland 1 Abstract Vision–language models (VLMs) have recently shown remarkable zero-shot performance in medical image understanding, yet their grounding ability, the extent to which textual concepts align with visual evidence, remains underexplored. In the medical domain, however, reliable grounding is essential for interpretability and clinical adoption. In this work, we present the first systematic benchmark for evaluating cross-modal interpretability in chest X-rays across seven CLIP-style VLM variants. We generate visual explanations using cross-attention and similarity-based localization maps, and quantitatively assess their alignment with radiologist-annotated regions across multiple pathologies. Our analysis reveals that: (1) while all VLM variants demonstrate reasonable localization for large and well-defined pathologies, their performance substantially degrades for small or diffuse lesions; (2) models that are pretrained on chest X-ray–specific datasets exhibit improved alignment compared to those trained on general-domain data. (3) The overall recognition ability and grounding ability of the model are strongly correlated. These findings underscore that current VLMs, despite their strong recognition ability, still fall short in clinically reliable grounding, highlighting the need for targeted interpretability benchmarks before deployment in medical practice. XBench code is available at https://github.com/Roypic/Benchmarkingattention. Figure 1: Overview of the unified evaluation framework for medical vision-language models. The framework integrates three main components: Dataset , Model , and Metrics . The Dataset Wrapper organizes multi-source datasets (36 diseases, 12,601 cases) including images, annotations, prompt templates, and augmentations from RSNA Pneumonia [ 1 ] , Covid19-rural [ 2 ] , ChestDet-10 [ 3 ] , SIIM [ 4 ] , CheXlocalize [ 5 ] , ChestXray14 Detection [ 6 ] , and Vindr-CXR [ 7 ] . The Model Wrapper standardizes inference logic across vision-language models, encapsulating the image encoder, text encoder, and fusion module; it supports representative models such as CLIP, BiomedCLIP, MedKILP, KAD, MAVL, CARZero, and DeViDe. The Metrics module unifies both Grounding Metrics (e.g., Pointing Game, Dice, IoU with fixed or searched thresholds) and Classification Metrics (e.g., AUC, Accuracy, F1), enabling comprehensive and comparable evaluation across datasets and models. 2 Introduction Deep learning has achieved remarkable progress in medical image analysis, enabling automated interpretation of chest radiographs at expert-level accuracy in certain diagnostic tasks. Recent advances in vision-language models (VLMs) [ 8 , 9 , 10 ] further extend this capability by jointly learning from paired image-text data, demonstrating strong zero-shot recognition and transferability across medical domains. However, in clinical settings, the value of such models extends beyond classification accuracy. Whether models’ predictions are grounded in meaningful visual evidence are equally important [ 11 , 12 , 5 ] , as reliable grounding, or cross-modal interpretability, is essential for clinical trust, model validation, and regulatory acceptance. While numerous VLMs [ 13 , 14 , 15 ] have shown promising performance on image-level tasks, their spatial reasoning and localization abilities remain poorly understood. Prior work has revealed that post-hoc saliency methods, though widely adopted for medical explainability, often fail to localize fine-grained or small-scale pathologies compared with radiologist annotations [ 5 ] . In particular, benchmarks like CheXlocalize [ 5 ] have highlighted large gaps between model-generated heatmaps and expert-drawn regions, underscoring the need for standardized and quantitative evaluation of grounding performance. To address this gap, we introduce XBench , the first comprehensive benchmark for evaluating cross-modal interpretability in chest X-rays. XBench integrates the Dataset , Model , and Metrics modules into a unified evaluation framework (Fig. 1 ), supporting seven representative CLIP-style VLMs spanning pretraining from natural images to chest X-ray–specific data. Grounding performance is assessed with Pointing Game, Dice, and IoU, while AUC, Accuracy, and F1 are jointly reported. Across 36 findings and 12,601 cases, XBench reveals systematic explainability patterns: domain-specific pretraining improves grounding for large, well-defined pathologies, but models still underperform on small lesions, obfuscated/overlapping regions, and diffuse or scale-variant findings. Together, this benchmark establishes a rigorous foundation for evaluating and improving the interpretability of medical vision–language models, paving the way toward clinically reliable multimodal AI. Method COVID-19 Pneumonia Pneumothorax CheXDet-10 CheXlocalize VinDR-CXR ChestX-ray14 Point Dice IoU Point Dice IoU Point Dice IoU Point Dice IoU Point Dice IoU Point Dice IoU Point Dice IoU CLIP [ 13 ] 15.62 18.31 10.92 7.39 20.20 11.97 0.46 3.16 1.64 7.10 12.99 8.10 3.22 7.96 4.33 2.47 7.31 4.25 7.42 12.36 7.11 BiomedCLIP [ 14 ] 3.12 16.56 9.69 12.64 20.20 11.97 1.01 3.17 1.64 5.41 12.68 7.92 3.24 8.16 4.46 3.95 7.55 4.39 9.05 12.67 7.37 MedKLIP [ 15 ] 28.12 19.25 11.24 42.78 33.07 21.25 1.55 3.79 2.00 31.79 23.32 15.38 18.12 17.91 10.98 23.74 19.04 12.02 40.06 27.74 18.21 KAD [ 8 ] 6.25 27.45 18.18 70.11 42.06 28.05 2.47 4.18 2.20 32.18 23.26 15.47 24.21 18.73 11.61 18.35 18.55 11.80 43.61 29.65 19.41 MAVL [ 16 ] 15.62 16.45 9.61 29.31 20.14 11.94 2.78 4.09 2.25 26.12 19.19 12.58 17.49 13.31 7.88 15.89 16.53 10.32 31.31 20.83 13.12 CARZero [ 17 ] 53.12 36.64 24.26 83.66 50.47 36.45 5.56 4.94 2.79 48.38 31.35 22.40 33.35 23.20 15.54 39.07 31.01 22.28 61.57 39.44 28.01 DeViDe [ 9 ] 3.12 28.36 18.56 70.77 40.16 26.48 3.40 4.36 2.33 35.26 26.56 18.12 27.02 18.22 11.22 21.47 20.43 13.03 49.16 30.57 20.04 Table 1: Grounding metrics across datasets. Each dataset group shows the mean Pointing Game and the best-threshold Dice/IoU. Single-disease datasets focus on one pathology; multi-disease show averages over classes. All values are percentages. Best and second-best in each column are in bold and underline , respectively. Table 2: Per-class Pointing Game performance on VinDR-CXR. Each model spans two rows to display all classes. The best result per class is shown in bold , and the second best is underlined . For most findings, the grounding performance of all models remains below 50%. Model Mean Classes Aortic enl. Atelectasis Calcif. Cardiomegaly Clav. fract. Consol. Emphysema Enl. PA ILD Infiltration Lung Opac. Lung cavity Lung cyst Mediast. shift Nodule/Mass Other lesion Pleural eff. Pleural thick. Pneumothorax Pulm. fibrosis Rib fract. CLIP 2.47 5.73 0.00 0.60 9.26 0.00 6.90 0.00 0.00 15.10 1.89 0.00 0.00 0.00 0.00 1.34 6.17 4.26 0.66 0.00 0.00 0.00 BioMedCLIP 3.95 7.73 1.20 0.00 7.90 0.00 0.00 0.00 0.00 2.80 3.51 0.00 0.00 50.00 0.00 1.25 1.12 4.85 0.62 0.00 1.91 0.00 MedKLIP 23.74 30.21 24.68 7.14 58.52 0.00 50.57 33.33 42.86 40.62 32.08 32.00 25.00 0.00 25.00 24.16 18.52 14.89 0.00 20.00 18.88 0.00 KAD 18.35 12.08 21.69 14.44 86.60 0.00 62.37 0.00 0.00 1.40 19.30 27.50 0.00 0.00 0.00 34.38 3.37 38.83 2.47 22.22 2.39 36.36 MAVL 15.89 30.73 12.99 3.57 12.59 0.00 34.48 33.33 14.29 33.85 18.87 26.67 12.50 0.00 6.25 9.40 18.52 5.32 0.00 33.33 17.86 9.09 CARZero 39.07 77.60 41.56 26.19 76.30 100.00 77.01 33.33 28.57 2.08 52.83 38.67 25.00 0.00 37.50 37.58 51.06 15.23 33.33 33.33 20.92 27.27 DeViDe 21.47 10.63 31.33 34.44 88.32 0.00 73.12 0.00 14.29 2.80 17.54 31.25 0.00 0.00 37.50 37.50 7.87 41.75 1.23 16.67 14.83 27.27 Table