Measuring What VLMs Don’t Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation
Aditya Parikh, Aasa Feragen, Sneha Das, Stella Frank · 2026
Read Measuring What VLMs Don’t Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation by Aditya Parikh, Aasa Feragen, Sneha Das, Stella Frank (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01625.txt.
Introduction
Aditya Parikh, Aasa Feragen, Sneha Das, and Stella Frank warn that good test scores can hide weak radiology reports. Their 2026 work studies AI models that turn X-ray images into written reports. Some decoding choices yield safe, generic text that scores well on word overlap yet drops key clinical terms.
They call that failure template collapse, since reports shrink to a few stock phrases. To expose it, they track how word links tied to patient groups shift between real and made reports. A weighted score sums that loss of clinical signal across groups. Fixed decoding erases the most signal, while random sampling adds variety but can add new bias. The authors urge diversity checks as a core part of judging report models.
Catalog record
| CREATOR | Aditya Parikh, Aasa Feragen, Sneha Das, Stella Frank |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22980 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01625
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring What VLMs Don’t Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation Aditya Parikh, Aasa Feragen, Sneha Das, Stella Frank Abstract Reliable deployment of Vision-Language Models (VLMs) in radiology requires validation metrics that go beyond surface-level text similarity to ensure clinical fidelity and demographic fairness. This paper investigates a critical blind spot in current model evaluation: the use of decoding strategies that lead to high aggregate token-overlap scores despite succumbing to template collapse , in which models generate only repetitive, safe generic text and omit clinical terminology. Unaddressed, this blind spot can lead to metric gaming , where models that perform well on benchmarks prove clinically uninformative. Instead, we advocate for lexical diversity measures to check model generations for clinical specificity. We introduce Clinical Association Displacement (CAD), a vocabulary-level framework that quantifies shifts in demographic-based word associations in generated reports. Weighted Association Erasure (WAE) aggregates these shifts to measure the clinical signal loss across demographic groups. We show that deterministic decoding produces high levels of semantic erasure, while stochastic sampling generates diverse outputs but risks introducing new bias, motivating a fundamental rethink of how “optimal” reporting is defined. Code available at [ 2 , 1 ] 1 Introduction and Related Work Radiology report generation (RRG) aims to automate the conversion of medical images into clinically actionable text, reduce documentation burden and support diagnostic decision-making [ 7 , 23 ] . While recent VLMs have made RRG more viable in clinical workflows [ 18 ] , in this paper we demonstrate that there is a major gap between scoring well on standard text similarity metrics and providing better clinical reporting or more equitable outcomes across patient populations. Metrics That Enable Gaming. Contemporary RRG evaluation relies on word-overlap metrics like BLEU [ 20 ] , ROUGE [ 15 ] , BERTScore [ 28 ] , measuring aggregate agreement between generated and reference (ground-truth) reports. Models evaluated on such metrics are vulnerable to template collapse : generating high-frequency “normal finding” templates that maximise token overlap despite omitting patient-specific details [ 26 ] . As normal radiographs dominate datasets, such generic outputs can achieve deceptively high scores with low diagnostic value [ 9 ] . Standard metrics cannot detect terminology redistribution , in which group-specific clinical terms are suppressed, over-generated, or reassigned across demographic groups. The risk increases with VLMs’ documented tendency to under-utilize visual evidence in favor of language priors [ 12 ] , which can lead to more generic outputs reproducing demographic biases. Current evaluation protocols focus on aggregate accuracy, leaving such biases undetected. Decoding as a Confounding Variable. The inference strategy (deterministic/stochastic decoding) directly shapes the diversity and specificity of generated reports [ 22 ] . Deterministic decoding optimises for mode selection, converging to a small set of high-probability templates; stochastic decoding introduces controlled randomness to explore lower-probability tokens. Prior work [ 14 ] has motivated stochastic decoding based on marginal gains in BLEU and ROUGE scores. However, selecting decoding strategies solely on average-metric improvements obscures critical fairness trade-offs: while stochasticity may reduce template collapse, it can simultaneously introduce or amplify demographic associations that were absent in reference reports [ 22 ] . Existing evaluation frameworks provide no mechanism to audit these lexical shifts. The Need for Complementary Metrics. Recent work has called for evaluation methods that go beyond surface similarity to measure clinical utility, demographic equity, and semantic fidelity [ 8 ] . Clinician-aligned scoring (e.g., RaTEScore [ 30 ] ) offers one avenue by incorporating expert judgment, but it does not directly quantify how models redistribute clinical terminology across demographic groups. Fairness metrics in NLP often rely on group-wise F1 gaps or counterfactual perturbation tests. What is missing is a vocabulary-level diagnostic that explicitly tracks whether the demographic association structure of clinical language is preserved, erased, or distorted during generation. Our Contributions. We identify decoding strategy as a hidden confounder in clinical evaluation for RRG, and thus treat inference configuration as a primary experimental factor: (i) We propose Clinical Association Displacement , a vocabulary-level framework quantifying shifts in group-associated word usage between human-authored and model-generated reports, identifying erased terminology as well as emergent and preserved bias; (ii) We introduce Weighted Association Erasure , a summary statistic that measures the global magnitude of association displacement, diagnosing whether a model systematically avoids clinically informative language; (iii) Using our radiology VLM ReX-QWEN as a testbed, we conduct an inference sensitivity analysis, showing that deterministic decoding inflates scores via template collapse (high erasure) while stochastic decoding recovers vocabulary but may increase bias; (iv) our baseline comparisons reveal fairness failures invisible to conventional metrics. Our work establishes that the choice of decoding strategy and metric jointly confounds clinical evaluation. By auditing both what models say and systematically don’t say , we enable transparent evaluation of whether RRG progress reflects genuine clinical utility or evaluation artifacts. Figure 1 : Clinical interpretation of CAD. Each point is a vocabulary term, mapped from its demographic association in the reference corpus ( z_{ref} , x-axis) to its association in predictions ( z_{pred} , y-axis). Points near the diagonal (gray band) indicate stable associations, while off-diagonal points highlight terms whose demographic association has shifted, reflecting erasure, new bias, or bias flips. 2 Methods We introduce a corpus-level diagnostic of global lexical shift, quantifying how vocabulary is preserved, erased, or newly introduced during generation. Such shifts may involve clinically meaningful terms that are differentially associated with demographic groups and therefore relevant for diagnostic interpretation. We therefore also introduce diagnostics for demographic-specific shifts, exposing shifts in lexical usage that are not captured by aggregate metrics. We formalize this through: Clinical Association Displacement (CAD) (Section 2.1 ) and Weighted Association Erasure (WAE) (Section 2.2 ). 2.1 Clinical Association Displacement (CAD) CAD is a vocabulary-level diagnostic framework that quantifies how the demographic association, both in polarity and magnitude, of individual clinical terms shifts between a human-reference corpus and generated outputs (see Fig. 1 ). Problem Formulation and Log Ratio: Let \mathcal{D}\in\{\mathrm{ref},\mathrm{pred}\} denote the reference and prediction corpora, where each report is associated with a binary group, here sex: g\in\{F,M\} . For each corpus \mathcal{D} , we compute, for each word w in the vocabulary \mathcal{V} , token counts c_{g}^{(\mathcal{D})}(w) counting the occurrences of w in reports with sex g . The total token count per group is N_{g}^{(\mathcal{D})}=\sum_{w\in\mathcal{V}}c_{g}^{(\mathcal{D})}(w) . To measure how strongly a word w is associated with female vs. male reports in a corpus \mathcal{D} , we use a Dirichlet-smoothed log ratio of unigram probabilities [ 17 ] . Let \alpha>0 (default: \alpha = 0.1) be a smoothing parameter (where ∼ denotes \alpha -smoothed estimates) and V=|\mathcal{V}| the vocabulary size. We define: \forall g\in\{F,M\}:\quad p_{g}^{(\mathcal{D})}(w)=\textstyle\frac{c_{g}^{(\mathcal{D})}(w)+\alpha}{N_{g}^{(\mathcal{D})}+\alpha V}=\textstyle\frac{\tilde{c}_{g}^{(\mathcal{D})}(w)}{\sum_{w^{\prime}\in\mathcal{V}}\tilde{c}_{g}^{(\mathcal{D})}(w^{\prime})},\quad s^{(\mathcal{D})}(w)=\textstyle\log\frac{p_{F}^{(\mathcal{D})}(w)}{p_{M}^{(\mathcal{D})}(w)}. s^{(\mathcal{D})}(w)>0 indicates a female association for w , while s^{(\mathcal{D})}(w)<0 indicates that w is more associated to male . We apply delta-method approximation under a multinomial model to estimate the variance of s^{(\mathcal{D})}(w) : \tilde{\mathrm{Var}}[s^{(\mathcal{D})}(w)]\approx\textstyle\frac{1}{\tilde{c}_{F}^{(\mathcal{D})}(w)}+\frac{1}{\tilde{c}_{M}^{(\mathcal{D})}(w)},\quad z^{(\mathcal{D})}(w)=\textstyle\frac{s^{(\mathcal{D})}(w)}{\sqrt{\tilde{\mathrm{Var}}[s^{(\mathcal{D})}(w)]}}, yielding a standardized association z^{(\mathcal{D})}(w) adjusting for frequency variance. We denote s^{(\mathrm{ref})}(w) , s^{(\mathrm{pred})}(w) and z^{(\mathrm{ref})}(w) , z^{(\mathrm{pred})}(w) for reference and predictions. Quantifying Displacement: We define Association Displacement z_{\mathrm{disp}}(w) as the standardized difference between prediction and reference associations, and a critical threshold z_{\mathrm{crit}} for significance testing: z_{\mathrm{disp}}(w)=\textstyle\frac{s^{(\mathrm{pred})}(w)-s^{(\mathrm{ref})}(w)}{\sqrt{\tilde{\mathrm{Var}}[s^{(\mathrm{pred})}(w)]+\tilde{\mathrm{Var}}[s^{(\mathrm{ref})}(w)]}},\quad z_{\mathrm{crit}}=\textstyle\Phi^{-1}\!\left(1-\frac{p^{\star}}{2}\right). Treating corpora as independent, z_{\mathrm{disp}}(w)\sim\mathcal{N}(0,1) under the