Cross-modal linkage risk in clinical vision-language models

Soroosh Tayebi Arasteh ∗, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn · 2026

Read Cross-modal linkage risk in clinical vision-language models by Soroosh Tayebi Arasteh ∗, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02276.txt.

Introduction

Tayebi Arasteh and co-authors audit a privacy risk in clinical vision-language models. Such models train on paired chest X-rays and radiology reports. They learn a shared embedding space from those pairs. Instance level links between image and report can persist there. That is a risk when images and reports are shared apart.

A de-identified image might be re-linked to its report. Cosine similarity alone could perform that re-linking. The task is formalized as image-to-report retrieval. Public paired cohorts give ground truth pairings for the audit. Models of increasing clinical specialization are evaluated. The cohorts hold 406,241 paired examples from 126,804 patients. MIMIC-CXR and CheXpert Plus supply the pairs. Linkage risk is measured rather than assumed.

Catalog record

CREATORSoroosh Tayebi Arasteh ∗, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02276

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-modal linkage risk in clinical vision-language models Soroosh Tayebi Arasteh ∗, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn Abstract Vision-language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared embedding space that can preserve instance-level image-report correspondence. This poses a privacy risk in settings where radiographs and reports are deliberately kept separate after acquisition, such as image-only data sharing or access-controlled reports, because a de-identified image may be re-linked to its original narrative report through cosine similarity alone. We formalized this as image-to-report retrieval and used public paired cohorts, in which the true pairing is known by design, as ground-truth benchmarks to audit the risk rather than as the privacy scenario. Evaluating VLMs of increasing clinical specialization on 406,241 paired examples from 126,804 patients across MIMIC-CXR (43,793 held-out pairs) and external CheXpert Plus (29,296 pairs), we found that re-linkage rose systematically with specialization: the strongest VLM retrieved the correct report at 15 times chance at a candidate pool of N = 100 N=100 , 50 times chance at N = 10,000 N=10{,}000 , and well above chance at full-database scale. The signal persisted under pathology-matched hard negatives that removed disease-label shortcuts, indicating correspondence beyond broad diagnostic categories. To reduce it without retraining, we froze both encoders and applied differentially private optimization only to the projection heads defining the alignment layer ( ε = 0.34 \varepsilon=0.34 , δ = 6 × 10 − 6 \delta=6\times 10^{-6} ). This reduced Recall@1 by 61.8% at N = 10,000 N=10{,}000 on MIMIC-CXR and transferred to CheXpert Plus without retraining, while image-side utility was largely preserved: macro AUROC for linear-probe classification across 14 labels shifted only from 79.63% to 79.43%. Targeted differentially private finetuning of the shared alignment layer can substantially reduce cross-modal re-linkage without materially degrading the image representations that make these models clinically useful. ∗ Correspondence to: Soroosh Tayebi Arasteh () Introduction Clinical artificial intelligence (AI) is increasingly built around vision-language models (VLMs) trained on paired medical images and free-text reports [ 24 , 4 ] . In chest radiography, this paradigm is especially attractive because chest radiographs and radiology reports are naturally paired at scale, allowing multimodal models to learn shared representations that support retrieval, representation learning, report generation, and zero-shot transfer [ 16 , 44 , 12 , 3 ] . Yet the same alignment that makes these models useful may also create a privacy vulnerability that is easy to overlook. If a chest radiograph and its report remain close in the shared embedding space, then a de-identified image may still be reconnectable to the report with which it was originally paired through similarity search alone. In radiology, that possibility is especially consequential because the report contains substantially richer and more specific information than a disease label, often including comparisons, qualifiers, technical context, and narrative findings that extend well beyond what is visible in a simple categorical annotation [ 11 , 15 ] . This raises a privacy question that is distinct from standard de-identification and distinct from the image-only privacy attacks more commonly studied in medical AI [ 38 , 26 ] . Prior work has focused mainly on settings such as membership inference, reconstruction risk, or classification under differential privacy (DP) [ 8 , 36 , 45 , 18 ] . Those lines of work are important, but they do not directly address cross-modal re-association in clinical VLMs. In multimodal radiology models [ 19 ] , the central learned object is not only an image representation or a text representation in isolation, but the alignment between them [ 44 , 12 , 3 ] . That alignment may preserve instance-level correspondence beyond broad pathology labels, making it possible to recover a paired report not merely because it describes edema, pleural effusion, or no finding, but because the model has learned a sharper image-report binding [ 2 ] . This matters in practice because public multimodal checkpoints are increasingly downloaded, reused, and adapted on clinical data, whereas the privacy assumptions attached to de-identified datasets were not designed with shared cross-modal embedding spaces in mind [ 24 , 21 ] . An important clarification concerns what these public datasets represent in this study. In resources such as MIMIC-CXR and CheXpert Plus the image-report pairing is intentionally published, so recovering a report from its radiograph within these datasets is not in itself a privacy breach. We therefore use them not as the privacy scenario but as ground-truth benchmarks: because the true pairing is known, they allow us to audit whether a model’s shared embedding space preserves enough instance-level structure to make reports recoverable from images. The deployment settings this audit speaks to are those in which radiographs and reports are deliberately held distinct after acquisition, including image-only data sharing, separate access control for narrative reports, institutional data enclaves, and vendor or model evaluation pipelines, as well as downstream releases where reports are not meant to be recoverable from image embeddings. In such settings, a VLM’s shared embedding space may weaken that intended separation by preserving pairwise image-report structure, and the benchmarks used here quantify how strongly a given model does so. Here we address that gap by formalizing cross-modal linkage risk as an image-to-report retrieval problem (see Fig. 1 ). We ask whether a held-out chest radiograph can retrieve its true paired report from a candidate pool, how that risk changes as the candidate pool grows, and whether the signal persists when distractor reports are chosen to be clinically similar rather than random [ 12 , 3 , 10 ] . To answer those questions, we evaluate four publicly available VLMs spanning increasing levels of domain specialization, from general-domain CLIP [ 27 ] to the radiology-specific BioViL-T [ 3 ] . We study retrieval on two large public chest radiograph resources: a held-out MIMIC-CXR test set [ 16 ] , which is a large public hospital chest radiograph dataset comprising 43,793 image-report pairs in our evaluation split, and an external CheXpert Plus test set [ 5 ] , which pairs the public CheXpert [ 14 ] chest radiograph benchmark with its corresponding report resource and comprises 29,296 image-report pairs in our external evaluation split. We then test a practical mitigation strategy that targets the final shared alignment layer rather than the full multimodal backbone, using head-only differentially private finetuning while keeping the image and text encoders frozen. A non-private entropy-regularized [ 30 ] alternative is included as a comparator. To determine whether reducing linkage comes at the cost of clinically meaningful image information [ 6 ] , we evaluate post-mitigation image embeddings on 14 chest radiograph abnormality tasks, spanning atelectasis, cardiomegaly, consolidation, edema, enlarged cardiomediastinum, fracture, lung lesion, lung opacity, no finding, pleural effusion, pleural other, pneumonia, pneumothorax, and support devices. Our starting assumption is that linkage risk should increase with clinical specialization, because models trained closer to the chest radiograph reporting distribution are more likely to preserve sharper image-report correspondence [ 44 , 12 , 43 , 3 ] . We further expect that if the signal reflects genuine instance-level binding rather than only broad disease similarity, it should remain detectable at database scale and under pathology-matched hard negatives, even if attenuated. Finally, because the retrieval risk is

Read the complete text