When are radiology reports useful for training medical image classifiers?

Herman Bergström, Zhongqi Yue, Fredrik D. Johansson · 2025

Read When are radiology reports useful for training medical image classifiers? by Herman Bergström, Zhongqi Yue, Fredrik D. Johansson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24385.txt.

Introduction

Herman Bergström and two co-authors ask when radiology reports actually help train medical image classifiers. Images for training often come with expert reports, but using reports at prediction time needs a radiologist's timely manual work, so image only models are preferred in practice. Prior studies mostly fine tuned pretrained image models on labels drawn from reports, ignoring tasks only weakly tied to the text.

The authors systematically test using reports during both pretraining and fine tuning, across diagnostic tasks with different links to the text. Their study shows where report based training genuinely improves image only classification and where it does not, guiding better use of paired medical data.

Catalog record

CREATORHerman Bergström, Zhongqi Yue, Fredrik D. Johansson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24385

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When are radiology reports useful for training medical image classifiers? Herman Bergström, Zhongqi Yue, Fredrik D. Johansson Abstract Medical images used to train machine learning models are often accompanied by radiology reports containing rich expert annotations. However, relying on these reports as inputs for clinical prediction requires the timely manual work of a trained radiologist. This raises a natural question: when can radiology reports be leveraged during training to improve image-only classification? Prior works are limited to evaluating pre-trained image representations by fine-tuning them to predict diagnostic labels, often extracted from reports, ignoring tasks with labels that are weakly associated with the text. To address this gap, we conduct a systematic study of how radiology reports can be used during both pre-training and fine-tuning, across diagnostic and prognostic tasks (e.g., 12-month readmission), and under varying training set sizes. Our findings reveal that: (1) Leveraging reports during pre-training is beneficial for downstream classification tasks where the label is well-represented in the text; however, pre-training through explicit image-text alignment can be detrimental in settings where it’s not; (2) Fine-tuning with reports can lead to significant improvements and even have a larger impact than the pre-training method in certain settings. These results provide actionable insights into when and how to leverage privileged text data to train medical image classifiers while highlighting gaps in current research. 1 Introduction Radiology reports containing key findings from medical images are routinely produced in clinical practice. These texts have recently received much attention within the machine learning literature ( Bannur et al. 2024 ; Yang et al. 2023 ; Tanno et al. 2025 ) , and have been shown to be predictive of various patient outcomes, such as hospital readmission ( Huang et al. 2019 ) , ICU mortality ( Lin et al. 2021 ) , and cancer progression ( Batch et al. 2022 ) . Requiring manually written reports for test-time predictions is undesirable if the same prediction could be made from the medical image itself, but the prevalence of reports in retrospective data makes them attractive to use during training. In particular, reports can be used to improve few-shot or small-sample training of medical image classifiers, either by i) incorporating them in a pre-training objective ( Zhang et al. 2022 ) , or ii) viewing them as privileged information (PI) when fine-tuning for a specific task ( Vapnik and Vashist 2009 ) . Figure 1 : Training a medical image classifier with or without reports, divided into: pre-training and fine-tuning. Note that the classifier input is only a medical image at test time. {V} : visual encoder; {T} : text encoder; f : classifier; g : a privileged classifier using both image and report for prediction. Prior work has almost exclusively focused on the potential role of reports in pre-training image encoders—either incorporating them as paired supervision through multimodal objectives ( Huang et al. 2021a ; Zhou et al. 2023 ) , or omitting them in favor of self-supervised learning ( Pérez-García et al. 2024 ; Zhou et al. 2021 ) —with evaluation performed via report-free fine-tuning on diagnostic classification tasks, where labels are often extracted from the reports themselves ( Johnson et al. 2019 ; Irvin et al. 2019 ; Bustos et al. 2020 ) . As a result, the roles of radiology reports in fine-tuning and their potential utility for tasks beyond diagnosis remain largely unexplored. For instance, pre-training with reports instead of relying on image-only self-supervision has been shown to yield representations that improve performance in diagnostic tasks ( Zhang et al. 2022 ; Huang et al. 2021a ) . Yet it is unclear whether these results generalize to prognostic tasks—which aim to predict future clinical events ( e.g. , 12-month readmission or 3-day discharge) that may not be explicitly captured in the original report—and intriguingly, whether incorporating reports during fine-tuning can compensate for the potential limitations of a self-supervised pre-trained representation or a small fine-tuning data set. To address these gaps, we compile a benchmark that systematically investigates the usefulness of radiology reports for training medical image classifiers during both pre-training and fine-tuning. We summarize our paradigm in Figure 1 . In the pre-training stage, we study six popular models that differ in both architecture (ResNet ( He et al. 2016 ) vs. ViT ( Dosovitskiy et al. 2021 ) ), and learning objective, including self-supervised learning ( Pérez-García et al. 2024 ; Zhou et al. 2020 ; Xiao et al. 2023 ) , CLIP-style text-image alignment ( Huang et al. 2021a ; Bannur et al. 2023 ; Zhang et al. 2023 ) , and masked image-text modeling ( Zhou et al. 2023 ) . For fine-tuning, we begin by highlighting that the relationships among the medical image (radiograph) X , report Z and label Y can vary significantly across diagnostic, prognostic, and other tasks, which in turn affects the utility of Z when training a classifier to predict Y from X . Motivated by this, we study a range of classification tasks with varying correlation and causality between Z and Y , and assess how using reports during fine-tuning by general distillation ( Lopez-Paz et al. 2016 ) impacts performance in each setting. To the best of our knowledge, this is the first study of distillation from radiology reports, offering new insights into how privileged textual information can be transferred to image-only classifiers. Our study leads to several interesting and novel findings: (1) Pre-training with report supervision is beneficial for diagnostic tasks at moderate sample sizes when the label is strongly correlated with the text. (2) Explicitly aligning image and text embeddings hurts downstream performance when the label is not captured well by the report, something that is more prominent for non-diagnostic prediction tasks. Crucially, we find that methods relying on text supervision in addition to self-supervision avoid this pitfall. (3) Incorporating reports during fine-tuning, even when they are not available at test time, can yield substantial gains in accuracy. 2 Method Our goal is to study when training using radiology reports helps solve a C -way medical image-only classification task at test time, i.e. , predicting the label y\in\{1,\ldots,C\} from only a medical image \mathbf{x} . We consider each training sample i in a data set \mathcal{D} as a triplet (X=\mathbf{x}_{i},Z=z_{i},Y=y_{i}) , where the image-label pair (\mathbf{x}_{i},y_{i}) is additionally annotated with a radiology report z_{i} derived from \mathbf{x}_{i} to describe clinically significant observations. For brevity, we omit the subscript i when it is unnecessary to highlight the sample index. Training is divided into pre-training and fine-tuning. Next, we describe training procedures for both stages, with or without Z . 2.1 Pre-training with and without reports In pre-training, we aim to learn a representation of images \mathbf{x} through a visual encoder V , which maps each \mathbf{x} into a sequence of visual tokens, e.g. , the patch embeddings in ViT ( Dosovitskiy et al. 2021 ) or convolutional feature maps in ResNet ( He et al. 2016 ) . When medical reports are used in pre-training, analogously, we use a text encoder T , such as a BERT-style transformer, which maps each {z} into a sequence of text tokens. We consider 7 different models pre-trained for medical images from the literature,

Read the complete text