CLIP-IT : CLIP -based Pairing of Histology I mages with Privileged T extual Information

Banafsheh Karimian, Giulia Avanzato Soufiane Belharbi, Mohammadhadi Shateri, Luke McCaffrey, and, Eric Granger · 2025

Read CLIP-IT : CLIP -based Pairing of Histology I mages with Privileged T extual Information by Banafsheh Karimian, Giulia Avanzato Soufiane Belharbi, Mohammadhadi Shateri, Luke McCaffrey, and, Eric Granger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16181.txt.

Introduction

Banafsheh Karimian and colleagues present CLIP-IT, a method that pairs histology images with helpful text. Their 2025 arXiv paper works in multimodal medical learning. Vision-language models read diagnostic cues well but need large paired datasets. Labels are costly, private, and compute heavy to gather. Free unpaired reports, though, may still carry useful diagnostic facts.

CLIP-IT retrieves the most relevant report for each image. A CLIP model trained on separate histology pairs does the matching. Reports come from the same disease and tissue type as the image. These pseudo-pairs share clinical meaning rather than exact links. Knowledge from the text is then distilled into the image model. The approach aims for strong diagnosis support without costly pairing.

Catalog record

CREATORBanafsheh Karimian, Giulia Avanzato Soufiane Belharbi, Mohammadhadi Shateri, Luke McCaffrey, and, Eric Granger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16181

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLIP-IT : CLIP -based Pairing of Histology I mages with Privileged T extual Information Banafsheh Karimian, Giulia Avanzato Soufiane Belharbi, Mohammadhadi Shateri, Luke McCaffrey, and, Eric Granger Abstract Multimodal learning has shown significant promise for improving medical image analysis by integrating information from complementary data sources. This is widely employed for training vision-language models (VLMs) for cancer detection based on histology images and text reports. However, one of the main limitations in training these VLMs is the requirement for large paired datasets, raising concerns over privacy, and data collection, annotation, and maintenance costs. To address this challenge, we introduce CLIP-IT method to train a vision backbone model to classify histology images by pairing them with privileged textual information from an external source. At first, the modality pairing step relies on a CLIP-based model to match histology images with semantically relevant textual report data from external sources, creating an augmented multimodal dataset without the need for manually paired samples. Then, we propose a multimodal training procedure that distills the knowledge from the paired text modality to the unimodal image classifier for enhanced performance without the need for the textual data during inference. A parameter-efficient fine-tuning method is used to efficiently address the misalignment between the main (image) and paired (text) modalities. During inference, the improved unimodal histology classifier is used, with only minimal additional computational complexity. Our experiments 1 1 1 github.com/BanafshehKarimian/ModalityPairing on challenging PCAM, CRC, and BACH histology image datasets show that CLIP-IT can provide a cost-effective approach to leverage privileged textual information and outperform unimodal classifiers for histology. Keywords: Cancer Detection, Histology, Deep Learning, Multimodal Learning, Vision-Language Models, Knowledge Distillation Figure 1: An overview of CLIP-IT , compared to unimodal and multimodal (vision-text) approaches. Unlike traditional methods, CLIP-IT applies multimodal learning to a unimodal dataset by pairing it with an external text source. 1 Introduction Cancer diagnosis is among the most critical and challenging tasks Kumaraswamy, 2022 . To ensure a comprehensive understanding of cancer, medical professionals rely on the results of multiple modalities, such as histology images, genomic profiles, and pathology reports. Although Machine and Deep Learning (ML/ DL) models have shown great potential in advancing cancer analysis, they remain largely unimodal, focusing on a single type of data, in contrast to how medical professionals analyze data. This reliance on unimodal models limits their ability to capture the complex nature of cancer as a multifaceted disease, potentially reducing model accuracy, robustness, and generalizability Abdullakutty et al., 2024 . Recent advancements in multimodal learning have shown significant improvements in addressing these challenges by integrating information from diverse data types. Multimodal methods use the complementary strengths of each modality to improve predictive accuracy. For instance, histology images have cell-level morphological details that are crucial for identifying tissue abnormalities, while textual data, such as pathology reports, have high-level clinical interpretations and contextual knowledge that are difficult to derive directly from images. By fusing the complementary information of these modalities, multimodal learning not only improves classification performance but also enhances model interpretability and robustness, particularly in handling data variability, noise, and incomplete information Baltrušaitis et al., 2019 . As shown in recent work, multimodal approaches have shown great potential for cancer detection and diagnosis, significantly pushing the boundaries of what ML can achieve in this field Abdullakutty et al., 2024 , Long et al., 2024 , Qu et al., 2024 . However, despite their potential, multimodal learning methods face a significant challenge in gathering paired datasets, where modalities are aligned. Collecting datasets such as histology images paired with pathology reports is resource-intensive and often requires extensive manual annotation. This limitation makes the scalability of multimodal learning systems challenging in practical settings. To address this critical gap, a novel multimodal approach, CLIP-IT , is introduced to utilize multimodal learning for improving the accuracy of unimodal image classification. It is done by pairing the main modality, the histology images, with semantically relevant textual modalities sourced from external datasets using a CLIP-based model. This process generates an augmented multimodal dataset, enabling multimodal learning without the need for resource-intensive paired dataset collection. We then propose to use knowledge distillation to both benefit from the extra information provided by the paired modality data and also remove the need for the extra modality at inference time. This makes it possible to use the paired modality as privileged information without adding any noticeable complexity during inference time. Fig. 1 shows our method compared to a) uni and b) multimodal methods. In unimodal learning, both the dataset and the model are unimodal. In multimodal learning, conventional approaches use paired multimodal datasets. In contrast, our approach performs multimodal learning with a unimodal dataset and an external dataset as privileged modality. This work highlights the broader implications of enabling multimodal learning with unimodal datasets, particularly in resource-constrained settings. By pairing a privileged modality, our approach not only allows multimodal learning but also sets a foundation for integrating heterogeneous datasets in medical applications. Contributions of this paper are as follows: • CLIP-IT uses CLIP to extend unimodal histology datasets by pairing them with histology reports from external textual modalities, creating an augmented multimodal dataset that includes complementary information of another modality. CLIP-IT is a universal method, applicable to any unimodal model and any dataset. • It then employs knowledge distillation to transfer knowledge from the text modality to the vision model, allowing our method to leverage the benefits of multimodal learning using the paired text as privileged information, not needed at test time. • Our extensive results show the efficacy of our approach, where multimodal learning with the augmented dataset improves classification accuracy compared to state-of-the-art methods. For PCAM, BACH and CRC, CLIP-IT improves the accuracy of unimodal backbones on average by 2.87%, 1.37%, and 0.65%, respectively. 2 Related Work (a) Integration of Histology Images with Other Modalities: Integrating multiple modalities has become a promising area in advancing classification, and diagnostic accuracy using histology images. One commonly used modality alongside histology images is genomic data Long et al., 2024 , Zhang et al., 2024d , Qu et al., 2024 , Xiong et al., 2024 , Cai et al., 2024 , Mejia et al., 2024 , Yang et al., 2024 , Zhang et al., 2024c . Genomic data often involves high-dimensional representations, which are challenging to integrate with other modalities. This complexity may result in increased computational costs and difficulty in extracting meaningful patterns. Another common modality is text, in the form of reports or text created from the clinical data Kim et al., 2024 . Recent methods have focused on aligning vision and text data for survival analysis with missing modalities in the data Qu et al., 2024 . PathM3 Zhou et al., 2024 , is proposed to use limited WSI and caption pairs to improve classification performance. The paper shows that the image integrated with paired text outperforms the image-only models. To the best of our knowledge, the previous methods that attempt to improve the classification of histology images, using text modality, leverage paired images and text of the same dataset, making multimodal learning expensive. (b) Prompt-based VLMs: VLMs have become a powerful tool for improving the diagnosis performance in medical imaging. Prompt-based methods are simple yet effective and are used for

Read the complete text