PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction

Manahil Raza, Ayesha Azam, Talha Qaiser, Nasir Rajpoot · 2025

Read PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction by Manahil Raza, Ayesha Azam, Talha Qaiser, Nasir Rajpoot (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20022.txt.

Introduction

Predicting cancer survival from tissue slides improves when other data join, and Manahil Raza with three co-authors add the pathologist's own report. Whole slide images hold billions of pixels; reports hold short expert summaries. Fusing such unequal partners risks the text being drowned out. Their PS3 model uses prototypes to build balanced representations of each modality.

Biological pathways join images and reports in the fusion. A transformer then integrates the three views for survival prediction. Reports carry interpreted findings and clinical context that pixels alone lack. The hypothesis is simple: the expert's summary should help the algorithm, as it helps tumor boards. Balanced fusion is the technical crux. The approach suits real clinical workflows, where reports already exist for every case.

Catalog record

CREATORManahil Raza, Ayesha Azam, Talha Qaiser, Nasir Rajpoot
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction Thanks: Accepted at ICCV 2025 Manahil Raza, Ayesha Azam, Talha Qaiser, Nasir Rajpoot Abstract Current multimodal fusion approaches in computational oncology primarily focus on integrating multi-gigapixel histology whole slide images (WSIs) with genomic or transcriptomic data, demonstrating improved survival prediction. We hypothesize that incorporating pathology reports can further enhance prognostic performance. Pathology reports, as essential components of clinical workflows, offer readily available complementary information by summarizing histopathological findings and integrating expert interpretations and clinical context. However, fusing these modalities poses challenges due to their heterogeneous nature. WSIs are high-dimensional, each containing several billion pixels, whereas pathology reports consist of concise text summaries of varying lengths, leading to potential modality imbalance. To address this, we propose a prototype-based approach to generate balanced representations, which are then integrated using a Transformer-based fusion model for survival prediction that we term PS3 (Predicting Survival from Three Modalities). Specifically, we present: (1) Diagnostic prototypes from pathology reports, leveraging self-attention to extract diagnostically relevant sections and standardize text representation; (2) Histological prototypes to compactly represent key morphological patterns in WSIs; and (3) Biological pathway prototypes to encode transcriptomic expressions, accurately capturing cellular functions. PS3, the three-modal transformer model, processes the resulting prototype-based multimodal tokens and models intra-modal and cross-modal interactions across pathology reports, WSIs and transcriptomic data. The proposed model outperforms state-of-the-art methods when evaluated against clinical, unimodal and multimodal baselines on six datasets from The Cancer Genome Atlas (TCGA). The code is available at: https://github.com/manahilr/PS3 . 1 Introduction Patient prognosis in oncology refers to the predicted progression and outcome of cancer, often determined by clinical, pathological and molecular factors [ 85 , 55 ] . It is essential to guide treatment strategies, inform risk assessment and facilitate patient-centered decision-making to optimize survival and quality of life [ 17 , 47 ] . Recently, integrating multiple data modalities have been shown to improve survival prediction in cancer research, with most studies focusing on combining genomic data such as transcriptomic data with histology whole slide images (WSIs) [ 66 , 68 , 10 ] . These approaches leverage the strengths of each modality—genomics captures molecular subtypes, while WSIs provide spatial and morphological insights into tumor characteristics [ 87 ] . However, pathology reports remain an under-utilized resource despite providing essential information to clinicians for estimating the prognosis of cancer patients. Generated by expert pathologists, they are mandatory for definitive cancer diagnosis [ 53 ] and contain critical clinical and prognostic information including biomarker status, tumor grading, staging and histological subtypes [ 33 , 51 , 30 ] . They often complement histology and genomic data by incorporating expert-driven diagnostic insights [ 67 , 1 ] . Moreover, as routinely produced components of clinical workflows, pathological reports serve as a readily available source of medical knowledge [ 29 ] . Despite their clinical significance, they remain largely untapped in computational survival prediction models. Early multimodal approaches often relied on late fusion techniques [ 8 , 38 , 78 ] , which integrated unimodal representations only at the final decision stage [ 16 ] . However, these methods struggled to model cross-modal interactions, which may be crucial for accurate prognosis. In contrast, more recent studies have investigated early fusion approaches for integrating WSIs and genomic data. Although early fusion captures fine-grained cross-modal interactions more effectively [ 65 , 26 , 7 , 81 ] , it remains computationally demanding due to the high dimensionality and heterogeneity of the data. Another key challenge in early fusion of WSIs, genomics and pathology reports is modality imbalance, arising from differences in structure and scale of the data modalities. WSIs are gigapixel-scale images typically divided into thousands of high-dimensional patches containing spatial and morphological information [ 58 ] . To process these vast amounts of image data, Multiple Instance Learning (MIL) [ 15 , 77 ] is commonly employed, allowing models to extract and aggregate relevant features from individual patches. In contrast, pathology reports provide concise textual summaries with significantly fewer raw data points. Further, genomic data, such as RNA-Seq expression levels are typically represented as isolated scalar values, lacking contextual information about their biological functions. This imbalance can lead to modality dominance, where data-rich modalities ( e.g . WSIs) can disproportionately influence the model compared to modalities with fewer tokens ( e.g . pathology reports). Addressing this imbalance is crucial for developing an effective multimodal fusion framework that ensures fair contributions from all modalities and improves predictive performance. To address these challenges, we propose a prototype-based multimodal fusion framework that standardizes and balances representations across three modalities. The prototypes represent large, variable-length inputs as compact embeddings, enabling more effective multimodal fusion. We exploit the inherent morphological redundancy in WSIs by identifying recurring histological patterns within tissue patches [ 76 ] . Inspired by previous works [ 64 , 65 , 31 ] , we use a Gaussian Mixture Model (GMM) to represent slides, with each mixture component corresponding to a distinct histological prototype . This clustering of similar morphological patterns enables compact WSI representations using key histological prototypes. Pathology reports, which contain significantly less raw data than WSIs, often exhibit unstructured or semi-structured formats, inconsistent formatting, varying lengths and levels of detail [ 51 , 73 , 60 ] . Rather than truncating their content, we construct diagnostic prototypes —standardized representations designed to capture the most diagnostically relevant sections of each report using self-attention-based mechanisms [ 74 ] . To ensure a biologically meaningful representation of transcriptomic data, we draw inspiration from prior methods [ 26 , 87 , 65 ] and transform the values into a set of 50 Cancer Hallmark pathway prototypes [ 41 ] . This representation aligns gene expression patterns with well-established cellular processes, providing a structured and interpretable genomic feature set. We propose a multimodal transformer-based framework PS3 , designed to P redict S urvival by integrating information from three ( 3 ) different data modalities: WSIs, transcriptomics and pathology reports. By transforming raw data into compact prototype-based representations, we reduce the disparity in token counts across modalities, achieving a more balanced token distribution. This allows our transformer-based method to effectively model self-attention and cross-attention mechanisms, thereby facilitating the fusion of complementary information from all three modalities. As a result, our approach enhances predictive performance, leading to robust survival prediction. We evaluate the proposed method on six cancer cohorts from The Cancer Genome Atlas (TCGA) [ 79 ] and demonstrate that it outperforms both unimodal and multimodal baselines. To summarize, our main contributions are: (1) a novel framework for representing pathology reports, WSIs and transcriptomic profiles using diagnostic, histological and pathway prototypes, respectively; (2) development of a multimodal transformer, PS3, to predict survival using three modalities; and (3) comprehensive experiments and ablation studies on six cancer cohorts, showcasing the effectiveness of PS3 for cancer survival prediction. 2 Related Work 2.1 Survival Prediction with Whole Slide Images Given the large number of patches in WSIs, many methods adopt MIL-based methods for efficient processing and analysis [ 2 ] . MIL typically involves three key steps: (1) dividing the WSI into hundreds or thousands of smaller patches (or instances), (2) extracting patch

Read the complete text