Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation

Ling Zhang, Boxiang Yun, Qingli Li, Yan Wang (✉) · 2025

Read Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation by Ling Zhang, Boxiang Yun, Qingli Li, Yan Wang (✉) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18658.txt.

Introduction

Zhang and co-authors teach a model to write pathology reports from slide images in this 2025 paper. Whole slide images are vast and redundant, and their raw features lack medical meaning. Pathologists, by contrast, reason with prior cases in mind.

Their framework, BiGen, copies that habit. It first retrieves relevant knowledge from a medical knowledge bank by matching the slide's most attended patches. Then a two-stream learning setup uses one learnable token for image features and one for the retrieved knowledge, aligning the two. A decoder merges both streams into the final report. On a breast cancer dataset it improved language metrics by 7.4 percent and a key classification score by 19.1 percent over prior methods.

Catalog record

CREATORLing Zhang, Boxiang Yun, Qingli Li, Yan Wang (✉)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18658

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation Ling Zhang, Boxiang Yun, Qingli Li, Yan Wang (✉) Abstract Automated pathology report generation from Whole Slide Images (WSIs) faces two key challenges: (1) lack of semantic content in visual features and (2) inherent information redundancy in WSIs. To address these issues, we propose a novel Historical Report Guided Bi -modal Concurrent Learning Framework for Pathology Report Gen eration (BiGen) emulating pathologists’ diagnostic reasoning, consisting of: (1) A knowledge retrieval mechanism to provide rich semantic content, which retrieves WSI-relevant knowledge from pre-built medical knowledge bank by matching high-attention patches and (2) A bi-modal concurrent learning strategy instantiated via a learnable visual token and a learnable textual token to dynamically extract key visual features and retrieved knowledge, where weight-shared layers enable cross-modal alignment between visual features and knowledge features. Our multi-modal decoder integrates both modals for comprehensive diagnostic reports generation. Experiments on the PathText (BRCA) dataset demonstrate our framework’s superiority, achieving state-of-the-art performance with 7.4% relative improvement in NLP metrics and 19.1% enhancement in classification metrics for Her-2 prediction versus existing methods. Ablation studies validate the necessity of our proposed modules, highlighting our method’s ability to provide WSI-relevant rich semantic content and suppress information redundancy in WSIs. Code is publicly available at https://github.com/DeepMed-Lab-ECNU/BiGen. 1 Introduction The automated analysis of Whole Slide Images (WSIs) in digital pathology has emerged as a significant research direction in the field of medical artificial intelligence recently [ 7 , 17 , 26 ] . Among these, the task of pathology report generation based on WSIs has garnered considerable attention due to its immense potential in assisting diagnosis and reducing the workload of physicians [ 3 , 10 , 21 , 22 ] . In recent years, there have been a significant number of studies on image captioning [ 15 , 25 , 27 , 29 ] . In the medical field, numerous studies have focused on generating radiology reports from radiological images [ 5 , 6 , 9 ] . There are also studies [ 12 , 16 , 23 ] that have focused on obtaining descriptions of small pathological regions of interest. Compared with these report generation tasks, generating reports from WSIs is more challenging and less studied. The challenge relies mainly on how to effectively extract semantically diagnostic features from WSIs with ultra-high resolution [ 13 , 28 ] . Existing approaches such as MI-Gen [ 3 ] directly apply a Transformer model to input image tokens, allowing each token to interact with other tokens. HistGen [ 10 ] employs local-global hierarchical visual encoding to input image tokens, which still relies on a self-attention mechanism to explore relationships among multiple tokens. Though global visual features can be learned via self-attention, these methods only learn visual features from WSI itself, which inevitably (1) lack of semantic content and (2) introduce considerable redundancy . To address these challenges, we propose to leverage historical diagnostic reports to guide the learning of semantically key pathological features for generating current WSI’s report. Concretely, we let the visual features be learned along with the relevant textual features, whose knowledge is retrieved from a pre-built knowledge bank. Our method contains two key components: (1) We propose to retrieve WSI-relevant report candidates from a pre-built knowledge bank to provide semantically rich diagnostic report information. (2) We propose a bi-modal concurrent learning strategy. It is instantiated via a learnable visual token and a learnable textual token which extract key pathological features and key knowledge features, respectively. The strategy also enables the learning of visual token to be guided by the semantically rich textual token. The knowledge retrieval emulates the process that pathologists recall historical diagnosis records from similar cases to aid current decision-making. The visual and textual token represent a more global perspective to mirror the way pathologists view WSIs and recall historical diagnosis records—they don’t necessarily examine each patch or each record but rather focus on key information to make a comprehensive diagnosis. The main contributions of this paper can be summarized as follows: • To the best of our knowledge, we first introduce an explicit knowledge retrieval mechanism into the pathology report generation task to enhance semantically rich diagnostic report information, establishing an interpretable cross-modal knowledge transfer paradigm. • We propose a bi-modal concurrent learning strategy to dynamically extract key features in WSIs and retrieved knowledge, effectively suppressing information redundancy while enhancing pathological semantic learning. • Experiments on the PathText (BRCA) dataset demonstrate that our method significantly outperforms existing benchmarks in report generation quality (BLEU-4=0.135, ROUGE-L=0.293) and Her-2 prediction metrics (F1=0.730). Figure 1: Overview of BiGen. The overall framework in (a), consists of a bi-modal concurrent learning encoder and a multi-modal decoder. "CA" and "FFN" indicates Cross-attention and Feed-Forward Network. "VTCA" and "TTCA" refer to the visual and textual token cross-attention. We construct a knowledge bank in (b) and retrieve WSI-relevant knowledge from the knowledge bank through high-attention patches in (c). 2 Method Our goal is to emulate pathologists’ diagnostic reasoning through the integration of visual evidence and historical knowledge, to provide rich semantic content and suppress redundant information in WSIs. For this, we introduce a novel Historical Report Guided Bi -modal Concurrent Learning Framework for Pathology Report Gen eration (BiGen), which involves a visual branch and a knowledge branch (as illustrated in Fig. 1 (a)). In the visual branch (Sec. 2.2 ) , compact visual features are iteratively aggregated through the visual token cross-attention (VTCA) module to focus on key pathological patterns. In the knowledge branch (Sec. 2.3 ), a knowledge retrieval module is designed to retrieve WSI-relevant knowledge features from a pre-constructed knowledge bank by matching high-attention patches from the visual branch. Furthermore, the retrieved knowledge features are aggregated through the textual token cross-attention (TTCA) module, sharing weights with the visual token cross-attention layers and obtaining rich semantic content. Finally, a multi-modal decoder (Sec. 2.4 ) integrates both visual and textual tokens to generate comprehensive diagnostic reports. 2.1 Problem Formulation The task of pathology report generation is formulated as a sequence-to-sequence problem, where the input consists of multiple image tokens extracted from a high-dimensional pathology image, and the output is a natural language report. Let \{\mathbf{w}_{i}\}_{i=1}^{M} denote the set of non-overlapping patches cropped from the WSI \mathbf{W} , where M is the number of patches. The objective is to generate the target pathology report \mathbf{Y}=\{\mathbf{y}_{n}\}_{n=1}^{N} , where N is the length of the report for \mathbf{W} . A pretrained visual extractor is used to extract feature embeddings from \{\mathbf{w}_{i}\}_{i=1}^{M} , followed by a linear projection layer, resulting in a feature matrix \mathbf{X}=\{\mathbf{x}_{i}\}_{i=1}^{M} , where \mathbf{x}_{i}\in\mathbb{R}^{d} and d is the embedding dimension. The training objective is to maximize the likelihood of generating the correct report given the patch embeddings \mathbf{X} . The total negative log-likelihood (NLL) loss is computed as: \mathcal{L}=-\sum_{n=1}^{N}\log P(\mathbf{y}_{n}|\{\mathbf{y}_{i}\}_{i<n},\mathbf{X}) , where \{\mathbf{y}_{i}\}_{i<n} represents the previously generated tokens in the report sequence. 2.2 Visual Token Cross-attention In the visual branch, we propose a visual token cross-attention module to extract a compact and informative visual representation of the WSI. Instead of performing self-attention over all patch features, we utilize a cross-attention mechanism, where a single learnable token attends to all patch features. Specifically, we initialize a visual token \mathbf{V}_{0}\in\mathbb{R}^{1\times

Read the complete text