Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation
Suryakant Singh, Saarthak Kapse, Joel Saltz, Prateek Prasanna · 2026
Read Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation by Suryakant Singh, Saarthak Kapse, Joel Saltz, Prateek Prasanna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01144.txt.
Introduction
Suryakant Singh and three co-authors present SCOUT, a concept-grounded multimodal transformer for generating pathology reports from whole-slide images. Report generation must integrate local tissue patterns, global slide context, and diagnostic concepts, yet existing approaches rely on fixed pretrained visual features and can miss key concepts. SCOUT maintains an evolving visual representation alongside recursively updated slide-level and concept-conditioned views.
Visual and contextual information co-evolve across encoder depth, and separate attention pathways feed an adaptive fusion module for each token. On more than twenty thousand slide-report pairs across several cancer types, SCOUT beats leading methods on standard text scores and on a clinical report quality score. Ablations confirm both iterative refinement and the fusion step contribute.
Catalog record
| CREATOR | Suryakant Singh, Saarthak Kapse, Joel Saltz, Prateek Prasanna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01144
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation Suryakant Singh, Saarthak Kapse, Joel Saltz, Prateek Prasanna Abstract Whole-slide images (WSIs) present a fundamental challenge for computational pathology due to their extreme resolution, multi-scale heterogeneity, and the requirement for clinically reliable interpretation. Although recent pathology foundation models have enabled fluent report generation, they often lack clinical grounding, failing to accurately represent key diagnostic concepts and relationships observed by pathologists. This limitation arises from the difficulty of integrating heterogeneous visual evidence spanning fine-grained cellular patterns, slide-level tissue architecture, and high-level diagnostic concepts, while maintaining interpretability and clinical coherence. Here we present SCOUT: Semantic Context-aware mOdality fUsion Transformer, a context-aware concept-grounded multimodal framework for pathology report generation that enables progressive conditioning of image representations by global slide information and explicit diagnostic concepts. The method integrates local histological patterns, whole-slide context, and expert-curated semantic descriptors within a unified learning paradigm, allowing visual features to be dynamically refined throughout the encoding process. By combining depth-aware contextual modulation with adaptive multimodal fusion during text generation, the framework produces clinically coherent reports while preserving complementarity across representational scales. We demonstrate that this approach improves generalization across datasets and enhances interpretability by explicitly linking visual evidence to diagnostic concepts and explanatory text. We evaluate SCOUT against three state-of-the-art baselines—WSI-Caption, HistGen, and BiGen—using identical CONCH1.5 visual features on TCGA-BRCA, MICCAI REG, and HistAI. Across these three benchmarks, SCOUT achieves the best BLEU-1 to BLEU-4 and METEOR scores on all datasets, and the best ROUGE-L on TCGA-BRCA and MICCAI REG while remaining competitive on HistAI. In particular, SCOUT reaches 0.436/0.303/0.202/0.156 BLEU-1/2/3/4, 0.204 METEOR, and 0.312 ROUGE-L on TCGA-BRCA, and 0.865/0.834/0.805/0.780 BLEU-1/2/3/4, 0.568 METEOR, and 0.874 ROUGE-L on MICCAI REG. Our results suggest that progressive contextual conditioning is a powerful and general strategy for multimodal medical report generation, providing a flexible foundation for integrating domain expertise into vision–language models for computational pathology. 1 Introduction Figure 1: End-to-end framework for our concept-grounded pathology report generation. The proposed framework integrates multi-scale histopathology information and curated clinical concepts to generate coherent and interpretable pathology reports. WSIs and pathology concepts constitute the primary inputs (left). Patch-level visual features are extracted using a frozen CONCH [ 20 ] encoder, slide-level representations are obtained via a TITAN [ 7 ] encoder, and concept embeddings are derived using GECKO [ 15 ] ( Feature Extraction ). These modality-specific features are first aligned through lightweight adaptive projection networks before being processed by a Context-aware Encoder . The encoder performs iterative depth-aware encoding, where self-attention is modulated via feature-wise linear modulation (FiLM [ 27 ] ) to condition patch representations on global slide context and expert-derived concepts, enabling progressive refinement across layers. Layer-wise outputs are aggregated to form a unified multimodal representation. A Context-adaptive Decoder then applies learned gated fusion and cross-attention mechanisms to dynamically integrate patch-, slide-, and concept-level features during text generation, producing structured and clinically meaningful pathology reports (right). Automated generation of structured reports from medical images has emerged as a promising application of multimodal machine learning, with the potential to reduce documentation burden and improve consistency in clinical workflows [ 10 , 4 , 31 , 22 , 11 , 13 , 9 ] . In computational pathology, this task is particularly challenging [ 8 , 12 , 21 ] : pathology reports synthesize fine-grained cellular observations, global tissue architecture, and standardized diagnostic concepts into coherent narratives that are both descriptive and explanatory. Report generation in this setting therefore requires not only visual understanding but also the integration of multi-scale evidence with structured medical knowledge. Recent advances in multimodal learning and pathology vision-language models have enabled increasingly fluent report generation from whole-slide images (WSIs). However, most existing approaches rely on fixed patch-level visual representations extracted prior to language modeling [ 4 , 8 , 12 , 21 ] or incorporate additional modalities derived from the same underlying patch features, without contextual adaptation. Such designs implicitly assume that these representations are universally optimal for all downstream linguistic and reasoning tasks. This assumption is particularly limiting in pathology, where diagnostically relevant features vary substantially across disease types, magnification levels, and reporting conventions. A central limitation of current pathology report generation methods is the lack of contextual adaptability during visual feature extraction. Local patch-level features often capture fine cellular detail but lack global context, while slide-level representations summarize tissue architecture at the cost of spatial specificity. Moreover, emerging approaches like HistGen [ 10 ] and BiGen [ 31 ] incorporate additional information, but as auxiliary signals, rather than as drivers of how visual features themselves are formed and refined. This separation between feature extraction and contextual reasoning limits both generalization and interpretability. Models may generate fluent reports while failing to accurately ground statements in relevant visual evidence, and often provide limited insight into how diagnostic concepts influence predictions. In contrast, human pathology practice involves iterative reasoning, where visual impressions are continuously refined based on global context and domain knowledge. Capturing this process computationally requires models in which visual representations are dynamically modulated by contextual and semantic signals throughout the learning pipeline. For instance, WSI-Caption [ 4 ] and related multiple instance learning approaches model interactions across large sets of patch tokens using Transformer-based aggregation, but rely on fixed pre-extracted visual features, which can dilute diagnostically relevant signals due to the overwhelming number of redundant regions. HistGen [ 10 ] attempts to address this through a local–global hierarchical encoder and a cross-modal context module that aligns visual and textual representations across granularities. However, its visual representations are still computed prior to language generation and refined largely through hierarchical aggregation and external memory, rather than being dynamically modulated by explicit semantic signals during encoding. As a result, the model improves multi-scale representation but lacks a mechanism for concept-driven feature refinement. Similarly, Bi-Gen [ 31 ] introduces a knowledge retrieval mechanism that augments visual features with semantically relevant textual information from a knowledge bank, coupled with a bi-modal concurrent learning strategy using shared cross-attention layers. While this improves semantic richness, the interaction between visual and retrieved knowledge is mediated through global tokens and late-stage fusion in the decoder, limiting fine-grained alignment between local visual evidence and diagnostic concepts throughout the feature hierarchy. In contrast, human pathology practice involves iterative reasoning, where initial visual impressions are continuously refined based on global context and domain knowledge. Capturing this process computationally requires models in which visual representations are dynamically modulated by contextual and semantic signals throughout the learning pipeline. In this work, we introduce a context-aware multimodal framework for pathology report generation that explicitly models the interaction between visual evidence and diagnostic concepts throughout the representation hierarchy. Unlike prior approaches that rely on fixed visual features, our method enables progressive conditioning of image representations via global slide context and structured semantic signals during encoding. Specifically, we introduce a depth-aware modulation mechanism that refines patch-level features at each encoder layer using global contextual embeddings and concept representations, allowing the model to dynamically emphasize diagnostically relevant patterns across spatial scales. In contrast to retrieval-based or token-level alignment strategies, this formulation treats diagnostic concepts as continuous conditioning signals that guide feature learning rather than auxiliary inputs. Further, to support multimodal reasoning