PolyPath : Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation

Faruk Ahmed †, Lin Yang, Tiam Jaroensri, Andrew Sellergren, Yossi Matias, Avinatan Hassidim, Greg S. Corrado, Dale R. Webster, Shravya Shetty, Shruthi Prabhakara, Yun Liu, Daniel Golden †∗, Ellery Wulczyn †∗, David F. Steiner †∗ · 2025

Read PolyPath : Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation by Faruk Ahmed †, Lin Yang, Tiam Jaroensri, Andrew Sellergren, Yossi Matias, Avinatan Hassidim, Greg S. Corrado, Dale R. Webster, Shravya Shetty, Shruthi Prabhakara, Yun Liu, Daniel Golden †∗, Ellery Wulczyn †∗, David F. Steiner †∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10536.txt.

Introduction

Faruk Ahmed and co-authors adapt a large multimodal model to write pathology reports from many slides at once. Their 2025 paper, PolyPath, answers a workflow gap: pathologists sum findings across slides, while earlier AI read only small regions or single slides. A model with a million token context can swallow up to forty thousand image patches from several whole slide images.

Expert pathologists judged the generated bottom line diagnoses clinically accurate, and equal or better than the original report text in about sixty-eight percent of multi-slide cases of up to five slides. Errors and misses fill the remaining third, so the team frames the system as a drafting partner, not a replacement. The work shows case level reading is within reach for current models.

Catalog record

CREATORFaruk Ahmed †, Lin Yang, Tiam Jaroensri, Andrew Sellergren, Yossi Matias, Avinatan Hassidim, Greg S. Corrado, Dale R. Webster, Shravya Shetty, Shruthi Prabhakara, Yun Liu, Daniel Golden †∗, Ellery Wulczyn †∗, David F. Steiner †∗
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10536

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PolyPath : Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation Faruk Ahmed †, Lin Yang, Tiam Jaroensri, Andrew Sellergren, Yossi Matias, Avinatan Hassidim, Greg S. Corrado, Dale R. Webster, Shravya Shetty, Shruthi Prabhakara, Yun Liu, Daniel Golden †∗, Ellery Wulczyn †∗, David F. Steiner †∗ Abstract The interpretation of histopathology cases underlies many important diagnostic and treatment decisions in medicine. Notably, this process typically requires pathologists to integrate and summarize findings across multiple slides per case. Existing vision-language capabilities in computational pathology have so far been largely limited to small regions of interest, larger regions at low magnification, or single whole-slide images (WSIs). This limits interpretation of findings that span multiple high-magnification regions across multiple WSIs. By making use of Gemini 1.5 Flash, a large multimodal model (LMM) with a 1-million token context window, we demonstrate the ability to generate bottom-line diagnoses from up to 40,000 768 × 768 768\times 768 pixel image patches from multiple WSIs at 10X magnification. This is the equivalent of up to 11 hours of video at 1 fps. Expert pathologist evaluations demonstrate that the generated report text is clinically accurate and equivalent to or preferred over the original reporting for 68% (95% CI: [60%, 76%]) of multi-slide examples with up to 5 slides. While performance decreased for examples with 6 or more slides, this study demonstrates the promise of leveraging the long-context capabilities of modern LMMs for the uniquely challenging task of medical report generation where each case can contain thousands of image patches. 1 1 footnotetext: These authors co-supervised the work. 2 2 footnotetext: Contact: {farukahmed, dangolden, ewulczyn, davesteiner} AT google.com. 1 Introduction Recent applications of vision–language modeling in digital histopathology have been predominantly designed to generate text describing individual regions of interest extracted from a single digitized histopathology image, or Whole Slide Image (WSI). An emerging line of research approaches the more practical clinical use case of slide-level text generation ( Chen et al., 2024 , Ahmed et al., 2024 ) . However, in the typical clinical use case, there can be multiple biological tissue parts associated with a case, with each part having multiple slides. Pathologists write up a report summarizing their part-level diagnostic findings by microscopically reviewing each of the available slides per part and integrating information across these slides. This many-to-one relationship of slides to clinical descriptions is a recognized challenge for vision–language modeling in this space ( Ahmed et al., 2024 ) . The common approach taken in recent literature is to restrict modeling and analysis to single-slide cases or to manually identify a single slide within a case or part that is most representative of the clinical findings in reports ( Zhou et al., 2024 , Guo et al., 2024 , Chen et al., 2024 , Shaikovski et al., 2024 , Xu et al., 2024 , Ahmed et al., 2024 ) . This strategy of selecting representative slides was also adopted in constructing one of the most widely used histopathology datasets, TCGA ( Cooper et al., 2018 ) . While the natural solution for multi-slide report generation is to perform vision–language modeling using all available slides as input, this approach remains underexplored in existing literature, likely due to computational challenges involved in processing multiple WSIs. Two recent works are of note. The HistoGPT vision–language model ( Tran et al., 2024 ) is capable of handling multiple slides per patient at 10X magnification (corresponding to approximately 1 micron per pixel). The work focused exclusively on dermatopathology cases, and the model was evaluated within a human-in-the-loop “expert-guidance” mode, where a human expert prompts the model with their preliminary impression. With this preliminary impression, the model was shown to generate reports on par with those written by human experts. In another line of work, Tan et al. (2024) proposed a multimodal model to generate reports using multiple slides with a total of 7422 slides from 1991 patients, covering kidney and colon cases. Slides at lower magnifications of 1.25X and 5X were used (corresponding to resolutions of approximately 8 and 2 microns per pixel respectively), which can be limiting since pathologists use higher magnifications for many diagnostic tasks ( Kim et al., 2020 ) . While the work did not conduct evaluation with human experts, the reports are pre-processed into a structured format for typical kidney and colon diagnoses and evaluated using natural language generation (NLG) metrics. Recent developments in long-context multimodal modeling ( Gemini et al., 2024 ) enables us to explore the computationally challenging task of report generation from multiple WSIs. In this work, we process up to 50 slides per part at a 10X magnification using the dataset in Ahmed et al. (2024) . The data consists of approximately 350K WSIs, spanning a wider range of tissue types and conditions than prior multi-slide works ( Tran et al., 2024 , Tan et al., 2024 ) . We evaluate the part-level report generation performance of our model with both NLG metrics and expert evaluation. Figure 1 : Model overview. PolyPath is an LMM based on Gemini 1.5 Flash that takes in tissue-containing image patches from multiple slides for a part as well as the part-level specimen label, generating part-level report text across a wide range of tissue sites, procedures, and pathologies. The flow above reflects usage at inference time; during training, the model is trained to predict both the label and finding sections. 2 Methods 2.1 Dataset We build off of the recent data curation effort in Ahmed et al. (2024) , where the primary dataset consists of de-identified retrospective data of approximately 350K WSIs with associated pathology reports from a tertiary teaching hospital. The study was reviewed by Advarra IRB (Columbia, Maryland) and deemed exempt from further review as all data are retrospective and de-identified. The dataset is representative of the distribution of surgical and biopsy procedures, and findings read by pathologists in the U.S. in a large hospital system. The vast majority of slides are hematoxylin and eosin (H&E) stained. This dataset was referred to as DS1 in Ahmed et al. (2024) . As is typical for pathology reports, each case has an associated report text, corresponding to the final diagnosis (i.e. “bottom-line” text). These report texts are structured into part-level sections. In histopathology, a part refers to a specific tissue specimen submitted for pathology review. There may be multiple parts submitted for a given case, and for each part it is typical for multiple slides to be prepared for pathologist review. For each part-level section in the pathology report, there is a label (description of anatomical site and surgical procedure) and a finding (description of diagnostic findings), as illustrated in Figure 1 . The label can often include information that is non-inferable from the WSIs alone (e.g., related to specimen location or procedure). For each part, we associate its corresponding report text with the set of WSIs that are available for that part. Since the number of slides per part is a key variable in this study, we define the following dataset subsets consisting of parts with varying number of slides per part: 1. P1 : Parts with a single slide, 2. P2-5 : Parts with 2 to 5 slides, 3. P6-9 : Parts

Read the complete text