Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning

Liu, Shih-Wen, Fan, Hsuan-Yu, Chu, Wei-Ta, Yang, Fu-En, Wang, Yu-Chiang Frank · 2025

Read Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning by Liu, Shih-Wen, Fan, Hsuan-Yu, Chu, Wei-Ta, Yang, Fu-En, Wang, Yu-Chiang Frank (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17645.txt.

Introduction

Shih-Wen Liu and co-authors teach a vision language model to write pathology reports by showing it similar past cases, in a 2025 paper. Reading tissue slides and drafting reports is expert work that resists automation. Human pathologists themselves lean on remembered cases.

Their framework retrieves whole slide image and report pairs that resemble the new case and feeds them as context. Adaptive feedback sharpens which examples get picked. On a standard benchmark the system set new highs across three overlap scores and held up across report lengths and disease kinds. The design wrings more value from training archives clinics already hold.

Catalog record

CREATORLiu, Shih-Wen, Fan, Hsuan-Yu, Chu, Wei-Ta, Yang, Fu-En, Wang, Yu-Chiang Frank
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17645

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning Liu, Shih-Wen, Fan, Hsuan-Yu, Chu, Wei-Ta, Yang, Fu-En, Wang, Yu-Chiang Frank Abstract Automating medical report generation from histopathology images is a critical challenge requiring effective visual representations and domain-specific knowledge. Inspired by the common practices of human experts, we propose an in-context learning framework called PathGenIC that integrates context derived from the training set with a multimodal in-context learning (ICL) mechanism. Our method dynamically retrieves semantically similar whole slide image (WSI)-report pairs and incorporates adaptive feedback to enhance contextual relevance and generation quality. Evaluated on the HistGen benchmark, the framework achieves state-of-the-art results, with significant improvements across BLEU, METEOR, and ROUGE-L metrics, and demonstrates robustness across diverse report lengths and disease categories. By maximizing training data utility and bridging vision and language with ICL, our work offers a solution for AI-driven histopathology reporting, setting a strong foundation for future advancements in multimodal clinical applications. \midlauthor National Cheng Kung University, Taiwan and National Cheng Kung University, Taiwan and \orcid 0000-0001-5722-7239 National Cheng Kung University, Taiwan and NVIDIA Research and NVIDIA Research † † year: 2025 † † workshop: Full Paper – MIDL 2025 † † volume: – nnn † † editors: Accepted for publication at MIDL 2025 keywords Multimodal In-Context Learning, Medical Report Generation, Histopathology Images, Vision-Language Models, HistGen Benchmark. 1 Introduction Histopathology reports are important diagnostic tools, providing detailed interpretations of histopathological findings that directly influence patient care. Composing a histopathology report demands significant expertise and is very time-consuming. Therefore, automating this process holds immense potential to enhance diagnostic efficiency and reduce workloads. Yet, this task is far from trivial, as it requires understanding the intricate visual features of histopathology images and generating accurate and structured documents. Recent advancements in medical AI have significantly improved the integration of visual and textual data for histopathology report generation. Models like HistGen ( Guo et al. 2024 ) and WsiCaption ( Chen et al. 2024 ) have made strides in bridging the gap between whole slide images (WSIs) and histopathology reports through multimodal approaches. Similarly, models like Quilt-LLaVA ( Seyfioglu et al. 2025 ) and LLaVA-Med ( Li et al. 2023 ) integrate medical imaging and language models for broader applications, including Visual Question Answering (VQA) and histopathology report generation. Despite these advances, these medical models overlook utilizing the similarity between a test image and the images (and their associated histopathology reports) in the training datasets. Taking similar images and their associated reports from previously confirmed cases as the reference is a common practice for expert pathologists to make a report efficiently and ensure report quality. Reviewing similar cases can facilitate maintaining diagnostic accuracy and consistency ( Doe and Smith 2023 ) . They also rely on the Diagnosis Learning Cycle ( Branson et al. 2021 ) , a conceptual framework designed to enhance diagnostic performance through feedback, reflection, and continuous learning, emphasizing the importance of learning from past experiences. Inspired by this, we propose to retrieve similar images and their associated reports from the database and then take them as important clues for generating a report of the test image. We employ a fine-tuned histopathology-specific vision language model (VLM) and investigate prompting techniques to generate accurate histopathology reports. We evaluate our framework on the HistGen benchmark, achieving state-of-the-art performance across BLEU, METEOR, and ROUGE-L metrics. We also test the proposed components separately and study performance obtained based on different settings. In summary, our contributions include: • Introducing an expert-inspired framework with multimodal in-context learning to utilize information from the training dataset to generate better histopathology reports. • Fine-tuning a histopathology-specific VLM to enhance its capabilities in generating histopathology reports. • Achieving state-of-the-art results on the HistGen benchmark and demonstrating the robustness and efficacy across various configurations. 2 Related Works Medical Report Generation. Medical report generation has received significant attention recently. Models like HistGen ( Guo et al. 2024 ) and WsiCaption ( Chen et al. 2024 ) have demonstrated success in generating histopathology reports by bridging WSIs and textual data through advanced encoding and generative approaches. HistGen employs local-global feature encoding to enhance contextual coherence in reports. WsiCaption uses a multiple-instance generative model to produce detailed clinical descriptions. Vision-language models like Quilt-LLaVA ( Seyfioglu et al. 2025 ) and LLaVA-Med ( Li et al. 2023 ) extend these capabilities to broader biomedical applications. LLaVA-Med adapts the LLaVA framework to handle diverse medical tasks, including report generation and VQA, by leveraging curriculum learning to align visual and textual data. Quilt-LLaVA focuses on histopathology-specific applications, aligning representations of WSIs and text using histopathology-specific datasets like Quilt-Instruct. While these models achieve automatic report generation, they often overlook contextual information from training datasets, diverse histopathology categories, or the common practices of expert pathologists. In-Context Learning and Retrieval-Augmented Generation. In-context learning (ICL) has emerged as a promising technique for improving model adaptability without retraining. Pioneered by models like GPT-3 ( Brown et al. 2020 ) , ICL enables models to work on tasks by incorporating task-specific examples directly into the input prompts. Retrieval-augmented generation (RAG) ( Lewis et al. 2020 ) is one of the ICL strategies that embodies this concept by utilizing retrieved data to increase model adaptability. While ICL and RAG have shown success in general-purpose language models, their application to medical AI remains limited. Existing multimodal models like Quilt-LLaVA and LLaVA-Med focus on directly encoding visual and textual data but lack robust mechanisms for retrieving and leveraging similar examples for better generation. Our work attempts to bridge the gaps by incorporating ICL into a report generation framework based on context derived from similar training examples, category-specific guidelines, and structured feedback. 3 Methodology 3.1 Base Model Figure 1 illustrates the proposed report generation framework. We take Quilt-LLaVA as the base model, which has been trained on diverse histopathology datasets for VQA tasks. It was trained based on histopathology image patches and was dedicated to VQA. To more effectively extract features from the entire histopathology image and focus on report generation, we replace its visual encoder with Vision Transformer Large (ViT-L) provided in HistGen ( Guo et al. 2024 ) . The ViT-L has been extensively pre-trained using the DINOv2 technique and is used to extract WSI 1 1 1 All histopathology data are represented in whole slide images in this work. Therefore, we use histopathology image and whole slide image interchangeably. features \mathbf{F}_{patch}\in\mathbb{R}^{n\times d} , where n is the number of patches in a WSI and d is the feature dimension. The extracted patch-based features \mathbf{F}_{patch} and m learnable query tokens \mathbf{Q}\in\mathbb{R}^{m\times d} are processed by transformer blocks. The goal is to consider the context between patches to form holistic WSI features: \hat{\mathbf{H}}=\text{TransformerBlock}(\mathbf{Q},\mathbf{F}_{patch}), (1) where \hat{\mathbf{H}}\in\mathbb{R}^{m\times d} represents the m contextualized tokens derived from \mathbf{Q} . Referring to Figure 1 , the contextualized tokens \hat{\mathbf{H}} are processed by a projector to be the embeddings more appropriate to a VLM, and the results are denoted as \mathbf{H}\in\mathbb{R}^{m\times d} in the following. Notice that the number of patches n in different WSIs may differ. But we always represent a WSI by the m contextualized tokens \mathbf{H} . The tokens \mathbf{H} are then passed to a VLM along with a text prompt \mathbf{P}_{text} to generate a

Read the complete text