MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi, Shahad Albastaki, Asim Khan, Muhammad Bilal, Moshira Ali Abdalla, Mohammed Bennamoun, Sajid Javed · 2026

Read MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding by Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi, Shahad Albastaki, Asim Khan, Muhammad Bilal, Moshira Ali Abdalla, Mohammed Bennamoun, Sajid Javed (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23067.txt.

Introduction

Alawode and co-authors present MLLM-HWSI, a language model for whole slide image understanding in pathology. Whole slide images hold diagnostic clues at many scales, from cells to the full slide. Most models compress a slide into one embedding, losing fine grounding. Their model aligns vision and language at four scales, likened to word, phrase, sentence, and paragraph. Two losses keep meaning coherent across scales. A light attention module packs cell data into compact tokens. The model sets new top scores on 13 benchmarks across six pathology tasks.

Catalog record

CREATORBasit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi, Shahad Albastaki, Asim Khan, Muhammad Bilal, Moshira Ali Abdalla, Mohammed Bennamoun, Sajid Javed
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94955
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi, Shahad Albastaki, Asim Khan, Muhammad Bilal, Moshira Ali Abdalla, Mohammed Bennamoun, Sajid Javed Abstract Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce MLLM-HWSI , a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight Cell–Cell Attention Fusion (CCAF) transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: GitHub . Figure 1 : Our proposed MLLM-HWSI model aligns WSIs across multiple scales e.g., cells, patches, regions, and WSI enabling fine-grained, context-aware, and interpretable pathology reasoning. 1 Introduction Cancer diagnosis and prognosis using gigapixel Whole Slide Images (WSIs) remain the clinical gold standard for histopathological assessment 80 ; 15 ; 63 ; 62 ; 87 . The rise of Computational Pathology (CPath) has opened new possibilities to accelerate diagnostic workflows, improve reproducibility, and enable earlier cancer detection through quantitative analysis of the histology landscape 24 ; 72 ; 31 . WSIs are inherently hierarchical, both biologically and structurally, capturing the full spatial organization of tissue across multiple magnifications and scales (Fig. 1 ) 23 ; 18 ; 87 ; 36 . This hierarchical organization reflects the architecture of tissue itself, where diagnostic cues emerge across nested levels, from cellular morphology to regional, and global structural patterns 10 ; 18 ; 87 ; 36 . At the cellular level , WSIs capture diverse morphological attributes including variations in nuclear size, cytoplasmic texture, and mitotic activity that collectively define the vocabulary of pathology 64 ; 59 ; 4 . At the regional level , these cells form micro-architectural structures such as glands, ducts, or solid nests, which define the syntax of tissue organization and carry diagnostic meaning 12 ; 60 . At the global WSI level , multiple regions integrate into a coherent tissue architecture, illustrating spatial relationships between tumor and normal areas, invasion of adjacent structures, and necrosis 78 ; 12 ; 18 ; 7 . This multiscale organization forms the biological foundation of histopathologic interpretation, underpinning how both human experts and computational models reason about cancer 7 ; 45 . Expert pathologists perceive a WSI not as a static but as a multiscale landscape 30 ; 12 ; 7 ; 39 . Diagnostic reasoning typically begins at low magnification, progresses to the examination of regional tissue morphology, and concludes in the inspection of cellular features 65 ; 67 . Pathologists interpret WSIs as structured narratives in which tissue architecture provides context, regions define syntax, and cells define vocabulary 48 ; 9 ; 26 . This process is bidirectional: global context informs local inspection, while local findings refine global understanding until a coherent finding is reached 30 ; 12 . In CPath, Multimodal Large Language Models (MLLMs) including Quilt-LLaVA 70 , SlideChat 20 , WSI-LLaVA 52 , TITAN 27 , PRISM 71 , and HistGen 35 have been proposed for a wide range of tasks, such as Visual Question Answering (VQA), morphological reasoning, and report generation 20 ; 70 . SOTA MLLMs such as SlideChat 20 and WSI-LLaVA 52 , aggregate patch-level embeddings into a single WSI-level representation aligned with corresponding reports 35 ; 27 . Although this aggregation captures a higher-level context, it neglects the hierarchical composition of WSIs, leading to the loss of fine-grained spatial semantics 20 ; 17 ; 53 . Also, existing models overlook the clinical workflow of expert pathologists, who integrate multi-scale visual cues obtained from progressive zooming and contextual reasoning 4 ; 32 . Figure 2 : Comparison of MLLM-HWSI with SOTA methods. In this work, we address these limitations by introducing a Hierarchical WSI-level MLLM (MLLM-HWSI) for comprehensive WSI understanding, including analysis, retrieval, pathological inference, and report generation (Figs. 1 - 2 ). Our approach decodes the inherent pathology language by interpreting individual cells as words , small patches as phrases that describe cellular neighborhoods, larger regions as sentences that depict tissue architecture, and the entire WSI as a paragraph that forms a coherent visual narrative of the disease 73 ; 21 ; 25 . We align the hierarchical structure of WSIs with pathology reports across multiple scales, ensuring that MLLM-HWSI mimics the standard diagnostic workflow of pathologists. By grounding textual description (e.g., pleomorphic nuclei, stromal invasion) in their corresponding visual counterparts, the model captures compositional reasoning underlying expert diagnosis. This multi-scale alignment enhances interpretability, enabling biologically grounded and explainable predictions (Fig. 2 ). MLLM-HWSI bridges the gap between tissue-level interpretation by pathologists and computational model reasoning. Unlike SlideChat 20 , TITAN 27 , and WSI-LLaVA 52 , which rely solely on global embeddings, our model decomposes each WSI into multiple semantic scales-cells, patches, regions, and global WSI—and learns distinct representations for each (Fig. 1 ). At the cellular scale, segmented cells are embedded to represent morphological and cytoplasmic features, and a lightweight Vision Transformer (ViT) with a Cell–Cell cross-Embedding Fusion (CCEF) module aggregates cellular information efficiently. At higher scales, a hierarchical encoder extracts patch, region, and WSI-level embeddings representing local tissue structure and global architecture. A Semantic Patch Filtering module further refines patch-level tokens. These embeddings are projected into a shared multimodal space through scale-specific Vision–Language (VL) projectors and aligned with corresponding textual descriptions. By jointly enforcing hierarchical alignment and cross-scale consistency, MLLM-HWSI preserves diagnostic relationships between local cellular features and global structural patterns. Aligned visual tokens are then fused with textual tokens during LLM pretraining, enabling multi-scale, evidence-based reasoning. MLLM-HWSI is optimized via a hierarchical contrastive alignment loss and a cross-scale consistency loss to maintain semantic coherence across spatial hierarchies. Finally, the fused multi-scale visual and textual tokens pre-train an LLM capable of multi-scale interpretative reasoning, mirroring how pathologists integrate detail and context into coherent diagnoses. We evaluate our proposed MLLM-HWSI model on six different WSI-level CPath tasks including zero-shot classification, retrieval, VQA, report generation, captioning, and cross-modal retrieval using 13 publicly available datasets. Compared to 24 SOTA CPath models, MLLM-HWSI achieves substantial performance improvements as shown in Fig. 2 . Our main contributions are: 1. We introduce a multi-scale hierarchical MLLM that performs cell-, patch-, region-, and WSI-level alignment with pathology reports, enabling unified multi-scale understanding and reasoning over WSIs. 2. We jointly optimize hierarchical contrastive alignment

Read the complete text