Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers

Yuntao Shou, Xiangyong Cao, Qian Zhao, Deyu Meng · 2026

Read Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers by Yuntao Shou, Xiangyong Cao, Qian Zhao, Deyu Meng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13386.txt.

Introduction

Yuntao Shou, Xiangyong Cao, Qian Zhao, and Deyu Meng work on controllable pathology image generation. Text-guided diffusion gives only coarse global control and cannot enforce fine structural limits. Paired data are also scarce, since labeling gigapixel slides by hand takes far too long. Their multi-agent framework drafts descriptions and diagnostic steps, then judges quality, with human checks for reliability. On that data they build IC-DiT, a layout-aware diffusion transformer with hierarchical attention. Tests on five histopathology datasets showed higher fidelity and stronger layout control than rivals.

Catalog record

CREATORYuntao Shou, Xiangyong Cao, Qian Zhao, Deyu Meng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39836
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13386

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers Yuntao Shou, Xiangyong Cao, Qian Zhao, Deyu Meng Abstract Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail. However, existing text-guided diffusion models offer only coarse global control and lack the ability to enforce fine-grained structural constraints. Progress is further limited by the absence of large datasets that pair patch-level spatial layouts with detailed diagnostic descriptions, since generating such annotations for gigapixel whole-slide images is prohibitively time-consuming for human experts. To overcome these challenges, we first develop a scalable multi-agent LVLM annotation framework that integrates image description, diagnostic step extraction, and automatic quality judgment into a coordinated pipeline, and we evaluate the reliability of the system through a human verification process. This framework enables efficient construction of fine-grained and clinically aligned supervision at scale. Building on the curated data, we propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer. Through hierarchical multimodal attention, IC-DiT maintains global semantic coherence while accurately preserving structural and morphological details. Extensive experiments on five histopathology datasets show that IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods. In addition, the generated images serve as effective data augmentation resources for downstream tasks such as cancer classification and survival analysis. 1 Introduction The synthesis of realistic and clinically reliable pathology images remains a challenging problem in computational pathology. Although recent text-guided diffusion models have demonstrated remarkable capability in generating photorealistic images with semantic control [ 64 , 55 , 16 , 27 ] , they still struggle to provide fine-grained spatial and morphological controllability. As shown in Fig. 1 , most existing diffusion pipelines rely on coarse global conditions such as prompts or latent embeddings, which offer only implicit control over spatial arrangements. However, histopathological images critically depend on precise cellular organization, tissue topology, and morphological continuity, making layout-controllable generation essential for producing diagnostically meaningful outputs [ 60 , 21 ] . Without explicit structural guidance, standard diffusion models often fail to maintain consistent glandular boundaries, cell distributions, or regional tissue compositions, limiting their applicability in clinical and educational scenarios [ 59 ] . Figure 1 : Comparison between traditional diffusion-based pathology synthesis and the proposed IC-DiT. (a) Conventional latent diffusion models rely on global cross-attention, offering limited control over spatial structure and semantics. (b) IC-DiT integrates multi-modal in-context conditioning with image, layout, visual, and text tokens, enabling anatomically coherent and spatially controllable pathology generation. To address the above limitations, we aim to propose a context-aware generative framework that integrates explicit spatial layout conditioning with textual and visual semantics. However, achieving such controllable synthesis presents two fundamental challenges. First, realizing layout-aware generation requires a training dataset that contains fine-grained spatial layouts paired with detailed diagnostic descriptions, yet no publicly available pathology dataset provides such patch-level semantic layouts together with high-quality captions. Second, existing diffusion-based pathology generators lack mechanisms for precise structural supervision, making it difficult for the model to learn how spatial constraints should be mapped to valid tissue morphology. Therefore, we devote ourselves to addressing these two challenges in this work. For constructing the desired dataset, unlike natural images that can be annotated with coarse captions by crowdworkers, histopathology data involve gigapixel whole-slide images with highly complex tissue morphology and subtle cellular differences [ 31 , 50 , 48 ] . Producing consistent patch-level annotations requires medical expertise, and the annotation burden grows exponentially with image scale. For example, generating high-quality descriptions for the entire BRCA cohort would require more than 40,000 hours of expert pathologists’ time [ 17 ] . Such heavy cost makes large-scale fine-grained annotation essentially infeasible and leaves current models without the conditional signals needed for controllable synthesis [ 8 , 18 , 1 ] . To overcome this bottleneck, we introduce an automated multi-agent large vision–language model (LVLM) annotation framework, enabling scalable generation of reliable pseudo-supervision for fine-grained pathology data. The framework orchestrates several specialized agents to mimic diagnostic reasoning. Specifically, Image-to-Text agent extracts morphological characteristics, a Key-Step Extraction agent decomposes them into interpretable diagnostic steps, and a Judge agent evaluates their consistency through automatic metrics. A human verification process then calibrates the scoring criteria, progressively improving domain alignment [ 62 , 30 , 7 , 49 ] . This multi-agent pipeline allows us to automatically construct fine-grained textual annotations at scale, significantly reducing expert workload while maintaining diagnostic relevance. Based on the curated dataset, we introduce In-Context Diffusion Transformer (IC-DiT), a unified generation framework that supports explicit layout-to-pathology synthesis. IC-DiT injects multimodal contextual information, including spatial layout masks, textual descriptions, and visual embeddings, into a diffusion transformer backbone. Within IC-DiT, a set of multimodal attention blocks performs hierarchical feature alignment in which textual semantics guide global appearance, layout tokens ensure structural fidelity, and visual embeddings enhance local morphological detail. This design enables controllable synthesis with fine-grained spatial accuracy and diagnostically faithful textures, addressing key limitations of previous diffusion-based approaches. The main contributions of this paper are summarized as follows. • To the best of our knowledge, we make the first attempt to incorporate large vision–language models (LVLMs) into a multi-agent annotation framework for histopathology, enabling scalable generation of fine-grained and clinically interpretable pseudo-labels through automated reasoning and human verification. • We propose a unified In-Context Diffusion Transformer (IC-DiT) that integrates textual, spatial, and visual embedding modalities via dedicated encoders and hybrid cross-modal attention blocks, achieving fine-grained controllability over both semantic appearance and structural layout during pathology image synthesis. • We conduct comprehensive experiments on multiple histopathology datasets, demonstrating that our method can obtain superior generation performance over state-of-the-art baselines in image fidelity, spatial consistency, and controllability. Additionally, our method can help augment the data for downstream high-level tasks, such as survival analysis and cancer classification. 2 Related Work 2.1 Diffusion Models for Digital Pathology Diffusion models have recently emerged as powerful generative tools for high-fidelity image synthesis and data augmentation in medical imaging, including digital pathology [ 13 , 15 , 20 , 23 , 38 ] . By modeling the data distribution through a learned denoising process, these models can generate realistic histopathological images, aiding in data scarcity and domain generalization issues. Yellapragada et al. [ 59 ] demonstrated the effectiveness of conditional diffusion models in synthesizing diverse and class-specific histology patches for cancer classification tasks. Similarly, Xu et al. [ 57 ] applied latent diffusion to perform high-resolution Whole Slide Image (WSI) synthesis, improving downstream segmentation and detection performance. 2.2 Diffusion Transformer Diffusion Transformer (DiT) [ 42 , 14 , 32 , 47 , 33 ] represents a recent advancement in generative modeling, combining the strengths of diffusion models and vision transformers. Unlike traditional convolution-based denoising networks, DiT replaces the U-Net backbone with a pure transformer architecture, offering greater modeling capacity and scalability for high-resolution image generation. DiT operates directly in the latent space using transformer blocks, and demonstrated state-of-the-art performance on ImageNet. Its self-attention mechanism enables efficient global context modeling, which is particularly beneficial for visual domains where spatial dependencies are critical, such as digital pathology. Compared to latent diffusion models that rely on autoencoders and operate in compressed latent spaces, DiT preserves

Read the complete text