Interactive Segmentation and Report Generation for CT Images

Yannian Gu 1,*, Wenhui Lei 1,*, Hanyu Chen 2, Xiaofan Zhang 1,†, Shaoting Zhang 1 · 2025

Read Interactive Segmentation and Report Generation for CT Images by Yannian Gu 1,*, Wenhui Lei 1,*, Hanyu Chen 2, Xiaofan Zhang 1,†, Shaoting Zhang 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03294.txt.

Introduction

Yannian Gu, Wenhui Lei, Hanyu Chen, Xiaofan Zhang, and Shaoting Zhang built an interactive tool in two thousand twenty-five for reporting on CT scans. Auto report writing can speed up clinic work, but set reports are hard to read and cannot be tuned while a doctor reviews an image. Their tool lets a clinician guide the outline of a lesion as the report forms.

The system pairs a three-dimensional lesion mask with a structured account of its traits. Tests across fifteen lesion types show it works well. The team says this is the first tool to join guided outlining and structured reports for 3D CT images in this way.

Catalog record

CREATORYannian Gu 1,*, Wenhui Lei 1,*, Hanyu Chen 2, Xiaofan Zhang 1,†, Shaoting Zhang 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interactive Segmentation and Report Generation for CT Images Yannian Gu 1,*, Wenhui Lei 1,*, Hanyu Chen 2, Xiaofan Zhang 1,†, Shaoting Zhang 1 Abstract Automated CT report generation plays a crucial role in improving diagnostic accuracy and clinical workflow efficiency. However, existing methods lack interpretability and impede patient-clinician understanding, while their static nature restricts radiologists from dynamically adjusting assessments during image review. Inspired by interactive segmentation techniques, we propose a novel interactive framework for 3D lesion morphology reporting that seamlessly generates segmentation masks with comprehensive attribute descriptions, enabling clinicians to generate detailed lesion profiles for enhanced diagnostic assessment. To our best knowledge, we are the first to integrate the interactive segmentation and structured reports in 3D CT medical images. Experimental results across 15 lesion types demonstrate the effectiveness of our approach in providing a more comprehensive and reliable reporting system for lesion segmentation and capturing. The source code will be made publicly available following paper acceptance. 1 Introduction CT (Computed Tomography) serves as an essential diagnostic tool, with radiological reports serving as the primary medium for communicating diagnostic findings to clinicians [ 13 , 22 ] . In recent years, CT report generation has gained significant attention, with advancements focusing on automating the clinical text generation process based on image analysis results [ 4 , 7 , 8 , 19 ] . Researchers have developed various approaches to generate reports from CT images, include traditional vision-based methods [ 10 ] , vision-language models [ 3 , 26 ] , and other methods [ 15 , 24 ] . However, current CT report generation methods lack interpretability and impede patient-clinician understanding, while offering no interactivity [ 16 ] . This produces reports that fail to explain findings clearly or adapt to new clinical cases, resulting in communication gaps and incomplete assessments. To address these limitations, interactive segmentation techniques have shown promising capabilities in related medical imaging analysis. Models like the Segment Anything Model (SAM) [ 12 , 20 ] enable clinicians to dynamically guide the segmentation process through intuitive interactions, providing immediate visual feedback and greater control over results. These techniques [ 11 , 17 , 18 ] successfully balance automation with expert input, allowing for precise refinements while maintaining workflow efficiency. However, current interactive segmentation methods lack integrated report generation capabilities, forcing radiologists to manually convert visual findings into structured clinical reports. In this work, we introduce the first interactive framework for lesion morphology reporting, integrating both intuitive visual segmentation and quantitative textual attribute description for a comprehensive lesion characterization. Our system enables radiologists to generate detailed clinical reports proposed by Lei et al. [ 14 ] , through minimal point-based interactions during image review, significantly reducing interpretation time while preserving diagnostic precision. By simultaneously generating accurate segmentation results with structured reports, our approach provides improved explanatory capabilities, creating direct visual-textual correspondence between report descriptions and anatomical features in CT images. Additionally, this integrated framework demonstrates considerable potential for facilitating multi-modal medical dataset annotation. Technically, our approach delivers two key innovations: (1) feature-space clustering-based point refinement that amplifies the impact of expert interactions, and (2) inter-task feature synergy that enables bidirectional information flow between segmentation and attribute description generation. The interactive nature of our system significantly enhances zero-shot performance, allowing radiologists to provide real-time feedback that helps the model adapt to novel lesion types not encountered during training. Our contributions are as follows: • We propose the interactive framework for joint lesion segmentation and attribute description that integrates point refinement and feature synergy, enabling comprehensive lesion profiling with minimal radiologist interaction; • Our approach significantly improves clinical workflow efficiency by reducing interpretation time while enhancing diagnostic communication through direct visual-textual correspondence between reports and segmentations; • Extensive experiments demonstrate superior performance in both segmentation accuracy and attribute description quality, including robust zero-shot capabilities for novel lesion types. 2 Methodology Figure 1: The model architecture . Our framework processes 3D CT images and user-provided points to simultaneously generate lesion segmentation masks and structured attribute descriptions. The model integrates visual tokens, point tokens, initial mask tokens, and initial IOU tokens into a hybrid encoding module, which forms the foundation for both visual and textual outputs. Key innovations include a clustering-based point refinement technique that optimizes the input points through clustering centers, and an inter-task feature synergy mechanism that enhances the performance of both segmentation and attribute description tasks concurrently. Our proposed interactive framework enables radiologists to generate detailed lesion profiles through minimal point interactions. As shown in Fig. 1 , the model consists of two main components: the Vision-Prompt Hybrid Token Encode module and the Synergistic Task Integration Decode module. Below, we describe each module in detail. 2.1 Vision-Prompt Hybrid Token Encode Module This module follows a dual-branch architecture, with image tokens extracted by the vision encoder and prompt tokens derived from the prompt encoder . Notably, to enhance the accuracy and reliability of user clicks, the prompt encoder integrates a feature-space clustering-based point refinement method. The refined prompt tokens are then combined with the image tokens in the hybrid token encoder , producing hybrid tokens that drive subsequent processing. Vision encoder \mathrm{E}_{I} adopts a ViT-style [ 5 ] architecture, consisting of a standard ViT with primary local window attention and several interleaved global attention layers. This design produces isotropic feature maps with consistent feature dimensions. The extracted image features \mathbf{I}=\mathrm{E}_{I}(\mathcal{I}) , where \mathcal{I} denotes the input image, are down-sampled for computational efficiency. Prompt encoder \mathrm{E}_{P} transforms point coordinates using positional embeddings, generating distinct representations based on point labels. This process converts sparse point prompts into embeddings that guide the model’s attention toward specific regions of interest. To compensate for the randomness and potential errors in individual point placements and improve interaction efficiency, we incorporate feature-space clustering that extends the influence of each point beyond its immediate location. For each user click p , we define a local feature window \mathbf{N}(p) and obtain features from the image features \mathbf{I} : \mathbf{I}_{p}=\{\mathbf{I}(q)|q\in\mathbf{N}(p)\}. (1) Applying K -means clustering, we partition \mathbf{I}_{p} into k clusters and select the point closest to each centroid \mu_{i} as guidance: q^{*}_{i}=\arg\min_{q\in C_{i}}\|\mathbf{I}(q)-\mu_{i}\|,i=1,\cdots,k. (2) These selected points provide structured feedback, enabling the model to refine predictions across semantically related regions rather than isolated pixels. The final point prompt tokens are obtained by \mathbf{P}=\mathrm{E}_{P}(q^{*}) . Hybrid token encoder \mathrm{E}_{H} combines image tokens \mathbf{I} , prompt tokens \mathbf{P} , mask tokens \mathbf{M} , and \mathbf{IOU} tokens via self-attention and cross-attention mechanisms, producing the hybrid token \mathbf{Z}=\mathrm{E}_{H}(\mathbf{I},\mathbf{P},\mathbf{M},\mathbf{IOU}) . Self-attention aggregates prompt context while cross-attention enables interaction with image information. The mask tokens \mathbf{M} help initialize and guide the lesion segmentation process, while the \mathbf{IOU} tokens provide an initial metric for optimizing the segmentation through the intersection-over-union score. 2.2 Synergistic Task Integration Decode Module After obtaining the hybrid tokens, we use the mask head and the attribute head to generate the segmentation masks and attribute descriptions, respectively. Recognizing the intrinsic relationship between these tasks, we introduce the inter-task feature synergy mechanism that enables bidirectional information flow, allowing both tasks to mutually enhance each other’s performance. Mask head \mathrm{H}_{seg} : Initial mask tokens \mathbf{M}_{0} interact with the hybrid token \mathbf{Z} through multiple Transformer layers to capture task-specific spatial and semantic features, formulated as \mathbf{M}=\mathrm{H}_{seg}(\mathbf{M}_{0},\mathbf{Z}) .

Read the complete text