A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports

Enobong Adahada*, Isabel Sassoon, Kate Hone and Yongmin Li · 2025

Read A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports by Enobong Adahada*, Isabel Sassoon, Kate Hone and Yongmin Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13796.txt.

Introduction

Enobong Adahada, Isabel Sassoon, Kate Hone, and Yongmin Li present Med-CTX, a system that outlines breast tumors in ultrasound images while explaining its choices, in this 2025 arXiv paper. The system reads the radiology report alongside the image, a pairing meant to ground its output in clinical language. That makes its reasoning easier for doctors to check.

Two vision branches, one based on ViT and one on Swin transformers, study the image, while a medical language model reads the report text. The parts are fused with an uncertainty-aware step, and the system outputs a mask, an uncertainty map, and a written rationale at once. On a standard breast dataset it beat older tools on overlap scores. Removing the report text hurt both accuracy and explanation quality.

Catalog record

CREATOREnobong Adahada*, Isabel Sassoon, Kate Hone and Yongmin Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13796

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports Enobong Adahada*, Isabel Sassoon, Kate Hone and Yongmin Li Abstract We introduce Med-CTX , a fully transformer based multimodal framework for explainable breast cancer ultrasound segmentation. We integrate clinical radiology reports to boost both performance and interpretability. Med-CTX achieves exact lesion delineation by using a dual-branch visual encoder that combines ViT and Swin transformers, as well as uncertainty aware fusion. Clinical language structured with BI-RADS semantics is encoded by BioClinicalBERT and combined with visual features utilising cross-modal attention, allowing the model to provide clinically grounded, model generated explanations. Our methodology generates segmentation masks, uncertainty maps, and diagnostic rationales all at once, increasing confidence and transparency in computer assisted diagnosis. On the BUS-BRA dataset, Med-CTX achieves a Dice score of 99% and an IoU of 95% , beating existing baselines U-Net, ViT, and Swin. Clinical text plays a key role in segmentation accuracy and explanation quality, as evidenced by ablation studies that show a -5.4% decline in Dice score and -31% in CIDEr . Med-CTX achieves good multimodal alignment (CLIP score: 85%) and increased confidence calibration (ECE: 3.2%), setting a new bar for trustworthy, multimodal medical architecture. I Introduction Despite significant technical advances, the clinical adoption of artificial intelligence (AI) in medical imaging remains limited because of the persistent trust gap, the system is unable to clearly defend its decisions and how it arrives at those decisions. While state-of-the-art (SOTA) systems can achieve high segmentation accuracy, they often function as opaque ”black boxes” providing predictions without communicating confidence, rationale, or clinical alignment. In highly critical medical environments, clinicians need more than accurate predictions, they need explainable decisions, actionable uncertainty estimates, and alignment with domain standards such as BI-RADS [ 1 ] [ 2 ] , and clinical or radiology texts or notes. The Breast Imaging Reporting and Data System (BI-RADS) [ 2 ] [ 1 ] provides a standardized way of reporting lesion assessment across mammography, ultrasound, and MRI, with category specific malignancy risks validated in multi-institutional studies [ 3 ] . Despite its clinical importance, BI-RADS descriptors are rarely integrated into deep learning pipelines, creating three critical gaps in AI assisted breast cancer diagnostics: • Diagnostic Variability : Inter radiologist disagreement rates exceed 25% for BI-RADS 4 lesions [ 3 ] , directly contributing to 28% of breast imaging malpractice claims [ 4 ] [ 5 ] . Because they function as black boxes, current AI systems make this variability worse. • Clinical Consequences : False negatives in BI-RADS 4 cases delay cancer diagnoses by 6 to 18 months [ 6 ] , while false positives increase unnecessary biopsies by 23% [ 6 ] . Neither outcome is captured by traditional segmentation metrics like Dice score. • Workflow Misalignment : 72% of radiologists report distrusting AI predictions that lack BI-RADS concordance, limiting clinical adoption of even technically accurate medical automated systems. Through the architectural integration of BI-RADS semantics, Med-CTX fills in these gaps and guarantees that diagnostic results are inherently in line with radiological standards. Existing segmentation architectures tend to treat uncertainty quantification, interpretability, and explanation generation as optional post processing steps rather than integral components [ 7 ] [ 8 ] . CNN-based models struggle to capture long range context, while transformer based methods often lack spatial precision. Most importantly, few systems leverage clinical texts such as BI-RADS descriptors or radiologist notes, despite their potential to guide and validate automated diagnostic interpretation. Transformer based models which were initially developed for Large Language Model (LLM) have reshaped vision tasks through their attention mechanisms since its adoption [ 9 ] . Vision Transformers (ViT) [ 10 ] and Swin Transformers [ 11 ] introduced scalable architectures capable of modeling global and local context. Meanwhile, multimodal models like CLIP [ 12 ] and SimVLM [ 13 ] demonstrate the potential of aligning text and image representations through contrastive pretraining. However, their applicability to medical image segmentation, particularly in the presence of uncertainty, is still a work in progress. To address these limitations, this research introduces the Medical Context Transformer Med-CTX , an end-to-end (ETE) uncertainty aware multimodal segmentation framework designed for clinical alignment, interpretability, and explanation generation. Med-CTX processes grayscale medical images alongside both structured (BI-RADS) and unstructured (radiology notes) clinical texts, and the key contributions include: • Multimodal Uncertainty Aware Fusion: A Cross-scale transformer encoder combines global and local attention mechanisms with text modulated uncertainty estimation, enabling pixel wise confidence maps grounded in clinical context. • Visual Decision Guidance: A novel RGB fusion of attention and uncertainty forms interpretable heatmaps that visually indicate model trust levels, that will enhance radiologist decision making. • Dual Pathway Explanation Generation: Med-CTX integrates neural language generation (via GRU decoders) with structured clinical reasoning to produce textual reports that include BI-RADS classification, malignancy risk, and confidence scoring. • Contrastive Pretraining: We adapt CLIP style contrastive learning to the medical domain, aligning visual and textual embeddings to improve segmentation accuracy and multimodal consistency. • Clinical Workflow Integration: Med-CTX enhances diagnostic accuracy and reduces decision time for radiologists by supporting their workflows with intelligent evaluation, artefact detection, and real-time uncertainty quantification. II Related Work II-A Medical Image Segmentation Medical image segmentation is the process of dividing medical images into anatomically or pathologically relevant sections, such as organs, lesions, or tumours [ 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] . It is essential for disease diagnosis, therapy planning, and disease progression monitoring. Historically dependent on radiologists’ manual annotation, segmentation is laborious and susceptible to interobserver variability which is the degree of difference (or inconsistency) between the observations, measurements, or judgments made by different people when they are assessing the same patient. Deep learning models have significantly advanced the field of medical image segmentation, with U-Net [ 24 ] [ 25 ] , ResUNet [ 25 ] , and nnU-Net [ 26 , 27 , 28 ] emerging as widely adopted or top tier in deep learning architectures. These models are exceptional when it comes to spatial regularization and local feature extraction. However, they lack the integration of textual and clinical context, which is crucial for model explainability and clinical trust, and are intrinsically constrained in their ability to capture long-range dependencies [ 29 ] . II-B Transformer Based Vision Architectures Transformers were originally developed for natural language processing (NLP) tasks [ 9 ] , but have been successfully adapted to vision through architectures such as the Vision Transformer (ViT) [ 10 ] , which divides an image into patches, and creates tokens to model global dependencies. In medical imaging, ViTs have demonstrated competitive performance across classification, segmentation, and synthesis tasks [ 30 ] . Despite ViTs’ effectiveness in high resolution medical images, it is nonetheless, limited by its lack of hierarchical structure, making it less effective at preserving fine grained spatial details that are crucial in clinical settings. To address this limitation, Swin Transformer [ 11 ] introduces shifted window based attention and a hierarchical pyramid structure. Swin-UNETR [ 31 ] demonstrates its effectiveness in medical image segmentation. However, Swin primarily captures local context and does not capture the same level of

Read the complete text