TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation
1 st Qiang Gao, 2 nd Yi Wang*, 3 rd Yong Zhang, 4 th Yong Li, 4 th Yongbing Deng, 5 th Lan Du, 6 th Cunjian Chen*, Qiang Gao 1 , Yi Wang 2∗ , Yong Zhang 2 , Yong Li 2 , Yongbing Deng 3 , Lan Du 1 , Cunjian Chen 1∗ · 2026
Read TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation by 1 st Qiang Gao, 2 nd Yi Wang*, 3 rd Yong Zhang, 4 th Yong Li, 4 th Yongbing Deng, 5 th Lan Du, 6 th Cunjian Chen*, Qiang Gao 1 , Yi Wang 2∗ , Yong Zhang 2 , Yong Li 2 , Yongbing Deng 3 , Lan Du 1 , Cunjian Chen 1∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10912.txt.
Introduction
Gao and six colleagues propose TAMISeg, a text-guided medical image segmentation frame using clinical prompts and semantic distillation to sharpen visual grasp while leaning less on pixel labels. It joins three parts: a consistency-aware encoder pre-trained with strong perturbations for robust features, and a semantic distillation module taught by a frozen DINOv3 teacher for sharper meaning. A scale-adaptive decoder segments anatomy across spatial scales. Tests on Kvasir-SEG, MosMedData+, and QaTa-COV19 show TAMISeg beats existing single- and multi-modal methods in qualitative and quantitative checks.
Catalog record
| CREATOR | 1 st Qiang Gao, 2 nd Yi Wang*, 3 rd Yong Zhang, 4 th Yong Li, 4 th Yongbing Deng, 5 th Lan Du, 6 th Cunjian Chen*, Qiang Gao 1 , Yi Wang 2∗ , Yong Zhang 2 , Yong Li 2 , Yongbing Deng 3 , Lan Du 1 , Cunjian Chen 1∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25967 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10912
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation Thanks: This work was supported in part by the Natural Science Foundation of Chongqing (CSTB2024NSCQ-KJFZZDX0036); Central University basic research young teachers and students research ability promotion sub-project(2023CDJYGRH-ZD06); Chongqing Technology Innovation and Application Development Project (CSTB2023TIAD-KPX0050); a key joint project of Chongqing Health Commission and Science and Technology Bureau (2024ZDXM007) * indicates the corresponding authors. 1 st Qiang Gao, 2 nd Yi Wang*, 3 rd Yong Zhang, 4 th Yong Li, 4 th Yongbing Deng, 5 th Lan Du, 6 th Cunjian Chen*, Qiang Gao 1 , Yi Wang 2∗ , Yong Zhang 2 , Yong Li 2 , Yongbing Deng 3 , Lan Du 1 , Cunjian Chen 1∗ Abstract Medical image segmentation remains challenging due to limited fine-grained annotations, complex anatomical structures, and image degradation from noise, low contrast, or illumination variation. We propose TAMISeg, a text-guided segmentation framework that incorporates clinical language prompts and semantic distillation as auxiliary semantic cues to enhance visual understanding and reduce reliance on pixel-level fine-grained annotations. TAMISeg integrates three core components: a consistency-aware encoder pretrained with strong perturbations for robust feature extraction, a semantic encoder distillation module with supervision from a frozen DINOv3 teacher to enhance semantic discriminability, and a scale-adaptive decoder that segments anatomical structures across different spatial scales. Experiments on the Kvasir-SEG, MosMedData+, and QaTa-COV19 datasets demonstrate that TAMISeg consistently outperforms existing uni-modal and multi-modal methods in both qualitative and quantitative evaluations. Code will be made publicly available at https://github.com/qczggaoqiang/TAMISeg . I Introduction Medical image segmentation plays an essential role in accurately delineating pathological regions, including pulmonary infectious diseases and colorectal polyps [ 22 ] . It enables accurate localization of lesions, significantly supporting clinical diagnosis, therapeutic planning, and disease monitoring. Deep learning-based methods have achieved significant progress in medical image segmentation, including CNN-based architectures such as U-Net [ 16 ] and U-Net++ [ 26 ] , as well as hybrid CNN-Transformer frameworks such as TransUNet [ 3 ] , which have been effectively applied to lesion segmentation in medical images. Despite the remarkable progress of these methods, achieving reliable performance remains challenging due to the inherent complexity of anatomical structures, limited pixel-level details, and frequent image degradation caused by noise, low contrast, or poor illumination. Additionally, most existing segmentation models rely solely on visual information and require a considerable amount of pixel-level annotations to achieve satisfactory performance [ 3 , 26 , 12 ] . However, obtaining such detailed annotations in clinical settings is often impractical due to the high cost and the need for significant expert involvement [ 11 ] . Furthermore, visual-only approaches remain vulnerable to performance degradation under challenging conditions, such as low-contrast tissues, image noise, and uneven illumination [ 12 , 13 ] . These limitations highlight the need for more robust and semantically enriched segmentation frameworks. While CondSeg [ 12 ] enhances feature robustness via consistency-driven visual learning, its CNN-based design captures only local features and lacks textual semantic cues, limiting its capability in modeling global context and complex anatomical structures. Motivated by the substantial performance gains demonstrated by MedCLIP [ 19 ] through the integration of textual and visual modalities, the task of text-guided medical image segmentation has garnered growing interest [ 14 , 9 , 6 , 22 , 25 ] . In clinical practice, diagnostic text reports are typically produced in parallel with medical images, making it possible to access corresponding textual information without incurring extra annotation costs [ 7 ] . However, existing text-guided segmentation methods still suffer from limited semantic discriminability, which restricts the model’s ability to extract rich contextual features from medical images. Additionally, many approaches adopt a uniform decoding mechanism, which overlooks the inherent scale variability of anatomical structures. As a result, small lesions are often missed, while large regions may be inaccurately localized. Furthermore, diffusion-based segmentation frameworks [ 21 , 20 ] have demonstrated strong performance through generative priors, yet their high computational complexity and training cost limit their efficiency and scalability in clinical applications. To address these limitations, we propose TAMISeg, a t ext- a ligned m edical i mage seg mentation framework that leverages clinical language prompts as auxiliary semantic cues to enrich visual understanding while reducing the reliance on dense pixel-level annotations. Rather than performing early fusion of text and image features, TAMISeg sequentially incorporates cross-modal knowledge through a pipeline of robust visual encoding, feature enhancement via distillation, and semantic alignment via a Cross-modal Alignment Module (CMA) with clinical text prompts. A Consistency-Aware Encoder (CAE) is pretrained with strong data perturbations to enhance robustness under noisy, low-contrast, or poorly illuminated conditions, ensuring reliable feature extraction in diverse clinical scenarios. To further improve the semantic discriminability of visual contextual representations, a Semantic Encoder Distillation (SED) module was proposed to enhance the multi-scale encoder features using supervision from a pretrained DINOv3 teacher model [ 17 ] , which captures global semantic dependencies through its Transformer-based architecture. Although knowledge distillation has been occasionally adopted in text-guided segmentation, our framework is among the first to integrate DINOv3-based semantic encoder distillation for enhancing multi-scale visual representations prior to text alignment. In addition, TAMISeg employs a Scale-Adaptive Decoder (SAD) that accommodates anatomical structures of different sizes through specialized decoding branches, enabling precise and scale-sensitive segmentation across varied lesions. The contributions of our work are summarized below. • We propose TAMISeg, a novel text-guided medical image segmentation framework that leverages clinical language prompts as auxiliary semantic cues to reduce reliance on dense visual annotations. • We introduce a Semantic Encoder Distillation module, which enriches pixel-level encoder outputs via supervision from a frozen DINOv3 teacher, improving the semantic discriminability of multi-scale encoder features. • We present a Scale-Adaptive Decoder that operates on text-aligned multi-scale features through parallel decoding branches specialized for anatomical structures of different sizes, improving segmentation accuracy across varying spatial scales. II Methods Fig. 1: Overview of the proposed TAMISeg framework. Phase I : Consistency-Aware Encoder. The encoder is trained with strong data perturbations, enforcing prediction consistency between original and augmented images to learn robust representations. Phase II : Cross-modal Fine-tuning. The pretrained encoder outputs four hierarchical features ( f_{1} – f_{4} ), which are distilled by a frozen DINOv3 teacher to enhance semantic quality. Clinical language prompts encoded by a frozen CXR-BERT model [ 1 ] guide these features via cross-modal attention, producing aligned maps ( {f_{t}}_{1} – {f_{t}}_{4} ). A Scale-Adaptive Decoder (SAD) then decodes multi-scale features through parallel branches specialized for varying anatomical sizes to generate the final segmentation. II-A Overview of TAMISeg We propose TAMISeg, a robust text-guided framework for medical image segmentation, designed to improve visual feature quality and effectively incorporate clinical language prompts as auxiliary semantic cues. The architecture, as illustrated in Fig 1 , follows a two-phase design: a consistency-aware pretraining phase for high-quality feature extraction (Phase I), and a fine-tuning phase where multi-scale features are enhanced via semantic encoder distillation and aligned with clinical language prompts before being decoded for precise segmentation (Phase II). Specifically, the first phase begins with a Consistency-Aware Encoder, which is pretrained under strong augmentations to ensure robustness to image degradation such as noise, low contrast, and illumination shifts. This encoder generates four hierarchical feature maps ( f_{1}