Hybrid(Transformer+CNN)-based Polyp Segmentation

Madan Baduwal Mississippi State University [email protected] https://madanbaduwal.github.io/polyp-seg · 2025

Read Hybrid(Transformer+CNN)-based Polyp Segmentation by Madan Baduwal Mississippi State University [email protected] https://madanbaduwal.github.io/polyp-seg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09189.txt.

Introduction

Madan Baduwal presents a hybrid neural network for segmenting colonic polyps in colonoscopy images. Accurate segmentation is hard because polyps vary in size and shape and often have ill-defined boundaries. Lighting, imaging protocols, folds, and fluid add further difficulty for existing networks such as U-Net variants and PraNet.

The model combines transformer and convolutional components. Boundary-aware attention targets polyps with unclear margins, while the architecture resists artifacts like specular highlights, motion blur, and fluid occlusions. In evaluations, recall improved to 0.9555 and accuracy to 0.9849, beating state-of-the-art methods. Baduwal argues the hybrid design offers more robust polyp segmentation for clinical screening.

Catalog record

CREATORMadan Baduwal Mississippi State University [email protected] https://madanbaduwal.github.io/polyp-seg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-09189

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hybrid(Transformer+CNN)-based Polyp Segmentation Madan Baduwal Mississippi State University [email protected] https://madanbaduwal.github.io/polyp-seg Abstract Colonoscopy is still the main method of detection and segmentation of colonic polyps, and recent advancements in deep learning networks such as U-Net, ResUNet, Swin-UNet, and PraNet have made outstanding performance in polyp segmentation. Yet, the problem is extremely challenging due to high variation in size, shape, endoscopy types, lighting, imaging protocols, and ill-defined boundaries (fluid, folds) of the polyps, rendering accurate segmentation a challenging and problematic task. To address these critical challenges in polyp segmentation, we introduce a hybrid (Transformer + CNN) model that is crafted to enhance robustness against evolving polyp characteristics. Our hybrid architecture demonstrates superior performance over existing solutions, particularly in addressing two critical challenges: (1) accurate segmentation of polyps with ill-defined margins through boundary-aware attention mechanisms, and (2) robust feature extraction in the presence of common endoscopic artifacts including specular highlights, motion blur, and fluid occlusions. Quantitative evaluations reveal significant improvements in segmentation accuracy (Recall improved by 1.76%, i.e., 0.9555, accuracy improved by 0.07%, i.e., 0.9849) and artifact resilience compared to state-of-the-art polyp segmentation methods. Keywords: Colorectal cancer(CRC), Polyp, Polyp segmentation, colonic polyp, colonoscopy, CNN, Transformer 1 Introduction Colon polyps are neoplastic polyps developing from the lining epithelium of the colon or rectum. Incidence epidemiological data suggest a prevalence of approximately 30% among individuals over the age of 50 years [ 42 ] . Although, in most cases, they are benign, a proportion among them, especially adenomatous polyps, are liable to malignancy by the stepwise adenoma-carcinoma sequence. This shift carries critical clinical import, as CRC is still the third most diagnosed malignancy within the United States, with 38 per 100,000 cases and a commensurate rate of death being 13 per 100,000 annually [ 42 ] . Colonoscopy is the cornerstone of preventing CRC, with the ability to detect and treat concomitantly via eliminating pre-malignant polyps. Its use in the initial treatment decreases CRC mortality and incidence by quite a lot [ 39 ] , showcasing its value in practice. Colonoscopy is a minimally invasive but very effective diagnostic procedure for detecting colorectal polyps when performed by skilled endoscopists. While helpful, existing colonoscopy tests fail in the detection of 22-28 % of polyps [ 40 ] , which could develop into advanced cancers and deteriorating clinical outcomes. The process entails using a colonoscope, a finger-thick, flexible tube containing a light source, and a video camera passed transanally to observe the colorectal mucosa. Integrated working channels allow concurrent removal of polyps or biopsy when suspicious lesions are encountered [ 40 ] . Polyp shape is highly heterogeneous during stages of development. As seen in Fig. 2 , variation of structural characteristics, size measurements, and color characteristics provides difficult diagnostic issues at a high level. These are the most demanding challenges for low-contrast small polyps ( <5 mm) in size. These constraints lead to low rates of detection, even being carried out by trained operators using standard image gathering methods. The clinical need for accurate segmentation of polyps beyond detection is the need to precisely demarcate lesion boundaries to direct therapeutic interventions. Online segmentation of polyp architectures (e.g., discrimination between adenomatous and hyperplastic growth patterns) would directly influence intraoperative decisions, allowing for the complete excision of neoplastic lesions with sparing of normal tissue when polyps are not neoplastic. Despite convolutional and transformer-based models achieving >90 % Dice scores on offline testing, clinical adoption is strictly limited by some inherent limitations. Current architectures cannot maintain diagnostic-grade segmentations in true real-time endoscopy settings primarily because of the computational delay involved in the processing of 1080p video streams at 30fps. This is compounded by morphological nuances: indistinct boundaries on sessile or flat polyps decrease segmentation accuracy by 15–20% compared with pedunculated lesions, and motion artifacts and specular reflections also degrade boundary precision. This underlying conflict between model complexity (to provide pixel-level precision) and inference speed (to provide clinical usability) identifies the medical imperative for lightweight but robust segmentation algorithms in gastrointestinal endoscopy. To address these challenges, our work presented in this paper contributes the following: • We suggest a hybrid approach in which pre-trained vision transformers are utilized for global feature extraction and light-weight CNNs for spatial fine-tuning to achieve precise polyp segmentation and minimize computational complexity. • Achieved the highest frames-per-second (FPS) alongside state-of-the-art (SOTA) results in performance metrics on various datasets (such as the Kvasir-SEG dataset [ 25 ] ), compared to other SOTA models like NanoNet [ 28 ] , ResUNet++ [ 27 ] , and ResUNet++ + CRF [ 24 ] . Figure 2: Comparative visualization of polyp segmentation results on the Kvasir-SEG dataset. From left to right: (a) Original endoscopic images, (b) Pixel-level ground truth annotations, and (c) Predicted segmentation masks generated by our proposed model. 2 Related work 2.1 Classic methods Early works proposed methods to address the problem of polyp segmentation using classical image processing techniques [ 38 ] . However, these methods struggled to achieve satisfactory performance due to the similarity between the polyps and the surrounding background. 2.2 Convolution networks Deep learning techniques [ 50 , 36 , 19 ] have immensely enhanced the performance of polyp segmentation. U-Net model [ 47 ] with its encoder-decoder in generic form and skip connections is a smooth benchmark for segmentation of polyps. Its variations, such as U-Net++ [ 70 ] , used nested skip connections to visualize fine details while ResUNet [ 68 ] and ResUNet++ [ 27 ] made use of residual blocks to achieve smooth gradient propagation and feature discovery. DoubleU-Net [ 23 ] further enhanced segmentation using a two-stage U-Net model, further advancing the detection of small polyps. Recent advancements have concerned attention mechanism boost and contextual modeling. PraNet [ 17 ] proposed reverse attention and boundary refining and achieved SOTA performance in a number of polyp benchmarks. DDANet [ 33 ] also utilized dilated dual attention blocks to boost contextual modeling, and UACANet-S/L [ 43 ] utilized channel attention to concentrate on polyp regions. For addressing size variation, MSRF-Net and MSRFE-Net [ 63 ] utilized multi-scale residual fusion and boosted segmentation of various polyp sizes. Real-time segmentation effort has also been made. Jha et al. [ 28 ] employed Conditional Random Field (CRF) post-processing for enhancing contextual information, whereas Thambawita et al. [ 55 ] proposed pyramid-based augmentation for generalization. ColonSegNet [ 22 ] was explicitly designed for real-time segmentation on the Kvasir and CVC datasets, but with some loss of accuracy for the sake of speed. Jha et al. [ 28 ] then created NanoNet, which is a light model with three variants (NanoNet-A, NanoNet-B, and NanoNet-C) that balance speed and accuracy. Among these, NanoNet-A has greater accuracy with more parameters, whereas NanoNet-C puts greater emphasis on speed with fewer parameters. 2.3 Transformers networks Transformers were originally proposed in natural language processing (NLP) and delivered outstanding performance [ 56 ] . Transformers use multi-head self-attention (MHSA) layers for capturing long-range dependencies. Dosovitskiy et al. [ 14 ] transferred transformers to computer vision, presenting the Vision Transformer (ViT), which represents images as a sequence of patch embeddings. Although ViT achieves good results in classification, low-resolution, single-scale feature maps are hard to use in dense prediction tasks such as segmentation

Read the complete text