AI‐Enabled Mucus Segmentation in Nasal Endoscopy with State Space and Attention‐Based Modeling Authors: Dipesh Gyawali, Jonathan Bidwell, Elena Karras, Thomas J Mundy, Kayla E Baker, Elena D Bartolone, Dhara Patel, Akio Fujiwara, Edward D McCoul Published: 2026 ABSTRACT ABSTRACTObjectiveMucus segmentation in nasal endoscopy is essential for sinusitis diagnosis, yet existing models struggle with separating thin mucus layers and serous secretions, delineating boundaries, and handling illumination artifacts like overexposure and reflections. We introduce SUM‐MucusNet, an artificial intelligence model designed to assist clinicians in identifying mucus across challenging settings.MethodsWe trained SUM‐MucusNet, a deep learning model, on 3486 annotated endoscopy images from 454 patients. Performance was measured against expert physician observations using the Dice coefficient, Intersection over Union (IoU), precision and recall, and the comparative analysis was performed with the existing models.ResultsSUM‐MucusNet achieved a Dice coefficient of 0.71 (95% CI: 0.70–0.72), representing a modest but statistically supported improvement over the next‐best model (Wilcoxon signed‐rank p = 0.027). In poor‐quality images, the model correctly segmented 89.3% of mucus regions, with improved boundary detection in cases of irregular mucus patterns and reduced false positives caused by glare and overexposure. The system's targeted architectural components address specific diagnostic challenges while maintaining real‐time processing at 17.3 frames per second.ConclusionThis work improves mucus segmentation in diagnostic nasal examinations by addressing image quality, boundary irregularity, and illumination challenges, advancing an under‐explored area in medical computer vision to support standardized, quantitative mucus assessment and longitudinal disease monitoring during diagnostic nasal endoscopy.Level of Evidence2.Keywords: artificial intelligence, computer‐assisted diagnosis, mucus segmentation, nasal endoscopy, sinusitis Introduction Sinusitis is one of the most prevalent upper respiratory conditions, leading to millions of medical visits and over $20 billion in healthcare expenditures annually in the US [1]. Diagnostic nasal endoscopy (DNE) is the optimal modality for assessing sinonasal conditions, yet interpretation relies on visual estimation, which introduces inter‐observer variability and limits quantitative longitudinal assessment [2]. Mucus is an essential feature for the diagnosis of sinusitis, one of the most prevalent and treatable indications for performing DNE. AI‐enabled mucus segmentation has the potential to standardize evaluations and reduce observer variability, providing consistent, reproducible measurements to support clinical decision‐making and disease monitoring [3]. However, the observation and interpretation of the finding of mucus can be challenging. Integrating accurate automated mucus segmentation into DNE workflows would enhance diagnostic reliability and enable quantitative tracking of treatment response over time [3]. Artificial intelligence (AI)‐assisted diagnostic tools represent a potential solution, although they must overcome unique challenges in the nasal cavity environment. Modern deep learning architectures, such as those used in colonoscopy and other endoscopic applications, have shown promise in medical imaging [4]. However, DNE presents distinct challenges such as thin, translucent layers, amorphous secretion patterns, and illumination artifacts that obscure boundaries, particularly in low‐contrast regions where mucus blends with surrounding tissue, causing boundaries to shift erratically between frames as shown in Figure 1. Modern architectures such as Swin‐UNet [5] and Swin‐UMamba [6] have achieved strong performance in medical image segmentation; however, even when fine‐tuned, these general‐purpose models often produce unstable segmentation results during DNE. Examples of challenges in DNE mucus segmentation: Poor image quality and low contrast (left), irregular morphology (middle), and illumination artifacts (right). The aim of this study was to develop and validate an AI model capable of accurately segmenting mucus in DNE images. Our specific objectives were to: (1) identify the primary challenges in automated mucus detection through systematic error analysis, (2) develop targeted solutions to address these challenges while maintaining real‐time performance, and (3) benchmark our system's performance against existing AI models. Related Work Mucus segmentation in DNE presents three distinct challenges beyond existing methods. First, mucus exhibits variable transparency and low contrast, unlike well‐defined structures in colonoscopy [7, 8, 9]. Transformer models like Swin‐UNet improve general segmentation but overlook subtle mucus patterns, while contrast enhancement methods [10] fail under DNE's dynamic lighting. Second, amorphous mucus morphology complicates segmentation compared to stable structures in bronchoscopy [11]. CNN models [12, 13] handle rigid boundaries but struggle with diffuse regions, while boundary‐aware networks [14, 15] fail to address mucus deformation in complex passages. Third, illumination artifacts from endoscopic imaging disrupt segmentation unlike controlled retinal environments [16], with existing techniques [17] unable to adjust for nasal cavity reflections. Materials and Methods We developed SUM‐MucusNet to address three mucus segmentation challenges: (1) detecting thin translucent layers, (2) delineating irregular boundaries, and (3) compensating for illumination artifacts [18, 19, 20]. The model integrates: (1) hierarchical Mamba modeling with Swin‐UNet for enhanced thin mucus differentiation, (2) deformable convolutions for adaptive boundary refinement [21], and (3) Convolutional Block Attention Module (CBAM)‐based attention to mitigate overexposure and reflections [22]. Data Collection, Annotation and Quality Control From 4216 endoscopy images captured during diagnostic examinations of 454 patients (2014–2024) at our tertiary center, we selected 3486 images using defined criteria: (1) adequate focus and resolution (≥ 1024 × 768, Karl Storz TP‐100), (2) minimal motion blur, (3) no surgical instrumentation obscuring anatomy, and (4) visible mucus present or confirmed absence for negative examples. Images were stratified to ensure balanced representation across anatomical sites (inferior turbinate, middle meatus, ethmoid recesses), mucus presentation (thin strands, thick accumulations, absent), and image quality levels (poor/moderate/good) based on contrast and clarity. Our exclusion criteria included incomplete examinations, technical failures and images with > 50% obscuration from blood or surgical debris. The dataset size is comparable to specialized endoscopy AI studies [15] and larger than existing nasal endoscopy segmentation datasets, which typically contain fewer than 500 annotated images. Patients underwent diagnostic nasal endoscopy for evaluation of sinonasal symptoms, predominantly chronic rhinosinusitis (CRS) or suspected inflammatory disease. The cohort comprised 454 patients (mean age 61 ± 1.2 years; 42% male, 58% female) undergoing diagnostic evaluation as shown in Table 1. The cohort included both chronic rhinosinusitis with nasal polyps (CRSwNP, n = 92, 20.26%) and without polyps (CRSsNP and normal, n = 362, 79.73%). The dataset included both positive (mucus present) and negative (mucus absent) examples, with mucus‐present images representing approximately 80% of the dataset to reflect typical clinical prevalence. All examinations were performed in patients with native, unoperated sinonasal anatomy; post‐surgical cases were excluded to maintain anatomical consistency during initial model development. This represents the primary clinical scenario where automated mucus detection would support diagnostic decision‐making in non‐surgical patients. TABLE 1Patient cohort characteristics.CharacteristicsValueTotal patients454Total images analyzed3486Images per patient, median [IQR]7 [4–12] *DemographicsAge (years), mean ± SD61 ± 1.2Male sex, n (%)191 (42%)Female sex, n (%)263 (58%)Clinical characteristicsPrimary indication: CRSwNP or inflammatory diseases92 (20.26%)Post‐surgical cases0 (0%) Patient cohort characteristics. Four medical graduates (MDs) completed a three‐phase standardized training protocol: (1) didactic instruction (4 h) on mucus characteristics, anatomical landmarks, and common artifacts; (2) supervised practice annotations of 100 calibration images with iterative feedback from fellowship‐trained rhinologist (EDM, > 15 years' experience); and (3) qualification requiring ≥ 80% Dice agreement with the expert annotations on 100 held‐out validation images before independent annotations. All 3486 images were independently annotated by all four annotators using COCO Annotator software. Mucus regions were delineated as polygon masks, with annotators blinded to each other's work. Inter‐rater reliability demonstrated: (1) Fleiss' kappa across four annotators κ = 0.80, indicating substantial agreement; (2) mean pairwise Dice coefficient between annotators (0.84 ± 0.06); and (3) each annotator's agreement with expert reference (mean Dice = 0.85 ± 0.04). Discrepancies (Dice < 0.70 between any annotator pair, occurring in 8.2% of images) were resolved through consensus review involving all four annotators and the supervising expert. The final consensus annotations served as ground truth for model training. Bottom‐Up Model Development We applied grounded theory using affinity diagramming to categorize segmentation challenges. Using YOLOv11 [17] pretrained on EndoVis 2017 [20] and fine‐tuned on our dataset (80–20 patient‐level split), we reviewed 1000 outputs under expert supervision to map failure patterns. Analysis identified three challenges: poor image quality (49.1%), irregular morphology (100%), and illumination artifacts (17.1%). Motion blur hindered thin mucus localization in low‐contrast areas. Mucus varied from thin strands to diffuse accumulations, with 16.9% particularly difficult. Overexposure and reflections caused false positives. Network Architecture SUM‐MucusNet combines multi‐scale feature fusion, boundary refinement, and illumination management (Figure 2). Multi‐Scale Feature Fusion: We integrated Mamba's state‐space modeling with Swin‐UNet for hierarchical representation and long‐range dependency modeling. Hierarchical attention in bottleneck and skip connections preserves fine details and ensures consistency in low‐quality regions. Multi‐Scale Feature Fusion: We integrated Mamba's state‐space modeling with Swin‐UNet for hierarchical representation and long‐range dependency modeling. Hierarchical attention in bottleneck and skip connections preserves fine details and ensures consistency in low‐quality regions. Boundary Refinement: Deformable convolutions (DCNv2) with 3 × 3 kernels in the bottleneck adapt sampling to improve segmentation where mucus blends with tissue or follows complex contours. Boundary Refinement: Deformable convolutions (DCNv2) with 3 × 3 kernels in the bottleneck adapt sampling to improve segmentation where mucus blends with tissue or follows complex contours. Illumination Artifact Mitigation: CBAM modules in skip connections provide channel attention to suppress excessive brightness from overexposure, while spatial attention suppresses glare‐affected pixels and enhances true mucus regions. Illumination Artifact Mitigation: CBAM modules in skip connections provide channel attention to suppress excessive brightness from overexposure, while spatial attention suppresses glare‐affected pixels and enhances true mucus regions. SUM‐MucusNet architecture: The encoder (left) features a Swin‐UNet backbone with Mamba's state‐space modeling, hierarchical attention, and deformable convolutions for boundary refinement. The CBAM module (top‐middle) enhances spatial and channel representation. The decoder (right) reconstructs segmentation with up‐sampling blocks. Evaluation and Implementation SUM‐MucusNet was implemented in PyTorch with 52 M parameters and trained on 4× NVIDIA L40S GPUs for 200 epochs using two‐phase training (Dice/Cross‐Entropy loss, then Weighted Dice/Focal Loss). Five‐fold cross‐validation ensured robustness. To assess the statistical significance of performance differences between SUM‐MucusNet and the next best model (Swin‐UMamba), we computed per‐image Dice Scores across 697 images. Bootstrap resampling (10,000 iterations with replacement) was performed to estimate 95% confidence intervals for each model's mean Dice coefficient and for the mean paired difference. A Wilcoxon signed‐rank test was used as a complementary non‐parametric test of significance. This study was approved by the Institutional Review Board of Ochsner Health (IRB# 2022.165). Results Overall Performance SUM‐MucusNet achieved a Dice coefficient of 0.71 (95% CI: 0.70–0.72) and an IoU of 0.63 on 697 expert‐annotated images (Table 2). This agreement is comparable to inter‐rater reliability between junior and senior ENT residents, suggesting utility for training and quality assurance. The system maintained real‐time processing at 17.3 FPS, exceeding the 15 FPS clinical threshold. TABLE 2Model performance on mucus segmentation dataset (EndoVis 2017 pretrained).ModelmIoUDice scorePrecisionRecallParameters (M)Inference speed (FPS)YOLOv80.570.650.640.723.840YOLOv110.570.650.650.722.640Swin‐Unet0.310.370.340.3810212.2Swin‐Umamba0.610.700.730.756017.5 SUM‐MucusNet 0.63 0.71 0.73 0.77 52 17.3 Note: Bold values indicate the best‐performing result for each metric. Model performance on mucus segmentation dataset (EndoVis 2017 pretrained). Note: Bold values indicate the best‐performing result for each metric. Comparative Model Performance SUM‐MucusNet showed consistent improvements over existing models (Figure 3). YOLO models achieved 65% Dice, Swin‐UNet 37%, and Swin‐UMamba 70%, which our model improved by 1%–2%. Figure 4 illustrates performance across three challenges. In poor quality images, SUM‐MucusNet distinguished thin mucus from blurred tissue. For irregular morphology, it accurately followed complex patterns. With severe illumination artifacts, it differentiated true mucus from reflections. Table 3 shows architectural component impacts, with performance detailed in Figure 5. Performance comparison of different segmentation models. Examples of SUM‐MucusNet per challenge (1) poor image quality, low contrast (top), (2) irregular mucus morphology (middle), and (3) illumination artifacts (bottom). TABLE 3Impact of architectural components on mucus segmentation performance.Model variant (All pretrained on Endovis 2017)DiceIoUIoU improvement (%)Swin‐Unet baseline0.370.31+ Mamba state‐space0.700.6196+ Hierarchical attention (Bottleneck)0.710.63103+ Deformable convolutions0.700.62100+ Hierarchical attention (Skip connections)0.700.62100+ CBAM (Spatial and channel recalibration)0.710.63103 Impact of architectural components on mucus segmentation performance. Impact of architectural components on performance. Compared with Swin‐UMamba, the next‐best model, SUM‐MucusNet achieved a mean Dice of 0.71 (95% CI: 0.70–0.72) versus 0.70 (95% CI: 0.69–0.71). Bootstrap analysis (10,000 resamples) of per‐image Dice scores yielded a mean paired difference of 0.010 (95% CI: −0.001 to 0.021). A Wilcoxon signed‐rank test confirmed that SUM‐MucusNet scores were significantly higher (p = 0.027), and SUM‐MucusNet achieved a higher Dice coefficient on 373 of 697 images (53.5%). While the aggregate improvement is modest, challenge‐specific analysis in Table 4 shows larger gains in poor image quality (+1.7%), irregular morphology (+2.3%), and illumination artifact cases (+5.3%), which together account for approximately 15%–20% of routine diagnostic examinations. TABLE 4True positives by challenge type and model (n = 1000).Challenge typeQuantitySwin‐UmambaSUM‐MucusNet% ImprovementPoor image quality + Tissue misclassified131115117 1.7% Irregular mucus shape1000887908 2.3% Illumination artifact291920 5.3% Note: Bold values represent total percentage improvement across different challenge type. True positives by challenge type and model (n = 1000). Note: Bold values represent total percentage improvement across different challenge type. Performance by Clinical Challenge Category Analysis of segmentation performance across different clinical scenarios revealed important patterns in Table 4. In cases with poor image quality—common in routine clinical practice, SUM‐MucusNet correctly identified 117 of 131 mucus regions (89.3%), compared to 115 for the next‐best model. For irregular mucus morphology, present in all images but particularly challenging in 16.9%, our model achieved a 90.8% true positive rate. In images with significant illumination artifacts (n = 29), performance improved from 65.5% to 69.0%; however, this subgroup is small (representing a difference of one additional true positive), and the finding should be considered preliminary. Discussion This study demonstrates the potential of our newly developed model, SUM‐MucusNet, to improve mucus segmentation in DNE by overcoming intrinsic challenges such as low image quality, irregular mucus morphology, and dynamic illumination artifacts. The model achieves a Dice coefficient of 0.71 (95% CI: 0.70–0.72) with expert‐labeled annotations, a modest but statistically supported improvement over existing approaches (Wilcoxon p = 0.027). While the aggregate Dice improvement of 1% is small in absolute terms, the bootstrap confidence interval for the paired difference (−0.001 to 0.021) and the consistency of improvement across 53.5% of test images suggest a genuine directional advantage rather than random variation. The clinical value of SUM‐MucusNet lies primarily in its targeted architectural innovations for specific diagnostic challenges and its potential for standardized, quantitative mucus assessment. SUM‐MucusNet enhances contrast sensitivity, boundary delineation, and illumination robustness—balancing accuracy and efficiency to meet the clinical requirement of 15 FPS, achieving 17 FPS in practice on our GPU hardware. The largest gains stem from hierarchical attention (+1% Dice) and CBAM‐based exposure normalization (+1% Dice), which stabilize segmentation under low contrast and dynamic lighting. Eliminating hierarchical attention in skip connections reduced the Dice score by 6%, underscoring the importance of preserving feature information across network layers (Figure 5). These improvements, though small in percentage, have potential clinical implications that warrant further investigation. As shown in Table 4, SUM‐MucusNet improves true positive segmentation across all major challenges, with a 1.7% gain in poor image quality cases, a 2.3% gain in irregular mucus shapes, and a 5.3% gain in illumination artifacts (though the latter subgroup of n = 29 is too small for robust quantitative conclusions). In real‐world settings, this means fewer missed mucus regions, more stable boundaries in complex patterns, and fewer false positives due to overexposure. At 17 FPS, these refinements reduce flickering artifacts from 119–238 frames to 113–225 frames, improving segmentation stability and real‐time usability. By achieving high segmentation accuracy, SUM‐MucusNet provides consistent automated measurements of mucus in AI‐assisted endoscopy. Hierarchical attention enhances detection of thin, transparent mucus, deformable convolutions refine irregular boundaries, and CBAM‐based recalibration prevents white mucus from being misclassified as reflections, ensuring reliable differentiation as shown in Figure 4. These refinements minimize segmentation errors, supporting smoother AI‐assisted diagnostics. Resolving Tissue Misclassification—Low Image Quality and Contrast SUM‐MucusNet enhances thin mucus detection by integrating Mamba's state‐space modeling with hierarchical attention, improving feature fusion across scales. As shown in Table 3, hierarchical attention improved Dice score by 5%–6%, capturing fine‐grained structures. The 1.7% gain in true positives for poor image quality (Table 4) translates to approximately 17 additional correct diagnoses per 1000 examinations. In clinical practice, this could help identify early‐stage sinusitis with minimal mucus accumulation, particularly by less experienced endoscopists. Enhancing Boundary Delineation—Irregular Mucus Boundaries Mucus accumulates in highly irregular structures, particularly the middle meatus and ethmoid recesses, where fixed convolutions fail to adapt, causing boundary inconsistencies. Deformable convolutions (DCNv2) enable dynamic kernel sampling aligned with local structures, reducing boundary errors by 42% and enabling accurate mucus volume measurements for treatment monitoring. This achieved a 2.3% gain in true positives for irregular shapes (Table 4), improving segmentation of amorphous structures while preserving healthy mucosa—particularly important near sinus ostia and nasal recesses for accurate disease monitoring. Mitigating Illumination Artifacts—Overexposure and Reflections CBAM‐based recalibration dynamically adjusts spatial and channel attention, reducing false positives from reflections by 61% and false negatives from overexposure by 37%. This contributes a 1% IoU increase (Table 2). The 5.3% improvement in illumination artifact cases (19 vs. 20 true positives, n = 29) should be interpreted with caution given the small subgroup size. This finding is hypothesis‐generating and requires validation in larger, multi‐center samples. The qualitative examples in Figure 4 illustrate the mechanism by which CBAM‐based recalibration differentiates mucus from specular reflections, but quantitative confirmation of this advantage awaits larger datasets. Limitations include single‐institution data limiting generalizability and lack of prospective clinical validation. The 1%–2% Dice improvement requires clinical validation to confirm diagnostic impact. Our model was trained on native anatomy only; future work should assess post‐surgical performance. Multicenter validation is needed to establish clinical utility across diverse settings. Additionally, our model was trained exclusively on native sinonasal anatomy and has not been validated in post‐surgical patients, who represent a significant proportion of rhinology practice. Future work should assess performance in altered anatomy, including prior endoscopic sinus surgery or septoplasty. Failure Mode Analysis and Model Limitations Despite achieving 71% Dice coefficient overall, qualitative analysis of failure cases revealed systematic patterns that highlight remaining challenges as shown in Figure 4. Extreme thin‐layer mucus (thickness < 0.5 mm) was frequently missed or under‐segmented, particularly when overlying mucosal edema of similar appearance. In these cases, even expert annotators showed lower agreement (mean Dice: 0.64), suggesting inherent task difficulty rather than pure model failure. Dense mucoid secretions with tissue‐like color presented another challenge, with the model occasionally misclassifying pale, thick mucus as normal mucosa, particularly in the sphenoethmoidal recess where lighting is typically reduced. This occurred in approximately 8% of cases with extensive mucus burden. Concurrent blood or purulence mixed with mucus caused segmentation instability, as the model was trained primarily on clear‐to‐white mucoid secretions. The presence of hemorrhagic or overtly purulent discharge degraded performance (Dice reduction of 12%–18%), though this represents < 5% of diagnostic examinations in our dataset. Finally, severe anatomical variants (e.g., concha bullosa, paradoxical middle turbinate) occasionally disrupted segmentation when combined with poor image quality, suggesting the model may benefit from explicit anatomical feature encoding in future iterations. These failure modes inform ongoing model refinement and define scenarios requiring heightened clinical vigilance. Importantly, these challenging cases represent < 15% of our test set, suggesting the model performs reliably in most routine diagnostic examinations. Our annotation protocol aimed to minimize bias through several mechanisms: (1) blinded independent annotation preventing annotator influence, (2) diverse annotator training levels ensuring varied perspectives, and (3) expert consensus resolution for disagreements. However, potential biases remain. All annotators trained at a single institution may share systematic interpretation patterns that differ from other clinical settings. Additionally, the reference standard (expert rhinologist annotations) represents one subspecialist's interpretation, which may not fully capture the variability in clinical practice. The high inter‐annotator agreement (κ = 0.80) suggests consistent mucus identification within our training paradigm, but external validation with annotations from multiple institutions would better assess the generalizability of our ground truth definitions. Conclusion SUM‐MucusNet represents a clinically viable approach to automated mucus segmentation in DNE. By achieving good agreement with expert clinician observations while maintaining real‐time performance, the system demonstrates potential for integration into clinical workflows. The targeted solutions for thin mucus detection, boundary delineation, and illumination artifact management address the primary challenges encountered in routine endoscopic examinations. The clinical implications extend beyond technical metrics. By providing consistent, quantitative mucus assessment, SUM‐MucusNet could support standardized evaluation for sinusitis and other sinonasal pathology across different practice settings. This standardization is particularly valuable for longitudinal disease monitoring and treatment response assessment. Future multicenter validation studies will be essential to confirm these benefits in diverse clinical environments and establish the system's role in supporting standardized mucus quantification for clinical decision‐making and disease monitoring. Funding This work was supported by a research grant from the Eye, Ear, Nose, and Throat Foundation (EE220702/EE220802). Ethics Statement Ochsner Clinic Foundation Institutional Review Board. Conflicts of Interest E.D.M. is a consultant for 3D Matrix, Advanced Rx, Sanofi/Regeneron, Stryker, and Zsquare. The other authors have nothing to disclose. Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request. References Associated Data Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request.