DenseUNet for Breast Cancer Segmentation in Histopathological Images Authors: Habib Rasi, Hossein Ebrahimnezhad, Mohammad Hossein Sedaaghi Published: 2026 ABSTRACT Background:Breast cancer remains one of the leading causes of mortality among women worldwide, highlighting the urgent need for accurate and efficient diagnostic tools. Histopathological image analysis plays a critical role in diagnosis by enabling cellularlevel tissue examination. However, manual assessment is often timeconsuming, subjective, and prone to variability, driving the need for automated solutions.Methods:This study proposes DenseUNet169, a deep learning framework that integrates established DenseNet and UNet for systematic evaluation in breast cancer segmentation. The model improves gradient flow, facilitates efficient training, and enhances segmentation accuracy by focusing on visually salient regions within histopathological images. Performance is evaluated using key metrics, including Intersection over Union, Dice Coefficient, Pixel Accuracy, Modified Hausdorff Distance, Surface Dice Overlap, Log Loss, and Jaccard Index.Results:Experiments conducted on a publicly available breast cancer segmentation dataset demonstrate competitive performance compared to alternative backbone configurations.Conclusion:Given the limited dataset size, the results should be interpreted as a proof of concept evaluation of the proposed architectural integration rather than evidence of largescale clinical deployment readiness. Introduction Histopathology remains a cornerstone in modern medical diagnostics, providing a vital means for individualized assessment, therapeutic planning, and disease prognosis.[1] The process involves a series of intricate steps, including fixation, tissue processing, sectioning, and staining, each essential to preserving cellular integrity and enabling detailed microscopic evaluation.[2] Fixation stabilizes tissue architecture by halting enzymatic degradation, whereas tissue processing replaces water content with a firm medium to facilitate fine slicing. Sectioning transforms these prepared blocks into ultra-thin slices, and staining imparts contrast to cellular structures, enabling pathologists to identify pathological features with precision.[345] Mastery of histopathological techniques is thus essential for the reliable preparation and interpretation of tissue samples.[6] In recent years, the integration of artificial intelligence (AI) into histopathological image analysis has revolutionized diagnostic workflows, particularly in breast cancer screening and monitoring,[78] Given that early-stage breast conditions often present asymptomatically, the deployment of semi-automated and scalable screening tools has become increasingly critical. AI-driven methods, especially those employing convolutional neural networks (CNNs), offer the potential to not only streamline image classification but also augment the diagnostic accuracy of clinical experts.[9] Deep learning, and CNNs in particular, have demonstrated significant promise in computational histopathology.[1011] Yet, existing models often remain narrowly focused on isolated tasks such as segmentation, lacking the holistic capacity to simultaneously identify spatial locations and morphological features of breast cancer tissues. Addressing this gap requires the development of advanced models that combine robust performance with interpretability two pillars essential for clinical adoption. To this end, we propose DenseUNet169, a deep learning framework that integrates established DenseNet and U-Net for segmentation of cancerous regions in hematoxylin and eosin (H and E)-stained histopathological images. This model is structured around two synergistic components: a segmentation module and a shared DenseNet169-based backbone network. By jointly optimizing these components, DenseUNet169 achieves superior feature extraction and end-to-end segmentation capabilities. Evaluations on breast cancer datasets demonstrate that our model consistently outperforms existing architectures, offering improvements in both detection accuracy and segmentation fidelity. It should be emphasized that the proposed framework does not introduce new learning principles; rather, it investigates the empirical impact of integrating DenseNet-based encoders within a U-Net segmentation paradigm. The contributions of this study are as follows: We present a system-level integration of DenseNet-based with U-Net architectures for breast cancer histopathological image segmentationWe provide a comprehensive empirical benchmark of U-Net variants with different backbone networks under a unified experimental protocolWe offer quantitative and qualitative insights into how backbone depth and dense connectivity influence segmentation accuracy and boundary delineation. We present a system-level integration of DenseNet-based with U-Net architectures for breast cancer histopathological image segmentation We present a system-level integration of DenseNet-based with U-Net architectures for breast cancer histopathological image segmentation We provide a comprehensive empirical benchmark of U-Net variants with different backbone networks under a unified experimental protocol We provide a comprehensive empirical benchmark of U-Net variants with different backbone networks under a unified experimental protocol We offer quantitative and qualitative insights into how backbone depth and dense connectivity influence segmentation accuracy and boundary delineation. We offer quantitative and qualitative insights into how backbone depth and dense connectivity influence segmentation accuracy and boundary delineation. The model’s performance is assessed using a suite of rigorous evaluation metrics, including Intersection over Union (IoU), Mean Pixel Accuracy, Dice Coefficient, Mean Modified Hausdorff Distance (MHD), and Mean Surface Dice Overlap. These metrics provide a comprehensive view of segmentation quality and relevance to computational histopathology research settings. By integrating architectural innovation and robust evaluation, this study advances the frontier of medical image segmentation. The outcomes offer meaningful insights for both clinical practice and the broader research community, paving the way for more transparent, accurate solutions in histopathology. The remainder of this paper is organized as follows: Section 2 surveys related literature to contextualise the research. Section 3 outlines the methodology, principles, CNN architectures, segmentation strategies, and evaluation metrics. Section 4 describes the dataset and training protocol. Section 5 presents experimental results, including performance comparisons and generalisation studies. Finally, Sections 6 and 7 discuss the implications of the findings and conclude the study. Related Works Accurate segmentation of cancerous tissues is a cornerstone in histopathological image analysis, serving as a foundation for evaluating tissue morphology, quantifying pathological features, and facilitating the precise localization of malignant regions. Over the years, a diverse array of deep learning-based methodologies has emerged to address inherent challenges in this domain, particularly those arising from complex tissue structures and overlapping cells. A seminal contribution by Amgad et al.[12] introduced fully CNNs to generate pixel-wise segmentation masks for joint region-and nucleus-level analysis in breast cancer tissue, enabling robust delineation of tumor-infiltrating lymphocytes. By unifying multiple segmentation tasks within a shared framework, their model achieved enhanced generalization and improved predictive performance on the PanNuke dataset. Subsequent studies have continued to refine segmentation accuracy, particularly in the context of overlapping or densely clustered nuclei. For instance, Zhao et al.[13] proposed a hematoxylin-aware color decomposition technique to isolate nuclear components from H and E images within a Triple U-Net architecture. This approach enhanced boundary localization of nuclei by leveraging optical properties of H and E staining, improving segmentation accuracy at the cellular level without requiring color normalization. In a related advancement, Zhou et al.[14] introduced the Contour-aware informative aggregation network, which integrates specialized decoders for nuclei instance segmentation and contour detection. This dual-decoder architecture significantly mitigated the impact of cell overlap through multilevel feature aggregation, leading to more precise segmentations on multiple public benchmarks. Recent efforts have increasingly prioritized both interpretability and computational efficiency in cancer segmentation models. van Rijthoven et al.[15] introduced HookNet, a multi-resolution encoder-decoder network that effectively integrates global context (low resolution) and local details (high resolution) via “hook” connections between parallel branches. This architecture yielded superior performance compared to conventional single-resolution frameworks on whole-slide image (WSI) segmentation tasks while maintaining clinical feasibility. To improve decision transparency, Gu et al.[16] implemented attention-guided CNNs with spatial-channel dual attention mechanisms that dynamically focus on visually salient regions within histopathological images. This design not only aligns the model’s focus with expert pathologists’ assessments but also enhances trust and interpretability in clinical settings. Similarly, Gour et al.[17] developed DeepRNNetSeg, a deep residual network tailored to learn discriminative nuclear features through skip connections and batch normalization, thereby boosting segmentation accuracy in heterogeneous histopathological environments. Schmitz et al.[18] explored multiscale fully convolutional networks with deep feature fusion, explicitly mimicking the diagnostic strategies of human pathologists by integrating fine-grained nuclear details and global tissue architecture. These models demonstrated high accuracy while maintaining a compact memory footprint compared to ensemble approaches, making them suitable for clinical deployment. Beyond segmentation, recent studies have proposed comprehensive systems for diagnosis and prognosis. Khaliliboroujeni et al.[19] presented an end-to-end computer-aided diagnosis system for detecting breast cancer metastases in WSIs. This approach integrates attention modules with regression and classification tasks, enabling dense tumor region prediction from low-resolution inputs. The system achieved higher accuracy than expert pathologists on the CAMELYON16/17 benchmarks and operated up to seven times faster. López-Pérez et al.[20] addressed annotation variability by proposing a novel crowd-sourced learning framework for histopathological segmentation. Their method involves two interacting networks: a segmentation network trained on consensus annotations, and an annotator network that models individual labeling tendencies. This dual-network system enhances scalability and accuracy by leveraging annotator behavior modeling via global image features, representing a substantial innovation in consensus-based segmentation. Complementing these efforts, recent frameworks have also targeted broader diagnostic automation. Boumaraf et al.[21] proposed a two-branch deep CNN architecture that facilitates robust classification of breast histopathology images into benign, malignant, or normal classes. This architecture addresses vanishing gradient issues through residual connections and offers improved efficiency compared to traditional manual feature engineering methods. Further, Drioua et al.[9] introduced dual approaches for semantic segmentation of breast cancer histopathology images: (1) an autoencoder architecture for unsupervised representation learning, and (2) an enhanced U-Net architecture with residual blocks for supervised segmentation. Evaluated on the BreakHis dataset, their supervised variant achieved Dice scores exceeding 0.89 for tumor region delineation. Collectively, these studies underscore a consistent trend: the convergence of accuracy, efficiency, and interpretability in CNN-based breast cancer segmentation. As summarized in Table 1, the diverse methodologies presented offer complementary strengths, contributing to the advancement of reliable and transparent AI-assisted pathology tools. Table 1Summary of key contributions in histopathological image segmentation for breast cancerAuthorsYearMethodologyAmgad et al.[12]2019Employed FCNNs for joint region/nucleus segmentation of tumor-infiltrating lymphocytes in breast cancerZhao et al.[13]2020Applied hematoxylin-aware color decomposition within Triple U-Net to enhance nucleus boundary detectionZhou et al.[14]2019Developed CIA-Net with dual decoders for nuclei instance segmentation and contour detection to handle overlapping cellsvan Rijthoven et al.[15]2024Introduced HookNet, a multi-resolution encoder-decoder for context-detail fusion in WSI segmentationGu et al.[16]2021Implemented spatial-channel attention mechanisms to align model focus with pathologist assessmentsGour et al.[17]2020Developed DeepRNNetSeg using deep residual learning to enhance nuclei segmentation in complex imagesSchmitz et al.[18]2021Employed multi-scale FCNs with deep fusion for high-accuracy, memory-efficient segmentation across scalesKhaliliboroujeni et al.[19]2022Designed a CAD system for fast, accurate metastasis detection in WSIs using attention mechanismsLópez-Pérez et al.[20]2024Proposed a dual-network framework for learning from crowdsourced labels with annotator behavior modelingBoumaraf et al.[21]2021Presented a two-branch CNN architecture with residual connections for breast cancer classificationDrioua et al.[9]2023Proposed autoencoder-based unsupervised segmentation and residual U-Net for supervised tumor delineationFCNNs – Fully convolutional neural networks; CIA-Net – Contour-aware Informative Aggregation Network; WSIs – Whole-slide images; CAD – Computer-aided diagnosis; CNN – Convolutional neural networks; FCNs – Fully convolutional networks Summary of key contributions in histopathological image segmentation for breast cancer FCNNs – Fully convolutional neural networks; CIA-Net – Contour-aware Informative Aggregation Network; WSIs – Whole-slide images; CAD – Computer-aided diagnosis; CNN – Convolutional neural networks; FCNs – Fully convolutional networks Methods The proposed DenseUNet169 architecture is designed with a tailored encoder–decoder structure that enhances segmentation accuracy in histopathological breast cancer imaging. This section outlines the methodological workflow, from data preprocessing and augmentation to model training and evaluation techniques. To standardise the input for deep learning, all histopathological images are resized to 256 × 256 pixels. To increase dataset diversity and mitigate overfitting, a comprehensive data augmentation pipeline is applied. Each image undergoes horizontal and vertical flipping, rotations at 90° and 180°, saturation adjustment, and central cropping, generating seven augmented samples per image. This expands the original dataset of 58 images to a total of 406 samples. The same augmentations (excluding saturation adjustment) are applied to the corresponding ground truth masks to preserve label fidelity. To avoid data leakage, cross-validation splitting was performed at the level of the original 58 images before any augmentation. Data augmentation was applied exclusively to the training subset within each fold, ensuring that augmented versions of the same image did not appear in both training and validation sets. The DenseUNet169 is optimised to leverage dense connectivity and enabling effective feature propagation and focusing on visually salient regions within histopathological images. During training, the model adjusts its weights and biases using the Adam optimiser with a learning rate of 0.001, and the ReLU activation function is used throughout the network. Performance is continuously monitored using a validation set, with the IoU metric guiding model checkpointing. The model achieving the highest IoU score is preserved for final deployment. ReLU: In the proposed architecture, the Rectified Linear Unit (ReLU) is employed as the primary activation function throughout the hidden layers, defined mathematically as f(x)=max(0,x). The selection of ReLU over traditional saturating functions (e.g., Sigmoid or Tanh) is driven by three critical theoretical advantages. First, its non-saturating linear behavior for positive inputs (x>0) ensures a constant gradient of 1, which effectively mitigates the vanishing gradient problem and accelerates convergence during backpropagation. Second, the piecewise linear formulation requires only simple thresholding, yielding superior computational efficiency. Third, by outputting exact zeros for negative inputs, ReLU induces sparse activations, which enhances the network’s representational capacity and provides an implicit regularization effect. To circumvent the potential ’dying ReLU’ issue where neurons may become inactive and output zero for all inputs the network weights are initialized using the He (Kaiming) initialization scheme, which is specifically designed to maintain the variance of activations and gradients in ReLU-based deep networks. In the segmentation phase, the best-performing model is used to infer segmentation masks on unseen test data. The output predictions are quantitatively compared against ground truth annotations using several metrics, including IoU, Dice Coefficient, Pixel Accuracy MHD, Surface Dice Overlap, Jaccard Score, and Log Loss. These comprehensive metrics provide a robust assessment of segmentation performance across spatial, surface, and probabilistic dimensions. To demonstrate the efficacy and robustness of the proposed DenseUNet169, its performance is benchmarked against six state-of-the-art segmentation methods. Comparative analysis is performed using the same evaluation metrics described earlier. This pipeline provides a controlled experimental framework for evaluating segmentation performance under consistent training conditions.[22] Figure 1 illustrates the complete pipeline of the proposed model for breast cancer segmentation, encompassing input preprocessing, feature extraction, segmentation prediction, and clinical validation. Methodology block diagram of the proposed method (DenseUNet169) Implemented architecture The implemented DenseUNet169 follows a U-Net style encoder–decoder framework in which the encoder is replaced by a DenseNet169 backbone pretrained on ImageNet. Skip connections are established between corresponding encoder and decoder stages. The encoder consists of the standard DenseNet169 convolutional blocks (dense blocks and transition layers), excluding the final classification layer. The decoder is composed of sequential upsampling blocks, each consisting of bilinear upsampling followed by convolution, batch normalization, and ReLU activation. The final segmentation output is generated using a 1 × 1 convolution with sigmoid activation for binary segmentation. Architectural configuration The structure of the DenseUNet is characterized as follows: Encoder: A DenseNet-based encoder effectively extracts multiscale featuresDecoder: Following the U-Net paradigm, the decoder reconstructs spatial resolution using upsampling operations, aided by skip connections that bridge encoder and decoder stagesOutput Layer: A final 1 × 1 convolution with appropriate activation (e.g., ReLU or sigmoid) is employed to generate precise segmentation masks. Encoder: A DenseNet-based encoder effectively extracts multiscale features Encoder: A DenseNet-based encoder effectively extracts multiscale features Decoder: Following the U-Net paradigm, the decoder reconstructs spatial resolution using upsampling operations, aided by skip connections that bridge encoder and decoder stages Decoder: Following the U-Net paradigm, the decoder reconstructs spatial resolution using upsampling operations, aided by skip connections that bridge encoder and decoder stages Output Layer: A final 1 × 1 convolution with appropriate activation (e.g., ReLU or sigmoid) is employed to generate precise segmentation masks. Output Layer: A final 1 × 1 convolution with appropriate activation (e.g., ReLU or sigmoid) is employed to generate precise segmentation masks. This architecture is designed to retain spatial fidelity, capture long-range dependencies, and improve feature selection dynamically core requirements for robust medical image analysis. Figure 2[23] presents a high-level overview of the proposed DenseUNet architecture. DenseUNet169 block diagram[23] Key architectural advantages The fusion of DenseNet and U-Net in the DenseUNet architecture yields several notable advantages: Efficient feature reuse: Dense connectivity promotes learning from compact and diverse feature representationsSpatial information retention: Skip connections preserve fine-grained spatial features essential for pixel-level tasksGlobal context awareness: The model captures long-range dependencies, facilitating robust global understanding in segmentationStable and fast convergence: DenseNet’s gradient flow mechanisms reduce the risk of vanishing gradients and accelerate trainingArchitectural versatility: The model is adaptable for various image-to-image translation tasks beyond segmentation. Efficient feature reuse: Dense connectivity promotes learning from compact and diverse feature representations Efficient feature reuse: Dense connectivity promotes learning from compact and diverse feature representations Spatial information retention: Skip connections preserve fine-grained spatial features essential for pixel-level tasks Spatial information retention: Skip connections preserve fine-grained spatial features essential for pixel-level tasks Global context awareness: The model captures long-range dependencies, facilitating robust global understanding in segmentation Global context awareness: The model captures long-range dependencies, facilitating robust global understanding in segmentation Stable and fast convergence: DenseNet’s gradient flow mechanisms reduce the risk of vanishing gradients and accelerate training Stable and fast convergence: DenseNet’s gradient flow mechanisms reduce the risk of vanishing gradients and accelerate training Architectural versatility: The model is adaptable for various image-to-image translation tasks beyond segmentation. Architectural versatility: The model is adaptable for various image-to-image translation tasks beyond segmentation. These architectural properties contribute to the observed performance improvements in the conducted experiments. Model implementation function The practical implementation of the proposed DenseUNet169 involves the following steps: Load pretrained backbone: Utilize a DenseUNet169 model pretrained on ImageNet, excluding its final classification layerIntegrate global mean fusion: Apply global average pooling to condense spatial dimensions, facilitating segmentation-specific representationAdd segmentation layer: Append a 1 × 1 convolutional layer with ReLU activation to produce class-specific segmentation outputsAssemble final model: Define input and output layers to complete the architecture for end-to-end segmentation training. Load pretrained backbone: Utilize a DenseUNet169 model pretrained on ImageNet, excluding its final classification layer Load pretrained backbone: Utilize a DenseUNet169 model pretrained on ImageNet, excluding its final classification layer Integrate global mean fusion: Apply global average pooling to condense spatial dimensions, facilitating segmentation-specific representation Integrate global mean fusion: Apply global average pooling to condense spatial dimensions, facilitating segmentation-specific representation Add segmentation layer: Append a 1 × 1 convolutional layer with ReLU activation to produce class-specific segmentation outputs Add segmentation layer: Append a 1 × 1 convolutional layer with ReLU activation to produce class-specific segmentation outputs Assemble final model: Define input and output layers to complete the architecture for end-to-end segmentation training. Assemble final model: Define input and output layers to complete the architecture for end-to-end segmentation training. Once implemented, this model is fine-tuned on breast histopathological data, ensuring domain-specific performance optimization. Segmentation module The segmentation module consists of a single binary mask prediction branch. Although accurate boundary delineation is an important objective, no explicit boundary supervision or auxiliary loss term was introduced. Improvements in contour representation arise from the architectural integration rather than from a dedicated boundary-aware training objective. Performance is evaluated using key metrics, including Intersection over Union, Dice Coefficient, Pixel Accuracy, Modified Hausdorff Distance, Surface Dice Overlap, Log Loss, and Jaccard Index (as detailed in Table 2). Table 2Evaluation metrics for segmentationMetricsFormulaParametersDescriptionIoU Cancerous tissueMeasures the overlap between predicted segmentation masks and ground truth[24]Dice coefficient (Dice) A represents the predicted segmentation maskBalances false positives and false negatives[25]B represents the ground truth segmentation maskMPA PixelsEvaluates pixel-wise classification performance. Indicates how many pixels are correctly labeled over the entire image[26]Mean MHD N is the total number of ground truth segmentation masksA metric to quantify the spatial dissimilarity between predicted and ground truth segmentation masks, with a particular focus on spatial deviations (the model’s ability to accurately represent the overall structure and spatial distribution of cancerous tissue)[2728]A represents the predicted segmentation maskB represents the ground truth segmentation maskd(a,b) denotes the distance between pixel a in the predicted mask and pixel b in the ground truth maskMSD N is the total number of ground truth segmentation masksEvaluates the concordance between the predicted and ground truth surfaces, offering a robust measure of segmentation accuracy (model’s ability to capture the structural intricacies and complexity of cancerous tissue)[28]A represents the predicted segmentation maskB represents the ground truth segmentation maskIoU – Intersection over Union; MPA – Mean pixel accuracy; MHD – Modified Hausdorff Distance; MSD – Mean Surface Dice Overlap Evaluation metrics for segmentation IoU – Intersection over Union; MPA – Mean pixel accuracy; MHD – Modified Hausdorff Distance; MSD – Mean Surface Dice Overlap Key components of the segmentation framework U-Net backbone: The segmentation framework builds upon the classical U-Net model, which is well-suited for biomedical image analysis due to its symmetrical encoder–decoder structure. The encoder progressively captures abstract semantic features, whereas the decoder reconstructs high-resolution segmentations through upsampling and skip connections, ensuring both contextual understanding and spatial precisionDense connectivity: Dense connections are integrated into the U-Net architecture, forming DenseUNet-style blocks. These connections concatenate feature maps from preceding layers directly into subsequent ones, facilitating continuous information flow and maximizing feature reuse. This design enriches the model’s representational capacity while reducing the risk of vanishing gradients during trainingTraining strategy: The segmentation module is trained on a carefully curated dataset of annotated breast histopathological images. The training regimen emphasizes class balance and includes regularization techniques to improve generalizability. The integration of boundary supervision further fine-tunes the network’s sensitivity to tumor contours, resulting in improved segmentation accuracy in regions with structural complexity U-Net backbone: The segmentation framework builds upon the classical U-Net model, which is well-suited for biomedical image analysis due to its symmetrical encoder–decoder structure. The encoder progressively captures abstract semantic features, whereas the decoder reconstructs high-resolution segmentations through upsampling and skip connections, ensuring both contextual understanding and spatial precision U-Net backbone: The segmentation framework builds upon the classical U-Net model, which is well-suited for biomedical image analysis due to its symmetrical encoder–decoder structure. The encoder progressively captures abstract semantic features, whereas the decoder reconstructs high-resolution segmentations through upsampling and skip connections, ensuring both contextual understanding and spatial precision Dense connectivity: Dense connections are integrated into the U-Net architecture, forming DenseUNet-style blocks. These connections concatenate feature maps from preceding layers directly into subsequent ones, facilitating continuous information flow and maximizing feature reuse. This design enriches the model’s representational capacity while reducing the risk of vanishing gradients during training Dense connectivity: Dense connections are integrated into the U-Net architecture, forming DenseUNet-style blocks. These connections concatenate feature maps from preceding layers directly into subsequent ones, facilitating continuous information flow and maximizing feature reuse. This design enriches the model’s representational capacity while reducing the risk of vanishing gradients during training Training strategy: The segmentation module is trained on a carefully curated dataset of annotated breast histopathological images. The training regimen emphasizes class balance and includes regularization techniques to improve generalizability. The integration of boundary supervision further fine-tunes the network’s sensitivity to tumor contours, resulting in improved segmentation accuracy in regions with structural complexity Training strategy: The segmentation module is trained on a carefully curated dataset of annotated breast histopathological images. The training regimen emphasizes class balance and includes regularization techniques to improve generalizability. The integration of boundary supervision further fine-tunes the network’s sensitivity to tumor contours, resulting in improved segmentation accuracy in regions with structural complexity This segmentation approach, combining dense connectivity and a refined U-Net backbone, demonstrates enhanced capability in identifying and delineating complex cancer morphologies. It contributes meaningfully to the automated analysis of histopathological images, paving the way for more accurate, efficient, and interpretable breast cancer diagnosis. Training objective and boundary-aware optimization Let the input histopathological image be denoted as: X ∈ ℝH×W×3 (1) And the corresponding ground-truth tumor mask as: Y ∈ {0,1}H×W (2) The network predicts a probability map: Ŷ = fθ(X) (3) Where fθ represents the DenseUNet169 model parameterized by θ. 1. Pixel-wise binary cross-entropy loss This term ensures pixel-level classification consistency. 2. Dice loss for overlap optimization To improve segmentation overlap and address class imbalance, Dice loss is employed: Where ∈ is a small constant for numerical stability. 3. Combined objective function The final segmentation loss is defined as: Where α = 0.5 and β = 0.5. Overlapping tumor regions commonly present weak intensity gradients and partially fused boundaries, posing significant challenges for precise delineation. To address this, the network leverages combined Dice-BCE (Binary Cross-Entropy) optimization, which promotes sharper probability transitions at object interfaces while inherently penalizing boundary misalignment through its region-overlap formulation. Concurrently, gating mechanisms applied to skip connections selectively suppress irrelevant background activations and preserve high-frequency edge information propagated from encoder feature maps to the decoder. This selective feature modulation, together with robust spatial detail transmission across the encoder-decoder pathways, effectively reduces merged segmentation artifacts and enhances contour continuity, particularly for closely adjacent or overlapping tumor structures, without requiring explicit edge supervision or a dedicated boundary optimization head. Evaluation metrics for segmentation To comprehensively assess the performance of the proposed segmentation framework, a set of well-established quantitative evaluation metrics is employed. These metrics capture both the spatial and contextual fidelity of the predicted segmentation masks compared to the ground truth annotations. The selected measures: Algorithm: DenseUNet169 training for breast cancer segmentation (pseudocode) Input: Labeled dataset , where Xi ∈ ℝH×W×C is a mammographic image and Yi ∈ {0,1}H×W its binary mask Number of folds K = 5 (stratified cross-validation) Output: Optimal model parameters θ* 1: Preprocessing 2: Resize all Xi, Yi to 256 × 256 3: Apply on-the-fly augmentation T (.): random flips, rotations (±90°), and intensity shifts 4: Cross-validation 5: IoUbest ← 0 6: for k = 1 to K do 7: Dtrain, Dval ← StratifiedSpli (D, k) 8: θ ← InitializeDenseUNet169(backbone = DenseNet169ImageNet) 10: for epoch = 1 to Emax do 11: for batch (Xb, Yb) ~ Dtrain do 12: Ŷb ← fθ(T(Xb)) 13: ∇θ ← ∇θℒ(ŶbUb) 14: θ ← AdamUpdate(θ,∇θ,x03B1) 15: end for 16: IoUk←Evaluate(fθ,Dval) 17: if IoUk>IoUbest then 18: θ*←θ:IoUbest ← IoUk 19: end if 20: if early stopping criterion met then break 21: end for 22: end for 23: Return θ* Model complexity analysis The proposed DenseUNet169 architecture consists of a DenseNet169 encoder backbone integrated within a U-Net style decoder. The total number of model parameters is approximately 20.3 million, of which 20.1 million are trainable, and 0.2 million correspond to nontrainable parameters (primarily Batch Normalization statistics). For an input resolution of 256 × 256 pixels, the computational complexity of the network is approximately 9.2 Giga Floating-Point Operations (GFLOPs) per forward pass. GFLOPs serves as a standard metric to quantify the computational cost and inference efficiency of deep learning models. A complexity of 9.2 GFLOPs indicates a highly efficient architecture that strikes an optimal balance between representational power and computational overhead. This relatively low computational footprint ensures that the proposed model is not only capable of capturing complex spatial features but is also highly suitable for deployment in resource-constrained environments or real-time clinical applications. While the majority of the parameters originate from the DenseNet169 encoder to ensure robust feature extraction, the decoder modules are deliberately designed to be lightweight, contributing additional representational capacity without significantly inflating the overall GFLOPs. Given the relatively small dataset size (58 original images), the model capacity may increase the risk of overfitting despite ImageNet pretraining. Therefore, performance results should be interpreted as proof-of-concept rather than large-scale deployment validation. For comparison, the DenseUNet121 configuration contains approximately 13.2 million parameters, representing a 35% reduction in model capacity relative to DenseUNet169 (DenseNet121 backbone ≈ 8M, Full DenseUNet121 ≈ 12-14M). Transfer learning and fine-tuning strategy The DenseNet169 encoder was initialized with ImageNet pretrained weights to leverage general visual feature representations. To mitigate overfitting risk given the limited dataset size (58 original images), a staged fine-tuning strategy was adopted. During the initial training phase, the encoder layers were frozen, and only the decoder modules were optimized. This allowed the segmentation-specific layers to adapt without perturbing pre-trained feature representations. After convergence of the decoder, the upper dense blocks of the encoder were unfrozen and fine-tuned using a reduced learning rate (〈 = 1e-3), whereas earlier layers remained frozen. This partial fine-tuning approach balances adaptation to histopathological features with regularization against overfitting. All experiments were conducted using cross-validation, and early stopping based on validation IoU was applied to further reduce overfitting. Dataset description The breast cancer segmentation project utilized 58 histopathological images with expert annotations. This dataset included 58 hematoxylin and eosin (H and E) stained histopathology images of breast cancer cells, along with corresponding ground truth data. H and E staining is commonly used in routine histology to improve visibility, as most cells are naturally transparent with little intrinsic pigment. Specialized stains that selectively bind to specific components are used to highlight biological structures such as cells. The main difficulty in these images is segmenting cells for classification into benign and malignant categories. The images were saved at a resolution of 256 × 256 pixels in Portable Network Graphic format, with no image compression applied during the steps of capturing, annotating, and compiling datasets. Various augmentations were implemented, including right-left flips, up-down flips, image rotations (90° for k = 1 and 180° for k = 2), saturation adjustments, and central cropping, resulting in a total of 406 data points. Within this dataset, 81.2% was allocated for training purposes, whereas the remaining 18.8% was designated for testing. Download link: https://www.kaggle.com/datasets/andrewmvd/breast-cancer-cell-segmentation. In this scenario, segmentation refers to the process of outlining and defining the borders of individual cells within images. This step plays a vital role in tasks like measuring cell characteristics, studying cell shapes, and aiding in the diagnosis and treatment of breast cancer. Researchers and data experts can utilize this dataset to create and refine machine learning models, especially deep learning models, designed for automating cell segmentation. By using this dataset, they can experiment with different methods to accurately pinpoint and separate cancer cells in breast tissue images, potentially revolutionizing cancer diagnosis and treatment strategies. The dataset consists of images accompanied by masks that show the detailed pixel-level segmentation of cancer cells. Researchers can employ this data to train segmentation algorithms, assess model accuracy, and potentially make significant contributions to the progress of medical image analysis, especially in the realm of breast cancer research and diagnostics. It is important to acknowledge that the dataset consists of only 58 original images, which represents a relatively small sample size for training deep learning models. Although data augmentation was applied to increase sample variability, augmentation does not replace genuine inter-patient variability or biological diversity. Consequently, there remains a potential risk of overfitting, and the findings should be interpreted with caution. Larger, multi-center datasets would be required to fully assess generalization performance. Experiments Breast cancer segmentation Our primary goal is the segmentation of breast images, a critical task in medical image analysis. Table 3 acts as a valuable reference, outlining learning rates, batch sizes, optimizer choices, and other crucial parameters customized for the optimal convergence of the respective model. The result of evaluation metrics obtained from the proposed method (DenseUNet169) is shown in Table 4. Table 3Hyperparameter tuning for breast image segmentation using DenseUNet169 on the datasetNetwork nameEpochsk-foldBatch-sizeLearning rateOptimizerDenseUNet169250581e-3Adam Hyperparameter tuning for breast image segmentation using DenseUNet169 on the dataset Table 4Evaluation metrics of the proposed method (DenseUNet169)Evaluation metricsFold-averageIoU0.782Dice coefficient0.868Pixel accuracy0.848MHD5.329Surface dice overlap0.768Log loss333.07Jaccard score0.771IoU – Intersection over union; MHD – Modified Hausdorff distance Evaluation metrics of the proposed method (DenseUNet169) IoU – Intersection over union; MHD – Modified Hausdorff distance A fold-average IoU with a percentage value of about 78.2% of the predicted regions correctly overlapped with the actual regions proves that the performance is strong in terms of the accuracy of overlap. The model reveals a very high Dice Coefficient value of segmentation, 0.868, where its perfect score will be 1.00. This is sensitive to class imbalance and small regions; therefore, a high score is important. A pixel accuracy of 0.848 indicates that 84.8% of the total pixels in the dataset are correctly classified, thus proving the reliability of the general performance. The MHD gives the spatial similarity between predicted boundaries and ground truth boundaries. A lower value is better, as this means that the two are more aligned. A score of 5.329 reflects a moderately precise capture of fine boundary details. Surface Dice overlap is a variant of the Dice coefficient but focuses on boundary regions. The score of 76.8% proves that the model-predicted boundaries are much closer to the ground truth boundaries. Therefore, acceptable accuracy for delineating the contours of objects was confirmed. Log Loss (cross-entropy loss) was computed as the cumulative pixel-wise loss across the full image resolution rather than as a normalized per-pixel average. Given the 256 × 256 resolution (65,536 pixels per image), the absolute value of Log Loss appears numerically large. Therefore, Log Loss values reported in this study should be interpreted comparatively across models rather than in absolute magnitude terms. Future work may report normalized per-pixel cross-entropy for improved interpretability. The value of 333.07 shows where the model can be improved in terms of the confidence calibration of its predictions, since the smaller the log loss value, the better. The Jaccard score characterizes the overlap, like IoU, but considers penalties for over-prediction and under-prediction. This score of 0.771 also agrees with IoU, strengthening the fact that this model performs very well in segmenting regions quite accurately. To provide an estimate of performance variability, we computed the mean and standard deviation across the five cross-validation folds for each evaluation metric. Given the limited number of folds (n = 5), statistical comparisons between closely performing models should be interpreted with caution, as small numerical differences may fall within expected sampling variance. Figure 3 displays a visual comparison of the original image, the ground truth, and the predicted mask generated by the proposed method, namely DenseUNet169. Original, ground truth and predicted mask of the proposed method (DenseUNet169) Figure 4 illustrates the model’s training and validation performance across 250 epochs within a 5-fold cross-validation setup, organized into four panels: Validation Accuracy (Top-left), validation loss (top-right), training accuracy (bottom-left), training loss (bottom-right). Validation accuracy, validation loss, train accuracy, and train loss of the proposed method (DenseUNet169) Minor variations in accuracy are likely due to batch effects or learning rate dynamics. Training loss declines rapidly and stabilizes between 0.03 and 0.05, indicating effective fitting of the training data. Overall, the training curves indicate stable convergence behavior within the evaluated dataset. However, due to dataset size limitations, conclusions regarding broader generalization remain tentative. However, fluctuations in validation performance, especially in specific folds, suggest sensitivity to data variability. Incorporating strategies such as regularization, early stopping, or adaptive learning rate scheduling may help mitigate these effects and enhance training stability. An ablation comparison between frozen and fully fine-tuned encoder configurations showed marginal performance differences (<1% IoU), whereas full fine-tuning exhibited higher validation variance. Therefore, partial fine-tuning was adopted for the final configuration. Analysis and comparison results To evaluate the performance of our proposed method, we conducted a comparative analysis against six established approaches: ResNet50V2, ResNet152V2, DenseUNet121, DenseUNet169, DenseUNet201, and SwinU-Net [Table 5]. To ensure methodological consistency, identical evaluation parameters were applied to all methods. Quantitative outcomes are systematically documented in Table 6, enabling a comprehensive performance assessment. Table 5Hyperparameter tuning for breast image segmentation using U-Net with different backbone architectures on the datasetNetwork nameEpochk-foldBatch-sizeLearning rateOptimizerSwinU-Net250511e-3AdamDenseUNet121250581e-3AdamDenseUNet169250581e-3AdamDenseUNet201250581e-3AdamResU-Net50V2250581e-3AdamResU-Net101V2250581e-3AdamResU-Net152V2250581e-3Adam Hyperparameter tuning for breast image segmentation using U-Net with different backbone architectures on the dataset Table 6Comparison between various methodsMethod nameFold-Average Evaluation metricsResU-Net50V2ResU-Net101V2ResU-Net152V2DenseUNet121DenseUNet169DenseUNet201SwinU-NetIOU0.7230.7610.7130.7820.7820.7510.475Dice coefficient0.8190.8420.8240.8680.8680.8480.616Pixel accuracy0.7880.8230.8000.8330.8480.8140.551MHD5.7825.3266.2005.3435.3295.7327.966Surface dice overlap0.7190.7490.6940.7590.7680.7320.303Log loss460.82391.51424.56361.78333.07402.41974.95Jaccard score0.7130.7550.6980.7770.7710.7390.448IoU – Intersection over Union; MHD – Modified Hausdorff distance Comparison between various methods IoU – Intersection over Union; MHD – Modified Hausdorff distance Although DenseUNet169 achieved the highest mean IoU and Dice values, the performance differences relative to DenseUNet121 are small. Without formal statistical significance testing on larger datasets, these differences should not be interpreted as definitive superiority but rather as indicative trends within this experimental setting. Figure 5 shows a bar chart of IoU, comparing the performance of the proposed DenseUNet169 model with several existing segmentation methods. The chart highlights that the proposed framework achieves higher accuracy across various evaluation benchmarks. Intersection over union bar chart to compare the proposed method (DenseUNet169) with other similar methods Figure 6 illustrates the pixel accuracy comparison curve, indicating the performance of the proposed DenseUNet169 model in relation to other state-of-the-art segmentation techniques. Pixel accuracy comparison curve of the proposed method (DenseUNet169) with other similar methods Summary Optimal Choice: DenseUNet169 balances accuracy, boundary precision, and reliability, making it ideal for segmentation-critical applications. Figure 7 presents a visual comparison of segmentation performance: input histopathological image, ground truth, outputs from the proposed DenseUNet169, and results from other methods, showing a comparative example of segmentation predictions. Original image, ground truth, and predicted mask of the proposed method (DenseUNet169) against other methods Figure 8 shows comparative curves of training accuracy, training loss, validation accuracy, and validation loss, depicting the performance of the proposed DenseUNet169 alongside other competing deep learning models. Validation accuracy, validation loss, train accuracy, train loss of different methods. (a)ResU-Net50V2, (b) ResU-Net101V2, (c) ResU-Net152V2, (d) DenseUNet121, (e) DenseUNet201, (f) SwinU-Net Furthermore, the training dynamics of competing methods across 250 epochs, offering insights into convergence behavior and stability. This study evaluates the segmentation performance of U-Net equipped with various convolutional backbones, including ResNet50V2, ResNet101V2, ResNet152V2, DenseNet121, DenseNet169, and DenseNet201, alongside Swin-UNet, a distinct transformer-based architecture. Within the U-Net framework, the DenseNet169 backbone achieved the highest IoU on the evaluated dataset. Swin-UNet, by contrast, yielded lower IoU scores relative to the convolutional variants, suggesting that its architectural design may be less suited to the texture and morphological patterns characteristic of breast histopathological images in this context. Notably, DenseNet121 and DenseNet169 demonstrated highly comparable performance across most evaluation metrics; although DenseNet169 exhibited marginal gains in Pixel Accuracy and Log Loss, the magnitude of these differences was minimal. Consequently, these results should be interpreted as evidence of comparable efficacy among high-capacity DenseNet variants rather than as establishing a definitive performance hierarchy. Collectively, the empirical analysis supports the utility of the U-Net architecture with DenseNet backbones for breast cancer image segmentation and provides a foundation for further investigation in clinical and research applications. Through this comparative analysis, we aim to demonstrate the empirical behavior of different backbone configurations within a controlled experimental setting. The results are intended to inform architectural choices in future segmentation research rather than to establish clinical deployment readiness [Table 7]. Table 7Row-wise visual segmentation comparison of the proposed model (highlighted) compares with five utilized models on different input images from the datasetOriginalGround TruthResU-Net50V2ResU-Net101V2ResU-Net152V2DenseUNet121DenseUNet169DenseUNet201 Row-wise visual segmentation comparison of the proposed model (highlighted) compares with five utilized models on different input images from the dataset Generalization study While 5-fold cross-validation was employed to reduce bias in performance estimation, the limited size of the dataset constrains definitive conclusions regarding model generalization. The observed performance consistency across folds suggests stable training behavior within this dataset; however, external validation on independent and larger cohorts would be necessary to establish robust generalizability. You can see the results of this comparison in Tables 6 and 7. Effectiveness of DenseUNet169 components The DenseUNet169 model works so well because it brings together the strengths of three proven techniques. DenseNet169 helps the model learn detailed and complex patterns in medical images by passing features efficiently through many connected layers. U-Net adds a powerful structure that helps the model not only understand the overall context of the image but also pinpoint important areas more accurately, thanks to its encoder-decoder layout with skip connections. Discussion This study presents a structured integration and comparative evaluation of established deep learning architectures, DenseNet169 and U-Net, for breast cancer histopathological image segmentation. Rather than proposing a novel theoretical framework, the work focuses on examining how specific architectural design choices interact under controlled experimental conditions and how these interactions influence segmentation performance. The DenseNet169 encoder facilitates dense feature propagation and multi-scale contextual representation, which is particularly beneficial for capturing subtle morphological variations in histopathological imagery. When combined with the U-Net decoder structure, the architecture enables effective spatial localization of tumor regions through hierarchical feature reconstruction. Importantly, the performance improvements observed in this study should be interpreted as incremental architectural refinements rather than fundamental methodological advances. The primary contribution lies in the systematic benchmarking of these configurations under identical preprocessing, augmentation, and evaluation protocols. By reporting multiple quantitative metrics and maintaining consistent experimental settings, this study provides a structured empirical reference for future investigations in medical image segmentation. However, several limitations must be acknowledged. First, although 5-fold cross-validation was employed to improve robustness of performance estimation, the limited dataset size (58 original images) restricts statistical power. Differences observed between closely performing backbones, such as DenseNet121 and DenseNet169, may reflect expected experimental variability rather than statistically significant superiority. Larger-scale evaluations would be necessary to draw definitive comparative conclusions. Second, despite the use of ImageNet pre-trained weights to stabilize optimization, the relatively high model capacity (approximately 20 million parameters) introduces potential overfitting risk in small-data regimes. While cross-validation and regularization strategies were applied, the results should be interpreted as proof-of-concept evidence of technical feasibility rather than confirmation of broad generalizability. Finally, it is essential to emphasize that this study evaluates segmentation performance at a technical level and does not constitute a clinical validation study or a diagnostic decision-support assessment. No downstream clinical tasks were performed, and no claims regarding diagnostic reliability or real-world deployment readiness are made. Future work should involve evaluation on larger and more diverse datasets, as well as formal clinical validation, to assess translational applicability. In summary, the present work contributes a disciplined empirical analysis of architectural integration strategies for histopathological image segmentation. Its value lies in methodological transparency, controlled benchmarking, and cautious interpretation of results within the constraints of limited data availability. Conclusion This study presented a DenseUNet-based segmentation framework integrating a DenseNet169 encoder within a U-Net architecture and evaluated its performance under controlled experimental conditions using a publicly available histopathological dataset. The work was intentionally positioned as a technical investigation into architectural integration strategies for semantic segmentation, rather than as the introduction of a novel theoretical construct or a clinically validated diagnostic system. Under the current experimental setting, the DenseNet169 backbone demonstrated segmentation performance comparable to other high-capacity backbone configurations, with incremental improvements observed in selected evaluation metrics. These findings suggest that dense feature propagation combined with structured decoder reconstruction can provide stable and competitive performance in histopathological tumor segmentation tasks. However, it is essential to interpret these results within the context of dataset constraints. The study was conducted on a dataset comprising 58 original images. Although data augmentation and cross-validation were employed to mitigate overfitting and improve robustness of performance estimation, the limited sample size restricts the strength of conclusions regarding generalization capability and real-world applicability. The reported results should therefore be understood as proof-of-concept evidence illustrating the empirical behavior of the proposed architectural integration under small-data conditions. Future research should focus on evaluation across larger, multi-institutional, and more diverse datasets to assess external validity. Additional studies incorporating rigorous statistical testing, external validation cohorts, and clinical workflow assessment will be necessary to determine translational potential and practical deployment readiness. In summary, this work contributes a disciplined empirical analysis of architectural configurations for breast cancer histopathological image segmentation, emphasizing methodological transparency, controlled benchmarking, and cautious interpretation of findings within data-limited scenarios. Ethics statement This study exclusively used a publicly available dataset obtained from Kaggle (Breast Cancer Cell Segmentation dataset). The dataset contains anonymized histopathological images and does not include patient-identifiable information. No new patient data were collected, and no human subjects were directly involved. Therefore, institutional review board (IRB) approval was not required. Conflicts of interest There are no conflicts of interest.