Enhancing weakly supervised 3D medical image segmentation through probabilistic-aware learning Authors: Runmin Jiang, Zhaoxin Fan, Junhao Wu, Lenghan Zhu, Xin Huang, Tianyang Wang, Heng Huang, Min Xu Published: 2026 ABSTRACT Abstract3D medical image segmentation is a challenging task with crucial implications for disease diagnosis and treatment planning. Recent advances in deep learning have significantly enhanced fully supervised medical image segmentation. However, this approach heavily relies on labor-intensive and time-consuming fully annotated ground-truth labels, particularly for 3D volumes. To overcome this limitation, we propose a novel probabilistic-aware weakly supervised learning pipeline, specifically designed for 3D medical imaging. Our pipeline integrates three innovative components: a Probability-based Pseudo-Label Generation technique for synthesizing dense segmentation masks from sparse annotations, a Probabilistic Multi-head Self-Attention network for robust feature extraction within our Probabilistic Transformer Network, and a Probability-informed Segmentation Loss Function to enhance training with annotation confidence. Demonstrating significant advances, our approach not only rivals the performance of fully supervised methods but also surpasses existing weakly supervised methods in CT and MRI datasets, achieving up to 18.1% improvement in Dice scores for certain organs. The code is available at https://github.com/runminjiang/PW4MedSeg.Keywords: 3D medical imaging, Weakly supervised learning, Segmentation, Probabilistic-based framework Introduction Medical image segmentation is pivotal in refining healthcare systems for accurate disease diagnosis and strategic treatment planning, as it delineates anatomical structures across various imaging modalities, providing crucial information for healthcare professionals [1]. Deep learning techniques have significantly impacted this field, evidenced by advancements in traditional supervised learning methods, particularly in 2D or 3D ‘U-shaped’ encoder–decoder architectures like U-Net [2–6]. Despite their wide usage, these methods often require intensive manual annotation, a process that can be both time-consuming and resource-intensive [7]. To mitigate these challenges, researchers have explored various strategies such as data augmentation [8–10], transfer learning [11,12], and domain adaptation [13,14] to reduce reliance on extensive labeled data. Nevertheless, weakly supervised training methods, employing minimal annotations like points and scribbles for generating pseudo-labels, have gained increasing attention [15–17]. These approaches, while addressing the issue of manual annotation, predominantly focus on 2D image segmentation and often overlook the complexities of 3D weak annotation. This oversight can lead to significant information loss, as these methods tend to directly use sparse weak annotations during training. Furthermore, the confidence level of the annotator is frequently disregarded, omitting a vital aspect of the segmentation process. In response to these challenges, we propose a novel weakly supervised pipeline for 3D medical image segmentation, emphasizing probability integration throughout training and inference. Inspired by the uncertainty model [18], our approach transforms sparse 3D point labels into dense annotations through Probability-based Pseudo-Label Generation. We further introduce a Probabilistic Multi-head Self-Attention mechanism within our Probabilistic Transformer Network to address class variance and noise in pseudo-labels. Complementing this is our Probability-informed Segmentation Loss Function, which incorporates annotation confidence, aligning closer with true segmentation boundaries. This holistic approach, encompassing pseudo-label generation, network structure, and loss function, effectively utilizes dense weakly supervised signals and reduces bias in confidence allocation, facilitating efficient segmentation with minimal annotation costs. Solid experiments conducted on the authoritative BTCV, CHAOS, and BraTS 2021 datasets, representing CT and MRI images respectively, demonstrate the substantial efficacy of our approach. While our method consistently delivers exceptional results on the mentioned datasets, it also achieves results similar to or even surpasses one of the fully supervised tests. Further, we conducted dedicated ablation experiments on our framework’s three critical components, encompassing pseudo-label generation, network structure, and loss function. Remarkably, all these components yielded positive results, collectively contributing to the enhanced accuracy of segmentation within our framework. These findings underscore our method’s potential as a robust and versatile solution for medical image segmentation in weakly supervised settings. The main contributions of our approach can be summarized as follows: Probabilistic-aware Framework: We introduce a novel probabilistic-aware weakly supervised learning pipeline. Through a comprehensive series of tests, we demonstrate that our method not only significantly enhances performance compared to state-of-the-art weakly supervised methods but also achieves results comparable to fully supervised approaches, highlighting its substantial real-world applicability. Probabilistic-aware Framework: We introduce a novel probabilistic-aware weakly supervised learning pipeline. Through a comprehensive series of tests, we demonstrate that our method not only significantly enhances performance compared to state-of-the-art weakly supervised methods but also achieves results comparable to fully supervised approaches, highlighting its substantial real-world applicability. Probability-based Pseudo-Label Generation: Within the framework, we innovate by converting sparse 3D point labels into comprehensive dense annotations, leveraging principles from the uncertainty model. This innovative approach minimizes the typical information loss associated with weak labels and enhances segmentation accuracy. Additionally, we simulated the diversity of real-world raw data to test the practicality of our method and achieved promising results. Probability-based Pseudo-Label Generation: Within the framework, we innovate by converting sparse 3D point labels into comprehensive dense annotations, leveraging principles from the uncertainty model. This innovative approach minimizes the typical information loss associated with weak labels and enhances segmentation accuracy. Additionally, we simulated the diversity of real-world raw data to test the practicality of our method and achieved promising results. Probabilistic Multi-head Self-Attention (PMSA): A critical component of our probabilistic transformer network, it effectively addresses the inherent class variance and noise found in pseudo-labels. It plays a pivotal role in enhancing segmentation performance by capturing and utilizing the probabilistic distributions of input–output mappings. Probabilistic Multi-head Self-Attention (PMSA): A critical component of our probabilistic transformer network, it effectively addresses the inherent class variance and noise found in pseudo-labels. It plays a pivotal role in enhancing segmentation performance by capturing and utilizing the probabilistic distributions of input–output mappings. Probability-informed Segmentation Loss Function: To complement the framework, we introduce a novel loss function that incorporates the annotator’s confidence level. This loss function aligns the segmentation process more closely with actual boundaries and captures the probabilistic nature of the segmentation task. It also plays a crucial role in reducing the bias in confidence allocation during model training. Probability-informed Segmentation Loss Function: To complement the framework, we introduce a novel loss function that incorporates the annotator’s confidence level. This loss function aligns the segmentation process more closely with actual boundaries and captures the probabilistic nature of the segmentation task. It also plays a crucial role in reducing the bias in confidence allocation during model training. Related work Medical image segmentation This task is dedicated to extracting objects of interest from medical images obtained through modalities such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). Fully Convolutional Networks (FCN) [19] and U-Net [2] have significantly advanced 2D medical image segmentation. Adjustments to U-Net by Guan et al. [20] and Ibtehaz et al. [21] have been put forward to enhance the precision of segmentation. For 3D volumetric medical image segmentation, Cicek et al. [22] introduces a 3D U-Net that handles spatial information from 2D slices, while Milletari et al. [4] presents V-Net with improved feature extraction and reduced computational costs. Shabani et al. [23] proposed a 3D decoder with a novel triple-attention mechanism (weighted and learnable) to improve segmentation accuracy in complex microscopy volumes. Similarly, Kumari et al. [24] addressed annotation ambiguity in semi-supervised medical image segmentation by introducing a latent distribution learning framework with mutual guidance between branches. However, the primarily discussed techniques are fully supervised methods tailored for 2D medical image segmentation. In contrast, our paper emphasizes weakly supervised approaches for 3D medical image segmentation, aiming for more efficient annotation processes. Weakly supervised segmentation Weakly supervised learning reduces annotation cost by using sparse annotations instead of fully annotated masks. Weak labels such as bounding boxes [25,26], scribbles [17], and points [16] have been utilized. Zhang et al. [27] integrates point-level annotation and sequential patch learning for CT segmentation. Roth et al. [28] designs a point-based loss function with an attention mechanism. Zou et al. [29] proposes a well-calibrated pseudo-labeling strategy, while Liu et al. [30] introduces an informative selection strategy. Zhang et al. [31] addressed the issue of information loss caused by missing modalities by introducing a learnable sorting mechanism within a State Space Model (SSM) architecture, which is conceptually relevant to handling incomplete supervision signals in 3D segmentation. In contrast, our work proposes a “dense” weak annotation approach from a probabilistic perspective. Probabilistic modeling in deep learning Probabilistic modeling in deep learning handles uncertainty and provides confidence intervals. Shirakawa et al. [32] uses a Bernoulli distribution to generate network structures. Choi et al. [33] estimates a probabilistic distribution using mixture density networks for object detection. Zhang et al. [34] introduces Bayesian attention belief networks, while Guo et al. [35] scales dot-product attention as Gaussian distributions. Our method is the first probabilistic modeling approach for 3D medical image segmentation, incorporating probability in annotation, network structure, and gradient backpropagation, offering advantages for training and inference. Method Overview Medical image segmentation, which is typically referred to as semantic segmentation of medical images, aims to partition the image into different non-overlapping regions with unique semantic labels. Given an image I and the semantic classes C1,C2,…,Ck, the semantic segmentation process is performed by dividing I into D1,D2,…,Dk (i.e., the subregions), which satisfies: where k is a positive integer no less than 2, and all pixels in the region Di are labeled with Ci(i=1,2,…,k). In a weak supervision setting, the model is trained on a training set denoted as I1,L1*,I2,L2*,…,In,Ln*, where Ii(i=1,2,…,n) is the image and Li*(i=1,2,…,n) is the weak label. During inference, the model outputs dense segmentation of the input images. As shown in Fig. 1, we introduce a novel weakly supervised training pipeline for 3D medical image segmentation, taking probabilistic features of both annotation process and network training into consideration. We illustrate our pipeline in the following aspects: (1) A probability-based pseudo-label generation scheme for generating “dense” weak annotations. (2) A probabilistic Transformer network, whose key component is the proposed gaussian-based multi-head self-attention mechanism. (3) The probability-informed loss function. Overview of the proposed probability-aware learning framework. (a) Probability-based pseudo-label generation transforms sparse 3D point annotations into dense Gaussian confidence maps. (b) The probabilistic multi-headed self-attention (PMSA) module models the dependency scores α as a Gaussian distribution and samples them via the reparameterization trick. (c) The KL loss ℒkl constrains the learned attention distribution by anchoring it to a data-informed prior. (d) The probability-informed segmentation loss ℒSeg combines Dice loss with a confidence-weighted cross-entropy term. Probability-based pseudo-label generation Sparse labels annotation In this paper, we explore weakly supervised 3D medical image segmentation to lower annotation costs, choosing 3D points for sparse labeling. This approach helps in generating high-quality pseudo dense labels by instructing annotators to select random, evenly distributed 3D points on the organ’s surface. Experimentally, we simulate this process by eroding the ground-truth label with a 5 × 5 × 5 structuring element and then applying Farthest Point Sampling (FPS) to pick points within this eroded region. FPS ensures uniform spatial coverage and maximizes inter-point distances, effectively preventing point clustering and spatial bias. Compared with random sampling, it better approximates expert annotation patterns, leading to more consistent and representative pseudo-labels across anatomical structures. This method ensures an even distribution of points, effectively mimicking real annotation and creating pseudo sparse labels that closely represent the organ’s surface distribution. Pseudo-label generation After acquiring sparse labels, directly using them for supervision leads to substantial information loss and may inadequately train a 3D medical image segmentation network. To overcome this, we introduce a method for generating dense 3D labels. This method is based on the idea that annotated points and their vicinity possess confidence scores, decreasing with distance from the point. Specifically, for an annotated point xa,ya,za, we apply a Gaussian function to model confidence scores, peaking at the annotated point and diminishing with distance. The confidence score P(x,y,z) for any point (x,y,z) is defined as: This process, applied to all annotated points, generates dense 3D labels by summing up the label maps from all points and normalizing the intensity to [0, 1]. This probability-based pseudo-label generation scheme effectively transforms sparse annotations into informative dense labels, improving the training of the segmentation network. The entire Probability-based Pseudo-Label Generation pipeline is illustrated in Fig. 2, while the algorithm of target generation is described in Algorithm 1. The pipeline for Probability-based Pseudo-Label Generation. The points are randomly sampled within the target organ. The Probability-based Pseudo-Label is generated by assigning confidence using the sampled points. The core idea of this algorithm is to generate a three-dimensional Gaussian distribution based on the coordinates of each sampled point and to accumulate these distributions to form the final label map. This label map can subsequently be used for training medical image segmentation models. The variance σ2 influences the width of the generated Gaussian distribution, thereby altering the shape of the label map. We empirically fix σ2=1, which provides an optimal trade-off between spatial coverage and boundary precision across organs of varying sizes. To clarify, the generation of dense labels from sparse annotations consists of two explicit steps: (1) we randomly sample a subset of annotated voxels and apply Farthest Point Sampling (FPS) to obtain a representative set of annotated points that are spatially well distributed across the organ; (2) for each annotated point xa,ya,za in this representative set, we generate a 3D Gaussian confidence map over the entire volume, where the confidence at any voxel (x,y,z) decreases smoothly according to Eq. (2). These individual confidence maps are then accumulated, resulting in a dense confidence map where regions closer to multiple annotated points receive higher values. Finally, the accumulated map is normalized to [0, 1] to form the final pseudo-label used for training. Each voxel is therefore assigned a continuous confidence value rather than a binary label, reflecting its spatial proximity to the sparse annotations and effectively representing the organ’s surface distribution. Probabilistic transformer network Though the proposed pseudo-label in Section 3.2.2 can reflect the confidence level of the annotator, the within-class variance is high, due to the inherent morphological variation of human organs and the randomness of the point-sampling process. Therefore, a probabilistic model is expected to capture the complex distribution. Network architecture Our framework adopts the contracting-expanding schema characteristic of the UNETR architecture, as illustrated in Fig. 3. Initially, a 3D volume x∈RH×W×D×C, with dimensions (H,W,D) and C input channels, is segmented into non-overlapping uniform patches of dimensions (P,P,P). This segmentation transforms the volume into a sequence xv∈RN×P3⋅C by flattening these patches, where N=(H×W×D)/P3 denotes the sequence length. Subsequently, these patches are mapped into a K-dimensional embedding space via a linear layer. Furthermore, a 1D learnable positional embedding Epos∈RN×K is incorporated into the mapped patches. The process can be defined as follows: Network architecture overview. Here, E∈RP3⋅C×K represents the patch embedding projection. The features are then passed through a series of Probabilistic Transformer blocks, which consist of alternating layers of PMSA and MLP blocks. The equations for these blocks are as follows: Probabilistic multi-head self-attention Weakly supervised 3D segmentation inevitably suffers from annotation noise and uncertainty in pseudo-labels, which can cause unstable attention learning and degraded feature consistency. To mitigate this, we extend the standard Multi-head Self-Attention (MSA) mechanism into a probabilistic formulation that explicitly models uncertainty in the attention weights. In the standard Transformer, MSA captures dependencies between different positions of an input sequence using multiple attention heads. Given an input sequence z∈RN×K, where N denotes the sequence length and K denotes the feature dimension, each head generates attention weights to compute the contextual relationships among tokens. The calculation of MSA can be expressed as follows: Here, Q,K, and V are obtained by linearly transforming the input sequence z into query, key, and value representations, respectively. The attention weights are computed by taking the dot product of the query and key vectors, scaled by the square root of the key dimension dk. The softmax function is applied to obtain the final attention weights. Finally, the attention values are computed by multiplying the attention weights with the value vectors. However, the Probability-based Pseudo-Label suffers from large in-class variance caused by the randomness of the point-sampling process and the inherent diversity of human organ structure. To guide our model to capture the variance within the proposed pseudo-label and encode the input properly, inspired by Guo et al. [35], we introduce our Probabilistic Multi-head Self-Attention module. In a single SA head, we assume that the dependency score αij follows a Gaussian distribution: αij~𝒩μij,σij2, where the mean μij and the variance σij2 are calculated with qi and kj using a multilayer perceptron (MLP). In order to allow the parameters to be updated through backpropagation, we adopt reparameterization trick [36]: where ϵ is a random variable that follows a standard normal distribution. For other parameters in the model, we set them as deterministic, and denote them as Θ. We assume that the dependency scores within the same PMSA layer are independent of each other, while the dependency scores of deeper PMSA layer are dependent on those of former PMSA layers: where αl denotes the dependency scores of the PMSA layer of the lth transformer block. This Gaussian modeling enables the network to capture attention uncertainty rather than deterministic point estimates, aligning with Bayesian deep learning principles and improving boundary-level robustness. With PMSA, the distribution of the output segmentation map y′ given the input image X′ can be computed according to: However, due to the intractability of the integral in Eq. (9), we sample α from pα∣X′,𝚯 for M times to approximate the integral, in which every αij is sampled independently each time: where αm denotes the dependency scores sampled at the mth time, and y* is the final segmentation output. More details about the sampling of dependency scores and the proof of Eq. (9) can be found in Appendix. Probability-informed segmentation loss function As discussed in Section 3.2, the proposed pseudo-label is considered a probability map, where the intensity of each point represents the annotator’s confidence in classifying it as the target organ. Therefore, to enable our model to be aware of the underlying confidence within the pseudo-label, we introduce a loss function which is a combination of DICE loss and Probability-weighted Cross Entropy (PCE) loss. The intuition is that points with prior confidence greater than a certain threshold are considered as the foreground of the basic label map, while we weight the loss function with the prior confidence of the annotator since voxels with low confidence deserve lower loss weights. Given the output y* and pseudo-label map S, the segmentation loss is formulated as: where ST is the thresholded map of S with a threshold T (set as 0.5), and for each voxel in the segmentation map, ℒpce is formulated as: where si and pi are the confidence of the ith voxel in S and y*, respectively, and N denotes the number of voxels in the segmentation map. The PCE loss is then averaged over all voxels to obtain ℒPCE: Moreover, to align the model’s learned distribution of dependency scores with a realistic expectation of the data, we set our prior distributions based on empirical observations of the data and domain knowledge. The KL divergence loss is introduced to enforce this alignment: The overall probability-aware segmentation loss function is formulated as: where w is a balance term to prevent ℒKL from dominating the update of parameters through backpropagation. Theoretically, the probability-informed segmentation loss function allows for a more nuanced model training that accounts for both the fidelity to the annotated data and the uncertainty inherent in pseudo-labels, thus maintaining the integrity of the learning process even in less-than-ideal conditions. Training and inference strategy Binary classification. Unlike binary segmentation map which is common in most deep learning tasks, the proposed probability-based pseudo-label suffers from large data size and a single point has confidence scores for multiple organ classes, the sum of which might be greater than 1, which could be ambiguous. Thus, our model is trained to make inference of a single organ class, which is formulated as a binary classification task for each point. Sampling of dependency scores. During training, to accelerate the process, we sample the dependency scores for only one time, while during inference, the dependency scores are sampled for M times, and the final output is calculated as Eq. (10). Experiments Implementation details Our model inherits the contracting-expanding pattern of UNETR [37] but substitutes the encoder with a stack of Probabilistic Transformer blocks, each connected to the decoder through skip connections. We implemented our method using PyTorch [38] and MONAI.1 All experiments were conducted on a single NVIDIA RTXA5000 GPU with 24 GB of memory. We set the number of transformer encoders to 12 (L = 12) with an embedding size of 768 (K=768). Each patch has a resolution of 16 × 16 × 16. During training, we used the AdamW optimizer with an initial learning rate of 0.0001 and a batch size of 1. For inference, we employed a sliding window approach with a 50% overlap. The number of sampled points for different labels is proportional to the volume of the corresponding organ: 200 points for the spleen, 400 points for the liver, and 50 points for each of the right and left kidneys. We conducted experiments on two authoritative datasets: the BTCV dataset [39], which includes multi-organ abdominal 3D CT scans acquired during the portal venous contrast phase, and the CHAOS dataset [40], involving the segmentation of four abdominal organs from MRI datasets acquired with two different sequences (T1-DUAL and T2-SPIR). We assessed the effectiveness of our methodology using two prevalent metrics: the DICE score, where higher scores indicate better performance, and the 95% Hausdorff Distance (HD95), where lower values are preferable. Results In this section, we present the results of our method, comparing it with leading pseudo-label generation and fully supervised learning methods. Comparison with state-of-the-art pseudo-label generation methods. Table 1 shows the quantitative results for four organs: spleen, liver, left kidney, and right kidney. We categorize the weakly supervised methods into two types of supervision: point-supervised learning and scribble-supervised learning. Point-supervised methods use a few annotated points to guide the segmentation, such as sparse [22], convex [41] and ours. ADNet [42] and ALPNet [43] are examples of using scribbles-supervised learning to generate pseudo-labels. Table 1Comparison with SOTA weakly supervised methods on the BTCV and CHAOS datasets using the Dice metric. “LK” stands for “Left Kidney”, and “RK” stands for “Right Kidney”. Throughout this text, these abbreviations will consistently be used to refer to these anatomical structures. Bold denotes the highest values, while Underline indicates the second-highest values.DatasetMethodSpleenLiverLKRKBTCVSparse [22]0.55150.43030.25320.2703Convex [41]0.82320.62680.40370.3272ADNet [42]0.3860.73890.17510.2382ALPNet [43]0.7455 0.7916 0.594 0.535 Swin-UNETR [44]0.71840.70920.25220.3099DynUNet [45] 0.8345 0.71710.47050.3220 Ours 0.8279 0.8157 0.7599 0.7164 CHAOSSparse [22]0.36930.51970.56750.559Convex [41] 0.7256 0.7659 0.5640.7048ADNet [42]0.56410.71010.6530.7652ALPNet [43]0.730.7036 0.7755 0.7706 Swin-UNETR [44]0.67180.60150.44640.3086DynUNet [45]0.66100.57670.27590.2298 Ours 0.7402 0.8205 0.6662 0.7716 Comparison with SOTA weakly supervised methods on the BTCV and CHAOS datasets using the Dice metric. “LK” stands for “Left Kidney”, and “RK” stands for “Right Kidney”. Throughout this text, these abbreviations will consistently be used to refer to these anatomical structures. Bold denotes the highest values, while Underline indicates the second-highest values. To strengthen the comparison, we also additionally train two strong 3D architectures, Swin-UNETR [44] (Transformer-based) and DynUNet [45] (CNN-based), under the same weakly supervised pseudo-label pipeline across all datasets. Swin-UNETR is a 3D transformer-based segmentation network with a Swin Transformer encoder and U-Net-like decoder. DynUNet, on the other hand, is a flexible 3D U-Net architecture adapted from nnU-Net and commonly used for medical image segmentation. In our weakly supervised baseline, both models are trained using the same Gaussian pseudo labels generated from 200 point annotations, with dice_pce loss instead of full voxel-wise supervision, serving as the Transformer-based and CNN-based weakly supervised baseline respectively. From Table 1, we can observe that our method achieves the best performance on both datasets, except for the left kidney on CHAOS dataset, where ALPNet is slightly better. Our method improves the Dice scores by up to 18.1% and 10.2% over the point-supervised methods, 58.4% and 17.6% over the scribble-supervised methods, and a large margin over the weakly supervised method on both datasets. In conclusion, these results demonstrate the effectiveness of our method in producing high-quality segmentation results. The quantitative comparison in Fig. 4 further highlights our method’s proficiency in acquiring more accurate and comprehensive segments. Moreover, our method is highly efficient, converging within 400 epochs on a single NVIDIA RTX 4090Ti (0.29 s per epoch), demonstrating strong scalability and suitability for real-world clinical deployment. Qualitative comparison with weakly supervised methods. Comparison with SOTA fully supervised methods. To highlight the efficacy of our proposed approach, we compare our weakly supervised method with state-of-the-art fully supervised methods on the BTCV dataset, including TransUnet [1], SwinUnet [6], UCTransNet [5] and UNETR [37]. It is paramount to note that this comparison is inherently imbalanced, as our method operates on notably sparser original annotations compared to the comprehensive annotations utilized by the aforementioned fully supervised methods. Despite this inherent disparity, as delineated in Table 2, our method exhibits performances that are remarkably on par with, and in certain metrics, even surpass, those achieved by fully supervised counterparts. For instance, our method eclipses UCTransNet in spleen segmentation, showcasing the distinct advantages of our probabilistic weakly supervised approach. Table 2Comparison with SOTA fully supervised methods on BTCV dataset using the Dice and HD95 metric.MethodSpleenLiverLKRKDICE↑HD95↓DICE↑HD95↓DICE↑HD95↓DICE↑HD95↓FullyTransUnet [1]0.869730.140.934110.210.782228.190.843129.24SwinUnet [6]0.829427.380.912913.500.801763.740.801028.12UCTransNet [5]0.817629.220.897217.360.782222.770.780527.71UNETR [37]0.930418.650.901739.260.915951.000.89456.35WeaklyOurs 0.8279 63.09 0.8157 127.16 0.7599 135.88 0.7164 116.22 Comparison with SOTA fully supervised methods on BTCV dataset using the Dice and HD95 metric. To provide a more intuitive understanding of our method’s performance, we present qualitative segmentation results in Fig. 8, which demonstrates our model’s capability to accurately delineate organ boundaries across different anatomical structures. The visual results confirm that our weakly supervised approach can achieve segmentation quality comparable to fully supervised methods, while requiring significantly less annotation effort. In Fig. 5, we provide visual comparisons of our method against several fully supervised approaches, demonstrating its effectiveness in accurately segmenting key regions of interest. Despite limited supervision during training, our approach achieves performance comparable to fully supervised methods, underscoring its potential as a strong alternative. Qualitative results of segmentation prediction. Qualitative comparison with fully supervised methods. In conclusion, our method demonstrates its prowess and superior adaptability, ensuring commendable accuracy even with limited annotations and emphasizing its potential as a robust solution in the realm of medical image segmentation. Generalization to more modalities and challenge datasets. To further assess cross-modality generalization and robustness under heterogeneous imaging conditions, we evaluate our weakly supervised framework on the BraTS 2021 benchmark [55], which provides four structural MRI modalities per case—T1 (pre-contrast anatomy), T1-CE/T1Gd (post-contrast, highlighting enhancing tumor core), T2 (fluid-sensitive, showing edema), and T2-FLAIR (CSF-suppressed, emphasizing abnormal edema). These modalities exhibit distinct tissue contrasts and acquisition heterogeneity across institutions, making BraTS 2021 a rigorous testbed for weakly supervised segmentation. We compare our method against seven state-of-the-art weakly supervised approaches (Grad-CAM [46], ScoreCAM [47], LFI-CAM [48], LayerCAM [49], Swin-MIL [50], AME-CAM [51], and AC-CAM [52]), one unsupervised baseline (C&F [53] in its fully unsupervised configuration), and two few-shot variants—the same C&F framework trained with limited supervision, and the Optimized U-Net [54]. As summarized in Table 3, our method achieves the best Dice and IoU performance across all four modalities, demonstrating superior adaptability and robustness to modality shifts, signal heterogeneity, and diverse contrast patterns. Table 3Quantitative comparison on BraTS 2021 across four MRI modalities per case: T1 (pre-contrast, basic anatomy), T1-CE (post-contrast, enhancing lesions), T2 (edema and fluid sensitivity), and T2-FLAIR (CSF suppression highlighting abnormal edema). The substantial contrast and noise variations across modalities make this benchmark highly challenging for weakly supervised segmentation, thereby providing a rigorous test for cross-modality generalization. Dice and IoU scores (↑ higher is better) are reported for weakly supervised (WSSS), unsupervised (UL), and few-shot (FSL) methods. Our method achieves the best performance across all modalities, indicating strong robustness and adaptability to modality heterogeneity. Bold denotes the highest values, while Underline indicates the second-highest values.MethodT1T1-CET2T2-FLAIRDice↑IoU↑Dice↑IoU↑Dice↑IoU↑Dice↑IoU↑ Weakly Supervised Semantic Segmentation (WSSS) Grad-CAM [46]0.11 ± 0.090.06 ± 0.060.13 ± 0.090.07 ± 0.050.05 ± 0.060.03 ± 0.030.15 ± 0.080.08 ± 0.05ScoreCAM [47]0.30 ± 0.130.18 ± 0.090.40 ± 0.190.24 ± 0.140.53 ± 0.180.38 ± 0.170.43 ± 0.210.31 ± 0.18LFI-CAM [48]0.57 ± 0.170.41 ± 0.150.61 ± 0.130.46 ± 0.120.67 ± 0.170.52 ± 0.160.61 ± 0.170.46 ± 0.16LayerCAM [49]0.57 ± 0.170.41 ± 0.150.60 ± 0.190.45 ± 0.170.62 ± 0.180.47 ± 0.180.61 ± 0.220.46 ± 0.21Swin-MIL [50]0.48 ± 0.170.33 ± 0.150.47 ± 0.170.31 ± 0.140.49 ± 0.170.35 ± 0.160.47 ± 0.180.33 ± 0.17AME-CAM [51]0.63 ± 0.120.47 ± 0.120.70 ± 0.100.54 ± 0.110.71 ± 0.130.55 ± 0.120.78 ± 0.100.64 ± 0.12AC-CAM [52]0.69 ± 0.110.54 ± 0.100.75 ± 0.090.61 ± 0.100.76 ± 0.070.61 ± 0.080.89 ± 0.060.81 ± 0.07Swin-UNETR [44]0.62 ± 0.040.45 ± 0.040.64 ± 0.040.47 ± 0.060.64 ± 0.040.46 ± 0.040.68 ± 0.040.52 ± 0.04DynUNet [45]0.66 ± 0.040.49 ± 0.040.65 ± 0.040.49 ± 0.040.65 ± 0.040.45 ± 0.040.72 ± 0.040.56 ± 0.05Ours 0.87 ± 0.02 0.77 ± 0.04 0.88 ± 0.03 0.79 ± 0.04 0.90 ± 0.03 0.82 ± 0.05 0.92 ± 0.03 0.85 ± 0.06 Unsupervised Learning (UL) C&F [53]0.20 ± 0.080.11 ± 0.050.18 ± 0.080.11 ± 0.050.23 ± 0.090.13 ± 0.060.21 ± 0.090.12 ± 0.05 Few-Shot Learning (FSL) C&F [53]0.57 ± 0.200.43 ± 0.190.65 ± 0.100.48 ± 0.100.61 ± 0.220.48 ± 0.210.70 ± 0.160.57 ± 0.16Opt. U-Net [54] 0.84 ± 0.06 0.72 ± 0.09 0.85 ± 0.09 0.74 ± 0.09 0.88 ± 0.06 0.80 ± 0.10 0.91 ± 0.06 0.85 ± 0.09 Quantitative comparison on BraTS 2021 across four MRI modalities per case: T1 (pre-contrast, basic anatomy), T1-CE (post-contrast, enhancing lesions), T2 (edema and fluid sensitivity), and T2-FLAIR (CSF suppression highlighting abnormal edema). The substantial contrast and noise variations across modalities make this benchmark highly challenging for weakly supervised segmentation, thereby providing a rigorous test for cross-modality generalization. Dice and IoU scores (↑ higher is better) are reported for weakly supervised (WSSS), unsupervised (UL), and few-shot (FSL) methods. Our method achieves the best performance across all modalities, indicating strong robustness and adaptability to modality heterogeneity. Bold denotes the highest values, while Underline indicates the second-highest values. Ablation study In the ablation study section, we investigate the integration of a probabilistic mechanism across three key aspects of our framework: pseudo-label generation, network structure, and loss function. This section also covers additional ablation studies exploring parameters like sampled points and variance selection, offering insights into their impact on our pipeline’s performance. Effectiveness of probability-based pseudo-label generation. We assess the performance of a probabilistic mechanism using a semi-supervised technique with random point selection at various thresholds, aiming to mimic real-world scenarios with irregular point distributions or feature absence. Our goal is to ascertain if this approach yields consistent results across typical real-world data distributions, bridging the gap between laboratory and real-life settings, and ensuring effectiveness in both controlled and varied authentic environments. Table 4 shows marked improvements in segmentation accuracy, evident from significant Dice Score increases, such as from 0.5425 to 0.8279 for the spleen in the BTCV dataset and from 0.6544 to 0.8205 for the liver in the CHAOS dataset. The HD95 metrics also improved, although they are sensitive to extreme cases, particularly in complex anatomical regions like the left kidney in the CHAOS dataset. This sensitivity is a common issue for weakly supervised methods and is not unique to our approach. Table 4Robustness analysis under noisy or conflicting annotations. This experiment evaluates model stability under varying annotation quality using random point selection at different ratios (10%–90%), simulating real-world scenarios with inconsistent or incomplete supervision. The probabilistic framework exhibits smooth performance degradation without collapse, demonstrating strong noise tolerance and generalization stability. The study further validates the efficacy of probability-based pseudo-label generation in bridging laboratory insights with real-world implementations. Bold denotes the highest values, while Underline indicates the second-highest values.DatasetRatioDice↑HD95↓SpleenLiverLKRKSpleenLiverLKRKBTCV10%0.54250.6996 0.5797 0.5439 369.62310.38 148.71 122.25 30%0.5747 0.8136 0.57840.5259388.08 133.17 297.53202.5450%0.61780.78450.39710.3894352.41281.86347.38325.8370%0.59610.77150.56860.3799354.61308.32326.35341.9590% 0.6652 0.80600.39270.3889 172.67 144.89320.72324.61Ours 0.8279 0.8157 0.7599 0.7164 63.09 127.16 135.88 116.22 CHAOS10%0.38030.65440.45640.4110 36.40 49.93 51.88 36.50 30%0.41790.71990.58580.612579.1451.06132.2441.7150%0.40540.65040.58190.5916 32.91 49.03 101.2262.4470%0.49030.73900.58510.620056.0461.85103.4052.1290% 0.5455 0.7450 0.6287 0.6299 53.5561.06101.0542.06Ours 0.7402 0.8205 0.6662 0.7716 53.11 48.75 93.48 36.01 Robustness analysis under noisy or conflicting annotations. This experiment evaluates model stability under varying annotation quality using random point selection at different ratios (10%–90%), simulating real-world scenarios with inconsistent or incomplete supervision. The probabilistic framework exhibits smooth performance degradation without collapse, demonstrating strong noise tolerance and generalization stability. The study further validates the efficacy of probability-based pseudo-label generation in bridging laboratory insights with real-world implementations. Bold denotes the highest values, while Underline indicates the second-highest values. These results demonstrate the method’s adaptability to real-world irregularities and its robustness across different clinical scenarios. The consistent performance across various organs and datasets proves its real-world applicability, narrowing the gap between lab and real-life settings. Additionally, the enhanced segmentation accuracy has important clinical implications, affecting clinical decisions and patient care. Our experiments highlight our method’s technical and clinical potential, suggesting it for widespread use due to its reliability in diverse conditions. Figs. 6 and 7 show a comparison of the visualizations of pseudo-labels generated by our method under different thresholds of random original labels in two multi-modal datasets. Since our method is based on sparse label generation, the pseudo-labels produced by our approach tend to be much smoother compared to the ground truth. This is a shortcoming of our method, as well as a challenge that needs to be addressed in the field of weak supervision. Consequently, the boundary information is slightly inferior, which also explains the slightly higher HD95 distance. However, our method can still produce pseudo-labels with an overall contour even when sampling from randomly labeled points with very low thresholds. This makes our method more advantageous for datasets with very few or poorly labeled annotations. Visualization results on BTCV dataset showing the comparison of pseudo-labels generated under different thresholds of random original labels. Visualization results on CHAOS dataset demonstrating the effectiveness of our method across different threshold settings. Effectiveness of probabilistic transformer network structure. We investigate the impact of the probabilistic mechanism in the network architecture. Table 5 presents our experimental results, comparing the performance of the Self-Attention (SA) and the Multi-head Self-Attention (MSA) methods. These results indicate the heightened accuracy and reliability of PMSA in producing segmentation results that closely align with the actual anatomical structures, demonstrating the significance of considering probabilistic modeling in our transformer network. Table 5Quantitative results of ablation study for network structure and loss function. For these parts of comparison, only the methods corresponding to that part are varied, while the methods in the other parts are kept constant. “Ours” column represents the results of our complete approach.DatasetMetricOrganNetworkLoss functionOursMSASADICECEDCEFocalBTCVDICE↑Spleen0.8170.60130.35610.63410.76650.7853 0.8279 Liver0.77190.78650.59920.77670.77530.4025 0.8157 LK0.42520.42070.25990.39630.61120.5653 0.7599 RK0.54450.3540.34030.48390.5060.3675 0.7164 HD95↓Spleen285.41385.36373.83362.6316.56350.44 63.09 Liver306.89295.71321.67300.85 82.13 344.48127.16LK330.94323.53371.04341.61 95.04 107.78135.88RK135.78325.88342.03318.26120.08216.56 116.22 CHAOSDICE↑Spleen0.71450.74810.29990.50580.44470.3442 0.7402 Liver0.75420.77810.65960.60920.70330.379 0.8205 LK0.62790.64850.35370.50370.4460.6297 0.6662 RK0.62840.67160.59520.65140.69260.5693 0.7716 HD95↓Spleen74.09164.91189.34122.3753.4474.93 53.11 Liver93.5275.39180.3245.81 32.02 76.8648.75LK84.26106.01133.96106.7946.39 31.59 93.48RK114.1498.64168.1757.9237.7556.69 36.01 Quantitative results of ablation study for network structure and loss function. For these parts of comparison, only the methods corresponding to that part are varied, while the methods in the other parts are kept constant. “Ours” column represents the results of our complete approach. Effectiveness of probability-informed segmentation loss function. We examine the effectiveness of our designed loss function, as presented in Table 5. The conclusive results underscore the outstanding efficacy of our approach. Our approach consistently achieves higher effectiveness scores, demonstrating its ability to deliver more accurate and coherent segments. Compared to our approach, the existing non-probabilistic loss functions, specifically DICE, Cross-Entropy (CE), combined Dice-Cross-Entropy (DCE), and Focal demonstrate suboptimal performance, especially in segmenting the liver and both kidneys. These findings underscore the limitations of the existing loss functions and underscore the superiority of our designed probability-informed loss function in achieving improved 3D medical image segmentation results. Robustness under noisy/conflicting annotations. To evaluate the model’s tolerance to annotation noise, we randomly subsample annotation points at ratios of 10%–90%, simulating spatially inconsistent or boundary-misaligned supervision. As shown in Table 4, our method demonstrates smooth and non-monotonic performance degradation across both BTCV and CHAOS datasets, maintaining stable Dice scores even with 90% annotation removal and showing no catastrophic collapse. This behavior indicates that the proposed Gaussian confidence modeling provides soft spatial tolerance to annotation uncertainty, while the probability-weighted loss adaptively downweights unreliable regions, jointly ensuring robustness to sparse and noisy labels. Parameters exploration In our study, we conduct in-depth ablation analyses on two crucial parameters. Specifically, the number of sampled points, as detailed in our annotation strategy (Section 3.2), plays a pivotal role in pseudo-label generation. Additionally, the selection of variance σ2 in computing the KL loss, a critical hyperparameter, is meticulously evaluated to determine its influence on segmentation accuracy. Impact of the number of sampled points. As shown in Table 6, we conducted a detailed investigation into the impact of the number of sampled points on segmentation performance. This experiment was designed to keep all other parameters constant, varying only the number of sampled points used in pseudo-label generation. The results from this comparative study provide intriguing insights into the optimal balancing of sampled points for effective segmentation. Table 6Illustration of the number of sampled points. Bold denotes the highest values, while Underline indicates the second-highest values.DatasetnDice↑HD95↓SpleenLiverLKRKSpleenLiverLKRKBTCV500.63920.10810.62760.5678336.25 127.16 135.88 292.41100 0.8001 0.7307 0.6366 0.4772 174.41 295.54 197.66 312.451500.54620.71640.3856 0.5756 348.11304.69333.50 183.65 200 0.8279 0.8157 0.7599 0.7164 63.08 265.79 266.17 116.22 CHAOS200 0.7356 0.8205 0.6662 0.6530 77.31 74.05 93.48 94.50 2500.69820.76980.56870.7716148.26103.01107.56 36.01 300 0.7402 0.7970 0.5978 0.6667 130.12 48.79 103.09112.753500.67110.78720.55160.6627 53.11 48.75 101.34 99.07 Illustration of the number of sampled points. Bold denotes the highest values, while Underline indicates the second-highest values. A key observation from the BTCV and CHAOS datasets is the non-linear relationship between the number of sampled points and the segmentation performance. Specifically, we noticed that both extremely low and high numbers of sampled points do not necessarily yield the best segmentation results. For instance, in the BTCV dataset, a sample size of 50 points resulted in suboptimal Dice Scores and HD95 metrics across all organs, suggesting inadequate coverage of the organ’s semantic space. Conversely, at 200 points, while some organs like the spleen and liver showed marked improvements in Dice Scores and reduced HD95 values, indicating better segmentation, others like the left kidney did not show a consistent pattern of improvement. This phenomenon can be attributed to the fact that a very low number of points may fail to provide sufficient information to cover the entire organ, leading to poor segmentation performance. On the other hand, a very high number of points could introduce noise or outliers, potentially hampering the segmentation accuracy. These additional points, rather than contributing useful information, might act as anomalies, detracting from the model’s ability to accurately delineate organ boundaries. Our results highlight the importance of an optimal range of sampled points in our probabilistic pseudo-label generation, striking a balance between comprehensive feature representation and minimizing noise. This balance is crucial for enhancing segmentation accuracy while efficiently utilizing limited annotation resources, proving especially beneficial in scenarios where full supervision is not feasible. The findings underscore the significance of carefully selecting the number of sampled points to achieve effective annotation efficiency and robust segmentation outcomes. Comparison of the selection of variance. When calculating the KL loss in our probability-informed segmentation loss function, the variance σ2 serves as a hyperparameter that needs to be manually determined. To ensure experimental rigor, we investigate the effects of different variances on segmentation accuracy. Table 7 illustrates that the choice of variance in the KL loss significantly influences the final results. We observe that when setting the variance to 1, our model achieves the highest DICE score and the lowest HD95 value. Based on these empirical findings, we establish σ2 as 1 in our method. By conducting this analysis, we enhance the reliability of our experimental setup and demonstrate the importance of selecting an appropriate variance for the KL loss. The chosen value of σ2 contributes to optimizing the segmentation performance and ensures the robustness of our method. Table 7Illustration of the number of selection of variance. Bold denotes the highest values, while Underline indicates the second-highest values.Dataset σ2 Dice Score↑HD95↓SpleenLiverLKRKSpleenLiverLKRKBTCV0.10.51040.74780.5474 0.5192 394.87 299.2 329.39 273.61 1 0.8279 0.8157 0.7599 0.7164 63.09 265.79 266.17 116.22 10 0.5754 0.7691 0.6103 0.3468 388.11 312.36 238.93 323.79CHAOS0.1 0.5472 0.5707 0.36690.3813 56.26 51.06 31.72 56.14 1 0.7402 0.8205 0.6662 0.7716 53.11 48.75 93.48 36.01 100.53990.5413 0.4058 0.6664 164.89 45.93 68.55 112.75 Illustration of the number of selection of variance. Bold denotes the highest values, while Underline indicates the second-highest values. Conclusion In this work, we present a novel probability-based framework for 3D medical image segmentation under weak supervision, showing marked accuracy improvements over state-of-the-art methods. This approach not only pioneers new and efficient segmentation strategies but also ensures precision with minimal annotations, promising significant real-world applicability. Acknowledgments This work was supported in part by U.S. NIH R01GM134020 and R35GM158094, NSF MCB-2205148, IIS-2211597, DBI-2238093, DBI-2422619, and UPMC Enterprises. We thank Haili Wang, Hoang-Thien Nguyen, and Xingjian Li for their help with the paper revision and discussions. Sampling of the dependency scores The dependency scores of a deeper PMSA layer are mutually independent but only rely on the former layers. Therefore, we have: where the mean μij and the variance σij2 are calculated with qi and kj using a multilayer perceptron (MLP), and αl denotes the dependency scores of the PMSA layer of the lth transformer block. Θ denotes all the deterministic parameters in the model. In this way, given the input image X′, the distribution of the output segmentation map y′ is calculated as: During inference, to approximate the integral of Eq. (9), we sample all the dependency scores independently for M times and calculate the final segmentation output y* where αm denotes the sampled dependency scores: Given that the dependency scores within the same PMSA layer are independent of each other, and the dependency scores of deeper PMSA layer are dependent on those of former PMSA layers, as indicated by Eq. (A.2), Eq. (9) could be written as: where M=∏l=1LMl, which we empirically set as 6 in our experiments. Footnotes Dataset link: https://github.com/runminjiang/PW4MedSeg https://monai.io/ CRediT authorship contribution statement Runmin Jiang: Writing – review & editing, Writing – original draft, Visualization, Validation, Methodology, Investigation, Formal analysis, Conceptualization. Zhaoxin Fan: Writing – review & editing, Supervision, Project administration, Methodology, Data curation, Conceptualization. Junhao Wu: Writing – review & editing, Visualization, Validation. Lenghan Zhu: Writing – review & editing, Visualization, Validation. Xin Huang: Supervision, Project administration. Tianyang Wang: Supervision, Project administration. Heng Huang: Supervision, Project administration. Min Xu: Writing – review & editing, Supervision, Resources, Project administration, Methodology, Funding acquisition, Data curation, Conceptualization. Declaration of Generative AI and AI-assisted technologies in the writing process During the preparation of this work the authors used ChatGPT in order to improve the language and clarity of the manuscript. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Data availability I have shared the link (https://github.com/runminjiang/PW4MedSeg) to the dataset and code on GitHub, as mentioned in the abstract. References Associated Data Data Availability Statement I have shared the link (https://github.com/runminjiang/PW4MedSeg) to the dataset and code on GitHub, as mentioned in the abstract.