A Multiple Sclerosis MRI Dataset with Tri-Mask Annotations for Lesion Segmentation Authors: Mahdi Bashiri Bawil, Mousa Shamsi, Aydin Ghalehasadi, Ali Fahmi Jafargholkhanloo, Abolhassan Shakeri Bavil Published: 2026 ABSTRACT Multiple sclerosis (MS) lesion segmentation from MRI is essential for diagnosis, monitoring, and treatment evaluation, yet existing datasets exhibit limited geographic diversity, scanner vendor bias, and inability to distinguish normal from pathological white matter hyperintensities. We present MS3SEG, a publicly available dataset comprising 100 MS patients from an Iranian cohort acquired on a 1.5 T Toshiba scanner, featuring multi-sequence MRI (T1-weighted, T2-weighted, T2-FLAIR in axial and sagittal planes) with novel tri-mask annotations. Expert annotators systematically delineated three classes on axial T2-FLAIR images: ventricles, normal white matter hyperintensities (age-related or CSF-contaminated regions), and abnormal white matter hyperintensities (MS lesions), incorporating automated quality control to minimize annotation errors. This tri-mask framework addresses a critical clinical challenge where benign periventricular hyperintensities are frequently misclassified as pathological lesions. Data are provided in multiple formats: raw DICOM, preprocessed NIfTI with co-registration and brain extraction, ground-truth masks, and visualization overlays. Baseline validation using U-Net, U-Net++, UNETR, and Swin UNETR across 5-fold cross-validation demonstrated dataset quality, with U-Net achieving Dice coefficients of 0.7469 for binary lesion segmentation and 0.6686 for multi-class abnormal hyperintensity segmentation. MS3SEG enables research on anatomically-aware segmentation, algorithm robustness to real-world clinical acquisition variations, and clinically relevant normal-versus-abnormal hyperintensity distinction, complementing existing 3D volumetric MS datasets. Background & Summary Multiple sclerosis (MS) is the most common chronic inflammatory, demyelinating, and neurodegenerative disease of the central nervous system in young adults, characterized by heterogeneous clinical presentations and disease courses1. The diagnosis of MS relies on demonstrating dissemination of lesions in space and time, as defined by the McDonald 2017 Criteria, with magnetic resonance imaging (MRI) serving as a fundamental paraclinical tool2,3. Recent updates to these diagnostic criteria, including the 2024 revisions, continue to refine the role of neuroimaging biomarkers while emphasizing the need for careful differential diagnosis to minimize the risk of misdiagnosis4,5. The disease manifests through focal demyelinating lesions in the brain and spinal cord, with pathological processes evolving along a continuum from acute inflammatory injury to progressive neurodegeneration6. Manual delineation of these lesions from MRI scans is time-consuming, subject to inter- and intra-observer variability, and increasingly impractical given the growing patient populations requiring monitoring7,8. This challenge has driven substantial interest in developing automated segmentation methods to support both clinical decision-making and research applications. MRI sequences play complementary roles in MS lesion detection and characterization. T2-weighted fluid-attenuated inversion recovery (T2-FLAIR) imaging is the primary sequence for identifying white matter hyperintensities (WMH) associated with MS lesions, as it suppresses cerebrospinal fluid (CSF) signal while enhancing lesion conspicuity9,10. T1-weighted (T1-w) imaging provides information about tissue integrity and can identify hypointense lesions (“black holes”) that correlate with axonal loss, while T2-weighted (T2-w) sequences offer additional sensitivity for detecting subtle abnormalities11. The axial plane is standard in clinical practice due to its efficiency and anatomical coverage, though sagittal T2-FLAIR sequences provide complementary information for periventricular and corpus callosum lesions9. Contemporary MRI protocols for MS emphasize standardized acquisition parameters and interpretation by trained neuroradiologists to ensure consistent lesion assessment across different scanners and institutions10. Despite progress in automated segmentation, current datasets present significant limitations. While several publicly available datasets exist, including the ISBI 2015 longitudinal challenge dataset12 and the MICCAI 2016 challenge dataset13, these resources are constrained by limited sample sizes and geographic homogeneity, with data predominantly derived from Western populations and scanners manufactured by GE or Siemens. A critical gap exists in the differentiation of normal versus abnormal WMH—not all hyperintensities visible on T2-FLAIR imaging represent pathological MS lesions14,15. Age-related WMH and CSF contamination artifacts, particularly adjacent to ventricles, can mimic MS lesions, potentially leading to false-positive segmentations by automated algorithms16. Additionally, existing datasets lack annotations of anatomical structures such as ventricles, which provide crucial spatial context for distinguishing pathological lesions from benign periventricular hyperintensities17. The absence of datasets from diverse scanner manufacturers limits the ability to evaluate generalizability of segmentation methods across different acquisition protocols and hardware platforms18. These limitations underscore the need for datasets that incorporate multi-class annotations, geographic diversity, and scanner variety to support the development of robust, clinically applicable automated segmentation tools. The objectives of this work are to:Introduce a new large-scale, publicly available MRI dataset of 100 MS patients with tri-mask annotations distinguishing ventricles, normal WMH, and abnormal WMH (MS lesions), acquired from a Middle Eastern populationProvide detailed analysis of the systematic annotation methodology, incorporating both expert delineation and automated quality control, to ensure reproducibility and high annotation qualityEstablish baseline segmentation performance using state-of-the-art deep learning architectures (U-Net19, U-Net++20, UNETR21, and Swin UNETR22) through rigorous k-fold cross-validationAddress geographic diversity by representing an understudied Middle Eastern (Iranian) population, where MS phenotypes may differ from Western cohorts23,24Provide scanner diversity through Toshiba platform data, under-represented in existing MS datasets regardless of acquisition type, contributing to vendor diversity within the clinical 2D imaging domainEnable research on anatomically-aware segmentation algorithms that leverage contextual information from ventricle locations and normal-appearing white matter Introduce a new large-scale, publicly available MRI dataset of 100 MS patients with tri-mask annotations distinguishing ventricles, normal WMH, and abnormal WMH (MS lesions), acquired from a Middle Eastern population Introduce a new large-scale, publicly available MRI dataset of 100 MS patients with tri-mask annotations distinguishing ventricles, normal WMH, and abnormal WMH (MS lesions), acquired from a Middle Eastern population Provide detailed analysis of the systematic annotation methodology, incorporating both expert delineation and automated quality control, to ensure reproducibility and high annotation quality Provide detailed analysis of the systematic annotation methodology, incorporating both expert delineation and automated quality control, to ensure reproducibility and high annotation quality Establish baseline segmentation performance using state-of-the-art deep learning architectures (U-Net19, U-Net++20, UNETR21, and Swin UNETR22) through rigorous k-fold cross-validation Establish baseline segmentation performance using state-of-the-art deep learning architectures (U-Net19, U-Net++20, UNETR21, and Swin UNETR22) through rigorous k-fold cross-validation Address geographic diversity by representing an understudied Middle Eastern (Iranian) population, where MS phenotypes may differ from Western cohorts23,24 Address geographic diversity by representing an understudied Middle Eastern (Iranian) population, where MS phenotypes may differ from Western cohorts23,24 Provide scanner diversity through Toshiba platform data, under-represented in existing MS datasets regardless of acquisition type, contributing to vendor diversity within the clinical 2D imaging domain Provide scanner diversity through Toshiba platform data, under-represented in existing MS datasets regardless of acquisition type, contributing to vendor diversity within the clinical 2D imaging domain Enable research on anatomically-aware segmentation algorithms that leverage contextual information from ventricle locations and normal-appearing white matter Enable research on anatomically-aware segmentation algorithms that leverage contextual information from ventricle locations and normal-appearing white matter The unique contributions of this dataset are multi-faceted. First, the three-class annotation system represents an unprecedented approach to MS lesion segmentation datasets. By distinguishing between ventricles, normal WMH, and abnormal WMH (true MS lesions), this dataset addresses a fundamental clinical reality: not all hyperintensities on T2-FLAIR imaging are pathological14,25. Age-related vascular changes produce WMH that are clinically insignificant, while CSF partial volume effects create periventricular hyperintensities that must be distinguished from genuine demyelinating lesions16,26. The inclusion of ventricle masks provides essential anatomical context, as periventricular MS lesions follow characteristic spatial patterns (“Dawson’s fingers”) that differ from artifact distributions. This tri-mask approach enables the development of segmentation algorithms that can make clinically meaningful distinctions—a capability that previous binary segmentation datasets cannot support. Such anatomically-informed segmentation has the potential to reduce false positives, improve diagnostic specificity, and better align automated methods with expert radiological interpretation. Second, this dataset provides representation of a Middle Eastern population, specifically from Iran. MS epidemiology varies substantially by geography, with the Middle East and North Africa (MENA) region showing rising prevalence and distinct disease characteristics23,24. Studies from the MENACTRIMS registry indicate that MS patients in the MENA region present at earlier ages and may experience more aggressive disease courses compared to Western populations23. The global prevalence of MS has increased across all regions, yet most publicly available datasets derive from Western populations, limiting understanding of disease manifestations across diverse genetic and environmental contexts27. Given that MS susceptibility is influenced by complex interactions between genetic factors, environmental exposures, and geographic latitude, datasets from underrepresented populations are essential for developing globally applicable automated analysis tools. Third, all data in this dataset were acquired using a Toshiba MRI scanner, in contrast to the GE and Siemens scanners that dominate existing MS datasets. Scanner manufacturer introduces systematic variations in image characteristics due to differences in radiofrequency coil design, gradient performance, and reconstruction algorithms18. Recent studies have demonstrated that deep learning models trained on data from one scanner manufacturer exhibit substantially degraded performance when tested on data from different manufacturers, even when imaging the same pathology18. By providing data from a Toshiba platform, this dataset enables explicit testing of cross-vendor generalizability for segmentation algorithms—a critical validation step before clinical deployment. This contribution addresses a known limitation in the field and supports the development of more robust, vendor-agnostic automated methods. We present this dataset in multiple formats to maximize utility: raw DICOM data as acquired; registered volumes with T1-w and T2-w co-registered to T2-FLAIR space; preprocessed data with standardized 256 × 256 dimensions and brain extraction; ground truth tri-mask annotations; paired visualizations concatenating T2-FLAIR images with corresponding masks; and RGB-coded overlays for intuitive quality assessment. Each patient includes T1-w, T2-w, and T2-FLAIR sequences in the axial plane, with additional sagittal T2-FLAIR. We present baseline segmentation results using four modern architectures evaluated in 5-fold cross-validation across multiple task formulations: three-class segmentation, binary ventricle segmentation, and binary lesion segmentation. This comprehensive resource addresses critical gaps in annotation granularity, geographic representation, and scanner diversity, providing the research community with a valuable tool for developing and validating next-generation MS lesion segmentation methods. Materials & Methods Data collection Patient cohort This dataset comprises MRI scans from 100 MS patients recruited retrospectively from the radiology archives of Golgasht Medical Imaging Center, Tabriz, Iran, between 2020 and 2021. All included patients had clinically confirmed MS diagnoses according to the McDonald 2017 Criteria2, which require demonstration of dissemination in space (DIS) and dissemination in time (DIT) through clinical and/or imaging evidence. The MRI scans included in this dataset represent single-timepoint acquisitions performed as part of routine clinical care. Due to the retrospective study design, detailed clinical metadata such as disease subtype (relapsing-remitting vs. progressive), disability scores, and treatment history were not systematically available. The imaging data directly support evaluation of the DIS component of the McDonald 2017 Criteria, which our tri-mask annotations are specifically designed to facilitate by distinguishing pathological MS lesions from benign periventricular hyperintensities. Inclusion criteria required: (1) definite MS diagnosis according to McDonald 2017 Criteria, (2) age between 18 and 59 years, and (3) adequate image quality for manual annotation. Exclusion criteria included: (1) concurrent neurological disorders that could confound MS lesion identification, (2) contraindications to MRI, and (3) poor image quality due to motion artifacts or technical failures. Each patient contributed a single imaging session. Demographic characteristics are summarized in Table 1.Table 1Patient Demographics.CohortPatientsFemale:MaleAge Mean ± SDAge RangeAge MedianAge IQRAll1002.85:135.5 ± 8.918-5535.029.0-42.0Female74N/A35.5 ± 8.818-5535.029.0-42.0Male26N/A35.6 ± 9.318-5534.029.0-41.8 Table 1Patient Demographics.CohortPatientsFemale:MaleAge Mean ± SDAge RangeAge MedianAge IQRAll1002.85:135.5 ± 8.918-5535.029.0-42.0Female74N/A35.5 ± 8.818-5535.029.0-42.0Male26N/A35.6 ± 9.318-5534.029.0-41.8 Patient Demographics. Ethics and data sharing This study received approval from the Tabriz University of Medical Sciences Research Ethics Committee (IRB approval number: IR.TBZMED.REC.1402.902). Written informed consent was obtained from all patients for the use of their anonymized imaging data for research purposes. To protect participant information, comprehensive anonymization was performed at the earliest stage of data processing. Patient identifiers were systematically removed from DICOM headers using the pydicom Python library, including name, date of birth, and all other personal health information fields. Each patient was assigned a numerical identifier (001–100) with no linkage to original clinical records maintained in the public dataset. All data handling and sharing procedures comply with institutional privacy policies and international research ethics standards. The dataset is released under a Creative Commons Attribution 4.0 International (CC-BY-4.0) license, permitting use, sharing, and adaptation with appropriate attribution. MRI Acquisition All scans were acquired on a single 1.5 Tesla Toshiba Vantage MRI scanner (Canon Medical Systems, Japan) using the institutional standard clinical imaging protocol. Each patient underwent a single scanning session that included T1-w, T2-w, and T2-FLAIR sequences in the axial plane, plus an additional T2-FLAIR sequence in the sagittal plane. Acquisition parameters were maintained nearly identical across patients, with variation typical of clinical practice arising from patient-specific field-of-view adjustments. This is a real-world clinical dataset collected retrospectively from routine diagnostic imaging, not a prospectively designed research cohort with strict protocol control. Original data were acquired in DICOM format. Detailed acquisition parameters are provided in Table 2. While parameters such as TE, TI, FA, and matrix size are identical across all 100 patients for the T2-w and T2-FLAIR sequences, certain parameters — specifically TR, slice thickness, inter-slice gap, and T1-w matrix size — vary continuously or in two discrete groups as a result of routine operator adjustments. Table 3 summarizes the exact distribution of these variable parameters across all 100 patients, demonstrating that the dataset maintains sufficient representation in every subgroup for the intended segmentation research.Table 2MRI Acquisition Parameters. TR, TE, and TI stand for time repetition, time echo, and time inversion.ModalityPlaneMatrix SizePixel SpacingSlice ThicknessSlice GapTRTETIFlip AngleT1-wAxial554 ± 60.5 × 554 ± 60.50.42 ± 0.045.86 ± 0.491.39 ± 0.42534.8 ± 11.715.0 ± 0.0N/A70.0 ± 0.0T2-wAxial704 ± 0.0 × 704 ± 0.00.33 ± 0.005.86 ± 0.491.39 ± 0.424729.7 ± 186.4105.0 ± 0.0N/A90.0 ± 0.0T2-FLAIRAxial256 ± 0.0 × 256 ± 0.00.90 ± 0.005.73 ± 0.521.27 ± 0.229915.0 ± 171.4100.0 ± 0.02500.0 ± 0.090.0 ± 0.0T2-FLAIRSagittal256 ± 0.0 × 257 ± 5.30.98 ± 0.015.15 ± 0.461.6 ± 0.79920.0 ± 174.7100.0 ± 0.02500.0 ± 0.090.0 ± 0.0Table 3Distribution of Key Variable Acquisition Parameters Across All 100 Patients.SequenceVariable ParameterSubgroups/RangeModal Value (n)Mean ± SDPatients with Non-Modal ValuesT1-w (axial)Matrix size512 × 512 (n = 67); 640 × 640 (n = 33)512 × 512 (n = 67)N/A (discrete)33 patients (640 × 640)T1-w (axial)TR484–550 ms540 ms (n = 72)534.8 ± 11.7 ms28 patientsT1-w (axial)ST5.0–6.9 mm—5.86 ± 0.49 mmN/A (continuous)T2-w (axial)TR3852–4800 ms4800 ms (n = 75)4729.7 ± 186.4 ms25 patientsT2-w (axial)ST5.0–6.9 mm—5.86 ± 0.49 mmN/A (continuous)T2-FLAIR (axial)TR9400–10000 ms10000 ms (n = 78)9915.0 ± 171.4 ms22 patientsT2-FLAIR (axial)ST4.7–6.9 mm—5.73 ± 0.52 mmN/A (continuous)T2-FLAIR (axial)Inter-slice gap (SBS - ST)0.7–2.1 mm1.2 mm (n = 69)1.27 ± 0.22 mm31 patientsT2-FLAIR (sagittal)TR9200–10100 ms10000 ms (n = 76)9920.0 ± 174.7 ms24 patientsT2-FLAIR (sagittal)ST4.0–6.7 mm—5.15 ± 0.46 mmN/A (continuous)T2-FLAIR (sagittal)Inter-slice gap (SBS - ST)0.4–2.4 mm2.1 mm (n = 54)1.60 ± 0.70 mm46 patientsParameters not listed were identical across all patients. Modal value = most frequently occurring value. ST = slice thickness; TR = repetition time; SBS = spacing between slices. Table 2MRI Acquisition Parameters. TR, TE, and TI stand for time repetition, time echo, and time inversion.ModalityPlaneMatrix SizePixel SpacingSlice ThicknessSlice GapTRTETIFlip AngleT1-wAxial554 ± 60.5 × 554 ± 60.50.42 ± 0.045.86 ± 0.491.39 ± 0.42534.8 ± 11.715.0 ± 0.0N/A70.0 ± 0.0T2-wAxial704 ± 0.0 × 704 ± 0.00.33 ± 0.005.86 ± 0.491.39 ± 0.424729.7 ± 186.4105.0 ± 0.0N/A90.0 ± 0.0T2-FLAIRAxial256 ± 0.0 × 256 ± 0.00.90 ± 0.005.73 ± 0.521.27 ± 0.229915.0 ± 171.4100.0 ± 0.02500.0 ± 0.090.0 ± 0.0T2-FLAIRSagittal256 ± 0.0 × 257 ± 5.30.98 ± 0.015.15 ± 0.461.6 ± 0.79920.0 ± 174.7100.0 ± 0.02500.0 ± 0.090.0 ± 0.0 MRI Acquisition Parameters. TR, TE, and TI stand for time repetition, time echo, and time inversion. Table 3Distribution of Key Variable Acquisition Parameters Across All 100 Patients.SequenceVariable ParameterSubgroups/RangeModal Value (n)Mean ± SDPatients with Non-Modal ValuesT1-w (axial)Matrix size512 × 512 (n = 67); 640 × 640 (n = 33)512 × 512 (n = 67)N/A (discrete)33 patients (640 × 640)T1-w (axial)TR484–550 ms540 ms (n = 72)534.8 ± 11.7 ms28 patientsT1-w (axial)ST5.0–6.9 mm—5.86 ± 0.49 mmN/A (continuous)T2-w (axial)TR3852–4800 ms4800 ms (n = 75)4729.7 ± 186.4 ms25 patientsT2-w (axial)ST5.0–6.9 mm—5.86 ± 0.49 mmN/A (continuous)T2-FLAIR (axial)TR9400–10000 ms10000 ms (n = 78)9915.0 ± 171.4 ms22 patientsT2-FLAIR (axial)ST4.7–6.9 mm—5.73 ± 0.52 mmN/A (continuous)T2-FLAIR (axial)Inter-slice gap (SBS - ST)0.7–2.1 mm1.2 mm (n = 69)1.27 ± 0.22 mm31 patientsT2-FLAIR (sagittal)TR9200–10100 ms10000 ms (n = 76)9920.0 ± 174.7 ms24 patientsT2-FLAIR (sagittal)ST4.0–6.7 mm—5.15 ± 0.46 mmN/A (continuous)T2-FLAIR (sagittal)Inter-slice gap (SBS - ST)0.4–2.4 mm2.1 mm (n = 54)1.60 ± 0.70 mm46 patientsParameters not listed were identical across all patients. Modal value = most frequently occurring value. ST = slice thickness; TR = repetition time; SBS = spacing between slices. Distribution of Key Variable Acquisition Parameters Across All 100 Patients. Parameters not listed were identical across all patients. Modal value = most frequently occurring value. ST = slice thickness; TR = repetition time; SBS = spacing between slices. Preprocessing Raw DICOM data underwent a comprehensive five-step preprocessing pipeline to generate standardized, analysis-ready volumes. All preprocessing was implemented using Python with FSL library integration and custom scripts. Step 1: anonymization Patient identifiers were removed from DICOM headers using Python with the pydicom library. All metadata fields containing personal health information were systematically stripped to ensure compliance with privacy regulations. Step 2: Format conversion DICOM files were converted to NIfTI format using dcm2niix, a widely adopted neuroimaging tool that preserves geometric information while enabling compatibility with standard analysis pipelines. Step 3: Co-registration Since manual annotations were performed on axial T2-FLAIR images, all other sequences required registration to T2-FLAIR space to ensure spatial alignment. T1-w and T2-w volumes were co-registered to the axial T2-FLAIR volume using FLIRT (FMRIB’s Linear Image Registration Tool) from FSL28. Registration employed an affine transformation with 12 degrees of freedom and mutual information as the cost function. This approach ensured that all modalities shared identical spatial coordinates with the annotation masks, enabling multi-modal analysis. Step 4: Standardization To facilitate batch processing and deep learning applications, all registered images were resized to a consistent 256 × 256 pixel matrix through cropping and padding operations while preserving the original pixel spacing. This standardization reduces memory requirements and enables efficient model training without resampling-induced information loss. Step 5: Brain extraction Non-brain tissue was removed using the Brain Extraction Tool (BET) from FSL29. BET was applied exclusively to the T2-FLAIR volume; the resulting brain mask was directly applicable to the co-registered T1-w and T2-w volumes due to spatial alignment established in Step 3. This approach ensures consistent brain boundaries across all modalities while minimizing processing time. Both raw DICOM data and preprocessed NIfTI volumes are provided in the public repository to support diverse research applications. Annotation methodology The dataset underwent meticulous manual annotation to generate ground-truth tri-masks distinguishing ventricles, normal white matter hyperintensities (WMH), and abnormal WMH (MS lesions) on axial T2-FLAIR images for all 100 patients. Manual segmentation of MS lesions presents significant challenges due to task complexity and natural variability in expert interpretations7. To address these challenges, we developed a systematic annotation protocol incorporating expert consensus review and automated quality control. Annotation team and training The annotation team consisted of one junior expert (trained medical imaging specialist) and one senior expert (experienced neuroradiologist with MS specialization). Prior to initiating annotations, multiple training sessions were conducted to establish standardized criteria for all three annotation classes. These sessions involved detailed examination of representative cases, discussion of challenging scenarios, and development of consistent decision rules for ambiguous regions. Annotation classes Three distinct classes were defined based on imaging characteristics and clinical significance, as shown in Table 4.Table 4Annotation Class Definitions.ClassLabelDefinitionIntensity on FLAIRClinical SignificanceBackground0Non-brain and normal brain parenchyma (all tissue except the three annotated classes)VariableNoneVentricles1CSF-filled ventricular spacesVery low (hypointense)Spatial context for lesion localization; source of CSF artifactsNormal WMH2Age-related hyperintensities or normal CSF-contaminated regions in periventricular areasHigh (hyperintense) but non-pathologicalShould be excluded from MS lesion burden calculations to avoid overestimationAbnormal WMH3MS-related demyelinating lesions with characteristic imaging and spatial featuresHigh (hyperintense) with MS-characteristic patternsTrue disease markers requiring clinical monitoring and therapeutic decision-making Table 4Annotation Class Definitions.ClassLabelDefinitionIntensity on FLAIRClinical SignificanceBackground0Non-brain and normal brain parenchyma (all tissue except the three annotated classes)VariableNoneVentricles1CSF-filled ventricular spacesVery low (hypointense)Spatial context for lesion localization; source of CSF artifactsNormal WMH2Age-related hyperintensities or normal CSF-contaminated regions in periventricular areasHigh (hyperintense) but non-pathologicalShould be excluded from MS lesion burden calculations to avoid overestimationAbnormal WMH3MS-related demyelinating lesions with characteristic imaging and spatial featuresHigh (hyperintense) with MS-characteristic patternsTrue disease markers requiring clinical monitoring and therapeutic decision-making Annotation Class Definitions. Annotation workflow Due to the labor-intensive nature of annotating approximately 2000 axial T2-FLAIR slices across 100 patients (mean: 20 slices per patient), we implemented a structured four-phase workflow designed to maximize annotation quality while maintaining efficiency. Phase 1: Primary annotation The junior expert performed initial annotations using a custom-developed Python-based graphical user interface (GUI). For each patient, T1-w and T2-w sequences were displayed alongside T2-FLAIR images to enable cross-modal verification (Fig. 1). The annotator proceeded slice-by-slice through the axial stack, systematically identifying and delineating structures in the following order: (1) ventricles, identified by characteristic very low T2-FLAIR intensity and anatomical location; (2) abnormal WMH (MS lesions), identified based on high T2-FLAIR intensity combined with MS-characteristic features including periventricular or juxtacortical location, ovoid morphology perpendicular to ventricles (Dawson’s fingers), minimum diameter of 3 mm, and confirmation of hyperintensity on T2-w images9; and (3) normal WMH, defined as hyperintensities lacking MS-specific characteristics, typically appearing as thin periventricular rims in the anterior and posterior horns of lateral ventricles where CSF contamination is physiologically normal, or as bilateral symmetric deep white matter changes consistent with age-related small vessel disease14.Fig. 1Custom Python-based graphical user interface for tri-mask annotation showing 2 × 2 synchronized viewer layout. Top panels display T1-weighted (left) and T2-weighted (right) reference images. Bottom panels show T2-FLAIR image without (left) and with (right) annotation overlay. Color scheme: blue (ventricles), green (normal white matter hyperintensities), red (abnormal white matter hyperintensities/MS lesions). The interface enables intuitive annotation with cross-modal verification and real-time visualization. Custom Python-based graphical user interface for tri-mask annotation showing 2 × 2 synchronized viewer layout. Top panels display T1-weighted (left) and T2-weighted (right) reference images. Bottom panels show T2-FLAIR image without (left) and with (right) annotation overlay. Color scheme: blue (ventricles), green (normal white matter hyperintensities), red (abnormal white matter hyperintensities/MS lesions). The interface enables intuitive annotation with cross-modal verification and real-time visualization. Phase 2: Expert review The senior expert independently reviewed each annotated case, overlaying the tri-masks on T2-FLAIR images with adjustable transparency to assess boundary accuracy and class assignments. Discrepancies were discussed, and modifications were implemented immediately. Feedback was systematically documented to refine the junior expert’s understanding of annotation criteria. This iterative review process continued until consensus was achieved for all cases. Phase 3: Statistical quality control An innovative automated quality control mechanism was implemented using statistical image processing. For each annotated region, pixel intensity distributions were analyzed to verify consistency with expected class characteristics. Specifically, ventricle annotations were validated to contain predominantly very low intensity pixels (defined relative to the image-specific intensity distribution), while both normal and abnormal WMH annotations were required to contain predominantly high intensity pixels. Annotations failing these intensity-based criteria were automatically flagged for manual re-examination. Additionally, morphological operations (opening and closing) were applied to refine annotation boundaries and eliminate isolated pixels likely representing annotation errors. Phase 4: Final consensus review Both experts conducted a final review of the statistically processed annotations. Cases flagged by automated quality control or exhibiting unusual patterns were re-examined, and manual corrections were applied where necessary. This hybrid approach—combining expert knowledge with objective statistical validation—was designed to minimize inter- and intra-rater variability while maintaining high annotation fidelity. Custom annotation tool A Python-based GUI was developed specifically for this tri-mask annotation task, prioritizing efficiency and usability. The interface architecture comprised four synchronized viewers arranged in a 2 × 2 grid: top-left displaying the T1-w slice, top-right displaying the T2-w slice, bottom-left displaying the T2-FLAIR slice for annotation, and bottom-right displaying the T2-FLAIR slice with real-time mask overlays in distinct colors (red: abnormal WMH, green: normal WMH, blue: ventricles). Key features included: (1) intuitive mouse-based controls (left-click: paint, right-click: erase); (2) independent visibility toggles for mask layers; (3) class selection and adjustable overlay transparency via script configuration; and (4) simple navigation tabs for moving between slices and patients. This minimalistic design required minimal training and reduced cognitive load during the time-consuming annotation process. A representative screenshot illustrating the four-panel layout is shown in Fig. 1. Annotation statistics Annotation statistics across the cohort are summarized in Table 5. Abnormal WMH (MS lesions) exhibited substantial inter-patient variability (mean count: 47.62 ± 33.71 connected components per patient; mean load: 1135.31 ± 1125.38 mm²), reflecting the heterogeneous nature of MS disease burden. Male patients demonstrated higher mean lesion counts (58.35 ± 37.45) and total lesion area (1452.47 ± 1065.46 mm²) compared to female patients (43.85 ± 31.71 lesions; 1023.87 ± 1131.54 mm²). Normal WMH were consistently present across nearly all patients (mean count: 68.28 ± 14.29; mean load: 579.44 ± 260.69 mm²) with comparatively lower variability, consistent with age-appropriate background hyperintensities. Ventricular volumes showed expected variation (mean: 3551.34 ± 1471.85 mm²) with males exhibiting larger volumes than females (4117.76 ± 1878.70 mm² vs. 3352.32 ± 1254.89 mm²), consistent with documented sex differences in brain morphometry.Table 5Annotation Statistics Across Patient Cohort.CohortabWMHnWMHVentricleCountLoadCountLoadCountLoadAll47.62 ± 33.711135.31 ± 1125.38, 703.2868.28 ± 14.29579.44 ± 260.69524.2732.69 ± 5.453551.34 ± 1471.85, 3279.28Female43.85 ± 31.711023.87 ± 1131.54, 628.9767.05 ± 14.83518.88 ± 217.35483.5732.23 ± 5.423352.32 ± 1254.89, 3140.37Male58.35 ± 37.451452.47 ± 1065.46, 1019.1871.77 ± 12.21751.79 ± 299.05663.3934.00 ± 5.404117.76 ± 1878.70, 3671.53Note: abWMH = abnormal white matter hyperintensities (MS lesions); nWMH = normal white matter hyperintensities; Count = number of connected components (mean ± SD); Load = total area in mm² (mean ± SD, and median). Table 5Annotation Statistics Across Patient Cohort.CohortabWMHnWMHVentricleCountLoadCountLoadCountLoadAll47.62 ± 33.711135.31 ± 1125.38, 703.2868.28 ± 14.29579.44 ± 260.69524.2732.69 ± 5.453551.34 ± 1471.85, 3279.28Female43.85 ± 31.711023.87 ± 1131.54, 628.9767.05 ± 14.83518.88 ± 217.35483.5732.23 ± 5.423352.32 ± 1254.89, 3140.37Male58.35 ± 37.451452.47 ± 1065.46, 1019.1871.77 ± 12.21751.79 ± 299.05663.3934.00 ± 5.404117.76 ± 1878.70, 3671.53Note: abWMH = abnormal white matter hyperintensities (MS lesions); nWMH = normal white matter hyperintensities; Count = number of connected components (mean ± SD); Load = total area in mm² (mean ± SD, and median). Annotation Statistics Across Patient Cohort. 579.44 ± 260.69 524.27 518.88 ± 217.35 483.57 751.79 ± 299.05 663.39 Note: abWMH = abnormal white matter hyperintensities (MS lesions); nWMH = normal white matter hyperintensities; Count = number of connected components (mean ± SD); Load = total area in mm² (mean ± SD, and median). Evaluation metrics Segmentation performance was evaluated using three complementary metrics that capture different aspects of segmentation quality12,25. All metrics were calculated volumetrically across the entire volume. The Dice Similarity Coefficient (DSC) quantifies volumetric overlap between predicted segmentation A and ground truth B:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${DSC}=\frac{2\left|A\cap B\right|}{\left|A\right|+\left|B\right|}$$\end{document}DSC=2A∩BA+Bwhere values range from 0 (no overlap) to 1 (perfect agreement). The Intersection over Union (IoU), also known as the Jaccard index, provides an alternative overlap measure that penalizes both false positives and false negatives:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${IoU}=\frac{\left|A\cap B\right|}{\left|A\cup B\right|}$$\end{document}IoU=A∩BA∪B The 95th percentile Hausdorff Distance (HD95) quantifies boundary delineation accuracy by measuring the distance between segmentation surfaces after excluding the 5% worst outliers:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${HD}95(A,B)=\max \left({h}_{95}\left(A,B\right),{h}_{95}\left(B,A\right)\right)$$\end{document}HD95(A,B)=maxh95A,B,h95B,Awhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${h}_{95}\left(A,{B}\right)$$\end{document}h95A, B is the 95th percentile of the distances from points in boundary \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A$$\end{document}A to their closest points in boundary \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$B$$\end{document}B. HD95 is measured in millimeters, with lower values indicating better boundary agreement. Together, these metrics provide comprehensive evaluation of both volumetric accuracy (DSC, IoU) and boundary precision (HD95), enabling thorough assessment of segmentation algorithm performance. Ethics approval and informed consent This retrospective study was approved by the Tabriz University of Medical Sciences Research Ethics Committee (IRB approval number: IR.TBZMED.REC.1402.902). Written informed consent was obtained from all patients for the use of their anonymized MRI data for research and publication purposes. All procedures performed were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments. Data Records The MS3SEG dataset is permanently archived at Figshare30 under a CC-BY-4.0 license. The repository contains four primary data categories: raw DICOM files preserving complete acquisition metadata, preprocessed NIfTI volumes suitable for immediate analysis, expert-annotated ground truth masks, and RGB-coded visualization overlays. Data are organized hierarchically by patient identifier with subdirectories for imaging sequences and processing stages. Raw DICOM data total approximately 4GB across 100 patients (30-40MB per patient), organized by patient and sequence (T1-w, T2-w, axial T2-FLAIR, sagittal T2-FLAIR). NIfTI data total approximately 2GB (2-12MB per patient) with spatial dimensions of 256 × 256 × N slices (N typically 18–24) and voxel spacing of 0.90 × 0.90 × 5.73 mm³ for the reference T2-FLAIR sequence. Images use 16-bit signed integers; masks use 8-bit unsigned integers. All sequences are co-registered to axial T2-FLAIR space with guaranteed voxel-wise correspondence. Ground truth annotations encode four classes as integer labels: background (0), ventricles (1), normal white matter hyperintensities (2), and abnormal white matter hyperintensities/MS lesions (3). Annotation files are 12-15KB per patient in compressed NIfTI format. Visualization overlays are provided as PNG files (40-60KB per patient) with color coding: blue (ventricles), green (normal WMH), red (abnormal WMH). Patient metadata are provided in JSON format including anonymized identifiers, age, sex, acquisition year, and cohort assignment. Cross-validation splits specify patient assignments for five folds plus the 20-patient held-out test set. File naming follows the convention patientID_[modality]_[processing].nii.gz for preprocessed data and patientID_[mask name].nii.gz for annotations. NIfTI headers preserve geometric information including orientation matrices, voxel spacing, and coordinate systems. Technical Validation The goal of this section is to demonstrate that MS3SEG, similar to other technically validated datasets in the field12,13,25, is suitable for segmentation tasks. The baseline results obtained with modern deep learning architectures demonstrate the dataset’s quality and establish performance benchmarks for future research. Experimental setup To ensure robust evaluation and address generalization challenges common in medical imaging due to limited data availability, we employed a 5-fold cross-validation strategy. The 100 patients were randomly divided into 80 patients for cross-validation (distributed across training and validation sets within each fold) and 20 patients held out as an independent test set for future evaluation. Within each fold, 64 patients were assigned to training and 16 to validation, ensuring no patient appears in both subsets. This configuration prevents data leakage and enables fair assessment of model generalization to unseen patients. The cross-validation split is detailed in Table 6.Table 6Cross-Validation Split Configuration.FoldTraining Patients (n=64 per fold)Validation Patients (n=16 per fold)Test Patients (n=20, constant)1003, 004, 006, 008-010, 012, 014, 016, 018, 020-022, 024, 026-028, 030, 033, 035-037, 039, 043, 047, 049-053, 055-058, 060-062, 064, 065-070, 072, 073, 075, 076, 079, 080, 082, 083, 085, 086, 089, 093-100007, 015, 017, 025, 029, 038, 041, 042, 044, 048, 059, 063, 087, 088, 090, 092001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0912002-004, 006, 007, 009, 010, 012, 015-018, 020, 021, 024-030, 033, 035-039, 041-044, 048-051, 055, 057-060, 062-067, 072, 073, 075, 076, 082, 085-090, 092, 093, 095-099008, 014, 022, 047, 052, 053, 056, 061, 068, 069, 070, 079, 080, 083, 094, 100001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0913002-004, 007, 008, 014-017, 020-022, 025, 027-030, 033, 036-044, 047, 048, 050, 052, 053, 055-057, 059-063, 065, 066, 068-070, 072, 073, 075, 076, 079, 080, 082, 083, 087-090, 092-096, 098-100006, 009, 010, 012, 018, 024, 026, 035, 049, 051, 058, 064, 067, 085, 086, 097001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0914002-004, 006-010, 012, 014, 015, 017, 018, 020, 022, 024-030, 035, 038, 039, 041, 042, 044, 047-053, 056, 058, 059, 061, 063-070, 075, 076, 079, 080, 082, 083, 085-090, 092, 094, 095, 097, 100016, 021, 033, 036, 037, 043, 055, 057, 060, 062, 072, 073, 093, 096, 098, 099001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0915002, 006-010, 012, 014-018, 021, 022, 024-026, 029, 033, 035-038, 041-044, 047-049, 051-053, 055-064, 067-070, 072, 073, 079, 080, 083, 085-088, 090, 093, 094, 096-100003, 004, 020, 027, 028, 030, 039, 050, 065, 066, 075, 076, 082, 089, 092, 095001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 091 Table 6Cross-Validation Split Configuration.FoldTraining Patients (n=64 per fold)Validation Patients (n=16 per fold)Test Patients (n=20, constant)1003, 004, 006, 008-010, 012, 014, 016, 018, 020-022, 024, 026-028, 030, 033, 035-037, 039, 043, 047, 049-053, 055-058, 060-062, 064, 065-070, 072, 073, 075, 076, 079, 080, 082, 083, 085, 086, 089, 093-100007, 015, 017, 025, 029, 038, 041, 042, 044, 048, 059, 063, 087, 088, 090, 092001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0912002-004, 006, 007, 009, 010, 012, 015-018, 020, 021, 024-030, 033, 035-039, 041-044, 048-051, 055, 057-060, 062-067, 072, 073, 075, 076, 082, 085-090, 092, 093, 095-099008, 014, 022, 047, 052, 053, 056, 061, 068, 069, 070, 079, 080, 083, 094, 100001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0913002-004, 007, 008, 014-017, 020-022, 025, 027-030, 033, 036-044, 047, 048, 050, 052, 053, 055-057, 059-063, 065, 066, 068-070, 072, 073, 075, 076, 079, 080, 082, 083, 087-090, 092-096, 098-100006, 009, 010, 012, 018, 024, 026, 035, 049, 051, 058, 064, 067, 085, 086, 097001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0914002-004, 006-010, 012, 014, 015, 017, 018, 020, 022, 024-030, 035, 038, 039, 041, 042, 044, 047-053, 056, 058, 059, 061, 063-070, 075, 076, 079, 080, 082, 083, 085-090, 092, 094, 095, 097, 100016, 021, 033, 036, 037, 043, 055, 057, 060, 062, 072, 073, 093, 096, 098, 099001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 0915002, 006-010, 012, 014-018, 021, 022, 024-026, 029, 033, 035-038, 041-044, 047-049, 051-053, 055-064, 067-070, 072, 073, 079, 080, 083, 085-088, 090, 093, 094, 096-100003, 004, 020, 027, 028, 030, 039, 050, 065, 066, 075, 076, 082, 089, 092, 095001, 005, 011, 013, 019, 023, 031, 032, 034, 040, 045, 046, 054, 071, 074, 077, 078, 081, 084, 091 Cross-Validation Split Configuration. Baseline methods and implementation Four representative architectures spanning both convolutional and transformer-based paradigms were evaluated to establish performance baselines. U-Net19 served as the foundational encoder-decoder architecture with proven effectiveness across medical image segmentation tasks. U-Net++20 extended this framework through nested skip pathways designed to reduce the semantic gap between encoder and decoder features. UNETR21 introduced transformer-based encoding, coupling a vision transformer encoder with a convolutional decoder to capture long-range spatial dependencies. Swin UNETR22 refined this approach using hierarchical shifted-window attention mechanisms, representing current state-of-the-art performance on several medical imaging benchmarks. All models were implemented in TensorFlow with consistent hyperparameters: Adam optimizer with learning rate 0.0001, batch size 4, unified focal loss function to address inherent class imbalance, and training duration of 30 epochs. Training was conducted on NVIDIA RTX 3060 GPU hardware. Input consisted exclusively of single-channel T2-FLAIR images, with no data augmentation applied, to establish clean baseline performance metrics that reflect model capacity rather than augmentation strategies. Experimental scenarios Three complementary experimental scenarios were designed to comprehensively evaluate dataset utility. Scenario 1 (Multi-Class Tri-Mask Segmentation) assessed simultaneous four-class segmentation (background, ventricles, normal WMH, abnormal WMH), directly testing the utility and difficulty of the novel tri-mask annotation framework. Scenario 2 (Binary Ventricle Segmentation) isolated ventricle segmentation as a standalone task, establishing baseline performance for this well-defined anatomical structure and demonstrating dataset utility for anatomical segmentation beyond lesion analysis. Scenario 3 (Binary Lesion Segmentation) focused on traditional MS lesion segmentation (abnormal WMH versus background), enabling direct performance comparison with existing MS datasets and literature. This multi-scenario design enables both validation of the complete tri-mask annotation system and benchmarking against conventional binary segmentation paradigms prevalent in existing research. Quantitative results Results for multi-class segmentation (Scenario 1) are presented in Table 7. For ventricle segmentation, U-Net achieved the highest performance metrics (DSC: 0.8897 ± 0.0049; IoU: 0.8026 ± 0.0056; HD95: 0.76 ± 0.01 mm), marginally surpassed by U-Net++ in DSC (0.8934 ± 0.0018) but with comparable IoU and slightly larger boundary errors. Transformer-based architectures demonstrated notably lower performance: UNETR achieved DSC of 0.8240 ± 0.0136 and Swin UNETR reached 0.8632 ± 0.0056. These results establish that convolutional architectures (U-Net, U-Net++) outperformed transformer-based methods (UNETR, Swin UNETR) for ventricle segmentation on this dataset, with performance differences ranging from 2.65 percentage points (U-Net vs Swin UNETR) to 6.57 percentage points (U-Net vs UNETR) in terms of DSC.Table 7Multi-Class Segmentation Results (Mean ± SD across 5 folds).MethodClassDSC ↑IoU ↑HD95 (mm) ↓U-NetVentricles0.8897 ± 0.00490.8026 ± 0.00560.76 ± 0.01U-Net++Ventricles0.8934 ± 0.00180.8073 ± 0.00290.88 ± 0.27UNETRVentricles0.8240 ± 0.01360.7018 ± 0.01911.71 ± 0.44Swin UNETRVentricles0.8632 ± 0.00560.7594 ± 0.00860.84 ± 0.02U-NetNormal WMH0.6452 ± 0.00910.4763 ± 0.01005.41 ± 0.56U-Net++Normal WMH0.6275 ± 0.00480.4572 ± 0.00515.19 ± 0.36UNETRNormal WMH0.5245 ± 0.00820.3555 ± 0.00755.72 ± 0.59Swin UNETRNormal WMH0.5591 ± 0.02110.3882 ± 0.02065.28 ± 0.16U-NetAbnormal WMH0.6686 ± 0.01440.4908 ± 0.023411.81 ± 1.12U-Net++Abnormal WMH0.6634 ± 0.01680.4966 ± 0.018811.88 ± 0.56UNETRAbnormal WMH0.5551 ± 0.04860.3854 ± 0.046217.46 ± 2.99Swin UNETRAbnormal WMH0.5886 ± 0.00790.4171 ± 0.007915.84 ± 1.11 Table 7Multi-Class Segmentation Results (Mean ± SD across 5 folds).MethodClassDSC ↑IoU ↑HD95 (mm) ↓U-NetVentricles0.8897 ± 0.00490.8026 ± 0.00560.76 ± 0.01U-Net++Ventricles0.8934 ± 0.00180.8073 ± 0.00290.88 ± 0.27UNETRVentricles0.8240 ± 0.01360.7018 ± 0.01911.71 ± 0.44Swin UNETRVentricles0.8632 ± 0.00560.7594 ± 0.00860.84 ± 0.02U-NetNormal WMH0.6452 ± 0.00910.4763 ± 0.01005.41 ± 0.56U-Net++Normal WMH0.6275 ± 0.00480.4572 ± 0.00515.19 ± 0.36UNETRNormal WMH0.5245 ± 0.00820.3555 ± 0.00755.72 ± 0.59Swin UNETRNormal WMH0.5591 ± 0.02110.3882 ± 0.02065.28 ± 0.16U-NetAbnormal WMH0.6686 ± 0.01440.4908 ± 0.023411.81 ± 1.12U-Net++Abnormal WMH0.6634 ± 0.01680.4966 ± 0.018811.88 ± 0.56UNETRAbnormal WMH0.5551 ± 0.04860.3854 ± 0.046217.46 ± 2.99Swin UNETRAbnormal WMH0.5886 ± 0.00790.4171 ± 0.007915.84 ± 1.11 Multi-Class Segmentation Results (Mean ± SD across 5 folds). Normal WMH segmentation proved substantially more challenging across all methods, with DSC values ranging from 0.5245 to 0.6452, representing a performance reduction of 29–36 percentage points compared to ventricle segmentation. The highest-performing method (U-Net, DSC: 0.6452 ± 0.0091) achieved approximately 72% of its ventricle segmentation performance. These results establish normal WMH segmentation as a difficult task requiring specialized approaches beyond standard architectures. U-Net again demonstrated superior performance (DSC: 0.6452 ± 0.0091; IoU: 0.4763 ± 0.01; HD95: 5.41 ± 0.56 mm), while UNETR struggled most significantly (DSC: 0.5245 ± 0.0082). U-Net++ and Swin UNETR showed intermediate performance (DSC: 0.6275 and 0.5591, respectively). Abnormal WMH (lesion) segmentation exhibited consistent architectural performance rankings. U-Net achieved the highest DSC of 0.6686 ± 0.0144, followed by U-Net++ (0.6634 ± 0.0168), Swin UNETR (0.5886 ± 0.0079), and UNETR (0.5551 ± 0.0486). Hausdorff distances for lesion segmentation were consistently larger than for ventricles (11.8–17.4 mm versus 0.7–1.7 mm), indicating reduced boundary delineation precision. The performance gap between top-ranked and lowest-ranked methods was 11.35 percentage points for abnormal WMH compared to 6.57 percentage points for ventricles. Binary segmentation results for ventricles (Scenario 2, Table 8) and lesions (Scenario 3, Table 9) demonstrated consistent performance across folds. For ventricle segmentation, U-Net achieved mean DSC of 0.8982 with minimal cross-fold variance (range: 0.8956–0.9033, SD: 0.0029), demonstrating robust generalization. Performance decreased progressively for U-Net++ (0.8862 ± 0.0082), Swin UNETR (0.8610 ± 0.0017), and UNETR (0.8457 ± 0.0062), consistent with multi-class results.Table 8Binary Ventricle Segmentation Results (DSC per fold).MethodFold 1Fold 2Fold 3Fold 4Fold 5Mean ± SDU-Net0.89800.90330.89560.89670.89730.8982 ± 0.0029U-Net++0.88010.87390.89200.89040.89460.8862 ± 0.0082UNETR0.84310.84070.85360.84010.85110.8457 ± 0.0062Swin UNETR0.85880.86340.86130.86080.86040.8610 ± 0.0017Table 9Binary Lesion Segmentation Results (DSC per fold).MethodFold 1Fold 2Fold 3Fold 4Fold 5Mean ± SDU-Net0.74730.75120.74110.74070.75430.7469 ± 0.0060U-Net++0.67780.63330.66580.59300.62190.6383 ± 0.0328UNETR0.64350.62750.63250.66320.64710.6428 ± 0.0137Swin UNETR0.61320.56040.65910.58410.62720.6088 ± 0.0392 Table 8Binary Ventricle Segmentation Results (DSC per fold).MethodFold 1Fold 2Fold 3Fold 4Fold 5Mean ± SDU-Net0.89800.90330.89560.89670.89730.8982 ± 0.0029U-Net++0.88010.87390.89200.89040.89460.8862 ± 0.0082UNETR0.84310.84070.85360.84010.85110.8457 ± 0.0062Swin UNETR0.85880.86340.86130.86080.86040.8610 ± 0.0017 Binary Ventricle Segmentation Results (DSC per fold). Table 9Binary Lesion Segmentation Results (DSC per fold).MethodFold 1Fold 2Fold 3Fold 4Fold 5Mean ± SDU-Net0.74730.75120.74110.74070.75430.7469 ± 0.0060U-Net++0.67780.63330.66580.59300.62190.6383 ± 0.0328UNETR0.64350.62750.63250.66320.64710.6428 ± 0.0137Swin UNETR0.61320.56040.65910.58410.62720.6088 ± 0.0392 Binary Lesion Segmentation Results (DSC per fold). Binary lesion segmentation performance ranked between ventricle and multi-class lesion segmentation tasks. U-Net achieved mean DSC of 0.7469 (range: 0.7407–0.7543, SD: 0.0060), outperforming U-Net++ (0.6383 ± 0.0328), UNETR (0.6428 ± 0.0137), and Swin UNETR (0.6088 ± 0.0392) by 10.86, 10.41, and 13.81 percentage points, respectively. Cross-fold standard deviation varied across methods: U-Net demonstrated the lowest variability (SD: 0.0060), while U-Net++ showed the highest (SD: 0.0328), representing a 5.5-fold difference. Comparing binary versus multi-class abnormal WMH segmentation for the same class, performance differed by 7.83 percentage points (0.7469 vs 0.6686), with binary segmentation achieving 11.7% higher relative performance. This finding underscores both the challenge and clinical importance of the tri-mask annotation: while more difficult for algorithms, this distinction directly addresses a key clinical problem where normal periventricular hyperintensities are frequently misidentified as pathological lesions. Qualitative analysis Representative segmentation results are visualized across three complementary figures. Figure 2 presents the complete tri-mask segmentation across four cases illustrating varying clinical scenarios: high lesion burden (row 1), moderate lesion burden with prominent ventricles (row 2), challenging small lesions coexisting with normal WMH (row 3), and cases requiring clear distinction between normal and abnormal WMH (row 4). Visual inspection reveals that U-Net consistently produces smoother, more anatomically plausible segmentations across all three tissue classes, with well-defined boundaries and minimal spurious predictions. In contrast, transformer-based methods (UNETR, Swin UNETR) occasionally generate fragmented or discontinuous predictions, particularly for normal WMH, suggesting difficulty in capturing the diffuse, irregular morphology characteristic of these structures despite transformers’ theoretical advantage in modeling long-range dependencies.Fig. 2Qualitative comparison of multi-class segmentation performance across baseline methods. Rows represent clinical scenarios: (1) high lesion burden, (2) moderate lesion burden with prominent ventricles, (3) small lesions with normal WMH, (4) normal versus abnormal WMH distinction. Columns show: input T2-FLAIR, ground truth, U-Net, U-Net++, UNETR, and Swin UNETR predictions. Color scheme: blue (ventricles), green (normal WMH), red (abnormal WMH/MS lesions). U-Net demonstrates superior boundary delineation and anatomical plausibility across all three classes. Qualitative comparison of multi-class segmentation performance across baseline methods. Rows represent clinical scenarios: (1) high lesion burden, (2) moderate lesion burden with prominent ventricles, (3) small lesions with normal WMH, (4) normal versus abnormal WMH distinction. Columns show: input T2-FLAIR, ground truth, U-Net, U-Net++, UNETR, and Swin UNETR predictions. Color scheme: blue (ventricles), green (normal WMH), red (abnormal WMH/MS lesions). U-Net demonstrates superior boundary delineation and anatomical plausibility across all three classes. To provide deeper insight into segmentation errors, Figs. 3, 4 present detailed error analysis for the two most clinically critical classes: abnormal WMH (MS lesions) and ventricles, respectively. These visualizations employ a color-coded error scheme where true positives (TP) appear in green, false positives (FP) in red, and false negatives (FN) in blue, enabling immediate identification of over-segmentation and under-segmentation patterns.Fig. 3Detailed error visualization for binary lesion segmentation across four baseline methods. Each row shows the same four clinical cases as Fig. 2. Columns display: (1) input T2-FLAIR image, (2) ground truth binary mask (white: lesions, black: background), (3) U-Net error map, (4) U-Net++ error map, (5) UNETR error map, and (6) Swin UNETR error map. Error color coding: green (true positives), red (false positives/over-segmentation), blue (false negatives/under-segmentation). Common error patterns include small lesion under-segmentation (blue regions) and periventricular over-segmentation (red regions) where normal WMH are incorrectly classified as lesions.Fig. 4Detailed error visualization for binary ventricle segmentation across four baseline methods. Each row shows the same four clinical cases as Figs. 2, 3. Columns display: (1) input T2-FLAIR image, (2) ground truth binary mask (white: ventricles, black: background), (3) U-Net error map, (4) U-Net++ error map, (5) UNETR error map, and (6) Swin UNETR error map. Error color coding: green (true positives), red (false positives), blue (false negatives). Predominantly green regions across all methods confirm high segmentation accuracy. Minor errors occur primarily at ventricular boundaries due to partial volume effects and in thin ventricular connections. Detailed error visualization for binary lesion segmentation across four baseline methods. Each row shows the same four clinical cases as Fig. 2. Columns display: (1) input T2-FLAIR image, (2) ground truth binary mask (white: lesions, black: background), (3) U-Net error map, (4) U-Net++ error map, (5) UNETR error map, and (6) Swin UNETR error map. Error color coding: green (true positives), red (false positives/over-segmentation), blue (false negatives/under-segmentation). Common error patterns include small lesion under-segmentation (blue regions) and periventricular over-segmentation (red regions) where normal WMH are incorrectly classified as lesions. Detailed error visualization for binary ventricle segmentation across four baseline methods. Each row shows the same four clinical cases as Figs. 2, 3. Columns display: (1) input T2-FLAIR image, (2) ground truth binary mask (white: ventricles, black: background), (3) U-Net error map, (4) U-Net++ error map, (5) UNETR error map, and (6) Swin UNETR error map. Error color coding: green (true positives), red (false positives), blue (false negatives). Predominantly green regions across all methods confirm high segmentation accuracy. Minor errors occur primarily at ventricular boundaries due to partial volume effects and in thin ventricular connections. For abnormal WMH segmentation (Fig. 3), all methods demonstrate robust ventricle delineation with sharply defined boundaries, qualitatively consistent with the high quantitative DSC values (>0.82). However, lesion segmentation quality varies considerably across cases and methods. A common failure mode across all architectures is under-segmentation of small lesions (<5 mm diameter), which often fall below detection thresholds or are fragmented into discontinuous components, manifesting as blue regions (false negatives) in the error visualizations. Conversely, models occasionally over-segment periventricular regions, producing false positives (red) that merge spatially adjacent but distinct lesions into single connected components, particularly in regions where multiple small lesions cluster. U-Net demonstrates the most balanced error profile with fewer false positives compared to U-Net++, while transformer-based methods (UNETR, Swin UNETR) show more scattered false positives in deep white matter regions and more extensive false negatives for small peripheral lesions. The distinction between normal and abnormal WMH remains challenging for all tested architectures. Frequent misclassification patterns visible in the error maps include: (1) normal periventricular WMH incorrectly labeled as lesions (false positives in red), particularly in regions immediately adjacent to the anterior and posterior horns of lateral ventricles where physiological CSF partial volume effects are most prominent; (2) thin periventricular hyperintensities representing CSF contamination misclassified as either lesions or ventricle extensions; and (3) symmetric bilateral deep white matter hyperintensities (typical of age-related changes) occasionally classified as lesions when they exceed typical size thresholds. These error patterns highlight the fundamental difficulty of this classification task and underscore the clinical value of the tri-mask annotation framework for developing algorithms that can make these subtle but clinically important distinctions. Ventricle segmentation error analysis (Fig. 4) reveals substantially more consistent performance across all methods compared to lesion segmentation. The error maps show predominantly green regions (true positives) with minimal red (false positives) or blue (false negatives), confirming the high DSC values (>0.84 for all methods). False positives typically occur at the periventricular boundaries where partial volume effects create ambiguous intensity profiles, particularly in the temporal horns and third ventricle regions. False negatives are rare and generally limited to extremely thin connecting passages between ventricular compartments (e.g., cerebral aqueduct, interventricular foramina) that may be below the spatial resolution of the imaging protocol. U-Net and U-Net++ demonstrate nearly identical error patterns with smooth, continuous boundaries. UNETR and Swin UNETR occasionally produce minor boundary irregularities, visible as thin alternating red-blue bands at ventricle edges, suggesting less stable boundary predictions despite comparable overall DSC values. The superior and more consistent performance for ventricle segmentation compared to lesion segmentation reflects the well-defined, high-contrast anatomical boundaries of ventricles compared to the irregular, variable-intensity characteristics of MS lesions. Usage Notes The MS3SEG dataset requires standard neuroimaging software including dcm2niix for format conversion, FSL for registration and brain extraction, and Python 3.9+ with nibabel, numpy, and scipy libraries. Deep learning implementations utilize TensorFlow framework with CUDA-capable GPUs (minimum 12GB memory recommended). Complete installation instructions are available in the accompanying GitHub repository. Researchers are strongly encouraged to use the provided 5-fold cross-validation splits documented in the repository to ensure fair comparison with baseline results. The 20 held-out test patients should be reserved exclusively for final model evaluation without hyperparameter tuning to avoid optimistic bias. All preprocessed volumes exhibit guaranteed spatial alignment across modalities, enabling direct multi-modal analysis without additional registration. Several considerations affect dataset usage. All acquisitions were performed on a Toshiba Vantage 1.5 T scanner, and models trained exclusively on this data may require domain adaptation for other manufacturers. This deliberate scanner selection addresses the documented predominance of GE and Siemens platforms in existing MS datasets, enabling explicit cross-vendor generalizability evaluation. Annotations are limited to axial T2-FLAIR images; T1-w and T2-w sequences lack tri-mask labels, while sagittal T2-FLAIR remains unannotated. The tri-mask annotations exhibit severe class imbalance with background comprising 99.55% of voxels, ventricles 0.3%, normal WMH 0.05%, and lesions 0.1%. Appropriate handling through weighted loss functions or balanced sampling is essential, as naive training approaches produce models biased toward background with poor lesion sensitivity. Slice thickness varies slightly across patients (5.73 ± 0.52 mm for axial T2-FLAIR), and algorithms sensitive to anisotropic spacing should account for this variability. The distinction between normal and abnormal WMH represents expert consensus but proves challenging for small periventricular hyperintensities where CSF, partial volume effects, and age-related changes create ambiguity. Inter-rater variability exists even among experienced neuroradiologists for this classification, and automated methods should not be expected to achieve perfect agreement. Primary applications include tri-mask segmentation algorithm development, cross-vendor generalizability testing, anatomically-aware approaches leveraging ventricle-lesion spatial relationships, and binary segmentation benchmarking. Secondary uses encompass transfer learning initialization, data augmentation for other MS datasets, and scanner harmonization research. Based on baseline validation, expected performance ranges are: ventricle segmentation DSC 0.85–0.90, binary lesion segmentation DSC 0.65–0.75, and multi-class normal WMH segmentation DSC 0.50–0.65. These performance ranges reflect both dataset characteristics and task difficulty. The modest normal WMH performance reflects the genuine difficulty of this clinically relevant distinction. Researchers should view these ranges as: (1) baselines for methodological comparison, (2) estimates of current capability limits, and (3) motivation for developing specialized approaches. Users should respect provided data splits, explicitly report scanner information in publications, implement class imbalance strategies, report multiple complementary metrics (DSC, IoU, HD95), and cite the original publication with CC-BY-4.0 license acknowledgment. The dataset structure accommodates future enhancements including longitudinal scans, additional annotation classes, multi-rater labels, and clinical outcome data. Users should carefully consider the acquisition protocol characteristics when designing studies using this dataset. The dataset uses 2D sequential slice acquisition with inter-slice gaps (~1.27 mm gap for ~5.7 mm slices), not 3D volumetric acquisition. This has several implications such as anisotropic resolution and partial volume effects. Cross-Dataset Comparison Considerations: Due to the 2D vs 3D acquisition difference, performance comparisons between methods evaluated on this dataset and those evaluated on 3D volumetric challenge datasets should be interpreted cautiously. Performance differences may reflect acquisition type mismatch rather than purely algorithmic superiority. This dataset is best positioned for:Testing robustness of methods to clinical acquisition variationsDeveloping domain adaptation techniques for 2D clinical deploymentBenchmarking within the 2D clinical imaging domainTraining models specifically intended for clinical 2D FLAIR data Testing robustness of methods to clinical acquisition variations Testing robustness of methods to clinical acquisition variations Developing domain adaptation techniques for 2D clinical deployment Developing domain adaptation techniques for 2D clinical deployment Benchmarking within the 2D clinical imaging domain Benchmarking within the 2D clinical imaging domain Training models specifically intended for clinical 2D FLAIR data Training models specifically intended for clinical 2D FLAIR data Despite these considerations, the dataset provides substantial value through its unique tri-mask annotations, real-world clinical acquisition characteristics, and representation of imaging conditions encountered in many clinical deployment scenarios. Users should be aware that due to the retrospective study design, extensive clinical metadata are not available for this dataset. Information regarding MS disease subtype (relapsing-remitting, secondary progressive, primary progressive), Expanded Disability Status Scale (EDSS) scores, disease duration, and treatment regimens at the time of scanning were not systematically extracted. The dataset focuses on single-timepoint imaging data with expert tri-mask annotations rather than comprehensive clinical phenotyping. Future dataset releases will incorporate additional clinical metadata as these data become systematically available.