Performances of experts and automated methods on new multiple sclerosis lesions detection: insights from the MSSeg2 challenge Authors: Arthur Masson, Benoit Combès, Anne Kerbrat, François-Daniel Ardellier, Sylvain Rabaste, Pauline Cloarec, Quentin Wdowik, Gilles Edan, Frédéric Cervenansky, Romain Casey, Sorina Camarasu-Pop, Axel Bonnet, Michael Kain, Michel Dojat, Francois Cotton, Olivier Commowick Published: 2026 ABSTRACT AbstractThe assessment of new Multiple Sclerosis (MS) lesions is a time-consuming and difficult task that may lead to an underestimation of patient disease activity. Efficient methods to detect new T2/FLAIR lesions are therefore critical to assist clinicians in this process. In this context, we proposed and organized in 2021 the MSSeg2 challenge aiming at comparing methods segmenting new MS lesions. For this purpose, we built a high-quality dataset of 100 pairs of FLAIR MRI with precise delineation of new MS lesions with a size superior to 3 mm. From these 100 pairs of MS patient images from various scanners and French clinical centers, 40 were shared to the challengers before they submitted their methods. 30 methods from 24 international teams were submitted and evaluated on the FLI-IAM dedicated platform on the 60 remaining images. Overall, we observed that even at lesion scale, expert annotations were variable (40% of lesions were annotated by two or fewer experts out of the four). The best expert exhibited a mean F1 score of 0.679 (SD = 0.345) while the best method exhibited a mean F1 score of 0.698 (SD = 0.295) on the 35 patients with new lesions. Moreover, we did not observe evidence of differences between the top-ranked methods and the best expert performances as evaluated by the F1 score (9 methods exhibited no clear evidence against no difference of mean F1). Similarly, we did not observe evidence of difference in performances between the top-ranked methods and the best expert in classifying patients into a 3 categories clinically relevant scale (0 lesion, 1 or 2 lesions and >2 lesions; 21 methods exhibited no clear evidence against no difference of performances with a best classification accuracy of 85 % for both the best expert and the best method). While results from current automated methods still remain perfectible, our results highlight their potential usefulness in detecting new FLAIR MS lesions.Subject terms: Computational biology and bioinformatics, Diseases, Medical research, Neurology, Neuroscience Introduction Multiple Sclerosis (MS) is an autoimmune inflammatory disease affecting the central nervous system and one of the main causes of disability in young adults. Since the 1990s, multiple treatments have been available to reduce the disease activity of relapsing-remitting MS patients, and their number still tends to increase. Finding the most suited medication for a given patient at a given time is of major importance to limit the occurrence of relapses and ultimately the progression of the patient’s disability. To closely monitor the efficiency of a given treatment, patients are typically followed-up with regular neurological examinations and Magnetic Resonance Images (MRI) imaging1–3. In particular, the appearance of new MS lesions over time is one of the main elements in deciding to change a patient’s treatment. A robust assessment of these lesions is therefore of major practical importance. However, such an assessment is a time-consuming and difficult task that may lead to an underestimation of disease activity. Efficient methods to detect new T2/FLAIR lesions are therefore critical to assist clinicians in this process4,5. There have been numerous studies on automatic cross-sectional MS lesion segmentation, as assessed by two major challenges on MS lesion segmentation held at the MICCAI international conference in 20086 and 20167. A third major competition took place at the IEEE ISBI international conference in 20158 and focused on the study of longitudinal lesion evolution with specific evaluation metrics based on segmentation volume evolution (in addition to the regular segmentation overlap metrics used in 2008). Yet, the quality of MRI images evolves rapidly, as do automatic lesion detection methods. Thus, there was a need for a challenge specifically focusing on new lesions detection to compare the latest automatic lesion segmentation methods. In this context, we proposed and organized in 2021 the MSSeg2 challenge, hosted at MICCAI, for methods aiming at segmenting new MS lesions. For this purpose, we built a high-quality dataset of 100 pairs of MRI from several scanners and French clinical centers with precise delineation of new MS lesions from independent annotations from four neuroradiologists as well as output segmentations from 30 participating methods from 24 international teams. Then, we evaluated the participating methods and experts with various metrics. The main objective of this paper consists in characterizing the performance of experts and methods on the basis of the ground-truth built from this segmentation challenge. Material and methods Challenge data The challenge dataset was constituted from the OFSEP9 cohort; all examinations were performed following a common acquisition protocol3,10. The OFSEP cohort was approved by the Ethics Committee and is registered at ClinicalTrial.gov (NCT02889965). All subjects gave written informed consent for their data to be used for research. Data storage and processing complied with GDPR and French regulations. Data was made available to participants through the Shanoir platform and access was conditioned to the agreement of a specific Data Use Agreement provided in Supplemental Material. MRI data were provided to the participating teams in NIfTI file format. No information regarding patient names or characteristics (e.g. age, acquisition date, weight) was included in the filenames, folder names or file metadata. Overall, 100 pairs of MR images from 100 distinct patients were included with the following requirements: the dataset should contain at least 40% of patients without new lesions and 40% with new lesions, and each scanner constructor must be represented by at least 10 patients. The presence of new lesions was assessed using the EDMUS clinical database11. In the end, images originated from 15 different MRI scanners from 3 major constructors (Siemens, Philips and GE) located in 10 different centers from universities as well as public and private hospitals in France. About half of these scanners had 1.5T magnets (7 scanners), while others had 3T magnets (8 scanners). The resulting dataset was heterogeneous (e.g. voxel sizes vary from 0.5 to 1.2 mm), reflecting real clinical data, with varying image quality. It comprised 72 females and 27 males, with a mean age of 44.76 years old ranging from 24 to 68 years old (excluding one patient for whom the information was not available). The choice was made to use the 3D FLAIR sequence only, as acquisition time is crucial for some centers and thus may correspond to the minimal imaging follow-up scenario. Thus, for each patient, a pair of FLAIR images from two time points was used. The duration between the two time points was 395 days on average, ranging from 70 to 1187 days. For each patient, both images were registered to a middle space between the two time points. Challenge participants 48 teams were initially willing to compete, with a potential overall number of 70 pipelines. A training dataset was then provided to all potential challengers. Finally, 24 teams chose to compete, providing 30 pipelines. Similar to the previous MSSeg Challenges6,7, these pipelines were executed on the FLI-IAM platform (composed of Shanoir and VIP) to automate the evaluation and eliminate any potential biases, such as participants manually adjusting their pipeline’s parameters for each patient. Preprocessing We provided preprocessing scripts along with their corresponding Docker and Singularity definition files. The script made heavy use of Anima, a set of medical image processing tools from the Empenn research team. The provided preprocessing consisted in four steps: i) a brain extraction, ii) a bias correction, iii) an optional normalization step from a given template, and iv) masking from the union of the masks of both time points. Note that the participants had the choice of using this script or their own preprocessing tools. Methods strategies Most participants used deep learning methods; only 2 teams submitted a pipeline not based on deep-learning based approaches. Details on the submitted methods are available in the associated proceedings12. Among the pipelines, the most common architecture is the UNet, the 3D version being more represented than the 2D one. Methods were executed on CPU or GPU depending on participant’s choices. Building the ground truth Experts delineation and adjudication The challenge first required building the ground truth segmentation masks consisting of the precise delineation of new MS lesions on each pair of FLAIR images. Four expert neuroradiologists (J. C., FD. A., S. R. and P. C.), specifically trained in MS and at the early stages of their careers (<5 years of experience) conducted the manual segmentation. These experts were sourced from four distinct French locations. The four experts manually delineated the new lesions on the second time-point, voluntarily omitting the growing lesions, which are of lesser importance for the MS evolution diagnosis. All experts followed the same segmentation protocol using ITK Snap along with a dedicated tutorial. Because of the high inter-expert variability reported in the literature13,14 and in the previous challenges, the lead expert (F. C.), a senior neuro-radiologist (> 20 years of experiences), validated or discarded the lesions which were reported by fewer than 3 experts (lesions found by 3 or 4 experts were automatically validated). Lesions below 3 mm were automatically discarded. Data splitting The dataset was split into a training set of 40 patients (provided to the challengers before they sent their methods for evaluation), and a testing set of 60 patients (not shared before the end of the evaluation). This ratio was chosen to ensure that deep learning-based methods had enough data to be trained exclusively on the given dataset, while also maintaining an adequate number of test cases for evaluation purposes. To evaluate the ability of methods to generalize to previously unknown image characteristics, the training dataset did not contain any image from GE scanners. Additionaly, all images from three specific centers were included in the testing set only, not in the training set. The dataset split was designed to ensure that the proportion of lesions in each location of the brain was preserved in both subsets. The localisation of each lesion was defined using an atlas registered onto the corresponding patient’s images, with five regions of interest: deep white matter, cortical and juxtacortical (hereafter simply referred to as juxtacortical), periventricular, cerebellum and brainstem. The lesion location was defined by the underlying region with the maximum number of overlapping voxels. The resulting split assigned 40% of the brainstem lesions to the training set and 60% to the testing set. The same proportions were applied to the deep white matter and periventricular lesions. For cerebellum lesions, 35% lesions were assigned to the training set and 65% in the testing set. Post-challenge experts delineation and adjudication Once the challenge was over, we investigated potential new lesions in the testing set that were detected by the participating methods but missed by all four experts, and thus were absent from the ground-truth. In total, 3003 lesions were detected by at least one of the methods but were nonexistent in the ground truth. Many of these lesions were obvious false positive, likely due to the large number of methods used and their potential lack of precision. Given the impracticality of having an expert review all 3,003 lesions, we chose to limit the task to lesions detected by at least five methods, or those detected by one of the top five methods with the highest F1 score based on the challenge’s preliminary results. This yielded a subset of 395 lesions, which were independently reviewed and classified as validated or discarded by two external senior experts (A. K. and Q. W.). An adjudication was then performed by the lead expert, F. C., who reviewed the lesions for which the two experts did not agree. Evaluation metrics Experts’ and methods’ performances were investigated using voxel-wise, lesion-wise, and patient-wise metrics. For each method and each expert, these metrics were computed individually on each patient of the testing dataset and then averaged to obtain a summarizing score. Voxel-wise metrics There are various metrics to assess lesion delineation. Similarly to the previous MSSEG challenge7, we used the Dice score (denoted as Dice) which accounts equally on the voxel-wise precision and sensitivity and is one of the most ubiquitous metrics. Additionally, to assess false detection on the patients without any new lesions, we used the volLes metric defined as the volume of new lesions in the predicted segmentation mask (with lower volume indicating higher performance). Lesion-wise metrics In practice, the precise delineation of lesions is rarely of primary interest and often constitutes an ill-posed problem due to the partial volume effect. Instead, robust detection of each new lesion is closer to the clinical goal in the follow-up of MS patients. To assess the performance of lesion-wise detection, we used the F1 score (denoted as F1) as defined in7, which is the lesion-wise equivalent of the Dice score. Additionally, to account for the false detection in patients without any new lesions, we used the nbLes metric, defined as the number of new lesions in the predicted segmentation mask (with lower numbers indication higher performance). Patient-wise metrics From a practical clinical perspective, this metric is the most important. Indeed, the latest guidelines for evaluating treatment response in MS patients recommend changing treatment based on the appearance of new brain lesions between two consecutive examinations. Consequently, in this study, we assessed the ability of methods and experts to classify correctly each patient in one of the three categories: i) no new lesion, ii) one or two new lesions, and iii) three or more new lesions. In particular, missing a few new lesions in a patient with many new lesions is not counted as an error according to this metric. Overall detection sensitivity, specificity and false positive rate In the context of this challenge, methods were required to output a binary segmentation mask that was used to compute the different metrics and in particular the F1 score. However, two output masks providing a similar F1 score can result from drastic differences in calibration of methods sensitivity and false positive rate (FPR). To investigate this aspect, we computed the overall lesion-wise sensitivity (defined as the ratio between the total number of ground-truth lesions detected and the total number of ground-truth lesions), specificity (defined as the ratio between the total number of ground-truth lesions detected and the total number of lesions detected) and false positive rate (defined as the ratio between the total number of false positive detection and the total number of images) for the models and the experts. Statistics Performance comparison with respect to the best expert To gain insight in the performance differences between the best expert and the other methods and experts, we computed the p-value associated with the null hypothesis “no difference in performance with respect to the performance of the best expert” using a Wilcoxon test for the Dice score, F1 score, nbLes and volLes metrics and a binomial test for equal probability with the performance of the best expert for the classification metric. Effect of patient, scanner and lesions characteristics on performance Multiple factors can affect the performances of methods. In particular, images quality can vary, the scanner used for the first examination of a patient differs from the scanner used for the second examination, the number of lesions can vary significantly between patients, and the size and location of lesions likely to affect their visibility. To facilitate interpretation of the results, only the detections of the four methods exhibiting the best performance have been included in this analysis. To investigate the association between the performance of methods in detecting lesions and these various factors, we fitted two independent logistic models. One model was used to assess the association between the variables’ effects and the method’s sensitivity, while the other focused on the method’s precision. For the first model, the dependent variables consisted of all ground truth lesions, including both true positives and false negatives. The second model used all lesions detected (i.e. true and false positives) by the four methods exhibiting the best performance. as dependent variables. The following independent variables (modeled as fixed-effects) were included: The actual method making the detection, The actual method making the detection, The lesion’s location, encoded as one of the following regions: deep white matter, periventricular, juxtacortical, brainstem, or cerebellum, The lesion’s location, encoded as one of the following regions: deep white matter, periventricular, juxtacortical, brainstem, or cerebellum, Whether the scanner changed between the two examinations (called hereafter device_changed), Whether the scanner changed between the two examinations (called hereafter device_changed), The volume of the considered lesion (called hereafter volume), The volume of the considered lesion (called hereafter volume), The total volume of those lesions (called hereafter patient_n_cs_lesions), The total volume of those lesions (called hereafter patient_n_cs_lesions), The number of new lesions in the second image (called hereafter patient_n_new_lesions), The number of new lesions in the second image (called hereafter patient_n_new_lesions), Whether the scanner was a GE scanner, which was only included in the testing set (called hereafter out_of_training), Whether the scanner was a GE scanner, which was only included in the testing set (called hereafter out_of_training), And the image voxel spacing (called hereafter voxel_volume). And the image voxel spacing (called hereafter voxel_volume). The beta coefficients associated to the fitted models and the p-values for the null hypothesis of no effect were reported. Similarly to what we did for the methods, we also investigated the association between the performances of the four experts and the same list of dependent variables (except the out_of_training, as this is not relevant for experts). Specifically, we fitted a logistic models to assess the association between variables’ effects and expert’s sensitivity. Given the high precision of experts, we did not perform a similar analysis for their precision. Results Ground truth analysis Overview of the final ground truth The final ground truth comprised 466 new lesions, most of them located in the deep white matter. Figure 1 shows the distribution of new lesions in the brain among the patients of the test and training sets. Moreover, overall patients had between 1 and 233 lesions at the first acquisition, with a mean of 33.67 lesions and a median of 25 lesions. In the second acquisition, 36 patients had no new lesions, 17 patients had 1 new lesion, and 47 have more than 1 new lesion. The mean number of new lesions was 4.89 with a range of [0, 54]. Number of lesions in each brain region in the training and final testing sets. Heterogeneity of experts annotations and adjudications Figure 2 displays the number of experts who delineated each of the 466 lesions in the final overall dataset. Notably, only 169 of the lesions were detected by all the experts, while 39 lesions were not detected by any of the 4 experts. Additionally, from a total of 395 candidate lesions that were not in the initial ground truth and were detected by at least five methods or by one of the top five methods, 81 of these lesions were validated and added to the ground truth in the test set. Figure 3 shows the result of this validation process by the senior experts. Unsurprisingly, most of these new lesions were relatively small, with half of them being below 17 mm. However, three new lesions larger than 100 mm were not segmented by any experts. Figure 4 provides examples of new lesions that were not part of the initial ground truth, but were detected by automatic methods and finally added to the ground truth. (Expert agreement scores: Number of lesions in the final ground truth (blue + red), respectively annotated by 1, 2, 3 and 4 experts or none (0). Number of lesions detected by automatic methods only and validated a posteriori by experts. Examples of new lesions detected by one or more methods, but not delineated by any expert. Factors that may explain why experts missed or ignored some lesions might include their unusual locations (bottom left corner, Patient 036 lesion 1), small size (bottom right corner, Patient 040 lesion 2) or the quality difference of the two images (top right corner, Patient 002 lesion 1). Top left: The periventricular region is often active and subject to interpretation making it error-prone. Patient 001 has 11 cross-sectional lesions, 22 new lesions, and 5 missed new lesions. This lesion was detected by 13 methods. Top right: The scanner changed between the two acquisitions of this patient. This explains the large quality difference between the two images, which may contribute to the difficulty for experts in detecting the lesion. Patient 002 has 43 cross-sectional lesions, 8 new lesions, and 5 missed new lesions. This lesion was detected by 15 methods. Bottom left: The unusual location of the lesion might explain why experts did not delineate it, even though its volume is relatively large. Patient 036 has 73 cross-sectional lesions, 15 new lesions and 4 new lesions missing from the ground truth. This lesion was detected by 11 methods. Bottom right: Experts have more difficulties finding small lesions. Patient 040 has 59 cross-sectional lesions, 16 new lesions and 1 new lesion missing from the ground truth. This lesion was detected by 9 methods. Finally, we observed that 32 lesions were detected by one or two experts, rejected during the first adjudication phase, then detected by some methods and re-added during the second adjudication phase, highlighting the difficulty to adjudicate cases with subtle and complex signal changes. Performance Methods and experts’ performances Voxel-wise and lesion-wise performance Figure 5 presents a summary of the performances of the methods and experts for the different metrics. In particular, the top row shows the average F1 and nbLes scores of each method and expert, computed on the 35 test patients having new lesions and on the 25 ones without new lesions. Similarly, its second row shows the average Dice and volLes scores. Additionally, the methods and experts whose scores differ significantly from that of the best expert () are highlighted in the Figure. Summary of methods and experts performances for the voxel-wise (top row), lesion-wise (middle row) and patient-wise (bottom row) metrics. Top row: Method (in blue) and expert (in yellow) performances as measured by i) Average Dice scores on the 35 test patients with new lesions (left) and ii) Average volLes scores on the 25 test patients without new lesions (right). The full method names are given in Supplemental Tables. Hatches show when the p-value associated to the null hypothesis “the performance is equivalent to the one of expert 1” is below 0.05. Middle row: Same than previously but using the F1 scores and nbLes instead of Dice and volLes. Bottom row: Average classification accuracy of the methods (in blue) and experts (in yellow) on the 60 test patients with the following 3 classes: “0 lesions”, “1 or 2 lesions”, “3 and more lesions”. Hatches show when the p-value associated to the null hypothesis “the performance is equivalent to the one of E1” is below 0.05. In particular, we observe that nine methods have scores that do not significantly differ from the best expert (i.e. E1), suggesting that a large portion of the pipelines performed well on the new lesion detection task. We observe the same general trend in the ranking for the dice and F1 scores, with the top methods closely matching expert’s (E1) performance, and the same five methods showing lower performances. This observation is supported by a strong Pearson correlation between the lesion-wise F1 score and the voxel-wise Dice score (r = 0.979, p = 1.30e-23) indicating that methods with precise and sensitive detection also tend to generate high-quality delineation. Regarding the averaged nbLes and volLes scores, lower values indicate fewer false detections and smaller errors. As expected, false detections from experts are rare. Only a moderate correlation (r = −0.467, p = 0.0053) was found between the F1 scores and the nbLes scores. Patient-wise performance We also evaluated the ability of experts and methods to correctly classify patients into three categories : i) no new lesions, ii) 1 or 2 new lesions, or iii) at least 3 new lesions. This task differs slightly from the detection of new lesions, as it does not require an exact count when there are more than two lesions present. The average classification accuracies are shown in the bottom row of Figure 5. Moreover, the methods and experts whose scores are significantly different from the score of the best expert’s score () are highlighted in the Figure. Remarkably, several methods demonstrate the ability to accurately classify patients, closely matching the performance of the best expert E1. Overall detection sensitivity, specificity and false positive rate Table 2 in Supplemental Material provides the overall new lesion detection sensitivity, specificity and FPR for the different models and for the experts. In particular for experts, we observe an important percentage of missed lesions (mean, min=, max=), a very low FPR (mean = 0.23, min=0.13, max=0.35) and a high precision (mean = 0.93, min = 0.88, max = 0.95). For methods, we observe an inverse trend where, overall, the percentage of missed lesions is lower than for experts (for the 5 best ranked methods, mean = , min = , max = ) while the FPR remains important (mean = 2.36, min = 0.88, max = 5.07) and the precision moderate (mean = 0.64, min = 0.42 , max = 0.77). Figure 2 in Supplemental Material shows the relation between the overall lesion-wise sensitivity and precision, still outlying these differences. Table 2Top: the pairwise comparison of the influence of the lesion location on the methods sensitivity to detect it. Bottom: the influence of the other factors on the methods sensitivity.BrainstemCerebellumJuxtacorticalDeep white matterVentriclesBrainstem–––––Cerebellum7.06e−01 (9.74e−02)––––Juxtacortical4.58e−01 (1.42e−01)−2.48e−01 (5.04e−01)–––Deep white matter8.95e−01 (1.51e−03)1.89e−01 (5.85e−01)4.37e−01 (1.99e−02)––Ventricles2.96e−01 (3.79e−01)−4.10e−01 (2.94e−01)−1.62e−01 (5.38e−01)−5.99e−01 (5.42e−03)–Beta coef.p valuesDevice_changed1.84e−012.88e−01out_of_training3.15e−013.24e−01Volume−6.29e−032.83e−08patient_n_cs_lesions−4.91e−031.59e−02patient_n_new_lesions2.09e−037.23e−01voxel_volume5.61e−011.25e−01Beta coefficients and p-value for the associated are given. Odd ratios can be computed as . Top: the pairwise comparison of the influence of the lesion location on the methods sensitivity to detect it. Bottom: the influence of the other factors on the methods sensitivity. Beta coefficients and p-value for the associated are given. Odd ratios can be computed as . Effect of patient, scanner and lesions characteristics on performance Influence on expert’s performance Based on the logistic model, we analyzed the influence of each variable on the experts’ sensitivity in detecting new lesions. Table 1 reports, for each pair of regions, the beta coefficients and p-values associated to the region in the row, using the region in the column as the baseline. Overall, the results illustrate the relative difficulty in identifying new lesion depending on its location. Additionally, Table 1 reports the results for other variables and highlights that i) experts have more difficulties in detecting small lesions, ii) experts are more likely to miss new lesions when the total number of cross-sectional lesions is high and when the number of new lesions is low and, iii) experts are more likely to miss new lesions on MR images with a coarser spacing and when the scanner device changed between the two acquisitions. Table 1Top: the pairwise comparison of the influence of the lesion location on the expert’s sensitivity to detect it. Bottom: the influence of the other factors on the expert’s sensitivity to detect new lesions.BrainstemCerebellumJuxtacorticalDeep white matterVentriclesBrainstem–––––Cerebellum9.91e−02 (8.11e−01)––––Juxtacortical−6.31e−01 (4.41e−02)−7.30e−01 (3.94e−02)–––Deep white matter−1.52e−01 (5.89e−01)−2.51e−01 (4.42e−01)4.78e−01 (8.23e−03)––Ventricles−6.59e−01 (5.08e−02)−7.58e−01 (4.33e−02)−2.81e−02 (9.13e−01)−5.07e−01 (1.65e−02)–Beta coef.p valuesdevice_changed4.17e−019.82e−03Volume−4.43e−034.91e−07patient_n_cs_lesions9.68e−031.05e−07patient_n_new_lesions−1.89e−021.26e−05voxel_volume9.20e−011.66e−04Beta coefficients and p-value for the associated are given. Odd ratios can be computed as . Top: the pairwise comparison of the influence of the lesion location on the expert’s sensitivity to detect it. Bottom: the influence of the other factors on the expert’s sensitivity to detect new lesions. Beta coefficients and p-value for the associated are given. Odd ratios can be computed as . Influence on method’s performance Detecting new lesions Table 2 presents the influence of various factors on the methods ability to detect new lesions. We observe that new lesions in the deep white matter were easier for methods to detect compared to those in the brainstem, juxtacortical and around the ventricles. Additionally, methods have more difficulties detecting lesions as their size decreased and when the number of lesions on the first MRI is lower. Avoiding false detection The results obtained from the model evaluating method’s precision are reported in Table 3. We observe that methods generate more false positives in the cerebellum, brainstem, near the cortex, and around the ventricles than in the deep white matter. Additionally, methods are more precise when the number of new lesions increases, and when the MR images have finer voxel spacing, but they are less precise on images that were not seen during the training. Table 3Top: the pairwise comparison of the influence of the lesion location on the methods sensitivity to avoid false detection. Bottom: the influence of the other factors on the method’s precision.BrainstemCerebellumjuxtacorticalDeep white matterVentriclesBrainstem–––––Cerebellum−3.78e−01 (3.49e−01)––––Juxtacortical1.70e−01 (6.16e−01)5.47e−01 (7.83e−02)–––Deep white matter7.41e−01 (1.87e−02)1.12e+00 (7.68e−05)5.71e−01 (1.82e−03)––Ventricles1.24e−01 (7.31e−01)5.01e−01 (1.31e−01)−4.62e−02 (8.55e−01)−6.17e−01 (4.03e−03)–beta coef.p valuesdevice_changedp2.13e−012.21e−01out_of_training1.77e+006.29ep12Volumep2.92e−041.81e−01patient_n_cs_lesions2.30e−031.14e−01patient_n_new_lesionsp5.94e−026.24ep31voxel_volume1.09e+001.74e−04Beta coefficients and p-value for the associated are given. Odd ratios can be computed as . Top: the pairwise comparison of the influence of the lesion location on the methods sensitivity to avoid false detection. Bottom: the influence of the other factors on the method’s precision. Beta coefficients and p-value for the associated are given. Odd ratios can be computed as . Discussion The analysis of the MSSeg2 challenge results provided valuable insights into expert and current method performances. First, similarly to other works, we observe that even at the lesion level, experts annotations are highly variable. Next, the results provided by the top-ranked methods are comparable to those from the best expert, both at lesion and patient levels. Finally, we document the relationships between experts and method performances and extrinsic factors. These observations are discussed in the following subsections. Expert annotations are variable Even at lesion level, expert annotations are variable. In particular, we observe that 40% of the lesions in the ground truth are annotated by two or fewer experts of the four. This observation is not surprising given that the “average expert” missed 43.80% of the overall lesions and that some specific lesions are more likely to be missed by most experts. Even more, 39 lesions were not detected by any of the four experts. All these lesions consist of lesions that were detected by some of the best ranked methods and then confirmed by the three raters checking process and included in the ground-truth. However, this lack of sensitivity could also be partly explained by the presence of ambiguous new lesions for whose existences cannot be assessed unambiguously from the available images. In particular, we observed that even senior expert adjudication was not fully repeatable for the most ambiguous cases. Interestingly, we observe that the performance assessment and subsequent method-to-expert comparison are sensitive to how the ground truth annotations were constructed. By adjudicating lesions detected by methods but not by experts and adding them in the ground truth when relevant, we reduce the bias toward experts in the results. Conversely, this may introduce another bias, as adjudication was performed non-blindly in two separate steps for experts and methods. Nonetheless, as shown in Fig. 4 it is clear from our experiences that some new lesions were missed by the experts and should be accounted for in the ground truth for an accurate performance assessment. The top-ranked methods exhibit performances comparable to those from the best expert, although their calibration differs Although automated methods are more prone to false positive detections (as indicated by the nbLes metric), we do not observe any evidence of performance differences between the top-ranked methods and the best expert when evaluated by the F1 score. Specifically, 9 out of the 30 methods exhibit no clear evidence against the null hypothesis of no difference in mean F1. However it is important to note that overall, segmentations from experts and methods tend to have different properties. In particular, experts tend to exhibit a lower sensibility than the best methods while reaching a lower FPR and a higher specificity. However, at this stage, it is not possible to formally conclude whether this is intrinsically related to differences in performance between methods and experts, or to how they are typically calibrated, which can be adjusted (e.g. using higher softmax output threshold for methods or modifying segmentation instruction for experts). More importantly, we do not observe evidence of differences between the top-ranked methods and the best expert in classifying patients in clinically relevant categories. Twenty-one methods exhibit no clear evidence against the null hypothesis of no difference of performances. Note that we deliberately chose metrics and rankings which provide valuable insights from a clinical perspective. For each method and expert, we computed the average F1 scores across all patients, disregarding their number of new lesions. As a result, a missed lesion on a patient with a single new lesion has a much larger impact on the F1 score than a missed lesion on a patient with many new lesions. In clinical practice, it is much more important to find all new lesions on patients with few new lesions than on patients with many new lesions. We also want to emphasize that we consider the MSSeg2 challenge as an opportunity to review, in a unified manner, the current performance of automated lesion detection, rather than to rank the methods. In fact, this ranking would have been affected by the choice of aggregating the F1 score. Figure 1 of the Appendix illustrates different ways of ranking the methods and experts based on F1 scores for each patient. While the same trends are observed across all cases, we note that the final ranking varies depending on how the scores are aggregated. Influence of image and lesions characteristics on experts’ and methods’ performances The diversity of patients and image quality in our testing dataset valuable insights into their associations with expert and method performances. For experts, we observe interesting associations between lesion-wise sensitivity and patients’ lesions involvement, with more difficulties encountered for patients with high number of cross-sectional lesions, and a low number of new lesions. For methods, we find significant associations between sensitivity and: i) lesions location (with noticeable difficulties near the ventricles, near the cortex and brainstem), ii) overall lesion involvement (with more challenges for patients with fewer cross-sectional lesions), and iii) lesion size (smaller lesions are more difficult). Regarding specificity, we observe significant associations with: i) lesion location, ii) voxel volume, iii) the type of data seen during the training, and iv) the number of new lesions on the patient being studied (with more false positive on patients with numerous new lesions). This last observation may be potentially due to the fact that the methods struggled to distinguish focal damage caused to newly occurring lesions from the growing or expanding lesions typically observed in highly active patients. Additionally, method performance tends to be lower when using data with quality not seen during the training (in our case, GE scanners). These observations highlight the importance of developing datasets tailored to specific complex settings. Relationship to existing open datasets and evaluations In most studies, automated methods for segmenting new MS lesions are evaluated independently using internal ground truth data. The datasets used in these different studies vary in terms of homogeneity, with many studies using a single scanner device (e.g.15–17). Moreover, despite some notable exceptions e.g.18,19, evaluation datasets rarely exceed 40 patients. The current limitation of open-access, dedicated datasets presents a significant obstacle to more homogeneous evaluations that involve a variety of acquisitions-specific and patients-specific conditions. To the best of our knowledge, before the release of the MSSeg2 dataset, only two main open longitudinal MS lesions datasets existed. The first one is those associated with the ISBI 2015 challenge20. This dataset contains follow-up data from 82 acquisition sessions across 19 unique patients (4 patients used for training and 15 for testing). It includes FLAIR, T1 weighted and T2 weighted acquisitions for each session. Ground truth lesion masks were built from the delineation of two experts. Interestingly, for the testing set, the challenge organizers released only the anatomical MR data and provided a web portal for challlengers to continuously test segmentation masks against the (unreleased) ground truth masks. To the best of our knowledge, the results reported in the publication and through the evaluation platform focused exclusively on voxel-wise metrics. Despite its strength, this dataset suffers from three main limitations: i) the lack of a substantial training set, ii) the lack of data acquisition heterogeneity that reflects current clinical practice and, iii) the lack of patients heterogeneity with only 17 patients in total. The second one is associated with a study21 that aims at building a ground truth dataset of new MS lesion segmentation masks and assess the performance of three new MS lesion segmentation methods. The dataset consists of pairs of FLAIR, T1-weighted, and T2-weighted acquisitions from 20 MS patients, all from a single scanner device. Ground truth lesion masks were built from the annotations of two experts. The overall corresponding dataset has been released and shares similar limitations than the the ISBI 2015 dataset mentioned above. In contrast, in our work, we presents an open evaluation based on 100 patients (40 for training and 60 for testing), utilizing delineation from four experts and a retrospective analysis of lesions detected by various competing methods. To our opinion, this evaluation, which facilitates a homogeneous comparison of a representative set of methods, stands out from others from several key aspects. First, the provided training set is of a reasonable size enabling the training of dedicated approaches that directly focus on new lesions detection without having to split the original problem into independent cross-sectional segmentation tasks. Several of these have been published by their authors in a dedicated issue12 using the ‘MSSeg2‘ training dataset22–30. Second, our dataset derived from the clinical data collected by OFSEP (the french organization responsible for constituting the national cohort of patients suffering from MS or related diseases and syndromes) reflects a broad diversity of patient characteristics and follow-up acquisition details. Third, we investigated the relative performances of experts and methods at the voxel, lesion and patient scales, considering various image quality. Overall, the presented evaluation aims to characterize the ability of methods to directly capture the new lesion involvement in real-life patient follow-ups, without experts interaction. It is worth noting that there is another class of studies that differ from the ones discussed above. These studies explore the ability of methods to assist radiologists in MRI reading. In this context, the target metrics cannot be directly assessed from ground truth data, as in our study. Specifically, false positive detections may have a lower impact in such settings, as they can easily be discarded by experts. Conversely, there is a risk that experts may place too much confidence in the sensitivity of the methods’ detection and fail to thoroughly check for potential lesions they could have missed. Dedicated study designs involving radiologists must be employed to properly assess these methods for such applications4,31. However, indirect insights can be drawn from our experiments. Notably, we observed that experts sensitivity to detect new lesions is only moderate and that methods are able, at the cost of a reasonable amount of false positives, to offer a much better sensitivity. As a consequence, we observed many occurrences of lesions detected by the top-ranked methods but only one or two of the experts, outlying the potential benefit of using automatically generated segmentation masks to help experts to increase their sensitivity. Limitations Although we have extracted valuable insights from the challenge, our study has some limitations. The first limitation relates to the rapid pace of evolution in deep learning architectures. Most participants leveraged the state-of-the-art UNet design to develop their methods, but several competing alternatives have gained in popularity since the beginning of the challenge, such as transformer-based architectures (e.g. SETR, UNETformer), or the “foundation models for image segmentation” (e.g. SAM, RadFM). While many of the challenge’s conclusions would likely remain unchanged with these new approaches, rerunning the experiments could help update some of the outcomes and highlight differences in evolution of methods performance compared to expert performance. For this purpose, the challenge data is available on Shanoir and the evaluation tool used for the challenge -segPerfAnalyzer- is part of the open binaries suite Anima. Secondly, even though some companies participated in the challenge, no commercial products were submitted. The performance of these specific tools remains unknown. However, it is likely that the 30 competing methods provided a comprehensive overview of the potential state-of-the-art implementations. Thirdly, there are alternatives that could have been considered instead of relying solely on the F1 score in the challenge. While the F1 score has the advantage of equally weighting precision and sensitivity, using more granular metrics that combine these two measures in a way that is more application-oriented could better address practical needs. However, the question of identifying the ideal target metric remains an open issue. Another way to proceed could be to ask challengers to provide instance masks with associated probability for each lesion instance instead of binary segmentation masks so that the performance of each model can be estimated for different sensitivity/FPR balance, as it has been performed in some recent challenges32. Finally, we designed the challenge with the goal of comparing the true capabilities of all methods, rather than focussing on their potential performance in a fixed setting. Hence, we allowed participants to train their models with all the data at their disposal. However, by requiring competitors to submit two versions of their method, - one trained with the challenge training data exclusively, and another trained with any additional dataset, - we could have provided fairer comparisons, both in terms of absolute and potential performance. Perspectives & conclusions Our results highlight the potential usefulness of current automated methods for detecting new MS lesions. Nevertheless, we also note that the results can still exhibit imperfections, suggesting that further improvement is possible. The MSSeg2 dataset is available for scientific research on Shanoir and we hope its design will serve as a valuable basis for identifying current shortcomings and fostering the development of new methods to address them. Moreover, with these results comes the need to better capture the practical utility of such tools. First, while the F1 score accounts equally for sensitivity and precision, this is a convenient choice as it eliminates the need to choose between the two types of errors. However, it is likely that these errors do not carry the same weight in practical situations, and their significance may vary depending on the application. Developing evaluation metrics that best align with practical needs remains an open research direction. Secondly, to date, all MS challenges have focused primary on brain MRI. Nevertheless, spinal cord lesions are known to have significant prognostic value, and there is a clear need for methods capable of addressing this organ. This task may prove more complex due to challenges related to varying characteristics of spinal cord data (e.g., inhomogeneous resolution, partial volume effects, subtle tissue contrast, and frequent acquisition artifacts). Acknowledgements Data collection has been supported by a grant provided by the French State and handled by the “Agence Nationale de la Recherche,” within the framework of the “France 2030” program, under the reference ANR-10-COHO-002, Observatoire Français de la Sclérose en Plaques (OFSEP). The FLI-IAM platform is supported by the French program “Investissement d’Avenir” run by the Agence Nationale pour la Recherche (ANR-11-INBS-0006). The pipelines were executed with VIP, using the Dirac service, on the computing resources provided to the biomed VO within the EGI e-infrastructure. Author contributions F.C., S.P., A.B., M.K., M.D. participated in the setup of the platform and running of the experiments on the France Life Imaging platform, the writing and proof-reading of the pipeline processing description and results in particular, and of the proof-reading of the whole article. O.C., F.C. participated in the challenge organization, design of the results evaluation experiments and metrics. A.M., B.C., A.K., O.C., G.E, F.C., R.C. participated in the constitution of the MSSeg2 MRI dataset (selection of the patients, expert guidance on the delineation of the lesions, data set split). A.K., F.C., FD.A., S.R, P.C, Q.W. participated in the building of the ground-truth segmentation masks (manual annotations, adjudication). A.M., B.C., F.C. participated in the analysis of challenge results. A.M. and B.C wrote the initial draft of the paper. All authors approved the submitted version. Data availability The dataset used to train the different models are available at https://shanoir.irisa.fr/shanoir-ng/welcome under the study name: “MSSeg2”. Case-wise performances of the different models and experts on the testing set are available here: https://zenodo.org/records/5552675 Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. O. Commowick passed away in December, 2022. References Associated Data Data Availability Statement The dataset used to train the different models are available at https://shanoir.irisa.fr/shanoir-ng/welcome under the study name: “MSSeg2”. Case-wise performances of the different models and experts on the testing set are available here: https://zenodo.org/records/5552675