SzCORE as a benchmark: report from the seizure detection challenge at the 2025 AI in Epilepsy and Neurological Disorders Conference

Jonathan Dan, Amirhossein Shahbazinia, Christodoulos Kechris, David Atienza · 2025

Read SzCORE as a benchmark: report from the seizure detection challenge at the 2025 AI in Epilepsy and Neurological Disorders Conference by Jonathan Dan, Amirhossein Shahbazinia, Christodoulos Kechris, David Atienza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18191.txt.

Introduction

Jonathan Dan, Amirhossein Shahbazinia, Christodoulos Kechris, and David Atienza wrote this report in two thousand twenty-five. It sums up a contest on finding seizures in brain wave records, run under the SzCORE frame at a meeting on AI in epilepsy care.

Hand review of long EEG records is still the norm, and models that shine in print often fail on new patients. The contest scored many top methods on a held-back set of records from dozens of subjects, with expert marks as truth. The best score was low, and top scorers were not the most steady across subjects. The gap between claimed and held-back results shows why shared, strict tests matter. The test base now stays open for future work.

Catalog record

CREATORJonathan Dan, Amirhossein Shahbazinia, Christodoulos Kechris, David Atienza
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18191

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SzCORE as a benchmark: report from the seizure detection challenge at the 2025 AI in Epilepsy and Neurological Disorders Conference Jonathan Dan, Amirhossein Shahbazinia, Christodoulos Kechris, David Atienza Abstract Reliable automatic seizure detection from long-term electroencephalogram recordings (EEG) remains an open challenge in machine learning as current models often fail to generalize across patients or clinical settings. Manual EEG review remains the standard of care, highlighting the need for robust models and standardized evaluation. To rigorously assess current machine learning algorithms for automatic seizure detection in long-term EEG from the epilepsy monitoring unit, we organized a challenge. A private dataset of continuous EEG recordings from 65 subjects, totalling 4,360 hours of data, was utilized to evaluate algorithm performance. Expert neurophysiologists annotated these recordings, establishing the ground truth for seizure events. Algorithms were required to generate accurate onset and duration annotations, with performance measured using event-based metrics, including sensitivity, precision, F1-score, and false positive rate per day. The SzCORE framework was employed to ensure standardized evaluation across submissions. The event-based F1-score served as the primary ranking criterion, reflecting the clinical significance of detecting seizures while minimizing false positives. The challenge attracted 30 submissions from 19 teams or individuals, with 28 algorithms successfully evaluated. Results revealed significant performance variability among state-of-the-art approaches, with the top F1 score of 43% (sensitivity 37%, precision 45%), highlighting the persistent difficulty of this task for current machine learning methodologies. This independent evaluation also exposed a notable gap between self-reported efficacies and challenge performance, underscoring the critical need for standardized, rigorous benchmarking in developing clinically viable ML models. A comparison with previous challenges and commercial systems indicates that the best algorithm in this contest surpassed prior methods. Critically, the challenge infrastructure transitions into a continuously open benchmarking platform, fostering reproducible research and accelerating the development of robust seizure detection algorithms by allowing ongoing submissions and integration of additional private datasets. Clinical centres can also adopt this platform to evaluate seizure detection algorithms on their EEG data using a standardized, reproducible framework. 1 Introduction Automated seizure detection is an active and critical area of machine learning research, particularly driven by the increasing use of ambulatory and long-term electroencephalography (EEG) monitoring in patients with epilepsy. These monitoring techniques, such as those used in epilepsy monitoring units (EMU) and home-based EEG, aim to provide continuous, real-time insights into a patient’s condition, improving diagnostic accuracy and enabling timely medical interventions. Real-time seizure detection can significantly benefit clinical practice by providing immediate alerts to healthcare professionals during in-hospital monitoring Kamitaki et al. 2019 . Meanwhile, offline detection helps reduce clinicians’ workload and detect subtle, often difficult-to-identify seizures Baumgartner and Koren 2018 . Despite significant advancements in the development of automated methods, algorithms often struggle to generalize across different patients, clinical settings, and seizure types. As a result, manual EEG review by clinicians remains the gold standard, underscoring the urgent need for robust, generalizable algorithms that can be deployed in real-world clinical environments. This gap in the state of the art presents a significant opportunity for innovation in machine learning and medical applications. One critical obstacle hindering progress in this field is the lack of standardized evaluation practices, which complicates direct comparisons across studies and prevents the community from identifying the best-performing algorithms. Many recent reviews of seizure detection methods provide valuable overviews of existing work but fail to compare state-of-the-art seizure detection algorithms directly. Miltiadous et al. Miltiadous et al. 2023 conducted a systematic review according to the PRISMA guidelines. They report the performance metrics reported by the authors for 89 studies in the Bonn dataset Andrzejak et al. 2001 , 20 studies in the Physionet CHB-MIT Scalp EEG database Goldberger et al. 2000 ; Shoeb 2009 , 16 in other datasets, and 65 in combinations of datasets. The results are organized by dataset. Shoebi et al. Shoeibi et al. 2021 focused on categorizing deep learning methods. They report the accuracy of the authors’ findings for 136 studies grouped by deep-learning technique. Similar approaches have been developed in other recent reviews Supriya et al. 2023 ; Siddiqui et al. 2020 ; Baumgartner and Koren 2018 . Although these reviews provide a comprehensive overview of the recently published work with a detailed analysis of the datasets used for training and evaluation, along with categorization of the most common machine learning techniques, they do not allow a direct comparison of performance due to heterogeneity in the data sets, evaluation metrics, and evaluation methodology. Most reviews acknowledge this limitation. To address these challenges, Dan et al. developed SzCORE Dan et al. 2024 , a framework that standardizes the validation methodology of seizure detection algorithms and enables direct comparison across studies. SzCORE proposes guidelines on EEG datasets, evaluation methodologies, and performance metrics. It defines the file format and data organization of EEG datasets. These follow a specific format that complies with the BIDS-EEG Gorgolewski et al. 2016 ; Pernet et al. 2019 and HED-SCORE Hermes et al. 2024 specifications. SzCORE defines event-based scoring to compute performance metrics. Event-based scoring compares the labels of an event (seizure) based on the overlap between the reference (ground truth) and the hypothesis (algorithm output) to count True Positives (TP), False Positives (FP) and False Negatives (FN). These counts then compute four performance metrics: sensitivity, precision, F1-score, and false positive rate per day (FP/24h). Algorithms evaluated with SzCORE can be directly compared as performance metrics are evaluated using the same methodology. By providing this framework, SzCORE aims to accelerate iterative improvements in machine learning models for seizure detection, allowing researchers to more reliably build upon prior art. A standardized evaluation framework, such as SzCORE, provides a consistent methodology for comparing algorithms, making it an ideal basis for self-evaluation. However, it does not provide a submission and evaluation pipeline for benchmarking algorithms on a private dataset. This paper presents the results of the seizure detection challenge we organized as part of the 2025 International Conference on Artificial Intelligence in Epilepsy and Neurological Disorders held in Breckenridge (Colorado, USA) from March 3 to 6, 2025. The challenge aimed to identify the best algorithms for automated seizure detection based on long-term scalp EEG from the EMU. The SzCORE framework was used for a standardized evaluation. This paper presents the first large-scale direct comparison of seizure detection algorithms using a standardized evaluation methodology on a substantial private dataset. Furthermore, by analyzing the characteristics of submitted algorithms alongside their performance, this work aims to shed light on effective strategies and persistent hurdles in developing automated seizure detection systems. The challenge’s source code and evaluation platform will remain online as a public service. This initiative provides a crucial and persistent resource for the machine learning community, promoting transparency, reproducibility, and continuous progress in this vital area of medical AI, opening up a new way of benchmarking seizure detection algorithms on private datasets. We describe the organization and timeline of the challenge in Section 2 , the dataset (Section 3 ), the evaluation procedure (Section 4 ) and the algorithmic contributions (Section 5 ). We analyze the submission results in terms of event-based F1-score and compute cost (Section 6 ). We compare them to self-reported results, previous challenges, and commercial software and discuss the limitations of the challenge

Read the complete text