Annotated interictal discharges in intracranial EEG sleep data and related machine learning detection scheme Authors: Rotem Falach, Maya Geva-Sagiv, Dawn Eliashiv, Lilach Goldstein, Ofer Budin, Guy Gurevitch, Genela Morris, Ido Strauss, Amir Globerson, Firas Fahoum, Itzhak Fried, Yuval Nir Published: 2024 ABSTRACT Interictal epileptiform discharges (IEDs) such as spikes and sharp waves represent pathological electrophysiological activities occurring in epilepsy patients between seizures. IEDs occur preferentially during non-rapid eye movement (NREM) sleep and are associated with impaired memory and cognition. Despite growing interest, most studies involving IED detections rely on visual annotations or employ simple amplitude threshold approaches. Alternatively, advanced computerized detection methods are not standardized or publicly available. To address this gap, we introduce a novel dataset comprising multichannel intracranial electroencephalography (iEEG) data recorded at two medical centers during overnight sleep with IED annotations performed by expert neurologists. Utilizing these annotations to train machine learning models via a gradient-boosting algorithm, we demonstrate automated IED detection with high precision (94.4%) and sensitivity (94.3%) that can generalize across individuals and surpass performance of a leading commercial software. The dataset featuring multi-channel annotations with sub-second resolution including hippocampus and medial temporal lobe (MTL) regions is made publicly available, together with the detection algorithm, to advance research on detection methodology, epilepsy, sleep, and cognition. Background & Summary Epilepsy is a common neurological condition present in approximately 50 million people worldwide1,2 that affects brain activity and leads to seizures. In addition to seizures, interictal epileptiform discharges (IEDs) are pathological electrophysiological events observed between seizures in patients with epilepsy. These events include spikes, poly-spikes, sharp waves, or spike and slow wave (spike-wave) complexes that last 20–70 ms for an interictal spike or 70–200 ms for sharp waves3. IED can appear several times per minute (much more frequently than seizures) and their rate varies depending on the pathology, age, vigilance state, drug therapy, circadian time, and other factors4–9. The prevalence of IED has been associated with long-term cognitive impairment, such as impaired language abilities or memory loss10–13. Although for a long while their clinical significance was debated14, precise localization of IED can aid diagnosis by suggesting a particular epilepsy syndrome or helping to identify the seizure onset zone (SOZ), a procedure that can be crucial in patients with drug-resistant epilepsy considered for resective epilepsy surgery15,16. In many types of epilepsy, pathological activity including seizures and IED occur preferentially during sleep17–19. Indeed, sleep is a potent physiological modulator of epileptic activity. Specifically, non-rapid eye movement (NREM) sleep, which comprises ~80% of sleep time in adults, is characterized by high neuronal synchrony at low (<4 Hz) frequencies, which increases with increasing sleep depth. Slow wave sleep (SWS), also known as N3 sleep, displays the highest degree of physiological synchronization within the brain20. Mechanisms facilitating slow wave activity, such as the reduction of cholinergic neuromodulation that normally leads to cortical de-correlation21, may concurrently serve as catalysts for the onset of pathological occurrences like IEDs and seizures. Indeed, IEDs occur maximally during NREM sleep whereas they show low rates and minimal spatial spread during REM sleep and wakefulness22. Thus, detection of IEDs during sleep represents an important challenge that holds significant value for both research and clinical purposes. The detection of IEDs serves as an initial milestone in any investigation for basic research or clinical evaluation. In mesial temporal lobe epilepsy (MTLE) patients, the seizure onset zone and IEDs occur primarily in deep limbic regions such as the medial temporal lobe (MTL). In such cases, intracranial EEG (iEEG) recorded from implanted depth electrodes often serves as a useful diagnostic procedure for drug-resistant epilepsy and is widely used to detect IEDs in patients with a known history of seizures. However, the lack of an objective definition of IEDs can cause disagreement even among EEG experts, and inter-observer reliability often hinders EEG interpretation23,24. One of the possible sources of classification errors is that IED morphology varies between subjects, due to different IED sources and brain pathologies. Visual annotation of IEDs by neurologists, although time-consuming and requiring extensive training, is considered the gold standard25. However, expert experience, as well as clinical softwares (e.g. “Persyst”) are mostly based on scalp EEG, and there are fewer clear definitions for intracranial recordings. Furthermore, the growing number of intracranial studies, incorporating continuous recordings from Deep Brain Stimulation (DBS) devices, emphasizes the increasing need for accurate IED detection from iEEG signals. Recent studies have explored various methodologies for identifying IEDs, ranging from template matching to frequency domain methods and beyond26. While these methods are extensively employed, the controversy surrounding their validation and comparison persists, primarily stemming from the absence of definitive ground-truth data. Aligned with this objective, this paper aims to furnish a cohesive dataset of iEEG sleep recordings from two different medical centers along with expert annotations of IEDs timings in Brain Imaging Data Structure (BIDS)27,28, a format that is both accessible and user-friendly. In order to demonstrate the validity of this dataset, and its potential to generalize to other patients’ recordings, we employed a machine learning (ML) algorithm to detect the IEDs and performed analysis of the detected results. While ML techniques have become increasingly prevalent in this domain, a notable gap exists in the literature concerning the utilization of gradient-boosting algorithms29 for IED detection. Gradient Boosting currently stands as a frontrunner in the field of classical ML based on feature extraction30. Its significant advantages in terms of resource efficiency and result interpretability distinguish it from neural networks. As the field of feature-based ML keeps improving, the incorporation of gradient boosting into the realm of IED detection holds the potential to significantly enhance both the accuracy of predictions and the understanding of underlying patterns in intracranial EEG data. To the best of our knowledge, the current dataset marks the first iEEG sleep dataset offering expert-annotated iEEG signals with a sub-second resolution that is made freely available. Furthermore, we utilized state-of-the-art ML techniques to develop and validate an automated system for detecting IEDs in intracranial EEG during sleep. In contrast to the variability associated with manual annotations by clinical personnel, automated computational tools hold promise in establishing new benchmarks for sensitivity and objectivity and advancing methodologies within the field. Methods Participants 25 drug-resistant epilepsy patients were implanted with depth iEEG electrodes as part of clinical evaluation for seizure foci identification and potential surgical treatment. Electrode locations were based solely on clinical criteria. All patients provided written informed consent to participate in the research study, under the approval of the Institutional Review Board at the Tel Aviv Sourasky Medical Center (TASMC, 9 patients), or the Medical Institutional Review Board at the University of California, Los Angeles (UCLA, 16 patients). In their consent, patients explicitly agreed for anonymized data to be shared and used in future scientific publications. UCLA Hospital IRB protocol: 10-000973, TLVMC IRB protocol: TLV-008-12. Participants recruited for the study were patients at UCLA (2007–2012, 2017–2021) or TASMC (2017–2023) who volunteered to participate in an overnight sleep research recording session. EEG Recordings For each patient, depth electrodes were placed in different regions according to clinical needs. Each depth electrode consisted of platinum iEEG contacts along the shaft. iEEG data were continuously recorded using Blackrock system throughout sleep referenced to a central scalp electrode and sampled at 2KHz (Figs. 1A, 2A). Channel selection was based on availability (Table 1), without consideration of IED presence. Channel names consist of an initial letter indicating the hemisphere (R = right, L = left), followed by 1–3 letters representing the brain region (A = amygdala, EC = entorhinal cortex, AH = anterior hippocampus, etc. A full list of channel abbreviation definitions is included in the dataset), and a number starting from 1, denoting the most mesial contact and increasing as it moves laterally. In addition, in 15 patients sleep scoring employed additional non-invasive simultaneous polysomnography using EEG channels from the C4 (right vertex), C3 (left vertex), and Pz (central parietal) derivations, electrooculogram channels (EOG1, EOG2), and chin electromyogram (EMG).Fig. 1Experimental data. (A) Raw data example from patient 12. The montage contains both hemisphere iEEG channels from most mesial to most lateral, from top to bottom: RA1-RA5 in black, REC1-REC5 in blue, RAH1-RAH5 in red, LA1-LA5 in green, LEC1-LEC5 in pink and LAH1-LAH5 in brown. Annotation examples of right hemisphere IED marked by the two left orange shades and left hemisphere IED marked by the right orange shade. Gray vertical lines represent 1 second and amplitude was auto-adjusted to best visualize IEDs. (B) Bar plot displays the rate of IEDs per minute in patient recordings. The bars are divided into left (blue) and right (green) hemispheric activity. Patients are sorted by total IEDs rate. (C) Visualization of electrode placements overlaid on a standard brain model, where color intensity corresponds to the frequency of patients recorded for each specific channel. Top: left and right hemisphere, bottom: superior view. (D) Time-frequency representation (spectrogram) of C3 electrode recorded in subject 14 during a full night sleep study. Warm colors indicate increased power in specific time-frequency windows (see color bar on the right, frequency on the left y-axis). Black trace marks the hypnogram, time-course of sleep-wake stages in one representative individual. W, wake; R, REM sleep; N1-N3, NREM sleep, stages 1–3.Fig. 2Dataset generation process. (A) Epilepsy patients implanted with depth electrodes were connected to an iEEG acquisition system for clinical monitoring during sleep. (B) The recorded signals underwent offline examination by a neurologist using clinical software. Bottom: annotation examples from two patients and annotators. PHG = parahippocampal gyrus, Bi-A = bi-synchronous amygdala, M-REC = mesial right entorhinal cortex. (C) The final dataset contains raw data and IEDs annotations exported into BIDS format.Table 1Recorded regions.12345678910111213141516171819202122232425HippocampusVVVVVVVVVVVVVVVVVVVVVVVVAmygdalaVVVVVVVVVVVVVVVVVVVVVEntorhinal cortexVVVVVVVVVVVVVVVParahippocampal gyrusVVVVVVVVVVVFrontal lobeVVVVVVVVVLateral temporalVVVVVVinsulaVVVVVOccipital lobeVThe table lists the specific brain regions recorded via iEEG electrodes for each patient. Regions with a “V” indicate where recordings were successfully obtained and reviewed by a neurologist. Experimental data. (A) Raw data example from patient 12. The montage contains both hemisphere iEEG channels from most mesial to most lateral, from top to bottom: RA1-RA5 in black, REC1-REC5 in blue, RAH1-RAH5 in red, LA1-LA5 in green, LEC1-LEC5 in pink and LAH1-LAH5 in brown. Annotation examples of right hemisphere IED marked by the two left orange shades and left hemisphere IED marked by the right orange shade. Gray vertical lines represent 1 second and amplitude was auto-adjusted to best visualize IEDs. (B) Bar plot displays the rate of IEDs per minute in patient recordings. The bars are divided into left (blue) and right (green) hemispheric activity. Patients are sorted by total IEDs rate. (C) Visualization of electrode placements overlaid on a standard brain model, where color intensity corresponds to the frequency of patients recorded for each specific channel. Top: left and right hemisphere, bottom: superior view. (D) Time-frequency representation (spectrogram) of C3 electrode recorded in subject 14 during a full night sleep study. Warm colors indicate increased power in specific time-frequency windows (see color bar on the right, frequency on the left y-axis). Black trace marks the hypnogram, time-course of sleep-wake stages in one representative individual. W, wake; R, REM sleep; N1-N3, NREM sleep, stages 1–3. Dataset generation process. (A) Epilepsy patients implanted with depth electrodes were connected to an iEEG acquisition system for clinical monitoring during sleep. (B) The recorded signals underwent offline examination by a neurologist using clinical software. Bottom: annotation examples from two patients and annotators. PHG = parahippocampal gyrus, Bi-A = bi-synchronous amygdala, M-REC = mesial right entorhinal cortex. (C) The final dataset contains raw data and IEDs annotations exported into BIDS format. Table 1Recorded regions.12345678910111213141516171819202122232425HippocampusVVVVVVVVVVVVVVVVVVVVVVVVAmygdalaVVVVVVVVVVVVVVVVVVVVVEntorhinal cortexVVVVVVVVVVVVVVVParahippocampal gyrusVVVVVVVVVVVFrontal lobeVVVVVVVVVLateral temporalVVVVVVinsulaVVVVVOccipital lobeVThe table lists the specific brain regions recorded via iEEG electrodes for each patient. Regions with a “V” indicate where recordings were successfully obtained and reviewed by a neurologist. Recorded regions. The table lists the specific brain regions recorded via iEEG electrodes for each patient. Regions with a “V” indicate where recordings were successfully obtained and reviewed by a neurologist. Sleep staging Manual sleep scoring was performed according to the established guidelines of the American Academy of Sleep Medicine31 facilitated by the sleep module within the Visbrain Python package32. EEG scalp electrodes were utilized whenever feasible; The data underwent resampling to 250 Hz and were visualized in 30-second epochs, accompanied by at least one synchronized EOG signal from electrodes placed above and below the eyebrows, and optionally an EMG signal from a submental electrode33,34. Verification of successful scoring was further corroborated by inspecting the time-frequency representation, depicted in Fig. 1D. In cases where only iEEG data were available, a validated automatic algorithm35,36 was employed to detect NREM in neocortical channels based on slow-wave and sleep spindle occurrence, while all the other epochs were marked as “wake/REM”. Electrode localization Pre-implantation, patients underwent T1-weighted MRI scans using a 3-Tesla scanner, providing detailed structural information. Post-implantation, CT scans were conducted and integrated with the pre-operative MRIs, and individual recording sites were visually identified and manually marked within each participant’s native MRI space using BioImage37, FreeSurfer38, and iELVIS39. Finally, the data from each subject were converted into a common brain average space (MNI) and visualized using NetPlotBrain40 (Fig. 1C). In instances where precise coordinate data were unavailable (112 electrodes from 7 patients), we utilized the average coordinates derived from all other patients for each channel. Manual annotations iEEG data underwent assessment by two neurologists. The first neurologist (D.E.) completed a fellowship in Clinical Neurophysiology and Epilepsy at UCLA, is now a Professor of Neurology at UCLA, Co-Director of the UCLA Seizure Disorders Center, and Director of the Clinical Neurophysiology and Epilepsy Training Program; D.E. reviewed a full montage of all intracranial channels for 10 UCLA patients. The second neurologist (L.G.) completed a clinical fellowship in Epilepsy and Clinical Neurophysiology at the Thomas Jefferson Epilepsy Center and is currently an attending physician at the EEG and Epilepsy Unit in TLVMC; L.G. assessed the other 15 patients. This neurologist reviewed a montage of the three most medial channels in the MTL probes (bilateral anterior hippocampus, amygdala, and entorhinal cortex), re-referenced offline to an additional EEG channel. Neurological annotation and assessment primarily employed a scalp reference montage, which was also exclusively used for any subsequent machine-learning based detections. In addition, for neurological annotations in patients who only had MTL signals we included an additional bipolar reference to facilitate easier data inspection and enhance confidence in the annotations. Conversely, for patients with many channels across multiple lobes (including MTL, lateral, frontal, etc.), an additional bipolar reference was not added, as this resulted in an excessive number of channels for neurologists to examine simultaneously. The visual analysis for IEDs was conducted based on the criteria for interictal epileptiform discharges as published by the International Federation of Clinical Neurophysiology, with relevant considerations for intracerebral recording specified by B. Frauscher et al.41. The expert neurologists, blinded to the patient’s clinical profile, identified and tagged 852 interictal activity epochs using Nicolet Reader software (Natus, USA) or Persyst software (Figs. 1B, 2B). Each IED tag was annotated with a specific timestamp and the brain location of abnormal activity. Further analysis involved visually inspecting the montage and converting each annotation into a list of specific channels where the IEDs were observed (Fig. 2C). To assess the inter-rater reliability of the annotations, we calculated the Cohen’s Kappa score for a subset of six patients each annotated independently by the two expert neurologists (where each neurologist also annotated three iEEG recordings from the other medical center, to facilitate cross-hospital comparisons). We compared the annotation timings with a 1-second resolution and computed the Kappa score to quantify agreement between the two raters. The average Kappa score for the original dataset was 0.63 ± 0.23, while the balanced dataset achieved a score of 0.71 ± 0.24, which is consistent with findings in previous literature42. Data Records The dataset is available at Figshare: 10.6084/m9.figshare.2613197843. Overall, our study involves 1–5 minutes of data per patient (N = 25), comprising a total of 857 iEEG channels (Table 1) and 76 minutes of data, along with precise sub-second timings of 852 IEDs annotated by expert neurologists. The dataset is organized by patient, with each patient’s data stored in a separate folder. The root folder contains a “participants” file containing clinical information such as participant age, sex and seizure onset zone (SOZ), and a readme file with a brief summary of the dataset. Within each patient’s folder, there are files for raw data, annotations, electrode information (coordinates available for 18 patients), and a metadata JSON containing additional details such as power line frequency, time from sleep onset (available for 22 patients), and a sleep scoring vector in 30-second resolution (available for 15 patients). The derivatives folder includes a channels file with channel abbreviation definitions and TSV files containing the relevant channels corresponding to each annotation. All file names follow a consistent BIDS naming convention, reflecting the type of data and the patient ID, to ensure easy navigation. Technical Validation Detection model The detection model underwent several stages (Fig. 3A). iEEG signal (in each channel that contained at least one IED annotation) was resampled to 1 kHz, band-pass filtered digitally between 0.1 Hz and 500 Hz, and subjected to a notch filter at 50/60 Hz to eliminate residual line noise offline. Next, Z-score normalization was applied to the raw channels and time-courses were segmented into 250 ms epochs, where each segment was categorized as either abnormal (IED occurrence) or normal, based on neurologist assessment. Each epoch comprised a vector of 250 time points, facilitating the extraction of the 14 best features out of all mne-features44 and Antropy package in terms of speed and accuracy, including kurtosis, entropy, teager-kaiser energy parameters45, peak-to-peak amplitude, Hjorth complexity and mobility46. In addition, we added two features of kurtosis and peak-to-peak amplitude that represent the channel properties. Initially, all 104 univariate features available in the MNE-features package were extracted, and the model was trained using the full feature set. We then refined the feature set using various feature importance metrics, including feature gain and frequency, permutation importance, and SHAP values (that quantify each feature's impact on a model's prediction)47. After several iterations, we reduced the number of features to optimize performance, ultimately replacing three features with those from the Antropy package to enhance processing speed while maintaining model accuracy.Fig. 3ML approach for data validation. (A) A diagram illustrating the steps involved in model implementation and evaluation, encompassing data segmentation, feature extraction, and performance evaluation through k-fold cross-validation and leave-one-out generalization tests. (B) Detection examples obtained from different patients using leave-one-out test. Each signal showcases a duration of 7 seconds from a specific channel. True positives are highlighted with a green background, false negatives with a pink background, false positives with an orange background, and true negatives with a white background. (C) Top: K-fold cross-validation metrics. Each point corresponds to a different testing fold excluded from the training dataset. Bottom: Leave-one-out generalization test results. Each point corresponds to a patient excluded from the training dataset. The LGBM model’s results are represented in blue, while clinical software results are depicted in orange, and the multi threshold algorithm results are in red. The median is denoted by a black horizontal line, and the mean by a black diamond. (D) Feature importance is illustrated by SHAP values sorted by the sum of the SHAP value magnitudes across all samples. (E) Various examples of IED morphologies are showcased through Event-Related Potentials (ERP) obtained from different channels. ML approach for data validation. (A) A diagram illustrating the steps involved in model implementation and evaluation, encompassing data segmentation, feature extraction, and performance evaluation through k-fold cross-validation and leave-one-out generalization tests. (B) Detection examples obtained from different patients using leave-one-out test. Each signal showcases a duration of 7 seconds from a specific channel. True positives are highlighted with a green background, false negatives with a pink background, false positives with an orange background, and true negatives with a white background. (C) Top: K-fold cross-validation metrics. Each point corresponds to a different testing fold excluded from the training dataset. Bottom: Leave-one-out generalization test results. Each point corresponds to a patient excluded from the training dataset. The LGBM model’s results are represented in blue, while clinical software results are depicted in orange, and the multi threshold algorithm results are in red. The median is denoted by a black horizontal line, and the mean by a black diamond. (D) Feature importance is illustrated by SHAP values sorted by the sum of the SHAP value magnitudes across all samples. (E) Various examples of IED morphologies are showcased through Event-Related Potentials (ERP) obtained from different channels. For model training and model evaluation purposes, we employed random under-sampling to balance the data, ensuring an equal number of epochs with and without IEDs. Balancing the data is crucial to prevent the model from being biased toward the majority class (non-IED epochs), which could result in poor performance and inaccurate evaluation metrics. Furthermore, under-sampling allowed us to train on more channels and increased the variability of IEDs in the training set, enhancing the model’s generalization without compromising key metrics. Although over-sampling techniques like SMOTE48 resulted in a higher F1 score, they produced a lower PRAUC. When comparing the original ratio model performance with under-sampled models at a 1:10 ratio, the original ratio yielded the lowest PRAUC, while the balanced and 1:10 ratios performed similarly but exhibited variability in other metrics. These findings underscore that different sampling strategies can affect various metrics in distinct ways, making it essential to understand the biases each approach introduces. Depending on the task, one might prioritize maximizing sensitivity over precision or vice versa. In the end, we opted for under-sampling due to its simplicity and ability to deliver clean, reliable results. This approach leverages only real data, making it more straightforward and offering robust, interpretable metrics. First, a 5-fold cross-validation was employed, randomly dividing the data into five non-overlapping folds, with one fold serving as the test set while the others were used for training the light gradient boosting machine (LGBM) algorithm49. Performance metrics such as accuracy, precision, sensitivity, specificity, F1 score, area under the curve of the receiver operating characteristic (AUCROC), and area under the curve of the precision-recall curve (AUCPR) were computed for each test fold, with the average of all folds considered as the final result. Next, feature importance was determined using the SHAP algorithm, elucidating the contribution of each feature to the model’s output and enhancing interpretability. We used the SHAP results to choose the top features with additional consideration of calculation speed to make the model faster and easier to use. Finally, a generalization test was conducted using the “leave one out” (LOO) method, wherein a specific subject’s data were withheld from the training dataset to assess the model’s performance on unseen data. To ensure algorithm quality even further, we tested the model’s generalizability between different annotators, by training the model on one expert data and test on the other. Additionally, we compared the model’s performance in the LOO test against both the clinical software and a multi-threshold algorithm commonly used in iEEG studies35,50. This method detects IEDs by identifying events where the envelope of the high-pass filtered (>250 Hz) time-course exceeds a +5 SD threshold or where both the amplitude and gradient surpass a +5 SD threshold, as well as a duration criterion (shorter than 70 ms). To this end, we performed automatic detection by the algorithm and the software on each channel. We exported annotations with one-second precision and converted the neurologist’s annotations to the same precision for a proper comparison. We then balanced the ratio between normal and abnormal epochs, and calculated the software metrics in the same approach as for our model, employing the scikit-learn python package. The 5-fold cross-validation average metrics are: accuracy = 94.3, precision = 94.4, sensitivity = 94.3, specificity = 94.4, f1 score = 94.3, ROCAUC = 94.3, PRAUC = 91.8 (Fig. 3C, top). The LOO test average metrics are accuracy = 91.2, precision = 91.2, sensitivity = 86.7, specificity = 93.1, f1 score = 87.5, ROCAUC = 89.9, PRAUC = 85.1, and the generalizability test achieved accuracies of 86% and 89%, indicating good generalizability across annotations from different experts and datasets from different medical centers. In addition, we found that our model outperformed the clinical software and the multi threshold algorithm (Fig. 3C, bottom). Notably, the software was trained on non-invasive data, making it likely less relevant for this purpose. SHAP analysis identified the top features as peak-to-peak amplitude, entropy and teager-kaiser parameters (Fig. 3D). Event-related potential (ERP) of detected IEDs In addition to quantitative metrics assessing the model’s performance, and to demonstrate potential utilization of the dataset and the detection method, we conducted additional analysis by generating a typical waveform for each channel using data recorded throughout the entire night. We applied the detection model to all channels of each patient, after manually excluding noisy channels. Detected epochs with model confidence higher than 80% were expanded to include 125 ms before and after the original 250 ms window, and centered around the peak that was identified by the highest amplitude. The duration of each event was calculated as the time between half amplitude before the peak and half amplitude after the peak51. Only events with a duration of 20–100 ms and amplitude of at least 3 STD were used for subsequent analysis. Finally, average event-related potentials were generated for each channel (Fig. 3E). Usage Notes The dataset is available in BIDS format, encompassing all essential information required for researchers to conduct their analyses using any software platform.