The Challenge of Out-Of-Distribution Detection in Motor Imagery BCIs
Merlijn Quincent Mulder 1 \orcid 0009-0002-0880-1059, Matias Valdenegro-Toro 1 \orcid 0000-0001-5793-9498, Andreea Ioana Sburlea 1 \orcid 0000-0001-6766-3464 and Ivo Pascal de Jong 1∗ \orcid 0000-0002-1497-8013 · 2026
Read The Challenge of Out-Of-Distribution Detection in Motor Imagery BCIs by Merlijn Quincent Mulder 1 \orcid 0009-0002-0880-1059, Matias Valdenegro-Toro 1 \orcid 0000-0001-5793-9498, Andreea Ioana Sburlea 1 \orcid 0000-0001-6766-3464 and Ivo Pascal de Jong 1∗ \orcid 0000-0002-1497-8013 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13324.txt.
Introduction
Merlijn Quincent Mulder and co-authors test out-of-distribution detection in motor imagery brain-computer interfaces. Classifiers trained on set classes can only guess blindly when new kinds of EEG data arrive. Such samples should be caught and rejected, not guessed at, the team argues. They train on some classes and test seven detection methods on unfamiliar ones. High uncertainty in EEG makes the task harder here than in most domains, they find. MC Dropout worked best, and strong in-class accuracy predicted strong detection.
Catalog record
| CREATOR | Merlijn Quincent Mulder 1 \orcid 0009-0002-0880-1059, Matias Valdenegro-Toro 1 \orcid 0000-0001-5793-9498, Andreea Ioana Sburlea 1 \orcid 0000-0001-6766-3464 and Ivo Pascal de Jong 1∗ \orcid 0000-0002-1497-8013 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55215 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13324
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Challenge of Out-Of-Distribution Detection in Motor Imagery BCIs Merlijn Quincent Mulder 1 \orcid 0009-0002-0880-1059, Matias Valdenegro-Toro 1 \orcid 0000-0001-5793-9498, Andreea Ioana Sburlea 1 \orcid 0000-0001-6766-3464 and Ivo Pascal de Jong 1∗ \orcid 0000-0002-1497-8013 Abstract Machine Learning classifiers used in Brain-Computer Interfaces make classifications based on the distribution of data they were trained on. When they need to make inferences on samples that fall outside of this distribution, they can only make blind guesses. Instead of allowing random guesses, these Out-of-Distribution (OOD) samples should be detected and rejected. We study OOD detection in Motor Imagery BCIs by training a model on some classes and observing whether unfamiliar classes can be detected based on increased uncertainty. We test seven different OOD detection techniques and one more method that has been claimed to boost the quality of OOD detection. Our findings show that OOD detection for Brain-Computer Interfaces is more challenging than in other machine learning domains due to the high uncertainty inherent in classifying EEG signals. For many subjects, uncertainty for in-distribution classes can still be higher than for out-of-distribution classes. As a result, many OOD detection methods prove to be ineffective, though MC Dropout performed best. Additionally, we show that high in-distribution classification performance predicts high OOD detection performance, suggesting that improved accuracy can also lead to improved robustness. Our research demonstrates a setup for studying how models deal with unfamiliar EEG data and evaluates methods that are robust to these unfamiliar inputs. OOD detection can improve the overall safety and reliability of BCIs. keywords Out-Of-Distribution Detection, Uncertainty, Machine Learning, Brain Computer Interfaces, Motor Imagery 1 Introduction Machine Learning models make mistakes when they are tested on data that is dissimilar to what the models were trained on. Especially under semantic shifts – where a test sample belongs to a class that the model was never trained on – the model cannot possibly give a correct classification. Instead of giving an incorrect classification, a robust and reliable Machine Learning model should detect these samples that are out-of-distribution (OOD) with respect to the training data, and abstain from making a classification on them. Machine Learning models used in Motor Imagery (MI) Brain-Computer Interfaces (BCIs) are also subject to this limitation. They are typically trained and evaluated with electroencephalography (EEG) data from two (e.g. left-hand, right-hand) to five (e.g. left, right, foot, tongue, rest) classes and are deployed with the assumption that all signals later must match one of these classes. In reality, it is very likely that users might perform actions not matching any of these classes at some point during usage [ 5 ] . Those OOD actions should be detected so that a device can abstain from making erroneous actions and instead prefer inaction. The basic mechanism through which a Machine Learning model can detect OOD samples is through uncertainty. By indicating high uncertainty estimates for OOD samples, and low uncertainty estimates for in-distribution (ID) samples, the model can indicate whether a sample is likely to be OOD or not. Achieving good separability between ID and OOD samples has only had limited attention in BCI research [ 5 , 8 ] , but in the broader Machine Learning literature, this topic has already attracted significant attention [ 7 ] . To achieve good OOD detection, various methods have been proposed. A unified discussion of these methods is presented in [ 32 ] . [ 18 ] highlighted that these different methods yield varying performance across tasks. Hence, there is a need for task-specific approaches when using uncertainty estimation methods for OOD detection. Notably, most methods have been developed for image classification problems, which risks that they cannot be applied to other domains with the same promise of performance. Particularly EEG is known to have large amounts of noise, which can lead to generally higher uncertainty estimates for the ID data, making OOD detection exceptionally difficult [ 8 ] . There has already been some interest in the task of OOD detection in EEG. [ 31 ] studied uncertainty and OOD detection in epilepsy, and proposed a novel anomaly detection method using deep support vector data description and model uncertainty, showcasing an increase in the robustness of a model trained for seizure detection. However, they used OOD detection to identify patients that were dissimilar to the training data for which the model would not work well. Detecting samples within-subject that are OOD requires per-sample OOD detection. [ 29 ] extensively studied uncertainty estimation under realistic EEG distribution shifts based on simulated instrumentation-related variabilities, but not brain-related variabilities, nor inference under unfamiliar classes. Figure 1 : Concept of out-of-distribution detection. A model is trained on several known classes. It has to be able to recognize when unknown data does not belong to any of the known classes, while still classifying the known classes correctly. [ 5 ] [ 5 ] previously studied OOD detection for Motor Imagery BCIs. They showed that OOD detection for BCI using Bayesian techniques, such as Markov Chain Monte Carlo samplers, is satisfactory only for some users. They argued that the use of effective prior distributions from computer vision is possibly not well-suited for BCI tasks, explaining the difference in OOD detection performance between the domains. Furthermore, classification and OOD detection are subject to the influences of the noise and variability in the signals between different distributions, making OOD detection more difficult for BCI tasks. Another factor is that patterns can differ between subjects, requiring within-subject training pipelines for good performance. The small, highly variable datasets further complicate OOD detection [ 21 ] . [ 5 ] focused on Bayesian approaches for OOD detection in BCIs, and it is unclear whether the methods developed in other domains transfer effectively to BCI tasks. The current state of the art for OOD detection in BCIs remains unknown, as do the factors that explain its performance. In this study we establish the state of the art of OOD detection for BCIs, exploring a wide range of OOD detection methods for Convolutional Neural Networks (CNN) used in BCIs to identify which methods have the biggest impact on improving BCI robustness. We additionally investigate which factors have an impact on OOD detection performance, looking at in-distribution classification performance, specific classes that may be easier or harder to detect, and the activation strength of the penultimate layer in the neural network. Through this, we establish the current state-of-the-art, explain why OOD detection is exceptionally difficult in BCIs, and show how OOD detection may improve as on-task performance improves. To this end, we benchmark seven OOD detection techniques spanning distance-based, Bayesian-based, and density-based approaches as defined in Section 2.2 . We will use three different datasets (from [ 27 , 22 , 23 ] ), available through Mother of all BCI Benchmarks (MOABB) [ 3 ] . As our CNN, we use EEGNeX, an optimised version of EEGNet that proved a reliable EEG signal decoding CNN-based architecture [ 4 ] . 1.1 Contributions By studying OOD detection in Motor Imagery BCIs, we make the following contributions: • We demonstrate leave-one-class-out OOD detection as a challenging and useful experimental setup to simulate unfamiliar inputs that a BCI model will be exposed to in practice. •