Mentality: A Mamba-based Approach towards Foundation Models for EEG
Saarang Panchavati, Corey Arnold & William Speier · 2025
Read Mentality: A Mamba-based Approach towards Foundation Models for EEG by Saarang Panchavati, Corey Arnold & William Speier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02746.txt.
Introduction
Saarang Panchavati, Corey Arnold, and William Speier present a 2025 biomedical AI paper exploring foundation models for EEG, the brain wave recordings used to diagnose epilepsy and other disorders. EEG data is noisy, high-dimensional, and strongly nonlinear, which limits older automated methods.
Their model uses a Mamba architecture, a selective state space design built for long sequences. Training first teaches the model to reconstruct masked EEG in a self-supervised way, then tunes it for seizure detection on a large set of seizure and non-seizure recordings. Ranking accuracy on test data held out from training was solid but not spectacular. The authors frame the work as a step toward large, clinically usable EEG foundation models, rather than a finished diagnostic tool, with scale and generality as the main goals.
Catalog record
| CREATOR | Saarang Panchavati, Corey Arnold & William Speier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02746
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mentality: A Mamba-based Approach towards Foundation Models for EEG Saarang Panchavati, Corey Arnold & William Speier Abstract This work explores the potential of foundation models, specifically a Mamba-based selective state space model, for enhancing EEG analysis in neurological disorder diagnosis. EEG, crucial for diagnosing conditions like epilepsy, presents significant challenges due to its noisy, high-dimensional, and nonlinear nature. Traditional machine learning methods have made advances in automating EEG analysis but often fail to capture its complex spatio-temporal dynamics. Recent advances in deep learning, particularly in sequence modeling, offer new avenues for creating more generalized and expressive models capable of handling such complexities. By training a Mamba-based model on a large dataset containing seizure and non-seizure EEG recordings through a self-supervised reconstruction task followed by a seizure detection task, we demonstrate the model’s potential, achieving an AUROC of 0.72 on a held-out test set. This approach marks a significant step toward developing large-scale, clinically applicable foundation models for EEG data analysis. 1 Introduction Electroencephalography (EEG), a vital, noninvasive tool for recording brain electrical activity, plays a key role in diagnosing and treating neurological disorders such as epilepsy ( Acharya et al. 2013 ) . Scalp EEG signals are noisy measurements of brain activity and are characterized by high inter-patient variability, making it challenging to analyze and interpret them ( Acharya et al. 2013 ) . Current clinical practice requires time-consuming, manual inspection for diagnosis ( Acharya et al. 2013 ) . While traditional machine learning models have helped to automate EEG analysis ( Hosseini et al. 2021 ) , they typically struggle with the high-dimensional, non-linear nature of the signal and fail to capture its spatio-temporal dynamics effectively. More expressive deep learning-based models may offer a solution to improve EEG analysis. In particular, foundation models are large-scale deep learning models trained on extensive datasets to generalize across various tasks. Such models have been proposed in medicine ( Moor et al. 2023 ) , but work on developing these large models in a clinical context remains limited. Recent advances in sequence modeling, specifically with selective state space modeling offer a unique opportunity for novel approaches to foundation models applied to biomedical data such as EEG. In particular, Mamba ( Gu & Dao 2023 ) has emerged as a powerful selective state space model with comparable performance to transformer-based approaches on language modeling, genomics, and audio. In this work, we aim to build a foundation model for EEG, first by relaxing the problem to a more focused context. We train a Mamba-based model on a large public dataset that contains both seizure and non-seizure EEG recordings, first on a self-supervised reconstruction task, and then a downstream seizure detection task. 2 Methods 2.1 Data We train and test our data on the Temple University Hospital EEG Seizure Corpus (TUSZ) v2.0.1 ( Shah et al. 2018 ) , which is one of the largest annotated seizure corpora to date. The training data subset contains 579 patients, with 2,138 seizure events. The testing subset contains a disjoint set of 43 patients, with 469 seizure events. We included 19 channels from the standard 10-20 EEG montage, all referenced to a reference electrode. All signals are resampled to 200 Hz, and we remove 60 and 120 Hz interference via a notch filter. Data are segmented into 10s non-overlapping windows. Windows are classified as “seizure” or “not seizure” if there is a seizure event in a window. Figure 1: Proposed Model Architecture. Inspired by SaShiMi, U-Net, and EEGNet, the model learns a state space representation of frequency-based filters, before downsampling to a smaller representation. The model upsamples back to reconstruct the signal, learning state space representations at each level of sampling till a final output. 2.2 Model Architecture and Training Mamba is a deep selective state space model proposed by Gu & Dao 2023 , capable of analyzing long-range dependencies in sequences. Mamba has been shown to outperform many state-of-the-art tasks in long-context modeling including audio and language. In this work we utilize Mamba “blocks”; which include a Mamba layer followed by layer normalization and a residual connection, which can be stacked together similarly to transformer blocks. In this work, we take inspiration from EEGNet ( Lawhern et al. 2018 ) , and SaShiMi ( Goel et al. 2022 ) for a full encoder/decoder architecture with Mamba blocks. EEGNet is a lightweight CNN-based model for processing EEG data. SaShiMi is a U-Net ( Ronneberger et al. 2015 ) like architecture interspersed with S4 blocks (a precursor to Mamba), where downsampling and upsampling are done by reshaping rather than repeated convolution and pooling. Our model begins with a 1D CNN layer of kernel size 100 (half the sampling rate of the signal) to learn “frequency-based” filters up to 50 Hz from each channel. Subsequently, a “channel mixing” linear layer is applied to try to learn the relationships between the channels. This is followed by several Mamba blocks to learn a state space representation of the data, emphasizing the temporal dynamics. The output is then downsampled through the traditional U-Net block, composed of a double convolution and mean pooling. We then pass the downsampled input through another set of Mamba blocks, to obtain a downsampled hidden representation of the data. During reconstruction, this is then upsampled through repeated Mamba-transpose convolution - -double convolution blocks. Following the strategy of the U-Net, we concatenate the output of the corresponding downsampling layer to form residual connections with each upsampling layer output to preserve features. We also add a final convolutional layer initialized with zeros to improve sparsity in the final output. In the downstream task, the downsampled hidden representation is max-pooled along the temporal dimension and then passed through linear layers to obtain a classification probability. To improve the reconstruction, we employ a combination of mean-squared-error (MSE) loss with a spectral loss, which computes the loss in the Fourier domain. 3 Results 3.1 Pretraining Our best model achieves a MSE of 0.0063. Removing the spectral loss resulted in a four-fold increase in MSE (0.025). Figure 2 demonstrates an example of the reconstruction on an example in the test data. Figure 2: Example reconstruction of input sequence on a test sample. Predicted time series are dashed, while ground truth are solid lines. The model captures the general patterns in the time series very well but sometimes fails to completely capture higher frequency components of the signals. The model with spectral loss reconstructs the signal much better than that without. 3.2 Seizure Detection We find that using a pretrained model followed by two linear layers, we can achieve a detection AUROC of 0.72. When we use a model trained from scratch, the detection AUROC drops to 0.64. 3.3 Model Interpretability A limitation of Mamba blocks is that they do not have a way to analyze the interpretability of their representations. In this work, we rely on analyzing model weights and saliency maps for interpretability. To identify channel importance we compute channel-wise saliency based on particular inputs. Figure 3 demonstrates a manner of highlighting channel importance for a seizure sample. We can also look at the weights of the initial convolutional layer (similar to the EEGNet approach) to identify what types of patterns or frequencies are activated