sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment

Yijun Liu · 2025

Read sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment by Yijun Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14468.txt.

Introduction

Picking the sentence a person just heard from brain signals alone is the goal of a 2025 arXiv paper by Yijun Liu. His framework, called SSENSE, maps recordings from electrodes inside the brain into the sentence space of a fixed language model. Training uses a contrastive loss, and the text side stays frozen.

The test data pairs brain recordings with speech transcripts from a subject watching movies. Data were scarce and came from one person, yet retrieval worked better than chance. Liu argues that ready made language representations can serve as a solid target for brain decoding.

Catalog record

CREATORYijun Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14468

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment Yijun Liu Abstract Interpreting neural activity through meaningful latent representations remains a complex and evolving challenge at the intersection of neuroscience and artificial intelligence. We investigate the potential of multimodal foundation models to align invasive brain recordings with natural language. We present SSENSE , a contrastive learning framework that projects single-subject stereo-electroencephalography (sEEG) signals into the sentence embedding space of a frozen CLIP model, enabling sentence-level retrieval directly from brain activity. SSENSE trains a neural encoder on spectral representations of sEEG using InfoNCE loss, without fine-tuning the text encoder. We evaluate our method on time-aligned sEEG and spoken transcripts from a naturalistic movie-watching dataset. Despite limited data, SSENSE achieves promising results, demonstrating that general-purpose language representations can serve as effective priors for neural decoding. 0 0 footnotetext: This work has been accepted for poster presentation at the CVPR 2025 Workshop on Multimodal Foundation Models (MMFM3). 1 Introduction The ability to decode internal mental content from brain activity—popularly referred to as ”mind-reading”—stands as a key challenge at the intersection of cognitive neuroscience and artificial intelligence. Recent breakthroughs in foundation models have enabled powerful cross-modal alignment between modalities such as vision and language, as demonstrated by CLIP Radford et al. 2021 and ALIGN Jia et al. 2021 . However, the extension of these capabilities to neural data, particularly high-temporal-resolution signals like stereo-electroencephalography (sEEG), remains underexplored. In this work, we investigate how to leverage foundation models for brain-to-language alignment. We propose SSENSE (Subject-wise sEEG-based Encoding for Sentence Retrieval), a framework that maps sEEG recordings into the sentence embedding space of a frozen CLIP model. SSENSE uses spectrogram representations of sEEG, processed by a neural encoder trained with a contrastive loss to align neural signals with their corresponding natural language descriptions. The objective is to enable zero-shot sentence retrieval from brain activity by grounding it in a shared semantic space. We evaluate SSENSE on a dataset comprising time-aligned sEEG recordings and movie transcript sentences collected from a single subject. Despite the limited amount of data, our method achieves promising results on the sentence retrieval task, demonstrating that frozen multimodal language models can serve as effective priors for neural decoding. Our findings provide initial evidence that contrastive brain-language alignment is feasible through embedding into pretrained semantic spaces. This opens new directions for integrating brain signals into the broader landscape of foundation models, with potential applications in cognitive neuroscience, language understanding, and assistive brain-computer interfaces. 2 Methods We train a sEEG encoder to align its neural embeddings with the corresponding sentence embeddings that were presented to the subject during movie watching. The overall framework is illustrated in Fig. 1 . Figure 1 : Proposed SSENSE multimodal framework for aligning sEEG signals with natural language. sEEG segments, zero-padded and transformed via the superlet method into time-frequency representations, are encoded using a dedicated sEEG encoder. Sentence embeddings are obtained from the frozen text encoder of CLIP. The model is trained using a contrastive InfoNCE loss to align sEEG and text representations in a shared embedding space. 2.1 sEEG Preprocessing Raw sEEG signals were first converted into time-frequency representations (spectrograms) using the superlet transform Moca et al. 2021 , a spectral analysis method previously employed in BrainBERT Wang et al. 2023 for self-supervised learning on sEEG data. To standardize input lengths across samples, sEEG segments corresponding to shorter sentences were zero-padded along the time dimension to a fixed length of 8,200 time points (approximately 4 seconds). 2.2 sEEG Encoder To project sEEG recordings into the CLIP sentence embedding space, we design a neural encoder that operates on spectrogram representations of the time-series neural signals. The encoder maps multi-electrode inputs into fixed-dimensional vectors aligned with CLIP’s 512-dimensional sentence embedding space. Given the visual structure of spectrograms, we adopt a modified ResNet-18 architecture He et al. 2016 , pretrained on ImageNet. To accommodate single-channel spectrogram inputs, we adapt the first convolutional layer to accept one input channel, and replace the original classification head with a linear projection that outputs 512-dimensional embeddings. Spectrograms from each electrode are processed independently by the encoder, and the resulting per-electrode embeddings are aggregated via mean pooling across electrodes to form a unified representation per sample. This design enables the model to capture both spatial and temporal patterns within each electrode’s signal for contrastive alignment with natural language. 2.3 Text Encoder We leverage the pretrained CLIP Radford et al. 2021 text encoder (ViT-B/32) as a frozen module to extract semantic representations of natural language stimuli. Each sentence is tokenized using the CLIP tokenizer and passed through the encoder to produce a 512-dimensional embedding. To ensure stable training and effective alignment in the shared embedding space, we apply L2 normalization to all text embeddings. Importantly, the CLIP model remains fixed throughout training, serving as a consistent target for contrastive alignment with brain-derived neural representations. This strategy enables the model to exploit rich language priors captured by large-scale vision-language pretraining while focusing learning capacity on the neural encoder. 2.4 Data Augmentation Strategies To enhance generalization, we explore data augmentation strategies inspired by the time-frequency masking techniques introduced in BrainBERT Wang et al. 2023 . Specifically, we apply time-frequency masking directly to the spectrogram inputs and randomly mask a subset of electrode channels during training. These augmentations aim to improve the model’s robustness to noise and inter-trial variability inherent in neural recordings. Pseudocode for both augmentation strategies is provided below. Algorithm 1 Time-Frequency Masking 1: Input tensor x\in\mathbb{R}^{B\times E\times C\times F\times T} , frequency mask ratio r_{f} , time mask ratio r_{t} 2: for b=1 to B do 3: if r_{f}>0 then 4: Sample M_{f}=\lfloor r_{f}\cdot F\rfloor random frequency indices 5: Set x[b,:,:,M_{f},:]=0 6: end if 7: if r_{t}>0 then 8: Sample M_{t}=\lfloor r_{t}\cdot T\rfloor random time indices 9: Set x[b,:,:,:,M_{t}]=0 10: end if 11: end for 12: Masked tensor x Algorithm 2 Electrode Masking 1: Input tensor x\in\mathbb{R}^{B\times E\times C\times F\times T} , mask ratio r 2: for b=1 to B do 3: Sample M=\lfloor r\cdot E\rfloor random electrode indices 4: Set x[b,\text{masked electrodes},:,:,:]=0 5: end for 6: Masked tensor x Method Recall@1 Recall@10 Recall@50 MRR SSENSE no masking 1.2 \pm 0.54 10.68 \pm 1.40 40.58 \pm 2.29 0.0498 \pm 0.0049 with electrode masking 0.72 \pm 0.25 9.00 \pm 1.21 40.00 \pm 1.76 0.0446 \pm 0.0044 with frequency and time masking 0.99 \pm 0.44 9.65 \pm 1.41 38.69 \pm 2.75 0.0475 \pm 0.0062 with electrode and frequency and time masking 1.06 \pm 0.57 9.38 \pm 1.78 40.37 \pm 1.78 0.0465 \pm 0.0077 Random sentence retrieval 0.34 3.42 17.12 0.0214 Table 1 : Sentence retrieval performance from sEEG signals. Mean and standard deviation were computed over 10 random seeds with different dataset splits. The training, validation, and test sets were fixed at 60%, 20%, and 20%, respectively. 2.5 Loss function We used InfoNCE Oord et al. 2018 to maximize the consine similarity between the positive sEEG-sentence pairs (sEEG and sentence recorded at the same time). Given a batch of N paired samples \{(z_{i}^{\text{sEEG}},z_{i}^{\text{text}})\}_{i=1}^{N} , where z_{i}^{\text{sEEG}} is the embedding of the sEEG input and z_{i}^{\text{text}} is the embedding of the corresponding text (from a frozen encoder), the InfoNCE loss for a positive pair (i) is: \mathcal{L}_{i}=-\log\frac{\exp\left(\frac{\text{sim}(z_{i}^{\text{sEEG}},z_{i}^{\text{text}})}{\tau}\right)}{\sum_{j=1}^{N}\exp\left(\frac{\text{sim}(z_{i}^{\text{sEEG}},z_{j}^{\text{text}})}{\tau}\right)} (1) where

Read the complete text