Learning Time-Scale Invariant Population-Level Neural Representations

Eshani Patel Yisong Yue Geeling Chau, Computing & Mathematical Sciences, Computation & Neural SystemsCalifornia Institute of Technology , Pasadena, CA 91125 [email protected] {yyue, gchau}@caltech.edu · 2025

Read Learning Time-Scale Invariant Population-Level Neural Representations by Eshani Patel Yisong Yue Geeling Chau, Computing & Mathematical Sciences, Computation & Neural SystemsCalifornia Institute of Technology , Pasadena, CA 91125 [email protected] {yyue, gchau}@caltech.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13022.txt.

Introduction

Eshani Patel, Yisong Yue, and Geeling Chau fix a fragility in base models for brain signals. Such models learn representations from many recording channels at once, and those codes decode downstream tasks well, until the preprocessing changes. If the time scale, how finely the signal is binned or sampled, differs between pretraining and use, scores drop. The authors show systematically that current representations are not invariant to that mismatch. Their remedy is time-scale augmented pretraining, which trains the model across varied time scales so it stops depending on the choice.

Across decoding tasks, the augmented pretraining consistently improved robustness to time-scale differences and built measurable invariance into the representation space. The broader claim: real laboratory preprocessing diversity is a variation the model must be built to absorb.

Catalog record

CREATOREshani Patel Yisong Yue Geeling Chau, Computing & Mathematical Sciences, Computation & Neural SystemsCalifornia Institute of Technology , Pasadena, CA 91125 [email protected] {yyue, gchau}@caltech.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Time-Scale Invariant Population-Level Neural Representations Eshani Patel Yisong Yue Geeling Chau, Computing & Mathematical Sciences, Computation & Neural SystemsCalifornia Institute of Technology , Pasadena, CA 91125 [email protected] {yyue, gchau}@caltech.edu Abstract General-purpose foundation models for neural time series can help accelerate neuroscientific discoveries and enable applications such as brain computer interfaces (BCIs). A key component in scaling these models is population-level representation learning, which leverages information across channels to capture spatial as well as temporal structure. Population-level approaches have recently shown that such representations can be both efficient to learn on top of pretrained temporal encoders and produce useful representations for decoding a variety of downstream tasks. However, these models remain sensitive to mismatches in preprocessing, particularly on time-scales, between pretraining and downstream settings. We systematically examine how time-scale mismatches affects generalization and find that existing representations lack invariance. To address this, we introduce Time-scale Augmented Pretraining (TSAP), which consistently improves robustness to different time-scales across decoding tasks and builds invariance in the representation space. These results highlight handling preprocessing diversity as a key step toward building generalizable neural foundation models. \workshoptitle Foundation Models for the Brain and Body 1 Introduction Building general-purpose representations of neural time series data is a foundational goal for neuroscience research. High-fidelity neural recordings such as intracranial electroencepholography (iEEG) capture complex activity patterns across multiple brain regions, but present significant modeling challenges due to inter-subject and session variability, and limited dataset sizes ( 8 ; 10 ) . As a result, many neuroscience studies and brain computer interface research use single-channel or subject-specific models, limiting their expressivity and generalizability ( 23 ; 19 ; 13 ; 21 ) . Recent advances in self-supervised learning have inspired the development of foundation models for neural signals. Time-series foundation modeling and large-scale univariate pretraining provide robust and rich temporal embeddings for downstream decoding ( 7 ; 1 ; 18 ; 20 ; 14 ) . However, one major limitation is a lack of learned spatial information at the level of populations of channels. Population-level pretraining on top of these temporal embeddings is one promising approach to provide the missing spatial component, demonstrating strong performance on downstream tasks while being computationally and sample-efficient to train ( 4 ; 15 ) . Here we will focus on improving models that learn population-level representations from temporal encoders to enable efficient and generalizable scaling for neuroscience foundation models. A key design limitation remains in this approach: the population-level layer is trained exclusively on the outputs of the temporal encoders. As a consequence, we empirically observe that these models are sensitive to preprocessing parameters used for input signals–such as the time-scales used for training. In practice, neural recordings vary widely in length across datasets and tasks ( 16 ; 21 ; 6 ) , so building invariance into our models along this dimension is important. In this work, we quantify the degradation we get from ignoring these preprocessing mismatches, and explore techniques to remedy this. In particular, we propose a new strategy: Time-scale Augmented Pretraining (TSAP). Applied to iEEG decoding tasks, TSAP consistently improves generalization across seen and unseen time-scales, outperforming interval-specific baselines ( Figure 1 b). Our analysis further demonstrates that TSAP reduces time-scale clustering in embedding space, leading to more invariant and transferable representations. Addressing this limitation is critical for realizing the promise of foundation models that are applicable across a wide range of tasks and experimental settings. Our contributions are: 1. A comparative evaluation of decoding performance on mismatched time-scales, confirming that pretraining and finetuning on the same interval length leads to better performance. 2. A novel Time-scale Augmented Pretraining (TSAP) strategy that exposes the model to a spectrum of interval lengths, improving generalization, even to unseen lengths. 3. Analysis of the embeddings from different time-scales across pretrained models. Figure 1: Schematic of our approach . (a) Previous population-level transformer (PopT) ( 4 ) pretrain on fixed temporal windows, which opens up the question of how sensitive these models are to inputs of different time-scales. (b) Our approach seeks to provide optimal performance for any input length. (c) Time-scale Augmented Pretraining (TSAP) use samples from varying input interval lengths to achieve invariance to input time-scales. 2 Methods Population Transformer Framework. To investigate the time-scale invariance of spatial aggregation pretrained transformers, we adopt the Population Transformer (PopT) ( 4 ) as our core framework, using architectural parameters and training configuration described in the original work. At a high level, for each electrode channel, c , a given interval i of length l (where l can be varied) is pushed through a frozen temporal encoder, in this case we use BrainBERT ( 20 ) . These temporal embeddings are summed with a positional embedding derived from its 3D electrode coordinates ( Figure 1 c), before being passed through the transformer encoder, yielding spatially contextualized channel representations and an aggregated [CLS] output. This [CLS] is then projected with a linear layer for downstream decoding. We adjust the PopT pretraining strategy below. Augmented Data and Training. To adapt the PopT framework for temporal invariance, we modified the data generation pipeline to expose the model to iEEG signals at multiple interval lengths. Specifically, we sampled recording segments of lengths l\in{1,2,4,5} seconds, with a fixed gap g between consecutive windows for each input channel. Each interval was independently encoded into BrainBERT embeddings, yielding distinct temporal representations for the same underlying signal at different scales. These embeddings include overlapping windows across interval lengths, which the temporal encoder maps into non-identical representations. This augmentation strategy, TSAP, encourages the model to generalize across multiple time-scales. Embedding analysis. To better understand the distributions of temporal embeddings and PopT representations, we perform 2D PCA on the embedding spaces. We take 100 samples from a specific subject-session from the Word Onset task, across the time-scales 1, 2, 3, 4, and 5 seconds. For temporal embedding PCA analysis, each sample is represented as a concatenation of all the channels embeddings, producing high-d n_{chan}*h_{dim} vectors. For pretrained PopT [CLS] token analysis, each sample is pushed through pretrained 5s and TSAP PopT models, producing the processed [CLS] token representations. To analyze the clustering, we performed K-Means clustering with 5 means, aligned the clusters to the true classes based on the mode interval of points assigned to the cluster, and plotted the confusion matrices of cluster assignments to true intervals. Figure 2: Performance drop from mismatch in input time-scales is recovered by TSAP. (a) Compared to the optimal (dotted line), models (x-axis) trained with mismatched time-scales perform much worse (below the line), while TSAP (dark blue) generally improves upon the optimal baseline. Shown are the Word Onset ROC AUC difference means and standard error across subjects and 5 seeds. (b) We see TSAP (dark blue) match or outperform all other models across all input lengths. Shown are the Word Onset ROC AUC mean and standard error across subjects and 5 seeds. 3 Experiments Data. We used iEEG, a type of neural time-series data collected via probes implanted within the brain to record local electrical signals at high temporal resolution and spatial precision. We used the publicly available BrainTreeBank dataset ( 21 ) . Data was collected from 10 subjects (total 1,688 electrodes, mean 167 electrodes per

Read the complete text