NeuralSet: A High-Performing Python Package for Neuro-AI
Jean-Rémi King, Corentin Bel, Linnea Evanson, Julien Gadonneix, Sophia Houhamdi, Jarod Lévy, Josephine Raugel, Andrea Santos Revilla, Mingfang (Lucy) Zhang, Julie Bonnaire, Charlotte Caucheteux, Alexandre Défossez, Théo Desbordes, Pablo Diego-Simón, Shubh Khanna, Juliette Millet, Pierre Orhan, Saarang Panchavati, Antoine Ratouchniak, Alexis Thual, Teon L. Brooks, Katelyn Begany, Yohann Benchetrit, Marlène Careil, Hubert Banville, Stéphane d’Ascoli, Simon Dahan, Jérémy Rapin · 2026
Read NeuralSet: A High-Performing Python Package for Neuro-AI by Jean-Rémi King, Corentin Bel, Linnea Evanson, Julien Gadonneix, Sophia Houhamdi, Jarod Lévy, Josephine Raugel, Andrea Santos Revilla, Mingfang (Lucy) Zhang, Julie Bonnaire, Charlotte Caucheteux, Alexandre Défossez, Théo Desbordes, Pablo Diego-Simón, Shubh Khanna, Juliette Millet, Pierre Orhan, Saarang Panchavati, Antoine Ratouchniak, Alexis Thual, Teon L. Brooks, Katelyn Begany, Yohann Benchetrit, Marlène Careil, Hubert Banville, Stéphane d’Ascoli, Simon Dahan, Jérémy Rapin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03169.txt.
Introduction
King and twenty-seven colleagues release a Python package that unifies brain data and AI models. Neuroscience and AI increasingly meet, but the software is fragmented: tools are split by recording type, from brain scans to electrode spikes, and built for small in-memory jobs. NeuralSet splits experimental metadata from lazy, memory-saving data extraction, so it can handle massive natural datasets spanning brain recordings and stimuli like text, audio, and video. One PyTorch-ready interface then covers local tests through cluster runs.
Standard preprocessing steps are harmonized with pretrained deep learning embeddings, and full provenance removes manual data wrangling. The package is built by a large team across Meta, Paris universities, and research institutes, reflecting how broad the neuro-AI community has become. By making diverse recordings and stimuli speak one language, NeuralSet aims to give the next generation of brain-AI research a shared, scalable base.
Catalog record
| CREATOR | Jean-Rémi King, Corentin Bel, Linnea Evanson, Julien Gadonneix, Sophia Houhamdi, Jarod Lévy, Josephine Raugel, Andrea Santos Revilla, Mingfang (Lucy) Zhang, Julie Bonnaire, Charlotte Caucheteux, Alexandre Défossez, Théo Desbordes, Pablo Diego-Simón, Shubh Khanna, Juliette Millet, Pierre Orhan, Saarang Panchavati, Antoine Ratouchniak, Alexis Thual, Teon L. Brooks, Katelyn Begany, Yohann Benchetrit, Marlène Careil, Hubert Banville, Stéphane d’Ascoli, Simon Dahan, Jérémy Rapin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03169
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NeuralSet: A High-Performing Python Package for Neuro-AI Jean-Rémi King, Corentin Bel, Linnea Evanson, Julien Gadonneix, Sophia Houhamdi, Jarod Lévy, Josephine Raugel, Andrea Santos Revilla, Mingfang (Lucy) Zhang, Julie Bonnaire, Charlotte Caucheteux, Alexandre Défossez, Théo Desbordes, Pablo Diego-Simón, Shubh Khanna, Juliette Millet, Pierre Orhan, Saarang Panchavati, Antoine Ratouchniak, Alexis Thual, Teon L. Brooks, Katelyn Begany, Yohann Benchetrit, Marlène Careil, Hubert Banville, Stéphane d’Ascoli, Simon Dahan, Jérémy Rapin Abstract Artificial intelligence (AI) is increasingly central to understanding how the brain processes information. However, the integration of neuroscience and modern AI is bottlenecked by a fragmented software ecosystem. Current tools are siloed by recording modality and optimized for small-scale, in-memory workflows, limiting the use of massive, naturalistic datasets. Here, we introduce NeuralSet, a Python framework that efficiently unifies the processing of diverse neural recordings (including fMRI, M/EEG, and spikes) and complex experimental stimuli (such as text, audio, and video). By decoupling experimental metadata from lazy, memory-efficient data extraction, NeuralSet harmonizes standard neuroscientific preprocessing pipelines with pretrained deep learning embeddings. This approach provides a single PyTorch-ready interface that scales seamlessly from local prototyping to high-performance cluster execution. By eliminating manual data wrangling and ensuring full computational provenance, NeuralSet establishes a scalable, unified infrastructure for the next generation of neuro-AI research. † † affiliation: FAIR, Meta † † affiliation: École Normale Supérieure - PSL Université, Paris † † affiliation: Foundation Adolphe de Rothschild Hospital, Paris † † affiliation: MIND, Inria † † affiliation: Université Paris Cité † † affiliation: Neurospin, CEA, Gif sur Yvette † † contribution: These authors contributed equally. † † contribution: These senior authors contributed equally. † † Code: https://github.com/facebookresearch/neuroai/ † † correspondence: [email protected] , [email protected] Figure 1: A. NeuralSet unifies the processing of neural recordings and the AI embedding of experimental conditions into a single PyTorch DataLoader. B. Specifying the extraction of any types of data requires changing the configuration of a single “extractor”. 1 Introduction The rise of naturalistic Neuro-AI. Neuroscience is undergoing a paradigm shift, transitioning from highly controlled, small-scale experiments to massive, multi-modal, and naturalistic datasets ( 12 ; 25 ) . This evolution is essential for understanding the biological foundations of intelligence in real-world contexts and effectively fuels the emergence of new discipline: Neuro-AI ( 22 ; 31 ) . Yet, the application of deep learning onto neuroscientific data ( 5 ; 6 ; 24 ) is severely bottlenecked by a fragmented software ecosystem, which leads most labs across the world, to re-implement similar home-made data processing workflows. As public datasets reach the terabyte scale 1 1 1 https://openneuro.org/ and experimental protocols increasingly incorporate complex stimuli like continuous speech and video ( 2 ; 11 ; 14 ) , the infrastructure required to process these data has failed to keep pace. A rich but fragmented software landscape. Over the past two decades, the neuroscience community has built a large set of open-source tools that have standardized analysis practices and enabled thousands of studies. Established softwares such as MNE-Python ( 10 ) , EEGLAB ( 7 ) , FieldTrip ( 15 ) , and Brainstorm ( 26 ) for electrophysiology, or Nilearn ( 1 ) and fMRIPrep ( 8 ) for neuroimaging, are the gold standard for signal processing and statistical inference. However, these tools were originally designed for a pre-deep-learning era: they typically rely on eager loading – assuming datasets can fit entirely into RAM – and lack native abstractions to temporally align neural time series with high-dimensional, pretrained embeddings from modern AI frameworks (e.g., HuggingFace transformers ( 30 ) ). Consequently, researchers must build ad-hoc pipelines that require manual data wrangling, manual caching, and complex backend configurations. Structure–data decoupling. To bridge this gap, we introduce NeuralSet, a Python package designed to efficiently unify the processing of diverse neural recordings and their associated experimental conditions. NeuralSet is built upon the principle of structure-data decoupling: it represents the logical structure of any experiment as lightweight, event-driven metadata, separating it entirely from the memory- and compute-intensive extraction of the underlying signals. This architecture enables lazy, on-demand data processing, allowing researchers to filter, recombine, and iterate on massive datasets without loading a single byte of raw data into memory until it is required by a PyTorch dataloader ( 17 ) . A unified, scalable interface. By providing a single, backend-agnostic interface, NeuralSet harmonizes the validated preprocessing stacks of domain-specific libraries with the embedding capabilities of modern AI models. It natively supports a comprehensive array of neural modalities: e.g. fMRI, EEG, MEG, iEEG, fNIRS, EMG, and spike trains, alongside the embedding of naturalistic text, audio, and video stimuli. Critically, its caching and hardware-agnostic execution relies on exca ( 21 ) , which allows researchers to scale seamlessly from local prototyping on a laptop to distributed execution on high-performance computing clusters. Ultimately, NeuralSet is designed to eliminate the friction of multi-task and multi-device data orchestration, providing the scalable and reproducible infrastructure necessary for the next generation of Neuro-AI research. Figure 2: Comparison of neuroscience software packages across neural devices, experimental tasks, and infrastructure features. Filled circles indicate full support, half-filled circles indicate partial or compatible support, and empty circles indicate no support. NeuralSet (bottom row, highlighted) is the only package supporting all categories. 2 Framework NeuralSet enforces a separation between the lightweight description of an experiment and the heavy extraction of data. The framework is organized around five core abstractions – Events , Extractors , Segments and Tensor Data – which together form a pipeline from raw neural and/or stimulus files to PyTorch-ready datasets. Figure 3: The NeuralSet pipeline from user perspective. Each step is shown alongside its Python code snippet, from defining a data source ( Study ) to iterating batched tensors in a standard PyTorch DataLoader . The entire pipeline is lazy: only the final iteration triggers data loading and feature extraction. 2.1 Events: describing what happens and when. The central insight of NeuralSet is that the logical structure of an experiment can be represented independently of the underlying data. Every piece of information – an fMRI run, a word spoken by a narrator, a stimulus image – is modelled as an event : a lightweight object (i.e. a Python dictionary) defined by a type , a start time, a duration , and a timeline (a unique identifier for a continuous recording session). For instance, a one-hour fMRI session in which a participant watches a movie is described by a handful of concurrent events on the same timeline: the fMRI acquisition, the video stimulus, and potentially hundreds of word-onset annotations derived from the soundtrack. A Study object assembles the events of an entire dataset into a single pandas DataFrame. This approach is consistent with, but not restricted to BIDS-compliant datasets ( 9 ) . Because the DataFrame contains only light metadata – not the signals themselves – researchers can explore, filter, and recombine experiments using standard pandas ( 13 ) operations (e.g. selecting all timelines of a given subject, or all word events in a specific language) without loading data into memory. Composable EventsTransform operations can then be chained to enrich, filter, or reorganize events prior to data extraction. Typical transforms include annotating words with their sentence context, assigning cross-validation splits, or chunking long audio and video events into shorter