NLP4Neuro: Sequence-to-sequence learning for neural population decoding

Jacob J. MorraKaitlyn E. FoukeKexin HangZichen HeOwen Traubert [0.1em] Timothy W. DunnEva A. Naumann [0.4em] Duke University, Durham NC 27705, USA · 2025

Read NLP4Neuro: Sequence-to-sequence learning for neural population decoding by Jacob J. MorraKaitlyn E. FoukeKexin HangZichen HeOwen Traubert [0.1em] Timothy W. DunnEva A. Naumann [0.4em] Duke University, Durham NC 27705, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02264.txt.

Introduction

Jacob Morra and co-authors test whether language models can read brain activity in a 2025 study called NLP4Neuro. Their data pair brain wide calcium imaging with behavior in larval zebrafish shown moving visual scenes. Large language models learn sequence patterns from text, and the team asks if that skill transfers to neural data.

Models with text training beat those trained from scratch at decoding behavior from brain signals. One recent model with a mixed expert design scored best, forecast tail moves over long spans, and flagged brain zones in ways that fit anatomy. The work hints that text trained models can aid the study of whole brain circuits.

Catalog record

CREATORJacob J. MorraKaitlyn E. FoukeKexin HangZichen HeOwen Traubert [0.1em] Timothy W. DunnEva A. Naumann [0.4em] Duke University, Durham NC 27705, USA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NLP4Neuro: Sequence-to-sequence learning for neural population decoding Jacob J. MorraKaitlyn E. FoukeKexin HangZichen HeOwen Traubert [0.1em] Timothy W. DunnEva A. Naumann [0.4em] Duke University, Durham NC 27705, USA Abstract Delineating how animal behavior arises from neural activity is a foundational goal of neuroscience. However, as the computations underlying behavior unfold in networks of thousands of individual neurons across the entire brain, this presents challenges for investigating neural roles and computational mechanisms in large, densely wired mammalian brains during behavior. Transformers, the backbones of modern large language models (LLMs), have become powerful tools for neural decoding from smaller neural populations. These modern LLMs have benefited from extensive pre-training, and their sequence-to-sequence learning has been shown to generalize to novel tasks and data modalities, which may also confer advantages for neural decoding from larger, brain-wide activity recordings. Here, we present a systematic evaluation of off-the-shelf LLMs to decode behavior from brain-wide populations, termed NLP4Neuro, which we used to test LLMs on simultaneous calcium imaging and behavior recordings in larval zebrafish exposed to visual motion stimuli. Through NLP4Neuro, we found that LLMs become better at neural decoding when they use pre-trained weights learned from textual natural language data. Moreover, we found that a recent mixture-of-experts LLM, DeepSeek Coder-7b, significantly improved behavioral decoding accuracy, predicted tail movements over long timescales, and provided anatomically consistent highly interpretable readouts of neuron salience. NLP4Neuro demonstrates that LLMs are highly capable of informing brain-wide neural circuit dissection. 1 Introduction Predicting behavior from neuron-scale network activity, or neural decoding, remains an open challenge in computational neuroscience; one which promises to improve our understanding of the function of neural circuitry in various model organisms, characterize individual differences, and advance our capacity to build brain-computer interfaces for treating neural dysfunctions affecting brain-wide circuits, such as in various neurodevelopmental or neurodegenerative disorders [ 45 , 31 ] . Machine learning (ML) is an increasingly used tool for neural decoding tasks, for two reasons: First, ML can be applied to myriad domains without the level of expertise required to build models from scratch; second, it achieves high fidelity performance across task types [ 31 ] . In other words, it works well as a general-purpose tool and requires minimal user interference. Taking these benefits further, recent progress in ML has leveraged pre-training; that is, learning from an often very large dataset to acquire generalized knowledge, and fine-tuning in order to sharpen that knowledge towards a new task (transfer learning). This trend is particularly apparent in natural language processing (NLP), wherein it has been shown that generalization is an emergent property of scale in off-the-shelf large language models (LLMs) [ 39 , 53 , 28 , 54 ] . It is an open question whether this emergence carries over to neural decoding, i.e. as opposed to using simpler [ 16 ] , linear [ 59 ] , or recurrent models; the latter being previously widely used in both NLP [ 52 , 17 ] and neuroscience [ 11 , 12 , 27 , 46 , 6 , 32 ] . However, many contemporary approaches in neural decoding specifically have moved to larger models with higher parameter counts, away from recurrent neural networks (RNNs) and towards transformers, as in [ 55 , 56 , 58 , 50 , 9 ] . Moreover, progress in NLP is expeditious. With LLMs reporting impressive abilities to transfer their language knowledge across domains – i.e. in generating candidate protein binders [ 10 ] , suggesting novel material design architectures [ 8 ] , and learning implicit graph structures [ 57 ] – and the continued release of open-access, state-of-the-art models, pre-trained on billions of tokens [ 22 ] , there is an opportunity to develop a modular, LLM-driven pipeline with the goal of advancing neural decoding. We present a two step process, NLP4Neuro: In the first, model-building stage (Fig. 1(b) ), we select from off-the-shelf LLMs, and evaluate their capacity to decode behavior from neural data. That is, to learn a function that maps, for a given sequence length s\in\{5,10,15,20\} , s tokens of neural population activity to s behavior outputs, measured simultaneously in a known model organism, the larval zebrafish . Using data acquired from simultaneous two-photon calcium imaging and head-fixed tail behavior tracking, neuron-scale population activity is matched to high-frequency tail kinematics. This stage can continue to be updated and improved with the addition of state-of-the-art models. In the second, model-interpretation stage, we use the best model as a practical tool for finding features (neurons) which contribute most to observed behavior. In particular, we use salience mapping from the model weights to inform each neuron’s spatial location and regional information [ 24 , 14 , 40 ] . We ground our approach in four experiments: In the first three experiments, we compare model paradigms, their pre-trained weights, and various token embedding strategies, to determine the resulting impacts on neural decoding. In experiment 1, we add linearly-trained and nonlinear RNNs to the model pipeline, alongside LLMs, to contrast RNNs and transformers, with varying parameter counts. In the second experiment, we assess the value of pre-trained weights for neural decoding. In the third experiment, we evaluate reasonable embedding strategies from the NLP domain (see Ssec. 3.3 ). Pre-trained weights and input embeddings are fundamental LLM features [ 51 ] . In the fourth experiment, we assess the viability of our best model as an explainable neural decoding tool. For RNN models, we first consider the reservoir computer (RC), which is best-in-class for modeling dynamical systems, whilst being linearly-trained and lightweight [ 19 ] . It is also used in NLP, i.e. in comprehension of narrative sequence structure [ 17 ] . We also consider the long short-term memory network (LSTM), which was previously state-of-the-art in natural language inference [ 52 ] . For LLMs, we consider variations of the transformer which include encoder-only models, e.g. Bidirectional Encoder Representations from Transformers (BERT) – the base uncased version – from [ 15 ] , decoder-only models, e.g. Generative Pretrained Transformer 2 (GPT-2) from [ 38 ] , and Mixture of Expert (MoE) models, i.e. DeepSeek Coder-7b (-c7b), from [ 22 ] , which performs similarly to GPT-3.5 Turbo in next token prediction, operating with 7b (7 billion) parameters instead of GPT-3.5’s 175b. We report that the MoE transformer, DeepSeek-c7b, achieves the strongest performance across all sequence-lengths except for s=15 , followed by BERT-base-uncased (-bu), whereas the LSTM is the best-performing RNN and achieves competitive scores for other sequence lengths. We find, furthermore, that DeepSeek-c7b outpaces the LSTM by a wider margin when assessing their performance from s=5 to s=20 , i.e. for longer timescales. Evaluating the impact of pre-training on the performance of transformer models alone, we find that pre-trained weights generate lower-error predictions on neural data for the MoE and encoder-only models. Regarding the neural data encoding strategies we employed, we find that relative position encoding improves performance across transformer models, but also that DeepSeek-c7b is insensitive to all strategies. Finally, we use DeepSeek-c7b, the best-performing model, to illustrate the use of NLP4Neuro as a tool for identifying single-neurons as features which may exert a controlling effect on tail behavior through salience mapping. 2 Related work Supervised models with benchmarks

Read the complete text