Learnable latent embeddings for joint behavioural and neural analysis.

Schneider S, Lee JH, Mathis MW. · 2023

Read Learnable latent embeddings for joint behavioural and neural analysis. by Schneider S, Lee JH, Mathis MW. (2023) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc10172131.txt.

Introduction

Schneider, Lee, and Mathis present a method for analyzing animal behavior and brain activity together, in a 2023 study. Linking what an animal does to what its neurons do is a core goal of neuroscience. As recordings grow larger, scientists need models that can find shared patterns across both kinds of data.

Their method learns compact embeddings, or summaries, that capture behavior and neural signals in one frame. It can then relate movements to brain states in fine detail. The tool, called CEBRA, opens new ways to study how brains produce adaptive behavior.

Catalog record

CREATORSchneider S, Lee JH, Mathis MW.
YEAR2023
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC10172131

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learnable latent embeddings for joint behavioural and neural analysis By Steffen Schneider, Jin Hwa Lee, Mackenzie Weygandt Mathis Published in: Nature DOI: 10.1038/s41586-023-06031-6 ABSTRACT Mapping behavioural actions to neural activity is a fundamental goal of neuroscience. As our ability to record large neural and behavioural data increases, there is growing interest in modelling neural dynamics during adaptive behaviours to probe neural representations1–3. In particular, although neural latent embeddings can reveal underlying correlates of behaviour, we lack nonlinear techniques that can explicitly and flexibly leverage joint behaviour and neural data to uncover neural dynamics3–5. Here, we fill this gap with a new encoding method, CEBRA, that jointly uses behavioural and neural data in a (supervised) hypothesis- or (self-supervised) discovery-driven manner to produce both consistent and high-performance latent spaces. We show that consistency can be used as a metric for uncovering meaningful differences, and the inferred latents can be used for decoding. We validate its accuracy and demonstrate our tool’s utility for both calcium and electrophysiology datasets, across sensory and motor tasks and in simple or complex behaviours across species. It allows leverage of single- and multi-session datasets for hypothesis testing or can be used label free. Lastly, we show that CEBRA can be used for the mapping of space, uncovering complex kinematic features, for the production of consistent latent spaces across two-photon and Neuropixels data, and can provide rapid, high-accuracy decoding of natural videos from visual cortex. MAIN A central quest in neuroscience is the neural origin of behaviour1,2. Nevertheless, we are still limited in both the number of neurons and length of time we can record from behaving animals in a session. Therefore, we need new methods that can combine data across animals and sessions with minimal assumptions, thereby generating interpretable neural embedding spaces1,3. Current tools for representation learning are either linear or, if nonlinear, typically rely on generative models and they do not yield consistent embeddings across animals (or repeated runs of the algorithm). Here, we combine recent advances in nonlinear disentangled representation learning and self-supervised learning to develop a new dimensionality reduction method that can be applied jointly to behavioural and neural recordings to show meaningful lower-dimensional neural population dynamics3–5. From data visualization (clustering) to discovery of latent spaces that explain neural variance, dimensionality reduction of behaviour or neural data has been impactful in neuroscience. For example, complex three-dimensional (3D) forelimb reaching can be reduced to between only eight and twelve dimensions6,7, and low-dimensional embeddings show some robust aspects of movements (for example, principal component analysis (PCA)-based manifolds in which the neural state space can easily be constrained and is stable across time8–10). Linear methods such as PCA are often used to increase interpretability, but this comes at the cost of performance1. Uniform manifold approximation and projection (UMAP)11 and t-distributed stochastic neighbour embedding (t-SNE)12 are excellent nonlinear methods but they lack the ability to explicitly use time information, which is always available in neural recordings, and they are not as directly interpretable as PCA. Nonlinear methods are desirable for use in high-performance decoding but often lack identifiability—the desirable property that true model parameters can be determined, up to a known indeterminacy13,14. This is critical because it ensures that the learned representations are uniquely determined and thus facilitates consistency across animals and/or sessions. There is recent evidence that label-guided variational auto-encoders (VAEs) could improve interpretability5,15,16. Namely, by using behavioural variables, such algorithms can learn to project future behaviour onto past neural activity15, or explicitly to use label priors to shape the embedding5. However, these methods still have restrictive explicit assumptions on the underlying statistics of the data and they do not guarantee consistent neural embeddings across animals5,17,18, which limits both their generalizability and interpretability (and thereby affects accurate decoding across animals). We address these open challenges with CEBRA, a new self-supervised learning algorithm for obtaining interpretable, consistent embeddings of high-dimensional recordings using auxiliary variables. Our method combines ideas from nonlinear independent component analysis (ICA) with contrastive learning14,19–21, a powerful self-supervised learning scheme, to generate latent embeddings conditioned on behaviour (auxiliary variables) and/or time. CEBRA uses a new data-sampling scheme to train a neural network encoder with a contrastive optimization objective to shape the embedding space. It can also generate embeddings across multiple subjects and cope with distribution shifts among experimental sessions, subjects and recording modalities. Importantly, our method relies on neither data augmentation (as does SimCLR22) nor a specific generative model, which would limit its range of use. JOINT BEHAVIOURAL AND NEURAL EMBEDDINGS We propose a framework for jointly trained latent embeddings. CEBRA leverages user-defined labels (supervised, hypothesis-driven) or time-only labels (self-supervised, discovery-driven; Fig. 1a and Supplementary Note 1) to obtain consistent embeddings of neural activity that can be used for both visualization of data and downstream tasks such as decoding. Specifically, it is an instantiation of nonlinear ICA based on contrastive learning14. Contrastive learning is a technique that leverages contrasting samples (positive and negative) against each other to find attributes in common and those that separate them. We can use discrete and continuous variables and/or time to shape the distribution of positive and negative pairs, and then use a nonlinear encoder (here, a convolutional neural network but can be another type of model) trained with a new contrastive learning objective. The encoder features form a low-dimensional embedding of the data (Fig. 1a). Generation of consistent embeddings is highly desirable and closely linked to identifiability in nonlinear ICA14,23. Theoretical work has shown that the use of contrastive learning with auxiliary variables is identifiable for bijective neural networks using a noise contrastive estimation (NCE) loss14, and that with an InfoNCE loss this bijectivity assumption can sometimes be removed24 (see also our theoretical generalization in Supplementary Note 2). InfoNCE minimization can be viewed as a classification problem such that, given a reference sample, the correct positive sample needs to be distinguished from multiple negative samples.Fig. 1Use of CEBRA for consistent and interpretable embeddings.a, CEBRA allows for self-supervised, supervised and hybrid approaches for both hypothesis-and discovery-driven analysis. Overview of pipeline: collect data (for example, pairs of behaviour (or time) and neural data (x,y)), determine positive and negative pairs, train CEBRA and produce embeddings. W1,...4 represent the neural network weights. b, Left, true 2D latent, where each point is mapped to the spiking rate of 100 neurons. Middle, CEBRA embedding after linear regression to the true latent. Right, reconstruction score is R2 of linear regression between the true latent and resulting embedding from each method. The ‘behaviour label’ is a 1D random variable sampled from uniform distribution of [0, 2π] that is assigned to each time bin of synthetic neural data, as visualized by the colour map. The orange line represents the median and each black dot an individual run (n = 100). CEBRA-Behaviour shows a significantly higher reconstruction score compared with pi-VAE, t-SNE and UMAP (one-way ANOVA, F(4, 495) = 251, P = 1.12 × 10−117 with post hoc Tukey’s honest significant difference P < 0.001). c, Rat hippocampus data derived from ref. 26. Cartoon from scidraw.io. Electrophysiology data were collected while a rat traversed a 1.6 m linear track ‘leftwards’ or ‘rightwards’. d, We benchmarked CEBRA against conv-pi-VAE (both with labels and without), autoLFADS, t-SNE and unsupervised UMAP. Note: for performance

Read the complete text