Multi-Modal Latent Variables for Cross-individual Primary Visual Cortex Modeling and Analysis

Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu †, Tiejun Huang · 2024

Read Multi-Modal Latent Variables for Cross-individual Primary Visual Cortex Modeling and Analysis by Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu †, Tiejun Huang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14536.txt.

Introduction

Each brain wires its visual cortex a little differently, which makes pooling data across animals a headache for neuroscience. Yu Zhu and co-authors build a model that puts many brains into one shared code space. Their 2024 paper presents a variational autoencoder that takes both neural activity and the visual stimulus as inputs.

Two levels of disentanglement sort the shared code into parts that track the stimulus and parts that track the animal. On a large mouse dataset the model beat rivals at aligning animals without any per-animal tuning, and it improved as data grew. A scoring method then traced code units back to real neurons and image regions, finding cell groups with distinct timing and patches tied to edges and brightness. The authors offer the frame as a scalable base for cortex research.

Catalog record

CREATORYu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu †, Tiejun Huang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14536

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal Latent Variables for Cross-individual Primary Visual Cortex Modeling and Analysis Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu †, Tiejun Huang Abstract Elucidating the functional mechanisms of the primary visual cortex (V1) remains a fundamental challenge in systems neuroscience. Current computational models face two critical limitations, namely the challenge of cross-modal integration between partial neural recordings and complex visual stimuli, and the inherent variability in neural characteristics across individuals, including differences in neuron populations and firing patterns. To address these challenges, we present a multi-modal identifiable variational autoencoder (miVAE) that employs a two-level disentanglement strategy to map neural activity and visual stimuli into a unified latent space. This framework enables robust identification of cross-modal correlations through refined latent space modeling. We complement this with a novel score-based attribution analysis that traces latent variables back to their origins in the source data space. Evaluation on a large-scale mouse V1 dataset demonstrates that our method achieves state-of-the-art performance in cross-individual latent representation and alignment, without requiring subject-specific fine-tuning, and exhibits improved performance with increasing data size. Significantly, our attribution algorithm successfully identifies distinct neuronal subpopulations characterized by unique temporal patterns and stimulus discrimination properties, while simultaneously revealing stimulus regions that show specific sensitivity to edge features and luminance variations. This scalable framework offers promising applications not only for advancing V1 research but also for broader investigations in neuroscience. 1 Institute of Automation, Chinese Academy of Sciences 2 School of Artifcial Intelligence, University of Chinese Academy of Sciences 3 Shanghai Artificial Intelligence Laboratory 4 Beijing Academy of Artificial Intelligence [email protected], [email protected], [email protected] Introduction The primary visual cortex (V1) plays a fundamental role in hierarchical visual information processing, making its functional characterization essential to systems neuroscience. Recent advances in calcium imaging technology ( Sofroniew et al. 2016 ) have accelerated V1 research by enabling recording of large neuronal populations in multiple subjects. A central goal in utilizing this technology is to identify distinct patterns of neural activity and establish their relationships with specific visual stimuli ( Stringer et al. 2019 ) . Current approaches to understanding V1 visual information processing primarily follow two paradigms. The first develops encoding models ( Sinz et al. 2018 ; Wang et al. 2023 ) that map visual stimuli to neural activity, simulating V1 function. The second focuses on decoding neural activity to understand the representation of visual features such as edges and motion ( Yoshida and Ohki 2020 ) . However, these approaches face fundamental limitations due to the restricted shared subspace between sparse, locally-recorded neural activity and complex visual features, making it challenging to fully capture brain-vision relationships. Furthermore, these models often make the oversimplified assumption that locally recorded V1 populations can fully encode complete visual stimuli, despite evidence that visual processing is distributed across the extensive V1 area according to retinotopic mapping (Figure 1 .A). Additionally, cross-individual neural heterogeneity presents an additional significant challenge. Neural responses to identical stimuli can vary substantially among individuals ( Guntupalli et al. 2016 ; Haxby et al. 2001 ) , complicating cross-individual modeling efforts (Figure 1 .B). Addressing this variability requires zero-shot or few-shot domain adaptation capabilities, a challenge that remains significant even in contemporary machine learning. Previous approaches ( Wang et al. 2023 ) requiring individual-specific fine-tuning have not adequately addressed this cross-individual variability. Figure 1: Challenges in modeling V1. (A) Retinotopic mapping (RL, AL and LM refers to other visual cortex) and the partially recorded field of view of calcium imaging (blue rectangle). (B) One-to-many challenge with cross-individual heterogeneity, especially neuron counts and firing patterns. To address these challenges, we introduce a m ulti-modal i dentifiable v ariational a uto e ncoder (miVAE) featuring two-level latent space disentanglement. For neural activity, we separately model idiosyncratic and preserved latent variables, capturing both subject-specific characteristics and functionally consistent information across individuals. For visual stimuli, we distinguish between neural activity-related and unrelated latent variables, refining the preserved variables to capture highly relevant neural correlates. We further enhance these highly correlated variables through latent modeling and introduce a novel score-based attribution strategy for comprehensive data interpretation. Validated on a mouse V1 dataset ( Turishcheva et al. 2023 ) using two-photon calcium imaging ( Grienberger and Konnerth 2012 ) , miVAE demonstrates superior cross-individual latent coding without requiring individual-specific fine-tuning, showing scalable performance with increasing dataset size and remarkable consistency across individuals exposed to identical visual stimuli. This cross-individual approach significantly enhances the scalability of data-driven research. Our attribution analysis successfully identifies key neuronal subpopulations with distinct stimulus-related responses and superior discriminative capabilities, while highlighting V1’s sensitivity to edge and luminance patterns. This framework presents a powerful tool for V1 research with potential applications across various sensory cortices. Our primary contributions include: • Multi-modal identifiable modeling with latent space refinement. miVAE effectively captures highly correlated variables between visual stimuli and neural responses, enhanced through pair-wise modeling in the latent space. • Scalable cross-individual performance. Our approach achieves state-of-the-art modeling and cross-individual alignment without subject-specific fine-tuning, demonstrating improved performance with increased data size. • Score-based attribution analysis with biological insights. This novel method maps latent variables to original data, identifying key neurons with distinct temporal patterns and stimulus discriminative capabilities, while revealing visual regions sensitive to edges and luminance. Related Work Latent Variable Modeling. The advent of calcium imaging has provided unprecedented access to neural data while simultaneously introducing significant analytical challenges. A prominent approach to managing this complexity involves mapping high-dimensional neural signals to low-dimensional latent spaces. Early research focused on extracting latent variables from neural population activities using simple, prior-driven designs ( Yu et al. 2008 ; Zhao and Park 2017 ) . This evolved into more sophisticated methods employing recurrent neural network (RNN)-based explicit dynamics modeling ( Pandarinath et al. 2018 ; Keshtkaran et al. 2022 ; Zhu et al. 2022 ) , which yielded highly interpretable results. The increasing scale of neural activity data led to the development of more efficient approaches, including Transformer-based methods with implicit latent representation learning ( Ye and Pandarinath 2021 ; Liu et al. 2022 ; Le and Shlizerman 2022 ; Ye et al. 2024 ; Antoniades et al. 2024 ) . While these demonstrated superior decoding performance, they often lacked interpretability and identifiability. To address these limitations, interpretable approaches such as pi-VAE ( Zhou and Wei 2020 ) and CEBRA ( Schneider, Lee, and Mathis 2023 ) introduced auxiliary variables for identifiable latent variables, building upon identifiable variational autoencoders (iVAE) ( Khemakhem et al. 2020 ) and nonlinear independent component analysis (ICA) ( Hyvarinen, Sasaki, and Turner 2019 ) . However, these methods prove inadequate for direct cross-individual analysis and fail to account for individual variability. Our approach differs by explicitly modeling cross-individual variability and separating neural activity components into stimulus-related and unrelated elements. Through multi-modal and bi-directed generative modeling, we effectively isolate highly relevant latent variables between neural activity and visual stimuli, ensuring interpretability while enabling sophisticated latent space modeling and analysis. Figure 2: miVAE for modeling V1 neural activity \mathbf{x} and the corresponding visual stimulus \mathbf{y} . \mathbf{x} and \mathbf{y} are decomposed in the latent space. For \mathbf{x} , it has the idiosyncratic latent variable \mathbf{z_{m}^{x}} and the cross-subject and functionally relevant preserved latent variable \mathbf{z_{c}^{x}} . For \mathbf{y}

Read the complete text