High-dimensional structure underlying individual differences in naturalistic visual experience
Chihye Han, Michael F. Bonner · 2025
Read High-dimensional structure underlying individual differences in naturalistic visual experience by Chihye Han, Michael F. Bonner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12653.txt.
Introduction
Chihye Han and Michael Bonner study how each brain builds its own view of a film in a 2025 paper. People who watch the same movie still live it in unlike ways. Brain scans during natural movie viewing let the team map where those private patterns sit in the visual system.
Breaking the scan signals into layers showed private patterns at every scale, from coarse to very fine. Each scale carried a distinct side of how a person processes scenes. The full pattern also predicted how well each viewer later recalled the film. Standard measures that average across viewers missed much of this detail. The authors frame private experience as spread across a wide geometric space.
Catalog record
| CREATOR | Chihye Han, Michael F. Bonner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
High-dimensional structure underlying individual differences in naturalistic visual experience Chihye Han, Michael F. Bonner Abstract How do different brains create unique visual experiences from identical sensory input? While neural representations vary across individuals, the fundamental architecture underlying these differences remains poorly understood. Here, we reveal that individual visual experience emerges from a high-dimensional neural geometry across the visual cortical hierarchy. Using spectral decomposition of fMRI responses during naturalistic movie viewing, we find that idiosyncratic neural patterns persist across multiple orders of magnitude of latent dimensions. Remarkably, each dimensional range encodes qualitatively distinct aspects of individual processing, and this multidimensional neural geometry predicts subsequent behavioral differences in memory recall. These fine-grained patterns of inter-individual variability cannot be reduced to those detected by conventional intersubject correlation measures. Our findings demonstrate that subjective visual experience arises from information integrated across an expansive multidimensional manifold. This geometric framework offers a powerful new lens for understanding how diverse brains construct unique perceptual worlds from shared experiences. 1 Introduction The human brain transforms visual input into subjective perceptual experiences. When presented with identical visual stimuli, individuals exhibit idiosyncratic patterns of neural activity across cortical regions involved in visual processing [ 1 ] . Idiosyncratic neural activity patterns remain stable across time [ 2 , 3 ] and carry meaningful information about behavior [ 2 , 4 ] . Individuals with shared personality traits [ 5 ] , shared context [ 6 ] , or shared perspectives [ 7 ] have more similar neural responses during naturalistic narrative experience. These findings underscore the importance of understanding differences in neural representations underlying differences in perception and behavior across people. However, the full extent of variability within rich neural structure that gives rise to unique visual experience remains largely unknown. Here, we investigated the dimensionality of neural representational structure underlying individual differences. Specifically, we quantified the extent to which inter-individual variability can be characterized through latent codes spanning multiple dimensional ranges. In neural population coding, dimensionality refers to the minimum number of axes needed to specify each response’s position in representational space [ 8 ] . A prevailing view in cognitive neuroscience holds that cortical representations embed complex, high-dimensional stimuli into lower-dimensional latent subspaces to support generalization [ 9 , 10 ] . However, recent studies leveraging large-scale neural datasets show that natural stimulus representations in visual cortex span at least thousands of dimensions, with dimensionality scaling unboundedly up to the limit of the data [ 11 , 12 ] . This observation suggests that neural representations are more high-dimensional than previously appreciated. Crucially, dimensions beyond the leading, high-variance components encode stimulus information that may be relevant for perception but is not well-characterized by conventional analyses [ 12 ] . These findings raise a critical question: Are there reliable individual differences across many latent dimensions of neural responses during naturalistic visual experience, or are they concentrated in a subset of dimensions that explain most of the variance in neural data? If neural representations are organized across multiple scales of variance, then individual differences in these representations might similarly span many latent dimensions. Alternatively, although stimulus-related variance is embedded in high-dimensional subspaces, it is possible that only a low-dimensional subspace might be relevant for individual differences in visual experience [ 13 , 14 , 15 ] . In this work, we use cross-decomposition to examine individual differences across the full spectrum of dimensions underlying cortical population codes. Our method combines spectral analysis with cross-validated functional alignment to characterize how neural representations vary between individuals at multiple representational scales. Using fMRI data of subjects viewing naturalistic movies, we quantify both the reliability and behavioral significance of intersubject variability in movie-evoked neural responses throughout the visual cortical hierarchy. Our findings reveal that reliable individual differences in visual processing manifest as distinct coactivation patterns distributed across multiple scales of neural representation, from dominant patterns explaining large amounts of variance to subtler patterns explaining increasingly smaller amounts. Crucially, we find that intersubject variability patterns are qualitatively distinct at each dimensional range and cannot be explained by conventional intersubject correlation (ISC) measures, revealing unique aspects of individual neural processing distributed across multiple levels of granularity. These multidimensional neural response patterns predict individual differences in movie recall, with neural-behavioral correlations distributed over a wide range of representational dimensions. These results highlight the multiscale structure of individual differences and establish a new spectral framework that advances our understanding of the high-dimensional neural architecture underlying subjective visual experience. 2 Results 2.1 High-dimensional structure in cortical responses to movies One key to understanding neural population codes is characterizing how variance is distributed across latent dimensions of neural activity. Traditional approaches like principal component analysis (PCA) decompose neural responses into orthogonal dimensions sorted by variance. However, PCA captures both stimulus-driven and noise-related variance, potentially obscuring reliable stimulus-evoked patterns. Here, we employ cross-decomposition [ 12 ] to isolate reliable stimulus-related variance in neural activity. Our approach isolates stimulus-dependent activations through cross-validation: basis sets derived from training data are used to compute covariance spectra on held-out test data. Like hyperalignment [ 16 ] , cross-decomposition performs the Procrustes transformation to map a pair of neural response data into a set of orthogonal latent dimensions that maximize the covariance between the data sets. Importantly, our method provides a spectral characterization of shared variance, decomposing the shared signal into latent dimensions ordered by explained covariance. We hypothesized that stimulus-related variance in movie-evoked neural responses would be distributed across multiple orders of magnitude of latent dimensions. While previous work used cross-decomposition to estimate the dimensionality of neural representations evoked by natural images [ 12 ] , it has not been previously applied to dynamic, naturalistic movie responses. Here we applied cross-decomposition to a dataset of 43 subjects viewing four naturalistic movies in an fMRI scanner [ 17 ] . The dataset was originally curated to study individual differences in neural event segmentation during movie viewing and contained movie stimuli with highly varied content. Thus, these movies provide an excellent stimulus set for testing the generalization of our findings using held-out stimuli that are qualitatively distinct from the training stimuli. Specifically, we used a one-movie-out cross-validation scheme, which provided a stringent test of reliability. For a pair of subjects, we extracted shared latent dimensions from a training set of movies, with one movie held out for testing (Fig. 1 a , left). During the testing phase, the held-out movie responses from both subjects were projected onto shared latent dimensions, yielding a cross-covariance spectrum for the held-out data (Fig. 1 a , right). We averaged the cross-covariance spectra across movie-folds and repeated this procedure for every subject pair. We applied this procedure to neural responses from a series of anatomically defined regions of interest (ROIs) spanning a processing hierarchy from low-level vision to high-level semantics, which we expect to be strongly modulated by movie stimuli (Fig. 1 b ). These regions included early visual regions, ventral visual stream, lateral visual stream, and higher-level regions in the posterior parietal and cingulate cortex (See Methods). These ROIs allowed us to examine on how movie-evoked responses are represented across the cortical hierarchy, from early visual processing to multimodal regions associated with complex semantic integration. Movie-evoked responses showed a characteristic power-law