Cross-Species RSA Reveals Conserved Early Visual Alignment but Divergent Higher-Area Rankings Across Human fMRI and Macaque Electrophysiology
Nils Leutenegger · 2026
Read Cross-Species RSA Reveals Conserved Early Visual Alignment but Divergent Higher-Area Rankings Across Human fMRI and Macaque Electrophysiology by Nils Leutenegger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22401.txt.
Introduction
Leutenegger tests whether links between learning rules and brain alignment hold across species, comparing human fMRI with macaque electrophysiology. All models align more with macaque early visual cortex than with human fMRI. In corrected re-evaluation, STDP keeps the top macaque V1/V2 alignment, while predictive coding falls to baseline level. At IT, model rankings show no correlation across species. A pretrained ResNet-50 reaches far higher IT alignment than custom CNNs, suggesting capacity, not learning rule, limits IT alignment.
Catalog record
| CREATOR | Nils Leutenegger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22809 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22401
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Species RSA Reveals Conserved Early Visual Alignment but Divergent Higher-Area Rankings Across Human fMRI and Macaque Electrophysiology Nils Leutenegger Abstract Does the relationship between learning rules and brain alignment generalize across species? We extend our prior finding that untrained CNNs match backpropagation at human V1 by testing the same five learning rules against macaque electrophysiology data. The rules are backpropagation (BP), feedback alignment (FA), predictive coding (PC), spike-timing-dependent plasticity (STDP), and an untrained random-weights baseline. The macaque data come from two publicly available datasets: MajajHong2015 (V4/IT, 3,200 stimulus presentations, 88/168 neurons) and FreemanZiemba2013 (V1/V2, 135 stimuli, 102/103 neurons). Using Representational Similarity Analysis (RSA) with identical model weights from our human study, we find that: (1) all models achieve substantially higher alignment with macaque early visual cortex ( ρ = 0.15 \rho=0.15 – 0.30 0.30 at V1/V2) than with human fMRI ( ρ = 0.01 \rho=0.01 – 0.08 0.08 ), consistent with the higher signal-to-noise ratio of electrophysiology; (2) STDP and PC produce the highest macaque V1/V2 alignment ( ρ ≈ 0.30 \rho\approx 0.30 and 0.28 0.28 ), consistent with their leading position among trained rules in human V1; (3) at IT, learning rule rankings show no detectable correlation across species (Kendall’s τ = 0.00 \tau=0.00 , p = 1.00 p=1.00 ), though this null result is expected given that n = 5 n=5 provides statistical power only at τ = ± 1.0 \tau=\pm 1.0 , and is further confounded by stimulus set differences; (4) a pretrained ResNet-50 baseline (ImageNet) achieves ρ = 0.25 \rho=0.25 at macaque IT, substantially above all custom CNN conditions ( ρ = 0.07 \rho=0.07 – 0.14 0.14 ), suggesting that IT alignment in our setup is limited by model capacity and training data rather than by the learning rule. Noise ceilings, multi-seed variability (5 seeds), and a stimulus-control analysis are reported. These results demonstrate that early visual alignment is robust across species and measurement modalities, while higher-area alignment is modulated by model capacity and stimulus domain. Keywords: cross-species, representational similarity analysis, visual cortex, learning rules, macaque electrophysiology, fMRI, Brain-Score 1 Introduction A central finding of computational neuroscience is that deep neural networks (DNNs) trained with backpropagation reproduce the representational hierarchy of the primate ventral visual stream, with early layers matching V1 and deeper layers matching IT cortex ( Yamins and DiCarlo 2016 ; Schrimpf et al. 2020 ) . However, most model–brain comparisons have been conducted within a single species (typically human fMRI), leaving open the question of whether model–brain alignment patterns generalize across species and measurement modalities. In a companion study ( Leutenegger 2026 ) , we showed that an untrained random-weights CNN exceeds backpropagation at human V1 ( \rho=0.076 vs. 0.034 , p<0.001 ), that STDP and PC preserve more V1-like structure than BP, and that all rules converge at IT. These findings raise a natural cross-species question: does the same pattern hold when evaluated against macaque electrophysiology, which offers single-neuron resolution and substantially higher signal-to-noise ratios than fMRI? We address this question by evaluating the same five learning rule conditions, using the identical trained weights from our human study, against two macaque neural datasets from the Brain-Score ecosystem ( Schrimpf et al. 2020 ) : MajajHong2015 ( Majaj et al. 2015 ) for V4 and IT (multi-electrode array recordings, 3,200 presentations), and FreemanZiemba2013 ( Freeman et al. 2013 ) for V1 and V2 (single-unit recordings, 135 texture stimuli). As a capacity control, we additionally evaluate a pretrained ResNet-50 (ImageNet) on all macaque datasets. Our contributions are: (1) the first systematic cross-species RSA comparison of four biologically plausible learning rules plus an untrained baseline, using identical model weights across species; (2) evidence that early visual alignment (V1/V2) is qualitatively conserved across human fMRI and macaque electrophysiology, with local learning rules (STDP, PC) outperforming BP in both species and achieving the highest macaque V1 alignment overall; (3) evidence that higher-area (IT) rankings show no detectable cross-species correspondence in our small-architecture setting, with a stimulus-control analysis showing that ranking differences are partially attributable to stimulus domain; (4) a ResNet-50 capacity control showing that IT alignment scales with model capacity and training data richness, suggesting that the IT convergence observed in our companion study may reflect architectural and data limitations rather than a fundamental property of learning rules. 2 Related Work Cross-species model–brain comparisons. The Brain-Score framework ( Schrimpf et al. 2020 ) provides standardized benchmarks for both human and macaque neural data, enabling systematic cross-species evaluation. Yamins and DiCarlo 2016 showed that task-optimized DNNs predict macaque IT responses, but did not compare multiple learning rules. Majaj et al. 2015 demonstrated that linear readouts from IT firing rates predict human object recognition, establishing a functional link between macaque IT and human behavior. To our knowledge, no study has systematically compared the cross-species alignment of multiple learning rules on the same architecture. Learning rules and brain alignment. Our companion study ( Leutenegger 2026 ) provides a detailed comparison of BP, FA, PC, STDP, and an untrained baseline against human fMRI; we refer readers there for a full review of biologically plausible learning rules and their relationship to brain data. 3 Methods 3.1 Models All five conditions use the identical architecture and weights from Leutenegger 2026 : a custom 3-layer CNN (Conv1: 32 filters, Conv2: 64, Conv3: 128, FC1: 512, FC2: 10) trained on CIFAR-10 for 40 epochs. The five learning rules are: Backpropagation (BP), Feedback Alignment (FA; Lillicrap et al. 2016 ), Predictive Coding (PC; Whittington and Bogacz 2017 ), STDP ( Bi and Poo 1998 ) , and Random Weights (untrained). Model weights are loaded from the companion study’s checkpoints; results are averaged across 5 random seeds trained with identical hyperparameters. One exception: the seed-0 STDP checkpoint from the companion study saved only convolutional weights (FC1 was randomly initialized), so STDP results at IT (FC1-based) use the mean across seeds 1–4 only; V1/V2 results (Conv1-based) are unaffected. For RSA evaluation, all stimuli are resized to 224\times 224 . As a capacity control, we evaluate a pretrained ResNet-50 (ImageNet-1K-V2 weights; He et al. 2016 ) with the layer mapping: layer1 \to V1/V2, layer2 \to V4, layer4 \to IT. 3.2 Neural Datasets Human fMRI. We use the THINGS-fMRI dataset (720 stimuli, 3 subjects) from Leutenegger 2026 . RSA values ( \rho ) for V1, V2, LOC, and IT are taken directly from that study. To enable cross-species comparison at V4, we additionally computed Conv2 \to V4 alignment on the same THINGS-fMRI data using the identical RSA pipeline; this ROI was not reported in the companion study. Macaque V4/IT: MajajHong2015. Multi-electrode array recordings from macaque V4 (88 neurons) and IT (168 neurons) in response to 3,200 presentations of HVM object images ( Majaj et al. 2015 ) . Loaded via the Brain-Score API ( MajajHong2015.public ). Responses were averaged across repetitions per stimulus, and neural RDMs were computed using correlation distance. Macaque V1/V2: FreemanZiemba2013. Single-unit recordings from macaque V1 (102 neurons) and V2 (103 neurons) in response to 135 texture stimuli ( Freeman et al. 2013 ) . Loaded via Brain-Score ( FreemanZiemba2013.public ). We note that these are texture stimuli, not object images; this stimulus mismatch is addressed in our stimulus-control analysis (Section 3.5 ). Layer-to-region mapping. For the