Beyond Object-Level Alignment: Do Brains and DNNs Preserve the Same Transformations?
Yukiyasu Kamitani · 2026
Read Beyond Object-Level Alignment: Do Brains and DNNs Preserve the Same Transformations? by Yukiyasu Kamitani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06420.txt.
Introduction
Yukiyasu Kamitani asks whether brains and deep neural networks preserve the same changes between stimuli, not just similar answers to single images. Standard alignment checks compare stimulus answers or overall shape. Drawing on category theory, Kamitani defines rough naturality: changing a stimulus, then translating between brain and model, should match translating first, then changing.
A Naturality Violation Score measures gaps from that match against a shuffled null. Synthetic tests show it catches misses that simpler scores miss. In human brain data, semantic features align toward higher visual areas and deeper network layers, while lower visual features align earlier, showing selective rather than uniform matching.
Catalog record
| CREATOR | Yukiyasu Kamitani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06420
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Object-Level Alignment: Do Brains and DNNs Preserve the Same Transformations? Yukiyasu Kamitani Abstract Brain–DNN alignment is usually assessed through stimulus-level correspondence or stimulus-set geometry. Inspired by category theory, we operationalize a different question: do brain and model preserve the same candidate transformations among stimuli? We formalize this as approximate naturality — if a proxy-defined stimulus change is propagated through the brain side and then translated to the model side, the result should match translating first and then propagating through the model side, so that the naturality square approximately commutes. We quantify deviations from commutativity by a Naturality Violation Score (NVS) normalized to a permutation null, shifting alignment from per-stimulus sameness to preservation of structure under an explicitly chosen comparison map. As a proof of concept, a controlled five-factor synthetic setting shows that NVS separates complementary alignment failures that aggregate object- and geometry-level scalars cannot resolve. Applied to fMRI responses from the GOD dataset (5 subjects), 3 vision DNNs, and 3 external embedding spaces used as limited World-Model proxies, the axis-resolved analysis reveals a hierarchy crossover: semantic axes align most strongly toward HVC and deeper DNN layers (animacy has the lowest pooled NVS among the tested axes, NVS animacy = 0.39 \mathrm{NVS}^{\mathrm{animacy}}=0.39 , compared with 0.52 0.52 for the next-best axis and 1.0 1.0 for the permutation-null baseline), whereas low- and mid-level visual axes align toward earlier visual cortex and shallower layers. Supporting analyses — a 15-axis appendix atlas, dissociation tests against RSA/CKA and encoding/decoding accuracy, and a W-less anchor-ablation control — support that the alignment is selective over candidate morphism families rather than uniform. NVS thereby turns brain–DNN comparison into a test of jointly preserved candidate transformations, relative to an explicitly chosen proxy space and permutation null, opening a path to richer proxy spaces and controlled world-side transformations. 1 Introduction When brain and model assign similar codes to the same image, do they also move between images in the same way? Existing alignment metrics — encoding ( 48 ) and decoding ( 19 ) accuracies, Brain-Score ( 41 ) , the Brain Hierarchy (BH) score ( 31 ) , RSA ( 25 ; 26 ) , CKA ( 24 ) , and Procrustes alignment — summarize how closely the two systems agree on individual stimuli or on overall stimulus-set geometry. In their standard use, these metrics do not directly test whether a specified stimulus transformation propagates compatibly through both systems: two systems can agree on per-stimulus predictions and on stimulus-set geometry yet still disagree about which changes among stimuli they treat as equivalent, stable, or meaningful. Inspired by category theory, we frame this as a question about approximate naturality : whether brain and model preserve the same proxy-defined candidate changes under translation, so that the two paths around the square approximately commute (Fig. 1 ; full categorical setup in App. A1 ). We use this as an operational analogy rather than as a claim that the fitted maps constitute strict functors — the tested operator families are empirical linear approximations defined on sampled stimulus pairs. This shifts alignment from objectwise sameness to preservation of structure under an explicitly chosen comparison map. Concretely, B (brain) and M (model) are ambient vector spaces with stimulus points b_{s}\in B , m_{s}\in M . A tested stimulus change r\colon s\to s^{\prime} is realized on each side by linear operators F_{B}(r)\colon B\to B and F_{M}(r)\colon M\to M (Fig. 1 , left), which we approximate per edge from sampled stimulus pairs (formalized in § 3 ). We call the cross-system maps \eta\colon B\to M and \eta^{\prime}\colon M\to B translators ( 44 ) : linear maps between brain activity and a target latent representation, with \eta a decoder and \eta^{\prime} an encoder. In the present framework they act as approximate intertwiners. For readability we write the forward square explicitly, \eta\circ F_{B}(r)\;=\;F_{M}(r)\circ\eta, (1) i.e., the two paths — propagate then translate, vs. translate then propagate — should agree (Fig. 1 , right). The same square is asked of \eta^{\prime}\colon M\to B in the reverse direction ( \eta^{\prime}\circ F_{M}(r)=F_{B}(r)\circ\eta^{\prime} ). Standard alignment metrics test only \eta(b_{s})\approx m_{s} at each stimulus individually; approximate naturality additionally constrains the change structure — whether the same candidate transformation is preserved across the two systems. Figure 1: Brain–DNN alignment as approximate naturality (motivational concept diagram). Left: the World Model W parameterizes which stimulus change r\colon s\to s^{\prime} is tested; the translators \eta\colon B\to M and \eta^{\prime}\colon M\to B connect the two sides. Right: the naturality square (the analogous square is also asked of \eta^{\prime} ). This is the motivational categorical reading; in practice F_{B}(r),F_{M}(r) are not estimated as endomorphisms of B,M but realized at first order via the proxy chain F_{W},\Phi_{B},\Phi_{M} (Fig. 2 ); F_{B},F_{M} are not enforced to be strict functors (App. A1 ). The candidate morphisms r are specified through an external comparison space W : the proxy through which the same candidate stimulus change is mapped into both brain and model spaces, sitting at the apex of the cospan that links them (Fig. 1 ; full categorical setup in App. A1 ). The choice of W determines which transformations can be tested and is therefore part of the scientific question, not a nuisance parameter. In this paper, W is a limited World Model proxy in the broad neuro-AI sense ( 16 ; 2 ) , given by an embedding function F_{W}\colon\mathbf{Stim}\to W ; we instantiate three: CLIP-text ( 38 ) , DINOv2 ( 34 ) , and DreamSim ( 13 ) , capturing language-grounded semantics, self-supervised visual structure, and human perceptual similarity, respectively. We do not treat these as full generative or causal world models, only as embedding spaces that supply a structured family of candidate morphisms. Differences within W parametrize world-structural changes via the linear representation hypothesis ( 29 ; 35 ) — the empirical observation that semantically meaningful features are encoded as approximately linear directions in modern embedding spaces, so that a single direction in W corresponds to one such change. Restricting to a single concept axis via a CAV ( 21 ) yields an axis-resolved version (formalized in § 3 ; throughout this paper, “axis” refers only to this CAV-based parameterization). The Naturality Violation Score (NVS) is defined in § 3 as the relative L^{2} residual of Eq. ( 1 ), evaluated in both translation directions ( \eta,\eta^{\prime} ) and normalized to a permutation null. We evaluate this in two stages. A five-factor synthetic toy world serves as a proof of concept (§ 4 ): morphism-level NVS recovers complementary failures that aggregate object- and geometry-level scalars collapse. On empirical data — GOD fMRI from 5 subjects, 3 vision DNNs, and 3 World Model proxies (§ 5 ) — we first fit conventional per-stimulus encoding/decoding mappings and only then evaluate naturality post hoc, without optimizing for commutativity. The axis-resolved analysis reveals a hierarchy crossover (Tab. A2 ): low-level photometric morphisms align at V1 \times shallow cells, semantic morphisms at HVC \times deep cells, and mid-level morphisms in between, with animacy the strongest of the six main axes. To keep the main narrative readable while preserving evidential support, the appendix is organized by scientific function rather than chronology: formal setup and data-processing details (Apps. A1 – A3 ), cross-subject