Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding

2026

Read Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21948.txt.

Introduction

Neural decoding tries to rebuild what a person saw from their brain activity, a central problem for brain-computer interfaces. It also promises insight into how the brain represents the visual world. The 2026 preprint behind this entry argues that current methods miss a basic mismatch in granularity.

Human vision and deep vision models work at different levels of detail. Deep models chase semantic invariance, recognizing the same object across many changes, while early human vision keeps fine spatial detail. The authors frame this as deep models with shallow alignment to the brain. Closing that gap could improve both decoding accuracy and our scientific picture of how seeing is organized in the cortex.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21948

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding Abstract Neural visual decoding is a central problem in brain–computer interface research, aiming to reconstruct human visual perception and to elucidate the structure of neural representations. However, existing approaches overlook a fundamental granularity mismatch between human and machine vision, where deep vision models emphasize semantic invariance by suppressing local texture information, whereas neural signals preserve an intricate mixture of low-level visual attributes and high-level semantic content. To address this mismatch, we propose Shallow Alignment, a novel contrastive learning strategy that aligns neural signals with intermediate representations of visual encoders rather than their final outputs, thereby striking a better balance between low-level texture details and high-level semantic features. Extensive experiments across multiple benchmarks demonstrate that Shallow Alignment significantly outperforms standard final-layer alignment, with performance gains ranging from 22% to 58% across diverse vision backbones. Notably, our approach effectively unlocks the scaling law in neural visual decoding, enabling decoding performance to scale predictably with the capacity of pre-trained vision backbones. We further conduct systematic empirical analyses to shed light on the mechanisms underlying the observed performance gains. Yang Du 1 Siyuan Dai 1 Yonghao Song 2 Paul M. Thompson 3 Haoteng Tang 4 Liang Zhan 1 1 Dept. of Electrical & Computer Engineering, University of Pittsburgh, USA 2 Dept. of Biomedical Engineering, Tsinghua University, China 3 Dept. of Neurology, University of Southern California, USA 4 Dept. of Computer Science, University of Texas Rio Grande Valley, USA [email protected] 1 Introduction Understanding how the brain encodes visual information is a fundamental problem in both neuroscience and artificial intelligence [ 18 , 37 , 28 , 40 , 25 ] . Recent advances in brain-computer interfaces, particularly studies based on electroencephalography (EEG) and magnetoencephalography (MEG), have demonstrated the feasibility of decoding visual stimuli directly from neural activity [ 34 , 15 , 17 , 33 ] . This task, commonly referred to as neural visual decoding, aims to retrieve perceived visual stimuli from non-invasive neural recordings. The core challenge lies in learning an alignment function that can translate high-dimensional, noisy neural dynamics into structured visual representations. Figure 1 : Overview of the proposed Shallow Alignment framework. The model aligns neural signals with intermediate visual representations to mitigate information-lossy alignment at high semantic levels. The prevailing paradigm in neural visual decoding adopts large-scale pre-trained vision models (e.g., CLIP) as feature extractors, aligning neural signals to visual representations via contrastive learning on the final-layer embeddings of these models [ 32 , 23 , 43 , 39 , 44 ] . However, these approaches overlook a fundamental granularity mismatch between human and machine vision. The inductive bias of contemporary vision models aims to maximize semantic invariance, systematically discarding local variations to optimize for categorization [ 21 , 30 , 20 ] . In contrast, visually evoked neural responses are not confined to a single level of abstraction. Instead, they encode information across multiple representational scales, ranging from low-level attributes such as contours, color, and spatial frequency to high-level semantic content. [ 7 , 3 , 6 , 11 ] . As a result, aligning neural signals to the final, most abstract layer of vision models forces a complex, multi-scale neural representation to match a texture-insensitive semantic embedding. This granularity mismatch introduces significant ambiguity into the contrastive objective, as distinct neural patterns with differing low-level characteristics may be mapped to similar high-level visual representations. Consequently, the resulting supervision signal is insufficiently informative to guide effective neural representation learning. To bridge this gap, we propose Shallow Alignment, a strategy that shifts the neural decoding target from the final output layer to the intermediate layers of vision encoders. We posit that intermediate representations offer a more appropriate granularity match for non-invasive neural signals, balancing high-level semantic abstraction with the retention of structural fidelity. Our contributions are summarized as follows: • We propose Shallow Alignment, a neural visual decoding framework that mitigates the granularity mismatch between neural signals and vision encoders, yielding consistent improvements across multiple benchmarks. • We resolve a Depth–Capacity Paradox, where increased semantic abstraction in deeper layers hinders effective alignment. By addressing this, we unlock the scaling law for neural decoding, enabling performance to scale with model capacity. • Through extensive empirical analysis showing superior alignment with intermediate layers, we provide evidence for a shared hierarchical granularity between human visual processing and deep vision models. 2 Related Work 2.1 Semantic Granularity in Neural Decoding Recent advancements in neural decoding have predominantly focused on aligning neural signals with the latent spaces of large-scale pre-trained vision models. [ 32 ] introduced a contrastive learning approach that incorporates plug-and-play self-attention and graph attention modules to capture spatial correlations in EEG electrodes for effective image decoding. [ 23 ] proposed an end-to-end zero-shot framework that utilizes a tailored EEG encoder named Adaptive Thinking Mapper (ATM) to align EEG signals with CLIP embeddings. Leveraging contrastive learning, [ 43 ] developed a method to align EEG signals not only with images but also with generated captions and depth maps, demonstrating that complementary multimodal information enhances brain signal decoding. However, emerging evidence suggests that maximizing semantic abstraction does not necessarily yield optimal decoding performance. For instance, [ 39 ] identified a ”System GAP” between human perception and digital stimuli, proposing an Uncertainty-Aware Blur Prior (UBP) that improves alignment by dynamically adjusting the blur radius based on sample uncertainty. Similarly, [ 44 ] introduced NeuroBridge leverages Cognitive Prior Augmentation to simulate perceptual variability via image transformations, including Gaussian blur, Gaussian noise, mosaic effects, and low-resolution downsampling. Notably, they also reported a counter-intuitive phenomenon where architecturally simpler backbones (e.g., ResNet-50) often exhibit competitive or even superior performance compared to large-scale foundation models (e.g., ViT) under standard settings. This observation motivates our hypothesis that the key to alignment lies not in maximizing semantic level of the image or neural representation, but in precisely calibrating the representational granularity between modalities. 2.2 Hierarchical Representations in Human and Artificial Vision Neuroscience has established that the primate visual cortex operates via a cascade of processing stages, where early areas (e.g., V1, V2) encode low-level features like spatial frequencies and orientation, while high-level object semantics emerge only in later stages (e.g., IT cortex) [ 7 , 6 ] . An interesting parallelism exists in deep artificial neural networks, which evolve from detecting low-level primitives in shallow layers to representing abstract concepts in deep layers [ 41 , 26 ] . In addition, non-invasive neural recordings (e.g., EEG/MEG) are inherently noisy and characterized by sparse information density. They are susceptible to multiple sources of contamination, including environmental interference, muscle artifacts, and intrinsic biological variability, leading to substantially reduced signal-to-noise ratios [ 13 , 35 ] . We posit that these create a fundamental granularity mismatch, where the intricate feature granularity of the brain signals is incompatible with the highly abstract, semantically compressed representations at the final output of the model. 2.3 Granularity Balance in Intermediate Representations Prior research has demonstrated the utility of intermediate layers for diverse tasks, ranging from elucidating training dynamics [ 1 ] to enhancing transfer learning [ 9 ] and improving robustness against distribution shifts [ 22 , 36 ] . Crucially, these works point to a fundamental trade-off in

Read the complete text