MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

Zexuan Chen, Sichao Liu, Runhao Lu, Huichao Qi, Alexandra Woolgar, Xi Vincent Wang Lihui Wang · 2026

Read MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding by Zexuan Chen, Sichao Liu, Runhao Lu, Huichao Qi, Alexandra Woolgar, Xi Vincent Wang Lihui Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24523.txt.

Introduction

Zexuan Chen and colleagues present MindAlign, a 2026 system decoding seen images from brain waves by aligning EEG, visual, and text in one space. It first pre-trains an EEG encoder by masked reconstruction, then aligns EEG, image, and model-written text through contrastive learning. On the Things-EEG2 200-way zero-shot test, it reached 54.1 percent Top-1 and 83.4 percent Top-5 accuracy, far above the prior baseline. It also generalized to MEG data, with decoding patterns matching known visual brain physiology.

Catalog record

CREATORZexuan Chen, Sichao Liu, Runhao Lu, Huichao Qi, Alexandra Woolgar, Xi Vincent Wang Lihui Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53947
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24523

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding Zexuan Chen, Sichao Liu, Runhao Lu, Huichao Qi, Alexandra Woolgar, Xi Vincent Wang Lihui Wang Abstract Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. A field-wide goal is to achieve accurate, generalizable decoding from non-invasive, temporally resolved signals, including electroencephalography (EEG). A major obstacle towards this goal is the low signal-to-noise ratio of EEG and the substantial inter-subject variability, which render direct end-to-end EEG–image supervision weak and unstable. To address this, we introduce a tri-modal contrastive framework for EEG-based visual decoding that aligns EEG, visual, and textual representations within a unified latent space. Our approach follows a two-stage design. First, we pre-train an EEG encoder via masked reconstruction on unlabeled trials, learning spatio-temporal regularities that transfer robustly to downstream tasks. Second, we jointly align EEG, image, and LLM-generated textual descriptions through contrastive learning, where text supervision acts as a semantic regularizer that injects linguistic structure into the shared space without overwhelming the primary EEG–image signal. The encoder integrates subject-specific adaptation, graph-attention over channels, and temporal-spatial convolutional embeddings. On the Things-EEG2 200-way zero-shot benchmark, our framework achieves 54.1% Top-1 and 83.4% Top-5 accuracy, substantially exceeding the strongest prior baseline (32.4% / 64.0%), with paired Wilcoxon tests confirming significance (p < 0.01) over all in-subject baselines. We validate generalization on Things-MEG. Analysis reveals that compact embedding geometries (CN-CLIP) outperform much larger backbones, and that decoding aligns with established neurophysiology of visual processing. This work is a critical step towards robust, semantically-grounded visual decoding from non-invasive temporal neural signals. The source code is publicly available in an https://github.com/anon-eeg/eeg_image_decoding . † † footnotetext: \dagger : equal contributors; \ddagger Corresponding author: [email protected] 1 INTRODUCTION Neuroscience has historically advanced through highly specialized studies of cognitive functions, resulting in a fragmented landscape of task-specific decoders tailored to individual experimental paradigms [ 39 , 59 ] . Developing robust brain-computer interfaces (BCIs) requires accurate, generalizable models of human visual processing from non-invasive neural signals. A major step in this direction has been the development of high-fidelity visual decoders of brain activities [ 29 , 30 , 40 ] with recent advances further accelerated by contrastive multimodal learning and high-quality visual neuroimaging datasets. Visual decoding from electroencephalography (EEG) provides a particularly demanding testbed, as models must extract semantic information from signals that are noisy, temporally entangled, and spatially diffuse. The growing evidence that deep network latent hierarchies converge with the representational geometry of the human brain [ 60 , 33 , 8 ] has driven a wave of methods that align AI features trained with neural activities. The problem typically decomposes into two subproblems: (1) mapping high-dimensional, low-SNR neural activity to a compact visual-semantic representation , and (2) aligning that representation with pretrained vision-language embedding spaces for recognition or retrieval . Large-scale vision-language models [ 44 , 36 , 55 ] have largely addressed (2), while large-scale EEG datasets [ 20 , 21 ] have driven recent progress on (1) [ 18 , 50 , 34 , 58 , 35 ] . Despite this progress, a critical barrier still limits EEG-based decoding accuracy . The low signal-to-noise ratio (SNR) of EEG and substantial inter-subject variability [ 12 , 2 , 43 ] make end-to-end EEG—image supervision weak and unstable, often yielding representations that fail to capture the richness of natural visual content and forcing per-subject models that cannot aggregate patterns across populations [ 24 ] . Beyond signal noise, previous approaches also rely on pairwise EEG—image contrast or indirect semantic-space regression [ 50 , 16 ] , leaving structured linguistic semantics — a complementary source of supervision — remains underexplored. More recent work — large-scale masked pre-training [ 42 ] , hierarchical channel-topology modeling [ 62 ] , and continual subject adaptation [ 65 ] — addresses these issues in isolation. Still, none combine self-supervision, structured channel modeling, and language-grounded supervision within a single framework. We address this gap by viewing EEG-based visual decoding as a cross-modal alignment problem in which a shared semantic space captures both visual appearance and linguistic meaning, and formulate decoding in two stages: (i) pre-train the EEG encoder with a masked reconstruction objective on unlabeled EEG, and (ii) transfer the pre-trained encoder and jointly align EEG, image, and LLM-generated text embeddings through contrastive learning. Image synthesis from brain activity is comparatively mature, we focus on decoding visual-semantic embeddings, evaluated through zero-shot image and text retrieval. Our framework achieves 54.1% Top-1 / 83.4% Top-5 accuracy in the 200-way zero-shot setting on Things-EEG2 , versus 32.4% / 64.0% for the strongest prior baseline. Figure 1 : Overview of the framework and decoding performance on Things-EEG2. (a) Tri-modal contrastive alignment. EEG signals, visual stimuli, and LLM-generated descriptions are encoded into a shared feature space, where corresponding triplets are aligned through contrastive learning and mismatched samples are separated. At inference, an EEG embedding retrieves the most similar image and text candidates in this space. (b) Qualitative top-5 retrievals on the Things-EEG2 test set, ranked by EEG–image similarity; correct matches are highlighted. (c) Quantitative comparison against state-of-the-art (SoTA) baselines (NICE [ 18 ] , NICE++ [ 50 ] , ATMS [ 34 ] , MCRL [ 35 ] ) under the in-subject (left) and cross-subject leave-one-subject-out (right) protocols, reporting Top-1 and Top-5 retrieval accuracy averaged over 10 participants. Our method leads in both settings, with the largest margin in-subject. Figure 1 summarizes the framework and headline results. Our main contributions are as follows. • A tri-modal EEG–image–text alignment framework aligning EEG representations with image features and LLM-generated text in a shared embedding space, where textual semantics provide complementary supervision and improve discriminability over pairwise EEG–image alignment. • A high-performance EEG encoder that integrates a subject-specific adaptation layer, graph-attention-based channel modeling, and temporal-spatial convolutional patch embeddings to capture inter-channel and temporal dependencies. • An Masked Autoencoder (MAE)-based pre-training strategy that initializes the EEG encoder with masked reconstruction and partially transfers weights to the alignment stage, yielding consistent gains. We further observe that the geometry of the visual target space may play an important role, with more compact embedding spaces outperforming larger backbones for EEG-to-image retrieval. 2 RELATED WORKS Contrastive Multimodal Learning for Visual Neural Decoding. Visual neural signal analysis follows two complementary paradigms [ 41 ] : encoding models predict neural activity from stimuli [ 28 , 47 , 8 ] , while decoding models reconstruct or identify stimuli from neural activity [ 29 , 18 , 34 , 35 , 16 , 51 ] . Both have benefited from contrastive objectives [ 44 ] that align neural activity with pretrained vision-language embeddings, motivated by the convergence between deep network hierarchies and the primate visual system [ 60 , 33 ] . CLIP [ 44 ] has since been applied to fMRI [ 5 , 48 , 57 ] and EEG [ 18 , 34 , 50 , 58 , 35 ] . For EEG specifically, prior work has used coarse text labels as auxiliary supervision [ 51 ] or indirect semantic-space regression [ 16 ] . With the emergence of multimodal LLMs such as LLaVA-1.5 [ 36 , 37

Read the complete text