Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction

Xiaoli Yang 1,2 , Huiyuan Tian 1 , Yurui Li 1 , Jianyu Zhang 1 , Shijian Li 1,2,∗ and Gang Pan 1,2 · 2026

Read Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction by Xiaoli Yang 1,2 , Huiyuan Tian 1 , Yurui Li 1 , Jianyu Zhang 1 , Shijian Li 1,2,∗ and Gang Pan 1,2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16370.txt.

Introduction

Yang and colleagues argue that brain signals from scalp electrodes may carry the gist of a sentence even when its exact wording cannot be recovered. Their Brain-CLIPLM system therefore recovers ordered meaning anchors first, then rebuilds sentence meaning around them with a retrieval-guided language model. On the ZuCo benchmark, accuracy peaked at an intermediate anchor size, and control tests showed the anchors held sentence-specific information beyond model guesswork. The paper reframes EEG-to-text decoding as recovering compressed meaning, not full sentences.

Catalog record

CREATORXiaoli Yang 1,2 , Huiyuan Tian 1 , Yurui Li 1 , Jianyu Zhang 1 , Shijian Li 1,2,∗ and Gang Pan 1,2
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54783
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16370

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction Xiaoli Yang 1,2 , Huiyuan Tian 1 , Yurui Li 1 , Jianyu Zhang 1 , Shijian Li 1,2,∗ and Gang Pan 1,2 Abstract Decoding natural language from non-invasive electroencephalography (EEG) remains fundamentally limited by low signal-to-noise ratio and restricted information bandwidth. This raises a fundamental question regarding whether sentence-level linguistic structure can be reliably recovered from such signals. In this work, we suggest that this assumption may not hold under realistic information constraints, and instead propose a semantic compression hypothesis in which EEG signals encode a compressed set of semantic anchors rather than full linguistic structure. Under our new perspective, direct sentence reconstruction becomes an overparameterized objective relative to the intrinsic information capacity of EEG. To address this mismatch, we introduce Brain-CLIPLM, a two-stage framework that decomposes EEG-to-text decoding into semantic anchor extraction via contrastive learning and sentence reconstruction using a retrieval-grounded large language model (LLM) with Chain-of-Thought (CoT) reasoning, following a granularity matching principle that aligns decoding complexity with neural information capacity. Evaluated on the Zurich Cognitive Language Processing Corpus, Brain-CLIPLM achieves 67.55% top-5 and 85.00% top-25 sentence retrieval accuracy, significantly outperforming direct decoding baselines, while cross-subject evaluation confirms robust generalization. Control analyses, including permutation testing, further demonstrate that EEG-derived representations carry sentence-specific information beyond language model priors. These results suggest that EEG-to-text decoding is better framed as recovering compressed semantic content rather than reconstructing full sentences, providing a biologically grounded and data-efficient pathway for non-invasive brain-computer interfaces. \helveticabold 1 Keywords: EEG decoding, neural coding, semantic compression, language reconstruction, semantic representation, contrastive learning, brain–computer interface \correspondance \extraAuth † † firstpage: 1 2 Introduction Decoding language from neural activity, which involves understanding how patterns of brain signals relate to meaning, remains a central problem in neuroscience and brain-computer interfaces (BCIs) ( 38 ; 16 ) . Progress in this direction would not only shed light on how semantic information is represented across distributed cortical systems ( 42 ; 3 ) , but also enable practical communication systems for individuals who have lost motor function while retaining cognition, such as patients with amyotrophic lateral sclerosis (ALS) or brainstem stroke ( 38 ; 22 ) . For such users, non-invasive solutions are particularly important. Among available modalities, electroencephalography (EEG) offers a compelling combination of portability, low cost, and millisecond temporal resolution, making it one of the most viable candidates for real-world BCI deployment ( 17 ; 26 ) . In practice, these systems do not require unrestricted language generation; instead, the immediate goal is to reliably decode short, semantically meaningful sentences that support everyday communication. Recent EEG-to-text studies have explored direct sequence reconstruction using encoder-decoder frameworks ( 43 ; 9 ; 25 ) and autoregressive generation with neural guidance ( 47 ) . While these approaches report improving benchmark performance, they share a critical assumption that sentence-level linguistic structure is recoverable from EEG. From an information-theoretic perspective, this assumption is difficult to justify. EEG signals are noisy and bandwidth-limited, and the amount of reliably decodable information is fundamentally constrained ( 37 ; 2 ) . Under typical recording conditions, a single EEG trial contains on the order of 10^{3} – 10^{4} bits of task-relevant information, whereas the space of possible sentences grows combinatorially with vocabulary size and sequence length ( 6 ) . Empirical evidence further shows that decoding performance can be strongly influenced by language model priors and may not reliably reflect neural input ( 43 ) . In addition, word-level decoding from non-invasive signals remains challenging even under controlled evaluation ( 10 ) . Together, these observations suggest that the core challenge lies in a mismatch between neural information content and linguistic complexity, rather than solely in model design. This raises the question of what form of linguistic representation EEG signals can realistically support. Psycholinguistic and neuroscientific evidence indicates that language comprehension relies heavily on a relatively small set of semantically informative words, so that humans can often recover the meaning of a sentence from a few key content words even when syntactic details are degraded, which is consistent with principles of efficiency and compression in cognition ( 29 ; 14 ) . At the neural level, word-level semantic representations tend to be more stable and more consistently decodable than full sentence structures ( 32 ) . Motivated by these observations, we propose a semantic compression hypothesis , which states that EEG signals do not encode full linguistic structure but instead preserve a compressed set of semantic anchors. We formalize this intuition using an information-theoretic framework. Let X denote EEG signals and Y the target sentence. Standard approaches aim to maximize I(X;Y) , yet in practice I(X;Y)\ll H(Y) , where H(Y) is the entropy of the sentence space ( 37 ; 6 ) . We therefore consider a factorized formulation X\rightarrow K\rightarrow Y , where K represents a low-dimensional set of semantic keywords. This leads to a granularity matching principle: neural decoding is most effective when the representational granularity of the target matches the intrinsic information capacity of the signal. Under this view, keyword-level representations provide a more appropriate interface between neural signals and language than full sentences. Building on this idea, we design Brain-CLIPLM as a principled framework that aligns the decoding objective with the intrinsic information capacity of EEG signals. Our approach instantiates the factorized formulation X\rightarrow K\rightarrow Y , in which semantic anchors serve as an intermediate representation bridging neural signals and language, naturally leading to a two-stage architecture. This perspective suggests that neural decoding is better understood as recovering compressed semantic representations that can be expanded by language models. In the first stage, we learn an EEG encoder using contrastive learning to align neural signals with embeddings of semantically meaningful words, producing a set of candidate keywords that capture the core content of the underlying sentence. In the second stage, we reconstruct full sentences from these semantic anchors using a large language model, augmented with chain-of-thought reasoning and retrieval-based grounding to improve coherence and robustness ( 45 ; 24 ) . This design decouples neural decoding from language generation, allowing each component to operate at an appropriate level of abstraction. While prior work has explored intermediate word-level representations, our goal is not simply to introduce another pipeline, but to provide a principled account of why such a decomposition is necessary under non-invasive constraints. This work makes three main contributions. First, we propose a conceptual framework for EEG-based language decoding grounded in the semantic compression hypothesis and information-theoretic constraints, offering a new perspective on the limitations of direct sentence reconstruction. Second, we develop Brain-CLIPLM as a practical instantiation of this idea, combining contrastive neural representation learning with large language model-based reconstruction. Third, we provide comprehensive empirical validation, including cross-subject analyses and controlled experiments, demonstrating that EEG-derived semantic representations capture sentence-specific information beyond language model priors. Together, these results suggest that EEG-to-text decoding is better understood as recovering compressed semantic content, rather than reconstructing full linguistic form directly. 3 Materials and methods 3.1 Dataset and preprocessing We evaluated the proposed framework on the Zurich Cognitive Language Processing Corpus (ZuCo 1.0), a publicly available dataset containing simultaneous EEG and eye-tracking recordings during natural reading ( 18 ; 19

Read the complete text