What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval

Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang · 2026

Read What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval by Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24524.txt.

Introduction

Xinyu Zhang and colleagues audit non-invasive brain-to-language retrieval, asking how much reported accuracy comes from real stimulus evidence rather than shortcuts. Signal-blind noise alone reached 66.3 percent Rank at 1 under variable-length decoding. It collapsed to near chance once fixed windows and identity splits were enforced. Under those controls, fixed-window retrieval recovered genuine MEG-audio discriminability, though most Top-1 errors picked the wrong sentence. A Group Context Bias intervention lifted Rank at 1 from 44 to 52 percent on Gwilliams and 22 to 29 percent on MOUS.

Catalog record

CREATORXinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE122852
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang Abstract In non-invasive neural language decoding, results can be inflated by sources that are not stimulus-evoked neural evidence: decoder priors, embedding-based metrics, and non-neural structural nuisances such as signal duration. The methodological challenge is therefore attribution: a reported gain is more informative when it can be traced to a specific source. We recast stimulus-locked MEG-to-audio retrieval as an auditing framework that separates apparent performance into three sources — structural shortcuts, window-level stimulus-locked evidence, and cross-window contextual aggregation — and provides a diagnostic for each. Signal-blind Gaussian noise reaches 66.3% Rank@1 (R@1) under variable-length decoding but collapses to near chance once fixed-duration windows and stimulus-identity splits are enforced, isolating structural leakage. Under these controls, fixed-window retrieval recovers measurable MEG–audio discriminability, while an oracle sentence-bucket diagnostic shows that 95.7% of Top-1 errors select the wrong sentence, localising the residual bottleneck to sentence-level competition. We audit this contextual source with Group Context Bias (GCB), an inference-time additive logit bias that pools sentence-consistent evidence across windows while leaving the base retrieval scores and candidate pool fixed. Used as a score-space intervention, GCB makes the contextual source measurable: R@1 shifts from 44% to 52% on Gwilliams and from 22% to 29% on MOUS under the same fixed setting. GCB is auditable under this design: its effect collapses under random-grouping perturbations and vanishes when local evidence is attenuated in MEG or is near chance in EEG, supporting its use as a controlled source-attribution intervention. These results suggest that brain-to-language performance should be source-attributed, not merely reported. † † footnotetext: \dagger : equal contributors; \ddagger Corresponding author: [email protected] 1 Introduction Non-invasive neural language decoding has advanced rapidly with the introduction of pretrained speech and language models, and end-to-end systems increasingly report strong performance on text generation and semantic matching benchmarks [ 42 , 14 , 43 , 44 , 27 ] . Yet reported accuracy in this setting is often not what it appears to be. Under a standard variable-length sentence-decoding protocol, replacing real MEG with Gaussian noise still yields 66.3% next-token R@1 accuracy — variable-duration inputs introduce a sequence-length shortcut [ 18 , 3 ] that attention-based models exploit through padding boundaries and mask patterns. Duration is one instance of a broader pattern: decoder priors [ 25 ] , embedding-based metrics [ 46 , 21 ] , training-evaluation leakage [ 10 , 45 ] , and other non-neural structural nuisances can each make signal-blind inputs appear informative, so reported gains can reflect evaluation shortcuts rather than decodable linguistic information in the neural signal. The methodological challenge is therefore attribution: how can we verify whether apparent brain-to-language decoding performance is genuinely supported by stimulus-locked neural evidence, rather than by structural shortcuts, decoder priors, or evaluation artefacts? Text generation is a poor endpoint for this kind of attribution. Neural evidence, decoder priors, decoding strategy, and text-level metrics are all entangled in the final output, so a high semantic score or fluent generation alone does not identify the source of the information being used. This ambiguity is especially acute when a neural encoder conditions a strong speech, language, or generative prior: plausible outputs can be produced even when the neural representation contributes little stimulus-specific evidence. Stimulus-locked MEG-to-audio retrieval offers a more controlled endpoint [ 13 ] : each neural window retrieves its time-aligned audio window from an explicit candidate pool, making the candidate set, target alignment, chance level, and rank errors directly observable. Retrieval therefore turns apparent decoding performance into auditable candidate-level decisions, allowing structural shortcuts, local stimulus-locked evidence, and contextual effects to be analysed separately. Within this auditable setting, retrieval performance arises from multiple sources, and we trace them in turn. To isolate window-level stimulus-locked evidence, we first use independent fixed-window retrieval: each window is decoded separately using fixed-duration windows and stimulus-identity splits, preventing variable duration and repeated stimulus content from acting as direct ranking cues [ 10 , 45 , 13 , 15 ] . This local baseline shows MEG–audio discriminability, indicating that the window-level stimulus-locked neural signal is decodable under the controlled retrieval setting. However, this local-only design factorises retrieval over individual windows and therefore does not condition on information beyond the current window, despite evidence that linguistic context modulates time-locked neural responses during natural speech [ 39 , 8 , 35 ] . The error structure confirms this gap: under a global candidate pool, 95.7% of Top-1 errors select a candidate from the wrong sentence, and an oracle diagnostic that restricts candidates to the ground-truth sentence raises R@1 from 44% to 88% with the model held fixed. Many failures, therefore, reflect unresolved sentence-bucket competition, where useful cross-window structure in the local logits is left unused by independent decoding. Cross-window contextual aggregation, therefore, emerges as a third performance source — one that should be modelled and attributed separately from local evidence rather than absorbed into a stronger encoder. We study it through an explicit logit-space correction, keeping the contextual contribution structurally decoupled from the frozen retrieval model. Together, these contributions form an attribution methodology for non-invasive neural language decoding. Concretely: • Quantifying structural nuisances. We show that variable-duration decoding enables a sequence-length shortcut: in a sentence-level setup, signal-blind Gaussian noise yields 66.3% R@1 because duration, padding, or mask structure remains informative without neural content. Fixed-duration retrieval windows and stimulus-identity splits collapse this shortcut and repeated-stimulus leakage to near chance, providing a retrieval endpoint that is largely independent of non-neural structure. • Isolating window-level stimulus-locked evidence. Under these controls, independent fixed-window retrieval recovers measurable MEG–audio discriminability across two datasets, isolating window-level evidence as a quantifiable source of decoding performance before any contextual aggregation is applied. • Auditing cross-window contextual aggregation. Beyond the oracle diagnostic, we audit the contextual source using GCB, a logit-space correction that leaves the frozen retrieval model, candidate pool, and within-bucket ordering untouched. Under frozen local logits, this intervention estimates the recoverable contextual contribution: R@1 shifts from 44% to 52% on Gwilliams and from 22% to 29% on MOUS under the same fixed setting, and the contrast vanishes under random grouping or evidence-limited controls. Figure 1 : Three performance sources in stimulus-locked M/EEG-to-audio retrieval. (a) Representative structural shortcut. Variable-duration inputs provide one concrete example of a structural shortcut: when neural activity is replaced by duration-preserving Gaussian noise, above-chance decoding indicates reliance on signal length or padding structure rather than neural content. (b) Fixed-duration, stimulus-locked M/EEG windows and aligned audio windows are encoded into a shared feature space with a contrastive objective, yielding an independent window-level retrieval baseline. (c) Cross-window contextual aggregation is isolated as a separate score-space intervention. Each window first produces local retrieval logits over the full candidate pool, organised here into sentence buckets \{A,B,C\} , and independent decoding can yield wrong-sentence near misses, such as a sentence- C window selecting a bucket- B candidate. GCB pools the fixed local logits across windows from the same query sentence to estimate bucket-level support, then adds a bucket-wise bias to the original logits. In this example, aggregated evidence supports bucket C , correcting the cross-sentence error while leaving the encoders, candidate pool, and within-bucket ordering unchanged. 2 Related Work End-to-end brain-to-text decoding and evaluation

Read the complete text