What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval
Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang · 2026
Read What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval by Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24524.txt.
Introduction
Xinyu Zhang and colleagues audit non-invasive brain-to-language retrieval, asking how much reported accuracy comes from real stimulus evidence rather than shortcuts. Signal-blind noise alone reached 66.3 percent Rank at 1 under variable-length decoding. It collapsed to near chance once fixed windows and identity splits were enforced. Under those controls, fixed-window retrieval recovered genuine MEG-audio discriminability, though most Top-1 errors picked the wrong sentence. A Group Context Bias intervention lifted Rank at 1 from 44 to 52 percent on Gwilliams and 22 to 29 percent on MOUS.
Catalog record
| CREATOR | Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 122852 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24524
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval Xinyu Zhang, Sichao Liu, Runhao Lu, Alexandra Woolgar, Lihui Wang Abstract In non-invasive neural language decoding, results can be inflated by sources that are not stimulus-evoked neural evidence: decoder priors, embedding-based metrics, and non-neural structural nuisances such as signal duration. The methodological challenge is therefore attribution: a reported gain is more informative when it can be traced to a specific source. We recast stimulus-locked MEG-to-audio retrieval as an auditing framework that separates apparent performance into three sources — structural shortcuts, window-level stimulus-locked evidence, and cross-window contextual aggregation — and provides a diagnostic for each. Signal-blind Gaussian noise reaches 66.3% Rank@1 (R@1) under variable-length decoding but collapses to near chance once fixed-duration windows and stimulus-identity splits are enforced, isolating structural leakage. Under these controls, fixed-window retrieval recovers measurable MEG–audio discriminability, while an oracle sentence-bucket diagnostic shows that 95.7% of Top-1 errors select the wrong sentence, localising the residual bottleneck to sentence-level competition. We audit this contextual source with Group Context Bias (GCB), an inference-time additive logit bias that pools sentence-consistent evidence across windows while leaving the base retrieval scores and candidate pool fixed. Used as a score-space intervention, GCB makes the contextual source measurable: R@1 shifts from 44% to 52% on Gwilliams and from 22% to 29% on MOUS under the same fixed setting. GCB is auditable under this design: its effect collapses under random-grouping perturbations and vanishes when local evidence is attenuated in MEG or is near chance in EEG, supporting its use as a controlled source-attribution intervention. These results suggest that brain-to-language performance should be source-attributed, not merely reported. † † footnotetext: \dagger : equal contributors; \ddagger Corresponding author: [email protected] 1 Introduction Non-invasive neural language decoding has advanced rapidly with the introduction of pretrained speech and language models, and end-to-end systems increasingly report strong performance on text generation and semantic matching benchmarks [ 42 , 14 , 43 , 44 , 27 ] . Yet reported accuracy in this setting is often not what it appears to be. Under a standard variable-length sentence-decoding protocol, replacing real MEG with Gaussian noise still yields 66.3% next-token R@1 accuracy — variable-duration inputs introduce a sequence-length shortcut [ 18 , 3 ] that attention-based models exploit through padding boundaries and mask patterns. Duration is one instance of a broader pattern: decoder priors [ 25 ] , embedding-based metrics [ 46 , 21 ] , training-evaluation leakage [ 10 , 45 ] , and other non-neural structural nuisances can each make signal-blind inputs appear informative, so reported gains can reflect evaluation shortcuts rather than decodable linguistic information in the neural signal. The methodological challenge is therefore attribution: how can we verify whether apparent brain-to-language decoding performance is genuinely supported by stimulus-locked neural evidence, rather than by structural shortcuts, decoder priors, or evaluation artefacts? Text generation is a poor endpoint for this kind of attribution. Neural evidence, decoder priors, decoding strategy, and text-level metrics are all entangled in the final output, so a high semantic score or fluent generation alone does not identify the source of the information being used. This ambiguity is especially acute when a neural encoder conditions a strong speech, language, or generative prior: plausible outputs can be produced even when the neural representation contributes little stimulus-specific evidence. Stimulus-locked MEG-to-audio retrieval offers a more controlled endpoint [ 13 ] : each neural window retrieves its time-aligned audio window from an explicit candidate pool, making the candidate set, target alignment, chance level, and rank errors directly observable. Retrieval therefore turns apparent decoding performance into auditable candidate-level decisions, allowing structural shortcuts, local stimulus-locked evidence, and contextual effects to be analysed separately. Within this auditable setting, retrieval performance arises from multiple sources, and we trace them in turn. To isolate window-level stimulus-locked evidence, we first use independent fixed-window retrieval: each window is decoded separately using fixed-duration windows and stimulus-identity splits, preventing variable duration and repeated stimulus content from acting as direct ranking cues [ 10 , 45 , 13 , 15 ] . This local baseline shows MEG–audio discriminability, indicating that the window-level stimulus-locked neural signal is decodable under the controlled retrieval setting. However, this local-only design factorises retrieval over individual windows and therefore does not condition on information beyond the current window, despite evidence that linguistic context modulates time-locked neural responses during natural speech [ 39 , 8 , 35 ] . The error structure confirms this gap: under a global candidate pool, 95.7% of Top-1 errors select a candidate from the wrong sentence, and an oracle diagnostic that restricts candidates to the ground-truth sentence raises R@1 from 44% to 88% with the model held fixed. Many failures, therefore, reflect unresolved sentence-bucket competition, where useful cross-window structure in the local logits is left unused by independent decoding. Cross-window contextual aggregation, therefore, emerges as a third performance source — one that should be modelled and attributed separately from local evidence rather than absorbed into a stronger encoder. We study it through an explicit logit-space correction, keeping the contextual contribution structurally decoupled from the frozen retrieval model. Together, these contributions form an attribution methodology for non-invasive neural language decoding. Concretely: • Quantifying structural nuisances. We show that variable-duration decoding enables a sequence-length shortcut: in a sentence-level setup, signal-blind Gaussian noise yields 66.3% R@1 because duration, padding, or mask structure remains informative without neural content. Fixed-duration retrieval windows and stimulus-identity splits collapse this shortcut and repeated-stimulus leakage to near chance, providing a retrieval endpoint that is largely independent of non-neural structure. • Isolating window-level stimulus-locked evidence. Under these controls, independent fixed-window retrieval recovers measurable MEG–audio discriminability across two datasets, isolating window-level evidence as a quantifiable source of decoding performance before any contextual aggregation is applied. • Auditing cross-window contextual aggregation. Beyond the oracle diagnostic, we audit the contextual source using GCB, a logit-space correction that leaves the frozen retrieval model, candidate pool, and within-bucket ordering untouched. Under frozen local logits, this intervention estimates the recoverable contextual contribution: R@1 shifts from 44% to 52% on Gwilliams and from 22% to 29% on MOUS under the same fixed setting, and the contrast vanishes under random grouping or evidence-limited controls. Figure 1 : Three performance sources in stimulus-locked M/EEG-to-audio retrieval. (a) Representative structural shortcut. Variable-duration inputs provide one concrete example of a structural shortcut: when neural activity is replaced by duration-preserving Gaussian noise, above-chance decoding indicates reliance on signal length or padding structure rather than neural content. (b) Fixed-duration, stimulus-locked M/EEG windows and aligned audio windows are encoded into a shared feature space with a contrastive objective, yielding an independent window-level retrieval baseline. (c) Cross-window contextual aggregation is isolated as a separate score-space intervention. Each window first produces local retrieval logits over the full candidate pool, organised here into sentence buckets \{A,B,C\} , and independent decoding can yield wrong-sentence near misses, such as a sentence- C window selecting a bucket- B candidate. GCB pools the fixed local logits across windows from the same query sentence to estimate bucket-level support, then adds a bucket-wise bias to the original logits. In this example, aggregated evidence supports bucket C , correcting the cross-sentence error while leaving the encoders, candidate pool, and within-bucket ordering unchanged. 2 Related Work End-to-end brain-to-text decoding and evaluation