Heterogeneous Neural Predictivity from Language Models During Naturalistic Comprehension

Xiao Jia · 2026

Read Heterogeneous Neural Predictivity from Language Models During Naturalistic Comprehension by Xiao Jia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26880.txt.

Introduction

Xiao Jia tests whether language model signals predict brain activity during natural story understanding. A model's internal features sum up a text or speech stream in rich detail. Those features can serve as predictors in models of brain responses. Jia analysed data from three public collections covering implanted probes, scalp sensors, and story listening.

Eight frozen language models supplied the features. Careful controls matched timing, nuisance signals, and model capacity. Positive prediction beyond low-level baselines was widespread. A stricter controlled criterion was met by a subset of cases, and removing model features changed scores in most of them. Advantages at the individual participant level were localised, not uniform. The author cautions that full strong claims need joint future data. The features clearly annotate brain activity, with bounded meaning.

Catalog record

CREATORXiao Jia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26880

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Heterogeneous Neural Predictivity from Language Models During Naturalistic Comprehension Xiao Jia Abstract Language-model representations provide structured, high-dimensional annotations of naturalistic language stimuli and can serve as informative neural predictors during comprehension. We analyzed locked derived data from Brain Treebank, MEG-MASC, and Podcast ECoG with eight frozen language models, blocked encoding models, and matched temporal, nuisance, and representation-capacity controls. Positive held-out prediction and gains over low-level baselines were widespread in source-level summaries. Across Brain Treebank and Podcast ECoG, 67 of 432 evaluable rows met a controlled predictive-only criterion, and model-side feature ablations changed prediction scores in most evaluable source rows. Brain-derived, timing-linked, acoustic, and implanted-signal controls confirmed component-level sensitivity of the analysis pipeline. These findings show that language-model-derived quantities can annotate neural activity during natural speech and text comprehension. Participant-level matched-control advantages were localized rather than uniform, response-profile and feature-specificity contrasts bounded representational or computational interpretations, and complete co-indexed integrated interpretation will require future jointly indexed coverage. Together, the analyses identify language-model features as useful neural predictors and separate predictive usefulness from claims about shared neural organization or language-processing computations. Keywords: naturalistic language comprehension, language models, cognitive computational neuroscience, neural encoding, positive controls, evidence calibration 1 Introduction Language-model representations have become effective quantitative probes of naturalistic language comprehension. Contextual embeddings, surprisal estimates, and layer-specific features can predict neural responses measured with fMRI, MEG, EEG, and intracranial electrophysiology, providing a tractable way to relate unfolding linguistic context to neural activity ( 13 ; 23 ; 3 ; 7 ; 27 ; 28 ; 22 ; 34 ; 12 ; 14 ) . This predictive success is scientifically useful because language-model features provide structured, high-dimensional representations of the stimulus that can account for variation in neural responses. The next question is what kind of neural and cognitive information this predictivity supports. Positive neural prediction can support several levels of inference. A feature may predict held-out neural responses because it tracks stimulus properties relevant to the measurement. Additional evidence can show whether the trained representation exceeds matched temporal, nuisance, and capacity controls, whether it reproduces organization across sampled neural units, or whether it depends selectively on a candidate language-related quantity. The present study treats these outcomes as separable empirical claims. Naturalistic stimuli make this separation important. Word onset, word rate, acoustic envelope, sentence position, discourse progression, lexical frequency, token predictability, and local transition statistics are intercorrelated. Neural measurements also contain temporal autocorrelation, and modern language-model features are high-dimensional, layer-indexed, and context-length-dependent ( 32 ; 4 ; 2 ; 36 ; 37 ) . Positive neural prediction may therefore combine language-related information with temporal, lexical, acoustic, and representation-capacity contributions. Recent language-neuroscience work motivates a more explicit link between computational measures and cognitive interpretation. Theoretical conclusions depend on the chain connecting constructs, tasks, measurements, analyses, and auxiliary assumptions ( 29 ) . Syntax–semantics, surprisal, and narrative-comprehension studies show the value of linking computational measures to specified neural response patterns and interpretive targets ( 24 ; 35 ; 26 ) . Hadidi and colleagues further show that shuffled training and testing partitions, activation-extraction choices, positional signals, and word-rate controls can strongly affect brain–language-model predictivity ( 9 ) . These results motivate a narrower question: how much of the observed signal supports predictive usefulness, model-specific advantage, shared neural organization, or candidate-computation interpretation. Here we characterize heterogeneous neural predictivity from language-model features across three naturalistic language datasets. We quantify positive neural predictions, gains over nuisance baselines, controlled predictive-only rows, participant-level consistency, and sensitivity to model-side ablation. We then compare these outputs with matched controls, response-profile tests, feature-specificity diagnostics, and reliability-bounded summaries to determine which interpretations are supported by the available derived data. 2 Materials and Methods 2.1 Participants and data sources This secondary analysis used preprocessed and derived data from previously released sources; raw neural recordings were governed by the original data providers and were not redistributed. Three datasets were treated as the primary naturalistic language sources because the available derived data contained neural time series or neural targets, word-level event grids, language-model features, nuisance variables, matched controls, and reliability or coverage metadata ( 33 ; 39 ; 8 ) . Brain Treebank contributed 10 participants, 26 subject-run units, and 248 modality-specific brain units. Podcast ECoG contributed 9 participants, 9 subject-run units, and 235 ECoG-derived brain units. MEG-MASC contributed 11 participants, 84 subject-run units, and 257 MEG-derived target units. These brain units are electrodes, sensors, source targets, time windows, or derived target profiles depending on the source dataset. They are distinct from independent participants. The sample size was determined by the secondary-data design and by the subset of publicly released or locally accessible derived data that could be matched across neural targets, word-event grids, model features, controls, and reliability metadata. No new participants were recruited, and participant expansion was outside the available derived-data scope. Inference is therefore bounded by the participant and participant-run coverage retained for each contrast. The matched participant-run predictive inference retained 26 Brain Treebank subject-run units from 10 participants, 44 MEG-MASC subject-run units from 11 participants, and 8 Podcast ECoG subject-run units from 8 participants after complete model-control matching. When a participant contributed multiple runs, runs were averaged within participant before participant-cluster bootstrap. These coverage counts define the predictive inference boundary; electrodes, sensors, layers, and table rows define nested analysis dimensions. The source datasets contain material beyond the subset with complete local feature matching. Brain Treebank records intracranial electrophysiology while neurosurgical participants watched naturalistic movie stimuli, with manually corrected transcripts, word onsets, part-of-speech labels, and dependency parses aligned to the audio track ( 33 ) . Podcast ECoG records intracranial responses while participants listened to a natural spoken podcast, with high-gamma preprocessed derivatives and linguistic feature annotations available from the source release ( 39 ) . MEG-MASC records English-speaking participants listening to naturalistic MASC stories across repeated MEG sessions that also include word-list and comprehension-question material ( 8 ) . The present manuscript analyzes the participant-run, event-grid, representation, and control rows that could be matched in the derived data. The source inventory retained Narratives and LPP multilingual as secondary or exploratory sources ( 18 ; 16 ) . Learning Brain was treated as validation-only, and Natural Stories was treated as stimulus–language-model-only or diagnostic because the available derived data lacked the neural-side coverage required for the present contrasts ( 6 ; 20 ) . Participant demographics, original exclusion criteria, ethics approvals, and consent procedures are governed by the source publications and repositories. The present manuscript reports the derived units available to this analysis and preserves de-identification and raw-data access boundaries. 2.2 Naturalistic stimuli and neural measurements The three primary datasets sample naturalistic comprehension with different measurement modalities. Brain Treebank provides intracranial recordings during naturalistic audiovisual or language stimuli. Podcast ECoG provides intracranial recordings during podcast comprehension. MEG-MASC provides MEG responses to natural speech. The local analysis inherited each source’s preprocessing, artifact rejection, and target definition from the released or derived artifacts. Word onset and event grids were used to align language-model features and nuisance variables to the neural targets. The primary inferential unit was the participant or participant-run whenever the matched data allowed that aggregation. Electrodes, sensors, target windows, layers, models, metrics, and candidate quantities were treated as nested or crossed analysis dimensions. Row counts describe coverage

Read the complete text