Towards Brain Passage Retrieval

Niall Mcguire, Yashar Moshfeghi · 2024

Read Towards Brain Passage Retrieval by Niall Mcguire, Yashar Moshfeghi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06695.txt.

Introduction

Niall McGuire and Yashar Moshfeghi propose in 2024 a search system that reads the brain directly, skipping the step where a user must turn a need into typed words. That step is hard for people with vague needs or with limits on typing.

Their framework maps brain signals from scalp sensors and text passages into one shared meaning space. A search then pulls passages that sit close to the brain pattern. Tests on a known data set with thirty readers beat older brain to text methods on precision by a clear margin. Extra tests showed that hard negative examples and a brain encoder built for the job were key parts of the gain.

Catalog record

CREATORNiall Mcguire, Yashar Moshfeghi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06695

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Brain Passage Retrieval Niall Mcguire, Yashar Moshfeghi Abstract . Information Retrieval (IR) systems primarily rely on users’ ability to translate their internal information needs into (text) queries. However, this translation process is often uncertain and cognitively demanding, leading to queries that incompletely or inaccurately represent users’ true needs. This challenge is particularly acute for users with ill-defined information needs or physical impairments that limit traditional text input, where the gap between cognitive intent and query expression becomes even more pronounced. Recent neuroscientific studies have explored Brain-Machine Interfaces (BMIs) as a potential solution, aiming to bridge the gap between users’ cognitive semantics and their search intentions. However, current approaches attempting to decode explicit text queries from brain signals have shown limited effectiveness in learning robust brain-to-text representations, often failing to capture the nuanced semantic information present in brain patterns. To address these limitations, we propose BPR ( B rain P assage R etrieval ), a novel framework that eliminates the need for intermediate query translation by enabling direct retrieval of relevant passages from users’ brain signals. Our approach leverages dense retrieval architectures to map EEG signals and text passages into a shared semantic space. Through comprehensive experiments on the ZuCo dataset, we demonstrate that BPR achieves up to 8.81% improvement in precision@5 over existing EEG-to-text baselines, while maintaining effectiveness across 30 participants. Our ablation studies reveal the critical role of hard negative sampling and specialised brain encoders in achieving robust cross-modal alignment. These results establish the viability of direct brain-to-passage retrieval and provide a foundation for developing more natural interfaces between users’ cognitive states and IR systems. Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730097 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Search interfaces CCS: Information systems CCS: Information systems Information retrieval CCS: Information systems Users and interactive retrieval An Investigation of EEG Query Representations † † cc-license: by 1. Introduction One of the fundamental challenges in Information Retrieval (IR) lies in bridging the gap between users’ core information needs (INs) and their external formulation as (textual) queries ( Belkin, 1980 ; Ingwersen, 1996 ; Moshfeghi et al., 2016 ) . The cognitive complexity of query formulation often results in queries that incompletely or inaccurately represent users’ true needs, while physical constraints of input mechanisms can create additional barriers to effective information access ( Belkin, 1980 ; Wolpaw et al., 2002 ) . This challenge is particularly acute for users with ill-defined information needs or physical impairments that limit traditional text input, where limitations in conventional interaction mechanisms can potentially exclude significant user populations from effective information access ( Wolpaw et al., 2002 ) . Foundational models of information-seeking behaviour illuminate the depth of this challenge. Taylor ( Taylor, 1968 ) demonstrates that users must progress through distinct stages, from a visceral, unconscious information need to a compromised form suitable for an IR system, a progression that often results in significant information loss. Kuhlthau ( Kuhlthau, 2005 ) extends this understanding by revealing how users struggle with uncertainty and vague thoughts during early search stages, precisely when they must articulate their needs most explicitly. The combination of these cognitive and physical barriers creates a fundamental gap between users’ internal states and their ability to externalise these states through conventional interaction mechanisms ( Ingwersen, 1996 ; Belkin, 1980 ) . Figure 1. A) Traditional EEG-to-text pipeline requiring intermediate query decoding before retrieval. The approach first translates EEG signals into text queries before applying traditional (lexical/neural) text retrieval methods. B) Our proposed direct EEG query retrieval framework that eliminates the translation step by learning a shared embedding space between EEG signals and text passages, enabling direct relevance scoring between brain activity and documents. Recent neuroscientific advances have revealed promising approaches to bridging this gap through direct measurement of users’ cognitive states during information interaction ( Moshfeghi et al., 2016 ; Eugster et al., 2016 ; Gwizdka et al., 2019 ) . Studies have demonstrated that core IR processes manifest as distinct brain signatures, from information need formation ( Moshfeghi et al., 2019 ; Michalkova et al., 2022 ) to relevance judgements ( Kauppi et al., 2015 ; Allegretti et al., 2015 ; Pinkosova et al., 2020 ) and search satisfaction ( Paisalnan et al., 2022 ) . These neurophysiological insights have motivated researchers to explore brain-based approaches for enhancing query formulation. Initial implementations focused on Steady-State Visually Evoked Potentials (SSVEP) ( Wolpaw et al., 2002 ) , these approaches primarily changed the input mechanism rather than addressing the fundamental gap between need and expression. More recent work using functional Magnetic Resonance Imaging (fMRI) and Magnetoencephalography (MEG) has demonstrated increasingly sophisticated semantic decoding capabilities ( Jayalath et al., 2024 ) , from mapping word-level representations ( Mitchell et al., 2008 ) to reconstructing continuous narratives ( Huth et al., 2016 ) and generating natural language from brain patterns ( Ye et al., 2024a ) . While these advances validate the possibility of capturing information needs directly from brain signals, both fMRI and MEG have severe practical limitations for real-world IR applications, requiring immobile participants in specialised facilities ( Ye et al., 2024b ; Kauppi et al., 2015 ) and involving prohibitive equipment costs ( Allegretti et al., 2015 ) . Electroencephalography (EEG) offers a more practical path forward, providing high temporal resolution and mobility at a significantly lower cost ( Binnie and Prior, 1994 ; McGuire and Moshfeghi, 2023b ; McGuire and Moshfeghi, 2023a ; Kingphai and Moshfeghi, 2023 ) . Recent studies have demonstrated EEG’s ability to detect relevance judgments ( Pinkosova et al., 2020 ; Allegretti et al., 2015 ; Gwizdka et al., 2017 ; Michalkova et al., 2024 ; Pinkosova et al., 2022 ) , satisfaction ( Paisalnan et al., 2022 ) , and enable direct document recommendation from brain signals during reading ( Eugster et al., 2016 ; Zhang et al., 2024 ) . However, current EEG-based approaches for document retrieval require the translation of brain signals into textual queries ( Wang and Ji, 2022 ; Lamprou et al., 2024 ) (see Figure 1 ), which recent studies have identified as ineffective for learning generalisable semantic representations from EEG and rely on the memorisation of training data and teacher forcing to generate a textual query ( Jo et al., 2024 ) . As a solution to this problem, we present BPR ( B rain P assage R etrieval ), a framework that eliminates this translation step entirely by directly mapping brain signals to dense passage representations. Rather than attempting to convert brain activity into text queries, BPR projects brain patterns into the same semantic space as passages, treating brain activity itself as a query representation (see Figure 1 ). Our approach builds on dense retrieval architectures ( Karpukhin et al., 2020 ) but adapts them for brain data through specialised EEG encoders and cross-modal negative sampling strategies. Our contributions include: • The development of BPR, a novel framework for direct EEG-to-passage retrieval that achieves a 8.81% improvement in Precision@5 over existing approaches by eliminating

Read the complete text