AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Xilin Jiang, Sukru Samet Dindar, Vishal Choudhari, Stephan Bickel, Ashesh Mehta, Guy M McKhann, Daniel Friedman, Adeen Flinker, Nima Mesgarani · 2025
Read AAD-LLM: Neural Attention-Driven Auditory Scene Understanding by Xilin Jiang, Sukru Samet Dindar, Vishal Choudhari, Stephan Bickel, Ashesh Mehta, Guy M McKhann, Daniel Friedman, Adeen Flinker, Nima Mesgarani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16794.txt.
Introduction
Xilin Jiang and eight co-authors present a brain driven listening system in a 2025 paper. Human listeners can focus on one speaker in a noisy room, but AI audio models treat all sounds equally. Their system adds brain recordings to a large audio language model, so responses follow what the listener is attending to.
Electrodes inside the skull record activity while a person listens to several talkers. The model first decodes which speaker is attended, then generates answers conditioned on that choice. Tests covered speaker description, speech transcription, and question answering in multi talker scenes. Both measured scores and human ratings improved with attention guidance. The work is a first step toward listener centered audio AI.
Catalog record
| CREATOR | Xilin Jiang, Sukru Samet Dindar, Vishal Choudhari, Stephan Bickel, Ashesh Mehta, Guy M McKhann, Daniel Friedman, Adeen Flinker, Nima Mesgarani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16794
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AAD-LLM: Neural Attention-Driven Auditory Scene Understanding Xilin Jiang, Sukru Samet Dindar, Vishal Choudhari, Stephan Bickel, Ashesh Mehta, Guy M McKhann, Daniel Friedman, Adeen Flinker, Nima Mesgarani Abstract Auditory foundation models, including auditory large language models (LLMs), process all sound inputs equally, independent of listener perception. However, human auditory perception is inherently selective: listeners focus on specific speakers while ignoring others in complex auditory scenes. Existing models do not incorporate this selectivity, limiting their ability to generate perception-aligned responses. To address this, we introduce Intention-Informed Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM (AAD-LLM), a prototype system that integrates brain signals to infer listener attention. AAD-LLM extends an auditory LLM by incorporating intracranial electroencephalography (iEEG) recordings to decode which speaker a listener is attending to and refine responses accordingly. The model first predicts the attended speaker from neural activity, then conditions response generation on this inferred attentional state. We evaluate AAD-LLM on speaker description, speech transcription and extraction, and question answering in multitalker scenarios, with both objective and subjective ratings showing improved alignment with listener intention. By taking a first step toward intention-aware auditory AI, this work explores a new paradigm where listener perception informs machine listening, paving the way for future listener-centered auditory systems. Demo and code available 1 1 1 https://aad-llm.github.io . 0 0 footnotetext: contribute equally. Figure 1: AAD-LLM is a brain-computer interface (BCI) for auditory scene understanding. It decodes neural signals to identify the attended speaker and integrates this information into a language model, generating responses that align with the listener’s perceptual focus. 1 Introduction The human auditory system does not process all sounds equally but selectively amplifies relevant elements while suppressing others based on the listener’s intent Mesgarani and Chang (2012) . In a multi-speaker setting, a listener may focus on a single speaker, tune into background music, or ignore speech entirely Cherry (1953) ; Osgood (1959) ; Shinn-Cunningham and Best (2008) . While modern auditory foundation models, including auditory large language models (LLMs), are designed for general-purpose auditory understanding, they do not inherently account for listener intent. In applications such as assistive hearing devices, however, listener-aware processing is critical. These systems must prioritize the content most relevant to the user to improve usability in complex acoustic environments. Existing models, such as LTU Gong et al. (2024) , SALMONN Tang et al. (2024) , and Qwen2-Audio Chu et al. (2024) , process all incoming audio equally, making them ineffective in scenarios where distinguishing between attended and unattended speech is essential. Although these models excel at general auditory scene understanding Sakshi et al. (2024) ; Wang et al. (2025) , they lack mechanisms to selectively process speech based on listener perception. In multi-speaker environments, they transcribe and analyze all speech sources indiscriminately, failing to separate what the user is actually attending to from background conversations Wu et al. (2024) . To address this, a listener-aware auditory AI must move beyond passive transcription and actively adapt its processing to reflect user intent. Studies have shown that the auditory cortex encodes speech features from the attended talker Mesgarani and Chang (2012) , allowing neural decoding methods to reconstruct or enhance the target speaker by comparing brain signal-derived representations with competing speech O’sullivan et al. (2015) . This line of research, also known as auditory attention decoding (AAD), has sought to infer a listener’s auditory focus from neural signals. Both invasive Han et al. (2019) ; Ceolini et al. (2020) ; Choudhari et al. (2024) and non-invasive methods Ciccarelli et al. (2019) ; Geirnaert et al. (2021) ; Vandecappelle et al. (2021) ; Pan et al. (2024) ; Pan et al. (2023) have been explored for attention-controlled speech extraction in hearing devices. However, AAD is primarily used for signal enhancement rather than guiding an AI’s interpretation of an auditory scene. While it could improve speech intelligibility, it does not enable models to reason about the attended content, such as summarizing speech or answering user queries based on what they perceive. Prior studies have attempted to integrate neural signals into large language models models, incorporating neural signals to enhance multimodal processing. Some efforts have used neural data for EEG-based text generation Jiang et al. (2024a) ; Kim et al. (2024) or fMRI-informed representations Zheng and Sun (2024) , while others have integrated brain signals to improve LLM semantic understanding Toneva and Wehbe (2019) ; Moussa et al. (2025) . However, these approaches focus primarily on language comprehension and semantic alignment, rather than using neural signals to refine auditory scene interpretation or speaker selection. This paper introduces Intention-Informed Auditory Scene Understanding (II-ASU), a framework where models align their interpretation of sound with listener intent. Instead of modifying the auditory signal like AAD, we integrate attentional signals into a language model to guide how it processes and responds to auditory scenes. This approach enables reasoning beyond speech separation, allowing models to adapt responses based on listener focus rather than treating all input speech equally. To implement this, we present Auditory Attention-Driven LLM (AAD-LLM), a prototype system that extends an auditory LLM with neural attention decoding. The model processes intracranial EEG (iEEG) to determine which speaker the listener is attending to, extracts speech representations while retaining both attended and ignored sources for contextual processing, and integrates the decoded attentional state into the LLM to generate responses that reflect listener perception. This work makes several contributions. It introduces II-ASU as a paradigm shift from passive auditory processing to listener-aligned interpretation; It proposes AAD-LLM as the first system to integrate brain signals into an auditory LLM for attention-driven scene understanding; Finally, it evaluates AAD-LLM across multiple auditory tasks, demonstrating improved alignment with listener perception. While this study focuses on selective attention, the broader II-ASU framework could extend to other intent-detection methods, including gaze tracking, head orientation, posture shifts, or explicit user input. By incorporating such attentional signals, future intention-aware auditory AI could dynamically adapt to user perception in a wide range of applications. 2 Intention-Informed Auditory Scene Understanding 2.1 Motivation and Goal Consider the scenario illustrated in Figure 1 : both a human and a machine listener are exposed to the same auditory scene, denoted as S . The human listener applies selective auditory attention forming an internal representation based on their intent I . When asked a question Q , an intention-informed auditory model, such as AAD-LMM, should generate an answer A that depends on Q , S , and I : \displaystyle A=\text{MachineListener}(Q,S,I) (1) However, existing auditory LLMs Gong et al. (2024) ; Hu et al. (2024) ; Chu et al. (2024) are not intention-informed. They function as: \displaystyle A^{\circ}=\text{AuditoryLLM}(Q,S) (2) without any awareness of what the listener actually perceives, leading to intention-uninformed response A^{\circ} . In real-world scenarios, the listener intention manifests in multiple ways—through facial expressions, eye gaze, head direction, verbal commands, or physical actions—but behavioral data capturing these cues remains scarce. As the first step towards intention-informed auditory scene understanding in LLMs, we focus on the most fundamental yet unexplored form of auditory intention: selective auditory attention. In this context, I represents the attention of the listener to specific sound source while ignoring the other. Unlike explicit commands, this cognitive state is abstract and non-trivial to encode as input for