Attention-guided Evidence Grounding for Spoken Question Answering

Ke Yang, Bolin Chen, Yuejie Li, Yueying Hua, Jianhao Nie, Yueping He, Bowen Li, Chengjun Mao · 2026

Read Attention-guided Evidence Grounding for Spoken Question Answering by Ke Yang, Bolin Chen, Yuejie Li, Yueying Hua, Jianhao Nie, Yueping He, Bowen Li, Chengjun Mao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16292.txt.

Introduction

Yang and co-authors answer spoken questions without first transcribing the speech. Cascaded systems that use speech recognition add delay and spread errors. Their AEG framework reads the speech model's own cross-modal attention. That attention locates the key evidence inside the model's latent space. A fine-tuning method, Learning to Focus on Evidence, sharpens the attention. Tests on SQuAD, HotpotQA, and MuSiQue cut hallucinations. Latency fell by about 62% versus strong cascaded baselines.

Catalog record

CREATORKe Yang, Bolin Chen, Yuejie Li, Yueying Hua, Jianhao Nie, Yueping He, Bowen Li, Chengjun Mao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30474
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16292

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention-guided Evidence Grounding for Spoken Question Answering Ke Yang, Bolin Chen, Yuejie Li, Yueying Hua, Jianhao Nie, Yueping He, Bowen Li, Chengjun Mao Abstract Spoken Question Answering (Spoken QA) presents a challenging cross-modal problem: effectively aligning acoustic queries with textual knowledge while avoiding the latency and error propagation inherent in cascaded ASR-based systems. In this paper, we introduce Attention-guided Evidence Grounding (AEG), a novel end-to-end framework that leverages the internal cross-modal attention of Speech Large Language Models (SpeechLLMs) to explicitly locate and ground key evidence in the model’s latent space. To address the diffuse attention distribution in pre-trained models, we propose Learning to Focus on Evidence (LFE), a supervised fine-tuning paradigm that calibrates the model’s attention mechanism to distinguish query-relevant segments from irrelevant context. Experiments on SQuAD, HotpotQA, and MuSiQue demonstrate that AEG reduces hallucinations and achieves strong efficiency gains, outperforming large-scale cascaded baselines (Whisper-Large-v3 + Reranker) while reducing inference latency by approximately 62%. I Introduction Spoken Question Answering (Spoken QA) [ 1 , 2 ] is a cross-modal task that requires models to answer spoken queries based on textual contexts. However, even when provided with the correct context, current Spoken QA systems frequently generate responses that are inconsistent with the source content, resulting in hallucinations [ 3 , 4 ] . This severely limits their deployment in critical, high-stakes scenarios such as medicine [ 5 ] and legal consultation [ 6 ] . Furthermore, existing approaches lack explicit evidence grounding, limiting interpretability and preventing user verification of supporting contexts. Therefore, improving the factual accuracy and interpretability of spoken QA systems is crucial for their reliable deployment in real-world applications. Inspired by human cognitive processes, our work explores a more intrinsic and intuitive approach to spoken QA. When humans perform a similar task, they intuitively follow a ”scan-then-focus” process: Given a large amount of context, they first scan and identify key information segments relevant to the question—i.e., the key evidence. Subsequently, they focus their attention on these key evidence to construct an answer, rather than attempting to process all context simultaneously. This two-stage process helps to reduce irrelevant interference during reasoning and ensures that the answer is based on verified information. We posit that a similar mechanism can be instilled in Speech Large Language Models (SpeechLLMs). Intuitively, as an LLM processes a query and its context, its internal attention mechanism is already dynamically computing the importance of different information segments [ 7 , 8 ] , creating a theoretical ”heat map” of relevance. Hence, attention offers a promising but largely under-exploited signal for evidence grounding [ 9 , 10 ] . Motivated by this insight, we first propose Grounding with Attention , which leverages the internal attention mechanism of a pre-trained SpeechLLM to locate and explicitly mark key evidence. These marked segments provide clear attribution for the model’s responses, enhancing interpretability by revealing the specific context used during answer generation. Fig. 1: A comparison demonstrating the critical role of Learning to Focus on Evidence (LFE). For the audio query ”When was the Governor ended?”, our complete AEG framework successfully grounds the answer in the correct evidence (Doc 3), while both AEG without LFE and baseline methods fail to identify relevant evidence, resulting in incorrect responses. While attention-based grounding works well in text-only settings, it is unreliable in cross-modal speech–text scenarios, where attention must align heterogeneous acoustic and text representations and thus requires explicit training. As a result, the raw attention of pre-trained SpeechLLMs is often diffuse and uncalibrated, failing to clearly distinguish key evidence from related context, as illustrated in Figure 1 . To address this issue, we propose Learning to Focus on Evidence (LFE) , a specialized training paradigm designed to ”teach” the LLM to perform the ”scan-then-focus” process. Through task-specific fine-tuning, LFE reshapes the model’s attention distribution, producing a sharper distinction between key evidence and irrelevant context. This enables the model to concentrate more effectively on key evidence. Overall, we present Attention-guided Evidence Grounding (AEG) , a controllable framework that starts from grounding with attention and provides a trainable way to guide a model’s focus on relevant evidence. By transforming the model’s implicit attention patterns into explicit evidence markers, AEG significantly enhances both the factual accuracy and the interpretability of the generated responses. Our main contributions are as follows: • We propose AEG , a framework that leverages the internal attention mechanism of SpeechLLMs to explicitly locate key evidence within the context, thereby guiding the model to generate well-grounded answers. • We introduce LFE , a specialized fine-tuning paradigm that calibrates the model’s attention distribution. This process sharpens the focus on critical information, enabling the model to filter out irrelevant noise effectively. • Experiments on multiple Spoken QA benchmarks demonstrate that AEG significantly improves evidence selection precision and enhances factual correctness, effectively boosting the overall reliability of the system against hallucinations. II Related Work Building on the success of Large Language Models (LLMs) in the text domain, a natural next step is extending this paradigm to other modalities, particularly speech. Traditionally, tasks like Spoken Question Answering have used cascaded ASR-LLM-TTS architectures [ 11 ] , but this approach suffers from error propagation, high latency, and loss of paralinguistic information. To address these limitations, the research trend is shifting towards integrated, end-to-end (E2E) SpeechLLMs [ 12 , 13 , 14 , 15 , 16 , 17 ] . These models directly process speech input and generate speech or text output, enabling a deeper fusion of acoustic and semantic information. Despite advancements in E2E architectures, SpeechLLMs still face the persistent challenge of ”hallucination”: the generation of outputs are plausible but factually incorrect or unfaithful to the context. Although Retrieval-Augmented Generation (RAG) can mitigate this issue, the ”faithfulness” of the model to the given context remains a critical concern. Research indicates that even when the correct documents are retrieved, models may still ignore context [ 18 ] , become ”confused” by contradictory information, or ”selectively” use information. Furthermore, model efficacy is often hampered by the ”lost in the middle” phenomenon [ 19 ] , where LLMs exhibit a strong positional bias, favoring information at the beginning and end of the context while overlooking critical evidence in the middle. Inspired by these challenges, recent research has explored more fine-grained contextual utilization, predominantly within the text modality. Relevant efforts include re-ranking documents based on attention scores to counteract positional bias [ 20 ] , employing auxiliary evidence extractors [ 21 ] , or leveraging self-attention mechanisms to highlight evidence [ 22 ] . However, these methods are limited to processing purely textual information. They fail to address a critical issue in multimodal models: how to effectively align and ground the user’s acoustic information with external textual knowledge. In contrast, our work focuses on achieving direct alignment between speech queries and textual knowledge passages within multimodal models. III Methodology III-A Task Definition We formally define the spoken question answering task as follows. Let Q_{A} be an audio query and C_{T}=\{c_{1},c_{2},...,c_{k}\} be a context set, which serves as the exclusive knowledge source for answering the query. The task takes the audio query Q_{A} and context set C_{T} as inputs, which are processed by a speech large language model (SpeechLLM) to generate a textual response R_{T}

Read the complete text