Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation
Wenyu Huang, Pavlos Vougiouklis, Mirella Lapata, Jeff Z. Pan · 2025
Read Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation by Wenyu Huang, Pavlos Vougiouklis, Mirella Lapata, Jeff Z. Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11754.txt.
Introduction
Wenyu Huang and co-authors probe how language models handle questions that need facts chained across several documents. They feed models search results in different orders and under different attention settings. One surprise is that encoder-decoder models from the Flan-T5 family beat much larger decoder-only models on these tasks.
Shuffling the key documents changes scores in telling ways: both model types do best when the documents arrive in the same order as the chain of reasoning. Letting decoder-only models look both ways across the text, by relaxing their one-way mask, closes much of the gap. The team also finds that attention peaks run higher when the final answer is right, and uses that signal to lift scores further.
Catalog record
| CREATOR | Wenyu Huang, Pavlos Vougiouklis, Mirella Lapata, Jeff Z. Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11754
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation Wenyu Huang, Pavlos Vougiouklis, Mirella Lapata, Jeff Z. Pan Abstract Multi-hop Question Answering (MHQA) adds layers of complexity to question answering, making it more challenging. When Language Models (LMs) are prompted with multiple search results, they are tasked not only with retrieving relevant information but also employing multi-hop reasoning across the information sources. Although LMs perform well on traditional question-answering tasks, the causal mask can hinder their capacity to reason across complex contexts. In this paper, we explore how LMs respond to multi-hop questions by permuting search results (retrieved documents) under various configurations. Our study reveals interesting findings as follows: 1) Encoder-decoder models, such as the ones in the Flan-T5 family, generally outperform causal decoder-only LMs in MHQA tasks, despite being significantly smaller in size; 2) altering the order of gold documents reveals distinct trends in both Flan T5 models and fine-tuned decoder-only models, with optimal performance observed when the document order aligns with the reasoning chain order; 3) enhancing causal decoder-only models with bi-directional attention by modifying the causal mask can effectively boost their end performance. In addition to the above, we conduct a thorough investigation of the distribution of LM attention weights in the context of MHQA. Our experiments reveal that attention weights tend to peak at higher values when the resulting answer is correct. We leverage this finding to heuristically improve LMs’ performance on this task. Our code is publicly available at https://github.com/hwy9855/MultiHopQA-Reasoning . 1 Introduction (a) Qwen 2.5, head 3 (b) FlanT5, head 1 Figure 1: Context attention distribution of Qwen 2.5 1.5B and FlanT5 large for the same question, both captured from the last layer (last encoder layer for FlanT5). The gold documents are Doc_5 and Doc_10, where the reasoning direction is Doc_10 \rightarrow Doc_5. With bi-directional attention, FlanT5 allows Doc_5 to assign attention and “see” Doc_10 (red dashed box), whereas Qwen 2.5 with causal mask cannot. Language Models (LMs) based on Transformer architectures ( Vaswani et al., 2017 ) have become essential tools for a wide variety of tasks, including question answering and conversational search ( Zhuang et al., 2023 ; Yi et al., 2024 ; Mo et al., 2024 ) . Among the different architectures, causal decoder-only configurations have become a popular choice for many of the most widely known LM families ( Llama Team, 2024 ; Qwen, 2025 ) . LMs exhibit a strong ability to reason within their input context, allowing for adaptability and effective generalisation across a wide range of tasks OpenAI (2024) . Albeit these capabilities, previous studies have highlighted significant challenges regarding the extent to which LMs can reason across different input contexts ( Zhang et al., 2023 ; Kaddour et al., 2023 ) . An exemplary case is the “lost in the middle” problem ( Liu et al., 2024 ) , where crucial information positioned in the middle of the context may be overlooked by LMs. As LMs are increasingly utilised in complex scenarios, their ability to reason across different contexts becomes critically important. The general Retrieval-augmented Generation (RAG) framework Lewis et al. (2020) ; Li et al. (2024) has recently become the cornerstone of many search-based conversational agents, such as Copilot and Doubao. In this setting, LMs frequently need to synthesise information from multiple retrieved search results to provide coherent answers. This study conducts an in-depth analysis of how LMs reason across various contexts, specifically focusing on multi-hop question answering (MHQA). Within the RAG framework, MHQA necessitates synthesising knowledge from multiple documents, presenting a more complex level of information integration than other more trivial question-answering tasks. To successfully perform MHQA, LMs must not only identify the most relevant documents within a given context but also reason with the information from these documents to determine the correct answer. A critical research question arises from the architecture of modern causal decoder-only Transformer-based LMs, which use a causal mask during both training and inference. This constraint hinders these models from performing bi-directional encoding ( Raffel et al., 2020 ) , as opposed to traditional encoder-decoder architectures that can capture interactions between documents more effectively (cf. Figure 1 ). This leads us to examine whether these widely used causal decoder-only LMs are limited by this constraint. Further, what might be the impact if we replaced the causal mask with a bi-directional (i.e. prefix) mask? To address these inquiries, we conduct a comprehensive investigation about how LMs model the MHQA task. Initially, we evaluate the MHQA performance of three widely adopted open-source LM families: the Flan-T5 ( Chung et al., 2024 ) family, representing the traditional encoder-decoder architecture, and the Qwen2.5 ( Qwen, 2025 ) and Llama 3.x ( Llama Team, 2024 ) families, exemplifying two popular causal decoder-only LMs. We design three types of document permutations to investigate the LMs’ behaviour in terms of: 1) the order of gold documents 1 1 1 In the context of MHQA, they usually refer to the documents needed to answer the single-hop questions into which the original multi-hop question is decomposed. , 2) the distance between them, and 3) their completeness. Our observations include the following: 1) The encoder-decoder model (Flan T5) is a superior MHQA solver when no fine-tuning takes place; 2) fine-tuned LMs tend to favour forward-placed documents ( Chen et al., 2024 ) (i.e., when the order of gold documents in the context mirrors the order of the reasoning chain, cf. Figure 2 ), a trend also observed in the Flan T5 models; 3) bi-directional attention with prefix mask can benefit LMs in MHQA tasks and offers better robustness when the order of gold documents is altered; 4) the distance between gold documents significantly affects performance, and; 5) while the removal of the first hop document reduces MHQA performance, a relatively high level of correctness is still maintained. Building on the above observations, we delve deeper to analyse how LMs perform MHQA by examining the attention distribution across layers. Our findings reveal that LMs typically assign higher attention score weights to at least one document when they correctly answer a multi-hop question. By sampling answers with different input document permutations, and retaining the answers for the inputs for which the LM assigned the largest peak attention score, we increased the accuracy of Qwen 7B from 28.6% to 33.7%. 2 Related Works 2.1 RAG and MHQA The RAG framework has been widely used to inject external knowledge into LMs ( Gao et al., 2024 ) , and mitigate their hallucination tendencies ( Huang et al., 2025 ) . In this framework, LMs generate a response by conditioning it on the top search results, which are provided as input context. This setup has shown promising performance in a variety of tasks, including knowledge graph question answering ( Luo et al., 2024 ; Huang et al., 2024 ) , open-domain dialogue generation ( Wang et al., 2024 ) , and multi-hop question answering ( Trivedi et al., 2022 ) . Some studies have focused on investigating how LMs use the input context, identifying discrepancies in how different parts of the input are processed. For instance, Mallen et al. (2023) find that retrieval may sometimes harm model predictions and Liu