Context-Length Robustness in Question Answering Models: A Comparative Empirical Study

Trishita Dhara, Siddhesh Sheth · 2026

Read Context-Length Robustness in Question Answering Models: A Comparative Empirical Study by Trishita Dhara, Siddhesh Sheth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15723.txt.

Introduction

Dhara and Sheth test how question answering degrades as context grows. Models now face long, noisy texts where the answer hides among filler. The pair ran controlled tests on SQuAD and HotpotQA. They added irrelevant context step by step and kept the answer signal fixed. Accuracy fell steadily as length grew in both sets. Multi-hop HotpotQA lost nearly twice as much accuracy as SQuAD. The authors argue context-length checks belong in standard reliability tests.

Catalog record

CREATORTrishita Dhara, Siddhesh Sheth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26323
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15723

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context-Length Robustness in Question Answering Models: A Comparative Empirical Study Trishita Dhara, Siddhesh Sheth Abstract Large language models are increasingly deployed in settings where relevant information is embedded within long and noisy contexts. Despite this, robustness to growing context length remains poorly understood across different question answering tasks. In this work, we present a controlled empirical study of context-length robustness in large language models using two widely used benchmarks: SQuAD and HotpotQA. We evaluate model accuracy as a function of total context length by systematically increasing the amount of irrelevant context while preserving the answer-bearing signal. This allows us to isolate the effect of context length from changes in task difficulty. Our results show a consistent degradation in performance as context length increases, with substantially larger drops observed on multi-hop reasoning tasks compared to single-span extraction tasks. In particular, HotpotQA exhibits nearly twice the accuracy degradation of SQuAD under equivalent context expansions. These findings highlight task-dependent differences in robustness and suggest that multi-hop reasoning is especially vulnerable to context dilution. We argue that context-length robustness should be evaluated explicitly when assessing model reliability, especially for applications involving long documents or retrieval-augmented generation. 1 Introduction Large language models have recently achieved strong performance on a wide range of question answering tasks, including both extractive and multi-hop reasoning benchmarks 5 ; 2 . In many practical applications, however, these models operate over long and heterogeneous contexts that contain a mixture of relevant and irrelevant information. Examples include retrieval-augmented generation, document analysis, and multi-document question answering systems. In such settings, models must reliably identify and reason over a small subset of context while ignoring distractors. Despite growing interest in long-context modeling 1 ; 4 , most evaluations of question answering systems assume a fixed or moderately sized input context. This assumption obscures an important aspect of model reliability: how performance changes as the total amount of available context increases while the underlying question and answer remain unchanged. As context length grows, additional irrelevant content may dilute the signal required for correct reasoning, potentially leading to systematic performance degradation. In this work, we study context-length robustness in large language models through a controlled empirical evaluation. Rather than altering question difficulty or answer ambiguity, we construct evaluation settings in which the answer-bearing content is preserved and progressively embedded within longer contexts by injecting additional irrelevant text. This design allows us to isolate the effect of context length itself from other confounding factors, such as changes in prompt formulation or task structure. We conduct experiments on two widely used question answering benchmarks with distinct characteristics: SQuAD 13 , which primarily involves single-span answer extraction from a given passage, and HotpotQA 17 , which requires multi-hop reasoning across multiple pieces of evidence. By applying equivalent context-length expansions to both datasets, we analyze how robustness to increasing context length varies across task types. Our results show that model accuracy consistently degrades as context length increases, even when the relevant information required to answer the question is retained. Moreover, the magnitude of degradation differs substantially across tasks: multi-hop reasoning exhibits significantly higher sensitivity to context expansion than single-span extraction. These findings indicate that robustness to long and noisy contexts is task-dependent and suggest that context-length robustness should be evaluated explicitly when assessing the reliability of question answering models. 2 Related Work Recent research has increasingly focused on the effects of context length on language model performance. Theoretical work by Shi et al. 14 proposes a framework explaining how intrinsic dimension and training dataset size influence optimal context length and the scaling behavior of language models. Complementary surveys, such as Pawar et al. 12 , provide comprehensive overviews of techniques proposed to support extended input sequences in transformers. Contemporaneous empirical investigations have demonstrated that extended context windows do not always translate into improved performance. For example, Du et al. 6 reveal that even with perfect retrieval of relevant evidence, LLM performance can decline as context length increases—a finding closely aligned with the robustness trends observed in our study. These results suggest that context length alone, independent of retrieval quality, can hinder reasoning performance. Beyond formal research publications, practitioners and industry observers have emphasized the practical importance and limitations of context length in real-world models. Articles such as those by Moesker 10 and exploratory work on context rotation strategies 15 highlight how context window design and token ordering can influence model behavior, underscoring the need for systematic robustness evaluations. Together, this body of work motivates our focus on quantifying the sensitivity of model accuracy to increasing context lengths across QA datasets and model families. Empirical analyses of long-context limitations. Multiple studies have shown that LLM performance degrades as context length increases, even when the answer-relevant information is present. Previous researches demonstrate that language models often fail to exploit long-range dependencies in practice, despite architectural capacity to process long sequences. Li et al. 9 further observe that models struggle with long in-context learning, exhibiting position-dependent performance degradation commonly referred to as the “lost-in-the-middle” effect. These findings suggest that long context availability does not necessarily translate into robust reasoning over extended inputs. Benchmarks for long-context understanding. Several benchmarks have been proposed to systematically evaluate long-context capabilities. BABILong 8 tests reasoning performance across contexts ranging from thousands to millions of tokens and shows that most models fail beyond relatively modest lengths. XL 2 Bench 11 evaluates extremely long documents across diverse tasks such as memory retrieval, detailed understanding, and open-ended generation, revealing substantial gaps between model performance and human-level comprehension. While these benchmarks provide valuable stress tests, they often combine multiple sources of difficulty, including task complexity, retrieval, and context length, making it challenging to isolate the specific effect of context expansion. Context window extension methods. Another line of work focuses on extending the maximum context window of LLMs. Position interpolation 3 enables context length extrapolation with limited fine-tuning and has been shown to reduce perplexity degradation at longer lengths. Subsequent approaches, including RoPE-based extensions and related interpolation techniques, further improve stability when processing longer sequences. However, these methods primarily address architectural or positional encoding constraints and do not directly evaluate whether models remain robust to irrelevant or distracting context once the window is extended. Prompt compression and retrieval-based approaches. To mitigate the computational and performance challenges of long contexts, several studies propose prompt compression or retrieval-based strategies. LongLLMLingua 7 introduces question-aware prompt compression to reduce irrelevant information and alleviate position bias, achieving significant gains on long-context benchmarks. Similarly, retrieval-augmented generation has been shown to recover performance using shorter effective contexts 16 . However, these approaches alter the input distribution by selectively removing or reordering information, making it difficult to assess intrinsic robustness to long context itself. Robustness and reasoning under interference. Recent work highlights that long-context failures are often tied to reasoning complexity rather than sheer length. Kuratov et al. 8 show that performance drops sharply for tasks requiring multi-hop reasoning, even when single-fact retrieval remains reliable. Related findings indicate that LLMs encode long-range context superficially, with limited sensitivity to semantic structure beyond a few thousand tokens. These observations motivate evaluations that control for evidence availability while varying only the amount of irrelevant context. Positioning of this work. In contrast to prior benchmarks

Read the complete text