Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
Owen Bianchi, Mathew J. Koretsky, Tanay Nayak, Chelsea X. Alvarado, Maya Willey, Adi Asija, Nicole Kuznetsov, Mike A. Nalls, Faraz Faghri, Daniel Khashabi · 2025
Read Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find by Owen Bianchi, Mathew J. Koretsky, Tanay Nayak, Chelsea X. Alvarado, Maya Willey, Adi Asija, Nicole Kuznetsov, Mike A. Nalls, Faraz Faghri, Daniel Khashabi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18148.txt.
Introduction
Owen Bianchi and co-authors show that language models struggle more to find answers hidden in short documents than in long ones, in a paper from 2025. Needle-in-a-haystack tests bury one relevant passage in a sea of filler. Past work blamed position and distractor count; this study isolates the size of the passage that holds the answer.
Across three benchmarks, eleven top models, and over 150,000 controlled runs, accuracy fell sharply as the answer-bearing passage shrank. Small passages also made models more sensitive to where the answer sat. The effect survived checks for position, repeated answer tokens, and distractor volume. The authors warn that agent systems, which stitch together many small scraps of information, face exactly this weakness.
Catalog record
| CREATOR | Owen Bianchi, Mathew J. Koretsky, Tanay Nayak, Chelsea X. Alvarado, Maya Willey, Adi Asija, Nicole Kuznetsov, Mike A. Nalls, Faraz Faghri, Daniel Khashabi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18148
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find Owen Bianchi, Mathew J. Koretsky, Tanay Nayak, Chelsea X. Alvarado, Maya Willey, Adi Asija, Nicole Kuznetsov, Mike A. Nalls, Faraz Faghri, Daniel Khashabi Abstract Large language models (LLMs) face significant challenges with needle-in-a-haystack tasks, where relevant information (“the needle“) must be drawn from a large pool of irrelevant context (“the haystack“). Previous studies have highlighted positional bias and distractor quantity as critical factors affecting model performance, yet the influence of gold context size , the length of the answer-containing document, has received little attention. We present the first systematic study of gold context size in long-context question answering, spanning three diverse benchmarks (general knowledge, biomedical reasoning, and mathematical reasoning), eleven state-of-the-art LLMs (including recent reasoning models), and more than 150K controlled runs. Our experiments reveal that LLM performance drops sharply when the gold context is shorter, i.e., smaller gold contexts consistently degrade model performance and amplify positional sensitivity , posing a major challenge for agentic systems that must integrate scattered, fine-grained information of varying lengths . This effect persists under rigorous confounder analysis: even after controlling for gold context position, answer token repetition, gold-to-distractor ratio, distractor volume, and domain specificity, gold context size remains a decisive, independent predictor of success. Our work provides clear insights to guide the design of robust, context-aware LLM-driven systems. \DeclareCaptionType prompt[Prompt][List of Prompts] 1 Introduction Large language models (LLMs) increasingly power applications that require reasoning over vast amounts of information, such as synthesizing evidence across scientific literature ( Sprueill et al., 2024 ; Bazgir et al., 2025 ; Wang et al., 2025 ; Gao et al., 2025 ) or navigating complex codebases ( Liu et al., 2023 ; Zhang et al., 2023 ; Bogomolov et al., 2024 ) . A critical stage in such systems is aggregation , the synthesis of retrieved evidence into an accurate, actionable response. This stage determines what content to include, cite, or ignore, and has direct implications for safety, reliability, and factual correctness. One specific, common variant of aggregation is needle-in-a-haystack (NIAH) scenarios, where relevant evidence (the ‘ gold context ’) is embedded within a large volume of topically related or superficially plausible but ultimately irrelevant or misleading, ‘ distractor context ’ ( Tay et al., 2021 ; Shaham et al., ) . Successful aggregation requires precise identification of essential evidence among a large number of distractor documents. In NIAH scenarios, prior work has explored phenomena such as positional bias ( Wang et al., 2023 ; Liu et al., ) , showing that models disproportionately attended to early content and that distractors degrade performance. Yet one key dimension remains underexplored: how the size of the gold context influences model performance. This question is critical for the design of agentic systems, where autonomous agents must perform NIAH-style aggregation over heterogeneous information streams produced by specialized components, whose evidence can vary widely in length . In this study, we present a systematic analysis of gold context size as an independent variable in LLM performance in long-context, single-needle NIAH settings. Our main finding (§ 3 ) is that smaller gold contexts consistently (1) degrade performance and (2) heighten sensitivity to positional bias , as illustrated in Fig. 1 . This exposes a form of brittleness not captured in prior work. Figure 1: LLM performance on a needle-in-a-haystack task (CARDBiomedBench). The y -axis is a measure of accuracy (higher is better). Varying both the size and position of the gold context within fixed distractor content, we find that LLMs perform worse and exhibit greater positional sensitivity when given short gold contexts (dashed line) compared to long ones (solid line). To support our analysis, we adapt three diverse datasets spanning mathematics, biomedical reasoning, general knowledge, systematically varying both the size and position of the gold context while keeping the distractor content fixed (§ 2 ). Our study comprises over 150K controlled runs across eleven state-of-the-art LLMs, both general-purpose and reasoning-focused, ensuring that the observed effects hold across architectures and tasks. We conduct extensive confounder analyses to ensure that our findings are not driven by spurious factors (§ 4 ). Even after controlling for gold position, answer-token repetition, gold-to-distractor ratio, distractor volume, and domain specificity, gold context size remains a strong, independent predictor of performance. The overall pattern is consistent—larger gold contexts outperform smaller ones—though the magnitude of the effect varies across conditions. For instance, smaller gold contexts are particularly vulnerable to primacy bias, whereas larger gold ones exhibit greater robustness to both positional variation and distractor interference. These findings have practical implications. In real-world deployments, factors such as context size, position, distractor length, and noise are typically uncontrolled. Our results show that heterogeneity in document length can induce aggregation failures, particularly when small but critical evidence is overshadowed by much longer passages. This situation naturally arises in retrieval-augmented and agentic systems, where an aggregator must isolate a single relevant piece of evidence from a mixture of heterogeneous sources. Practitioners can mitigate this fragility by monitoring length disparities among retrieved documents and by avoiding pipelines that combine extremely short and very long contexts, thereby reducing size-induced attention asymmetries. In summary, our contributions are as follows: • Novel determinant of long-context performance: To the best of our knowledge, we are the first to demonstrate that the size of gold contexts functions as a hidden variable in LLM performance on long-context NIAH. Smaller golds degrade accuracy and amplify positional bias, underscoring a potential fragility in real-world applications. • Robust to confounders: We identify and analyze five potential confounding variables, (1) gold context position, (2) answer token repetition, (3) gold-to-distractor ratio, (4) distractor volume, and (5) domain specificity, demonstrating that gold context size remains a decisive predictor of success despite these factors. • Large-scale experimentation: We repeat our experiments and aggregate findings across eleven state-of-the-art LLMs, three diverse benchmarks, three sizes of gold, and six positions in the context window totaling over 150k controlled runs. 2 Experimental Setup This section outlines our design objectives, benchmark adaptations, baseline validations. 2.1 Desiderata We begin by outlining the core desiderata guiding our experimental design. Realism. In real-world agentic systems, aggregation involves synthesizing outputs from multiple specialized agents, each retrieving information from their domain of expertise. Usually, one agent returns the document containing the correct answer (the “gold context"), while others provide distractors, topically relevant but ultimately uninformative. We simulate this by inserting a gold context of varying size at different positions within a fixed-length sequence of distractors. Document order is randomized to reflect natural uncertainty in agent contributions and retrieval quality. Gold Size Variability. We constructed three nested gold variants for each benchmark: • Small Gold : Minimal span sufficient to answer the question. • Medium Gold : Additional explanatory or supporting content. • Large Gold : Complete reasoning process and/or extended relevant context. These were wrapped in pseudo-documents (titles, questions). Variants are hierarchically structured ( small \subset medium \subset large ) and validated for sufficiency. See Figure 9 for examples. Performance is high and uniform when observing only the gold of any size (Appendix ). Distractors. To simulate realistic scenarios, we curate distractors topically relevant and lexically similar to the question but lacking the answer. The total distractor