Reasoning on Multiple Needles In A Haystack

Yidong Wang · 2025

Read Reasoning on Multiple Needles In A Haystack by Yidong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04150.txt.

Introduction

Yidong Wang tackles why chat models sag on long prompts that hide a chain of clues in a sea of filler, in a paper from 2025. The stock test hides one fact in long text, which checks find skill alone. The hard form hides a chain of clues that must be found and linked in steps, and scores fall as text grows.

Wang shows the fall comes from thought that shrinks as input grows, not from lost find skill. Split thought into a find stage and a link stage, he urges, and add a mirror step that sends the model back for more rounds. A model trained on that loop holds its score far out in length. The same loop lifts a top model on hard math tests.

Catalog record

CREATORYidong Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04150

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoning on Multiple Needles In A Haystack Yidong Wang Abstract The Needle In A Haystack (NIAH) task has been widely used to evaluate the long-context question-answering capabilities of Large Language Models (LLMs). However, its reliance on simple retrieval limits its effectiveness. To address this limitation, recent studies have introduced the Multiple Needles In A Haystack Reasoning (MNIAH-R) task, which incorporates supporting documents ( Multiple needles ) of multi-hop reasoning tasks into a distracting context ( Haystack ). Despite this advancement, existing approaches still fail to address the issue of models providing direct answers from internal knowledge, and they do not explain or mitigate the decline in accuracy as context length increases. In this paper, we tackle the memory-based answering problem by filtering out direct-answer questions, and we reveal that performance degradation is primarily driven by the reduction in the length of the thinking process as the input length increases. Building on this insight, we decompose the thinking process into retrieval and reasoning stages and introduce a reflection mechanism for multi-round extension. We also train a model using the generated iterative thinking process, which helps mitigate the performance degradation. Furthermore, we demonstrate the application of this retrieval-reflection capability in mathematical reasoning scenarios, improving GPT-4o’s performance on AIME 2024. 1 Introduction With advancements in context window extension technologies ( Dao et al., 2022a ; Chen et al., 2023 ; Xiong et al., 2023 ; Bai et al., 2024 ) , models such as Qwen2.5 ( Qwen-Team, 2025 ) now support context windows of up to 1M tokens. However, the Needle In A Haystack (NIAH) task ( Kamradt, 2023 ) , once a key benchmark for long-context evaluation, has become less effective ( Vodrahalli et al., 2024 ; Hsieh et al., 2024 ) . While new benchmarks ( Zhang et al., 2024b ; Karpinska et al., 2024 ; Bai et al., 2023 ) aim to assess long-context understanding, their scalability issues limit their ability to evaluate models with very large context windows. Recent works ( Hsieh et al., 2024 ; Li et al., 2024 ; Vodrahalli et al., 2024 ) introduced the Multiple Needles In A Haystack Reasoning (MNIAH-R) task as a scalable diagnostic task for reasoning, which incorporates supporting documents of multi-hop reasoning tasks within a pool of distracting information. However, these studies did not provide a comprehensive analysis of the underlying causes of the accuracy decline with increasing context length, nor did they explore potential solutions to address this issue. Additionally, the problem of models relying on internal knowledge for direct answers was not sufficiently addressed. To address the memory-based answering issue, we evaluate models by focusing on questions where models answer correctly based on supporting documents in multi-hop reasoning tasks, but incorrectly when answering directly, ensuring that models do not rely on internal knowledge. The experimental results show that, before filtering, models’ accuracy decreases slightly with fluctuations as the context length increases, with minimal differences in the rate of decline between models. However, after filtering, the accuracy drops significantly as the context length increases, with open-source models experiencing a greater decline than commercial models. We investigate the causes of accuracy decline and find that it is not related to needles placement or the distance between them, but rather to the reduction in thinking process length as input length increases. Based on these observations, we decompose the thinking process into retrieval and reasoning stages and introduce a reflection mechanism for multi-round extension, exploring the Test-Time Scaling Law ( Snell et al., 2024 ) . On this basis, we train a model using the generated iterative thinking process, reducing the accuracy drop from 25.8% to 4.6%. Additionally, we apply the retrieval-reflection capability to a mathematical reasoning context, improving the Pass@1 score of GPT-4o on AIME 2024 from 9.3 to 15.3. Our contributions are as follows: • We demonstrate that memory-based responses significantly impact MNIAH-R performance. After filtering, we assess models’ capabilities, highlighting a performance gap between open-source and commercial models. • We identify that the accuracy drop is related to a shorter thinking process, which can be mitigated by decoupling retrieval from inference and incorporating a reflection mechanism for iterative thinking. • We train a model with retrieval-reflection ability and apply it to mathematical scenarios. 2 MNIAH-R Dataset Setup For multi-hop reasoning tasks, we use the HotpotQA ( Yang et al., 2018 ) and IRE ( Wu et al., 2024 ) datasets. HotpotQA contains 113k wikipedia-based question-answer pairs requiring reasoning over multiple documents, and we focus on its dev_distractor subset, which includes clearly defined standard answers and their corresponding supporting documents. The IRE dataset introduces a stepwise counterfactual benchmark with 782 factual and counterfactual instances to assess multi-step reasoning. Please refer to Appendix B.1 for more details about dataset. Model Evaluation We select six long-context models: GPT-4o ( OpenAI, 2024b ) , GPT-4o-mini ( OpenAI, 2024a ) , Claude-3.5-Sonnet ( Anthropic, 2024 ) , Llama-3-8B-ProLong-64k-Instruct ( Princeton NLP group, 2024 ) , Qwen-2.5-1M ( Qwen-Team, 2025 ) , and GLM-4-9B-Chat-1M ( THUDM, 2024 ) . We use greedy decoding for inference and employ DeepSeek-V3 ( Guo et al., 2025 ) to assess the correctness of model responses. For further details on the models introduction and evaluation prompt, please refer to Appendix B.2 . Dataset Filtering To tackle the issue of memory-based answering, we evaluate models by concentrating on questions where models can answer correctly when supported by the provided documents, but fail to do so when answering directly without relying on external context. By emphasizing these types of questions, we mitigate the tendency of the model to recall answers from previous training data and provide direct responses. For a detailed breakdown of the data statistics before and after filtering, please refer to Appendix B.3 . Results Analysis As shown in Figure 1 and 2 , before filtering, although there is a noticeable difference in accuracy between individual models, the decrease in accuracy with increasing context length is negligible. This does not adequately reflect the impact of context length on the performance of different models. After filtering, we observe a clear decline in accuracy for each model as context length increases. Notably, the open-source model, represented by the dashed line, exhibits a steeper decline compared to the closed-source model, represented by the solid line. Figure 1: Performance on MNIAH-R before filtering . Figure 2: Performance on MNIAH-R after filtering . 3 Explanations for Accuracy Decrease To explain the decrease in accuracy with increasing context length, we further investigate the intersection of the models’ filtered questions. First, we find that the accuracy at the intersection follows a similar decreasing trend with context length; please see the Appendix B.3 for details. Building on this, we explore three potential factors inspired by related works: needles placement ( Liu et al., 2024 ) , distance between Needles ( Ankush Gola, 2024 ) , and thinking process length ( Team et al., 2025 ) . Needles Placement We first examine the impact of supporting documents ( needles ) placement within the context on accuracy decline. Needle position is measured using depth percent , which indicates the relative position of a needle within the context window as a percentage, with 0% representing the start and 100% the end. Each

Read the complete text