The First Drop of Ink : Nonlinear Impact of Misleading Information in Long-Context Reasoning

Muhan Gao, Zih-Ching Chen, Kuan-Hao Huang · 2026

Read The First Drop of Ink : Nonlinear Impact of Misleading Information in Long-Context Reasoning by Muhan Gao, Zih-Ching Chen, Kuan-Hao Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10828.txt.

Introduction

Muhan Gao, Zih-Ching Chen, and Kuan-Hao Huang ask how much misleading material it takes to spoil a language model's long-context reasoning. Retrieval systems and agents pile up long contexts, and earlier work showed that relevant-looking but wrong documents hurt answers, without pinning down the dose curve.

Holding context length fixed while raising the share of hard distractors, the team finds a sharp early plunge followed by a flat tail, an effect they call the first drop of ink. Attention analysis shows a few distractors seize outsized attention, and only cutting them to near zero restores performance. Builders of retrieval pipelines learn that upstream precision matters more than trimming context length.

Catalog record

CREATORMuhan Gao, Zih-Ching Chen, Kuan-Hao Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56546
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10828

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The First Drop of Ink : Nonlinear Impact of Misleading Information in Long-Context Reasoning Muhan Gao, Zih-Ching Chen, Kuan-Hao Huang Abstract As large language models are increasingly deployed in retrieval-augmented generation and agentic systems that accumulate extensive context, understanding how distracting information affects long-context performance becomes critical. Prior work shows that semantically relevant yet misleading documents degrade performance, but the quantitative relationship between the proportion of distractors and performance remains unstudied. In this work, we systematically vary the hard-distractor proportion in fixed-length contexts, revealing a striking nonlinear pattern: as the proportion of hard distractors increases, performance drops sharply within the first small fraction, while the remainder of the range yields only marginal additional decline. We term this “ The First Drop of Ink ” effect, analogous to how a single drop of ink contaminates water. Our theoretical and empirical analyses grounded in attention mechanics show that hard distractors capture disproportionate attention even at small proportions, with diminishing marginal impact as their proportion grows. Controlled experiments further show that filtering gains mainly come from context-length reduction rather than distractor removal; substantial recovery requires reducing the hard-distractor proportion to near zero, highlighting the importance of upstream retrieval precision. 1 Introduction Recent advances in long-context language models ( 1 ) have given rise to applications that aggregate extensive documents into a single context. Deep research pipelines ( 27 ) , for instance, autonomously retrieve and synthesize information from numerous sources, while long-document analysis systems enable users to query entire books or legal documents ( 18 ; 4 ; 10 ; 36 ) . These applications accumulate large volumes of text, often exceeding 100K tokens, before generating a final response. However, as models ingest more documents, they inevitably encounter information that is topically relevant yet ultimately misleading. Figure 1 : The First Drop of Ink effect. Left: Conventional linear assumption (top, red dashed line) versus empirically observed nonlinear degradation (bottom, blue curve): a small fraction of hard distractors is sufficient to severely degrade accuracy. Middle: Hard distractors receive similar attention logits as gold documents ( 8\approx 9\gg 1 ), dominating the softmax competition even at low proportions. Right: With 100 distractor documents, attention on gold drops 76% by adding only 10% hard distractors. This convex relationship explains The First Drop of Ink . Prior work on long-context language models has primarily focused on how the position ( 25 ) and length ( 3 ; 23 ; 22 ) of relevant information affect performance. Less attention has been paid to the surrounding context itself. While research on short-context reasoning tasks ( 34 ; 42 ) and retrieval-augmented generation (RAG) systems ( 21 ; 15 ) demonstrates that distractors can cause non-negligible performance drops, and 11 reveals that this effect amplifies as context length grows, how performance degrades in long contexts as the proportion of misleading documents increases remains unexplored. A natural question arises: how does performance change as the proportion of distractors grows in long-context reasoning? In this work, we systematically vary the proportion of hard distractors within fixed-length contexts and identify The First Drop of Ink effect as in Figure 1 : as hard distractor proportion increases, performance drops sharply within the first small fraction, then plateaus with only marginal further decline. We provide a theoretical analysis grounded in the softmax attention mechanism, showing that attention on the gold document is a convex function of hard distractor proportion, with empirical validation on retrieval heads ( 40 ; 49 ) . This explains the observed nonlinearity: hard distractors dominate the softmax denominator even at small proportions, implying that partially removing them yields negligible recovery and only near-complete removal restores performance. These findings challenge the prevailing assumption in long-context applications that accumulating more documents improves performance. As long as even a small fraction of hard distractors remains in the context, performance is severely degraded; consequently, post-hoc filtering in most cases yields only marginal recovery. This suggests that preventing hard distractors from entering the context in the first place is more critical than filtering them afterward. Contribution. (1) We identify The First Drop of Ink effect across multiple models and datasets: as the proportion of hard distractors increases, performance degrades sharply within the first small fraction, then plateaus. (2) We provide a theoretical explanation showing that attention on the gold document is a strictly convex function of hard distractor proportion, and validate this empirically through attention logit measurements on retrieval heads. (3) We design controlled experiments to disentangle the effects of context length and distractor composition, showing that conventional filtering yields gains primarily from context reduction, and removing hard distractors only provides substantial benefit when their proportion is reduced to near zero. 2 Related Work Long-context understanding and evaluation. The ability to process long context has emerged as a critical capability for large language models (LLMs), with the context window extended from 4K to over 1M tokens ( 1 ; 37 ; 41 ; 6 ; 28 ) . This expansion has motivated efforts to more effectively understand and evaluate long-context ability. Among various evaluation approaches, the “Needle-in-a-Haystack” (NIAH) paradigm ( 17 ) is preferred due to its controllability and ease of construction ( 13 ; 46 ; 2 ; 50 ) . A “needle” (a fact or short passage required to answer a query) is inserted into a “haystack” of unrelated filler text, and the model must locate and use the needle while ignoring the surrounding context. Under this paradigm, 25 identify the ”Lost-in-the-Middle” phenomenon: LLMs prioritize information at the start and end of contexts while neglecting middle portions. This limitation persists across various scenarios ( 20 ; 8 ) , motivating follow-up studies to develop methods for mitigating positional bias ( 14 ; 51 ; 38 ) . Beyond position, needle length also affects retrieval accuracy ( 3 ; 23 ) . Recent work has also examined the haystack itself. In original settings ( 17 ; 13 ) , the haystack consists of irrelevant documents, posing no semantic confusion with the target needle. 43 use synthetically generated biographies to improve coherence between needles and haystack, better approximating realistic retrieval conditions. Further studies introduce semantically related distractors into the haystack and observe non-negligible performance degradation ( 21 ; 11 ) . However, under the long context setting, how performance varies with the proportion of distractors in the haystack remains unexplored. Information aggregation in agentic systems. The rise of agentic AI has fundamentally transformed how LLMs interact with external information. Rather than responding to a single query with a fixed context, modern systems such as deep research pipelines ( 27 ; 48 ) , multi-agent collaboration frameworks ( 39 ; 12 ; 24 ) , and tool-augmented agents ( 33 ; 30 ; 45 ) autonomously gather, aggregate, and synthesize information across multiple retrieval rounds. These systems routinely accumulate contexts exceeding 100K tokens before producing a final response ( 35 ) . Recent work has shown that even context length alone can degrade performance ( 7 ) , further underscoring the challenges of information aggregation at scale. Figure 2 : Accuracy as a function of hard distractor proportion at 128K context length across three models ( Llama-3.1-8B-Instruct , Qwen2.5-7B-Instruct , and Qwen3-Next-80B-Instruct ) on Natural Questions, TriviaQA,

Read the complete text