Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks
Chuyifei Zhang, Hongyu Cui, Xiaowen Huang, Jitao Sang · 2026
Read Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks by Chuyifei Zhang, Hongyu Cui, Xiaowen Huang, Jitao Sang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23170.txt.
Introduction
Zhang and three coauthors audit how task position in long contexts affects reasoning scores. Reasoning benchmarks do not control task position, filler, and context length together, unlike retrieval tests. Their Context Rot Evaluation varies all three and tests nine models on math and challenge questions. Scores can fall sharply when a target task moves from the end to the middle, growing with context length. Newer releases show smaller falls, and a probe copy at the end restores middle scores, pointing to position as the cause.
Catalog record
| CREATOR | Chuyifei Zhang, Hongyu Cui, Xiaowen Huang, Jitao Sang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62562 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23170
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks Chuyifei Zhang, Hongyu Cui, Xiaowen Huang, Jitao Sang Abstract Position-controlled evaluation is widely adopted for retrieval tasks such as Needle-in-a-Haystack and RULER. Mainstream reasoning benchmarks, by contrast, fail to control the positional placement of target tasks within long contexts. We audit 11 popular long-context benchmarks and find that none jointly controls task position, filler content, and context length for reasoning. An audit of four recent flagship long-context releases finds no main result-table entry for NIAH, RULER, or LongBench-family benchmarks, while agentic and coding benchmarks appear in main result-tables across all four. These audited vendor result tables therefore provide little direct visibility into position-controlled reasoning behavior. In response to these gaps in benchmark design and vendor evaluation practice, we propose Context Rot Evaluation (CRE), a controlled framework that systematically varies all three factors. We evaluate nine LLMs on GSM8K and ARC-Challenge across two rounds: an initial five-model set and four newer vendor releases. Models can drop sharply when the target task moves from the end of the context to the middle, and the drop grows worse with context length for vulnerable models. MiMo-v2-Flash, for example, drops 88pp at 64K under with_solutions filler, with middle accuracy falling to 8%. The newer releases show smaller middle-position drops under the same filler. At 64K, three of the four stay within ± \pm 6pp of end-position accuracy. MiMo-V2.5-Pro narrows the drop from MiMo-v2-Flash’s 88pp to 32pp. Under questions_only_v2 filler, middle-position drops persist across all four (range − 16 -16 pp to − 56 -56 pp across 8K, 32K, and 64K). At 8K, a diagnostic probe that adds a target-task copy at the end brings middle-position accuracy within ± \pm 4pp of end-position baseline across all nine models. This pattern is consistent with a positional explanation. In the initial five-model set, 76% of middle-position errors match surrounding filler text versus 22% at the end position, consistent with filler-answer interference as a dominant error mode. These results expose a structural evaluation gap in current reasoning benchmark design and vendor evaluation practice: when task position is not controlled, positional vulnerabilities that grow with context length cannot be measured as a separate effect. 1 Introduction Modern long-context LLMs claim effective context windows of 128K tokens or more. Whether these claims hold up in practice depends on how we evaluate them. For retrieval tasks, position control is already standard: Needle-in-a-Haystack ( Kamradt 2023 ) sweeps answer position across context depth, and RULER ( Hsieh et al. 2024 ) adds configurable task complexity. By contrast, mainstream long-context reasoning benchmarks follow a different practice: task position is rarely controlled or explicitly reported. Representative suites including SCROLLS ( Shaham et al. 2022 ) , LongBench ( Bai et al. 2024 ) , L-Eval ( An et al. 2023 ) , and LooGLE ( Li et al. 2023 ) evaluate models on natural documents whose task-relevant information sits at positions inherited from the source rather than chosen experimentally. This asymmetry between retrieval and reasoning benchmarks introduces an evaluation gap. Natural-document reasoning benchmarks can only report aggregated model performance on long-context tasks, but they cannot disentangle success or failure from task position. If a model fails at some positions but not others, those failures stay uncharacterized when benchmarks do not control task position. None of the surveyed benchmarks jointly controls task position, filler content, and context length for reasoning tasks. Whether this gap matters empirically remains an open question. We then audit how four recent flagship releases (DeepSeek-V4-Pro, MiMo-V2.5-Pro, Kimi-K2.6, and GLM-5.1) report their own evaluation results, and the same gap appears in vendors’ own tables. Across the four releases, the seven long-context benchmark families tracked in this audit (NIAH, RULER, LongBench, HELMET, \infty Bench, BABILong, LOFT) are absent from all twenty-eight main result-table audit cells. By contrast, each vendor’s headline comparison includes SWE-Bench and browser- or terminal-oriented agentic/coding benchmarks, such as BrowseComp, Terminal-Bench, ClawEval, or GDPVal. DeepSeek-V4-Pro, for example, reports SWE-Bench Verified 80.6, BrowseComp 83.4, and Terminal-Bench 67.9 in its main Instruct table. NIAH and RULER are absent, and LongBench-V2 (51.5 EM) appears only in the Base eval table. GLM-5.1 reports NIAH and RULER inside a 9B-proxy architecture ablation for sparse attention, not as primary positional fidelity claims for the deployed product. The full vendor disclosure matrix is in Appendix G . These four vendors lead their main result-tables with agent and coding task scores rather than with positional reasoning scores. Strong scores on agent and coding benchmarks would not, on their own, characterize position-controlled reasoning behavior. Position-controlled benchmarks isolate the variable that these vendor tables leave uncharacterized: how reasoning performance changes with task position. This measurement gap motivates CRE. Our work bridges these gaps. We first audit 11 long-context benchmarks ( Section 2 ) and document a structural asymmetry: position-controlled evaluation is well established for retrieval but largely absent from mainstream reasoning benchmark design. Building on this audit, we propose Context Rot Evaluation (CRE), a controlled framework that systematically varies three factors—task position, filler content, and context length—for reasoning tasks. CRE evaluates nine LLMs across two rounds: an initial set (Qwen 2.5-7B-Instruct, MiMo-v2-Flash, GLM-4.7-FlashX, DeepSeek-V3.2 in reasoning mode, and Kimi k2.5) plus four newer vendor releases (DeepSeek-V4-Pro, MiMo-V2.5-Pro, Kimi-K2.6, and GLM-5.1), tested across three context tiers (8K, 32K, 64K) and three filler types (worked solutions, unanswered questions, neutral Wikipedia/news prose). The reasoning tasks are GSM8K ( Cobbe et al. 2021 ) (math word problems) and ARC-Challenge ( Clark et al. 2018 ) (science multiple choice). With this setup, we characterize a model-dependent vulnerability spectrum and analyze diagnostic probe behavior across filler types and context lengths. End-to-middle drops reach 94pp on GSM8K under structured-filler conditions; 76% of middle-position errors match filler content versus 22% at the end, consistent with filler-answer interference as a dominant error mode. Our contributions are: • A systematic position audit of 11 long-context benchmarks: position-aware evaluation is comparatively mature for retrieval but largely unaudited for mainstream reasoning; synthetic exceptions like BABILong and LongReason do not close this gap ( Sections 2 and F ). • Evidence that task position, filler content, and context length jointly shape reasoning failures. End-to-middle drops reach 94pp on GSM8K, with directional cross-domain support on ARC-Challenge (range -6 pp to -40 pp); benchmarks that do not control task position miss these effects ( Section 4 ). • A model-dependent vulnerability spectrum: some models drop sharply at the middle position, others stay nearly immune. Diagnostic probe behavior is consistent with a positional explanation of these failures ( Section 4.2 ). 2 Related Work Position effects in long-context LLMs have been documented since Liu et al. 2023 . They showed that multi-document QA models attend less to information in the middle of the context than to information at the beginning or end. This result motivated a line of synthetic retrieval-style benchmarks that manipulate position directly. Needle-in-a-Haystack ( Kamradt 2023 ) established the basic protocol of sweeping answer position across context depth. RULER ( Hsieh et al. 2024 ) extended this idea into a configurable synthetic suite covering retrieval, variable tracking, and aggregation tasks. Later benchmarks such as LongPiBench ( Tian et al. 2025 ) , NoLiMa ( Modarressi et al. 2025 ) , and Counting-Stars ( Song