Not All Needles Are Found: How Fact Distribution and Don’t Make It Up Prompts Shape Literal Extraction, Logical Inference, and Hallucination Risks in Long-Context LLMs
Amirali Ebrahimzadeh Department of Electrical Engineering & Computer Science University of Michigan, Ann Arbor, MI 48109, USA [email protected] Seyyed M. Salili ∗ Independent Researcher Ann Arbor, MI 48106, USA [email protected] · 2026
Read Not All Needles Are Found: How Fact Distribution and Don’t Make It Up Prompts Shape Literal Extraction, Logical Inference, and Hallucination Risks in Long-Context LLMs by Amirali Ebrahimzadeh Department of Electrical Engineering & Computer Science University of Michigan, Ann Arbor, MI 48109, USA [email protected] Seyyed M. Salili ∗ Independent Researcher Ann Arbor, MI 48106, USA [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-02023.txt.
Description
Long-context language models are being trusted as working memory for autonomous tasks, yet their reliability depends on where the facts sit. Amirali Ebrahimzadeh and Seyyed Salili build an extended needle-in-a-haystack benchmark that separates literal extraction, logical inference, and hallucination, and apply it to four current models.
Two failure modes stand out. Distributional collapse: performance degrades sharply when evidence is scattered across the context instead of concentrated. And a safety tax: prompts telling the model not to make things up cause over-conservative refusals of facts that are actually present, sharply cutting accuracy. The authors blame poor context utilization more than missing knowledge, and call for model-specific robustness work before long-horizon deployment.
Catalog record
| CREATOR | Amirali Ebrahimzadeh Department of Electrical Engineering & Computer Science University of Michigan, Ann Arbor, MI 48109, USA [email protected] Seyyed M. Salili ∗ Independent Researcher Ann Arbor, MI 48106, USA [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-02023