Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations
Ahmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon · 2026
Read Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations by Ahmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19239.txt.
Description
Large language models can carry out long procedures yet stumble on a trivial last step: reporting a value they computed moments earlier. The paper calls this procedural hallucination, a failure to carry out a clear, checkable instruction even when the right answer sits in the model's own context. Ahmed Karim and five colleagues study it with long-context tasks where the correct answer is a single known token.
They find that many mistakes happen at readout, when the model must fetch and state its result. Some failures are gating errors, where the model never switches into answer mode at all. Others are binding errors, where it enters answer mode but picks the wrong candidate. Splitting the failure this way matters, because each kind points to a different fix for making models follow procedures they already understand.
Catalog record
| CREATOR | Ahmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-19239