Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations

Ahmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon · 2026

Read Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations by Ahmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19239.txt.

Description

Large language models can carry out long procedures yet stumble on a trivial last step: reporting a value they computed moments earlier. The paper calls this procedural hallucination, a failure to carry out a clear, checkable instruction even when the right answer sits in the model's own context. Ahmed Karim and five colleagues study it with long-context tasks where the correct answer is a single known token.

They find that many mistakes happen at readout, when the model must fetch and state its result. Some failures are gating errors, where the model never switches into answer mode at all. Others are binding errors, where it enters answer mode but picks the wrong candidate. Splitting the failure this way matters, because each kind points to a different fix for making models follow procedures they already understand.

Catalog record

CREATORAhmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-19239

Browse the full library · About the Tower of Babel