Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Heads

Artem Vazhentsev, Lyudmila Rvanova, Gleb Kuzmin, Ekaterina Fadeeva, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin, Artem Shelmanov · 2025

Read Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Heads by Artem Vazhentsev, Lyudmila Rvanova, Gleb Kuzmin, Ekaterina Fadeeva, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin, Artem Shelmanov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20045.txt.

Description

Chat models state falsehoods with full confidence, and catching those slips usually takes heavy extra computation or labeled training data. Artem Vazhentsev and co-authors find a cheaper signal in a 2025 paper: inside the model, certain attention heads pull back from earlier words when the text goes wrong. Their method hunts those heads down on its own.

The signal from those heads is blended with the model's own word-by-word confidence in a rolling estimate of doubt. One pass through the text is all it takes, adding almost no compute. Across a dozen test sets and nine models, the method beat the best prior doubt scores. Since it needs no labels or tuning, it can ride along inside other tools as a light safety check.

Catalog record

CREATORArtem Vazhentsev, Lyudmila Rvanova, Gleb Kuzmin, Ekaterina Fadeeva, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin, Artem Shelmanov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20045

Browse the full library · About the Tower of Babel