LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations

William Lugoloobi, Thomas Foster, William Bankes, Chris Russell · 2026

Read LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations by William Lugoloobi, Thomas Foster, William Bankes, Chris Russell (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09924.txt.

Description

Running a reasoning model at full stretch on every question wastes compute, yet guessing which questions need the extra effort is hard. A 2026 paper led by William Lugoloobi, with three co-authors, asks whether a model already knows, before it writes, how likely it is to succeed. The team trains simple linear probes on the model's internal activations from before generation.

Those probes beat surface clues like question length on mathematics and coding tasks, and they reveal a model-specific sense of difficulty distinct from human judgment. Routing questions across a pool of models with these probes beat the best single model while cutting inference cost by as much as seventy percent. Internal signals, the paper suggests, can make inference both stronger and cheaper.

Catalog record

CREATORWilliam Lugoloobi, Thomas Foster, William Bankes, Chris Russell
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09924

Browse the full library · About the Tower of Babel