LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations
William Lugoloobi, Thomas Foster, William Bankes, Chris Russell · 2026
Read LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations by William Lugoloobi, Thomas Foster, William Bankes, Chris Russell (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09924.txt.
Description
Running a reasoning model at full stretch on every question wastes compute, yet guessing which questions need the extra effort is hard. A 2026 paper led by William Lugoloobi, with three co-authors, asks whether a model already knows, before it writes, how likely it is to succeed. The team trains simple linear probes on the model's internal activations from before generation.
Those probes beat surface clues like question length on mathematics and coding tasks, and they reveal a model-specific sense of difficulty distinct from human judgment. Routing questions across a pool of models with these probes beat the best single model while cutting inference cost by as much as seventy percent. Internal signals, the paper suggests, can make inference both stronger and cheaper.
Catalog record
| CREATOR | William Lugoloobi, Thomas Foster, William Bankes, Chris Russell |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09924