Perplexity Cannot Always Tell Right from Wrong
Petar Veličković, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, Razvan Pascanu · 2026
Read Perplexity Cannot Always Tell Right from Wrong by Petar Veličković, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, Razvan Pascanu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22950.txt.
Description
Petar Veličković and four co-authors question a favorite yardstick for language models: perplexity. Perplexity measures how surprised a model is by a text, and it serves both as a training loss and a quick quality score. Earlier work noted flaws in it, mostly from experiments.
The authors give a sharper account of when perplexity can, and cannot, tell right answers from wrong ones. A metric that looks simple can quietly mislead model choices. Knowing its limits helps teams judge models by measures tied to real correctness. The authors move past scattered empirical complaints, aiming for a principled account of what perplexity can and cannot certify.
Catalog record
| CREATOR | Petar Veličković, Federico Barbero, Christos Perivolaropoulos, Simon Osindero, Razvan Pascanu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22950