Filtered Reasoning Score: Evaluating Reasoning Quality on a Model’s Most Confident Traces
Manas Pathak, Xingyao Chen, Shuozhe Li, Amy Zhang, Leqi Liu · 2026
Read Filtered Reasoning Score: Evaluating Reasoning Quality on a Model’s Most Confident Traces by Manas Pathak, Xingyao Chen, Shuozhe Li, Amy Zhang, Leqi Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11996.txt.
Description
Pathak and four colleagues argue that benchmark accuracy alone cannot reveal thinking quality, since models can reach right answers through flawed thinking or rote memory. They propose a score rating traces on faithfulness, coherence, utility, and factuality. Lumping all traces misleads, above all over long runs where shaky correct traces may be luck. Their Filtered Reasoning Score instead uses only the top-K percent surest traces. Models tied on accuracy then differ sharply on thinking quality, and higher scores predict better results on other benchmarks.
Catalog record
| CREATOR | Manas Pathak, Xingyao Chen, Shuozhe Li, Amy Zhang, Leqi Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71928 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11996