How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains
Reza Khanmohammadi , Erfan Miahi , Simerjot Kaur, Charese H. Smiley , Ivan Brugere , Kundan Thind , Mohammad M. Ghassemi · 2026
Read How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains by Reza Khanmohammadi , Erfan Miahi , Simerjot Kaur, Charese H. Smiley , Ivan Brugere , Kundan Thind , Mohammad M. Ghassemi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08134.txt.
Description
Reza Khanmohammadi and co-authors benchmark confidence estimators for large reasoning models. Their 2026 arXiv paper notes that miscalibration undermines these models in high-stakes domains. Judging confidence for long, multi-step outputs is harder than for short answers. The team introduces the Reasoning Model Confidence estimation Benchmark, a public resource. It holds 347,496 reasoning traces from six popular reasoning models across distinct families. The benchmark lets researchers compare confidence methods on equal footing before such models are trusted in medicine, law, or finance.
Catalog record
| CREATOR | Reza Khanmohammadi , Erfan Miahi , Simerjot Kaur, Charese H. Smiley , Ivan Brugere , Kundan Thind , Mohammad M. Ghassemi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 314 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08134