How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains

Reza Khanmohammadi , Erfan Miahi , Simerjot Kaur, Charese H. Smiley , Ivan Brugere , Kundan Thind , Mohammad M. Ghassemi · 2026

Read How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains by Reza Khanmohammadi , Erfan Miahi , Simerjot Kaur, Charese H. Smiley , Ivan Brugere , Kundan Thind , Mohammad M. Ghassemi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08134.txt.

Description

Reza Khanmohammadi and co-authors benchmark confidence estimators for large reasoning models. Their 2026 arXiv paper notes that miscalibration undermines these models in high-stakes domains. Judging confidence for long, multi-step outputs is harder than for short answers. The team introduces the Reasoning Model Confidence estimation Benchmark, a public resource. It holds 347,496 reasoning traces from six popular reasoning models across distinct families. The benchmark lets researchers compare confidence methods on equal footing before such models are trusted in medicine, law, or finance.

Catalog record

CREATORReza Khanmohammadi , Erfan Miahi , Simerjot Kaur, Charese H. Smiley , Ivan Brugere , Kundan Thind , Mohammad M. Ghassemi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE314 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08134

Browse the full library · About the Tower of Babel