Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth · 2026
Read Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations by Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08064.txt.
Description
Yuxi Xia and five co-authors test how much users should trust language model confidence scores. Their 2026 arXiv paper notes that confidence estimates shape user trust and real choices. Past work judged these methods mostly by calibration and by how well scores rank right over wrong answers. Calibration checks whether stated confidence matches accuracy in practice. The authors argue that this is not enough, and they test confidence methods across changes in language instead. That shift matters when the same question is worded in new ways.
Catalog record
| CREATOR | Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08064