Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations

Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth · 2026

Read Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations by Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08064.txt.

Description

Yuxi Xia and five co-authors test how much users should trust language model confidence scores. Their 2026 arXiv paper notes that confidence estimates shape user trust and real choices. Past work judged these methods mostly by calibration and by how well scores rank right over wrong answers. Calibration checks whether stated confidence matches accuracy in practice. The authors argue that this is not enough, and they test confidence methods across changes in language instead. That shift matters when the same question is worded in new ways.

Catalog record

CREATORYuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08064

Browse the full library · About the Tower of Babel