Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction

Jon-Paul Cacioli · 2026

Read Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction by Jon-Paul Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17716.txt.

Description

Cacioli tests whether a validity screen for LLM confidence signals predicts selective prediction. The screen classifies confidence signals as Valid, Indeterminate, or Invalid. Twenty frontier LLMs from seven families were tested on 524 items across six tracks. Valid models show a mean Type 2 AUROC of 0.624, invalid models 0.357, Indeterminate 0.554, Cohen's d 2.81, rising from Invalid to Valid. Split-half validation gives median d of 1.77 across 1,000 splits, and the three tiers account for 47 percent of variance. DeepSeek-R1 drops from 85.3 percent accuracy at full coverage to 11.3 percent at 10 percent.

Catalog record

CREATORJon-Paul Cacioli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22444
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17716

Browse the full library · About the Tower of Babel