Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals

Jon-Paul Cacioli · 2026

Read Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals by Jon-Paul Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17714.txt.

Introduction

Cacioli turns clinical validity screening into a portable protocol for LLM confidence data. Confidence signals guide abstention, routing, and safety calls, yet no practice checks whether a signal carries item information. The protocol uses indices (L, Fp, RBS), a structural sign (TRIN), and an item stat from a 2x2 table. Its three tiers, Invalid, Indeterminate, and Valid, draw on four schools. On 20 LLMs across 524 items, four models are Invalid and two Indeterminate. Cross-benchmark checks on 18 models with MMLU confirm the screen.

Catalog record

CREATORJon-Paul Cacioli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56633
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17714

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals Jon-Paul Cacioli Abstract LLM confidence signals are used for abstention, routing, and safety-critical decisions. No standard practice exists for checking whether a confidence signal carries item-level information before building on it. Clinical personality assessment solved this problem decades ago: the PAI (Morey, 1991, 2007) and MMPI-3 (Ben-Porath & Tellegen, 2020) mandate validity screening before substantive interpretation. We transfer this principle as a portable protocol for benchmark-based LLM confidence data. The protocol specifies a two-stage architecture. Stage A (validity screening) determines whether the confidence signal is interpretable. Stage B (substantive analysis) proceeds only if Stage A is passed. Stage A requires three core indices computed from a single correctness-by-confidence contingency table, L (confidence on errors), Fp (low confidence on correct items), and RBS (inverted monitoring). A structural indicator (TRIN) and an item-sensitivity statistic (r(confidence, correct)) complete the screen. A three-tier classification system (Invalid, Indeterminate, Valid) draws on the MMPI-3/PAI interpretation sequence, the MCMI adjustment philosophy (Millon, Grossman & Millon, 2015), and the Jacobson-Truax Reliable Change Index framework (Jacobson & Truax, 1991) to handle borderline cases with appropriate uncertainty. We validate the protocol on 20 frontier LLMs across 524 items. Four models are classified as Invalid, two as Indeterminate. Valid-profile models show mean r(confidence, correct) = .18 (15/16 significant). Invalid-profile models show mean r = -.20 (d = 2.48, p = .001). Subsampling analysis shows stable classification at 100-150 items for clear cases, with honest instability for borderline models near threshold values. The failure-mode demonstration shows that unscreened invalid models produce AUROC at chance (0.499-0.509) and zero selective prediction gain, creating false assurance that confidence-based safeguards are functioning. We propose the protocol as a required screening step, analogous to the MMPI-3 validity interpretation sequence. A minimal reporting table (VRS Table) specifies what must accompany any metacognitive, calibration, or selective prediction metric. Two cross-benchmark validations confirm the protocol transfers. On external data from Yang et al. (2024), the screen classified 1 of 11 models as Invalid across 10 benchmarks (rho = .894 between L and AUROC). On 18 battery models screened with MMLU and verbalized confidence (0-100), all battery-Valid models remained Valid, while all three battery-Invalid models shifted to Valid under the continuous probe format, confirming that validity is a property of the model-probe-task interaction. 1 Introduction 1.1 The problem: unscreened confidence Every LLM metacognition benchmark, calibration study, and selective prediction system treats the model’s confidence signal as carrying information. Downstream systems use these signals for abstention (Wen et al., 2025), routing, human-in-the-loop escalation, and safety-critical decisions. The entire selective prediction literature assumes that confidence discriminates correct from incorrect responses. Nobody checks this assumption before building on it. Clinical personality assessment solved this structural problem decades ago. The MMPI-3 (Ben-Porath & Tellegen, 2020) mandates a fixed interpretation sequence. Check content non-responsiveness, check over-reporting, check under-reporting, and only then interpret substantive scales. The PAI (Morey, 1991, 2007) mandates an equivalent sequence. ICN and INF first, then NIM and MAL, then PIM and DEF. If validity is compromised, the profile is flagged as uninterpretable regardless of what the clinical scales show. This is not optional. It is the first step of competent interpretation (van Scoyoc, 2017; Burchett & Bagby, 2022). We propose that any study reporting confidence-based metrics for LLMs without validity screening is methodologically incomplete, analogous to interpreting PAI clinical scales without checking PIM and NIM. The four principles of classical psychometrics are reliability, validity, standardisation, and equivalence (Rust, Kosinski & Stillwell, 2021). Equivalence demands that a test be free from bias. A confidence signal dominated by a response set (blanket endorsement, blanket withdrawal, or fixed responding) is biased in the classical psychometric sense. It systematically misrepresents the relationship between confidence and correctness. The present protocol addresses this fourth principle. Cacioli (2026d) introduced six validity indices for LLM confidence data, mapped to PAI and MMPI-3 scales, and demonstrated their psychometric properties on 20 frontier models. The present paper extracts from that derivation study a portable, benchmark-agnostic screening protocol that any researcher or deployment team can apply to their own confidence data. 1.2 The construct being protected A confidence signal is interpretable for downstream use only if it shows non-trivial item-level dependence on correctness and is not dominated by response-style artefacts such as blanket endorsement, blanket withdrawal, inversion, or patterned responding. This is the construct the protocol screens. The scope is deliberately narrow. The protocol does not screen for good calibration, high metacognitive sensitivity, or optimal abstention policy. It screens only for the minimum condition that makes those downstream analyses meaningful, that the confidence signal carries item-level information about correctness. Failing the screen does not prove the model lacks metacognitive information. It means the elicited confidence signal is not sufficiently interpretable for standard analysis under this protocol. A different elicitation method, probe format, or task domain may yield a different result. Validity is a property of the interaction between model, probe, and task, not an intrinsic property of the model alone (Cacioli, 2026d; Rust et al., 2021). The protocol screens response validity, not construct validity. Passing does not imply that the benchmark measures what it claims to measure, that the model possesses metacognition, or that the confidence signal reflects an internal state. It implies only that the confidence output carries item-level information about correctness under the tested conditions. 1.3 What this paper is not This paper does not introduce new validity indices. Those are derived and validated in the companion paper (Cacioli, 2026d). This paper specifies how to use them as a portable protocol. What to compute, in what order, with what thresholds, and with what reporting requirements. The relationship between the two papers is analogous to the relationship between the MMPI-3 Technical Manual (derivation and validation) and the MMPI-3 Administration Manual (how to use the instrument). This paper does not claim that the protocol is sufficient for all confidence evaluation needs. It claims only that it is necessary as a first step. 1.4 Scope The protocol is designed for settings with item-level ground truth (correctness labels for each item), one confidence output per item (binary or binarisable to high/low), repeated benchmark-style evaluation (multiple items, same model), and black-box or grey-box access (no requirement for logits or internal states). The protocol is not designed for open-ended generation without stable correctness labels, multi-step reasoning traces where correctness is graded, multi-label tasks where partial credit applies, interactive agents with changing context windows, or confidence expressed as free text without a canonical parse. These are scope limitations, not claims of irrelevance. Extensions to these settings are possible but require additional validation. 1.5 Prior work Three research communities address the question of whether LLM confidence signals are meaningful, each from a different angle. The confidence calibration literature asks whether stated probabilities match empirical frequencies. Xiong et al. (2024) surveyed confidence elicitation methods and documented pervasive overconfidence when LLMs verbalise confidence, predominantly in the 80-100% range regardless of actual accuracy. Geng et al. (2024) surveyed calibration methods including temperature scaling, verbalized uncertainty, and ensemble-based approaches. Dai (2026) demonstrated that verbal confidence scales suffer severe discretisation artefacts. These studies all assume that the confidence signal contains item-level information and ask how well-calibrated that information is. None checks the

Read the complete text