Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report

Jon-Paul Cacioli · 2026

Read Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report by Jon-Paul Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17707.txt.

Introduction

Cacioli applies validity scaling from clinical personality assessment to data probing 20 frontier models across 524 items. LLM evaluation screens nothing before interpreting substantive scales. Six indices are used: L for keeping confidence on errors, K for betting on errors, F for dropping agreed items, and Fp for dropping correct answers. RBS covers flipped watching; TRIN covers fixed answers. A tiered system finds four models construct-invalid and two elevated. Valid-profile models show item-linked confidence with mean r of 0.18, while invalid-profile models do not, at -0.20.

Catalog record

CREATORJon-Paul Cacioli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33955
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17707

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report Jon-Paul Cacioli Abstract Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI ( 10 ; 11 ) and MMPI-3 ( 2 ) to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). These are calibrated against a 20-model derivation sample with family-level comparison groups. A tiered classification system, validated against eight synthetic response policies, identifies four models as construct-level invalid and two as elevated. The indices show strong convergent validity (L–K: r = .82 r=.82 ; F–Fp: r = .99 r=.99 ; dual-probe convergence: r = .88 r=.88 ) and appropriate discriminant structure (L–F: r = − .10 r=-.10 ). Valid-profile models produce item-sensitive confidence, with KEEP decisions tracking correctness (mean r = .18 r=.18 , 14 of 16 significant). Invalid-profile models do not (mean r = − .20 r=-.20 ). The group difference is large ( d = 2.17 d=2.17 , p = .001 p=.001 ) and robust to leave-one-out removal of any single model. Chain-of-thought training produces two opposite response distortions: systematic over-withdrawal (DeepSeek-R1: 37% WITHDRAW+BET contradiction on correct items) and blanket confidence (Qwen Think: L = .974 L=.974 ). Retrospective and prospective probes yield inconsistent validity profiles in 12 of 20 models. Two latent dimensions, corresponding to the under-reporting and over-reporting blocs of clinical assessment, account for 94.6% of index variance. We argue the mapping from clinical validity to LLM evaluation is structural, not anthropomorphic. The methodology screens response patterns for interpretability regardless of what produces them. The operational principle is simple: screen before you interpret. 1 Introduction 1.1 The problem Every LLM metacognition benchmark treats the model’s confidence signal as informative. Downstream systems use these signals for abstention, routing, and human-in-the-loop decisions. Nobody checks whether the signal actually tracks correctness before building on it. Clinical assessment solved this problem decades ago. The PAI ( 10 ; 11 ) and MMPI-3 ( 2 ) include validity scales that detect distorted response patterns before substantive scales are interpreted. In forensic, clinical, and personnel selection contexts, a profile with elevated validity scales is flagged as uninterpretable regardless of what the clinical scales show. This is not optional. It is the first step of any competent interpretation. The three metacognitive profiles identified in 5 across 20 frontier LLMs represent, in clinical terms, three response styles. Two of them produce confidence signals that contain no item-level information. The present paper provides the formal methodology for detecting this. The central question is whether a model’s confidence signal carries item-level information about correctness. Validity indices are the diagnostic machinery for answering that question. This is a methods paper. We introduce a validity-screening framework that can be applied to any metacognitive probe dataset. 1.2 The structural parallel We are not claiming that LLMs have personalities, experience doubt, or engage in impression management. The mapping from clinical validity to LLM evaluation is structural, not psychological. What is shared between clinical self-report and LLM probe data is the data format (binary responses to structured items), the problem (response patterns dominated by a general strategy rather than item-specific information), the consequence (if the pattern is dominated by a response set, the data is uninformative), and the solution (screen for validity before interpreting content). What is not shared is the cause of response distortion. In humans, distortion arises from motivation, psychopathology, or deliberate malingering. In LLMs, it arises from training objectives, RLHF, or architectural constraints. We make no claims about LLM phenomenology. We make no claims about LLM experience of doubt or confidence. The methodology does not require these claims. It requires only that the data be structured as self-report and that the question “does this signal carry information?” be answerable. The validity framework is a statistical methodology for detecting whether response patterns carry item-level information. A random number generator that produced 95% KEEP would be flagged by the same indices, not because it has a personality disorder but because its data is uninformative. The methodology is substrate-independent. This parallels the application of signal detection theory to LLMs ( 4 ) . SDT was developed for human psychophysics, but d^{\prime} measures sensitivity regardless of what the detector is. The methodology transfers because the measurement problem transfers. 1.3 The policy objection A sceptical reader will argue that LLM KEEP/WITHDRAW/BET decisions are policy outputs, not introspective reports. They are trained decision rules. Therefore “response validity” is the wrong construct, and what we observe is simply different policies. We agree that LLMs produce policy outputs. The validity framework does not require introspection. It requires only one thing: that the confidence signal carries item-level information about correctness. A policy that produces KEEP on 99% of items regardless of correctness is a policy whose output is uninformative for any downstream use that depends on confidence discriminating correct from incorrect responses. To be explicit about what this paper does not claim: we do not claim LLMs introspect, we do not claim LLMs have personalities or psychological states, we do not claim clinical constructs transfer psychologically, and we do not claim validity indices measure latent internal states. We claim only that some confidence signals carry item-level information and others do not, and that this distinction can be detected. We retain the term “validity” because the statistical structure of the problem is identical to the structure addressed by clinical validity scales: detecting when response patterns suppress item-level information. 1.4 Prior work Several lines of work address LLM confidence and metacognition. 17 surveyed confidence elicitation methods and found pervasive overconfidence when LLMs verbalise confidence, predominantly in the 80–100% range. 7 showed that LLMs can sometimes discriminate questions they answer correctly from those they do not. 16 reviewed LLM metacognition through AUROC and meta- d^{\prime} . 1 tested metacognitive control using a second-chance paradigm and found limited strategic deployment. 6 demonstrated that verbal confidence scales suffer from severe discretisation. 8 identified a pronounced choice-supportive bias that reinforces initial confidence. 14 argued that LLMs exhibit “metacognitive myopia,” lacking monitoring and control processes that produce systematic biases. 12 introduced a decision-theoretic metric for LLM confidence reliability, showing that even frontier models remain prone to overconfidence on complex tasks. 3 applied Type-2 SDT to decompose metacognitive efficiency from task performance, finding fully inverted rankings. 5 identified three behavioural profiles across 20 frontier LLMs using a cross-domain monitoring battery. An emerging field of “LLM Psychometrics” ( 9 ) applies psychometric instruments and principles to LLM evaluation broadly, including personality, values, and cognitive biases. Our work contributes to this field by importing the specific methodology of clinical validity scaling, which has not previously been operationalised for LLM confidence data. None of the metacognition studies above screens for response validity before analysing the confidence signal. They all assume the data is interpretable. 1.5 The validity scaling framework in clinical assessment Clinical personality assessment uses validity scales organised into three categories. Content non-responsiveness scales detect inconsistent or fixed responding: VRIN and TRIN on the MMPI-3 ( 2 ) , ICN

Read the complete text