A Confidence–Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks

Zhilong Zhao, Yindi Liu · 2025

Read A Confidence–Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks by Zhilong Zhao, Yindi Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02029.txt.

Introduction

Zhilong Zhao and Yindi Liu wrote this paper, published in two thousand twenty-five, on checking the work of language models that code text into categories. Such coding is now done at scale by AI, but human experts often disagree, so there is no clean answer key.

The pair studied more than five thousand coding calls made by eight top models across ten categories. A model's own stated confidence tracked how much the models agreed with each other. Adding a diversity measure, based on how spread out the answers were, made the pair of signals explain agreement almost fully. From this they build a three-tier flow: easy cases pass on their own, which cuts hand work by about two thirds. Tests in other fields showed the method travels well. The authors frame it as a base method, with harder tasks still to come.

Catalog record

CREATORZhilong Zhao, Yindi Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Confidence–Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks Zhilong Zhao, Yindi Liu Abstract Abstract LLMs enable qualitative coding at large scale, but assessing reliability remains challenging where human experts seldom agree. We investigate confidence-diversity calibration as a quality assessment framework for accessible coding tasks where LLMs already demonstrate strong performance but exhibit overconfidence. Analysing 5 680 coding decisions from eight state-of-the-art LLMs across ten categories, we find that mean self-confidence tracks inter-model agreement closely (Pearson r=0.82 ). Adding model diversity —quantified as normalised Shannon entropy—produces a dual signal explaining agreement almost completely ( R^{2}=0.979 ), though this high predictive power likely reflects task simplicity for current LLMs. The framework enables a three-tier workflow auto-accepting 35 % of segments with <5 % error, cutting manual effort by 65 %. Cross-domain validation confirms transferability ( \kappa improvements of 0.20–0.78). While establishing a methodological foundation for AI judgement calibration, the true potential likely lies in more challenging scenarios where LLMs may demonstrate comparative advantages over human cognitive limitations. Introduction Qualitative research has long faced a structural trade–off between scale and reliability. Traditional workflows typically depend on two–coder adjudication, yet resulting Cohen’s \kappa often remains only “moderate” while labour costs grow super–linearly with corpus size. 1 Yet in domains lacking a single ground truth, researchers still need principled criteria for deciding when an AI label can be trusted. We investigate this challenge in accessible coding tasks where LLMs already demonstrate strong performance but exhibit overconfidence. Drawing on calibration research in metacognitive monitoring , we treat self–confidence as a first–order cue and complement it with panel diversity , an error signal that reveals collective blind spots. 2 Our study centres on how analysts allocate attention within human–AI collaboration, translating cognitive signals into workflow decisions that optimise expert effort. Computational text-analysis methods—ranging from early topic models to state-of-the-art large language models (LLMs)—are rapidly entering the methodological toolkit of social scientists and health researchers, promising to automate labour-intensive qualitative tasks such as open coding and theme extraction 3 , 4 , 5 , 6 , 7 , 8 . By partitioning the confidence–diversity space into actionable zones, we created a tiered review system that efficiently allocates human attention. Indeed, empirical analyses and methodological reviews consistently show that inter-coder reliability in qualitative studies often falls only in the “moderate” range even among trained researchers 1 , 9 . Disagreement reflects interpretive uncertainty, disciplinary lenses and contextual nuances that resist reduction to binary truth values 10 , 11 . This makes it difficult to judge when an AI-generated label should be trusted, because standard accuracy metrics presuppose an authority against which the model can be scored. Most recent work therefore turns to metacognitive signals. Self-reported confidence has emerged as a convenient proxy: if a model “knows that it knows”, high-confidence predictions might be prioritised while low-confidence ones are routed for review. However, modern neural networks are notorious for poorly calibrated probabilities 12 , 13 , and confidence alone cannot expose blind spots shared by an otherwise unanimous ensemble of models. Here we propose to complement confidence with model diversity —the dispersion of predictions across differently trained or prompted LLMs. Diversity has long served as an error bound in ensemble learning 14 , yet remains under-explored in qualitative research. We analyse eight state-of-the-art LLM variants —four single-response models and their chain-of-thought counterparts—that each coded 71 interview segments against a ten-category framework, yielding 5 680 decisions. A ninth model is introduced only in cross-domain validation (see Methods). Our approach focuses on coding tasks that, while presenting moderate challenges for human coders ( \kappa=0.67 ), are relatively straightforward for LLMs ( \kappa=0.89 ). This complexity differential provides an ideal testbed for developing calibration methods. We show that mean self-confidence explains much of the observed inter-model agreement (Pearson r=0.82 ), and that adding model diversity produces a dual indicator that predicts agreement almost perfectly ( R^{2}=0.979 )—though this high predictive power likely reflects task accessibility for current LLMs. Finally, we show that this confidence–diversity principle transfers across six public datasets spanning finance, medicine and law within similar complexity ranges, and that its predictive accuracy remains robust to the exact weighting of the two signals: although an in-sample grid search favours w^{\ast}=0 , the intuitive 0.6/0.4 split performs within 0.03 MAE across domains. However, these validation tasks share the characteristic of being relatively accessible to current LLMs. Related Work At the intersection of qualitative content analysis, model–calibration theory and ensemble–uncertainty estimation, three strands of scholarship inform our study. Reliability in qualitative coding. Classic content–analysis texts demonstrate that interpretive categories must be both conceptually coherent and empirically reliable. Formal checks usually rely on chance–corrected agreement metrics such as Cohen’s \kappa or Krippendorff’s \alpha ; empirical studies and methodological guidelines consistently classify values above 0.80 as “almost perfect” agreement—and report that such cases are uncommon in practice. 15 , 16 , 11 , 17 , 18 Recent work therefore explores whether computational models—from earlier topic-modelling pipelines to contemporary LLMs—can assist human coders at scale while preserving rigour, reporting promising early results in political science, communication and social research. 6 , 3 , 4 Yet even with multiple annotators, majority voting improves stability only marginally because humans (and often models) share similar blind spots, particularly in accessible coding scenarios where individual models already perform well. Calibration of modern neural networks. Large–scale studies reveal that soft–max probabilities produced by deep classifiers are often miscalibrated; simple post–hoc schemes such as temperature scaling shrink but do not eliminate over–confidence. 12 , 19 , 13 Broader surveys propose dedicated metrics to quantify calibration quality. 20 Deep ensembles add epistemic uncertainty, with Bayesian approximations such as SWAG further refining predictive distributions. 21 , 22 However, all these methods presuppose access to an oracle ground–truth label, a luxury rarely available in exploratory qualitative research. Ensemble voting and uncertainty in NLP. Disagreement among independently prompted or fine–tuned models has been leveraged to flag hard examples, measure trust scores, and reduce hallucinations. 23 Recent work on uncertainty–aware majority voting extends this logic to crowdsourced labels. 24 Diversity within the ensemble therefore serves as an additional uncertainty cue that complements confidence; classic machine–learning theory offers quantitative diversity indices. 25 Our study unifies confidence and diversity in the absence of ground truth for accessible qualitative coding tasks, showing that their linear combination predicts agreement with R^{2}=0.979 in this specific domain, establishing a methodological foundation for more challenging scenarios. Results Human baseline comparison To contextualise the dual–signal gains we first benchmarked model performance against a traditional two–coder baseline and our full three–tier review workflow (Table 1 ). Across the 5 680 coding decisions the eight–model majority already surpasses typical human reliability, achieving a chance–corrected Cohen’s \kappa=0.89 and halving the error rate relative to double human coding. This substantial performance advantage reflects the relative accessibility of our coding tasks for current LLMs. Integrating the dual–signal triage and expert reconciliation lifts effective reliability to \kappa=0.93 while trimming the residual error rate to 4.1%. These figures derive from the audited interview corpus of 71 segments (see Methods). Table 1: Reliability of human versus AI coding. Condition Cohen’s \kappa Error rate (%) Decisions ( n ) Human pair (research assistants) 0.67 11.4 1 420 AI majority (8 models)

Read the complete text