When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression Journal: arXiv
Marcell T. Kurbucz · 2026
Read When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression Journal: arXiv by Marcell T. Kurbucz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12780.txt.
Introduction
Kurbucz builds a calibration checking tool for pseudo-label flows. Calibrated outputs of models are used as inputs to later targets such as effects, rates or gaps for a latent group seen on a subset. A usual practice cuts the calibrated score at a trust cut and treats the hard label as truth. Building on a finding, he derives a closed-form formula for the shrinkage bias that cutting induces in the later regression slope. That bias is known before fitting runs from rest score spread.
Catalog record
| CREATOR | Marcell T. Kurbucz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40751 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression Journal: arXiv Marcell T. Kurbucz Abstract Calibrated probability outputs of trained classifiers are increasingly used as inputs to downstream regression estimands such as effects, prevalences, or disparities for a latent group observed only on a small labelled subset. A standard practice is to threshold the calibrated score at a confidence cutoff and treat the hard label as the truth. Building on a recent identification result for the underlying moment equation, we develop a calibration-aware diagnostic apparatus for pseudo-labelling pipelines. We derive a closed-form expression for the attenuation bias that confidence thresholding induces in the downstream regression coefficient, and show that the bias can be predicted, before any inference is run, from the residual score variance V ∗ = 𝔼 [ Var ( p ∣ X ) ] V^{*}=\mathbb{E}[\operatorname{Var}(p\mid X)] on the unlabelled set after partialling out the downstream controls X X . We further obtain a sharp sensitivity bound under bounded calibration drift, and identify the boundary V ∗ = 0 V^{*}=0 , which holds iff p p is a deterministic function of X X ; this motivates a structural separation between classifier features W W and downstream controls X ⊊ W X\subsetneq W . Five controlled simulations and a UCI Adult illustration trace the predictions. The contribution is operational: a ( V ∗ , κ ) (V^{*},\kappa) decision rule that practitioners can compute from any classifier output to decide whether confidence thresholding is safe. 1 Introduction Modern machine-learning pipelines increasingly emit calibrated probability scores as primary outputs rather than intermediate classification decisions, and these scores are then used as inputs to a downstream inferential task: a clinical risk classifier feeds a population-level resource-allocation calculation; a fairness audit uses a proxy classifier for an unobserved protected attribute and estimates a disparity in some outcome [ 7 ] ; prevalence estimation in biomedical imaging labels a large unlabelled population and regresses the outcome of interest on the predicted class; pattern recognition systems for medical, biometric, and behavioural data feed calibrated risk scores into operational decision rules. In each case the classifier is trained on a small labelled subset L and applied to a much larger unlabelled subset U , with the resulting probability used in a downstream regression. The dominant practice in pseudo-labelling [ 11 , 22 , 29 , 26 , 23 ] converts the calibrated probability to a hard label by thresholding at some confidence level \tau_{thr} , typically 0.9 or 0.95 . The choice has been recognised as empirically consequential [ 22 , 29 , 1 ] , but no quantitative account exists of how thresholding biases a downstream estimand. This paper supplies that quantitative account in a deliberately restricted regression setting that admits a sharp closed-form analysis. The downstream task is the partial association between a latent binary group indicator G and an outcome Y , controlling for a covariate vector X , formalised in Section 3.1 as the partially linear regression ( 1 ); the classifier is trained on a (possibly richer) feature set W\supseteq X to produce a calibrated probability p=f(W) on the unlabelled set U . The companion identification paper [ 9 ] establishes that the partial association \tau in ( 1 ) is point-identified by a moment equation in the calibrated probability p , with V^{*}=\mathbb{E}[\operatorname{Var}(p\mid X)] as the asymptotic-precision quantity. That paper is identification-theoretic; it does not analyse the pseudo-labelling practice that motivated the question, nor does it construct a practitioner-facing decision rule. The present paper takes the identification result as given and develops its consequences for pseudo-labelling. Concretely, we make three contributions, formalised in Section 4 as five predictions (P1–P5) and verified empirically in Section 5 : (i) a closed-form attenuation factor \kappa(\tau_{thr}) for hard pseudo-labelling, expressed as a ratio of residual covariance moments and computable on the unlabelled set alone—the central new object of this paper; (ii) a bias-variance regret result for confidence-thresholded abstention (FixMatch-style), with closed-form bias and an explicit variance inflation, neither of which appears in [ 9 ] ; and (iii) a one-page diagnostic procedure, the (\widehat{V^{*}},\widehat{\kappa}) decision rule, that operationalises the choice between the supervised, soft, and hard estimators (Algorithm 1 ). The sandwich-variance result and the calibration-drift sensitivity bound, recalled in Section 3.3 and Section 4.4 for self-containedness, are taken from the companion paper and applied here in the pseudo-labelling setting. The paper does not propose a new pseudo-labelling algorithm, predict classification accuracy, or compete with FixMatch on its native loss: it studies a single-pass regression on pseudo-labels and provides a calibration-aware diagnostic for the downstream estimand. Five controlled simulations and a public-data illustration on UCI Adult [ 8 ] trace the predictions. The remainder is organised as follows. Section 2 situates the work in the literature; Section 3 formalises the setup and recalls the anchor identification result; Section 4 develops the five predictions and the diagnostic procedure; Section 5 reports the experiments; Section 6 discusses operational implications and limitations. 2 Related work The work draws on four threads. Pseudo-labelling [ 11 ] converts predicted class probabilities of unlabelled examples into hard labels and trains the classifier on the augmented set. FixMatch [ 22 ] adds a consistency loss and restricts the pseudo-label to confident examples; FlexMatch [ 29 ] and FreeMatch [ 26 ] replace the global threshold with class-adaptive ones; CoMatch [ 13 ] adds graph regularisation; Mean Teacher [ 23 ] uses an EMA of the classifier as a target. Uncertainty-aware [ 20 ] and class-imbalanced [ 25 ] variants further refine the procedure, while [ 1 ] document the confirmation-bias dynamics of iterative retraining and [ 24 ] survey the broader semi-supervised landscape. The unifying observation is that confidence thresholding works empirically but its theoretical justification is heuristic; our contribution is the precise, closed-form bias that any confidence threshold induces in a downstream regression coefficient. The second thread is the calibration of probabilistic predictors. Post-hoc calibration via Platt scaling [ 19 ] , isotonic regression [ 28 ] , or related constructions [ 17 ] produces scores that can be interpreted as probabilities. Modern deep networks are systematically overconfident [ 5 ] , with newer architectures calibrating better but imperfectly [ 15 ] , and calibration degrades under distribution shift [ 18 ] . None of this work translates calibration quality into a downstream-estimand bias bound, which we do in the restricted regression setting. The third thread is soft-label learning. [ 6 ] introduced soft-target distillation; [ 16 ] clarified the role of label smoothing. The soft-versus-hard distinction in our setting is structurally similar but in a different functional: there the target is the classifier itself, here the target is a downstream regression coefficient, and the closed-form attenuation factor we derive is new to the pseudo-labelling literature. The fourth thread, measurement error and misclassification, has a long history in econometrics. [ 12 ] showed that misclassification of a binary regressor attenuates the estimated effect, and [ 14 ] obtained identification under an instrumental variable. Our setting differs because we observe a calibrated probability rather than a noisy binary label; the identification argument therefore changes, but the attenuation phenomenon is structurally analogous. The semiparametric backbone of the moment estimator is