Calibration Strategies for Robust Causal Estimation: Theoretical and Empirical Insights on Propensity Score-Based Estimators

Sven Klaassen, Jan Rabenseifner, Jannis Kueck, Philipp Bach · 2025

Read Calibration Strategies for Robust Causal Estimation: Theoretical and Empirical Insights on Propensity Score-Based Estimators by Sven Klaassen, Jan Rabenseifner, Jannis Kueck, Philipp Bach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17290.txt.

Introduction

Sven Klaassen and three co-authors study calibration for propensity score estimators, in work from 2025. Methods such as inverse probability weighting and double machine learning lean on estimated treatment probabilities, which wobble when groups barely overlap or samples are small. The paper refines calibration schemes and studies how the split between estimation and calibration data affects validity.

Theory for calibrated estimators inside the double learning frame is paired with wide simulations. Calibration cut variance and bias for weighting estimators, even in small samples, and steadied flexible learners such as gradient boosting without losing double robustness. With the right splitting scheme, adding calibration never hurt.

Catalog record

CREATORSven Klaassen, Jan Rabenseifner, Jannis Kueck, Philipp Bach
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Calibration Strategies for Robust Causal Estimation: Theoretical and Empirical Insights on Propensity Score-Based Estimators Sven Klaassen, Jan Rabenseifner, Jannis Kueck, Philipp Bach Abstract The partitioning of data for estimation and calibration critically impacts the performance of propensity score based estimators like inverse probability weighting (IPW) and double/debiased machine learning (DML) frameworks. We extend recent advances in calibration techniques for propensity score estimation, improving the robustness of propensity scores in challenging settings such as limited overlap, small sample sizes, or unbalanced data. Our contributions are twofold: First, we provide a theoretical analysis of the properties of calibrated estimators in the context of DML. To this end, we refine existing calibration frameworks for propensity score models, with a particular emphasis on the role of sample-splitting schemes in ensuring valid causal inference. Second, through extensive simulations, we show that calibration reduces variance of inverse-based propensity score estimators while also mitigating bias in IPW, even in small-sample regimes. Notably, calibration improves stability for flexible learners (e.g., gradient boosting) while preserving the doubly robust properties of DML. A key insight is that, even when methods perform well without calibration, incorporating a calibration step does not degrade performance, provided that an appropriate sample-splitting approach is chosen. Key words: Balancing, Calibration, Double Machine Learning, IPW, Sample Splitting † † footnotetext: \dagger Corresponding author email: [email protected] 1 Introduction 1.1 Motivation In many settings of the causal inference literature, researchers are interested in the effect of a (binary) treatment D\in\{0,1\} on an outcome Y\in\mathbb{R} . If the treatment is not assigned randomly, a common assumption is the so-called unconfoundedness assumption Y(0),Y(1)\perp D\mid X assuming that the potential outcomes Y(1) and Y(0) are independent of the actual treatment status D conditional on control variables X . Let \displaystyle m_{0}(x):=P(D=1|X=x)=\mathbb{E}[D|X=x]. (1) be the propensity score. As famously shown in Rosenbaum and Rubin (1983) conditioning on the propensity score is sufficient to effectively account for the confounding through X Y(0),Y(1)\perp D\mid m_{0}(X). Consequently, propensity scores are a cornerstone of modern causal inference for addressing confounding in observational studies. They enable balancing treatment and control groups, allowing for unbiased estimation of treatment effects under unconfoundedness. Commonly, propensity scores are used in methods such as inverse probability weighting (IPW), matching, stratification, Bayesian causal inference and more recently double machine learning (DML). Moreover, effective propensity adjustment requires sufficient overlap for propensity score-based estimators. When overlap between treatment and control groups is limited, or treatment assignment is unbalanced, propensity scores can become extreme (i.e. close to 0 or 1) leading to instability of the causal estimates. In such cases, estimators can suffer from inflated variance as extreme weights in IPW disproportionately amplify small errors in propensity score estimation. Similarly, matching algorithms may struggle to find suitable matches, resulting in biased estimates. These challenges highlight the importance of robust and well-calibrated propensity score models to maintain the reliability of causal estimates. To mitigate instability, researchers often enforce common support by trimming or bounding propensity scores. For example, observations with propensity scores below a certain threshold or outside a predefined range are excluded from analysis. While this approach can reduce variance, it does so at the expense of bias, as it discards valuable information and reduces sample size. Such trade-offs are particularly problematic in small-sample settings, where the exclusion of even a few observations can significantly impact the precision and validity of treatment effect estimates. To use the propensity score for balancing, the property \displaystyle m_{0}(X)=\mathbb{E}[D|m_{0}(X)] (2) is crucial, e.g. see Theorem 2 in Rosenbaum and Rubin (1983) . In the classification literature, Equation ( 2 ) is known as the so-called calibration property. 1 1 1 To be precise, this notion of calibration is also referred to as conditional calibration, which is equivalent to probabilistic calibration for binary outcomes ( Gneiting and Ranjan, 2011 ) . Intuitively, a (binary) classifier \hat{m}(\cdot) is well calibrated if the percentage of positive labels ( D=1 ) is approximately m for all instances with \hat{m}(X)\approx m . As the true propensity score is the conditional expectation it is calibrated m_{0}(X)=\mathbb{E}[D|X]=\mathbb{E}[\mathbb{E}[D|X]|\mathbb{E}[D|X]]=\mathbb{E}[D|m_{0}(X)].\quad P\text{-a.s.} In most settings the true propensity score is not known such that it is typically estimated via some classification algorithm such as logistic regression, random forest, boosting methods or even deep neural networks. Consequently, the resulting estimator \hat{m}(\cdot) of m_{0}(\cdot) might not be calibrated, e.g. the percentage of treated units with \hat{m}(X)\approx m might differ substantially from m for certain values of m\in(0,1) . Accurately estimating treatment probabilities is crucial for valid causal inference. Inverse propensity score estimators aim to balance covariates between treatment and control groups. This balancing ensures more reliable and unbiased treatment effect estimates by aligning the covariate distributions, improving the comparability of the treatment groups in terms of observable characteristics. Whereas methods like isotonic regression and Platt scaling refer to the calibration of predictions, critical questions remain about the optimal integration of these calibration techniques in causal estimation workflows. 1.2 Related Literature Recent advances in calibration for propensity score estimation and causal inference emphasize three interconnected themes: the adaptation of machine learning calibration techniques to causal settings, stabilization strategies for inverse probability weighting (IPW), and theoretical insights into finite-sample performance. Initial studies by Deshpande and Kuleshov (2023) showed significant improvements using single split calibration on data with deterministic treatment assignments and complex settings with hidden confounders. They specifically highlight variance reduction properties and analyze the regret of recalibration of propensity scores. Their proofs are independent of the sample size N , as their calibration framework consists of splitting the data into training and calibration sets. This distinction is crucial for our work, as we focus on sample size dependent calibration algorithms. Gutman et al. (2024) demonstrated that post-processing propensity scores using methods such as Platt scaling, referred to as post-calibration, improves treatment effect estimation by correcting propensity score distortions beyond covariate balancing. Calibration significantly benefits miscalibrated models like tree-based methods (e.g. gradient boosting), while logistic regression, contrary to the findings of Deshpande and Kuleshov (2023) , shows minimal gains. This suggests that post-calibration allows flexible learners to retain their predictive accuracy while enhancing robustness to data challenges such as small sample sizes, model misspecification, class imbalance, or limited overlap, thus challenging the conventional trade-off between model complexity and calibration. Ballinari and Bearth (2025) use nested cross-fitting, reserving distinct data folds for propensity estimation and calibration, which is a straightforward application of the double machine learning theory of Chernozhukov et al. (2018) . They reported instability in small samples, a limitation attributed to reduced effective sample sizes for both steps. These studies collectively identify a tension between calibration flexibility and stability, particularly in finite-sample regimes. Efforts to address these challenges have taken different paths. For instance, van der Laan et al. (2024a) proposed a stratified calibration for treated and control groups to stabilize IPW weights. In another work, they extended calibration frameworks to estimate heterogeneous treatment effects ( van der Laan et al., 2023 ) . Meanwhile, van der Laan et al. (2024b) extend automatic debiased machine learning (autoDML) ( Chernozhukov et al., 2022 ; Chernozhukov et al., 2024 ) by calibrating both outcome regression and Riesz representer. They demonstrate that the calibration step yields doubly robust asymptotically linear estimators, reducing nuisance rate requirements. Theoretical work by

Read the complete text