A Large-Scale Empirical Comparison of Meta-Learners and Causal Forests for Heterogeneous Treatment Effect Estimation in Marketing Uplift Modeling
Aman Singh · 2026
Read A Large-Scale Empirical Comparison of Meta-Learners and Causal Forests for Heterogeneous Treatment Effect Estimation in Marketing Uplift Modeling by Aman Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06123.txt.
Introduction
Singh benchmarks four methods for estimating individual-level treatment effects in marketing, using nearly 14 million customer records from Criteo Uplift v2.1. The S-Learner, T-Learner, and X-Learner use LightGBM base learners, while Causal Forest comes from EconML. Judged by Qini score and cumulative gain, the S-Learner leads at 0.376, with its top 20 percent of ranked customers capturing about 78 percent of extra conversions. A driver analysis points to feature f8, and uncertainty analysis flags about 2 percent of customers as confident persuadables.
Catalog record
| CREATOR | Aman Singh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14218 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.06123
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Large-Scale Empirical Comparison of Meta-Learners and Causal Forests for Heterogeneous Treatment Effect Estimation in Marketing Uplift Modeling Aman Singh Abstract Estimating Conditional Average Treatment Effects (CATE) at the individual level is central to precision marketing, yet systematic benchmarking of uplift modeling methods at industrial scale remains limited. We present UpliftBench , an empirical evaluation of four CATE estimators: S-Learner, T-Learner, X-Learner (all with LightGBM base learners), and Causal Forest (EconML), applied to the Criteo Uplift v2.1 dataset comprising 13.98 million customer records. The near-random treatment assignment (propensity AUC = 0.509) provides strong internal validity for causal estimation. Evaluated via Qini coefficient and cumulative gain curves, the S-Learner achieves the highest Qini score of 0.376, with the top 20% of customers ranked by predicted CATE capturing 77.7% of all incremental conversions, a 3.9 × \times improvement over random targeting. SHAP analysis identifies f8 as the dominant heterogeneous treatment effect (HTE) driver among the 12 anonymized covariates. Causal Forest uncertainty quantification reveals that 1.9% of customers are confident persuadables (lower 95% CI > 0 >0 ) and 0.1% are confident sleeping dogs (upper 95% CI < 0 <0 ). Our results provide practitioners with evidence-based guidance on method selection for large-scale uplift modeling pipelines. 1 Introduction Standard A/B testing yields Average Treatment Effects (ATE), campaign-level estimates that cannot distinguish between customers who respond positively, negatively, or not at all to a marketing intervention. In large-scale digital marketing, this limitation creates three economically costly targeting errors: wasted spend on always-converters, negative returns from sleeping dogs who react adversely to treatment, and missed revenue from high-responders who are under-prioritized. Uplift modeling, grounded in the Neyman-Rubin potential outcomes framework [ 10 ] , addresses this by estimating the Conditional Average Treatment Effect: \tau(\mathbf{x})=\mathbb{E}[Y(1)-Y(0)\mid\mathbf{X}=\mathbf{x}] (1) where Y(1) and Y(0) are potential outcomes under treatment and control, and \mathbf{X} is the covariate vector. Individual-level CATE estimates enable precision targeting: deploy treatment only where \tau(\mathbf{x})>\text{threshold} . Despite a growing literature on CATE estimation methods, systematic empirical comparisons at industrial scale remain sparse. Existing benchmarks [ 4 , 8 ] are largely conducted at smaller scales or on synthetic data. We contribute UpliftBench : a reproducible, large-scale benchmark comparing meta-learner and Causal Forest approaches on the Criteo Uplift v2.1 dataset, with full evaluation infrastructure including Qini scoring, policy simulation, uncertainty quantification, and SHAP-based HTE attribution. 2 Related Work Meta-learners for CATE estimation. [ 8 ] introduced the X-Learner, demonstrating improved CATE estimation under treatment imbalance relative to S-Learner and T-Learner approaches. The S-Learner fits a single outcome model with treatment as a feature; the T-Learner fits separate models per treatment arm. Both are reviewed in [ 3 ] , who analyze their bias-variance tradeoffs theoretically. Causal Forests. [ 11 ] introduced Causal Forests as a nonparametric approach to CATE estimation, providing asymptotically valid confidence intervals via honest splitting. [ 1 ] extended this to generalized random forests. EconML [ 2 ] provides the production implementation used in this work. Uplift benchmarking. [ 4 ] conduct a systematic comparison of uplift modeling methods across multiple datasets, finding that ensemble-based methods generally outperform simpler approaches. [ 6 ] examine evaluation metrics including Qini coefficient and AUUC in the context of marketing attribution. Our work extends this literature to the 13.98M-record scale with LightGBM-based meta-learners and full uncertainty quantification. Evaluation metrics. The Qini coefficient [ 9 ] measures the area between the model’s cumulative gain curve and the random targeting baseline, providing a scalar ranking metric for uplift models. We adopt Qini as our primary evaluation metric, supplemented by cumulative gain curves and policy simulation. 3 Dataset We use the Criteo Uplift v2.1 dataset [ 5 ] , a public benchmark released by the Criteo AI Lab for uplift modeling research. Table 1 summarizes key properties. The near-random propensity score (AUC \approx 0.51) confirms near-random treatment assignment, lending the CATE estimates high internal validity. The 85/15 treatment imbalance motivates inclusion of the X-Learner, specifically designed for imbalanced settings. We model the visit outcome throughout; conversion modeling is left for future work given its 0.29% base rate. Table 1: Criteo Uplift v2.1 Dataset Summary Property Value Records 13,979,592 Features 12 covariates (f0–f11) Treatment 85% treated / 15% control Primary Outcome visit (4.7% positive rate) Secondary Outcome conversion (0.29% rate) Propensity AUC 0.509 4 Methods 4.1 Pipeline Overview The full pipeline proceeds as follows: (1) Preprocessing : 80/20 stratified train-test split (random_state=42); StandardScaler normalization applied to all 12 covariates. (2) Propensity estimation : Logistic Regression propensity score estimated on training data; score appended as a 13th feature for meta-learner training. (3) CATE estimation : Four methods trained on the training split. (4) Evaluation : Qini coefficient, cumulative gain curves, and policy simulation computed on the held-out test set. (5) Interpretability : SHAP values computed on T-Learner predictions to attribute HTE to individual covariates. 4.2 CATE Estimators All meta-learners use LightGBM as the base learner, a gradient-boosted decision tree framework optimized for speed and performance on large tabular datasets. S-Learner. A single outcome model \mu(\mathbf{x},t) is trained with treatment indicator t as an additional feature. CATE is estimated as \hat{\tau}(\mathbf{x})=\mu(\mathbf{x},1)-\mu(\mathbf{x},0) . The S-Learner regularizes toward zero treatment effect, reducing variance at the potential cost of bias. T-Learner. Separate outcome models \mu_{1}(\mathbf{x}) and \mu_{0}(\mathbf{x}) are trained on treated and control populations respectively. CATE is estimated as \hat{\tau}(\mathbf{x})=\mu_{1}(\mathbf{x})-\mu_{0}(\mathbf{x}) . This approach captures group-specific patterns but may overfit under the 85/15 treatment imbalance. X-Learner [ 8 ] . A two-stage cross-fitting approach that imputes individual treatment effects by applying each group’s outcome model to the opposite group, then re-fitting on the imputed effects weighted by propensity scores. Designed to handle treatment imbalance. Causal Forest [ 11 ] . EconML’s doubly robust Causal Forest with honest splitting, providing point estimates and 95% bootstrap confidence intervals at the individual level. Evaluated on a 10% stratified subsample ( \approx 280K records) due to computational constraints. 4.3 Evaluation Metrics Qini Coefficient. Primary ranking metric. Measures the area between the model’s cumulative gain curve and the random targeting baseline: \text{Qini}=\int_{0}^{1}\left[G(\phi)-\phi\cdot G(1)\right]d\phi (2) where G(\phi) is the fraction of incremental conversions captured when targeting the top \phi fraction of customers ranked by descending predicted CATE. Policy Simulation. Translates Qini curves into actionable budget vs. incremental conversion tradeoff estimates for campaign planning. Uncertainty Quantification. Causal Forest 95% confidence intervals segment customers into confident persuadables (lower CI >0 ), confident sleeping dogs (upper CI <0 ), and uncertain segments. 5 Results 5.1 Model Comparison Table 2 summarizes performance across all four estimators. Figure 1 shows the Qini curves; the S-Learner dominates across all targeting thresholds. Figure 2 shows the CATE distributions for S-Learner and T-Learner, illustrating the variance difference. Figure 1: Qini curves for all four CATE estimators. S-Learner achieves the highest cumulative gain across all targeting thresholds. Table 2: Model Performance on Criteo Uplift v2.1 Model CATE Mean CATE Std Qini Top 20% Top 50% S-Learner 0.0070 0.0223 0.3759 77.7% 95.8% X-Learner 0.0074 0.0248 0.3615 76.0% 92.8% T-Learner 0.0074 0.0267 0.3503 75.1% 92.4% Causal Forest 0.0072 0.0377 0.2524 – – Random Baseline – – \approx 0.007 20.0% 50.0% The S-Learner achieves the highest Qini score (0.3759) and top-decile capture rate with the lowest CATE variance, indicating the most stable ranking behavior for campaign targeting.