Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation
George Panagopoulos · 2026
Read Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation by George Panagopoulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10430.txt.
Description
George Panagopoulos examines a split in how treatment-effect methods are judged. Estimating varied treatment effects with machine learning attracts both academics and industry, but the two camps test under different conditions. Methods papers use simulated benchmarks and metrics needing what-if outcomes, while real applications use visible metrics based on ranking or test outcomes. His 2026 paper compares those test regimes head to head.
Across standard simulated benchmark families and real-world datasets, covering meta-learners with multiple base learners plus special causal models, Panagopoulos finds two gaps. What-if metrics do not reliably pick out the estimators favored by visible metrics, even on matching simulated benchmarks. Rankings from simulated benchmarks fail to carry over to real datasets. Simple meta-learners with strong base models prove consistently competitive, unlike specialized causal models. Progress, he argues, should be judged with observable metrics and real-data validation, not simulated benchmarks alone.
Catalog record
| CREATOR | George Panagopoulos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10430