Adversarial Debiasing for Unbiased Parameter Recovery
Luke C Sanford, Megan Ayers, Matthew Gordon, Eliana Stone · 2025
Read Adversarial Debiasing for Unbiased Parameter Recovery by Luke C Sanford, Megan Ayers, Matthew Gordon, Eliana Stone (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12323.txt.
Introduction
Luke Sanford and co-authors published this methods study in 2025. Social scientists increasingly use machine learning predictions as stand-ins for hard-to-measure outcomes, then run regressions on those predictions. The catch is that prediction errors can bend the regression results. This paper shows how that bias arises and offers a diagnostic test for detecting it. The settings covered are any where a model's predictions serve as the outcome variable.
Their fix uses adversarial training. A second model component pushes the predictor to make errors that carry no systematic tilt. Simulations and a real exercise with satellite measures of African forest cover demonstrate the approach. Naive model predictions produced biased estimates of the true effects. Predictions from the adversarial model recovered the true coefficients instead. The authors present the test and the debiasing method as practical tools for the growing field of prediction-based research.
Catalog record
| CREATOR | Luke C Sanford, Megan Ayers, Matthew Gordon, Eliana Stone |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12323
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adversarial Debiasing for Unbiased Parameter Recovery Luke C Sanford, Megan Ayers, Matthew Gordon, Eliana Stone Abstract Advances in machine learning and the increasing availability of high-dimensional data have led to the proliferation of social science research that uses the predictions of machine learning models as proxies for measures of human activity or environmental outcomes. However, prediction errors from machine learning models can lead to bias in the estimates of regression coefficients. In this paper, we show how this bias can arise, propose a test for detecting bias, and demonstrate the use of an adversarial machine learning algorithm in order to de-bias predictions. These methods are applicable to any setting where machine-learned predictions are the dependent variable in a regression. We conduct simulations and empirical exercises using ground truth and satellite data on forest cover in Africa. Using the predictions from a naive machine learning model leads to biased parameter estimates, while the predictions from the adversarial model recover the true coefficients. A Preprint Keywords Machine Learning \cdot Remote Sensing \cdot Adversarial Debiasing 1 Introduction Efforts to combat climate change, alleviate poverty, and improve food security rely on studies that evaluate the effectiveness of policy interventions. These studies increasingly rely on satellite remote sensing data and machine learning methods to measure outcomes (biodiversity, forest carbon, wealth, agricultural productivity) of interest. Converting multi-spectral, multi-temporal, and spatial data into these dependent variables previously relied on simple heuristics for dimension reduction (for example, determining the date of max greenness). More recently, researchers have turned to machine learning methods paired with labeled data to generate more precise measures of these outcomes. With the increasing availability of remote sensing data, decreasing cost of computation, and continuing improvement of machine learning tools, we are in the midst of a measurement revolution in social, economic, and environmental sciences. These advances in measurement derive from the field of remote sensing, which generally develops tools to improve measurement of particular outcomes, Y using satellite data. These approaches generate \widehat{Y}_{i}(k_{i})=Y_{i}+\nu_{i} , where k are the satellite imagery features we use to train our machine learning model and \nu_{i} is the measurement error for a given observation. This model, f(k_{i},w) , finds weights w that minimize some function of \nu in the labeled data, for example the sum of squared error: \sum_{j}\nu_{j}^{2}=\sum_{j}(\widehat{Y}_{j}(k_{j})-Y_{j})^{2} . While this is a sensible approach for descriptive tasks (how much deforestation occurred in the Amazon in 2024?), it can generate bias when \widehat{Y} is used as a dependent variable for causal inference tasks (how much deforestation in the Amazon was caused by the expansion of road networks?). This bias can occur because of differential measurement error across some independent variable, for example, if forest cover is over-estimated far from roads and under-estimated near roads. Then, researchers who estimate the effect \beta of roads X_{i} on forest cover \widehat{Y_{i}} according to the model Y_{i}=\alpha+\beta X_{i}+e_{i} will obtain in expectation: \mathbb{E}[\widehat{\beta}]=\beta+\frac{Cov(e,X)}{Var(X)}+\frac{Cov(\nu,X)}{Var(X)}. (1) \beta is the true effect, the second term is standard omitted variable bias, and the third term is measurement error bias caused by using predicted forest cover as the dependent variable rather than the true values. This is problematic when there is a correlation between the independent variable and the measurement error. We describe why we expect this type of correlation to be common in studies that use remote sensing data to generate dependent variables. We then construct a statistical test for bias arising from measurement error bias and develop a power calculation that allows researchers to estimate the number of ground truth points they will need to acquire. While other work in this area takes the model predictions \widehat{Y} as given and seeks to correct them using ground truth labels in the form of \tilde{Y}=m(\widehat{Y},Y,X,k) , we develop a class of models that generate \widehat{Y} such that Cov(\nu,X)=0 . To achieve this, we borrow an approach from the algorithmic fairness literature where prediction errors are constrained to be uncorrelated with legally protected categories like race or gender Zhang et al. (2018) . Rather than having the adversarial component of the model attempt to predict one or more protected categories, we task it with predicting treatment status (i.e. the independent variable of interest X ). This approach generates predictions that are both precise and unbiased for the researcher’s causal task. Figure 1: The model architecture of the debiasing approach 1.1 Contributions The main contribution of the paper is a novel approach to the increasingly recognized issue of using machine-learned proxies for causal tasks. While all other approaches we know of attempt to correct off-the-shelf measurements or measurements from a fixed model, we propose an adversarial component that can be added to any measurement model to generate predictions that avoid measurement error bias. This eliminates extra bias correction steps in the causal estimation process, which can decrease precision. This natively fits workflows in which researchers are constructing their own measurement models–something we expect to be more common in remote sensing with the advent of geospatial foundation models. We also develop what to our knowledge is the first statistical test for measurement error bias and the first power calculation workflow that guides researchers in understanding how much ground truth data they will need to label to ensure their results are robust to measurement error bias. 1.2 Related Work This paper contributes to a growing literature that has begun to document the problem of non-random measurement error in machine-learning models (see Jain (2020) for a review 1 1 1 For topic specific reviews, see Balboni et al. (2022) on deforestation, Gibson et al. (2021) and Bluhm and McCord (2022) on night lights, and Fowlie et al. (2019) on air pollution. ). A number of recent papers propose econometric estimators that can correct for the non-classical measurement error in some cases. Alix-Garcia and Millimet (2022) proposes a misclassification model that requires users to specify the variables that may induce measurement error (e.g. cloud cover, satellite angle). Proctor et al. (2023) suggest a multiple imputation approach that may be sensitive to functional form specifications. Egami et al. (2023) address an analagous problem in the NLP realm, developing a method similar to Proctor et al. (2023) for using gold-standard labeled data to adjust labels provided by an LLM. Zhang (2021) propose analytical bias correction techniques and MLE estimators to handle attenuation bias when classifier predictions are used as outcomes, but assume non-differential measurement error with respect to regressors. Torchiana et al. (2023) provide an approach based on hidden Markov models to correct for misclassification bias that does not require ground-truth data but requires a stronger set of structural assumptions about data generating processes. In contrast, bias estimation and correction methods ( Angelopoulos et al., 2023 , such as prediction-powered inference introduced in) and our proposed adversarial debiasing method make i.i.d sampling assumptions but do not require assumptions of functional forms or specific sources of measurement error, and are designed to target differential measurement error. 2 Adversarial Debiasing 2.1 Detecting Bias Unlike for omitted variable bias, an estimate of measurement error bias is directly obtainable if researchers have access to or can generate some ground-truth values of Y (for example, by visually interpreting high-resolution satellite imagery). Let j\in J index observations in the labeled set.