Decomposing Inequalities using Machine Learning and Overcoming Common Support Issues

Emmanuel Flachaire, Bertille Picard · 2025

Read Decomposing Inequalities using Machine Learning and Overcoming Common Support Issues by Emmanuel Flachaire, Bertille Picard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13433.txt.

Introduction

Emmanuel Flachaire and Bertille Picard rebuild the Kitagawa-Oaxaca-Blinder decomposition of group gaps with machine learning tools. The classic method splits a mean gap between two groups into a part explained by observed traits and a part left unexplained. Results hinge on the reference outcome chosen, plus overlap and model errors.

Their version uses potential outcomes and extends a weighted reference approach with a doubly robust estimator. Neyman orthogonality and double machine learning let the method avoid trimming and guessing beyond the data. Two real examples show the gains in flexibility, though the authors warn that adding useless explanatory variables can still distort results under this reference choice.

Catalog record

CREATOREmmanuel Flachaire, Bertille Picard
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE107 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13433

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decomposing Inequalities using Machine Learning and Overcoming Common Support Issues Emmanuel Flachaire, Bertille Picard Abstract The Kitagawa-Oaxaca-Blinder decomposition splits the difference in means between two groups into an explained part, due to observable factors, and an unexplained part. In this paper, we reformulate this framework using potential outcomes, highlighting the critical role of the reference outcome. To address limitations like common support and model misspecification, we extend Neumark’s (1988) weighted reference approach with a doubly robust estimator. Using Neyman orthogonality and double machine learning, our method avoids trimming and extrapolation. This improves flexibility and robustness, as illustrated by two empirical applications. Nevertheless, we also highlight that the decomposition based on the Neumark reference outcome is particularly sensitive to the inclusion of irrelevant explanatory variables. JEL : C46, D31 Keywords : inequality, decomposition methods, machine learning 1 Introduction The Kitagawa-Oaxaca-Blinder decomposition ( 19 ; 24 ; 4 ) is widely used in empirical studies to study differences in distributional statistics between two groups or two periods ( 8 ; 9 ; 10 ) . 1 1 1 While the decomposition method is also referred to as the “Oaxaca-Blinder decomposition” in the economics literature, similar principles had been introduced earlier by Kitagawa in demographics and sociology. The links between these contributions are discussed in 23 . Given the generality of our counterfactual framework in this paper, we do not attribute the method to any single author and instead cite all three contributions. It separates an observed difference into an explained part due to individuals’ observed characteristics and an unexplained part. This method is typically used to decompose average wage differences between men and women. In classical decompositions, two major methodological choices are made. The first methodological choice consists of selecting a reference outcome. In the context of gender wage gap decompositions, the reference outcome is the wage an individual would receive in a counterfactual setting where men and women would be paid according to the same wage model. This means that each characteristic, such as education or experience, determines the reference outcome in the same way for men and women. In other words, the reference outcome is a counterfactual outcome that would be identical for an individual in one group and an individual in another group, sharing the same observable characteristics. Classical decompositions in the literature assume that the reference outcome is either the wage model of men or of women. Empirically, the final decompositions vary according to this methodological choice, with no consensus which group’s potential outcome should be chosen as the reference ( 11 ) . The second traditional methodological choice is to impose assumptions about outcome models, such as linearity, invertibility, and exogeneity, to perform ordinary least squares estimates. These standard assumptions can be relaxed by using reweighting techniques ( 11 ) and machine learning methods ( 28 ) . However, these techniques require an additional common support assumption. In this paper, we propose a method to combine the advantages of both approaches, avoiding the restrictive assumptions of a standard linear model through machine learning and reweighting methods, without relying on the common support assumption. This is achieved by changing the reference outcome. Instead of choosing the outcome of one of the two groups as the reference outcome, we propose using a propensity score-weighted combination of the two potential outcomes as a reference, following 21 in the standard linear framework. To understand why changing the reference outcome helps relaxing these standard assumptions simultaneously, we underline the idea that the common support hypothesis is needed whenever both reweighting techniques are used and the outcome of one of the two groups is taken as the reference outcome. Indeed, the combination of these two choices forces us to reweight the observed outcomes by the probability of belonging to the reference group, conditional on individual characteristics. This implies divisions by the conditional probability of belonging to the reference group, which is feasible only when these probabilities are bounded away from zero (hence the common support assumption). Intuitively, therefore, it is necessary for individuals to have comparable counterparts in the reference group, to avoid unstable weights driven by very small estimated probabilities. While this condition may be credible in situations close to randomized controlled trials, as in impact evaluation programs, it becomes more questionable in the context of inequality decomposition. For instance, using a large data set of 1.7 million employees in Switzerland, 28 find that with the 10 most important variables for explaining male wages, 55% of women in the private sector have no comparable men with respect to these variables. They show that the results are sensitive to the trimming used to have observationally comparable men and women, and thus that common support is a key issue. However, dividing by conditional probabilities is no longer required when choosing a reference outcome defined as a propensity score-weighted sum of the two potential outcomes. This choice of reference outcome was originally proposed by 21 within a linear framework, where it was limited to the use of only a few categorical variables. This reference outcome is sensitive to gender composition within sectors and is consistent with the standard concept of equal redistribution conditional on a set of characteristics. We extend this reference outcome to the case of a large set of characteristics, either categorical or continuous, and derive several estimators of the unexplained part of the mean difference. Unlike those based on standard reference outcomes, we show that the resulting reweighting estimators do not require the imposition of the common support hypothesis. To tackle the assumption on the correct parametric specification of the reference outcome or the propensity score regressions, we consider Machine Learning (ML) techniques. These methods allow us to estimate potentially complex non-linear functions by mapping inputs to outputs and/or by performing variable selection when a large number of characteristics is available. Although it may be tempting to introduce ML estimates of unknown functions directly into standard estimators, this naive approach is not recommended because it provides biased estimators that converge at slower rates than the {n}^{-1/2} rate obtained in the parametric framework. A more appropriate approach has been developed in recent literature on treatment effects. 6 provided a so-called double machine learning procedure to recover debiased estimators that achieve root-n convergence rates for the Average Treatment Effect (ATE) and Average Treatment Effect on the Treated (ATT). When the unexplained part in the Kitagawa-Oaxaca-Blinder decomposition can be interpreted as an ATT ( 11 ) , double machine learning can thus be used to estimate it with good statistical properties ( 28 ) . Our main contribution can be summarized as follows: drawing on Rubin’s conceptual framework ( 27 ) and reformulating the decomposition problem in terms of potential outcomes, we rewrite the unexplained part when the reference outcome is directly inspired by what is proposed by 21 . This means that we choose as a reference outcome a combination of the potential outcomes of the two groups, weighted by conditional probabilities of being in each group. While the unexplained part can be analogous to an Average Treatment effect on the Treated (ATT) or an Average Treatment effect on the Untreated (ATU) when choosing standard reference outcomes, this is no longer the case when

Read the complete text