Learning Representational Disparities
Pavan Ravishankar, Rushabh Shah, Daniel B. Neill, Machine Learning for Good Laboratory, New York University (NYU), Courant Institute of Mathematical Sciences, Department of Computer Science, NYU, Robert F. Wagner Graduate School of Public Service, NYU, Center for Urban Science, Progress, Tandon Scho · 2025
Read Learning Representational Disparities by Pavan Ravishankar, Rushabh Shah, Daniel B. Neill, Machine Learning for Good Laboratory, New York University (NYU), Courant Institute of Mathematical Sciences, Department of Computer Science, NYU, Robert F. Wagner Graduate School of Public Service, NYU, Center for Urban Science, Progress, Tandon Scho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17533.txt.
Introduction
Pavan Ravishankar, Rushabh Shah, and Daniel Neill build a fairness method that studies how decision makers see people in a 2025 paper. When a human's decision leads to unequal outcomes, part of the cause may be representation: the decision maker effectively sees a distorted version of each person. The authors name that gap a representational disparity, between the view behind real decisions and the view a fairer decider would hold.
Their algorithm learns that gap as an interpretable neural model, framed as a multi-objective optimization. Under stated simplifying assumptions, they prove the learned weights can fully cancel the outcome disparity in their model. The point is practical: an interpretable gap suggests exact nudges to the human decision, forms, prompts, or checklists, rather than a black box override.
Catalog record
| CREATOR | Pavan Ravishankar, Rushabh Shah, Daniel B. Neill, Machine Learning for Good Laboratory, New York University (NYU), Courant Institute of Mathematical Sciences, Department of Computer Science, NYU, Robert F. Wagner Graduate School of Public Service, NYU, Center for Urban Science, Progress, Tandon Scho |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17533
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Representational Disparities Pavan Ravishankar, Rushabh Shah, Daniel B. Neill, Machine Learning for Good Laboratory, New York University (NYU), Courant Institute of Mathematical Sciences, Department of Computer Science, NYU, Robert F. Wagner Graduate School of Public Service, NYU, Center for Urban Science, Progress, Tandon School of Engineering, NYU Abstract We propose a fair machine learning algorithm to model interpretable differences between observed and desired human decision-making, with the latter aimed at reducing disparity in a downstream outcome impacted by the human decision. Prior work learns fair representations without considering the outcome in the decision-making process. We model the outcome disparities as arising due to the different representations of the input seen by the observed and desired decision-maker, which we term representational disparities. Our goal is to learn interpretable representational disparities which could potentially be corrected by specific nudges to the human decision, mitigating disparities in the downstream outcome; we frame this as a multi-objective optimization problem using a neural network. Under reasonable simplifying assumptions, we prove that our neural network model of the representational disparity learns interpretable weights that fully mitigate the outcome disparity. We validate objectives and interpret results using real-world German Credit, Adult, and Heritage Health datasets. 1 Introduction Many human decisions are made with the aid of machine learning algorithms, which have had a significant impact across various domains such as healthcare [ 6 ] . Machine learning algorithms have frequently been criticized in the fairness literature due to their potential to create or exacerbate disparities [ 1 , 2 , 19 ] . However, human decisions can also exhibit demographic and other biases, whether intentional (e.g., racial animus) or unintentional, with undesirable impacts, e.g., in policing [ 8 ] . Moreover, even when presented with ostensibly “fair” algorithmic predictions, human decision-makers tend to deviate from these predictions in systematically biased ways [ 10 ] , suggesting a need to provide more concrete and interpretable recommendations for behavioral change. In this paper, we propose an algorithmic aid to mitigate downstream disparity in outcomes resulting from human decisions. We describe a methodological solution to model and correct differences between an observed human decision-making process, with resulting disparity in outcomes impacting some protected class, and a desired (fairer) human decision-making process, which differs from the observed decisions in a systematic and interpretable way, and mitigates the observed disparity. The differences serve as concrete aids to nudge the human decision-maker toward fairer behavior [ 25 ] , as measured by a reduction in outcome disparity. This approach aligns with the core principles of the algorithm-in-the-loop framework [ 10 ] , preserving the agency of the human decision-maker while providing them with algorithmic recommendations geared toward improving the fairness of the combined (algorithm + human) system. Concretely, we assume a decision process (Figure 1 ) in which a human decision-maker (e.g., housing agency) makes a consequential decision H (e.g., whether to give an applicant a housing voucher) that impacts a downstream outcome Y (e.g., whether the applicant is able to obtain housing). The decision could be based on the applicant’s values of a binary sensitive attribute S and other non-sensitive attributes \mathbf{X} . Critically, the human decision \mbox{Pr}(H\>|\>S,\mathbf{X}) , the outcome model \mbox{Pr}(Y\>|\>H,S,\mathbf{X}) , or both, could be biased in a way that differentially impacts the protected class S=1 . For example, a biased human decision-maker may provide housing vouchers less often to minoritized individuals, or such individuals may be less likely to obtain housing with or without a voucher. In the former case, the desired decision may be to correct H to be uncorrelated with S , but in the latter case, outcome disparities may persist unless minoritized individuals receive housing vouchers more often than non-minoritized individuals to compensate for the downstream biases in the outcome variable. Figure 1 : Data Generation Process In this paper, based on the observation that value-based (economic) decisions generally rely on memory-based representations [ 21 , 7 , 22 , 24 ] , we model the differences between the observed and desired decision-making processes, and resulting disparities in outcomes, as arising due to different representations of the inputs seen by the observed and desired decision-makers, which we term representational disparities . Concretely, we learn a shallow, mechanistically interpretable neural network model where a portion of the hidden layer models the representational disparities, thus explaining the differences between the observed and desired decisions and their resulting outcomes. The key contributions of the paper are as follows: 1. Methodology: We mathematically formulate the goal of the desired human decision-maker to reduce the disparity in outcomes between the protected and non-protected class. 2. Optimization: We learn the representational disparities between the observed and the desired decision-makers by formulating the problem as an multi-objective optimization problem with the primary objectives of (i) mitigating outcome disparity between protected and non-protected class using fair (desired) decisions; and (ii) learning representational disparities that represent interpretable differences between observed and desired decisions. 3. Theory: Under reasonable simplifying assumptions, we prove that the weights learned by the optimization procedure result in representational disparities that are interpretable and if corrected, will fully mitigate the disparity in outcomes. In more general settings, convergence to these desired weights is not guaranteed but can be achieved in practice via multiple random initializations. 4. Experiments: We validate the optimization objectives using synthetically created data sets and investigate representational disparities using real-world German Credit, Adult Income, and Heritage Health datasets. We compare our results to a foundational work Fair Representation Learning [ 26 ] , and show that our methodology provides multiple advantages including higher accuracy, increased interpretability and consistency in its recommendations (thus facilitating nudges toward fairness), and greater reduction in outcome disparity, by accounting for biases in the distribution of the outcome conditional on the human decision. The remainder of the paper is organized as follows: Section 2 reviews related work. Section 3 describes the model by formalizing the notations, setting, and optimization problem. Section 4 discusses theorems that prove that the representational disparities learned are interpretable and mitigate disparities in outcomes. Section 5 validates each objective of the optimization problem using synthetic data, investigates disparities in real-world datasets, and compares our model with competing approaches. Finally, Section 6 discusses conclusions and future work. 2 Related Work A vast corpus of literature on algorithmic fairness mainly considers the algorithm, disregarding the role of the human. Stage-specific fairness approaches analyze fairness from the lens of a particular stage of the machine learning pipeline such as data or predictions [ 19 , 2 ] , while pipeline-aware approaches analyze how bias propagates across multiple pipeline stages such as from data to the predictions, either qualitatively [ 23 , 3 ] or quantitatively [ 20 ] . Only a few works have discussed the fairness problem with both the algorithm and the human decision-maker in purview: our work is built on the algorithm-in-the-loop framework [ 10 ] , which assumes that a human decision-maker makes a consequential decision with support from an algorithmic prediction or recommendation. Numerous works have proposed methodologies and frameworks to learn fair representations [ 26 , 28 , 27 , 18 , 17 ] . A framework for fair representation is proposed by defining an \epsilon