Beyond Internal Data: Bounding and Estimating Fairness from Incomplete Data
Varsha Ramineni, Hossein A. Rahmani, Emine Yilmaz, David Barber · 2025
Read Beyond Internal Data: Bounding and Estimating Fairness from Incomplete Data by Varsha Ramineni, Hossein A. Rahmani, Emine Yilmaz, David Barber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13040.txt.
Introduction
Varsha Ramineni and co-authors estimate AI fairness when the needed data can never be joined. Their 2025 paper starts from a common bind: firms hold the model data, while census files hold the group data, and privacy law keeps them apart. Auditors still must judge whether a model treats groups fairly.
The method builds every joint dataset that fits both sources. Fairness scores are then computed across that whole set, giving bounds rather than a single guess. Simulations and real tests showed the bounds track the true score well. The authors offer the approach to regulators and firms. It works with the partial data audits actually have under privacy law in industry settings.
Catalog record
| CREATOR | Varsha Ramineni, Hossein A. Rahmani, Emine Yilmaz, David Barber |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13040
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Internal Data: Bounding and Estimating Fairness from Incomplete Data Varsha Ramineni, Hossein A. Rahmani, Emine Yilmaz, David Barber Abstract Ensuring fairness in AI systems is critical, especially in high-stakes domains such as lending, hiring, and healthcare. This urgency is reflected in emerging global regulations that mandate fairness assessments and independent bias audits. However, procuring the necessary complete data for fairness testing remains a significant challenge. In industry settings, legal and privacy concerns restrict the collection of demographic data required to assess group disparities, and auditors face practical and cultural challenges in gaining access to data. In practice, data relevant for fairness testing is often split across separate sources: internal datasets held by institutions with predictive attributes, and external public datasets such as census data containing protected attributes, each providing only partial, marginal information. Our work seeks to leverage such available separate data to estimate model fairness when complete data is inaccessible. We propose utilising the available separate data to estimate a set of feasible joint distributions and then compute the set plausible fairness metrics. Through simulation and real experiments, we demonstrate that we can derive meaningful bounds on fairness metrics and obtain reliable estimates of the true metric. Our results demonstrate that this approach can serve as a practical and effective solution for fairness testing in real-world settings where access to complete data is restricted. 1 Introduction It is well established that Artificial Intelligence (AI) systems have the potential to perpetuate, amplify, and systemise harmful biases [ 6 , 7 ] . Therefore, rigorous testing for bias is imperative to mitigate harms, especially given the increasing influence of AI in high-stakes domains such as lending, hiring, and healthcare. Such concerns have fuelled active research on bias detection and mitigation [ 24 ] , and ensuring the fairness of AI systems has become an urgent policy priority for governments around the world [ 10 , 32 ] . For instance, the EU AI Act imposes strict safety testing on high-risk systems [ 14 ] , while New York City Local Law 144 mandates independent bias audits for AI used in employment decisions [ 17 ] . Despite this progress, procuring the necessary data for fairness testing remains a significant challenge. Influential works in ethics and fairness of machine learning have highlighted the centrality of datasets [ 20 , 3 ] , emphasising how representative model testing and evaluation data is crucial [ 5 , 28 ] . To effectively uncover biases, complete datasets that include demographic information and their relationship with model features are essential. However, having access to such datasets that can reliably be used for evaluating fairness may not always be possible in practice. As a motivating example, consider a bank that uses an AI system to assess loan applicants based on non-protected variables such as savings and occupation . The bank wants to perform an internal audit as to whether its AI system inadvertently discriminates against certain ethnic groups. Most common fairness metrics assess disparities in decision outcomes across protected attribute groups, such as ethnicity . Therefore, the bank requires protected attribute data of the applicants, alongside data of non-protected attributes required by the model to make a loan decision. Figure 1 : Illustration of our approach to bound and estimate fairness from incomplete data. The internal dataset (e.g., from a bank) contains non-protected attributes like savings and occupation, but lacks protected demographic attributes. A separate external public dataset includes protected attributes (such as ethnicity) and overlaps partially with the internal dataset (e.g., occupation attribute is common in both datasets). ( Top ) The first approach assumes a strong structural independence assumption to produce a single estimated joint distribution, and therefore a single fairness estimate. ( Bottom ) The second approach uses the marginal distributions from both sources to estimate the set of joint distributions consistent with the observed marginals. This allows fairness metrics to be computed over the space of feasible distributions, enabling bounding and estimation of fairness even in the absence of complete data. Although protected attributes such as ethnicity, sex, age etc. are crucial to assess bias, their collection and use in modelling are heavily restricted under regulations such as GDPR [ 1 , 31 ] . Hence, most internal datasets collected by organisations that use AI systems for decision making (such as the bank in our example) do not contain such protected attributes [ 23 ] . Similarly, procuring the necessary data is also a huge complexity for auditors, hindering the effective implementation of algorithmic auditing laws [ 17 ] . In an external audit of fairness, the auditing agency often has access only to the black box loan predictions and is not provided any data by the bank since existing regulations often do not allow data holders to release datasets that pause privacy concerns. For this external audit the agency needs a joint distribution of both the attributes needed by the black box loan classifier and the protected attributes. Therefore, the development of principled testing approaches capable of effectively uncovering biases in limited data scenarios is essential [ 23 ] . Our work addresses this challenge of estimating classifier fairness in scenarios where complete data including protected attributes is inaccessible. In practice, data relevant for fairness testing is often split across separate sources, such as internal datasets held by institutions and external public datasets each providing only partial, marginal information. We propose leveraging such separate data with overlapping variables, which are more accessible than complete datasets containing all variables [ 15 ] . Specifically, in addition to using an internal dataset that lacks protected attribute information, we propose utilising external public data, such as census datasets which provide representative demographic information. For example, the UK Office for National Statistics [ 25 ] offers multivariate data from the 2021 Census, providing access to customisable combinations of census variables. Similar resources are available from the US Census Bureau [ 29 ] . In our motivating example above, the internal dataset used by the bank only includes information about non-protected attributes \{\textit{savings},\;\textit{occupation}\} . Therefore to estimate fairness it is valuable to utilise an external public dataset such as \{\textit{occupation},\;\textit{ethnicity}\} that contains required protected attribute information and is representative of the population where the model will be deployed. As illustrated in Figure 1 , we propose an approach to leverage available separate incomplete data to estimate the set of all joint distributions consistent with the available marginal information. This enables us to compute fairness metrics over the full range of feasible distributions, allowing for estimation and bounding of fairness even when protected attributes are missing from the model’s input data. This is important to robustly estimate fairness metrics, as the true distribution underlying the data is often unknown or difficult to determine. We validate our approach through simulation and real experiments with realistic scenarios of separated data with limited overlap. Our results demonstrate that we can derive meaningful constraints on fairness metrics and obtain reliable estimates of the true metric, highlighting the effectiveness of our method in evaluating fairness when complete data is unavailable. These findings suggest that our approach offers a robust and practical solution for fairness evaluation in contexts where full datasets are inaccessible.