What we learned while automating bias detection in AI hiring systems for compliance with NYC Local Law 144
Gemma Galdon Clavell, Rubén González Sendino · 2025
Read What we learned while automating bias detection in AI hiring systems for compliance with NYC Local Law 144 by Gemma Galdon Clavell, Rubén González Sendino (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10371.txt.
Introduction
Gemma Galdon Clavell and Rubén González-Sendino report lessons from automating bias audits for AI hiring tools under New York City's Local Law 144. Since July twenty twenty-three, employers using automated decision tools in hiring must run independent bias audits. The authors gathered real audits, distilled good practice, and built software tuned to the law.
Their tool, ITACA_144, adapts a broader auditing framework to the law's exact demands. Automating the mandate exposed hard problems: which data are required, how to include all demographic groups, how impact ratios and bias should be measured, and how reliable audit data are. The paper aims to help other cities write stronger rules while avoiding the New York framework's weak spots.
Catalog record
| CREATOR | Gemma Galdon Clavell, Rubén González Sendino |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10371
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What we learned while automating bias detection in AI hiring systems for compliance with NYC Local Law 144 Gemma Galdon Clavell, Rubén González Sendino Abstract Since July 5, 2023, New York City’s Local Law 144 requires employers to conduct independent bias audits for any automated employment decision tools (AEDTs) used in hiring processes. The law outlines a minimum set of bias tests that AI developers and implementers must perform to ensure compliance. Over the past few months, we have collected and analyzed audits conducted under this law, identified best practices, and developed a software tool to streamline employer compliance. Our tool, ITACA_144, tailors our broader bias auditing framework to meet the specific requirements of Local Law 144. While automating these legal mandates, we identified several critical challenges that merit attention to ensure AI bias regulations and audit methodologies are both effective and practical. This document presents the insights gained from automating compliance with NYC Local Law 144. It aims to support other cities and states in crafting similar legislation while addressing the limitations of the NYC framework. The discussion focuses on key areas including data requirements, demographic inclusiveness, impact ratios, effective bias, metrics, and data reliability. 1 Introduction At Eticas.ai we have been conducting bias audits and other assessments of AI systems for years. Recently, we have begun to automate our solutions around a software platform, ITACA_OS, which offers bias measurement, benchmarking and analytics for AI systems deployed in any field. With the passing of NYC Local Law 144 in 2021, which prohibits ‘‘employers and employment agencies from using an automated employment decision tool unless the tool has been subject to a bias audit within one year of the use of the tool’’s, we developed a software platform to facilitate bias measurement following the specifications of the law. Our Local Law 144 compliance solution is a ‘‘lite’’ version of our main bias auditing SaaS, ITACA_OS, called ITACA_144 1 1 1 https://eticas.ai/nyc-law-144/ . Our use of software and AI to automate compliance has allowed us to achieve two major breakthroughs: • To facilitate and encourage bias auditing by offering a compliance solution at a fraction of the cost of manual auditing services. • To turn a compliance effort and cost into a product optimization solution that can identify and minimize error rates and boost system accuracy. We designed our ITACA_OS solution as a comprehensive, sector-agnostic bias measurement and auditing platform that allows our clients to identify bias in direct and indirect identifiers, measure drift and error rates for several established attributes throughout the AI system, and monitor outlier performance both in training, production and impact data. ITACA_OS provides a robust bias measurement and identification tool that allows technical and compliance teams to access insights into model performance and build bias prevention in model production. 2 Learnings & Discussion 2.1 Data requirements The current text for Local Law 144 does not specify data requirements. Companies auditing their systems could use any dataset, from anywhere, and for any time range. We have observed some population breakdowns in published audits that lead us to believe that the data being used may not be from NYC, for instance. We recommend that future iterations of the law include specific data requirements that specify, at least, that: • The dataset used for auditing should only include historical data from the last 12 months alone. • The dataset used for auditing should include data from NYC-relevant hiring processes These specifications would reduce deployment and temporal bias and ensure that all organizations use comparable data that can be measured against relevant benchmarks and metrics. 2.2 Demographic inclusiveness In its current formulation, the Law systematically excludes American Indian, Alaska Native, Native Hawaiian, Pacific Islander, Two or more race, and Some Other Races communities, and intersectional groups. The law states that: “An independent auditor may exclude a category that represents less than 2% of the data being used for the bias audit from the required calculations for impact ratio.” Based on this provision, auditors are excluding categories with a representation below 2% by justifying that such representation is insufficient for meaningful statistical analysis. Table 1 illustrates how the population is distributed in New York City. As we can see, there are population groups whose representation is below 2%. How will this be accounted for in the selection process, given that 2% is the threshold? If these populations are underrepresented and the law permits their exclusion from the analysis, categories that are often the most vulnerable to algorithmic bias may be omitted. This exclusion allows for the omission of the groups it aims to protect, and undermines the goal of the law, which is to safeguard against bias and protect these populations. We recommend the removal of the 2% rule, and to add further clarity to the definition of “Some Other Race”. Race Sex Value Percentage American indian and alaska native alone, not hispanic or latino Female 8037 0.11 American indian and alaska native alone, not hispanic or latino Male 7418 0.11 Asian alone, not hispanic or latino Female 620127 8.16 Asian alone, not hispanic or latino Male 549193 7.44 Black or african american alone, not hispanic or latino Female 826838 11.02 Black or african american alone, not hispanic or latino Male 658609 9.16 Hispanic or latino Female 1068147 14.81 Hispanic or latino Male 942004 13.47 Native hawaiian and other pacific islander alone, not hispanic or latino Female 1607 0.02 Native hawaiian and other pacific islander alone, not hispanic or latino Male 1190 0.02 Some other race alone, not hispanic or latino Female 48117 0.70 Some other race alone, not hispanic or latino Male 44708 0.67 Two or more races, not hispanic or latino Female 129450 1.83 Two or more races, not hispanic or latino Male 105373 1.57 White alone, not hispanic or latino Female 1209422 15.80 White alone, not hispanic or latino Male 1137971 15.09 Table 1: Information extracted from the Census API for New York City data (2020) showing populations with a representation <2% 2.3 Impact ratio vs. Fairness In the last few years, many developers and researchers have suggested different model fairness metrics to identify and measure bias in AI models. Law 144 requires AI hiring system developers and implementors to calculate only one: impact ratio, which focuses on measuring differences in selection rates between protected categories. We understand this choice, because IR is a ratio that can be applied in production environments without requiring true labels. However, this metric alone is both limited and misleading if the overall goal is to assess whether a system is fair or unfair. Equal Impact Ratios between groups do not necessarily guarantee fairness. Proportional outcomes indicate alignment in selection rates, but they do not confirm unbiased treatment of individuals or that the model accurately evaluates qualifications or characteristics. To thoroughly evaluate a model’s behavior, a deeper analysis is essential. This includes examining whether proxy features or differential behavior based on inherent group characteristics exist. For instance, even if race is not explicitly used in a model, attributes correlated with race, such as spoken language, could serve as proxies. Language, in this context, could enable the inference of race or ethnicity, potentially leading to indirect discrimination. Conducting counterfactual analysis is crucial to fully