PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

Sumin Yu, Juhyeon Park, Taesup Moon · 2026

Read PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset by Sumin Yu, Juhyeon Park, Taesup Moon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22714.txt.

Introduction

Yu, Park, and Moon present PopResume, a dataset for fair auditing of AI resume screeners. The dataset holds 60.8 thousand resumes across five occupations. It is built from population statistics, so attribute links stay natural. The authors split a protected trait's effect into two paths. One path runs through job-relevant skills, the other through demographic proxies. Testing four language models and four vision-language models reveals five bias patterns. Standard summary metrics miss these patterns, the authors report.

Catalog record

CREATORSumin Yu, Juhyeon Park, Taesup Moon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77193
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22714

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset Sumin Yu, Juhyeon Park, Taesup Moon Abstract We present PopResume, a population-representative resume dataset for causal fairness auditing of LLM- and VLM-based resume screening systems. Unlike existing benchmarks that rely on manually injected demographic information and outcome-level disparities, PopResume is grounded in population statistics and preserves natural attribute relationships, enabling path-specific effect (PSE)-based fairness evaluation. We decompose the effect of a protected attribute on resume scores into two paths: the business necessity path, mediated by job-relevant qualifications, and the redlining path, mediated by demographic proxies. This distinction allows auditors to separate legally permissible from impermissible sources of disparity. Evaluating four LLMs and four VLMs on PopResume’s 60.8K resumes across five occupations, we identify five representative discrimination patterns that aggregate metrics fail to capture. Our results demonstrate that PSE-based evaluation reveals fairness issues masked by outcome-level measures, underscoring the need for causally-grounded auditing frameworks in AI-assisted hiring. 1 Introduction The use of large language models (LLMs) in resume screening and hiring decisions is expanding rapidly 18 ; 9 , raising growing concerns regarding fairness and discrimination 6 ; 33 ; 34 . Given the potential for automated decisions to produce systematic disparities that disadvantage individuals based on protected attributes such as gender or race, organizations deploying automated hiring systems are now expected to demonstrate compliance with anti-discrimination law 30 ; 31 ; 22 . Figure 1: Prior works inject protected attributes, making causal framework–based evaluation infeasible, and measure outcome disparity; our population-representative resume dataset instead enables causal effect–based evaluation and pathway decomposition. In actual audit and compliance environments, group-level outcome disparities are often identified using simple statistical criteria such as selection rate comparisons 31 . While these metrics serve as useful early indicators of potential risk, they do not explain the mechanisms behind the disparities . This limitation is critical, as legal liability ultimately depends not only on the presence of disparity, but also on how it arises. Recent studies investigating bias in LLM-based hiring systems 14 ; 12 ; 32 share the same limitation: they primarily focus on statistical disparities between demographic groups, without identifying the underlying causal pathways through which protected attributes influence hiring outcomes, as illustrated in Figure 1 (left). However, legal and policy frameworks around employment discrimination hinge on a crucial distinction: disparities arising through legitimate job-related pathways, e.g., role-essential qualifications, may be permissible ( business necessity ), whereas disparities driven by proxy variables such as name or address that can disadvantage specific protected groups are unacceptable ( redlining ). Therefore, meaningful fairness auditing requires understanding not just the magnitude of outcome disparities, but the causal pathways through which protected attributes produce them. Path-specific effect (PSE)-based fairness frameworks 21 ; 20 ; 7 offer a principled approach to this challenge, enabling answering how and through which pathways protected attributes contribute to observed disparities. However, enabling such pathway-level analysis requires data with explicit control over all relevant attributes, such as protected attributes, legitimate qualifications, and proxy variables, simultaneously. Existing benchmarks for LLM-based hiring systems fail to meet this requirement in two distinct ways. Real-world resume datasets 11 ; 23 typically do not contain ground-truth demographic information about applicants, as such attributes are withheld due to privacy concerns. Current evaluation benchmarks 12 ; 32 ; 14 ; 1 ; 33 ; 19 , on the other hand, often manually inject and perturb demographic information, which can disrupt the realistic relationships between attributes, making causal framework-based evaluation infeasible. We address these gaps by introducing legally-grounded causal fairness auditing framework for LLM- and vision language model (VLM)-based resume screening systems. Building on the PSE framework, we decompose the causal effect of a protected attribute on resume scores into two pathways: (1) the business necessity path, mediated by job-relevant qualifications (education, work experience), and (2) the redlining path, mediated by demographic proxies (name, address). This decomposition enables auditors to identify not only that a hiring system discriminates, but how and whether the discrimination is legally defensible. To support this analysis under realistic conditions, we construct PopResume , a dataset derived from U.S. population-level statistics ( e.g., ACS PUMS 29 ) that preserves the natural relationships among attributes and enables causal effect estimation, which contains 60.8k resumes across five occupations. Our contributions are as follows. • We propose a PSE-based fairness auditing framework for LLM/VLM-based resume screening systems that separates legally permissible outcome disparities mediated by job-relevant qualifications from legally impermissible ones transmitted through demographic proxies. • We construct a large-scale resume dataset grounded in U.S. population statistics, preserving realistic distribution and enabling causal effect-based fairness evaluation. 2 Preliminaries First, we review the legal foundations of employment discrimination law that motivate our fairness criteria ( Section 2.1 ), and then introduce the causal modeling framework and PSEs that underpin our analysis ( Section 2.2 ). 2.1 Legal Background Title VII of the US Civil Rights Act 30 prohibits employment discrimination based on race, color, religion, sex, and national origin, which are referred to as protected attributes . The legal theory typically applied to automated decisions is the disparate impact doctrine 16 ; 3 , which holds that even a seemingly neutral practice can be unlawful if it produces outcomes that disproportionately disadvantage a protected group. To establish disparate impact, plaintiffs must first demonstrate a disparity in outcomes between groups, commonly assessed using statistical heuristics such as the Four-Fifths rule. This step identifies the presence of a disparity but does not determine its causal origin. Critically, however, disparate impact analysis does not end with identifying statistical disparities. Under the burden-shifting framework established by Title VII, employers may justify the observed disparity by demonstrating that it arises from job-related criteria consistent with business necessity , recognizing that disparities mediated through legitimate, job-relevant factors, such as professional qualifications or required skills, are permissible. Accordingly, legal liability hinges on a fundamentally causal question: whether the observed disparity arises through permissible pathways ( business necessity ) or impermissible ones ( redlining ). This motivates the PSE-based analysis that can distinguish between these two types of effects. (a) Standard Fairness Model (SFM). (b) Mediator decomposition in our framework. Figure 2: In our framework, the mediator set \mathbf{W} is decomposed into business necessity components \mathbf{B} and redlining-related components \mathbf{R} . Blue edges denote BIE pathways, while red edges denote RIE pathways. 2.2 Standard Fairness Model & Causal Effects We use notations (\mathbf{X},X,\mathbf{x},x) to denote random vectors, random variables, and corresponding realizations, respectively. Standard Fairness Model (SFM). To formalize the causal relationships between protected attributes and hiring outcomes, we build on the structural causal model (SCM) framework 20 . An SCM induces a causal graph \mathcal{G} , where nodes correspond to variables \mathbf{V} , directed edges represent causal relationships, and dashed bi-directed edges indicate hidden confounders. Following 21 , we adopt the standard fairness model (SFM) with \mathbf{V}=\{X,\mathbf{Z},\mathbf{W},Y\} . SFM partitions variables into four groups: a protected attribute X , an outcome Y , confounders \mathbf{Z} that are not causally affected by X , and mediators \mathbf{W} that may be causally influenced by X , as depicted in Figure 2(a) . This abstraction allows fairness analysis without requiring the complete causal graph to be specified, while preserving the causal pathways

Read the complete text