On the Effectiveness and Generalization of Race Representations for Debiasing High-Stakes Decisions
Dang Nguyen, Chenhao Tan · 2025
Read On the Effectiveness and Generalization of Race Representations for Debiasing High-Stakes Decisions by Dang Nguyen, Chenhao Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06303.txt.
Introduction
Dang Nguyen and Chenhao Tan test racial bias in language models used for admissions and hiring calls. Their paper appeared in two thousand twenty-five. They built simple tasks with fake applicant profiles. Race could be inferred from names alone. Two small instruction-tuned models showed strong skews. One granted admission to far more white than black applicants. The other hired far more Asian than white applicants.
Prompt tricks failed to fix the skew. Every prompting strategy they tried fell short. The team instead found race-related directions inside the models' activations. Averaging those signals across races cut the bias sharply. They then check whether the fix transfers to new settings. The work warns that surface fixes are weak for high-stakes tools.
Catalog record
| CREATOR | Dang Nguyen, Chenhao Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06303
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Effectiveness and Generalization of Race Representations for Debiasing High-Stakes Decisions Dang Nguyen, Chenhao Tan Abstract Understanding and mitigating biases is critical for the adoption of large language models (LLMs) in high-stakes decision-making. We introduce Admissions and Hiring , decision tasks with hypothetical applicant profiles where a person’s race can be inferred from their name, as simplified test beds for racial bias. We show that Gemma 2B Instruct and LLaMA 3.2 3B Instruct exhibit strong biases. Gemma grants admission to 26% more White than Black applicants, and LLaMA hires 60% more Asian than White applicants. We demonstrate that these biases are resistant to prompt engineering: multiple prompting strategies all fail to promote fairness. In contrast, using distributed alignment search, we can identify “race subspaces” within model activations and intervene on them to debias model decisions. Averaging the representation across all races within the subspaces reduces Gemma’s bias by 37-57%. Finally, we examine the generalizability of Gemma’s race subspaces, and find limited evidence for generalization, where changing the prompt format can affect the race representation. Our work suggests mechanistic approaches may provide a promising venue for improving the fairness of LLMs, but a universal race representation remains elusive. 1 Introduction While it is well-recognized that LLMs may exhibit racial biases in high-stakes decisions ( Tamkin et al., 2023 ) , it remains an open question how we can effectively mitigate such biases. In this work, we explore two possible approaches: 1) prompt engineering, which treats the model as a black box and leverages its ability to follow instructions; 2) representation-based debiasing by identifying how the model encodes biases internally. To that end, we first introduce synthetic tasks and datasets, Admissions and Hiring , in which models are given hypothetical applicant profiles and are asked whether to accept or reject them. We assess models’ biases by giving them applicant names that are highly suggestive of their race, and measure the disparity in outcomes across races using our novel fairness metric, BiasScore . Working with Gemma 2B Instruct and LLaMA 3.2 3B Instruct, we find that models exhibit racial biases in Admissions and Hiring . In both decision tasks, Gemma favors White and Asian applicants over Black and Latino applicants, and LLaMA favors Asian and Latino applicants over Black and White ones. Figure 1 : We consider two decision tasks: Admissions and Hiring , and examine two approaches to control model behavior. In prompt engineering, we attempt to debias by adding various instructions. To modify model internals, (1) we learn a “race subspace” to intervene so that the target decision becomes the counterfactual decision (i.e., as if the inferred race is the same as that from the source text). (2) We debias model decisions by averaging the representation in the race subspace across a batch of samples, which removes the variance in race representations between applicants. Our first approach to debiasing model decisions builds on the model’s ability to follow instructions, which has proven effective in many cases ( Brown et al., 2020 ; Wei et al., 2022 ; Tamkin et al., 2023 ; Tseng et al., 2024 ) . However, we find that different prompting strategies to promote fairness end up exacerbating the biases or even completely derailing model generation. For example, asking Gemma to not discriminate based on sensitive attributes in Hiring can increase its BiasScore by 137%, and doing so for LLaMA can cause it to indiscriminately accept all applicants. Thus, prompt engineering can unpredictably alter model behavior in high-stakes decision-making, rendering it unreliable. Seeking a more principled and effective approach, we attempt to pinpoint the mechanism causing biased decisions in LLMs and modify it to mitigate biases. We use distributed alignment search ( Geiger et al., 2024 ; Wu et al., 2024b ) to identify a subspace in models’ hidden representations dedicated to encoding race, a “race subspace”, with 74-85% interchange intervention accuracy (see exact definitions in Geiger et al. (2024) and Section 5 ) indicating strong race representation. Hence, models implicitly keep track of the applicant’s race despite it not being given in their profile. We debias model decisions by averaging the values of all races in the subspace, thereby erasing the variance between different races, or by projecting out the race subspace from the model’s hidden representation, which “deletes” race information altogether. Compared to prompt engineering, representation-based debiasing is much more effective. Race averaging reduces Gemma’s bias by 37-57% and LLaMA’s bias by 32% in Admissions . This suggests that mechanistic approaches are promising for mitigating biases in LLM decisions. Finally, we investigate whether LLMs dedicate the same subspace for race across different decision tasks. We find mixed results, as the race subspace in Admissions can be used to debias Hiring , but changing the prompt template in Admissions or explicitly providing race can cause our debiasing approaches to fail. The generalization results are even weaker in the case of LLaMA. These findings suggest that LLMs may represent race differently even with slight task modifications, which calls for more research into understanding why this happens and whether universal debiasing methods are possible. In summary, our main contributions include: • We introduce Admissions and Hiring as simplified tasks and datasets for evaluating LLMs’ implicit racial biases. • We demonstrate the ineffectiveness of prompt engineering in debiasing. • We find “race subspaces” using distributed alignment search, and successfully debias models via representation-based interventions. • We discover limited generalizability in the race subspaces, indicating a challenge to the generalizability of LLM bias mitigation research. We release code and data at https://github.com/ChicagoHAI/llm-prediction-bias . 2 Related Work Our work sits at the intersection of several active research areas: fairness and bias in NLP, prompt engineering, and mechanistic interpretability. Fairness and Bias in NLP. The increasing deployment of LLMs in real-world applications comes with concerns about fairness and bias ( Tamkin et al., 2023 ) , including gender ( Bolukbasi et al., 2016 ; Zhao et al., 2018 ; Rudinger et al., 2018 ; Sheng et al., 2019 ) , race ( Shaikh et al., 2022 ; An et al., 2024 ) , and other sensitive attributes. These biases can manifest in different ways, from stereotyping associations to discriminatory decisions in downstream tasks. Prior works have explored various techniques to measure and mitigate them, including data augmentation, bias-aware training objectives, and post-training interventions ( Gallegos et al., 2024 ) . Our work builds upon prior work on debiasing semantic representations ( Bolukbasi et al., 2016 ; Kurita et al., 2019 ) , with a focus on LLMs. Prompt Engineering. Prompt engineering has emerged as a popular approach for steering LLM behavior without requiring model retraining. By carefully crafting input prompts, users can influence the model’s outputs and improve performance on various tasks ( Lu et al., 2021 ; Wei et al., 2022 ; Min et al., 2022 ) . Notably, Wu et al. (2025) found that prompting outperforms representation-based model steering techniques on the AxBench benchmark, further demonstrating its competitiveness as a baseline. Our work examines the efficacy of prompt engineering in the context of high-stakes decision making and demonstrates its limitation for debiasing compared to interventions on internal representations. Mechanistic Interpretability. A growing body of research is dedicated to understanding the