Social Group Bias in AI Finance

Thomas R. Cook, Sophia Kazinnik · 2025

Read Social Group Bias in AI Finance by Thomas R. Cook, Sophia Kazinnik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17490.txt.

Introduction

Thomas Cook and Sophia Kazinnik test racial bias in AI loan decisions in a 2025 paper. Banks now use large language models for high-stakes calls. Such models can carry social bias into those calls if no one checks. The team built a test where fake mortgage applicants differ only in race. Every other trait stays the same, so any gap in answers points to bias.

The gaps they found were large, past the levels seen in human lending history. They then traced how race signals spread through the model's inner layers as it worked. A control-vector tweak cut the racial gaps by up to 70 percent. On average the cut was 33 percent, with no loss in overall task skill.

Catalog record

CREATORThomas R. Cook, Sophia Kazinnik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17490

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Social Group Bias in AI Finance Thomas R. Cook, Sophia Kazinnik Abstract Financial institutions increasingly rely on large language models (LLMs) for high-stakes decision-making. However, these models risk perpetuating harmful biases if deployed without careful oversight. This paper investigates racial bias in LLMs specifically through the lens of credit decision-making tasks, operating on the premise that biases identified here are indicative of broader concerns across financial applications. We introduce a reproducible, counterfactual testing framework that evaluates how models respond to simulated mortgage applicants identical in all attributes except race. Our results reveal significant race-based discrepancies, exceeding historically observed bias levels. Leveraging layer-wise analysis, we track the propagation of sensitive attributes through internal model representations. Building on this, we deploy a control-vector intervention that effectively reduces racial disparities by up to 70% (33% on average) without impairing overall model performance. Our approach provides a transparent and practical toolkit for the identification and mitigation of bias in financial LLM deployments. 1 Introduction Financial institutions are rapidly deploying artificial intelligence tools into areas ranging from customer service chatbots to credit underwriting. 1 1 1 For example, Risk.net, 2024 reports that large banks are rapidly deploying large language models to automate core functions. The Bank for International Settlements (BIS) reports similar findings. Notably, credit underwriting tasks like credit scoring and collateral valuation are increasingly augmented by AI algorithms in recent years. See also, Elliot et al., 2024 This rapid adoption is driven by the remarkable capacity of these tools to improve worker productivity ( Bick et al., 2024 ; Brynjolfsson et al., 2025 , e.g.,) . 2 2 2 The McKinsey Global Institute estimates that generative AI could add $200–340 billion annually to the global banking sector, mainly through increased productivity. Yet, alongside these benefits, there is growing concern that AI may inadvertently introduce or amplify discriminatory bias into financial decision-making. We investigate this question in the context of consumer lending, the largest consumer finance market in the United States. Racial bias in mortgage lending has long been documented for its adverse economic and social consequences. Recent research shows that, even after controlling for observable risk factors, minority borrowers pay on average about 7–8 basis points more in interest, an excess cost that aggregates to roughly $765 million per year ( Popick, 2022 ; Hurtado and Sakong, 2024 ) . Our central research question is therefore whether LLM-driven credit decisions yield systematically different outcomes for equally qualified applicants solely on the basis of race. Our findings confirm these disparities and caution against trying to mitigate bias solely through adjustments to prompt instructions. Effective mitigation demands interventions within the model’s internal reasoning. To test this, we design a controlled experiment in which LLMs assess loan applicants with identical financial profiles but subtly different racial cues, allowing us to trace internal reasoning pathways and systematically evaluate targeted mitigation strategies. It is unsurprising that these models learn and reproduce systematic discrepancies in their responses to different social groups, given that the vast text corpora used to train LLMs contain historical social biases. 3 3 3 Barocas and Selbst, 2016 provide a broader discussion on biases in big data. The training data is likely to include text scraped from various social media websites, where expressions of social-group prejudice are often found ( Matamoros-Fernández and Farkas, 2021 ) . Aware of these biases, LLM developers attempt to mitigate them by filtering offensive content from training data. However, discerning which data actually contains social-group prejudice can be difficult. 4 4 4 It can, for instance, be difficult to determine whether a social media post espouses social-group prejudice or the lived experience of a member of a social group. See Tomasev et al., 2024 . More generally, LLM developers make an effort to align these models to follow instructions and reduce undesirable social bias, using methods like reinforcement learning with human feedback (RLHF) and direct preference optimization ( Christiano et al., 2017 ; Ouyang et al., 2022 ; Rafailov et al., 2024 ) . These efforts, however, are imperfect, and undesirable social biases often persist in LLMs ( Abid et al., 2021 ; Wan et al., 2023 ; Taubenfeld et al., 2024 ) . And even though most LLMs are trained to avoid biases, they can still demonstrate biased responses in certain special cases or particular scenarios. 5 5 5 This tendency of models to produce unaligned responses in response to particular prompts is the basis for research on “jailbreaking” LLMs ( Wei et al., 2024 ; Nasr et al., 2023 ) . In other words, it is entirely possible that social biases not expressed by an LLM in general settings may nevertheless persist and manifest when the LLM is deployed in a financial context. In contrast to studies that use proprietary AI models, we deploy local, open-source LLMs whose parameters are fully accessible for analysis and intervention. 6 6 6 A local, open-source LLM is a language model whose code and trained parameters are freely published, giving one full access to inspect, modify, and run experiments in a deterministic way. This design enables us to run deterministic experiments, while ensuring that our results are reproducible and not subject to the hidden updates or stochastic outputs of a remote AI service. 7 7 7 Additionally, we note that using local large language models ensures our study mirrors how lenders are likely to adopt LLM technology in practice. Banks are likely to want to running AI models in-house (i.e., locally) due to concerns about data privacy and compliance requirements. Institutions will likely look to open-source LLMs for this purpose. Because these models are open-source, we can also examine the model activations – the internal mechanism of the LLM’s “black box.” By inspecting the model’s internal representations across its neural network layers, we can observe how the information about protected characteristics is encoded during the decision process. Specifically, we measure the concept intensity scores of model outputs on sensitive features at each layer, providing a detailed view of the model’s decision-making logic. 8 8 8 This process is similar to tracing a product through each stage of a supply chain to determine exactly where defects (here, biases) originate and how they’re amplified downstream. A concept intensity score is a single number that tells you how strongly a particular idea or feature is represented in a model’s internal activations at a given layer. Building on this analysis, we propose a mitigation technique grounded in representation engineering ( Zou et al., 2023 ) . We construct ‘‘control vectors’’ that can be injected into the model’s hidden states to steer the model away from bias. 9 9 9 Extending the supply chain analogy, our mitigation technique is similar to introducing quality control checkpoints at key stages of the production process. By injecting control vectors, we intervene within the model’s internal stages, much like correcting defects early in production before they reach later stages. In simpler terms, a control vector nudges the model’s internal representation in a direction that counteracts the influence of a sensitive attribute, reducing biased outcomes at the source (i.e., within the model’s cognition rather than solely in its final output). Our findings confirm that LLMs can indeed exhibit significant bias in credit decisions.

Read the complete text