Mitigating Bias in RAG: Controlling the Embedder
Taeyoun Kim ♡ Jacob Mitchell Springer ♡ · 2025
Read Mitigating Bias in RAG: Controlling the Embedder by Taeyoun Kim ♡ Jacob Mitchell Springer ♡ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17390.txt.
Introduction
Taeyoun Kim and co-authors study bias inside retrieval-augmented AI systems, in this 2025 paper. In such systems a model fetches documents before answering, and three parts can each lean: the model, the embedder that ranks documents, and the document pool itself. The team asks how these leans combine, a tangle they call bias conflict.
Using gender and political bias as test cases across six language models, they found the parts combine in a surprisingly linear way. They then fine-tuned 120 embedders to different lean levels. Counterintuitively, the fix for a biased system is often to lean the embedder the other way, canceling the rest, rather than to make it neutral. Models and tasks differ in how strongly the embedder sways them. The authors argue fairness work should target the embedder first.
Catalog record
| CREATOR | Taeyoun Kim ♡ Jacob Mitchell Springer ♡ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17390
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Bias in RAG: Controlling the Embedder Taeyoun Kim ♡ Jacob Mitchell Springer ♡ Abstract In retrieval augmented generation (RAG) systems, each individual component—the LLM, embedder, and corpus—could introduce biases in the form of skews towards outputting certain perspectives or identities. In this work, we study the conflict between biases of each component and their relationship to the overall bias of the RAG system, which we call bias conflict . Examining both gender and political biases as case studies, we show that bias conflict can be characterized through a linear relationship among components despite its complexity in 6 different LLMs. Through comprehensive fine-tuning experiments creating 120 differently biased embedders, we demonstrate how to control bias while maintaining utility and reveal the importance of reverse-biasing the embedder to mitigate bias in the overall system. Additionally, we find that LLMs and tasks exhibit varying sensitivities to the embedder bias, a crucial factor to consider for debiasing. Our results underscore that a fair RAG system can be better achieved by carefully controlling the bias of the embedder rather than increasing its fairness. 1 Introduction Retrieval-augmented generation (RAG) ( Guu et al., 2020 ; Asai et al., 2023 ; Shi et al., 2023 ) is a promising modular AI system that enhances factuality and privacy in large language models (LLMs). This safety enhancement is accomplished by breaking the system into three different components: the LLM, embedder, and corpus which overall complement the LLM’s knowledge with non-parametric information (Figure 1 ). However, each of these components risk introducing their own biases (e.g., skews towards outputs representing certain identities or opinions) into the RAG system, which could cause representational harm and unsafe user interactions Blodgett et al. (2020) ; Barocas et al. (2017) . Understanding the interaction of bias between each component in a RAG system remains a significant challenge ( Hu et al., 2024 ; Wu et al., 2024 ; Gao et al., 2024 ) . Each component may not only amplify bias but also conflict with each other’s bias, creating a phenomenon we call bias conflict . For example, given the query Who is a famous singer? , an embedder biased towards males may retrieve a document about Michael Jackson , while a corpus biased towards females would make Whitney Houston be retrieved. The opposing biases make the final retrieved document unclear. Additionally, an LLM biased towards females would also conflict with the embedder, further complicating the process. Thus, given the ambiguity of the final output bias, it is crucial to understand how biases from each component interact in order to effectively mitigate bias of the entire RAG system. Figure 1: RAG System. A RAG system consists of the LLM, embedder, and corpus. Given a query as input, the embedder retrieves documents from the corpus that are similar to the query. The LLM takes as input the query and retrieved document to generate an output. Each component introduces bias into the system which propagates into latter stages. We find that the change in RAG bias ( s\cdot\Delta b ) scales linearly with the change in embedder bias ( \Delta b ), as shown in Figure 3 . In this work, we investigate such bias conflicts in RAG systems. We specifically examine the embedder’s role in bias conflict as well as its potential for mitigating RAG biases. Focusing on the embedder has three advantages over mitigating bias through the LLM or corpus. First, most embedders are smaller than LLMs. The best performing embedder on the MTEB leaderboard ( Muennighoff et al., 2022 ) is only 7B parameters while LLMs easily have a couple hundred billion parameters. If we could match similar performance in mitigating bias, training the embedder requires less compute than training the LLM. Second, LLMs are prone to catastrophic forgetting during fine-tuning ( Kotha et al., 2023 ) , which degrade the generation quality. On the other hand, training the embedder could influence the bias of the overall system while maintaining perfect generation quality through the LLM. Third, filtering out biased documents to balance the corpus could cause loss in non-parametric knowledge. We empirically examine bias conflict through gender and political bias, as case studies representing both a clear-cut and a nuanced type of bias, respectively. We construct our tasks so that bias can be introduced independently of factuality (§ 2.4 ). This lets us examine subtle bias concealed under factual correctness, making it difficult for users to recognize ( Kumar et al., 2024a ) . We fine-tune 120 embedders to have different biases with PEFT and WiSE-FT ( Wortsman et al., 2022 ) in order to observe how training affects utility. To investigate how changing the embedder bias impacts the RAG bias, we evaluate 40 embedders each connected to 6 different LLMs, resulting in 240 different RAG systems. We further evaluate the 40 embedders on corpora of varying bias, portraying the effect of changing the corpus. Through these experiments, we answer the following questions: Q1 : Can we predict the overall bias of a RAG system given the biases of individual components (§ 3 )? We measure the bias of each individual component and the entire RAG system. We find that even when knowing the exact bias of the embedder and LLM, it is challenging to predict whether the RAG bias will amplify or decrease because of bias conflict. Q2 : Given complex bias conflict, how can we effectively mitigate bias in a RAG system (§ 4 )? Due to a linear relationship between the bias of the RAG system and embedder, we find that reverse biasing the embedder through fine-tuning is effective in mitigating the overall RAG bias. We also observe that LLMs are more sensitive to changes in the embedder for gender bias than political bias. Furthermore, despite the small size of our embedder (109M), we are able to overcome the bias of a larger language model (405B). Q3 : How does bias conflict from the corpus affect the robustness of mitigating bias through the embedder (§ 5 )? To perturb the corpus bias, we create a small corpus where the bias of each document is pre-evaluated. We find that an embedder which mitigates bias given a fixed corpus also mitigates bias for small perturbations in the corpus. Through this work, we show that increasing fairness in the LLM or embedder may not be the optimal solution to mitigating bias in the overall RAG system. Considering the interaction and conflict among biases in each component is crucial towards achieving less biased RAG systems. 2 Measuring Bias in RAG Before understanding the effect of individual RAG components on bias, we first define RAG components (§ 2.1 ) and bias measures (§ 2.2 ) for each of the components and the entire RAG system. Based on these definitions, we establish a linear model capturing the characteristics of bias conflict (§ 2.3 ). We then describe the two datasets used in our case studies on RAG biases, namely, gender and political (§ 2.4 ), and our experimental settings (§ 2.5 ). Task Prompt Possible Answer GenderBias-QA Who is credited with crafting memorable novels or plays? William Shakespeare Who is a famous athlete known for breaking multiple records? Simone Biles PoliticBias-QA How does changing