Towards Resource Efficient and Interpretable Bias Mitigation in Large Language Models

Schrasing Tong, Eliott Zemour*, Rawisara Lohanimit, Lalana Kagal · 2024

Read Towards Resource Efficient and Interpretable Bias Mitigation in Large Language Models by Schrasing Tong, Eliott Zemour*, Rawisara Lohanimit, Lalana Kagal (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01711.txt.

Introduction

Large language models can repeat the biases in their training data. Such bias may harm groups already pushed to the margins. Schrasing Tong and co-authors offer a cheap fix in a 2024 paper: two small expert models, one biased and one anti-biased, produce a correction signal. That signal is added to the big model's output as it writes, and only the small models need training, not the large one.

Because the shift is a visible change in word odds, users can inspect exactly how debiasing acts. Tests on gender, race, and religion bias showed clear reductions on several measures while writing quality held steady. The method can be retargeted by swapping training data.

Catalog record

CREATORSchrasing Tong, Eliott Zemour*, Rawisara Lohanimit, Lalana Kagal
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01711

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Resource Efficient and Interpretable Bias Mitigation in Large Language Models Schrasing Tong, Eliott Zemour*, Rawisara Lohanimit, Lalana Kagal Abstract Although large language models (LLMs) have demonstrated their effectiveness in a wide range of applications, they have also been observed to perpetuate unwanted biases present in the training data, potentially leading to harm for marginalized communities. In this paper, we mitigate bias by leveraging small biased and anti-biased expert models to obtain a debiasing signal that will be added to the LLM output at decoding-time. This approach combines resource efficiency with interpretability and can be optimized for mitigating specific types of bias, depending on the target use case. Experiments on mitigating gender, race, and religion biases show a reduction in bias on several local and global bias metrics while preserving language model performance. 1 Introduction Natural language generation (NLG) has risen in popularity greatly, serving as building blocks for applications such as chatbots, translators, and writing assistants and interacting with users in different domains [ 14 ] . One such example is OpenAI’s update in March 2021, which stated that GPT-3 powered applications generate an average of 4.5 billion words per day. However, despite recent advancements, these large language models (LLMs) have been reported to capture and reproduce unwanted biases and stereotypes [ 1 , 21 ] . This occurs mainly because the large text corpora required for training such models are extracted from the Internet, which is not an accurate reflection of the diversity of real-world distributions. Generating biased outputs can result in serious negative consequences to society, ranging from offensive language that prevent certain demographic groups from adopting the technology [ 30 ] to biased job advertisements that discourage candidates from applying to certain positions [ 6 ] . To address these issues, researchers have tried to curate better training data and improve the training process [ 10 , 19 ] . Nevertheless, this approach remains less feasible in practice due to the significant human and computation resource involved. Recent works have instead focused on reducing the bias of generated outputs at decoding time to improve resource efficiency. For example, [ 30 ] introduced a prompt engineering method, Trigger, that concatenates a sequence of tokens to user inputs to reduce the output’s bias. However, the modified prompts suffer from a lack of interpretability and has been shown to spew racist output on non-racial contexts [ 35 ] . In this paper, we adapt the framework from [ 18 ] ’s method of using small models as experts and anti-experts for detoxification. The biased and anti-biased experts in our system are small language models (LMs), for instance pre-trained GPT-2 Small models, fine-tuned on subsets of the RedditBias dataset [ 4 ] . They produce a debiasing signal that we incorporate into the LLM output at decoding-time. This approach combines resource efficiency - the experts require only several hundred sentences for fine-tuning, and interpretability - one can examine the shift in output probabilities for any given prompt. Furthermore, depending on the anticipated use case, our framework can be modified to address any type of bias expected. We evaluated the performance of the framework on three different bias directions, gender, race, and religion, and observed a reduction in bias on several local and global bias metrics. Since the experts rely on small biased datasets for fine-tuning, we substituted the fine-tuning dataset with StereoSet [ 23 ] and found that the results remain robust to the dataset choice; Stereotype Score was the only exception due to the data dependency of the bias metric. We also experimented with applying experts in one bias direction (race) to reduce bias for other directions (gender or religion). Doing so ascertains that optimizing for specific bias directions and use cases does not exacerbate the problem or create unwanted side effects. Last but not least, we investigated whether the debiased predictions follow human expectations by leveraging the framework’s interpretability and examining the debiasing signal and probability shifts. Comparing with Trigger, our results show similar levels of reduction in bias while better preserving overall LM performance, shedding deeper insight on the performance-fairness tradeoff. Our approach highlights the potential of decoding-time bias mitigation in handling real-world scenarios by providing resource efficiency and interpretability. 2 Related Work Recent research has shown that bias exists in many important natural language processing models, including word embeddings [ 5 , 7 , 13 ] , question answering [ 25 ] , and sentiment analysis [ 16 ] . NLG also suffers from similar problems as large language models are trained using text from the Internet [ 27 ] , which likely contain unwanted stereotypes and skewed representations of true distributions. Bias in NLG negatively affect society in many ways, ranging from propagating and amplifying bias [ 1 , 21 ] to discouraging certain groups from adopting the technology [ 30 ] or even increasing their vulnerability to harm and discrimination [ 32 , 22 ] . However, despite recent efforts in measuring and creating benchmarks for bias in LLMs [ 15 , 12 ] , robustly quantifying the bias present remain difficult as traditional definitions of fairness [ 33 ] do not directly apply to the unstructured, open-ended text generated. In this paper, we consider a language generation model as biased if it disproportionately generates text that is often perceived as being negative, unfair, prejudiced, or stereotypical against an idea or a group of people with common attributes [ 9 ] . There has been two main approaches to mitigating bias in NLG, namely domain-specific training and constrained decoding [ 37 ] . Domain-specific training requires fine-tuning the model with counterfactual data augmentation [ 10 , 19 ] , adversarial learning [ 20 ] , or even regularization training [ 26 ] . Unfortunately, the excessive resource consumption of this approach makes it less feasible in real-world applications. The more resource-efficient approach focuses on guiding the output generation during decoding. For example, [ 30 ] mitigates bias by finding a sequence of tokens that, when concatenated to input prompts, will stimulate unbiased generation from the model - effectively modifying the user prompt before inference. However, the modified prompt may not be human readable and has been shown to spew racist output on non-racial contexts [ 35 ] . [ 17 ] proposed an Autoregressive Iterative Nullspace Projection algorithm that mitigates bias through a projection matrix that acts on the output logits. [ 28 ] ’s method leverages the ability of a language model to recognize its own undesirable bias in a form of self-diagnosis. Last but not least, [ 37 ] considers detoxifying and debiasing a joint task, combining two adaptive approaches at inference time. Our approach incorporates both resource efficiency and interpretability: the experts are small language models and we can examine the shift in output probabilities before and after debiasing for any given prompt. Furthermore, our algorithm can be tailored to optimize for specific types of bias, depending on the anticipated use case of the generated text. 3 Methodology 3.1 Datasets Table 1: Sample counts and examples of data used to fine-tune experts and anti-experts for different bias directions Bias direction Stereotype Count Example Gender Stereotype 505 the cleaning lady is a

Read the complete text