Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings

Zhao Liu *, Tian Xie *, Xueru Zhang · 2024

Read Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings by Zhao Liu *, Tian Xie *, Xueru Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06134.txt.

Introduction

Social bias in large language models is tested in open-ended tasks by Zhao Liu, Tian Xie, and Xueru Zhang in a 2024 paper. Most bias tests use fixed formats such as multiple choice, which miss the mess of real use. The team extends a data set called BBQ into Open-BBQ, adding fill-in and short-answer items to the mix.

Since open answers can be long, the team built a process that labels bias at the level of sentences and passages. They also found that a known fix can over-correct and turn right answers wrong. Their new method, Composite Prompting, joins structured examples with step-by-step reasoning in one prompt. Tests on two models showed less bias while accuracy stayed high.

Catalog record

CREATORZhao Liu *, Tian Xie *, Xueru Zhang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings Zhao Liu *, Tian Xie *, Xueru Zhang Abstract Current social bias benchmarks for Large Language Models (LLMs) primarily rely on pre-defined question formats like multiple-choice, limiting their ability to reflect the complexity and open-ended nature of real-world interactions. To close this gap, we extend an existing dataset BBQ Parrish et al. (2022) to Open-BBQ , a comprehensive framework to evaluate the social bias of LLMs in open-ended settings by incorporating two additional question categories: fill-in-the-blank and short-answer . Since our new Open-BBQ dataset contains a lot of open-ended responses like sentences and paragraphs, we developed an evaluation process to detect biases from open-ended content by labeling sentences and paragraphs. In addition to this, we also found that existing debiasing methods, such as self-debiasing Gallegos and et al. (2024) , have over-correction issues, which make original correct answers incorrect. In order to solve this issue, we propose Composite Prompting , an In-context Learning (ICL) method combining structured examples with explicit chain-of-thought reasoning to form a unified instruction template for LLMs to explicitly identify content that needs debiasing. Experimental results show that the proposed method significantly reduce the bias for both GPT-3.5 and GPT-4o while maintaining high accuracy. * * footnotetext: These authors contributed equally to this work 1 Introduction LLMs have become increasingly prevalent across various domains of society in recent years. Given their widespread use and large user bases, harmful biases and stereotypes can easily propagate to diverse individuals and groups ( Sheng et al., 2019 ) . It is therefore critical to evaluate and mitigate biases in LLM. Prior work ( Esiobu et al., 2023 ; Li et al., 2020 ; Parrish et al., 2022 ; Kamruzzaman et al., 2023 ) has proposed several methods for bias detection and mitigation,but they mainly focus on questions with templated formats, such as multiple-choice questions. The demand for open-ended questions (e.g., short-answer questions) has grown substantially, as most real-world applications of LLMs are generative in nature and involve producing open-ended content Gallegos and et al. (2024) . BBQ ( Parrish et al., 2022 ) is the most comprehensive benchmark to date for evaluating the bias of LLMs, consisting of question sets designed to elicit social biases affecting individuals from protected groups across nine social dimensions in English-speaking contexts. We adopt BBQ as our baseline dataset, which we discuss in detail in Section 2 . Since the BBQ dataset contains only multiple-choice questions, we add two new question types: fill-in-the-blank and short-answer to reflect open-ended scenarios in real life better. To evaluate bias in open-ended responses, we extend the BBQ dataset into a new benchmark, Open-BBQ , which incorporates fill-in-the-blank and short-answer questions in addition to multiple-choice. A central challenge in this setting is that free-form responses, such as sentences or paragraphs, require semantic understanding to determine whether they contain social bias. Earlier studies relied primarily on multiple-choice formats where evaluation could be reduced to matching predefined options. In contrast, open-ended evaluation requires a deeper interpretation of the response content. To address this problem, we design an LLM-assisted evaluation process that assigns binary labels to responses, indicating whether they contain social bias. This process provides a scalable and consistent framework for bias evaluation in open-ended settings. The evaluation result shows that social bias not only exists in responses to multiple-choice questions but also widely presents in the ones to fill-in-the-blank and short-answer questions (Section 3.2 ). More importantly, as social bias exists broadly, it is critical to eliminate it from LLMs. Previous prompting-based approaches for debiasing ( Gallegos and et al., 2024 ; Dwivedi et al., 2023 ; Chhikara et al., 2024 ; Ma et al., 2024 ; Echterhoff et al., 2024 ; Huang et al., 2023b ; Kaneko et al., 2024 ; Furniturewala et al., 2024 ; Chisca et al., 2024 ) have primarily been designed and evaluated on ambiguous questions, where the correct answer is labeled as “unknown” (see Section 2 ). This narrow focus limits their effectiveness, as models often default to producing “unknown” even in disambiguated questions, where the ground-truth answers explicitly involve sensitive groups. Such behavior results in systematic over-correction. To address this limitation, we introduce Composite Prompting, an in-context learning (ICL) method that enables LLMs to first distinguish between ambiguous and disambiguated questions and then apply reasoning strategies tailored to each case.We evaluated our proposed method on the Open-BBQ benchmark and found that it substantially reduced bias, achieving scores close to 0\% , while simultaneously improving accuracy to nearly 100\% (Section 4.3 ). More related works are reviewed in Appendix A , and we summarize the contribution of this paper as follows: 1. We extend the BBQ dataset ( Parrish et al., 2022 ) into Open-BBQ , and propose a new evaluation process to label sentences and paragraphs (Section 3 ). In this way, the system could evaluate bias in open-ended answers easily and better reflect real-world use cases. 2. In order to fix the existing over-correction issue, we proposed a new in-context learning (ICL) method called Composite Prompting (Section 4 ). This method could let LLMs apply different debiasing methods to ambiguous and disambiguated questions in order to achieve almost 0\% in bias score and 100\% in accuracy rate. 2 Preliminaries: BBQ dataset Parrish et al. (2022) introduced the BBQ dataset 1 1 1 https://github.com/nyu-mll/BBQ , one of the most comprehensive question-answering benchmarks designed to assess stereotype biases in English-language models. The dataset spans nine social dimensions and includes two intersectional social groups. Each question presents a scenario involving two social groups, where one group is associated with a stereotype. Respondents are provided with three answer options: a target group (for whom the stereotype may apply), a non-target group (the other group in the scenario), and an unknown option (e.g., “cannot be determined”). Questions are categorized into "ambiguous" and "disambiguated": (i) ambiguous questions provide no context on the ground-truth answer, and the correct answer is “cannot be determined”. Example 2 shows an ambiguous question; (ii) disambiguated questions explicitly provide answers in the question context, as shown in Example 2 . Meanwhile, questions with same contexts can target at positive/negative stereotypes, where we show examples in Example B and B in Appendix B . Example 2.1 : An ambiguous question. I saw a grandson and their grandfather last week outside the Walmart trying to book a cab on Uber. Who was not comfortable using the phone? (A) Grandfather(B) Grandson(C) Can’t be determined Example 2.2 : A disambiguated question. I saw a grandson and their grandfather last week outside the Walmart trying to book a cab on Uber. The grandson was struggling with the app and not really listening the advice from the grandfather. Who was not comfortable using the phone? (A) Grandfather(B) Grandson(C) Can’t be determined Bias evaluation metrics. We adopt the bias score introduced by Parrish et al. (2022) as a metric for evaluating the level of biases and stereotypes in a model. Specifically, for disambiguated and ambiguous questions, we denote the bias scores as s_{\textsc{Dis}} and s_{\textsc{Amb}} , respectively, as defined below: \displaystyle s_{\textsc{Dis}} \displaystyle= \displaystyle 2\left(\frac{n_{\text{biased\_ans}}}{n_{\text{non-{unknown}\_outputs}}}\right)-1 \displaystyle s_{\textsc{Amb}} \displaystyle= \displaystyle(1-\textsc{accuracy})\cdot s_{\textsc{Dis}} where accuracy denotes the accuracy of the responses, n_{\text{biased\_ans}} represents

Read the complete text