Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models

Seunguk Yu, Juhwan Choi, Youngbin Kim · 2025

Read Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models by Seunguk Yu, Juhwan Choi, Youngbin Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19121.txt.

Introduction

Seunguk Yu, Juhwan Choi, and Youngbin Kim build a multilingual data set to test ethical bias in language models. Their paper is from two thousand twenty-five. Social bias in such models is well logged, but whether it shifts with the tongue in use was less clear. They guess that tongue-linked gaps drive part of the bias.

Their set draws news pieces on human rights themes and turns them into touchy questions and answers in many tongues. Two strict tests then weigh bias across tongues and themes. In most cases, the tests rule out the no-bias claim. Bias shows across tongues and across models alike. The set is free for later work.

Catalog record

CREATORSeunguk Yu, Juhwan Choi, Youngbin Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19121

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models Seunguk Yu, Juhwan Choi, Youngbin Kim Abstract Despite the recent strides in large language models, studies have underscored the existence of social biases within these systems. In this paper, we delve into the validation and comparison of the ethical biases of LLMs concerning globally discussed and potentially sensitive topics, hypothesizing that these biases may arise from language-specific distinctions. Introducing the Multilingual Sensitive Questions & Answers Dataset ( MSQAD ), we collected news articles from Human Rights Watch covering 17 topics, and generated socially sensitive questions along with corresponding responses in multiple languages. We scrutinize the biases of these responses across languages and topics, employing two statistical hypothesis tests. The results suggest that the null hypotheses are rejected in most cases, indicating biases arising from cross-language differences. It indicates that ethical biases in responses are widespread across various languages, and notably, these biases are prevalent even among different LLMs. By making the proposed MSQAD openly available, we aim to facilitate future research endeavors focused on examining cross-language biases in LLMs and their variant models 1 1 1 https://github.com/seungukyu/MSQAD . 1 Introduction The advancement of large language models (LLMs) has enabled widespread access to extensive pre-trained models, which are instrumental in addressing task-specific user requirements Zhao et al. (2023) . Numerous versions of LLMs have been deployed, each tailored based on distinct tuning processes and the characteristics of individual datasets Anthropic (2024) ; Google (2024) ; OpenAI (2023) . As models have developed and progressed, there have been reports of the potential risk of incorporating socially biased information into them Taubenfeld et al. (2024) ; Wan et al. (2023) ; Yeh et al. (2023) . Figure 1: Results of instructing the same model to respond to a socially sensitive question in the MSQAD. The underlined and highlighted texts indicate key parts of the question, both in the original languages and their English translations . Despite being given the same question, we observed significant differences in the output contents depending on the language used. In the meantime, culture and language are inherently interconnected with cultural meanings encoded in linguistic symbols and expressed through linguistic behavior Kramsch (2014) ; Jiang (2000) . Therefore, the cultural characteristics of a language can be inferred from large corpora in that language. In summary, since culture and language have historically been closely intertwined, a corpus in a specific language inherently reflects the culture of that language Rabiah (2018) ; Sharifian (2017) . However, the inherent biases in analyzing ethical factors across languages in LLMs remain unexplored. While recent studies have examined the multilingual aspects of LLMs, they focused on improving performance in general tasks rather than addressing language-specific biases from social or cultural perspectives Zhao et al. (2024) ; Huang et al. (2023) ; Yuan et al. (2024) . In this study, we validate cross-language biases of LLMs on globally discussed and potentially sensitive questions. Given that LLMs are predominantly English -centric and unevenly distributed across languages, owing to imbalances in the training corpus Liu et al. (2025) ; Li et al. (2024) , we define ethical biases as situations where the informativeness and morality of responses change depending on the language used 2 2 2 To evaluate the variation in morality or informativeness of responses across languages, we employ two statistical tests and formulate corresponding null hypotheses. . We then measure these biases by examining how LLMs’ responses to our sensitive questions varied across different languages. Therefore, it was essential to develop a series of questions on sensitive topics that could be universally applicable across languages. To accomplish this, we collected news information from Human Rights Watch on 17 topics, including Children’s Rights , Refugees and Migrants , and Women’s Rights . We employed LLM to generate socially sensitive and controversial questions based on that information, which were then expanded into multiple languages. Semantically equivalent questions and prompt constructions were provided to obtain responses in each language, creating what we propose to refer to as a Multilingual Sensitive Questions & Answers Dataset ( MSQAD ). Examples of the question and acceptable responses in each language are shown in Figure 1 . When asked how Malaysia should address allegations of torture related to refugees , responses in English , Chinese , and German were more specific, suggesting concrete actions that Malaysia should take. In contrast, responses in other languages, such as Hindi , were less detailed and more concise. We hypothesize that there would be no significant differences between responses to the same questions under identical conditions, only except for the language used. To evaluate our hypothesis, we apply several statistical hypothesis tests commonly used in NLP research to ensure that the results were not due to chance Zmigrod et al. (2022) ; Dror et al. (2018) . The results consistently reject the null hypotheses, indicating significant ethical biases arising from differences in the language used. Furthermore, by conducting experiments across various LLMs under the same conditions, we validate how responses varied according to the model used for each language. The contributions of our study are as follows: • We propose the Multilingual Sensitive Questions & Answers Dataset ( MSQAD ), enabling the LLM to generate both acceptable and non-acceptable responses to socially sensitive questions. We generate controversial questions from global news topics and relevant responses in multiple languages. • We conduct statistical examinations to assess the degree of ethical bias in responses when the prompt constructions were semantically identical but the used language varied. We reveal that there are significant biases across languages in nearly all cases, with some languages exhibiting a prejudice for specific topics over other languages. • We further validate the statistical tests by experimenting with different LLMs to verify the bias in responses due to model choices. We observe that even for questions with the same topics, there are significant language-specific differences based on the model used. Figure 2: Process of constructing the MSQAD and validating ethical biases across languages with the dataset. The blue, green, and yellow sections depict the stages of collecting news, constructing dataset through LLM, and conducting cross-lingual examinations across languages using statistical hypothesis tests, respectively. 2 Related Work 2.1 Data Construction through LLMs Recent progress in LLMs has led to studies focusing on constructing specific datasets required for each task Xu et al. (2024) ; Mosca et al. (2023) ; Abdullin et al. (2023) . Researchers have employed prompting techniques tailored to each context Brown et al. (2020) , allowing them to utilize the high-quality texts generated by LLMs as newly-constructed datasets. Other studies have focused on socially biased texts and constructed related datasets Lee et al. (2023) ; Hartvigsen et al. (2022) ; Rosenthal et al. (2021) . Although using model-generated texts to represent specific demographics is significant, it was often limited to certain groups or languages. To address this, we propose the Multilingual Sensitive Questions & Answers Dataset ( MSQAD ), which adopts a broader multilingual perspective by generating globally sensitive questions and enabling responses in multiple languages. 2.2 Bias Covered in LLMs There has been a steady stream of research analyzing the potential risks inherent in LLMs Gallegos et al.

Read the complete text