A Systematic Analysis of Biases in Large Language Models

Xulang Zhang, Rui Mao, Erik Cambria · 2025

Read A Systematic Analysis of Biases in Large Language Models by Xulang Zhang, Rui Mao, Erik Cambria (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15792.txt.

Introduction

Xulang Zhang, Rui Mao, and Erik Cambria examine social biases in four widely used large language models in a 2025 paper, dated December 16. Language models now shape how people get information and make decisions, so hidden leanings matter. The study probes bias along politics, ideology, alliance, language, and gender lines.

News summarization tests political neutrality, stance classification probes ideology, and United Nations voting patterns reveal alliance leanings. Multilingual story completion exposes language bias, and World Values Survey prompts reveal gender related affinities. The models aim for neutrality and impartiality, the authors find, yet they still show biases and affinities of different types across the tests.

Catalog record

CREATORXulang Zhang, Rui Mao, Erik Cambria
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15792

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Systematic Analysis of Biases in Large Language Models Xulang Zhang, Rui Mao, Erik Cambria Abstract Large language models (LLMs) have rapidly become indispensable tools for acquiring information and supporting human decision-making. However, ensuring that these models uphold fairness across varied contexts is critical to their safe and responsible deployment. In this study, we undertake a comprehensive examination of four widely adopted LLMs, probing their underlying biases and inclinations across the dimensions of politics, ideology, alliance, language, and gender. Through a series of carefully designed experiments, we investigate their political neutrality using news summarization, ideological biases through news stance classification, tendencies toward specific geopolitical alliances via United Nations voting patterns, language bias in the context of multilingual story completion, and gender-related affinities as revealed by responses to the World Values Survey. Results indicate that while the LLMs are aligned to be neutral and impartial, they still show biases and affinities of different types. 1 Main Humans have the propensity to trust the suggestions and decisions made by automated systems to be neutral and reliable. However, as Large Language Models (LLMs) are progressively getting integrated into the daily lives and decision-making processes of users around the world, there are growing doubts about whether LLMs are able to give fair and unbiased responses 33 . As such, to the betterment of Artificial Intelligence (AI) safety, it is important to examine the various biases that may be propagated into LLMs during the training procedure, so as not to perpetuate prejudice, stereotypes, and harmful messaging to a global user base. Existing works have explored different types of bias analyses on LLMs. Biases against certain demographics have long been a focal point of research in this field, e.g., gender bias 27 ; 45 ; 19 ; 35 , racial bias 47 , ableist bias 21 , and various harmful stereotypes and biases in LLMs 8 ; 24 ; 16 ; 40 ; 31 and VLMs 1 ; 49 . With different probing methodologies, these works have consistently shown that LLMs have varying degrees of biases on different subject matters, mirroring human prejudice and discrimination in their decision-making and generated text. Furthermore, with the widespread use of LLMs around the world, there is a growing scrutiny in the culture bias 23 ; 9 ; 38 ; 43 ; 34 of LLMs. Notably, LLMs are susceptible to the semantic anglocentrism inherited from the predominantly US-based English training corpora 22 ; 46 ; 13 ; 50 . It can be concluded that most LLMs are ill-equipped to handle cultural nuances, not only because of the lack of knowledge on certain cultural practices and conventions, but also the misalignment with non-English-speaking cultures embedded in the semantic space. In a similar vein, as interests grow in employing LLMs as tools for media bias analyses 11 ; 30 , some works explored the inherent political bias of LLMs by prompting LLMs to generate a stance on selected political and ideological questions and topics 41 ; 17 ; 6 ; 44 . Other studies investigated whether LLMs show signs of human-like social identity bias, confirming that with explicitly or implicitly assigned identity, most LLMs exhibit a similar degree of ingroup solidarity and outgroup hostility as humans demonstrate in the pretraining data and in real life 15 ; 25 . More interestingly, Laurito et al. showcased that LLMs have AI-AI bias, where they consistently favor texts generated by LLMs over humans 29 . Therefore, it is apparent that LLMs are not agents of fairness and neutrality that we aim for them to be. Despite the plethora of existing works dissecting LLMs’ bias, we have only scratched the surface. As many LLMs nowadays are becoming closed-source or resource-demanding for local deployment and finetuning, it is important for the research community to develop accessible ways to probe LLMs for their biases in various domains. In this paper, we meticulously designed a set of experiments to systematically examine the underlying biases and predispositions of four widely used LLMs (namely, Qwen, DeepSeek, Gemini, and GPT) in areas of politics, ideology, alliance, language, and gender. While some existing works explored some of these areas individually, as introduced above, this paper aims to provide a more comprehensive evaluation through rigorous experiment design, offering a novel perspective into the nature of these biases presented in widely-used LLMs. We examined the LLMs’ political leaning by analyzing their generated text from the task of news summarization, showing that while the LLMs are, at large, politically neutral, some of them do show a slight inclination to certain political leanings. We investigated their biases on different ideologically charged topics by analyzing their performance in news stance classification, discovering that the LLMs show different perceptions of progressiveness in areas of election, race, immigration, LGBT, and abortion. We probed their preference towards certain geopolitical alliances by analyzing their prompted voting results on United Nations General Assembly (UNGA) roll calls, which suggests that each LLM demonstrates unique patterns of agreements and disagreements with real-life UNGA delegates. We studied whether they demonstrate different musing patterns when responding to different languages by analyzing their story completion results, leading to findings about the clusters of thinking in different languages. We explored their affinities to specific genders by analyzing their answers to sections of the World Value Survey, which indicates that all four LLMs are slightly more inclined to represent women’s values. 2 Results In this paper, we conducted five carefully designed experiments to investigate the biases of four widely used LLM architectures, namely Qwen, DeepSeek, Gemini, and GPT (specific model versions listed in Section 4.1 ,) in the areas of politics, ideology, alliance, language, and gender. 2.1 Political Bias To probe the political bias of the LLMs, we prompt the LLMs to write neutral summaries of news reports on political events ( n=1,018 ) from center-leaning media outlets. The summaries are then compared with the news articles on the same political events covered by left-leaning and right-leaning outlets. Our motivation is that, by comparing the similarities between LLMs’ summaries and the reporting from the left and the right, we can examine if what is perceived by the LLMs to be neutral writing may in fact be slanted more towards one political leaning. To achieve this, we measure the similarities by using the cosine similarity between the context embeddings of the involved paragraphs, generated by the Qwen Embedding model. Fig. 1 shows the results of the political bias experiment. For all subfigures, points in the upper diagonal signify that the neutral summaries generated by the LLM are more similar to the right-leaning reporting, and vice versa for the lower diagonal. From Fig. 1 (a)-(d), we can see that for all four LLMs, the similarity points are mostly distributed along the diagonal line, indicating that they are able to be politically neutral in most cases. Additionally, the data points tend to aggregate in the top right for all the scatter plot graphs, i.e., higher similarities to cosine similarities to the news reporting to both sides. This can be interpreted as a better quality of summarization, i.e., superior at capturing key information and the core essence of the reported event, imitating general news article writing style, etc. From Fig. 1 (e), it can

Read the complete text