Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models

Mats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, David Garcia · 2025

Read Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models by Mats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, David Garcia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16148.txt.

Introduction

Mats Faulborn and co-authors build a theory-grounded way to measure political bias in language models, in a 2025 paper. Models now serve as simulated people, search tools, and text analysts. Bias in them can skew all those uses. Past tests leaned on a popular compass quiz that is not a valid survey tool. Prompt wording also swung the results.

Their measure draws on political science and survey design. It tests many prompt forms and lets models answer freely, not just pick from set options. A careful yardstick matters because loose tests have produced clashing claims about model leanings. The work aims to settle how bias should be measured before arguing over how large it is.

Catalog record

CREATORMats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, David Garcia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16148

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models Mats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, David Garcia Abstract Prompt-based language models like GPT4 and LLaMa have been used for a wide variety of use cases such as simulating agents, searching for information, or for content analysis. For all of these applications and others, political biases in these models can affect their performance. Several researchers have attempted to study political bias in language models using evaluation suites based on surveys, such as the Political Compass Test (PCT), often finding a particular leaning favored by these models. However, there is some variation in the exact prompting techniques, leading to diverging findings and most research relies on constrained-answer settings to extract model responses. Moreover, the Political Compass Test is not a scientifically valid survey instrument. In this work, we contribute a political bias measured informed by political science theory, building on survey design principles to test a wide variety of input prompts, while taking into account prompt sensitivity. We then prompt 11 different open and commercial models, differentiating between instruction-tuned and non-instruction-tuned models, and automatically classify their political stances from 88,110 responses. Leveraging this dataset, we compute political bias profiles across different prompt variations and find that while PCT exaggerates bias in certain models like GPT3.5, measures of political bias are often unstable, but generally more left-leaning for instruction-tuned models. 1 Introduction In the past years, Large Language Models (LLMs) have emerged as a transformative technology with applications in a large variety of social domains including medicine Singhal et al. 2023 ; Thirunavukarasu et al. 2023 ; Zhou et al. 2024 , finance Jeong 2024 ; Li et al. 2023c ; Wu et al. 2023 , education Elkins et al. 2023 ; Kasneci et al. 2023 and academia Beltagy et al. 2019 ; Meyer et al. 2023 ; Porsdam Mann et al. 2023 . More and more people use these technologies for complex information integration tasks such as search to become informed about or to summarise historical and current events Sharma et al. 2024 . However, researchers and policy-makers have found evidence of societal bias in language technologies, including LLMs. Political bias is especially problematic in some of these information gathering, search, or even content analysis-related tasks, because they can perpetuate certain existing real-life biases Sharma et al. 2024 . Given these challenges, researchers, policy-makers, industry and other involved stakeholders must ensure that emerging technologies do not contribute to such problems but rather, in an optimistic view, help to solve them. To achieve this, it is of vital importance to develop valid measures of political bias in LLMs. Careful bias analyses require evaluating varied inputs and outputs of the models with evaluation techniques that have high construct validity. However, most research on political bias in LLMs is limited to static settings where the model is forced to answer with a single answer token, thereby only evaluating highly constrained outputs, that are not ecologically valid , i.e., in real-world usage, users of LLMs rarely restrict them to single tokens. So far, few studies have investigated political bias in an open-answer setting, while also accounting for prompt sensitivity Feng et al. 2023 ; Röttger et al. 2024 ; Wright et al. 2024 . Adding to these problems, no clear conceptual definition of political bias has been brought forward and most previous work relies on poorly documented and popular science survey inventories like the Political Compass Test (PCT). In addition, a large portion of the literature focuses on investigating closed-source models that lack openness about the technical details of the system and stringent documentation of possible model changes that go undocumented Fujimoto and Takemoto 2023 ; Hartmann et al. 2023 ; Motoki et al. 2023 ; Rozado 2023 ; Rutinowski et al. 2024 . This study addresses these gaps by 1) introducing a theory-driven definition of political bias that relies on a scientifically valid survey inventory — the political leaning items in the World Values Survey EVS/WVS 2022 , 2) developing a political bias measure with inherent consideration of 30 prompt variations applied over eleven commercial and open-weight models. Using this novel methodology, we find that instruction-tuned models exhibit considerable left-leaning political bias. However, we also show that PCT exaggerates political bias for certain models, the variation of input prompts significantly affects the resulting bias measures, and that constrained answer settings lead to unpredictable model outputs. Based on our finding, we provide concrete suggestions for measuring political bias in LLMs. Our code, data, and prompts are available. 1 1 1 anonymized for review 2 Operationalizing Political Bias of Language Models While there has been a plethora of recent work on political bias in LLMs (see Section 5.1 ), there is still a lack of a general definition of political bias. One notable exception is Liu et al. 2022 , who distinguish between political bias that is measured when a political entity is mentioned in the prompt (direct bias) or not mentioned in the prompt (indirect bias). However, this definition still does not explain the concept of political bias itself. Generally speaking, when talking about political bias, we mean divergences of political attitudes and ideas on a ideological spectrum from left to right. In political science, ideology is generally defined as a set of political ideas that are interconnected and stable Campbell et al. 1960 , as well as interdependent Converse 1964 . In the context of LLMs, this implies that there needs to be some consistency to the political ideas fabricated by a model. In practice this implies that if a model argues for both sides of the political spectrum, we cannot reliably position it on an ideological scale and therefore the model would not be considered politically biased or ideological. Recent political science literature suggests that ideology among the US public is better described with a left-right scale along two dimensions rather than one Carmines et al. 2012b . Therefore, we disaggregate our political bias measure into one cultural and one economic dimension. Toward a Political Bias Measure. Given language model responses to political statements and political leaning extracted from these responses (see Section 3.4 ), we define the political bias measures as follows. For a model m and political directions d\in\left\{left,right\right\} we define the count of model answers agreeing with statements d as A , disagreeing with d as D and neutral answers w.r.t d as N . Agreement of m with d is now computed as the proportion of answers that agree with d relative to all other valid answers not labelled as unrelated: \displaystyle P_{agree,m,d}=\frac{A}{A+D+N} (1) Disagreement is computed accordingly. The bias with respect to d is now computed as: \displaystyle Bias_{m,d}=P_{agree,m,d}-P_{disagree,m,d} (2) The bias measure is now positive if a model disproportionately agrees with the statements associated with one political side and negative if the model disproportionately disagrees. Finally, we compute total political bias for one model m by subtracting right political bias from left political bias and dividing the result by two: \displaystyle\frac{Bias_{right,m}-Bias_{left,m}}{2} (3) The resulting measure ranges from -1 to 1 and is negative for left-leaning political bias and positive for right-leaning political bias, disaggregated by the economic and cultural

Read the complete text