Don’t Change My View: Ideological Bias Auditing in Large Language Models
Paul Kröger, Emilio Barkett · 2025
Read Don’t Change My View: Ideological Bias Auditing in Large Language Models by Paul Kröger, Emilio Barkett (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12652.txt.
Introduction
If a model's politics can be quietly steered, the steering should at least be detectable from outside. Paul Kröger and one co-author adapt a statistical auditing method to that job in 2025, hunting for ideological bias injected into large language models without ever opening the model.
The audit compares output distributions across prompts on a chosen theme; a systematic shift betrays steering toward a political or religious stance. Because the method is model-agnostic and needs only black-box access, it suits independent checks of commercial systems. Experiments validate the approach and frame it as a post hoc accountability tool, separate from the harder questions of prevention.
Catalog record
| CREATOR | Paul Kröger, Emilio Barkett |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12652
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Don’t Change My View: Ideological Bias Auditing in Large Language Models Paul Kröger, Emilio Barkett Abstract As large language models (LLMs) become increasingly embedded in products used by millions, their outputs may influence individual beliefs and, cumulatively, shape public opinion. If the behavior of LLMs can be intentionally steered toward specific ideological positions, such as political or religious views, then those who control these systems could gain disproportionate influence over public discourse. Although it remains an open question whether LLMs can reliably be guided toward coherent ideological stances and whether such steering can be effectively prevented, a crucial first step is to develop methods for detecting when such steering attempts occur. In this work, we adapt a previously proposed statistical method to the new context of ideological bias auditing. Our approach carries over the model-agnostic design of the original framework, which does not require access to the internals of the language model. Instead, it identifies potential ideological steering by analyzing distributional shifts in model outputs across prompts that are thematically related to a chosen topic. This design makes the method particularly suitable for auditing proprietary black-box systems. We validate our approach through a series of experiments, demonstrating its practical applicability and its potential to support independent post hoc audits of LLM behavior. 1 Introduction LLMs have become core components of many digital services, powering chatbots, coding assistants, search engines, and more ( Guu et al. 2020 ; Lewis et al. 2020 ; Li et al. 2022 ; Raffel et al. 2020 ; Chen et al. 2021 ; Ouyang et al. 2022 ) . In particular, chatbots are increasingly more capable of helping users research and form opinions by generating summaries, explanations, and arguments. Because these models may influence users’ beliefs, it is essential that their outputs remain factual, balanced, and free from systematic bias ( Bolukbasi et al. 2016 ; Caliskan et al. 2017 ; Bender et al. 2021 ; Feng et al. 2023 ; Motoki et al. 2024 ) . From a societal perspective, the widespread adoption of LLMs raises the crucial question of whether individuals or corporations that control these systems could intentionally steer them toward specific ideological positions in an attempt to influence their users’ opinions. Even though it is still an open question whether LLMs can reliably be guided toward a consistent ideological stance ( Santurkar et al. 2023 ; Ceron et al. 2024 ; Kim et al. 2025 ) , it is conceivable that providers may attempt to do so, especially through lightweight interventions such as modifying the system prompt. For instance, a provider might try to bias a model toward conservative Christian values, potentially influencing its responses on issues such as abortion. 1 1 1 While current techniques may not ensure that a model consistently reflects a given ideological identity ( Santurkar et al. 2023 ) , this does not necessarily prevent providers from attempting such steering. One practical and low-effort mechanism for influence is the system prompt. For example, see the recent controversy regarding xAI’s Grok model Goggin 2025 ; Vanian 2025 ; Steedman 2025 ; O’Brien 2025 ; Morrow 2025 ; Robertson 2025 ; Conger 2025 ; Kerr 2025 . Whether such interventions should be legally restricted is ultimately a regulatory question. However, the mere possibility that a few powerful actors could significantly influence public opinion poses a societal risk. As a first step toward addressing this issue, we propose a method for monitoring whether such steering occurs. Specifically, we repurpose the training-free, model-agnostic framework introduced by Levin et al. 2025 , originally designed for detecting unauthorized use of proprietary system prompts. We apply this framework to the task of ideological bias auditing by monitoring whether a model’s responses to prompts on sensitive topics remain consistent over time. More concretely, we assume a fixed LLM and aim to detect steering attempts that result in a change in model outputs. The approach requires no access to model internals and is suitable for black-box settings, making it well-suited for independent post hoc audits. For each sensitive topic of interest, we generate a set of task prompts and track model outputs over time. If responses exhibit significant distributional shifts, the system flags the change for further inspection. The framework is summarized in Figure 1 . Figure 1: Our general framework works as follows: We assume a base LLM deployed via a chat interface. To monitor whether this model’s outputs remain consistent over time, we generate topic-specific prompts and sample responses periodically. If the provider makes changes to the model (e.g., via a system prompt), trying to steer its behavior, our framework identifies distributional shifts in the outputs and alerts the user. This allows independent audits of potential ideological manipulation. One might ask whether systematic monitoring is truly necessary, given that users could probably recognize drastic changes in model behavior on their own. However, there are several challenges that make this complicated in practice. Even subtle shifts in framing or emphasis, potentially unrecognizable to humans, can meaningfully affect human judgments and opinions Tversky and Kahneman 1981 ; Entman 2006 ; Chong and Druckman 2007 ; Beratšová et al. 2016 . Furthermore, without a principled approach, it is difficult to determine whether variations in model responses reflect inherent stochasticity or indicate a more fundamental change in behavior. Our framework addresses this problem by applying the statistical hypothesis test proposed by Levin et al. 2025 with a user-defined significance threshold \alpha , enabling more reliable detection of distributional shifts. Manual inspection also does not scale well when attempting to monitor model outputs across a wide range of topics, further underscoring the need for automated auditing techniques. To illustrate the issues discussed above, we provide an example in the Appendix, where two responses are generated from the same model and for the same task prompt but with different system prompts. In this work, we make two primary contributions. First, we adapt a well-established, training-free statistical method to the novel setting of detecting ideological steering in LLMs. Second, we demonstrate the practical utility of our approach through a series of experiments. While the results are encouraging, we outline important limitations and directions for future work in Section 5 . 2 Related Work While still an area of active research, biases in artificial intelligence systems have been rigorously examined ( Bolukbasi et al. 2016 ; Gordon et al. 2020 ; Raghavan et al. 2020 ; Norori et al. 2021 ) . Numerous studies have identified systematic biases in LLMs, producing a wide range of interesting findings Bang et al. 2021 ; Bender et al. 2021 ; Nadeem et al. 2021 ; Sheng et al. 2021 ; Hosseini et al. 2023 ; Schramowski et al. 2022 ; Smith et al. 2022 ; Liu et al. 2024 . One notable result is that these models tend to express left-leaning or liberal views on social issues, while often exhibiting libertarian attitudes on economic topics ( Hartmann et al. 2023 ; Bang et al. 2024 ; Motoki et al. 2024 ; Rozado 2024 ; Exler et al. 2025 ; Bang et al. 2024 ; Santurkar et al. 2023 ; Rettenberger et al. 2025 ) . To uncover these patterns, researchers frequently employ sophisticated prompting strategies,