Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personas Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25), April 28-May 2, 20
Pietro Bernardelle, Leon Fröhling, Stefano Civelli, Riccardo Lunardi, Kevin Roitero, Gianluca Demartini · 2024
Read Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personas Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25), April 28-May 2, 20 by Pietro Bernardelle, Leon Fröhling, Stefano Civelli, Riccardo Lunardi, Kevin Roitero, Gianluca Demartini (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14843.txt.
Introduction
Bernardelle, Fröhling, Civelli, Lunardi, Roitero, and Demartini map in twenty twenty-four how fake personas steer the politics of large language models. Bias studies usually treat a model as one fixed voice, yet users often prompt models to speak as someone else. Using a big library of synthetic persona sketches, the team ran models through a standard political compass test under many personas.
Persona-prompted answers clustered heavily in the left-libertarian corner of the compass. When then pushed with explicit ideology prompts, models swung strongly toward right-authoritarian answers. Swings toward left-libertarian answers were much smaller, an asymmetry the authors flag. They read the lopsided response as a hint of leanings baked into training.
Catalog record
| CREATOR | Pietro Bernardelle, Leon Fröhling, Stefano Civelli, Riccardo Lunardi, Kevin Roitero, Gianluca Demartini |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14843
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personas Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25), April 28-May 2, 2025, Sydney, NSW, Australia DOI: XX.XXXX/XXXXXXX.XXXXXXX ISBN: 979-8-4007-1331-6/25/04 CCS: Information systems Language models Pietro Bernardelle, Leon Fröhling, Stefano Civelli, Riccardo Lunardi, Kevin Roitero, Gianluca Demartini Abstract . The analysis of political biases in large language models (LLMs) has primarily examined these systems as single entities with fixed viewpoints. While various methods exist for measuring such biases, the impact of persona-based prompting on LLMs’ political orientation remains unexplored. In this work we leverage PersonaHub, a collection of synthetic persona descriptions, to map the political distribution of persona-based prompted LLMs using the Political Compass Test (PCT). We then examine whether these initial compass distributions can be manipulated through explicit ideological prompting towards diametrically opposed political orientations: right-authoritarian and left-libertarian. Our experiments reveal that synthetic personas predominantly cluster in the left-libertarian quadrant, with models demonstrating varying degrees of responsiveness when prompted with explicit ideological descriptors. While all models demonstrate significant shifts towards right-authoritarian positions, they exhibit more limited shifts towards left-libertarian positions, suggesting an asymmetric response to ideological manipulation that may reflect inherent biases in model training. † † cc-license: by (a) Mistral-7B-Instruct-v0.3 (b) Llama-3.1-8B-Instruct (c) Qwen2.5-7B-Instruct (d) Zephyr-7b-beta Figure 1 . Political compass distribution of PersonaHub personas when impersonated by different LLMs. Darker regions indicate higher density of personas on a logarithmic scale. The white dot represents the leaning of the original LLM (without any form of persona prompting). The bar charts show the marginal distributions along each axis. 1. Introduction Large Language Models (LLMs) have been shown to exhibit distinct political biases when evaluated through the Political Compass Test (PCT) ( Feng et al. 2023 ; Röttger et al. 2024 ) , typically leaning towards economically left and socially liberal positions. Parallel research has demonstrated that these models can dynamically adopt the perspective of different personas through zero-shot prompting, increasing diversity in their outputs and behavior patterns ( Fröhling et al. 2024 ) while accurately simulating responses across demographic groups ( Argyle et al. 2023 ) . However, this adaptability comes with potential drawbacks, as it can amplify stereotypes ( Liu et al. 2024 ) and significantly alter political expression ( Bang et al. 2024 ) . This malleability in LLMs’ behavior raises questions about the stability of their political orientations and the potential for deliberate manipulation through persona-based prompting. Recent work has introduced PersonaHub ( Ge et al. 2024 ) , a collection of synthetic persona descriptions generated through LLM bootstrapping. While these personas have been shown to increase diversity in various natural language processing (NLP) tasks, their influence on LLMs’ political biases remains unexplored. Our study bridges this gap by investigating how different PersonaHub personas affect LLMs’ responses to the PCT. Specifically, we examine both the distribution of political orientations across personas and the potential for manipulation through explicit ideological prompting. We focus on testing ideological malleability towards right-authoritarian and left-libertarian positions, as these represent maximally distant points on the political compass while also allowing us to examine both reinforcement and resistance to LLMs’ inherent left-libertarian bias. Our investigation addresses two questions: RQ1: How do diverse persona descriptions affect LLMs’ positions on the Political Compass Test? RQ2: To what extent the introduction of explicit ideological elements in persona descriptions can influence LLMs’ Political Compass Test outcomes? We show that synthetic personas predominantly cluster in the left-libertarian quadrant (Figure 1 ), with Mistral, Llama, and Qwen exhibiting similar distributions. Zephyr demonstrates the highest concentration of personas around its centroid, while Llama’s personas distribution displays the greatest variance. Additionally, explicit ideological prompting can substantially alter this baseline distribution, though models respond differently to such interventions. In our experiments, Llama exhibited the strongest shift toward right-authoritarian positions, while Mistral showed the most pronounced movement toward left-libertarian orientations. 2. RELATED WORK Personas in LLMs Recent work has shown that LLMs can effectively adopt and simulate different personas through appropriate prompting. PersonaHub ( Ge et al. 2024 ) introduced a collection of over 1 billion synthetically generated persona descriptions, showing that these personas can help diversify model outputs in a controlled way. This approach has been successful in diversifying data annotations, where personas help elicit a wider range of valid perspectives ( Fröhling et al. 2024 ) . The ability of LLMs to reliably simulate different personas has also been explored in the context of simulating human samples ( Argyle et al. 2023 ) , where LLMs successfully predicted survey responses across different demographic groups with high accuracy. Our work builds on these findings by systematically mapping personas from PersonaHub onto the political compass, providing the first large-scale analysis of how synthetic personas distribute across political ideological space over multiple LLMs. LLMs & Political Compass Test Multiple studies have investigated the political biases of LLMs using the PCT as an evaluation framework. Initial work by Hartmann et al. 2023 found that ChatGPT exhibited consistently left-libertarian leanings. Subsequent studies confirmed this tendency across different LLMs while highlighting challenges in reliably eliciting political stances from more recent models with stronger guardrails ( Röttger et al. 2024 ) . Recent methodological innovations have focused on making PCT evaluations more robust, including through masked token prediction ( Feng et al. 2023 ) and ensuring stability across different prompting approaches ( Röttger et al. 2024 ) . Our work advances this line of research by combining PCT evaluation with persona-based prompting, offering a novel methodology for understanding not just the default political biases of LLMs, but also their capacity to simulate diverse political perspectives through persona adoption. Table 1. Average distance between each LLM-generated persona response and the group’s central point, measured across three political configurations: baseline (plain description), right-authoritarian descriptor, and left-libertarian descriptor. Lower values indicate tighter clustering of persona responses within the political compass coordinate system. Model Avg Distance Base Right-Authoritarian Left-Libertarian Mistral 1.518 1.766 0.950 Llama 1.758 1.801 1.606 Qwen2.5 1.560 1.348 0.833 Zephyr 1.164 1.175 1.163 3. METHODOLOGY Data We utilize PersonaHub, a collection of 1 billion diverse personas 1 1 1 with 200,000 publicly accessible through Huggingface , to examine how persona descriptions shape the political orientations of LLMs. To systematically evaluate these influences, we employ the PCT, which assesses political perspectives through 62 distinct statements across six key domains. While previous research by Lunardi et al. 2024 has shown that LLMs’ responses may vary based on question phrasing, we maintain the original PCT formulations to ensure methodological consistency. Our analysis includes 12.4 million categorical responses (200,000 personas x 62 statements), collected by prompting models to select from predefined political stance options. Models For our analysis, we select four open-source language models: Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, Qwen-2.5-7B, and Zephyr-7B-beta. These models were chosen for their open-source nature and relatively small parameter size (7-8B), enabling easy reproducibility while maintaining sufficient diversity for generalizable insights. Experimental setup Our experimental setup comprises of two phases. First, we establish a baseline by having each model complete the PCT while impersonating each persona description. Second, we investigate ideological malleability by injecting explicit political descriptors (‘‘right authoritarian" or ‘‘left