Gender and Political Bias in Large Language Models: A Demonstration Platform
Wenjie Lin, Hange Liu, Xutao Mao, Yingying Zhuang, Jingwei Shi, Xudong Han, Tianyu Shi, Jinrui Yang · 2025
Read Gender and Political Bias in Large Language Models: A Demonstration Platform by Wenjie Lin, Hange Liu, Xutao Mao, Yingying Zhuang, Jingwei Shi, Xudong Han, Tianyu Shi, Jinrui Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16264.txt.
Introduction
Can language models predict how European lawmakers will vote, and do their errors fall unevenly across groups? Wenjie Lin and seven co-authors build ParlAI Vote, a web platform that links parliament debates, speeches, and roll-call votes with data on gender, age, country, and party group. Users can browse debates and compare real outcomes with model forecasts, broken down by group.
The platform shows where leading models show steady bias in both vote forecasts and gender tasks. It also displays model reasoning, so users can see what cues drive errors. By joining data, models, and visuals in one place, it eases audits and teaching.
Catalog record
| CREATOR | Wenjie Lin, Hange Liu, Xutao Mao, Yingying Zhuang, Jingwei Shi, Xudong Han, Tianyu Shi, Jinrui Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 10 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gender and Political Bias in Large Language Models: A Demonstration Platform Wenjie Lin, Hange Liu, Xutao Mao, Yingying Zhuang, Jingwei Shi, Xudong Han, Tianyu Shi, Jinrui Yang Abstract We present ParlAI Vote 1 1 1 https://euro-parl-vote-demo.vercel.app/ Video: https://www.youtube.com/@Jinrui-sf2jg , an interactive system for exploring European Parliament debates and votes, and for testing LLMs on vote prediction and bias analysis. This platform connects debate topics, speeches, and roll-call outcomes, and includes rich demographic data such as gender, age, country, and political group. Users can browse debates, inspect linked speeches, compare real voting outcomes with predictions from frontier LLMs, and view error breakdowns by demographic group. Visualizing the EuroParlVote benchmark and its core tasks of gender classification and vote prediction, ParlAI Vote highlights systematic performance bias in state-of-the-art LLMs. The system unifies data, models, and visual analytics in a single interface, lowering the barrier for reproducing findings, auditing behavior, and running counterfactual scenarios. It supports research, education, and public engagement with legislative decision-making, while making clear both the strengths and the limitations of current LLMs in political analysis. Introduction LLMs are now widely used to read and analyze political text ( Potter et al. 2024 ; Li et al. 2024 ) , such as text classification ( Liu and Shi 2024 ) , stance detection ( Lan et al. 2024 ; Wagner et al. 2024 ) , coalition negotiation modeling ( Moghimifar et al. 2024 ) , and debate analysis ( Liu et al. 2024 ; Heide-Jørgensen, Eady, and Rasmussen 2025 ) . These uses lead to two core needs. First, they need practical tools to examine whether model performance is valid and equitable across demographic groups. Von der Heyde et al. 2024 shows that LLMs fail to reliably predict individual voting behavior in the 2024 European Parliament elections, with accuracy varying across contexts and requiring detailed prompts, highlighting strong limitations and biases in using LLM-synthetic samples for public opinion prediction. Rettenberger, Reischl, and Schutera 2025 also find that open-source LLMs show consistent political bias on European Union issues: larger models align more with left-leaning parties while smaller models are closer to neutral. Second, researchers require a transparent way to connect language to votes rather than treating prediction as a black box ( Liang et al. 2024 ; Wang et al. 2024 ; Yi et al. 2025 ; Zhou et al. 2024 ; Zhou et al. 2025 ) . The European Parliament is a transparent and practical setting for such work because debates, roll-call votes, and member attributes are public and detailed ( Van Aggelen et al. 2016 ) , but these signals are spread across sources and formats, which raises the cost of systematic analysis and teaching. There is a developed search engine, HowTheyVote ( HowTheyVote 2025 ) , where users can find summaries and votes of Members of the European Parliament (MEPs). To further evaluate LLMs in politically sensitive contexts, we previously introduced the EuroParlVote Benchmark ( Yang, Han, and Baldwin 2025 ) , which systematically links debate topics and speeches to roll-call outcomes and includes rich demographics. Until now, however, there has been no research-oriented and AI-powered visualization system that links debate content to legislative outcomes and demographics while also supporting model-based evaluation. On top of this dataset, we build an interactive visualization platform, ParlAI Vote, that enables exploration of issue-speech-vote connections. The system integrates frontier and open-source LLMs to simulate MEPs’ voting behavior. This combination of data visualization and predictive AI creates a novel environment for investigating political behaviors. This system allows researchers and even the general public to intuitively understand how debate-related contents relate to voting outcomes, while demonstrating the potential and limitations of LLMs in political science. Our system makes three contributions: • To the best of our knowledge, we introduce ParlAI Vote, the first AI-powered system that unifies debates, roll-call votes, and demographic data of the European Parliament into a live, explorable interface, making a complex political benchmark accessible to researchers and the public. • Our demo lets users experiment with different LLMs of their choice to predict MEPs’ voting behavior from debates, enabling comparative exploration of model capabilities and biases while highlighting both the potential and risks of AI in democratic decision-making. • Our system offers an intuitive way for users to observe how LLMs implicitly encode stereotypical associations and biases. By turning gender and political bias evaluations into interactive exploration, the demo makes issues of fairness tangible, fostering transparency and accountability discussions in AI for societal domains. System Description To manage the dataset of 969 roll-call votes, the main page interface provides essential search, filter, and sorting tools. These features are crucial for allowing users to quickly isolate specific legislative events or analyze political trends over time. The vote page provides an integrated interface for exploring roll-call votes in the European Parliament. At the top, users see the debate title, metadata (vote date, report ID, number of participants), and the overall outcome. The centerpiece is the Vote Breakdown visualization, which displays how MEPs voted. By default, votes are grouped by political affiliation and ordered along the left–right spectrum, making ideological alignments immediately visible. Users can pivot this breakdown by country, gender, or age, enabling analysis of how demographics intersect with political behavior and highlighting subgroup divergences that are invisible in aggregate results. The core of the system is the set of interactive AI Prediction modules, which include both Vote Prediction and Gender Prediction. For each debate speaker (Figure 1 ), users can: 1. Access Ground Truth: Display the MEP’s actual vote and/or gender as a baseline reference. This anchoring in real-world outcomes is essential for assessing model reliability, making it clear not only if a model is wrong but also how its errors are distributed across different groups. 2. Demographic Impact Explorer: Predict votes or gender using only the debate speech, then compare outcomes when demographic attributes are introduced. A counterfactual mode allows users to modify variables (e.g., adding political group or demographic information), making fairness concerns visible by showing how sensitive model predictions are to individual attributes. 3. Compare Models: Run multiple LLMs side-by-side on the same input to directly contrast accuracy, robustness, and bias. This feature makes divergences across models visible, helping users understand trade-offs between proprietary and open-weight systems. 4. Inspect Reasoning: Examine the rationales behind predictions to see whether models rely on substantive argumentation or superficial cues. This transparency addresses the “black-box” problem and highlights sources of bias that may be invisible in raw accuracy scores. Figure 1: Vote Prediction results on Ecodesign Regulation by Marc Angel when providing Topic and Speech referred as context and using Llama3.2 as LLM predictor. Findings Leveraging this system ( Yang, Han, and Baldwin 2025 ) , we find consistent evidence of gender bias: female MEPs are disproportionately misclassified as male in gender prediction, and vote simulation performance drops significantly when speakers are labeled as female. Proprietary models such as GPT-4o and Gemini-2.5 demonstrate lower misclassification rates and more balanced treatment of female speakers, whereas open-weight models (e.g., LLaMA-3.2, Mistral) exhibit stronger male bias and weaker calibration. Attempts to mitigate these disparities through LoRA ( Hu et al. 2021 ) fine-tuning were unsuccessful, often worsening performance for female MEPs. On the political dimension,