Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset

2025

Read Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20508.txt.

Introduction

Paul Lerner and François Yvon assess the political fairness of multilingual language models through translation quality rather than English opinion surveys. Their framing treats fair translation as the test: if speeches from some political camps are translated systematically better than others, the model is not politically neutral. The study rests on a new twenty-one-way multiparallel version of the EuroParl corpus with each speaker political affiliation attached.

The dataset holds one point five million sentences over three years, with more than a thousand speakers from seven countries and twelve European parties. Comparing translation quality across affiliations, the authors observe systematic differences: majority parties of the left and right are translated better than outsider parties. The dataset itself is a contribution for future multilingual fairness work.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20508

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset Abstract The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left, center, and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties. Keywords: Parallel Dataset, Political Fairness, Political Biases, Large Language Models Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset Paul Lerner, François Yvon Sorbonne Université, CNRS, ISIR 75005, Paris, France [email protected], [email protected] Abstract content 1. Introduction Parliamentary debates have long been used as a major source of parallel texts in the history of Machine Translation ( Brown et al. 1990 ) . Notably, the EuroParl dataset ( Koehn 2005 ) enabled the study of an increasing diversity of language pairs as the European Union (EU) was growing and played a key part in WMT’s shared tasks ( Koehn and Monz 2006 ; Bojar et al. 2015 ; Kocmi et al. 2022 ) . Given their content, parliamentary debates are also relevant for various fields, including political sciences ( Høyland et al. 2014 ; Deligiaouri 2016 ; Schwalbach 2024 ; Sebők et al. 2025 ) . Unfortunately, existing corpora derived from the European Parliament’s debates (including the noteworthy EuroParl 1 1 1 Also available via OPUS ( Tiedemann 2012 ) . ) share two limitations (see Table 1 ): • they are biparallel, i.e., they pair texts in two languages, although speeches were originally translated into n languages, which prevents any multilingual comparable analysis; • the associated metadata is incomplete. To fill this gap, we build on LinkedEP ( Noy et al. 2017 ) to present 21-EuroParl, a 21-way multiparallel dataset aligned at the sentence level with extensive metadata, including information about the speaker’s political affiliation (national and EU party), see example on Figure 1 . The dataset contains a grand total of 72,234 instances and spans over three years, from 2009 to 2011 (included). It comprises speeches from 1000+ speakers coming from 27 countries (see Figure 2 ), 12 EU parties, 25 EU committees, and hundreds of national parties. Figure 1: A fully worked-out example of 21-EuroParl: a German ( de ) source sentence is aligned to translations in 20 other languages. Metadata includes the speaker’s political affiliation. Dataset # Lang. Pairs Sent. Align. Multiparallel EU Committee EU Party National Party OL LinkedEP 420 ✗ ✓ ✓ ✓ ✓ ✓ MPDE 42 ✗ ✓ ✗ ✗ ✗ ✓ DCEP 253 ✓ ✗ ✗ ✗ ✗ ✗ EuroParl-UdS 2 ✓ ✗ ✓ ✓ ✓ ✓ EuroParl v10 420 ✓ ✗ ✗ ✓ ✗ ✓ Europarl-ST 12 ✓ ✓ ✓ ✓ ✓ ✓ 21-EuroParl 420 ✓ ✓ ✓ ✓ ✓ ✓ Table 1: 21-EuroParl compared to related parallel datasets. OL: Original Language. To showcase the potential of 21-EuroParl, we conduct a study on the political fairness of multilingual Large Language Models (LLMs). This question is critical as LLMs are used daily through chatbots by hundreds of millions of users Milmo et al. 2023 ; Top Websites Ranking 2025 and are increasingly embedded in democratic procedures and processes Small et al. 2023 ; Tessler et al. 2024 ; Revel and Penigaud 2025 .However, their biases have so far mostly been assessed through simulated surveys, written in English ( Feng et al. 2023 ; Rozado 2023 ; Hartmann et al. 2023 ; Santurkar et al. 2023 ; Durmus et al. 2024 ; Motoki et al. 2024 ; Potter et al. 2024 ; Boelaert et al. 2025 ; Röttger et al. 2024 ; Ceron et al. 2024 ) . The propose an alternative framing of political biases and ask the following question: does LLM translation quality systematically vary depending on the political content of the source text? The answer is not straightforward as existing Machine Translation metrics are not well calibrated across language pairs. We therefore introduce a novel way to aggregate translation scores across multiple language pairs and partisan speeches based on Borda counts. We find systematic differences in scores depending on the source languages, as well as the party affiliation, across several model sizes and families. In summary, our contributions are threefold: • a new, 21-way multiparallel version of EuroParl, including the political affiliation of speakers (Section 3 ); • a new Borda-based method to aggregate rankings across language pairs (Section 4 ); • observations regarding the politically unfair translation of LLMs (Section 5 ). We make our dataset, code, and translation outputs available for reproducibility 2 2 2 https://github.com/PaulLerner/21-EuroParl . Figure 2: Countries covered in 21-EuroParl, members of the European Union in 2009-2011 (Kolja21, CC BY 3.0, via Wikimedia Commons) 2. Related Work 2.1. From Parliamentary Speeches into Multilingual Corpora The preparation and release of EuroParl based on the processing of speeches at the EP ( Koehn 2005 ) has had a major impact on machine translation for European languages. Since then, multiple works have, like ours, extended EuroParl with additional information or metadata, such as the original source language or the political affiliation of speakers. For instance, CoStEP ( Graën et al. 2014 ) includes multiple automatic linguistic annotations enabling various types of cross-lingual corpus analyses. However, they are often limited to a small number of languages, see Table 1 . EuroParl-UdS ( Karakanta et al. 2018 ) extends EuroParl with metadata about the original source language and the political affiliation of the speaker, focusing on only three languages. MPDE ( Amponsah-Kaakyire et al. 2021 ) augments EuroParl with information about the original source language and about the use of pivot translation through a third language (e.g. fr-en then en-de instead of fr-de directly). As this information is not explicitly available, they approximate it by considering all translations from 2004 onward as performed through a pivot language. Moreover, they are limited to 7 languages and do not align texts at the sentence level. Iranzo-Sánchez et al. 2020 focus on speech translation and, like us, rely on LinkedEP to collect metadata. However, their study is also limited to a subset of four languages, dubbed Europarl-ST. The undocumented EuroParl v10 now includes speaker metadata, notably their EU party affiliation. 3 3 3 https://www.statmt.org/europarl/v10/ However, it is lacking the mention of the relevant EU committee as well as the national party affiliation. Moreover, it is only biparallel, which prevents comparable analysis across languages. Related to parliamentary debates, the DCEP dataset covers various documents from the European Parliament, such as press releases ( Hajlaoui et al. 2014 ) . However, to avoid overlap with EuroParl, it does not contain the verbatim reports of speeches made in the European Parliament’s plenary. Moreover,

Read the complete text