ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech

2025

Read ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08247.txt.

Introduction

Marios Koniaris, Argyro Tsipi, and Panayiotis Tsanakas present ParliaBench, a benchmark for making lawmaker speeches with large language models. A good lawmaker speech needs more than fluent language: it must sound politically real and stay ideologically steady. Existing tests score standard text quality and miss those political sides, while models get little specialized training for parliament.

A data set of United Kingdom lawmaker speeches supports systematic training and scoring. Evaluation mixes computational metrics with model judged assessments of linguistic quality, coherence, and political realism. Two new embedding metrics measure spectrum and party alignment. Fine tuning five models and scoring twenty eight thousand speeches shows clear gains from specialization.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech Abstract Parliamentary speech generation presents specific challenges for large language models beyond standard text generation tasks. Unlike general text generation, parliamentary speeches require not only linguistic quality but also political authenticity and ideological consistency. Current language models lack specialized training for parliamentary contexts, and existing evaluation methods focus on standard NLP metrics rather than political authenticity. To address this, we present ParliaBench, a benchmark for parliamentary speech generation. We constructed a dataset of speeches from UK Parliament to enable systematic model training. We introduce an evaluation framework combining computational metrics with LLM-as-a-judge assessments for measuring generation quality across three dimensions: linguistic quality, semantic coherence, and political authenticity. We propose two novel embedding-based metrics, Political Spectrum Alignment and Party Alignment, to quantify ideological positioning. We fine-tuned five large language models (LLMs), generated 28k speeches, and evaluated them using our framework, comparing baseline and fine-tuned models. Results show that fine-tuning produces statistically significant improvements across the majority of metrics and our novel metrics demonstrate strong discriminative power for political dimensions. Keywords: Parliamentary Speech Generation,LLM Evaluation,Political Authenticity,Benchmark Evaluation ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech Marios Koniaris , Argyro Tsipi Panayiotis Tsanakas National Technical University of Athens School of Electrical and Computer Engineering, Division of Computer Science Iroon Polytechniou 9, Zographou Campus, 15780 Athens, Greece Email: [email protected], [email protected], [email protected] Abstract content 1. Introduction Democracy thrives through debate. Democratic parliaments are open forums where elected representatives engage in arguments over policy Back et al. 2021 . These debates provide unique insights into political reasoning and ideological positioning. Researchers in political science and computational linguistics increasingly seek to understand and model parliamentary debates. Yet generating authentic parliamentary speech presents significant challenges that extend well beyond typical text generation tasks. The complexity becomes apparent when considering specific examples. A Labour MP discussing taxation policy must sound distinctly different from a Conservative counterpart not just in policy position, but most significantly, in their fundamental approach to governance. Large language models have shown promise across political applications, from sentiment analysis Bestvater and Monroe 2023 and election forecasting Li et al. 2024b to synthetic survey data generation Argyle et al. 2023 ; Bisbee et al. 2024 . Authentic parliamentary speech generation, however, remains challenging. Evaluation approaches for generated political text fall short for these specialized requirements. Although evaluation methods for text generation tasks have evolved from simple overlap metrics Papineni et al. 2002 ; Lin 2004 toward more sophisticated embedding-based approaches Zhang et al. 2020 ; Zhao et al. 2019 , they still focus on surface-level similarity rather than political authenticity. Parliament Proceedings 671k Speeches Process & Enrich Data + External Sources Parliamentary (Benchmark) 448k Speeches LLMs 5 Architectures Fine-tuning Methodology QLoRA + Training Fine-tuned Models 5 Specialized Models AI Generated Dataset 28k Generated Evaluation Framework 3 Assessment Categories Performance Analysis Validation Results Data Processing Pipeline Model Development Evaluation & Analysis Figure 1: Methodology Overview: Data processing (top row) creates Dataset A from UK Parliament proceedings. Model development (middle row) fine-tunes base architectures using Dataset A. Evaluation pipeline (bottom row) applies the assessment framework to generated speeches (Dataset B) for performance analysis. Similarly, domain-specific benchmarks have emerged for legislative summarization Kornilova and Eidelman 2019 , opinion alignment Santurkar et al. 2023 , and other specialized fields Xie et al. 2024 ; Li et al. 2024a . Parliamentary speech generation, however, lacks the evaluation framework needed to assess ideological consistency and parliamentary conventions simultaneously. Contributions. We address these limitations by establishing a benchmark resource designed specifically for evaluating parliamentary speech generation. First, we develop a curated dataset containing 448k speeches from UK Parliament (Section 3 ). Second, we introduce a multi-dimensional evaluation framework that assesses both linguistic quality and political authenticity (Section 4 ). Third, we fine-tune five large language models and generate 28k parliamentary speeches to establish baseline performance (Section 5 ). Finally, we demonstrate the framework’s effectiveness through systematic evaluation (Section 6 ). Figure 1 provides an overview of the complete ParliaBench framework and experimental methodology. 2. Related Work Recent advances in large language models (LLMs) have enabled a wide range of applications across political domains. Li et al. 2024b outlines several applications of LLM in political contexts, covering predictive, generative, and simulation-based approaches. The use of LLMs as substitutes for human experts in annotating political texts across multiple languages is explored in Heseltine and von Hohenberg 2024 , while Gunes and Florczak 2023 employ LLMs for classifying U.S. Congressional bills. Argyle et al. 2023 investigate LLMs as proxies for specific human subpopulations in social science research and Bisbee et al. 2024 raise concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs. Agent-based LLMs are utilized as coalition negotiators Moghimifar et al. 2024 and as U.S. senators simulating legislative processes Baker and Azher 2024 . However, these approaches predominantly emphasize analytical and simulation capabilities rather than authentic speech generation quality. Traditional text generation evaluation has evolved from reference-based metrics like BLEU Papineni et al. 2002 and ROUGE Lin 2004 toward embedding based approaches that better capture semantic similarity. BERTScore Zhang et al. 2020 uses contextualized embeddings to compute token-level similarity, while MoverScore Zhao et al. 2019 measures semantic transportation cost using Earth Mover’s Distance. For reference-free evaluation, Zhu and Bhat Zhu and Bhat 2020 propose GRUEN, assessing grammaticality and semantic coherence. Domain-specific datasets like BillSum Kornilova and Eidelman 2019 for legislative summarization and OpinionQA Santurkar et al. 2023 for opinion alignment provide targeted evaluation resources, though gaps remain in generative parliamentary speech assessment. The emergence of LLM-as-a-Judge evaluation Zheng et al. 2023 offers scalable alternatives for nuanced assessment, achieving over 80% agreement with human evaluators in complex judgment tasks. Liu and Sun 2023 further validate this approach, demonstrating GPT-4’s high alignment with human thematic coding in political analysis. This methodology has been successfully applied across diverse contexts, from general LLM benchmarking through competitive debates Moniri et al. 2025 to long-context reasoning evaluation in parliamentary debates Tiwari et al. 2025 . These approaches demonstrate the viability of automated evaluation for argumentative and political content, though models exhibit documented biases toward Western, educated populations Durmus et al. 2023 and systematic preferences in political simulations Qi et al. 2024 . Recent advances in retrieval-augmented generation and chain-of-thought reasoning provide enhanced capabilities for contextually-grounded political text generation, though their application to parliamentary speech evaluation remains underexplored. Regarding parliamentary speech data, structured corpora like ParlaMint Erjavec et al. 2025 provide multi-lingual parliamentary proceedings. Embedding-based approaches for political analysis, introduced in Rheault and Cochrane 2020 , demonstrate that embeddings can capture ideological positioning in parliamentary text. Domain-specific evaluation frameworks have emerged across professional fields, including FinBen Xie et al. 2024 and LexEval Li et al. 2024a . Political science applications have developed specialized benchmarks for election prediction and legislative analysis, yet these focus primarily on classification and analysis tasks. Parliamentary speech generation has attracted recent computational interest, with work exploring European Parliament consensus building Zhang et al. 2025 and political impersonation authenticity Herbold et al. 2024 , but evaluation frameworks remain underdeveloped. Existing approaches focus on narrow aspects like style mimicry rather than systematic quality assessment across linguistic and political authenticity dimensions that parliamentary speech generation

Read the complete text