When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis
Juergen Dietrich · 2026
Read When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis by Juergen Dietrich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27228.txt.
Introduction
Dietrich tests whether language models keep assigned advocate roles when analyzing political statements in multi-model chains. Systems increasingly assign distinct evaluator models adversarial roles to yield multi-perspective assessments, assuming roles hold steady. Using the TRUST pipeline and a stance sorter that reads reasoning text without surface vocabulary, he measures role loyalty across 60 statements, half English and half German, on four drift scores. Two failure types emerge in the results.
Catalog record
| CREATOR | Juergen Dietrich |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63952 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27228
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis Juergen Dietrich Abstract Democratic discourse analysis systems increasingly rely on multi-agent LLM pipelines in which distinct evaluator models are assigned adversarial roles to generate structured, multi-perspective assessments of political statements. A core assumption is that models will reliably maintain their assigned roles. This paper provides the first systematic empirical test of that assumption using the TRUST pipeline. We develop an epistemic stance classifier that identifies advocate roles from reasoning text without relying on surface vocabulary, and measure role fidelity across 30 English and 30 German political statements using four complementary metrics: the Role Drift Index (RDI, frequency of role departure), Expected Drift Distance (EDD, magnitude of drift), Directional Drift Index (DDI, direction of drift), and Entropy-based Role Stability (ERS, consistency across runs). We identify two failure modes — the Epistemic Floor Effect (fact-check results create an absolute lower bound below which the legitimizing role cannot be maintained) and Role-Prior Conflict (training-time knowledge overrides role instructions for factually unambiguous statements) — as manifestations of a single underlying mechanism: Epistemic Role Override (ERO), confirmed by a mirror-symmetric Logos-accuracy profile across the critical and charitable roles. Model choice significantly affects role fidelity: Mistral Large outperforms Claude Sonnet by 28 percentage points (pp; 67% vs. 39%) and exhibits a qualitatively different failure mode — role abandonment without polarity reversal — compared to Claude’s active switch to the opposing stance. Role fidelity is language-robust across English and German. Fact-check (FC) provider choice is not universally neutral: Perplexity significantly reduces Claude’s role fidelity on German statements ( Δ = − 15 \Delta=-15 pp, p = 0.007 p=0.007 ) while leaving Mistral unaffected. These findings have direct implications for the validation of multi-agent LLM systems in quality-critical applications: a system validated without role fidelity measurement may systematically misrepresent the epistemic diversity it was designed to provide. Keywords: role fidelity, adversarial multi-agent LLM, epistemic floor effect, role-prior conflict, role drift index, political statement analysis, democratic discourse analysis, fact-checking, charitable interpretation, computer system validation 1 Introduction The use of LLMs in automated analysis of political speech raises a fundamental design challenge: how can a single evaluation pipeline capture the genuine diversity of perspectives that political statements require? The TRUST democratic discourse analysis pipeline addresses this by assigning three LLM components — a critical, a balanced, and a charitable advocate — to evaluate each statement from distinct epistemic positions before a supervisor aggregates their assessments into a composite quality score [ 1 ] . This multi-agent design draws on the insight that structured disagreement is itself a quality mechanism, analogous to adversarial peer review [ 2 ] . A prior conceptual risk analysis of the TRUST architecture identified peer-preservation — the spontaneous tendency of model components to protect peer models from deactivation — as a structural hazard in multi-agent configurations, and proposed prompt-level identity anonymization as a mitigation [ 1 ] . An empirical follow-up study measured identity-dependent scoring bias across all active identity exposure channels in TRUST, finding a homogeneity-dependent channel sign pattern and establishing ensemble heterogeneity as a key structural advantage for normative evaluation tasks [ 3 ] . That study also documented substantial run-level variance in identity bias coefficients (standard deviation, SD \approx 0.18 – 0.32 ), raising the question of whether this variance reflects genuine stochasticity or systematic role departure by individual model components. The present paper addresses that question directly. We operationalize role fidelity as the degree to which an advocate’s reasoning text reflects its assigned epistemic stance, independent of surface vocabulary. We develop a classifier for this purpose, introduce four complementary role drift metrics (RDI, EDD, DDI, ERS), and identify two systematic failure modes — Epistemic Floor Effect and Role-Prior Conflict — that manifest as a single underlying mechanism: Epistemic Role Override (ERO). These failure modes appear to be general properties of instruction-following models rather than pipeline-specific artifacts [ 4 , 5 ] , with direct implications for the validation of multi-agent LLM systems in quality-critical applications. Our main contributions are: (1) an epistemic stance classifier for advocate roles that achieves 80%, 89%, and 67% accuracy for the critical, balanced, and charitable roles respectively; (2) four complementary metrics — RDI, EDD, DDI, and ERS — that together characterize the frequency, magnitude, direction, and consistency of role drift; (3) the identification of the Epistemic Floor Effect and Role-Prior Conflict as named, reproducible failure modes; (4) an empirical comparison of Claude Sonnet and Mistral Large as charitable advocates across English and German statements; and (5) an empirical comparison of Gemini 2.5 Flash and Perplexity sonar-pro as fact-check providers, establishing conditions under which provider choice is and is not interchangeable. Section 2 situates the work within the TRUST pipeline architecture and related research on role-playing, stance detection, and political discourse analysis. Section 3 describes the statement dataset, the role fidelity classifier, the drift metrics, and the experimental design. Section 4 reports classifier performance, the two failure modes, and empirical comparisons across models, languages, and fact-check providers. Section 5 discusses implications for multi-agent system design and validation. Section 6 concludes. 2 Background This section situates the present work within the TRUST research programme and relevant prior literature on role-playing, sycophancy, and stance detection in LLMs. 2.1 The TRUST Pipeline TRUST ( democracy-intelligence.de ) evaluates political statements along three rhetorical dimensions inspired by Aristotle’s classical framework: 1 1 1 Aristotle, Rhetoric , ca. 322 BCE. Logos (factual argumentation quality, -2 to +2 ), Ethos (respect and conduct toward political opponents and social groups, -2 to +2 ), and Pathos (emotional appeal and social cohesion framing, -2 to +2 ). Three advocate components — critical, balanced, and charitable — evaluate each statement from distinct epistemic positions using different underlying LLMs to prevent monoculture effects. A rule-based supervisor layer applies fixed scoring weights per dimension and maps the weighted sum to a final composite quality score on an A–E scale. If score variance across advocates exceeds a threshold after Round 1, a second deliberative iteration (Round 2) is triggered. This multi-perspective design follows the multi-agent debate (MAD) paradigm [ 6 ] and leverages structured epistemic disagreement as a quality mechanism. 2.2 Related Work Role-playing and persona maintenance in LLMs has attracted increasing attention [ 4 , 5 ] . LLMs are known to drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior even under explicit instruction [ 4 ] . Recent work addresses this through training interventions such as contrastive learning [ 5 ] and reinforcement learning [ 4 ] , reducing persona inconsistency by over 55%. However, these approaches target dialogue-level persona drift — gradual deviation from an assigned character in open-ended interaction. The failure mode identified in the present paper is qualitatively distinct: Epistemic Role Override is triggered by factual verification rather than conversational drift. Whether it can be mitigated by prompt engineering — and to what degree — is an empirical question addressed in Section 4.4. General instruction-following benchmarks [ 7 ] measure format and content compliance but do not assess directional consistency of adversarial stances. Sycophancy research has shown that models revise outputs toward user-expressed preferences [ 8 ] , which is related but distinct: sycophancy measures responsiveness to external