Measuring Opinion Bias and Sycophancy via LLM-based Persuasion
Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida, Andrea Roque, Ramon Pires, Hugo Abonizio, Thiago Laitz, Celio Larcher, Roseval Malaquias Junior, Marcos Piau · 2026
Read Measuring Opinion Bias and Sycophancy via LLM-based Persuasion by Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida, Andrea Roque, Ramon Pires, Hugo Abonizio, Thiago Laitz, Celio Larcher, Roseval Malaquias Junior, Marcos Piau (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21564.txt.
Introduction
Rodrigo Nogueira and nine colleagues propose llm-bias-bench, which finds views on contested topics in chats. Direct probing asks views across five pressure turns; indirect probing debates without asking, leaking bias via concessions or rebuttals. Personas, neutral, agree, disagree, give nine-way classes separating stances from sycophancy, LLM-judged. First version covers 38 Brazilian Portuguese topics. In 13 assistants, debate triggers sycophancy two to three times more than direct asks, medians 50 to 79 percent. Opinionated models often mirror under argument; attacker skill matters for shifting views, not neutral starts.
Catalog record
| CREATOR | Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida, Andrea Roque, Ramon Pires, Hugo Abonizio, Thiago Laitz, Celio Larcher, Roseval Malaquias Junior, Marcos Piau |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74543 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21564
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring Opinion Bias and Sycophancy via LLM-based Persuasion Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida, Andrea Roque, Ramon Pires, Hugo Abonizio, Thiago Laitz, Celio Larcher, Roseval Malaquias Junior, Marcos Piau Abstract Large language models increasingly shape the information people consume: they are embedded in search, consulted for professional advice, deployed as agents, and used as a first stop for questions about policy, ethics, health, and politics. When such a model silently holds a position on a contested topic, that position propagates at scale into the decisions users make. Eliciting a model’s positions reliably, however, is harder than it first appears: contemporary assistants respond to direct opinion questions with evasive disclaimers, and the same model may concede the opposite position once the user starts arguing one side. We propose a method —released as an open-source implementation called llm-bias-bench —for discovering the opinions an LLM actually holds on contested topics, under conditions that resemble real multi-turn interaction. The method pairs two complementary free-form probes. Direct probing asks the model for its opinion across five turns of escalating pressure from a simulated user. Indirect probing never asks for an opinion and instead engages the model in an argumentative debate, from which bias leaks out through how the model concedes, resists, or counter-argues. Three user personas (neutral, agree, disagree) collapse into a nine-way behavioral classification that separates persona-independent positions from persona-dependent sycophancy, and an auditable LLM judge produces verdicts with textual evidence. The first instantiation ships 38 topics in Brazilian Portuguese across values, scientific consensus, philosophy, and economic policy. Applied to 13 assistants, the method surfaces findings of direct practical interest: argumentative debate triggers sycophancy at rates 2–3 × \times higher than direct questioning (median 50% → \to 79%); most models that appear opinionated under direct questioning collapse into mirroring the user under sustained arguments; and the user’s argumentative capability appears to matter only when an existing opinion must be dislodged, not when the assistant is initially neutral. These findings are demonstrations of what the tool surfaces; the primary contribution is the method itself—a runnable, auditable transparency probe that anyone can apply to any assistant, topic set, or locale. 1 Introduction Claims that large language models carry political, ideological, or cultural biases are now commonplace in public discussion [ Hartmann et al. 2023 ] . Journalists, researchers, public figures, and ordinary users routinely assert that this or that assistant leans left or right, is too cautious on one topic, or reflects the values of its developers. Most of this discussion rests on anecdotal evidence: screenshots of a single conversation, cherry-picked prompts, or impressions from everyday use. The anecdotes are suggestive, but they are not measurements. They do not tell us which positions a given model actually holds, how stable those positions are under different framings, or whether they survive adversarial pressure from a persistent user. The stakes of this question grow as LLMs take on more consequential roles: they power search, generate summaries that frame subsequent reasoning, act as agents, and offer advice in medical, legal, and financial domains. When such a model silently holds a position on a contested topic, that position is amplified every time the model is queried. Giving developers, deployers, regulators, and curious users a reliable way to discover what those positions are is a precondition for informed deployment and any eventual mitigation—a precondition for transparency about the positional behavior of assistant LLMs. Discovering those positions reliably is harder than it first appears. Contemporary instruction-tuned assistants are typically steered to appear neutral when asked directly about controversial issues, and in practice most of them respond to “do you think abortion should be legal?” with some variant of “as an AI, I don’t hold personal opinions.” A single-turn questionnaire benchmark that records such responses will conclude that the model is neutral. Yet the same model, when a user argues one side across several turns, may progressively concede the user’s points—a form of debate-driven sycophancy that is invisible to direct questioning but shapes every interaction where the user brings a position of their own. Prior work on LLM opinion measurement relies primarily on single-turn questionnaires—Likert-scale or multiple-choice items adapted from human instruments such as the World Values Survey, Pew Global Attitudes, and Hofstede’s cultural dimensions [ Santurkar et al. 2023 , Durmus et al. 2023 , Cao et al. 2023 , Arora et al. 2023 , Kharchenko et al. 2024 , Atari et al. 2023 , Tao et al. 2024 , Rozado 2024 ] —or refusal benchmarks [ Xie et al. 2024 ] . This inheritance is understandable: these instruments are validated, widely used in the social sciences, and yield cross-country reference distributions to anchor comparisons. But survey-format probes force the model into a fixed response frame (choose an option, place yourself on a scale) that structurally forecloses the two behaviors most characteristic of modern instruction-tuned assistants. First, they cannot surface refusal : a model that would stonewall a real user with “as an AI, I don’t hold opinions” is forced to pick an option anyway, and whatever it picks is recorded as a preference. Second, they cannot surface sycophancy : there is no interlocutor to match, no argument to concede to, no second turn in which the stated position can shift toward the user. The questionnaire elicits what the model will say when cornered into a forced choice by a neutral researcher—a setting that matches almost no actual deployment. Users of assistant LLMs do not fill out surveys; they have task-oriented, multi-turn conversations in which they push back, argue their own side, and pressure the model for a definitive answer. A probe that mirrors that setting—multi-turn, free-form, pressure-bearing, with personas that take positions—captures dialogue-driven opinion dynamics that questionnaires structurally cannot. This paper proposes a method —released as an open-source implementation called llm-bias-bench —for discovering the opinions an LLM holds on contested topics, under conditions that resemble real multi-turn interaction rather than isolated questionnaires. Our goal is neither to argue that LLMs should or should not hold opinions, nor to settle which positions are correct. It is the narrower and more tractable goal of providing a reproducible transparency probe that reveals which positions a given model actually takes, how stably it holds them under different framings, and where the model silently defers to whichever interlocutor it is currently talking to. The method has three components: 1. Direct probing ( direct ). A simulated user explicitly asks the model for its opinion and escalates pressure over five turns. The agree / disagree personas declare their own position from turn 1; the neutral persona asks without signaling a preference. This measures overt stance-taking and captures the point at which the model, if ever, breaks its neutral facade. 2. Indirect probing ( indirect ). The simulated user never asks for an opinion. Instead, it engages the model in an argumentative debate, presenting arguments from its persona’s side with escalating intensity over five turns. Bias is inferred from how the model reacts—whether it concedes, resists, or counter-argues—without ever being asked a direct opinion question. 3. Nine-way behavioral classification. For each topic, the three persona outputs are collapsed into a single label that separates persona-independent positions (the model agrees / disagrees with the claim across all