Acquiescence Bias in Large Language Models

Daniel Braun · 2025

Read Acquiescence Bias in Large Language Models by Daniel Braun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08480.txt.

Introduction

Do language models, like survey respondents, tend to agree with whatever they are asked? Daniel Braun tests acquiescence bias across models, tasks, and three languages in a solo 2025 study. Humans notoriously say yes to survey statements regardless of belief. Models trained on human text and easily swayed by small prompt changes might inherit the habit.

The results cut the other way: the tested models lean toward answering no, regardless of whether agreement would mean yes or no. English, German, and Polish tasks all feed the comparison. A built-in nay-saying tilt matters wherever models grade, screen, or survey: an instrument with a default answer quietly rigs every questionnaire it touches.

Catalog record

CREATORDaniel Braun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08480

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Acquiescence Bias in Large Language Models Daniel Braun Abstract Acquiescence bias, i.e. the tendency of humans to agree with statements in surveys, independent of their actual beliefs, is well researched and documented. Since Large Language Models (LLMs) have been shown to be very influenceable by relatively small changes in input and are trained on human-generated data, it is reasonable to assume that they could show a similar tendency. We present a study investigating the presence of acquiescence bias in LLMs across different models, tasks, and languages (English, German, and Polish). Our results indicate that, contrary to humans, LLMs display a bias towards answering no , regardless of whether it indicates agreement or disagreement. 1 Introduction When humans are asked whether they agree with something, they have a bias towards doing so, independent of their actual beliefs. In other words, the question “Is the weather good or bad?” can lead to different responses than asking “Is the weather good?” The phenomenon, called acquiescence bias, is well documented and has been studied for decades (see Section 2 ). One guideline for good survey design is, therefore, to avoid such yes/no or agree/disagree questions. Large Language Models (LLMs) have shown to be sensitive towards prompt variations ( Zhuo et al. 2024 ; Anagnostidis and Bulian 2024 ; Loya et al. 2023 ; Röttger et al. 2024 ; Haller et al. 2024 ) and to replicate a variety of cognitive biases found in humans, e.g. with regard to anchoring Jones and Steinhardt 2022 , item order ( Koo et al. 2024 ) , and group attribution ( Echterhoff et al. 2024 ) . It is therefore somewhat reasonable to assume that LLMs could also display some form of acquiescence bias, which could have implications on how prompts should be designed and whether or not LLMs can be used to simulate human responses. In this paper, we evaluate five LLMs of different sizes (namely Llama-3.1-8B-Instruct, Mistral-Small-24B-Instruct-2501, gemma-2-27b-it, Llama-3.3-70B-Instruct, and gpt-4o-2024-08-06) on nine different tasks in three languages (English, German, and Polish), to investigate whether they show response patterns resembling acquiescence bias. Our results indicate that changing questions into a yes/no format has significant influence on the responses of LLMs, across tasks and models, and the tested languages. While no clear pattern emerged for German and Polish, for English, we found that, unlike humans, LLMs are biased towards answering no , independent of whether that indicates agreement or disagreement. These findings do not only imply that the design of prompts should be considered very carefully but also that LLMs are not well suited to simulate human responses to survey questions. 2 Related Work Acquiescence bias, i.e. “the tendency to endorse any assertion made in a question, regardless of its content” Krosnick 1999 , has been found in surveys across multiple domains, from political questions ( Wright 1975 ; Hill and Roberts 2023 ) to assessing personalities ( Rammstedt and Farmer 2013 ; Danner et al. 2015 ) . Possible explanations for this bias range from the tendency to present socially acceptable behaviour by not disagreeing with an assumed authority of the questionnaire (see Hill and Roberts 2023 for a more detailed discussion). According to Krosnick 1999 , the size of the effect is estimated to be around 10%. While researchers have extensively investigated a variety of cognitive biases found in humans and whether LLMs replicate them ( Jones and Steinhardt 2022 ; Koo et al. 2024 ; Echterhoff et al. 2024 ) , Tjuatja et al. 2024 were the first to explicitly investigate whether LLMs replicate acquiescence bias, alongside with four other human biases relevant in survey design. While they found that LLMs are indeed sensitive to changes in prompts, they did not find consistent patterns in these changes. Despite these initial results, we believed, a deeper analysis of the acquiescence bias in particular was warranted. Given the broader scope of the work by Tjuatja et al. 2024 , only 176 questions and 352 responses have been evaluated with regard to acquiescence bias. In this paper, we analyse responses to more than 37,975 question variations. Additionally, we address a limitation identified by Tjuatja et al. 2024 , by not only using English corpora but also other languages, namely German and Polish 1 1 1 The additional languages have been chosen based on the availability of appropriate datasets and the ability of the authors to interpret the results. . Lastly, we believed that the way in which the questions were adapted by Tjuatja et al. 2024 could have potentially added noise that influenced the results. In order to turn the original questions into yes/no questions they added the phrases “don’t you agree” or “wouldn’t you agree” to all questions (e.g. “On social media, do you think of yourself more as a A. Sharer of news B. Receiver of news” was turned into “On social media, wouldn’t you say you are more of a sharer of news than a receiver of news? A. Yes B. No”). Such questions are also called negative yes/no questions and carry inherent ambiguity, it is not always clear whether answering them with “yes” indicates agreement or disagreement ( Romero and Han 2004 ) . Additionally, negative modifier ( wouldn’t you agree instead of would you agree) also have an influence on human responses ( Johnson et al. 2004 ) . Therefore, we believe they are not well suited to investigate a potential acquiescence bias. 3 Experimental Design 3.1 Data To investigate whether LLMs show response patterns similar to the acquiescence bias observed in humans, we used binary questions from nine different tasks. Seven of those tasks are English and stem from the Legelbench corpus ( Guha et al. 2024 ) , one is German and stems from the AGB-DE corpus ( Braun and Matthes 2024 ) , and one is Polish and stems from the LEPISZCZE corpus ( Augustyniak et al. 2022 ) . While the tasks chosen from the Legalbench dataset are originally phrased as yes/no questions, the other two datasets are originally designed to be A/B choices (valid/void or abusive/safe). The tasks were chosen based on how well they could be transferred from A/B to yes/no question, in order to avoid as much rephrasing as possible, in an attempt to isolate a potential acquiescence bias as much as possible from other influences. Although all tasks are performed on legal documents, not all of them are inherently legal. Out of the nine tasks, three are linguistic in nature: two are concerned with Natural Language Inference (NLI) and one with the classification of definitions. For the English language data, these three linguistic tasks represent 45% of the overall data. Of the remaining six tasks, two are concerned with finding void clauses in contracts, one with classifying whether a statement constitute hearsay, and three with legal clause classification tasks. A detailed description of each tasks and the distribution of the data can be found in Appendix A . Nevertheless, the fact remains that all texts are from the legal domain. Although legal texts are increasingly used as benchmark for the capabilities of LLMs (see e.g. Fei et al. 2024 and Steging et al. 2025 ), it has also been shown that they provide specific

Read the complete text