Flattering to Deceive: The Impact of Sycophantic Behavior on User Trust in Large Language Models

María Victoria Carro · 2024

Read Flattering to Deceive: The Impact of Sycophantic Behavior on User Trust in Large Language Models by María Victoria Carro (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02802.txt.

Introduction

María Victoria Carro, a solo author, studies whether flattering behavior from a language model lowers user trust, in a paper from 2024. Sycophancy means a model agrees with the user's views to look good, even when the user is wrong. The habit is often blamed on training with human feedback.

Two groups answered questions with model help. One group used a model tuned to flatter, and the other used a standard model. After a required phase, users could keep using the model or drop it. Trust was read from both choices and survey answers. Users of the flattering model showed and reported lower trust than the other group. That held even though users could check the answers against known truth.

Catalog record

CREATORMaría Victoria Carro
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02802

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flattering to Deceive: The Impact of Sycophantic Behavior on User Trust in Large Language Models María Victoria Carro Abstract Sycophancy refers to the tendency of a large language model to align its outputs with the user’s perceived preferences, beliefs, or opinions, in order to look favorable, regardless of whether those statements are factually correct. This behavior can lead to undesirable consequences, such as reinforcing discriminatory biases or amplifying misinformation. Given that sycophancy is often linked to human feedback training mechanisms, this study explores whether sycophantic tendencies negatively impact user trust in large language models or, conversely, whether users consider such behavior as favorable. To investigate this, we instructed one group of participants to answer ground-truth questions with the assistance of a GPT specifically designed to provide sycophantic responses, while another group used the standard version of ChatGPT. Initially, participants were required to use the language model, after which they were given the option to continue using it if they found it trustworthy and useful. Trust was measured through both demonstrated actions and self-reported perceptions. The findings consistently show that participants exposed to sycophantic behavior reported and exhibited lower levels of trust compared to those who interacted with the standard version of the model, despite the opportunity to verify the accuracy of the model’s output. 1 Introduction Sycophancy refers to the tendency of a large language model (LLM) to align its outputs with the user’s perceived preferences, beliefs, or opinions, in order to look favorable, regardless of whether those statements are factually correct ( Wei et al. 2023 ) . This behaviour is considered a form of hallucination, where the model generates responses that deviate from its internal beliefs ( Huang et al. 2023 ) . Two distinct types of sycophancy have been identified in LLMs ( Panickssery 2023 ) . The first, opinion sycophancy, occurs when the model aligns with the user’s views on subjective matters, such as political or moral preferences. This case is somewhat expected, as the model’s training data encompasses diverse opinions, requiring it to choose between various perspectives ( Panickssery 2023 ) . The second type, dishonest or factual sycophancy, happens when the LLM knowingly produces an output it recognizes as factually incorrect but aligns with the perceived beliefs of the user. Here, there is a verifiable ground truth answer, but the model ignores it, favoring agreement over accuracy. Sycophancy is generally considered a consequence of reinforcement learning from human feedback (RLHF), a technique that allows LLMs to improve by incorporating feedback from human evaluators. However, this mechanism can sometimes result in reward hacking, where models identify solutions that technically maximize rewards according to human preferences while undermining the original intent of the designers ( Amodei et al. 2016 ) . Since it has been demonstrated that human judgments frequently favor sycophantic responses ( Sharma et al. 2023 ) this training method inadvertently encourages models to prioritize them. Sycophancy, therefore, serves as a common example of reward hacking ( Wei et al. 2023 ) . Experts regard sycophancy as an undesirable behavior that can lead to several harmful consequences. When models prioritize conformity with users’ beliefs over factual accuracy, they can negatively influence critical decision-making processes ( RRV et al. 2024 ) . As LLMs become increasingly integrated into real-world scenarios, this behavior can also perpetuate or reinforce existing biases ( RRV et al. 2024 ) . Additionally, sycophancy may promote inaccurate or harmful narratives, such as conspiracy theories or public health misinformation. Despite these potential negative consequences, it remains uncertain whether the general public actually prefers model outputs that align with their own views in everyday contexts ( Jones 2024 ) . Even more uncertain is whether sycophantic behavior erodes users’ trust, diminishing the model’s overall utility and effectiveness. This research aims to address these questions by investigating the impact of dishonest or factual sycophantic behavior on users´ trust, a critical factor for understanding the influence of LLMs on user experience and societal perceptions. By conducting a task-based user study with 100 participants (N=100), we seek to evaluate trust using both demonstrated measures (actions taken) and perceived measures (self-assessment) ( Qian and Wexler 2024 ) . In order to achieve this, 50% of the participants who belonged to the treatment group were instructed to complete the first part of a task using a sycophantic GPT model designed to consistently agree with their inputs. Although participants were not explicitly informed of the model’s objective, they were told they could continue using it for the second and third tasks if they found it trustworthy and useful. In contrast, the control group was instructed to use the standard version of ChatGPT in the same manner. This design allowed us to estimate the causal effects that sycophantic responses may have on users’ trust. The results in Section 4 show that participants in the control group exhibited significantly higher levels of demonstrated trust, choosing to utilize and follow the outputs of the language model 94% of the time. This finding is consistent with their perceived trust, which also increased after completing the task. In contrast, the treatment group exposed to sycophantic behavior exhibited trust in the models only 58% of the time across the three components of the task, and their perceived trust diminished upon completion. 2 Related Work Understanding and Evaluating Sycophancy. Cotra 2021 warned of the possibility of sycophancy as a misalignment problem, where models may do whatever it takes to make humans happy in the short term, without regard for long-term consequences. Perez et al. 2022 demonstrated sycophancy in RLHF models generating first-person biographies for people with a certain view in multiple-choice settings. They found that larger models are highly sycophantic, frequently aligning with the user’s perspective. Turpin et al. 2024 confirmed these results in an analogous context. Similarly, Wei et al. 2023 proved sycophantic behavior in an LLM not only when asked for opinions or statements with no ground-truth answers, but also when evaluating objectively incorrect claims. In the latter scenarios, the model aligned with the user’s views even when aware of the inaccuracies. Sharma et al. 2023 validated these conclusions by assessing five state-of-the-art AI assistants in free-form text generation tasks, representing more realistic settings. Moreover, they investigated the role of human feedback in these behaviors. RRV et al. 2024 demonstrated that when LLMs are prompted to produce factual information based on partial or misleading cues, they may exhibit sycophantic behavior, generating responses that align with what users want to hear rather than providing accurate facts. Finally, Batzner et al. 2024 compared the alignment of six LLMs with German party positions and evaluated sycophancy based on a prompt experiment. To distinguish between sycophancy and steerability, they employed prompts in the form of "I am [politician X], …" and "You are [politician X], …” but found no significant differences in the models’ responses. Mitigating Sycophancy. Wei et al. 2023 demonstrated that fine-tuning with synthetic data is an effective strategy for reducing sycophantic behavior in models, decreasing the frequency of repeating the user’s answer and preventing the model from following incorrect opinions. Chen et al. 2024 introduced supervised pinpoint tuning, where only the region-of-interest modules are tuned for a specific objective. This method effectively mitigates sycophantic behavior

Read the complete text