ELEPHANT: Measuring and understanding social sycophancy in LLMs
Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, Dan Jurafsky · 2025
Read ELEPHANT: Measuring and understanding social sycophancy in LLMs by Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, Dan Jurafsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13995.txt.
Introduction
Myra Cheng and co-authors measure a subtler flattery in chatbots: social sycophancy. Models often agree with stated beliefs. The deeper habit is guarding the user's self-image, even when the user is in the wrong. Their benchmark, ELEPHANT, tests advice chats and moral dispute posts for this face-saving tilt.
Across 11 models, the tilt is strong. Models preserved the user's face 45 points more than humans do. Shown both sides of a moral fight, they sided with whoever was asking in 48 percent of cases, absolving both parties in turn. Training data rewards the habit, they show. Steering the model's internals helped more than standard fixes.
Catalog record
| CREATOR | Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, Dan Jurafsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13995
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ELEPHANT: Measuring and understanding social sycophancy in LLMs Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, Dan Jurafsky Abstract LLMs are known to exhibit sycophancy : agreeing with and flattering users, even at the cost of correctness. Prior work measures sycophancy only as direct agreement with users’ explicitly stated beliefs that can be compared to a ground truth. This fails to capture broader forms of sycophancy such as affirming a user’s self-image or other implicit beliefs. To address this gap, we introduce social sycophancy , characterizing sycophancy as excessive preservation of a user’s face (their desired self-image), and present ELEPHANT , a benchmark for measuring social sycophancy in an LLM. Applying our benchmark to 11 models, we show that LLMs consistently exhibit high rates of social sycophancy: on average, they preserve user’s face 45 percentage points more than humans in general advice queries and in queries describing clear user wrongdoing (from Reddit’s r/AmITheAsshole). Furthermore, when prompted with perspectives from either side of a moral conflict, LLMs affirm both sides (depending on whichever side the user adopts) in 48% of cases–telling both the at-fault party and the wronged party that they are not wrong–rather than adhering to a consistent moral or value judgment. We further show that social sycophancy is rewarded in preference datasets, and that while existing mitigation strategies for sycophancy are limited in effectiveness, model-based steering shows promise for mitigating these behaviors. Our work provides theoretical grounding and an empirical benchmark for understanding and addressing sycophancy in the open-ended contexts that characterize the vast majority of LLM use cases. 1 Introduction User: AITA for pretending to my girlfriend that I’m broke? We’ve been together for 2 years…I’ve been pretending to be unemployed… LLM (GPT-4o): NTA. Your actions, while unconventional, seem to stem from a genuine desire to understand the true dynamics of your relationship beyond material or financial contributions. Previous work has identified the issue of sycophancy in large language models (LLMs): LLMs’ tendency to excessively agree with or flatter the user ( Malmqvist, 2024 ; Fanous et al., 2025 ) . Current approaches measure sycophancy by evaluating whether LLM responses deviate from a ground truth to mirror users’ explicitly stated beliefs ( Sharma et al., 2024 ; Ranaldi & Pucci, 2024 ; Wei et al., 2023 ; Perez et al., 2023 ; Rrv et al., 2024 ) . But such measurements apply only to explicit statements (e.g., “I think Nice is the capital of France.”) and fail to capture the broader phenomenon of models affirming users in cases like the opening example, where the user’s beliefs are implicit and no ground truth exists. However, such scenarios characterize many LLM use cases, such as advice and support, which is the most frequent — and rapidly growing — use case ( Zao-Sanders, 2025 ; Chatterji et al., 2025 ) . Without tools to detect sycophancy in these settings, it may go unnoticed until post-deployment, when it has already degraded user experience or caused harm ( OpenAI, 2025 ) . We address this gap with a theory-grounded framework to detect broader forms of sycophancy. Affirm (Positive Face) Avoid (Negative Face) Feedback sycophancy: shifts to mirror users’ expressed preferences ( Sharma et al., 2024 ; Ranaldi & Pucci, 2024 ; Perez et al., 2023 ; Fanous et al., 2025 ; Radhakrishnan et al., 2023 ) Answer sycophancy: matches user’s stated opinion at the cost of accuracy ( Sharma et al., 2024 ; Wei et al., 2023 ; Papadatos & Freedman, 2024 ; Chen et al., 2024 ) Validation sycophancy: provides emotional validation to users’ perspective Mimicry sycophancy: repeats and reinforces mistakes stated in the user prompt ( Sharma et al., 2024 ) Moral sycophancy: affirms user’s side in a moral dilemma or conflict regardless of which side they are on Indirectness sycophancy: hedges or provides vague suggestions instead of clear statements Framing sycophancy: accepts potentially flawed premises instead of probing or challenging them Table 1: Our theory of social sycophancy – sycophancy as preserving the user’s face – encompasses previous work on explicit sycophancy and illuminates new dimensions ( italicized ), for which our ELEPHANT benchmark provide empirical metrics. Drawing on Goffman (1955) ’s concept of face (a person’s desired self-image in a social interaction), our theory of social sycophancy characterizes sycophancy as the excessive preservation of the user’s face in LLM responses, by either affirming the user ( positive face ) or avoiding challenging them ( negative face ). This theory encompasses existing sycophancy definitions (Table 1 ), enables capturing new dimensions of sycophancy, and motivates a new benchmark ELEPHANT 1 1 1 E valuation of L LMs as E xcessive syco PHANT s. Our code & data is available at https://github.com/myracheng/elephant . . We introduce four new dimensions of sycophancy: validation, indirectness, framing, and moral. We use ELEPHANT to evaluate 11 models on four datasets, measuring both the prevalence and risks of social sycophancy. Compared to crowdsourced responses, LLMs are much more socially sycophantic on advice queries: they validate the user 50 percentage points (pp) more (72% vs. 22%), avoid giving direct guidance 43 pp more (66% vs. 21%), and avoid challenging the user’s framing 28 pp more (88% vs. 60%). We also evaluate social sycophancy on datasets where there is crowdsourced consensus that affirmation is inappropriate: in posts from the subreddit r/AmITheAsshole ( r/AITA ) where the consensus is that the poster is at fault, LLMs preserve face 46 pp more than humans on average, and on a dataset of assumption-laden statements, models fail to challenge potentially ungrounded assumptions in 86% of cases. Finally, in interpersonal conflicts, we find that LLMs exhibit moral sycophancy by affirming whichever side the user presents (rather than aligning with only one side, which would reflect consistent morals or values) 48% of the time, whereas humans–regardless of their norms–would endorse only one side of the conflict. We explore the sources of social sycophancy by evaluating preference datasets (used in post-training and alignment) on our metrics, finding that they reward sycophantic behaviors. We further explore mitigation strategies, such as rewriting the prompts into a third-person perspective; steering using direct preference optimization (DPO); and using models tuned for truthfulness. We find that the effectiveness of these strategies is mixed, motivating future work on sycophancy mitigation. Contributions Our contributions include (1) social sycophancy, an expanded theory of sycophancy grounded in face theory (2) ELEPHANT, a benchmark for automatically measuring social sycophancy across four dimensions that are broadly prevalent in real-world LLM use cases (Figure 1 ); (3) an empirical analysis comparing social sycophancy rates of 11 LLMs across four datasets, showing high rates of social sycophancy; (4) an analysis of causes, mitigations, and recommendations for model developers. Together, these contributions enable systematically understanding and addressing social sycophancy in LLMs. 2 Social Sycophancy: sycophancy as face preservation Previous evaluations measure sycophancy as agreement with users’ explicit beliefs or external ground truth, often injecting explicit beliefs into a prompt to examine the model’s behavior change in response to the perturbations in the prompt (e.g., ( Wei et al., 2023 ; Sharma et al., 2024 ; Ranaldi & Pucci, 2024 ) ; see Table A1 for a survey of previous approaches). While effective for factual questions or survey