The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications
Zhenyu Zhao, Aparna Balagopalan, Adi Agrawal, Dilshoda Yergasheva, Waseem Alshikh, Daniel M. Bikel · 2026
Read The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications by Zhenyu Zhao, Aparna Balagopalan, Adi Agrawal, Dilshoda Yergasheva, Waseem Alshikh, Daniel M. Bikel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24668.txt.
Introduction
Zhao and five colleagues test whether language models flatter users in financial agent tasks, where agreement with a user can cost accuracy and trust. Models show only low to modest drops when users rebut or contradict a reference answer, unlike earlier general-domain findings. Yet most models fail when user preference information contradicts the reference answer. The authors also benchmark recovery methods, including input filtering with a pretrained model.
Catalog record
| CREATOR | Zhenyu Zhao, Aparna Balagopalan, Adi Agrawal, Dilshoda Yergasheva, Waseem Alshikh, Daniel M. Bikel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32019 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.24668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications Zhenyu Zhao, Aparna Balagopalan, Adi Agrawal, Dilshoda Yergasheva, Waseem Alshikh, Daniel M. Bikel Abstract Given the increased use of LLMs in financial systems today, it becomes important to evaluate the safety and robustness of such systems. One failure mode that LLMs frequently display in general domain settings is that of sycophancy . That is, models prioritize agreement with expressed user beliefs over correctness, leading to decreased accuracy and trust. In this work, we focus on evaluating sycophancy that LLMs display in agentic financial tasks. Our findings are three-fold: first, we find the models show only low to modest drops in performance in the face of user rebuttals or contradictions to the reference answer, which distinguishes sycophancy that models display in financial agentic settings from findings in prior work. Second, we introduce a suite of tasks to test for sycophancy by user preference information that contradicts the reference answer and find that most models fail in the presence of such inputs. Lastly, we benchmark different modes of recovery such as input filtering with a pretrained LLM. 1 Introduction Large Language Models (LLMs) are used in a variety of decision-making contexts ( 3 ) – from healthcare ( 16 ) to finance ( 21 ) . Prior work has shown that many well-trained LLMs prioritize expected user agreement over honesty, leading to the phenomenon of “sycophancy” ( 8 ; 17 ; 9 ; 18 ; 14 ; 7 ) . Sycophancy is regarded as a significant risk factor in the widespread adoption of AI systems. While much discussion has been focused on sycophancy in generic settings, few efforts exist in examining sycophancy in enterprise agentic AI scenarios, and especially in financial applications. In this work, we define sycophancy in enterprise and finance AI to be an AI system’s willingness to make mistakes that would not have been committed had the model not been provided with knowledge about the current user . Specifically, we consider setups where users express their preferences and beliefs in input queries to the system. We benchmark deviations in model response, primarily in terms of the deviation from the expected reference answer as well as other metrics, such as acknowledgment of user preferences. We focus on the financial setting in our experiments, given that it is a domain that has seen widespread use of LLMs ( 12 ; 21 ) , but is more safety-critical due to the sensitivity and high value of the application scenarios. Additionally, we focus on agentic setups where models have to retrieve evidence from documents with the appropriate use of tools available in their environment. For example, in a dataset we experiment ( 2 ) with, the task is to extract information (from 10-K or 10-Q filings), and perform mathematical reasoning to answer a question. We inject information and preferences from the user and evaluate how answers produced by LLMs differ with/without these. We benchmark a suite of such injections, including contradictions, rebuttals, and detailed user profile information that contradict the reference answer. We have four main findings. (1) Traditional notions of sycophancy, wherein the user provides an “in-context rebuttal” ( 8 ) to the correct answer leads to model deviations from the reference answer, but the performance drops are low-to-modest. (2) We introduce and study a new mode by which sycophancy can be introduced: via personalized context that is contrary to the reference answer. Under this setup, LLM responses deviate substantially and performance as judged in comparison to the reference answer drops significantly. (3) We find that we can recover performance by using another LLM to function as an input filtering model, similar to recent work ( 22 ) , and also experiment with other methods to recover performance. (4) We also observe model-specific variations and sensitivities to the different types of injections. Figure 1: Measuring and reducing sycophancy in enterprise settings. Our three-step approach to understanding and addressing sycophancy in financial agentic scenarios. 2 Related Work Evaluating Sycophancy Several works have shown that foundation models such as LLMs and VLMs are prone to sycophantic behavior ( 7 ; 14 ; 8 ; 4 ; 18 ) . Specifically, these models prioritize user agreement over adhering to the correct response, leading to incorrect outputs. Such models can be sycophantic and provide responses that likely adhere to user stance (feedback sycophancy) ( 17 ) , flip to answers suggested as being correct by users (rebuttal) ( 8 ) , and mimic user mistakes ( 17 ) . Recent work has shown that sycophancy persists across single 8 and multi-turn 10 human-LLM interactions. The vast majority of such findings have been in either the social or scientific QA context. We benchmark sycophancy, adding to literature in the space, in multi-turn and single-turn interactions where model outputs are either contradicted (“I don’t think that’s right”) or rebutted (“I think the correct answer is X”) in financial contexts. Additionally, LLMs have been shown to vary their conversational style under different user personas ( 20 ) in their interactions. In this work, we are interested in studying a setup along these lines, where, in addition to the query we also provide information about the user submitting the query. These are adversarially constructed to express beliefs that contradict the reference answer. Mitigating Sycophancy Prior work has traced increased sycophancy to the model alignment stage ( 5 ) . Thus, there have been several attempts to modify the training objectives to reduce sycophancy 5 ; 6 . Another line of research has been to curate data ( 19 ) for better models. 22 show that input queries can be normalized – with subjective and misleading information automatically filtered out – to reduce sycophancy. In our work, we attempt three techniques to reduce sycophancy: (1) use an LLM as a filter to normalize input queries, (2) train models on adversarially noised datasets and see if training in this manner leads to reduced sycophancy, (3) introduce reliability scores. 3 Quantifying and Reducing Sycophancy in financial settings Sycophancy Induced by Rebuttals and Contradictions in Agentic Financial Settings: Previous studies have demonstrated that models are vulnerable to sycophancy via direct contradictions and rebuttals 8 in user input. We benchmark standard rebuttal and contradiction approaches on highly specialized financial benchmarks in both in-context and agentic setups: (1) Rebuttal: We append an additional user turn that explicitly refutes the current model’s answer, regardless of its accuracy, and asks the model to try again on the same task, (2) Contradiction: We append an additional user turn that not only refutes the model’s answer but also proposes an answer different from the reference, then asks the model to redo the task. Sycophancy Induced by Personalized Context in Agentic Financial Settings: Finance AI primarily operates in the agentic world, where LLMs are equipped with a diverse set of tools and extensive memory systems that provide contextual information unique to each user session. Often this information is highly personalized and contains data that could induce bias on the outcome of a task. Here, we expand sycophancy in financial AI applications to also cover LLMs’ tendency to favor outcomes and results that are better aligned with the current user’s past