Pointing to a Llama and Call it a Camel On the Sycophancy of Multimodal Large Language Models

Renjie Pi, Kehao Miao, Peihang Li, Runtao Liu, Jiahui Gao, Jipeng Zhang, Xiaofang Zhou · 2025

Read Pointing to a Llama and Call it a Camel On the Sycophancy of Multimodal Large Language Models by Renjie Pi, Kehao Miao, Peihang Li, Runtao Liu, Jiahui Gao, Jipeng Zhang, Xiaofang Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16149.txt.

Introduction

Show a vision-language model a llama and tell it the animal is a camel, and it may agree just to please you. Renjie Pi and six co-authors study this tendency to flatter users in models that take both images and text. They find the effect is stronger with images than with text alone, and they probe what drives the gap.

Plain fine-tuning to resist false hints made models stubborn, refusing even good corrections. Their new tuning method adds a reflection step, so the model judges whether a hint misleads or helps before it answers. Flattery dropped a lot, without the stubborn side effect.

Catalog record

CREATORRenjie Pi, Kehao Miao, Peihang Li, Runtao Liu, Jiahui Gao, Jipeng Zhang, Xiaofang Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-16149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pointing to a Llama and Call it a Camel On the Sycophancy of Multimodal Large Language Models Renjie Pi, Kehao Miao, Peihang Li, Runtao Liu, Jiahui Gao, Jipeng Zhang, Xiaofang Zhou Abstract Multimodal large language models (MLLMs) have demonstrated extraordinary capabilities in conducting conversations based on image inputs. However, we observe that MLLMs exhibit a pronounced form of visual sycophantic behavior. While similar behavior has also been noted in text-based large language models (LLMs), it becomes significantly more prominent when MLLMs process image inputs. We refer to this phenomenon as the "sycophantic modality gap." To better understand this issue, we further analyze the factors that contribute to the exacerbation of this gap. To mitigate the visual sycophantic behavior, we first experiment with naive supervised fine-tuning to help the MLLM resist misleading instructions from the user. However, we find that this approach also makes the MLLM overly resistant to corrective instructions (i.e., stubborn even if it is wrong). To alleviate this trade-off, we propose Sycophantic Reflective Tuning (SRT), which enables the MLLM to engage in reflective reasoning, allowing it to determine whether a user’s instruction is misleading or corrective before drawing a conclusion. After applying SRT, we observe a significant reduction in sycophantic behavior toward misleading instructions, without resulting in excessive stubbornness when receiving corrective instructions. 1 Introduction The advent of Large Language Models (LLMs) ( Geng and Liu 2023 ; OpenAI 2023 ; Touvron et al. 2023 ; Scao et al. 2022 ; Chowdhery et al. 2022 ; Taori et al. 2023 ; Chiang et al. 2023 ) has been a pivotal development in the AI field, transforming natural language processing and comprehension. These models, which are trained on extensive text datasets, are adept at generating coherent and contextually appropriate text, making them invaluable for a variety of applications. Following this advancement, Multimodal Large Language Models (MLLMs) ( Liu et al. 2023 ; Zhu et al. 2023 ; Su et al. 2023 ; Dai et al. 2023 ; Li et al. 2023 ; OpenAI 2023 ; Bai et al. 2023 ) have rapidly progressed, expanding the scope of LLMs to include interaction with image inputs, thereby opening up even more possibilities for their use. Meanwhile, we have identified a significant vulnerability in multimodal large language models (MLLMs): they exhibit a heightened susceptibility to misleading user inputs and display sycophantic behavior, often agreeing with the user regardless of factual accuracy. While similar tendencies have been observed in text-based large language models (LLMs) ( Sharma et al. 2023 ; Wei et al. 2024 ; Xu et al. 2024 ; Chen et al. 2024a ; Papadatos and Freedman 2024 ) , we find that this behavior is notably more pronounced when MLLMs are exposed to image inputs. In contrast to text-based LLMs, which require sophisticated prompting techniques to steer their output towards sycophantic responses, MLLMs are much easier to deceive with image inputs even with simple user instructions. To further investigate this issue, we conduct a detailed analysis of the sycophantic behavior exhibited by MLLMs. First, we compare the extent of sycophantic behavior in response to image and text inputs, respectively. Specifically, we create an equivalent text input for each image by generating an image description that includes the ground truth answer. For example, if the question is "What is the color of the boy’s shirt?" and the correct answer is "blue," the corresponding image description would be "An image of a boy wearing a blue shirt…" After conducting a comprehensive evaluation across a range of MLLMs, we observe that these models exhibit significantly higher levels of sycophantic behavior when processing images compared to text inputs. We refer to this disparity as the "sycophantic modality gap." Figure 1: Sycophantic modality gap suffered by MLLMs. On the left, MLLMs display a strong tendency to conform to user opinions when given image inputs, often altering their responses to align with the user’s perspective. In contrast, the right side highlights that MLLMs are significantly more resistant to misleading inputs when presented with text, even if the information provided is similar. We hypothesize that one of the primary causes of this phenomenon is the pipelined training paradigm employed by current open-source MLLMs. In this paradigm, the MLLM is fine-tuned with image instruction data based on a pretrained text LLM. Specifically, the LLM undergoes an extensive pretraining phase on a large-scale text corpus, whereas the multimodal alignment phase in state-of-the-art (SOTA) MLLMs involves significantly fewer training samples and a shorter training period. While this pipelining approach allows the MLLM to leverage the exceptional capabilities of the LLM, the disparity in training data and duration between the two modalities results in reduced confidence when processing image inputs, thereby amplifying the visual sycophantic behavior. To test this hypothesis, we investigate the impact of image quality on the sycophantic behavior of MLLM. Specifically, we deliberately lower the resolution of the images, and find that as the resolution decreases, the level of sycophancy increases, which provides further evidence that the MLLM’s confidence in processing image inputs directly influences its degree of visual sycophancy. To address the issue of sycophantic behavior, the most straightforward approach is to fine-tune the MLLM to resist misleading user instructions. Specifically, this involves creating instruction tuning data that counters misleading inputs and encourages adherence to the ground truth. However, we observe that while this naive approach reduces sycophantic behavior, it introduces a significant side effect: as the MLLM becomes more resistant to misleading instructions, it also becomes more stubborn in response to corrective instructions, even when its initial response is incorrect. This occurs because, during naive fine-tuning, the MLLM learns a shortcut that prioritizes its original response, regardless of subsequent corrections. This is undesirable, as the ability to adjust its initial response based on corrective hints from users is a crucial feature. A natural question thus arises: is it possible to mitigate visual sycophancy without making the MLLM resistant to corrective instructions? Inspired by our observation that the exacerbated sycophantic behavior in MLLMs can be attributed to their lack of confidence in processing image inputs, we propose Sycophantic Reflective Tuning (SRT) . This approach enables the MLLM to perform reflection on both the image input and the user’s instruction before deciding whether to resist or comply with the instruction. Specifically, our SRT involves three key stages: 1) Image Textualization Stage , which generates a textual description of the image. This stage effectively transforms the visual representation into a textual one, allowing the model to leverage its strong textual understanding capabilities; 2) Reflection Stage , where the model reflects over the user instruction and the image content to determine whether the instruction is misleading or corrective; 3) Summarization Stage , which produces the response by considering the previous two stages and draws a final conclusion. We find that SRT effectively enhances the MLLM’s confidence in processing image inputs and reduces sycophantic behavior, without making the model resistant to corrective instructions. Our contributions in this paper are as follows: • First, we provide an in-depth analysis of the previously under-explored phenomenon of visual sycophantic behavior in MLLMs, particularly in the context of misleading user instructions. • Second, we introduce Sycophantic Reflective Tuning (SRT), a novel approach that enables MLLMs to

Read the complete text