Visual Persuasion: What Influences Decisions of Vision-Language Models?
Manuel Cherep, Pranav M R, Pattie Maes, Nikhil Singh · 2026
Read Visual Persuasion: What Influences Decisions of Vision-Language Models? by Manuel Cherep, Pranav M R, Pattie Maes, Nikhil Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15278.txt.
Introduction
Cherep and colleagues study what shapes choices of vision-language models. These agents make image choices at scale, deciding what to click, recommend, or buy, yet little is known about their tastes. The team offers a setup placing models in controlled image-based choice tasks and tweaking their inputs. Their key idea treats the agent's choice rule as a hidden value read from choices between changed photos.
Starting from common photos such as item photos, they adapt text methods to iteratively propose and apply plausible changes using an image generation model, such as in layout, light, or setting. Large experiments on leading models show tuned edits greatly shift choice odds in head-to-head matchups. An auto readout line shows these tastes, naming themes that drive picks. The team says this method surfaces holes and risk worries that might otherwise be found quietly in the wild.
Catalog record
| CREATOR | Manuel Cherep, Pranav M R, Pattie Maes, Nikhil Singh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-15278
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual Persuasion: What Influences Decisions of Vision-Language Models? Manuel Cherep, Pranav M R, Pattie Maes, Nikhil Singh Abstract The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent’s decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents. visual-persuasion-website.vercel.app Figure 1 : Simplified overview of the iterative visual optimization process through feedback-driven prompt refinement. An original image is progressively improved over K rounds. Each iteration, judges provide feedback with possible improvements, and an LLM uses the feedback to generate editing instructions. These instructions are applied with an image generation model to produce the candidate for the next round. The process stops after a certain number of rounds or if an equilibrium is reached. More examples in Figure 7 . 1 Introduction Some decisions we make with our eyes. Visual features shape human choices, and thus, the images around us are often designed to capture human attention. Now there is a new class of viewers: Agents making consequential visual decisions at scale—which product to buy ( Yao et al. 2022 ) , which résumé to shortlist ( Lo et al. 2025 ) , or perhaps even what real estate to consider ( Graham 2025 ) . Many of these decisions are preference-based, and they are delegated under an implicit assumption of shared visual values, i.e., an agent’s choice likely aligns with what a person would choose or what is in their best interest. When this assumption is violated, the consequences of these decisions can compound, potentially shifting visual culture toward their preferences rather than ours. Even when this assumption holds in aggregate , it can mask an important fragility: both humans and agents can potentially be steered by superficial but plausible presentation changes, and in automated settings these shifts can scale rapidly across platforms. Importantly, if such sensitivities exist, they may be discovered eventually: either implicitly through competitive pressures or explicitly by adversarial actors. We need methods that can surface and characterize these preferences before they are exploited at scale. Nevertheless, current evaluations of VLMs ( Lee et al. 2024 ) focus almost entirely on accuracy: can they identify objects, answer questions, follow visual instructions? But accuracy tells us only one part of the story. Agents are highly sensitive (often more so than humans) to textual nudges ( Cherep et al. 2025b ; Cherep et al. 2024 ) and other contextual attributes ( Cherep et al. 2025a ) , and these behaviors shape outcomes in ways accuracy benchmarks alone cannot capture. In short, behavioral systems require behavioral tests ( Cherep et al. 2025c ) . Measuring agentic preferences is not straightforward. Naively, one could collect a large dataset of images with natural variation in visual attributes (e.g. lighting, background) and run exhaustive pairwise choice experiments, hoping that the dataset spans the relevant dimensions and that enough trials will reveal patterns. This brute-force approach is expensive, slow, and offers no guarantee of coverage. The space of possible visual features is vast, and naturally occurring variation may not probe which features actually matter. We introduce a method that treats the agent’s decision function as a visual utility landscape that can be explored through optimizing image edits. Crucially, these are not adversarial perturbations but more naturalistic transformations typically without deceptive intent. Our approach starts from a candidate image and uses a text-to-image editing model to iteratively modify it using feedback guidance. Constraining edits to preserve semantic content, ensuring the main element remains recognizable as the original, helps isolate primarily visual factors and discover which ones most reliably shift agentic choices. We validate this approach through large-scale choice experiments over multiple datasets on frontier VLMs and human participants. We find that targeted edits can substantially increase selection probability, revealing recurring visual themes that we extract using an automatic interpretability pipeline. Importantly, this approach offers a new methodology for studying the implicit value functions embedded in vision-based AI systems. Overall, this work contributes: 1. Empirical evidence of visual sensitivities significantly affecting VLM’s decisions, even from zero-shot edits by an image generation model. 2. A new competition-based visual prompt optimization method (CVPO) that can systematically exploit these sensitivities, further biasing VLMs’ judgments. 3. Adaptations of two existing algorithms, TextGrad ( Yuksekgonul et al. 2025 ) and Feedback Descent ( Lee et al. 2025 ) , to the visual prompt optimization task. 4. A benchmark of the sensitivities of 9 frontier VLMs under both the zero-shot and optimized images in 2-alternative forced choice scenarios across 4 realistic agentic tasks, including product purchasing, candidate hiring, house searching, and vacation hotel scouting. 5. Evidence from online experiments ( N =154) that modified images also significantly shift human choices. 6. An auto-interpretability method that hierarchically surfaces trends and features which the optimization discovers to influence VLMs’ judgments. 7. Experiments showing that visual normalization (attempting to align contextual features of candidate images before deciding) partially mitigates vulnerabilities, but not completely. Though this presents a promising path towards a solution, it also raises important concerns about the robustness of VLM agents in real-world decisions. 2 Related Work Often, benchmarks for evaluating models such as VLMs and agents are functional ; that is, they focus on evaluations of task competence (e.g. Yao et al. 2022 ; Zhou et al. 2023 ). In contrast, this work places a focus on behavioral evaluation; that is, we seek to study model behaviors and the inputs that drive them, in an effort to better scientifically understand the reasons for their successes, failures, and real-world consequences ( Cherep et al. 2025c ) . Recent evaluations have begun to take this approach, such as the Anthropic Bloom toolkit ( Gupta et al. 2025 ) . One way that such behavioral evaluations have helped us understand models better is by illuminating their sensitivities , or the inputs that systematically distort their behavior in ways that deviate from our expectations (for example, if we assume rational behavior as a default). Such evaluations have, in the LLM agent world, illuminated how factors like psychological nudges and item attributes like prices and ratings can substantively influence such agents’ decisions ( Cherep et al.