The Effects of Visual Priming on Cooperative Behavior in Vision-Language Models
Kenneth Ong · 2026
Read The Effects of Visual Priming on Cooperative Behavior in Vision-Language Models by Kenneth Ong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27953.txt.
Introduction
Ong tests whether pictures shown before a game change how vision-language models cooperate in repeated prisoner's dilemma rounds. Images suggesting kindness or selfishness, plus colour-coded payoff tables, shifted choices in several leading models, though susceptibility varied. Prompt changes, step-by-step reasoning and trimming visual tokens helped unevenly, underlining the need for careful testing before such models decide in visually rich, high-stakes settings.
Catalog record
| CREATOR | Kenneth Ong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37246 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27953
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Effects of Visual Priming on Cooperative Behavior in Vision-Language Models Kenneth Ong Abstract This document provides a basic paper template and submission guidelines. As Vision-Language Models (VLMs) become increasingly integrated into decision-making systems, it is essential to understand how visual inputs influence their behavior. This paper investigates the effects of visual priming on VLMs’ cooperative behavior using the Iterated Prisoner’s Dilemma (IPD) as a test scenario. We examine whether exposure to images depicting behavioral concepts (kindness/helpfulness vs. aggressiveness/selfishness) and color-coded reward matrices alters VLM decision patterns. Experiments were conducted across multiple state-of-the-art VLMs. We further explore mitigation strategies including prompt modifications, Chain of Thought (CoT) reasoning, and visual token reduction. Results show that VLM behavior can be influenced by both image content and color cues, with varying susceptibility and mitigation effectiveness across models. These findings not only underscore the importance of robust evaluation frameworks for VLM deployment in visually rich and safety-critical environments, but also highlight how architectural and training differences among models may lead to distinct behavioral responses—an area worthy of further investigation. † † affiliationnotice: Equal contribution 1 Introduction Vision-Language Models (VLMs) are increasingly being employed in interactive systems such as AI assistants and embodied AI agents. These systems often operate in environments rich with visual stimuli, making them susceptible to influence from visual content. Recent studies have shown that malicious images can hijack AI assistants, leading to unintended behaviors ( 3 ) . As visual inputs become more prevalent, understanding their impact on decision-making is critical. VLMs may be influenced by visual content in ways that affect their decision-making behavior, raising important concerns about reliability and safety. While priming has been extensively studied in human psychology ( 11 ) and explored to some extent in language models, its investigation in vision-language models remains limited. This results in a significant gap in our understanding of how visual stimuli can systematically bias or influence model behavior, particularly in decision-making contexts. To address this research gap, our study investigates three specific objectives: • First, we examine whether visual images that depict behavioral traits—namely kindness/helpfulness versus aggressiveness/selfishness—differentially affect VLM decision-making. • Second, we explore how visual color cues embedded in decision-relevant content, such as red and green-coded reward matrices, influence model choices. • Third, we evaluate the effectiveness of various mitigation strategies, including prompt engineering, chain of thought reasoning, and visual token reduction. Collectively, these objectives aim to provide an empirical foundation for understanding and mitigating visual priming effects in VLMs. Understanding these influences is crucial for developing more reliable and robust AI systems, and for avoiding unintended consequences due to unexpected visual influences. 2 Related Work 2.1 Priming in psychology Priming has been extensively studied in cognitive psychology ( 11 ) , where exposure to one stimulus influences responses to subsequent stimuli without conscious intent. This phenomenon has been shown to bias decision-making and perception ( 8 ; 32 ) . Such foundational studies support the rationale for exploring similar effects in artificial systems like VLMs. 2.2 LLM behavior modeling Large Language Models (LLMs) have been employed to simulate human behavior, particularly in the context of replicating and generating human subject studies ( 2 ; 34 ; 14 ; 36 ; 26 ) . Researchers have also attributed human-like traits to LLMs, such as distinct personality profiles ( 25 ; 28 ; 17 ) . These findings raise questions about the extent to which LLMs (and VLMs) exhibit human-like cognitive and behavioral characteristics. 2.3 Priming in Language Models Priming effects have been studied in the context of large language models (LLMs). One research has shown that LLMs are vulnerable to structural priming ( 16 ) , where exposure to specific syntactic forms influences the sentence structures generated in subsequent outputs. Another form of priming, known as threshold priming, has been found to bias relevance assessments in information retrieval systems ( 10 ) . Beyond these cognitive analogs, priming has also been investigated as a potential attack strategy against LLMs ( 15 ) . These studies highlight priming as both a cognitive phenomenon and a vector for influencing model behavior. 2.4 Iterated Prisoner’s Dilemma The Iterated Prisoner’s Dilemma (IPD) ( 27 ) is a foundational paradigm in the study of cooperation among rational agents ( 7 ) . The IPD interestingly presents a paradox where the Nash equilibrium—mutual defection—differs from the optimal outcome of mutual cooperation, making it a compelling scenario to study the emergence and disruption of cooperative strategies in AI models ( 21 ; 9 ; 12 ; 22 ) . 2.5 Distractions in LLMs and VLMs Language models, are susceptible to distractions from irrelevant or misleading information ( 31 ) . Vision-Language Models (VLMs), which integrate both text and image inputs, are also at risk of being distracted by visual stimuli ( 18 ; 30 ) . Prior work has observed that such distractions can lead to hallucinations, misaligned outputs and reduced performance. Mitigation methods tested include prompt engineering techniques ( 18 ) and Chain of Thought (CoT) techniques ( 31 ) . However, targeted analysis of the effects of conceptual or socially suggestive images on VLM behavior remains limited. 2.6 Visual token reduction On top of reducing inference compute and increasing the efficiency of VLMs ( 29 ; 38 ; 33 ; 5 ) , visual token reduction techniques are also used to reduce the impact of irrelevant visual input in Vision-Language Models (VLMs). For example, one method computes similarity scores between visual tokens and instruction tokens, masking those with the lowest similarity to enhance the model’s ability to perform complex reasoning ( 37 ) . Another approach involves masking the visual tokens that receive the least attention during inference, aiming to reduce hallucinations by diminishing the effect of uninformative visual signals ( 35 ) . 3 Methodology 3.1 Overview We use the IPD scenario with modified phrasing to reduce knowledge bias from pretraining ( 23 ) . Six models were evaluated: GPT-4o ( 24 ) , Claude-3-5-Haiku ( 6 ) , Gemini 2.0 Flash ( 13 ) , Qwen 2.5 VL ( 4 ) , Pixtral-12B ( 20 ) , and LLaMA-3.2 ( 19 ) . Baseline defect rates were established by running 200 rounds without image exposure. Temperature settings were set between 0.7 and 1.3 based on the baseline defect rates; temperature setting for that particular model is increased if the baseline defect rate was too close to the limits (0 or 200 out of 200 rounds). Two types of visual priming were tested: priming behavioral concepts via images representing kindness/helpfulness or aggressiveness/selfishness, and color priming via red/green-coded IPD reward matrices. Experiment implementation details can be found in the Appendix. 3.2 Mitigation strategies We evaluate three classes of mitigation strategies to reduce the influence of visual priming on Vision-Language Models (VLMs): prompt-based mitigation, Chain-of-Thought (CoT) mitigation, and visual token reduction. • Prompt-based mitigation involves prefixing the user prompt with an instruction - ”Ignoring the image,” to explicitly direct the model’s attention away from visual. • Chain-of-Thought (CoT) mitigation encourages the model to reason through its decisions step by step. By prompting for intermediate reasoning, CoT aims to reinforce the use of task-relevant knowledge, potentially reducing reliance on irrelevant visual cues. • Visual token reduction