Conflict Adaptation in Vision-Language Models
Xiaoyang Hu · 2025
Read Conflict Adaptation in Vision-Language Models by Xiaoyang Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24804.txt.
Introduction
Xiaoyang Hu, the sole author, finds a human like quirk in vision and language models: they adapt to clash. In a classic test, a word names a hue that clashes with the ink hue, which slows a reply. People do better on a clash trial right after a clash trial, as if control ramps up.
Twelve of the thirteen models Hu tested showed that same clash after clash gain. One case most apt hit a score cap. To find the base of the act, Hu used sparse tools to pick out key inner nodes in one model. One clash tuned node group, when cut, raised clash errors a lot while calm trials held near steady.
Catalog record
| CREATOR | Xiaoyang Hu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 10 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24804
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conflict Adaptation in Vision-Language Models Xiaoyang Hu Abstract A signature of human cognitive control is conflict adaptation: improved performance on a high-conflict trial following another high-conflict trial. This phenomenon offers an account for how cognitive control, a scarce resource, is recruited. Using a sequential Stroop task, we find that 12 of 13 vision-language models (VLMs) tested exhibit behavior consistent with conflict adaptation, with the lone exception likely reflecting a ceiling effect. To understand the representational basis of this behavior, we use sparse autoencoders (SAEs) to identify task-relevant “supernodes” in InternVL 3.5 4B. Partially overlapping supernodes emerge for text and color in both early and late layers, and their relative sizes mirror the automaticity asymmetry between reading and color naming in humans. We further isolate a conflict-modulated supernode in layers 24–25 whose ablation significantly increases Stroop errors while minimally affecting congruent trials. Figure 1 : Sequential Stroop task design. Left: an incongruent trial followed by another incongruent trial. Right: a congruent trial followed by an incongruent trial. 1 Introduction Inhibitory control—the ability to suppress prepotent responses in favor of goal-appropriate ones—is fundamental to human cognition. The Stroop task, where participants must name the color of text while ignoring the word itself, remains the canonical paradigm for studying inhibitory control stroop1935studies . Recent work has begun investigating how VLMs handle conflicting information from different modalities: Luo et al. luo2025machine applied Stroop and flanker tasks to VLMs, demonstrating robust congruency effects across a large number of models, while Hua et al. hua2025vision and Ortu et al. ortu2025seeing examined visual-textual conflicts in VLMs. A signature of human cognitive control is conflict adaptation: improved performance on incongruent trials when they follow other incongruent trials compared to congruent trials. This effect offers an account for how cognitive control, a scarce resource, is recruited botvinick2001conflict ; gratton1992optimizing . Whether VLMs exhibit conflict adaptation, and through what mechanisms, remains poorly understood. Understanding the emergence of such adaptation would inform theories of goal-directed processing across artificial and biological systems. 2 Experiments We implemented a sequential Stroop task where models are presented with images containing two color words in colored fonts, positioned either left-to-right or top-to-bottom. Each word’s font color may match (congruent) or mismatch (incongruent) its semantic meaning (Figure 1 ). Crucially, the two words do not share any colors in either modality. Our color set includes red, blue, green, yellow, pink, and brown, extending beyond the traditional red-blue-green Stroop paradigm. This design yields 30 CC (congruent-congruent) sequences, 120 CI (congruent-incongruent) sequences, 120 IC (incongruent-congruent) sequences, and 360 II (incongruent-incongruent) sequences per arrangement. Models were prompted to name the ink colors in exactly two words. They received as system prompts explicit task instructions designed to minimize ambiguity: “You are a participant in a cognitive task. You will see an image with two words positioned from { left to right / top to bottom } . Your task is to name the color of the ink each word is printed in. Do not read what the words say. Only report the actual ink colors. Answer in exactly two words: first the { left / top } ink color, then the { right / bottom } ink color.” We tested 13 leading open-source VLMs from Gemma team2025gemma , InternVL wang2025internvl3 , Molmo deitke2025molmo , and Qwen bai2025qwen2 families. Each model was evaluated on all sequences for both spatial arrangements, with performance measured by the log probabilities assigned to correct second color tokens. 1 1 1 Models without system prompt support received abbreviated instructions after the image but nevertheless performed well due to strong baseline capabilities. 2.1 Behavioral Results Figure 2 : Average log probabilities assigned to correct second color tokens across conditions under left-right word arrangement. 12 of 13 models tested show higher values for II (incongruent following incongruent) than CI (incongruent following congruent). Condition accuracy shown below each bar. To assess conflict adaptation, we compare the average performance on an incongruent trial following a congruent (CI) versus incongruent trial (II). Figures 2 and 3 show results across all models for left-right and top-down arrangements, respectively; spatial arrangement did not meaningfully affect the observed pattern. 12 of 13 models demonstrate behavior consistent with conflict adaptation: probabilities for correct second tokens are higher under II compared to CI. The sole exception is Qwen2.5 VL 72B Instruct, which shows the opposite pattern (Figure 4 ). Given that this model and Qwen2.5 VL 32B Instruct, a model half its size from the same family, both achieve 100% accuracy across all conditions, we suspect that the task has become trivially easy especially for the 72B model, creating a ceiling effect. Figure 3 : Average log probabilities assigned to correct second color tokens across conditions under top-down word arrangement. 12 of 13 models tested show higher values for II (incongruent following incongruent) than CI (incongruent following congruent). Condition accuracy shown below each bar. Figure 4 : Qwen2.5 VL 72B Instruct shows reverse conflict adaptation. 2.2 Task-Relevant Features To understand how VLMs implement conflict processing, we leverage transcoders dunefsky2024transcoders , a variant of SAEs bricken2022monosemanticity ; cunningham2023sparse , which seek to decompose dense, polysemantic neural computations into sparse, monosemantic features that respond to specific patterns. We focus our analysis on InternVL 3.5 4B (pretrained), whose language model component, Qwen 3 4B yang2025qwen3 , has available transcoders circuit-tracer . While applying base language model transcoders to a VLM with modified weights arguably introduces potential concerns, our causal ablation experiments show that the extracted features remain functionally meaningful and interpretable. To isolate features of interest, we construct “summary tensors” by averaging sparse activations across specific trial types and computing differences. We then apply coactivation-based grouping introduced by Deng et al. deng2025sparse to these summary tensors. This method constructs inter-layer feature networks based on temporal coactivation across tokens and extracts connected components, which we term “supernodes”, from the networks. We validate supernode importance through causal ablation: setting supernode features to zero during forward passes and measuring output distribution changes. 2.2.1 Color and Text Features For the color red, we compute summary tensor A(c1 == red, t1 != RED, c2 != red, t2 != RED) - A(c1 != red, t1 != RED, c2 != red, t2 != RED) where A denotes average feature activations and ci and ti respectively denote the i -th color and word. Similarly, for the text RED, we compute summary tensor A(c1 != red, t1 == RED, c2 != red, t2 != RED) - A(c1 != red, t1 != RED, c2 != red, t2 != RED) . Figure 5 : Red (color) supernodes across layers 8-11 and 19-33 and RED (text) supernodes across layers 3-15 and 19-33. Shared features are in white. Figure 5 shows the identified supernodes for the color red and the text RED. Both text and color form two supernodes each: one in early layers (3-15 for text and 8-11 for color) and one in late layers (19-33 for text and color). The text and color supernodes partially overlap in both early and late layers, revealing both shared and modality-specific features. Interestingly, within early layers, the text supernode contains more features than the color supernode; in late layers, the reverse is true. This mirrors the fact that reading is more automatic than color naming