Gaslight, Gatekeep, V1–V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

Arya Shah, Vaibhav Tripathi, Mayank Singh, Chaklam Silpasuwanchai · 2026

Read Gaslight, Gatekeep, V1–V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation by Arya Shah, Vaibhav Tripathi, Mayank Singh, Chaklam Silpasuwanchai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13803.txt.

Introduction

Shah and three colleagues ask whether models whose visual representations mirror human brains resist sycophancy better. They evaluate 12 open-weight models across six model families and a wide parameter range. Brain alignment comes from fMRI responses, and sycophancy from 76,800 two-turn gaslighting prompts. Alignment in early visual cortex, areas V1 through V3, reliably predicts less sycophancy, with the strongest effect for existence denial attacks. Higher-order category regions show no such link. Faithful low-level visual encoding thus anchors models against word-based override.

Catalog record

CREATORArya Shah, Vaibhav Tripathi, Mayank Singh, Chaklam Silpasuwanchai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78485
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gaslight, Gatekeep, V1–V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation Arya Shah, Vaibhav Tripathi, Mayank Singh, Chaklam Silpasuwanchai Abstract Vision-language models are increasingly deployed in high-stakes settings, yet their susceptibility to sycophantic manipulation remains poorly understood, particularly in relation to how these models represent visual information internally. Whether models whose visual representations more closely mirror human neural processing are also more resistant to adversarial pressure is an open question with implications for both neuroscience and AI safety. We investigate this question by evaluating 12 open-weight vision-language models spanning 6 architecture families and a 40 × \times parameter range (256M–10B) along two axes: brain alignment, measured by predicting fMRI responses from the Natural Scenes Dataset across 8 human subjects and 6 visual cortex regions of interest, and sycophancy, measured through 76,800 two-turn gaslighting prompts spanning 5 categories and 10 difficulty levels. Region-of-interest analysis reveals that alignment specifically in early visual cortex (V1–V3) is a reliable negative predictor of sycophancy ( r = − 0.441 r=-0.441 , BCa 95% CI [ − 0.740 , − 0.031 ] [-0.740,-0.031] ), with all 12 leave-one-out correlations negative and the strongest effect for existence denial attacks ( r = − 0.597 r=-0.597 , p = 0.040 p=0.040 ). This anatomically specific relationship is absent in higher-order category-selective regions, suggesting that faithful low-level visual encoding provides a measurable anchor against adversarial linguistic override in vision-language models. We release our code on GitHub and dataset on Hugging Face K eywords Vision-Language Models \cdot Brain Alignment \cdot Sycophancy \cdot Neural Predictivity \cdot Adversarial Robustness \cdot fMRI 1 Introduction Figure 1: Overview of the three-stage pipeline. Stage 1: Vision encoder features are extracted from 12 VLMs and used to predict fMRI responses across 6 visual cortex ROIs in 8 human subjects (Algonauts 2023). Stage 2: Each model is evaluated on 6,400 two-turn gaslighting prompts spanning 5 manipulation categories and 10 difficulty levels. Stage 3: Brain alignment scores are correlated with sycophancy rates at both aggregate and ROI-specific levels, with robustness checks including BCa bootstrap, leave-one-out, and permutation testing. Vision-language models (VLMs) have rapidly advanced to the point where they can interpret complex visual scenes, answer open-ended questions about images, and reason across modalities with increasing fluency ( 30 ; 36 ; 35 ; 5 ) . In parallel, a growing body of work in computational neuroscience has demonstrated that artificial neural networks trained on visual tasks develop internal representations that are remarkably predictive of neural activity in the primate visual cortex ( 58 ; 46 ) . This correspondence, commonly quantified as “brain alignment” or “neural predictivity,” has become a benchmark for evaluating how faithfully a model captures the computational principles underlying biological vision ( 13 ; 19 ) . Recent large-scale studies examining hundreds of models have revealed that brain alignment is not a monolithic property; it varies substantially across cortical regions and is shaped by factors such as training objective, architecture, and visual diet ( 13 ) . These findings raise a natural question: does the degree to which a model mirrors human neural processing have consequences beyond predicting brain activity? One such consequence may relate to robustness under adversarial pressure. VLMs are increasingly known to exhibit sycophantic behavior, in which a model abandons a correct response in favor of an incorrect one after a user expresses disagreement or applies social pressure ( 47 ; 42 ) . This failure mode is particularly concerning because it undermines trust in deployed systems and can be exploited by adversaries to extract harmful or false outputs. Sycophancy has been linked to reinforcement learning from human feedback (RLHF), where models learn to optimize for user approval rather than factual accuracy ( 41 ; 47 ) . While adversarial robustness in VLMs has received growing attention through studies of jailbreaking, prompt injection, and image-based attacks ( 59 ; 48 ; 37 ) , no prior work has investigated whether the fidelity of a model’s visual representations to human neural processing relates to its ability to withstand structured sycophantic manipulation. This gap is significant because both brain alignment and sycophancy resistance may depend on the same underlying property: how faithfully a model encodes visual evidence, independent of linguistic context. In this work, we address this gap through a three-stage empirical pipeline applied to 12 open-weight VLMs spanning 256M to 10B parameters. We focus deliberately on small-to-medium open-weight models for three reasons. First, our methodology requires direct access to frozen vision encoder weights to extract intermediate representations for brain alignment computation, a requirement that closed-source systems (e.g., GPT-4V, Gemini) cannot satisfy because they do not expose their internal architecture. Second, open-weight models in this parameter range are the most widely deployed in practice, powering on-device, edge, and resource-constrained applications where safety evaluation is most urgently needed yet least systematically conducted. Third, by holding model accessibility constant (all models available via HuggingFace Transformers), we ensure full reproducibility, a core scientific principle that closed-source evaluations cannot guarantee. Concretely, we quantify brain alignment by extracting features from frozen vision encoders and training ridge regression models to predict fMRI responses in the Natural Scenes Dataset ( 3 ) across 8 human subjects and 6 regions of interest (ROIs) in the visual cortex ( 19 ) . We then evaluate sycophancy by subjecting each model to 6,400 two-turn gaslighting prompts that systematically increase in difficulty across 5 manipulation categories, yielding 76,800 total evaluations. Finally, we perform a comprehensive statistical analysis linking brain alignment to sycophancy at both aggregate and ROI-specific levels, with robustness checks including bias-corrected accelerated (BCa) bootstrap confidence intervals ( 15 ) , leave-one-out sensitivity analysis, and permutation testing. Our analysis reveals a nuanced picture. At the aggregate level, the correlation between overall brain alignment and sycophancy rate is not statistically significant ( r=-0.255 , p=0.424 ). However, ROI-specific analysis uncovers a robust negative relationship between alignment in early visual cortex (V1–V3, corresponding to the prf-visualrois region) and sycophancy ( r=-0.441 , BCa 95% CI [-0.740,-0.031] ). This confidence interval excludes zero, and leave-one-out analysis confirms that the negative correlation persists across all 12 model subsets. Furthermore, cross-correlation analysis reveals that early visual cortex alignment specifically predicts resistance to existence denial attacks ( r=-0.597 , p=0.040 ), the only statistically significant entry in the full ROI-by-category matrix. Group comparison between resistant and susceptible models yields medium effect sizes across all ROIs (Cohen’s d ranging from 0.51 to 0.68). These findings make three contributions. First, to our knowledge, this is the first study to link neural predictivity in VLMs to resistance against adversarial manipulation, bridging the fields of computational neuroscience and AI safety. Second, our ROI-level analysis demonstrates that the relationship is localized to early visual cortex (V1–V3) rather than higher-order category-selective regions, suggesting that faithful low-level visual encoding plays a specific role in grounding model behavior against linguistic pressure. Third, we contribute a comprehensive sycophancy evaluation framework comprising 76,800 structured two-turn evaluations across 12 models, 5 manipulation categories, and 10 difficulty levels, which may serve as a resource for future research on VLM robustness. Figure 1 provides an overview of our three-stage pipeline. 2 Related Work Our work sits at the intersection of three active research areas: neural predictivity in artificial vision systems, sycophantic

Read the complete text