VISOR++:Universal Visual Input based Steering for Large Vision Language Models

Ravikumar Balakrishnan, Mansi Phute · 2025

Read VISOR++:Universal Visual Input based Steering for Large Vision Language Models by Ravikumar Balakrishnan, Mansi Phute (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25533.txt.

Introduction

Ravikumar Balakrishnan and Mansi Phute steer vision language models with a single image in a 2025 paper. Prompts can be overridden, and direct control needs access to model internals. Closed services allow neither route. The pair craft images that push a model toward a chosen stance.

One image can serve many models at once, with no runtime access. Tests on two open models covered refusal, flattery, and self preservation stances. Crafted images matched the effect of direct internal control in both directions. Early tests show shifts on unseen models, even closed ones. Scores on a large general test set stayed near intact, which suggests narrow, targeted control.

Catalog record

CREATORRavikumar Balakrishnan, Mansi Phute
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25533

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VISOR++:Universal Visual Input based Steering for Large Vision Language Models Ravikumar Balakrishnan, Mansi Phute Abstract As Vision Language Models (VLMs) are deployed across safety-critical applications, understanding and controlling their behavioral patterns has become increasingly important. Existing behavioral control methods face significant limitations: system prompting is a popular approach but could easily be overridden by user instructions, while applying activation-based steering vectors requires invasive runtime access to model internals, precluding deployment with API-based services and closed-source models. Finding steering methods that transfer across multiple VLMs is still an open area of research. To this end, we introduce universal visual input based steering for output redirection (VISOR++), a novel approach that achieves behavioral control through optimized visual inputs alone. We demonstrate that a single VISOR++ image can be generated for an ensemble of VLMs that by itself can emulate each of their steering vectors. By crafting universal visual inputs that induce target activation patterns, VISOR++ eliminates the need for runtime model access while remaining deployment-agnostic. This means that when an underlying model supports multimodal capability, model behaviors can be steered by inserting an image input completely replacing runtime steering vector based interventions. We first demonstrate the effectiveness of the VISOR++ images on open-access models such as LLaVA-1.5-7B and IDEFICS2-8B along three alignment directions: refusal, sycophancy and survival instinct. Both the model-specific steering images and the jointly optimized images achieve performance parity closely following that of steering vectors for both positive and negative steering tasks. We also show the promise of VISOR++ images in achieving directional behavioral shifts for unseen models that include both open-access and closed-access models. At the same time, VISOR++ images are able to preserve 99.9% performance on 14,000 unrelated MMLU evaluation tasks highlighting their specificity to inducing only behavioral shifts. 1 Introduction Vision-Language Models (VLMs) process both images and text to enable applications ranging from visual question answering and image captioning to multimodal reasoning and code generation from screenshots ( Achiam et al. 2024 ; Touvron et al. 2023 ) . These models are increasingly deployed in production systems, including safety-critical domains like healthcare, autonomous systems, and content moderation, where they at times outperform text-only models even on purely textual tasks due to their richer pre-training. As VLMs become core infrastructure for both multimodal and text-based applications, ensuring their behavioral alignment and resistance to adversarial manipulation becomes essential for preventing harmful outputs and maintaining system reliability. Researchers have developed methods for bypassing alignment in Large Language Models (LLMs), including prompt engineering ( Liu et al. 2023 ) , adversarial suffixes ( Zou et al. 2023 ) , and steering vectors ( Turner et al. 2023 ; Panickssery et al. 2023 ) . Numerous attacks targeting VLMs have been explored, including manipulation of image embeddings, adversarial patching, prompt injection, and inpainting techniques ( Bailey et al. 2023 ; Qi et al. 2023 ; Shayegani et al. 2023 ) . Steering vectors, in particular, function by manipulating the activation space of a model. A popular steering technique involves computing the steering vector as the difference between the activations corresponding to the undesired and desired outputs. When added to the model’s activation layers during inference, it induces targeted behavioral shifts. While powerful, the practical application of steering vectors is fundamentally constrained by their requirement for white-box access to model internals, including the need to compute and manipulate activations at runtime, an assumption that does not hold in many realistic settings. The above limitation is significant since, on the one hand, inaccessibility of model internals in production systems creates a false sense of security against activation-based attacks. On the other hand, the applicability of guard-railing using steering becomes severely restricted since the majority of the VLMs are served via APIs without access to inference pipelines. In order to make the steering techniques for VLMs practicable, we introduce VISOR++ ( V isual I nput based S teering for O utput R edirection), a technique that optimizes perturbations in the input image space to mimic the behavior of steering vectors in the latent activation space. We successfully demonstrate the existence of universal images that can steer model behavior across a range of input text prompts for three different behavioral dimensions. We show that both per-model steering images as well as a single image trained across an ensemble of models can achieve similar levels of steering as their corresponding steering vectors in most cases. Additionally, we show promise in terms of transferability of steering images to unseen models even when trained under a limited ensemble size of 2, especially when trying to induce negative behavior. We believe our findings provide interesting insights towards understanding the relationship between visual inputs and model hidden states and helps take a firm step towards developing truly transferable behavioral steering images. Figure 1: Conventional Steering techniques apply steering vector(s) addition to one or more model layers and even potentially at specific token positions to induce steering effects and must be model specific. VISOR++ operates strictly in the input space and can be passed along with the input prompt to induce the same steering effect across potentially several models. The significant contributions of VISOR++ are the following: 1. Visual Input based Steering: We shift the steering mechanism from the model supply chain to the visual input domain. We show that carefully optimized images can replicate the effects of the activation space steering and enable practical deployment without requiring runtime access to model internals. 2. Universal Steering over key behavioral dimensions: We showcase the effectiveness and universality of steering by using the same image to influence the model behavior for a range of inputs for each of the three behavioral dimensions including refusal, sycophancy and survival instinct. At the same, we show that such images don’t negatively influence VLM performance on unrelated tasks (e.g., MMLU benchmark). 3. Generality and Transferability: A single steering image effectively achieves steering for two distinct model architectures. Furthermore, those same images can clearly influence behavioral steering directions on unseen models providing promise for fully transferable steering images when expanded to larger ensembles. 2 Related Work 2.1 Steering in LLMs Steering vectors in LLMs have been used to modify LLM output to reflect desired behavior. A popular method of computing such steering vectors is by finding the difference of activations induced in the model by contrastive pairs of prompts ( Cao et al. 2024 ; Panickssery et al. 2023 ; Wu et al. 2025 ) . These “contrastive” pairs represent two opposing concepts (e.g., compliance and refusal, sycophancy and disparagement). Researchers have found that adding such vectors to models’ hidden states can alter model sentiment, toxicity, and topics in GPT-2-XL without any further optimization ( Turner et al. 2023 ) . Contrastive Additive Addition (CAA) ( Panickssery et al. 2023 ) demonstrated robust control of sycophancy, hallucination, and corrigibility. Recent work addresses basic steering limitations: GCAV ( Cao et al. 2025 ) manages multi-concept interactions through input-specific weights. Feature Guided Activation Additions (FGAA) ( Tennenholtz et al. 2025 ) use Sparse Autoencoder features for precise control. Style vectors effectively control writing style ( Konen et al. 2024 ) . These approaches improve upon naive vector addition but increase complexity. Researchers have also found high variability in steering effectiveness across

Read the complete text