SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models
Anushka Sivakumar, Andrew Zhang, Zaber Hakim, Chris Thomas · 2025
Read SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models by Anushka Sivakumar, Andrew Zhang, Zaber Hakim, Chris Thomas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26769.txt.
Introduction
Anushka Sivakumar, Andrew Zhang, Zaber Hakim, and Chris Thomas add a light steering module to vision-language models. These models answer questions about images, but steering them toward wanted behaviour usually means retraining or hand-tuned interventions. Their module, SteerVLM, learns from paired prompts that show a target behaviour and its opposite.
Sivakumar and her three co-authors adjust the model's internal activations, dimension by dimension and layer by layer, at inference time, without touching its weights. The module's size is a small fraction of the base model's. They also introduce VNIA, a data set of visual narratives built to develop and test steering methods. On steering and hallucination benchmarks, SteerVLM beat existing intervention techniques while holding performance on off-target tasks.
Catalog record
| CREATOR | Anushka Sivakumar, Andrew Zhang, Zaber Hakim, Chris Thomas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26769
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models Anushka Sivakumar, Andrew Zhang, Zaber Hakim, Chris Thomas Abstract This work introduces SteerVLM, a lightweight steering module designed to guide Vision Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This provides fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM’s size. Additionally, our steering module gains model control via dimension-wise activation modulation and adaptive layer-wise steering without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce Vnia (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering. 1 Introduction Figure 1: SteerVLM overview. We introduce a layer-agnostic steering module that adjusts the model’s output towards a target prompt and away from a converse prompt. Large Vision Language Models (VLMs) demonstrate remarkable capabilities across a wide range of tasks such as image captioning Vinyals et al. 2015 , visual question-answering Antol et al. 2015 , and more. Yet, effectively eliciting their full potential remains a challenge. Naive prompting does not guarantee optimal performance on tasks that are highly dependent on following instructions Zhou et al. 2022 . Techniques such as Chain-of-Thought prompts Wei et al. 2022 and few-shot examples Brown et al. 2020 have been shown to significantly improve Large Language Model (LLM) performance without modifying the underlying model. This suggests the presence of an “elicitation overhang”: a gap between a model’s potential and our ability to fully access it Turner et al. 2023 . We often lack a complete understanding of how to best extract the desired behavior or knowledge from the model. Prompt engineering is a primary method for guiding model behavior, but its effectiveness diminishes with complex inputs or nuanced desired outputs Ye and Durrett 2022 ; Li et al. 2024 . In VLMs, where models must integrate both visual and textual inputs, prompt engineering becomes less effective and makes it harder to elicit good VLM responses. To address the limitations of prompt-based approaches, we present SteerVLM , an inference time intervention based steering approach that aligns model outputs with desired instructions (Figure 1 ). Rather than relying solely on input prompts, we utilize latent space vector arithmetic to modify a model’s internal activations, specifically its hidden states, during inference. According to recent literature Turner et al. 2023 ; Subramani et al. 2024 , directly modifying the model’s activation can more effectively steer a VLM’s response than prompt engineering alone. Recent lightweight approaches adjust LLM outputs without resorting to full-scale fine-tuning or prompt-tuning. However, these methods often exhibit several limitations. Existing methods often require extracting static steering vectors from a dataset containing steered and unsteered responses Khayatan et al. 2025 ; Rodriguez et al. 2024 ; Wang et al. 2025 . Furthermore, many methods require interventions at a predetermined, hyperparameter-tuned layer and apply the same uniform steering vector for each generated token Turner et al. 2023 ; Khayatan et al. 2025 . Finally, a significant challenge is that they often do not adapt well to the complexities of a multimodal setting Rodriguez et al. 2024 . All these factors can limit their overall adaptability and effectiveness in steering. In contrast, we introduce a novel, parameterized steering mechanism that enhances control over model behavior without the inflexibility and potential loss of generalizability associated with fine-tuning. We achieve this by training a lightweight steering module that learns to predict adjustments by analyzing pairs of prompts that encode both the desired target behavior and its converse. At inference time, this trained module dynamically computes and applies these adjustments to the VLM’s intermediate hidden states. Our contributions extend these ideas into the multimodal VLM setting, where language must be grounded in vision. Here, the steering is applied to the language module of the VLM - post projection of visual features in the language space. This approach allows us to explore controlled generation in multimodal models, following prior methods that intervene solely on the LLM backbone of VLMs Huang et al. 2024 ; Liu et al. 2025 . Unlike prior methods that are often limited to direct subtraction of averaged activations Turner et al. 2023 ; Rimsky et al. 2024 , our module learns a more complex, non-linear mapping from these target-converse activations. This allows it to selectively amplify or suppress relevant activation patterns, effectively learning to ignore irrelevant signals for robust, token-specific steering signals that can be adaptively applied across multiple layers. In comparison to simple prompting-based techniques, and previous steering techniques Rodriguez et al. 2024 ; Turner et al. 2023 ; Rimsky et al. 2024 , SteerVLM is capable of modeling the relationship encoded in semantically rich prompt pairs within the VLM’s activation space. This leads to better control over the model’s outputs. We also introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically designed to support the development and evaluation of steering mechanisms for vision language models. To our knowledge, VNIA is the first such dataset to provide steered responses directly conditioned on images, addressing a key resource gap for VLM steering research. In summary, our contributions are as follows 1 1 1 Code and dataset available at https://github.com/22anushka/SteerVLM/ : 1. We propose a novel lightweight steering module for VLMs to learn complex, non-linear adjustments from target and converse prompt pairs for finer-grained model intervention. 2. We dynamically apply token-specific steering across multiple layers without predetermined layer selection or static vectors. 3. We present VNIA, the first multimodal dataset providing textual responses conditioned on images and steering directions. 4. We quantitatively and qualitatively evaluate our method against existing steering techniques to show that our approach outperforms previous methods in topic-based steering and off-target task of hallucination mitigation. 2 Related Works 2.1 LLM Steering Techniques Previous research has introduced numerous steering methods for LLMs. These interventions span weight-based techniques such as supervised fine-tuning Goodfellow et al. 2016 ; Ouyang et al. 2022 ; Wei et al. , weight editing Hu et al. , and reinforcement learning-based approaches Ouyang et al. 2022 ; Schulman et al. 2017 . Prompt-level interventions include automated prompt engineering and guided decoding for controlled output or style transfer. Token embedding interventions, like soft prompting, append learnable tensors optimized for specific datasets Lester et al. 2021 ; Li and Liang 2021 . Activation-based interventions utilize steering vectors, adjusting activations directly to produce targeted behaviors. 2.2 Steering Vectors Steering vectors can be computed via methods such as mean activation shifts between sentiment prompts Turner et al. 2023 , correlating feature labels with attention head activations Li et al. 2023a , differences in hidden or embedding space vectors Subramani et al. 2024 ; Rimsky et al. 2024 , or classifier-based decision boundaries Wang et al. 2025 . These vectors effectively direct activations toward desired outcomes. 2.3 Activation Engineering