One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs
Youxu Shi, Suorong Yang*, Dong Liu* · 2026
Read One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs by Youxu Shi, Suorong Yang*, Dong Liu* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23041.txt.
Introduction
Youxu Shi, Suorong Yang, and Dong Liu steer vision-language models away from making things up. These models do well on tasks that mix images and text, but they still hallucinate and fail on safety, even at large scale. Steering nudges a model's internal state to improve behavior with little cost.
Their one-shot optimized steering vector is tuned once, then applied broadly. Prior steering traded efficiency against effectiveness in awkward ways. A cheap, reusable vector could make image-aware assistants more truthful without retraining. Steering is attractive because it is lightweight: one vector can adjust behavior without retraining the whole multimodal model.
Catalog record
| CREATOR | Youxu Shi, Suorong Yang*, Dong Liu* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23041
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs Youxu Shi, Suorong Yang*, Dong Liu* Abstract Vision Language Models (VLMs) achieve strong performance on multimodal tasks but still suffer from hallucination and safety-related failures that persist even at scale. Steering offers a lightweight technique to improve model performance. However, steering, whether input-dependent or input-independent, achieves a meaningful trade-off between efficiency and effectiveness. In this work, we observe that steering vectors can generalize across inputs when tasks share aligned semantic intent. Based on this insight, we propose OSGA ( O ne-shot S teering with G enerative A nchor), an input-independent framework that improves model performance with a single optimization instance. OSGA first selects an informative sample via a variance-based data selection strategy and learns a single steering vector with a contrastive objective with generative anchor regularization. The resulting vector can be universally applied at a certain layer during inference time without modifying model parameters. Experiments across multiple benchmarks show that a single OSGA-optimized steering vector consistently improves hallucination mitigation and safety enhancement with negligible overhead, highlighting one-shot steering as a practical and scalable solution for reliable VLMs. 1 Introduction Recently, Vision Language Models (VLMs) ( 44 ; 49 ; 1 ) have demonstrated impressive capabilities across a wide range of complex tasks, including image captioning, visual question answering, and multimodal reasoning. Despite these advances, hallucinations ( 28 ; 36 ; 11 ) and safety-related issues ( 57 ; 14 ; 31 ; 46 ) remain fundamental obstacles to their reliable deployment, often manifesting as responses that contradict the visual evidence or produce offensive, biased, or misleading content. Crucially, these issues persist even as models scale ( 52 ; 17 ) , indicating that simply increasing model capacity or training data is insufficient to address these risks. To mitigate hallucinations in VLMs, existing methods can be broadly categorized into training-based and training-free approaches. Training-based methods include supervised fine-tuning (SFT) ( 42 ; 23 ; 4 ) , and reinforcement learning from human feedback (RLHF) ( 6 ; 56 ; 60 ) , which can introduce substantial training costs. Without incurring expensive training costs, another line of work focuses on lightweight and training-free strategies, such as inference-time interventions ( 62 ; 20 ; 35 ) , post-hoc revision ( 58 ; 10 ) , and hidden-state editing ( 15 ) . Among these, model steering ( 30 ; 33 ; 41 ) emerges as a promising direction due to its simplicity and efficiency. By injecting a steering vector into the model’s hidden representations ( 32 ) , they directly guide the model’s outputs toward a desired semantic direction at inference time without relying on retraining. While model steering techniques have been widely employed in large language models (LLMs), their applications to VLMs remain largely underexplored, in part due to the complexity of multimodal representations and visual grounding. For instance, steering a VLM to suppress hallucinated objects may induce overly conservative generative behavior, leading the model to excessively suppress content generation to avoid potential errors. Moreover, the characteristics of visual tasks can vary across different inputs in terms of complexity and the visual concepts. Therefore, existing methods ( 33 ; 40 ; 41 ) compute input-specific steering vectors, a form of input-dependent steering, which adjusts the model based on the specific input. While effective, these approaches introduce non-negligible inference overhead. In contrast, input-independent approaches ( 30 ; 19 ) compute a single steering vector applied to all inputs, offering a more efficient and lightweight solution, but with somewhat limited effectiveness. As a result, a critical question arises: How to bridge this gap between the two poses a meaningful trade-off and remains an open research question? Table 1 : Comparisons between sample-dependent and sample-independent steering using the CHAIR metric. Methods CHAIR S \downarrow CHAIR I \downarrow Average \downarrow Recall \uparrow Baseline 53.0 14.0 33.50 81.00 Sample-dependent 47.8 12.7 30.25 80.10 Sample-independent 51.4 \pm 3.11 13.5 \pm 1.92 31.98 \pm 1.63 80.11 \pm 1.12 We observe that steering vectors exhibit cross-input transferability when applied to tasks that share aligned semantic objectives and response patterns. Although steering vectors are typically constructed in an input-dependent manner, i.e., generating a steering vector for each sample, their influence on model behavior is not strictly confined to the original input instance. For instance, a steering vector optimized from a single image-caption pair for hallucination mitigation can remain effective when applied to other image-caption instances. As shown in Table 1 , while sample-specific steering achieves the strongest hallucination suppression, a single steering vector derived from one randomly selected image generalizes well to 500 additional images and consistently mitigates hallucinations. Notably, computing steering vectors on a per-sample basis incurs substantial inference overhead. Taken together, these observations suggest that input-dependent steering vectors may implicitly encode task-level or semantic-level control directions that transcend individual samples. Such behavior points to the existence of a latent input-independent component embedded within ostensibly instance-specific steering signals, motivating a shift from per-sample steering toward more shared and reusable control mechanisms. In this paper, we propose OSGA (One-shot Steering with Generative Anchor), a lightweight framework for steering VLMs using only a single training instance. The core idea is to learn a compact steering vector that can generalize across different unseen inputs, enabling efficient and scalable inference-time model steering. To this end, OSGA first identifies a highly informative anchor example via a variance-based selection strategy, which prioritizes data points that induce diverse and discriminative model responses. Based on this anchor, we optimize a steering vector that is injected into the decoder layers of the VLM via a contrastive learning objective. To further enhance optimization stability, we introduce a generative anchor regularization term that constrains the learned steering vector toward a semantically meaningful reference derived from the model’s own generative behavior. Once optimized, the steering vector can be applied universally at inference time without modifying model parameters or requiring additional per-input computation. This design enables efficient, stable, and effective performance improvements. Extensive experiments across diverse VLM architectures and benchmarks demonstrate that OSGA consistently improves performance on hallucination mitigation and safety enhancement tasks. Remarkably, a single optimized steering vector generalizes well across unseen inputs and sub-tasks, achieving a favorable trade-off between faithfulness, safety, and task performance. These results highlight the untapped potential of one-shot optimization as a practical and scalable tool for improving the reliability of VLMs. The contributions are as follows: (1) To the best of our knowledge, we are the first to explore one-shot optimized steering in VLMs, demonstrating that hallucinations can be effectively mitigated by introducing only a single token-level steering vector (4096 dimensions), incurring negligible parameter overhead. (2) We propose OSGA, a one-shot, input-independent steering framework that learns a generalizable steering vector from a single, automatically selected anchor datum, enabling efficient and retraining-free hallucination mitigation for VLMs. (3) Extensive experiments across various VLM architectures and benchmarks demonstrate the robustness and generality of our method, consistently improving hallucination mitigation and safety performance. 2 Related work 2.1 VLMs Hallucination and Safety Contemporary VLMs ( 2 ; 53 ; 48 ; 61 ) face two primary challenges: hallucination and safety. Whether a VLM can perform well on these two tasks is crucial for determining its reliability and practical deployment. For VLMs, hallucination generally