Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination

Yangneng Chen, Junlin Li, Weijun Yao, Xilai Ma, Guodong Du, Wenya Wang, Jing Li · 2026

Read Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination by Yangneng Chen, Junlin Li, Weijun Yao, Xilai Ma, Guodong Du, Wenya Wang, Jing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10622.txt.

Introduction

Yangneng Chen and six colleagues trace vision-language hallucinations to vocabulary hijacking. Large vision-language models progress fast in mixed tasks, yet hallucinations still hurt trust. Hallucinations produce text clashing with image input, often blamed on weak visual attention. Their 2026 paper studies attention patterns via the logit lens.

The authors find specific image tokens, called Inert Tokens, draw extra attention. Mapped into vocabulary space, these tokens always decode to a fixed set of stray words, called Hijacking Anchors. That rigid meaning collapse motivates Hijacking Anchor-Based Identification, a strategy to localize these Inert Tokens. A new metric, the Non-Hijacked Visual Attention Ratio, finds heads resilient to hijacking and critical for factual truth. Hijacking-Aware Visual Attention Boost then strengthens these heads' focus on salient visual content, cutting hallucinations with no extra compute.

Catalog record

CREATORYangneng Chen, Junlin Li, Weijun Yao, Xilai Ma, Guodong Du, Wenya Wang, Jing Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10622

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination Yangneng Chen, Junlin Li, Weijun Yao, Xilai Ma, Guodong Du, Wenya Wang, Jing Li Abstract Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal tasks, yet their reliability is persistently undermined by hallucinations—generating text that contradicts visual input. Recent studies often attribute these errors to inadequate visual attention. In this work, we analyze the attention mechanisms via the logit lens, uncovering a distinct anomaly we term Vocabulary Hijacking . We discover that specific visual tokens, defined as Inert Tokens , disproportionately attract attention. Crucially, when their intermediate hidden states are projected into the vocabulary space, they consistently decode to a fixed set of unrelated words (termed Hijacking Anchors ) across layers, revealing a rigid semantic collapse. Leveraging this semantic rigidity, we propose Hijacking Anchor-Based Identification (HABI) , a robust strategy to accurately localize these Inert Tokens. To quantify the impact of this phenomenon, we introduce the Non-Hijacked Visual Attention Ratio (NHAR) , a novel metric designed to identify attention heads that remain resilient to hijacking and are critical for factual accuracy. Building on these insights, we propose Hijacking-Aware Visual Attention Enhancement (HAVAE) , a training-free intervention that selectively strengthens the focus of these identified heads on salient visual content. Extensive experiments across multiple benchmarks demonstrate that HAVAE significantly mitigates hallucinations with no additional computational overhead , while preserving the model’s general capabilities. Our code is publicly available at https://github.com/lab-klc/HAVAE . † † footnotetext: Corresponding author. 1 Introduction Large Vision-Language Models (LVLMs), extending large language models (LLMs) to process visual inputs, represent a major advancement in artificial intelligence ( Liu et al. 2023 ; Chen et al. 2023 ; Zhu et al. 2023 ; Wang et al. 2024 ) . Nevertheless, LVLMs remain susceptible to hallucinations ( Sun et al. 2024 ; Zhou et al. 2024 ; Huang et al. 2024 ; Bai et al. 2024 ) , where generated text fails to align with visual content. Such inconsistencies undermine LVLM reliability in multimodal tasks, limiting their practical deployment. Figure 1: Attention maps for generating “phone”. (a) The average of all heads exhibits the Vocabulary Hijacking phenomenon, with focus scattered on the background. (b) In contrast, heads selected by HAVAE concentrate attention precisely on the target object. Substantial research on mitigating hallucinations LVLMs has produced various intervention strategies ( Yu et al. 2024a ; Leng et al. 2024 ; Chen et al. 2024 ; Li et al. 2025 ) . A critical consensus has emerged : insufficient attention to visual tokens during generation is a primary driver of hallucinations. This insight has inspired numerous attention-based methods ( Jiang et al. 2025 ; Liu et al. 2024 ; Yang et al. 2025 ) . However, a fundamental question remains unresolved: Which specific attention heads are actually critical? Figure 2: Illustration of the Vocabulary Hijacking phenomenon in LLaVA-1.5 7B. (a) Attention heatmap for the token “phone”, where several irrelevant background tokens capture excessive attention (highlighted in red boxes), which we term Inert Tokens . (b) Logit lens analysis of the top-10 attended visual tokens and their neighbors. Inert Tokens (red boxes) exhibit rigid non-semantic traces, contrasting sharply with target tokens (yellow boxes) that evolve from meaningless words into image-aligned concepts. To demystify attention mechanisms, ( Jiang et al. 2025 ) applied Logit Lens to vision tokens, identifying a transition from Visual Information Enrichment to Semantic Refinement . Adopting this framework, we define the Trace as the layer-wise sequence of vocabulary words decoded from a token’s hidden states. Typically, a functional Trace evolves from meaninglessness to rich semantics; for instance, Token #395 in Figure 2 shifts from nonsensical outputs to image-aligned concepts like “phone” after layer 20. Further investigating the top-attended tokens reveals a critical anomaly: LVLMs exhibit a tendency to persistently allocate excessive attention to certain background image tokens, whose Traces are dominated by a small set of fixed, meaningless words. For example, in Figure 2 , tokens #595 and #157 receive high attention despite being irrelevant to the generated “phone”. Their Traces remain monopolized by “kwiet”, failing to exhibit the standard semantic evolution. We term this phenomenon Vocabulary Hijacking , wherein specific vision tokens, designated as Inert Tokens , exhibit inertia across layers while attracting substantial attention. To characterize this, we introduce the Vocabulary Hijacking Score (VHS) . Statistical analysis reveals that the Traces of Inert Tokens are dominated by a small subset of words with anomalously high VHS (e.g., “kwiet”), which we formally define as Hijacking Anchors . Leveraging this finding, we propose Hijacking Anchor-Based Identification (HABI) . By quantifying the proportion of a token’s Trace that coincides with Hijacking Anchors, HABI effectively localizes Inert Tokens. While Vocabulary Hijacking superficially resembles the Visual Attention Sink ( Kang et al. 2025 ) , empirical results demonstrate significant divergences (detailed in Appendix C ), confirming that our framework captures this anomalous attention phenomenon more precisely and robustly. Building upon this foundation, we investigate the intrinsic link between Vocabulary Hijacking and hallucinations. Statistical experiments reveal a strong positive correlation: excessive attention to Inert Tokens significantly increases the propensity for hallucination. Motivated by this, we propose Non-Hijacked Visual Attention Ratio (NHAR) to identify critical heads focusing on salient visual information, and introduce Hijacking-Aware Visual Attention Enhancement (HAVAE) to selectively amplify them. As shown in Figure 1 , HAVAE-selected heads exhibit precise grounding on target objects, contrasting sharply with the scattered attention of generic heads. Extensive experiments validate HAVAE’s superiority: it achieves SOTA performance across hallucination benchmarks with zero additional computational overhead , while fully preserving general capabilities. This highlights an optimal balance between performance and practical efficiency. Our main contributions are summarized as follows: • We unveil the novel mechanism of Vocabulary Hijacking and propose HABI , which reliably localizes Inert Tokens across various LVLMs. • We introduce HAVAE , a training-free intervention that leverages our novel metric, NHAR , to identify and selectively reinforce the attention heads most critical for hallucination mitigation. • Through extensive experimentation across diverse architectures, we demonstrate that HAVAE establishes a new SOTA in hallucination mitigation with zero computational overhead, while preserving general capabilities. 2 Related Work Reducing Hallucinations in MLLMs. Mitigating hallucinations in MLLMs is primarily addressed through fine-tuning ( Yu et al. 2024a ; Yu et al. 2024b ) or training-free inference interventions. Prominent strategies include contrastive decoding ( Leng et al. 2024 ; Chen et al. 2024 ) , activation steering ( Li et al. 2025 ) , and attention intervention ( Liu et al. 2024 ; Jiang et al. 2025 ; Yang et al. 2025 ) . While attention-based methods are most relevant to our work, they face a critical unresolved problem: determining precisely which heads require intervention. In the absence of a principled identification mechanism, these approaches are often forced to rely on heuristics. Our work bridges this gap by proposing a precise metric to target the specific heads essential for factual grounding. Attention Sink in Language Models. The “Attention Sink” phenomenon, where vacuous tokens monopolize attention, is established in LLMs ( Xiao et al. 2023 ) and recently extended to LVLMs as Visual Attention Sinks (VAS) ( Kang et al. 2025 ) . Along similar lines, works like

Read the complete text