See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment

Mohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui, Adinath Madhavrao Dukre, Rafiq Ali, Sara Atito, Yutong Xie, Imran Razzak · 2026

Read See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment by Mohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui, Adinath Madhavrao Dukre, Rafiq Ali, Sara Atito, Yutong Xie, Imran Razzak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09749.txt.

Introduction

Azeez and seven co-authors trace false objects in multimodal language models to uneven attention: output over-attends loud or frequent content, leaving rare or edge objects unlinked. Every object merits an equal chance during output. Their DOP-OBC plan, training-free and broadly usable, pairs a Dominant Object Penalty damping over-focus on loud regions with an Outlier Boost Coefficient raising focus toward rare spotted objects. Tests across image and video models show fewer false objects on CHAIR and POPE benchmarks with GPT-4o-judged caption gains.

Catalog record

CREATORMohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui, Adinath Madhavrao Dukre, Rafiq Ali, Sara Atito, Yutong Xie, Imran Razzak
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35989
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09749

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment Mohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui, Adinath Madhavrao Dukre, Rafiq Ali, Sara Atito, Yutong Xie, Imran Razzak Abstract Multimodal large language models (MLLMs) frequently hallucinate objects that are absent from the visual input, often because attention during decoding is disproportionately drawn to visually dominant or frequently occurring content. We observe that this inequity in attention allocation is a root cause of object hallucination: when rare, small, or contextually peripheral objects receive insufficient attention, the model fails to ground its generation in the full visual scene. We argue that every object in an image, regardless of its size, frequency or visual salience, deserves equal representational opportunity during decoding. To this end, we propose DOP-OBC , a training-free and architecture-agnostic decoding strategy built on the principle of equitable attention . Two complementary object-aware signals work in tandem: a Dominant Object Penalty (DOP) that softly suppresses attention over-concentration on visually dominant regions, and an Outlier Boost Coefficient (OBC) that amplifies attention toward rare yet confidently detected objects. These signals are injected as per-row logit modulations within the causal attention mask, requiring no weight updates and preserving autoregressive decoding properties. Extensive experiments across image and video MLLMs demonstrate consistent reductions in object hallucination on CHAIR and POPE benchmarks, alongside improvements in GPT-4o-assessed captioning quality across correctness, consistency, detail, context and temporal dimensions. DOP-OBC establishes that fairness in attention allocation is not merely a design principle but a practical and effective path toward more faithful multimodal generation. Figure 1 : DOP–OBC Method Overview. (I) Standard causal masking in transformer attention. (II) Equity-aware modulation adjusts row-wise logit amplitudes: the Dominant Object Penalty (DOP) softly suppresses over-attended, visually dominant objects to free representational capacity, while the Outlier Boost Coefficient (OBC) amplifies rare yet confidently detected objects. (III) These object-aware signals are integrated with an upper-triangular positional decay register to form an equitable attention matrix at inference time, requiring no weight updates. 1 Introduction Multimodal Large Language Models (MLLMs) have achieved remarkable progress in vision–language understanding, demonstrating strong capabilities in image captioning, visual question answering, instruction following, and multimodal reasoning [ 17 , 18 , 3 ] . By combining powerful vision encoders with large language models, these systems generate fluent and contextually coherent descriptions across diverse domains. However, despite their impressive generative quality, a critical reliability gap persists: models frequently produce outputs that are only partially grounded in the visual evidence. Among the most prevalent and concerning failure modes is object hallucination , where the model omits visually present objects or confidently describes objects that do not exist in the scene [ 22 , 12 ] . The persistence of this issue across architectures and benchmarks suggests that the limitation is not purely representational, but fundamentally tied to how multimodal information is integrated and decoded. Figure 2 : Example showing that DOP–OBC produces a more complete, grounded description than the LLaVA-1.5 baseline: the method adds correct background entities (highlighted in green in the figure text), and the accompanying attention-map comparison illustrates the reallocation of attention toward those previously neglected objects. Figure 2 illustrates this imbalance concretely. In the example, LLaVA-1.5 correctly captures the dominant elements of the scene, namely the motorcycle and its rider, yet completely overlooks a clearly visible girl standing beside her bicycle. The correlated attention maps clarify the source of this failure. Although the visual encoder activates across multiple regions, the decoder progressively concentrates attention on the dominant foreground object. In the green attention weight plots, most visual tokens receive uniformly low weights, while a narrow band corresponding to the motorcycle region accumulates strong and persistent attention. As decoding advances, this concentration intensifies, leading to attention collapse and suppression of peripheral yet visually present objects. Importantly, this is not a perception failure, since the encoder representations already contain signals for the girl and the bicycle. Rather, the issue arises during decoding, where attention allocation becomes structurally biased toward large, salient, and frequently occurring objects due to token abundance and language priors. Simply redistributing attention equally does not resolve this structural imbalance, as equality assigns identical resources while equity compensates for disparities in scale and frequency. To address this challenge, we propose DOP–OBC , a training-free and architecture-agnostic decoding strategy that explicitly rebalances attention at inference time. As summarized in Figure 1 , our approach injects two complementary object-aware controls directly into the attention computation. The Dominant Object Penalty (DOP) suppresses over-attended objects by reducing their row-wise logit amplitude, while the Outlier Boost Coefficient (OBC) amplifies rare yet confidently detected objects through calibrated scaling. These modulations are integrated within an upper-triangular positional decay register that preserves autoregressive causality while stabilizing long-range grounding. In Figure 2 , this mechanism suppresses excessive focus on the motorcycle and elevates the underrepresented girl and bicycle. The resulting attention maps show a more balanced and sustained green weight distribution across decoding steps, enabling faithful grounding of all relevant objects. Importantly, this reallocation operates entirely at inference time and requires no retraining, auxiliary models, or architectural modification. The main contributions of this paper are as follows: • We identify a systematic attention imbalance in existing MLLMs, where visually dominant objects consume disproportionate decoding capacity, suppressing small, rare, or peripheral objects. We show that this inequity is a key driver of object hallucination and remains unaddressed by prior methods. • We propose DOP-OBC, a training-free and architecture-agnostic decoding strategy that corrects this imbalance through two complementary plug-in signals: a Dominant Object Penalty that attenuates over-attended objects and an Outlier Boost Coefficient that amplifies rare yet confidently detected ones. Both operate as per-row modulations within an upper-triangular attention register without modifying model weights. • We validate DOP-OBC across multiple image and video MLLMs, reducing hallucination on CHAIR and POPE and improving all five GPT-4o-evaluated video captioning dimensions. These results show that equitable attention allocation is an effective inference-time approach for faithful multimodal generation. 2 Related Work 2.1 Vision-language models Recent progress in multimodal large language models (MLLMs) has substantially advanced vision–language integration, improving multimodal reasoning, instruction following, and conversational capabilities [ 18 , 17 , 3 ] . LLaVA aligns a pretrained LLM with a visual encoder through instruction tuning, enabling open-ended image-conditioned generation [ 18 ] . Flamingo introduces an interleaved visual-text token architecture that supports few-shot multimodal reasoning [ 1 ] . InstructBLIP extends BLIP-2 with instruction tuning to enhance image-grounded dialogue and general multimodal understanding [ 3 ] . For video understanding, Video-LLaVA and related frameworks incorporate temporal modeling into LLaVA-style architectures to enable coherent video dialogue and reasoning [ 13 ] . Despite these advances, object hallucination and grounding failures remain persistent challenges. Rohrbach et al. [ 22 ] first systematically analyzed object hallucination in image captioning, showing that language priors often override visual evidence. POPE [ 12 ] further demonstrates that modern MLLMs frequently generate objects unsupported by the input image, even when confidence is high. More recently, FarSight [ 23 ] identifies decoding-time attention collapse and positional drift as key contributors to hallucination in LLaVA-style models. These findings indicate that hallucination is not solely a representation issue but is closely tied to attention allocation during decoding, motivating inference-time attention rebalancing strategies

Read the complete text