Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
Xin Zou, Haolin Deng, Yibo Yan, Shuliang Liu, Zhiwei Jin, Chen Chen, Haonan Lu, Xuming Hu, [2.5pt] HKUST (GZ) · 2026
Read Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation by Xin Zou, Haolin Deng, Yibo Yan, Shuliang Liu, Zhiwei Jin, Chen Chen, Haonan Lu, Xuming Hu, [2.5pt] HKUST (GZ) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29805.txt.
Introduction
Zou and seven colleagues tackle hallucination in multimodal large language models, systems that answer questions about images but sometimes state things the image does not show. Their method, Oriented Pickup Preference Optimization, or OPPO, is built on an observation about attention.
The authors first show that when the parts of an image tied to the question are made stronger through the model's own attention, its answers get more accurate. Many errors, they argue, come not from failing to see but from failing to trust what was seen. OPPO therefore trains the model to prefer answers grounded in stronger visual evidence over the same answer under weaker evidence. Tests on hallucination and general benchmarks show OPPO beating the baselines it is compared with.
Catalog record
| CREATOR | Xin Zou, Haolin Deng, Yibo Yan, Shuliang Liu, Zhiwei Jin, Chen Chen, Haonan Lu, Xuming Hu, [2.5pt] HKUST (GZ) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29805
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation Xin Zou, Haolin Deng, Yibo Yan, Shuliang Liu, Zhiwei Jin, Chen Chen, Haonan Lu, Xuming Hu, [2.5pt] HKUST (GZ) Abstract Multimodal Large Language Models (MLLMs) are prone to hallucination as their generation preferences are insufficiently calibrated to visual evidence, causing them to fall back on linguistic priors, rather than faithful grounding. In this work, we start from an empirical observation: when query-relevant visual evidence is explicitly strengthened using the model’s own attention, generation becomes more accurate, suggesting that many failures do not arise solely from missing perception, but from an insufficient tendency to trust the evidence the model has already attended to. Motivated by this finding, we propose Oriented Pickup Preference Optimization ( OPPO ), an evidence-aware alignment objective that learns preferences over the strength of visual evidence, rather than only response quality. Concretely, OPPO contrasts the same faithful response under stronger, anchored, weaker-evidence views, turning naive visual preference into ordered visual-evidence alignment. We further combine this objective with fine-grained span-level and token-level regularization to stabilize the training. Besides, we provide a theoretical analysis showing that ordered evidence margins induce a positive lower bound on local visual sensitivity. Extensive evaluations across hallucination and general-purpose benchmarks demonstrate that OPPO consistently outperforms baseline methods. 1 Introduction The rapid evolution of MLLMs ( Qwen2.5-VL ; Qwen3-VL ; jin2025andesvl ; wang2025internvl3_5 ; team2026kimi ) has fundamentally advanced document intelligence ( hu2025mplug ; liao2025doclayllm ) and complex reasoning tasks ( hu2024bliva ; vu2025describe ) . Yet, their failures still share a common pattern: when visual evidence is weak, atypical, or conflicts with semantic expectations, the model often trusts its internal linguistic prior more than visual evidence, leading to what is known as multimodal hallucination huang2025survey . Fig. 1 illustrates this shared pattern across generalized and text-scene hallucination. This deficiency poses significant risks in safety-critical kim2025vru and text-based VQA tasks shu2025semantics ; huang2025ocr , where any wrong in object or character recognition can completely undermine the model’s reliability chen2025can . Figure 1 : Examples of hallucination in MLLMs. The root of this misalignment may stem from a fundamental limitation in current post-training paradigms han2025learning ; shu2025large ; han2025self . Previous preference alignment methods typically treat the visual input as a static and passive condition while focusing on optimizing response likelihood huang2025visual ; tang2026revisiting , thus they may improve response ranking without explicitly enforcing evidence sensitivity. Consequently, when confronted with low-quality or counter-intuitive visual signals, the decoder tends to “short-circuit” the evidence and retreat to its parameterized knowledge he2025mmboundary , i.e. , the model prefers to predict “what should be there” based presumptively on linguistic probability, rather than to decode “what is actually there” directly according to task-relevant visual groundedness. This disconnect raises a pivotal question: How can we explicitly align the model’s generation preferences with the strength of its visual evidence, rather than merely ranking the final text outputs? Interestingly, our preliminary analysis ( Section 2.1 ) reveals a promising paradox, i.e. , even when MLLMs generate hallucinatory content, their cross-modal attention maps often successfully localize the relevant visual regions. This observation suggests that the failure is not necessarily a lack of “sight”, but rather a policy-level deficiency: the model does not know how to prioritize the evidence it has already attended to. To bridge this gap, we propose oriented pickup preference optimization ( OPPO ), an evidence-aware alignment paradigm that transforms preference learning from pure answer ranking into evidence-aware ranking . Specifically, we identify query-relevant regions through attention priors, synthesize stronger and weaker visual views, and train the model to prefer faithful responses more strongly when the supporting evidence is stronger. Compared with conventional DPO-style objectives, OPPO adds an orthogonal supervision signal, i.e. , it teaches the model that the confidence assigned to a faithful answer should scale with the strength of the evidence that supports it. To make this supervision more stable and localized, we further introduce span-level and token-level regularization, so that the alignment signal is concentrated on critical content and remains consistent throughout autoregressive generation. Ultimately, OPPO ensures that the model’s responses are not just linguistically plausible, but are causally anchored in the visual evidence it perceives. Figure 2 : Comparison of preference optimization paradigms. (a) DPO focuses on linguistic alignment but is vision-insensitive. (b) Multimodal DPO utilizes stochastic visual augmentation but suffers from high gradient noise and low margin gain. (c) Our method constructs a deterministic visual preference chain to align \nabla_{\theta}L with a query-aware evidence axis, thus maximizing the stable decision margin. To further clarify the mechanics of this causal anchoring, we contrast our approach with existing preference optimization paradigms in Figure 2 . Direct Preference Optimization (DPO) operates on the language manifold, but it often yields a decision boundary that is insensitive to fine-grained visual hallucinations. Existing multimodal variants of DPO attempt to bridge this gap by introducing visual perturbations; however, their reliance on random noise or generic cropping often misses the core visual evidence. This leads to a “noisy” latent space where the distinction between positive and negative samples remains ill-defined. In contrast, OPPO explicitly constructs a visual preference chain along a query-aware “evidence axis”. By leveraging attention priors to warp and amplify task-relevant regions, we pull the faithful samples toward high-confidence regions of the latent space while pushing hallucinatory counterparts across the decision hyperplane. This mechanism forces the model to ground its preference scores on explicit visual features, thereby enlarging the optimization margin and effectively mitigating the gradient noise inherent in naive visual preference learning. Overall, our contributions are summarized as follows: • We propose to redefine MLLM hallucination as a policy-level distrust of internal visual signals rather than a mere deficiency in visual perception, an insight that allows us to pivot the alignment objective from stochastic linguistic preference to structurally-ordered evidence-aware calibration. • We propose OPPO , which anchors the decision boundary to an ordered visual preference chain, reinforced by span-level regularization and a theoretical lower bound on local visual sensitivity. • We demonstrate that OPPO consistently outperforms baseline methods, effectively mitigating hallucinations while preserving general capabilities across both generalized and text-scene benchmarks. 2 Preliminary and Motivation Direct Preference Optimization for MLLMs . To align MLLMs with human intent, RLHF bai2022training and RLAIF leerlaif frameworks optimize a policy \pi_{\theta} to maximize the KL-constrained expected reward: \max_{\pi_{\theta}}\mathbb{E}_{(x,v)\sim\mathcal{D},y\sim\pi_{\theta}(y|x,v)}[r({x},{y},{v})]-\beta\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}(\cdot\,|{x},{v})||\pi_{\mathrm{ref}}(\cdot\,|{x},{v})], (1) where \mathcal{D} denotes a preference dataset, {v} is an image, and \beta is the regularization parameter. DPO rafailov2023direct bypasses explicit reward modeling by deriving a closed-form solution, the reward is reparameterized: r({x},{y},{v})=\beta\log\frac{\pi_{\theta}({y}|{x},{v})}{\pi_{\mathrm{ref}}({y}|{x},{v})}+\beta\log Z({x},{v}). (2) Here, Z({x},{v}) is a partition function dependent only on the inputs ({x},{v}) . By integrating Bradley-Terry model bradley1952rank and preference pairs ({y}_{w},{y}_{l}) , the partition function Z cancels out, yielding DPO objective: \mathcal{L}_{\mathrm{DPO}}=-\mathbb{E}_{\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x,v)}{\pi_{\text{ref}}(y_{w}|x,v)}-\beta\log\frac{\pi_{\theta}(y_{l}|x,v)}{\pi_{\text{ref}}(y_{l}|x,v)}\right)\right], (3) where \sigma(\cdot) is sigmoid function, which enables direct policy optimization without explicit rewarding. Multimodal Direct Preference Optimization . To align MLLMs with visual preferences, multimodal DPO extends the original framework by contrasting visual contexts (v_{w},v_{l}) instead of textual responses. Given the instruction x and response y , the objective optimizes the policy \pi_{\theta} to favor the preferred image v_{w} over the dispreferred or hallucinatory image v_{l} . The loss function is defined as: \mathcal{L}_{\mathrm{MDPO}}=-\mathbb{E}_{\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y|x,v_{w})}{\pi_{\text{ref}}(y|x,v_{w})}-\beta\log\frac{\pi_{\theta}(y|x,v_{l})}{\pi_{\text{ref}}(y|x,v_{l})}\right)\right]. (4) By maximizing the likelihood