Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
Jiawei Kong, Hao Fang, Shunxiang Liao, Jinyu Li, Bin Chen, Hao Wu, Shu-Tao Xia, Min Zhang · 2026
Read Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization by Jiawei Kong, Hao Fang, Shunxiang Liao, Jinyu Li, Bin Chen, Hao Wu, Shu-Tao Xia, Min Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27906.txt.
Introduction
Kong and colleagues reduce hallucinations in multimodal reasoning models by treating the reasoning chain as a condition for the answer, not part of one lump output. Their RC-DPO method contrasts the same answer under different reasoning chains and mines grounded chains with tree search. Across models and benchmarks it curbed hallucinations and made reasoning more reliable.
Catalog record
| CREATOR | Jiawei Kong, Hao Fang, Shunxiang Liao, Jinyu Li, Bin Chen, Hao Wu, Shu-Tao Xia, Min Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65475 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27906
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization Jiawei Kong, Hao Fang, Shunxiang Liao, Jinyu Li, Bin Chen, Hao Wu, Shu-Tao Xia, Min Zhang Abstract Multimodal Large Reasoning Models introduce the reasoning paradigm, demonstrating strong capabilities on complex vision-language tasks. However, they still suffer from severe hallucinations. Existing training-based methods typically mitigate hallucinations through response-level direct preference optimization (DPO), where the Chain-of-Thought (CoT) and the final answer are treated as a monolithic output and optimized jointly. We reveal that this formulation performs similarly to answer-only optimization, suggesting that it primarily learns answer-level preference, while leaving CoT-level supervision insufficiently exploited. To address this issue, we explicitly formulate a CoT-oriented preference term and derive Reasoning-Conditioned Direct Preference Optimization (RC-DPO), which models the CoT as a condition for answer generation and contrasts the preference for the same preferred answer under different CoT conditions, promoting answer-supportive reasoning chain alignment. To further improve optimization, we introduce a reasoning-enhanced preference data generation strategy that employs Monte Carlo Tree Search to discover visually grounded and logically consistent CoTs as positive samples, and attention-guided CoT token pruning to construct negative ones. Extensive experiments across various models and benchmarks show that RC-DPO effectively mitigates hallucinations and improves the reliability of the multimodal reasoning process. 1 Introduction Large Reasoning Models (LRMs), such as OpenAI o3 and DeepSeek-R1 11 , have recently achieved remarkable progress on various tasks by explicitly generating intermediate reasoning steps before producing final responses 5 ; 4 . Building upon this, Multimodal Large Reasoning Models (MLRMs) 26 ; 37 ; 14 ; 33 extend such reasoning capabilities to vision-language scenarios. Further enhanced by post-training techniques such as reinforcement learning, these models have achieved substantial gains on challenging multimodal tasks, including visual mathematics 22 , scientific question answering 23 , and chart understanding 25 . Figure 1: A visual question answering example generated by MM-Eureka-7B 26 , where the hallucinated content is highlighted in red . Despite their strong performance, recent studies 35 ; 8 have shown that MLRMs remain prone to hallucinations, generating content that appears plausible yet is inconsistent with the visual input. Unlike conventional Multimodal Large Language Models (MLLMs), hallucinations in MLRMs can arise not only in the final answer but also within the intermediate Chain-of-Thought (CoT), as illustrated in Figure 1 . This poses a unique challenge for hallucination mitigation: unsupported visual evidence, such as nonexistent objects or relations, may first emerge in the CoT and then propagate to the final answer through subsequent reasoning steps. Therefore, mitigating hallucinations in MLRMs requires moving beyond answer-level correction toward the explicit alignment of intermediate reasoning chains. Recent studies have explored post-alignment methods for mitigating hallucinations in MLRMs, with Direct Preference Optimization (DPO) commonly employed to align model outputs with curated preference pairs 40 ; 38 . However, these methods typically construct preference pairs over complete responses, where the CoT and final answer are concatenated and jointly optimized under a single response-level preference signal. While such coarse-grained supervision can reduce hallucinations to some extent, it does not explicitly distinguish CoT-level alignment from answer-level alignment. As a result, the optimization may be biased toward answer-level shortcuts , leaving the reasoning chain insufficiently optimized and limiting the overall effectiveness of hallucination mitigation. This raises a natural question: How can we move beyond response-level preference optimization and explicitly align reasoning chains with the correct answer? To address this, we propose R easoning- C onditioned D irect P reference O ptimization (RC-DPO), a fine-grained preference optimization framework that explicitly models the relationship between reasoning chains and final answers. Instead of only comparing complete responses, RC-DPO treats the CoT as a condition for answer generation and contrasts the likelihood of the same preferred answer under different CoT conditions. Specifically, given a multimodal input and a preferred answer, RC-DPO encourages the model to assign higher preference to the answer when conditioned on a visually grounded and logically valid CoT than on a hallucinated or corrupted CoT. By isolating the effect of the reasoning chain, this controlled comparison enables the model to better distinguish quality differences between reasoning chains and provides a direct signal for learning answer-supportive CoT preferences. To support this objective, we further construct reasoning-conditioned preference data by searching for high-quality positive reasoning chains with Monte Carlo Tree Search (MCTS), and generating negative reasoning chains through an attention-guided CoT token pruning strategy that removes visually salient CoT tokens. By combining response-level preference optimization with reasoning-conditioned preference learning, RC-DPO explicitly aligns intermediate reasoning chains with the answers they support, enabling more faithful multimodal reasoning. In summary, our contributions are as follows: • We revisit existing DPO-based hallucination mitigation methods for MLRMs and reveal that their coarse-grained supervision may leave CoT insufficiently optimized. • We propose RC-DPO, a fine-grained preference optimization framework that aligns reasoning chains by contrasting the same preferred answer under different CoT conditions. • We further introduce a reasoning-conditioned preference data generation pipeline with MCTS-based positive sampling and attention-guided negative construction. • Experimental results on four MLRMs across nine benchmarks demonstrate the effectiveness of RC-DPO in mitigating hallucinations and improving multimodal reasoning ability. 2 Observation and Motivation In this section, we present the motivation behind our RC-DPO for mitigating hallucinations in MLRMs. We first provide evidence of insufficient CoT optimization from two complementary perspectives, i.e. , training dynamics and inference-time hallucination decomposition. We further highlight the necessity of CoT-level hallucination mitigation by analyzing the conditional dependency between CoT and answer hallucinations. Figure 2: Motivating observations for RC-DPO. (a) Loss-ratio curves of CoT and answer segments during OPA-DPO training. (b) Segment-level CHAIR S scores of the original model, OPA-DPO, and the answer-only variant OPA-DPO {}_{\text{Ans}} . (c) Conditional distribution between CoT CHAIR i scores and answer CHAIR i scores. 2.1 Answer Bias in Response-level DPO Observation. We analyze the training dynamics of OPA-DPO 38 by decomposing each response into CoT and answer spans and tracking their loss ratios during training. As shown in Figure 2 (a), the policy-reference preference gap grows faster on the answer segment than on CoT, indicating that answer-level signals are optimized more rapidly during response-level DPO. To further examine this bias, we implement an answer-only variant, (c_{w},a_{w})\succ(c_{w},a_{l}) , where the same CoT is shared and only the final answer differs. This removes CoT-level preference from the comparison and provides only answer-level supervision. As shown in Figure 2 (b), its CHAIR S score is close to that of OPA-DPO. Insights. These results suggest that response-level DPO may be biased toward answer-level optimization, with much of its gain arising from answer-level preference learning, while CoT preference remains insufficiently captured. This motivates a more fine-grained objective that explicitly separates CoT alignment from answer-level alignment to reduce reliance on shortcut learning. Figure 3: Overview of RC-DPO. We first construct SFT data to align the CoT format. Then, we construct positive and negative reasoning samples through MCTS sampling and attention-guided token pruning, respectively. Finally, we optimize the policy model with both response-level and reasoning-conditioned preference objectives. 2.2 CoT as the Hallucination Bottleneck Observation. We evaluate CHAIR S on OPA-DPO outputs, including the complete response and its decomposed CoT and answer segments. As shown in Figure 2 (b), although OPA-DPO reduces hallucinations compared with the original model, CoT hallucination