Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs
Yunhun Nam, Jongheon Jeong · 2026
Read Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs by Yunhun Nam, Jongheon Jeong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28401.txt.
Description
Yunhun Nam and Jongheon Jeong teach vision-language models to stop hallucinating objects that are not in the picture. Preference training can improve faithfulness, but only if the preferred and rejected answers are built well. Intervention-based pairs drift too far from what the model would say, while sampled pairs often ignore the image during construction.
Nam and Jeong propose ViPSy, which first extracts a visual cue from object content recurring across semantically aligned variants of an image. The model's own rollouts are then conditioned on that cue, so candidate answers stay close to its natural style while leaning on real visual evidence. Aligned this way, the model set a new best mark for cutting hallucination, reducing hallucination rates by about a third on one benchmark and a quarter on another.
Catalog record
| CREATOR | Yunhun Nam, Jongheon Jeong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28401