Probing Visual Language Priors in VLMs Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee Abstract Despite recent advances in Vision-Language Models (VLMs), they may over-rely on visual language priors existing in their training data rather than true visual reasoning. To investigate this, we introduce ViLP, a benchmark featuring deliberately out-of-distribution images synthesized via image generation models and out-of-distribution Q&A pairs. Each question in ViLP is coupled with three potential answers and three corresponding images: one that can be resolved by text priors alone and two that demand visual reasoning. Although, humans achieve near-perfect accuracy, modern VLMs falter; for instance, GPT-4 achieves only 66.17% on ViLP. To alleviate this, we propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form “good-bad" image pairs for self-training. Our training objectives compel VLMs to focus more on the actual visual inputs, and we demonstrate their effectiveness in boosting the performance of open-source VLMs, including LLaVA-v1.5 and Cambrian. 1 Introduction Vision-Language Models (VLMs) have advanced text-image interaction, bridging the gap between visual and textual data ( Achiam et al., 2023 ; Team et al., 2023 ) . However, a persistent challenge for learning-based models, such as ResNets and CLIPs, lies in their reliance on learned priors from the training data, sometimes overlooking visual cues when answering questions ( Agrawal et al., 2016 ; Prabhu et al., 2023 ) . For example, when shown a torus-shaped soccer ball (Figure 1 ), a model might incorrectly identify it as a sphere due to strong language priors. Simultaneously, these models may adhere to learned visual priors ( Thrush et al., 2022 ; Sterz et al., 2024 ) , making it difficult to comprehend out-of-distribution visual cues, such as a zebra with atypical spot patterns (Figure 1 ), which humans would easily discern. This raises an important question: do today’s VLMs still over-rely on learned visual language priors, especially given that they rely on far fewer image-text pairs compared to the internet-scale text corpora used for pretraining? To investigate this, we probe the Vi sual L anguage P riors of VLMs by constructing Question-Image-Answer (QIA) triplets that deliberately deviate from the training data distribution. Unlike existing benchmarks that typically rely on internet-sourced images ( Goyal et al., 2017 ; Tong et al., 2024 ) , which inadvertently favor the visual language priors embedded in the training data of VLMs, we utilize modern image generation models, including DALL·E-3 ( Ramesh et al., 2021 ) and Flux , to synthesize out-of-distribution images supporting also out-of-distribution answers. Using the image generation models also helps our QIAs exhibit notable variation in texture, shape, conceptual combinations, hallucinated elements, and proverb-based contexts. Our benchmark, ViLP, contains 300 carefully designed questions, each paired with three distinct answers: a Prior Answer and two Test Answers , resulting in a total of 900 QIA triplets. To further challenge the priors of VLMs, we amplify language priors in questions by introducing distractor facts: each question is structured to present a distractor fact followed by a question . The distractor fact directly leads to the Prior Answer. In contrast, the two Test Answers are crafted to challenge the priors by requiring both textual and visual cues for accurate reasoning. While human participants achieved \sim 98% accuracy easily, current VLMs exhibit considerable difficulty, as evidenced by a significant performance drop on our benchmarks, with GPT-4o ( OpenAI, 2024 ) scoring only 66.17%. Motivated by the results of ViLP, we propose Image-DPO, a self-improving approach to enhance VLM visual reasoning performance by increasing reliance on visual inputs. Our method employs self-generated VQAs using image generation and editing models ( Podell et al., 2023 ; Ren et al., 2024 ; Brooks et al., 2023a ) and applies controlled corruptions to create “good" and “bad" image pairs for DPO-like training ( Rafailov et al., 2024 ) . Experiments with open-source VLMs, including LLaVA-v1.5 ( Liu et al., 2024a ) and Cambrian ( Tong et al., 2024 ) , demonstrate its effectiveness. Moreover, we theoretically show that our objective optimizes an upper bound of the RLHF objective ( Ouyang et al., 2022 ) . The proposed ViLP dataset, Image-DPO code, and our synthetic data appear in https://vilp-team.github.io/ . Figure 1 : Sample data from ViLP. For the same question, ViLP  provides three distinct images and corresponding answers (upper-left corner). All questions follow a consistent structure, combining a distractor fact with a question . The Prior Answer (first column) can be directly inferred from the question, while Test Answers (second \& third column) rely on visual cues. Our answers are designed to be single words, and both the model and human evaluators are tasked with open-domain answering, rather than selecting from predefined options. To support this, we have developed a robust synonym and plural detection pipeline, ensuring that open-ended responses do not hinder the evaluation process. This approach also enables evaluation without relying on LLMs. Please refer to Appendix A.1 for more data samples from ViLP. We investigate the impact of image styles in Appendix A.3 , where we generate more realistic images using 4o image generation . Furthermore, we include both qualitative and quantitative comparison results with Winoground ( Thrush et al., 2022 ) , Whoops! ( Bitton-Guetta et al., 2023a ) , and HallusionBench ( Guan et al., 2023 ) in Appendix A.4 . 2 Related Work VQA Dataset: Significant efforts have produced VQA datasets from various angles, including general VQA ( Agrawal et al., 2015 ; Gurari et al., 2018 ; Fu et al., 2023 ; Liu et al., 2023d ; Li et al., 2023a ; Yu et al., 2023b ; Liu et al., 2024a ) , reading text or charts ( Singh et al., 2019a ; Mathew et al., 2020 ; Mathew et al., 2021 ; Masry et al., 2022 ) , complex reasoning ( Lu et al., 2022 ; Lu et al., 2023 ) , composition probing ( Hudson & Manning, 2019 ; Ma et al., 2022 ; Thrush et al., 2022 ; Hsieh et al., 2023 ; Li et al., 2024a ) , hallucinations ( Rohrbach et al., 2018 ; Li et al., 2023c ; Guan et al., 2023 ) , common-sense reasoning ( Bitton-Guetta et al., 2023b ; Bitton-Guetta et al., 2023a ) , and more ( Majumdar et al., 2024 ; Sterz et al., 2024 ) . We propose a benchmark that tests VLMs’ visual reasoning when questions, answers, and images defy common patterns. Following the balanced dataset design of ( Goyal et al., 2017 ) , each question is accompanied by three answers: one aligns with language priors, and two deviate, prompting reliance on visual cues. By leveraging state-of-the-art image generation models, our benchmark challenges these priors more effectively than previous datasets built from internet images ( Goyal et al., 2017 ; Tong et al., 2024 ) . Furthermore, unlike the “trick” category in ( Sterz et al., 2024 ) , we first generate question-answer pairs before synthesizing images under specified constraints, creating more difficult visual reasoning tasks. Comprehensive comparisons with existing datasets appear in Appendix A.4 . Vision Language Models and Language Priors : Multimodal reasoning is crucial for machine intelligence, with VLMs integrating visual perception, text reasoning, instruction following, and generation for complex tasks ( Tan & Bansal, 2019 ; Li et al., 2019 ; Kim et al., 2021 ; Wang et al., 2021b ; Wang et al., 2021a ; Alayrac et al., 2022 ; Li et al., 2023b ; Chen et al., 2022 ; Jia et al., 2021 ; Shen et al., 2021 ; Singh et al., 2021 ; Liu et al., 2023c ; Liu et al., 2023a ; Zhao et al., 2023a ; Chen et al., 2023 ; Zhu et al., 2024b ; Li et al., 2024c ; Dai et al., 2023 ; Li et al., 2024c ; Yu et al., 2024 ; Dai et al., 2024 ; Deitke et al., 2024 ) . Inspired by the success of large language models ( Brown et al., 2020 ; OpenAI, 2023a ; Touvron et al., 2023a ; Touvron et al., 2023b ; Chiang et al., 2023 ) and pre-trained visual encoders ( Radford et al., 2021 ; Desai & Johnson, 2020 ; Caron et al., 2021 ; Chen et al., 2024 ) , many recent methods leverage relatively small vision-language paired datasets ( Liu et al., 2024a ; Tong et al., 2024 ) to fine-tune connectors between LLMs and visual backbones ( Liu et al., 2024a ) . However, these datasets are far smaller than the vast text corpora for LLM pre-training ( OpenAI, 2023b ; Soldaini et al., 2024 ) , and freezing the visual encoder and LLM parameters often preserves language biases, causing visual inputs to be overshadowed ( Thrush et al., 2022 ; Sterz et al., 2024 ) . This challenge is amplified by deliberately generated images that expose such biases, as shown in our study. Previous works and datasets ( Goyal et al., 2016 ; Agrawal et al., 2017 ; Dancette et al., 2021 ; Wu et al., 2022 ; Ramakrishnan et al., 2018 ; Gouthaman & Mittal, 2020 ) addressed these issues with curated simulators ( Johnson et al., 2016 ) or internet imagery ( Zhang et al., 2016 ) . In this paper, we present a novel VQA benchmark featuring carefully designed questions, fact-based distractors, rare-distribution answers, and image generation techniques to produce realistic visuals that challenge learned visual language priors (Figure 1 ). Self-Rewarding VLM : Self-rewarding LLM ( Yuan et al., 2024 ) has shown that LLMs can generate and improving themselves in the process via Directed Peference Optimization (DPO) ( Rafailov et al., 2024 ) . This approach extends to VLMs by generating new answers for DPO training ( Zhou et al., 2024a ; Deng et al., 2024 ; Zhou et al., 2024b ; Wang et al., 2024c ; Wang et al., 2024a ; Yue et al., 2024 ; Liu et al., 2024b ; Xiao et al., 2024 ) . Our work aligns with these self-rewarding VLMs but differs in two key ways: (1) our proposed Image-DPO generates multiple images for a single question-image pair (rather than multiple answers); (2) rather than relying solely on existing images ( Zhu et al., 2024a ) , Image-DPO creates diverse new images using pre-trained generative models (SDXL ( Podell et al., 2023 ) , GroundedSAM ( Ren et al., 2024 ) , InstructPix2Pix ( Brooks et al., 2023a ) ). Furthermore, Image-DPO deliberately corrupts images to produce multiple degraded versions that serve as rejected data in DPO training. Concurrent works ( Wang et al., 2024a ; Xie et al., 2024 ) explore similar methods but lack a benchmark to verify enhanced visual focus, fail to establish theoretical connections between their proposed objective and DPO, and utilize limited image transformations (e.g., only randomly cropping). In contrast, we introduce ViLP (Section 3 ) to assess visual reasoning and provide theoretical foundations (Appendix B ), alongside multi-category image corruptions (semantic editing, Gaussian blurring, pixelation). 3 ViLP  Benchmark 3.1 Design Principles “What’s the tall animal with the longest neck shown?” Humans readily guess “giraffe” based on learned priors, , yet as shown in the bottom-right of Figure 1 , it could be an elephant or camel – where visual reasoning corrects the answer. This highlights a potential shortfall in Vision-Language Models (VLMs), which may over-rely on learned visual language priors instead of true visual reasoning, particularly since VLMs are typically fine-tuned on limited image-text data, which is several orders of magnitude smaller than the trained text corpus ( Liu et al., 2024a ; Tong et al., 2024 ) . Specifically for the scope of visual language priors in this paper, we target (1) strong language priors that lead VLMs to derive answers solely from text, and (2) potential visual priors causing models to overlook critical uncommon visual cues (e.g., unusual zebra spots in Figure 1 ). To evaluate how VLMs handle learned visual language priors, we introduce ViLP, a specialized benchmark of out-of-distribution Question-Image-Answer (QIA) triplets guided by two core principles. First, text-only inference ensures that each question can be answered with high confidence using textual clues alone. Second, visual inference requires that the correct answer—sometimes contradicting common sense—only emerges once an out-of-distribution image is considered. By forcing models to integrate both textual and visual information, ViLP reveals whether they truly engage in visual reasoning or merely rely on memorized patterns. Mathematically, let Q be a question, I an image, and A=\{a_{\text{prior}},\dots,a_{\text{test}},\dots\} the set of possible answers. We define P(a\mid Q) as the probability of answer a given Q alone and P(a\mid Q,I) as the probability given both Q and I . We consider a prior model p , which may represent either human cognition ( P_{\text{human}} ) or a VLM/LLM’s learned visual-language prior ( P_{\theta} ). For constructing our benchmark, we used the following guidances: Criterion One : The question Q alone should strongly favor a_{\text{prior}} , where \delta_{1} is a high-confidence threshold. a_{\text{prior}} usually satisfies common knowledge, such as “soccer ball is a sphere" and “Einstein created E=mc^{2} " (Figure 1 ). P(a_{\text{prior}}\mid Q)\geq\delta_{1} (1) Criterion Two : With the image I , the correct answer shifts to a_{\text{test}} , where \delta_{2} is another high-confidence threshold. Also, a_{\text{test}} is significantly different from a_{\text{prior}} to ensure that the image has a substantial impact on the answer, where D is a divergence measure, and \delta_{3} is a threshold indicating significant difference. For instance, the image in the 1st row and the 3rd column of Figure 1 turns the answer to torus . P(a_{\text{test}}\mid Q,I)\geq\delta_{2},\,D\big(P(a_{\text{prior}}\mid Q),P(a_{\text{test}}\mid Q,I)\big)\geq\delta_{3} (2) Criterion Three : The answer a_{\text{test}} should be rare and unlikely from Q alone, while a_{\text{prior}} becomes clearly incorrect when considering I . This is enforced by a low-confidence threshold \delta_{4} (e.g., Newton as a_{\text{test}} inferred from the image, contradicting Einstein, shown by the image from the 3rd row and 2nd column of Figure 1 ). P(a_{\text{test}}\mid Q)\leq\delta_{4},\,P(a_{\text{prior}}\mid Q,I)\leq\delta_{4} (3) In designing ViLP, we leverage the human cognition prior P_{\text{human}} as our guiding principle, ensuring each QIA configuration aligns with typical human expectations while requiring visual evidence to override strong textual assumptions. We then compare the learned priors of VLMs and LLMs, denoted P_{\theta} , P_{\text{human}} to evaluate whether these models genuinely engage in visual reasoning rather than relying on memorized patterns. 3.2 Question-Image-Answer Generation Following Criterion Three , a_{\text{test}} should be highly improbable based on Q alone yet the correct choice when paired with I . Since such images do not exist naturally, we use generative models like DALL·E-3 ( Ramesh et al., 2021 ) and Flux to blend unusual elements that override typical language priors. We incorporate substantial human input and leverage advanced LLMs such as OpenAI-o1 and Claude-3.5-Sonnet to ensure alignment with all the criteria. More details, including text prompts and average cost, are provided in Appendix A.2 . Note that as more advanced image generative models—such as the recently introduced 4o image generation —become available, we anticipate generating increasingly abundant and high-quality data for our benchmark, yet future updates will remain consistent with the dataset construction criteria outlined in Section 3.1 . (a) Image-DPO vs. Text-DPO : In the green dashed box, we illustrate Image-DPO, which uses a single Q&A pair paired with multiple corrupted images. In contrast, the purple dashed box presents Text-DPO, involving a single input image paired with multiple, distinct Q&A pairs. (b) Illustration of Image DPO. We construct chosen and rejected pairs by corrupting the image with a set of perturbations while keeping the Q&A unchanged. Perturbations include semantic editing , Gaussian blurring , and pixelation . The mathematical formulations and implementation details are provided in Appendix B and Appendix C , respectively. For each question, we design three answers: one a_{\text{prior}} inferred solely from Q , and two a_{\text{test}} that defy language priors, requiring visual cues for correctness. We rely on GPT-4 to generate text prompts, produce large-scale images, and then conduct human filtering and refinement. This process faces two main challenges: (1) producing diverse out-of-distribution QA pairs, and (2) synthesizing images that defy specific priors, sometimes necessitating hundreds of samples to find one that accurately matches Q and a_{\text{test}} . Table 1 : Category Data . Type Frequency Texture 16 Shape 20 Conceptual combinations 276 Hallucinated Components 151 Proverbs 17 Ultimately, we curated 300 questions, each paired with three distinct image-answer sets, totaling 900 QIA triplets. These cover a broad range from low-level recognition ( texture , shape ) to high-level reasoning ( conceptual combinations , hallucinated components , proverbs ). Table 1 summarizes their categorical distribution, with each question spanning an average of 1.6 categories. To reinforce text priors, we present a distractor fact before the question . Rigorous human review ensures that all final QIA triplets are clear and interpretable, as reflected by our human evaluation results in Table 2 . Besides, in Appendix A.3 , we investigate the impact of image styles by generating more realistic images via 4o image generation and comparing them to those produced by DALL·E-3 ( Ramesh et al., 2021 ) and Flux . We find that realistic images can increase the difficulty of the task, highlighting their importance for future studies. 3.3 Dataset Evaluation All of our questions are designed to elicit single-word answers, an approach that is more efficient and more reliable than sentence-based evaluations that rely on LLM judgment. By avoiding using LLM, we reduce API fees, computational overhead, and the risk of occasional inaccuracies due to incorrect model reasoning. We explicitly instruct the model to provide a single-word answer, and we evaluate the correctness of each response using a binary system. To ensure a fair evaluation, we devote significant efforts to building a comprehensive set of synonyms and plural for each answer to detect other valid alternative answers. This ensures that the model is only penalized for actual errors, not for providing synonymous or alternative correct responses. 4 Image DPO Inspired by our benchmark, we propose Image DPO , a self-improvement method for enhancing VLMs’ visual reasoning, featuring a new objective and a data generation pipeline using VLMs themselves and pre-trained image models. 4.1 Objective Figure 3 : Qualitative examples. We show the results from GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro, and Llama-3.2-Vision-90B for some challenging cases. Please refer to Appendix A.6 for more failure case analysis. Existing approaches for VLM self-improvement follow the way used in DPO paper ( Rafailov et al., 2024 ) , where the model is trained to distinguish between good and bad answers for a fixed image and question (Figure 2(a) right). However, this straightforward adaptation may not the best for vision models, as the model sometimes distinguish good and bad answers from the text alone without needing to analyze the image. In contrast, we propose Image DPO , a vision-focused objective that creates good and bad question-image-answer pairs by corrupting the image while keeping the question and answer unchanged (Figure 2(a) left). An example of our synthetic data is illustrated in Figure 2(b) . Formally, given an image I_{w} , a question Q , and its corresponding answer A , we generate a corrupted image I_{l} via image-editing operations, including Gaussian blur, pixelation, or semantic modifications. The triplet (Q,I_{l},A) forms a degraded question-image-answer pair compared to (Q,I_{w},A) . We train the model to distinguish between good and bad triplets using the objective 4 , where \pi_{\theta} is the target VLM, \pi_{\text{ref}} is the reference VLM (typically an earlier version of \pi_{\theta} ), S is the dataset of good and bad triplets, \sigma is the sigmoid function, and \alpha is a scaling factor. \displaystyle L(\pi_{\theta},\pi_{\text{ref}}) \displaystyle=-\mathbb{E}_{Q,I_{w},I_{l},A\sim S}\Bigl[ (4) \displaystyle\log\sigma\Bigl(\alpha\frac{\pi_{\theta}(A\mid Q,I_{w})}{\pi_{{\text{ref}}}(A\mid Q,I_{w})}\;-\;\alpha\frac{\pi_{\theta}(A\mid Q,I_{l})}{\pi_{{\text{ref}}}(A\mid Q,I_{l})}\Bigr)\Bigr] Intuitively, since the textual inputs and outputs are identical in both good and bad cases, the gradients of this objective push the model to rely more on the vision branch, driving a shift in gradient direction when processing normal images I_{w} compared to corrupted images I_{l} (Figure 15 ). This behavior encourages the model to focus more on image inputs rather than relying solely on text-based reasoning, thereby enhancing its performance on visual-related tasks. Our experiments demonstrate Image-DPO objective (Eq. 4 ) outperforms varies self-improve VLM baselines on ViLP. Proposition 1 . Let \mathcal{L}_{\mathrm{RLHF}}(\pi_{\theta},\pi_{\mathrm{ref}};\mathcal{S}) be the KL-constrained reward maximization objective in Appendix Eq. 7 ( Rafailov et al., 2024 ) , where the dataset \mathcal{S}=\{(Q,A,I_{w},I_{l})\} contains good images I_{w} and corrupted images I_{l} . Let \mathcal{L}_{\mathrm{ImageDPO}}(\pi_{\theta},\pi_{\mathrm{ref}};\mathcal{S}) be the objective from Eq. 4 , which compares \bigl(Q,I_{w},A\bigr) against \bigl(Q,I_{l},A\bigr) . Then for any policy \pi_{\theta} and reference model \pi_{\mathrm{ref}} , we have \mathcal{L}_{\mathrm{RLHF}}(\pi_{\theta},\pi_{\mathrm{ref}};\mathcal{S})\;\;\leq\;\;\mathcal{L}_{\mathrm{ImageDPO}}(\pi_{\theta},\pi_{\mathrm{ref}};\mathcal{S}). Proof Sketch. Following ( Rafailov et al., 2024 ) , we express the optimal KL-constrained policy in terms of a latent reward function. Applying a Bradley–Terry preference model to question-image-answer triplets \bigl(Q,I_{w}/I_{l},A\bigr) and using Jensen’s inequality yields an upper bound whose minimization is equivalent to \mathcal{L}_{\mathrm{ImageDPO}}(\pi_{\theta},\pi_{\mathrm{ref}};\mathcal{S}) . A full derivation appears in Appendix B . 4.2 Data Generation Training VLMs demands large-scale question-image-answer (QIA) triplets, which are often scarce. To address this, we introduce a scalable data generation pipeline (Appendix Figure 16 ) that repurposes existing image datasets via VLM themselves and image generative models. Given a seed image from COCO ( Lin et al., 2014 ) , Text2VQA ( Singh et al., 2019b ) , or Visual Genome ( Krishna et al., 2017 ) , VLMs are tasked with simultaneously selecting appropriate functions (e.g., image generation or editing models) and generating corresponding instructions. These instructions are then used to produce new images, in addition to the seed image, as illustrated in Figure 17 . The same VLMs are then employed to generate QA pairs for these newly created images. Following, we apply the mentioned three types of image corruptions to the generated images, constructing good bad pars (Q,I_{w},A) and (Q,I_{l},A) . Specifically, we employ Stable Diffusion XL ( Podell et al., 2023 ; Rombach et al., 2022 ) for image generation, and use Instruct-Pix2Pix ( Brooks et al., 2023a ) , and Grounded-SAM ( Rombach et al., 2022 ; Ren et al., 2024 ) for image editing. Example generated data, prompts, and more details are included in Appendix C . 5 Experiments Table 2 : Benchmarking on ViLP . Please refer to the left text for symbol definitions. †  indicates the model often fails to follow the instructions. Model ViLP F ViLP P Score Prior Score Prior Baseline Human 98.33 99.67 98.67 96.67 GPT-4o (text only) 0.0 92.33 0.17 71.33 API call only GPT-4o 66.17 91.00 56.00 87.67 GPT-4V 57.67 88.33 38.33 85.33 GPT-4o-Mini 57.67 89.00 46.67 84.67 Claude-3.5-Sonnet 70.00 84.33 59.33 86.67 Claude-3-Opus 59.17 74.00 43.00 82.67 Claude-3-Sonnet 48.83 83.67 40.33 81.33 Claude-3-Haiku 43.67 82.67 34.83 82.33 Gemini-1.5-Pro 60.50 79.33 48.00 83.00 Gemini-1.5-Flash 54.50 83.33 69.17 79.67 Open weights Llama-3.2-Vision-11B 67.33 76.67 61.17 79.33 Llama-3.2-Vision-90B 64.00 91.67 63.17 83.33 MolmoE-1B 48.67 57.33 47.83 69.00 Molmo-7B-O 57.83 60.67 47.33 76.33 Molmo-7B-D 54.5 69.00 46.17 72.33 Molmo-72B 60.33 85.00 47.17 82.33 Qwen2-VL-7B 50.50 83.00 48.67 80.33 Qwen2-VL-72B 56.50 92.33 53.83 83.00 InternVL2-8B 47.00 66.67 43.00 75.00 InternVL2-76B 42.67 47.67 50.84 74.33 LLaVA-1.5-7B 29.67 71.33 37.67 65.67 LLaVA-1.5-13B 35.33 81.00 41.50 73.67 Cambrian-1-8B † 8.67 43.67 32.50 63.67 LLaVA-OneVision-7B 54.17 82.33 49.67 75.00 LLaVA-OneVision-72B † 1.67 3.00 5.22 11.67 We introduce ViLP, a new benchmark comprising 300 questions. Each question is paired with three unique images and their corresponding answers—one QIA prior and two QIA test —for a total of 900 QIAs. The QIA prior examples (300 in total) align with common language priors (i.e., they can usually be answered correctly by relying on textual cues alone). In contrast, the QIA test examples (600 in total) challenge these priors by requiring visual reasoning. ViLP features two evaluation settings: • ViLP F , where both distractor facts and the questions are provided; • ViLP P , where only the questions themselves are given (i.e., no distractor facts). We report two metrics in Table 2 : average accuracy on QIA test (noted as Score ) and average accuracy on QIA prior (noted as Prior ). Our benchmark emphasizes the performance in Score . Figure 4 : Comparison of benchmark scores under different image transformations. Solid line and dotted line refer to ViLP F -Score and ViLP F -Prior, respectively. Is the QIA easy for humans? We begin by evaluating our benchmark through a human study. Participants achieved nearly perfect accuracy on ViLP F -Prior and over 98% on ViLP F -Score and ViLP P -Score, confirming that our question-image-answer combinations are unambiguous for human interpretation. Notably, despite QIA test being designed as out-of-distribution examples, humans were still able to correctly distinguish them. Humans performed slightly better on ViLP F -Prior when distractor facts were provided, as they could easily identify that these facts aligned with the correct answers. Moreover, ViLP F -Score was marginally lower when facts were introduced, as the distractor facts added some noise and caused minor confusion, although the impact of this noise is relatively small. These findings are consistent with the design principles of our benchmark. Figure 5 : Qualitative results before and after removing distactor facts . GPT-4o and LLaVA-1.5-13B models yield completely opposite behaviors. Are our QIAs aligned with the learned priors of VLMs? We tested GPT-4o (text only) on our questions (removing all image references). Despite no visual content, it correctly answered 92.33% on ViLP F -Prior. The accuracy drops to 71.33% once distractor facts are removed, showing that these facts significantly guide the answer. For QIA test , GPT-4o (text only) accuracy nearly falls to 0% (ViLP F -Score & ViLP P -Score), indicating the QIA test cannot be answered using text alone. How do VLMs perform on our benchmark? Although our benchmark questions are distinguishable for humans, they are challenging for VLMs. Even the advanced VLM models like GPT-4o, have a clear performance gap (66.17% v.s. 98.33%) compared to humans’ performance on ViLP F -Score, indicating the difficulty of these questions for VLMs. Claude-3.5-Sonnet achieved the best score 70%, while most of the commercial VLMs are below 60%. Figure 3 highlights sample outputs from top commercial and open-source models, including GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro, and Llama-3.2-Vision-90B. They face significant challenges when addressing these cases in our ViLP, whereas humans can arrive at correct answers after consideration. Notably, it is encouraging to see that some open-source models achieved over 60% accuracy on ViLP F -Score, with performance nearing that of their commercial counterparts, including Llama-3.2-Vison and Molmo-72B. Additionally, we provide more detailed failure case analysis in Appendix A.6 . Table 3 : Effectiveness of Image-DPO on General VQA benchmarks. VLMs ViLP F {}_{\text{Score}} ViLP P {}_{\text{Score}} NB {}_{\text{Q}} NB {}_{\text{I}} NB {}_{\text{G}} NB {}_{\text{B}} MM-Vet CHAIR S \downarrow CHAIR I \downarrow LLaVA-1.5-7B 29.67 37.67 37.7 43.8 12.7 67.3 31.1 49.1 14.8 LLaVA-1.5-7B + Image-DPO 34.17 \uparrow 4.5 39.33 \uparrow 1.66 39.79 \uparrow 2.09 45.47 \uparrow 1.67 14.16 \uparrow 1.46 68.45 \uparrow 1.15 32.3 \uparrow 1.2 45 \uparrow -4.1 12.3 \uparrow -2.5 LLaVA-1.5-13B 35.33 41.5 39.6 44.6 14.8 68.9 36.1 48.3 14.1 LLaVA-1.5-13B + Image-DPO 38.17 \uparrow 2.84 42.5 \uparrow 1 42.68 \uparrow 3.08 47.37 \uparrow 2.77 17.16 \uparrow 2.36 70.36 \uparrow 1.46 37.5 \uparrow 1.4 42.6 \uparrow -5.7 11.6 \uparrow -2.5 Cambrian-8B 8.67 32.5 44.6 47.9 19.4 71.5 51.4 14.5 4.7 Cambrian-8B + Image-DPO 20.83 \uparrow 12.16 39.3 \uparrow 6.83 46.5 \uparrow 1.9 50.2 \uparrow 2.3 20 \uparrow 0.6 72 \uparrow 0.5 51.7 \uparrow 0.3 11.4 \uparrow -3.1 4.4 \uparrow -0.3 Do distractor facts really distract? In ViLP F setting, we add a distractor fact before the question . Since these facts implicitly suggest incorrect answers for QIA test , we expected this change to make the questions more suggestive and lower the ViLP F -Score, as the distractors would mislead the VLMs. Surprisingly, GPT-4o benefits from distractor facts , improving accuracy on QIA test . We hypothesize these facts highlight question focus, narrowing the search space. However, weaker models like LLaVA-1.5-13B ( Liu et al., 2023b ) often get misled by the distractors, hurting their Score but boosting Prior . For instance, as shown in Figure 5 , with including distractor facts , LLaVA-1.5-13B consistently predicts the distractor fact as the answer. However, once the distractors are removed, it can then predict correctly. For bad instruction-following models like Cambrian-8B ( Tong et al., 2024 ) , distractor facts significantly hinder adherence to explicit instructions, such as providing single-word answers. With facts, Cambrian-8B fails to follow instructions in 62% of cases, compared to 30% without (a nearly 2x increase). Manual review shows 59% of these failures are contextually correct, yielding an adjusted accuracy of 47.92%. Similarly, LLaVA-OneVision-72B ( Li et al., 2024b ) often generates detailed analyses despite explicit single-word prompts. This trend highlights a concerning trend: focusing on improving performance on well-established benchmarks may come at the cost of basic instruction-following abilities, ultimately limiting the practical utility of these models in real-world applications. How image transformations affect the results? We also investigate how image transformations, including resizing, Gaussian blur, and pixelation, affect ViLP  performance. The results, shown in Figure 4 , reveal that the ViLP F -Score rapidly decreases as the severity of the transformations (x-axis) increases, while the ViLP F -Prior score remains around 50%. Interestingly, GPT-4o, when using degraded images, performs worse in ViLP F -Prior than when no images are used, i.e., GPT-4o (text only) in Table 2 . Comparison of ViLP with other VQA datasets: To highlight the distinctions between ViLP and existing benchmarks, including Winoground ( Thrush et al., 2022 ) , Whoops! ( Bitton-Guetta et al., 2023a ) , and HallusionBench ( Guan et al., 2023 ) , we conduct a comparative analysis of both their high-level design principles and low-level data formats. This comparison incorporates qualitative and quantitative insights as detailed in Appendix A.4 . Table 4 : Comparisons of Image-DPO on ViLP. Model ViLP F ViLP P Score Prior Score Prior LLaVA-1.5-7B 29.67 71.33 37.67 65.67 +HADPO ( Zhao et al., 2023b ) 33.00 74.33 38.50 65.00 +RLHF-V ( Yu et al., 2023a ) 29.50 75.00 36.33 65.33 +EOS ( Yue et al., 2024 ) 31.33 67.00 38.67 65.67 +SIMA ( Wang et al., 2024b ) 27.83 68.67 36.17 66.00 +V-DPO ( Xie et al., 2024 ) 29.5 0 72.67 37.83 67.67 +Text-DPO 31.34 71.67 37.83 65.67 +Image-DPO 34.17 75.00 39.33 68.00 5.1 Image DPO In this section, we evaluate Image-DPO (Section 4.1 ) on both ViLP  and general VQA benchmarks. As an ablation baseline , we introduce Text-DPO , which uses the same Question-Image-Answe (QIA) generation process as we used in Image-DPO but applies LLM self-rewarding objective ( Yuan et al., 2024 ) (the standard DPO objective). In Text-DPO, good and bad pairs stem from VLM-generated positive and negative answers, while the question and image remain fixed. As shown in the right of Figure 2(a) , the green box depicts Image-DPO, generating corrupted images via semantic edits, Gaussian blur, and pixelation while keeping the question and answer constant; the purple box illustrates Text-DPO, which fixes the image and varies the answers with associated ratings. This setup parallels other VLM self-rewarding work ( Zhou et al., 2024a ; Deng et al., 2024 ; Zhou et al., 2024b ; Wang et al., 2024c ; Wang et al., 2024a ) . For baselines, we compare with VLM self-improvement methods, including SIMA ( Wang et al., 2024c ) , HADPO ( Zhao et al., 2023a ) , and EOS ( Yue et al., 2024 ) , by using their publicly available checkpoints. Additionally, we train models using the dataset and code provided in RLHF-V ( Yu et al., 2023a ) and V-DPO ( Xie et al., 2024 ) . All models use LLaVA-7B for a comprehensive comparisons as many paper only release 7B checkpoints. Table 4 shows that Image-DPO achieves the highest across all the metrics. Besides, we evaluate the proposed Image-DPO algorithm across three VLM models—Cambrian-8B, LLaVA-1.5-7B, and LLaVA-1.5-13B—using several popular VLM benchmarks that focus on different aspects, including compositionality & biases (NaturalBench ( Li et al., 2024a ) ), general visual reasoning (MM-Vet ( Yu et al., 2023c ) ), and hallucinations (CHAIR ( Rohrbach et al., 2018 ) ). The results, presented in Table 3 , show consistent performance improvements across both datasets and models, further demonstrating the effectiveness of our Image DPO method. TextDPO on Corrupted Images. Does Image-DPO’s improvement stem from the objective itself, or is it merely due to training on more perturbed data? To investigate this, we conduct an ablation where we train TextDPO with corrupted images. Specifically, for each pair of (Q,I,A_{w}) and (Q,I,A_{l}) used in Text-DPO, we apply the same corruptions as Image-DPO to I to form (Q,I^{\prime},A_{w}) and (Q,I^{\prime},A_{l}) . Results show that LLaVA-v1.5-7B trained with this TextDPO variant achieves a ViLP F -Score of 31, a ViLP P -Score of 37.5, and 30.3 on MMVET. In contrast, ImageDPO outperforms it across all metrics, achieving a ViLP F -Score of 34.17, a ViLP P -Score of 39.33, and 32.3 on MMVET. 6 Conclusion In conclusion, we present the ViLP  benchmark to probe the challenge of visual language bias in Vision-Language Models (VLMs). By utilizing advanced image generation models and designing questions that demand visual cues for accurate responses, our benchmark includes images that defy language priors, revealing the limitations of current VLMs. Our method, Image-DPO, which incorporates self-generated VQA pairs and image corruption for training, has demonstrated promising improvements in enhancing visual reliance, as evidenced by performance gains on models like LLaVA-v1.5 and Cambrian. 7