Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models

Ashish Seth, Dinesh Manocha, Chirag Agarwal · 2024

Read Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models by Ashish Seth, Dinesh Manocha, Chirag Agarwal (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20622.txt.

Description

A 2024 arXiv research paper by Ashish Seth, Dinesh Manocha, Chirag Agarwal. Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks. However, these models still suffer from hallucinations, particularly when required to implicitly recognize or infer diverse visual entities from images for complex vision-language tasks. To address this challenge, we propose Hallucinogen , a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks to evaluate the extent of hallucination in state-of-the-art LVLMs.

Catalog record

CREATORAshish Seth, Dinesh Manocha, Chirag Agarwal
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20622

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models Ashish Seth, Dinesh Manocha, Chirag Agarwal Abstract Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks. However, these models still suffer from hallucinations, particularly when required to implicitly recognize or infer diverse visual entities from images for complex vision-language tasks. To address this challenge, we propose Hallucinogen , a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks to evaluate the extent of hallucination in state-of-the-art LVLMs. Our benchmark provides a comprehensive study of the implicit reasoning capabilities of these models by first categorizing visual entities based on the ease of recognition in an image as either salient (prominent, visibly recognizable objects such as a car) or latent entities (such as identifying a disease from a chest X-ray), which are not readily visible and require domain knowledge or contextual reasoning for accurate inference. Next, we design hallucination attacks for both types of entities to assess hallucinations in LVLMs while performing various vision-language tasks, such as locating or reasoning about specific entities within an image, where models must perform implicit reasoning by verifying the existence of the queried entity within the image before generating responses. Finally, our extensive evaluations of eleven LVLMs, including powerful open-source models (like LLaMA-3.2 and DeepSeek-V2), commercial models like Gemini, and two hallucination mitigation strategies across multiple datasets, demonstrate that current LVLMs remain susceptible to hallucination attacks. 1 Introduction Figure 1: Examples of different object hallucination attacks, where hallucination prompts from Hallucinogen (right) are able to make the LVLM hallucinate response. (Left) When explicitly asked to identify a non-existent object, such as “person,” LVLMs like LLaVA1.5 Liu et al. 2024b generate a correct response. (Right) However, in the case of an implicit object hallucination attack, where the question requires first implicitly determining an object’s presence before describing its position, the LVLMs produce a hallucinated response. In recent years, Large Language Models (LLMs) have made significant advancements in natural language understanding and natural language generation, significantly advancing the field of artificial intelligence Achiam et al. 2023 ; Dubey et al. 2024 ; Zhao et al. 2023 . Building on the exceptional capabilities of LLMs, researchers have developed Large Vision-Language Models (LVLMs), which have demonstrated outstanding performance on multimodal tasks such as image captioning and VQA Zhu et al. 2023 ; Ye et al. 2023 ; Wang et al. 2024 ; Dubey et al. 2024 ; Liu et al. 2024b . These models use LLMs as their foundational architecture, integrating visual features as supplementary inputs and aligning them with textual features through visual instruction tuning Liu et al. 2023 ; Liu et al. 2024b . Despite these advancements, LVLMs continue to struggle with the issue of hallucination — a phenomenon characterized by the misidentification or misclassification of visual objects in an image Li et al. 2023 ; Lovenia et al. 2023 . This potentially leads to harmful consequences, especially when users lacking sufficient domain knowledge place undue reliance on these models. Hallucinogen vs. Existing Benchmarks. Prior works have introduced a series of benchmarks Lovenia et al. 2023 ; Li et al. 2023 ; Guan et al. 2023 ; Yin et al. 2024 and mitigation strategies Leng et al. 2024 ; Huang et al. 2024 ; Zhou et al. 2023 to evaluate and mitigate hallucinations in LVLMs. However, as illustrated in Fig. 1 , we find that existing benchmarks predominantly rely on explicit closed-form attacks , which directly prompt the underlying LVLM to identify a specific visual entity, such as a “car,” expecting a simple “Yes” or “No” response. For example, POPE Li et al. 2023 utilizes simple visual object detection prompts like “Is <object> present in the image?” . In contrast, Hallucinogen introduces implicit open-form hallucination attacks , which pose a more significant challenge for LVLMs to defend against. For instance, in a complex vision-language task that requires the model to identify the surrounding visual context of a specific object using a prompt like, “Describe the context and surrounding of the <object> in the image.” , LVLMs must first implicitly verify whether the object mentioned in the prompt is present in the image before generating a factually accurate response. This additional layer of reasoning increases the likelihood of LVLMs mistakenly assuming the presence of a visual entity due to pre-existing biases from strong LLM priors, such as spurious correlations between non-existent objects and the overall visual scene Liu et al. 2024a ; Liu et al. 2025 . Main Contribution. To address these shortcomings, we propose Hallucinogen , a novel benchmark for evaluating hallucinations in LVLMs. Unlike existing benchmarks, which primarily rely on simple, single-object identification prompts, Hallucinogen introduces a diverse set of contextual-reasoning prompts, which we call as hallucination attacks . We categorize these attacks into two types: explicit and implicit hallucination attacks. Prior benchmarks have shown to mainly focus on explicit attacks, where LVLMs are directly asked to identify non-existent visual entities in an image, often leading to hallucinated responses. In contrast, we introduce implicit attacks, which employ more complex and indirect queries. Rather than explicitly asking about a specific entity, these prompts leverage contextual or relational cues in the visual and textual input, inducing LVLMs to infer visual entities not present in a target image. Additionally, based on the visual ease of recognizing entities in an image, we categorize them as either salient or latent entities. Salient entities refer to prominent, visibly recognizable objects, like a “car,” that can be easily identified without requiring additional context. In contrast, latent entities are those that are not readily visible and necessitate domain knowledge or contextual reasoning for accurate inference, e.g., diagnosing a “disease” from a biomedical image like a chest X-ray. Furthermore, we design implicit hallucination attacks for both types of entities and utilize these attacks to identify hallucinated responses when LVLMs are challenged with complex vision-language tasks such as locating or reasoning about specific visual entities in an image. We summarize our main contributions below: Figure 2: Illustration of various types of hallucination attacks in Hallucinogen . We broadly define two categories of hallucination attacks: explicit and implicit attacks. An explicit attack involves directly prompting LVLMs to accurately identify the presence or absence of existing or non-existing visual entity. In contrast, an implicit attack employs more complex queries that do not explicitly inquire about a specific visual entity but instead require the model to implicitly assess its presence in the image to generate a factually accurate response. Furthermore, for implicit attacks, we propose a range of visual-language tasks with varying levels of difficulty, from correctly locating a visual entity to understanding its surrounding context. • We propose Hallucinogen , a novel benchmark for evaluating hallucination in LVLMs. Unlike prior benchmarks, Hallucinogen introduces a diverse set of complex contextual reasoning prompts, referred to as hallucination attacks, specifically designed to query LVLMs about visual entities that may not be present in a target image. Our benchmark consists of 90,000 image-prompt pairs with 6,000 visual-entity pairs equally divided between salient and latent entities. Furthermore, for robust evaluation, each image is associated with 15 diverse implicit hallucination attack prompts. • We show that LVLMs are also capable of hallucinating reasoning and using Chain-of-Thought reasoning

Read the complete text