LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models DOI: XXXXXXX.XXXXXXX CCS: Computing methodologies Computer vision CCS: Computing methodologies Visual languages CCS: Computing methodologies Natural language processing CCS: Computing methodologies Visual content-based indexing and retrieval
Zhiyuan Jiang, Weihao Hong, Xinlei Guan, Tejaswi Dhandu, Miles Q. Li, Meng Xu, Kuan Huang, Umamaheswara Rao Tida, Bingyu Shen, Daehan Kwak, Boyang Li · 2026
Read LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models DOI: XXXXXXX.XXXXXXX CCS: Computing methodologies Computer vision CCS: Computing methodologies Visual languages CCS: Computing methodologies Natural language processing CCS: Computing methodologies Visual content-based indexing and retrieval by Zhiyuan Jiang, Weihao Hong, Xinlei Guan, Tejaswi Dhandu, Miles Q. Li, Meng Xu, Kuan Huang, Umamaheswara Rao Tida, Bingyu Shen, Daehan Kwak, Boyang Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18803.txt.
Introduction
Jiang and ten colleagues present Ghost-100, a benchmark of 800 images in eight categories and three task families, where each target is absent, illegible, or indeterminate by design. Every image pairs with five prompts from a five level intensity framework, varying only directive force so tone alone varies. A dual track protocol measures crossing from grounded refusal to unsupported commitment, and the confidence of fabrication once it occurs. Across nine open weight vision language models, the two measures dissociate, and several models peak in sensitivity at intermediate tone levels.
Catalog record
| CREATOR | Zhiyuan Jiang, Weihao Hong, Xinlei Guan, Tejaswi Dhandu, Miles Q. Li, Meng Xu, Kuan Huang, Umamaheswara Rao Tida, Bingyu Shen, Daehan Kwak, Boyang Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64146 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18803
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models DOI: XXXXXXX.XXXXXXX CCS: Computing methodologies Computer vision CCS: Computing methodologies Visual languages CCS: Computing methodologies Natural language processing CCS: Computing methodologies Visual content-based indexing and retrieval Zhiyuan Jiang, Weihao Hong, Xinlei Guan, Tejaswi Dhandu, Miles Q. Li, Meng Xu, Kuan Huang, Umamaheswara Rao Tida, Bingyu Shen, Daehan Kwak, Boyang Li Abstract . Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding has operational consequences, yet their behavior under coercive prompt phrasing remains undercharacterized. Existing hallucination benchmarks rely on neutral prompts and binary detection, leaving open how the incidence and intensity of fabrication respond to graded linguistic pressure across distinct task types. We present Ghost-100, a benchmark of 800 synthetic images spanning eight categories across three task families: text-illegibility, time-reading, and object-absence. Each image follows a negative-ground-truth principle guaranteeing the queried target is absent, illegible, or indeterminate by construction. Every image is paired with five prompts from a 5-Level Prompt Intensity Framework, isolating tone as the sole independent variable. We adopt a dual-track evaluation: a rule-based H-Rate measuring how often models cross from grounded refusal into unsupported commitment, and a GPT-4o-mini-judged H-Score (1-5) characterizing the confidence and specificity of fabrication. We also release a three-stage automated validation workflow, which confirms 717 of 800 images as strictly compliant. Evaluating nine open-weight VLMs, we find that H-Rate and H-Score dissociate across model families, reading-style and presence-detection subsets respond differently to prompt pressure, and several models exhibit non-monotonic sensitivity peaking at intermediate tone levels: patterns that aggregate metrics obscure. 1. Introduction Vision-Language Models (VLMs) have demonstrated strong capability in visual question answering, image captioning, and multimodal reasoning ( 11 ; 16 ) , and are now integrated into applications where reliable visual grounding carries operational consequences — including medical image analysis, security screening, and decision support systems. In such settings, hallucination poses a particular risk: the model generates content unsupported by the visual input, most critically when a user prompt presupposes the existence of an object, person, or attribute that is simply not present ( 20 ; 36 ) . Unlike factual errors in text-only models, visual hallucination is difficult to detect because a fabricated response may be syntactically fluent and contextually plausible even when it directly contradicts the image ( 25 ) . Compounding this, current VLMs exhibit a tendency toward sycophantic behavior : when a prompt strongly implies a particular answer, the model may defer to the prompt’s implicit expectation rather than the visual evidence, producing fabricated content with high confidence ( 34 ; 25 ) . This susceptibility is not uniformly distributed across models or task types, and the question of how hallucination behavior evolves under prompt phrasing and escalating pressure remains the central concern of this work ( 7 ; 9 ) . Figure 1 . Overview of our framework. The Ghost-100 benchmark comprises 800 synthetically generated images across 8 categories and 3 task families. All 800 images are paired with five escalating prompt tones (L1–L5) and evaluated on 9 open-weight VLMs. In parallel, a retrospective three-stage quality audit finds that 717/800 images satisfy the workflow’s strict compliance criteria. Stronger prompt pressure consistently induces higher H-Rate and H-Score in susceptible models, while robust models maintain stable refusal across all levels. Prior work has produced a range of VLM hallucination benchmarks probing object presence, attribute accuracy, and spatial relations ( 13 ; 19 ; 25 ; 31 ) , but three gaps limit their ability to characterize pressure-induced fabrication. First, prompt manipulation is typically treated as a binary factor distinguishing adversarial from benign phrasing ( 22 ) , which collapses the graded continuum along which real-world user prompts actually escalate — from neutral inquiry, to insistence, to coercive demand — and leaves open how hallucination behavior evolves under monotonically increasing pressure. Second, evaluation almost universally relies on binary detection , reporting whether a hallucination occurs but not how confidently or richly the model commits to fabricated content; this conflates a hedged guess with an elaborately specified false claim, even though the two carry very different practical risks ( 5 ) . Third, task coverage concentrates on object presence and absence ( 3 ; 19 ) , while categories defined by finer-grained visual constraints — such as text rendered illegible or a temporal display made structurally unreadable — are largely absent, leaving an important class of failure modes unexamined. Compounding these gaps, the field also lacks a principled mechanism for verifying that benchmark images faithfully implement their intended design constraints, a prerequisite for reproducible evaluation ( 29 ; 38 ) . We address these gaps with Ghost-100 , a procedurally constructed benchmark of 800 synthetically generated images spanning eight categories across three task families: text-illegibility , time-reading , and object-absence . As shown in Figure 1 , the three families are selected to cover qualitatively distinct forms of absent evidence — textual content that is present but rendered unreadable, temporal information that is structurally obscured, and objects entirely absent from the scene despite strong contextual cues suggesting their presence. All images are synthetically generated under a negative-ground-truth principle that guarantees the queried target is verifiably absent or indeterminate by construction, eliminating the label noise inherent in curated natural image datasets ( 14 ; 29 ) and ensuring that unsupported affirmative responses can be interpreted as hallucinations with minimal annotation ambiguity. To probe how prompt pressure shapes model behavior, each image is paired with five prompts drawn from a structured 5-Level Prompt Intensity Framework (L1–L5), ranging from neutral inquiry to explicit coercive demand, with the image and task identity held fixed so that tone is isolated as the sole independent variable. Model responses are evaluated under a dual-track protocol that decouples the two axes of hallucination behavior: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment, and a GPT-4o-mini-judged H-Score on a 1–5 scale characterizing the confidence and specificity of fabrication once it occurs. We additionally release a three-stage automated validation workflow — covering perceptual screening, design compliance, and semantic discriminability — as a reusable quality-assurance tool, and apply it retrospectively to the full 800-image collection, where it confirms 717 images as strictly compliant under conservative criteria. Evaluation is conducted on nine representative open-weight VLMs ( 1 ; 30 ; 32 ; 35 ) using two complementary metrics: a binary H-Rate measuring whether the model commits to a hallucinated response, and a fine-grained 1–5 H-Score assessing the degree of fabrication. Results show that H-Rates vary substantially across models under identical conditions, and that the reading-style and presence-detection task subsets respond to prompt pressure in qualitatively different ways. A particularly notable pattern is that several models exhibit near-zero hallucination on the presence-detection subset regardless of pressure level, while remaining highly susceptible on reading-style tasks — suggesting that robustness is task-specific rather than a general property of the model ( 7 ; 19 ) . At the highest pressure level, time-reading categories approach hallucination saturation across most models, and non-monotonic behavior observed in the presence-detection subset points to conflicting signals between task comprehension and safety alignment ( 2 ; 8 ) . These findings indicate that hallucination vulnerability is governed jointly by