Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models

Chengsheng Zhang, Chenghao Sun, Xinyan Jiang, Wei Li, Xinmei Tian · 2026

Read Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models by Chengsheng Zhang, Chenghao Sun, Xinyan Jiang, Wei Li, Xinmei Tian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25642.txt.

Description

Zhang and four colleagues propose Prefill-Time Intervention, a new way to cut hallucinations in large vision-language models. Past steering-vector methods worked only at decoding time, where errors pile up step by step and worsen later outputs. Prefill-Time Intervention acts just once during prefill, improving the initial key-value cache before errors can accumulate. It is modality-aware, steering keys toward visually grounded objects and values to filter background noise. Tests show strong hallucination cuts across decoding strategies, models, and benchmarks.

Catalog record

CREATORChengsheng Zhang, Chenghao Sun, Xinyan Jiang, Wei Li, Xinmei Tian
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56198
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25642

Browse the full library · About the Tower of Babel