Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
Chengsheng Zhang, Chenghao Sun, Xinyan Jiang, Wei Li, Xinmei Tian · 2026
Read Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models by Chengsheng Zhang, Chenghao Sun, Xinyan Jiang, Wei Li, Xinmei Tian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25642.txt.
Description
Zhang and four colleagues propose Prefill-Time Intervention, a new way to cut hallucinations in large vision-language models. Past steering-vector methods worked only at decoding time, where errors pile up step by step and worsen later outputs. Prefill-Time Intervention acts just once during prefill, improving the initial key-value cache before errors can accumulate. It is modality-aware, steering keys toward visually grounded objects and values to filter background noise. Tests show strong hallucination cuts across decoding strategies, models, and benchmarks.
Catalog record
| CREATOR | Chengsheng Zhang, Chenghao Sun, Xinyan Jiang, Wei Li, Xinmei Tian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56198 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25642