When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu · 2026

Read When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models by Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08245.txt.

Description

Harshvardhan Saini, Samyak Jha, Yiming Tang, and Dianbo Liu trace vision-language model hallucinations to a geometric cause. Decoder-based models must align visual embeddings with the text space for attention to work. The authors argue the alignment overshoots. Visual features are pulled so far onto the language manifold that language bias drowns fine visual detail.

The study gives the first measured account of this over-alignment. It shows the language bias sits in the top principal components of the representation. Rather than costly black-box decoding fixes, the authors pursue geometric debiasing at the source. The direction has direct stakes for medical imaging and autonomous systems.

Catalog record

CREATORHarshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08245

Browse the full library · About the Tower of Babel