Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani · 2026

Read Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs by Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01710.txt.

Description

Hasanebrahimi and co-authors recalibrate zero-shot vision-language scores by embedding density. Models like CLIP classify well with no task training. Their predictions stay sensitive to spurious correlations, though. Context cues can dominate over semantic content in outputs. Fine-tuning and prompt fixes carry their own costs and risks. Density-Aware Translation refines image-text similarity scores instead. A local geometric density term, from group reference sets, does the work. CLIP embeddings show a modality gap and an anisotropic shell shape.

Common patterns cluster near the mean of that shell. Rare patterns get pushed outward, skewing alignment. Their relative measure rescales scores by density. Overconfident scores in diffuse regions get suppressed. Benchmark tests improve worst-group and average accuracy consistently.

Catalog record

CREATORAfsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01710

Browse the full library · About the Tower of Babel