Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs
Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani · 2026
Read Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs by Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01710.txt.
Description
Hasanebrahimi and co-authors recalibrate zero-shot vision-language scores by embedding density. Models like CLIP classify well with no task training. Their predictions stay sensitive to spurious correlations, though. Context cues can dominate over semantic content in outputs. Fine-tuning and prompt fixes carry their own costs and risks. Density-Aware Translation refines image-text similarity scores instead. A local geometric density term, from group reference sets, does the work. CLIP embeddings show a modality gap and an anisotropic shell shape.
Common patterns cluster near the mean of that shell. Rare patterns get pushed outward, skewing alignment. Their relative measure rescales scores by density. Overconfident scores in diffuse regions get suppressed. Benchmark tests improve worst-group and average accuracy consistently.
Catalog record
| CREATOR | Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01710