Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?
Yiwei Yang, Chung Peng Lee, Shangbin Feng, Dora Zhao, Bingbing Wen, Anthony Z. Liu, Yulia Tsvetkov, Bill Howe, [1ex] University of Washington · 2025
Read Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? by Yiwei Yang, Chung Peng Lee, Shangbin Feng, Dora Zhao, Bingbing Wen, Anthony Z. Liu, Yulia Tsvetkov, Bill Howe, [1ex] University of Washington (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18322.txt.
Introduction
Yiwei Yang and co-authors built a test set to catch AI models that lean on false clues, in a 2025 study. Large models that read images and text can learn shortcuts: features that hint at answers without being the real reason. Past tests for this flaw used staged tasks. This team mined real errors made by GPT-4o on photo question sets instead.
After sifting by model and human checks plus fake counter-tests, the set held 124 kinds of false clue. Each kind pairs one real item with ten made ones, for 1,364 questions in all. Fifteen models were tried, and the best scored only about thirty-seven percent. Training on made examples that stress the clues lifted scores sharply, which hints that varied practice teaches models to look at the whole scene.
Catalog record
| CREATOR | Yiwei Yang, Chung Peng Lee, Shangbin Feng, Dora Zhao, Bingbing Wen, Anthony Z. Liu, Yulia Tsvetkov, Bill Howe, [1ex] University of Washington |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18322
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? Yiwei Yang, Chung Peng Lee, Shangbin Feng, Dora Zhao, Bingbing Wen, Anthony Z. Liu, Yulia Tsvetkov, Bill Howe, [1ex] University of Washington Abstract Finetuning can cause spurious correlations to arise between non-essential features and the target labels, but benchmarks to study these effects involve contrived settings and narrow tasks. In contrast, we consider spurious correlations in multi-modal Large Vision Language Models (LVLMs) pretrained on extensive and diverse datasets without explicit task supervision. We develop a benchmark by sourcing GPT-4o errors on real-world visual-question-answering (VQA) benchmarks, then curating a subset through LVLM-human annotation and synthetic counterfactual evaluation to identify errors caused by spurious correlations. This process yields SpuriVerse, a novel benchmark comprised of 124 distinct types of spurious correlations extracted from real-world datasets, each containing 1 realistic and 10 synthetic VQA samples for a total of 1364 multiple choice questions. We evaluate 15 open and closed-source LVLMs on SpuriVerse, finding that even state-of-the-art closed-source models struggle significantly, achieving at best only 37.1% accuracy. Fine-tuning on synthetic examples that emphasize the spurious correlation improves performance to 78.40%, suggesting that training on diverse spurious patterns generalizes to unseen situations: models appear to learn to avoid "shortcuts" and attend to the overall image context. * * footnotetext: Equal contribution. 1 Introduction Real-world data frequently contains spurious correlations —- patterns predictive of the target during training but irrelevant to the true label ( Leino et al. 2018 ) . For example, in ImageNet, butterflies often appear with flowers ( Singla and Feizi 2021 ) , while the Waterbirds dataset was constructed to have waterbird images commonly appear on water backgrounds ( Sagawa et al. 2019 ) . Models trained on such data may learn to rely on these spurious features (backgrounds, co-occurring objects, texture, etc.), leading to errors when the correlations no longer hold. An example of such errors is a landbird incorrectly classified as a waterbird simply due to its water background ( Sagawa et al. 2019 ) . This phenomenon is especially problematic in critical domains such as medicine, where models trained to detect pneumonia might rely on unrelated features (e.g., metal tokens) rather than genuine disease indicators ( Zech et al. 2018 ) . In traditional supervised training where a model is finetuned for a downstream application, spurious correlation arises from the association between spurious features and target labels captured by the model through standard loss minimization ( Sagawa et al. 2019 ; Kirichenko et al. 2022 ) . In the regime of pre-training Large Vision-Language Models (LVLMs), not only are the training data more diverse and extensive, but the objective is no longer predicting a task-specific target label. In this generalized regime, we hypothesize that correlations between spurious features and target concepts can still persist and compromise LVLMs’ performance. However, modern LVLMs’ evaluation suites do not examine whether zero-shot LVLMs rely on spurious features to make incorrect predictions, where fine-tuning for a particular target label is not available. In this work, we provide a benchmark to evaluate an LVLM’s susceptibility to spurious correlations in generalized settings. In particular, we are interested in spurious correlations that appear outside of contrived tasks where the correlation is a result of non-representative training data such as Gender -to- BlondHair in CelebA ( Liu et al. 2015 ) and Background -to- BirdType in WaterBirds ( Sagawa et al. 2019 ) . Instead of contrived spurious correlations, we focus on situations where a model may overrely on a dominant correlation that is not necessarily irrelevant but is not generally reliable (Figure 1 ). We address two important challenges: (1) Existing benchmarks often study spurious correlations by collecting a training set with an imbalanced distribution across spurious features and target labels ( Sagawa et al. 2019 ; Lynch et al. 2023 ; Joshi et al. 2023 ; Li et al. 2023b ; Arjovsky et al. 2019 ) . These benchmarks are particularly useful for investigating whether a model learns spurious correlations when trained on a skewed distribution. However, when evaluating pre-trained LVLMs, curating an imbalanced dataset is ineffective due to distribution shifts. (2) The presence of spurious correlations often relies on annotations of spurious features and target label . For example, in CelebA ( Liu et al. 2015 ) , the comprehensive list of annotated face attributes allows one to recognize a strong correlation between gender and hair color in the training set. Observing that the trained model achieves low accuracy on non-blond males then confirms that the spurious correlation is captured by the model. However, this approach only applies to a specific target label identified beforehand. While one may consider annotating an extensive list of potential spurious features, and examining correlation among all possible pairs, it is still impractical to annotate them on LVLM’s web-scale pre-training data. Instead of guessing the candidates for spurious features , we choose a bottom-up approach, described in Figure 2 : we (1) identify errors of a strong LVLM on real-world visual-question-answering (VQA) benchmarks; (2) derive a subset of errors attributable to spurious correlations via human-LVLM collaboration; then (3) validate these derived samples by evaluating multiple LVLMs on synthetic counterfactual examples with and without the spurious features. A significant accuracy drop when spurious feature(s) are present indicates the model’s reliance on spurious correlation. This process yields SpuriVerse, a novel multimodal benchmark comprised of 124 distinct spurious correlations extracted from real-world datasets ( Li et al. 2024a ; Schwenk et al. 2022 ; Li et al. 2023a ; Li et al. 2024b ) , each containing 11 VQA samples—1 from the original dataset, referred to as the anchor and 10 synthetic examples, generated as part of the validation process, referred to as the spurious group —for a total of 1364 multiple choice questions involving visual understanding. SpuriVerse affords evaluation of the susceptibility of LVLMs to diverse spurious correlations in a generalized setting. Moreover, this diversity enables investigation of whether models can learn to ignore spurious correlations as a meta-skill to generalize to unseen situations . Figure 1 : SpuriVerse consists of 124 distinct spurious correlations, each of which is comprised of 1 original example and 10 synthetic images. Both original and synthetic images share the same multiple-choice question. Each image contains a feature that is spuriously correlated with one of the choices, causing the model to make an error. We evaluate on 15 LVLMs, finding that even state-of-the-art closed-source models such as Qwen-VL-Max struggle significantly, achieving only 37.10% accuracy. No prompt-based methods such as as Chain-of-Thought ( Wei et al. 2022 ) effectively mitigates the vulnerability. However, fine-tuning on a subset of SpuriVerse’s anchor and evaluating on the held-out set of anchor , improves accuracy from 35.20% to 45.60% on previously unseen (not fine-tuned) spurious correlations for Qwen-2.5-VL-7b-Instruct. Furthermore, we boost the performance to 78.40% by fine-tuning the same model on the spurious group counterparts of the original training subset, where the held-out set is still previously unseen spurious correlations. These results suggest that models can learn a general meta-skill to avoid being distracted by a dominant correlation and pay more attention to the overall scene. However, we also observe a trade-off in overall performance (14% on Qwen-2.5-VL-7b-Instruct), suggesting that models