Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering
Michal Štefánik Timothee Mickus Marek Kadlčík Michal Spiegel Josef Kuchař · 2025
Read Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering by Michal Štefánik Timothee Mickus Marek Kadlčík Michal Spiegel Josef Kuchař (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18407.txt.
Introduction
Michal Štefánik and co-authors asked in 2025 whether testing AI models on unfamiliar data truly exposes their reliance on shortcuts. Question answering models often pass such tests by latching onto shallow cues rather than reading for meaning.
The team compared popular shifted test sets against a catalogue of known shortcut failures in question answering. The shifted sets varied wildly in quality: some spotted shortcut reliance worse than a plain in family test. Part of the problem is that the same shortcuts appear in both the home and shifted data. Skill at training on a set also proved a poor guide to its value for testing. The authors give a firmer method for judging generalization tests.
Catalog record
| CREATOR | Michal Štefánik Timothee Mickus Marek Kadlčík Michal Spiegel Josef Kuchař |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18407
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering Michal Štefánik Timothee Mickus Marek Kadlčík Michal Spiegel Josef Kuchař Abstract A large body of recent work assesses models’ generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. Despite their practicality, such evaluations build upon a strong assumption: that OOD evaluations can capture and reflect upon possible failures in a real-world deployment. In this work, we challenge this assumption and confront the results obtained from OOD evaluations with a set of specific failure modes documented in existing question-answering (QA) models, referred to as a reliance on spurious features or prediction shortcuts. We find that different datasets used for OOD evaluations in QA provide an estimate of models’ robustness to shortcuts that have a vastly different quality, some largely under-performing even a simple, in-distribution evaluation. We partially attribute this to the observation that spurious shortcuts are shared across ID+OOD datasets, but also find cases where a dataset’s quality for training and evaluation is largely disconnected. Our work underlines limitations of commonly-used OOD-based evaluations of generalization, and provides methodology and recommendations for evaluating generalization within and beyond QA more robustly. 1 Introduction Improving the generalization of language models (LMs), i.e., their capability to perform well beyond patterns covered by their limited training data Chollet 2019 ; Guo et al. 2023 , presents one of the most important challenges in modern NLP, with direct implications to their practical applicability in a wide variety of tasks. The most common approach towards evaluating LMs’ ability to generalize is to assess their performance on so-called out-of-distribution (OOD) datasets: datasets of the same task(s) but of different origins — an approach made practical by the large variety of pre-existing datasets for an extensive array of NLP tasks. We expect a model that generalizes well to achieve high scores on OOD data. However, as we hold limited knowledge of the properties of datasets, it is difficult to ensure that OOD datasets can comprehensively capture real-world failures to generalize observed in LMs’ practical applications. Figure 1: How to pick the most robust model? Ranking of popular QA models by two facets of generalization: Out-of-Distribution evaluations (left) and Reliance on prediction shortcuts/spurious features (right); models are ordered by the average ranking across all spurious features (last column). On the other hand, a growing body of work has been documenting and classifying the systematic generalization failures of LMs as due to prediction bias Utama et al. 2020 , prediction shortcuts Mikula et al. 2024 , or a reliance on spurious features Zhou et al. 2021 . These failures are characterized by a model’s over-reliance on a feature that can explain the training data well but is not representative of the task overall. In this case study, we confront these two orthogonal views on generalization — OOD performance and non-reliance on shortcuts — by using them to establish two sets of independent rankings for a selection of the most popular QA models. We focus on extractive QA where previous work provides the most extensive documentation of prediction shortcuts, allowing for a robust assessment. We find that many OOD datasets can accurately portray models’ robustness to shortcuts, but an uninformed selection of OOD datasets can also deliver a ranking that is not at all correlated with a reliance on shortcuts, underperforming even traditional in-domain evaluations. Finally, by assessing reliance on shortcuts for models trained on datasets used as OOD, we show that different QA datasets exhibit the same types of shortcuts. Nevertheless, we find that a dataset’s usefulness in uncovering shortcuts does not entail that it can be used to train more robust models. 2 Background Evaluating generalization in QA Evaluations on out-of-distribution (OOD) datasets are decisively the most popular method for evaluating generalization of language models — to the point that performance on OOD is often even referred interchangeably with the term of generalization Yang et al. 2023a . Within QA, among many others, Awadalla et al. 2022 train models on SQuAD Rajpurkar et al. 2016 as in-distribution (ID) and evaluate generalization on TriviaQA Joshi et al. 2017 , NewsQA Trischler et al. 2017 , SearchQA Dunn et al. 2017 or NaturalQuestions Kwiatkowski et al. 2019 as OOD. Clark et al. 2019 evaluates even shortcut-eliminating method by training on SQuAD and evaluating on AdversarialQA Jia and Liang 2017 as OOD. Yogatama et al. 2019 train on SQuAD as ID and evaluate on TriviaQA as OOD. Despite its known blindspots, SQuAD still remains the default training dataset for a majority of the most popular QA models on HuggingFace. With a primary objective of improving generalizations, these works assume that OOD evaluations can also uncover reliance on non-representative, spurious features. 1 1 1 While we focus on QA, we can easily find such an assumption in other tasks, including NLI ( Du et al. 2021 ; Korakakis and Vlachos 2023 ) or classification Yang et al. 2023b . In this work, we question this assumption and find cases where OOD evaluations are largely independent from a reliance on non-representative, spurious features. Prediction shortcuts A complementary yet less prevalent approach to assessing models’ generalization aims to exploit functional failures identified in previous models. One approach towards this goal consists in identifying models’ prediction shortcuts , i.e. a reliance on spurious features that are not representative for the learned task in general. Such shortcuts were previously identified in NLI Nie et al. 2020 , in-context learning Wei et al. 2023 or question answering Mikula et al. 2024 . While these shortcuts are difficult to identify, with their knowledge, we can test the model specifically for the reliance on each of the shortcuts. This can be done by constructing synthetic data Clark et al. 2019 or subsetting existing data Mikula et al. 2024 into subset(s) where we make sure that a specific shortcut is not applicable. In NLI, where the impact of specific prediction shortcuts is widely studied, a common practice is to evaluate on datasets specifically constructed to exploit reliance on shortcuts McCoy et al. 2019 . However, in open-ended tasks including QA or multitask benchmarks such as MMLU Hendrycks et al. 2021 , the applicability of shortcuts across different datasets used for OOD evaluations remains under-studied. 3 Methodology Figure 2: Kendall \tau Correlation (a ratio of changed orderings ) between the rankings provided by two different facets of generalization: ( rows ) performance on OOD datasets; and ( columns ) reliance on prediction shortcuts. First row and column contain an average rankings across all evaluated datasets and shortcuts. Our goal is to uncover if, and to what extent can commonly-performed OOD evaluations capture models’ failures attributed to models’ reliance on prediction shortcuts. We approach this by reproducing several OOD evaluations used in previous work and comparing the results of these evaluations with models’ measured sensitivity to prediction shortcuts representing previously reported failures modes. Models For our assessments, we aim to pick a set of existing QA models that are the most widely-used in practice. Towards this goal and with a minor preference for diversity, we pick five among the thirty most-downloaded models on HuggingFace 2 2 2 https://huggingface.co/models?pipeline_tag=question-answering&sort=downloads ; accessed 21/04/25. The selected models total over 2.4 million monthly