Shortcut Learning Susceptibility in Vision Classifiers
Pirzada Suhail & Amit Sethi · 2025
Read Shortcut Learning Susceptibility in Vision Classifiers by Pirzada Suhail & Amit Sethi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09150.txt.
Introduction
Suhail, Goel, and Sethi tested in twenty twenty-five how prone three kinds of pic sort tools are to learn-ing short cuts in place of true traits. Short cuts are false cues in data that cut train loss but fail on new pics. They plant cues tied to class by spot and by bright-ness in a test set.
Tools are then read on sets with and without the planted cues to gauge their lean. At low learn rates one tool kind, the swirl net, held back from the cues most. Glass-like tools with no spot code took the cues in near full and lost the true traits. Learn rate shifts the lean in all kinds.
Catalog record
| CREATOR | Pirzada Suhail & Amit Sethi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Shortcut Learning Susceptibility in Vision Classifiers Pirzada Suhail & Amit Sethi Abstract Shortcut learning, where machine learning models exploit spurious correlations in data instead of capturing meaningful features, poses a significant challenge to building robust and generalizable models. This phenomenon is prevalent across various machine learning applications, including vision, natural language processing, and speech recognition, where models may find unintended cues that minimize training loss but fail to capture the underlying structure of the data. Vision classifiers such as Convolutional Neural Networks (CNNs), Multi-Layer Perceptrons (MLPs), and Vision Transformers (ViTs) leverage distinct architectural principles to process spatial and structural information, making them differently susceptible to shortcut learning. In this study, we systematically evaluate these architectures by introducing deliberate shortcuts into the dataset that are positionally correlated with class labels, creating a controlled setup to assess whether models rely on these artificial cues or learn actual distinguishing features. We perform both quantitative evaluation by training on the shortcut-modified dataset and testing them on two different test sets—one containing the same shortcuts and another without them—to determine the extent of reliance on shortcuts. Additionally, qualitative evaluation is performed by using network inversion-based reconstruction techniques to analyze what the models internalize in their weights, aiming to reconstruct the training data as perceived by the classifiers. We evaluate shortcut learning behavior across multiple benchmark datasets, including MNIST, Fashion-MNIST, SVHN, and CIFAR-10, to compare the susceptibility of different vision classifier architectures to shortcut reliance and assess their varying degrees of sensitivity to spurious correlations. 1 Introduction Machine learning models are expected to learn meaningful patterns from data to make accurate predictions and generalize well across different domains. However, in many cases, models do not learn the intended task-relevant features but instead rely on shortcut learning, where they exploit spurious correlations in the training data that happen to be predictive of the labels. Rather than learning intended patterns, models often latch onto unintended statistical correlations present in the training data, leading to poor generalization across different domains. While shortcut learning may improve performance on in-distribution test data, it significantly degrades model robustness when evaluated on out-of-distribution samples in a real-world settings where these spurious correlations no longer hold. This issue is not limited to computer vision but extends to various domains, including natural language processing (NLP), where models rely on dataset biases rather than true linguistic understanding, and speech recognition, where models can leverage background noise instead of learning speech patterns. In computer vision, different classification architectures process input data in fundamentally distinct ways, which may lead to varying tendencies toward shortcut learning. Convolutional Neural Networks (CNNs) operate through convolutional filters that extract spatial patterns, Multi-Layer Perceptrons (MLPs) learn feature representations in a fully connected manner without spatial biases, and Vision Transformers (ViTs) leverage self-attention mechanisms to model long-range dependencies. Since each of these architectures encodes spatial and structural information differently, their responses to shortcuts may vary. However, the specific nature of their susceptibility to shortcut learning remains unclear. In this paper, we aim to systematically evaluate how CNNs, MLPs, and ViTs learn from data when shortcut cues are deliberately introduced, providing insights into their respective tendencies toward shortcut reliance and generalization. To systematically assess shortcut susceptibility, we introduce deliberate shortcuts into the training dataset. Specifically, we modify certain pixel regions in the images to correlate deterministically with class labels, embedding artificial cues that serve as an unintended yet easily learnable path to minimize training loss. These modifications serve as an alternative path for models to reduce training error, creating a controlled setup where we can analyze whether models rely on these shortcuts or learn the actual distinguishing features in the images. To measure the extent to which models depend on shortcut cues, we train CNNs, MLPs, and ViTs on the modified datasets and evaluate them on two test sets: one containing the same shortcuts and another without any shortcuts. If a model primarily learns the artificial shortcut instead of the actual class-relevant features, its performance is expected to drop significantly when evaluated on the test set without shortcuts. By comparing the performance gap across architectures, we quantify each model’s reliance on shortcuts and determine whether certain architectures are inherently more vulnerable. Beyond performance metrics, we also conduct qualitative evaluations using network inversion-based reconstruction techniques to analyze what the models internalize in their weights. By reconstructing the training data as perceived by the classifiers, we investigate whether the models primarily store shortcut-based representations or capture meaningful semantic features. We also explore how learning rate variations impact shortcut reliance across different architectures. Models trained with a low learning rate may learn more nuanced and meaningful features, whereas a high learning rate may encourage faster convergence at the cost of relying on simpler, easily learnable patterns, such as shortcuts. We conduct experiments across multiple benchmark datasets, including MNIST, Fashion-MNIST, SVHN, and CIFAR-10 to provide insights into how different vision architectures process data and how susceptible they are to shortcut learning. 2 Related Works Shortcut learning is a phenomenon where machine learning models prioritize learning simple, potentially misleading cues from data that do not generalize well beyond the training set. This issue has been identified as a fundamental limitation of deep learning models, leading to poor robustness and transferability Geirhos et al. 2020 . Shortcut learning has been observed across various domains, including vision, natural language processing, and speech recognition, where models often exploit unintended correlations in the training data to minimize loss instead of learning meaningful patterns. The study in Hermann et al. 2024 examines shortcut learning from a theoretical perspective by investigating the factors that influence whether a model will rely on a shortcut. The authors propose that a feature’s predictivity and availability determine whether it will be used as a shortcut, with more easily extractable (but less predictive) features being favored over harder-to-extract (but more predictive) features. This phenomenon is particularly concerning in medical AI, where it has been linked to algorithmic unfairness. In Brown et al. 2023 , the authors propose a method to detect shortcut learning in clinical machine learning models by applying multitask learning to identify improper correlations that may cause biased predictions. Beyond classification, the phenomenon has also been studied in medical image segmentation, where commonly used dataset preparation techniques, such as zero-padding and center-cropping, introduce unintended shortcuts that influence segmentation accuracy Lin et al. 2024 . The paper Bleeker et al. 2024 investigates shortcut learning in vision-language models and evaluates how contrastive learning-based models tend to latch onto unintended patterns in multi-caption training scenarios. By injecting synthetic shortcuts into image-text data, the authors show that contrastive losses often fail to encourage models to learn all task-relevant information, instead reinforcing the simplest available features. Shortcut learning has also been studied in the context of Vision Transformers. The work in Ma et al. 2024 explores how ViTs might be particularly prone to shortcut learning due to their reliance on self-attention mechanisms. The authors introduce a saliency-guided ViT model that leverages computational visual saliency maps to guide ViTs toward learning meaningful features rather than background artifacts. This paper exploits network inversion-based reconstruction techniques to analyze different model architectures for their shortcut learning susceptibility. Inversion has been studied in Kindermann & Linden 1990 ; Jensen