Data Leakage and Deceptive Performance: A Critical Examination of Credit Card Fraud Detection Methodologies

Khizar Hayat,University of Nizwa,Sultanate of Oman [email protected], Baptiste Magnier · 2025

Read Data Leakage and Deceptive Performance: A Critical Examination of Credit Card Fraud Detection Methodologies by Khizar Hayat,University of Nizwa,Sultanate of Oman [email protected], Baptiste Magnier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02703.txt.

Introduction

Khizar Hayat and Baptiste Magnier show how sloppy evaluation can fake excellence in credit card fraud detection, in this 2025 paper. The field chases fancy models, yet four flaws recur in its papers: data leakage from careless preprocessing, vague method reporting, weak time-based validation, and recall chasing that guts precision.

In their case study, a minimal neural network fed leaked data beat most published systems on paper, with near-perfect recall that meant nothing. The authors argue rigor must come before architecture, a lesson they extend to machine learning research at large.

Catalog record

CREATORKhizar Hayat,University of Nizwa,Sultanate of Oman [email protected], Baptiste Magnier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02703

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data Leakage and Deceptive Performance: A Critical Examination of Credit Card Fraud Detection Methodologies Khizar Hayat,University of Nizwa,Sultanate of Oman [email protected], Baptiste Magnier Abstract This study critically examines the methodological rigor in credit card fraud detection research, revealing how fundamental evaluation flaws can overshadow algorithmic sophistication. Through deliberate experimentation with improper evaluation protocols, we demonstrate that even simple models can achieve deceptively impressive results when basic methodological principles are violated. Our analysis identifies four critical issues plaguing current approaches: (1) pervasive data leakage from improper preprocessing sequences, (2) intentional vagueness in methodological reporting, (3) inadequate temporal validation for transaction data, and (4) metric manipulation through recall optimization at precision’s expense. We present a case study showing how a minimal neural network architecture with data leakage outperforms many sophisticated methods reported in literature, achieving 99.9% recall despite fundamental evaluation flaws. These findings underscore that proper evaluation methodology matters more than model complexity in fraud detection research. The study serves as a cautionary example of how methodological rigor must precede architectural sophistication, with implications for improving research practices across machine learning applications. Keywords Credit Card Fraud, Data Leakage, Preprocessing Flaws, Methodological Rigor, Model Performance 1 INTRODUCTION The rise of deep learning has profoundly transformed how we approach complex classification and regression problems. Given its data-hungry nature, the prevailing emphasis has been on acquiring large volumes of training data to boost model performance. Meanwhile, the implementation of deep learning models has become increasingly trivialized, thanks to the proliferation of high-level APIs and libraries—especially in Python—that abstract much of the underlying complexity. Today, even large language mode… However, this convenience comes at a cost. The focus has shifted so heavily toward automation and performance metrics that critical aspects of the modeling pipeline are often overlooked or poorly documented. Researchers frequently delegate key steps to automated tools, sometimes without fully understanding or reporting them. In doing so, we risk putting the cart before the horse —prioritizing model tuning over foundational concerns such as data quality and preparation. The initial steps that precede model training are often loosely grouped under the term “preprocessing,” yet they receive scant attention in many studies. It is not uncommon for papers to devote pages to re-explaining ubiquitous evaluation metrics like accuracy, precision, recall, or confusion matrices, while providing vague or incomplete details on more foundational questions such as: • How were categorical, ordinal, or fuzzy variables handled? • What strategy was used for data splitting (e.g., random, stratified, time-based)? • When and how was normalization or standardization applied; and to which subsets? • Were oversampling or undersampling techniques limited to the training set, or did they inadvertently affect the test data? • Was feature selection or dimensionality reduction performed before or after data splitting? This vagueness often extends to the methodological core. Sleek pipeline diagrams are commonly included but tend to omit essential details. For instance, one might encounter the use of 2D Convolutional Neural Networks (CNNs) applied to tabular data without any justification. Was there a compelling reason for employing a spatial model on non-spatial input? If so, how was the data reshaped to accommodate this architecture? Such decisions are non-trivial and require clear, transparent reporting. These questions are far from minor. They directly affect the reproducibility, interpretability, and trustworthiness of machine learning models. When unaddressed, they may introduce silent data leakage, bias, or misleading performance metrics; ultimately undermining the credibility of the research. Thorough documentation and transparency in preprocessing are not optional; they are essential to rigorous, responsible data science. In this work, we undertake a critical review of existing literature in light of the concerns previously discussed, using credit card fraud detection as a case study-specifically focusing on a widely referenced benchmark dataset [ 1 ] . This domain poses distinctive challenges, with extreme class imbalance being a primary issue. In such contexts, seemingly high accuracy scores are not uncommon, yet they can be misleading. Methodological oversights or missteps, whether inadvertent or otherwise, can easily lead to inflated performance metrics and a skewed perception of model efficacy. One recurring issue we have observed is the mishandling of resampling techniques - particularly when oversampling or undersampling is performed before the train-test split - leading to data leakage. To underscore this point, we intentionally apply a simple yet flawed MLP-based approach. Despite its simplicity, the model yields impressively high metrics, thereby demonstrating how superficial performance gains can mask deeper methodological flaws. The rest of this paper is organized as follows: Section provides background on credit card fraud detection and dataset characteristics. Section 3 reviews the literature on deep learning-based fraud detection. Section 4 describes our methodology, followed by experimental results in Section . Finally, Section 5 concludes the paper. 2 Credit Card Fraud Detection The widespread use of credit and debit cards has greatly enhanced financial convenience, but it has also led to increased fraudulent activity. Payment card fraud typically involves unauthorized transactions intended to obtain goods, services, or cash. While fraud represents a tiny fraction of all transactions, the absolute financial impact is substantial [ 2 ] , making it a significant area of concern for financial institutions. 2.1 Dataset Description We use the popular European credit card fraud dataset [ 1 ] for benchmarking. It contains 284{,}807 transactions made by European cardholders over two days in September 2013, out of which only 492 are fraudulent, just 0.17\% , highlighting extreme class imbalance. The dataset comprises 30 features: 28 anonymized principal components (V1 - V28) derived via PCA, along with Time and Amount . The target variable Class is binary, with 1 indicating fraud and 0 denoting legitimate transactions. Due to privacy concerns, the original feature labels and semantics were withheld. Figure 1: Snapshot of the first five rows of the dataset. 2.2 Handling Imbalanced Data In fraud detection, the misclassification of minority class instances (fraudulent cases) carries significant consequences, as failing to detect fraud can lead to financial losses and security risks. Traditional classifiers often bias toward the majority class due to the skewed distribution of data, leading to poor performance on the minority class. Resampling techniques address this issue by adjusting the class distribution, either by oversampling the minority class, undersampling the majority class, or combining both approaches to improve model performance. 2.2.1 Oversampling Methods Oversampling techniques aim to increase the representation of the minority class by either duplicating existing samples or generating synthetic samples. These methods help the model learn patterns from the minority class more effectively. • Random Oversampling : Duplicates minority samples without introducing new information. This method is simple but can lead to overfitting due to repeated samples. • SMOTE (Synthetic Minority Oversampling Technique) [ 3 ] : Generates synthetic samples between nearest neighbors of minority instances. This method helps to create a more balanced dataset and can improve model performance. • ADASYN (Adaptive Synthetic Sampling) [ 4 ] : Focuses on generating synthetic samples for harder-to-classify minority samples through adaptive weighting. This method is particularly useful for improving the classification of difficult minority instances. • Borderline-SMOTE [ 5 ] : Creates synthetic samples near class boundaries for better discrimination. This method helps to improve the classification of minority instances near the decision boundary. 2.2.2 Undersampling Methods Undersampling techniques aim to reduce

Read the complete text