The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks
João Manoel Herrera Pinheiro, Suzana Vilas Boas de Oliveira, Thiago Henrique Segreto Silva, Pedro Antonio Rabelo Saraiva, Enzo Ferreira de Souza, Leonardo André Ambrosio, Marcelo Becker · 2025
Read The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks by João Manoel Herrera Pinheiro, Suzana Vilas Boas de Oliveira, Thiago Henrique Segreto Silva, Pedro Antonio Rabelo Saraiva, Enzo Ferreira de Souza, Leonardo André Ambrosio, Marcelo Becker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08274.txt.
Introduction
The authors test how data prep choices, called feature scaling, affect machine learning in 2025. Scaling puts inputs on a shared range before training. Past studies covered few methods. This team tries twelve scaling methods across fourteen learning methods. Sixteen data sets cover both class and number tasks.
Scores track accuracy and error, plus training time and memory use. Tree based group methods did well with or without scaling. Other common methods swung a lot with the choice of scaler. Those swings hit both scores and cost. All code and results are shared in public. The study gives model by model guidance on when scaling matters. It should help users pick a sound method.
Catalog record
| CREATOR | João Manoel Herrera Pinheiro, Suzana Vilas Boas de Oliveira, Thiago Henrique Segreto Silva, Pedro Antonio Rabelo Saraiva, Enzo Ferreira de Souza, Leonardo André Ambrosio, Marcelo Becker |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 137 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08274
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks João Manoel Herrera Pinheiro, Suzana Vilas Boas de Oliveira, Thiago Henrique Segreto Silva, Pedro Antonio Rabelo Saraiva, Enzo Ferreira de Souza, Leonardo André Ambrosio, Marcelo Becker Abstract This research addresses the critical lack of comprehensive studies on feature scaling by systematically evaluating 12 scaling techniques - including several less common transformations - across 14 different Machine Learning algorithms and 16 datasets for classification and regression tasks. We meticulously analyzed impacts on predictive performance (using metrics such as accuracy, MAE, MSE, and R²) and computational costs (training time, inference time, and memory usage). Key findings reveal that while ensemble methods (such as Random Forest and gradient boosting models like XGBoost, CatBoost and LightGBM) demonstrate robust performance largely independent of scaling, other widely used models such as Logistic Regression, SVMs, TabNet, and MLPs show significant performance variations highly dependent on the chosen scaler. This extensive empirical analysis, with all source code, experimental results, and model parameters made publicly available to ensure complete transparency and reproducibility, offers model-specific crucial guidance to practitioners on the need for an optimal selection of feature scaling techniques. I Introduction Machine Learning progress has been notorious in several domains of knowledge engineering, notably driven by the rise of big data [ 1 , 2 ] , its applications in healthcare [ 3 , 4 , 5 , 6 ] , forecasting [ 7 , 8 , 9 , 10 ] , precision agriculture [ 11 ] , wireless sensor networks [ 12 ] , language tasks [ 13 , 14 , 15 ] and many other domains [ 16 , 17 , 18 , 19 , 20 , 21 ] . All different applications compose the field of Machine Learning [ 14 ] , which has become a major subarea of computer science and statistics due to its crucial role in the modern world [ 22 ] . Although these methods hold immense potential for the advancement of predictive modeling, their improper application has introduced significant obstacles [ 23 , 24 , 25 ] . One such obstacle is the indiscriminate use of preprocessing techniques, particularly feature scaling [ 26 ] . Feature scaling is a mapping technique in a preprocessing stage by which the user tries to give all attributes the same weight [ 27 , 28 ] . In some applications, this data transformation can improve the performance of Machine Learning models [ 29 ] . Consequently, applying a scaling method without a careful evaluation of its suitability for the specific problem and model may be inadvisable and could negatively impact results. This practice risks undermining the validity of the claims regarding model performance and may lead to a feedback loop of overconfidence in the results, known as overfitting [ 30 ] . Reproducibility is a critical problem in Machine Learning [ 31 , 32 , 33 , 34 ] . It is often undermined by factors such as missing data or code, inconsistent standards, and sensitivity to training conditions [ 35 ] . Feature scaling, in particular, if not documented or applied correctly, can significantly affect model performance and hinder the replication of results. The absence of rigorous evaluation not only hampers reproducibility, but can also lead to the adoption of practices with poor generalizability across different datasets or domains. As Machine Learning methods continue to shape research by their use in a wide range of applications, it is essential to critically assess and justify each step of the modeling pipeline, including feature scaling, to ensure robust and replicable findings [ 36 ] . The primary objective of this study is to evaluate the impact of different data scaling methods on the training process and performance metrics of various Machine Learning algorithms across multiple datasets. We employ 14 widely used Machine Learning models for tabular data, including Linear Regression, Logistic Regression, Support Vector Machines (SVM), K-Nearest Neighbors, Multilayer Perceptron, Random Forest, TabNet, Naive Bayes, Classification and Regression Trees (CART), Gradient Boosting Trees, AdaBoost, LightGBM, CatBoost, and XGBoost. These models were evaluated using 12 different data scaling techniques, in addition to a baseline without scaling, across 16 datasets covering both classification and regression tasks. The selected models represent the state of the art in tabular data analysis, offering a favorable balance between predictive performance and computational efficiency, often outperforming deep learning techniques in this context [ 37 , 38 , 39 , 40 ] . In Section II we cover some related work in similar studies while Section III explains more about each algorithm, each feature scaling technique, the study diagram, evaluation metrics, and how the models were trained. In Section IV we represent the final results of this study and some discussion. The limitations of our study are discussed in Section V . Lastly, in Section VI we give a final conclusion of the current experiments and future works. II Related Work Despite its fundamental role in Machine Learning pipelines, the impact of feature scaling remains an underexplored area in the literature. Most existing studies examine only a limited number of algorithms [ 41 ] or datasets, and often provide minimal analysis of the specific effects of different scaling techniques [ 42 ] on each particular algorithm’s performance. Studies that comprehensively evaluate various scaling methods across a broad range of models and datasets, such as the approach taken in this work, are scarce. In many Machine Learning cases, preprocessing is briefly mentioned, with scaling treated as a routine step rather than a variable worthy of in-depth investigation. For some Machine Learning models, feature scaling is extremely necessary, such as K-Nearest Neighbors [ 43 ] , Neural Networks [ 44 , 45 ] [ 46 ] , and SVM [ 47 ] [ 48 ] . In object detection, data scaling has a crucial impact [ 49 ] . In [ 50 ] , the authors compared six normalization methods in an SVM classifier to improve intrusion data, and the Min-Max Normalization showed the best performance. In [ 51 ] , the authors evaluated eleven Machine Learning algorithms, including Logistic Regression, Linear Discriminant Analysis, K-Nearest Neighbors, Classification and Regression Trees, Naive Bayes, Support Vector Machine, XGBoost, Random Forest (RF), Gradient Boosting, AdaBoost, and Extra Tree, across six different data scaling methods: Normalization, Z-score Normalization, Min-Max Normalization, Max Normalization, Robust Scaler, and Quantile Transformer. However, they focused on only one dataset, the UCI - Heart Disease [ 52 ] . Despite that, their results are interesting; models based on Decision Trees showed the best performance without any scaling method, while K-Nearest Neighbors and Support Vector Machine achieved the lowest performance. In a work on diabetes diagnosis using models such as Random Forest, Naive Bayes, K-Nearest Neighbors, Logistic Regression and Support Vector Machine [ 53 ] the researchers compared only three preprocessing scenarios: Normalization, Z-score Normalization, and no feature scaling. Their findings suggested that Random Forest, Naive Bayes, and Logistic Regression showed little sensitivity to these specific scaling approaches, and in some cases, their performance even worsened post-scaling. Another problem is data leakage and the reproducibility of Machine Learning models. As shown in [ 36 ] , many studies used preprocessing steps, such as