Regularisation in neural networks: a survey and empirical analysis of approaches

Christiaan P. Opperman, Anna S. Bosman, Katherine M. Malan · 2026

Read Regularisation in neural networks: a survey and empirical analysis of approaches by Christiaan P. Opperman, Anna S. Bosman, Katherine M. Malan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23131.txt.

Introduction

Christiaan P. Opperman, Anna S. Bosman, and Katherine M. Malan survey methods that help neural networks generalize. Networks ace many tasks, yet they can stumble on data they have not seen. A large family of tricks, called regularisation, has grown up over the years to fight that weakness.

The authors review those approaches and test them empirically side by side. A clear map of what works, and at what cost, saves practitioners from folklore. Generalization is the difference between a demo and a tool people can rely on. The empirical analysis keeps the survey honest, testing whether celebrated techniques still earn their place in current practice.

Catalog record

CREATORChristiaan P. Opperman, Anna S. Bosman, Katherine M. Malan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-23131

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Regularisation in neural networks: a survey and empirical analysis of approaches Christiaan P. Opperman, Anna S. Bosman, Katherine M. Malan Abstract Despite huge successes on a wide range of tasks, neural networks are known to sometimes struggle to generalise to unseen data. Many approaches have been proposed over the years to promote the generalisation ability of neural networks, collectively known as regularisation techniques. These are used as common practice under the assumption that any regularisation added to the pipeline would result in a performance improvement. In this study, we investigate whether this assumption holds in practice. First, we provide a broad review of regularisation techniques, including modern theories such as double descent. We propose a taxonomy of methods under four broad categories, namely: (1) data-based strategies, (2) architecture strategies, (3) training strategies, and (4) loss function strategies. Notably, we highlight the contradictions and correspondences between the approaches in these broad classes. Further, we perform an empirical comparison of the various regularisation techniques on classification tasks for ten numerical and image datasets applied to the multi-layer perceptron and convolutional neural network architectures. Results show that the efficacy of regularisation is dataset-dependent. For example, the use of a regularisation term only improved performance on numeric datasets, whereas batch normalisation improved performance on image datasets only. Generalisation is crucial to machine learning; thus, understanding the effects of applying regularisation techniques, and considering the connections between them is essential to the appropriate use of these methods in practice. † † impactstatement: Generalisation is one of the most important properties of a neural network (NN), as it determines whether the NN would produce usable outputs outside of the lab setting. Current practices of improving NN generalisation via regularisation techniques are haphazard and unreliable. Our work sheds light on the interplay between the various existing regularisation techniques, and points out directions for future research that can streamline regularisation and make it more robust. We demonstrate that some popular methods of regularisation, such as dropout, may be ineffective on smaller models and datasets, and challenge the universality of regularisation techniques. We show that there is a strong need to tailor the regularisation approach to the task at hand, and that no one-size-fits-all solution exists. I Introduction Generalisation ability is the performance of an artificial neural network (NN) on previously unseen data instances. Generalisation is one of the most important characteristics of NNs, as it determines the applicability of the final model to real-world data. Even though multiple theories have been proposed in an attempt to explain generalisation, varying from linking generalisation to geometric properties of the loss landscape [ 43 ] to attributing generalisation to the size and structure of the neural architectures [ 6 ] , this phenomenon is still not well understood [ 48 , 59 , 104 ] . The generalisation theories have, however, led to the development of regularisation techniques , which are commonly used to potentially improve generalisation. Some regularisation techniques were discovered incidentally, through research that investigated approaches to improve other properties of a NN, such as memory usage [ 10 ] . Due to the lack of theoretical studies, regularisation techniques do not guarantee improved generalisation in NNs. Furthermore, the choice of which approach to use is a non-trivial design decision. In this paper, we survey the existing regularisation techniques, and make the following specific contributions: 1. We propose a taxonomy of regularisation techniques (summarised in Fig. 1 ) under four broad categories, namely: data-based strategies, architecture strategies, training strategies, and loss function strategies. Each category is broken down into specific subcategories and approaches. This visual overview of the wide range of alternatives to promote generalisation provides a useful starting point for researchers and practitioners working in the field. 2. For each set of techniques proposed, we provide a brief description of the rationale underlying the general approach, and give references to sources that provide further details to guide researchers and practitioners on the implementation of the techniques. 3. We highlight contradictions and correspondences between regularisation techniques, suggesting that certain combinations of approaches may be in conflict with each other and should be used with caution until further studies provide guidance on ways to handle these apparent contradictions. 4. Finally, to evaluate the relative effectiveness of the regularisation techniques, we perform benchmark tests on two distinct NN architectures (multi-layer perceptron and convolutional neural network), on ten different datasets (numerical and image classification) for the following techniques: geometric transformation, SMOTE, weight perturbation, pruning, dropout, batch normalisation, layer normalisation, weight normalisation, and regularisation terms. 5. We observe that regularisation performance varies significantly across datasets and NN architectures, and in many cases, regularisation hinders rather than helps. Fig. 1 : A tree representation of the proposed taxonomy of regularisation methods. Related work. This paper is not the first survey of regularisation techniques. Kukačka et al. [ 62 ] defined a high-level taxonomy of regularisation techniques based on which part of the NN the techniques affect: data, NN architecture, loss function, regularisation term, or optimisation. The taxonomy proposed in our paper is similar to Kukačka et al.’s [ 62 ] , but our survey includes additional strategies under loss functions and, furthermore, discusses the contradictions and correspondences between these techniques. Further, Kukačka et al.’s [ 62 ] review was carried out in 2017, and therefore excludes some of the most recent discoveries such as double descent and sharpness optimisation. Moradi et al. [ 73 ] performed a review of regularisation techniques in 2020, but did not provide a usable taxonomy, nor discuss the links and contradictions between the approaches. Similarly, in 2022, Tian and Zhang [ 96 ] performed a theoretical survey of regularisation techniques in machine learning, but did not provide a taxonomy or compare the approaches. Survey by Santos et al. [ 92 ] only considers more recent regularisation techniques, and focuses on image datasets and convolutional neural networks. Our paper puts prior work in a wider modern context of understanding generalisation, and provides a more holistic and practical view of regularisation. A traditional review approach was followed for the survey of techniques. We used the surveys discussed above as a basis to establish the main categories of the taxonomy. For each of the main categories, existing category-specific surveys were reviewed to identify subcategories. Further, we adopted a snowballing approach by mining the reference lists of the review articles, forward-tracking of citations in time, and performing target-specific searches on generalisation and regularisation techniques. Paper structure. This paper is organised into two main sections. Firstly, Section II describes the proposed taxonomy, providing a detailed view of the four categories of approaches that have been proposed to promote generalisation, and the contradictions and correspondences between them. Secondly, Section III describes the benchmark tests that were performed on a subset of the regularisation techniques, and discusses the benchmarking results. Section IV concludes the paper. II Taxonomy Fig. 1 summarises the proposed taxonomy of regularisation techniques. Each of the main categories (data-based strategies, architecture strategies, training strategies, and loss function strategies) is discussed below in Sections II-A to II-D . Section II-E provides an overview of the contradictions and correspondences between the families of regularisation techniques. II-A Data-based strategies Data-based regularisation strategies manipulate the training dataset

Read the complete text