What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics
Kunwoong Kim, Dongha Kim · 2026
Read What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics by Kunwoong Kim, Dongha Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29791.txt.
Introduction
Kim and Kim study outlier detection, the task of flagging data points that do not fit the normal pattern, in the fully unsupervised setting where no labelled anomalies exist. Their focus is the inlier-memorization effect: deep models learn normal patterns earlier in training than abnormal ones.
Working with a simple autoencoder, a model that learns by reconstructing its input. The authors prove that, under mild assumptions, the model memorises inliers during parts of early training while failing to memorise outliers. They characterise how strong the effect is and how long it lasts, and how it depends on the data and starting weights. From the theory they derive practical tips on preprocessing and initialisation, reaching state-of-the-art results on the ADBench datasets.
Catalog record
| CREATOR | Kunwoong Kim, Dongha Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics Kunwoong Kim, Dongha Kim Abstract Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect , a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods. 1 Introduction Outlier detection (OD) aims to identify anomalous instances (outliers) that deviate from the underlying structure of normal data (inliers) ( 8 ; 7 ) . It plays a crucial role in a wide range of applications, including fraud detection ( 23 ) , cybersecurity ( 2 ) , medical diagnosis ( 15 ) , and data preprocessing for downstream learning tasks. Among various settings, unsupervised outlier detection (UOD), where both inliers and outliers are contained in the training data and no information about anomalousness is accessible, is the most general and particularly challenging setting. In this scenario, the model must learn the patterns of inliers while being trained on contaminated data. To this end, a recent line of work exploits the inlier-memorization (IM) effect ( 29 ) : when a deep unsupervised model is trained on data containing both inliers and outliers, it prioritizes reducing the loss of inliers before that of outliers in early training steps. Thus, the per-sample loss of an under-fitted model effectively serves as an outlier score, and this insight motivates various methodologies ( 29 ; 61 ; 10 ; 28 ) . These approaches consistently demonstrate strong empirical performance across diverse datasets, highlighting the practical importance of the IM effect in UOD. Despite these successes, the IM effect itself remains largely underexplored from a theoretical perspective. Existing works primarily rely on empirical observations or heuristic arguments (for example, attributing the IM effect to the higher density and prevalence of inliers in the data distribution), but lack a rigorous understanding of how long the effect persists and how strong the resulting separation is, i.e., its persistence and strength, and how they depend on key factors such as the data distribution and model initialization. This gap limits both the interpretability of the IM effect and the principled design of more reliable IM-based outlier detection methods. In this work, we theoretically study the IM effect. For a simple autoencoder trained by minimizing reconstruction error via gradient descent, we theoretically show that, under mild assumptions, an under-fitted autoencoder memorizes inliers while failing to memorize outliers. We characterize the conditions under which the IM effect exhibits persistence (i.e., the range of training steps over which the effect holds) and strength (reflecting how many outliers can be separated from inliers), from two key perspectives: the data distribution and weight initialization. In particular, we show that the IM effect persists longer and exhibits stronger separability when inliers form dense clusters and outliers are sparse and well-separated, and that it further persists longer when the initialization captures inlier structure. Figure 1 illustrates the IM effect and the factors governing its strength and persistence. Figure 1 : Inlier-memorization (IM) effect and the factors that influence its strength (how large the loss gap is) and persistence (how long the loss gap persists). Building on these insights, we provide two simple yet effective guidelines for enhancing the IM effect in practice. First, we show that using embedding representations can induce more favorable data structures, making the IM effect clearer. Second, we demonstrate that applying EMA-based weight initialization helps preserve inlier information during early training, allowing the IM effect to last longer. These guidelines are intentionally simple and broadly applicable, requiring only minimal modifications to existing methods. We empirically verify that incorporating these two components into existing IM-based outlier detection approaches consistently leads to improved performance, achieving gains over state-of-the-art IM-based methods. The remainder of this paper is organized as follows. Section 2 briefly reviews related research on OD problems. In Section 3 , we present a theoretical study of the IM effect, including the problem setup, a formal description of the phenomenon, and the corresponding theoretical analysis. Section 4 provides empirical validation and discusses practical implications, including guidelines for data preprocessing and weight initialization, followed by concluding remarks in Section 5 . Our contributions can be summarized as follows: \diamond We theoretically study the inlier-memorization (IM) effect, showing that under-fitted autoencoders memorize inliers earlier than outliers, and identifying key factors that govern this behavior, including the data distribution and weight initialization. \diamond Inspired by our theoretical study, we propose two simple yet effective strategies for IM-based outlier detection and show that they consistently improve performance over existing IM-based methods, achieving state-of-the-art results. 2 Related Works Recent advances have shown that deep learning provides a powerful framework for UOD. Classical approaches include density-based methods such as LOF ( 6 ) , cluster-based methods such as CBLOF ( 22 ) , robust statistics-based approaches such as MCD ( 14 ) , and tree-based methods such as Isolation Forest (IF) ( 39 ) . Boundary-based methods such as OCSVM ( 51 ) and SVDD ( 56 ) learn a separating boundary for normal data, with deep extensions including DeepSVDD ( 49 ) and DeepSAD ( 48 ) . Self-supervised learning has also been widely applied to OD tasks ( 54 ; 16 ) . Representative approaches include SimCLR ( 9 ) , which learns informative representations via contrastive learning, and ICL ( 52 ) , which detects anomalies by maximizing mutual information between masked and unmasked data. NeuTraL AD ( 45 ) further learns neural transformations to generate diverse views, enabling effective anomaly detection. A wide range of deep learning techniques with diverse modeling perspectives have been applied to UOD. For example, RDA ( 62 ) enhances autoencoder-based anomaly detection by improving robustness to outliers during representation learning, while DSEBM ( 60 ) detects anomalies using deep energy-based models that assign low energy to normal samples and high energy to anomalous ones. AnoGAN ( 50 ) employs generative adversarial networks to learn the manifold of normal data, and GAAL ( 40 ) generates informative potential outliers to better define the decision boundary. More recently, DTE ( 41 ) leverages diffusion-based models for likelihood-based detection, and ROBOD ( 12 ) improves robustness via a hyper-ensemble framework. Recently, the IM