Adaptive Regularization for Large-Scale Sparse Feature Embedding Models

Mang Li, Wei Lyu Institute of Intelligent TechnologyAlibaba International Digital Commerce GroupHang Zhou, China {mang.ll, lw386934}@alibaba-inc.com · 2025

Read Adaptive Regularization for Large-Scale Sparse Feature Embedding Models by Mang Li, Wei Lyu Institute of Intelligent TechnologyAlibaba International Digital Commerce GroupHang Zhou, China {mang.ll, lw386934}@alibaba-inc.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06374.txt.

Introduction

Mang Li and Wei Lyu explain a training quirk that plagues click and conversion prediction models in search, ads, and recommendation. Models fed by huge sparse category features often shine after one pass over the data, then degrade sharply with more passes. Heuristic patches exist, but the root cause stayed unclear.

Using Rademacher complexity, a measure of how much a model class can fit noise, the authors give a theoretical account of the one-epoch overfitting effect. Guided by that analysis, their adaptive regularization caps the size budget of embedding layers as training proceeds. The method blocked the multi-epoch collapse, improved even single-epoch results, and is already running in production systems.

Catalog record

CREATORMang Li, Wei Lyu Institute of Intelligent TechnologyAlibaba International Digital Commerce GroupHang Zhou, China {mang.ll, lw386934}@alibaba-inc.com
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06374

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Regularization for Large-Scale Sparse Feature Embedding Models Mang Li, Wei Lyu Institute of Intelligent TechnologyAlibaba International Digital Commerce GroupHang Zhou, China {mang.ll, lw386934}@alibaba-inc.com Abstract The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising, and recommendation domains. These models which rely heavily on large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, they have not clearly identified the fundamental cause of this phenomenon. In this work, we provide a theoretical analysis that explains why overfitting occurs in models that use large-scale sparse categorical features. Based on this analysis, we propose an adaptive regularization method to address it. Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch. This method has already been deployed in online production systems. 1 Introduction Click-through rate (CTR) and conversion rate (CVR) estimation are critical for advertising, search and recommendation (ASR) applications. E-commerce platforms like Amazon and Taobao rely on optimizing CTR and CVR estimation to boost gross merchandise volume (GMV), while advertising platforms at Google and Meta depend on it to drive revenue growth. In the past decade, as deep learning has been widely adopted in ASR applications, most estimation models have been built on deep learning frameworks and rely on large-scale, sparse categorical features ( 3 ; 10 ; 14 ). Recent work ( 28 ) demonstrates, through extensive experiments, that such models commonly suffer from the one-epoch overfitting phenomenon, where model performance drops sharply after the first epoch of training. It empirically suggests that the optimal practice is to train the models for only one epoch. At the same time, their empirical analysis also reveals a strong connection between feature sparsity and the one-epoch phenomenon. In ASR scenarios, categorical features can easily reach the scale of billions, and most feature values occur extremely infrequently ( 25 ; 27 ), which makes models especially susceptible to overfitting after the first epoch. Additionally, 19 reported a similar one-epoch phenomenon during supervised fine-tuning (SFT) of large language model (LLM), though they argue that moderate overfitting can actually be beneficial. We leave it in future work. In this paper, we focus on addressing the multi-epoch overfitting problem for models with large-scale sparse features in ASR applications. 15 was the first to address the one-epoch problem using a heuristic approach. They introduced a multi-epoch data augmentation method (MEDA) that can be easily applied to estimation models with large-scale sparse categorical features. Later, they extended it to a continual learning paradigm ( 6 ). To mitigate overfitting during multi-epoch training, MEDA reinitializes the embeddings of categorical features and their corresponding optimizer states at the beginning of each epoch. Although this approach effectively alleviates multi-epoch overfitting, it is fundamentally heuristic because the embedding parameters are only reinitialized at epoch boundaries. It neither ensures optimal convergence nor explains the root cause of overfitting. Moreover, this embedding reinitialization approach may result in the loss of substantial information, potentially resulting in suboptimal model performance. Nevertheless, it provides important inspiration for subsequent research. Another method, proposed by 24 , adopts an LLM-style paradigm. In this approach, generative pretraining is used to obtain embeddings for categorical features, which remain frozen during subsequent training. This strategy also helps to mitigate overfitting. However, it requires significant training resources to obtain the pretrained embeddings, and the performance cannot be fairly compared to single-epoch training because the parameter budget used during the pretraining phase is not included. Therefore, it cannot be regarded as a solution to the one-epoch problem, although it does highlight the pivotal role of the embedding layer in this phenomenon. There are several well-established and general-purpose methods to mitigate overfitting, such as dropout ( 23 ), L_{1} and L_{2} regularization ( 21 ; 17 ), and weight decay ( 11 ). However, in real-world industrial applications, data is typically high-dimensional and sparse, with different features exhibiting varying degrees of sparsity. Consequently, methods like AdamW ( 16 ), which apply the same weight decay across all parameters, tend to be suboptimal for addressing overfitting. Specifically, such approaches may reduce the fitting accuracy for relatively dense features while failing to effectively control overfitting in sparse features. This paper presents a foundational theoretical analysis of the root cause of one-epoch overfitting, and proposes a method that adaptively determines regularization coefficients according to the effective update interval of each categorical feature value, and integrates these coefficients into the optimizer’s update rules. The proposed approach is particularly well-suited for models with large-scale sparse features. It reduces regularization for relatively dense features and neural network parameters, while assigning appropriate regularization strengths to individual categorical features. As a result, this method not only prevents the sharp performance degradation observed in multi-epoch training, but also enhances performance within a single epoch. 2 Preliminary In the industrial ASR domain, mainstream estimation models typically combine embedding layers with a variety of dense multi-layer perceptron (MLP) backbones, such as 9 . Taking an e-commerce platform estimation model as an example, these models process large-scale sparse inputs, including item IDs, brand IDs, and seller IDs, where the scale of categorical features can range from millions to billions. Before being processed by the estimation model, these sparse features are first mapped to low-dimensional dense representations via embedding layers. The one-hot categorical feature {\bm{x}}_{i}(t)\in\mathbb{R}^{N_{i}} is defined as {\bm{x}}_{i}(t)=\begin{bmatrix}0,0,\ldots,1,\ldots,0\end{bmatrix}^{\top},i\in[S],t\in[T] , which can be represented by an embedding vector {\bm{e}}_{i}\left(t\right)\in\mathbb{R}^{d_{i}} shown below {\bm{e}}_{i}\left(t\right)={\bm{E}}_{i}^{\top}{\bm{x}}_{i}(t) (1) where {\bm{E}}_{i}\in\mathbb{R}^{N_{i}\times d_{i}} is the embedding matrix for feature i and d_{i} is the embedding size. T\in\mathbb{N} is the number of training samples. S\in\mathbb{N} denotes the number of categorical features. N_{i}\in\mathbb{N},\forall i\in[S] denotes the number of distinct values for feature i . At each update step, an embedding lookup is performed to retrieve the dense representations for the sparse features via equation 1 , which are then fed into the MLP layers. For simplicity, we separate the model into the embedding component and the MLP component denoted as f . In our theoretical analysis, we use a basic DNN backbone and provide experiments in section 4 to demonstrate the generalizability of our method with other backbone architectures. 2.1 Neural Network Definition The general form of basic DNN function can be defined as f\left(\cdot\right)={\bm{W}}_{L}\sigma_{L-1}\left({\bm{W}}_{L-1}\sigma_{L-2}\left(\dots\sigma_{1}\left({\bm{W}}_{1}\cdot\right)\right)\right) (2) where L denotes the number of MLP layers, \sigma(\cdot) is the ReLU function ( 1 ), {\bm{W}}_{l} is the linear projection matrix of layer l,l\in[L] . In this paper, we only discuss the impact of embedding layer so we can define the t -th sample output of DNN as y(t)=f\left(\begin{bmatrix}{\bm{E}}_{1}^{\top}{\bm{x}}_{1}(t);{\bm{E}}_{2}^{\top}{\bm{x}}_{2}(t);\dots;{\bm{E}}_{S}^{\top}{\bm{x}}_{S}(t)\end{bmatrix}\right) (3) where y(t) is the logit output for CTR or CVR estimation model. 2.2 Rademacher Complexity Bound Rademacher complexity is a standard tool to control the uniform convergence of given classes of predictors ( 13 , 26 ). Formally, given a real-valued function class \mathcal{H} with the input {\bm{z}} , we define the empirical Rademacher complexity \widehat{\mathcal{R}}_{T}\left(\mathcal{H}\right) as \widehat{\mathcal{R}}_{T}\left(\mathcal{H}\right)=\mathbb{E}_{\bm{\epsilon}}\left[\sup_{h\in\mathcal{H}}\frac{1}{T}\sum_{t=1}^{T}\epsilon_{t}\,h({\bm{z}}_{t})\right] (4) where \bm{\epsilon}=(\epsilon_{1},\dots,\epsilon_{T}) is a vector whose entries \epsilon_{t} are independent and uniformly distributed in \{-1,+1\} . Using standard arguments, such bounds, as long as the norm of {\bm{z}}_{t} is bounded, can be converted to bounds on the generalization error,

Read the complete text