Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy

Tingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, Enhong Chen · 2024

Read Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy by Tingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, Enhong Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00430.txt.

Introduction

Tingjia Shen and nine co-workers wrote a 2024 paper on systems that suggest the next item a user may like. Such systems learn from the order of past choices. Scaling laws say how scores grow as models grow, but they do not fit these systems in a plain way. The team built a law made for this case.

Their law links scores to data quality, not just data size. They use a measure of how complex the data pattern is to grade quality. Tests fit two common scores across many data and model sizes. The fit was strong for large models. The law can help teams pick the best set-up for a given budget.

Catalog record

CREATORTingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, Enhong Chen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00430

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy Tingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, Enhong Chen Abstract Scaling Laws have emerged as a powerful framework for understanding how model performance evolves as they increase in size, providing valuable insights for optimizing computational resources. In the realm of Sequential Recommendation (SR), which is pivotal for predicting users’ sequential preferences, these laws offer a lens through which to address the challenges posed by the scalability of SR models. However, the presence of structural and collaborative issues in recommender systems prevents the direct application of the Scaling Law (SL) in these systems. In response, we introduce the Performance Law for SR models, which aims to theoretically investigate and model the relationship between model performance and data quality. Specifically, we first fit the HR and NDCG metrics to transformer-based SR models. Subsequently, we propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics. Our method enables accurate predictions across various dataset scales and model sizes, demonstrating a strong correlation in large SR models and offering insights into achieving optimal performance for any given model configuration. 1 Introduction Sequential Recommendation (SR), focused on suggesting the next item for a user based on their past sequential interactions to capture dynamic user preferences ( Wang et al. 2021a ; Wang et al. 2019 ; Xie et al. 2024b ; Wang et al. 2021b ) , has gained significant attention in commercial, internet, and diverse scenarios ( Fang et al. 2020 ; Hidasi et al. 2016 ) . However, as the volume of user data increases, more expansive recommendation models are being implemented. The high computational requirements of these recommendation models ( Ding et al. 2023 ) lead to considerable expenses and unpredictability during development. This places added stress on developers to allocate resources efficiently ( Acun et al. 2021 ) and GPU consumption ( Lin et al. 2022 ) . To anticipate how recommendation models will perform without executing full-scale experiments, researchers have crafted a range of scaling laws to evaluate the models’ effectiveness in various scenarios. Figure 1: Distinction between Performance Law and Scaling Law. Performance typically shows decay as the model size increases. Scaling laws were first explored in the context of Large Language Models ( Kaplan et al. 2020 ) . Specifically, since the introduction of the Chinchilla scaling law, which models the final pre-training loss L(N,D) as a function of the number of model parameters N and the number of training tokens D , models such as LLaMA2 ( Touvron et al. 2023 ) , Mistral ( Jiang et al. 2023 ) , and Gemma ( Team et al. 2024 ) have applied this principle. However, merely increasing model size does not indefinitely enhance performance. A line of research focuses on the generalization behavior of over-parameterized neural networks. Recent experiments reveal that over-trained transformers exhibit an inverted U-shaped scaling behavior ( Belkin et al. 2019 ) , which existing empirical scaling laws cannot explain ( Niu et al. 2024 ) . Following the success of scaling laws in large language models, many researchers have also sought to apply these principles to recommendation models, such as HSTU ( Zhai et al. 2024 ) and Wukong ( Gu et al. 2022 ) . However, the current endeavor to transfer large model scaling laws to the analysis of recommendation systems faces two challenges: (1) Compared to the loss metrics typically examined in scaling laws, performance is the more pertinent outcome of interest for recommendation models. Although some studies have highlighted the relationship between performance and loss ( Du et al. 2024 ) , these relationships remain unstable and lack quantitative fitting analysis. (2) In addition to data scale, data quality has increasingly become a focal point of concern, significantly impacting recommendation model datasets, which typically exhibit structural and collaborative characteristics and higher noise and redundancy levels. This positions data quality as a critical factor influencing outcomes, yet it remains unaddressed in the prevailing scaling laws for recommendation models. In response to Challenge (1), we propose the Performance Law for recommendation models. By fitting the hit rate (HR) and normalized discounted cumulative gain (NDCG) metrics of sequential recommendation models, we can utilize a few initial small-scale training points to infer the impact of model layers and item embedding dimensions on model performance. Toward Challenge (2), we introduced Approximate Entropy (ApEn) as a novel measure of data quality. We thoroughly validated, both theoretically and experimentally, the appropriateness of replacing data scale with data scale/approximate entropy, providing a more accurate substitute for the number of training tokens D in the original scaling law. Overall, all we need are the amount and quality of training data, along with initial experiments to balance parameters that have a core impact on model performance (such as the number of candidates, where performance can vary significantly between having 1,000 candidates versus 100). We obtained a surprisingly accurate correlation coefficient within large sequential models of different sizes and across various datasets. We also verified that the Performance Law can provide both the globally optimal and optimal performance for a given model size. 2 Related Work 2.1 Sequential Recommendation The focus of recommendation systems has undergone significant transformations. Among these, sequential recommendation ( He et al. 2023 ; Guo et al. 2021 ; Guo et al. 2023 ) is a technique that aims to delve into and understand users’ interest patterns by analyzing their historical interactions ( Tong et al. 2024 ; Wu et al. 2024 ) . Initially, techniques such as markov chain and matrix factorization were employed ( He et al. 2016 ) . However, with the emergence of neural networks, deep learning approaches have been developed for sequential recommendation tasks. Convolutional Neural Networks(CNN) ( Tang & Wang 2018b ) , Graph Neural Networks (GNN) ( Wu et al. 2019 ) from GCE-GNN ( Wang et al. 2020 ) to SR-GNN ( Wu et al. 2018 ) and URLLM ( Shen et al. 2024 ) . Multilayer Perceptron (MLP) ( Zhou et al. 2022 ) modeled SR from various perspectives, while the Diffusion-based method ( Xie et al. 2024a ; Xie et al. 2024b ) and Fourier transform-based ( Wang et al. 2024 ; Han et al. 2023 ) method are used to attempt noise reduction. However, the compatibility of RNN ( Hidasi 2015 ) with sequences has ultimately garnered more attention. Early work like GRU4Rec ( Hidasi & Karatzoglou 2018 ) and Caser ( Tang & Wang 2018a ) were introduced to improve recommendation accuracy. Another notable technique in sequential recommendation is the attention mechanism. SASRec ( Kang & McAuley 2018 ) , for instance, utilizes self-attention to independently learn the impact of each interaction on the target behavior. On the other hand, BERT4Rec ( Sun et al. 2019 ) incorporates bi-directional transformer layers after conducting pre-training tasks. Since LLaMA4Rec ( Luo et al. 2024 ) and HSTU ( Zhai et al. 2024 ) demonstrated improvements in recommendation performance brought by large models and large datasets, It is meaningful to study how the model performance would change as the

Read the complete text