On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspective DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Retrieval model
Donald Loveland, Mingxuan Ju, Tong Zhao, Neil Shah, Danai Koutra · 2025
Read On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspective DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Retrieval model by Donald Loveland, Mingxuan Ju, Tong Zhao, Neil Shah, Danai Koutra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11318.txt.
Introduction
Donald Loveland and colleagues ask why weight decay, a routine shrinkage setting, matters so much in training recommendation models. Collaborative filtering turns user-item history into dense ID embeddings, and tuning that shrinkage is normally make-or-break, for reasons nobody had pinned down. Theory and tests give a surprise: weight decay mainly writes popularity into the size of the embedding vectors.
Turning the knob thus steers the model toward or away from popular items. Their fix, PRISM, preloads that popularity into the starting vector sizes, so decay is no longer needed. Performance rose by up to 4.77 percent and training ran about thirty-eight percent faster.
Catalog record
| CREATOR | Donald Loveland, Mingxuan Ju, Tong Zhao, Neil Shah, Danai Koutra |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11318
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspective DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Retrieval models and ranking Donald Loveland, Mingxuan Ju, Tong Zhao, Neil Shah, Danai Koutra Abstract . Collaborative filtering (CF) enables large-scale recommendation systems by encoding information from historical user-item interactions into dense ID-embedding tables. However, as embedding tables grow, closed-form solutions become impractical, often necessitating the use of mini-batch gradient descent for training. Despite extensive work on designing loss functions to train CF models, we argue that one core component of these pipelines is heavily overlooked: weight decay . Attaining high-performing models typically requires careful tuning of weight decay, regardless of loss, yet its necessity is not well understood. In this work, we question why weight decay is crucial in CF pipelines and how it impacts training. Through theoretical and empirical analysis, we surprisingly uncover that weight decay’s primary function is to encode popularity information into the magnitudes of the embedding vectors. Moreover, we find that tuning weight decay acts as a coarse, non-linear, knob to influence preference towards popular or unpopular items. Based on these findings, we propose PRISM ( P opularity-awa R e I nitialization S trategy for embedding M agnitudes), a straightforward yet effective solution to simplify the training of high-performing CF models. PRISM pre-encodes the popularity information typically learned through weight decay, eliminating its necessity. Our experiments show that PRISM improves performance by up to 4.77% and reduces training times by 38.48%, compared to state-of-the-art training strategies. Additionally, we parameterize PRISM to modulate the initialization strength, offering a cost-effective and meaningful strategy to mitigate popularity bias. 1. Introduction The vast amount of online data poses significant challenges for content navigation and discovery ( Cooley et al. [n. d.] ) . In response, recommender systems have become essential to distill content into a manageable set of personalized recommendations ( Gomez-Uribe and Hunt 2016 ; Yang et al. 2023 ; Dou et al. 2016 ; Sankar et al. 2021 ; van den Oord et al. 2013 ) . At the core of many recommender system is collaborative filtering (CF), a technique that leverages historical user-item preferences to generate new recommendations ( Su and Khoshgoftaar 2009 ; Chen et al. 2018 ) . Among the most prominent CF methods is matrix factorization (MF), which translates sparse historical interaction data into dense embeddings by learning low-rank user and item matrices ( Koren et al. 2009 ; Wang et al. 2022 ; Xue et al. 2017 ) . Recent advancements in MF have also considered incorporating non-linearities through multilayer perceptrons (MLPs) ( He et al. 2017 ; Covington et al. 2016 ; Cheng et al. 2016 ) , as well as message-passing, to enable the encoding of richer user-item interactions ( He et al. 2020 ; Wang et al. 2019 ) . When training CF models, closed-form solutions become impractical with large datasets ( Bokde et al. 2015 ) , leading to the use of iterative learning algorithms, such as mini-batch gradient descent ( Rendle et al. 2009 ; Wu et al. 2024 ) . However, despite their empirical success, gradient-based methods cannot guarantee an optimal solution ( Peng et al. 2019 ) . Thus, significant effort has been devoted to designing losses that improve the optimization of these approaches ( Wang et al. 2022 ) . One core design consideration is deciding which geometric properties of the embedding vectors to optimize. Early loss functions, such as Bayesian Personalized Ranking (BPR) ( Rendle et al. 2009 ) , leverage the dot product similarity between interacted pairs, promoting the learning of both magnitude and angle information. In contrast, more recent loss functions have shifted to prioritize meaningful angles, e.g. through the use of cosine similarity ( Wu et al. 2024 ) or normalized embeddings ( Chen et al. 2024 ; Wang et al. 2022 ) . We refer to these as angle-based losses given the loss values are not impacted by embedding magnitude. These losses have shown exceptional model performance, leading to a continued prioritization of angle-based optimization ( Park et al. 2023 ; Lin et al. 2024 ) . Despite angle-based losses superior performance, a notable inconsistency frequently arises in their evaluation – the application of weight decay . Most angle-based losses recommend careful tuning of weight decay ( Wu et al. 2024 ; Wang et al. 2022 ; Park et al. 2023 ) , yet they lack justification for its necessity given weight decay regularizes the embedding magnitudes . Even more surprising, angle-based losses display high sensitivity to weight decay strength, leading to significant performance degradation if not properly tuned (shown in Figure 1 ). This seemingly contradictory yet necessary application of weight decay on losses that do not explicitly leverage magnitude information leads us to pose the question: Why is weight decay crucial to learning high-performing CF models, and what properties does it encode? Understanding how weight decay influences training allows us to discern the roles of magnitude and angle information. However, practically, the process of tuning weight decay becomes infeasible as the datasets grow, potentially leading to poor performance if the optimal strength is not found. Moreover, incorporating weight decay can be costly given it requires frequent parameter updates to large embedding tables which may not be loaded in memory. Thus, this inconsistency between angle-based losses and weight decay not only highlights the importance of deeply understanding weight decay’s role, but also motivates the exploration of alternative strategies capable of efficiently encoding similar properties. We begin by systematically analyzing state-of-the-art (SOTA) loss functions to evaluate the properties they encode into the embedding matrices. Although many of these losses are angle-based, we unexpectedly discover that the magnitudes of the embedding vectors capture the underlying popularity information, a behavior more typically associated with losses that use both magnitude and angle. Through comprehensive analysis, we show that weight decay is the direct cause of this phenomenon, resulting from imbalanced gradient updates across parameters due to mini-batch gradient descent. Surprisingly, we find that this magnitude encoding is crucial for angle-based loss functions to achieve SOTA performance; without the popularity encoding from weight decay, performance significantly regresses. In fact, more complex losses (e.g. DirectAU ), are often out-performed by simple baselines (e.g. BPR ) when weight decay is not applied, raising important questions around what designs are necessary for achieving high performance. Thus, our first contribution is the insight that weight decay is essential for angle-based losses to achieve SOTA through the encoding of popularity information, and angle information is only partially responsible for their high performances. Building on our analysis, we question whether weight decay is necessary for achieving high performance. As angle-based losses focus on angle information, while weight decay encodes magnitude information correlated with popularity, we introduce a cost-effective initialization strategy called PRISM : a P opularity-awa R e I nitialization S trategy for the embedding M agnitudes. PRISM splits the learning of angle and magnitude by directly integrating popularity information into the embedding magnitudes, allowing training to focus on learning angle information. By replacing weight decay with PRISM, our models achieve