Leveraging Historical and Current Interests for Continual Sequential Recommendation Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Retrieval models and ranking CCS: Information systems Recommender systems CCS: Information systems Personalization
Gyuseok Lee, Hyunsik Yoo, Junyoung Hwang, SeongKu Kang, Hwanjo Yu · 2025
Read Leveraging Historical and Current Interests for Continual Sequential Recommendation Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Retrieval models and ranking CCS: Information systems Recommender systems CCS: Information systems Personalization by Gyuseok Lee, Hyunsik Yoo, Junyoung Hwang, SeongKu Kang, Hwanjo Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07466.txt.
Introduction
Gyuseok Lee, Hyunsik Yoo, Junyoung Hwang, SeongKu Kang, and Hwanjo Yu design a recommender that keeps learning from endless behavior streams without forgetting old tastes, in a paper from 2025. Retraining transformers on every new click is costly, and naive updates erase long-term preferences. Continual learning offers a middle path.
Their transformer variant uses a custom linear attention with a normalization step that steadies learning across users with uneven activity. A shared pool of learnable interests lets knowledge transfer between people and cushions forgetting. New users borrow history from similar veterans. On three real datasets it beat leading models at both retention and adaptation.
Catalog record
| CREATOR | Gyuseok Lee, Hyunsik Yoo, Junyoung Hwang, SeongKu Kang, Hwanjo Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07466
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Historical and Current Interests for Continual Sequential Recommendation Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Retrieval models and ranking CCS: Information systems Recommender systems CCS: Information systems Personalization Gyuseok Lee, Hyunsik Yoo, Junyoung Hwang, SeongKu Kang, Hwanjo Yu Abstract . Sequential recommendation models based on the Transformer architecture show superior performance in harnessing long-range dependencies within user behavior via self-attention. However, naively updating them on continuously arriving non-stationary data streams incurs prohibitive computation costs or leads to catastrophic forgetting. To address this, we propose C ontinual S equential T ransformer for Rec ommendation ( \proposed ) that effectively leverages well-preserved historical user interests while capturing current interests. At its core is Continual Sequential Attention (CSA), a linear attention mechanism that retains past knowledge without direct access to old data. CSA integrates two key components: (1) Cauchy–Schwarz Normalization that stabilizes training under uneven interaction frequencies, and (2) Collaborative Interest Enrichment that mitigates forgetting through shared, learnable interest pools. We further introduce a technique that facilitates learning for cold-start users by transferring historical knowledge from behaviorally similar existing users. Extensive experiments on three real-world datasets indicate that \proposed outperforms state-of-the-art baselines in both knowledge retention and acquisition. 1. Introduction Sequential recommendation (SR) has gained prominence in both academic research and practical applications by capturing sequential patterns in user behavior to enhance next item prediction ( Kang and McAuley 2018 ; Fang et al. 2020 ; Steck et al. 2021 ; Yue et al. 2024 ) . Earlier studies use Markov chains ( Rendle et al. 2010 ; Chen et al. 2012 ) and Recurrent Neural Networks (RNNs) ( Wu et al. 2017 ; Hidasi 2015 ) to model temporal dependencies in user behavior sequences. However, both approaches encounter difficulties in capturing long-range dependencies as sequence length increases ( Kang and McAuley 2018 ) . To address this challenge, Transformer-based SR models ( Kang and McAuley 2018 ; Sun et al. 2019 ; Zhou et al. 2020 ; Wu et al. 2020 ; Fan et al. 2022 ) employ self-attention to focus on the most relevant behaviors regardless of their positional distance ( Vaswani 2017 ) . With its remarkable performance, the Transformer has become the dominant approach in the field of SR. Nevertheless, applying Transformer-based SR models to real-world applications presents non-trivial challenges. As user behavior sequences are continuously arriving, models should adapt to new information for timely recommendations ( Cho et al. 2021 ; Li et al. 2020 ; Costa and Dolog 2019 ) . A simple approach is to retrain the model using the entire user behavior sequences accumulated along data streams. However, training on the whole sequences is extremely time-consuming ( Wang and Shen 2023 ; Mi et al. 2020 ) , due to the quadratic complexity of self-attention with respect to input length ( Katharopoulos et al. 2020 ; Wang et al. 2020 ; Choromanski et al. 2020 ) . Moreover, this approach is impractical in scenarios where not all historical interactions are accessible (e.g., privacy issues) or where memory constraints prevent storing complete sequences during both training and inference ( Zhang and Kim 2023 ; Wang and Shen 2023 ) . A more cost-effective alternative is to fine-tune the model using only new sequences. However, such overreliance on transient interests risks forgetting historical user interests that may reemerge later, thereby significantly limiting recommendation accuracy ( Wang and Shen 2023 ; Xu et al. 2020 ; Wang et al. 2023 ) . Continual learning (CL) ( Lee et al. 2024 ; Kirkpatrick et al. 2017 ; Li and Hoiem 2017 ) , a well-established approach to updating a model with non-stationary data streams, has been actively studied for recommendation ( Xu et al. 2020 ; Mi et al. 2020 ; Wang et al. 2021 ; Wang et al. 2023 ; Zhu et al. 2023 ) . With the goal of adapting to new data while preserving previously learned knowledge, there are two dominant CL approaches: (1) Regularization-based methods ( Wang et al. 2021 ; Xu et al. 2020 ; Wang et al. 2023 ) impose constraints on the parameter space to prevent significant changes from previously trained parameters. (2) Replay-based methods ( Mi et al. 2020 ; Cai et al. 2022 ; Zhu et al. 2023 ) store small portions of historical data and reuse them in subsequent training, with external memory being updated over time. Although existing CL methods retain interests from the relatively recent past, they are insufficient to preserve those from the distant past, gradually forgetting long - term preferences. Specifically, regularization-based methods rely on the most recently learned parameters to transfer acquired knowledge into the subsequent training process. As this greedy preservation continues, new information gradually dilutes historical knowledge, leading to unavoidable forgetting. Similarly, replay-based methods are constrained by limited memory capacity, which necessitates continually updating stored data with newer interactions. This reduces access to older data, further hindering the retention of historical user interests. Furthermore, most CL methods have focused on non-sequential models (e.g., matrix factorization ( Jakomin et al. 2020 ; Yu et al. 2016 ) , graph neural networks ( Wang et al. 2021 ; Wang et al. 2023 ) ), leaving their application in SR relatively underexplored. We propose C ontinual S equential T ransformer for Rec ommenda tion ( \proposed ), which continuously updates a transformer-based SR model with non-stationary data streams. \proposed aims to preserve historical interests and leverage them to adapt to current ones, ultimately capturing the trajectory of user interests over time. To facilitate the preservation of historical knowledge, we borrow the idea of linear attention ( Katharopoulos et al. 2020 ; Wang et al. 2020 ; Choromanski et al. 2020 ) . Linear attention approximates self-attention by performing linear computations at each position and sequentially accumulating hidden states over time, similar to the hidden state propagation in RNNs. This allows the model to partially retain historical knowledge through parametric memories without direct access to all historical sequences, while emulating the expressive power of self - attention ( Munkhdalai et al. 2024 ; Lee et al. 2023 ) . However, naively applying linear attention to continual SR yields suboptimal results due to two challenges: (1) unstable training, caused by an imbalance in the number of interactions per user along data streams; hidden states accumulate user behaviors over time, resulting in disproportionately large values for active users and small values for less active users. This imbalance leads to uneven magnitudes and updates across users, making optimization highly unstable. (2) inevitable forgetting, caused by the continual update of hidden states with new sequences over time. This causes historical knowledge to be continuously overwritten and gradually forgotten. As a solution, we introduce Continual Sequential Attention (CSA) , a specialized linear attention for continual SR, featuring two novel components: (1) Cauchy-Schwarz Normalization to resolve unstable learning. Leveraging the Cauchy-Schwarz inequality, we dynamically adjust the magnitudes of hidden states to address the imbalance caused by disparities in the number of interactions. (2) Collaborative Interest Enrichment to alleviate inevitable forgetting. We utilize learnable interest pools that store historical interests. For each user context, we retrieve the most relevant interests from the pools and use them to compensate for forgotten user interests.