Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems Conference: FAccTRec@RecSys’25; September 22–26, 2025; Prague, Czech Republic

Pierre Lubitzsch † Olga Ovcharenko † Hao Chen † Maarten de Rijke ‡ Sebastian Schelter † · 2025

Read Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems Conference: FAccTRec@RecSys’25; September 22–26, 2025; Prague, Czech Republic by Pierre Lubitzsch † Olga Ovcharenko † Hao Chen † Maarten de Rijke ‡ Sebastian Schelter † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17076.txt.

Introduction

Pierre Lubitzsch, Olga Ovcharenko, Hao Chen, Maarten de Rijke, and Sebastian Schelter argue in a 2025 paper for a real-world benchmark for unlearning in recommender systems. Such systems lean on user data to tailor their picks. Privacy law gives users a right to be forgotten, so a model must be able to drop a person's data after training.

The main past benchmark missed that real task, they say. It stuck to one method type, left out tasks such as next-basket picks, and asked for drops far larger than real ones. Their plan ties the test to true loads and time caps.

Catalog record

CREATORPierre Lubitzsch † Olga Ovcharenko † Hao Chen † Maarten de Rijke ‡ Sebastian Schelter †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17076

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems Conference: FAccTRec@RecSys’25; September 22–26, 2025; Prague, Czech Republic Pierre Lubitzsch † Olga Ovcharenko † Hao Chen † Maarten de Rijke ‡ Sebastian Schelter † Abstract . Modern recommender systems heavily leverage user interaction data to deliver personalized experiences. However, relying on personal data presents challenges in adhering to privacy regulations, such as the GDPR’s “right to be forgotten”. Machine unlearning (MU) aims to address these challenges by enabling the efficient removal of specific training data from models post-training, without compromising model utility or leaving residual information. However, current benchmarks for unlearning in recommender systems—most notably CURE4Rec—fail to reflect real-world operational demands. They focus narrowly on collaborative filtering, overlook tasks like session-based and next-basket recommendation, simulate unrealistically large unlearning requests, and ignore critical efficiency constraints. In this paper, we propose a set of design desiderata and research questions to guide the development of a more realistic benchmark for unlearning in recommender systems, with the goal of gathering feedback from the research community. Our benchmark proposal spans multiple recommendation tasks, includes domain-specific unlearning scenarios, and several unlearning algorithms—including ones adapted from a recent NeurIPS unlearning competition. Furthermore, we argue for an unlearning setup that reflects the sequential, time-sensitive nature of real-world deletion requests. We also present a preliminary experiment in a next-basket recommendation setting based on our proposed desiderata and find that unlearning also works for sequential recommendation models, exposed to many small unlearning requests. In this case, we observe that a modification of a custom-designed unlearning algorithm for recommender systems outperforms general unlearning algorithms significantly, and that unlearning can be executed with a latency of only several seconds. 1. Introduction Modern recommender systems rely heavily on user interaction data, such as clicks, ratings, and purchases to build personalized models. The right-to-be-forgotten and machine unlearning . This personalization enhances the user experience, but at the same time makes it challenging to adhere to privacy rights such as the “right-to-be-forgotten” from the GDPR ( GDPR.eu nd ) regulation in Europe and similar regulations adopted outside of Europe ( California Privacy Protection Agency nd ; Dig nd ) . These regulations mandate that responsible, ethical, and legally compliant AI systems give their users the right to request the timely deletion of their personal data from databases and models trained on them ( Stoyanovich et al. 2022 ) . Being able to efficiently remove data from models is also important for security-related scenarios, e.g., to quickly react to spam interactions ( Buzzfeed 2019 ) or unintended data leakage ( Carlini et al. 2019 ) . On a technical level, the outlined challenges give rise to “machine unlearning” (MU), the problem of efficiently removing the influence of selected training data points from a machine learning model post training. After unlearning the data points there should be no information about these points in the model, it should ideally behave as if they had never been in the training set. Efficient machine unlearning is an active area of research ( Cao and Yang 2015 ; Ginart et al. 2019 ; Izzo et al. 2021 ; Schelter et al. 2021 ; Wang and Schelter 2022 ; Wu et al. 2020a ; Wu et al. 2020b ) . The evaluation of unlearning algorithms is especially challenging since multiple aspects, such as the model utility after unlearning, the amount of information about the unlearned data still contained in the model (unlearning completeness), and the unlearning efficiency have to be considered simultaneously. Shortcomings of existing benchmarks for unlearning in recommender systems . We argue in Section 2.1 that current unlearning benchmarks like CURE4Rec ( Chen et al. 2024 ) do not sufficiently address the real-world demands of unlearning in recommender systems. They narrowly focus on collaborative filtering (CF) ( Rendle et al. 2009 ; Sarwar et al. 2001 ; He et al. 2020 ; Xia et al. 2023 ; Ren et al. 2023 ) , overlooking other vital tasks, such as session-based recommendation (SBR) ( Hidasi et al. 2016 ; Jannach and Ludewig 2017 ; Tan et al. 2016 ; Wu et al. 2019 ; Kang and McAuley 2018 ; Li et al. 2017 ) and next-basket recommendation (NBR) ( Rendle et al. 2010 ; Hu et al. 2020 ; Faggioli et al. 2020 ; Qin et al. 2021 ; Yu et al. 2020 ; Hu and He 2019 ) that are common in e-commerce and frequently require unlearning capabilities. Moreover, these benchmarks simulate unlearning by deleting large chunks of data—typically 2–5% of all interactions—whereas actual systems face continuous, small-scale deletion requests, such as when individual users withdraw consent for personalization. The choice of data to unlearn is also misrepresented: rather than being random, real scenarios often evolve around specific types of content, like interactions with sensitive items ( Schelter et al. 2023 ) or interactions from spammers ( Buzzfeed 2019 ) . Operational efficiency is another overlooked factor; in practice, unlearning must be fast, inexpensive, and ideally executable directly on deployed models ( Schelter et al. 2021 ) . The unlearning procedures in CURE4Rec are only an order of magnitude faster than retraining, which undermines real-time responsiveness. Finally, promising general-purpose unlearning algorithms, such as those highlighted in the NeurIPS 2023 competition ( Triantafillou et al. 2024 ) , remain underexplored for recommendation-specific applications. Towards a real-world aligned benchmark for unlearning in recommender systems . The goal of this paper is to propose the desiderata for a real-world aligned benchmark for unlearning in recommender systems, and to gather initial feedback. In Section 2.2 , we formulate four extended research questions to address the shortcomings of existing benchmarks, and layout desiderata and design decisions for a new benchmark. In particular, we propose to cover NBR and SBR in addition to CF. For each recommendation task, we need different models, unlearning settings, and datasets to get insights about the unlearning performance for different data distributions. We consider various approximate unlearning algorithms for neural networks, one designed for recommendation models ( Li et al. 2023b ) , two adapted from the NeurIPS unlearning competition ( Triantafillou et al. 2024 ) , and three designed for graph neural networks (GNNs) ( Wu et al. 2023b ; Wu et al. 2023a ; Dong et al. 2024 ) that are common architectures for recommender systems. As recommendation models get fully retrained in the range between once per week (Amazon Prime ( Amazon Web Services 2025 ) ) and once per month (Tencent ( Chen et al. 2025 ) ), it is unrealistic to unlearn a large chunk of data collected over a longer period of time. In contrast to existing benchmarks, we propose to issue a number of small unlearning requests in a sequential manner instead of unlearning a single forget set all at once, as unlearning requests are usually time-sensitive. We propose to evaluate the unlearned models based on retained model utility, unlearning completeness, and unlearning efficiency. Moreover, we present a first preliminary experiment in Section 3 , designed according to our benchmark desiderata. Here, we evaluate the ability of three unlearning algorithms in a next-basket recommendation scenario, where we repeatedly unlearn the interactions with sensitive items (chosen in a domain-specific manner) from a neural model on

Read the complete text