Measuring the stability and plasticity of recommender systems

Maria João Lavoura, João Vinagre, Robert Jungnickel · 2025

Read Measuring the stability and plasticity of recommender systems by Maria João Lavoura, João Vinagre, Robert Jungnickel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03941.txt.

Introduction

Maria Joao Lavoura and co-authors propose a way to judge recommendation systems over time in a 2025 paper. Standard tests train a model once and score it on held out data, which gives only a snapshot. Real services retrain often on fresh data, and past scores may not predict how a model handles new patterns.

Their method tracks two traits as models retrain: stability, the power to keep old patterns, and plasticity, the speed of adapting to change. Early tests with three algorithm types on a book rating data set show distinct profiles for each trait. A trade off between the two traits may exist. The authors discuss limits and ways to sharpen the method.

Catalog record

CREATORMaria João Lavoura, João Vinagre, Robert Jungnickel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03941

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring the stability and plasticity of recommender systems Maria João Lavoura, João Vinagre, Robert Jungnickel Abstract . The typical offline protocol to evaluate recommendation algorithms is to collect a dataset of user-item interactions and then use a part of this dataset to train a model, and the remaining data to measure how closely the model recommendations match the observed user interactions. This protocol is straightforward, useful and practical, but it only captures performance of a particular model trained at some point in the past. We know, however, that online systems evolve over time. In general, it is a good idea that models reflect such changes, so models are frequently retrained with recent data. But if this is the case, to what extent can we trust previous evaluations? How will a model perform when a different pattern (re)emerges? In this paper we propose a methodology to study how recommendation models behave when they are retrained. The idea is to profile algorithms according to their ability to, on the one hand, retain past patterns – stability – and, on the other hand, (quickly) adapt to changes – plasticity. We devise an offline evaluation protocol that provides detail on the long-term behavior of models, and that is agnostic to datasets, algorithms and metrics. To illustrate the potential of this framework, we present preliminary results of three different types of algorithms on the GoodReads dataset that suggest different stability and plasticity profiles depending on the algorithmic technique, and a possible trade-off between stability and plasticity. Although additional experiments will be necessary to confirm these observations, they already illustrate the usefulness of the proposed framework to gain insights on the long term dynamics of recommendation models. 1. Introduction The stability-plasticity dilemma has been identified by Mermillod et al. 2013 as a fundamental trade-off of learning systems. It consists of the tension between two potentially conflicting requirements: stability , the ability to retain previously learned concepts, and plasticity , the ability to learn new ones. A system with high plasticity will tend to be forgetful, whereas a highly stable system will struggle to adapt to changes. Traditional offline evaluation protocols are agnostic to time, and therefore do not account for such properties. However, it is reasonable to hypothesize that different types of algorithms will learn models with disparate stability and plasticity profiles. The desirable stability and plasticity are dependent on the dynamics of the task at hand. Highly dynamic domains, such as news or social media, may favor models with high plasticity, while domains with few and slow changes, such as music or books, may benefit more from models with high stability. Furthermore, other technical considerations may come at play, such as the need to deal with cold-start or robustness to shilling attacks ( Günes et al. 2014 ) . This suggests that evaluating stability and plasticity of recommendation models has implications on early choices with high potential downstream impact. In the fields of continual learning ( Wang et al. 2024 ) and incremental stream-based learning ( Gama 2012 ) , researchers have looked at this problem, typically focusing on classic supervised machine learning in a sequence of well identified tasks – the so-called task-incremental learning. The problem in recommendation is that such division does not exist naturally, so most of the methods to measure stability and plasticity used in previous research are not applicable to recommendation models. In addition, past work focuses disproportionally on the problem of catastrophic forgetting – i.e. through the lens of stability. There are comparably very few works that focus on plasticity, be it exclusively or complementarily. Given that classic evaluation methods are not designed to evaluate stability and plasticity, it is impractical to use them for that purpose. This paper addresses the problem by proposing a framework, including a methodology and metrics, that answers questions related to the ability of recommendation models to retain past concepts and assimilate new ones. Our approach is double-sided. On the one hand, we ask the question of how to measure the stability and plasticity of models, and to what extent these two are conflicting. On the other hand, we also want to know what is the benefit of stability or plasticity in given problems. Our research questions are the following: • RQ1: How can we measure and compare the stability and plasticity of different recommendation models using a common evaluation framework? • RQ2: Do different algorithmic approaches exhibit distinct trade-offs? Our two contributions are (1) an algorithm- and metric-agnostic methodology that can be used systematically to profile algorithms with respect to their stability-plasticity profile, and (2) preliminary insights on the stability-plasticity trade-offs of different classes of algorithms across metrics related to accuracy and diversity. We structure the remainder of the paper with and overview of related work in Section 2 , a description of our methodology in Section 3 , and results in Section 4 . In Section 5 we present our conclusions, limitations and future work. 2. Related work The concepts of stability and plasticity have been discussed for quite some time by researchers studying continual machine learning ( Zenke et al. 2017 ; Wang et al. 2024 ) , as a broader framing of the problem of catastrophic forgetting ( French 1999 ) , suggesting there is a dilemma, or a trade-off, between the ability of intelligent systems to retain what they have learned from past tasks and their ability to learn new tasks ( Mermillod et al. 2013 ) . In the related literature, there is the fundamental notion of tasks. A continual learning problem essentially consists of incrementally learning neural models over a sequence of distinct tasks, typically classification tasks ( van de Ven and Tolias 2019 ) . The challenge is not only to maximize performance on the most recently learned tasks, but also to minimize loss in previous ones. The main distiction in existing approaches is whether they require task identifiers in the data, or are able to infer tasks autonomously. One of the issues of research task-incremental continual learning is the lack of standard tools and metrics to measure forgetting and retention ( Kemker et al. 2018 ) . One interesting approach towards systematically measuring stability and plasticity is proposed by Lopez-Paz and Ranzato 2017 through the concepts of forward and backward information transfer. The two concepts are captured by metrics that reflect the ability of past models to perform on future tasks and future models to perform in pasts tasks, respectively. In recommender systems, the problem of continual learning has been mostly – and indirectly – approached through stream-based learning ( Al-Ghossein et al. 2022 ) , in particular under the assumption that forgetting is beneficial ( Matuszyk et al. 2018 ) . Our proposal draws inspiration from the above work, however a direct application of methods designed for classic machine learning is not possible for multiple reasons. First, the idea of a sequence of tasks is highly ambiguous in recommendation problems. A notable contribution has been made by Yuan et al. 2021 , where tasks are defined as distinct platforms – i.e. social media, marketplaces, etc – users navigate in. However this would only work in a scenario where user representations are common to all

Read the complete text