The Unreasonable Effectiveness of Data for Recommender Systems CCS: Computing methodologies Ranking CCS: Computing methodologies Machine learning
Youssef Abdou · 2026
Read The Unreasonable Effectiveness of Data for Recommender Systems CCS: Computing methodologies Ranking CCS: Computing methodologies Machine learning by Youssef Abdou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06420.txt.
Introduction
Abdou investigates how recommendation performance evolves as data grows. Collecting and processing data costs time and energy, yet gains may stop. Using LensKit and RecBole on 11 datasets, he trains on nine sizes, measuring NDCG@10. Raw NDCG usually rises with size, with no saturation. Min-max normalization reveals a positive trend: around 75 percent at the largest size achieve the best. A late-stage analysis supports the upward trend, with a non-negative range. For recommenders, more data remains beneficial, while weaker scaling concentrates in atypical datasets and the outlier RecBole BPR.
Catalog record
| CREATOR | Youssef Abdou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19035 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.06420
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Unreasonable Effectiveness of Data for Recommender Systems CCS: Computing methodologies Ranking CCS: Computing methodologies Machine learning Youssef Abdou Abstract . In recommender systems, collecting, storing, and processing large-scale interaction data is increasingly costly in terms of time, energy, and computation, yet it remains unclear when additional data stops providing meaningful gains. This paper investigates how offline recommendation performance evolves as the size of the training dataset increases and whether a saturation point can be observed. We implemented a reproducible Python evaluation workflow with two established toolkits, LensKit and RecBole , included 11 11 large public datasets with at least 7 7 million interactions, and evaluated 10 10 tool–algorithm combinations. Using absolute stratified user sampling, we trained models on nine sample sizes from 100,000 100{,}000 to 100,000,000 100{,}000{,}000 interactions and measured N D C G @ 10 NDCG@10 . Overall, raw N D C G NDCG usually increased with sample size, with no observable saturation point. To make result groups comparable, we applied min–max normalization within each group, revealing a clear positive trend in which around 75 % 75\% of the points at the largest completed sample size also achieved the group’s best observed performance. A late-stage slope analysis over the final 10 10 – 30 % 30\% of each group further supported this upward trend: the interquartile range remained entirely non-negative with a median near 1.0 1.0 . In summary, for traditional recommender systems on typical user–item interaction data, incorporating more training data remains primarily beneficial, while weaker scaling behavior is concentrated in atypical dataset cases and in the algorithmic outlier RecBole BPR under our setup. 1. Introduction 1.1. Background Recommender systems rely on a limited number of parameters, with the dataset size being significant but practically controllable. This work investigates the relationship between dataset size and system performance to guide researchers and practitioners in making informed design choices aligned with their objectives. 1.2. Research Problem Since large volumes of high-quality, usable data are difficult to gather, the value of relying on ever-larger datasets should be clearly justified. Researchers have highlighted challenges such as scalability, data sparsity, and privacy concerns across domains like e-commerce and media streaming ( 12 ) . At the same time, industrial recommender systems must balance potential performance gains against system cost and latency ( 20 ) . Related evidence from practice suggests that non-algorithmic factors can dominate measured performance: changing the GUI has been shown to boost CTR by as much as 66\% , potentially exceeding gains from algorithm tuning ( 4 ) . This influence is therefore a key confounder when conducting experiments on most standard-scale RecSys datasets. However, larger datasets can be more GUI-diverse and might therefore mitigate potential bias, since they are often accumulated over longer timelines during which the interface likely evolves. Data pruning has on one hand been shown to result in up to a 3\% increase in performance when detrimental users are excluded ( 11 ) , while on the other hand a broader empirical study concluded that pruning should generally be avoided whenever possible ( 3 ) . Resolving this discrepancy in settings that incorporate larger data volumes remains an open question. Model training, particularly on large datasets, is computationally expensive and resource-intensive. Our longest observed cases occurred among LensKit runs on iPinYou at the 1m scale, where every LensKit run required more than 20 hours of end-to-end runtime. We observed similarly long end-to-end runtimes across different libraries and datasets. Researchers describe deep learning model training as especially time-consuming on large datasets, which can take up to several months ( 6 ) . Recent work by Vente et al. ( 17 ) emphasizes the considerable environmental cost of research on recommender systems, especially when using deep learning and large-scale datasets. Their analysis of full papers from the 2023 ACM Recommender Systems Conference shows that the average study consumed approximately 6,854 kWh of electricity, and that deep learning models required about eight times more energy than traditional machine learning methods without demonstrating clear performance benefits. The study further reports that the carbon footprint of recommender-system experiments has increased dramatically over the past decade, highlighting how larger datasets and more complex models intensify energy consumption and CO 2 emissions. Consequently, the pursuit of larger datasets must be critically assessed; the environmental and computational costs require that the need for such data abundance be clearly justified. 1.3. Research Question How does the performance of recommender systems evolve as the size of the training dataset increases? In particular, does performance improve steadily with more data or does it reach a point of diminishing returns? Is there an optimal dataset size at which a given recommender system likely achieves its best performance, beyond which additional training data yields no significant improvement? If such a saturation point exists, it needs to be identified, as recognizing it would greatly help reduce unnecessary computational costs and energy consumption. 1.4. Research Hypothesis We hypothesize that, within recommender systems, predictive performance continues to improve as training data grows across the scales studied, rather than reaching a clear saturation point. This expectation is consistent with prior work showing that larger datasets often yield higher accuracy and more stable results ( 1 ) , and with the broader view that scale can have an “unreasonable” positive effect ( 9 ) . At the same time, earlier studies show that the apparent benefit of additional data can depend on the model and evaluation design ( 5 ; 16 ) . Determining whether saturation occurs is therefore crucial: evidence of saturation would shift attention toward understanding bottlenecks, whereas evidence against it would justify further investment in efficient large-scale data collection. 2. Methodology 2.1. Overview We created a standard Python project that includes the libraries for two established tools, LensKit ( 7 ) and RecBole ( 19 ) , as well as other necessary utility libraries. We gathered freely available datasets from a list ( 13 ; 14 ) maintained by the RecBole team, prioritizing the larger options. An experimental run generally consists of multiple iterations as follows: first, load a prepared dataset; sample it to a specified number of rows using a chosen sampling strategy; select an algorithm from the list associated with the corresponding tool; provide the sampled dataset to the tool along with the selected algorithm; extract the resulting performance metric value; and finally, record it in the results file. The implementation and experiment scripts are publicly available ( 15 ) . 2.2. Environment Python 3.11 and pinned library versions were required for compatibility across the workflow. These explicit versions are kept in the requirements file. The main utility libraries used were NumPy, pandas, and Matplotlib, while the selected recommender frameworks were LensKit 2025.2.0 and RecBole 1.2.1. The project is configured for SLURM-based ( 18 ) execution and includes scripts for running jobs sequentially and in parallel, a Singularity ( 10 ) container definition, and a Makefile ( 8 ) . The OMNI HPC cluster of the University of Siegen was used for most computationally intensive jobs, where GPU-equipped nodes made the full experiment suite practical. 2.3. Configurations Nearly all configurations are defined in the constants module. This includes settings shared