A Worrying Reproducibility Study of Intent-Aware Recommendation Models CCS: Information systems Recommender systems
Faisal Shehzad, Maurizio Ferrari Dacrema, Dietmar Jannach · 2025
Read A Worrying Reproducibility Study of Intent-Aware Recommendation Models CCS: Information systems Recommender systems by Faisal Shehzad, Maurizio Ferrari Dacrema, Dietmar Jannach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10143.txt.
Description
Recommender systems that guess why a user clicked promise better suggestions, but Faisal Shehzad, Maurizio Ferrari Dacrema, and Dietmar Jannach could not make the promise reproduce in a 2025 study. They took five recent intent-aware models from top venues and tried to rerun them as published.
In two cases, even the authors' own code with the reported best settings failed to reach the reported numbers. Worse, every one of the five was beaten by at least one traditional, non-neural model once the baselines were properly tuned. The authors place the failure in a pattern seen across neural recommendation research: weak baselines and loose reporting inflate gains. Their call is for stricter scholarly practice, tuned comparisons and reproducible setups as the price of a claim.
Catalog record
| CREATOR | Faisal Shehzad, Maurizio Ferrari Dacrema, Dietmar Jannach |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10143