A Worrying Reproducibility Study of Intent-Aware Recommendation Models CCS: Information systems Recommender systems

Faisal Shehzad, Maurizio Ferrari Dacrema, Dietmar Jannach · 2025

Read A Worrying Reproducibility Study of Intent-Aware Recommendation Models CCS: Information systems Recommender systems by Faisal Shehzad, Maurizio Ferrari Dacrema, Dietmar Jannach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10143.txt.

Description

Recommender systems that guess why a user clicked promise better suggestions, but Faisal Shehzad, Maurizio Ferrari Dacrema, and Dietmar Jannach could not make the promise reproduce in a 2025 study. They took five recent intent-aware models from top venues and tried to rerun them as published.

In two cases, even the authors' own code with the reported best settings failed to reach the reported numbers. Worse, every one of the five was beaten by at least one traditional, non-neural model once the baselines were properly tuned. The authors place the failure in a pattern seen across neural recommendation research: weak baselines and loose reporting inflate gains. Their call is for stricter scholarly practice, tuned comparisons and reproducible setups as the price of a claim.

Catalog record

CREATORFaisal Shehzad, Maurizio Ferrari Dacrema, Dietmar Jannach
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-10143

Browse the full library · About the Tower of Babel