Informfully Recommenders – Reproducibility Framework for Diversity-aware Intra-session Recommendations Conference: Proceedings of the Nineteenth ACM Conference on Recommender Systems; September 22–26, 2025; Prague, Czech Republic Proceedings of the Nineteenth ACM Conference on Recommender Systems (R

Lucien Heitz, Runze Li, Oana Inel, Abraham Bernstein · 2025

Read Informfully Recommenders – Reproducibility Framework for Diversity-aware Intra-session Recommendations Conference: Proceedings of the Nineteenth ACM Conference on Recommender Systems; September 22–26, 2025; Prague, Czech Republic Proceedings of the Nineteenth ACM Conference on Recommender Systems (R by Lucien Heitz, Runze Li, Oana Inel, Abraham Bernstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13019.txt.

Introduction

Heitz and co-authors introduce Informfully Recommenders, a toolkit for testing recommendation systems that aim for diversity, not just accuracy. News and media recommenders can trap users in narrow feeds, and researchers who study fairer designs lacked a shared, reproducible way to run their experiments.

Built on the Cornac framework, the toolkit covers the whole pipeline: preparing datasets, training diversity-aware models, re-ranking items within a session, and scoring results with many diversity metrics. The authors show it off with a large offline experiment in the news domain. By standardizing each stage, the toolkit lets different teams compare diversity methods fairly instead of each rebuilding the test bench.

Catalog record

CREATORLucien Heitz, Runze Li, Oana Inel, Abraham Bernstein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13019

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Informfully Recommenders – Reproducibility Framework for Diversity-aware Intra-session Recommendations Conference: Proceedings of the Nineteenth ACM Conference on Recommender Systems; September 22–26, 2025; Prague, Czech Republic Proceedings of the Nineteenth ACM Conference on Recommender Systems (RecSys ’25), September 22–26, 2025, Prague, Czech Republic DOI: 10.1145/3705328.3748148 CCS: Information systems Recommender systems CCS: Human-centered computing Open source software CCS: Information systems Information retrieval diversity Lucien Heitz, Runze Li, Oana Inel, Abraham Bernstein Abstract . Norm-aware recommender systems have gained increased attention, especially for diversity optimization. The recommender systems community has well-established experimentation pipelines that support reproducible evaluations by facilitating models’ benchmarking and comparisons against state-of-the-art methods. However, to the best of our knowledge, there is currently no reproducibility framework to support thorough norm-driven experimentation at the pre-processing, in-processing, post-processing, and evaluation stages of the recommender pipeline. To address this gap, we present Informfully Recommenders, a first step towards a normative reproducibility framework that focuses on diversity-aware design built on Cornac. Our extension provides an end-to-end solution for implementing and experimenting with normative and general-purpose diverse recommender systems that cover 1) dataset pre-processing, 2) diversity-optimized models, 3) dedicated intra-session item re-ranking, and 4) an extensive set of diversity metrics. We demonstrate the capabilities of our extension through an extensive offline experiment in the news domain. † † cc-license: by 1. Introduction Recommender systems (RSs) help users to find their way in the vast online information space, shape the public discussion, and serve as a foundation for public cohesion ( Helberger et al. 2018 ; Bernstein et al. 2021 ; Heitz et al. 2022 ) . In the news domain, for example, they fulfill an important “democratic role” ( Helberger 2019 ) for political opinion formation and informational self-determination ( Sargeant et al. 2022 ) . Therefore, depending on the target domain, RSs face unique challenges that require balancing societal norms and values on the one hand ( Helberger et al. 2018 ) as well as technical performance and economic goals of platform owners on the other hand ( Cohen 2002 ; Richardson et al. 2007 ) . In the context of this paper, we refer to such RS instances as a normative RS (NRS). We follow the definition of Vrijenhoek et al. 2023 where normativity is understood as the practice of operationalizing societal norms and values as part of the RS pipeline. Development and evaluation of NRSs is challenging. When looking at a target objective for NRSs, such as diversity, there is disagreement on the conceptual level on the precise notion ( Loecherbach et al. 2020 ; Vrijenhoek et al. 2024 ) . Additionally, online user studies for assessing the algorithm’s impact on users remain the exception in RS research ( Bauer et al. 2024 ) , and the understanding of NRSs as well as re-rankers remains limited ( Treuillier et al. 2022 ) . These studies, however, are vital to assess the performance of NRSs, because with the predominant focus on offline evaluation, it is unclear how algorithms impact users ( Jannach and Bauer 2020 ) . One reason for this lack of proper assessment is the requirement to have sufficiently rich datasets with complementary information on items, participants’ backgrounds ( Iana et al. 2023 ; Heitz et al. 2024c ) , normative models/re-rankers ( Castells et al. 2021 ) , diversity metrics ( Vrijenhoek et al. 2024 ) , and visualizations ( Beel and Dixon 2021 ; Heitz et al. 2024a ) for the evaluation with users. Despite recent efforts to promote more normative and beyond accuracy perspectives (e.g., see RecSys 2024 Challenge ( Kruse et al. 2024b ; Kruse et al. 2024a ; Heitz et al. 2024b ) ), there have not yet been any dedicated end-to-end pipelines proposed for the systematic evaluation of NRS. To address these shortcomings, we present Informfully Recommenders, a first open-source reproducibility framework for norm-aware approaches, such as diversity. 1 1 1 Informfully Recommenders: https://github.com/Informfully/Recommenders Experiment configuration files: https://github.com/Informfully/Experiments We provide a complementary online documentation of the codebase with an extended technical description of each of the pipeline stages that we outline in this paper: https://informfully.readthedocs.io Our framework’s contributions include: 1) six out-of-the-box dataset augmentation functions to add norm-relevant dimensions to items (supporting multiple languages), 2) three random walks and two lightweight diversity models for norm-aware recommendations, 3) three diversity-optimized re-ranker algorithms for use with existing models, together with two intra-session re-rankers for the user simulator, 4) five traditional and five normative diversity metrics for assessing recommendations, and 5) compatibility with the Informfully Research Platform ( Heitz et al. 2024a ) to support item visualization for online user studies. It is designed as an extension to the well-established Cornac framework ( Salah et al. 2020 ; Truong et al. 2021a ) , providing an end-to-end solution for implementing and experimenting with NRSs. Table 1. Overview of open-source reproducibility framework. The comparison looks at supported datasets, models, re-rankers, and metrics, as well as augmentation, simulation, and visualization capabilities. Overview of open-source reproducibility framework. The comparison looks at supported datasets, models, re-rankers, and metrics, as well as augmentation, simulation, and visualization capabilities. Framework Modes Models Re-rankers Metrics Data Aug-mentation User Simulator Item Visualization ClayRS ( Lops et al. 2023 ) OFF TRA ACC Cornac ( Salah et al. 2020 ; Truong et al. 2021b ; Truong et al. 2021a ) + A/B ( Ong et al. 2024 ) ON, OFF TRA ACC, BEY \checkmark daisyRec 2.0 ( Sun et al. 2022 ) OFF TRA ACC, BEY Elliot ( Anelli et al. 2021 ) OFF TRA ACC, BEY FuxiCTR ( Zhu et al. 2021 ; Zhu et al. 2022 ) OFF TRA ACC LensKit ( Ekstrand 2020 ) OFF TRA ACC Microsoft Recommenders ( Argyriou et al. 2020 ) OFF TRA ACC, BEY RecBole ( Zhao et al. 2021 ) OFF TRA ACC, BEY \checkmark ReChorus 2.0 ( Li et al. 2024 ) OFF TRA STA ACC RecList ( Chia et al. 2022 ) OFF TRA \checkmark RecPack ( Michiels et al. 2022 ) OFF TRA STA ACC, BEY \checkmark Informfully Recommenders ON, OFF NOR, TRA STA, DYN ACC, BEY \checkmark \checkmark \checkmark This enables Informfully Recommenders to assist the systematic integration, evaluation, and assessment of societal values and norm-awareness into recommender systems. We show the applicability of our extension in the context of news recommendations—a domain closely linked to normative societal values in general ( Helberger 2019 ; Vrijenhoek et al. 2021 ; Vrijenhoek et al. 2023 ) and diversity in particular ( Vrijenhoek et al. 2022 ; Heitz et al. 2024b ) . 2. Related Work In this section, we compare RS reproducibility frameworks to show their respective shortcomings for assessing NRSs. Table 1 presents a comparison of open-source reproducibility frameworks that can be used to tackle news recommendations and diversity. 2 2 2 This list is based on the ACM RecSys repository of evaluation framework recommendations: https://github.com/ACMRecSys/recsys-evaluation-frameworks We compare the capabilities of the frameworks on the following dimensions: Modes:: Shows if frameworks support on line user experiment (ON) or if they are focusing on off line benchmarking (OFF). Models:: Lists available model types. Options include nor mative (NOR) and tra ditional models (e.g, accuracy, TRA). Re-rankers:: We differentiate between sta tic re-ranking of candidate lists after

Read the complete text