Civic Ground Truth in News Recommenders: A Method for Public Value Scoring

James Meese, Kyle Herbertson · 2025

Read Civic Ground Truth in News Recommenders: A Method for Public Value Scoring by James Meese, Kyle Herbertson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22865.txt.

Introduction

News recommenders can serve public values, not just clicks. James Meese and Kyle Herbertson propose a civic scoring method in 2025. Prior work used expert input or audience feedback. Their approach is called civic ground truth. It embeds civic values into recommendation systems.

Labels come from a large structured audience survey. The survey is nationally representative. Automated metadata enrichment extends labels to wider corpora. That makes the labels generalizable across news sets. The method aims at civic minded systems. It builds on editorial and public service goals. Structured public input grounds the value scores. This offers a path beyond engagement only ranking.

Catalog record

CREATORJames Meese, Kyle Herbertson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22865

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Civic Ground Truth in News Recommenders: A Method for Public Value Scoring James Meese, Kyle Herbertson Abstract Research in news recommendation systems (NRS) continues to explore the best ways to integrate normative goals such as editorial objectives and public service values into existing systems. Prior efforts have incorporated expert input or audience feedback to quantify these values, laying the groundwork for more civic-minded recommender systems. This paper contributes to that trajectory, introducing a method for embedding civic values into NRS through large-scale, structured audience evaluations. The proposed civic ground truth approach aims to generate value-based labels through a nationally representative survey that are generalisable across a wider news corpus, using automated metadata enrichment. [orcid=0000-0003-1910-6166, [email protected], ] [orcid=0009-0001-3764-6222, [email protected], url=https://github.com/KalykeXIII, ] keywords recommendation ,public interest ,labelling ,survey ,civic signals ,similarity-based inference , † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: Normalize at the 19th ACM Recommender Systems Conference, September 22 - 26, 2025, Prague, Czech Republic † † address: School of Media and Communication, RMIT University, Latrobe St, Melbourne, Australia † † address: School of Media and Communication, RMIT University, Latrobe St, Melbourne, Australia 1 Introduction Research in news recommendation systems (NRS) has recognized the importance of ’beyond precision’ metrics, working to design recommender systems that support normative outcomes. Most notably, Sanne Vrijenhoek and colleagues have established evaluation metrics Vrijenhoek et al. 2024 , which can quantify the extent to which the output of recommendation systems aligns with normative standards. Two of these metrics have been empirically tested: activation, which refers to ’variations in journalistic writing style’ and alternative voices, ’the extent to which minorities are being represented’ ( Mattis et al. 2025 , p. 652) . It was found that certain deployments had a limited effect on users with audiences showing ’limited appetite for extreme versions of news recommenders that adhere to ideals of critical models of democracy’ ( Mattis et al. 2025 , p. 663) . Alongside this scholarly activity, similar efforts are being championed in industry contexts. Expert feedback is a common mechanism, where editorial objectives are embedded in NRS. For example, Swedish Radio has developed a public service algorithm allowing journalists to signal whether their story addresses key public service goals through metadata labelling. The Dutch public service organisation NPO (Nederlandse Publieke Omroep) also has an active audience survey, polling thousands of audience members daily about the public value of content they have watched through a mobile app. This enables the public service media organisation to generate a public value metric, which they use as a weighting mechanism within their wider recommender system. These different industry and scholarly projects provide some useful insight into the current state of normative NRS. On the academic side there are theoretically-informed Helberger 2021 normative tools enabling post-hoc auditing to be conducted Vrijenhoek et al. 2024 ; Vrijenhoek et al. 2024 and various empirical assessments are underway. In parallel, industry deployments are increasing. Our paper details efforts to advance this work by devising structured public value signals from a rating task, which can be extrapolated across a wider news corpus. Subsequent user evaluation will assist with further refinements before it is ultimately used as a weighting signal. The method is called civic ground truth and aims to build on the NPO model. Our paper proceeds as follows. We begin by outlining our concept of civic ground truth and setting out the contribution to the field, before detailing the methods for our first wave study, which involves metadata enrichment and a representative national survey, followed by semi-supervised extrapolation to generate public value scores across a corpus of news. We end with a brief discussion and conclusion. 2 Civic Ground Truth We define civic ground truth as audience-generated judgements about the societal importance of news. While the term ground truth implies a stable or coherent benchmark, socio-technical research has challenged these definitions noting the often provisional and ad-hoc nature of these datasets Jaton 2025 . Therefore, while the opinions of the public may fluctuate over time, we suggest they still provide a vital normative baseline for recommendation. In doing so the project directly accounts for the views of citizens, in contrast to current work where user input has been productively sought for empirical testing rather than value construction Heitz et al. 2022 ; Heitz et al. 2023 ; Mattis et al. 2025 ; Shin and Zhou 2024 . We capture these views through classification tasks conducted as a nationally representative survey, where participants rate items based on perceived personal and public interest. Respondents are guided with a brief description of the public interest and are then asked to also rate items based on certain sub-dimensions of public value. As the literature has noted, the concept of the ’public interest’ is ’broad, vague and loosely constructed’ ( Iosifidis 2011 , p.25) despite being a foundational concept of media law and policy Napoli 2019 ; Pickard 2019 ; Christians 2009 . However, recent work has shown that normative goals can be translated into technical areas despite these conceptual challenges Vrijenhoek et al. 2024 . Our scale continues this work by drawing on legislative descriptions of what news is considered valuable to Australians Commonwealth of Australia 2021 , previous conceptual efforts to identify ’public affairs’ content Dwyer and Wilding 2023 and pre-existing normative implementations such as RAdio Vrijenhoek et al. 2024 . Such an approach means that our labelling process is not wholly bottom-up, as respondents are given normative frames to evaluate content against. However, we offer a path towards a representative evaluation of a media corpus, rather than just capturing ratings post-consumption as NPO does. We also use these evaluations to inform future recommender testbeds and develop civic weightings that can inform re-ranking algorithms. Our survey captures the civic orientation of respondents drawing on previously validated batteries Prochazka and Schweiger 2019 and our own scale, which will be validated through expert review, Exploratory Factor Analysis and Confirmatory Factor Analysis. These questions will enable us to capture a range of civic profiles from issue-specific engagement with news to respondents who are disengaged with news. We now outline these methods in more detail below. 3 Methodology A persistent challenge in NRS research is the lack of sustained access to high-quality news datasets Vrijenhoek 2023 . While some datasets are available, they are typically released as static snapshots rather than actively maintained resources. This makes reproducibility and longitudinal analysis difficult. In the Australian context, such datasets are particularly scarce. To address this limitation, this study adopts a dataset-agnostic approach by relying solely on information that is regularly and publicly available to news consumers. This includes article headlines, featured images, bylines (author names), and publication dates. This offers greater methodological adaptability to future data sources, irrespective of their origin. Rather than sampling from existing datasets, the study constructs a ’complete’ dataset, collecting all articles published on the aggregate news site News.com.au over a one-month period. Securing a representative dataset of all news in Australia is challenging, especially given growing restrictions around web scraping. Given this wider context, we have selected one of the most visited commercial news websites in

Read the complete text