HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems DOI: XXXXXXX.XXXXXXX Conference: Companion Proceedings of the ACM Web Conference 2026; April 28–May 2, 2026; Sydney, Australia ISBN: 978-1-4503-XXXX-X/2026/04 CCS: Information systems Recommender systems CCS: Human-cente

Sushant Mehta · 2026

Read HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems DOI: XXXXXXX.XXXXXXX Conference: Companion Proceedings of the ACM Web Conference 2026; April 28–May 2, 2026; Sydney, Australia ISBN: 978-1-4503-XXXX-X/2026/04 CCS: Information systems Recommender systems CCS: Human-cente by Sushant Mehta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19197.txt.

Introduction

Sushant Mehta builds a human centered way to grade chat based recommenders. Language models have given recommenders new skills: they understand plain requests, explain picks, and hold a conversation. Old grading methods, though, focus on classic scores like accuracy of the ranked list.

Those scores miss how the system treats the person using it. The framework, HELM, adds the human side to the grade: trust, clarity, and the feel of the exchange. As recommenders turn into chat partners, grading what users experience matters as much as grading the list itself.

Catalog record

CREATORSushant Mehta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19197

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems DOI: XXXXXXX.XXXXXXX Conference: Companion Proceedings of the ACM Web Conference 2026; April 28–May 2, 2026; Sydney, Australia ISBN: 978-1-4503-XXXX-X/2026/04 CCS: Information systems Recommender systems CCS: Human-centered computing Interactive systems and tools CCS: Computing methodologies Natural language processing Sushant Mehta Abstract . The integration of Large Language Models (LLMs) into recommendation systems has introduced unprecedented capabilities for natural language understanding, explanation generation, and conversational interactions. However, existing evaluation methodologies focus predominantly on traditional accuracy metrics, failing to capture the multifaceted human-centered qualities that determine the real-world user experience. We introduce Helm ( H uman-centered E valuation for L LM-powered reco M menders), a comprehensive evaluation framework that systematically assesses LLM-powered recommender systems across five human-centered dimensions: Intent Alignment , Explanation Quality , Interaction Naturalness , Trust & Transparency , and Fairness & Diversity . Through extensive experiments involving three state-of-the-art LLM-based recommenders (GPT-4, LLaMA-3.1, and P5) across three domains (movies, books, and restaurants), and rigorous evaluation by 12 domain experts using 847 recommendation scenarios, we demonstrate that Helm reveals critical quality dimensions invisible to traditional metrics. Our results show that while GPT-4 achieves superior explanation quality (4.21/5.0) and interaction naturalness (4.35/5.0), it exhibits a significant popularity bias (Gini coefficient 0.73) compared to traditional collaborative filtering (0.58). We release Helm as an open-source toolkit to advance human-centered evaluation practices in the recommender systems community. 1. Introduction The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in recommender systems research ( 33 ; 22 ) . LLM-powered recommenders offer transformative capabilities, including elicitation of natural language preferences, rich explanation generation, and fluid conversational interactions ( 7 ; 15 ) . Systems such as P5 ( 9 ) , TALLRec ( 1 ) and Chat-REC ( 7 ) have demonstrated that LLMs can unify various recommendation tasks within a single text-to-text framework, while commercial deployments in platforms like ChatGPT plugins showcase their practical potential. However, the evaluation of these systems remains anchored in traditional accuracy-centric metrics – Hit Rate, NDCG, and Mean Reciprocal Rank – that fundamentally misalign with the human-centered qualities that distinguish LLM-powered recommenders ( 24 ; 35 ) . This methodological gap is particularly concerning given that the main value proposition of LLM integration lies in enhanced user understanding, natural interaction, and transparent reasoning ( 18 ) . Consider a movie recommendation scenario: a traditional collaborative filtering system might achieve higher NDCG by recommending popular blockbusters, while an LLM-powered system provides a thoughtful recommendation for an independent film with a compelling explanation of why it matches the user’s stated mood and preferences. Traditional metrics would favor the former, yet the latter may deliver a superior user experience through better intent understanding and trust-building explanation ( 37 ) . The human-centered recommender systems community has long advocated for multidimensional evaluation that includes user satisfaction, trust, and social responsibility ( 27 ; 8 ) . The ResQue framework ( 27 ) established the foundational constructs for user-centric evaluation, while recent work on trustworthy recommendation systems ( 8 ) has formalized the dimensions of explainability, fairness, and controllability. However, no comprehensive framework addresses the unique evaluation challenges posed by LLM-powered systems. We address this gap by introducing Helm ( H uman-centered E valuation for L LM-powered reco M menders), a systematic evaluation framework comprising five dimensions: (1) Intent Alignment : How accurately does the system understand and address the user’s underlying goals, including implicit preferences and contextual needs? (2) Explanation Quality : How effectively do generated explanations help users understand recommendations and support their decision-making? (3) Interaction Naturalness : How fluid, coherent, and human-like are conversational interactions with the system? (4) Trust & Transparency : Does the system foster user trust through honest uncertainty communication and consistent behavior? (5) Fairness & Diversity : Does the system provide equitable recommendations across user groups and item distributions? Through extensive experiments with 12 domain experts evaluating 847 recommendation scenarios across three LLM-based systems and three domains, we demonstrate that Helm reveals quality dimensions invisible to traditional metrics. Our contributions are: • A comprehensive human-centered evaluation framework specifically designed for LLM-powered recommender systems, grounded in established HCI and recommender systems theory. • Rigorous empirical validation through expert evaluation, establishing reliability metrics and demonstrating discriminative validity across systems. • Novel findings on the trade-offs between LLM capabilities and fairness, revealing that stronger language understanding correlates with increased popularity bias. • An open-source evaluation toolkit and annotated dataset to support reproducible human-centered evaluation research. 2. Related Work 2.1. Human-Centered Recommender Systems The vision of human-centered recommender systems has evolved from early user-centric design principles ( 18 ) to comprehensive frameworks addressing user experience, trust, and societal impact ( 8 ) . Konstan and Terveen ( 18 ) traced this evolution, emphasizing that recommender systems should be designed around user objectives rather than optimization metrics alone. The ResQue framework ( 27 ) operationalized this vision through 32 questionnaire items that measure recommendation quality, interface adequacy, and behavioral intentions. Subsequent work extended ResQue for conversational systems (CRS-Que ( 16 ) ), adding constructs for dialogue coherence and response appropriateness. Our framework builds upon these foundations while addressing LLM-specific evaluation challenges. Trustworthy recommender systems research ( 8 ) has identified five critical dimensions: explainability, fairness, privacy, robustness, and controllability. We integrate these dimensions within our human-centered framework while adding LLM-specific considerations such as hallucination detection and chain-of-thought reasoning evaluation. 2.2. LLM-Powered Recommender Systems The integration of LLMs into recommendation has followed two primary paradigms ( 33 ) : discriminative approaches that use LLMs for feature extraction or re-ranking, and generative approaches that frame recommendation as language generation. The P5 paradigm ( 9 ) pioneered unified text-to-text recommendation, enabling a single model to handle rating prediction, sequential recommendation, explanation generation, and direct recommendation through natural language prompts. TALLRec ( 1 ) demonstrated that lightweight LoRA fine-tuning achieves strong performance with minimal training data, while LLMRec ( 32 ) combined LLM knowledge with graph neural networks. Conversational recommender systems ( 15 ; 5 ) represent a closely related research thread. Chat-REC ( 7 ) augmented ChatGPT with retrieval mechanisms for interactive recommendations, while recent work has explored emotional alignment ( 36 ) and multi-turn preference elicitation. Evaluation of LLM-based recommenders has primarily relied on traditional metrics ( 2 ; 12 ) , with limited attention to human-centered qualities. Dai et al. ( 2 ) evaluated ChatGPT’s ranking capabilities using point-wise, pair-wise, and list-wise protocols but focused exclusively on accuracy. Di Palma et al. ( 4 ) proposed robustness-aware evaluation addressing hallucinations but did not incorporate user studies. 2.3. Evaluation Beyond Accuracy The “beyond accuracy” movement in recommender systems ( 24 ; 17 ) has established metrics for diversity, novelty, serendipity, and coverage. Kaminskas and Bridge ( 17 ) provided a comprehensive survey of these objectives, highlighting the difficulty of optimizing multiple criteria simultaneously. Fairness in recommender systems ( 31 ; 14 ) has received growing attention, with research addressing consumer fairness (equal treatment across user groups) and provider fairness (equitable item exposure). Recent work has revealed concerning biases in LLM-based systems ( 3 ) , particularly in cold-start scenarios. Explainability evaluation ( 37 ; 29 ) has evolved from simple readability metrics to

Read the complete text