User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems

Jianling Wang, Yifan Liu, Yinghao Sun, Xuejian Ma, Yueqi Wang, He Ma, Zhengyang Su, Minmin Chen, Mingyan Gao, Onkar Dalal, Ed H. Chi, Lichan Hong, Ningren Han, Haokai Lu · 2025

Read User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems by Jianling Wang, Yifan Liu, Yinghao Sun, Xuejian Ma, Yueqi Wang, He Ma, Zhengyang Su, Minmin Chen, Mingyan Gao, Onkar Dalal, Ed H. Chi, Lichan Hong, Ningren Han, Haokai Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05522.txt.

Introduction

Jianling Wang and co-authors tackle exploration in large recommendation systems in a 2025 paper. Recommenders tend to loop on what users already like, so showing new content is risky and poorly measured. Large language models could suggest fresh items using their broad knowledge, if they can be aligned with user taste.

The challenge is to steer the model toward a user's preferences without erasing its knowledge and reasoning. The team uses user feedback alignment for that steering. Better exploration matters because it shapes what millions of people discover, not just what they repeat.

Catalog record

CREATORJianling Wang, Yifan Liu, Yinghao Sun, Xuejian Ma, Yueqi Wang, He Ma, Zhengyang Su, Minmin Chen, Mingyan Gao, Onkar Dalal, Ed H. Chi, Lichan Hong, Ningren Han, Haokai Lu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05522

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems Jianling Wang, Yifan Liu, Yinghao Sun, Xuejian Ma, Yueqi Wang, He Ma, Zhengyang Su, Minmin Chen, Mingyan Gao, Onkar Dalal, Ed H. Chi, Lichan Hong, Ningren Han, Haokai Lu Abstract Exploration, the act of broadening user experiences beyond their established preferences, is challenging in large-scale recommendation systems due to feedback loops and limited signals on user exploration patterns. Large Language Models (LLMs) offer potential solutions by leveraging their world knowledge to recommend novel content outside these loops. A key challenge is aligning LLMs with user preferences while preserving their knowledge and reasoning. To enhance planning for new user interests using LLMs, this paper introduces a novel approach that combines hierarchical planning with LLM inference-time scaling. It aims to improve recommendation relevancy without compromising novelty. We decouple novelty and user-alignment, training separate LLMs for each objective. We then scale up the novelty-focused LLM’s inference and select the best-of-n predictions using the user-aligned LLM. Live experiments demonstrate efficacy, showing significant gains in both user satisfaction (measured by watch activity and active user counts) and exploration diversity. * * footnotetext: indicates equal contribution 1 Introduction Large Language Models (LLMs) present a significant opportunity to revolutionize recommendation systems Wu et al. 2024 , due to their powerful reasoning, planning, and world knowledge capabilities. Traditional recommendation backbones, such as collaborative filtering and content-based methods, typically suggest items by identifying similar users based on past interactions, which often reinforce existing preferences and perpetuate feedback loops Chaney et al. 2018 ; Mansoury et al. 2020 . LLMs can overcome these limitations by leveraging their vast world knowledge to generate novel and diverse recommendations that go beyond a user’s historical interactions, thus driving long-term user engagement Chen 2021 . Among recent advancements leveraging LLMs for recommendation systems Bao et al. 2023 ; Lin et al. 2024 ; Wang et al. 2024a , the hierarchical planning paradigm Wang et al. 2024c stands out as a promising and deployable approach that combines an LLM, which provides high-level guidance, with traditional recommenders for efficient item-level serving. As this solution has been adopted in industry, the subsequent challenge lies in effectively integrating real-world human feedback into the LLM. While human feedback is key to optimizing LLMs Ouyang et al. 2022 , systematically incorporating it into recommendation systems remains an under-explored area, offering both challenges and opportunities for future research. Using real-world human feedback is challenging because recommendation systems rely on noisy implicit signals (e.g., clicks or dwell time) instead of explicit comparative judgments (e.g., side-by-side comparisons). This makes it hard to translate such feedback into robust training objectives for LLMs that align with users’ true preferences. More importantly, balancing novelty and relevance – two usually competing objectives – is crucial for exploration in recommendation systems as relevant novel content drives sustained user satisfaction. Initial experiments with the hierarchical planning Wang et al. 2024c framework, using an LLM as a novelty model to identify novel interest clusters and subsequently retrieve relevant items, demonstrated the potential of this approach. However, aligning the novelty model’s predictions with user preferences remains challenging. Directly fine-tuning with more users’ interaction history data yielded neutral results and raised concerns about memorization and loss of novelty. Attempts at RLHF Ouyang et al. 2022 with a reward model also proved unsuccessful as it undermined the controlled generation capability (see in Sec. 3 ). To address these challenges, we propose a novel, decomposed approach that leverages two specialized LLMs for high-level planning: a novelty model and an alignment model. To balance novelty and relevance, the alignment LLM is trained specifically to evaluate and rate the predictions of the novelty model based on observed user feedback. This separation allows for the independent optimization of novelty generation and preference alignment. Moreover, to further improve the system’s ability to generate relevant novel predictions, we scale inference-time compute by generating multiple independent predictions from the novelty model using a high temperature setting. The alignment model then acts as a selector, choosing the most user-aligned outputs from the novelty model. This combination of specialized models, training signals derived from collective user behaviors, and repeated sampling significantly increases the likelihood of generating recommendations that are both novel and relevant. In summary, this paper presents a system that has been deployed on a commercial short-form video recommendation platform serving billions of users. The key contributions are: (1) Collective User Feedback Alignment : We introduce an LLM-based alignment model specifically trained to evaluate the novelty model’s predictions based on collective user behaviors. By aggregating implicit signals(e.g. clicks and dwell time) for interest clusters transition across many users, we enable the system to learn user preferences with reduced noise and bias. (2) Inference-Time Scaling : We demonstrate the effectiveness of repeated sampling at inference time, allowing the alignment model to select the most relevant predictions from a diverse set of candidates generated by the novelty model, thereby improving exploration. (3) Decomposed Novelty and Preference Modeling : We propose a novel paradigm that decouples novelty generation and preference modeling into two specialized LLMs. This separation enables independent optimization for each objective. Consequently, it directly addresses the core challenge of balancing novelty with relevance via specialized models, leading to a significantly improved operating curve for user interest exploration. 2 Related Work This research builds upon two primary streams of existing work: the application of LLMs to recommendation systems and the ongoing efforts to improve recommendation exploration. LLMs for Recommendation Systems . The advances in LLM capabilities have recently drawn a lot of attention to their potential in recommendation systems Bao et al. 2023 ; Geng et al. 2023 ; Hou et al. 2024 ; Li et al. 2023 ; Liu et al. 2023 ; Wang et al. 2024b . One promising direction involves augmenting traditional recommendation models with LLM-powered feature engineering, including supplementary textual features or embeddings that encode world knowledge Xi et al. 2024 ; Ren et al. 2024 . Another approach focuses on directly generating recommendations using LLMs; e.g., Hou et al. 2024 and Gao et al. 2023 have experimented with prompting off-the-shelf LLMs to produce ranked lists of recommendations. Meanwhile, there are also work involving fine-tuning LLMs Singh et al. 2024 ; Bao et al. 2023 ; lin2024rella to better align them with the recommendation domain, whether through incorporating domain-specific knowledge, generating new tokens, or predicting user preferences for specific user-item pairs. However, few of these methods are truly equipped to handle query-per-second (QPS) requirements of real-time applications. Wang et al. 2024a addresses this by employing LLMs as data augmentation tools for conventional recommendation systems during training, thereby boosting performance without incurring additional serving costs. Recommendation Exploration . Improving user interest exploration is key to broadening preferences and fostering long-term engagement Chen et al. 2021 ; Chen 2021 ; Su et al. 2024 . However, a key challenge lies in the inherent closed-loop nature of existing recommendation systems Chaney et al. 2018 ; Mansoury et al. 2020 ; Wang et al. 2023 . Training data is primarily derived from past user-item interactions, limiting the system’s ability to explore truly novel interests. While methods like PIE Mahajan et al. 2023 offer improvements through user-creator affinity and online bandit formulations, they remain

Read the complete text