Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery an
Jingxin Liu, Xiang Gao, YiSha Li, Xin Li, Haiyang Lu, Ben Wang · 2024
Read Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery an by Jingxin Liu, Xiang Gao, YiSha Li, Xin Li, Haiyang Lu, Ben Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10381.txt.
Introduction
Jingxin Liu and co-authors present, in a 2024 conference paper, a reinforcement learning system that decides when to slip a live stream into a user's short video feed. Each request allows at most one live card, and greedy placement can sour long-term use. Standard learning methods wobble and diverge at industrial scale, which blocks deployment.
Their design steadies the learner with supervised help: multi-task reward learning curbs error build-up in the value side. Splitting the state into groups for both halves of the system cuts prediction variance. A shaped reward stops the system from over-allocating live streams. Offline scoring and online A/B tests were used to judge it.
Catalog record
| CREATOR | Jingxin Liu, Xiang Gao, YiSha Li, Xin Li, Haiyang Lu, Ben Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10381
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada DOI: 10.1145/3711896.3737264 ISBN: 979-8-4007-1454-2/2025/08 CCS: Information systems Recommender systems CCS: Computing methodologies Reinforcement learning Jingxin Liu, Xiang Gao, YiSha Li, Xin Li, Haiyang Lu, Ben Wang Abstract . In the context of a short video & live stream mixed recommendation scenario, the live stream recommendation system (RS) decides whether to allocate at most one live stream to the video feed for each user request. The inappropriate policy which ignores the long-term negative impact of live stream allocation can significantly affect app usage duration and user retention. To maximize long-term user engagement, it is crucial to determine an optimal policy for accurate live stream allocation. Recently, reinforcement learning (RL) has been widely applied in recommendation systems to capture long-term user engagement. However, traditional RL algorithms often face divergence and instability problems, which restricts application and deployment in large-scale industrial recommendation systems, especially in the aforementioned challenging scenario. To address these challenges, we propose a novel S upervised L earning-enhanced M ulti- G roup A ctor C ritic algorithm (SL-MGAC). Specifically, we introduce a supervised learning-enhanced actor-critic framework that incorporates variance reduction techniques, where multi-task supervised reward learning helps restrict bootstrapping error accumulation during critic learning. Additionally, we design a multi-group state decomposition module for both actor and critic networks to reduce prediction variance and improve model stability. We also propose a novel reward function to prevent overly greedy live stream allocation. Empirically, we evaluate the SL-MGAC algorithm using offline policy evaluation (OPE) and online A/B testing. Experimental results demonstrate that the proposed method not only outperforms baseline methods under platform-level constraints, but also exhibits improved stability in online recommendation scenarios. Figure 1. Structure of a short video & live stream mixed recommendation system(RS). The decision making of SL-MGAC takes place in the final stage of live stream RS. 1. Introduction We consider a challenging sequential decision making scenario in a short video & live stream blended recommendation system, as shown in Fig. 1 . The system consists of three components: a live streaming recommendation system, a short video recommendation system, and a blending server. For each user request with timestamp t , the live stream recommendation system decides whether to inject the recommended live stream into the video feed, while the short video recommendation system suggests B (with B<10 ) videos. The blending server then mixes and rearranges the single live stream with the B videos to form the final recommendation list. Our objective is to find a personalized optimal live stream allocation policy that not only maximizes user long-term engagement with live streams, but also meets the constraint of not causing a negative impact on app usage duration and user retention, which can be modeled as an infinite request-level Constrained Markov Decision Process (CMDP) ( Altman 2021 ) . Currently, deep reinforcement learning has shown great potential in learning policies to maximize long-term rewards in various research domains, such as computer vision, robotics, natural language processing, gaming, and recommendation systems ( Afsar et al. 2022 ; Li 2017 ) . In the context of recommendation systems, reinforcement learning is applied to optimize long-term user engagement ( Zou et al. 2019 ) and improve user retention ( Cai et al. 2023 ) . Numerous RL applications have been proposed for real-world recommendation systems, including slate recommendation ( Ie et al. 2019 ; Deffayet et al. 2023 ; Liu et al. 2023 ) , personalized search ranking ( Miao et al. 2021 ) , and advertisement allocation ( Liao et al. 2022 ) . However, the intrinsic issues of divergence and instability associated with traditional reinforcement learning algorithms (RL) ( François-Lavet et al. 2015 ; Fujimoto et al. 2018 ; Kumar et al. 2019 ; Dasagi et al. 2019 ) are significantly exacerbated in the short video & live stream mixed recommendation system. A primary possible reason for this is the drastic fluctuations in both the live stream supply scale and user interaction behaviors over time, as shown in Fig. 7 in Appendix B . RL models often struggle to learn effective policies from data that exhibit such high variance across both time scales and user scales. In practice, we observe that RL models frequently encounter issues of policy deterioration or model collapse ( Dohare et al. 2023 ) . Most importantly, the live-stream allocation agent is the final module of the live-stream RS , as shown in Fig 1 . If the RL agent becomes highly unstable or collapses in the online environment, it may lead to an excessive injection of live-stream content into the short-video feed. In turn, this can negatively impact the exposure of advertisements and E-commerce videos, potentially causing severe system malfunctions and significant platform losses. Hence, in the industrial RS, this RL application is more risky than existing RL methods ( Cai et al. 2023 ; Zhang et al. 2022 ; Zhang et al. 2024 ) , which are only applied in the multi-rank score aggregation of the ranking stage. Furthermore, like advertisement, live stream is not welcomed by every user. Allocating too many live streams for a user will significantly interrupt the user’s short video interest and lead to a decrease in the user’s app usage duration, which will eventually affect the user engagement and user retention. Therefore, we cannot merely focus on the user feedback for a single request, but rather should optimize the long-term user experience. To address the aforementioned problems, we propose a novel S upervised L earning-enhanced M ulti- G roup A ctor C ritic algorithm, SL-MGAC. Given the significant variation in users’ interest in live stream content, which introduces high variance at both the feature and feedback levels, we incorporate separate M ulti- G roup S tate D ecomposition (MG-SD) modules within both the actor and critic networks. Additionally, we combine multi-task supervised reward learning with traditional critic learning to not only restrict temporal difference (TD) error accumulation during model training but also improve the accuracy of Q-value estimation. Through experiments, we compare the SL-MGAC method with competitive baselines using offline policy evaluation (OPE) ( Uehara et al. 2022 ) and online A/B tests to demonstrate its effectiveness and stability. The main contributions of this paper are as follows. • We introduce SL-MGAC, a novel RL model that outperforms existing methods in maximizing the user engagement of live stream while satisfying the platform-level constraints on app usage duration and user retention. • We propose new variance reduction ( Greensmith et al. 2004 ; Kumar et al. 2019 ; Anschel et al. 2017 ; Mao et al. 2018 ; Romoff et al. 2018 ) techniques, including multi-group state decomposition, distribution discretization for reward learning, reward normalization, and Q value normalization. • We integrate multi-task supervised reward learning with traditional critic learning to alleviate TD error accumulation and improve the accuracy of Q-value estimation. • We successfully deploy the SL-MGAC model in a challenging short video and live stream