HyperZero : A Customized End-to-End Auto-Tuning System for Recommendation with Hourly Feedback

Xufeng Cai, Ziwei Guan, Lei Yuan, Ali Selman Aydin, Tengyu Xu, Boying Liu, Wenbo Ren, Renkai Xiang, Songyi He, Haichuan Yang, Serena Li, Mingze Gao, Yue Weng, Ji Liu · 2025

Read HyperZero : A Customized End-to-End Auto-Tuning System for Recommendation with Hourly Feedback by Xufeng Cai, Ziwei Guan, Lei Yuan, Ali Selman Aydin, Tengyu Xu, Boying Liu, Wenbo Ren, Renkai Xiang, Songyi He, Haichuan Yang, Serena Li, Mingze Gao, Yue Weng, Ji Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18126.txt.

Introduction

Xufeng Cai, Ziwei Guan, Lei Yuan, and a team of co-authors build HyperZero, an automatic tuner for the scoring formulas inside a giant recommendation system. Such systems first predict many signals for each item. Then they blend them into one value score. The blend weights quietly shape what billions of users see.

General tuning tools exist. But they take weeks or months to find good settings. That is far too slow when a poor blend degrades the service every day. HyperZero is built to land a workable blend within two or three days instead. It is tuned to the quirks of hourly feedback from live traffic. The authors describe the system's design for this fast cycle. They suggest the frame can stretch to other tuning jobs across recommendation platforms.

Catalog record

CREATORXufeng Cai, Ziwei Guan, Lei Yuan, Ali Selman Aydin, Tengyu Xu, Boying Liu, Wenbo Ren, Renkai Xiang, Songyi He, Haichuan Yang, Serena Li, Mingze Gao, Yue Weng, Ji Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18126

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HyperZero : A Customized End-to-End Auto-Tuning System for Recommendation with Hourly Feedback Xufeng Cai, Ziwei Guan, Lei Yuan, Ali Selman Aydin, Tengyu Xu, Boying Liu, Wenbo Ren, Renkai Xiang, Songyi He, Haichuan Yang, Serena Li, Mingze Gao, Yue Weng, Ji Liu Abstract . Modern recommendation systems can be broadly divided into two key stages: the ranking stage, where the system predicts various user engagements (e.g., click-through rate, like rate, follow rate, watch time), and the value model stage, which aggregates these predictive scores through a function (e.g., a linear combination defined by a weight vector) to measure the value of each content by a single numerical score. Both stages play roughly equally important roles in real industrial systems; however, how to optimize the model weights for the second stage still lacks systematic study. This paper focuses on optimizing the second stage through auto-tuning technology. Although general auto-tuning systems and solutions - both from established production practices and open-source solutions - can address this problem, they typically require weeks or even months to identify a feasible solution. Such prolonged tuning processes are unacceptable in production environments for recommendation systems, as suboptimal value models can severely degrade user experience. An effective auto-tuning solution is required to identify a viable model within 2-3 days, rather than the extended timelines typically associated with existing approaches. In this paper, we introduce a practical auto-tuning system named HyperZero that addresses these time constraints while effectively solving the unique challenges inherent in modern recommendation systems. Moreover, this framework has the potential to be expanded to broader tuning tasks within recommendation systems. Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada DOI: 10.1145/3690624.3709409 ISBN: 979-8-4007-1245-6/25/08 CCS: Information systems Recommender systems † † cc-license: by 1. Introduction Recommendation systems are widely deployed across various scenarios to facilitate user access to pertinent content ( 4 , 9 , 10 ) . In this context, numerous configuration hyperparameters play critical roles in achieving optimal performance and enhancing the overall user experience. To clarify this concept, we first provide a simplified view of how recommendation systems determine the order in which the content should be shown to users. The system usually consists of two components: 1) the ranking stage, and 2) the value model (VM) stage. In the ranking phase, a collection of models predicts the probabilities of different interactions between the user and the contents, such as p(\mathrm{click}) , p(\mathrm{share}) , and p(\mathrm{follow}) . In the value model phase, these predictions are aggregated to determine the holistic value of each content to the user, which is then used to finally sort and present the best contents: \displaystyle\mathrm{score}=\theta_{0}\ast p(\mathrm{click})+\theta_{1}\ast p(\mathrm{share})+\theta_{2}\ast p(\mathrm{follow})+\cdots, where \theta_{0} , \theta_{1} , and \theta_{2} are among the most common examples of hyperparameters in a recommendation system. To optimize these hyperparameters, the team needs to balance multiple aggregated online objectives such as total view counts, peak capacity cost, and user daily average usage time. These metrics often require days to calculate and are subject to considerable delay and noise. Traditional tuning approaches, even those refined by engineering teams over the years, typically rely on daily system feedback and require 2-3 weeks to find a reasonable balance. Figure 1 . Hyperparameter tuning in common scenarios and recommendation systems respectively. In practice, however, these configuration parameters often need to be swiftly adapted to evolving business requirements, demanding high efficiency in tuning. For example, when a new interface is launched, there may be a need on the recommendation system to simultaneously promote longer videos to account for regionally nuanced user preferences while adhering to other business constraints. This necessitates an efficient solution for tuning configuration parameters that optimizes for multiple objectives under strict constraints within a short time frame, raising the following question: How do we reduce the tuning time from weeks to days? To formalize this hyperparameter tuning problem in the context of recommendation systems, we first introduce a conceptual system that utilizes the tuning process, as illustrated on the left of Figure 1 . This conceptual system consists of two main phases: the estimator module, which estimates the objectives based on user feedback on the current hyperparameter, and the optimizer module, which utilizes the estimation from the estimator module and updates the hyperparameter. To address the efficiency concerns raised above, it is important that the tuning system leverages hourly system feedback. However, the conceptual workflow described faces substantial implementation challenges in practice. In particular, we illustrate the real-world system environment for hyperparameter tuning on the right of Figure 1 , and discuss the emerging challenges as follows. (1) Estimator : Non-i.i.d. hourly feedback. Hourly system feedback data received by the prediction module exhibits high variance and is usually correlated over time, violating the independently and identically distributed (i.i.d.) condition required by most algorithms, which leads to a lack of performance guarantee. (2) Optimizer : Multi-objective optimization. Hyperparameter tuning typically involves multiple objectives on system metrics. However, each metric is a random variable with an unknown distribution, making direct estimation impossible. Furthermore, the gradients or Hessians of these objectives are not available, making (stochastic) first- or second-order optimization algorithms inapplicable. Existing zeroth-order optimization methods, which require only function value evaluations, are also not directly applicable due to the presence of generic constraints and non-i.i.d. hourly data. (3) Delayed environment for system feedback. There exist inherent system latencies associated with: 1) the hyperparameter change taking effect in the recommendation system, and 2) collecting user and system feedback after the hyperparameter change. These delays can span several hours and potentially offset the efficiency gains achieved by utilizing hourly data in the tuning process. In this paper, we contribute to addressing the efficiency and capability challenges arising in hyperparameter tuning for recommendation systems. We overcome all aforementioned technical obstacles and propose an end-to-end system for Hyper parameter auto-tuning via Zero th-order optimization, namely HyperZero . 1.1. Contributions Our main contributions are summarized as follows. • End-to-end auto-tuning with hourly feedback. We develop an end-to-end hyperparameter auto-tuning system for recommendation systems. Our proposed HyperZero framework is capable of optimizing multiple generic objectives and constraints and leverages hourly system feedback to significantly reduce the iteration cycle from weeks to days. • Semi-i.i.d. hourly signal. To address the challenge of the non-i.i.d. hourly feedback, we propose a semi-i.i.d. (independently distributed) signal that tracks the metric delta between the test and base hyperparameters. This normalized signal is independently distributed on an hourly basis and characterizes the performance gain over the base hyperparameters. We further derive the mean and variance estimates for the delta signal using a high-order Taylor series approximation. • Efficient zeroth-order constrained optimization. HyperZero accommodates multiple tuning objectives by solving a constrained optimization problem with generic objectives and constraints. We develop an efficient zeroth-order optimization framework that combines a Gaussian process estimator (GP) with Thompson sampling (TS) for arbitrary problem forms. The GP facilitates continuous estimation of random hourly signals with unknown distributions, while new candidates are proposed by solving the constrained problem using TS based on GP estimations. Additionally, we design

Read the complete text