xMTF: A Formula-Free Model for Reinforcement-Learning-Based Multi-Task Fusion in Recommender Systems

Yang Cao, Changhao Zhang, Xiaoshuang Chen, Kaiqiao Zhan, Ben Wang · 2025

Read xMTF: A Formula-Free Model for Reinforcement-Learning-Based Multi-Task Fusion in Recommender Systems by Yang Cao, Changhao Zhang, Xiaoshuang Chen, Kaiqiao Zhan, Ben Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05669.txt.

Introduction

Yang Cao and co-authors remove the fixed formula from a key recommender step in this 2025 paper. Recommenders predict many feedback types, like clicks and likes, then fuse them into one ranking score. Reinforcement learning can tune that fusion for long-term satisfaction, but older methods only nudge coefficients inside a preset formula. The formula caps what the learner can find.

Their model, xMTF, replaces the formula with a learnable fusion cell built from single-variable monotone pieces. A math theorem guarantees such pieces can express any fit fusion shape. A two-stage training plan teaches the cell in phases. In offline and live online tests, xMTF beat the formula-based methods. The authors say widening the search space is what unlocks the gain.

Catalog record

CREATORYang Cao, Changhao Zhang, Xiaoshuang Chen, Kaiqiao Zhan, Ben Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

xMTF: A Formula-Free Model for Reinforcement-Learning-Based Multi-Task Fusion in Recommender Systems Yang Cao, Changhao Zhang, Xiaoshuang Chen, Kaiqiao Zhan, Ben Wang Abstract . Recommender systems need to optimize various types of user feedback, e.g., clicks, likes, and shares. A typical recommender system handling multiple types of feedback has two components: a multi-task learning (MTL) module, predicting feedback such as click-through rate and like rate; and a multi-task fusion (MTF) module, integrating these predictions into a single score for item ranking. MTF is essential for ensuring user satisfaction, as it directly influences recommendation outcomes. Recently, reinforcement learning (RL) has been applied to MTF tasks to improve long-term user satisfaction. However, existing RL-based MTF methods are formula-based methods, which only adjust limited coefficients within pre-defined formulas. The pre-defined formulas restrict the RL search space and become a bottleneck for MTF. To overcome this, we propose a formula-free MTF framework. We demonstrate that any suitable fusion function can be expressed as a composition of single-variable monotonic functions, as per the Sprecher Representation Theorem. Leveraging this, we introduce a novel learnable monotonic fusion cell (MFC) to replace pre-defined formulas. We call this new MFC-based model eXtreme MTF (xMTF). Furthermore, we employ a two-stage hybrid (TSH) learning strategy to train xMTF effectively. By expanding the MTF search space, xMTF outperforms existing methods in extensive offline and online experiments. Conference: Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Proceedings of the ACM Web Conference 2025 (WWW ’25), April 28-May 2, 2025, Sydney, NSW, Australia DOI: 10.1145/3696410.3714959 ISBN: 979-8-4007-1274-6/25/04 CCS: Information systems Recommender systems 1. Introduction Figure 1. MTL and MTF in recommender systems. Figure 2. Overall structure of xMTF. Recommender systems are playing an increasingly important role in various platforms, e.g. E-commerce ( Gu et al., 2020 ; Linden et al., 2003 ; Zhou et al., 2018 ) , videos ( Tang et al., 2017 ; Wu et al., 2018 ) , news ( Liu et al., 2010 ; Zheng et al., 2018 ) , etc. Practical recommender systems often need to optimize various types of user feedback. For instance, on video platforms, user satisfaction is influenced by watch time and interactions such as likes, follows, and shares ( Zhang et al., 2024 ; Cai et al., 2023a ) . In such scenarios, recommender systems must consider multiple feedback types to deliver the final recommendation. A typical recommender system handling multiple types of feedback has two components: a multi-task learning (MTL) module, predicting feedback such as click-through rate and like rate; and a multi-task fusion (MTF) module, integrating these predictions into a single score for item ranking ( Zhang et al., 2022 ) . Figure 1 illustrates the relationship between MTL and MTF. MTF is essential for ensuring user satisfaction with the recommender systems, as it directly affects recommendation outcomes. However, compared to MTL, which has been extensively studied ( Ma et al., 2018 ; Tang et al., 2020 ; Su et al., 2024 ; Yu et al., 2020 ; Yang et al., 2023 ; Chen et al., 2018 ) , MTF poses more challenges. MTF integrates the predictions of multiple user feedback to generate a single score, reflecting the user’s overall satisfaction. However, users typically do not provide direct feedback indicating overall satisfaction with each item. Instead, overall satisfaction is indicated by long-term feedback such as session length, daily watch time, and retention, which cannot be directly linked to individual recommended items. Recently, there has been growing interest in using reinforcement learning (RL) for MTF ( Changhua et al., 2019 ; Zhang et al., 2022 ; Liu et al., 2024 ; Cai et al., 2023a ; Zhang et al., 2024 ; Chen et al., 2024 ) . RL-based approaches regard users as the environment and the recommender system as the agent, treating fusion weights as RL actions. These approaches have proven effective in MTF by modeling long-term user satisfaction. However, existing RL-based approaches are all formula-based MTF approaches, i.e., to define a fusion formula f(o_{1},\cdots,o_{K};a_{1}, \cdots a_{K}) with K predictions o_{1},\cdots,o_{K} and the coefficients \boldsymbol{a}=[a_{1},\cdots,a_{K}] , regarding the coefficients \boldsymbol{a} as the actions of RL. Table 1 shows typical fusion formulas in recent research. By pre-defining the fusion formula, we only need to optimize a few coefficients via RL, which makes RL learning easier. Nevertheless, there are several issues with pre-defined fusion formulas. First, different formulas result in varying recommendation outcomes, making it challenging to determine the best one, especially since different users might benefit from different formulas. Second, pre-defined formulas restrict the MTF search space, leading to inferior model performance. Table 1. Typical fusion formulas in existing research. Index Formula Literature 1 z_{i}=\sum_{k=1}^{K}a_{k}o_{ki} ( Cai et al., 2023a ; Zhang et al., 2024 ) 2 z_{i}=\sum_{k=1}^{K}a_{k}\log\left(o_{ki}+\beta_{k}\right) ( Zhang et al., 2022 ) ( \beta_{k} is pre-defined) 3 z_{i}=\prod_{k=1}^{K}o_{ki}^{a_{k}} ( Changhua et al., 2019 ) To address this issue, this paper provides a formula-free MTF framework, called eXtreme MTF (xMTF), as shown in Figure 2 . We first show that any suitable fusion function (shown in Figure 2 (a)) can be expressed as a combination of single-variable monotonic functions for each prediction o_{ki} (shown in Figure 2 (b)). Leveraging this, xMTF introduces a novel monotonic fusion cell (MFC) to learn these monotonic transformations of input prediction values, as shown in Figure 2 (c). MFCs have the following important features: • MFCs capture the monotonicity between the output and the input variables, which is an important intrinsic structure of MTF, making the MFCs interpretable. • MFCs are learnable , replacing pre-defined fusion formulas in existing methods, thus expanding the MTF search space. • The learnable MFCs provide personalized fusion functions for different users and predictions. In contrast, formula-based approaches can only personalize a few coefficients in pre-defined formulas. The larger search space complicates xMTF training, and existing RL-based methods are not directly applicable. To overcome this, we provide a two-stage hybrid (TSH) training approach, shown in Figure 2 (d), comprising an RL-based outer stage and a supervised-learning-based inner stage. The outer stage contains fewer parameters as RL actions, learning long-term rewards, while the inner stage, with more parameters, learns the knowledge of the outer stage by supervised learning. TSH enables effective xMTF training, showing significant improvement over existing approaches. In summary, the contributions of this paper are: • We propose xMTF, a formula-free MTF framework, to expand the search space of MTF. To the best of the authors’ knowledge, we are the first to discuss a generalized MTF model rather than formula-based MTF approaches. • We introduce learnable MFCs to replace pre-defined formulas in MTF. MFCs capture the intrinsic monotonicity of MTF and enable personalized fusion functions, leading to superior performance over formula-based approaches. • We propose TSH to train xMTF effectively, addressing the challenge of increasing search space. • Extensive offline and online experiments show the effectiveness of xMTF, and xMTF has been applied to our online system, serving over 100 million users. 2. Related Work 2.1. Multi-Task Learning MTL simultaneously predicts multiple user feedback, e.g. click-through rate and like rate. Multi-gate Mixture-of-Experts (MMoE) ( Ma et al., 2018 ) employs multiple expert networks to capture task-specific patterns while sharing information. Progressive Layered Extraction (PLE) ( Tang et al.,

Read the complete text