Goal-Conditioned Supervised Learning for Multi-Objective Recommendation

Shijun Li, Hilaf Hasson, Jing Hu, Joydeep Ghosh · 2024

Read Goal-Conditioned Supervised Learning for Multi-Objective Recommendation by Shijun Li, Hilaf Hasson, Jing Hu, Joydeep Ghosh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08911.txt.

Introduction

Shijun Li, Hilaf Hasson, Jing Hu, and Joydeep Ghosh propose a learning method for recommendation systems in this 2024 paper. Online shops must balance several goals at once, such as clicks, sales, and user satisfaction. Training one model for conflicting goals is often complex and costly.

Their method extends goal-conditioned learning, where goals become lists of target values rather than single numbers. The design drops the need for complex model shapes and extra constraints. It also filters out noisy training cases that never reach good long-term rewards. A new goal-picking step chooses targets that are both wanted and reachable. Tests on recommendation tasks showed strong, scalable performance.

Catalog record

CREATORShijun Li, Hilaf Hasson, Jing Hu, Joydeep Ghosh
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08911

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Goal-Conditioned Supervised Learning for Multi-Objective Recommendation Shijun Li, Hilaf Hasson, Jing Hu, Joydeep Ghosh Abstract Multi-objective learning endeavors to concurrently optimize multiple objectives using a single model, aiming to achieve high and balanced performance across diverse objectives. However, this often entails a complex optimization problem to balance the learning of potentially conflicting objectives, leading to solutions with higher memory requirements and computational complexity. This paper introduces a Multi-Objective Goal-Conditioned Supervised Learning (MOGCSL) framework for automatically learning to achieve multiple objectives from offline sequential data. MOGCSL extends the conventional GCSL method to multi-objective scenarios by redefining goals from one-dimensional scalars to multi-dimensional vectors. It benefits from naturally eliminating the need for complex architectures and optimization constraints. Moreover, MOGCSL inherently disentangles uninformative or noisy training instances that fail to achieve desirable long-term rewards across multiple objectives. We also introduce a novel goal-selection algorithm for MOGCSL to model and identify desired and achievable goals for inference. In this paper, we focus on its application to the next action prediction problem in commercial-grade recommender systems. In this context, any viable solution needs to be reasonably scalable and also be robust to large amounts of noisy data that is characteristic of this application space. We show that MOGCSL performs admirably on both counts by extensive experiments. Also, analysis and experiments are included to explain its strength in discounting the noisier portions of training data in recommender systems with multiple objectives. Code is available at: https://github.com/xiwenchao/MOGCSL . 1 Introduction Multi-objective learning techniques typically aim to train a single model to determine a policy for multiple objectives, which are often defined on different types of tasks [ 1 , 2 , 3 ] . For instance, two common tasks in recommender systems are to recommend items that users may click or purchase, and hence the corresponding two objectives are defined as pursuing higher click rate and higher purchase rate. However, learning for multiple objectives simultaneously is often nontrivial, particularly when there are potential inherent conflicts among these objectives [ 2 ] . Existing approaches to multi-objective learning broadly address this optimization issue by formulating and optimizing a loss function that takes into account multiple objectives in a supervised learning paradigm. Some research focused on designing model architectures specifically for multi-objective scenarios, such as the shared-bottom structure [ 4 ] , the experts and task gating networks [ 4 ] , and so on. Another line of research studies how to constrain the optimization process based on various assumptions regarding how to assign reasonable loss weights [ 5 ] or dynamically adjust gradients [ 6 ] . We remark that these approaches often introduce substantial space and computational complexity [ 4 , 3 ] . Also, they treat all the data samples uniformly. In recommender systems, sometimes none of the items shown to a user is of interest, rendering their choice uninformative. Also users are often distracted or their interests temporarily change. These are some of the reasons that the observed interaction data in real settings suffers from substantial uninformative or noisy components that are better left discounted. Existing approaches do not cater well to this aspect of our focus application. To address these issues, we propose a novel method called Multi-Objective Goal-Conditioned Supervised Learning (MOGCSL) . In this framework, we first apply goal-conditioned supervised learning (GCSL) [ 7 ] to the multi-objective recommendation problem by introducing a multi-dimensional goal. At its core, MOGCSL aims to learn primarily from the behaviors of those sessions where the long-term reward ends up being high, thereby discounting noisy user choices coming with low long-term rewards. Unlike conventional GCSL, however, we represent the reward gained from the environment with a vector instead of a scalar. Each dimension of this vector indicates the reward for a certain objective. The “goal” in MOGCSL can then be defined as a vector of desirable cumulative rewards on all of the objectives given the current state. By incorporating these goals as input, MOGCSL learns to rely primarily on high-fidelity portions of the data with less noise corresponding to multiple objectives, which helps to better predict users’ real preference. Extensive experiments indicate that MOGCSL significantly outperforms other baselines and benefits from lower complexity. For inference of GCSL, most previous works employ a simple goal-choosing strategy for scalar goals [ 8 , 9 ] (e.g., some multiple of the average goal in training). To study how to choose reasonable and desirable goals within multi-objective scenarios, we introduce a goal-choosing algorithm that estimates the distribution of achievable goals using variational auto-encoders, and automates the selection of desirable high goals on multiple objectives as input for inference. This algorithm addresses the key challenge of selecting achievable “high” multi-dimensional goals, a problem that has remained unexplored yet. By comparing the proposed method with simpler statistical strategies, we gain valuable insights into the goal-choosing process and trade-offs for practical implementation. Our key contributions can be summarized as follows: • We introduce a supervised framework called MOGCSL for multi-objective recommendations that, by design, selectively leverages training data with desirable long-term rewards on multiple objectives for inference. We implement this approach using a transformer encoder optimized with a standard cross-entropy loss, avoiding more complex architectures or optimization constraints that are typical for multi-objective learning. Empirical experiments on real-world e-commerce datasets demonstrate the superior efficacy and efficiency of MOGCSL. • As a part of MOGCSL, we conduct a formal analysis of goal properties for inference with a theorem. Then we introduce a goal-choosing algorithm that can model the distribution of achievable goals over interaction sequences and choose desirable “high” goals across multiple objectives. • We conduct a comprehensive analysis of MOGCSL’s working mechanism and are the first to reveal its ability to effectively mitigate harmful effects of noisy instances in the training data, which is a crucial challenge in practical applications like recommender systems. 2 Related Work Multi-Objective Learning. Multi-objective learning typically investigates the construction and optimization of models that can simultaneously achieve multiple objectives. Existing research focuses mainly on resolving the problem by model architecture designs [ 4 , 10 ] and optimization constraints [ 5 , 11 ] . All of these works give equal weight to all instances in the training data that they learn from, instead of forcefully distinguishing noisy data from non-noisy data by considering their different effects on multiple objectives. This is fine in many applications but problematic in commercial recommendation systems. Moreover, these approaches often introduce substantial space and computational complexity [ 3 ] , making them more challenging for large-scale applications in the real world. For example, MMOE [ 4 ] requires constructing separate towers for each objective. DWA [ 5 ] necessitates recording and calculating loss change dynamics for each training epoch, while PE [ 11 ] demands substantial computational resources to solve an optimization problem for Pareto efficiency. A recent work FAMO [ 12 ] aims to improve the efficiency of multi-objective learning, but still suffers from the complexity of calculating gradient similarities to determine task weights. Goal-Conditioned Supervised Learning. In contrast, we propose resolving the multi-objective optimization dilemma within the framework of GCSL [ 7 , 8 , 13 ] . Typically, GCSL can be directly combined with various sequential models with minor adaptations and

Read the complete text