Enabling Pareto-Stationarity Exploration in Multi-Objective Reinforcement Learning: A Multi-Objective Weighted-Chebyshev Actor-Critic Approach

Fnu Hairi, Jiao Yang, Tianchen Zhou, Haibo Yang, Chaosheng Dong, Fan Yang, Michinari Momma, Yan Gao, Jia Liu · 2025

Read Enabling Pareto-Stationarity Exploration in Multi-Objective Reinforcement Learning: A Multi-Objective Weighted-Chebyshev Actor-Critic Approach by Fnu Hairi, Jiao Yang, Tianchen Zhou, Haibo Yang, Chaosheng Dong, Fan Yang, Michinari Momma, Yan Gao, Jia Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21397.txt.

Introduction

Hairi and eight co-authors present a learning algorithm for tasks with several competing goals, in a 2025 paper. Reinforcement learning agents often must balance rewards that pull in opposite directions, and good compromises form a set of Pareto-stationary points. Exploring that set in a systematic way, with proof of how much data it takes, was an open gap. The authors build an actor-critic method around a weighted-Chebyshev way of blending the goals.

Their algorithm, called MOCHA, is shown to reach good compromise points within a bounded number of samples. The bound reveals a notable dependence on the smallest weight given to any goal. The work gives multi-objective learning a firmer theory base, which matters for real systems that juggle cost, safety, and quality at once.

Catalog record

CREATORFnu Hairi, Jiao Yang, Tianchen Zhou, Haibo Yang, Chaosheng Dong, Fan Yang, Michinari Momma, Yan Gao, Jia Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21397

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enabling Pareto-Stationarity Exploration in Multi-Objective Reinforcement Learning: A Multi-Objective Weighted-Chebyshev Actor-Critic Approach Fnu Hairi, Jiao Yang, Tianchen Zhou, Haibo Yang, Chaosheng Dong, Fan Yang, Michinari Momma, Yan Gao, Jia Liu Abstract In many multi-objective reinforcement learning (MORL) applications, being able to systematically explore the Pareto-stationary solutions under multiple non-convex reward objectives with theoretical finite-time sample complexity guarantee is an important and yet under-explored problem. This motivates us to take the first step and fill the important gap in MORL. Specifically, in this paper, we propose a M ulti- O bjective weighted- CH ebyshev A ctor-critic ( 𝖬𝖮𝖢𝖧𝖠 \mathsf{MOCHA} ) algorithm for MORL, which judiciously integrates the weighted-Chebychev (WC) and actor-critic framework to enable Pareto-stationarity exploration systematically with finite-time sample complexity guarantee. Sample complexity result of 𝖬𝖮𝖢𝖧𝖠 \mathsf{MOCHA} algorithm reveals an interesting dependency on p min p_{\min} in finding an ϵ \epsilon -Pareto-stationary solution, where p min p_{\min} denotes the minimum entry of a given weight vector 𝒑 \bm{p} in WC-scalarization. By carefully choosing learning rates, the sample complexity for each exploration can be 𝒪 ~ ​ ( ϵ − 2 ) \tilde{\mathcal{O}}(\epsilon^{-2}) . Furthermore, simulation studies on a large KuaiRand offline dataset, show that the performance of 𝖬𝖮𝖢𝖧𝖠 \mathsf{MOCHA} algorithm significantly outperforms other baseline MORL approaches. I Introduction Multi-objective systems [ 1 ] have gained significant attention due to their applicabilities in many real-world applications. For example, on commercial platforms like Booking.com, in addition to the overall ratings for satisfaction, hotels receive various customer ratings for subcategories such as value-for-money , comfort , and cleanliness . These ratings potentially provide a more nuanced recommendation strategy than the traditional overall rating-based recommendation system. From the perspective of a decision-maker (whether a client or a recommender system), the goal is to develop decision-making strategies that maximize all these ratings to deliver an ideal service experience. Despite these multiple ratings seemingly providing more insights about the hotels, they sometimes can conflict. For instance, hotels with high cleanliness ratings often cost more and, leading to low value-for-money ratings. Consequently, Pareto optimality is more suitable solution concepts in this case, where overall balanced solutions can be provided for a further decision-making. Another example is short-term video recommender system on video platforms. In general, the recommender system aim to engage users more on the platform by maximizing the pleasant experience while minimizing the negative ones. Specifically, on Kuaishou [ 2 ] platform, it considers multiple objectives such as "WatchTime", "Likes", "Forward", "Comments", "Dislikes" to optimize. Reinforcement learning (RL) [ 3 ] provides a learning framework where agents learn policies through experience by trial and error. Although the definition for RL is over scalar rewards, it can be naturally extended to vectorized reward settings [ 4 ] , known as multi-objective reinforcement learning (MORL). The key differences between single objective RL and MORL lie in their objective goals and solution concepts. In single-objective RL, the goal is to learn optimal policies that maximizes the long-term accumulated rewards as follows \displaystyle\max_{\pi}\mathbb{E}[\gamma r_{1}+\gamma^{2}r_{2}+\cdots+\gamma^{t}r_{t}+\cdots] where \pi denotes a candidate policy in the policy space, \gamma the discount factor and the expectation is subject to usual caveats about appropriate distribution. Similarly, in MORL, the goal extends to finding a policy that maximizes the following vectorized objective \displaystyle\max_{\pi}\mathbb{E}[\gamma\mathbf{r}_{1}+\gamma^{2}\mathbf{r}_{2}+\cdots+\gamma^{t}\mathbf{r}_{t}+\cdots] where \mathbf{r}_{t} denotes the vectorized reward. A policy \pi is Pareto optimal if it is not dominated by any other policy \pi^{\prime} . However, in general, there are more than one Pareto optimal solutions for multi-objective problems. Pareto front (PF) is a set that includes all Pareto optimal solutions. One of the main research endeavors is to characterize PF systematically. Generally speaking, when the objective functions are non-convex or the PF is non-connected, it is shown challenging to characterize PF [ 5 , 6 , 7 , 8 ] . In this paper, we consider the concept of Pareto stationarity front (PSF), which consists of all Pareto stationary solutions (see the definitions later). By definition, PSF is a superset of PF. The goal of the paper is to develop an algorithm that characterizes the set of PSF in MORL problems. In this paper, we propose a M ulti- O bjective weighted- CH ebyshev A ctor-critic ( \mathsf{MOCHA} ) method by drawing inspirations and insights from the MORL and multi-objective optimization (MOO) literature. More specifically, to enable systematic Pareto-stationarity front exploration with low sample complexity in MORL, our proposed \mathsf{MOCHA} method takes advantage of approach of multiple temporal-difference (TD) learning in the critic component and multi-gradient-descent algorithmic (MGDA) techniques in the actor component, originally proposed in [ 4 ] , then judiciously integrates the weighted-Chebyshev (WC). The rationale behind our approach is three-fold: (i) Combining the strengths of value-based and policy-based RL approaches, the actor-critic framework has been shown to offer state-of-the-art performance in RL; (ii) in the MOO literature, it has been shown that an optimal solution under the WC-based scalarization approach (also known as hypervolume scalarization) provably achieves the Pareto front even when the Pareto front is non-convex [ 9 ] ; and (iii) for MOO problems, the MGDA method is an efficient approach for finding a Pareto-stationary solution [ 10 ] . Finally, the above connections leads us to generalize the WC and actor-critic framework to our \mathsf{MOCHA} method for MORL. Main Challenges: However, to show that \mathsf{MOCHA} enjoys systematic Pareto-stationarity exploration with provable low sample complexity remains highly non-trivial due to the fact: In the MOO literature, WC- and MGDA-based techniques are developed with very different goals in mind, facilitating Pareto-front exploration and achieving Pareto-stationarity, respectively. To date, it remains unclear how to combine them to achieve systematic Pareto-stationarity exploration with finite-time convergence and low sample complexity simultaneously even for general MOO problems, not to mention generalizing them to the more specially structured MORL problems and the associated theoretical performance analysis. Indeed, to our knowledge, there is no such result in the literature on integrating WC- and MGDA- techniques for designing MORL policies. II Related Work In this section, we provide an overview of related work in multi-objective reinforcement learning. Without learning framework, multi-objective optimization [ 11 , 10 ] has been studied extensively with various problem settings, solution concepts and corresponding approaches. Notably, the weighted-Chebyshev formulation and multiple-gradient descent algorithm used in this paper can be traced back to their standard adoption in MOO [ 11 ] and [ 10 ] , respectively. MGDA can be viewed as an extension of the standard gradient descent method to MOO, which dynamically performs a linear combination of all objectives’ gradients in each iteration to identify a common descent direction for all objectives. [ 12 ] and [ 13 ] established finite-time convergence of \mathcal{O}(1/T) to Pareto stationarity point for MGDA and Stochastic MGDA respectively. Also, the finite-time convergence rate of MGDA has recently been established under different MOO settings, including convex and non-convex objective functions [ 13 , 14 ] and decentralized data [ 6 ] , etc. [ 15 ] proposed a weight/direction vector oriented stochastic gradient descent algorithm in MOO. MORL (also referred to as multi-criteria reinforcement learning) dates back to at least [ 16 ] , where a Q-learning based algorithm is proposed for constraint setting. In solving Pareto stationary solution, [ 17 ] proposed an actor-critic where in

Read the complete text