PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning
Tianmeng Hu, Biao Luo · 2026
Read PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning by Tianmeng Hu, Biao Luo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19579.txt.
Introduction
Hu and Luo present PA2D-MORL, a method for reinforcement learning with several conflicting objectives. It aims to approximate the full Pareto policy set well. Pareto ascent directions guide the choice of scalarization weights. A multi-objective gradient then improves all objectives jointly. Many policies are tuned under an evolutionary frame, each from a different direction. Adaptive fine-tuning adds density and spread along the Pareto front. On robot control tasks, it beats the prior best method in quality and stability.
Catalog record
| CREATOR | Tianmeng Hu, Biao Luo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37166 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19579
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PA2D-MORL: Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning Tianmeng Hu, Biao Luo Abstract Multi-objective reinforcement learning (MORL) provides an effective solution for decision-making problems involving conflicting objectives. However, achieving high-quality approximations to the Pareto policy set remains challenging, especially in complex tasks with continuous or high-dimensional state-action space. In this paper, we propose the Pareto Ascent Directional Decomposition based Multi-Objective Reinforcement Learning (PA2D-MORL) method, which constructs an efficient scheme for multi-objective problem decomposition and policy improvement, leading to a superior approximation of Pareto policy set. The proposed method leverages Pareto ascent direction to select the scalarization weights and computes the multi-objective policy gradient, which determines the policy optimization direction and ensures joint improvement on all objectives. Meanwhile, multiple policies are selectively optimized under an evolutionary framework to approximate the Pareto frontier from different directions. Additionally, a Pareto adaptive fine-tuning approach is applied to enhance the density and spread of the Pareto frontier approximation. Experiments on various multi-objective robot control tasks show that the proposed method clearly outperforms the current state-of-the-art algorithm in terms of both quality and stability of the outcomes. Introduction Deep Reinforcement Learning (RL) is a promising approach for solving decision-making and planning problems, and it has found widespread applications in various fields such as robot control ( 19 ) , autonomous driving ( 10 ) , gaming ( 20 ) , and recommender systems ( 5 ) . In Deep RL, an agent’s behavior is guided by a reward function that defines the optimization objective. However, decision-making problems can involve multiple conflicting objectives, which is an important challenge for real-world applications of RL ( 8 ) . For instance, an autonomous driving task may have two objectives: vehicle speed and passenger comfort, while a robot control task may involve forward speed and energy efficiency. In such multi-objective decision-making problems, a single optimal policy does not exist because conflicting objectives cannot be simultaneously optimized. Instead, there exists a set of Pareto-optimal policies. Adapting to requirements and application scenarios, human users must select a trade-off among multiple objectives that reflects their objective preferences, and different preferences align with different Pareto-optimal policies. In this context, multi-objective reinforcement learning has attracted increasing attention and many advanced methods have been proposed. Among them, the multi-policy MORL ( 3 ; 25 ; 24 ; 11 ) is dedicated to searching a non-dominated policy set consisting of a series of policies aligned to different objective preferences. Compared with the single-policy approach ( 2 ; 21 ; 14 ) , the multi-policy approach has great advantages in terms of applicability and flexibility, despite being more complicated. On the one hand, by visualizing the non-dominated policy set, users are free to select satisfactory policies to suit various demands. On the other hand, the multi-policy MORL can also be combined with methods that focus on real-world applications, such as safe RL ( 12 ) , since the policies that satisfy the safety constraints can be easily identified from the non-dominated policy set. Recently, deep policy gradient methods and the actor-critic architecture have also been introduced into multi-objective reinforcement learning to address complex tasks with continuous or high-dimensional state-action spaces. The Pareto policy adaptation (PPA) ( 14 ) method designs a loss function based on a multi-objective gradient and attempts to search for a single solution in a given preference distribution. It is easy to implement, but the policy needs to be fine-tuned or retrained when preferences change. In contrast, the Prediction-Guided MORL (PGMORL) ( 24 ) method employs an intuitive prediction model and an evolutionary architecture to address multi-objective robot control problems and achieve high-quality sets of non-dominated policies. However, the limitations of the prediction model can negatively affect the performance of policies and the stability of the results. Therefore, instead of introducing predictive models, we explore completely different solutions. In this paper, we focus on multi-policy MORL and propose a novel policy gradient-based approach to obtain high-quality approximations of the Pareto policy set, outperforming current state-of-the-art approaches. The proposed method leverages the Pareto ascent direction to decompose a multi-objective decision-making problem into a series of single-objective decision-making problems, and achieves parallel improvement of multiple policies with different directions in an evolutionary framework. A non-dominated policy set is maintained during training and gradually approximates the Pareto frontier. The main contributions of this paper can be summarized as follows: • We propose a novel multi-policy MORL method for Pareto policy set approximation via Pareto Ascent Directional Decomposition (PA2D-MORL). For a non-Pareto optimal policy, the method finds a decomposition of the original multi-objective problem by solving the Pareto ascent direction that improves all objectives simultaneously and avoids introducing any prior objective preference. In this way, an optimization direction that depends on the policy parameters can be determined automatically, avoiding the use of human-designed prediction models. • We propose a partitioned greedy randomized approach to select policies for updating under an evolutionary framework, which helps the policies to move towards a higher-performance and wider objective space. This encourages effective exploration and exploitation, and avoids getting stuck in the same local optimum for long term. • We propose a Pareto adaptive fine-tuning method, which selects several policies to be fine-tuned according to the current distribution of non-dominated policies in the objective space, thus improving the quality of the Pareto set approximation. • We evaluate the proposed method in seven MuJoCo environments that are modified for multi-objective tasks, and achieve state-of-the-art performance in terms of both the quality of the Pareto set approximation and the stability of the results. Related Work As reinforcement learning is being used to solve complex real-world decision-making problems, many of which have multiple conflicting objectives, MORL is an effective way to solve such problems. Current MORL methods can be broadly divided into two categories: single-policy and multi-policy methods. Some relevant work is presented below. Single-Policy Methods Single-policy methods transform multi-objective problems into single-objective problems. A typical approach is to use specific weights to scalarize the multi-objective rewards into a scalar reward, and then run the reinforcement learning algorithm to find an optimized policy. For example, Duan et al. artificially designed a synthetic multi-objective reward function and applied policy gradient DRL methods to several continuous control problems ( 6 ) . Chen et al. applied MORL to hyper-parameter optimization problems, designing a scalarization function combining accuracy and latency as a synthetic reward to guide policy updating ( 2 ) . Furthermore, the lexicographic-MORL ( 21 ) optimizes the rewards lexicographically according to the importance and constraints of different objectives. The Pareto policy adaptation ( 14 ) method uses a gradient solver to calculate the policy optimization direction and modify the loss function. This method can learn a single optimized policy for a given preference distribution. Typically, single-policy MORL employ some prior information to find a single solution that meets the requirements. However, these methods have several problems: 1) requirements and application scenarios may change, making the policy need to be fine-tuned or re-trained; 2) user preferences may be abstract and difficult to quantify accurately. An effective way to solve above problems is to find a coverage set of optimized policies, allowing users to select policies based on their preferences and demands. Multi-Policy Methods Multi-policy MORL learns a