TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning

Dilina Rajapakse, Juan C. Rosero, Ivana Dusparic · 2026

Read TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning by Dilina Rajapakse, Juan C. Rosero, Ivana Dusparic (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21988.txt.

Introduction

Rajapakse and co-authors explain the choices of agents trained for several goals at once. In multi-objective learning, trade-offs are the real decision, yet standard explainers assume one reward. Their TREX framework runs the expert policy under many user preferences. It clusters the trajectories into meaningful behavior segments over time. To score each segment, it retrains policies with that cluster removed and measures how rewards and actions shift. Tests on 3 MuJoCo tasks show it can isolate and price behavior patterns.

Catalog record

CREATORDilina Rajapakse, Juan C. Rosero, Ivana Dusparic
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38399
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21988

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning Dilina Rajapakse, Juan C. Rosero, Ivana Dusparic Abstract Reinforcement Learning (RL) has demonstrated its ability to solve complex decision-making problems in a variety of domains, by optimizing reward signals obtained through interaction with an environment. However, many real-world scenarios involve multiple, potentially conflicting objectives that cannot be easily represented by a single scalar reward. Multi-Objective Reinforcement Learning (MORL) addresses this limitation by enabling agents to optimize several objectives simultaneously, explicitly reasoning about trade-offs between them. However, the “black box" nature of the RL models makes the decision process behind chosen objective trade-offs unclear. Current Explainable Reinforcement Learning (XRL) methods are typically designed for single scalar rewards and do not account for explanations with respect to distinct objectives or user preferences. To address this gap, in this paper we propose TREX , a TR ajectory based EX plainability framework to explain Multi-objective Reinforcement Learning policies, based on trajectory attribution. TREX generates trajectories directly from the learned expert policy, across different user preferences and clusters them into semantically meaningful temporal segments. We quantify the influence of these behavioural segments on the Pareto trade-off by training complementary policies that exclude specific clusters, measuring the resulting relative deviation on the observed rewards and actions compared to the original expert policy. Experiments on multi-objective MuJoCo environments - HalfCheetah, Ant and Swimmer, demonstrate the framework’s ability to isolate and quantify the specific behavioural patterns. 1 Introduction As the application of Reinforcement Learning for decision-making tasks increases, Explainable AI (XAI) techniques have been extended to Explainable Reinforcement Learning (XRL) with the aim of interpreting agent behavior. Before deploying RL agents in real-world systems, it is crucial to understand the underlying logic behind their decisions to ensure safety and trustworthiness. For example, if an autonomous vehicle engages in sudden braking, it is vital to know if this decision was driven by an external factor, such as an animal crossing, or a sensor failure. Most complex real-world decision-making domains involve multiple, often conflicting, objectives, requiring the use of Multi-Objective Reinforcement Learning (MORL). In the case of multiple objectives, optimality is defined not by a single policy, but by a set of Pareto-optimal solutions over varying user-preferences. In a typical multi-objective scenario, the RL agent is provided with a user-preference vector that tells the agent how it is expected to priotize the objectives when making decisions. For example, in a task with two objective, a user-preference of (0.5,0.5) denotes both objectives are given the same importance, whereas (0.75, 0.25) tells the agent to favour objective 1 over objective 2 in the given ratio. With MORL policies, explanations must account for learned objective trade-offs and user-specified preferences. Relative to single objective XRL methods, current literature on Explainable Multi-Objective Reinforcement Learning (XMORL) techniques is scarce [ 18 ] . The existing literature primarily focuses on methods such as reward decomposition and policy selection (e.g: [ 9 ] , [ 20 ] ), resulting in a lack of methods capable of explaining the behaviour of MORL policies. In this paper, we introduce TREX , a TR ajectory based EX plainability framework, an explainability framework specifically developed for multi-objective reinforcement learning tasks, and demonstrate its applicability in standard MORL environments from the MO-gymnasium library 1 1 1 https://mo-gymnasium.farama.org/ . TREX aims to discover influential behaviours learned by an expert MORL policy using trajectory analysis techniques. It generates trajectories from the expert agent across different user-preferences, with the notion that these trajectories reflect the behavioural patterns learned by the model. For each user-preference we decompose the trajectories into temporal segments and cluster them, essentially grouping semantically similar behaviour patterns in the trajectories. We are then able to measure how these behaviours contribute towards each objective and shape the trade-off at a given user-preference. For example, in the multi-objective HalfCheetah, a MuJoCo environment from MO-gymnasium where an agent has to optimize for speed vs energy of the robot, certain behaviours drive conflicting outcomes - ‘long leaps’ maximize the robot’s speed at the cost of increased energy consumption, while ‘short strides’ conserve energy by sacrificing speed. Consequently, when a user prioritizes energy, speed centric behaviours such as ‘leaping’ negatively impact the preferred trade-off. Our framework moves beyond just qualitative observations to provide quantitative analyses of these behavioural patterns. The analyses are based on training policies that exclude specific clusters and we measure the relative deviations in the objective returns and objective trade-off, compared to an expert policy. These metrics serve as a proxy for influence, allowing us to pinpoint exactly which behaviours drive objective fulfilment and determining the magnitude of their impact on the Pareto trade-off. Overall, the contributions of this paper are as follows: • We propose TREX , a post-hoc explainability framework for multi-objective RL, through the behaviours learned by the agent. • We show how TREX can help quantitatively identify influential behavioural patterns learned by the policy. We then show how these behaviours contribute towards each objective and balance the trade-offs to accommodate a given user-preferences. 2 Related Work In this section we provide a brief overview of the relevant background and related work to contextualize our contribution. We first introduce reinforcement learning as a machine learning paradigm (section 2.1 ) and its multi-objective counterpart (section 2.2 ), followed by an overview of existing approaches for explainability in reinforcement learning (section 2.3 ), with special focus on the trajectory based approaches (section 2.4 ). Finally, we introduce the gap in explainability on Multi-Objective Reinforcement Learning approaches (section 2.5 ). 2.1 Reinforcement Learning Reinforcement Learning (RL) is a machine learning framework in which an agent learns through trial-and-error interactions with its environment, where the Agent iteratively selects an action and receives feedback in the form of rewards or penalties [ 19 ] . By iteratively interacting with the environment, the agent can autonomously acquire policies that achieve optimal or near-optimal behavior without requiring explicit supervision. This learning paradigm achieved contributions in a wide range of domains, including video games [ 4 ] , traffic signal control [ 10 ] , computer networks [ 13 ] , robotics [ 7 ] , and healthcare [ 21 ] , to mention a few. 2.2 Multi-Objective Reinforcement Learning Multi-Objective Reinforcement Learning (MORL) extends standard RL to settings in which agents must simultaneously optimize multiple, potentially conflicting objectives. A common distinction in MORL methods is based on the number of policies learned. In Single-policy approaches the aim is to learn one policy that balances all objectives, most often by reducing the multi-objective problem to a scalar one by the use of methods like reward scalarization or utility functions. Representative examples include weighted-sum methods [ 1 ] and modified Q-learning architectures [ 11 ] . In contrast, multi-policy approaches seek to learn a set of policies corresponding to different trade-offs between objectives. These methods aim to explore the Pareto front and allow policies to be selected or adapted based on user preferences or contextual requirements. Examples include policy adaptation methods [ 24 ] , that learn and generalize across multiple preference settings, as well as preference-conditioned policies such as Pareto-conditioned networks [ 17 ] . See [ 8 ] for a comprehensive survey on MORL. 2.3 Explainable Reinforcement Learning A commonly used taxonomy

Read the complete text