Post Hoc Extraction of Pareto Fronts for Continuous Control

Raghav Thakar, Gaurav Dixit, Kagan Tumer · 2026

Read Post Hoc Extraction of Pareto Fronts for Continuous Control by Raghav Thakar, Gaurav Dixit, Kagan Tumer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02628.txt.

Introduction

Raghav Thakar, Gaurav Dixit, and Kagan Tumer extract trade-off fronts from pretrained controllers in this 2026 paper. Real agents must balance goals like speed, stability, and energy use. A Pareto front collects the policies that trade those goals optimally. Standard multi-objective learning builds fronts from scratch, ignoring existing specialists.

Preferences often emerge only after a policy was trained on one goal. Their offline method, MAPEX, reuses specialist policies, critics, and replay buffers. Specialist evaluations merge into a mixed advantage signal for each action. A behavior cloning loss, weighted by that signal, trains balanced new policies. On five simulated control suites, fronts matched established baselines. The sample cost was a tiny fraction of retraining from scratch, they report.

Catalog record

CREATORRaghav Thakar, Gaurav Dixit, Kagan Tumer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37023
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.02628

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Post Hoc Extraction of Pareto Fronts for Continuous Control Raghav Thakar, Gaurav Dixit, Kagan Tumer Abstract Agents in the real world must often balance multiple objectives, such as speed, stability, and energy efficiency in continuous control. To account for changing conditions and preferences, an agent must ideally learn a Pareto frontier of policies representing multiple optimal trade-offs. Recent advances in multi-policy multi-objective reinforcement learning (MORL) enable learning a Pareto front directly, but require full multi-objective consideration from the start of training. In practice, multi-objective preferences often arise after a policy has already been trained on a single specialised objective. Existing MORL methods cannot leverage these pre-trained ‘specialists’ to learn Pareto fronts and avoid incurring the sample costs of retraining. We introduce Mixed Advantage Pareto Extraction (MAPEX), an offline MORL method that constructs a frontier of policies by reusing pre-trained specialist policies, critics, and replay buffers. MAPEX combines evaluations from specialist critics into a mixed advantage signal, and weights a behaviour cloning loss with it to train new policies that balance multiple objectives. MAPEX’s post hoc Pareto front extraction preserves the simplicity of single-objective off-policy RL, and avoids retrofitting these algorithms into complex MORL frameworks. We formally describe the MAPEX procedure and evaluate MAPEX on five multi-objective MuJoCo environments. Given the same starting policies, MAPEX produces comparable fronts at 0.001 % 0.001\% the sample cost of established baselines. 1 Introduction Many real-world continuous control problems require agents to balance multiple, even conflicting, objectives. In legged locomotion, for example, a robot must simultaneously optimise forward speed, gait stability, and energy efficiency. In such settings, there is no single optimal solution, and agents must instead discover a set of non-dominated policies that capture the range of feasible trade-offs. Learning this Pareto front enables downstream stakeholders to select behaviours that reflect their current preferences or adapt to changing operational conditions. Typically, multiple objectives are handled through scalarisation into a single weighted sum and training via standard reinforcement learning (RL). While accessible and tractable, this approach yields only a single, fixed trade-off. Simply repeating this procedure for every trade-off is inefficient and practically infeasible without sophisticated experience and representation sharing. Multi-Objective RL (MORL), and specifically multi-policy MORL algorithms partially address this through various architectural improvements to learn the entire frontier directly. Methods like MORL/D 8 divide the problem into several single-objective problems through scalarisation, and use single-objective RL with buffer sharing to train policies for each scalarisation. Other works like PG-MORL 28 and MOPDERL 25 maintain populations of policies, combining RL with evolutionary principles to jointly improve the policies in the population and cover the objective space. PSL-MORL 16 , a notable recent work, trains a hypernetwork to produce a fresh policy for every desired trade-off. Despite successfully learning Pareto fronts, these methods suffer from a critical limitation: requiring full multi-objective consideration from the outset of training. This rigidity creates a disconnect with practical scenarios, where multi-objective preferences often arise retroactively, typically only after a robust policy for a primary task has already been trained. For instance, a preference for more stability may only emerge after observing a locomotion policy highly optimised for speed. To obtain new trade-offs in these scenarios, practitioners must 1) discard pre-trained policies and incur the sample costs of re-training, and 2) retrofit their algorithm into complex multi-objective learning frameworks. Currently, no method reuses disjoint specialist policies and training data to recover Pareto fronts efficiently and with minimal added algorithmic complexity. In this work we present Mixed Advantage Pareto Extraction (MAPEX) 1 1 1 Code: https://github.com/raghavthakar/MAPEX , a novel method that fully leverages prior-trained single-objective policies, critics, and replay buffers to produce Pareto fronts of policies. Our key insight is that agents learn optimal trade-offs by intelligently blending expert behaviour on each objective. MAPEX implements this by blending the single-objective evaluation from each specialist critic into a multi-objective mixed advantage value and weighting a behaviour cloning loss with it. MAPEX bypasses the need to retrofit bespoke or standard off-policy RL into complex multi-objective learning frameworks. It preserves both, the simplicity, and intricacies of these algorithms by providing a realistic pathway to learning multi-objective behaviours from single-objective training data. If training from scratch, MAPEX allows allocating the entire sample budget to training high-performing single-objective specialists, from which high-quality Pareto fronts can later be extracted at a minimal sample cost. We provide a detailed view of MAPEX’s Pareto extraction procedure, and perform Pareto extraction from specialists trained using both standard and bespoke off-policy RL. Across five multi-objective MuJoCo environments, MAPEX produces fronts comparable to, or better than established baselines from the literature. When extracting Pareto fronts from the same starting policies, other methods consume up to 1000 \times more samples than MAPEX to produce similar fronts. 2 Background 2.1 Actor–Critic Methods and Offline Reinforcement Learning Actor–critic methods: These methods perform reinforcement learning using an actor , which defines the policy, and a critic , which estimates the expected return of the actor’s behaviour 24 . By relying on learned value estimates rather than Monte Carlo returns alone, actor–critic methods enable efficient and stable policy optimisation, and are commonly used in continuous control. These methods are often implemented in an off-policy setting, where experience collected by one or more behaviour policies is stored in a replay buffer and reused for policy updates 10 ; 9 . Policy improvement is commonly guided by the advantage function, A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s), (1) where Q^{\pi}(s,a) is the critic’s action-value function, and V^{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}[Q^{\pi}(s,a)] . In practice, particularly in deterministic actor–critic methods, the value function is often approximated by Q^{\pi}(s,\pi(s)) , yielding A^{\pi}(s,a)\approx Q^{\pi}(s,a)-Q^{\pi}(s,\pi(s)). (2) Offline Reinforcement Learning: Offline reinforcement learning studies the problem of learning a policy from a fixed dataset of transitions, without any further interaction with the environment during training. A key challenge in this setting is distributional shift: the dataset may contain diverse experiences that are hard to generalise to, leading to unreliable value estimates 18 ; 13 . Advantage-weighted regression (AWR) 19 provides a simple and stable mechanism for policy improvement in this setting by casting reinforcement learning as a weighted supervised learning problem. Given a critic-derived advantage estimate A(s,a) , the AWR policy update solves \pi^{+}=\arg\max_{\pi}\;\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\log\pi(a|s)\;\exp\!\left(\tfrac{1}{\beta}A(s,a)\right)\right], (3) where \beta>0 is a temperature parameter. This objective emphasises actions that are predicted to improve performance while sampled purely from the observed data, making it well suited to off-policy and offline reinforcement learning problems. We use an AWR-inspired regression weighting in MAPEX to learn multi-objective behaviours from static single-objective replay buffers. 2.2 Multi-Objective Sequential Decision-Making Multi-objective sequential decision-making problems are commonly modelled as Multi-Objective Markov Decision Processes (MOMDPs). A MOMDP is defined by the tuple \langle\mathcal{S},\mathcal{A},\mathcal{T},\gamma,\mathbf{R}\rangle , containing the state space \mathcal{S} , the action space \mathcal{A} , the transition dynamics \mathcal{T}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1] , and a scalar discount factor \gamma\in[0,1) . Unlike standard MDPs, the reward function \mathbf{R}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}^{k} returns a vector of k rewards, each corresponding to a distinct objective. A policy \pi:\mathcal{S}\rightarrow\mathcal{A} induces an expected return for each objective. We characterise policy performance directly through its expected long-term returns. Let \mathbf{J}(\pi)\in\mathbb{R}^{k} denote the vector of objective returns achieved by policy \pi , with components J_{i}(\pi)=\mathbb{E}_{s_{0}\sim\rho_{0},\;a_{t}\sim\pi(\cdot\mid s_{t})}\!\left[\sum_{t=0}^{\infty}\gamma^{t}r_{i}(s_{t},a_{t})\right],\;i\in\{1,\dots,k\}. (4) where r_{i} is the reward function associated with the i -th objective and \rho_{0} is the initial state distribution. From a multi-objective

Read the complete text