GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

Zhiheng Jiang, Yunzhe Wang, Ryan Marr, Ellen Novoseller, Benjamin T. Files, Volkan Ustun · 2026

Read GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning by Zhiheng Jiang, Yunzhe Wang, Ryan Marr, Ellen Novoseller, Benjamin T. Files, Volkan Ustun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20753.txt.

Introduction

Zhiheng Jiang and five co-authors build a test bed for AI that must balance rival goals. Their 2026 paper presents GraphAllocBench, a benchmark for preference-conditioned policy learning. In this setup, one model must serve many users with different trade-offs. The model takes the user's stated preferences as input. It then produces a policy tuned to those weights.

The goal is to cover the Pareto front, the set of best compromises among goals. A single trained model should flex at run time, not need retraining for each taste. The benchmark tests how well policies land on or near that front. It gives researchers a shared yardstick for multi-objective reinforcement learning. Such tools matter wherever one system must serve people who value outcomes in unlike ways.

Catalog record

CREATORZhiheng Jiang, Yunzhe Wang, Ryan Marr, Ellen Novoseller, Benjamin T. Files, Volkan Ustun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20753

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning Zhiheng Jiang, Yunzhe Wang, Ryan Marr, Ellen Novoseller, Benjamin T. Files, Volkan Ustun Abstract Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) aims to approximate diverse Pareto-optimal solutions by conditioning policies on user-specified preferences over objectives. This enables a single model to flexibly adapt to arbitrary trade-offs at run-time by producing a policy on or near the Pareto front. However, existing benchmarks for PCPL are largely restricted to toy tasks and fixed environments, limiting their realism and scalability. To address this gap, we introduce GraphAllocBench, a flexible benchmark built on a novel graph-based resource allocation sandbox environment inspired by city management, which we call CityPlannerEnv. GraphAllocBench provides a rich suite of problems with diverse objective functions, varying preference conditions, and high-dimensional scalability. We also propose two new evaluation metrics – Proportion of Non-Dominated Solutions (PNDS) and Ordering Score (OS) – that directly capture preference consistency while complementing the widely used hypervolume metric. Through experiments with Multi-Layer Perceptrons (MLPs) and graph-aware models, we show that GraphAllocBench exposes the limitations of existing MORL approaches and paves the way for using graph-based methods such as Graph Neural Networks (GNNs) in complex, high-dimensional combinatorial allocation tasks. Beyond its predefined problem set, GraphAllocBench enables users to flexibly vary objectives, preferences, and allocation rules, establishing it as a versatile and extensible benchmark for advancing PCPL. Code: https://github.com/jzh001/GraphAllocBench 1 Introduction Reinforcement learning (RL) has achieved remarkable progress in domains such as video games 31 , robotics 18 ; 24 ; 42 , and large language model fine-tuning 26 . Much of this success is built on single-objective RL algorithms such as Proximal Policy Optimization (PPO) 30 , which optimize a single scalar reward function. However, many real-world and simulated environments involve multiple, often conflicting objectives rather than a single scalar reward. For example, in games and training simulations, Non-Player Characters (NPCs) must balance dealing damage, avoiding damage, and maximizing survival time 28 ; 29 to create engaging and adaptive experiences. In such cases, objectives carry different levels of importance; we term such weights preferences over objectives. A common approach to multi-objective RL (MORL) is scalarization 37 (See Appendix C.3 ), where objectives are collapsed into a single reward using preference weights. While straightforward, using a predetermined scalarization ties the learned policy to one fixed preference and requires retraining for new ones. Other existing approaches to MORL (See Section 2.2 ) include training populations of policies via evolutionary algorithms 7 or training multiple single-objective models to approximate the Pareto front 40 . These methods are generally inefficient, as each new preference requires retraining from scratch. Gradient-based optimization with vector-valued rewards 9 has also been explored, but often suffers from convergence issues 22 . Preference-conditioned Policy Learning (PCPL) in MORL, for instance PD-MORL 3 , offers a more scalable solution. Instead of training a separate model for each preference, a single preference-conditioned policy can adapt to arbitrary user-specified preferences. Compared to alternatives such as role embeddings 23 , policy pools 12 ; 43 , or mixture-of-experts models 25 , preference-conditioned methods provide a more direct and flexible mechanism for handling diverse objectives. Despite algorithmic progress in the field, current multi-objective benchmarks 10 are insufficient for pushing progress forward for PCPL. Most existing testbeds utilize either continuous optimization problems 8 ; 15 or simple 2D grid-based games 41 ; 36 . These lack flexibility in scaling observations, actions, objectives and dependency structures, making them insufficient to evaluate the robustness of preference-conditioned methods in more complex problems. For instance, in Deep Sea Treasure 36 , the objectives are to maximize treasure and minimize time, as limited by the game mechanics. To address this gap, we introduce CityPlannerEnv , a Gymnasium-based 35 sandbox environment inspired by city-scale resource allocation that enables multi-objective planning in graph-based environments of arbitrary and scalable complexity. Cities must allocate limited resources across diverse and often conflicting demands – such as reducing congestion, fostering economic growth, and promoting sustainability – according to shifting preferences. The environment models this process as a sequential allocation problem, represented through bipartite resource-demand dependency graphs. Agents must not only balance competing objectives but also anticipate future events and adapt to varying risk profiles (e.g., conserving resources for emergencies versus aggressively pursuing short-term gains). In CityPlannerEnv , agents allocate resources add or remove productions for each demand at each step to pursue custom-defined objectives based on user preferences. This incremental allocation mechanism draws inspiration from the Multi-step Colonel Blotto Game (MCBG), a well-studied model for competitive resource allocation on graphs 2 ; 1 ; 33 . Dependencies between resources and demands are represented as bipartite graphs, and objectives can be defined over any subset of productions. Agents seek to plan actions that maximize the expected cumulative reward over time while respecting preference trade-offs. As the number of resources, demands, and objectives increases – and as conditions change over time – the allocation problem becomes increasingly challenging for PCPL. We further introduce GraphAllocBench , a benchmark for PCPL that defines a suite of challenging problems designed using CityPlannerEnv. GraphAllocBench leverages the flexibility of CityPlannerEnv to create customizable problems by varying the numbers of demands, resources, objectives, and objective shapes, providing targeted stress tests for PCPL algorithms. For example, GraphAllocBench includes test problems spanning diverse optimization challenges, including difficult objective functions, diverse and non-convex Pareto fronts, sparse observation spaces, complex dependency structures, and high-dimensional graph observations. For performance characterization, the benchmark includes the hypervolume ratio along with two novel supplementary metrics – Proportion of Non-Dominated Solutions (PNDS) and Ordering Score (OS) – to assess how well policies satisfy varied preferences. To establish baseline performance and highlight how the benchmark challenges existing methods, we combine PPO 30 with Multi-Layer Perceptron (MLP) and Heterogeneous Graph Neural Network (HGNN) 14 feature extractors. We further employ Smooth Tchebycheff Scalarization 20 to approximate challenging non-convex Pareto fronts and demonstrate how graph neural networks can learn preference-conditioned, graph-aware policies in environments with complex dependency structures, going beyond prior work that primarily focused on 2D grids 3 or simpler graph topologies 19 ; 11 . Our contributions are as follows: • GraphAllocBench benchmark: We introduce GraphAllocBench , a flexible and challenging graph-based benchmark for preference-conditioned policy learning, based on our novel, customizable sandbox environment CityPlannerEnv , designed to capture complex resource allocation scenarios with complex bipartite dependency structures, discrete non-convex Pareto Fronts, and difficult objective functions. • Novel evaluation metrics: Beyond the standard hypervolume metric, we introduce the ordering score (OS) and the proportion of non-dominated solutions (PNDS) to evaluate preference-consistency and robustness, addressing limitations of existing multi-objective evaluation practices for PCPL (see Section 2 ). • HGNN-based PCPL: We extend baseline MLP policies with Heterogeneous Graph Neural Networks (HGNNs) and preference conditioning, showing improved generalization and preference-awareness on high-dimensional graph observations. 2 Background 2.1 Pareto Front In multi-objective optimization, a solution is Pareto optimal if it cannot be improved in any objective without degrading at least one other objective. Let’s consider a multi-objective maximization problem with N objectives, represented by a vector function: \displaystyle\mathbf{F}(x)=(f_{1}(x),f_{2}(x),\dots,f_{N}(x)), where x is a solution in the decision space X . A solution x_{1}\in X is said to dominate another solution x_{2}\in X , denoted as x_{1}\succ x_{2}

Read the complete text