PB²: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning

Brahim Driss, Alex Davey, Riad Akrour · 2025

Read PB²: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning by Brahim Driss, Alex Davey, Riad Akrour (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13741.txt.

Introduction

Brahim Driss, Alex Davey, and Riad Akrour fix, in a 2025 paper, a blind spot in learning from human picks. Such learning skips set reward rules and grades acts by which of two paths a judge likes more. Lone agents rush to a thin slice of what judges want and stall there. A kept crowd of agents roams the pick space far more wide, the team shows. Crowd acts split in clear ways, so judges can tell paths apart and grade them with sense. That split feeds a sharper reward model as well.

Lone ways fail in set traps: stuck in poor spots, starved for grades, or wrecked when judges err on like paths. The crowd held firm when such slips were fed in.

Catalog record

CREATORBrahim Driss, Alex Davey, Riad Akrour
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PB²: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning Brahim Driss, Alex Davey, Riad Akrour Abstract Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions. However, current PbRL methods face a critical challenge in effectively exploring the preference space, often converging prematurely to suboptimal policies that satisfy only a narrow subset of human preferences. In this work, we identify and address this preference exploration problem through population-based methods. We demonstrate that maintaining a diverse population of agents enables more comprehensive exploration of the preference landscape compared to single-agent approaches. Crucially, this diversity improves reward model learning by generating preference queries with clearly distinguishable behaviors, a key factor in real-world scenarios where humans must easily differentiate between options to provide meaningful feedback. Our experiments reveal that current methods may fail by getting stuck in local optima, requiring excessive feedback, or degrading significantly when human evaluators make errors on similar trajectories, a realistic scenario often overlooked by methods relying on perfect oracle teachers. Our population-based approach demonstrates robust performance when teachers mislabel similar trajectory segments and shows significantly enhanced preference exploration capabilities, particularly in environments with complex reward landscapes. 1 Introduction Reinforcement learning (RL) has demonstrated remarkable success across a wide range of applications, from game playing to robotic control. However, the effectiveness of traditional RL methods remains heavily dependent on carefully designed reward functions, which are often challenging to specify for complex tasks involving subjective outcomes or intricate human preferences ( Hadfield-Menell et al. 2017 ) . Preference-based reinforcement learning (PbRL) offers a promising alternative by enabling agents to learn directly from human feedback through preferences between pairs of behavior trajectories ( Christiano et al. 2017 ; Lee et al. 2021a ) . This approach eliminates the need for hand-crafted reward functions and provides a more intuitive interface for humans to express their intentions. Despite these advantages, PbRL faces a fundamental challenge: policies optimized for the current reward model often generate similar queries for preference elicitation, limiting the informativeness of human feedback needed to improve the model. Current PbRL implementations typically use single-policy approaches that often converge too early to behaviors representing only a limited subset of human preferences. This happens because these methods increasingly generate similar trajectory segments pairs for human evaluation, limiting the range of behaviors presented. As a result, the reward model learns from less diverse examples over time, leading to poor exploration and suboptimal policies. The problem becomes worse when humans need to evaluate trajectory segments pairs with only minor differences. In such cases, evaluators often give inconsistent feedback, adding noise that significantly degrades learning performance. Existing approaches attempt to address preference illiciation challenges through various query selection strategies: uncertainty-based methods targeting uncertain preference regions ( Marta et al. 2023 ) , policy-aligned techniques focusing on current behavior ( Hu et al. 2024 ) , ensemble-based strategies leveraging model disagreement ( Liang et al. 2022 ) , and information-theoretic approaches maximizing information gain ( Biyik et al. 2020 ; Biyik and Sadigh 2018 ) . While improving sample efficiency, these methods operate within a single-agent framework, limiting behavioral diversity during evaluation. This becomes problematic when humans must compare similar trajectories, often leading to inconsistent feedback ( Huang et al. 2025 ) . Without mechanisms for maintaining diversity, existing approaches remain vulnerable to local optima, requiring excessive feedback and degrading under realistic conditions of human evaluation inconsistency. In this paper, we present PB²: a novel population-based approach to preference space exploration in PbRL. Our key insight is that simultaneously training multiple distinct policies facilitates more thorough preference landscape exploration than conventional single-policy methods. By collecting experiences across different policies to construct comparison pairs, we substantially enhance the variety of behaviors evaluated while preserving alignment with expressed human preferences. This behavioral diversity significantly improves reward model training by creating preference queries that are more easily distinguishable, an essential consideration often overlooked by existing methods that implicitly assume humans can reliably evaluate even subtly different behaviors. As illustrated in Figure 1 , our approach implements this insight through a feedback loop where diverse policies generate distinct trajectories, human preferences on these trajectories train a reward model, and a discriminator maintains population diversity while encouraging behaviors that align with current preferences. Our contributions are as follows: 1. We identify the preference exploration problem in PbRL, demonstrating how single-agent methods frequently fail by converging to suboptimal local minima in the preference space. 2. We propose a population-based framework for PbRL that maintains policy diversity while optimizing for human preferences, significantly enhancing exploration of the preference landscape. 3. We demonstrate through experiments that PB² produces more distinguishable queries that improve reward learning efficiency, achieving greater robustness when human feedback is inconsistent and improving performance with limited feedback. We validate these claims through three complementary experimental evaluations: (1) a systematic evaluation across DMControl locomotion tasks with varying similarity thresholds \epsilon to simulate human judgment inconsistency, and (2) a qualitative demonstration of how PB² escapes local optima in complex preference landscapes where single-agent methods remain trapped and (3) a comparative analysis in navigation tasks with extremely limited feedback, demonstrating PB²’s feedback efficiency in this limited setting. The remainder of this paper is organized as follows: Section 2 reviews related work in preference-based RL and population-based methods. Section 3 establishes preliminaries and formalizes the preference exploration problem. Section 4 presents our population-based approach in detail. Section 5 describes our experimental setup, and Section 6 presents and analyzes our results. Finally, Section 7 discusses limitations, future directions, and broader implications of our work. Figure 1: Overview of PB² . After an initial unsupervised exploration phase, we sample experience from different agents and form comparison pairs to train a reward model using human feedback. A discriminator is trained on the experience of the agents, with the goal of maintaining diversity in the population. The discriminator exploration bonus is added to the learned reward, in order to encourage the discovery of different behaviours, aligned with the current human preferences, thereby helping to elicit more informative and refined preferences. 2 Related Work Preference-based Reinforcement Learning Preference-based Reinforcement Learning (PbRL) enables agents to learn from human feedback through trajectory comparisons ( Christiano et al. 2017 ; Lee et al. 2021a ; Wirth et al. 2017 ) . Despite its intuitive interface, PbRL faces a fundamental exploration challenge, as policies optimized for the current reward model often generate similar queries for preference elicitation, limiting the informativeness of human feedback needed to improve the model. Recent methods address this through reward uncertainty ( Liang et al. 2022 ) , semi-supervised learning ( Park et al. 2022 ) , model-based approaches ( Liu et al. 2023 ) , bi-level optimization ( Liu et al. 2022 ) , dynamics encoding ( Metcalf et al. 2022 ) , and query-policy alignment (QPA) ( Hu et al. 2024 ) , though they remain primarily exploitative. Our approach extends QPA’s insights while addressing exploration-exploitation tradeoffs through diversity that generates distinguishable behaviors crucial for meaningful human feedback. Population-based Reinforcement Learning Population-based methods maintain multiple agents to enhance exploration, from Population-Based Training ( Jaderberg et al. 2017 ) to extensions incorporating Bayesian optimization ( Wan et al.

Read the complete text