Active Query Selection for Crowd-Based Reinforcement Learning

Jonathan Erskine, Taku Yamagata, Raúl Santos-Rodgríguez · 2025

Read Active Query Selection for Crowd-Based Reinforcement Learning by Jonathan Erskine, Taku Yamagata, Raúl Santos-Rodgríguez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19132.txt.

Introduction

Jonathan Erskine, Taku Yamagata, and Raul Santos-Rodriguez present in 2025 a smarter way to spend scarce human feedback when training agents by preference. Such training suits tasks where a clear reward is hard to write, but expert judges are costly and scarce. Their frame joins two ideas: model the crowd of judges, noise and all, and ask only the most telling questions.

An older advising algorithm was extended to many trainers, with each one's reliability learned on the fly. A doubt-based rule picks which actions to ask about. Agents trained this way learned faster in most test worlds, from simple games to a blood sugar control task for type one diabetes.

Catalog record

CREATORJonathan Erskine, Taku Yamagata, Raúl Santos-Rodgríguez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19132

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Active Query Selection for Crowd-Based Reinforcement Learning Jonathan Erskine, Taku Yamagata, Raúl Santos-Rodgríguez Abstract Preference-based reinforcement learning has gained prominence as a strategy for training agents in environments where the reward signal is difficult to specify or misaligned with human intent. However, its effectiveness is often limited by the high cost and low availability of reliable human input, especially in domains where expert feedback is scarce or errors are costly. To address this, we propose a novel framework that combines two complementary strategies: probabilistic crowd modelling to handle noisy, multi-annotator feedback, and active learning to prioritize feedback on the most informative agent actions. We extend the Advise algorithm to support multiple trainers, estimate their reliability online, and incorporate entropy-based query selection to guide feedback requests. We evaluate our approach in a set of environments that span both synthetic and real-world-inspired settings, including 2D games (Taxi, Pacman, Frozen Lake) and a blood glucose control task for Type 1 Diabetes using the clinically approved UVA/Padova simulator. Our preliminary results demonstrate that agents trained with feedback on uncertain trajectories exhibit faster learning in most tasks, and we outperform the baselines for the blood glucose control task. Introduction Preference-based reinforcement learning has become a powerful paradigm for tailoring the behaviour of autonomous agents with human trainers. By incorporating human judgments, typically in the form of pairwise comparisons indicating whether a particular action is preferred over the other, human preferences can guide exploration, improve sample efficiency, and prevent unsafe behaviours during training ( Kaufmann et al. 2024 ) . However, practical deployment remains limited by two central challenges: (1) the cost of collecting high-quality feedback from crowds of humans with mixed skills and motives and (2) the limited availability of reliable human trainers, particularly in domains where expertise is scarce or errors carry high risk (e.g., healthcare, robotics). One approach to mitigate the scarcity of expert feedback is to aggregate input from multiple non-expert trainers, a technique broadly studied in the learning from crowds literature. These methods aim to infer both the true label and trainer reliability by modelling the noise inherent in human inputs. However, while crowd-based strategies can scale feedback collection ( Buecheler et al. 2010 ) , not all trainers are equally reliable. A second, complementary strategy to reduce annotation burden is active learning. In supervised settings, active learning algorithms prioritize annotation of the most uncertain or informative data points to accelerate training with fewer labels. In RLHF, however, relatively little work has been done to adapt such ideas to interactive feedback-driven learning. Yet, not all agent behaviours are equally valuable to label; querying human feedback only on high-uncertainty actions could yield greater improvements in policy quality for the same feedback budget. Many recent works collect feedback as pairwise comparisons of agent actions or trajectories ( Abdelkareem, Shehata, and Karray 2022 ) and tend to convert these comparisons into a reward signal ( Chakraborty et al. 2024 ) and/or generate models which can learn to predict these preferences ( Frick et al. 2024 ) . In our experiments we instead prescribe to the practice of collecting feedback directly on the policy as an additional signal to environmental rewards, as demonstrated by Griffith et al. 2013 . In this work, we propose a novel feedback aggregation framework that combines active learning, learning from crowds, and policy feedback: probabilistic crowd modelling to handle noisy, multi-trainer feedback, and entropy-based active selection of state-action pairs to maximize the utility of human input. Building on the framework provided in ( Griffith et al. 2013 ) , our method aims to: 1. Handle feedback from multiple trainers of varying reliability. 2. Infer trainer consistency levels in an online setting. 3. Actively request feedback only for agent behaviours deemed highly uncertain. Figure 1 illustrates our proposed method. To evaluate our approach, we simulate human feedback using a pre-trained oracle and assess performance across a set of diverse domains, including classic 2D environments (Taxi, PACMAN, Frozen Lake) and a blood-glucose monitoring and control task for Type 1 Diabetes using the clinically approved UVA/Padova simulator ( Man et al. 2014 ) . Experimental results show that our method significantly improves learning speed and feedback efficiency, particularly in deterministic settings where uncertainty is easier to quantify and resolve. Figure 1: Standard reinforcement learning training loop and our extended method. We collect a trajectory \boldsymbol{\tau} of state–action pairs and extract a subset \boldsymbol{\tau}_{\mathcal{H}}\subseteq\boldsymbol{\tau} consisting of the top- n pairs ranked by entropy \mathcal{H}(\mathcal{O}_{s,a}) . This subset is labelled by a crowd of trainers with diverse skill levels (some might be adversarial). We estimate each trainer’s reliability based on their feedback and weight their input accordingly when aggregating their feedback. Finally, the aggregated results are merged with the underlying RL policy \pi_{R}(a\mid s) , allowing us to sample an action to execute in the environment. This approach improves the efficiency of human feedback while accounting for varying trainer reliability. Related Work Learning From Crowds Learning from crowds is a well-established research area that addresses the challenge of aggregating noisy, unreliable, or biased labels provided by multiple trainers, often with varying expertise. A naive approach treats all trainers as equally reliable. However, performance can be improved by adjusting for trainer reliability ( Snow et al. 2008 ) . One of the earliest attempts to compute trainer reliability explicitly was a probabilistic model that jointly estimates the true labels and the error rates of each trainer via Expectation-Maximization (EM) ( Dawid and Skene 1979 ) . This foundational model has since inspired a range of probabilistic and Bayesian methods ( Whitehill et al. 2009 ; Raykar et al. 2010 ) , which learn trainer-specific confusion matrices and incorporate priors over worker accuracy. Beyond probabilistic modelling, recent work leverages deep learning to model crowd annotations at scale. Rodrigues and Pereira 2018 proposed learning with crowds using Gaussian processes and variational inference, while more recent neural architectures integrate crowd modelling directly into end-to-end training ( Guan, Houlsby, and Cesa-Bianchi 2018 ; Chu, Ma, and Wang 2020 ) , enabling better generalization in image classification and NLP tasks. In the reinforcement learning domain, Siththaranjan, Laidlaw, and Hadfield-Menell 2024 address the challenge of collecting preferences from humans with diverse preferences by modelling prefernce distributions and identifying hidden contexts and improving adversarial robustness. Active Learning Active learning (AL) is a supervised learning framework that aims to reduce annotation cost by selecting only the most informative data points for labelling. This selection is typically guided by an acquisition function that quantifies a model’s uncertainty or expected gain from receiving a label. The foundational idea is that the learner queries those instances for which its current predictions are least confident. This approach, known as uncertainty sampling , was introduced by Lewis and Gale 1994 . Subsequent strategies have proposed entropy-based selection (using Shannon information), margin sampling (difference between top prediction confidences), and expected model change (querying points likely to cause the greatest update to the model) ( Settles 2009 ) . These techniques have proven effective across a wide range of domains, including natural language processing, computer vision, and structured prediction. While traditionally applied to classification tasks, active learning has recently seen broader use in more complex learning settings. Notably, several recent works

Read the complete text