Active Human Feedback Collection via Neural Contextual Dueling Bandits

Arun Verma, Xiaoqiang Lin, Zhongxiang Dai, Daniela Rus, Bryan Kian Hsiang Low · 2025

Read Active Human Feedback Collection via Neural Contextual Dueling Bandits by Arun Verma, Xiaoqiang Lin, Zhongxiang Dai, Daniela Rus, Bryan Kian Hsiang Low (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12016.txt.

Introduction

Arun Verma and four co-authors present Neural-ADB, an algorithm that spends a human feedback budget where it teaches the most, in this 2025 arXiv paper. Preference labels are pricey, and older selection methods assume the hidden reward is a simple straight-line rule. Real rewards bend, in recommendations and in aligning language models alike.

Their method grows from the dueling bandit framework, letting pairs of options compete while a neural model learns the reward's curved shape. Theory says the learned policy's worst-case gap to the best one shrinks steadily as labels accumulate. Experiments on preference datasets backed the claim. The authors aim the tool at teams that must learn from scarce human judgment.

Catalog record

CREATORArun Verma, Xiaoqiang Lin, Zhongxiang Dai, Daniela Rus, Bryan Kian Hsiang Low
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12016

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Active Human Feedback Collection via Neural Contextual Dueling Bandits Arun Verma, Xiaoqiang Lin, Zhongxiang Dai, Daniela Rus, Bryan Kian Hsiang Low Abstract Collecting human preference feedback is often expensive, leading recent works to develop principled algorithms to select them more efficiently. However, these works assume that the underlying reward function is linear, an assumption that does not hold in many real-life applications, such as online recommendation and LLM alignment. To address this limitation, we propose Neural-ADB , an algorithm based on the neural contextual dueling bandit framework that provides a principled and practical method for collecting human preference feedback when the underlying latent reward function is non-linear. We theoretically show that when preference feedback follows the Bradley-Terry-Luce model, the worst sub-optimality gap of the policy learned by Neural-ADB decreases at a sub-linear rate as the preference dataset increases. Our experimental results on preference datasets further corroborate the effectiveness of Neural-ADB . 1 Introduction Collecting human preference feedback is essential in many real-life applications, like online recommendations ( Kohli et al., 2013 ; Wu et al., 2023 ; Zhang and Wang, 2023 ; Yang et al., 2024 ) , content moderation ( Avadhanula et al., 2022 ) , medical treatment design ( Lai and Robbins, 1985 ; Bengs et al., 2021 ) , prompt optimization ( Lin et al., 2024 ) , and aligning large language models ( Bai et al., 2022 ; Menick et al., 2022 ; Mehta et al., 2023 ; Chaudhari et al., 2024 ; Das et al., 2024 ; Ji et al., 2024 ) , to ensure systems effectively align with user preferences and exhibit desired behaviors. However, this process is often costly due to the need for skilled evaluators, the complexity of tasks, and the time-intensive nature of producing high-quality, reliable human feedback. To address the challenge of balancing cost and effectiveness in aligning systems, this paper proposes principled and practical algorithms for efficiently collecting human feedback sequentially and adaptively to achieve the desired system behavior. Specifically, we aim to answer the following fundamental question: How to achieve desired system behavior while using as minimum human feedback as possible? Recent works ( Mehta et al., 2023 ; Das et al., 2024 ) have modeled the problem of active human feedback collection as an active version of the contextual dueling bandit problem ( ADB for brevity) ( Saha, 2021 ; Bengs et al., 2022 ; Li et al., 2024 ) , where context-arm pair in the contextual dueling bandits corresponds to a task for which human preference feedback is collected and then proposed algorithms to select context-arm pairs for human feedback sequentially and adaptively by exploiting collected preference dataset, i.e., past context-arm pairs with their preference feedback. The preference feedback between two context-arm pairs is commonly assumed to follow the Bradley-Terry-Luce (BTL) model 1 1 1 For more than two context-arm pairs, preferences are typically modeled using the Plackett-Luce model ( Soufiani et al., 2014 ) . ( Hunter, 2004 ; Bengs et al., 2022 ; Li et al., 2024 ; Lin et al., 2024 ; Verma et al., 2025 ) in which the probability of preferring a context-arm pair over others is proportional to the exponential of its reward. In many real-life applications, the number of context-arm pairs (e.g., user-movie pair in online movie recommendation) can be large or even infinite. Therefore, the reward for each context-arm pair is assumed to be an unknown function of its feature vector, such as a linear function ( Mehta et al., 2023 ; Das et al., 2024 ) . To better align the system for optimal performance, we consider two key components: context selection and arm selection . The context selection aims to encourage diversity by exploring the context space, such as selecting prompts as diverse as possible in prompt optimization. Whereas arm selection focuses on identifying the arms that help learn the best arm for each context, such as selecting the most effective pair of responses to a given prompt that maximizes the system’s learning ( Lin et al., 2024 ; Verma et al., 2025 ) . Since the goal is to identify the best arm for each context, selecting suboptimal arms provides less useful information than choosing better arms. Existing methods for active contextual dueling bandits ( Mehta et al., 2023 ; Das et al., 2024 ) fail to incorporate an efficient arm selection strategy during the data collection process, thereby limiting the ability of these methods to achieve optimal performance. An efficient arm selection strategy requires estimating the reward function to guide the arm selection process effectively. Since the reward function may not always be linear in practice, this paper parameterizes the reward function via a non-linear function , which needs to be estimated from the available preference dataset by using methods like Gaussian processes ( Williams and Rasmussen, 2006 ; Srinivas et al., 2010 ) or neural networks ( Zhou et al., 2020 ; Zhang et al., 2021 ) . However, Gaussian processes have limited expressive power and fail to optimize highly complex functions. In contrast, neural networks (NNs) have greater expressive power, making them well-suited for modeling complex functions ( Dai et al., 2023 ; Lin et al., 2023 ; Lin et al., 2024 ; Verma et al., 2025 ) . In this paper, we propose a neural active contextual dueling bandit algorithm, Neural-ADB , which uses an NN to estimate the unknown reward function using the available preference dataset. The context selection in Neural-ADB is adapted from Das et al. (2024) , while arm selection strategies are based on, respectively, upper confidence bound (UCB) and Thompson sampling (TS), and adapted from Verma et al. (2025) . Due to the differences in context selection strategy, arm selection strategies, and the use of a non-linear reward function, our theoretical analysis is completely different than related existing work ( Mehta et al., 2023 ; Das et al., 2024 ) . One of the key theoretical contributions of this paper is providing an upper bound on the maximum Mahalanobis norm of a vector from the fixed input space, measured with respect to the inverse of a positive definite Gram matrix that is constructed using finite, adapted samples from that space. Building on this result, we prove that the worst sub-optimality gap (defined in Eq. 1 ) of the policy learned by Neural-ADB decreases at a sub-linear rate as the preference dataset size increases. Specifically, our key contributions can be summarized as follows: • We introduce the setting of active contextual dueling bandits with a non-linear reward function in Section 2 . In Section 3 , we propose a neural active contextual dueling bandit algorithm, Neural-ADB , which uses an NN to estimate the unknown reward function from the available preference dataset and then uses this estimate into the arm selection strategies. • We prove an upper bound on the maximum Mahalanobis norm of a vector from the fixed input space, as measured with respect to the inverse of a positive definite Gram matrix ( Theorem 1 ), where the gram matrix is constructed using finite, adapted samples from that input space. We show that this upper bound decays at a sub-linear rate as the number of samples

Read the complete text