Pref-GUIDE : Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning

Zhengran Ji [email protected], Boyuan Chen [email protected], Project Website: http://generalroboticslab.com/Pref-GUIDE · 2025

Read Pref-GUIDE : Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning by Zhengran Ji [email protected], Boyuan Chen [email protected], Project Website: http://generalroboticslab.com/Pref-GUIDE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07126.txt.

Introduction

Zhengran Ji and Boyuan Chen teach robots from live human likes instead of scores. Their 2025 paper tackles feedback in online learning, where a person must steer an agent as it acts. Score-style feedback is noisy and wobbles over time, which bends the learned reward. Their frame, Pref-GUIDE, turns scores into comparisons between short behavior windows and drops murky ones.

A voting twin pools reward models across many users into shared preferences. Across three hard test worlds, the method beat score baselines, and voting even topped hand-built dense rewards. Preference form makes live feedback steadier, they argue.

Catalog record

CREATORZhengran Ji [email protected], Boyuan Chen [email protected], Project Website: http://generalroboticslab.com/Pref-GUIDE
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07126

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pref-GUIDE : Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning Zhengran Ji [email protected], Boyuan Chen [email protected], Project Website: http://generalroboticslab.com/Pref-GUIDE Abstract Training reinforcement learning agents with human feedback is crucial when task objectives are difficult to specify through dense reward functions. While prior methods rely on offline trajectory comparisons to elicit human preferences, such data is unavailable in online learning scenarios where agents must adapt on the fly. Recent approaches address this by collecting real-time scalar feedback to guide agent behavior and train reward models for continued learning after human feedback becomes unavailable. However, scalar feedback is often noisy and inconsistent, limiting the accuracy and generalization of learned rewards. We propose Pref-GUIDE , a framework that transforms real-time scalar feedback into preference-based data to improve reward model learning for continual policy training. Pref-GUIDE Individual mitigates temporal inconsistency by comparing agent behaviors within short windows and filtering ambiguous feedback. Pref-GUIDE Voting further enhances robustness by aggregating reward models across a population of users to form consensus preferences. Across three challenging environments, Pref-GUIDE significantly outperforms scalar-feedback baselines, with the voting variant exceeding even expert-designed dense rewards. By reframing scalar feedback as structured preferences with population feedback, Pref-GUIDE offers a scalable and principled approach for harnessing human input in online reinforcement learning. Figure 1: Pref-GUIDE . Real-time scalar human feedback is often inconsistent, noisy, and varies across individuals. Pref-GUIDE addresses this by (a) converting scalar feedback into local pairwise preferences to achieve temporal consistency, and (b) aggregating reward models across human evaluators to form consensus-based rewards. (c) These improvements yield more robust reward learning and enable effective continual policy training after human feedback becomes unavailable. 1 Introduction Incorporating human feedback into reinforcement learning (RL) has become increasingly important for training agents in environments where task goals are difficult to formalize Christiano et al. 2017 ; Cao et al. 2021 ; Rafailov et al. 2023 ; Ramesh et al. 2024 ; Zhang et al. 2024b ; Ji et al. 2024 . While manually designed reward functions can work well in constrained or well-specified settings, they often fail to capture the full complexity of real-world objectives or align with human intent. In such cases, human feedback offers a more flexible and adaptive alternative for guiding agent behavior. Two major paradigms have emerged to leverage human feedback in RL. Preference-based RL Christiano et al. 2017 ; Cao et al. 2021 ; Ouyang et al. 2022 ; Rafailov et al. 2023 ; Ramesh et al. 2024 uses offline datasets of pairwise trajectory comparisons to train reward models that reflect human judgments. These models are then used to generate reward signals during policy learning. Alternatively, real-time scalar feedback MacGlashan et al. 2017 ; Arumugam et al. 2019 ; Knox & Stone 2009 ; Warnell et al. 2018 ; Zhang et al. 2024b allows humans to directly influence the learning process by providing moment-to-moment evaluations during agent interaction. GUIDE Zhang et al. 2024b , a recent method in this category, introduces a two-phase framework: during the human-in-the-loop phase , scalar feedback is combined with environment rewards to guide agent learning; in the subsequent post-human-guidance phase , a regression model trained on the collected human feedback continues to guide the agent after the human input is no longer available. While effective in early demonstrations, both paradigms face critical limitations. Preference-based methods rely on rich trajectory sets and offline human labeling, making them unsuitable for real-time and interactive settings Wirth et al. 2017a . Moreover, the co-evolution of the policy and reward model can introduce instability Wirth et al. 2017a . Real-time scalar feedback, while more adaptable, suffers from two key challenges, especially in obtaining an effective reward model for post-human-guidance phase : Human feedback is inconsistent over time. Evaluators naturally shift their expectations over the course of training Wang et al. 2022 , which makes it difficult to learn stable reward functions from raw feedback. For example, in an object search task, exploratory behavior may initially be rewarded but later penalized once goal-directed behavior is expected. This temporal inconsistency introduces non-stationarity, making it difficult for a regression model to generalize feedback into a stable reward model. Human feedback is not always reliable. Variability in cognitive ability, fatigue, attention lapses, and evaluation Wang et al. 2022 ; Wirth et al. 2017a ; Zhang et al. 2024a ; Zhang et al. 2024b contribute to intra- and inter-evaluator noise. These inconsistencies reduce the signal-to-noise ratio in scalar feedback, ultimately degrading policy performance when learned directly. Our goal in this study is to improve the post-human-guidance phase of real-time human-guided RL. While we adopt the same human-in-the-loop phase as GUIDE, our contributions focus on transforming and aggregating the collected human feedback to support more stable and effective continual training without requiring additional human input. To this end, we propose Pref-GUIDE (Figure 1 ), a framework that transforms noisy real-time scalar feedback into structured preference data to train more robust reward models. Pref-GUIDE consists of two key components: Pref-GUIDE Individual transforms scalar human feedback from each evaluator into temporally local pairwise preferences. This design is motivated by our observation (Section 4.1 ) that human feedback tends to be relatively consistent within short time intervals. By comparing agent behaviors within short temporal windows, we generate multiple preference pairs from each scalar signal, which effectively reuses feedback to create more training data while maintaining temporal coherence. We further introduce a no-preference margin Lee et al. 2021b to handle ambiguous or low-confidence feedback. Pref-GUIDE Voting builds on this idea by aggregating individual reward models across a population of human evaluators. Since feedback quality varies across individuals due to noise or evaluator bias Wang et al. 2025 ; Meyers et al. 2023 ; von Rueden et al. 2015 , we use a consensus-based relabeling strategy to extract a more robust reward signal. By combining predictions from multiple evaluator-specific models, we reduce individual noise and improve the robustness of the learned reward (Figure 1 ). We evaluate Pref-GUIDE across three challenging visual RL environments Zhang et al. 2024a ; Zhang et al. 2024b , where agents must act based on partial visual observations. Our results show that Pref-GUIDE Individual outperforms regression-based baselines when feedback quality is high, and Pref-GUIDE Voting maintains strong performance across diverse user inputs. In more complex tasks, our method even surpasses policies trained with expert-designed dense rewards, demonstrating the power of structured and population preference learning from real-time human feedback. Extensive ablation studies further validate our design choices and provide insight into their contributions. By converting noisy and unreliable human signals into structured and population-aggregated preference data, Pref-GUIDE offers a scalable and principled framework to leverage human feedback in RL, towards more consistent improvements even after human supervision ends. 2 Related Work 2.1 Real-Time Human-in-the-Loop Reinforcement Learning Incorporating real-time human feedback into RL has been studied through several paradigms, each grounded in different assumptions about the nature and role of feedback. One major class of methods, such as TAMER Knox & Stone 2009 and Deep TAMER Warnell et al. 2018 , treats human input as a one-step reward signal, allowing agents to update their policies myopically based on predicted human evaluations. A second line of work, such as COACH MacGlashan et al. 2017 and Deep

Read the complete text