Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback Volume: 44 Issue: 3

J. Kompatscher 1 , D. Shi 1 , G. Varni 2 , T. Weinkauf 3 , and A. Oulasvirta 1 1 Aalto University, Finland 2 University of Trento, Italy 3 KTH Royal Institute of Technology, Sweden · 2025

Read Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback Volume: 44 Issue: 3 by J. Kompatscher 1 , D. Shi 1 , G. Varni 2 , T. Weinkauf 3 , and A. Oulasvirta 1 1 Aalto University, Finland 2 University of Trento, Italy 3 KTH Royal Institute of Technology, Sweden (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04340.txt.

Introduction

Jan Kompatscher and co-authors present a new tool in a 2025 paper for teaching AI systems from human choices. The usual method shows a person two sample behaviors and asks which one is better. That pair-by-pair method is slow when a training run produces thousands of samples for review.

Their tool instead shows whole groups at once. One view maps all samples in a tree of clusters, and a second view places two picked groups side by side for judgment. A smart suggestion step points users to the groups most worth comparing. In tests on six robot tasks, the method raised final rewards and produced better policies with fewer errors.

Catalog record

CREATORJ. Kompatscher 1 , D. Shi 1 , G. Varni 2 , T. Weinkauf 3 , and A. Oulasvirta 1 1 Aalto University, Finland 2 University of Trento, Italy 3 KTH Royal Institute of Technology, Sweden
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04340

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback Volume: 44 Issue: 3 J. Kompatscher 1 , D. Shi 1 , G. Varni 2 , T. Weinkauf 3 , and A. Oulasvirta 1 1 Aalto University, Finland 2 University of Trento, Italy 3 KTH Royal Institute of Technology, Sweden Abstract Reinforcement learning from human feedback (RLHF) has emerged as a key enabling technology for aligning AI behavior with human preferences. The traditional way to collect data in RLHF is via pairwise comparisons: human raters are asked to indicate which one of two samples they prefer. We present an interactive visualization that better exploits the human visual ability to compare and explore whole groups of samples. The interface is comprised of two linked views: 1) an exploration view showing a contextual overview of all sampled behaviors organized in a hierarchical clustering structure; and 2) a comparison view displaying two selected groups of behaviors for user queries. Users can efficiently explore large sets of behaviors by iterating between these two views. Additionally, we devised an active learning approach suggesting groups for comparison. As shown by our evaluation in six simulated robotics tasks, our approach increases the final policy returns by 69.34%. It leads to lower error rates and better policies. We open-source the code that can be easily integrated into the RLHF training loop, supporting research on human-AI alignment. \SpecialIssuePaper \CGFccby ccs Human-centered computing Visual analytics ccs Computing methodologies Learning from critiques † † year: 2025 † † year: 2025 † † editors: W. Aigner, N. Andrienko, and B. Wang † † subject: Eurographics Conference on Visualization CGF Vol No † † teaser: The standard RLHF uses pairwise comparisons and therefore requires a large number of comparisons leading to a high workload. The comparison pairs are suggested by the system and cannot be chosen by the user. Our RLHF approach provides more agency to the user and demands less work: we leverage the user’s visual abilities to effectively explore the behavior space via hierarchical grouping in the “exploration view” and to select groups for comparison. (a) Standard RLHF requires high workload due to pairwise comparisons, and provides no ability to steer the process. (b) Our RLHF approach requires less work due to groupwise comparisons, and the user can steer the process actively. 1 Introduction exploration view comparison view individual behaviors groups of behaviors previously compared behaviors suggested comparisons yes no Figure 1 : Our user interface consists of 2 connected views: The Exploration View on the left displays the sampled behaviors from the model in a hierarchical radial chart. Users can select groups or individual behaviors for comparison using a mouse. Suggestions for comparisons are shown as gray lines, while previously made comparisons are visualized in color. The Comparison View shows 2 groups of videos, and the user is tasked to provide the preference, i.e., to state which group comes closer to the desired behavior of the agent. Users can edit these groups at any time by adding or deleting videos, or even moving them from one group to the other. Reinforcement learning (RL) is a method for training AI models from experience [ SB18 ] . During the last decade, it has been successfully applied to a range of difficult tasks such as Go [ SHM∗16 ] , Dota 2 [ BBC∗19 ] , and Atari Games [ MKS∗15 ] . The main idea is to reward the preferred behaviors of the model, and punish the unpreferred ones. To achieve such a goal, a reward function , which judges the AI model’s behavior with a numerical value that serves as a reward, is used. Behaviors in this context refer to segments of state-action sequences of the agent. For example, this could mean part of a written response, a generated image, or a time series of angles, positions, and torques of the joints of a robot. However, it has turned out to be highly challenging to define reward functions that are related to human preferences in the form of mathematical equations [ Mah96 , GCJ∗24 ] . Therefore, research has turned to exploit human feedback as a guidance for RL. Reinforcement Learning from Human Feedback (RLHF) is one of the most popular methods of this kind [ CLB∗17 ] . It works as follows: repeatedly, two different behaviors of the AI model (e.g., images or videos) are presented to human evaluators. The evaluator can then choose which of the two they prefer. This preference data is used to train a reward model , which serves as the reward function [ CLB∗17 , ZSW∗19 , OWJ∗22 ] . This concept has been used for hard-to-formulate objectives such as safety [ DPS∗23 ] , factuality [ SSC∗23 ] , or aesthetics [ WDR∗23 ] , and to fine-tune models to generate images better aligned with human preferences [ BJD∗23 , LLR∗23 ] , and to improve the grounding of textual explanations of images [ YYZ∗23 ] . Standard RLHF [ CLB∗17 ] uses pairwise comparisons , i.e., asking users to repeatedly compare pairs of behaviors (Fig. -a), until a sufficient number of comparisons is reached. This is highly laborious work that leaves no agency for the users, who cannot choose which behaviors to compare. The main limitations are: • Time inefficiency: Comparing one pair of behaviors at a time is time-consuming. Gathering enough human feedback requires over 700 comparisons for a simple behavior like a robot walking forward [ CLB∗17 ] , which is labor-intensive and costly. • Lack of user agency: Users have their own idea of what the desired agent’s behavior should look like. With the standard RLHF, users have no ability to explore and select behaviors interactively , to provide more effective feedback. • Inability to leverage contextual information: the standard approach does not show and leverage valuable contextual information such as an overview of all behaviors, or a list of the comparisons already done by the user. This makes it impossible for users to understand the broader behavior space. Thus, the standard RLHF can be impractical in applications that require specialized expertise (e.g., medical doctors), and in cases where users want to teach a model for a creative purpose (e.g., game design). Such cases would benefit from more agency and a lower workload than the standard RLHF approach can offer [ DKF22 ] . Our work substantially expands the reward elicitation interface by visualizing the entire behavior space in a hierarchical manner and allowing users to freely navigate it, thereby providing more agency for the users. We enable them to categorize behaviors into different groups and compare these groups with each other. As we will show in our evaluation, this increases efficiency; i.e., more preferences can be recorded in the same time than with the standard approach, and produces higher final policy returns by letting the user provide more informative feedback. Furthermore, we augment the visualization of the behavior space indicating the achieved comparison progress and suggesting what to compare next. This allows to review previous decisions and adapt the work accordingly. See Fig. 1 for an overview of the user interface. Previous work on user interfaces for RLHF addresses modular environments for rapidly developing such interfaces and analyzing the

Read the complete text