Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Tung M. Luu, Younghwan Lee, Donghoon Lee, Sunho Kim, Min Jun Kim, Chang D. Yoo · 2025
Read Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models by Tung M. Luu, Younghwan Lee, Donghoon Lee, Sunho Kim, Min Jun Kim, Chang D. Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12822.txt.
Introduction
Tung Luu and colleagues improve how robot learners earn rewards from ratings given by large vision-language models. Writing good reward rules by hand takes expert effort, and asking humans for feedback is slow and costly. Rating-based learning asks for a score on a whole behavior rather than a pick between two, which carries more information. Writing in twenty twenty-five, the team asks a vision-language model to rate recorded agent behavior directly. They also fix two weak points of rating methods: lopsided rating data and noisy labels, which together made earlier training unstable.
Across both low-level control and higher-level tasks, their method learned better reward functions than prior vision-model feedback methods. Sample efficiency improved as well, since each rating teaches more than a single comparison. The authors present AI ratings as a scalable substitute for much human supervision in reward learning.
Catalog record
| CREATOR | Tung M. Luu, Younghwan Lee, Donghoon Lee, Sunho Kim, Min Jun Kim, Chang D. Yoo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-12822
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models Tung M. Luu, Younghwan Lee, Donghoon Lee, Sunho Kim, Min Jun Kim, Chang D. Yoo Abstract Designing effective reward functions remains a fundamental challenge in reinforcement learning (RL), as it often requires extensive human effort and domain expertise. While RL from human feedback has been successful in aligning agents with human intent, acquiring high-quality feedback is costly and labor-intensive, limiting its scalability. Recent advancements in foundation models present a promising alternative–leveraging AI-generated feedback to reduce reliance on human supervision in reward learning. Building on this paradigm, we introduce ERL-VLM, an enhanced rating-based RL method that effectively learns reward functions from AI feedback. Unlike prior methods that rely on pairwise comparisons, ERL-VLM queries large vision-language models (VLMs) for absolute ratings of individual trajectories, enabling more expressive feedback and improved sample efficiency. Additionally, we propose key enhancements to rating-based RL, addressing instability issues caused by data imbalance and noisy labels. Through extensive experiments across both low-level and high-level control tasks, we demonstrate that ERL-VLM significantly outperforms existing VLM-based reward generation methods. Our results demonstrate the potential of AI feedback for scaling RL with minimal human intervention, paving the way for more autonomous and efficient reward learning. The code is available at: https://github.com/tunglm2203/erlvlm . 1 Introduction Reinforcement Learning (RL) has achieved remarkable success in various domains, including games ( Silver et al. 2017 ; Vinyals et al. 2019 ; Perolat et al. 2022 ) , robotics ( Kalashnikov et al. 2018 ; Chen et al. 2022 ) , and autonomous systems ( Bellemare et al. 2020 ; Zhou et al. 2020 ; Kaufmann et al. 2023 ) . A critical factor driving these successes is the design of an appropriate reward function, which both captures the task objective and allows an RL agent to learn efficiently. This reward function is often manually defined and refined by human domain experts, a process that is not only time-consuming but also susceptible to errors ( Skalse et al. 2022 ) . In this work, we aim to eliminate the dependence on manually designed reward functions by training agents with reward functions derived from data. In this context, RL from human feedback (RLHF) has emerged as a powerful paradigm, enabling the learning of reward functions directly from human feedback ( Knox & Stone 2009 ; Christiano et al. 2017 ; Ibarz et al. 2018 ) . One obstacle for employing RLHF at scale is its reliance on a substantial amount of human feedback, which is both costly and labor-intensive. Recent advancements in vision-language models (VLMs) have demonstrated their ability to understand intricate relationships between images and text. When integrated with large language models, large VLMs have shown a high degree of alignment with human judgment ( Gilardi et al. 2023 ; Ding et al. 2023 ; Bao et al. 2024 ; Yoon et al. 2025 ) . These capabilities position large VLMs as promising proxies for human feedback in training RL agents. Recent studies have explored this approach for training robotic tasks by querying large VLMs for pairwise preferences between trajectories ( Guan et al. 2024 ; Wang et al. 2024a ) . However, querying VLMs for preferences has a few disadvantages. First, individual preferences convey little information, requiring a large number of queries to learn meaningful reward functions, which can lead to sample inefficiency. In addition, since preferences are inherently derived by comparing at least two trajectories, the number of input and (potential) output tokens processed by the VLM effectively doubles. This leads to higher computational costs and a slower querying process, especially for long trajectories. Finally, when querying VLMs with trajectories of similar quality ( e.g. , negative-negative or positive-positive pairs), the model often provides invalid preferences by “hallucinating” that one trajectory is better than the other ( Guan et al. 2024 ) . To address this, ( Wang et al. 2024a ) introduces an “unsure” label for indistinguishable trajectories and discards them during reward learning. While this mitigates query ambiguity, it is inefficient as it prevents the full utilization of all queried samples. Figure 1 : Overview of ERL-VLM : The figure illustrates an example of a PickupObject task in ALFRED along with its corresponding task description. During each feedback session, trajectory segments \sigma are randomly sampled from the replay buffer and sent to the VLM teacher to obtain rating feedback \tilde{y} . The reward model \hat{r}_{\psi} is then trained using the collected rating dataset and subsequently used to compute rewards as the agent interacts with the environment. To address these challenges, we propose ERL-VLM ( E nhancing R ating-based L earning to Effectively Leverage Feedback from V ision- L anguage M odels), a method that efficiently utilizes feedback from large VLMs, such as Gemini ( Reid et al. 2024 ) , to generate reward functions for training RL agents. Unlike prior approaches that rely on pairwise comparisons, ERL-VLM queries large VLMs for absolute evaluations of individual trajectories, represented on a Likert scale ( i.e. , a range of discrete ratings such as “very bad”, “bad”, “ok”, “good”, and “very good”). This approach allows VLMs to provide more expressive feedback, reduces query ambiguity, and ensures that all queried samples are fully utilized for reward learning. Additionally, we introduce simple yet effective enhancements to existing rating-based RL methods to mitigate instability in reward learning caused by data imbalance and noisy rating labels from VLMs. We demonstrate that ERL-VLM can generate reward functions that successfully train RL agents across a variety of vision-language navigation discrete action and continuous robotic control environments. Notably, ERL-VLM operates solely on image observations of the agent and a language task description, without requiring privileged state information or access to environment source code, as required in prior work ( Ma et al. 2024 ; Xie et al. 2024 ; Wang et al. 2024b ; Venuto et al. 2024 ; Han et al. 2024 ) . Our contributions are summarized as follows: • We present ERL-VLM, a novel method for learning reward functions from VLM-provided feedback. ERL-VLM enables agents to learn new tasks using only a human-provided language task description, significantly reducing the need for extensive human effort in reward design. • We propose simple yet effective enhancements to existing rating-based RL methods, improving stability in reward learning from VLM feedback and thereby improving the overall agent performance. • We demonstrate that ERL-VLM outperforms prior VLM-based reward generation methods, successfully training RL agents across a variety of vision-language navigation and robotic control tasks. Additionaly, we conduct extensive ablation studies to identify key factors driving ERL-VLM’s performance gains and provide deeper insights into its effectiveness. 2 Related Work Inverse Reinforcement Learning. Inverse Reinforcement Learning (IRL) aims to recover reward functions from expert demonstrations, enabling the learned reward functions to guide agents in replicating the demonstrated behaviors. Various IRL approaches have been developed, including maximum entropy IRL ( Ziebart et al. 2008 ; Wulfmeier et al. 2015 ) , non-linear IRL ( Levine et al. 2011 ; Finn et al. 2016 ) , adversarial IRL ( Ho & Ermon 2016 ; Fu et al. 2018 ) , and behavioral cloning IRL ( Szot et al. 2023 ) . While these methods have shown