Policy Learning from Large Vision-Language Model Feedback Without Reward Modeling

Tung M. Luu, Donghoon Lee, Younghwan Lee, Chang D. Yoo · 2025

Read Policy Learning from Large Vision-Language Model Feedback Without Reward Modeling by Tung M. Luu, Donghoon Lee, Younghwan Lee, Chang D. Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23391.txt.

Description

Luu, Donghoon Lee, Younghwan Lee, and Yoo train robots from the opinions of a vision-language model. Offline reinforcement learning learns from stored experience, but the stored data usually need reward labels, and writing reward functions is slow expert work. Their method, PLARE, skips rewards entirely: the model is shown pairs of video segments and asked which better matches the spoken task.

The robot's policy is then trained straight from those preference labels with a contrastive objective, never building a reward model. On MetaWorld manipulation tasks, the approach matches or beats methods that do construct rewards from such models. A real robot arm confirms the gains carry into the physical world, which the authors offer as proof of practical value.

Catalog record

CREATORTung M. Luu, Donghoon Lee, Younghwan Lee, Chang D. Yoo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23391

Browse the full library · About the Tower of Babel