Revisiting the learning objectives of vision-language reward models
Simon Roy, Samuel Barbeau, Giovanni Beltrame, Christian Desrosiers, Nicolas Thome · 2025
Read Revisiting the learning objectives of vision-language reward models by Simon Roy, Samuel Barbeau, Giovanni Beltrame, Christian Desrosiers, Nicolas Thome (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20675.txt.
Introduction
Simon Roy and co-authors strip a fashionable research area down to one variable. Vision-language models can supply dense rewards for embodied agents without human labels, and recent papers keep adding fancier learning objectives. But differing data, backbones, and evaluations make the claimed progress impossible to attribute.
Under a unified approach with identical backbones, fine-tuning data, and Meta-World evaluation environments, the team compares recent reward models head to head. Accuracy is judged by consistency with ground-truth reward and correlation with expert progress. A simple triplet loss beats the state-of-the-art methods outright. Much of the field's recent improvement, the authors suggest, came from data and architecture differences rather than the celebrated objectives. Simplicity wins the controlled race.
Catalog record
| CREATOR | Simon Roy, Samuel Barbeau, Giovanni Beltrame, Christian Desrosiers, Nicolas Thome |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20675
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting the learning objectives of vision-language reward models Simon Roy, Samuel Barbeau, Giovanni Beltrame, Christian Desrosiers, Nicolas Thome Abstract Learning generalizable reward functions is a core challenge in embodied intelligence. Recent work leverages contrastive vision language models (VLMs) to obtain dense, domain-agnostic rewards without human supervision. These methods adapt VLMs into reward models through increasingly complex learning objectives, yet meaningful comparison remains difficult due to differences in training data, architectures, and evaluation settings. In this work, we isolate the impact of the learning objective by evaluating recent VLM-based reward models under a unified framework with identical backbones, finetuning data, and evaluation environments. Using Meta-World tasks, we assess modeling accuracy by measuring consistency with ground truth reward and correlation with expert progress. Remarkably, we show that a simple triplet loss outperforms state-of-the-art methods, suggesting that much of the improvements in recent approaches could be attributed to differences in data and architectures. 1 Introduction The increasing momentum of embodied intelligence has motivated the study of generalizable reward models that do not rely on hand-crafted human supervision. Recent work leverages large vision-language models (VLMs) as general-purpose reward functions that measure progress through alignment between visual observations and language goals. 13 showed that CLIP ( 12 ) can be used as a zero-shot reward model for downstream policy learning, though it struggles with domain shift related to action specification and environmental dynamics. To mitigate this effect, many methods have turned to large-scale video demonstration datasets ( 1 ; 3 ) to finetune VLMs, enabling a better understanding of task-relevant behaviors. Nevertheless, the question of extracting meaningful signals of task progress from video demonstrations remains a central challenge to vision-language reward modeling. As a result, increasingly complex learning objectives have been proposed to generate more accurate rewards ( 16 ; 9 ; 8 ; 7 ; 5 ) . However, most of these methods were pre-trained using different datasets and architectures, making it difficult to isolate the choice of learning objective from downstream performance. For instance, R3M ( 9 ) was pre-trained on Ego4D ( 3 ) yet compares to methods pre-trained on ImageNet ( 14 ; 11 ) . LIV ( 7 ) was initialized with CLIP weights and pre-trained on EpicKitchen ( 9 ) yet compares to methods utilizing a frozen BERT encoder 2 and vision encoder pre-trained on Ego4D from scratch ( 9 ) . In this work, we systematically compare objectives under a unified framework, holding the pre-trained model backbone, finetuning data, and downstream evaluation environments constant. This setup allows us to decouple the impact of the learning objectives from any other confounding factors. We assess model performance via two distinct benchmarks, evaluating consistency with ground truth reward and alignment with expert progress. While initially introduced as a minimal baseline for comparison with recent methods, our results show that a very simple triplet loss 15 can surpass current state-of-the-art learning objectives. More broadly, our findings indicate that simpler ranking-based learning objectives offer greater accuracy and robustness, suggesting that much of the apparent progress in recent methods could instead be due to differences in data and model architecture. 2 Methodology State-of-the-art contrastive reward modeling approaches rely on combinations of learning objectives, either in their original form or adapted for alignment with language. For our experiments, we focus on a set of loss functions that can be used to reproduce many popular reward models. TCN ( 16 ) reduces the embedding distance of images which are closer in time and increases the distance of images which are temporally further apart. Given an image encoder \pi_{img} , a sequence of images \{I_{t}\}_{t=0}^{k} , their corresponding encodings z_{t}=\pi_{img}(I_{t}) , batches of [I_{i},I_{j>i},I_{k>j}]^{1:B} and a similarity function \mathcal{S} , TCN minimizes the following objective: \mathcal{L}_{\text{TCN}}=-\frac{1}{B}\sum_{b\in B}\log\frac{e^{\mathcal{S}(z^{b}_{i},z^{b}_{j})}}{e^{\mathcal{S}(z^{b}_{i},z^{b}_{j})}+e^{\mathcal{S}(z^{b}_{i},z^{b}_{k})}+e^{\mathcal{S}(z^{b}_{i},{z^{\neq b}_{i}})}} (1) We adapt \mathcal{L}_{\text{TCN}} to language by replacing I_{k>j} with the language annotation for the given sequence. Inserting the language embedding v=\pi_{\text{text}}(l) results in the following objective \mathcal{L}_{\text{TCN}_{\text{text}}}=-\frac{1}{B}\sum_{b\in B}\log\frac{e^{\mathcal{S}(z^{b}_{j>i},v^{b})}}{e^{\mathcal{S}(z^{b}_{j>i},v^{b})}+e^{\mathcal{S}(z^{b}_{i},v^{b})}+e^{\mathcal{S}(z^{\neq b}_{j>i},v^{b})}} (2) VIP ( 8 ) uses a goal-conditioned approach that separates embeddings of temporally adjacent images while bringing together embeddings of images that are farther apart in time. Given batches of [I_{i},I_{j>i},I_{j+1},I_{k\geq j+1}]^{1:B} , VIP is defined as: \mathcal{L}_{\text{VIP}}=\frac{1-\gamma}{B}\sum_{b\in B}\left[-\mathcal{S}(z^{b}_{i},z^{b}_{k})\right]+\log\frac{1}{B}\sum_{b\in B}\mathrm{exp}\left[\mathcal{S}(z^{b}_{j},z^{b}_{k})+1-\gamma\mathcal{S}(z^{b}_{j+1},z^{b}_{k})\right] (3) Just like TCN, we adapt \mathcal{L}_{\text{VIP}} to language by replacing I_{k>h} with the embedded language annotation v=\pi_{\text{text}}(l) for the given sequence. \mathcal{L}_{\text{VIP}_{\text{text}}}=\frac{1-\gamma}{B}\sum_{b\in B}\left[-\mathcal{S}(z^{b}_{i},v^{b})\right]+\log\frac{1}{B}\sum_{b\in B}\mathrm{exp}\left[\mathcal{S}(z^{b}_{j},v^{b})+1-\gamma\mathcal{S}(z^{b}_{j+1},v^{b})\right] (4) Combinations of these losses reproduce popular reward modeling methods minus regularization terms. R3M is obtained by combining both TCN versions: \mathcal{L}_{\text{R3M}}=\mathcal{L}_{\text{TCN}}+\mathcal{L}_{\text{TCN}_{\text{text}}} (5) Note that R3M adapts TCN to language by training an MLP to predict a similarity score from the concatenated vector [ z_{0},z_{i},v ], where z_{0} is the first image in the sequence. This approach introduces a non-negligible amount of additional parameters, especially for large embeddings like those from SigLIP2. For fairness, we omit the initial embedding in our setup. LIV is a combination of both VIP versions plus an InfoNCE objective ( 10 ) , the latter which we also include in our experiments: \mathcal{L}_{\text{LIV}}=\mathcal{L}_{\text{VIP}}+\mathcal{L}_{\text{VIP}_{\text{text}}}+\frac{1}{B}\sum_{b\in B}\left[-\log\frac{e^{\mathcal{S}(z^{b}_{k},v^{b})}}{\frac{1}{B}\sum_{j\neq b}^{B}\left[e^{\mathcal{S}(z^{j}_{k},v^{b})}\right]}\right] (6) Triplet ( 15 ) applied to this context reduces the embedding distance between the language goal and the later images of the sequence, and pushes earlier images further away. We propose it as a simple baseline for task progress and show its remarkable effectiveness. Given batches of [I_{i},I_{j>i},l]^{1:B} , we use the language annotation as the anchor, the later image as the positive and the earlier image as the negative example. The triplet loss is defined as: \mathcal{L}_{\text{Triplet}}=\sum_{b\in B}\max\Big(0,\mathcal{S}(v^{b},x_{i}^{b})-\mathcal{S}(v^{b},x_{j}^{b})+\alpha\Big) (7) where \alpha is the margin controlling how far apart the negative should be from the positive. 3 Experiments 3.1 Finetuning Learning objectives. We finetune using different configurations of the losses described in 2 (1) \mathcal{L}_{\text{Triplet}} (ours) (2) \mathcal{L}_{\text{TCN}_{\text{text}}} (3) \mathcal{L}_{\text{R3M}} (4) \mathcal{L}_{\text{VIP}_{\text{text}}} (5) \mathcal{L}_{\text{VIP}_{\text{text}}}+\mathcal{L}_{\text{VIP}} (6) \mathcal{L}_{\text{LIV}} Model backbone To isolate the performance of the learning objective from model capacity and pre-training, we employ the base version of SigLIP2 ( 17 ) as the backbone for all methods. We finetune using LoRA ( 4 ) to prevent overfitting. Data Each model is finetuned on expert demonstrations from the Meta-World environments ( 18 ) . To evaluate out-of-domain performance, we remove 3 tasks from the training data (see 3.2 ). For the remaining tasks, we collect 3 expert trajectories per task with randomized end-effector initializations, yielding 132 total. To ensure consistent dataset size, we cap the maximum number of samples at 50 thousand and generate them evenly across trajectories. Each timestep is recorded from 3 camera views to enable multi-view training. The timesteps within a sample remain fixed between epochs, but their associated views are randomly reassigned at each iteration, acting as a form of data augmentation to promote view-invariant representations and reduce overfitting. Implementation details For a complete list of the implementation details, refer to A.1 . 3.2 Evaluation We wish to learn generalizable reward models which can leverage pre-training knowledge to predict progress on unseen tasks. To this end, we exclude button-press , drawer-open , and door-open environments from the finetuning data and use them for subsequent evaluation metrics. Button-press is relatively simple, as success depends mainly on end-effector proximity to a target. In contrast, drawer-open and door-open are compositional, requiring navigation to the object and subsequent manipulation. Note that we evaluate