SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
Philip Schroeder Thomas Weng Karl Schmeckpeper, Eric Rosen Stephen Hart Ondrej Biza · 2026
Read SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning by Philip Schroeder Thomas Weng Karl Schmeckpeper, Eric Rosen Stephen Hart Ondrej Biza (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28730.txt.
Introduction
Schroeder and co-authors introduce SOLE-R1, a video-language model that serves as the sole reward for robot reinforcement learning. Strong evaluator models can fail under partial observability and reward hacking. SOLE-R1 reads raw video and a language goal, then reasons step by step over time and space. It outputs dense estimates of task progress as rewards. A synthesis pipeline created training traces tied to continuous progress labels. Across four simulated settings and a real robot, robots learned unseen tasks from scratch. SOLE-R1 succeeded on 24 unseen tasks.
Catalog record
| CREATOR | Philip Schroeder Thomas Weng Karl Schmeckpeper, Eric Rosen Stephen Hart Ondrej Biza |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92661 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28730
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning Philip Schroeder Thomas Weng Karl Schmeckpeper, Eric Rosen Stephen Hart Ondrej Biza Abstract Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today’s strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. To address this limitation, we introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including GPT-5 and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. 1 Introduction Large language models (LLMs) and vision-language models (VLMs) have demonstrated impressive capabilities across a wide range of reasoning and perception tasks [ 16 , 52 ] . These capabilities have motivated growing efforts to leverage foundation models as sources of supervision for robot learning, replacing or reducing the need for task-specific reward engineering and human annotation. Ideally, a robot could acquire new skills entirely from scratch by interacting with the world, while receiving guidance derived solely from pretrained foundation models. In practice, however, this idealized paradigm remains out of reach. When used as reward functions or evaluators for reinforcement learning (RL), current state-of-the-art models, such as GPT-5 and Gemini-3-Pro, exhibit systematic failures in grounded visual reasoning. Despite exhibiting impressive visual captioning and question-answering abilities, they lack robustness to partial observability and distribution shift [ 20 , 50 ] . As a result, when robot policies are trained using rewards derived from these models, the policies quickly discover behaviors that exploit failures in perception or reasoning, eliciting high predicted reward, or perceived success, without achieving true success. To address this limitation, we introduce SOLE-R1, a foundation model with video-language reasoning explicitly designed to guide online RL. SOLE-R1 ( S elf- O bserving LE arner) generates per-timestep chain-of-thought (CoT) reasoning directly from raw video observations, yielding a dense estimate of task progress relative to goals specified in natural language. To generate the training data for SOLE-R1, we develop a video trajectory and reasoning synthesis pipeline which produces CoT traces that reason about changes over time, aligned with continuous progress signals. We generate over one million CoT reasoning examples from more than 40,000 real-world and simulated videos. We also carefully curate a diverse collection of general spatial and multi-frame temporal reasoning data to serve as a foundational layer of our training mixture. Together, this training induces video-native reasoning that explicitly integrates both spatial and temporal structure (Figure 2 ). We propose a two-stage hybrid recipe for training SOLE-R1: (1) supervised fine-tuning (SFT) to develop high-quality spatiotemporal CoT reasoning, (2) RL with verifiable rewards (RLVR) to further develop accurate progress prediction, boosted by the strong CoT reasoning developed during SFT. We demonstrate that SOLE-R1 reasoning can successfully serve as the SOLE signal for learning unseen tasks through online RL. In this setting, the robot begins with a random policy and learns entirely through interaction, guided only by SOLE-R1’s predicted rewards - without access to any ground-truth rewards, task-specific tuning, demonstrations, or other off-policy trajectories. In total, SOLE-R1 achieves zero-shot success on 24 unseen manipulation tasks across 4 simulator environments and a real-robot setting, while strong baseline reasoning models (e.g., GPT-5, Gemini-3, and other VLM rewarders) succeed on fewer than 10 tasks. SOLE-R1 generalizes to unseen task families (pick-and-place, articulated object manipulation, button/lever/knob interactions), scene layouts, camera viewpoints, and robot embodiments. We further demonstrate that SOLE-R1 can steer strong pre-trained policies to learn novel tasks and validate its reasoning through large-scale offline evaluations. Figure 1 : SOLE-R1 is a video-language reasoning model designed to guide online RL with per-timestep chain-of-thought reasoning and progress prediction. In large-scale experiments across 40 tasks, SOLE-R1 outperforms strong baseline models with zero-shot online RL. Our contributions include the following: • We introduce SOLE-R1 , a video-language reasoning model designed for guiding online RL with per-timestep CoT reasoning and progress prediction. • We introduce a video trajectory and reasoning synthesis pipeline to generate training data for video-based spatiotemporal CoT reasoning and progress estimation using videos from real-world and simulation. • We propose a hybrid training framework combining SFT for strong multi-frame spatial and temporal CoT reasoning with RLVR to emphasize progress prediction quality and calibration. • We perform extensive experiments showing SOLE-R1 reasoning can serve as the sole signal for learning unseen tasks through online RL , generalizing to new tasks and environments better than strong baseline vision-language reasoning models. • We release the SOLE-R1 model checkpoints and full training dataset, along with the online RL code and algorithms for reproducing all experiments. Figure 2 : SOLE-R1 Training Data Mixture. The dataset combines foundational spatial reasoning (single-image and depth), multi-frame temporal reasoning, and our synthesized video trajectories with chain-of-thought explanations and dense progress supervision, jointly enabling reasoning over space and time for progress prediction. 2 SOLE-R1 SOLE-R1 is a vision-language model designed to perform grounded natural language reasoning over videos of robot trajectories and predict task progress estimates, which can directly serve as dense rewards for online RL. To develop this form of reasoning, we construct a video trajectory and reasoning synthesis pipeline to generate video-based spatiotemporal CoT reasoning and progress estimation training data, layered on top of a foundational mixture of general spatial and multi-frame temporal reasoning, and integrated with a hybrid training framework. In Section 2.1 , we introduce our framework for SOLE-R1 video-language reasoning and its integration as a dense reward for online RL. In Section 3 , we describe our method for generating diverse video trajectories with CoT reasoning traces for video-based progress prediction training and our layering of this with a carefully curated foundation of general spatial and temporal reasoning data. Finally, in Section 4 , we outline our hybrid training framework. 2.1 Video-native temporal progress reasoning We design SOLE-R1 to perform video-native temporal reasoning for goal-conditioned tasks. Given a natural-language goal g\in G and a video stream of observations \{o_{t}\}_{t=1}^{T} , the model produces (i) a per-timestep CoT explanation \{m_{t}\}_{t=1}^{T} describing what has changed since the last timestep and what remains to be done , and (ii) a dense scalar progress estimate \{p_{t}\}_{t=1}^{T} used as a reward signal for online RL. Temporal conditioning. At timestep t , the input x_{t} comprises the goal g , a sliding temporal context window of frames up to the present and the previous prediction, x_{t}=[g,\;o_{0};o_{t-K+1:t},\;p_{t-1}], where o_{0}