ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
Philip Schroeder 1 Ondrej Biza 2 Thomas Weng 2 Hongyin Luo 1 James Glass 1 1 MIT CSAIL 2 RAI Institute [email protected] · 2025
Read ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks by Philip Schroeder 1 Ondrej Biza 2 Thomas Weng 2 Hongyin Luo 1 James Glass 1 1 MIT CSAIL 2 RAI Institute [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01943.txt.
Introduction
Philip Schroeder and co-authors teach vision-language models to reason over long robot videos in this 2025 paper. A robot's camera stream is long, and models lose the thread when fed the whole thing at once. Their framework, ROVER, chops the stream into subtask segments and reasons over each in turn, keeping the whole task in view.
They test it on open robot video collections and a new set of 543 kitchen-task clips, including clumsy, perturbed runs. ROVER beat strong baselines at judging progress, narrating frames, and answering questions. Zooming into segments cut the model's invented details, worst during mistakes. Its cost grows in a straight line with video length, a scaling win over rivals.
Catalog record
| CREATOR | Philip Schroeder 1 Ondrej Biza 2 Thomas Weng 2 Hongyin Luo 1 James Glass 1 1 MIT CSAIL 2 RAI Institute [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01943
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks Philip Schroeder 1 Ondrej Biza 2 Thomas Weng 2 Hongyin Luo 1 James Glass 1 1 MIT CSAIL 2 RAI Institute [email protected] Abstract Vision-language models (VLMs) have exhibited impressive capabilities across diverse image understanding tasks, but still struggle in settings that require reasoning over extended sequences of camera frames from a video. This limits their utility in embodied settings, which require reasoning over long frame sequences from a continuous stream of visual input at each moment of a task attempt. To address this limitation, we propose ROVER (Reasoning Over VidEo Recursively), a framework that enables the model to recursively decompose long-horizon video trajectories into segments corresponding to shorter subtasks within the trajectory. In doing so, ROVER facilitates more focused and accurate reasoning over temporally localized frame sequences without losing global context. We evaluate ROVER, implemented using an in-context learning approach, on diverse OpenX Embodiment videos and on a new dataset derived from RoboCasa that consists of 543 videos showing both expert and perturbed non-expert trajectories across 27 robotic manipulation tasks. ROVER outperforms strong baselines across three video reasoning tasks: task progress estimation, frame-level natural language reasoning, and video question answering. We observe that, by reducing the number of frames the model reasons over at each timestep, ROVER mitigates hallucinations, especially during unexpected or non-optimal moments of a trajectory. In addition, by enabling the implementation of a subtask-specific sliding context window, ROVER’s time complexity scales linearly with video length, an asymptotic improvement over baselines. Demos, code, and data available at: https://rover-vlm.github.io Figure 1: ROVER is a recursive framework for reasoning over camera video that decomposes a task into subtasks to maintain a compact temporal context, improving reasoning accuracy and efficiency. 1 Introduction Vision-language models (VLMs) have demonstrated remarkable generalization capabilities across a wide range of tasks, including image captioning, visual question answering, and grounding language in complex visual scenes [ 3 , 22 , 52 ] . These models excel at extracting high-level semantic understanding from visual inputs and can generate fluent natural language that reflects this understanding [ 18 , 56 ] . As a result, VLMs have become a key component in efforts to build general-purpose AI systems. Given their success, VLMs are increasingly viewed as promising foundations for embodied intelligence: systems that perceive, reason about, and act within the physical world [ 38 , 9 , 19 , 15 , 31 , 57 , 25 , 54 , 63 ] . Embodied tasks require VLMs to recognize and reason about what is happening at each moment in time (given a continuous stream of visual input from camera video) when attempting to complete a task. Existing frameworks for VLM reasoning in this setting either perform highly localized reasoning over a small set of frames [ 57 , 25 , 54 , 63 ] or attempt to reason over the entire sequence by concatenating all video frames into a single context [ 31 ] . Both strategies have limitations: the former sacrifices global context, while the latter is computationally expensive and overwhelms the model with irrelevant or redundant visual input. In this work, we seek to advance VLM capabilities in embodied settings by enabling high-precision frame-by-frame reasoning, without compromising efficiency or losing global context for the full task trajectory. With this motivation, we propose R easoning O ver V id E o R ecursively (ROVER). ROVER is a framework that enables the model to recursively decompose the task corresponding to the given video input, reducing the number of camera frames the VLM must reason over at any given moment during the trajectory (Figure 1 ). Instead of generating a long, single line of reasoning spanning all timesteps of the video input for a task attempt (e.g., opening a door), ROVER decomposes the task and generates a separate line of reasoning for each subtask (e.g., grasping the door handle). When a subtask is complete, the corresponding line of reasoning terminates and a new one is created for the video input for the next subtask (e.g., pulling the door open). We show that the decomposition not only improves accuracy by focusing the reasoning on relevant temporal segments, but also enables the implementation of a subtask-specific sliding context window, which further reduces the number of frames the model must reason over at each moment of a trajectory. We evaluate ROVER, implemented using an in-context learning approach, in the setting of robotic manipulation tasks using a large-scale dataset of videos collected from robot-mounted and third-person camera viewpoints during both successful and unsuccessful task attempts. We create this dataset by automatically perturbing expert demonstrations collected in RoboCasa [ 39 ] to produce diverse trajectories, ranging from near-optimal to fully random action sequences. In addition, we compute ground-truth task progress estimates at each timestep based on geometric distance to goal states. The generated dataset includes 543 videos across 27 tasks, each collected in a random kitchen scene. We leverage this dataset to evaluate ROVER across three benchmarks for embodied reasoning over camera video: 1) frame-level task progress estimation, 2) frame-level natural language reasoning, and 3) video question answering (QA). We also test ROVER on previous benchmarks based on diverse real-world OpenX Embodiment videos [ 41 ] . ROVER outperforms baselines across all settings: it achieves a stronger correlation with ground-truth task progress, lower frame-level reasoning error rates, and higher video QA accuracy. Specifically, we observe that ROVER reduces model hallucinations, especially during moments of a trajectory that deviate from expected or optimal behavior. We show these types of hallucinations during non-optimal states are most likely to occur when the model is reasoning over long sequences of camera frames. ROVER mitigates these hallucinations by reducing the number of frames the model must reason over at each timestep. In addition, by leveraging a sliding window within each subtask, ROVER’s time complexity scales linearly with video length, an asymptotic improvement over baselines. Overall, the contributions of this work including the following: • We propose ROVER , a recursive framework that improves video reasoning accuracy and efficiency in embodied settings. • We introduce a protocol to generate diverse non-expert trajectories with ground-truth progress labels from expert demonstrations in simulation. • We release a large-scale video dataset featuring both expert and non-expert robotic task executions, annotated with natural language descriptions and fine-grained task progress signals. • We establish a video reasoning benchmark that evaluates models on frame-level progress estimation, frame-level natural language reasoning, and video QA across diverse task trajectories. 2 Problem setup We model tasks as goal-conditioned finite-horizon partially observed Markov decision processes: \displaystyle\mathcal{M}:=\langle S,A,P,R,\Omega,O,T,G\rangle. (1) The tuple \langle S,A,P,R,G\rangle forms a Markov Decision Process, where S is the (unobserved) state space, A is the action space, P:S{\times}A\rightarrow\Delta(S) is the state-transition function and R:S{\times}G\rightarrow\mathbb{R} is the reward function. An agent observes the world through O:S\rightarrow\Delta(\Omega) , the observation function, where \Omega is the set of observations. Finally, T is the time horizon and G specifies the task semantically. Conditioned on a history h_{t}=(a_{<t},o_{{\leq}t})\in H and a goal g\in G , we aim to find a policy \pi:H{\times}G\rightarrow A that maximizes the \gamma -discounted return J_{\pi}=\mathbb{E}_{\pi}\left[\sum_{t=0}^{T-1}\gamma^{t}R(s_{t},a_{t},g)\right] . The goal-conditioned value function V^{\pi} is defined as the expected cumulative reward when