Test-Time Adaptation for Generalizable Task Progress Estimation
Christos Ziakas, Alessandra Russo · 2025
Read Test-Time Adaptation for Generalizable Task Progress Estimation by Christos Ziakas, Alessandra Russo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10085.txt.
Introduction
Christos Ziakas and Alessandra Russo wrote this paper in two thousand twenty-five. Its text presents VITA, a zero-shot value gauge for robot tasks that adapts itself at test time, built on a vision and language model.
Such models can grade how close a scene is to a goal, but their frozen codes cap their reach and their sense of time. VITA takes one gradient step at run time on a self-taught loss, and each step sharpens the grade. Steps stack along a run, so past events soak into the weights. A sampling trick that picks unlike clips curbs shortcut learning. Trained in one room, it grades tasks in new rooms and on new robot forms, and its grades can shape offline training rewards.
Catalog record
| CREATOR | Christos Ziakas, Alessandra Russo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-10085
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test-Time Adaptation for Generalizable Task Progress Estimation Christos Ziakas, Alessandra Russo Abstract We propose a test-time adaptation method that enables a progress estimation model to adapt online to the visual and temporal context of test trajectories by optimizing a learned self-supervised objective. To this end, we introduce a gradient-based meta-learning strategy to train the model on expert visual trajectories and their natural language task descriptions, such that test-time adaptation improves progress estimation relying on semantic content over temporal order. Our test-time adaptation method generalizes from a single training environment to diverse out-of-distribution tasks, environments, and embodiments, outperforming the state-of-the-art in-context learning approach using autoregressive vision-language models. 1 Introduction Vision-Language Models (VLMs) have demonstrated remarkable generalization capabilities across diverse tasks and domains, learning from large-scale, unstructured web data without human supervision ( Radford et al. 2021 ; Alayrac et al. 2022 ) . In contrast, despite significant advances in learning generalist policies for robotics ( Brohan et al. 2023 ; Ghosh et al. 2024 ) and 3D virtual environments ( Reed et al. 2022 ; Team et al. 2024 ) , state-of-the-art methods in these domains have yet to achieve comparable success due to their reliance on expert demonstrations, limiting their scalability. World models have emerged as a promising solution ( Ha & Schmidhuber 2018 ; Hafner et al. 2021 ) , enabling agents to learn in simulated environments. Despite the potential of world models to generate realistic visual trajectories in diverse, open-ended environments ( Hughes et al. 2024 ; Silver & Sutton 2025 ) , a critical challenge remains: How can agents effectively learn from videos at scale without relying on human supervision? One promising approach is to learn a generalizable goal-conditioned value function directly from expert visual trajectories and their corresponding task descriptions, using it as a reward and supervision signal for reinforcement learning and imitation learning, respectively ( Chen et al. 2021 ; Ma et al. 2022 ) . In this framework, goal-conditioned value estimation has been formulated as task progress estimation : predicting the extent to which an agent has advanced toward completing a task, based on visual observations and a natural language task description ( Dashora et al. 2025 ) . Prior work on progress estimation has employed contrastive VLMs ( Radford et al. 2021 ) , leveraging the similarity between a task description and a visual observation in their shared multimodal representation. However, these methods do not incorporate temporal context from the visual trajectory during inference ( Ma et al. 2023 ; Baumli et al. 2023 ; Rocamonde et al. 2024 ) . In contrast, autoregressive VLMs ( Alayrac et al. 2022 ) incorporate temporal context by conditioning on the full visual trajectory within the prompt; however, the state-of-the-art approach ( Ma et al. 2024 ) discards temporal order by shuffling the trajectory in inference to mitigate a bias toward monotonically increasing predictions, inherited from chronologically ordered datasets used to pre-train VLMs. Both existing VLM architectures rely on pre-trained representations and in-context learning for generalization, which limits their robustness in tasks that require temporal reasoning ( Pătrăucean et al. 2023 ) . In this work, we propose a method for training a progress estimation network that adapts to both visual and temporal contexts in unseen environments, using sequences of multimodal representations derived from expert visual trajectories and their corresponding task descriptions. At test time, instead of relying on similarity-based approaches or in-context learning, our method updates the parameters of the progress estimation network to adapt to context before making task progress predictions. As the network sequentially updates its parameters during inference, it captures temporal context, forming an implicit memory of the past ( Sun et al. 2024 ) . To mitigate reliance on temporal cues in progress estimation ( Xu et al. 2024 ; Singh et al. 2021 ) , we propose a gradient-based meta-learning strategy ( Finn et al. 2017 ; Sun et al. 2020 ) that trains our model for test-time adaptation by optimizing the self-supervised loss over diverse sub-trajectories selected via dissimilarity-based sampling. 2 Methodology 2.1 Problem Formulation In video trajectories, task progress estimation is equivalent to learning a goal-conditioned value function under reward functions that measure task completion. Therefore, we formulate task progress estimation as the problem of learning a goal-conditioned value function that predicts the degree of task completion from visual-language representations. Formally, the value function is defined as: V:\mathcal{O}\times\mathcal{G}\rightarrow[0,1] which maps an observation o_{t}\in\mathcal{O} and a goal specification g\in\mathcal{G} to a scalar value indicating the predicted progress toward goal completion, where V(o_{t};g)=0 corresponds to the start of the task and V(o_{t};g)=1 to task completion. Task progress is often aligned with temporal position in expert demonstrations, based on the assumption that such trajectories exhibit monotonically increasing progress toward goal completion ( Lee et al. 2021 ; Ma et al. 2024 ; Dashora et al. 2025 ) . Given an expert trajectory \tau=(o_{1},\ldots,o_{T})\sim\pi_{E} , temporal progress is defined using normalized timestep indices: V^{\pi_{E}}(o_{t};g)=\frac{t}{T} where T is the trajectory length. This provides supervision for learning a goal-conditioned value function V , which can generalize beyond expert data to estimate task progress. 2.2 Model Architecture Figure 1 : Test-Time Adaptation for Progress Estimation. 1) At each timestep, multimodal representation \boldsymbol{x}_{t} , and its context window \mathcal{W}_{\text{ctx}}=\{\boldsymbol{x}_{t-k},\ldots,\boldsymbol{x}_{t}\} , serve as input to adaptation module f_{\text{adapt}} . 2) f_{\text{adapt}} is updated using self-supervised loss over \mathcal{W}_{\text{ctx}} . 3) The adapted representation is passed through frozen MLP head h to produce scalar task progress score. Our method comprises three modules: (1) a multimodal encoder that extracts visual-language representations from visual trajectories and task descriptions; (2) a test-time adaptation module updated with a learned self-supervised objective; and (3) a regression head that predicts task completion. 2.2.1 Multimodal Input Representation We use a frozen contrastive vision-language encoder, CLIP ( Radford et al. 2021 ) , to extract representations from visual observations and goal descriptions. Given a visual trajectory \tau=(o_{1},\ldots,o_{T}) and a language task description g , we concatenate their representations at each timestep t to form a sequence of joint multimodal representations (x_{1},x_{2},\ldots,x_{T}) , where x_{t}=[\phi_{v}(o_{t});\phi_{g}(g)]\in\mathbb{R}^{2d} and \phi_{v} and \phi_{g} denote the visual and language encoders, respectively. CLIP is pre-trained with a contrastive objective to align paired image-text inputs in a shared representation space. As a result, representations of visual observations that are semantically closer to goal completion tend to be closer to the representations of the goal description. 2.2.2 Test-Time Adaptation To adapt the multimodal input representation to both visual and temporal context, we employ an adaptation module f_{\text{adapt}} following the test-time training paradigm ( Sun et al. 2020 ) . At inference time, the parameters of f_{\text{adapt}} are updated online using the self-supervised loss \ell_{\text{self}} , based on the test trajectory. We learn a reconstruction objective based on linear projections as a self-supervised task optimized for task progress estimation in training ( Sun et al. 2024 ) . In particular, f_{\text{adapt}} is trained to reconstruct a projected target representation from a corrupted input generated by the projection matrices P_{V}\in\mathbb{R}^{d^{\prime}\times d} and P_{K}\in\mathbb{R}^{d^{\prime}\times d} , respectively. At each timestep t , the model receives a sliding window of size k+1 with the local representations \mathcal{W}_{ctx}=\{x_{t-k},\ldots,x_{t}\} , and performs parameter updates for a number of epochs t_{\text{ep}} by