Test-Time Training for Visual Foresight Vision-Language-Action Models
Sangwu Park, Wonjoong Kim, Yeonjun In, Sein Kim, Hongseok Kang, Chanyoung Park · 2026
Read Test-Time Training for Visual Foresight Vision-Language-Action Models by Sangwu Park, Wonjoong Kim, Yeonjun In, Sein Kim, Hongseok Kang, Chanyoung Park (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08215.txt.
Introduction
Sangwu Park, Wonjoong Kim, Yeonjun In, Sein Kim, and coauthors make foresight-based robot policies adapt at test time. Visual Foresight VLA models first predict a future image and then choose actions from it, so distribution shift strikes twice: the prediction degrades and the action built on it follows.
Their method uses the model's own predicted future image and the observation that arrives later as a natural training pair, updating the model as it runs. An adaptive filter blocks careless updates that could do harm. In experiments the approach reduced the out-of-distribution weakness at modest extra inference cost, with no change to the architecture and no auxiliary data.
Catalog record
| CREATOR | Sangwu Park, Wonjoong Kim, Yeonjun In, Sein Kim, Hongseok Kang, Chanyoung Park |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08215
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test-Time Training for Visual Foresight Vision-Language-Action Models Sangwu Park, Wonjoong Kim, Yeonjun In, Sein Kim, Hongseok Kang, Chanyoung Park Abstract Visual Foresight VLA (VF-VLA) has become a prominent architectural choice in the recent VLA due to its impressive performance. Nevertheless, the inherent design of VF-VLA makes it particularly vulnerable to out-of-distribution (OOD) shifts. Because the quality of action directly depends on the accuracy of the predicted future visual information, OOD conditions affect both stages at once. To address this vulnerability, we propose Test-Time Training Visual Foresight VLA (T 3 VF), a test-time training approach motivated by the observation that the predicted future image and its subsequent observation form a natural supervision pair. To further address the practical challenges that arise from indiscriminate test-time updates, we introduce an adaptive update filtering mechanism. Empirically, T 3 VF mitigates the OOD vulnerability of VF-VLA at a modest additional inference cost, without requiring any architectural modification or auxiliary modules 1 1 1 Our source code is available at https://github.com/sangwu99/T3VF.git . . 1 Introduction Vision-Language-Action (VLA) models have become a central paradigm for generalist robotic manipulation ( 5 ; 8 ; 2 ) . Among them, a recent line of research adopts a two-stage formulation in which the model first predicts the future visual state that the robot is expected to reach and then generates actions conditioned on this prediction. Models that adopt this formulation are called Visual Foresight VLA (VF-VLA), and their impressive performance has made this design one of the prominent architectural choices in the recent VLA literature ( 14 ; 12 ; 10 ; 3 ; 13 ; 11 ) . Despite the effectiveness of VF-VLA, its inherent design makes it particularly vulnerable to out-of-distribution (OOD) shifts. Because the quality of action generation directly depends on the accuracy of the predicted future visual information ( 14 ) , OOD conditions affect both the visual prediction stage and the action generation stage at once. As summarized in Fig. 1 , recent VF-VLA such as WorldVLA ( 3 ) and Mantis ( 11 ) show substantial performance drops on LIBERO-Plus ( 4 ) compared to their in-distribution success rates on LIBERO ( 6 ) . This support the dual-stage exposure of both pathways under OOD shifts. Figure 1 : Performance comparison on LIBERO (In-Distribution) and LIBERO-Plus (Out-of-Distribution) To address this vulnerability, we propose T est- T ime T raining V isual F oresight VLA (T 3 VF ), a test-time training approach motivated by a simple intuition that the predicted future image and its later observation form a natural self-supervision pair. In VF-VLA, at a given step the model predicts a future image from the current observation, and after executing the corresponding action, the actual image at that future step is observed. This observed image can be regarded as an oracle for the earlier prediction, allowing the test-time training to improve visual prediction. Accordingly, action generation also improves through the dependence. However, this approach faces two practical challenges. First, a discrepancy between a prediction and its oracle does not always indicate a useful supervision signal. Such discrepancy can result from an inaccurate visual prediction or from an action-side error despite a correct prediction, which serves as noise during training. Second, the difficulty varies both across episodes and within an episode. Even with a proxy metric that can separate the two error sources discussed above, applying a fixed threshold cannot reflect this variation. Under such threshold, most steps in easier episodes or segments pass and allow excessive noise to enter training. In contrast, most steps in harder ones are skipped and fail to provide training signal. To address the first challenge, we use the variance of the action at each step as the proxy metric. A low variance indicates that the model is internally consistent about the action it intends to execute. Such consistency makes a prediction error more plausibly attributable to the visual pathway and therefore a useful signal for updating the model. In contrast, high variance indicates that the source of the error cannot be reliably attributed. Since such an error may not provide a useful supervision signal, the corresponding step is skipped. To address the second challenge, we replace the fixed threshold with an adaptive variance buffer that maintains a running window of recent variances and permits an update only when its variance falls within the lower range of this buffer. Because this criterion is based on relative ranking rather than an absolute cutoff, it adapts naturally to the scale of variance in each episode as well as to its fluctuation between steps. The main contributions of this work are as follows: • First identification of OOD vulnerability in VF-VLA. We identify the amplified OOD vulnerability arising from the dual-stage exposure inherent to VF-VLA. • Test-time training with predicted-attained pairs. We propose a test-time training approach leverages the correspondence between predicted images and their later observations, complemented by an adaptive update filtering. • Empirical effectiveness. T 3 VF improves the average success rate on LIBERO-Plus by 5% over the base VF-VLA, demonstrating that it mitigates the OOD vulnerability. 2 Related Works Visual Foresight VLA refer to VLA models that predict a future image to be reached by the robot from the current observation and language instruction, and condition action generation on this prediction. The first group formulates future visual pixels and actions as autoregressive token sequences on a shared vocabulary ( 14 ; 12 ; 10 ; 3 ) . The second group predicts future visual information as compressed latent features or implicitly aligns latent representations with future visual states, avoiding the redundancy and computational overhead of autoregressive pixel-token generation. ( 11 ; 13 ) . Both groups share a common structure: action generation is conditioned on predicted visual information. Consequently, both the visual prediction stage and the action generation stage are exposed to OOD shifts ( 9 ; 4 ) , leading to amplified vulnerability under such shifts. To the best of our knowledge, this vulnerability of VF-VLA remains unaddressed. Another line of work adapts VLA models through test-time reinforcement learning. ( 1 ; 7 ) . They differ from ours in that they require a separate reward model, incur substantial overhead from online RL, and target VLA models in general rather than VF-VLA. For complete related work, please refer to Appendix A . 3 Method 3.1 Preliminaries A VF-VLA consists of a VLM backbone P , an image head I_{h} , and an action head A_{h} . Given the language instruction l , the current observation o_{t} , and a placeholder q for any learnable query tokens, 2 2 2 The specific forms of placeholder q , \mathcal{L}_{\mathrm{img}} and \mathcal{L}_{\mathrm{act}} depend on the design choices of each VF-VLA implementation. backbone extracts following representations (h_{t}^{\mathrm{inst}},\,h_{t}^{\mathrm{img}},\,h_{t}^{\mathrm{act}})\;=\;P(l,\,o_{t},\,q). (1) This step computes h_{t}^{\mathrm{act}} conditionally on h_{t}^{\mathrm{img}} following the autoregressive nature of the VLM backbone, which discussed in Sec. 1 . The image head predicts future observation \hat{o}_{t+n} at a fixed gap of n steps ahead, and the action head generates the action \hat{a}_{t} , \hat{o}_{t+n}\;=\;I_{h}\!\left([h_{t}^{\mathrm{inst}},h_{t}^{\mathrm{img}}],\,o_{t}\right),\quad\hat{a}_{t}\;\sim\;A_{h}(h_{t}^{\mathrm{act}}). (2) During training, the model is optimized through two objectives 2 , {}\mathcal{L}_{\mathrm{train}}\;=\;\mathcal{L}_{\mathrm{img}}(\hat{o}_{t+n},\,o_{t+n})\;+\;\lambda\,\mathcal{L}_{\mathrm{act}}(\hat{a}_{t},\,a_{t}), (3) where o_{t+n} is the future ground-truth observation, a_{t} is the demonstration action, and \lambda balances