Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
Huihan Liu, Changyeon Kim, Bo Liu, Minghuan Liu, Yuke Zhu · 2026
Read Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning by Huihan Liu, Changyeon Kim, Bo Liu, Minghuan Liu, Yuke Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03818.txt.
Introduction
Huihan Liu and four colleagues find that big pretrained robot brains forget far less than small ones trained from nothing. Continual learning, adding skills without erasing old ones, is a classic robot headache, studied mostly in small imitation policies. Their 2026 paper tests modern pretrained vision-language-action models and finds the dynamics transformed: the pretrained models are remarkably resistant to catastrophic forgetting.
Plain experience replay, the simplest remedy, works startlingly well on them, sometimes producing zero forgetting with only a small replay buffer. Pretraining itself deserves the credit, pairing low forgetting with strong learning of the new skill. Even when performance on an old task dips mid-training, the knowledge is not gone: brief fine-tuning recovers the seemingly lost skill rapidly. Scale, the authors conclude, changes continual learning from damage control into accumulation.
Catalog record
| CREATOR | Huihan Liu, Changyeon Kim, Bo Liu, Minghuan Liu, Yuke Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49467 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03818
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning Huihan Liu, Changyeon Kim, Bo Liu, Minghuan Liu, Yuke Zhu Abstract Continual learning is a long-standing challenge in robot policy learning, where a policy must acquire new skills over time without catastrophically forgetting previously learned ones. While prior work has extensively studied continual learning in relatively small behavior cloning (BC) policy models trained from scratch, its behavior in modern large-scale pretrained Vision-Language-Action (VLA) models remains underexplored. In this work, we found that pretrained VLAs are remarkably resistant to forgetting compared with smaller policy models trained from scratch. Simple Experience Replay (ER) works surprisingly well on VLAs, sometimes achieving zero forgetting even with a small replay data size. Our analysis reveals that pretraining plays a critical role in downstream continual learning performance: large pretrained models mitigate forgetting with a small replay buffer size while maintaining strong forward learning capabilities. Furthermore, we found that VLAs can retain relevant knowledge from prior tasks despite performance degradation during learning new tasks. This knowledge retention enables rapid recovery of seemingly forgotten skills through finetuning. Together, these insights imply that large-scale pretraining fundamentally changes the dynamics of continual learning, enabling models to continually acquire new skills over time with simple replay. Code and more information can be found at project website . Figure 1 : Comparison of continual learning performance between a pretrained Vision-Language-Action (VLA) model ( GR00T N1.5 ; 32 ) and a non-pretrained small policy model ( BC-Transformer ; 25 ). Each checkpoint corresponds to a model obtained by sequentially training over ten tasks under Experience Replay (ER), where the parameters at the start of training for checkpoint i are initialized from checkpoint i\!-\!1 . Each matrix entry (i,j) denotes the success rate on Task j after training on Task i . The columns track how a given task performance evolves as training continues (top to bottom). We compare a pretrained VLA model (top) with a non-pretrained small BC policy (bottom) across multiple LIBERO benchmark suites. 1 Introduction Continual learning ( 10 ; 30 ; 25 ) remains a long-standing challenge in robot policy learning, which requires robot policies to acquire new skills over time while preserving previously learned behaviors. In contrast to learning multi-task policies at once ( 22 ; 14 ; 5 ; 4 ) , continual learning requires the policy to balance the ability of learning new tasks (plasticity) and preserving previously acquired knowledge (stability) ( 10 ; 30 ) . The learning is non-trivial, as the policy tends to fail and result in catastrophic forgetting ( 10 ; 30 ; 19 ; 28 ) , where the learning of new skills often leads to large degradation in previously learned behaviors, making naive sequential finetuning ineffective ( 41 ) . Prior work on continual learning in robotics has primarily focused on relatively small policy models trained from scratch or with limited pretraining ( 25 ; 7 ; 19 ; 29 ; 43 ) . In these settings, catastrophic forgetting is often pervasive ( 25 ) , and people typically require the use of large replay buffers ( 7 ) or carefully designed regularization techniques ( 19 ) for mitigation. Recently, large-scale pretrained Vision-Language-Action models (VLAs) have demonstrated strong generalization and downstream task transfer in robotic manipulation. Pretrained on diverse multimodal data ( 37 ; 42 ; 35 ) and robot trajectories, these models rely less on task-specific parameter updates and instead adapt by reusing and reconfiguring existing representations. This raises a central question: do large pretrained VLAs behave differently in continual learning from the smaller models, and if so, in what ways? Our answer follows by observing that pretrained VLAs exhibit vastly different dynamics from smaller policy models in continual learning. In particular, we conduct a comprehensive empirical study that reveals a set of unexpected dynamics of VLAs in continual learning: Pretrained VLAs are remarkably resistant to forgetting compared with smaller policy models trained from scratch (Sec. 2 ). Specifically, we found that simple Experience Replay (ER) works surprisingly well on VLAs, often achieving zero forgetting even with a small replay data size ( e.g. , 2\% of training data in LIBERO benchmark suites). Furthermore, sometimes it even enables positive backward transfer on previously learned tasks, i.e. , further improving the performance of previous tasks with only the replay data stored. These dynamics of learning have not been seen before on smaller models. Pretraining plays an integral role in improving continual learning performance in both forward and backward transfer (Sec. 4 ). We found that the pretrained knowledge in VLAs is especially effective for mitigating forgetting even at a small replay dataset size ( e.g. , 2\% of training data). Furthermore, pretraining reduces forgetting while still maintaining a high success rate on new tasks, avoiding the trade-off between preserving knowledge and forward transfer ability. VLAs preserve relevant knowledge of prior tasks despite performance degradation from learning new ones (Sec. 5 ). Even when performance on previous tasks seems to degrade from catastrophic forgetting, the underlying knowledge is still preserved in the VLA’s internal representation. Evidence for this lies in the fact that a few finetuning steps can quickly restore performance on past tasks. 2 Preliminaries Table 1 : Continual learning performance on LIBERO benchmarks : Average Success Rate (SR \uparrow ) and Negative Backward Transfer (NBT \downarrow ). Results are with sample size = 1000 (20% of dataset size per task), in accordance with LIBERO’s official setting. BC-DP , BC-T , BC-ViT are abbreviations for BC-Diffusion Policy , BC-Transformer , BC-ViT respectively. LIBERO-Spatial LIBERO-Object LIBERO-Goal LIBERO-10 Average Method SR ( \uparrow ) NBT ( \downarrow ) SR ( \uparrow ) NBT ( \downarrow ) SR ( \uparrow ) NBT ( \downarrow ) SR ( \uparrow ) NBT ( \downarrow ) SR ( \uparrow ) NBT ( \downarrow ) Pi0 0.879\pm 0.008 0.019\pm 0.019 0.897\pm 0.011 -0.011\pm 0.034 0.732\pm 0.015 -0.005\pm 0.010 0.563\pm 0.028 -0.068\pm 0.018 0.768\pm 0.017 -0.016\pm 0.022 GR00T 0.940\pm 0.013 0.007\pm 0.009 0.975\pm 0.004 0.019\pm 0.013 0.943\pm 0.004 0.023\pm 0.017 0.820\pm 0.017 0.059\pm 0.035 0.919\pm 0.011 0.027\pm 0.021 BC-DP 0.663\pm 0.074 0.050\pm 0.070 0.847\pm 0.069 0.025\pm 0.056 0.809\pm 0.089 0.230\pm 0.101 0.464\pm 0.024 0.201\pm 0.044 0.696\pm 0.068 0.127\pm 0.071 BC-T 0.659\pm 0.058 0.299\pm 0.096 0.595\pm 0.112 0.132\pm 0.120 0.709\pm 0.022 0.356\pm 0.042 0.376\pm 0.034 0.192\pm 0.019 0.585\pm 0.066 0.245\pm 0.080 BC-ViT 0.513\pm 0.069 0.171\pm 0.065 0.543\pm 0.268 0.077\pm 0.134 0.661\pm 0.024 0.319\pm 0.022 0.316\pm 0.062 0.204\pm 0.063 0.508\pm 0.142 0.193\pm 0.082 Figure 2 : Negative Backward Transfer (NBT) across different replay buffer sizes. Each subplot shows NBT as a function of replay buffer size ( \{0.2\%,2\%,20\%\} ) for all methods across the four benchmarks and their average. Shaded regions indicate \pm 1 standard deviation across seeds. Higher NBT indicates more forgetting; values near zero indicate no forgetting. Results and discussion for LIBERO-10 are reported separately in Tab. 5 in Appendix A.2 . Table 2 : Comparison with non-ER continual learning baselines (Sequential, EWC) on LIBERO benchmarks. LIBERO-Object LIBERO-10 Model Method SR ( \uparrow ) NBT ( \downarrow ) SR ( \uparrow ) NBT ( \downarrow ) Pi0 Sequential 0.910 0.696 0.644 0.562 EWC 0.910 0.608 0.622 0.543 ER 0.898 -0.007 0.586 -0.070 GR00T Sequential 0.964 0.752 0.852 0.758