CRL-VLA: Continual Vision–Language–Action Learning Qixin Zeng, Shuo Zhang, Hongyin Zhang, Renjie Wang, Han Zhao, Libang Zhao, Runze Li, Donglin Wang, Chao Huang Abstract Lifelong learning is critical for embodied agents in open-world environments, where reinforcement learning fine-tuning has emerged as an important paradigm to enable Vision-Language-Action (VLA) models to master dexterous manipulation through environmental interaction. Thus, Continual Reinforcement Learning (CRL) is a promising pathway for deploying VLA models in lifelong robotic scenarios, yet balancing stability (retaining old skills) and plasticity (learning new ones) remains a formidable challenge for existing methods. We introduce CRL-VLA, a framework for continual post-training of VLA models with rigorous theoretical bounds. We derive a unified performance bound linking the stability-plasticity trade-off to goal-conditioned advantage magnitude, scaled by policy divergence. CRL-VLA resolves this dilemma via asymmetric regulation: constraining advantage magnitudes on prior tasks while enabling controlled growth on new tasks. This is realized through a simple but effective dual-critic architecture with novel Goal-Conditioned Value Formulation (GCVF), where a frozen critic anchors semantic consistency and a trainable estimator drives adaptation. Experiments on the LIBERO benchmark demonstrate that CRL-VLA effectively harmonizes these conflicting objectives, outperforming baselines in both anti-forgetting and forward adaptation. † † affiliationnotice: Equal contribution 1 Introduction Reinforcement Learning (RL) post-training has become a central paradigm for aligning Vision–Language–Action (VLA) models with complex embodied reasoning and robotic manipulation tasks ( 4 ; 13 ) . By optimizing policies through interaction, RL post-training enables adaptive, goal-directed behaviors beyond what supervised learning can provide. Sustaining such adaptability requires lifelong learning across non-stationary task streams. However, in large-scale settings, Continual Reinforcement Learning (CRL) aims to equip agents with the ability to acquire new skills over time without catastrophic forgetting ( 28 ) . This problem fundamentally requires balancing scalability, stability, and plasticity under non-stationary task distributions ( 22 ) . Despite its importance, CRL for VLA models remains underexplored, particularly in robotic manipulation scenarios involving multi-modal observations and long-horizon dynamics ( 33 ; 22 ) . Most existing CRL methods enforce stability via parameter- or function-space regularization, but they face severe limitations when applied to modern VLA architectures. Experience replay ( 23 ; 1 ) reuses outdated transitions whose gradients often conflict with those of new tasks, inducing off-policy errors that destabilize continual learning. Second-order regularization methods ( 15 ) are computationally prohibitive due to the scale and dimensionality of multi-modal Transformers ( 13 ; 18 ) . Recent approaches favor lightweight constraints such as KL regularization ( 27 ) , but such transition-level constraints enforce only short-horizon behavioral similarity and fail to preserve long-term task performance ( 14 ) . Backbone-freezing strategies ( 7 ) hinge on the fragile assumption that pretrained multi-modal representations remain aligned with language goals; under task shifts, value learning drifts without explicit language conditioning, degrading adaptation and collapsing plasticity ( 10 ; 6 ) . In this work, we show that forgetting in continual VLA learning is fundamentally driven by the goal-conditioned advantage magnitude, which directly links policy divergence to performance degradation on prior tasks. This perspective reframes the stability–plasticity dilemma as an asymmetric regulation problem: suppressing advantage magnitudes on previous tasks to ensure stability, while permitting controlled growth on new tasks to enable plasticity. Based on this insight, we propose CRL-VLA , a continual learning framework that disentangles stable value semantics from adaptive policy learning via a novel Goal-Conditioned Value Formulation (GCVF) . Our method employs a dual-critic architecture in which a frozen critic anchors long-horizon, language-conditioned value semantics, while a trainable critic estimator drives adaptation. Stability is further enforced by combining trajectory-level value consistency with transition-level KL regularization, while Monte Carlo (MC) estimation and standard RL objectives support efficient learning on new tasks without catastrophic forgetting. Our contributions are summarized as follows: • We propose CRL-VLA , a principled continual post-training framework for VLA models that identifies the goal-conditioned advantage magnitude as the key quantity governing the stability–plasticity trade-off. • CRL-VLA operationalizes this insight via a dual-critic architecture with a novel goal-conditioned value formulation, enabling asymmetric regulation that preserves prior skills while efficiently adapting to new tasks. • Extensive experiments show that CRL-VLA consistently outperforms strong baselines on continual VLA benchmarks, achieving superior knowledge transfer and resistance to catastrophic forgetting. 2 Related work 2.1 Reinforcement Learning for VLA Models VLA models have gained interest as unified robotic manipulation policies, with recent work exploring online RL to address SFT limitations and improve generalization and long-horizon reasoning. SimpleVLA-RL ( 17 ) builds upon OpenVLA-OFT and GRPO, showing that online RL substantially improves long-horizon planning under demonstration scarcity. RIPT-VLA ( 29 ) applies the REINFORCE leave-one-out estimator to QueST and OpenVLA-OFT architectures. ARFM ( 37 ) enhances VLA action models via adaptive offline RL, balancing signal and variance to improve generalization and robustness. RLinf-VLA ( 35 ) implements a novel hybrid fine-grained pipeline allocation mode scalling up VLA model training with diverse RL algorithms and benchmarks. Recent work introduces IRL-VLA, a closed-loop RL framework for VLA autonomous driving that achieved award-winning performance in benchmarks ( 9 ) . In contrast, our work aims to investigate the knowledge transfer of VLA models and their ability to learn new tasks stably and continuously. 2.2 Continual Reinforcement Learning Continual reinforcement learning generally falls into three paradigms. Regularization-based methods constrain updates to important parameters using Fisher information ( 3 ) or path integrals ( 36 ) , or enforce behavioral consistency via KL divergence ( 27 ) . Replay-based strategies ( 23 ) mitigate forgetting by interleaving past experience, though recent benchmarks ( 31 ) highlight their scalability limits in high-dimensional robotics. Architecture-based approaches ( 33 ) prevent interference by expanding network capacity, a strategy often impractical for large-scale pretrained models. While limited works explore value preservation via distillation ( 26 ; 24 ) , they typically neglect the value drift inherent in non-stationary, language-conditioned tasks. Unlike these approaches, our work focuses on the continuous learning process of VLA models for general visual language manipulation tasks. 2.3 Continual learning in VLA model To date, there are no studies on continual learning specifically for VLA models, but similar research methods exist. Recent work ( 34 ) improves policy stability under task variations, providing useful insights for subsequent lifelong learning approaches, although it is evaluated in a non-continual setting. Recently, several works have extended these paradigms to the VLA domain. VLM2VLA ( 7 ) treats actions as a special language tokens and employs LoRA to align pre-trained VLMs with low-level robot control. Stellar-VLA ( 32 ) introduces a skill-centric knowledge space to evolve task representations continually across diverse robot manipulation scenarios. DMPEL ( 16 ) proposes a dynamic mixture of progressive parameter-efficient experts to achieve lifelong learning without high storage overhead. ChatVLA ( 39 ) addresses spurious forgetting by decoupling multi-modal understanding from high-frequency action execution. Unlike these work, we aim to investigate a continuous post-training framework for VLA models and explicitly model the stability-plasticity trade-off between new and old tasks. 3 Preliminaries We study continual post-training of VLA policies in a goal-conditioned reinforcement learning setting. Each task is formulated as a goal-conditioned MDP \mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{G},P,r,\gamma\rangle , where s\in\mathcal{S} is the state, g\in\mathcal{G} is a language instruction, and a\in\mathcal{A} is a robotic control command. For a given policy \pi and goal g , we define the state-value function as V^{\pi}(s,g)=\mathbb{E}_{\tau\sim\pi,g}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},g)|s_{0}=s] , which represents the expected cumulative return starting from state s . The action-value function is defined as Q^{\pi}(s,a,g)=\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},g)|s_{0}=s,a_{0}=a] . The advantage function A^{\pi}(s,a,g)=Q^{\pi}(s,a,g)-V^{\pi}(s,g) measures how much better an action is compared to the average action at that state. The state occupancy measure d_{g}^{\pi}(s)=(1-\gamma)\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}\mathbf{1}(s_{t}=s)] describes the distribution of states visited by policy \pi when executing goal g . Different policies induce different state visitation distributions, which is fundamental to understanding how policy updates affect task performance. The goal-conditioned policy is \pi(a\mid s,g) , with expected return for goal g given by J_{g}(\pi)=\mathbb{E}_{\tau\sim\pi,g}\Big[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},g)\Big] . Performance on old tasks under distribution p_{\mathrm{old}}(g) is J_{\mathrm{old}}(\pi)=\mathbb{E}_{g\sim p_{\mathrm{old}}}[J_{g}(\pi)] . New policy’s performance \pi^{\prime} on new tasks under distribution p_{\mathrm{new}}(g) is J_{\mathrm{new}}(\pi^{\prime})=\mathbb{E}_{g\sim p_{\mathrm{new}}}[J_{g}(\pi^{\prime})] . A pretrained VLA policy \pi_{\theta_{0}} is adapted sequentially to a task stream \mathcal{T}=\{\mathcal{T}_{1},\ldots,\mathcal{T}_{K}\} . At stage k , the agent interacts only with \mathcal{T}_{k} and updates to \pi^{\theta}_{k} via on-policy RL, without access to interaction data, rewards, or gradients from previous tasks \{\mathcal{T}_{i}\}_{i1 , we freeze the converged MC critic as the new GCV critic and initialize a fresh MC critic. PPO Loss Implementation and KL Constraint Following 25 , we optimize the clipped surrogate objective for each task to ensure stable policy updates. Specifically, we constrain the policy update by enforcing a KL divergence penalty between the updated policy \pi_{\theta} and the reference policy \pi^{\prime} , thereby adhering to the trust region principle. Building upon this framework, the surrogate constraint for policy divergence constrains D_{\mathrm{new}} . The overall PPO loss is then expressed as: \mathcal{L}_{\text{PPO}}(\theta)=\mathbb{E}_{(s,a,g)\sim\mathcal{B}_{\mathrm{new}}}\Bigg[\min\bigg(r_{\theta}(s,a,g)\hat{A}_{\theta}(s,a,g),\text{clip}\big(r_{\theta}(s,a,g),1-\epsilon,1+\epsilon\big)\hat{A}_{\theta}(s,a,g)\bigg)\Bigg], (14) where \epsilon is the clipping hyperparameter that restricts the probability ratio r_{\theta}(s,a,g)=\frac{\pi_{\theta}(a|s,g)}{\pi_{\mathrm{old}}(a|s,g)} . To strictly enforce the trust region, we implement an early stopping criterion based on the approximate KL divergence: D_{\mathrm{KL}}(\pi^{\prime}\|\pi_{\theta})\approx\mathbb{E}_{(s,g)\sim\mathcal{B}_{\mathrm{new}}}\left[\log\frac{\pi^{\prime}(a|s,g)}{\pi_{\theta}(a|s,g)}\right]\leq d_{\text{targ}}, (15) where d_{\text{targ}} is the target KL threshold . The advantage \hat{A}_{\theta} is computed using Generalized Advantage Estimation (GAE) ( schulman2015high ) : \hat{A}_{t}=\sum_{k=0}^{\infty}(\gamma\lambda)^{k}\delta_{t+k}^{V}, (16) where \gamma\in[0,1] is the discount factor , \lambda\in[0,1] is the GAE smoothing parameter , and \delta_{t}^{V}=r_{t}+\gamma V(s_{t+1})-V(s_{t}) denotes the TD error. B.2 Action-Value (Q-based) Implementation We extend the Dual-Critic Architecture and Goal-Conditioned Value Formulation detailed in the methodology to action-value functions. This extension applies the same principles of independent dimension management to Q(s,a,g) , enabling precise control over stability and plasticity in the action-value space. Goal-Conditioned Action-Value Formulation Consistent with the value function design, we leverage the shared Vision-Language Model’s embedding as the global state representation \phi(s,g) . To parameterize the action-value function Q(s,a,g) , we concatenate the continuous action vector a with the language-grounded state embedding \phi(s,g) before the Multi-Layer Perceptron processing. This joint encoding ensures that Q -values inherit the metric smoothness and semantic understanding from the pretrained VLM embedding space, addressing language-goal tracking issues while incorporating action information. Dual-Critic Architecture via Action State Value To operationalize the control of M_{\mathrm{old}} and M_{\mathrm{new}} within the Q-learning framework, we employ the Dual-Critic Architecture with critic decoupling: 1) Frozen GCV Q-critic \theta_{\text{GCV}} . Initialized from the old task’s converged Q-network and frozen during new task training. It serves as the anchor for stability, providing reference values Q_{\text{old}}(s,a,g) to regularize backbone updates on historical data. 2) Trainable MC Q-critic \theta_{\text{MC}} . Updated on new task trajectories using MC returns. This critic remains plastic to capture the reward structure of the new task, naturally bounding M_{\mathrm{new}} via the environment’s reward limits. Q-based Goal-Conditioned Value Consistency (Q-GCV) The Q-based GCV mechanism enforces consistency between the current backbone’s predictions (via the frozen GCV critic) and the historical Q-values stored in the replay buffer \mathcal{B}_{\text{old}} . Unlike the value function which targets MC returns, the Q-function targets the specific state-action values Q_{\text{old}} recorded from previous interactions. The loss is formulated as the Mean Squared Error (MSE): \mathcal{L}^{Q}_{\text{GCV}}(\phi)=\beta_{Q}\mathbb{E}_{(s,a,g)\sim\mathcal{B}_{\text{old}}}\left[\left\|Q_{\phi,\theta_{\text{GCV}}}(s,a,g)-Q_{\text{old}}\right\|^{2}\right]. (17) This objective constrains the shared backbone \phi to maintain representations that, when passed through \theta_{\text{GCV}} , reconstruct the historical action values, thereby minimizing performance degradation on old tasks ( M_{\mathrm{old}} ). Q-based Monte Carlo Learning To exploit the plasticity dimension ( M_{\mathrm{new}} ), the Trainable MC Q critic is optimized to fit the returns G_{t}^{g_{\text{new}}} from the current policy on the new task: \mathcal{L}^{Q}_{\text{MC}}(\phi,\theta_{\text{MC}})=\mathbb{E}_{(s,a,g)\sim\mathcal{B}_{\text{new}}}\left[\left\|Q_{\phi,\theta_{\text{MC}}}(s,a,g)-G_{t}^{g_{\text{new}}}\right\|^{2}\right]. (18) This allows the shared backbone to learn features necessary for the new task without being overly inhibited by the frozen GCV critic, as the optimization of \theta_{\text{MC}} drives the embedding towards relevant new structures. B.3 Training Procedure and Hyperparameters All experiments are conducted based on OpenVLA-OFT ( 13 ) , where the official pretrained checkpoints are adopted as the base model and LoRA adapters are applied for efficient task adaptation. Our implementation is built upon the RIPT-VLA codebase ( 29 ) , and all unspecified training configurations strictly follow the settings reported in prior work. Training is performed with a LoRA rank of 32, and the detailed hyperparameter configuration is summarized in Tab. 7 . Unless otherwise stated, all hyperparameters are kept consistent across experiments to ensure fair comparison. All experiments are executed on the following hardware platform: CPU: Intel(R) Xeon(R) Platinum 8358 @ 2.60GHz; GPU: NVIDIA A100-SXM4-80GB. Table 3 : More performance comparison on the multi-task learning scenario. Specific task settings in Appendix C.3 . \uparrow denotes higher is better, while \downarrow denotes lower is better. The best and second best results are highlighted accordingly. Method FAR (\uparrow) BWT (\uparrow) FT (\uparrow) SL ( 20 ) 0.60 0.00 -0.70 MTL ( 20 ) 0.54 0.17 -0.70 ER ( 21 ) 0.52 -0.05 -0.65 LWF ( 19 ) 0.43 0.00 -0.65 CRL-VLA (V) 0.63 -0.10 -0.70 CRL-VLA (Q) 0.33 -0.45 -0.70 Table 4 : Comparison of continual learning metrics under different parameter settings (Only \mathcal{L}^{V}_{\mathrm{GCV}} , Only \mathcal{L}^{V}_{\mathrm{GCV}} , and KL), where all experiments are conducted on the Task-4 benchmark (Appendix C.4 ). Parameter Value FAR (\uparrow) BWT (\uparrow) Forgetting (\downarrow) FT (\uparrow) \mathcal{L}^{Q}_{\mathrm{GCV}} 0.00001 0.97 -0.02 0.02 -0.13 0.0001 0.94 -0.05 0.05 -0.09 0.001 0.93 -0.05 0.05 -0.06 0.01 0.98 0.02 0.00 -0.09 0.1 0.93 0.00 0.02 -0.06 \mathcal{L}^{V}_{\mathrm{GCV}} 0.00001 0.91 -0.05 0.06 -0.06 0.0001 0.91 0.00 0.00 -0.19 0.001 0.98 0.00 0.00 -0.06 0.01 0.94 -0.06 0.06 -0.03 0.1 0.93 -0.03 0.02 -0.06 KL 0 0.91 -0.03 0.03 -0.03 0.000001 0.95 -0.02 0.02 -0.09 0.00001 0.98 -0.02 0.02 -0.06 0.0001 0.95 -0.02 0.02 -0.11 0.001 0.91 -0.05 0.05 -0.03 0.01 0.96 0.02 0.02 -0.06 0.1 0.90 -0.02 0.06 -0.09 Table 5 : Ablation study on the effects of the \mathcal{L}^{V}_{\mathrm{MC}} loss coefficient and the \mathcal{L}^{Q}_{\mathrm{MC}} loss coefficient on performance and stability. All experiments are conducted on the Task-4 benchmark (Appendix C.4 ). Parameter Coef FAR (\uparrow) BWT (\uparrow) Forgetting (\downarrow) FT (\uparrow) \mathcal{L}^{V}_{\mathrm{MC}} 10 0.93 0.00 0.02 -0.06 1 0.95 -0.02 0.02 -0.03 0.1 0.93 -0.03 0.03 -0.06 0.01 0.90 -0.05 0.05 -0.13 0.001 0.85 -0.08 0.08 -0.13 \mathcal{L}^{Q}_{\mathrm{MC}} 10 0.98 0.05 0.00 -0.06 1 0.99 0.05 0.00 0.00 0.1 0.94 0.03 0.00 -0.06 0.01 0.85 -0.06 0.09 -0.06 0.001 0.92 -0.03 0.08 -0.06 Table 6 : Ablation study of the Goal-Conditioned Value (GCV) network on model performance and stability. All experiments are conducted on the Task-4 benchmark (Appendix C.4 ). Enabling GCV leads to higher success rates (FAR) and significantly reduced forgetting. Used GCV FAR (\uparrow) BWT (\uparrow) Forgetting (\downarrow) Forward Transfer (\uparrow) False 0.29 -0.06 0.06 -0.72 True 0.31 -0.02 0.02 -0.70 Appendix C Task and Experiment Details To systematically evaluate the robustness and scalability of our method, we construct multiple benchmarks by randomly sampling tasks from a shared task pool. Each benchmark contains a different number of object manipulation tasks, serving as controlled evaluation settings with varying task complexity and combinatorial difficulty. All benchmarks are fixed across methods and runs to ensure fair comparison. Specifically, we evaluate our method on a subset of the LIBERO benchmark suite, consisting of tasks randomly sampled from the original collection to encompass various task cardinalities. We denote these benchmarks as Task-1 through Task-4 . Each benchmark comprises multiple task sets defined by natural language directives, primarily focusing on robotic pick-and-place behaviors. This setup allows us to rigorously assess the model’s instruction-following capabilities and its generalization across diverse task scales. we next detail the language instructions used for each benchmark. Each task consists of one or more natural language directives. C.1 Task-1 Benchmark • Task 0 : “pick up the black bowl from table center and place it on the plate” • Task 1 : “pick up the black bowl next to the ramekin and place it on the plate” • Task 2 : “pick up the black bowl on the cookie box and place it on the plate” C.2 Task-2 Benchmark • Task 0 : “pick up the orange juice and place it in the basket”, “pick up the chocolate pudding and place it in the basket”, “pick up the bbq sauce and place it in the basket” • Task 1 : “pick up the cream cheese and place it in the basket”, “pick up the milk and place it in the basket”, “pick up the tomato sauce and place it in the basket” C.3 Task-3 Benchmark • Task 0 : “pick up the bbq sauce and place it in the basket”, “pick up the chocolate pudding and place it in the basket” • Task 1 : “pick up the ketchup and place it in the basket”, “pick up the salad dressing and place it in the basket” C.4 Task-4 Benchmark • Task 0 : “put the wine bottle on the rack”, ”put the cream cheese in the bowl” • Task 1 : “push the plate to the front of the stove”, ”put the bowl on the stove.” • Task 2 : “put the wine bottle on top of the cabinet”, ”open the top drawer and put the bowl inside.” C.5 Baselines 1) Sequential Learning (SL) ( 20 ) : This is the most straightforward continual learning method, where the VLA model is simply fine-tuned on new tasks without any specific mechanisms to prevent forgetting. 2) Multi-Task Learing (MTL) ( 20 ) : This baseline algorithm learns from both new and old tasks simultaneously. 3) Learning Without Forgetting (LWF) ( 19 ) : This algorithm preserves existing knowledge without requiring the old data by having the new model imitate the old model’s predictions on new data during training (knowledge distillation). 4) Experience Replay (ER) ( 21 ) : It is a classic example of combining reinforcement learning and supervised learning, explicitly using episode memory to store samples from past tasks and converting them into gradient constraints to prevent the gradients of new tasks from degrading the performance on old tasks. Table 7 : Training hyperparameter configuration. Parameter Value General LoRA Rank 32 Gradient Accumulation Steps 1 PPO Epochs 1 PPO Clip Range 0.2 PPO Clip High 0.2 Max Step Batch Size 2 Learning Rate (LoRA modules) 1.0\times 10^{-4} Learning Rate (Action head) 5.0\times 10^{-5} Weight Decay 1.0\times 10^{-4} Gradient Clip Norm (model) 1.0 Gradient Clip Norm (header) 1.0 Total Steps M 12 Eval Interval I_{interval} 1 Update Times N 10 CRL-VLA Weight Q-Network Constraint ( \mathcal{L}^{Q}_{\mathrm{GCV}} ) 0.01 Value-Head Constraint ( \mathcal{L}^{V}_{\mathrm{GCV}} ) 0.01 KL Divergence Scale ( \lambda_{\text{KL}} ) 1.0\times 10^{-6} Q Loss Coef ( \mathcal{L}^{Q}_{\mathrm{MC}} ) 1.0 Value Loss Coef ( \mathcal{L}^{V}_{\mathrm{MC}} ) 1.0 Global Context Vector (GCV) Enabled