DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulation DOI: XXXXXXX.XXXXXXX Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 Zebin Yang 1,2 , Yijiahao Qi 4 , Tong Xie 1,2 , Bo Yu 3 , Shaoshan Liu 3 , Meng Li 1,2,5 Abstract . Vision-Language-Action (VLA) models have shown remarkable success in robotic tasks like manipulation by fusing a language model’s reasoning with a vision model’s 3D understanding. However, their high computational cost remains a major obstacle for real-world applications that require real-time performance. We observe that the actions within a task have varying levels of importance: critical steps demand high precision, while less important ones can tolerate more variance. Leveraging this insight, we propose DySL-VLA, a novel framework that addresses computational cost by dynamically skipping VLA layers based on each action’s importance. DySL-VLA categorizes its layers into two types: informative layers, which are consistently executed, and incremental layers, which can be selectively skipped. To intelligently skip layers without sacrificing accuracy, we invent a prior-post skipping guidance mechanism to determine when to initiate layer-skipping. We also propose a skip-aware two-stage knowledge distillation algorithm to efficiently train a standard VLA into a DySL-VLA. Our experiments indicate that DySL-VLA achieves 2.1% improvement in success length over Deer-VLA on the Calvin dataset, while simultaneously reducing trainable parameters by a factor of 85.7 and providing a 3.75 × \times speedup relative to the RoboFlamingo baseline at iso-accuracy. Our code is available on https://github.com/PKU-SEC-Lab/DYSL_VLA . 1. Introduction Inspired by the success of Vision-Language Models (VLMs) ( 1 ; 21 ) , Vision-Language-Action (VLA) models have emerged ( 4 ; 14 ; 34 ) , enabling a promising paradigm for end-to-end robotic control. By tokenizing robot control signals, these models take images as environmental observations and language instructions as the task goal, then generate the next control command for the robot to fulfill the task ( 16 ; 15 ) . Leveraging the vast, internet-scale knowledge embedded within VLMs ( 4 ; 15 ) , VLA models have already shown remarkable generalization capabilities in complex robotic tasks like manipulation ( 8 ; 14 ) . However, deploying VLA models on real-world robots poses a significant challenge. Their immense computational demands lead to high latency and power consumption ( 39 ; 41 ; 17 ; 37 ; 32 ) , which conflict with the limited resources and battery capacity of most robotic platforms ( 13 ; 30 ) . Consequently, existing VLA systems, like RT-2 (1-3 Hz) ( 4 ) and OpenVLA (3-5 Hz) ( 15 ) , have slow action generation speeds compared to the high-frequency low-level control required for real-time physical interaction (20-50+ Hz) ( 14 ) . Figure 1. Different actions in robot manipulation have different importance. We show an example when the robot is performing task “Grasp the black cup and drop it into basket”. (a) shows the task completion rates when adding noise with different magnitudes to VLA model weights at different action steps. When adding noise at important action steps, the task completion rate drops faster as noise magnitude increases. We sample 50 times on each noise magnitude for each step range. We show the robot status at (b) step 25, (c) step 75, and (d) step 125 when using the origin VLA model. While existing VLA acceleration methods, such as quantization ( 24 ; 6 ) , pruning ( 42 ; 5 ) , and knowledge distillation ( 41 ; 6 ) , have not fully solved this problem, they often overlook a crucial insight: in robot manipulation, the importance of different actions is not equal. For instance, the act of grasping or releasing an object is far more critical to a task’s success than the preparatory pre-grasp movements, which is also shown in Figure 1 . By applying a uniform approach to all action predictions, these methods miss key opportunities for acceleration on less important actions and, therefore, offer limited speedup. Similarly, early exit methods ( 39 ; 28 ) attempt to take advantage of this feature by dynamically adjusting the computational load, but they risk discarding crucial information by exiting before the final layers are fully processed. This trade-off can compromise the model’s overall accuracy and effectiveness ( 41 ) . To address the high latency and computational demands of VLA models, we propose DySL-VLA, a method that dynamically skips unnecessary layers during inference. Our approach is based on a key finding: not all VLA layers contribute equally to action prediction. Specifically, we observed that activation distributions change significantly after certain “informative” layers. Our dynamic-static layer skipping method leverages this insight by statically keeping the most critical layers while dynamically skipping others. We also found that the success of a manipulation task is highly sensitive to the accuracy of a few key actions. To account for this and ensure training convergence, we introduce a prior-post skipping guidance and a skip-aware two-stage knowledge distillation method. We summarize our contributions as follows: • We conduct comprehensive analysis of layer-wise performance and action importance variations in VLA prediction. • We propose DySL-VLA to accelerate VLA inference via dynamic-static layer skipping. We also propose prior-post skipping guidance and skip-aware two-stage knowledge distillation to ensure the correctness of important actions and improve training convergence. • Extensive experiments show that DySL-VLA shows 3.75 \times latency reduction over RoboFlamingo, and 2.1% average successful length improvement over DeeR-VLA, with 85.7 \times trainable parameters and 13.7 \times training steps reduction. 2. Background Figure 2. VLA model architecture. Vision-language-action Model. Numerous studies have investigated instructing robots by natural language ( 7 ) . Among them, VLA models are fine-tuned from pretrained VLMs to increase generalization and conduct robot control in an end-to-end way ( 2 ; 40 ) , which shows good performance and becomes the mainstream. As shown in Figure 2 , in each frame, VLA model predicts action given current image observation and language instruction ( 16 ; 15 ) . Though achieving high performance, VLA model shows long real-time latency and low control frequency ( 35 ; 29 ) , which comes from the high computation cost of the LLM backbone. In this paper, we mainly focus on accelerating the LLM backbone of the VLA models, which accounts for most of the parameters and inference latency (84.3% for OpenVLA ( 15 ) and 75.4% for OpenVLA-oft ( 14 ) ). Efficient Model Inference. Existing VLA acceleration works use pruning ( 38 ; 31 ; 42 ) , quantization ( 24 ; 18 ) , and mixture-of-layers ( 41 ) to accelerate VLA models, while these methods ignore the importance difference of each action and allocate an equal amount of computation to each prediction, wasting the acceleration opportunities on unimportant actions. In addition, methods such as quantization and pruning require specialized kernels, which increase the difficulty of deployment. Early-exit methods ( 36 ; 25 ) halting forward propagation at a certain layer based on intermediate predictions, which can dynamically apply computation on different actions. But skipping all final layers results in a significant loss of information. To solve this, DeeR-VLA ( 39 ) largely trains the LLM backbone and multiple action heads to recover model performance. However, this will introduce high computation and memory costs in the training stage. The large-scale fine-tuning on specific scenarios may also break the generalization ability of VLA models. Table 1. Comparison between different methods. Method Approach Action Importance Specialized Training Aware Kernel Free Modules SparseVLM ( 42 ) Pruning ✗ ✗ – FastV ( 5 ) Pruning ✗ ✗ – QAIL ( 24 ) Quantization ✗ ✗ LLM Backbone MoLe-VLA ( 41 ) Mixture-of-Layers ✗ ✓ LLM Backbone DeeR-VLA ( 39 ) Early-exit ✓ ✓ LLM Backbone & Action Heads DySL-VLA Dynamic-static ✓ ✓ Light-weight Adapters Layer Skipping & Skipping Controllers Compared with existing works, our method adaptively applies more computation to important actions using layer skipping methods. We systematically examine the role of each layer and use dynamic-static layer skipping to reduce information loss after skipping. We use pre-skip prediction and post-skip verification to ensure correct skipping decisions. For training efficiency, we only train light-weight skipping controllers and adapters instead of the LLM backbone. The comparison of different methods for VLA acceleration is shown in Table 1 . 3. Efficient VLA Inference via Dynamic-Static Layer-Skipping 3.1. Observation and Overview To achieve high acceleration rate and model performance after layer skipping, there are two questions to answer. 1) When to skip layers? 2) Which layer to skip? Observation 1: the importance across different VLA layers varies a lot, while skipping informative layers may cause low model performance. When deciding which layers to skip, existing layer skipping works ( 39 ; 20 ; 26 ; 9 ) either empirically skip with the same interval or directly skip all the final layers. However, these strategies do not consider the different importance of VLA layers. We evaluate the amount of information contained in each layer of VLA models by calculating the average cosine similarity of each layer’s output activation. As shown in Figure 3 (a) and (b), we find that some VLA layers significantly change the activation distribution compared to other layers. As shown in Figure 3 (c) and (d), skipping these informative layers will introduce significant performance drops. Figure 3. The average cosine similarity between the output activations of different VLA layers for (a) RoboFlamingo-3B and (b) RoboFlamingo-9B. The similarity between the input and output activations of each layer and the model performance when skipping each VLA layer in a zero-shot manner for (c) RoboFlamingo-3B and (d) RoboFlamingo-9B. Figure 4. (a) The ratio of different numbers of kept layers in VLA model inference when only using skipping controllers. The inference latency for different numbers of kept layers using (b) RoboFlamingo-3B in FP32 and (c) RoboFlamingo-9B in FP16. Observation 2: VLA systems show high sensitivity on important actions, and more restrictions are needed to decide skipping positions. To decide when to skip layers, existing works ( 12 ; 20 ; 26 ) use skipping controllers (usually feedforward networks) before LLM layers to predict skipping probability, and conduct layer skipping if the probability exceeds a threshold. However, directly transferring this method to VLA models shows low accuracy. This is because even small errors on important actions may cause task failure, which is shown in Figure 1 . However, as shown in Figure 4 (a), when only using skipping controllers, the number of kept layers remains at a relatively low level for all actions. This is because a normalization loss is usually introduced in the training process to force the skipping controllers to skip more layers. And when only using the skipping controllers, the correctness of important actions can not be guaranteed. In addition, the skipping controllers will introduce non-negligible extra inference latency, which comes from the serial nature of the inference of skipping controllers and the VLA layers ( 36 ) . As shown in Figure 4 (b) and (c), when the skipping mechanism is introduced before each layer and half of the layers are activated, the mechanism will gain little latency reduction over the baseline model. DySL-VLA overview. Based on these observations, we propose DySL-VLA, accelerating VLA models by dynamic-static layer skipping. To achieve low information loss and high speedup, we propose dynamic-static layer skipping to statically keep the informative layers and dynamically skip unnecessary layers (Section 3.2 ). To keep enough layers for important action, we propose prior-post skipping guidance to guide the skipping decision (Section 3.3 ). Finally, we propose skip-aware two-stage knowledge distillation to conduct low-cost training and improve training convergence (Section 3.4 ). 3.2. Dynamic-static Layer Skipping Problems of existing layer skipping works. Existing layer skipping works do not consider the different importance of VLA layers, thus showing sub-optimal performance. As shown in Figure 5 (b) and (c), early exit methods either need to train multiple action heads ( 39 ) or introduce adapters ( 11 ) to fit the final action head. However, these methods skip all final layers, some of which are informative, thus showing lower performance. ( 20 ; 12 ) only skip one layer each time, as shown in Figure 5 (d). This fine-grained skipping mechanism shows a low acceleration rate because the skipping controller and adapter introduce extra inference cost. Figure 5. The inference mode of (a) original VLA model, (b) using early exit with multiple action heads, (c) using early exit with adapters, (d) using traditional layer skipping methods, and (e) using dynamic-static layer skipping. The modules with light colour are not activated in current inference. We set the same legend for VLA layers in (a), (b), (c), (d), and dynamic layers in (e). To solve these problems, we propose a dynamic-static layer skipping mechanism, as shown in Figure 5 (e). As discussed in Section 3.1 , some informative layers in VLA models significantly change the activation distribution. We define them as static layers and statically keep these layers in model inference. At the same time, although other layers contain less information, we find that directly skipping all of these layers will cause extremely low accuracy. We define these layers as dynamic layers and dynamically skip them. Before each dynamic layer, we determine whether to perform layer skipping (the decision mechanism will be discussed in Section 3.3 ). If we decide to conduct layer skipping, we directly skip to the next static layer, which shows a higher speedup compared with skipping only one layer each time. And we train adapters (light-weight feedforward layers) to summarize the skipped layers and fit the activation for the next static layer. This is reasonable as the skipped layers will not change the activation distribution much and contain less information, which is within the adapter’s fitting ability. By using dynamic-static layer skipping, we can achieve a higher speedup with low information loss. At the same time, our method does not need to train multiple action heads or the LLM backbone, which reduces the training cost. Figure 6. The proportion of action prediction steps in different continuity ranges and the accuracy loss when conducting layer skipping at the steps in different continuity ranges for (a) RoboFlamingo-3B and (b) RoboFlamingo-9B. 3.3. Prior-post Skipping Guidance In this section, we discuss our strategy to determine whether to conduct layer skipping before each dynamic layer. As discussed in Section 3.1 , just using skipping controllers ( 12 ; 20 ) is neither accurate nor efficient, and we should keep enough layers for important actions. Based on this, we find that the trajectory continuity can reflect the importance of the current action. Here we define the continuity at step t as: (1) \displaystyle C_{t}=-\frac{1}{k}\sum_{j=t-k+1}^{t}||\delta A_{j}||_{2}=-\frac{1}{k}\sum_{j=t-k+1}^{t}||A_{j}-A_{j-1}||_{2}, where \delta A_{j} represents the difference between action j and j-1 , and we consider the trajectory of last k actions. As shown in Figure 6 , we find that the trajectory has good continuity at most of the time, which means adjacent actions have similar magnitude and direction. This phenomenon comes from the training data collection process for VLA models, as robot operators tend to keep uniform speeds when executing non-critical motions. However, the continuity will be broken when the robots are conducting fine operations, e.g., grasping or releasing objects. Unlike free-space movements that follow smooth trajectories, these fine operations include frequent stops, micro-corrections, and hesitation, which break the natural flow of motion into disjointed segments ( 33 ) . We find that these actions show higher importance in task completion, and conducting layer skipping on these steps will introduce huge accuracy loss, as shown in Figure 6 . Based on these observations, we can approximate action importance based on action continuity to guide layer skipping determination. As shown in Figure 7 , besides skipping controllers, we propose pre-skip prediction to approximate the proper skipping positions between static layers. Between each two adjacent static layers, we define a skipping-allow point, which is initialized after the front static layer. Before the point, the skipping controllers are disabled, and the layers are forcibly kept. The skipping controller after the point is activated and determines layer skipping. We dynamically move the skipping-allow point according to the continuity change of the previous k steps: (2) \displaystyle l_{i} \displaystyle=l_{i}+\delta l,~~\text{when}~~C_{t}-C_{t-1}<-\eta~~\text{and}~~l_{i}\eta~~\text{and}~~l_{i}>s_{i-1}, where l_{i} is the id of the layer after the skipping-allow point, \eta is threshold ( \eta>0 ), s_{i} is the i_{th} static layer, \delta l is an adaptive moving stride defined as \delta l=\lceil\frac{C_{t}-C_{t-1}}{\eta}\rceil . Note that when detecting decreased continuity, the skipping allowance point rapidly shifts forward according to continuity change to prioritize action accuracy during critical phases. Conversely, when continuity improves, it gradually moves backward. This hysteresis-like system design maximizes the correctness of essential actions. And we decide the position of the skipping-allow points according to the trajectory of the previous k steps instead of a single recent step ( k=1 ), as the change of action difference ( \delta A_{t}-\delta A_{t-1} ) will become low when important actions occur in a continuous mode. An example of skipping-allow points moving is shown in Figure 8 . Note that the pre-skip prediction only closes or activates the skip controllers, and whether to conduct layer skipping is still determined by the activated skip controller itself. Figure 7. Prior-post skipping guidance. The modules with light colour are not activated in current inference. By using pre-skip prediction, we can keep enough layers for the important actions, thus improving model performance. At the same time, the extra latency cost caused by skipping controllers can also be reduced, as the pre-skip prediction will close most of the skipping controllers that will probably decide not to skip. However, only using pre-skip prediction is not enough. This is because we can not get the current action prediction before current model inference, so the continuity decrease can only be detected after the first important action has been predicted, whose correctness can not be guaranteed. The issue becomes more pronounced when the action chunk technique is used ( 14 ; 2 ; 35 ) , where multiple actions will be predicted in a single inference. To solve this problem, we also propose post-skip verification to add a feedback mechanism, which is shown in Figure 7 . When we first detect the continuity decrease ( \delta C_{t}=C_{t}-C_{t-1}<\eta_{1} and \delta C_{t-1}=C_{t-1}-C_{t-2}>\eta_{1} ), we re-predict the current action without any layer skipping. Then we recompute the continuity change using the re-predicted action. This process not only ensures the correctness of the initial critical action prediction but also enhances the detection accuracy of continuity degradation. And it will not introduce much extra cost, as important actions only occupy a small proportion and usually appear continuously. Figure 8. Skipping-allow point changes in different steps. Here we use 4 dynamic layers between 2 static layers as an example, with 5 possible positions for skipping-allow point. 3.4. Skip-aware Two-stage Knowledge Distillation As discussed in Section 2 , by freezing the LLM backbone and just training the lightweight skipping controllers and adapters, our training cost can be largely reduced. However, the training strategy is nontrivial, as simply training controllers and adapters together may cause a convergence problem. This is because the controllers and adapters are both randomly initialized, and the training of them will impact each other. At the beginning of training, as the adapters are not trained, the controllers will refuse layer skipping to avoid huge task loss. And this will further affect the adapter training. To solve this problem, we propose skip-ware two-stage knowledge distillation. In the first stage, as shown in Figure 9 (a), we only train all the adapters to summarize the information of the following dynamic layers. The loss function of the first stage is: (4) \displaystyle loss_{1}=\sum_{i}\|adapter_{i}(x_{i})-L_{s_{i}-1}(L_{s_{i}-2}(\dots(L_{i}(x_{i}))))\|_{F}, where x_{i} is the input to dynamic layer i , L_{i} is the i_{th} layer and s_{i} is the layer id of next static layer. After the basic capability of adapters are developed, in the second stage, we can then train the controllers and adapters together, as shown in Figure 9 (b). However, in the forward path, if we use the controller itself to decide the skipping place, we find that the model will continuously skip at the first few dynamic layers. So, to fully train each controller, between two static layers, we just select one dynamic layer i ( s_{i-1}