MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation

Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, Shanghang Zhang, Peking University; Beijing Academy of Artificial Intelligence · 2025

Read MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation by Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, Shanghang Zhang, Peking University; Beijing Academy of Artificial Intelligence (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20384.txt.

Introduction

Rongyu Zhang and co-authors present a leaner robot control model in a 2025 paper. Large vision-language models let robots follow spoken instructions and act in the world. Their size makes real-time use costly, and trimming them often cuts the final layers that carry task meaning.

Their design treats each model layer as an expert that can be switched on or off. A router that reads space and time cues activates only the layers a step needs. The idea borrows from brain science claims that early layers do much of the work. Skipping idle layers cuts compute and storage demands. The team tests the model on robot manipulation tasks to check speed and success.

Catalog record

CREATORRongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, Shanghang Zhang, Peking University; Beijing Academy of Artificial Intelligence
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20384

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, Shanghang Zhang, Peking University; Beijing Academy of Artificial Intelligence Abstract Multimodal Large Language Models (MLLMs) excel in understanding complex language and visual data, enabling generalist robotic systems to interpret instructions and perform embodied tasks. Nevertheless, their real-world deployment is hindered by substantial computational and storage demands. Recent insights into the homogeneous patterns in the LLM layer have inspired sparsification techniques to address these challenges, such as early exit and token pruning. However, these methods often neglect the critical role of the final layers that encode the semantic information most relevant to downstream robotic tasks. Aligning with the recent breakthrough of the Shallow Brain Hypothesis (SBH) in neuroscience and the mixture of experts in model sparsification, we conceptualize each LLM layer as an expert and propose a M ixture- o f- L ay e rs V ision- L anguage- A ction model ( MoLe-VLA or simply MoLe ) architecture for dynamic LLM layer activation. We introduce a Spatial-Temporal Aware Router (STAR) for MoLe to selectively activate only parts of the layers based on the robot’s current state, miming the brain’s distinct signal pathways specialized for cognition and causal reasoning. Additionally, to compensate for the cognition ability of LLM lost in MoLe, we devise a cognition self-knowledge distillation (CogKD) to enhance the understanding of task demands and generate task-relevant action sequences by leveraging cognition features. Extensive experiments in both RLBench simulation and real-world environments demonstrate the superiority of MoLe-VLA in both efficiency and performance, achieving performance improvement of 8% mean success rate across ten tasks while reducing at most × 5.6 \times 5.6 computational costs in LLM. Figure 1 : Overview of our proposed MoLe-VLA : Our proposed framework integrates dynamic layer activation, a novel Spatial-Temporal Aware Router (STAR), and self-knowledge distillation (CogKD) to achieve efficient and adaptive performance in robotic applications. MoLe reduces computational costs while enhancing model performance, enabling resource-constrained platforms to benefit from MLLMs. 1 Introduction The rapid advancements in multimodal large language models (MLLMs) [ 27 , 3 , 17 , 2 , 32 ] have demonstrated their ability to integrate complex language and visual representations, inspiring the development of generalist robots and embodied agents capable of vision-language comprehension, human interaction, and flexible problem-solving in manipulation tasks. Preliminary vision language action (VLA) models [ 25 , 16 , 21 , 19 ] , such as RT-2 [ 6 ] and OpenVLA [ 16 ] , have shown the feasibility of using MLLMs for end-to-end robotic control, enabling robust policies and emergent abilities, including generalization to unseen objects and understanding novel commands. However, deploying MLLMs in real-world robotic systems faces significant challenges due to their high computational demands, including substantial memory usage, power consumption, and time delays, which conflict with robotic platform resource-constrained and real-time requirements. For example, a 7B VLA model running on a commercial-grade GPU like the RTX 4090 generally achieves an inference frequency of approximately 5-12 Hz, which falls significantly short of the 50-1000 Hz control frequency required by the Franka robotic arm. Recent studies [ 43 , 34 ] have uncovered significant redundancy in LLM layer, particularly in robotic tasks, where homogeneous patterns across layers lead to high computational costs with limited performance gains. For instance, DeeR [ 43 ] demonstrated that using all 24 layers of the Flamingo [ 21 ] model improves task success rates by only 3.2\% compared to using six layers, while computational costs increase 4x on the Calvin LH-MTLC [ 28 ] . Similarly, our analysis of OpenVLA [ 16 ] with RLBench [ 13 ] in Fig. 1 (A) reveals that cosine similarity between consecutive layer outputs exceeds 90% , while features from the first and last layers differ significantly. This suggests the potential for skipping adjacent layers to reduce computation but also highlights the limitations of early-exit strategies [ 43 , 10 ] , as shown in Fig. 1 (B), where discarding deeper layers risks losing critical semantic information. Inspired by the Shallow Brain Hypothesis (SBH) [ 37 ] , which suggests that the brain balances deep hierarchical structures with shallow, parallel cortico-subcortical loops for cognition and causal reasoning, we propose a selective layer activation strategy in VLA models. As shown in Fig. 1 (C), our approach mirrors the brain’s dynamic depth-parallelism balance, activating only task-relevant layers to enhance efficiency and adaptability, embodying principles of SBH in VLA model design. In this paper, we introduce a M ixture- o f- L ay e rs V ision- L anguage- A ction model ( MoLe-VLA ) incorporating a novel layer-selection router at the input stage of LLMs for its sparsity. Our design emulates the brain’s decision-making process described in the SBH by dynamically selecting optimal forward pathways with varying layer combinations. Inspired by the routing mechanism in mixture-of-experts (MoE) [ 23 , 46 , 47 ] , which enables horizontal expert-wise activation within a single LLM layer, we extend this concept vertically to achieve layer-wise activation. Specifically, we treat each LLM layer as an independent expert and utilize a biologically inspired router to manage layer skipping, mimicking the brain’s selective activation of cortico-subcortical loops. Unlike Mixture-of-Depth (MoD) [ 34 ] , which assigns input tokens to different experts and risks token-wise inconsistencies due to varying perception levels across layers, our proposed MoLe dynamically selects the most relevant layers while processing input features holistically. Traditional MoE or MoD routers, which rely on simple linear layers, often fail to capture critical spatial-temporal information necessary for reasoning in dynamic, embodied intelligence tasks. To address this limitation, we propose the Spatial-Temporal Aware Router (STAR) , which independently processes spatial features from visual inputs and temporal dependencies from textual inputs. By combining these essential properties into a unified representation, STAR aligns the selection of LLM layers with the demands of the current environment. STAR dynamically activates the most relevant layers by generating softmax probabilities for each layer and selecting the top- k layers with the highest probabilities. By fully leveraging spatial-temporal information, STAR ensures accurate and efficient adaptation to the dynamic nature of embodied intelligence tasks, achieving optimal performance with reduced computational overhead. Nonetheless, skipping certain layers inevitably reduces the cognitive expressiveness of the model. To address this, we propose Cognitive self-Knowledge Distillation (CogKD) , a novel approach to preserve grasping ability while mitigating cognitive collapse. In CogKD, the original full-layer model serves as the teacher, while the MoLe layer-skipping model acts as the student. Inspired by [ 19 ] , we introduce a learnable cognition token , which efficiently integrates visual tokens and language guidance to enhance comprehension of task demands and produce task-relevant action sequences. By analyzing the similarity between cognition tokens and student tokens, we identify tokens of interest (ToIs) that represent task-critical information the student needs to learn. These ToIs provide precise guidance for adaptively re-weighting the distillation process, ensuring the student model focuses on key cognitive features while consistently benefiting from the layer-skipping efficiency. The effectiveness of MoLe in both performance and efficiency enhancement is demonstrated in real-world and RLBench simulation environments based on various VLA models against state-of-the-art baselines. Extensive

Read the complete text