DTP:A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Models

Chenyang Li 1 , Jieyuan Liu 2 , Bin Li 3 , Bo Gao 4 , Yilin Yuan 4 , Yangfan He 5 , Yuchen Li 6 , Jingqun Tang 7 1 Australian National University, 2 University of California, San Diego, 3 Chinese Academy of Sciences, 4 Beijing Institute of Graphic Communication, 5 University of North Carolina at Cha · 2026

Read DTP:A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Models by Chenyang Li 1 , Jieyuan Liu 2 , Bin Li 3 , Bo Gao 4 , Yilin Yuan 4 , Yangfan He 5 , Yuchen Li 6 , Jingqun Tang 7 1 Australian National University, 2 University of California, San Diego, 3 Chinese Academy of Sciences, 4 Beijing Institute of Graphic Communication, 5 University of North Carolina at Cha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16065.txt.

Introduction

Chenyang Li and a large team trim waste tokens in robot vision and action models. Their 2026 preprint notes a flaw in such models. By default, the models may gaze too much at parts of the view that do not serve the task. The team calls those parts stray tokens. Stray gaze can hurt the acts the robot picks.

Their fix, named DTP, cuts stray tokens from the chain. The cut keeps gaze on the parts that guide the act. A lean chain should run fast and act with more care. The method is framed as plain yet strong. It targets models that join sight, words and acts for robot tasks in real rooms.

Catalog record

CREATORChenyang Li 1 , Jieyuan Liu 2 , Bin Li 3 , Bo Gao 4 , Yilin Yuan 4 , Yangfan He 5 , Yuchen Li 6 , Jingqun Tang 7 1 Australian National University, 2 University of California, San Diego, 3 Chinese Academy of Sciences, 4 Beijing Institute of Graphic Communication, 5 University of North Carolina at Cha
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16065

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DTP:A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Models Chenyang Li 1 , Jieyuan Liu 2 , Bin Li 3 , Bo Gao 4 , Yilin Yuan 4 , Yangfan He 5 , Yuchen Li 6 , Jingqun Tang 7 1 Australian National University, 2 University of California, San Diego, 3 Chinese Academy of Sciences, 4 Beijing Institute of Graphic Communication, 5 University of North Carolina at Chapel Hill, 6 Baidu Search, 7 Bytedance Abstract Vision-Language Action (VLA) models have shown remarkable progress in robotic manipulation by leveraging the powerful perception abilities of Vision-Language Models (VLMs) to understand environments and directly output actions. However, by default, VLA models may overly attend to image tokens in the task-irrelevant region, which we describe as ‘distracting tokens’. This behavior can disturb the model from the generation of the desired action tokens in each step, affecting the success rate of tasks. In this paper, we introduce a simple yet effective plug-and-play D istracting T oken P runing (DTP) framework, which dynamically detects and prunes these distracting image tokens. By correcting the model’s visual attention patterns, we aim to improve the task success rate, as well as exploring the performance upper boundaries of the model without altering its original architecture or adding additional inputs. Experiments on the SIMPLER Benchmark ( 16 ) show that our method consistently achieving relative improvements in task success rates across different types of novel VLA models, demonstrating generalizability to transformer-based VLAs. Further analysis reveals a negative correlation between the task success rate and the amount of attentions in the task-irrelevant region for all models tested, highlighting a common phenomenon of VLA models that could guide future research. We also publish our code at: https://anonymous.4open.science/r/CBD3 . Figure 1: Overview of our proposed Distracting Token Pruning (DTP) method for improving Vision-Language-Action (VLA) models in robotic manipulation. Left: Input consisting of the current visual observation and natural language instruction. Middle: Comparison between the original VLA model (top) may focus on task-irrelevant regions that lead to task failure, while our DTP-enhanced approach (bottom) creates more focused attention on task-critical areas, leading to the improvements in the task success rate. 1 Introduction The success of Vision-Language Models (VLMs) in understanding and reasoning about visual content has opened new possibilities for embodied AI. Trained on large-scale image–text data, VLMs ( 19 ; 12 ; 2 ; 20 ; 21 ; 6 ; 1 ) excel at visual recognition, reasoning, and visual question answering. Building on this, Vision-Language-Action (VLA) models extend VLMs to generate executable robot actions, bridging high-level semantic reasoning with low-level control. Early VLAs such as RT-1 ( 4 ) and RT-2 ( 3 ) pioneered action tokenization for robotic control, while later models like OpenVLA ( 14 ) achieved strong performance despite smaller sizes. These advances established the core VLA pipeline: a visual encoder ( 26 ; 28 ; 31 ; 22 ) , a language model ( 29 ; 7 ; 25 ) , and discrete action token generation. Recent works further improve VLA capabilities. SpatialVLA ( 24 ) and ( 27 ; 17 ; 15 ) incorporate depth for better 3D understanding, Nora ( 10 ) employs tokenizers such as FAST ( 23 ) for precise continuous control, and world-model approaches ( 5 ; 30 ) leverage dynamics modeling for action prediction. Despite these architectural advances, VLAs may still fail in manipulation tasks due to redundant visual tokens, with task-irrelevant visual tokens may receive disproportionately high attention during cross-attention, distracting the model from task-relevant regions (Figure 1 ). This raises a concern: Do distracting tokens affect task success, and can pruning them improve performance? We initially tried manually inspecting visual attention patterns and pruning image tokens for the model. However, this approach has two major limitations: (1) reviewing attention by human for each action token is extremely time-consuming, especially since the model can output hundreds to thousands action tokens for each episode, making full test-suite analysis impractical; (2) human intuition about ”good” attention patterns may not align with the model’s preferences. Although ideal attention should focus on task-relevant objects and regions, different VLA models may have their own visual pattern preference, so manual pruning does not necessarily lead to improved performance. To address these two problems, we propose Distracting Token Pruning (DTP) , a plug-and-play framework that dynamically prunes distracting tokens. DTP contains three stages: (1) Relevance-based important region construction, where prompt–image token interactions identify task-relevant visual tokens, and (2) Action attention analysis, where the attention heatmap reveals which image tokens the model focuses at each action generation step. (3) We then apply an intersection-based strategy, selectively pruning visual tokens in the unimportant region, if their attention exceeds the maximum attention within the important region (scaled by tolerance \tau ). Experiments on the SIMPLER Benchmark ( 16 ) prove that DTP effectively corrects visual attention patterns, improves task success rates, and generalizes across transformer-based VLAs. In summary, our work makes the following contributions: (1) We introduce the Distracting Token Pruning (DTP) framework, a novel intersection-based method that automatically identifies and prunes distracting tokens, improving task success rates by addressing common attention weaknesses in VLA models. (2) We explore the performance upper bound of VLAs by varying tolerance \tau , seeking the ideal visual attention patterns that align with model preferences and maximize achievable performance. (3) We analyze attention values in unimportant regions, revealing a negative correlation with task success and offering insights for building more robust VLAs. 2 Related Work Vision-Language Models (VLMs) have become the foundation for multimodal reasoning, showing strong generalization across visual understanding, captioning, and visual question answering. Large-scale pretraining on internet-scale image–text data has enabled models such as PrismaticVLM 12 , InternVL2.5 6 , PaliGemma 2 2 , and Qwen2.5-VL 1 to achieve impressive performance on various visual-language tasks. These models combine high-capacity visual encoders such as SigLIP 31 and DINOv2 22 with large language models ( 29 ; 7 ; 25 ) , producing flexible representations that can be adapted across a wide range of tasks. While primarily designed for static vision-language tasks, the success of VLMs in structured reasoning has inspired their extension into embodied AI domains. In particular, their ability to jointly ground natural language with perceptual input provides a natural interface for robotic control. This transition, however, requires not only high-level semantic understanding but also the capacity to map instructions and observations into temporally grounded motor actions — a gap that gave rise to Vision-Language-Action (VLA) models. Vision-Language-Action Models (VLAs) extend the capabilities of VLMs by directly generating robot actions, effectively bridging perception and motor control. Early work such as RT-1 ( 4 ) and RT-2 ( 3 ) established the paradigm of representing actions as discrete tokens, enabling pretrained language models to output robot commands in a manner analogous to text generation. Building on this foundation, OpenVLA ( 14 ) and OpenVLA-OFT ( 13 ) further reduced the model size and improved inference efficiency. Recent efforts have pushed VLAs in two complementary directions. First, improvements in vision encoding aim to overcome limitations of purely 2D perception. SpatialVLA ( 24 ) , GeoVLA ( 27 ) , EVO-0 ( 17 ) , and PointVLA ( 15 ) incorporate the depth information

Read the complete text