Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach

Chenglong Lu 1 , Shen Liang 2 ✉, Xuewei Wang 3 , Wei Wang 1 · 2025

Read Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach by Chenglong Lu 1 , Shen Liang 2 ✉, Xuewei Wang 3 , Wei Wang 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23459.txt.

Introduction

Chenglong Lu and co-authors teach vision transformers to drop useless image tokens by playing a game, in this 2025 paper. These models split an image into tokens and pay a cost that grows with the square of the token count. Pruning tokens saves that cost, but hand-made rules cannot adapt to each image, and they ignore that pruning happens layer after layer. The authors frame the whole process as a sequence of decisions.

They model pruning as a Markov game and train it with a multi-agent reinforcement learning method. Each token gets its own agent, deciding its fate in context. Reward functions balance saved compute against lost accuracy. A learned, input-aware policy should beat fixed rules, above all on images whose informative tokens sit in unexpected places.

Catalog record

CREATORChenglong Lu 1 , Shen Liang 2 ✉, Xuewei Wang 3 , Wei Wang 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach Chenglong Lu 1 , Shen Liang 2 ✉, Xuewei Wang 3 , Wei Wang 1 Abstract Vision Transformers (ViTs) have computational costs scaling quadratically with the number of tokens, calling for effective token pruning policies. Most existing policies are handcrafted, lacking adaptivity to varying inputs. Moreover, they fail to consider the sequential nature of token pruning across multiple layers. In this work, for the first time (as far as we know), we exploit Reinforcement Learning (RL) to data-adaptively learn a pruning policy. Formulating token pruning as a sequential decision-making problem, we model it as a Markov Game and utilize Multi-Agent Proximal Policy Optimization (MAPPO) where each agent makes an individualized pruning decision for a single token. We also develop reward functions that enable simultaneous collaboration and competition of these agents to balance efficiency and accuracy. On the well-known ImageNet-1k dataset, our method improves the inference speed by up to 44% while incurring only a negligible accuracy drop of 0.4%. The source code is available at https://github.com/daashuai/rl4evit . [email protected], [email protected], [email protected], [email protected] I Introduction Vision Transformers (ViTs) [ 1 , 2 ] have achieved state-of-the-art performance in many computer vision tasks, yet they typically have high computational costs that scale quadratically with the number of tokens. However, in practice, a ViT’s final prediction often depends on a subset of the most informative tokens [ 3 ] , with the other tokens being redundant. This opens up the opportunity for token pruning to reduce the computation cost while maintaining high accuracy. To date, several token pruning policies have been proposed, most of which [ 4 , 5 , 6 , 7 , 8 ] draw on pre-defined, handcrafted pruning policies based on the attention values of the tokens, lacking adaptivity to varying inputs. Another existing work, DynamicViT [ 9 ] , exploits Multi-Layer Perceptrons (MLPs) to automatically learn whether to prune each token. However, it still relies on manually defined pruning ratios for each layer, resulting in limited flexibility. Furthermore, existing works largely overlook the sequential nature of the tokens, where the decision to preserve or prune certain tokens in one layer can significantly impact the outcomes for the subsequent layers. Fig. 1: Comparisons between our RL4EViT and some state-of-the-art token pruning methods on the ImageNet-1k dataset [ 10 ] , using DeiT-S and DeiT-B as backbones. Our RL4EViT has achieved the best trade-off between accuracy and inference speed. In view of the drawbacks of existing handcrafted features, in this work, we propose Reinforcement Learning for Efficient Vision Transformer (RL4EViT) , where we draw on Reinforcement Learning (RL) to learn a token pruning policy in a data-adaptive manner. To the best of our knowledge, this is the first work applying RL to this task, which can achieve the best trade-off between accuracy and efficiency when compared with the state-of-the-art (Fig. 1 ). Concretely, we formulate token pruning as a sequential decision problem. As is shown in Fig. 2 , we incorporate a token pruning layer after each Transformer block 1 1 1 For simplicity but without loss of generality, in all sections in this paper but Section IV , we assume that all Transformer blocks are proceeded by token pruning layers. In practice, we can selectively add token pruning layers after a subset of Transformer blocks. , where we utilize Multi-Agent Proximal Policy Optimization (MAPPO) [ 11 ] to prune the output tokens of the Transformer block. The multi-agent setting allows for individualized treatment of each token as compared with a single-agent setting. We also propose reward functions that enable the agents to both compete and collaborate with each other, effectively balancing efficiency and accuracy. The global decision across all token pruning layers is obtained via a Markov Game [ 12 ] , thus preserving important inter-layer sequential information. Fig. 2: An illustration of our Reinforcement Learning for Efficient Vision Transformers (RL4EViT) method for token pruning. For a given ViT, each Transformer block is proceeded by a token pruning layer, where we utilize Multi-Agent Proximal Policy Optimization (MAPPO) [ 11 ] to prune the output tokens of the Transformer block. The global reward across all token decision layers is obtained via a Markov Game [ 12 ] to preserve important sequential information. Note that in practice, instead of directly removing the pruned tokens, we set their attention weights to -1000 to facilitate mini-batch-based training. In summary, our main contributions are as follows: • We propose RL4ViT where we exploit RL for data-adaptive token pruning in ViTs. To the best of our knowledge, this is the first time RL has been applied to this task. • We present MAPPO-based token pruning layers, where the multi-agent setting enables individualized decision-making for each token. We propose novel reward functions for an effective trade-off between efficiency and accuracy, and utilize a Markov Game to calculate the global reward to capture sequential semantics across all token pruning layers. • We conduct extensive experiments on the well-known ImageNet-1k dataset, which show that our RL4EViT can improve the inference speed by up to 44% while incurring only a negligible accuracy drop of 0.4%. The rest of this paper is organized as follows: Section II reviews the related work. Section III details our RL4EViT method. Section IV reports the experimental results. Section V concludes the paper. II Related Work Addressing their quadratic complexity against the number of tokens, researchers have proposed multiple token pruning methods for ViTs. Most existing works [ 4 , 5 , 6 , 7 , 8 ] favor preserving tokens with high attention weights over those with low ones. These methods all use pre-defined, handcrafted pruning policies that lack adaptivity to diverse inputs. DynamicViT [ 9 ] , on the other hand, uses MLPs to predict whether to preserve or prune each token by balancing accuracy and efficiency. However, the number of tokens to prune in each layer remains manually defined, limiting its flexibility. Alongside token pruning, many methods entail token merging [ 7 , 8 , 5 ] . Instead of directly discarding less important tokens, these methods merge them into more compact representations using techniques such as spatial attention [ 7 , 8 ] and pooling [ 5 ] . Such methods are more robust against errors in token pruning, at the cost of efficiency decay. III RL4EViT for Token Pruning We are now in a position to present the details of our RL4EViT method, which integrates MAPPO token pruning layers into the ViT architecture and exploits a Markov Game setup to prune redundant tokens (Fig. 2 ). As illustrated in Alg. 1 , RL4EViT takes the following as the input: • image dataset \mathcal{D} ; • ViT components : patch embedding network M , Transformer blocks F^{l} (l=1,2,\ldots,L) entailing self-attention and Feed-Forward Networks (FFNs), and the image classifier C (can be replaced with other downstream task solvers); • RL components : an actor network \pi_{\theta} and a critic network V_{\phi} , as core modules of MAPPO (detailed in Section III-A1 ). RL4ViT is trained based on a pre-trained ViT (line 1 of Alg. 1 ) with no token pruning. Then, we alternate between optimizing the RL components and optionally fine-tuning the ViT

Read the complete text