Video, How Do Your Tokens Merge?
Sam Pollard, Michael Wray · 2025
Read Video, How Do Your Tokens Merge? by Sam Pollard, Michael Wray (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03885.txt.
Introduction
Sam Pollard and Michael Wray asked in 2025 how video models should throw away information. Video transformers are ravenously costly because their input tokens multiply across space and time. Token merging. This fuses similar tokens as the network processes them, is a strong diet: it plugs into any vision transformer, needs no retraining. And, unlike dropping tokens, it carries the merged information forward instead of deleting it. Yet merging had barely been tested on video with temporal hardness.
The authors ran training-free merging across four video transformer designs and three action recognition data sets, coarse and fine-grained. They were hunting for best practices. These covered where in the network to merge, how aggressively, and how to pair tokens. Merging delivered roughly a two-and-a-half-fold speedup while costing under one percentage point of accuracy on their headline setup, and the study codifies when the trick is safe.
Catalog record
| CREATOR | Sam Pollard, Michael Wray |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03885
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video, How Do Your Tokens Merge? Sam Pollard, Michael Wray Abstract Video transformer models require huge amounts of compute resources due to the spatio-temporal scaling of the input. Tackling this, recent methods have proposed to drop or merge tokens for image models, whether randomly or via learned methods. Merging tokens has many benefits: it can be plugged into any vision transformer, does not require model re-training, and it propagates information that would otherwise be dropped through the model. Before now, video token merging has not been evaluated on temporally complex datasets for video understanding. In this work, we explore training-free token merging for video to provide comprehensive experiments and find best practices across four video transformers on three datasets that exhibit coarse and fine-grained action recognition. Our results showcase the benefits of video token merging with a speedup of around 2.5 2.5 X while maintaining accuracy (avg. − 0.55 % -0.55\% for ViViT). Code available at https://github.com/sjpollard/video-how-do-your-tokens-merge . Figure 1 : Video Token Merging reduces computation of video transformer models by successively merging tokens without re-training or additional learned parameters. We show how an input video has its tokens merged across different layers. 1 Introduction Vision transformers Dosovitskiy et al. 2021 (ViTs) have quickly been established as the architecture of choice for solving the majority of computer vision problems. The long range dependencies that are afforded by self-attention significantly improve the flexibility and reasoning of models, when supplied with enough training data. This performance is not free as the quadratic complexity of the attention mechanism leads to exceedingly poor scaling. This is especially prevalent in the video domain, in which the sequence length of tokens increases with both spatial and temporal dimensions. In the recent push for long video understanding Soldan et al. 2022 ; Carreira et al. 2024 ; Grauman et al. 2022 , video vision transformers are therefore requiring more and more resources for both training and inference. Assuming that a 5 minute video at 224\times 224 pixels consists of the typical 16\times 16 spatial patches, when sampling 1 frame per second, the transformer sequence length is 58,800 tokens. With this increase in complexity, it’s simply not enough to hope that hardware scales up at the same rate: this requires a huge number of resources per video and is significantly longer than the sequence lengths that can be handled in a single window of context for current transformers. It is also worth noting that this sampling rate may not even be enough for fine-grained action understanding, where actions average 2.6 s in duration Damen et al. 2022 . Figure 2 : The merging process first separates tokens into two sets. Similarities are calculated and a one-to-many bipartite matching between tokens in each set is found. Finally, the top r edges are kept and these are merged based on the similarity between tokens. Recently, token merging of transformers has been proposed for images Bolya et al. 2023 , as a drop-in method for increasing the throughput of ViT models by reducing the token sequence length. This has several benefits over other methods: the method does not require re-training of a pre-trained model and the metric to merge tokens is already calculated as part of the forward pass. In this work, we explore the implementation of token merging in the video domain without fine-tuning , allowing tokens to merge freely in and between frames, to showcase its viability on video models that have been built upon ViT. We evaluate video token merging on Kinetics- 400 Kay et al. 2017 , Something-Something v 2 Goyal et al. 2017 , and EPIC-KITCHENS- 100 Damen et al. 2022 , which are datasets commonly used for video understanding, using both third and first person videos as well as containing actions with differing granularities. To summarise, our contributions are as follows: (i) we explore best practices for training-free token merging for video; (ii) we provide a comprehensive evaluation of video transformer models on common action recognition benchmarks, demonstrating a training-free speedup of 2.5 X; and (iii) we provide an in-depth analysis of how tokens are merged within spatio-temporal models. 2 Related Work Efficiency of transformers is a popular topic, with a variety of possible approaches. Recent works in natural language processing (NLP) have focused on the creation of more computationally efficient attention modules to reduce the overhead of the quadratic complexity Wang et al. 2020 , attention modules that make better use of modern hardware Dao et al. 2022 , and new architectures that scale linearly with input size Gu and Dao 2023 . Others make use of domain specific knowledge to reduce the range of tokens that self-attention is applied to in vision problems Bertasius et al. 2021 ; Patrick et al. 2021 . Alternatively, in Touvron et al. 2021 , knowledge distillation is applied to make transformers more efficient learners at training time. Transformers are resilient to the dropping (or pruning) of input tokens, attention heads, and even entire blocks, which Meng et al. 2022 has demonstrated. The dropout of input tokens is of particular interest for vision transformers because it exploits the data redundancy present in the natural image and video domain. Accordingly, we now introduce works whose focus is to reduce the token sequence length, whether by dropping them directly or merging them into more compact features. 2.1 Token Dropout A simple method to reduce the sequence length in transformers is to drop tokens, i.e. token dropout. This has been applied to vision problems in multiple ways: tokens can be dropped at random Han et al. 2022 ; Liu et al. 2023 or via a learned module Rao et al. 2021 ; Yin et al. 2022 ; Chen et al. 2023 . In particular, Liu et al. 2023 introduces a scheme whereby image classification training is expedited by applying random token dropout to vanilla ViTs, however, at inference time the whole token sequence is used. Similarly, Han et al. 2022 trains a partial masked auto-encoder for video understanding by dropping out significant portions of the input and then partially reconstructing the video. Dropout is well established in the literature, but emphasis has not been placed on the speedup of transformers at inference time without re-training. 2.2 Token Merging Tokens have been merged in a variety of ways to reduce the information lost by dropout. In Liang et al. 2022 innattentive tokens are fused into a single “background” token, while others exploit semantics to improve features of interest Zeng et al. 2022 ; Zhou et al. 2023 . Other works Lee et al. 2024 optimise token merging through trainable parameters. Most recently, Koner et al. 2024 trains models with an extra stream of compressed tokens to reduce attention overhead. Alternatively, there are many works that make use of token merging, as established in Bolya et al. 2023 ; Bolya and Hoffman 2023 , where image tokens are merged via a weighted average by reusing attention keys as a similarity metric. The primary motivation of the method is that it can be applied to ViTs without re-training or additional learned modules . This has been extended to VLMs Cao et al. 2023 ; Ren et al. 2023 ; Choi et al. 2024 ;