Accelerating Vision Transformers with Adaptive Patch Sizes
Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, Kris M. Kitani, [0.2em] Carnegie Mellon University · 2025
Read Accelerating Vision Transformers with Adaptive Patch Sizes by Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, Kris M. Kitani, [0.2em] Carnegie Mellon University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18091.txt.
Introduction
Rohan Choudhury, with five co-authors, speeds up vision transformers by cutting images into patches of mixed sizes. Standard models use one patch size everywhere, so sharp high-resolution pictures create very long token lists. Their adaptive method gives big patches to plain regions and small patches to busy regions of the same image.
Throughput rose sharply on large models while task scores held steady. The scheme can be added to an already trained model and settles in about one extra pass. It also cut time on dense tasks like detection and segmentation. Vision teams get faster models without retraining from scratch.
Catalog record
| CREATOR | Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, Kris M. Kitani, [0.2em] Carnegie Mellon University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18091
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accelerating Vision Transformers with Adaptive Patch Sizes Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, Kris M. Kitani, [0.2em] Carnegie Mellon University Abstract Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting in long input sequence lengths for high-resolution images. We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones. APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance. It can be applied to a previously fine-tuned ViT and converges in as little as 1 epoch. It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up to 30% faster training and inference in visual QA, object detection, and semantic segmentation. Our project page is available at this link . * * footnotetext: Equal Contribution 1 Introduction Vision Transformers (ViTs) ( Dosovitskiy et al. 2020 ) have become the dominant paradigm for visual recognition, but their scalability is limited by the quadratic cost of self-attention with respect to sequence length. Since inputs are divided into fixed-size patches, image resolution directly determines sequence length: higher resolution images yield disproportionately long token sequences despite much higher redundancy. Many prior works have proposed solutions to this issue, typically by merging a fixed proportion of similar tokens ( Bolya et al. 2022 ) or pruning uninformative ones with auxiliary predictors ( Rao et al. 2021 ; Yin et al. 2022 ) . While these reduce theoretical FLOPs, they face two drawbacks. Firstly, a fixed reduction ratio is mismatched to image complexity: merging only half the tokens in a pure white image is insufficient, while merging half the tokens in a busy cityscape is harmful. Secondly, pruning during the forward pass introduces padding and irregular shapes, often negating speedups in practice ( Dehghani et al. 2021 ) . In contrast to vision transformers, language models rely on adaptive tokenizers such as Byte-Pair Encoding ( Sennrich et al. 2016 ) and SentencePiece Kudo & Richardson 2018 , which flexibly assign tokens of varying lengths depending on subword frequency. This reduces input sequence size while improving performance, suggesting that variable-granularity tokenization can be more efficient than fixed-size splits. Figure 1: Adaptive Patch Sizing. We present APT, Adaptive Patch Transformers, which significantly accelerate vision transformer training and inference by patchifying images based on their content. Complex regions receive more, smaller tokens, while simpler, homogeneous regions receive fewer. Our key insight is that a similar idea can be applied to vision transformers. As illustrated in Figure 1 , ViTs use the same amount of computation on a uniform green background as on the complex patches on the head of the bird, despite the significant difference in visual complexity. We introduce the Adaptive Patch Transformer (APT), which addresses this mismatch by varying patch sizes within a single image . Regions that are smooth and redundant can be represented with large patches, while regions rich in detail are allocated smaller patches. This content-aware patchification preserves important information where it matters while reducing redundancy elsewhere. To do this, APT computes entropy at multiple scales and assigns larger patch sizes to regions with the lowest entropy, resulting in significantly fewer input tokens. We then down-sample the larger patches and combine their patch embeddings with the information from the original large patch using a zero-initialized MLP, allowing APT to converge without harming the network. APT speeds up ViT inference and training by almost 40 \% , with even larger boosts for higher resolution images and larger models. When initialized from a self-supervised or large-scale pretrained checkpoint, APT reaches the same performance as the original ViT after fine-tuning. If applied directly to an already fine-tuned ImageNet checkpoint, APT incurs only a small accuracy drop without additional training. With our zero-initialized MLP, this gap can be closed in as little as a single epoch of fine-tuning. We also find that unlike most prior token reduction works, APT can successfully accelerate vision transformers on a wide range of image understanding tasks, such as visual question answering, object detection, and semantic segmentation, while matching the baseline performance. In summary, we (1) introduce the Adaptive Patch Transformer (APT), which accelerates Vision Transformers by up to 40 \% through content-aware patch sizes, with larger gains at higher resolutions and model scales; (2) show that APT preserves the accuracy of standard pretrained models across resolutions and scales; and (3) demonstrate that APT extends beyond ImageNet, performing well on dense prediction and vision-language tasks. 2 Related Work Vision Transformers and Patchification. Vision Transformers (ViTs) ( Dosovitskiy et al. 2020 ) are currently the de facto standard architecture for computer vision backbones ( Xu et al. 2022 ; Kirillov et al. 2023 ; Peebles & Xie 2022 ) . In contrast to language models, which typically use subword tokenizers ( Sennrich et al. 2016 ; Kudo & Richardson 2018 ) with varying numbers of bits per token, ViTs patchify images into equally sized patches, each becoming a token. This can result in an enormous number of tokens, especially at high resolution. Transformer-based generative models ( Peebles & Xie 2022 ; Esser et al. 2020 ) use visual tokenizers, typically using a variational auto-encoder ( Kingma et al. 2013 ; Van Den Oord et al. 2017 ) , to project images into a compressed latent space, reducing the input size significantly. Some recent works explore adaptive visual tokenizers ( Yan et al. 2024 ; Duggal et al. 2024 ) , which dynamically allocate more tokens to more complex visual inputs, but do not meaningfully speed up training or generation. As a result, image understanding tasks are typically limited to lower resolution. Reducing ViT Tokens. Accelerating ViTs by removing tokens is a rich area of research. Methods such as pruning ( Yu & Xiang 2023 ; Yang et al. 2023 ; Zheng et al. 2022 ) , compressed representations ( Wu et al. 2018 ; Park & Johnson 2023 ) , or quantization ( Liu et al. 2021b ; Li et al. 2022c ; Moon et al. 2024 ) remove redundancies or compactly encode parameters, reducing inference time and memory usage. Alternative attention mechanisms, such as linearized ( Katharopoulos et al. 2020 ; Lu et al. 2021 ) or local window attention ( Liu et al. 2021a ; Wei et al. 2023 ; Chen et al. 2023b ) , improve efficiency by limiting token interactions. More related to our work are methods that exploit the inherent redundancy of images ( Meng et al. 2022 ; Yin et al. 2022 ; Kong et al. 2022 ; Rao et al. 2021 ) and videos ( Choudhury et al. 2025 ; Ding et al. 2023 ) by pruning uninformative tokens. While these works are content-aware, most require learning which tokens are unhelpful, negating any training speedup and preventing inference on batch sizes greater than 1. Several methods instead merge tokens based on similarity ( Bolya et al. 2022 ; Bolya & Hoffman 2023