ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
Seungdong Yoa, Seungjun Lee, Hyeseung Cho, Bumsoo Kim, Woohyung Lim · 2024
Read ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition by Seungdong Yoa, Seungjun Lee, Hyeseung Cho, Bumsoo Kim, Woohyung Lim (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16491.txt.
Introduction
Vision transformers read images as grids of small patches, but a single patch rarely means anything on its own, which wastes computation. Yoa and co-authors propose ImagePiece, a re-tokenizing step borrowed from language processing: neighboring patches merge until each token carries real meaning. The merging follows a longest-match habit, grouping locally coherent pieces first.
The trick stacks cleanly on top of existing token-cutting methods. On a standard image benchmark it sped up a small transformer by over half while slightly raising accuracy. At extreme speed settings it beat rival methods by more than eight accuracy points. The authors pitch it as a general front end for faster image recognition.
Catalog record
| CREATOR | Seungdong Yoa, Seungjun Lee, Hyeseung Cho, Bumsoo Kim, Woohyung Lim |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition Seungdong Yoa, Seungjun Lee, Hyeseung Cho, Bumsoo Kim, Woohyung Lim Abstract Vision Transformers (ViTs) have achieved remarkable success in various computer vision tasks. However, ViTs have a huge computational cost due to their inherent reliance on multi-head self-attention (MHSA), prompting efforts to accelerate ViTs for practical applications. To this end, recent works aim to reduce the number of tokens, mainly focusing on how to effectively prune or merge them. Nevertheless, since ViT tokens are generated from non-overlapping grid patches, they usually do not convey sufficient semantics, making it incompatible with efficient ViTs. To address this, we propose ImagePiece, a novel re-tokenization strategy for Vision Transformers. Following the MaxMatch strategy of NLP tokenization, ImagePiece groups semantically insufficient yet locally coherent tokens until they convey meaning. This simple retokenization is highly compatible with previous token reduction methods, being able to drastically narrow down relevant tokens, enhancing the inference speed of DeiT-S by 54% (nearly 1.5 × \times faster) while achieving a 0.39% improvement in ImageNet classification accuracy. For hyper-speed inference scenarios (with 251% acceleration), our approach surpasses other baselines by an accuracy over 8%. Introduction Figure 1: An illustration of the ImagePiece pipeline compared to WordPiece in NLP. While text is tokenized into meaningful tokens, image tokens from patches often contain irrelevant or non-semantic information. The retokenization in ImagePiece enables these non-semantic tokens to be merged into meaningful tokens, particularly when they have the potential to become meaningful after retokenization. “All that is gold does not glitter, not all those who wander are lost.” — J.R.R. Tolkien Tokenization involves breaking down text into smaller units called tokens, such as words, and is a crucial preprocessing step for nearly all Natural Language Processing (NLP) tasks. Modern NLP models like BERT ( Devlin et al. 2018 ) , GPT ( Brown et al. 2020 ) , and XLNet ( Yang et al. 2019 ) tokenize text into subword units. These subword units strike a balance between words and characters, preserving linguistic meaning while minimizing out-of-vocabulary issues even with a relatively small vocabulary. Subword tokenizer such as wordpiece ( Devlin et al. 2018 ) employs a greedy longest-match-first strategy, iteratively selecting the longest prefix of the remaining text that matches a vocabulary token, a method known as Maximum Matching or MaxMatch. Following the pioneering success of Transformers in natural language processing, Vision Transformers ( Yin et al. 2022 ) have showcased remarkable performance across a broad spectrum of computer vision tasks. In the context of images, tokenization involves splitting an image into non-overlapping grid patches and feeding the sequence of linear embeddings of these patches into a Transformer. Here, image patches are treated similarly to tokens (words) in NLP. However, unlike textual tokens stemmed from subwords where most elements contain meaning that contributes to understanding, image patch-based tokens differ significantly as they often 1) contain totally irrelevant semantics with the overall meaning (e.g., roads, sky, background) or 2) require a broader context to convey meaning (for example, in Fig 1 , the blue patch itself (a) hardly contains any semantics while its collection with nearby patches (b) can be correctly recognized as a ‘bus’). This phenomenon becomes a more serious issue in further works streamlining ViTs; i.e., reducing the number of input tokens ( Liu et al. 2021 ; Dong et al. 2022 ; Fan et al. 2021 ; Li et al. 2022 ; Xiao et al. 2021 ; Chu et al. 2021 ; Wang et al. 2021 ; Graham et al. 2021 ) . In ViTs, token reduction is mainly achieved through two primary methods: 1) token pruning : removing inattentive tokens or 2) token merging : fusing redundant (or similar) tokens into new abstractions. Both approaches effectively reduce the complexity of Vision Transformers (ViTs). However, due to the limitations of patch-based tokenization, this results in either 1) discarding tokens too hastily before their meaning has fully contextualized, or 2) prematurely smoothing semantic tokens with the nearest-similarity non-semantic noise, rather leading to more serious degradation than token pruning in certain scenes. To address this, we propose ImagePiece , a novel re-tokenization strategy for Vision Transformers. Following the MaxMatch strategy of NLP tokenization ( Devlin et al. 2018 ) , ImagePiece groups semantically insufficient (bottom-k) tokens to a level that convey meaning. To add local inductive bias for visual scenes, we add a local coherence bias module consisting of overlapping convolution layers to enhance non-semantic patches with similar positions to have higher similarity. This facilitates the re-tokenization to merge nearby non-semantic tokens until they form meaningful tokens (or end up to be irrelevant with the overall meaning). Then, the attention score of the semantic abstraction of inattentive tokens are measured again: if the abstraction gains relevance with the overall visual semantic, the token is re-organized. For further efficiency, tokens that do not participate in the final semantic even after our re-tokenization are discarded. ImagePiece poses several major advantages: 1) The locally coherent chunks of scattered inattentive tokens are facilitated to be merged to form a semantically meaningful abstraction, enabling the model to perform well-informed contextualization or pruning. 2) The attentive tokens that contain semantically important information are preserved from being merged (or smoothed), mitigating the limitation of previous token-merging pipelines. Benefiting from these advantages, ImagePiece is able to drastically narrow down relevant tokens, enhancing the inference speed of DeiT-S by 54% (nearly 1.5 \times faster) while achieving a 0.39% improvement in ImageNet classification accuracy. This is notable given that other baselines in efficient ViTs ( Liang et al. 2022 ; Bolya et al. 2023 ) suffer from an unignorable degradation in performance as a trade-off for this speed gain. Our contributions are threefold: • We introduce ImagePiece, a novel retokenization strategy for efficient vision transformers that condense semantically irrelevant tokens. After condensing, the semantics of the tokens are re-evaluated, reorganizing tokens that have obtained meaning by gaining more contextual information. • With local cohesive bias, non-semantic tokens that are positionally nearby are facilitated to be condensed together, adding local inductive bias of visual scenes. The condensed tokens that have no relevance after several iterations are discarded for efficiency. • Our method achieves superior performance in ImageNet classification and demonstrates robust performance in hyper-speed inference, outperforming existing baselines. Related Work Vision Transformers. Transformers ( Vaswani et al. 2017 ) , originally prevalent in NLP, have recently attracted significant attention in computer vision, primarily due to their exceptional ability to model long-range dependencies. Vision Transformers (ViTs) ( Dosovitskiy et al. 2021 ) first introduced Transformer backbones to the field of computer vision, and a diverse array of studies ( Touvron et al. 2021a ; Yuan et al. 2021 ; Zhou et al. 2021 ; Touvron et al. 2021b ; Pan et al. 2022 ; Pan, Cai, and Zhuang 2022 ; Liu et al. 2021 ; Dong et al. 2022 ; Li et al. 2022 ; Xiao et al. 2021 ; Wang et al. 2021 ; Graham et al. 2021 ) on ViTs have demonstrated their success in various aspects, ranging from architectural improvements to optimization techniques. One of the core differences between NLP and computer vision applications of transformers lies in how input data is tokenized. In NLP, methods like