Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens
Jaihyun Lew, Soohyuk Jang, Jaehoon Lee, Seungryong Yoo, Eunji Kim, Saehyung Lee, Jisoo Mok, Siwon Kim, Sungroh Yoon · 2024
Read Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens by Jaihyun Lew, Soohyuk Jang, Jaehoon Lee, Seungryong Yoo, Eunji Kim, Saehyung Lee, Jisoo Mok, Siwon Kim, Sungroh Yoon (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04680.txt.
Introduction
Jaihyun Lew and eight co-authors rethink how vision models chop up images in a 2024 paper. Vision transformers split a photo into a grid of square patches. Each patch becomes a token the model can weigh against others. A square patch can mix two objects into one token, though.
Their method cuts the image into superpixels instead. Each superpixel hugs one visual region, so its token holds a single idea. Odd shapes and sizes make that swap hard to wire in. A two step pipeline first extracts, then merges features with care. Tests show gains in accuracy and toughness across tasks. The approach plugs into existing vision frameworks without deep changes.
Catalog record
| CREATOR | Jaihyun Lew, Soohyuk Jang, Jaehoon Lee, Seungryong Yoo, Eunji Kim, Saehyung Lee, Jisoo Mok, Siwon Kim, Sungroh Yoon |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04680
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens Jaihyun Lew, Soohyuk Jang, Jaehoon Lee, Seungryong Yoo, Eunji Kim, Saehyung Lee, Jisoo Mok, Siwon Kim, Sungroh Yoon Abstract Transformers, a groundbreaking architecture proposed for natural language processing (NLP), have also achieved remarkable success in computer vision. A cornerstone of their success lies in the attention mechanism, which models relationships among tokens. While the tokenization process in NLP inherently ensures that each token maintains semantic integrity without containing multiple meanings, the grid-based tokenization of Vision Transformer (ViT) relies on uniformly partitioned square image patches, which may result in an arbitrary mixing of visual concepts within a token. In this work, we propose a novel tokenization pipeline that replaces the grid-based tokenization with superpixels, encouraging each token to capture a distinct visual concept. Unlike square image patches, superpixels are formed in varying shapes, sizes, and locations, making direct substitution challenging. To address this, our pipeline first generates pixel-level embeddings and efficiently aggregates them within superpixel clusters, producing superpixel tokens that seamlessly replace patch tokens in ViT. Extensive experiments demonstrate that our approach enhances the performance of ViT on various downstream tasks and introduces intriguing properties such as adaptive inference and semantic integrity in tokens. 1 Introduction Figure 1 : A high-level overview of tokenization . (Top) The conventional grid-like tokenization in ViT and (Bottom) our proposed superpixel tokenization. Transformers [ 58 ] , originally proposed for natural language processing (NLP), have demonstrated exceptional performance across diverse tasks such as machine translation [ 62 , 22 , 5 ] , natural language understanding [ 30 , 38 ] , and text generation [ 46 , 34 ] . The transformative power of Transformers was again attested by the release of large language models [ 7 , 56 ] . Similarly, the Vision Transformer (ViT) [ 14 ] , which extended the Transformer architecture to image recognition, changed the landscape of computer vision research. The subsequent research efforts to broaden the usage of ViTs to various visual tasks, including detection [ 50 , 8 , 74 ] , segmentation [ 35 , 72 , 53 ] , generation [ 15 , 42 , 69 ] , and multi-modal applications [ 47 ] , have made the ViT the dominant architecture in computer vision. In NLP, the success of Transformers is commonly attributed to the attention mechanism, which captures contextual dependencies by modeling the relationship among input texts, which are expressed as tokens. The widely adopted tokenization methods in NLP, such as Byte Pair Encoding [ 51 ] , WordPiece [ 52 ] and SentencePiece [ 32 ] , first divide each sentence into words based on whitespaces. For instance, when tokenizing “ John loves Mary ,” each word is separated into: [“ John ”, “ ”, “ loves ,” “ ”, “ Mary ”]. This initial word-level splitting of sentences prevents multiple words with different meanings from being fused into a single token, preserving semantic integrity in tokens. In contrast, in computer vision, fixed-size grid patches form the basis of tokenization in ViT and its variants [ 14 , 55 , 39 ] . As visualized in Figure 1 , when naïvely sliced square image patches are used as tokens, a single token can simultaneously contain two or more visual concepts, e.g., eye and fur. This is analogous to splitting “ John loves Mary ” into arbitrary fixed-size fragments like [“ Joh ,” “ n l ,” “ ove ”, “ s M ”, “ ary ”], disregarding semantic boundaries. Consequently, this grid-based tokenization deviates from the semantic integrity inherent in NLP tokenization, leading to tokens with arbitrarily mixed visual concepts. In this work, we employ the concept of superpixels to design a tokenization process. A superpixel [ 2 , 11 , 36 ] is a cluster of connected pixels in an image that share similar properties, such as color, texture, or position, and is often used as a unit of semantic representation in various studies [ 17 , 16 ] . Building on this property, superpixels emerge as a viable alternative to square tokens for preserving semantic integrity. Figure 1 illustrates how leveraging superpixels alters tokenization of ViT. Grouping pixels into superpixels divides an image into cohesive regions that capture similar semantics, thereby minimizing the mixture of unrelated semantic information in each token. Despite the potential advantages of superpixels in tokenization, the different shapes and sizes of each superpixel present significant challenges in utilizing them as a direct replacement for visual tokens [ 28 , 67 , 71 ] . In ViTs, tokens are generated from fixed-size square patches at predefined locations through simple flattening and linear projection operations. Because superpixels consist of varying numbers of pixels, de facto flattening and linear projection operations in ViT are incompatible with superpixels. Moreover, the location of each superpixel changes dynamically depending on the image. The variability of superpixel locations makes it infeasible to directly apply the positional embeddings of ViT, which are designed for fixed locations. The inadequacy of original ViT operations necessitates novel techniques to address irregular geometry and composition to reap the inherent benefits of superpixels. We thus propose a novel tokenization pipeline that overcomes the aforementioned problems and enables the effective incorporation of superpixels into ViT tokenization. The proposed pipeline consists of two major technical components: pixel-level embedding and superpixel-aware aggregation . The first stage generates pixel-level embeddings consisting of local and positional features. Second, these pixel-level embeddings are then aggregated according to superpixel clusters. The aggregation process employs pooling operations to remove the irregularity and variability of superpixels. Collectively, this two-stage process allows the proposed tokenization approach to leverage the benefits of superpixels while preserving important details of an image. We demonstrate the superiority of our Su perpixel-Tokenized V i sion T ransformer (SuiT) compared to other baselines. SuiT outperforms in a variety of tasks including ImageNet-1K classification [ 12 ] , transfer learning [ 75 ] , and zero-shot segmentation [ 64 ] . We further analyze intriguing properties that emerge using our tokenization pipeline, such as adaptive inference and semantic integrity. Our contribution can be summarized as follows: • We propose an effective superpixel-based tokenization pipeline that overcomes the challenges associated with the use of superpixels. • We empirically show that our tokenization pipeline induces notable properties including adaptive inference and semantic integrity in tokens. • We demonstrate the effectiveness of our approach through experiments on image classification, transfer learning, and zero-shot segmentation. 2 Related Work Superpixel-based Vision Models Superpixels, which are perceptually meaningful clusters of pixels, have been a foundational concept in computer vision [ 2 , 11 , 36 ] . Previous studies have attempted to use superpixels in several tasks such as segmentation [ 33 , 41 ] and object detection [ 21 ] . A recent work, STViT [ 26 ] , applies superpixel-like clustering named supertoken attention in the Transformer backbone. However, unlike superpixels which are formed at the pixel level, clusters in STViT are formed at the square token level, resulting in coarser granularity. CoC [ 40 ] borrows the concept of superpixels within neural networks, but does not actually exploit superpixels in the network. Similarly, SPFormer [ 41 ] utilizes