LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information
Ke Wang, Hong Xuan · 2024
Read LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information by Ke Wang, Hong Xuan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08771.txt.
Introduction
Ke Wang and Hong Xuan tackle a cost problem in large models that read both images and text, in a paper from twenty twenty-four. In systems like LLaVA, each image is split into many small tokens that eat up the model's limited space. Long prompts with several images or videos then run slowly and work worse. The authors build on an open model and add a method that shrinks the image tokens on the fly.
Their method, called Dynamic Feature Map Reduction, cuts the token count based on what the image itself holds. Tests show the trimmed model does better across varied image lengths than the base model. The gain matters most where computing power is scarce, such as in schools, and may also help build training data at larger firms.
Catalog record
| CREATOR | Ke Wang, Hong Xuan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08771
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information Ke Wang, Hong Xuan Abstract Multi-modal large language models (MLLMs) utilizing instruction-following data, such as LLaVA, have achieved great progress in the industry. A major limitation in these models is that visual tokens consume a substantial portion of the maximum token limit in large language models (LLMs), leading to increased computational demands and decreased performance when prompts include multiple images or video. Industry solutions often mitigate this issue by increasing computational power, but this approach is less feasible in academic environments with limited resources. In this study, we propose Dynamic Feature Map Reduction (DFMR) based on LLaVA-1.5 to address the challenge of visual token overload. DFMR dynamically compresses the visual tokens, freeing up token capacity. Our experimental results demonstrate that integrating DFMR into LLaVA-1.5 significantly improves the performance of LLaVA in varied visual token lengths, offering a promising solution for extending LLaVA to handle multi-image and video scenarios in resource-constrained academic environments and it can also be applied in industry settings for data augmentation to help mitigate the scarcity of open-domain image-text pair datasets in the continued pretraining stage. Figure 1: Overall Structure of LLaVA-Dynamics. The core module is the Dynamic Feature Map Reduction (DFMR), which dynamically compresses the visual tokens from the visual encoder based on the calculated image information. Results Summary. The proposed DFMR module demonstrates clear improvements in all benchmark tasks over the baseline and the original LLaVA-1.5, see Table 1 . 1 Introduction Recently, the multimodal large language models (MLLMs), especially based on instruction following data, LLaVA family [ 1 , 2 , 3 ] , have achieved great progress in the industry. One of the major limitations of these models is that the visual tokens consume a substantial portion of the maximum token limit in the pre-trained large language model (LLM). To address this, researchers apply the AnyRes method [ 1 ] , which splits the single input image into multiple sub-images and uses the visual tokenizer to tokenize all the sub-images and optionally the original images. The method has proved effective and can be widely applied in industrial settings with sufficient computation power. However, in academic settings, where computing resource is constrained. The increase of visual tokens is limited by the maximum GPU memory and maximum token length in the pre-trained LLM. To address the problems, researchers have concentrated on reducing the number of visual tokens. Li et al. [ 4 ] introduced BLIP-2, which utilizes a Q-former to compress image information into a fixed set of visual tokens, effectively condensing the visual input. Recent works have proposed methods that leverage textual guidance to identify and retain important visual tokens [ 5 , 6 ] . Arif et al. [ 7 ] presented HiRED, an attention-guided token dropping technique that employs CLIP attention masks to split images and selectively discard less relevant tokens, enhancing efficiency in resource-constrained environments. Additionally, Cai et al. [ 8 ] developed Matryoshka Multimodal Models, which manage image representations by organizing visual tokens in a coarse-to-fine hierarchy. Generally, the current approach focuses on the use of external information outside of the image itself to facilitate the compression of visual tokens, instead of fully leveraging the information the image can provide. In this paper, we propose DFMR (Dynamic Feature Map Reduction), a method that dynamically adjusts the compression ratio of visual tokens based solely on the intrinsic content of each image. By introducing a novel metric that quantifies the information within an image, DFMR determines the optimal number of visual tokens required for effective representation. This enables the model to allocate computational resources more efficiently—focusing on complex images rich in information while compressing simpler ones to reduce redundancy. To validate the effectiveness of DFMR, we conduct experiments on various mainstream multimodal evaluation benchmarks. The results demonstrate that our method improves performance across all eight tasks compared with LLaVA-1.5 and our designed baseline in compressed visual tokens. In summary, the key contributions of this paper are: • We propose DFMR for dynamically compressing visual tokens based on the intrinsic information of images. • We develop a metric to quantify image information, guiding compression ratio determination. • We show that DFMR improves model performance in scenarios involving compressed visual tokens. 2 Related Work MLLMs. The most promising approach to bridging visual and textual information involves integrating visual information into LLMs. Flamingo [ 9 ] proposed using cross-attention mechanisms to incorporate encoded image information directly into LLM layers. In contrast, BLIP-2 [ 4 ] introduced a Q-Former to effectively bridge encoded visual information into the input space of LLMs. LLaVA [ 1 , 2 , 3 ] adopted a simpler approach by utilizing a multilayer perceptron (MLP) to map encoded visual information into the LLM’s input space while incorporating high-quality data and adaptive image segmentation techniques to handle multiple images and videos efficiently. Visual token compression. The compression of visual tokens can be broadly categorized into three approaches: QFormer-like, LLM-assisted, and handcrafted metrics methods [ 10 ] . The QFormer-like method [ 4 , 11 ] maps any number of images to a fixed number of visual tokens, which often results in information loss for tasks requiring fine-grained details. The LLM-assisted method [ 5 , 6 ] utilizes LLMs to perform visual token pruning but does not fully exploit the internal information pre-trained from images. The heuristic approach [ 7 , 8 ] relies on handcrafted metrics to evaluate the importance of visual tokens. Our proposed method falls into the third category, where we propose a metric to measure the intrinsic information represented by the image and dynamically compress the image based on this metric. 3 Method 3.1 Overall Structure The overall structure of the proposed model is illustrated in Fig. 1 . Our model is developed based on LLaVA-1.5 [ 2 ] , with an additional module, Dynamic Feature Map Reduction (DFMR), inserted between the vision encoder and the projector. The DFMR dynamically compresses the visual tokens based on the image’s inherent information. Given a resized input given image \mathbf{I}\in\mathbb{R}^{H\times W\times 3} , it is passed through the vision encoder to obtain visual tokens \mathbf{V}\in\mathbb{R}^{N_{v}\times D_{v}} , where N_{v}=H^{\prime}\times W^{\prime} is the number of visual tokens determined by the image feature map size H^{\prime} and W^{\prime} after the vision encoder, and D_{v} is the embedding dimension of the visual tokens. The obtained visual tokens \mathbf{V} are then passed through the proposed DFMR module to produce compressed visual tokens \mathbf{V}^{\prime}\in\mathbb{R}^{N_{c}\times D_{v}} , where N_{c}=\left(\frac{H^{\prime}}{s}\times\frac{W^{\prime}}{s}\right) and s is the compression factor of dynamic pooling. The compressed visual tokens \mathbf{V}^{\prime} are flattened and connected to the LLM via the projector, resulting in projected visual tokens \mathbf{V}^{p}\in\mathbb{R}^{N_{c}\times D_{l}} , where D_{l} is the embedding dimension of the LLM. For a given text input, we tokenize it to obtain textual tokens \mathbf{T}\in\mathbb{R}^{N_{t}\times D_{l}} , where N_{t} is the number of textual tokens. The projected visual tokens \mathbf{V}^{p} and textual tokens \mathbf{T} are concatenated to form the input sequence to the LLM \mathbf{S}=[\mathbf{V}^{p};\mathbf{T}]\in\mathbb{R}^{(N_{c}+N_{t})\times D_{l}} , which is then fed into the LLM for next token prediction. 3.2 Dynamic Feature Map Reduction (DFMR) DFMR conducts an s\times s average pooling operation on the original visual tokens \mathbf{V}\in\mathbb{R}^{N_{v}\times D_{v}} . First, it