Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning
Bonan LiUCAS & NUS [email protected], Zicheng [email protected] [email protected] [email protected] [email protected] · 2025
Read Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning by Bonan LiUCAS & NUS [email protected], Zicheng [email protected] [email protected] [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11945.txt.
Introduction
Bonan Li, Zicheng Zhang, Songhua Liu, Weihao Yu, and Xinchao Wang present LLaVA-Meteor, a leaner way to feed images into a large language model. Vision instruction tuning makes models answer questions about pictures, but shrinking the image into tokens usually trades accuracy for speed. Their top-down compression claims both.
A trainable fusion module, built on fast state-space operators, lets every visual token sense the whole scene and the user's instruction before compression. A local-to-single scanning pattern then preserves fine dependencies. Tokens are cut hard while core information survives. Efficient vision-language models matter for phones, robots, and cheap serving.
Catalog record
| CREATOR | Bonan LiUCAS & NUS [email protected], Zicheng [email protected] [email protected] [email protected] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11945
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning Bonan LiUCAS & NUS [email protected], Zicheng [email protected] [email protected] [email protected] [email protected] Abstract Visual instruction tuning aims to enable large language models to comprehend the visual world, with a pivotal challenge lying in establishing an effective vision-to-language projection. However, existing methods often grapple with the intractable trade-off between accuracy and efficiency. In this paper, we present LLaVA-Meteor , a novel approach designed to break this deadlock, equipped with a novel Top-Down Compression paradigm that strategically compresses visual tokens without compromising core information. Specifically, we construct a trainable Flash Global Fusion module based on efficient selective state space operators, which aligns the feature space while enabling each token to perceive holistic visual context and instruction preference at low cost. Furthermore, a local-to-single scanning manner is employed to effectively capture local dependencies, thereby enhancing the model’s capability in vision modeling. To alleviate computational overhead, we explore a Visual-Native Selection mechanism that independently assesses token significance by both the visual and native experts, followed by aggregation to retain the most critical subset. Extensive experiments show that our approach reduces visual tokens by 75%–95% while achieving comparable or superior performance across 12 benchmarks, significantly improving efficiency. 1 Introduction Visual instruction tuning [ 42 , 40 , 31 , 30 ] has emerged as a promising pipeline to extend large language models (LLMs) into the visual domain, achieving notable progress and demonstrating impressive performance across a wide range of vision-language tasks [ 27 , 63 , 24 , 2 , 47 , 21 ] , e.g. , image captioning [ 46 , 62 , 69 ] , visual question answering [ 23 , 61 ] and open-vocabulary object detection [ 22 , 60 ] . To perceive the visual world, current mainstream approaches ( e.g. , LLaVA [ 42 ] ) typically project sequential visual tokens into the linguistic space, which are subsequently incorporated with textual representation and jointly interpreted by the LLMs decoder. However, the sheer number of visual tokens significantly increases the computational burden on LLMs. For instance, the widely used CLIP ViT-L/336px [ 53 ] encodes a single \text{672}\times\text{1008} image into \text{48}\times\text{72}=\text{3456} tokens. To mitigate this, a variety of token compression techniques [ 25 , 20 , 57 , 65 , 66 , 73 , 9 , 54 ] have been proposed to condense visual information into a reduced set of tokens, thereby enhancing model efficiency. Broadly, these approaches fall into two categories: Fusion and Selection. Fusion-based methods either apply predefined merging operations ( e.g. , PixelShuffle [ 28 , 7 , 11 ] ), which often compromise the structural integrity of the features, or learn to condense the full token sequence into a smaller fixed set via trainable modules ( e.g. , Q-Former [ 3 , 30 , 4 , 10 ] ), which typically require substantial additional parameters and large-scale datasets for effective training. In contrast, Selection-based approaches identify a subset of important tokens using task-agnostic visual cues ( e.g. , class token attention [ 66 , 43 , 70 ] ), but often overlook instruction relevance, limiting their alignment with downstream tasks. Figure 1 : Illustration of visual token compression strategies. (a) Naive and learnable fusion methods reduce token count via predefined rules or learned aggregation, but may lose structural fidelity or require heavy training. (b) Vision-based selection filters important tokens using attention, yet ignores instruction relevance. (c) Our Top-Down Compression combines Flash Global Fusion for global context propagation and instruction preference summarization, with Visual-Native Selection to jointly evaluate token importance from visual and native perspectives. Despite notable progress, achieving the best of both worlds—accuracy and efficiency—remains an open challenge in visual instruction tuning. Existing fusion-based compression methods typically reduce the number of tokens either by predefined merging rules or through learned token aggregation. While effective in shortening the input length, such approaches often disrupt the structural integrity of visual features or introduce significant training complexity. To address this, we introduce a novel fusion module that retains the full token sequence and enhances each token’s representation through global information propagation at low cost. Additionally, we design a dedicated instruction token that implicitly encodes instruction-level preferences by attending to the full visual context during global fusion. The instruction token is not conditioned on explicit prompts; rather, it learns to focus on instruction-relevant regions via weak supervision over large-scale vision-language data, where common user attention patterns such as phone number, faces, or salient objects are consistently reinforced. On the other hand, selection-based methods aim to identify the most important tokens using task-agnostic cues, such as attention scores from the class token. However, these approaches often overlook instruction relevance, leading to suboptimal token retention for instruction-following tasks. To mitigate this issue, we propose a native expert based on the instruction token, which complements the visual expert (class token) by providing instruction task guidance for token importance scoring. Integrating these two complementary components, we propose a unified visual token compression framework, termed Top-Down Compression. This paradigm first performs top-level global fusion to establish semantic context and task-aware priors, and subsequently applies bottom-level token selection guided by both visual and instruction-informed experts. As illustrated in Figure 1 , the fusion stage (Top) facilitates holistic context exchange and distills instruction priors into the instruction token. In the subsequent selection stage (Down), each token is evaluated from two perspectives: a visual expert that highlights globally salient regions, and a native expert that captures task-specific relevance. Based on these complementary signals, we select a compact and informative token subset to be forwarded to the LLMs, thereby achieving an effective balance between semantic richness and computational efficiency. Building upon the proposed Top-Down Compression strategy, we present a novel and efficient vision-language model named LLaVA-Meteor , which comprises two key components: Flash Global Fusion (FGF) and Visual-Native Selection (VNS). In the FGF module, we leverage state space models (SSMs) [ 15 ] as the core operator due to their linear computational complexity and strong capability for long-range sequence modeling. This enables efficient propagation of global contextual information across the visual token sequence. To further enhance spatial understanding, we incorporate a local-to-single scanning strategy, which captures local dependencies by summarizing information from spatial neighborhoods before each token. In addition, a learnable instruction token is inserted at the center of the token sequence and trained to aggregate holistic semantic information while implicitly encoding instruction-related cues. In the VNS module, we adopt a dual-expert evaluation mechanism. The class token, obtained from a frozen vision encoder, serves as a visual expert and provides attention-based scores to reflect general visual importance. Meanwhile, the instruction token, enriched by the Flash Global Fusion module, serves as a native expert that evaluates instruction relevance by computing its similarity with each token in the sequence. To synthesize these two complementary perspectives, we aggregate their scores at each token index and select the Top- K tokens with the highest combined importance as the final visual input to the LLMs. In summary, our contributions can be summarized as follows: ( i ) We construct the vision-language model LLaVA-Meteor upon the simple and effective Top-Down Compression paradigm, simultaneously achieving strong performance and high efficiency; ( ii ) We introduce a lightweight fusion