EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models
Yinan Liang 1 , Ziwei Wang 2 , Xiuwei Xu 1 , Jie Zhou 1 , Jiwen Lu 1 1 Tsinghua University 2 Nanyang Technological University {liangyn24, xxw21}@mails.tsinghua.edu.cn [email protected] {jzhou, lujiwen}@tsinghua.edu.cn · 2025
Read EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models by Yinan Liang 1 , Ziwei Wang 2 , Xiuwei Xu 1 , Jie Zhou 1 , Jiwen Lu 1 1 Tsinghua University 2 Nanyang Technological University {liangyn24, xxw21}@mails.tsinghua.edu.cn [email protected] {jzhou, lujiwen}@tsinghua.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15369.txt.
Introduction
Yinan Liang, Ziwei Wang, Xiuwei Xu, Jie Zhou, and Jiwen Lu prune large vision-language models automatically, using almost no data. These models reason well but are heavy to deploy on small devices. Ordinary pruning needs the original training data to choose cut ratios, an impossible ask at web scale. Their method searches for a pruning policy with only sixty-four samples.
The search maximizes generalization to unseen data while holding accuracy, formalized through structural risk minimization. Task performance and generalization jointly steer an iterative hunt through the policy space. The vision projector is retrained along the way to raise the space's ceiling. On four visual question answering benchmarks the pruned model kept about eighty-three percent accuracy on the science set. It also ran nearly twice as fast as the original. Efficiency, here, is searched for rather than hand-tuned.
Catalog record
| CREATOR | Yinan Liang 1 , Ziwei Wang 2 , Xiuwei Xu 1 , Jie Zhou 1 , Jiwen Lu 1 1 Tsinghua University 2 Nanyang Technological University {liangyn24, xxw21}@mails.tsinghua.edu.cn [email protected] {jzhou, lujiwen}@tsinghua.edu.cn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15369
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models Yinan Liang 1 , Ziwei Wang 2 , Xiuwei Xu 1 , Jie Zhou 1 , Jiwen Lu 1 1 Tsinghua University 2 Nanyang Technological University {liangyn24, xxw21}@mails.tsinghua.edu.cn [email protected] {jzhou, lujiwen}@tsinghua.edu.cn Abstract While multimodal large language models demonstrate strong performance in complex reasoning tasks, they pose significant challenges related to model complexity during deployment, especially for resource-limited devices. In this paper, we propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning. Conventional methods rely on the training data of the original model to select the proper pruning ratio for different network components. However, these methods are impractical for large vision-language models due to the unaffordable search costs caused by web-scale training corpus. In contrast, our approach only leverages a small number of samples to search for the desired pruning policy by maximizing its generalization ability on unknown training data while maintaining the model accuracy, which enables the achievement of an optimal trade-off between accuracy and efficiency for large visual language models. Specifically, we formulate the generalization gap of the pruning strategy using the structural risk minimization principle. Based on both task performance and generalization capability, we iteratively search for the optimal pruning policy within a given search space and optimize the vision projector to evolve the search space with higher upper bound of performance. We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering. Using only 64 samples for pruning policy search, EfficientLLaVA achieves an accuracy of 83.05% on ScienceQA, along with a × \times 1.8 speedup compared to the dense LLaVA-v1.5-7B model. 1 Introduction Multimodal large language models have been widely adopted in complex reasoning tasks such as visual question answering [ 34 , 10 ] , embodied task planning [ 12 , 28 ] and dialogue systems [ 16 ] . Although they achieve excellent performance in high-level tasks with rich commonsense, the inference stage of large vision-language models (LVLMs) in deployment requires large memory footprint and long latency because of the tremendous neurons. In order to deploy the powerful LVLMs on mobile devices such as cellphones, navigation robots, and autonomous vehicles with a strict computational resource limit, we are required to reduce the model complexity without obvious degradation of performance. Figure 1 : The trade-off between accuracy and inference speed on ScienceQA with EfficientLLaVA. Network pruning aims to remove unimportant components including layers [ 38 ] , channels [ 25 ] , tokens [ 41 , 42 ] , and neurons [ 44 ] to reduce the storage and computational complexity without harming the task performance, which have been widely studied for computer vision [ 4 , 13 ] , natural language processing [ 33 , 25 ] and system control [ 36 ] . The components importance can be defined as the l_{1} and l_{2} magnitude of weights and activations [ 35 ] , and the Jacobian and Hessian matrix regarding the loss functions [ 30 ] . Since different components usually have various sensitivity to pruning in downstream tasks, automatic pruning assigns the optimal pruning ratio to each component in order to achieve higher accuracy-efficiency trade-off. In conventional methods, the training data of the original model is leveraged to evaluate the sampled pruning policy based on the accuracy and the model complexity, which provides feedback to the search algorithm to select better candidates. However, LVLMs require extensive training corpus, and the search cost associated with existing automatic pruning methods is prohibitively high. In this paper, we present EfficientLLaVA that automatically prunes LVLMs to achieve low model complexity and unaffected task performance with an acceptable search cost. Unlike existing automatic pruning methods that leverage the large-scale training data of the original models for pruning policy search, our method only uses a few samples to assign the optimal pruning ratio for weight matrix in each layer. Since we maximize the generalization ability of the pruning policy to the unknown web-scale training corpus, the acquired solution achieves a satisfying trade-off between accuracy and efficiency in a wide variety of downstream tasks. More specifically, we first formulate the generalization gap of the pruning policy between the proxy samples for policy search and the unknown training set via structural risk minimization principle, which can be evaluated by the Frobenius norm of the weight matrix in the pretrained LLMs. We then evolve the pruning policy candidates with the goal of achieving higher task performance and generalization ability, where the weights of MLP and attention are pruned based with the selected pruning ratio. The vision projector is optimized to enhance the upper bound of accuracy and the generalization ability for all policies in the pruning space, which is estimated via the Euclidean distance between the sampled policy and neighbor candidates. The policy candidate selection and the pruning space evolution are implemented iteratively to acquire satisfying accuracy-efficiency trade-off in the extremely large pruning space with acceptable cost. Fig. 1 demonstrates the trade-off between accuracy and inference latency on ScienceQA for uniform pruning methods SparseGPT [ 7 ] and our EfficientLLaVA. We conducted extensive experiments in a wide variety of multimodal reasoning tasks including short-answer, option-only for multiple-choice and natural QA, and the results shows that EfficientLLaVA can achieve 83.05% accuracy on ScienceQA and \times 1.8 speedup compared to the dense LLaVA-v1.5-7B model. Our contributions can be summarized as follows: • We introduce EfficientLLaVA, a novel pruning method for large vision-language models that effectively reduces model complexity while preserving task performance with minimal data usage. • Our approach leverages the structural risk minimization principle to maximize the generalization ability of pruning policies, enabling efficient adaptation to unknown training data. • Extensive experiments demonstrate that EfficientLLaVA achieves a significant speed-up in inference while maintaining competitive accuracy across various multimodal reasoning tasks. 2 Related Work Large Vision-language Models: Large Vision-language Models(LVLMs) fuse visual and linguistic modals via bridging the gap between different modalities that achieved outstanding performance, such as in-context predictions [ 22 , 32 ] , multi-image and chain-of-thought reasoning [ 40 , 6 ] . To utilize a learnable connector module projecting visual information into language modal space, token-level fusion and feature-level fusion are presented for encoding multimodal information to LLMs. For token-level fusion, LLaVA [ 20 ] series utilized linear MLPs to align the image tokens with language tokens, finetuning LVLMs with visual instruction samples to enhance zero-shot capabilities. MM1 [ 27 ] compared the significance of various architectural aspects and found that the number of visual tokens and input resolution were more important than the projector design. For feature-level fusion, Flamingo [ 1 ] inserted extra cross-attention layers to connect pre-trained vision-only and language-only models, significantly boosting learning efficiency. CogVLM [ 37 ] integrated a visual expert module into every Transformer layer to fuse vision and language features. Existing methods usually focus on the performance of LVLMs, ignoring real-world deployment particularly in environments with limited resources. Despite light-weight LVLMs such as MobileVLM [ 5 ] , TinyLLaVA [ 45 ] , and quantinized methods such as GPTQ [ 8 ] , AWQ [ 17 ] have reduced the size of LMMs, the cost still exceeds the resource constraint of mobile