The Effect of Compression Techniques on Large Multimodal Language Models in the Medical Domain

Tanvir A. Khan, Aranya Saha, Ismam N. Swapnil, Mohammad A. Haque · 2025

Read The Effect of Compression Techniques on Large Multimodal Language Models in the Medical Domain by Tanvir A. Khan, Aranya Saha, Ismam N. Swapnil, Mohammad A. Haque (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21976.txt.

Introduction

Tanvir Khan, Aranya Saha, Ismam Swapnil, and Mohammad Haque wrote this machine learning paper, published in 2025. It tests ways to shrink large multimodal language models, which read both images and text, so they can run on modest hardware in medical settings.

Big models are costly to run, a real barrier in clinics. The team fine-tuned a vision-language model for medical use, then pruned parts of it and rounded its numbers down with care for which parts matter most. They also add a new way to pick which layers to prune. The squeezed seven-billion-parameter model fits in four gigabytes of graphics memory, a seventy percent saving. It also scored slightly higher than models shrunk by older methods at the same size. The paper maps the trade-offs of a prune, fine-tune, then quantize pipeline.

Catalog record

CREATORTanvir A. Khan, Aranya Saha, Ismam N. Swapnil, Mohammad A. Haque
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21976

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Effect of Compression Techniques on Large Multimodal Language Models in the Medical Domain Tanvir A. Khan, Aranya Saha, Ismam N. Swapnil, Mohammad A. Haque Abstract Multimodal Large Language Models (MLLMs) hold huge potential for usage in the medical domain, but their computational costs necessitate efficient compression techniques. This paper evaluates the impact of structural pruning and activation-aware quantization on a fine-tuned LLAVA model for medical applications. We propose a novel layer selection method for pruning, analyze different quantization techniques, and assess the performance trade-offs in a prune-SFT-quantize pipeline. Our proposed method enables MLLMs with 7B parameters to run within 4 GB of VRAM, reducing memory usage by 70 % 70\% while achieving 4 % 4\% higher model performance compared to traditional pruning and quantization techniques in the same compression ratio. keywords Multimodal LLM, Model Pruning, Quantization, Supervised Finetuning 1 Introduction We present a unified and efficient pipeline for deploying multimodal large language models (MLLMs) in domain-specific settings, with a focus on dermatological visual question answering (VQA). Real-time clinical workflows require low-latency models, yet the substantial memory and computational demands of LLMs and MLLMs hinder their deployment in resource-constrained edge or cloud environments. To tackle these challenges, we introduce a compression pipeline that integrates structural pruning, supervised fine-tuning (SFT), and quantization for task specific purposes. Our approach begins with structured pruning to remove redundant parameters, reducing the model’s size while preserving essential functionality. We employ a novel layer pruning criterion for domain specific applications. Since pruning can lead to performance drops, we leverage SFT to restore task-specific capabilities. Following pruning and fine-tuning, we incorporate Activation-aware Weight Quantization, which significantly boosts memory efficiency with minimal performance degradation. We show that traditional calibration free quantization techniques (such as bitsandbytes Dettmers et al. 2022 , hqq Badri and Shaji 2023 ) fails to retain language modeling capacity of pruned LLMs. We validate our method on dermatological VQA tasks using the LLaVA (Large Language and Vision Assistant) Liu et al. 2023 model, demonstrating its effectiveness for skin disease diagnosis. Our results show that compressed MLLMs can retain strong performance while being suitable for deployment in real-world clinical environments. 2 Related Work 2.1 Multi-modal Large Language Models Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like text across diverse domains. These models, trained on vast amounts of textual data, leverage deep learning architectures such as transformers to capture complex linguistic patterns. Despite their success in natural language processing (NLP), traditional LLMs are limited to text-based inputs, restricting their applicability in fields requiring multimodal understanding Wu et al. 2023 . To address this limitation, Multimodal Large Language Models (MLLMs) extend LLMs by integrating multiple data modalities, such as images, audio, and structured medical data, alongside text. By incorporating vision-language pretraining techniques, these models can interpret and generate responses based on both textual and visual information, making them highly effective in perception-driven tasks. In healthcare, MLLMs can play a crucial role in medical image interpretation, diagnostic assistance, and patient-doctor interaction modeling Santomauro et al. 2023 . They can analyze dermatological images, radiology scans, and pathology slides while cross-referencing clinical notes for comprehensive assessments. Additionally, MLLMs enhance medical chatbots and virtual assistants, improving accessibility to healthcare information and decision support for professionals. By bridging the gap between language and vision, these models contribute to more accurate and context-aware medical diagnostics. As MLLMs continue to evolve, they hold the potential to revolutionize healthcare by enabling AI-driven, multimodal clinical decision-making and patient care. However, the deployment of Multimodal Large Language Models (MLLMs) is constrained by the high cost of model serving. LLMs contain billions of parameters, making their inference computationally expensive, particularly for consumer applications. Deploying these models on GPUs requires significant memory and processing power, which may not be feasible for many organizations. In the medical domain, where data privacy and security are paramount, cloud-based solutions are often impractical, necessitating on-premise deployment . To mitigate the computational burden and reduce serving costs, model compression techniques can be applied. Pruning and Quantization are the most popular model compression techniques. These techniques, when applied effectively, enable efficient deployment of MLLMs without substantial performance degradation. By leveraging compression, MLLMs can be made more accessible and cost-effective, facilitating their adoption in real-world healthcare applications. 2.2 Pruning Pruning removes less significant parameters from a model, reducing memory and computational requirements while preserving core functionality. However, pruning large language models (LLMs), particularly Transformer-based architectures, presents unique challenges due to their dense attention mechanisms and inter-layer dependencies. Pruning can be broadly classified into two types, Unstructured and Structured. Structural pruning is preferred when it is important to save memory resources as it does not require specialized hardware optimized for sparse matrix multiplication. A notable development in structural pruning is depth pruning , which involves removing entire Transformer blocks. Kim et al. Kim et al. 2024 explored this technique in LLaMA, showing that selective removal of less impactful layers leads to large speedups in inference time. They further highlighted the importance of continued pretraining (CPT) to restore performance after aggressive pruning. Depth pruning presents a compelling solution for memory-constrained deployment, especially when coupled with fine-tuning strategies to recover performance. Different works have proposed different algorithms to determine which layers of model contributed the least to the generation of output. The baseline method for layer importance detection is weight magnitude Han et al. 2015 . The sum of weight magnitude is taken as the layer importance and the layers with the least magnitude are removed. ShortGPT Men et al. 2024 uses the cosine similarity between the embeddings of the input and output of a layer to derive layer importance. 2.3 Quantization Quantization is a compression technique that maps floating-point values to low-bit integer representations, reducing both memory usage and computation cost. It can be expressed for a given weight group w and an input x as a typical linear transformation is y=wx . In standard post-training quantization, the quantized weights \hat{w} are computed as: Q(w)=\Delta\cdot\text{Round}\left(\frac{w}{\Delta}\right),\quad\Delta=\frac{\max(|w|)}{2^{N-1}}, (1) where N is the bit-width (e.g., 3 or 4), and \Delta is the quantization scale derived from the maximum absolute value in the weight group. Quantization is broadly categorized into Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). QAT integrates quantization during training, using backpropagation to update quantized weights Liu et al. 2023 . While effective, it is computationally intensive and difficult to scale to LLMs. Recent works such as QLoRA, PEQA, and LoftQ Dettmers et al. 2023 combine QAT with Parameter-Efficient Fine-Tuning (PEFT) to address this limitation. L4Q Jeon et al. 2024 further introduces a LoRA-wise learned step size to improve generalization across tasks. In contrast, PTQ offers a training-free alternative, making it highly suitable for LLM compression. Common configurations include weight-only INT4 (W4A16) and full INT8 (W8A8) quantization. PTQ can be subdivided into weight-only, weight-activation, and key-value (KV) cache quantization. Among PTQ methods, Activation-aware Weight Quantization (AWQ) Lin et al. 2024 has gained prominence. AWQ identifies and preserves 0.1%–1% of high-impact weights by analyzing activation distributions instead of weight magnitudes. It applies per-channel scaling to protect important features, maintaining hardware efficiency while achieving strong performance in both vision and language tasks. AWQ outperforms traditional round-to-nearest (RTN) methods and rivals mixed-precision approaches without the hardware complexity. There are

Read the complete text