Benchmarking and Enhancing VLM for Compressed Image Understanding
Zifu Zhang, Tongda Xu, Siqi Li, Shengxi Li, Yue Zhang, Mai Xu, Yan Wang · 2025
Read Benchmarking and Enhancing VLM for Compressed Image Understanding by Zifu Zhang, Tongda Xu, Siqi Li, Shengxi Li, Yue Zhang, Mai Xu, Yan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20901.txt.
Introduction
Zifu Zhang and six co-authors test how vision-language models cope with heavily compressed images. Models usually train on lightly compressed, high-bitrate pictures, so their skill on small, low-bitrate files was unknown. Storage and bandwidth pressures make that question practical, not academic.
Their benchmark holds over one million images, compressed with widely used codecs across many tasks. Breaking down the errors showed two sources: information truly lost in compression, and a fixable failure to generalize. Only the second can be repaired, and their universal adaptor does that. One adaptor lifted performance by ten to thirty percent across codecs and bitrates, without retraining the underlying model for each format.
Catalog record
| CREATOR | Zifu Zhang, Tongda Xu, Siqi Li, Shengxi Li, Yue Zhang, Mai Xu, Yan Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20901
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking and Enhancing VLM for Compressed Image Understanding Zifu Zhang, Tongda Xu, Siqi Li, Shengxi Li, Yue Zhang, Mai Xu, Yan Wang Abstract With the rapid development of Vision-Language Models (VLMs) and the growing demand for their applications, efficient compression of the image inputs has become increasingly important. Existing VLMs predominantly digest and understand high-bitrate compressed images, while their ability to interpret low-bitrate compressed images has yet to be explored by far. In this paper, we introduce the first comprehensive benchmark to evaluate the ability of VLM against compressed images, varying existing widely used image codecs and diverse set of tasks, encompassing over one million compressed images in our benchmark. Next, we analyse the source of performance gap, by categorising the gap from a) the information loss during compression and b) generalisation failure of VLM. We visualize these gaps with concrete examples and identify that for compressed images, only the generalization gap can be mitigated. Finally, we propose a universal VLM adaptor to enhance model performance on images compressed by existing codecs. Consequently, we demonstrate that a single adaptor can improve VLM performance across images with varying codecs and bitrates by 10%-30%. We believe that our benchmark and enhancement method provide valuable insights and contribute toward bridging the gap between VLMs and compressed images. 1 Introduction Figure 1: Visualization of VLM performance drop due to image compression and improvement by our method, measured by BD-Metric. The boom of multimedia services and applications has resulted in dramatic increase in image data, creating significant challenges in terms of transmission bandwidth and storage capacity. To address this, efficient image compression methods are essential for reducing data volume while maintaining or enhancing the subjective visual quality for human perception. Over the past few decades, numerous advanced compression standards have been introduced, such as JPEG ( 45 ) , and VTM ( 4 ) , alongside recent end-to-end learned compression approaches ( 3 ; 33 ; 11 ; 55 ) and generative compression methods ( 35 ; 27 ; 53 ) . Figure 2: Comparative visualization of four image compression technique: uncompressed, traditional codec (JPEG), learning-based codec (ELIC), and generative codec (StableCodec), highlighting their impact on visual clarity and semantic preservation through targeted question-answering. All forms of compression-induced distortion affect the ability of VLMs to understand images. In parallel, the advent of Big Data has transformed the way intelligent machines interact with the world, leading to extensive research into compression techniques for machine vision tasks, as opposed to human vision. Notable examples include image coding for machines (ICM) ( 20 ) and feature coding for machines (FCM) ( 41 ) , emerging standards introduced by the moving picture experts group (MPEG). However, previous works ( 22 ; 56 ; 6 ) have largely focused on specific computer vision tasks, such as object detection and instance segmentation, using fixed backbone networks, which limits their generalization capabilities. With the continuous advancements in the multimodal field, VLMs have developed rapidly ( 46 ; 7 ; 10 ) . Current VLMs are not only capable of understanding complex images ( 49 ) but also performing tasks such as object detection and segmentation ( 13 ) , further generalizing and unifying visual tasks. This makes VLMs a promising and important direction for future development. To enhance the ability of VLMs to process compressed images, one approach is to optimize existing codecs to minimize the bitrate without compromising VLM performance. Research based on VCM and FCM has demonstrated good results when transferring tasks to VLM vision ( 21 ; 23 ) . However, these methods are specific to a particular codec or VLM, limiting the generalization capabilities. On the other hand, another approach is to improve the VLM’s capacity to understand compressed images from various codecs, independent of specific compression distortions. This approach could enhance generalization, but to date, no research has investigated this area. Furthermore, although several benchmarks exist to evaluate VLMs’ performance in tasks such as VQA ( 18 ) , spatial relations ( 24 ; 31 ; 26 ) , text understanding ( 32 ) , and knowledge answering ( 50 ) , no benchmark has been developed specifically for assessing VLM performance on compressed images. In this paper, we present a comprehensive benchmark designed to evaluate the ability of VLMs to understand and process compressed images. Our benchmark includes 11 widely-used codecs and 3 series of VLMs, ranging from 1 to 32 billion parameters, and assesses both coarse-grained and fine-grained metrics across more than 1 million compressed images. This large-scale analysis allows us to quantify the performance degradation of VLMs due to image compression, revealing that compression can significantly impair the model’s ability to interpret visual content, as reflected in the objective results shown in Figure 1 and the subjective examples in Figure 2 . Based on this, we further break down the observed performance gap into two distinct components: the information gap during compression, which directly impacts the fidelity of image features, and the generalization gap of VLMs, which limits their ability to adapt to compressed images. Through visualizations and examples, we show that while the loss of information during compression is inherent and cannot be easily mitigated, the generalization gap represents a gap that can be addressed by improving the model’s ability to handle compressed inputs. To close this generalization gap, we propose a lightweight VLM adaptor that enhances VLM performance on compressed images across various codecs and bitrates. Empirical results show a 10%-30% improvement in understanding images compressed with different codecs, making it a promising solution for real-world applications involving image compression. To wrap up, our contribution can be summarized as follows: • We establish a comprehensive benchmark to explore the impact of image compression on VLM understanding, including more than 10 codecs and 1 million compressed images. • We analyse the performance gap between compressed and uncompressed images, and propose the information and generalization gap, illustrated using JPEG and Qwen-VL2.5-3B. • To eliminate generalization gap, we introduce a universal lightweight VLM adaptor to enhance performance on heavily compressed images across different codecs. Emprically, our adaptor improve by 10%-30%. 2 Benchmarking VLM for Compressed Images Understanding 2.1 Overview Image Codecs. We selected 11 state-of-the-art codecs with three representative categories, including traditional codecs, learning-based codecs, and generative codecs, as shown in Table 1 . Specifically, for traditional codecs, we chose the widely used JPEG ( 45 ) , HM ( 43 ) and VTM ( 4 ) . For learning-based codecs, we selected commonly used ELIC ( 15 ) , TCM ( 29 ) , and MLICpp ( 19 ) . For generative codecs, we selected the GAN-based HiFiC ( 35 ) and MS-ILLM (ILLM) ( 37 ) , as well as the diffusion-based DiffEIC (D.EIC) ( 27 ) , RDEIC ( 28 ) , and StableCodec (S.Codec) ( 53 ) . All compression methods were applied to the original dataset with four different bitrate levels to cover a wide range of bitrates. Detailed parameter settings are provided in the Appendix A . Datasets and Tasks. To assess the impact of image compression on VLMs, we curate seven tasks covering coarse-grained and fine-grained evaluation. Coarse-grained tasks focus on semantic understanding, evaluated using the POPE (