LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model

Yuan Xue †, Qi Zhang ‡, Chuanmin Jia ♯, and Shiqi Wang † † School of Computer Science, City University of Hong Kong, Hong Kong SAR, China ‡ National Engineering Research Center of Visual Technology, Peking University, Beijing, China ♯ Wangxuan Institute of Computer Technology, Peking University, Bei · 2024

Read LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model by Yuan Xue †, Qi Zhang ‡, Chuanmin Jia ♯, and Shiqi Wang † † School of Computer Science, City University of Hong Kong, Hong Kong SAR, China ‡ National Engineering Research Center of Visual Technology, Peking University, Beijing, China ♯ Wangxuan Institute of Computer Technology, Peking University, Bei (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03841.txt.

Introduction

Yuan Xue, Qi Zhang, Chuanmin Jia, and Shiqi Wang build a frame to squeeze images for low-level machine tasks, not for human eyes. Their paper is from two thousand twenty-four.

Squeezing for machines most often serves high-level tasks, such as finding things. But real images are often of poor grade to start with. Low-level tools, such as image repair, can lift that grade, so their squeeze needs count too. The team's frame tunes squeeze and the low-level task as one. A large sight-language model guides the work. The frame is the first of its kind for this class of tasks.

Catalog record

CREATORYuan Xue †, Qi Zhang ‡, Chuanmin Jia ♯, and Shiqi Wang † † School of Computer Science, City University of Hong Kong, Hong Kong SAR, China ‡ National Engineering Research Center of Visual Technology, Peking University, Beijing, China ♯ Wangxuan Institute of Computer Technology, Peking University, Bei
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03841

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model Yuan Xue †, Qi Zhang ‡, Chuanmin Jia ♯, and Shiqi Wang † † School of Computer Science, City University of Hong Kong, Hong Kong SAR, China ‡ National Engineering Research Center of Visual Technology, Peking University, Beijing, China ♯ Wangxuan Institute of Computer Technology, Peking University, Beijing, China Abstract Image Compression for Machines (ICM) aims to compress images for machine vision tasks rather than human viewing. Current works predominantly concentrate on high-level tasks like object detection and semantic segmentation. However, the quality of original images is usually not guaranteed in the real world, leading to even worse perceptual quality or downstream task performance after compression. Low-level (LL) machine vision models, like image restoration models, can help improve such quality, and thereby their compression requirements should also be considered. In this paper, we propose a pioneered ICM framework for LL machine vision tasks, namely LL-ICM. By jointly optimizing compression and LL tasks, the proposed LL-ICM not only enriches its encoding ability in generalizing to versatile LL tasks but also optimizes the processing ability of down-stream LL task models, achieving mutual adaptation for image codecs and LL task models. Furthermore, we integrate large-scale vision-language models into the LL-ICM framework to generate more universal and distortion-robust feature embeddings for LL vision tasks. Therefore, one LL-ICM codec can generalize to multiple tasks. We establish a solid benchmark to evaluate LL-ICM, which includes extensive objective experiments by using both full and no-reference image quality assessments. Experimental results show that LL-ICM can achieve 22.65% BD-rate reductions over the state-of-the-art methods. 1 Introduction Image Coding for Machines (ICM) has become an emerging research topic that combines visual signal compression and understanding. Existing methods can be categorized according to the number of generated bitstreams: a single versatile bitstream for multiple tasks jointly, two bitstreams for human perception and machine task respectively and multiple bitstreams for diverse tasks separately. These ICM bitstreams can be compact representations of both original image [ 1 ] and extracted deep features [ 2 ] , which usually have different levels of capability on texture reconstruction and semantic preservation. However, all current methods are limited by only optimizing for high-level (HL) vision tasks, while neglecting practical compression requirements for low-level (LL) ones. Low-level vision tasks, such as denoising, debluring, inpainting, etc. , have been studied for decades. The main purpose of these tasks is to recover the details introduced by image capturing and delivering, and enhance the visual quality of the image content. Early works tend to propose one neural network for one specific LL vision task. Recently, considering the similarity of these tasks on pixel-level content analysis and processing, more and more works are trying to create a single model to solve diverse LL vision tasks simultaneously. By introducing large vision-language modeling, state-of-the-art all-in-one LL vision models can achieve impressive performance on important LL vision tasks. These methods are also more suitable for real-world applications where distortion types unknown or combined. Figure 1: ICM frameworks from the simplest to the proposed. \mathbf{X} is the original image, \mathbf{\hat{X}} is the compressed image, and \mathbf{\hat{X}_{H}} is the enhanced image by downstream task model. \mathbf{Cls.A} and \mathbf{Seg.B} are the results of HL tasks A and B. Image compression is usually the subsequent processing step after image capturing, which is essential for content delivery with diverse network conditions. However, compression also leads to information lost and quality degradation. Since LL visual enhancement cannot always be performed before compression due to diverse restrictions, there are two approaches to improve its performance. First, we can design LL vision models that consider compression distortions or even have the capability of removing them. But this approach has two notable drawbacks: 1) There are many types of compression distortions from different codecs like JPEG, VVC, AVS, neural codecs, etc. , making such model hard to architect and train. 2) Without leveraging the power of LL vision models, the compression efficiency is suboptimal. For example, some pixels could have been compressed more severely and restored or enhanced appropriately in the downstream. Similarly, some signals are visually unpleasant or broken, which should not be preserved after compression. Second, and more ideally, we can create an end-to-end compression and LL vision model, which jointly optimizes the performance of enhancing the visual perceptual quality and saving the bit-rate cost at the same time. In this work, we propose the first ICM framework for LL vision tasks, namely LL-ICM. Our goal is to achieve the rate-perception optimum by breaking the long lasting isolation between compression and LL visual enhancement via a unified model. Considering the diversity and complexity of distortion types in the real world, it is expensive or even impractical to customize LL-ICM model for each individually. Therefore, we further extend our framework so that it can adapt to versatile LL vision tasks using a single all-in-one model. Specifically, we introduce large-scale vision-language model for generalized feature embedding generation and exploit diffusion procedure to remove distortions efficiently. Our primary contributions are delineated as follows. • We propose LL-ICM, the first compression framework for LL vision. By jointly optimizing compression and LL vision processing, LL-ICM can improve the performance of both compared to existing frameworks that only consider compression or LL task alone. • We integrate large vision-language models (VLM) into LL-ICM. Based on the generalized feature representations learned by these models, we can use a diffusion model to reduce arbitrary type of distortions and increase the perceptual quality of compressed images. Therefore, one LL-ICM model can handle multiple LL vision tasks. • We create a large-scale benchmark to evaluate the performance of LL-ICM through objective evaluations. Experimental results indicate that LL-ICM significantly outperforms existing compression methods on both full and no-reference quality measurements. Specifically, LL-ICM achieves 22.65% BD-rate reductions compared to state-of-the-art neural codecs. 2 Method \SubSection Exsiting ICM frameworks We first give a brief introduction to the existing ICM frameworks and analyze their drawbacks. Fig. 1(a) is a conventional image codec optimized for signal fidelity without considering the performance of downstream tasks on compression outputs. As machine intelligence evolves, the compression needs for high-level visual understanding are getting more and more attention, leading to the establishment of HL-ICM framework shown in Fig. 1(b). In this framework, the codec is optimized for HL task models, whether their weights are fixed or not. Most ICM methods focus on HL vision tasks. However, LL tasks are also important for applications in the real world. The reason is that the quality of original images is usually not guaranteed. After compressed and transmitted, we need to enhance their quality by LL vision processing. The simplest ICM framework for LL-vision tasks is illustrated in Fig. 1(c), where the compressed image is processed by LL task models, but LL vision models do not consider the compression artifacts. In recent years, this issue has been addressed by the framework shown in Fig. 1(d), where LL vision models are trained with adaption to the compression distortion. However, the compression efficiency is neglected. Therefore, the LL-ICM framework in Fig. 1(e) offers a better solution that jointly improves the performance of compression and LL vision processing. Nevertheless, there are many LL vision tasks and

Read the complete text