UniMIC: Towards Universal Multi-modality Perceptual Image Compression

Yixin Gao, Xin Li, Xiaohan Pan, Runsen Feng, Zongyu Guo, Yiting Lu, Yulin Ren, Zhibo ChenUniversity of Science and Technology of China {gaoyixin, pxh123, fengruns, guozy, luyt31415, renyulin}@mail.ustc.edu.cn,{xin.li, chenzhibo}@ustc.edu.cn · 2024

Read UniMIC: Towards Universal Multi-modality Perceptual Image Compression by Yixin Gao, Xin Li, Xiaohan Pan, Runsen Feng, Zongyu Guo, Yiting Lu, Yulin Ren, Zhibo ChenUniversity of Science and Technology of China {gaoyixin, pxh123, fengruns, guozy, luyt31415, renyulin}@mail.ustc.edu.cn,{xin.li, chenzhibo}@ustc.edu.cn (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04912.txt.

Introduction

Yixin Gao and seven co-authors present UniMIC, a frame that boosts many image codecs at once, in a 2024 study. Image compression trades file size against how good the result looks to the eye. Most work rebuilds one codec at a time. UniMIC instead keeps a shelf of proven codecs as its base tools. Text prompts join the codec output to guide a diffusion model that restores fine looks at the decoder end.

Two prompt types carry content notes and compression notes at varied lengths. A shared compensator then lifts the look of images from any base codec. Tests show clear gains in the size-versus-looks trade for both classic and learned codecs. The gains hold even at very low bit rates. The authors plan to share the code in a public repo.

Catalog record

CREATORYixin Gao, Xin Li, Xiaohan Pan, Runsen Feng, Zongyu Guo, Yiting Lu, Yulin Ren, Zhibo ChenUniversity of Science and Technology of China {gaoyixin, pxh123, fengruns, guozy, luyt31415, renyulin}@mail.ustc.edu.cn,{xin.li, chenzhibo}@ustc.edu.cn
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04912

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniMIC: Towards Universal Multi-modality Perceptual Image Compression Yixin Gao, Xin Li, Xiaohan Pan, Runsen Feng, Zongyu Guo, Yiting Lu, Yulin Ren, Zhibo ChenUniversity of Science and Technology of China {gaoyixin, pxh123, fengruns, guozy, luyt31415, renyulin}@mail.ustc.edu.cn,{xin.li, chenzhibo}@ustc.edu.cn Abstract We present UniMIC, a universal multi-modality image compression framework, intending to unify the rate-distortion-perception (RDP) optimization for multiple image codecs simultaneously through excavating cross-modality generative priors. Unlike most existing works that need to design and optimize image codecs from scratch, our UniMIC introduces the visual codec repository, which incorporates amounts of representative image codecs and directly uses them as the basic codecs for various practical applications. Moreover, we propose multi-grained textual coding, where variable-length content prompt and compression prompt are designed and encoded to assist the perceptual reconstruction through the multi-modality conditional generation. In particular, a universal perception compensator is proposed to improve the perception quality of decoded images from all basic codecs at the decoder side by reusing text-assisted diffusion priors from stable diffusion. With the cooperation of the above three strategies, our UniMIC achieves a significant improvement of RDP optimization for different compression codecs, e . g ., traditional and learnable codecs, and different compression costs, e . g ., ultra-low bitrates. The code will be available in https://github.com/Amygyx/UniMIC . 1 Introduction (a) VTM, 0.0079bpp (b) UniMIC(VTM-based), 0.0134bpp (c) HM, 0.0437bpp (d) UniMIC(HM-based), 0.0492bpp (e) JPEG, 0.2945bpp (f) UniMIC (JPEG-based), 0.2986bpp (g) MS-ILLM, 0.0581bpp (h) UniMIC (MS-ILLM-based), 0.0622bpp (i) ELIC, 0.0279bpp (j) UniMIC (ELIC-based), 0.0319bpp (k) mbt2018, 0.0729bpp (l) UniMIC (mbt2018-based), 0.0769bpp (m) cheng20-msssim, 0.1138bpp (n) UniMIC (cheng20-msssim-based), 0.1184bpp (o) cheng20-mse, 0.0901bpp (p) UniMIC (cheng20-mse-based), 0.0947bpp Figure 1 : Visual comparisons of our proposed UniMIC framework with eight representative basic codecs, including the hand-crafted codec VTM Bross et al. 2021 , HM Sullivan et al. 2012 , JPEG Wallace 1992 , MSE-optimized neural codec ELIC He et al. 2022 , cheng20-mse Cheng et al. 2020 , mbt2018 Minnen et al. 2018 , MS-SSIM-optimized cheng20-msssim on Kodak Dateset. Our method achieves more realistic and clear reconstructions than all basic codecs. Recent years have witnessed the great advancement of image compression Liu et al. 2023 ; He et al. 2022 ; Guo et al. 2021a ; Wu et al. 2021 ; Cheng et al. 2020 ; Minnen and Singh 2020 ; Minnen et al. 2018 ; Li et al. 2021b in improving the rate-distortion optimization when compressing the images into the bit streams. Early traditional image codecs, e . g ., JPEG Wallace 1992 and BPG Bellard , remove the content redundancy through several hand-crafted techniques, e . g ., intra-frame prediction, transform coding and entropy coding. In contrast, neural image compression Cheng et al. 2020 ; Zhu et al. 2022 ; Yang et al. 2020 ; Guo et al. 2021b ; Feng et al. 2023b ; Ballé et al. 2018 ; Minnen et al. 2018 ; Minnen and Singh 2020 ; He et al. 2022 exploits deep neural networks to establish the nonlinear transform, quantization, and entropy modeling in an end-to-end training manner, thereby owning the potential to model the data distribution more accurately and adaptively than traditional codecs. Based on the optimization objectives, existing works on image compression can be roughly divided into three types: (i) improving the objective quality of decoded images with the same bitrate Liu et al. 2023 ; He et al. 2022 ; Guo et al. 2021a ; Cheng et al. 2020 ; Minnen and Singh 2020 ; Minnen et al. 2018 , e . g ., PSNR and MS-SSIM; (ii) enhancing the perceptual quality Agustsson et al. 2019 ; Mentzer et al. 2020 ; Muckley et al. 2023 ; Yang and Mandt 2024 ; Xu et al. 2024 ; and (iii) assisting in the application of machine vision Sun et al. 2020 ; Feng et al. 2023a . However, most works are still designed based on the single-modality compression, i . e ., only reducing the redundancy in visual space, which overlooked the cross-modality redundancy modeling and information compensation. Thanks to the fast development of cross-modality generative foundation models, e . g ., Stable Diffusion Rombach et al. 2022 and DALLE-3 Betker et al. 2023 , some works have taken the first step to investigate how to incorporate the textual modality as the assistance for perceptual image compression, i . e ., enhancing the perceptual quality of decoded images, constructing the multi-modality compression. For instance, Lei et al . Lei et al. 2023 propose compressing images with text and sketch based on a conditional diffusion model. PerCo Careil et al. 2023 achieves realistic reconstructions even at 0.003 bpp with vector-quantized latent and a concise caption through fine-tuning a large-scale text-to-image diffusion model. Compared with existing GAN/Diffusion-based perceptual image compression, multi-modality compression enables ultra low-bitrate compression Careil et al. 2023 and better perceptual quality. Nonetheless, the above works entail the redesign of codecs for different coding requirements/scenarios and lack the universality for perceptual image compression at different bitrates. In this work, we propose the first universal multi-modality perceptual image compression framework, dubbed UniMIC, aiming at establishing unified rate-distortion-perception (RDP) optimization by excavating the cross-modality generation priors for multiple codecs simultaneously. Instead of incorporating the textual modality by designing new image codecs like existing works Lee et al. 2024 ; Qin et al. 2023 , which can merely support limited applications, e . g ., the ultra low-bitrate compression, our UniMIC can reuse the visual compression capability of various popular and mature image codecs by constructing the compositional codec repository. The compositional codec repository is composed of traditional codecs, e . g ., JPEG Wallace 1992 , HEVC Sullivan et al. 2012 and VVC Bross et al. 2021 , learned-based codecs, including ELIC He et al. 2022 , and MS-ILLM Muckley et al. 2023 , etc, covering broad application scenarios in both industry and academic fields. To achieve universal textual modality compression suitable for all these basic visual codecs, we propose multi-grained textual coding as the information compensation from textual modality. In particular, three-level variable-length content prompt coding modes are provided to transfer the content descriptions of images with different lengths to the decoding side, where a longer content description will deliver more visual information about images. It is noteworthy that our universal perceptual image compression entails the adaptation for various compression codecs. Consequently, we also design the compression prompt as part of textual coding, which intends to transfer the compression syntax, including the type of basic codec, compression quality, and optimization objective, as the guidance on the decoding side. To achieve universal perceptual optimization with textual modality, we propose the universal perception compensator by excavating the cross-modality generation priors from pre-trained large generative foundation models, i . e ., Stable Diffusion Rombach et al. 2022 . Concretely, we design the universal visual adapter, which incorporates the compression prompts, i . e ., compression syntax, and decoded visual images into the perceptual compensator through scale and shift modification, thereby enabling the adaptation capability for different codecs. The content prompt is injected into the perceptual compensator by reusing the cross-modality generation module in stable diffusion. Based on the above three innovations, our UniMic significantly improves the RDP optimization and achieves multi-modality compression for all image codecs in our proposed

Read the complete text