Predicting Satisfied User and Machine Ratio for Compressed Images: A Unified Approach

Qi Zhang, Shanshe Wang, Xinfeng Zhang, Siwei Ma, Jingshan Pan, Wen Gao · 2024

Read Predicting Satisfied User and Machine Ratio for Compressed Images: A Unified Approach by Qi Zhang, Shanshe Wang, Xinfeng Zhang, Siwei Ma, Jingshan Pan, Wen Gao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17477.txt.

Introduction

Qi Zhang and co-authors predict how happy humans and machines will be with squeezed images in a 2024 paper. Shrinking an image for storage or sending dulls it for both eyes and algorithms, so a joint forecast can guide the squeeze.

One model scores both shares at once: the share of viewers who will be content, and the share of machine reads that will still pass. A feature reader is warmed up on machine labels faked by human quality models. Mixer and attention parts then fuse deep and difference traits. Tests beat past single task scorers on both shares.

Catalog record

CREATORQi Zhang, Shanshe Wang, Xinfeng Zhang, Siwei Ma, Jingshan Pan, Wen Gao
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17477

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predicting Satisfied User and Machine Ratio for Compressed Images: A Unified Approach Qi Zhang, Shanshe Wang, Xinfeng Zhang, Siwei Ma, Jingshan Pan, Wen Gao Abstract Nowadays, high-quality images are pursued by both humans for better viewing experience and by machines for more accurate visual analysis. However, images are usually compressed before being consumed, decreasing their quality. It is meaningful to predict the perceptual quality of compressed images for both humans and machines, which guides the optimization for compression. In this paper, we propose a unified approach to address this. Specifically, we create a deep learning-based model to predict Satisfied User Ratio (SUR) and Satisfied Machine Ratio (SMR) of compressed images simultaneously. We first pre-train a feature extractor network on a large-scale SMR-annotated dataset with human perception-related quality labels generated by diverse image quality models, which simulates the acquisition of SUR labels. Then, we propose an MLP-Mixer-based network to predict SUR and SMR by leveraging and fusing the extracted multi-layer features. We introduce a Difference Feature Residual Learning (DFRL) module to learn more discriminative difference features. We further use a Multi-Head Attention Aggregation and Pooling (MHAAP) layer to aggregate difference features and reduce their redundancy. Experimental results indicate that the proposed model significantly outperforms state-of-the-art SUR and SMR prediction methods. Moreover, our joint learning scheme of human and machine perceptual quality prediction tasks is effective at improving the performance of both. I Introduction A large amount of images are consumed by not only humans but also machines every day. Image quality is an important attribute that significantly affects the user experience and machine analysis performance, and thereby higher image quality is preferred by both. However, images are usually compressed before spread, and the compression process inevitably degrades the image quality. It is essential to assess the compressed image quality so that more efficient compression methods can be developed according to the assessment result. In the past decades, many image quality assessment (IQA) models have been proposed to predict the image quality perceived by humans [ 1 ] . Traditional metrics, like PSNR and SSIM, are designed based on the pixel-wise difference between the reference and distorted images. These metrics are applied widely, but are not well correlated with human perception. Recently, many deep learning-based IQA metrics [ 2 , 3 , 4 ] have been developed. Most of them utilize the difference of extracted deep features to predict a perceptual quality score. Since these features are learned to contain more task-related information, deep IQA metrics have outperformed traditional ones, and are becoming more broadly-used nowadays. However, existing IQA metrics seldom model the characteristics of the human vision system (HVS), making them still inconsistent and unreliable, especially when the compressed image quality varied subtlely [ 5 ] . Specifically, HVS has inherent flaws that can only perceive quality difference beyond a threshold, which is called Just Noticeable Difference (JND) [ 6 ] . By collecting many JND samples from different humans, the Satisfied User Ratio (SUR) of a compressed image can be calculated [ 7 ] , which is the ratio of subjects who cannot perceive the quality difference between the original image and compressed one. Compared with other aforementioned IQA metrics, SUR directly reflects the HVS characteristics, and is more consistent with general human perception. Several methods have been developed to predict the SUR of compressed images [ 8 , 9 ] . However, the prediction error can be further decreased by improving the network architecture. Different from humans, machines perceive images by understanding their contents. Some recent works reveal that machine vision systems (MVS) also have JND characteristics [ 10 ] , and the machine diversity is also addressed in [ 11 ] , where the concept of Satisfied Machine Ratio (SMR) is introduced. Similar to SUR, SMR is defined as the ratio of machines that cannot perceive the quality difference between the original image and compressed one. Therefore, it is reasonable to evaluate the image quality for machines by predicting its SMR, which has not been well studied yet. In many applications, images are consumed by large populations of both humans and machines. It is meaningful to predict SUR and SMR simultaneously, which can guide the development of better compression methods to satisfiy both kinds of consumers. Since SUR and SMR are both related to the perceptual quality of images, leveraging such relationship and designing a unified model to predict them can be more effective than predicting them separately. In this paper, we propose a learning-based model to achieve this. Our contributions can be summarized as follows: • We propose a unified model to predict SUR and SMR of compressed images simultaneously. The joint learning of HVS- and MVS-perception related features implicitly leads to more effective feature extraction for both tasks. • We introduce a pre-train scheme that does not rely on large-scale SUR-annotated data, which is hard to obtain. Instead, we use diverse IQA models to generate normalized quality labels for compressed images, which aligns with the annotation process of SUR labels. • We introduce a Difference Feature Residual Learning (DFRL) module to learn more discriminative difference features. We then use a Multi-Head Attention Aggregation and Pooling (MHAAP) layer to aggregate the multi-layer difference features and pool them into a smaller representation. Moreover, we propose to use an MLP-Mixer network to fuse the spatial and channel information of the aggregated representation for SUR and SMR prediction. • Experimental results show that the proposed model outperforms state-of-the-art SUR and SMR prediction models. And the joint learning scheme of SUR and SMR prediction tasks can improve the performance of both. II Revisit SUR and SMR We first give a unified formulation of SUR and SMR. Let I_{0} be the original image, and I_{q_{1}},I_{q_{2}},\ldots,I_{q_{K}} be its compressed versions. The image quality is controlled by compression parameters q_{k} , which degrades as k increases. For any human or machine subject X , we use the following function to determine where it is satisfied with the image quality of I_{q_{k}} : S(X;I_{q_{k}})=\begin{cases}1,&\text{if}\ P(X,I_{0})=P(X,I_{q_{k}})\\ 0,&\text{otherwise}\end{cases}, (1) where P(\cdot) is the perception result of X on the image. For humans, a subjective test can be conducted to check and record whether participants can perceive the quality difference between I_{0} and I_{q_{k}} or not [ 7 ] . For machines, we can run them on I_{0} and I_{q_{k}} and know if the two results are the same or similar enough [ 11 ] . Then, by collecting S(X;I_{q_{k}}) from a large population of X , SUR or SMR (jointly denoted as R ) can be calculated as R(I_{q_{k}})=\frac{|\{X\in\mathbb{X}\mid S(X;I_{q_{k}})=1\}|}{|\mathbb{X}|}, (2) where \mathbb{X} is the set of subjects and |\cdot| counts its elements. Therefore, SUR and SMR of I_{q_{k}} are the ratios of subjects who are satisfied with its quality. Because of such acquisition process, SUR and SMR reflect the general perceptual characteristics of humans and machines instead of individuals, removing the subject bias and making them more consistent and reliable. This advantage is particularly significant since the ultimate receivers of images are often large and diverse populations of humans or machines. SUR and SMR are also more direct and promising quality indicators than other IQA metrics, because in practice images are compressed to satisfy some percentage

Read the complete text