Subjective Visual Quality Assessment for High-Fidelity Learning-Based Image Compression
Mohsen Jenadeleh 1 , Jon Sneyers 2 , Panqi Jia 3 , Shima Mohammadi 4 , João Ascenso 4 , Dietmar Saupe 1 · 2025
Read Subjective Visual Quality Assessment for High-Fidelity Learning-Based Image Compression by Mohsen Jenadeleh 1 , Jon Sneyers 2 , Panqi Jia 3 , Shima Mohammadi 4 , João Ascenso 4 , Dietmar Saupe 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06301.txt.
Introduction
Mohsen Jenadeleh and co-authors run a large human test of a new image codec in a 2025 study. Learning-based compression promises finer images at small file sizes, and JPEG AI is its newest standard. Human judgment remains the gold standard for judging quality.
The team built fifty images at fine distortion steps and gathered over ninety-six thousand comparisons from four hundred fifty-nine people. Scores were placed on a just-noticeable-difference scale. Most automated quality metrics rated the codec too kindly compared with human eyes. One metric aligned best overall yet still overshot. The full data set is public, and a new statistical test for metric gaps is introduced.
Catalog record
| CREATOR | Mohsen Jenadeleh 1 , Jon Sneyers 2 , Panqi Jia 3 , Shima Mohammadi 4 , João Ascenso 4 , Dietmar Saupe 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Subjective Visual Quality Assessment for High-Fidelity Learning-Based Image Compression Thanks: This research is funded by the DFG (German Research Foundation) – Project ID 496858717, titled “JND-based Perceptual Video Quality Analysis and Modeling”. D.S. is funded by DFG Project ID 251654672. Mohsen Jenadeleh 1 , Jon Sneyers 2 , Panqi Jia 3 , Shima Mohammadi 4 , João Ascenso 4 , Dietmar Saupe 1 Abstract Learning-based image compression methods have recently emerged as promising alternatives to traditional codecs, offering improved rate-distortion performance and perceptual quality. JPEG AI represents the latest standardized framework in this domain, leveraging deep neural networks for high-fidelity image reconstruction. In this study, we present a comprehensive subjective visual quality assessment of JPEG AI-compressed images using the JPEG AIC-3 methodology, which quantifies perceptual differences in terms of Just Noticeable Difference (JND) units. We generated a dataset of 50 compressed images with fine-grained distortion levels from five diverse sources. A large-scale crowdsourced experiment collected 96,200 triplet responses from 459 participants. We reconstructed JND-based quality scales using a unified model based on boosted and plain triplet comparisons. Additionally, we evaluated the alignment of multiple objective image quality metrics with human perception in the high-fidelity range. The CVVDP metric achieved the overall highest performance; however, most metrics including CVVDP were overly optimistic in predicting the quality of JPEG AI-compressed images. These findings emphasize the necessity for rigorous subjective evaluations in the development and benchmarking of modern image codecs, particularly in the high-fidelity range. Another technical contribution is the introduction of the well-known Meng–Rosenthal–Rubin statistical test to the field of Quality of Experience research. This test can reliably assess the significance of difference in performance of quality metrics in terms of correlation between metrics and ground truth. The complete dataset, including all subjective scores, is publicly available at https://github.com/jpeg-aic/dataset-JPEG-AI-SDR25 . PubID: pubid: ©2025 IEEE 2025 17th International Conference on Quality of Multimedia Experience (QoMEX) I Introduction Image compression remains a fundamental research area in image processing, having undergone significant advancements over the years. Traditional image compression standards—such as JPEG [ 1 ] , JPEG 2000 [ 2 ] , AVIF [ 3 ] , HEIC [ 4 ] , VVC/H.266 intra coding [ 5 ] , and JPEG XL [ 6 ] are designed to reduce data redundancy while preserving visual fidelity. These codecs use hand-engineered transformations, including the discrete cosine transform (DCT) and discrete wavelet transform (DWT), followed by quantization and entropy coding. More recently, deep learning-based image compression has emerged as a promising alternative, leveraging neural networks to further optimize compression efficiency with a perceptual quality target [ 7 , 8 ] . Unlike traditional codecs, learning-based methods employ end-to-end trainable architectures for the encoding and decoding processes and have demonstrated enhanced adaptability, enabling the preservation of critical visual details while achieving lower bitrates. Furthermore, these approaches facilitate adaptive quantization, content-aware bitrate allocation, and more effective entropy modeling, positioning them as viable solutions for the evolving challenges in modern image compression. One of the most recent advancements in the field is the development of the JPEG AI standard [ 9 , 10 , 11 ] , a state-of-the-art image compression standard being developed by the Joint Photographic Experts Group (JPEG). Unlike conventional transform-based codecs such as JPEG, JPEG 2000, and JPEG XL, this new standard employs deep learning-based image coding techniques to learn optimal encoding and decoding strategies. By leveraging neural network-driven models, JPEG AI achieves higher compression efficiency while maintaining superior visual fidelity, signaling a transformative shift towards AI-powered end-to-end image compression. However, this type of methods generate novel types of artifacts, distinct from traditional blocking and ringing distortions, while achieving competitive performance compared to conventional approaches [ 12 ] . Objective metrics such as SSIM [ 13 ] , MS-SSIM [ 14 ] , PSNR, and VMAF [ 15 ] offer some insights and have been used by researchers to evaluate compression algorithms [ 16 , 17 , 18 , 19 ] . However, the artifacts produced by learning-based codecs necessitate comprehensive subjective studies to assess their impact on perceived image quality [ 20 , 21 ] . Figure 1: JPEG AI encoder and decoder architecture (blue modules correspond to neural networks). Despite the increasing prevalence of learning-based image compression, relatively few studies have focused on its subjective quality assessment. In [ 21 ] , a subjective study was conducted using the absolute category rating (ACR) method, in which seven source images were compressed with image learning compression solution at four different bitrates—ranging from very low to very high—using three early learning-based compression algorithms. Similarly, in [ 20 ] , a subjective evaluation methodology based on JPEG AIC-2 Annex A [ 22 ] and a triplet comparison approach was employed to assess subjects’ preferences between images compressed with two learning-based algorithms, LBIC-CO and LBIC-PO. This study included 46 source images, each encoded at five different bitrates, and collected ratings from 20 subjects. Additionally, in [ 23 ] , a dataset comprising 100 source images with varying resolutions was constructed, where images were compressed using three traditional codecs and seven learning-based compression algorithms, each at three or four bitrates, ranging from very low to very high quality. A double-stimulus method with a five-category degradation rating scale was then used to collect responses from 40 subjects. In all of these studies, subjective quality assessment relied on single- or double-stimulus category ratings or user preference. However, as interest grows in the perceptual evaluation of high-fidelity compressed images, new methodologies are being introduced. To address this, JPEG AIC-3 has proposed a subjective test methodology for estimating the perceptual quality of compressed images, particularly in the high-quality to perceptually lossless range, using Just Noticeable Difference (JND) units. In [ 24 ] , the JPEG AIC-3 methodology was applied to evaluate the performance of traditional codecs, including JPEG, JPEG 2000, AVIF, VVC, and JPEG-XL. In this study, the JPEG AIC-3 subjective test methodology was applied to conduct a large-scale crowdsourcing study of JPEG AI compressed images with fine-grained distortion levels and reconstructed the subjective scores in JND units. The main contributions of this work are: • A dataset of JPEG AI-compressed images and triplet comparisons with plain and boosted distortions according to JPEG AIC-3. • A large-scale crowdsourcing study with 459 participants. • Data analysis of JPEG AIC-3 by subject screening, outlier detection and handling, and maximum likelihood estimation of an exponential unified model for perceived plain and boosted distortion in JND units. • Performance evaluation of 15 full-reference image quality assessment (IQA) metrics on our dataset. • Introduction of the Meng-Rosenthal-Rubin statistical test to assess the significance of differences in the correlations of quality metrics with ground truth. II JPEG AI standard The scope of JPEG AI standardization [ 9 ] is the creation of a learning-based image coding standard offering a single-stream, compact compressed domain representation, targeting both human visualization, and effective performance for image processing and computer vision tasks, with the goal of supporting a royalty-free baseline. The standardization process is divided into two versions, where version 1 focuses on high perceptual quality and fidelity, reconstructing images through entropy decoding and image synthesis from a latent tensor representation. This has been the main target until now. The International Standard (IS) for