VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression
Kyle Sargent, Ruiqi Gao, Philipp Henzler, Charles Herrmann, Aleksander Hołyński, Li Fei-Fei, Jiajun Wu, Jason Zhang, [0.5em] Stanford University, Google Research, Google DeepMind · 2025
Read VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression by Kyle Sargent, Ruiqi Gao, Philipp Henzler, Charles Herrmann, Aleksander Hołyński, Li Fei-Fei, Jiajun Wu, Jason Zhang, [0.5em] Stanford University, Google Research, Google DeepMind (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15701.txt.
Introduction
Kyle Sargent and co authors present VLIC, an image compression system that uses vision language models as perceptual judges, in a 2025 paper. Compression tuned to plain distortion scores often clashes with what human viewers prefer. Older perceptual losses needed neural networks calibrated on large sets of human judgments.
The authors find, surprisingly, that leading vision language models can repeat human two choice judgments about image pairs with no extra training. VLIC, a diffusion based compressor, is post trained directly on those binary model judgments instead of distilling them into a separate loss network. The system reaches competitive or state of the art human aligned compression on several data sets and user studies.
Catalog record
| CREATOR | Kyle Sargent, Ruiqi Gao, Philipp Henzler, Charles Herrmann, Aleksander Hołyński, Li Fei-Fei, Jiajun Wu, Jason Zhang, [0.5em] Stanford University, Google Research, Google DeepMind |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15701
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression Kyle Sargent, Ruiqi Gao, Philipp Henzler, Charles Herrmann, Aleksander Hołyński, Li Fei-Fei, Jiajun Wu, Jason Zhang, [0.5em] Stanford University, Google Research, Google DeepMind Abstract Evaluations of image compression performance which include human preferences have generally found that naive distortion functions such as MSE are insufficiently aligned to human perception. In order to align compression models to human perception, prior work has employed differentiable perceptual losses consisting of neural networks calibrated on large-scale datasets of human psycho-visual judgments. We show that, surprisingly, state-of-the-art vision-language models (VLMs) can replicate binary human two-alternative forced choice (2AFC) judgments zero-shot when asked to reason about the differences between pairs of images. Motivated to exploit the powerful zero-shot visual reasoning capabilities of VLMs, we propose V ision- L anguage Models for I mage C ompression (VLIC), a diffusion-based image compression system designed to be post-trained with binary VLM judgments. VLIC leverages existing techniques for diffusion model post-training with preferences, rather than distilling the VLM judgments into a separate perceptual loss network. We show that calibrating this system on VLM judgments produces competitive or state-of-the-art performance on human-aligned visual compression depending on the dataset, according to perceptual metrics and large-scale user studies. We additionally conduct an extensive analysis of the VLM-based reward design and training procedure and share important insights. More visuals are available on our website . Figure 1 : Left. We propose a new post-training technique for diffusion autoencoders which uses Vision-Language Models to judge different decodings of the same image and leverage these judgements to improve the autoencoder through Diffusion DPO. Right. Our method, VLIC, demonstrates substantial improvements in the overall reconstruction quality, as well as better alignment to human perception. 1 Introduction Figure 2 : Qualitative results on standard image compression datasets. Top : We compare VLIC with HiFiC 34 and PO-ELIC 18 on a CLIC 2022 image 46 at various bits per pixel (bpp). Bottom : We compare our approach with HiFiC and PerCo on MS-COCO 28 . We find that our approach represents perceptually relevant fine details, faces, and textures more faithfully Compressing images and videos is necessary for storing and transmitting the rich sensory multimedia captured every day. This process inherently involves making trade-offs between compression rate ( i.e . file size) and visual quality. Ideally, the assessment of visual quality should align well with human perception, prioritizing details to which humans are more sensitive. For instance, humans are sensitive to perturbations to faces and text, but not to high-entropy natural textures such as grass or fur. Historically, visual quality has been assessed via classical metrics such as PSNR and SSIM 49 . However, these metrics are poorly aligned with human perception and often contradict human judgments of visual quality 57 ; 13 . To address this, prior work has focused on learning neural network approximations to human judgment of visual similarity, with varying approaches that target low-level similarity 57 ; 13 and semantic or high-level similarity 17 . While these learned perceptual metrics enable training neural compression methods that are better aligned with human perception, they are not without issues. Directly optimizing on these perceptual metrics can exploit their null-space, improving network performance with only limited gains in actual human-perceived quality 23 ; 8 . Moreover, trained perceptual metrics do not necessarily generalize beyond the datasets of human judgments used to calibrate them. For instance, networks calibrated on low-level visual differences may not agree with human judgments on images with high-level semantic differences 17 . In this work, we propose an alternative to the dominant paradigm of learned and differentiable perceptual metrics used during image compression network training. First, we show that Vision-Language Models (VLMs) are effective zero-shot perceptual judges of visual similarity (Figure 1 ). Specifically, we show that an off-the-shelf VLM (Gemini 2.5-Flash 10 ) can replicate human judgments on multiple human visual judgment datasets, namely on BAPPS 23 and our own dataset of human judgments collected on images compressed by various compression baselines. The fact that VLM reasoning can replicate human perceptual judgments is an encouraging finding because, as VLMs are improved through considerable investments, improved automatic perceptual judges may result without additional effort to collect human judgment data and train perceptual metrics. However, it is not clear how to convert the binary 2AFC judgments produced by VLMs into an optimizable perceptual metric which can be exploited by existing GAN-based perceptually oriented compression systems. Therefore, motivated by a desire to maximally exploit VLMs for human-oriented visual compression, we instead design a diffusion-based visual compression system similar to recent diffusion-based approaches for visual compression 41 ; 21 ; 7 . Since diffusion-based visual techniques can leverage the rich existing literature on diffusion model post-training with preferences 47 , we can benefit from VLM perceptual judgments without having to use them to train a separate perceptual metric. Concretely, we make the following contributions: 1. We show that an off-the-shelf VLM (Gemini 2.5-Flash 10 ) can replicate human judgments of visual similarity on multiple human judgment datasets zero-shot. 2. We present a diffusion-based visual compression system based on FlowMo 41 extended with an additional entropy coder. We show that VLM-generated preferences, can be used to post-train this system via Diffusion DPO 47 , improving performance. Moreover, we show that ensembling VLM preferences with those of a traditional perceptual metric, LPIPS, adds additional benefits and exceeds the performance of post-training with either reward alone. 3. We quantitatively study VLIC and show it achieves either competitive or state-of-the art performance relative to strong existing compression baselines, depending on the dataset, and conduct several large-scale user studies. We additionally provide several empirical analyses outlining best practices for VLM-based compression post-training, conduct ablation studies on reward design, and analyze and discuss important failure modes. 2 Related Work Perceptually-oriented Image Compression. Prior work has considered GAN-based 19 and diffusion-based 20 perceptually-oriented image compression. GAN-based techniques such as HiFiC 34 and PO-ELIC 18 are quite popular and are often fast to decode, particularly if a factorized entropy model is used 35 ; 18 . Diffusion-based visual compression can be separated into two distinct styles. The first style, based on diffusion autoencoders 38 , attempts to learn perceptually-oriented visual compression end-to-end with a discrete latent bottleneck 53 ; 3 ; 41 ; 8 ; 1 ; 54 or with the decoder conditioned on a prior learnt representation 21 . This style sometimes has the added benefit of producing a tractable latent space for downstream generative modeling. The second style involves using a trained diffusion model as an entropy coder over the diffusion model’s reverse process 36 ; 44 . Our model architecture belongs to the first style of diffusion-based visual compression, and is derived from FlowMo 41 with a few modifications, while our training scheme is a novel combination of diffusion autoencoder training with Diffusion DPO 4 . Approximations to Human Perception of Visual Similarity. Various work has explored designing proxies for human perception, such as LPIPS 57 , E-LPIPS 23 , DreamSim 17 , and DISTS 13 . These models have been used to calibrate learned image compression models 34 ; 18 but have also been used to generate different human visual content which humans