LGDM: Latent Guidance in Diffusion Models for Perceptual Evaluations

Shreshth Saini, Ru-Ling Liao, Yan Ye, Alan C. Bovik · 2025

Read LGDM: Latent Guidance in Diffusion Models for Perceptual Evaluations by Shreshth Saini, Ru-Ling Liao, Yan Ye, Alan C. Bovik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00327.txt.

Introduction

Judging image quality without a clean reference copy is a hard task for software, yet people do it at a glance. Shreshth Saini and three co-authors argue in a 2025 paper that image-generating diffusion models already organize pictures in ways that match human taste. Their method steers sampling inside such a model using quality-related features, then reads rich detail maps from the model's inner layers at several sizes and steps.

Those detail maps track human quality scores closely. Tested on standard quality datasets, the approach reached top performance without training a new judge from scratch, and it can be added to any similar pretrained generator.

Catalog record

CREATORShreshth Saini, Ru-Ling Liao, Yan Ye, Alan C. Bovik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00327

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LGDM: Latent Guidance in Diffusion Models for Perceptual Evaluations Shreshth Saini, Ru-Ling Liao, Yan Ye, Alan C. Bovik Abstract Despite recent advancements in latent diffusion models that generate high-dimensional image data and perform various downstream tasks, there has been little exploration into perceptual consistency within these models on the task of No-Reference Image Quality Assessment (NR-IQA). In this paper, we hypothesize that latent diffusion models implicitly exhibit perceptually consistent local regions within the data manifold. We leverage this insight to guide on-manifold sampling using perceptual features and input measurements. Specifically, we propose P erceptual M anifold G uidance (PMG), an algorithm that utilizes pretrained latent diffusion models and perceptual quality features to obtain perceptually consistent multi-scale and multi-timestep feature maps from the denoising U-Net. We empirically demonstrate that these hyperfeatures exhibit high correlation with human perception in IQA tasks. Our method can be applied to any existing pretrained latent diffusion model and is straightforward to integrate. To the best of our knowledge, this paper is the first work on guiding diffusion model with perceptual features for NR-IQA. Extensive experiments on IQA datasets show that our method, LGDM, achieves state-of-the-art performance, underscoring the superior generalization capabilities of diffusion models for NR-IQA tasks. † † affiliationnotice: *Work done during internship at Alibaba Group. Figure 1 : An overview of our proposed approach: (a) shows the transition of latent samples across latent manifolds, highlighting the steps of DDIM and our PMG. (b) depicts the content bias (green) on the manifold \mathcal{M}_{0}\approx\mathcal{D}(\mathcal{Z}_{0}) , showing that the guidance term in red (PMG) pushes a data sample ( x^{\prime}_{0|t}\sim\mathcal{D}(z^{\prime}_{0|t}) ) towards the perceptually consistent region (yellow) on the manifold. Here \mathcal{D} is decoder of VAE. 1 Introduction Score-based diffusion models have advanced significantly in recent years and have achieved remarkable success at synthesizing high-quality images across diverse scenes, views, and lighting conditions ( Ho et al. 2020 ; Song & Ermon 2019 ; Song et al. 2021b ; Zhang et al. 2023a ) . Latent Diffusion Models (LDMs), which embed data into a compressed latent space, enhance computational efficiency ( Rombach et al. 2022 ) . Diffusion models provide strong data priors that effectively capture the intricacies of high-dimensional data distributions, making them powerful for generative tasks. Conditional generation using posterior sampling has become crucial for solving various real-world low-level vision problems ( Kawar et al. 2022 ; Chung et al. 2024 ; Rout et al. 2024 ; Song et al. 2024 ) . Additionally, several methods leverage the rich internal representations of diffusion models by extracting either hand-selected single or subsets of features from a denoising U-Net for downstream tasks ( Tumanyan et al. 2023 ; Ye et al. 2023 ; Xu et al. 2023 ; Baranchuk et al. 2022 ) . Despite these advancements in addressing tasks like inverse problems, segmentation, and semantic keypoint correspondence, there has been little exploration into perceptual consistency of diffusion models for No-Reference Image Quality Assessment (NR-IQA). NR-IQA aims to evaluate image quality in line with human perception without a high-quality reference image ( Wang & Bovik 2006 ) . It plays a crucial role in optimizing parameters for image processing tasks, such as resizing, compression ( Feng et al. 2023 ; Liu et al. 2023 ) , and enhancement ( Hou et al. 2024 ; Fei et al. 2023 ; Zhang et al. 2024 ) . Fig. 2 illustrate a diverse spectrum of distortions frequently encountered in real‐world datasets. Early NR-IQA methods used hand-crafted natural scene statistics features ( Zhang et al. 2015 ; Mittal et al. 2012 ; Saad et al. 2012 ) , and have evolved into learning-based quality metrics ( Madhusudana et al. 2022 ; Tu et al. 2021 ; Ke et al. 2021 ; Saini et al. 2024 ) . While learning-based methods show promise, they often lack generalizability. With the advent of generative models, some authors have explored the use of pixel diffusion models for NR-IQA tasks ( Fu et al. 2024 ; Li et al. 2024b ; Babnik et al. 2024 ; Wang et al. 2025 ) . While these approaches show impressive progress, they are often ad hoc , focusing on tasks like quality feature denoising and image restoration by converting NR-IQA problems into Full-Reference IQA (FR-IQA) ones. Additionally, training on specific IQA datasets limits their generalizability. By contrast, our goal is to utilize pretrained latent diffusion models without fine-tuning, leveraging perceptual guidance to extract intermediate multi-scale and multi-time features, termed diffusion hyperfeatures ( Luo et al. 2024 ) , for NR-IQA. Our proposed method robustly handles breadth of degradations, capturing both local texture artifacts and larger‐scale structural inconsistencies as shown in Fig. 2 . Figure 2 : Exemplar Authentic and Synthetic distorted images. Here (a) shows Camera Motion Blur, (b) shows compression artifacts, (c) shows noise, and (d) shows a night scene with grains. Best viewed when zoomed in. At the core of our method is the manifold hypothesis: real data does not occupy the entire pixel space but instead lies on a smaller underlying manifold . Previous works ( Chung et al. 2022 ; He et al. 2024 ; Sun et al. 2023 ) have used the manifold concept for guided sample generation and solving inverse problems. In IQA, deep models aim to learn distortion manifolds that correlate highly with human perceptual quality ( Agnolucci et al. 2024 ; Su et al. 2023 ; Guan et al. 2018 ; Gao et al. 2024 ) . These manifolds represent regions within the data manifold that contain perceptually consistent samples, with content bias further narrowing these regions (Fig. 1 ). Figure 3 : Median SRCC scores of NR-IQA methods across authentic distortion IQA datasets, demonstrating the superior performance of our method. Towards further advancing progress in this direction, we propose P erceptual M anifold G uidance (PMG) to ensure perceptually consistent on-manifold sampling, conditioned on weak perceptual quality features. Fig. 1 provides an overview and conceptual visualization of our approach. Unlike previous state-of-the-art CNN or transformer-based IQA models that only utilize the final feature layer, we extract intermediate multi-scale and multi-time features, termed diffusion hyperfeatures ( Luo et al. 2024 ) , from a denoising U-Net for NR-IQA. As shown by ( Ghildyal et al. 2024 ) , intermediate features of foundation models outperform state-of-the-art learned metrics based on final feature layers. Our method, L atent G uidance in D iffusion M odel (LGDM), is a framework for extracting perceptually consistent diffusion hyperfeatures from unconditionally pretrained latent diffusion models: • We introduce a novel approach for leveraging unconditional latent diffusion models to tackle the challenging task of NR-IQA without any fine-tuning or additional training of feature extraction model. • We design a manifold guidance scheme (PMG) that ensures the sampling process remains on the manifold and close to perceptually consistent region. We provide extensive theoretical proof that our perceptual guidance keeps gradient updates on the tangent spaces of the data manifold, maintaining proximity to the local perceptually consistent manifold. We also utilize intermediate multi-scale and multi-time features from the denoising U-Net, resulting in high correlation with human perceptual judgments. • Extensive experiments on both authentic and synthetic IQA benchmarks demonstrate that our method achieves state-of-the-art performance. To the best of our

Read the complete text