CLPIPS: A Personalized Metric for AI-Generated Image Similarity

Khoi Trinh, Jay Rothenberger, Scott Seidenberger, Dimitrios Diochnos, Anindya Maiti · 2026

Read CLPIPS: A Personalized Metric for AI-Generated Image Similarity by Khoi Trinh, Jay Rothenberger, Scott Seidenberger, Dimitrios Diochnos, Anindya Maiti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01234.txt.

Introduction

Trinh and colleagues introduce CLPIPS, a personalized metric for judging similarity between AI-generated images and targets. Existing metrics such as LPIPS and CLIP often miss human judgments, especially in context-specific or user-driven tasks. CLPIPS adapts LPIPS to human judgments by fine-tuning only its layer combination weights on human-ranked image pairs, using margin ranking loss. On a human-subject dataset of iterative regeneration and ranking, it aligns more strongly with human judgments than baseline LPIPS.

Catalog record

CREATORKhoi Trinh, Jay Rothenberger, Scott Seidenberger, Dimitrios Diochnos, Anindya Maiti
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35585
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.01234

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLPIPS: A Personalized Metric for AI-Generated Image Similarity Khoi Trinh, Jay Rothenberger, Scott Seidenberger, Dimitrios Diochnos, Anindya Maiti Abstract Iterative prompt refinement is central to reproducing target images with text-to-image generative models. Previous studies have incorporated image similarity metrics (ISMs) as additional feedback to human users. These existing ISMs such as LPIPS and CLIP provide objective measures of image likeness but often fail to align with human judgments, particularly in context-specific or user-driven tasks. In this paper, we introduce Customized Learned Perceptual Image Patch Similarity (CLPIPS), a customized extension of LPIPS that adapts a metric’s notion of similarity directly to human judgments. We aim to explore whether lightweight, human-augmented fine-tuning can meaningfully improve perceptual alignment, positioning similarity metrics as adaptive components for human-in-the-loop workflows with text-to-image tools. We evaluate CLPIPS on a human-subject dataset in which participants iteratively regenerate target images and rank generated outputs by perceived similarity. Using margin-ranking loss on human-ranked image pairs, we fine-tune only the LPIPS layer-combination weights and assess alignment via Spearman’s rank correlation and Intraclass Correlation Coefficient. Our results show that CLPIPS achieves stronger correlation and agreement with human judgments than baseline LPIPS. Rather than optimizing absolute metric performance, our work emphasizes improving alignment consistency between metric predictions and human ranks, demonstrating that even limited human-specific fine-tuning can meaningfully enhance perceptual alignment in human-in-the-loop text-to-image workflows. 1 Introduction † † This study was reviewed and approved by the authors’ Institutional Review Board (IRB). Generative AI has made it possible to create images from text prompts, but reproducing a specific target image through prompt inference remains a challenge. In such tasks, users iteratively adjust prompts to better match a given reference image 18 ; 17 . A key bottleneck is producing an image similarity metric that aligns with human perception. Existing image similarity metrics (ISMs) such as Learned Perceptual Image Patch Similarity (LPIPS) 20 and CLIP-based scores 13 can provide feedback on image likeness. However, these ISMs’ alignment with humans’ subjective judgment remains unverified, especially in iterative workflows. Iterative refinement has been a common strategy across many domains where humans progressively improve outputs using feedback, including writing, programming, and design. Flower 5 provides a model on how writers plan, create, and revise their work iteratively. Madaan et al. 11 show that iterative refinement is effective in significantly improving outputs in text and code generation tasks through self-feedback mechanisms. Møller and Aiello 12 show that stepwise prompt refinement can show improvement in text summarization tasks. Du et al. 3 provides the R3 framework that has demonstrated the effectiveness of iterative revision in producing high-quality textual outputs by incorporating user feedback at each stage of the revision. Moreover, with the current rapid development of text-to-image models, the ability to iteratively reconstruct or approximate a target image using text prompts is relevant in several practical and economically meaningful contexts. These include but are not limited to: prompt recovery or prompt stealing, where users attempt to replicate proprietary or undisclosed prompts from observed outputs; bypassing prompt marketplaces or subscription gated prompt repositories by approximating high performing prompts through interaction alone; educational settings, where novice users learn how prompt structure maps to visual outcomes through guided feedback; creative restoration tasks, such as approximating lost, damaged, or incomplete visual artifacts when only partial references remain; and auditing and interpretability workflows, where understanding how prompts influence outputs helps diagnose model behavior. Recent work by Trinh et al. 17 found that off-the-shelf ISMs only moderately reflect human judgment in similarity for prompt refinement tasks. When such metrics fail to reflect human perception, metric-guided refinement can become unreliable, as users may be encouraged to pursue prompt changes that improve numerical scores while degrading perceptual similarity, analogous to benchmark overfitting or metric gaming in broader machine learning contexts 14 ; 8 . This gap motivates the need for more personalized or adapted metrics that better mirror human perception 17 . Rather than introducing yet another similarity metric optimized for raw accuracy or benchmark performance, in this paper, we focus on how well a metric’s notion of similarity mirrors that of a human. To this end, we introduce CLPIPS: Customized Learned Perceptual Image Patch Similarity , a novel, fine-tuned ISM built upon LPIPS. CLPIPS is designed to better align with individual human judgments of image similarity by fine-tuning LPIPS using human generated ranking data. While LPIPS provides a strong baseline for perceptual distance, CLPIPS further adjusts this distance function using human similarity ranking. This calibration effectively guides the metric to what users consider important in the context of image comparison. We evaluated CLPIPS on a dataset in which users iteratively generated prompts to match a target image and subsequently provided perceptual similarity rankings. Our evaluation utilizes two key metrics that quantify correlations between CLPIPS and human rankings: Spearman’s rank correlation coefficient ( \mathbf{\rho} ) and the Intraclass Correlation Coefficient (ICC). We show that CLPIPS achieves a higher correlation with human similarity rankings than baseline LPIPS, with improvements that are statistically significant. Finally, we discuss how CLPIPS can lay a foundation for future on-the-fly personalization of similarity metrics during human-AI workflows. Our work offers the following contributions. 1. Alignment-Oriented and Data-Efficient Similarity Metric: We propose CLPIPS, a customized extension of LPIPS that is fine-tuned on human similarity judgments. CLPIPS employs a lightweight tuning strategy that calibrates LPIPS’ deep features using human-ranked image data while updating only a small set of weights. To our knowledge, this is one of the first image similarity metrics explicitly adapted to individual/crowd preferences in an image regeneration context. For replicability purposes, the code and dataset will be made publicly available pending acceptance. 2. Evaluation of Human Judgments: We evaluate CLPIPS against baseline LPIPS using a dataset of human similarity rankings derived from iterative image regeneration workflows. The dataset provides human similarity rankings over image sets, enabling robust evaluation of alignment via Spearman’s rank correlation and Intraclass Correlation Coefficient (ICC). We further assess the statistical significance and generalization of these improvements on held-out image sets. 3. Insights: We demonstrate that even a modest amount of human-specific training data can noticeably improve metric alignment with subjective perception. We discuss the implications for interactive generative systems and outline how CLPIPS could enable on-the-fly metric adaptation to each user’s preferences in future applications. 2 Background & Related Work 2.1 Learned Perceptual Image Patch Similarity and other ISMs Accurately quantifying visual similarity as humans perceive it is a long-standing challenge in computer vision. Traditional metrics like Euclidean Distance, structural similarity index measure (SSIM), and peak signal-to-noise ratio (PSNR) capture pixel-wise differences but often fall short in reflecting the perceptual nuances emphasized by human judgment. To address this, Zhang et al. 20 proposed using deep features from convolutional neural networks (CNNs). This approach resulted in Learned Perceptual Image Patch Similarity (LPIPS), which computes feature embeddings from multiple convolutional layers of pre-trained networks (e.g., AlexNet, VGG, or SqueezeNet), normalizes these activations channel-wise, and calculates the weighted L2 distance between feature maps. The final similarity score averages these distances across spatial dimensions and layers. LPIPS leverages the hierarchical nature of CNNs, allowing the metric to capture higher-order image structures and context-dependent visual patterns that impact human perceptual judgments. LPIPS has since been widely adopted as a perceptual similarity measure. Other

Read the complete text