F-Bench: Rethinking Human Preference Evaluation Metrics for Benchmarking Face Generation, Customization, and Restoration
Lu Liu, Huiyu Duan, Qiang Hu, Liu Yang, Chunlei Cai, Tianxiao Ye, Huayu Liu, Xiaoyun Zhang, Guangtao Zhai · 2024
Read F-Bench: Rethinking Human Preference Evaluation Metrics for Benchmarking Face Generation, Customization, and Restoration by Lu Liu, Huiyu Duan, Qiang Hu, Liu Yang, Chunlei Cai, Tianxiao Ye, Huayu Liu, Xiaoyun Zhang, Guangtao Zhai (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13155.txt.
Introduction
Lu Liu and co-authors build, in a 2024 paper, a benchmark for judging AI-made face images by human taste. Generators now create, edit, and restore faces with ease, yet their output often carries odd flaws, false detail, or a drift in identity. The authors argue that scoring such images needs a framework built for faces, not borrowed from general image tests.
Their database, FaceQ, holds 12,255 images made by twenty-nine models across the three tasks of generation, customization, and restoration. One hundred eighty raters supplied nearly thirty-three thousand opinion scores, graded on quality, realism, identity match, and fit to the text prompt. The paper uses this base to rethink how preference metrics for face work should be built and checked.
Catalog record
| CREATOR | Lu Liu, Huiyu Duan, Qiang Hu, Liu Yang, Chunlei Cai, Tianxiao Ye, Huayu Liu, Xiaoyun Zhang, Guangtao Zhai |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13155
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
F-Bench: Rethinking Human Preference Evaluation Metrics for Benchmarking Face Generation, Customization, and Restoration Lu Liu, Huiyu Duan, Qiang Hu, Liu Yang, Chunlei Cai, Tianxiao Ye, Huayu Liu, Xiaoyun Zhang, Guangtao Zhai Abstract Recent artificial intelligence (AI) generative models have demonstrated remarkable capabilities in image production, and have been widely applied to face image generation, customization, and restoration. However, many AI-generated faces (AIGFs) still suffer from issues such as unique distortions, unrealistic details, and unexpected identity shifts, underscoring the need for a comprehensive quality evaluation method for AIGFs. To this end, we introduce FaceQ , the first comprehensive AI-generated Face image database with fine-grained Q uality annotations aligned with human preferences, which consists of 12K images and 491K ratings across multiple dimensions. Using the FaceQ database, we establish F-Bench , a benchmark for comparing and evaluating face generation, customization, and restoration models, highlighting strengths and weaknesses across various prompts and evaluation dimensions. Additionally, we assess the performance of existing image quality assessment (IQA) methods on FaceQ, and further propose a large multimodal model (LMM) based F ace quality Eval uator ( F-Eval ) to accurately assess the multi-dimensional quality of generated faces in a one-for-all manner. Extensive experimental results demonstrate the state-of-the-art performance of our F-Eval. The project page is: https://mediax-sjtu.github.io/F-Bench/ . Figure 1 : Data construction pipeline of F-bench and content overview of FaceQ database . From left to right , (a) Diverse input source data for face generation, customization, and restoration. (b) F-Bench benchmarks 29 face generative models, including 14 face generation models, 6 face customization models, and 9 face restoration models. (c) Multi-dimensional subjective preference evaluation in F-Bench . (d) 3D scatter plots of MOS distributions across FaceQ three subsets. † † footnotetext: ∗ Equal contribution. † Corresponding author. 1 Introduction Table 1 : Comparison between FaceQ and existing AIGC quality assessment databases (top) as well as natural face quality assessment databases (bottom) . FaceQ is the first large-scale AI-generated, customized and restored face quality assessment database. Database Domain Source Images Scores Dimensions AGIQA-3K ( TCSVT2024 ) [ 51 ] General AI-generated 2,982 MOS Perception, Alignment AIGCIQA ( CVPR2024 ) [ 89 ] General AI-generated 2,400 MOS Quality, Authenticity, Correspondence RichHF-18K ( CVPR2024 ) [ 59 ] General AI-generated 17,760 MOS Plausibility, Aesthetics, Text-image Alignment, Overall PIQ23 ( CVPR2023 ) [ 13 ] Face Real 5,116 Pair Overall, Details, Exposure FaceQ Face AI-generated, customized and restored 12,255 MOS Quality, Authenticity, ID Fidelity, Correspondence Generative models, including VAE [ 47 ] , GAN [ 26 ] , and diffusion models [ 33 , 82 , 74 ] , have made remarkable progress in recent years, and have been extended to the fields of text-to-image generation [ 3 ] , text-driven image editing [ 44 ] , image restoration [ 102 ] , etc , leading to high-quality artificial intelligence generated images (AIGIs). As an important aspect of AIGIs, many studies have also investigated face generation [ 103 ] , face customization [ 57 , 111 ] , and face restoration [ 62 , 90 ] , producing creative, high-quality face images. Despite these advances, many AI-generated faces (AIGFs) still suffer from perceptual deficiencies. Many natural face image quality assessment databases have been established in the literature [ 48 , 51 , 89 , 59 ] . However, natural images are mainly degraded by distortions such as noise, blur, compression [ 113 , 66 , 23 ] , etc. , while AIGIs generally suffer from unique distortions including unrealistic structures and text-image inconsistencies [ 59 , 89 ] , etc . Recently, many studies have investigated human preferences for AIGIs [ 59 , 89 , 50 , 39 , 106 ] . However, these general AIGI quality assessment works may not be well aligned with human preferences on AIGFs, since the perceptual quality of AIGFs can be significantly affected by small deviations in facial attributes,( e.g. , eye size, skin texture, identity shifts), while the number of AIGFs in these AIGI databases is limited. Moreover, current AIGI quality assessment works mainly focus on the image generation, while human preferences for face customization have rarely been considered, where identity fidelity is significant. The above limitation highlights the necessity of establishing a comprehensive quality evaluator for AIGFs. To this end, we propose FaceQ , the first AI-generated Face image Q uality assessment database with three popular tasks. Specifically, FaceQ contains 12K+ images obtained from 29 generative models across three tasks: 1) face generation, 2) face customization, and 3) face restoration, as shown in Fig. 1 . Based on the generated images, we collect 491K+ quality ratings from multiple dimensions including quality, authenticity, identity (ID) fidelity, and text-image correspondence, respectively, finally forming 32K+ mean opinion scores (MOSs). Using FaceQ, we establish a human preference benchmark, i.e. , F-Bench , for face generation, customization, and restoration models, and thoroughly analyze the generation performance across different evaluation perspectives, models, and categories. In addition, we also compare the correlation of existing image quality assessment (IQA), face quality assessment (FQA), AIGI quality assessment, and preference evaluation metrics with human preferences. Furthermore, we present F-Eval , the first generated F ace image quality Eval uation method designed to perform comprehensive multi-dimensional quality evaluation for generated, customized, and restorted faces in a one-for-all manner. Specifically, F-Eval leverages large multimodal models (LMMs) [ 11 ] and adopts instruction tuning [ 63 ] for preference-related vision-language adaption. We also apply low-rank adaptation (LoRA) [ 35 ] techniques for more efficient feature refining. Extensive experimental results demonstrate that F-Eval outperforms existing quality assessment methods in terms of multiple dimensions relevant to human preference for all three tasks. The main contributions of our work are highlighted as follows: • We construct FaceQ , the first generated face quality assessment database consisting 491K+ human preference ratings from quality, authenticity, ID fidelity, correspondence dimensions, on 12K+ generative face images across three tasks. • We establish F-Bench , a comprehensive human preference benchmark for 29 face generation, customization, and restoration methods. It also extensively evaluates the alignment of 26 existing quality assessment methods with human preferences. • We present F-Eval , a LMM-based quality assessment model to predict multi-dimensional human preference on generated faces. Extensive experiments demonstrate the state-of-art QA performance of the proposed F-Eval. 2 Related Work Figure 2 : Rating comparisons of eight dimensions. Each column presents a pair of intuitive examples of each dimension, with red indicating the better rating and blue indicating the worse one. From left to right , the subsets are face generation, face customization, and face restoration subsets. The last row displays the corresponding prompts, reference image-prompt pairs, and the GT-LQ image pairs. Face Generation & Customization & Restoration. (1) For face generation models, earlier works such as GANs [ 26 , 43 , 87 ] have demonstrated significant influences. Recently, diffusion models [ 74 , 77 , 73 ] have achieved rapid developments in text-to-image generation. These models can be categorized into pixel-space diffusion models [ 72 , 73 , 76 , 1 ] , and latent space diffusion models [ 74 , 58 , 71 , 17 ] . (2) For face customization, this is a recently defined task that aims to create images of a specific