EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation

Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, Chongyi Li · 2024

Read EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation by Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, Chongyi Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18150.txt.

Introduction

Shuhao Han and ten co-authors present EvalMuse-40K in two thousand twenty-four. The benchmark grades text to image models with human annotations. Auto metrics for image text alignment lacked large test sets. Fine grained checks were missing from the small sets that existed.

Their set holds forty thousand image text pairs with rich labels. Balanced prompt sampling and re-annotation keep the data sound. Two new scorers come with the benchmark itself. FGA-BLIP2 fine tunes a vision language model end to end. PN-VQA uses a positive negative question style for zero shot checks. Both score well, and model rankings follow.

Catalog record

CREATORShuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, Chongyi Li
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18150

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, Chongyi Li Abstract Text-to-Image (T2I) generation models have achieved significant advancements. Correspondingly, many automated metrics emerge to evaluate the image-text alignment capabilities of generative models. However, the performance comparison among these automated metrics is constrained by the limited scale of existing datasets. Additionally, these datasets lack the capacity to assess the performance of automated metrics at a fine-grained level. In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks. In the construction process, we employ various strategies such as balanced prompt sampling and data re-annotation to ensure the diversity and reliability of our benchmark. This allows us to comprehensively evaluate the effectiveness of image-text alignment metrics for T2I models. Meanwhile, we introduce two new methods to evaluate the image-text alignment capabilities of T2I models: FGA-BLIP2 which involves end-to-end fine-tuning of a vision-language model to produce fine-grained image-text alignment scores and PN-VQA which adopts a novel positive-negative VQA manner in VQA models for zero-shot fine-grained evaluation. Both methods achieve impressive performance in image-text alignment evaluations. We also apply our methods to rank existing AIGC models, providing results that can serve as a reference for future research and foster the development of T2I generation. 1 1 footnotetext: Intern in Media Evaluation Lab, ByteDance Inc 1 Introduction Table 1 : In comparison to existing T2I model evaluation benchmarks, EvalMuse-40K collects a large number of human annotations (Ann.). Furthermore, EvalMuse-40K offers fine-grained annotations at the element (Ele.) level and categorizes these elements into different skills in image-text alignment. Additionally, EvalMuse-40K includes annotations for structural problems in generated images. To ensure reliable evaluation of automated metrics, we also randomly generate image-text pairs from 20 different T2I models. Benchmark Dataset Size Alignment Faithfulness T2I Models Prompt Image Ann. Likert Ele.-Ann. Ele.-Category Structure Num. PartiPrompt [ 47 ] 1.6K - - ✗ ✗ ✗ ✗ - DrawBench [ 34 ] 200 - - ✗ ✗ ✗ ✗ - T2I-CompBench [ 17 ] 6K - - ✗ ✗ ✓ ✗ - TIFA160 [ 16 ] 160 800 1.6K ✓ ✗ ✓ ✗ 5 GenAI-Bench [ 20 ] 1.6K 9.6K 28.8K ✓ ✗ ✗ ✗ 6 Gecko [ 42 ] 2K 8K 108K ✓ ✓ ✗ ✗ 4 EVALALIGN [ 37 ] 3K 21K 132K ✓ ✗ ✓ ✓ 8 EvalMuse-40K (ours) 4K 40K 1M ✓ ✓ ✓ ✓ 20 Recently, advanced Text-to-Image (T2I) models [ 24 , 30 , 10 , 31 , 33 , 34 , 47 ] are capable of generating numerous impressive images. However, these models may still generate images that fail to accurately match the input text, such as inconsistency in quantities [ 19 , 44 , 43 ] . Given the high cost and inefficiency of manual evaluation, developing a reliable automatic evaluation metric and corresponding benchmark is vital. They can effectively evaluate the performance of existing models and provide guidance for improvements in future models. Since traditional evaluation metrics, such as FID [ 13 ] and CLIPScores [ 12 ] , are not well-suited for assessing the consistency of T2I models, recent works [ 16 , 42 , 45 ] have explored the construction of new evaluation metrics. These methods introduce diverse evaluation approaches leveraging various multi-modal models. For example, VQAScore [ 21 ] is constructed by asking the Visual Question Answering (VQA) model “Does this figure show {text}?” and obtaining the likelihood of answer “Yes” as the alignment score. However, such a simple VQA manner cannot handle the fine-grained matching problem well. TIFA [ 16 ] and VQ2 [ 45 ] decompose the prompt into multiple elements, formulate related questions for each, and then average the answers to generate a final alignment score. These methods allow for fine-grained evaluation of image-text alignment but are also limited by the performance of the VQA model. Moreover, the relationship between element-based alignment scores and overall human preferences has yet to be explored. Therefore, to better explore the performance of existing T2I evaluation methods, we contribute a new benchmark, EvalMuse-40K , featuring fine-grained human annotations of image-text pairs. EvalMuse-40K includes 4K prompts, 40K image-text pairs, and more than 1M fine-grained human annotations. To ensure the diversity of prompts, EvalMuse-40K includes 2K real prompts and 2K synthetic prompts, where the real prompts are sampled from DiffusionDB [ 41 ] . We categorize the real prompts from multiple dimensions and use the MILP [ 39 ] strategy to ensure the category balance of the final sampled prompts. The synthesized prompts are then constructed for specific skills in image-text alignment, such as quantity and location. By synthesizing specific prompts and sampling from real prompts, EvalMuse-40K is able to accurately evaluate specific model problems while also providing a more comprehensive assessment of model performance in real-world scenarios. For fine-grained evaluation, we use a large language model for the elemental splitting of prompts and question generation, and to increase the diversity of the generated images, we generate images using a variety of T2I models. Compared with the previous benchmark (see Tab. 1 for details), EvalMuse-40K not only scores image-text alignment but also performs more fine-grained annotations for elements split from the prompts. Additionally, we annotate different types of structural problems that may appear in generated images. This comprehensive and fine-grained human evaluation in EvalMuse-40K enables an analysis of current automated T2I model evaluation metrics and their correlation with human preferences, ultimately supporting the improvement and development of the T2I evaluation system. In addition to the comprehensive benchmark, we introduce two methods, FGA-BLIP2 and PN-VQA, for image-text alignment evaluation. FGA-BLIP2 uses vision-language models to jointly fine-tune the image-text alignment scores and element-level annotations so that the models can output the overall scores and determine whether the generated images match the elements in the prompt. Additionally, we employ a variance-weighted optimization strategy to account for prompts’ ability to generate diverse images. PN-VQA constructs specific positive-negative question-answer pairs, allowing MLLMs to evaluate the fine-grained alignment of image-text pairs. With these two methods, we can obtain a scoring model aligned with human preferences and perform fine-grained alignment analysis and evaluation. Finally, we select some representative prompts to regenerate images and analyze existing AIGC models using our proposed evaluation methods. Our data, models, code, and results serve as valuable resources to support further research. To summarize, our contributions are listed as follows. • EvalMuse-40K collects a large number of human annotations and ensures a more robust evaluation of automated metrics by using a well-balanced set of prompts and a diverse range of generative models. • EvalMuse-40K categorizes elements during fine-grained annotation, enabling the evaluation of existing automated metrics’ accuracy in assessing specific skills at a fine-grained level. • We enable fine-grained evaluation by fine-tuning a scoring model, FGA-BLIP2, achieving impressive performance across multiple datasets. • We introduce a new automated metric, PN-VQA, which employs positive-negative questioning to enhance the correlation with human ratings. 2 Related Work Figure 1 : Overview construction of EvalMuse-40K, consisting of (a) data collection, (b) data annotation, and (c) evaluation methods. (a) We collect 2K real-world

Read the complete text