Re-Thinking the Automatic Evaluation of Image-Text Alignment in Text-to-Image Models
Huixuan Zhang, Xiaojun Wan · 2025
Read Re-Thinking the Automatic Evaluation of Image-Text Alignment in Text-to-Image Models by Huixuan Zhang, Xiaojun Wan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08480.txt.
Introduction
Huixuan Zhang and Xiaojun Wan wrote a paper in two thousand twenty-five questioning how image generators are graded. Text-to-image models often miss details of the prompt, so automatic scores judge how well a picture matches its text. Existing checks mostly ask whether the scores agree with human ratings.
The authors argue a trustworthy grading framework needs more than agreement. They name two further properties it should have. Then they test mainstream frameworks across many metrics and models and show the current ones fall short on those properties. The paper closes with recommendations for building alignment evaluations that deserve trust.
Catalog record
| CREATOR | Huixuan Zhang, Xiaojun Wan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08480
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Re-Thinking the Automatic Evaluation of Image-Text Alignment in Text-to-Image Models Huixuan Zhang, Xiaojun Wan Abstract Text-to-image models often struggle to generate images that precisely match textual prompts. Prior research has extensively studied the evaluation of image-text alignment in text-to-image generation. However, existing evaluations primarily focus on agreement with human assessments, neglecting other critical properties of a trustworthy evaluation framework. In this work, we first identify two key aspects that a reliable evaluation should address. We then empirically demonstrate that current mainstream evaluation frameworks fail to fully satisfy these properties across a diverse range of metrics and models. Finally, we propose recommendations for improving image-text alignment evaluation. 1 Introduction Text-to-Image (T2I) models have demonstrated remarkable capabilities in generating high-quality, realistic images Betker et al. 2023 ; Esser et al. 2024 . Despite these advances, they still face challenges in accurately interpreting and adhering to user prompts. Common failures include generating incorrect object counts, attributes, or spatial relationships ( Li et al. 2024 ) . Nevertheless, evaluating text-image alignment remains a persistent and unresolved problem in the field. Several frameworks exist for evaluating image-text alignment. Model-based approaches include CLIPScore ( Hessel et al. 2021 ) and VQAScore ( Lin et al. 2024 ) . Component-based methods decompose text prompts into fine-grained elements and assess alignment through techniques like question generation and answering (QG/A) Hu et al. 2023 ; Cho et al. 2023 . Additionally, detector-based frameworks such as UniDet-Eval Huang et al. 2023 leverage object detection for evaluation. Figure 1: A robustness failure case of VQAScore. The right image is visually the same as the left one, yet their calculated VQAScore differs a lot. Despite the vast number of evaluation frameworks, few studies have thoroughly investigated the trustworthiness of image-text alignment assessment. Most existing work focuses narrowly on aligning automatic evaluation results with human judgments Li et al. 2024 ; Wiles et al. 2025 . When addressing trustworthiness, researchers primarily concentrate on improving human evaluation protocols to better validate automatic metrics Otani et al. 2023 ; Wiles et al. 2025 . However, these approaches overlook the critical point that a truly trustworthy evaluation framework must encompass more dimensions than mere correlation with human assessment. Additional discussion of related works is provided in Appendix A . In this work, we identify two critical properties for trustworthy evaluation frameworks: Robustness and Significance . Through empirical analysis, we demonstrate that current evaluation methods fail to fully satisfy these criteria, highlighting an important research direction for improving text-image alignment assessment. For example, as illustrated in Figure 1 , while two visually similar images receive substantially different VQAScore evaluations, revealing clear deficiencies in Robustness. In summary, our work makes two key contributions: (1) identifying Robustness and Significance as fundamental requirements for trustworthy evaluation frameworks, and (2) systematically demonstrating how current frameworks fail to meet these criteria. 2 Methodology 2.1 Preliminaries We would like to briefly introduce the common image-text alignment evaluation framework that we discuss in this research. Generally, given a benchmark consisting a set of N prompts P=\{p_{1},...,p_{N}\} , a text-to-image model M is used to generate images I=\{I_{1},...,I_{N}\} based on these prompts. 1 1 1 Though there can be multiple images generated using a certain prompt, many practices only generate one image per prompt. An automatic metric J is used to evaluate the alignment between the image-text pair (p_{i},I_{i}) , and outputs a score s_{i}=J(p_{i},I_{i}) . The final evaluation result is the average of all scores: s_{M}=\dfrac{1}{N}\sum_{i=1}^{N}s_{i} . If there are multiple models M_{1},...,M_{K} , the scores s_{M_{1}},...,s_{M_{K}} also provides a ranking of these models, which is also a part of the evaluation result that people focus on. 2.2 Aspects We would like to propose two important aspects that a trustworthy image-text alignment evaluation should focus on. Robustness Robustness requires that evaluation results remain consistent under reasonable perturbations of the input pair (p_{i},I_{i}) . In this work, we specifically examine two critical dimensions of robustness: (1) robustness to randomness and (2) robustness to image perturbations. Robustness to Randomness Most state-of-the-art text-to-image models employ denoising diffusion processes, where the generation output for a given prompt p_{i} depends on the sampled noise prior. This inherent randomness introduces variability in evaluation results. A trustworthy evaluation framework must maintain consistent model rankings despite this randomness - otherwise, the evaluation fails to reliably indicate which model performs better. To assess robustness under randomness, we systematically evaluate model performance across different random seeds. Robustness to Image Perturbation We make a fundamental assumption that for visually similar images I_{i} and I_{i}^{{}^{\prime}} , their evaluation scores J(p_{i},I_{i}) and J(p_{i},I_{i}^{{}^{\prime}}) should also be close. Large discrepancies would indicate potential metric flaw rather than true model capability. To assess robustness against image perturbations, we apply a minimal transformation: given an image I with pixel values I_{c,h,w}\in[0,255],c,h,w corresponds to the channels, height and width of the image, we achieve a perturbed image I^{{}^{\prime}} with pixel values 2 2 2 We confirm the perturbation’s visual imperceptibility through manual inspection of multiple cases, verifying that modified images remain indistinguishable from originals. : I^{{}^{\prime}}_{c,h,w}=\begin{cases}I_{c,h,w}+1&\text{if }I_{c,h,w}<255\\ I_{c,h,w}&\text{otherwise}\end{cases} (1) The robustness metric is computed by calculating the performance gap as: \Delta J_{i}=|J(p_{i},I_{i})-J(p_{i},I^{{}^{\prime}}_{i})| (2) Significance The property of Significance examines whether an observed performance difference (e.g., s_{M_{1}}>s_{M_{2}} ) reflects a meaningful superiority of model M_{1} over M_{2} . To quantify this, we employ two complementary approaches: 1. Statistical Testing: For evaluation score sets S_{M_{1}}=(s_{M_{1}}^{1},...,s_{M_{1}}^{N}) and S_{M_{2}}=(s_{M_{2}}^{1},...,s_{M_{2}}^{N}) , we conduct a paired t-test to assess statistical significance. 2. Dominance Ratio: We compute the empirical probability of M_{1} over M_{2} as: R=\dfrac{1}{N}\sum_{i=1}^{N}\mathbb{I}[s_{M_{1}}^{i}>s_{M_{2}}^{i}] (3) Model Name VQAScore CLIPScore DSGScore 42 3407 5096 42 3407 5096 42 3407 5096 Stable-Diffusion-3 91.18(1) 90.90(1) 91.06(1) 26.39(1) 26.34(1) 26.36(1) 93.66(1) 91.99(1) 93.52(1) Stable-Diffusion-XL 86.63(3) 86.01(3) 85.77(3) 25.93(2) 25.81(2) 25.85(2) 89.68(3) 90.04(3) 90.44(2) Pixart-Sigma-XL 87.04(2) 87.16(2) 86.72(2) 25.78(3) 25.71(3) 25.75(4) 90.52(2) 90.53(2) 89.99(3) Stable-Diffusion-1.5 76.26(4) 75.79(4) 77.32(4) 25.76(4) 25.58(4) 25.76(3) 83.23(4) 82.64(4) 83.88(4) Table 1: Results on MSCOCO using different metrics and different random seeds. The value under metric name is the corresponding random seed used. The value in the brackets is the ranking under the same seed. 3 Experiment Setup For our evaluation, we select four widely-used text-to-image generation models that represent diverse architectures and capabilities: Stable-Diffusion-3 (SD3) Esser et al. 2024 , Stable-Diffusion-XL (SD-XL) Podell et al. 2023 , Stable-Diffusion-1.5 (SD1.5) Rombach et al. 2022 , and PixArt-Sigma-XL (Pixart) Chen et al. 2024a . This carefully chosen set ensures comprehensive coverage of the current state-of-the-art in diffusion-based text-to-image generation. For metrics, we employ three widely applicable metrics: CLIPScore Hessel et al. 2021 , VQAScore Li et al. 2024 , and DSGScore Cho et al. 2023 . For benchmarking, we use MSCOCO Lin et al. 2014 , selecting 1,000 prompts from its validation split to assess general text-to-image generation capabilities following common practice Esser et al. 2024 . Additional implementation details are provided in Appendix B . 4 Experiment Results and Analysis 4.1 Analysis of Robustness Robustness to Randomness Our investigation first examines evaluation robustness under random seed variations. Table 1 reveals that both CLIPScore and DSGScore produce inconsistent model rankings across different random seeds, demonstrating their failure to maintain robust evaluation outcomes. Notably, this inconsistency cannot be solely attributed to narrow score margins between models. For instance, CLIPScore