Multimodal Language Models as Text-to-Image Model Evaluators
Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha, Melissa Hall, Amy Zhang, Michal Drozdzal, Adriana Romero-Soriano · 2025
Read Multimodal Language Models as Text-to-Image Model Evaluators by Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha, Melissa Hall, Amy Zhang, Michal Drozdzal, Adriana Romero-Soriano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00759.txt.
Introduction
Jiahui Chen and co-authors let a multimodal model judge text to image systems in 2025. Image generators improve so fast that fixed test sets age out quickly. The team built a framework where one large multimodal model acts as the examiner. It writes fresh test prompts and scores the images they produce.
Its consistency scores track human judgment better than earlier metrics. Just twenty generated prompts nearly recovered the official rankings of three distinct benchmarks. That is a small fraction of the prompts those benchmarks use. A tireless examiner model could keep evaluation pace with generator progress.
Catalog record
| CREATOR | Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha, Melissa Hall, Amy Zhang, Michal Drozdzal, Adriana Romero-Soriano |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00759
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Language Models as Text-to-Image Model Evaluators Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha, Melissa Hall, Amy Zhang, Michal Drozdzal, Adriana Romero-Soriano Abstract The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate T2I progress. We present Multimodal Text-to-Image Eval ( MT2IE ) , an evaluation framework in which a single multimodal large language model (MLLM) acts as an evaluator agent, iteratively generating the evaluation prompts and scoring the resulting images. We show that MT2IE ’s image-text consistency scores have higher correlation with human judgment than metrics previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance: closely recovering the official T2I model rankings of three structurally distinct benchmarks from just 20 generated evaluation prompts, 28 28 – 105 × 105\times fewer than the benchmarks’ own prompt sets. When compared to existing evaluation metrics such as CLIPScore, VIEScore, and VQAScore, MT2IE ’s T2I model rankings are more faithful and far more consistent across multiple evaluation seeds when using the same number of prompts. MT2IE can also adapt evaluation to the model being tested: rewriting each prompt based on the model’s own measured performance to produce a bespoke per-model benchmark that still recovers the official rankings and keeps the evaluated model in an informative scoring range. We hope that these results will encourage the development of dynamic and interactive evaluation frameworks, and mitigate the deprecation of automatic evaluation benchmarks. † † date: August, 2026 † † correspondence: [email protected] 1 Introduction With the ever-growing use of generated images in academic and commercial settings ( Rombach et al., 2022 ; Podell et al., 2023 ; Sinha et al., 2024 ) , the evaluation of text-to-image (T2I) generative models is a rapidly growing and highly relevant area of research. This work focuses on evaluating image-text consistency: how well a generated image matches its corresponding text prompt. Consistency directly impacts the overall quality of a generated image ( Astolfi et al., 2024 ; Xu et al., 2023 ; Kirstain et al., 2023 ) , and its assessment has gained increasing attention as users’ text prompts become longer and more semantically complex. Generally, static sets of text prompts are used to benchmark T2I models. These prompts are either drawn from the real world, typically sourced from image captioning datasets ( e.g. , COCO ( Lin et al., 2014 ) , Conceptual Captions ( Changpinyo et al., 2021 ) ), or designed to test specific image generation capabilities ( e.g. , ImagenHub ( Ku et al., 2023 ) , ConceptMix ( Wu et al., 2024 ) , T2I-CompBench ( Huang et al., 2023 ) , HPDv2 ( Wu et al., 2023b ) ); in both cases, generated images are most commonly assessed with automatic metrics. For image-text consistency evaluation, alignment-based or generated question-answering-based (GQA) metrics are commonly used. Alignment-based metrics judge image-text consistency by computing the similarity between the generated image and its prompt. For example, CLIPScore ( Hessel et al., 2021 ) leverages the CLIP model ( Radford et al., 2021 ) to calculate cosine similarity between generated images and text prompts. However, its effectiveness is limited by the weaknesses of CLIP, which struggles with capturing compositional vision-language aspects such as spatial relationships, object cardinality, and attribute binding ( Tong et al., 2024 ; Thrush et al., 2022 ; Yuksekgonul et al., 2023 ) . GQA metrics ( Hu et al., 2023 ; Cho et al., 2024 ; Lin et al., 2024 ) have been introduced to address these limitations and enable finer-grained image-text consistency evaluation. These use large language models (LLMs) to generate question sets about the text prompt and then use visual question answering (VQA) models to score the generated questions given the prompt’s generated image. Figure 1: We present Multimodal Text-to-Image Eval ( MT2IE ), our novel evaluation methodology for text-to-image (T2I) generation. Previous approaches to T2I model evaluation require large, static benchmarks and often multiple external models. Our approach is a single system that generates prompts and scores the resulting images. We show that MT2IE can generate just 20 prompts while still closely recovering the official model rankings of approaches that use far more data. Recently, multimodal LLMs (MLLMs) ( Bai et al., 2025 ; Liu et al., 2023 ; Deitke et al., 2024 ) have been developed to enhance large language models (LLMs) with vision understanding capabilities, resulting in state-of-the-art performance across some vision-language tasks. Various approaches use MLLMs for image-text consistency evaluation, often via some form of visual question answering ( Lu et al., 2023 ; Ku et al., 2024 ; Zhang et al., 2023 ; Meng et al., 2024 ; Tan et al., 2024 ) . Although these approaches leverage MLLMs for improved evaluation, they still require fixed prompt datasets without any ability for dynamic evaluation. This can create a narrow focus on optimizing T2I performance for predefined prompts, rather than driving broad improvements. Additionally, rapid advancements in model capabilities can render static prompt sets ineffective, as they become overly simplistic and fail to provide meaningful evaluation ( Corry et al., 2021 ; Ott et al., 2022 ; Gupta et al., 2024 ) . Across benchmarks and metrics, the use of static evaluation frameworks is prone to overfitting and vulnerable to deprecation. In this work, we present Multimodal Text-to-Image Eval ( MT2IE ) , a unified framework that moves beyond static evaluation by iteratively generating dynamic prompts and scoring image generations. We explore the potential of MLLMs as evaluator agents that interact with a T2I model. Instead of relying on a static benchmark of prompts for image generation, the MLLM is now responsible for both generating text prompts and scoring the generated images. This removes the need to curate a prompt set for benchmarking while performing end-to-end evaluation with a single model. This additionally allows us to leverage the MLLM to generate increasingly complex prompts for generation, pushing the limits of the T2I model, and adaptive prompts, automatically rewriting prompts according to the T2I model’s performance. We use only off-the-shelf open-source MLLMs, in contrast with previous work that either relied on proprietary GPT-4 variants ( Zhang et al., 2023 ) or found open-source MLLMs’ judgments ineffective and uncorrelated with human judgments ( Ku et al., 2024 ; Jiang et al., 2024b ) . Ranking 8 recent T2I models across three structurally distinct benchmarks (GenEval ( Ghosh et al., 2023 ) , T2I-CompBench ( Huang et al., 2023 ) , GenAIBench ( Li et al., 2024a ) ), we find that MT2IE closely recovers each benchmark’s official model ranking from a single 20-prompt run—more faithfully and far more consistently than existing metrics given the same prompt budget. Our contributions are as follows: 1. We validate MLLMs as image-text consistency judges: on the TIFA160 benchmark ( Hu et al., 2023 ) , a single open-source MLLM correlates with human judgment better than every automatic metric and multi-model GQA pipeline we compare against, replacing the separate question-generation, filtering, and VQA models those pipelines require. 2. We show that the same MLLM can generate the benchmark: MT2IE closely recovers the T2I model rankings from three widely used