Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help

Xuyang Guo and Jiayan Huo and Yingyu Liang and Zhenmei Shi and Zhao Song and Jiahao Zhang and Zhen Zhuang · 2025

Read Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help by Xuyang Guo and Jiayan Huo and Yingyu Liang and Zhenmei Shi and Zhao Song and Jiahao Zhang and Zhen Zhuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06884.txt.

Introduction

Guo leads a team benchmarking whether image generators can count objects in a 2025 paper. Diffusion models draw stunning pictures but often ignore numbers in the prompt. A picture asking for five cats may deliver four or seven.

Their benchmark, T2ICountBench, isolates counting across many leading open and private models. Difficulty levels are structured, and human checks keep scoring honest. Every top model failed to count reliably, and accuracy fell steeply as object counts rose. Polishing the prompt wording did not fix the problem. The authors call counting a fundamental blind spot, not a prompting issue. Future model designs need real numerical understanding. The benchmark is offered for tracking progress.

Catalog record

CREATORXuyang Guo and Jiayan Huo and Yingyu Liang and Zhenmei Shi and Zhao Song and Jiahao Zhang and Zhen Zhuang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE134 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06884

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help Xuyang Guo and Jiayan Huo and Yingyu Liang and Zhenmei Shi and Zhao Song and Jiahao Zhang and Zhen Zhuang Abstract Generative modeling is widely regarded as one of the most essential problems in today’s AI community, with text-to-image generation having gained unprecedented real-world impacts. Among various approaches, diffusion models have achieved remarkable success and have become the de facto solution for text-to-image generation. However, despite their impressive performance, these models exhibit fundamental limitations in adhering to numerical constraints in user instructions, frequently generating images with an incorrect number of objects. While several prior works have mentioned this issue, a comprehensive and rigorous evaluation of this limitation remains lacking. To address this gap, we introduce T2ICountBench , a novel benchmark designed to rigorously evaluate the counting ability of state-of-the-art text-to-image diffusion models. Our benchmark encompasses a diverse set of generative models, including both open-source and private systems. It explicitly isolates counting performance from other capabilities, provides structured difficulty levels, and incorporates human evaluations to ensure high reliability. Extensive evaluations with T2ICountBench reveal that all state-of-the-art diffusion models fail to generate the correct number of objects, with accuracy dropping significantly as the number of objects increases. Additionally, an exploratory study on prompt refinement demonstrates that such simple interventions generally do not improve counting accuracy. Our findings highlight the inherent challenges in numerical understanding within diffusion models and point to promising directions for future improvements. 1 Introduction Generative modelling is widely regarded as one of the most essential problems in today’s AI community, encompassing tasks such as natural language generation [ 7 , 1 , 35 ] , image synthesis [ 18 , 17 , 75 ] , video generation [ 63 , 28 , 59 ] , and speech synthesis [ 42 , 52 , 61 ] . Among various generative approaches, Diffusion Models (DMs) have demonstrated remarkable success across multiple domains, particularly in text-to-image and text-to-video generation [ 53 , 65 , 76 ] . Notable models like Diffusion Transformers (DiTs) [ 48 ] and Video LDM [ 8 ] have been shown to produce high-resolution and realistic images and videos, forming the foundation of advanced generative AI tools, including OpenAI Sora [ 43 ] and Kling [ 31 ] . Despite these advancements, diffusion-based models exhibit fundamental limitations in adhering to numerical constraints in user instructions. Prior empirical studies have shown that text-to-image diffusion models often struggle with basic object counting tasks [ 57 , 29 , 47 ] . Specifically, when given prompts specifying an exact number of objects (e.g., “generate an image with 7 apples on a wooden table”), the generated content frequently fails to match the requested quantity. These limitations become even more pronounced in complex scenarios, such as “generate an image with 7 apples on a table, separated by 3 oranges.” Such failures raise concerns about the reliability of such generative models and highlight their inherent difficulty in following precise numerical constraints. However, existing empirical studies on the counting ability of text-to-image models suffer from key limitations. Many benchmark studies evaluate only a small number of possibly outdated generative models [ 57 , 47 ] , with most models dating back to 2022–2023. Additionally, some benchmarks are too general and fail to disentangle counting ability from other factors such as adherence to style and shape constraints [ 29 , 44 , 70 ] . These shortcomings necessitate the need for a comprehensive, up-to-date, and specialized benchmark dedicated to evaluating the counting ability of text-to-image models. To address this gap, we introduce T2ICountBench , a novel benchmark designed to rigorously assess the counting ability of state-of-the-art text-to-image models in 2025. Our benchmark covers a diverse set of generative models, including both open-source and private image generation systems [ 45 , 5 , 74 ] . Unlike prior works, T2ICountBench explicitly isolates counting performance from other capabilities and provides structured difficulty levels, spanning object counts from 1 to 15. Additionally, our benchmark incorporates human evaluations to ensure high reliability and robustness. With the proposed T2ICountBench, we conduct a comprehensive evaluation to determine whether diffusion-based text-to-image models can accurately generate objects under numerical constraints. Our results show that most existing models exhibit significant failures in simple counting tasks, frequently generating the wrong number of objects. To highlight the non-trivial nature of this limitation, we also explore whether simple prompt refinements—decomposing a difficult counting task (e.g., generating 15 objects) into smaller subtasks—can improve performance. Our contributions are summarized as follows: • We present a comprehensive and rigorous benchmark, T2ICountBench, for evaluating the counting ability of text-to-image diffusion models. This benchmark effectively exposes the inherent limitations of these models in generating the exact number of objects. • We conduct extensive ablation studies on various factors influencing counting performance, including the number of objects, scene type, and style. Our findings indicate that as the number of objects increases from 1 to 15, model accuracy significantly drops, reaching around 10% for higher counts. We also find that complex background scenes will further adversely affect counting ability. • We performed an exploratory study to investigate whether simple prompt refinements could alleviate counting limitations. Our results indicate that such refinements generally do not improve counting performance, highlighting the inherent challenge of text-to-image diffusion models in counting. Roadmap. In Section 3 , we introduce our new benchmark to evaluate the counting capability of text-to-image diffusion models. In Section 4 , we show the main findings from our counting benchmark. In Section 5 , we discuss the possibility of improving text-to-image diffusion models with prompt refinement. In Section 6 , we show the conclusion of this paper. 2 Related Works Benchmarks on Text-to-Image Generation. The rapid advancement and real-world impact of text-to-image models have driven the development of evaluation benchmarks, particularly following the emergence of diffusion models. Early benchmarks [ 50 , 15 , 27 ] primarily relied on captions sourced from well-established datasets such as MS COCO, focusing on generating simple objects and scenes that could be automatically evaluated using pre-trained vision models. For instance, DALL-Eval [ 15 ] employs a 3D renderer to generate synthetic scenes for training text-to-image models, subsequently assessing them with object detection models. It also incorporates fairness considerations by evaluating social biases such as gender and skin tone. GenEval [ 21 ] as an object-focused automatic evaluation framework that uses object detection and related vision models to assess fine-grained compositional and text-to-image alignment. Addressing DALL-Eval’s limited scope, TIFA [ 27 ] expands evaluation criteria by leveraging a pretrained visual question-answering (VQA) model, enabling assessments beyond synthetic captions and 3D-rendered scenes to include more diverse conditions such as geolocation and weather variations. More recent benchmarks have shifted toward evaluating advanced capabilities of text-to-image models. HPDv2 [ 66 ] and Gecko [ 71 ] incorporate human preference-based ranking to assess alignment with aesthetic preferences. Another key research direction focuses on compositional text-to-image generation, which involves associating arbitrary attributes with objects beyond predefined datasets like COCO and reasoning about complex object relationships. Representative benchmarks in this area include T2I-CompBench [ 29 ] , ConceptMix [ 70 ] , and GenAI-Bench [ 37 ] . Additionally, Commonsense-T2I [ 19 ] and PhyBench [ 39

Read the complete text