MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning

Yuxuan Luo, Yuhui Yuan, Junwen Chen, Haonan Cai, Ziyi Yue, Yuwei Yang, Fatima Zohra Daha, Ji Li, Zhouhui Lian · 2025

Read MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning by Yuxuan Luo, Yuhui Yuan, Junwen Chen, Haonan Cai, Ziyi Yue, Yuwei Yang, Fatima Zohra Daha, Ji Li, Zhouhui Lian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10963.txt.

Introduction

Yuxuan Luo and co-authors introduce a benchmark for knowledge images in 2025. Charts, diagrams, and mind maps have long helped people learn, a point backed by learning theory. Making such images with models demands both world knowledge and precise visual placement.

Their set holds more than four thousand expert-checked image and prompt pairs. It spans ten fields and six school levels, with each target laid out as a graph of items and links. A score rates both factual match and visual clarity. Sixteen top models were tested, and even the best scored only about half, showing how hard the task is.

Catalog record

CREATORYuxuan Luo, Yuhui Yuan, Junwen Chen, Haonan Cai, Ziyi Yue, Yuwei Yang, Fatima Zohra Daha, Ji Li, Zhouhui Lian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-10963

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning Yuxuan Luo, Yuhui Yuan, Junwen Chen, Haonan Cai, Ziyi Yue, Yuwei Yang, Fatima Zohra Daha, Ji Li, Zhouhui Lian Abstract In this paper, we introduce knowledge image generation as a new task, alongside the Massive Multi-Discipline Multi-Tier Knowledge-Image Generation Benchmark ( MMMG ) to probe the reasoning capability of image generation models. Knowledge images have been central to human civilization and to the mechanisms of human learning—a fact underscored by dual-coding theory and the picture-superiority effect 1 1 1 Both dual-coding theory and the picture-superiority effect principles suggest humans remember visuals more effectively than words, partly because visual information can engage multiple cognitive encoding pathways. This work was supported by the National Natural Science Foundation of China (Grant No.: 62372015) and Key Laboratory of Intelligent Press Media Technology. . Generating such images is challenging, demanding multimodal reasoning that fuses world knowledge with pixel-level grounding into clear explanatory visuals. To enable comprehensive evaluation, MMMG offers 4,456 4,456 expert-validated (knowledge) image-prompt pairs spanning 10 10 disciplines, 6 6 educational levels, and diverse knowledge formats such as charts, diagrams, and mind maps. To eliminate confounding complexity during evaluation, we adopt a unified Knowledge Graph (KG) representation. Each KG explicitly delineates a target image’s core entities and their dependencies. We further introduce MMMG-Score to evaluate generated knowledge images. This metric combines factual fidelity, measured by graph-edit distance between KGs, with visual clarity assessment. Comprehensive evaluations of 16 16 state-of-the-art text-to-image generation models expose serious reasoning deficits—low entity fidelity, weak relations, and clutter—with GPT-4o achieving an MMMG-Score of only 50.20 50.20 , underscoring the benchmark’s difficulty. To spur further progress, we release FLUX-Reason ( MMMG-Score of 34.45 34.45 ), an effective and open baseline that combines a reasoning LLM with diffusion models and is trained on 16,000 16,000 curated knowledge image–prompt pairs. Figure 1 : 40,000 Years of Knowledge Image: From Cave Paintings to Generative AI. 1 Introduction Reasoning-based large language models (LLMs) such as OpenAI-o1/o3 openai2024o1 ; openai2025o3mini ; openai2025o3 and DeepSeek-R1 guo2025deepseek excel on math and coding tests (AIME 2024 AIME2024 , Codeforces CodeForces , GPQA Diamond GPQA , MATH-500 MATH500 , MMLU MMLU , SWE-bench SWE-bench ) thanks to rigorous, reasoning-focused benchmarks. By contrast, widely used text-to-image benchmarks T2Icompbench ; CommonsenseT2I still focus on instruction following and compositionality—e.g., attribute binding—while largely overlooking reasoning. The lack of reasoning-oriented benchmarks has left text-to-image generation models lagging significantly behind reasoning-focused LLMs. Measuring reasoning in image generation is non-trivial. Current evaluations emphasize prompt-following, aesthetics, and visual–text rendering, typically quantified by CLIP ruiz2023dreambooth , FID saharia2022photorealistic , OCR and Aesthetic discus04342023aesthetic25 scores. Yet producing such images seldom requires complex logical, domain-specific knowledge reasoning. Motivated by how humans leverage visuals to think, we introduce a new task—knowledge image generation: given only a vague user prompt, the model must autonomously infer the pertinent concepts (or entities) and relationships, and render them in a coherent knowledge image—such as a diagram, chart, infographic, or other annotated visual—that faithfully conveys the intended information. Throughout human history, knowledge images have propelled progress for nearly 40,000 years, serving as a lasting bridge that turns abstract ideas into concrete, shareable visual forms (Figure 1 ). Cognitive science also supports this direction—the dual-coding theory dualcoding and the picture-superiority effect picture_superiority_effect suggest humans encode information more robustly when language and imagery are combined. Creating such visuals, however, is intrinsically difficult: a model must fuse broad world knowledge with spatial composition, select salient entities, and faithfully ground relations in pixel space. Figure 2 visualizes several knowledge representations across disciplines. To advance text-to-image reasoning, we introduce the Massive Multi-Discipline Multi-Tier Knowledge-Image Generation Benchmark ( MMMG ). MMMG comprises 4,456 expert-validated prompt–image pairs spanning ten academic disciplines—Biology ( 850 ), Chemistry ( 328 ), Mathematics ( 399 ), Engineering ( 582 ), Geography ( 352 ), Economics ( 623 ), Sociology ( 479 ), Philosophy ( 210 ), History ( 327 ), and Literature ( 306 );—and six educational tiers: pre-school ( 591 ), primary school ( 680 ), secondary school ( 693 ), high school ( 936 ), undergraduate ( 744 ), and PhD ( 812 ). Each sample is annotated with a high-quality knowledge graph that lists the necessary entities and their dependencies, enabling format-agnostic coverage and requiring models to generalize across domains and reasoning levels. Its benefits are twofold: first, it abstracts core concepts into an interpretable graph, reducing the diversity and complexity of knowledge visuals; second, it enables objective fidelity evaluation via graph-edit distance between the ground-truth and generated graphs. Figure 2 : Representative knowledge images generated with GPT-4o. Disciplines (L-R): Economics, Oceanography, Environmental Engineering, Astrophysics, Climate Science. Details are in the supplementary. Benchmarking reasoning fidelity in generated images requires more than perceptual metrics. We therefore introduce MMMG-Score , which combines the graph-edit distance between knowledge graphs with a visual-clarity score derived from foundational segmentation models. Specifically, we employ the OpenAI-o3 reasoning LLM to analyze each image–prompt pair, and predict an initial knowledge graph. For the visual-clarity component, we run Segment Anything Model v2 (SAM-2) SAM2 on the generated images and penalize overly cluttered and disorganized outputs that may “hack” the reasoning LLM to extract the unreliable knowledge graphs yet fail to convey the knowledge clearly. The importance of this visual-clarity metric is examined in the experimental section. Benchmark Scale Focus Domains Metrics World Knowledge Explanatory GenEval ghosh2023geneval 553 Compositionality counting, colors, position, attribute binding Accuracy ✗ ✗ T2I-CompBench++ T2ICompbench++ 6,000 Compositionality Object-Attribute Binding BLIP-VQA, UniDet, CLIP ✗ ✗ DPG-Bench DPGBench 1,065 Prompt Adherence Dense Scene Generation CLIP, Human Eval ✗ ✗ Commonsense-T2I CommonsenseT2I 1,000+ Commonsense Reasoning Everyday Scenarios Accuracy ✗ ✗ Winoground-T2I WinogroundT2I 11,000 Compositionality 20 Types Contrastive Accuracy ✗ ✗ TIFA hu2023tifa 1,000 Faithfulness General Knowledge VQA-based ✗ ✗ TypeScore sampaio2024typescore 1,000 Text Fidelity Scene Text OCR-based ✗ ✗ WISE WISE 1,000 Commonsense Reasoning Science, Culture, Space-Time LLM-Judged ✓ ✗ MMMG (Ours) 4,456 Disciplinary Knowledge 10+ Academic Fields Readability, Graph Edit Distance ✓ ✓ Table 1 : Comparison with previous Text-to-Image (T2I) benchmark. We conduct comprehensive evaluations of 16 state-of-the-art text-to-image models—LlamaGen, JanusFlow, Emu-3, SimpleAR, Janus-Pro, CogView, SEED-X, SDXL-1.0, SDXL-1.0-refiner, Infinity, FLUX.1-[dev], FLUX.1-[pro], Ideogram 2.0, HiDream-l1-Full, BAGEL, and GPT-4o image generation—on the MMMG benchmark, reporting their FID, aesthetic, WISE WISE , and MMMG-Score metrics. We also conduct human studies to confirm that MMMG-Score aligns best with user judgements, underscoring the value of knowledge-graph–based evaluation. Our MMMG benchmark presents significant challenges: even the GPT-4o image generation achieves only MMMG-Score of 46.66 , while the next-best model, the open-source HiDream-l1-Full, reaches just MMMG-Score of 25.72 . To catalyze further research, we release FLUX-Reason , a fully reproducible and open-source baseline that pairs a reasoning-oriented LLM (e.g., OpenAI-o3 or DeepSeek-R1) with a diffusion model (FLUX.1-[dev]) trained on 16,000 curated knowledge-image pairs. Although its MMMG-Score of 30.52 still trails that of GPT-4o, FLUX-Reason serves as an open source baseline and underscores the new challenges posed by the MMMG benchmark for next-generation reasoning-oriented text-to-image generation models. 2 Related Work Benchmarks for Text-to-Image Generation. Many benchmarks have been developed to assess both the limitations and progress of recent text-to-image models. We summarize the comparison between MMMG and prior benchmarks in Table 1 . GenEval ghosh2023geneval introduces object detectors for fine-grained,

Read the complete text