Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani · 2026

Read Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models by Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29689.txt.

Introduction

Ghiasvand and four colleagues evaluate whether multimodal large language models can critique photographs the way humans do. Open-ended aesthetic critique has no single correct answer. Most earlier evaluation compared models against numeric scores, not against the written critiques people actually give. The team uses the Reddit Photo Critique Dataset, scoring five open-weight models against ranked human critiques for each photo.

Six prompt conditions separate persona framing, aspect hints, and length control. A grounding control feeds each model the wrong photograph. Reference-based similarity turns out to mislead: strict metrics show weak alignment only with human critiques, while a coarse embedding score reports broad overlap that the control traces to a stable house style. Behaviourally, models write two to three times more than humans even under a length cap. The authors argue such metrics reward fluent breadth over human selectivity.

Catalog record

CREATORSajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani Abstract Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single correct answer, and most aesthetic evaluation has measured models against numeric scores rather than the written critiques people actually give. We evaluate MLLM critiques against ranked human references and ask whether they are close to human ones. Using the Reddit Photo Critique Dataset, we score five open-weight MLLMs against multiple ranked human critiques per photo with reference-based similarity metrics, under six prompt conditions that disentangle persona framing, aspect hinting, length control, and single- versus multi-pass generation, and add an image-grounding control that feeds each model the wrong photograph. We find that reference-based similarity gives a misleading picture. Stricter lexical and learned metrics show only weak alignment with human critiques, while a coarse embedding cosine reports broad topical overlap that the grounding control traces to a stable house style rather than image-specific observation. Behaviorally, the models diverge from humans in consistent ways the scores do not surface: even under a length cap they write two to three times as much, cover nearly every aesthetic aspect where humans are selective, engage each aspect more uniformly and at greater depth, and repeat themselves across critiques of the same photo where humans vary. We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique, and discuss implications for evaluating and training open-ended multimodal generation. 1 Introduction Figure 1 : Overview of our evaluation pipeline. From the Reddit Photo Critique Dataset we retain 1{,}693 posts, each with an image, the original post, and three top-ranked human critiques (sample shown is illustrative). Five open-weight MLLMs critique every post under six prompt conditions that independently vary persona framing, aspect hinting, and length/format control. We then assess the generated critiques by reference-based similarity to the human critiques (ROUGE-L, SBERT-cos, BERTScore, BLEURT), an image-grounding control (correct vs. shuffled image), and behavioral comparison along length, diversity, aspect coverage, and engagement. When a photographer posts an image to an online critique community and asks for feedback, what they receive back is not a number. It is a short paragraph: a few targeted observations about what works, what does not, and what to try next. Computational work on image aesthetics, however, has historically reduced this act to scoring . Benchmarks such as AVA [ 24 ] , predictors such as NIMA [ 32 ] , and attribute-level datasets [ 15 , 12 , 5 ] pair each image with a well-defined numeric target, so that model quality reduces to a correlation against a ground-truth score. Aesthetic critique , which is textual, open-ended, and unavoidably multiply-valued, is the form photographic feedback actually takes, and it is precisely the form that resists this kind of evaluation. Instruction-tuned multimodal large language models (MLLMs) have made such critique trivially easy to elicit. Open-weight families such as Qwen2-VL [ 36 ] , the LLaVA-NeXT family [ 20 ] , LLaVA-OneVision [ 16 ] , InternVL3 [ 43 ] , and Llama-3.2-Vision [ 10 ] all produce fluent, structured commentary about an image from a single prompt, and a growing body of work repurposes them as aesthetic raters and explainers [ 37 , 11 , 42 ] , with several reports suggesting they approach or even surpass human references on aesthetic- assessment tasks [ 42 ] . This invites a natural question, and the one we ask here: when an MLLM critiques a photograph, is it critiquing like a human, or merely producing text that scores as if it were? The question is difficult because open-ended critique has no single correct response and is conventionally scored against human references using similarity metrics [ 19 , 39 , 31 ] . Such metrics are convenient and reproducible, but they are known to correlate only weakly with human judgment when many distinct responses are equally acceptable [ 25 , 23 ] . The difficulty is sharpest when no reference is canonical: a critique can be excellent yet share little surface form with any reference, or mediocre yet score well by echoing reference vocabulary. Recent work on open-ended social reasoning reaches the same conclusion outside the visual domain, evaluating LLM responses to r/ChangeMyView prompts against multiple human references with a max-mapping over similarity metrics and finding that aggregate similarity obscures persistent human–model gaps [ 1 ] . A high similarity score, in other words, is not by itself evidence of human-level critique. We therefore evaluate five open-weight MLLMs on photo critique against ranked human references from the Reddit Photo Critique Dataset [ 34 ] , which supplies several independently upvoted critiques per photo. This multiplicity is central to our design: competent critiques of the same image routinely disagree, so we adopt a charitable multiple-reference, max-mapping protocol that rewards a model critique for resembling any human reference rather than an arbitrary one, together with a leave-one-out human ceiling that calibrates the range competent human critiques occupy under each metric. To separate how a critique is elicited from how a model critiques, we evaluate every model under six prompt conditions that independently vary persona framing, aspect hinting, length control, and single- versus multi-pass generation. And to test whether scores reflect genuine engagement with the photograph rather than a generic critiquing register, we add a shuffled-image control that feeds each model the wrong image alongside the correct title and body. Figure 1 summarizes this pipeline, from the 1,693 retained posts through the five models, six prompt conditions, and three evaluation axes. Across these analyses we find that reference-based similarity substantially overstates MLLM critique quality. The stricter lexical and learned metrics show only weak alignment between model and human critiques, yet a coarse embedding cosine reports broad topical overlap, so read naively these scores can suggest parity with human critique. The behavioral evidence shows why that reading is mistaken. Even under a human-plausible length cap, models write two to three times as much as humans and cover nearly every aesthetic aspect where humans are selective, with composition coverage above 98% for every model even when the aspect hint is removed. Asked for several critiques of the same photo they largely repeat themselves while humans vary, and when given the wrong image a model’s critiques resemble each other more than either resembles the human references, indicating that much of the measured similarity reflects a stable house style rather than image-specific observation. Qualitatively, the models walk through a fixed set of aspects rather than answer the question asked, reframe likely flaws as intentional choices, and stay uniformly positive where human critics diagnose problems and suggest fixes. We read these gaps as a measurement artifact, not as evidence that these models critique like humans. Our contributions are as follows: • We frame open-ended aesthetic critique as an evaluation problem in its own right and build a benchmark from the Reddit Photo Critique Dataset in which every photo carries three ranked human references, scored under a charitable multiple-reference max-mapping protocol with a leave-one-out human ceiling for calibration. • We evaluate five open-weight MLLMs under

Read the complete text