Do Large Language Models Judge Error Severity Like Humans?
Diege Sun, Guanyi Chen, Zhao Fan, Xiaorong Cheng, Tingting He · 2025
Read Do Large Language Models Judge Error Severity Like Humans? by Diege Sun, Guanyi Chen, Zhao Fan, Xiaorong Cheng, Tingting He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05142.txt.
Introduction
Diege Sun and co-authors ask whether language models judge mistakes as harshly as people do, in a 2025 paper. Models now grade generated text for a living, yet their sense of error severity may not match ours. The team planted controlled meaning errors into image descriptions. Four error types were tested: age, gender, clothing type, and clothing color.
People rated the errors in both text-only and image-shown settings. Seeing the image made color and type errors feel worse to human judges. Most models scored gender errors oddly low but color errors oddly high, unlike people. One model matched the human ranking of error types, yet still blurred their gaps. A text-only model surprised the team with strong results.
Catalog record
| CREATOR | Diege Sun, Guanyi Chen, Zhao Fan, Xiaorong Cheng, Tingting He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05142
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do Large Language Models Judge Error Severity Like Humans? Diege Sun, Guanyi Chen, Zhao Fan, Xiaorong Cheng, Tingting He Abstract Large Language Models (LLMs) are increasingly used as automated evaluators in natural language generation, yet it remains unclear whether they can accurately replicate human judgments of error severity. In this study, we systematically compare human and LLM assessments of image descriptions containing controlled semantic errors. We extend the experimental framework of van Miltenburg et al. 2020 to both unimodal (text-only) and multimodal (text + image) settings, evaluating four error types: age, gender, clothing type, and clothing colour. Our findings reveal that humans assign varying levels of severity to different error types, with visual context significantly amplifying perceived severity for colour and type errors. Notably, most LLMs assign low scores to gender errors but disproportionately high scores to colour errors, unlike humans, who judge both as highly severe but for different reasons. This suggests that these models may have internalised social norms influencing gender judgments but lack the perceptual grounding to emulate human sensitivity to colour, which is shaped by distinct neural mechanisms. Only one of the evaluated LLMs, Doubao, replicates the human-like ranking of error severity, but it fails to distinguish between error types as clearly as humans. Surprisingly, DeepSeek-V3, a unimodal LLM, achieves the highest alignment with human judgments across both unimodal and multimodal conditions, outperforming even state-of-the-art multimodal models. 1 Introduction Large Language Models (LLMs) are increasingly being explored not only as generators of text ( Xuanfan and Piji 2023 ) , but also as evaluators, offering a scalable alternative to human judgment in Natural Language Generation (NLG) evaluation ( Gu et al. 2024 ; Gao et al. 2025 ) . Their use as “LLM-as-a-judge” has shown promise in areas such as summarisation and dialogue ( Liu et al. 2023 ; Liusie et al. 2024 ) , where traditional metrics, such as BLEU ( Papineni et al. 2002 ) and ROUGE ( Lin 2004 ) , often fall short of capturing semantic nuances and user preferences. Nonetheless, key questions remain about whether LLMs can reliably mirror human judgment Bavaresco et al. 2024 ; Gao et al. 2025 . One underexplored challenge is the assessment of error severity in generated text. van Miltenburg et al. 2020 observed that, in image captioning, human evaluators do not treat all errors equally: for example, given Figure 1 , the caption “ a man wearing a red shirt ” is judged more severely incorrect than “ a man wearing a yellow coat ”. Whether (multi-modal) LLMs’ assessments of error severity align with such human intuitions remains an open question. Figure 1: Image 34419 from MSCOCO: A man wearing a yellow shirt on a tennis court plays tennis. van Miltenburg et al. 2020 focused on multimodal NLG evaluation, where human judges evaluate outputs with access to both text and visual input. They showed that humans assign different levels of severity to different types of errors, suggesting nuanced perceptions of correctness. However, many real-world NLG tasks, such as summarisation, dialogue, and data-to-text generation, are unimodal (text-only), and most current LLMs are also unimodal. This raises a critical question: Are the severity distinctions observed in multimodal evaluations dependent on the presence of visual context, or do they reflect more general patterns in human judgment? In other words, it remains unclear how severity judgments of human beings are formed in unimodal settings, and how the presence of visual context may influence those judgments. With these two questions about human severity judgement in mind, we are further interested in whether unimodal LLMs’ severity judgement aligns with that of humans and whether visual context has a similar impact on multimodal LLMs. We therefore come up with two core research questions: • Do Large Language Models judge error severity like humans? We examine the extent to which LLM-based evaluations agree with human assessments across a range of error types in NLG. • How does the presence of images influence human judgment of errors in NLG? We explore whether access to visual context alters human perceptions of error severity and whether this impact is mirrored in LLM-based assessments. To address these two questions, we begin by updating the experimental setup of van Miltenburg et al. 2020 , allowing human judges to evaluate outputs in Chinese under two conditions: with access to both visual and textual inputs (multimodal), or with access to only textual inputs (unimodal). This design enables us to examine how severity judgments are formed in unimodal settings and to assess the influence of visual context. Next, we investigate the extent to which LLMs’ assessments of error severity align with those of humans by replicating the experiment using a range of unimodal and multimodal LLMs. Finally, we conduct a systematic comparison of human and LLM behaviour, evaluating which models best approximate human judgments of error severity across unimodal and multimodal conditions. 2 Hypotheses Following van Miltenburg and Elliott 2017 and van Miltenburg et al. 2020 , we are interested in four types of errors in Chinese Image Descriptions: age , gender , clothing-colour (henceforth, colour ), and clothing-type (henceforth, type ) (See Appendix A for examples of each error). The current section discusses the hypotheses of our experiment. van Miltenburg et al. 2020 asked participants to rate the quality of image descriptions, each containing a single type of error, with reference to the corresponding image. Lower ratings indicated more severe errors. Their findings revealed the following ranking of error types in terms of severity: \text{{colour}}\prec\text{{gender}}\prec\text{{type}}\prec\text{{age}} (1) where A\prec B means error type A receives lower scores (i.e., was judged more severe) than B . Notably, the differences in severity between colour and age , colour and type , as well as gender and age , were found to be statistically significant. In this study, we allow participants to access either both visual and textual inputs or only textual inputs to explore the impact of visual context. Previous studies in cognitive science have shown that visual context influences the processing of colour perception ( Adelson 1993 ; Chen et al. 2019 ) , age perception ( Pilz and Lou 2022 ) , object type recognition ( Fenske et al. 2006 ) , and gender representation ( Guilbeault et al. 2024 ) in the human brain. Based on these findings, we expect that the presence of visual context will significantly affect the ratings given by participants. However, we do not expect it to alter the severity ranking of errors observed in van Miltenburg et al. 2020 . For the LLM replications, considering the potential of LLMs to approximate human judgments, we hypothesise that these models will replicate human behaviour in our updated experiment. Formally, we formulate the following hypotheses: \mathcal{H}_{1} Humans will reproduce the same severity ranking of error types as reported by van Miltenburg et al. 2020 , under both multimodal and unimodal conditions. \mathcal{H}_{2} The presence of visual context will have a significant effect on human judgments. \mathcal{H}_{3} Both multimodal and unimodal LLMs will replicate the same severity ranking of error types as observed in human judgments. \mathcal{H}_{3} Visual context will influence multimodal LLMs in a manner similar to its effect on human judgments. 3 Experiments We elaborate on experimental