Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim Abstract While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability 1 1 1 Source code: [Anonymized for Review]. . † † affiliationnotice: Equal contribution 1 Introduction Large Language Models (LLMs) have emerged as automated evaluators across diverse domains, commonly referred to as LLM-as-a-Judge ( 32 ; 20 ; 31 ) . This paradigm has been rapidly adopted not only in natural language processing tasks such as summarization ( 15 ; 13 ; 18 ) and dialogue evaluation ( 34 ; 5 ) , but also extends to vision-language models ( 28 ; 7 ; 30 ) , and reward modeling for reinforcement learning from human feedback ( 55 ; 62 ) . The appeal is clear: LLM judges offer scalable, cost-effective evaluation without the bottleneck of human annotation. Despite their widespread adoption, a fundamental question remains unresolved: Can we trust the judgments of LLM-as-a-Judge? LLM judges are increasingly used to support critical evaluation decisions, even though principled methods for verifying the reliability of their judgments remain limited. The reliability of LLM-based evaluation can be examined through two fundamental dimensions: intrinsic consistency , which concerns whether a judge behaves as a stable and coherent measurement instrument under equivalent evaluation conditions, and human alignment , which concerns how closely its assessments correspond to those of human evaluators. However, existing work tends to address two dimensions separately and relies on metrics that provide limited diagnostic insight. Studies focusing on intrinsic reliability have proposed a range of approaches, including inter-rater agreement ( 27 ; 53 ) , internal consistency ( 45 ) , and uncertainty quantification ( 52 ; 59 ) . While these methods capture partial aspects of judge behavior, they operate at the level of observed scores and cannot separate stable measurement characteristics of the judge from variation driven by the evaluated samples. Alignment, which has been widely studied in prior work, is commonly assessed through aggregate agreement signals between LLM and human judgments, typically quantified using correlation and agreement based metrics ( 20 ; 32 ) . However, agreement at the outcome level reflects similarity in evaluation results, rather than whether the judge itself functions as a reliable measurement instrument. As a result, the reliability of LLM judges remains insufficiently characterized under existing evaluation approaches. To address this gap, we introduce a unified framework for assessing LLM judge reliability that integrates intrinsic consistency and human alignment into a single diagnostic view. We leverage Item Response Theory (IRT), a measurement framework widely used to analyze latent traits from observed responses, to examine LLM judging behavior. IRT models observed scores as probabilistic functions of latent traits and item characteristics, enabling judgments to be analyzed in terms of underlying evaluation signals rather than outcome-level scores alone ( 33 ; 14 ) . For rating-based judgments with ordered categorical outputs (e.g., Likert-style ratings), we adopt the Graded Response Model (GRM; 44 ), a polytomous IRT model that characterizes how latent evaluation signals give rise to discrete score categories. Just as educational testing uses IRT to assess whether exam items reliably measure student ability, we use GRM to assess whether LLM judges reliably measure subject quality. Our diagnostic framework is organized into two conceptual phases. The first phase examines intrinsic consistency by applying prompt perturbations and examining whether judges produce consistent measurements under semantically equivalent variations. Phase 2 evaluates human alignment by comparing latent evaluation signals inferred from LLM judges with those of human annotators, conditioned on judges that exhibit sufficient intrinsic consistency. This sequential design ensures that alignment is interpreted only when measurement behavior is internally consistent. We evaluate the framework across diverse evaluation settings, illustrating how IRT-GRM produces interpretable diagnostic signals that characterize measurement behavior of LLM judges. By identifying multiple sources of unreliability, the framework provides a practical basis for determining when LLM-based evaluation can be considered reliable. 2 Related Work LLM-as-a-Judge is now widely used as a scalable alternative or complement to human evaluation ( 10 ; 20 ; 31 ; 49 ) , with recent work further systematizing this paradigm via structured design choices including Chain-of-Thought prompting ( 56 ) to elicit explicit reasoning ( 54 ; 52 ; 9 ; 42 ) , checklist-based evaluation ( 17 ) , external validation tools ( 16 ) , self-improvement processes ( 57 ) , and agent-based pipelines ( 68 ) . Despite its growing adoption, concerns have been raised about whether LLM-based evaluation can produce judgments that are sufficiently reliable and consistent to substitute for human evaluation ( 6 ; 2 ) . Yet existing evidence offers limited guidance on whether failures in LLM judging stem from unreliable scoring or divergence from human judgment. Intrinsic Consistency Prior work has assessed consistency by examining whether an LLM judge’s outcomes remain stable across different inputs and evaluation settings. These include measuring inter-rater agreement across LLM evaluators ( 27 ; 53 ) , as well as evaluating internal consistency by repeating judgments under different random seeds ( 38 ) and summarizing stability with metrics such as McDonald’s \omega ( 45 ) . However, relying only on final scores or discrete verdicts provides limited insight into the stability and confidence of LLM judgments, and recent work has incorporated uncertainty modeling ( 60 ; 65 ; 59 ) , including leveraging token probabilities associated with ratings or verdicts to quantify evaluation confidence or uncertainty ( 52 ; 58 ; 47 ) . In addition, several works have examined multiple dimensions of reliability in LLM-based evaluation, including structural instability ( 61 ) , rating indeterminacy ( 21 ) , and biases that affect judging outcomes ( 64 ; 48 ; 51 ) . However, these approaches capture only partial aspects of reliability without separating the judge’s measurement properties from sample quality, leaving researchers unable to diagnose the source of observed inconsistencies in measurements. Human Alignment Human alignment captures the extent to which LLM-based judgments agree with human evaluations, often serving as a key objective of LLM-as-a-Judge benchmarks and evaluation frameworks ( 32 ; 67 ) . It is commonly measured by comparing LLM-based and human evaluations using correlation-based metrics (Kendall’s \tau , Spearman’s \rho , or Pearson’s r ) ( 61 ; 19 ; 9 ; 2 ) and agreement metrics, including percent agreement, pairwise accuracy, Cohen’s \kappa ( 11 ) , Krippendorff’s \alpha ( 23 ) , or Scott’s \pi ( 46 ; 17 ; 21 ; 2 ; 66 ; 50 ; 22 ) . However, relying on single-point scores or discrete verdict matches may provide an incomplete picture of human alignment, as such summaries can mask disagreement and uncertainty in human evaluation. To address this, recent work explores distributional alignment that predicts human rating distributions ( 8 ) and statistical frameworks that analyze discrepancies at the distribution level ( 41 ) . Moreover, recent work proposes procedural approaches to justify or guarantee human agreement, including selective evaluation with escalation ( 26 ) and statistical tests for replacing human annotators with LLM judges ( 4 ) . Yet these metrics offer limited insight into the nature of misalignment, unable to distinguish inconsistent measurements or apply different circumstances. 3 Methodology We propose a framework for diagnosing LLM-as-a-judge reliability based on Item Response Theory (IRT). Our approach consists of three components: (1) fitting a Graded Response Model (GRM) to separate measurement properties from true quality differences, (2) generating controlled prompt variations to probe sensitivity, and (3) extracting interpretable reliability metrics grounded in social science. 3.1 Graded Response Model for LLM Judges We adopt the GRM ( 44 ) to jointly estimate sample quality and measurement properties. For a rating scale with K unique score values, the probability that an evaluator with a variation of judgment prompt p\in\mathcal{P} associates score value Y_{pj} of evaluation subject j\in\mathcal{D} (instance to be evaluated) with at least k\in\{1,2,\cdots,K\} is: P(Y_{pj}\geq k\mid\theta_{j})=\frac{1}{1+\exp(-\alpha_{p}(\theta_{j}-\beta_{pk}))} (1) where \theta_{j}\in\mathbb{R} is the latent genuine quality of subject j , \alpha_{p}\in\mathbb{R}^{+} is the discrimination parameter for the variation p , and \{\beta_{pk}\}_{k=1}^{K}\in\mathbb{R} is a monotonically increasing sequence of thresholds defining levels of latent quality at which the judge transitions between adjacent score values. After fitting GRM, each prompt variant (original, typo, newline, paraphrase) corresponds to its own (\alpha_{p},\boldsymbol{\beta}_{p}) , while all variants share the latent quality \theta_{j} per sample. This separation allows us to attribute score variation to either prompt effects (captured by \alpha,\beta ) or true quality differences ( \theta ). Also, to follow the common practice of GRM, we apply priors \theta_{j}\sim\mathcal{N}(0,1) , \alpha_{p}\sim\text{LogNormal}(0,0.5) , and \beta_{pk}\sim\mathcal{N}(0,1) with an ordered constraint ( \beta_{pk}<\beta_{p,k+1} for all k0.7 indicates acceptable reliability. Diagnostic Interpretation. The combination of C_{V} and \rho enables differential diagnosis: high C_{V} with acceptable \rho points to prompt sensitivity as the primary issue, while low \rho regardless of C_{V} indicates fundamental model limitations. 3.4.2 Phase 2: Human Alignment Once a judge passes Phase 1 criteria, we examine how its quality perception compares to human judgment. Two metrics capture different aspects of this comparison. As we only use original prompts instead of using all variants, we omit the index p in the following equations for simplicity. Discrimination Breadth Ratio ( \theta_{\text{ratio}} ). We compare the range of latent quality estimates between the LLM judge and human annotators: \displaystyle\theta_{\text{ratio}} \displaystyle= \displaystyle\theta_{\text{range}}^{(\text{LLM})}/\theta_{\text{range}}^{(\text{Human})} (5) \displaystyle\theta_{\text{range}} \displaystyle= \displaystyle\text{median}(\theta_{j}\mid Y_{j}=\max Y_{j^{\prime}}) (6) \displaystyle-\text{median}(\theta_{j}\mid Y_{j}=\min Y_{j^{\prime}}) This ratio reveals whether the LLM discriminates quality differences with similar breadth to humans. A ratio less than 1 indicates the LLM perceives a narrower quality spectrum—it is hypersensitive , failing to distinguish samples that humans would rate differently. A ratio greater than 1 indicates the LLM perceives a wider spectrum—it is insensitive , amplifying quality differences beyond human perception. A ratio near 1 suggests comparable discrimination breadth. Note that \theta_{\text{range}} alone is uninterpretable without human reference: a value of 1.5 could indicate appropriate, insufficient, or excessive discrimination depending on the human baseline. Human Alignment ( D_{W} ). To assess absolute alignment with human judgments, we compute the Wasserstein distance between LLM and human distributions of \hat{\theta}_{j} : D_{W}=W_{1}(\hat{\theta}^{(\text{LLM})},\hat{\theta}^{(\text{Human})}) (7) We choose Wasserstein over alternatives because correlation captures only linear relationships ignoring distributional shape, and KL divergence is asymmetric and undefined for non-overlapping support. Wasserstein captures both location shift and distributional differences, providing an interpretable cost of transforming one distribution into another. Diagnostic Interpretation. The combination of \theta_{\text{ratio}} and D_{W} enables nuanced comparison: \theta_{\text{ratio}}\approx 1 with high D_{W} indicates the judge discriminates with appropriate breadth but is systematically more lenient or strict, while \theta_{\text{ratio}}\neq 1 with low D_{W} suggests the judge’s overall perception aligns with humans despite differing sensitivity. Both metrics deviating from ideal values indicates fundamental differences in how the LLM perceives quality. 4 Demonstration 4.1 Judge prompts and Benchmarks We select Judge methods that provide rating-based evaluations across diverse criteria. All methods were evaluated on human annotations, enabling intrinsic reliability diagnostics (Phase 1). We provide detailed description in Appendix C . NLP. We evaluate two LLM-as-a-judge methods: G-Eval ( 32 ) and HelpSteer-2 ( 55 ) to mirror popular usages of LLM-based quality evaluation. We evaluated reliability of G-Eval on two datasets: SummEval ( 15 ) and TopicalChat ( 34 ) . For SummEval, we use the original human annotation script from the dataset, measuring summary quality with four 5-point scales. For TopicalChat, we adapt the SummEval format with metric definitions from 34 , evaluating dialogue quality with two binary scales (understandability, groundedness) and three 3-point scales (naturalness, coherence, engagingness). For HelpSteer-2, we use the original prompt from 55 , measuring response quality with five 5-point scales. Benchmark Gemini-2.5 GPT-4o GPT-4o-mini Qwen3-30b Qwen3-235b Llama-4-m Llama-4-s C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho SummEval Relevance 0.25 0.91 0.05 0.92 0.04 0.93 0.17 0.86 0.09 0.89 0.36 0.83 0.17 0.92 Consistency 0.07 0.55 0.05 0.91 0.92 0.88 0.15 0.60 0.08 0.66 0.25 0.63 0.07 0.78 Fluency 0.21 0.89 0.06 0.90 0.60 0.89 0.15 0.90 0.09 0.91 0.13 0.81 0.15 0.92 Coherence 0.09 0.89 0.29 0.94 0.06 0.92 0.22 0.87 0.16 0.92 0.18 0.84 0.15 0.88 TopicalChat Understandability 0.18 0.39 0.16 0.48 0.20 0.49 0.03 0.53 0.27 0.34 0.21 0.46 0.48 0.43 Naturalness 0.29 0.85 0.28 0.57 0.11 0.80 0.18 0.87 0.23 0.87 0.49 0.71 0.33 0.81 Coherence 0.27 0.79 0.14 0.84 0.17 0.81 0.13 0.85 0.17 0.86 0.25 0.81 0.33 0.82 Engagingness 0.15 0.78 0.52 0.72 0.20 0.79 0.21 0.87 0.16 0.86 0.21 0.70 0.11 0.80 Groundedness 0.17 0.72 0.07 0.71 0.12 0.70 0.18 0.73 0.30 0.71 0.10 0.71 0.18 0.73 HelpSteer-2 Helpfulness 0.03 0.86 0.08 0.84 0.30 0.67 0.27 0.72 0.37 0.78 0.13 0.64 0.28 0.66 Correctness 0.10 0.72 0.11 0.76 0.12 0.60 0.30 0.68 0.25 0.76 0.24 0.54 0.28 0.54 Coherence 0.16 0.68 0.24 0.67 0.40 0.54 0.21 0.58 0.78 0.60 0.25 0.40 0.31 0.49 Complexity 1.08 0.87 0.16 0.89 0.30 0.81 0.39 0.88 0.32 0.89 0.25 0.77 0.31 0.85 Verbosity 0.17 0.87 0.04 0.83 0.21 0.74 0.20 0.76 0.74 0.76 0.33 0.75 0.45 0.83 VIEScore CIG-SC 1.32 0.94 0.46 0.93 0.56 0.93 0.32 0.94 0.37 0.92 0.36 0.90 0.29 0.81 CIG- PQ 1.11 0.94 0.30 0.96 0.82 0.93 0.62 0.94 0.47 0.94 0.17 0.92 0.46 0.83 MIE-SC 1.01 0.94 0.37 0.95 0.52 0.94 0.45 0.94 0.94 0.95 0.54 0.93 0.55 0.87 MIE- PQ 1.14 0.92 1.02 0.94 0.58 0.93 0.40 0.94 0.61 0.90 0.32 0.89 0.58 0.80 TIE-SC 1.00 0.94 0.32 0.94 1.11 0.95 0.64 0.93 0.43 0.95 0.56 0.93 0.80 0.88 TIE- PQ 1.14 0.93 0.68 0.92 0.30 0.93 0.16 0.92 0.67 0.91 0.84 0.90 0.77 0.80 Table 1: Phase 1 intrinsic consistency results. CV denotes prompt consistency; \rho denotes marginal reliability (higher is better). Highlighted cells indicate CV \leq 0.10 and \rho\geq 0.70 . Gemini-2.5 refers to Gemini 2.5 Flash; Llama-4-m and Llama-4-s refer to Llama-4-Maverick and Llama-4-Scout, respectively. For VIEScore, Qwen3-30B and Qwen3-235B refer to their VL variants Vision. We evaluate one VLM-as-a-judge method: VIEScore ( 28 ) to mirror popular usages of image quality evaluation. We follow the same prompt on the same dataset. We evaluated reliability of VIEScore on ImageHub dataset ( 29 ) containing three subsets: control-guided image generation (CIG), text-guided image editing (TIE), and mask-guided image editing (MIE). VIEScore measures image-text alignment with two 11-point scales: semantic consistency (SC) and perceptual quality (PQ). Thus, we separately examined VIEScore on these subsets. 4.2 Selected Models We select judge models based on two criteria: (1) frontier models widely adopted in LLM-as-a-Judge deployments, and (2) coverage across modalities to enable cross-modal reliability comparison across diverse benchmarks. All evaluations are conducted via the OpenRouter API ( 36 ) with temperature set to 0 for deterministic outputs. Specifically, we used seven models from four families: Gemini 2.5 Flash ( 12 ) , GPT-4o and GPT-4o-mini ( 25 ) , Qwen3-30B-A3B-instruct and Qwen3-235B-A22B-instruct ( 63 ) , and LLaMA-4-Maverick and LLaMA-4-Scout ( 35 ) . Note that, for vision tasks, we used Qwen3-VL ( 1 ) instead of Qwen3 to ensure text and vision inputs. 4.3 Procedure for Analysis Step 1. Generating Prompt Variations. We generate three prompt variations for each evaluation prompt: typo, newline, and phrase. For typo variations, we select the five high-attention tokens 2 2 2 After excluding metric names, we chose five highest words. from the final layer of Qwen3-8B ( 63 ) and apply character-level perturbations to those tokens using the AugLy library ( 37 ) . For newline variations, we randomly add three newlines between sentence segments. For paraphrase variations, we extract five verbs or adjectives using NLTK POS tagging ( 3 ) , then use GPT-4o-mini to generate synonyms for those tokens. To ensure comparability across tasks, identical variations are applied to all models; details are provided in Appendix B . Step 2. Fitting IRT-GRM. We fit GRM using PyMC ( 43 ) with the NUTS sampler ( 24 ) via NumPyro ( 40 ) backend. We run 4 chains with 1000 warmup and 1000 sampling iterations, setting target acceptance rate to 0.95. For binary scale, we fit a 2-parameter logistic model instead as GRM assumes K>2 . Detailed hyperparameters are provided in Appendix A . 5 Phase 1: Instinsic Consistency Table 1 presents the Phase 1 diagnostic results, where we highlight results meeting consistency criteria: C_{V}\leq 0.10 and \rho\geq 0.7 . With two metrics, we analyze LLM-as-a-judge methods along three axes: modality, model, and task. Topical Chat Gemini-2.5 GPT-4o GPT-4o-mini Qwen3-30b Qwen3-235b Llama-4-m Llama-4-s C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho Detail Prompt Understandability 0.12 0.36 0.21 0.43 0.11 0.56 0.12 0.67 0.14 0.40 0.13 0.47 0.43 0.36 Naturalness 0.12 0.84 0.10 0.83 0.21 0.64 0.14 0.88 0.13 0.87 0.19 0.84 0.44 0.72 Coherence 0.20 0.78 0.10 0.78 0.21 0.84 0.13 0.83 0.21 0.81 0.14 0.84 0.25 0.81 Engagingness 0.11 0.76 0.16 0.69 0.17 0.70 0.13 0.80 0.23 0.80 0.10 0.82 0.14 0.61 Groundedness 0.04 0.73 0.03 0.68 0.07 0.71 0.10 0.73 0.22 0.69 0.22 0.72 0.14 0.71 Detail Prompt + CoT Understandability 0.13 0.50 0.22 0.58 0.28 0.60 0.14 0.64 0.44 0.46 0.14 0.67 0.18 0.66 Naturalness 0.18 0.84 0.01 0.85 0.47 0.70 0.23 0.87 0.06 0.87 0.06 0.87 0.09 0.80 Coherence 0.21 0.82 0.10 0.79 0.37 0.81 0.18 0.82 0.10 0.84 0.06 0.82 0.53 0.79 Engagingness 0.12 0.81 0.06 0.73 0.39 0.75 0.16 0.79 0.10 0.85 0.03 0.84 0.25 0.67 Groundedness 0.06 0.71 0.14 0.49 0.16 0.71 0.11 0.68 0.10 0.58 0.03 0.62 0.17 0.59 Likert 5 Naturalness 0.20 0.93 0.15 0.95 0.32 0.91 0.12 0.92 0.49 0.94 0.17 0.94 0.47 0.92 Coherence 0.20 0.90 0.09 0.91 0.13 0.95 0.12 0.93 0.28 0.93 0.14 0.92 0.13 0.92 Engagingness 0.13 0.92 0.19 0.91 0.10 0.93 0.16 0.94 0.25 0.94 0.27 0.94 0.07 0.91 Likert 7 Naturalness 0.08 0.95 0.15 0.95 0.32 0.87 0.22 0.90 0.34 0.93 0.26 0.81 0.24 0.90 Coherence 0.49 0.87 0.31 0.95 0.17 0.97 0.31 0.96 0.19 0.97 0.22 0.95 0.14 0.94 Engagingness 0.34 0.93 0.18 0.96 0.28 0.94 0.25 0.96 0.26 0.96 0.30 0.94 0.26 0.92 Detail Prompt + CoT + Likert 5 Naturalness 0.25 0.93 0.06 0.94 0.29 0.88 0.16 0.93 0.22 0.95 0.11 0.92 0.55 0.92 Coherence 0.18 0.89 0.18 0.89 0.10 0.94 0.09 0.91 0.19 0.92 0.17 0.90 0.33 0.89 Engagingness 0.15 0.90 0.08 0.87 0.26 0.92 0.07 0.90 0.11 0.94 0.17 0.94 0.27 0.88 Table 2: Ablation results on TopicalChat with varying instruction detail, chain-of-thought prompting, and rating scales. CV denotes prompt consistency (lower is better); \rho denotes marginal reliability (higher is better). Highlighted cells indicate CV \leq 0.10 and \rho\geq 0.70 . Gemini-2.5 refers to Gemini 2.5 Flash; Llama-4-m and Llama-4-s refer to Llama-4-Maverick and Llama-4-Scout, respectively. 5.1 Prompt Consistency ( C_{V} ) Modality: Vision is more prompt-sensitive than NLP. NLP judges on SummEval, TopicalChat, or HelpSteer-2 generally maintain C_{V}<0.30 , with several model-criterion pairs achieving C_{V}<0.10 . In contrast, VIEScore exhibits substantially higher C_{V} , ranging from 0.16 to 1.32. This gap is most pronounced for Gemini-2.5, which showed score from 0.03 to 0.29 on NLP tasks but exceeds C_{V}>1.0 across all VIEScore subtasks. These results suggest that vision-language evaluation is inherently more sensitive to prompt wording than text-based evaluation. Model: Scale improves prompt robustness in NLP but not in vision. In NLP benchmarks, larger models consistently achieve lower C_{V} . Qwen3-235B outperforms Qwen3-30B on SummEval (Relevance: 0.09 vs. 0.17, Fluency: 0.09 vs. 0.15, Coherence: 0.16 vs. 0.22). GPT-4o similarly surpasses GPT-4o-mini (SummEval Fluency: 0.06 vs. 0.60; HelpSteer-2 Helpfulness: 0.08 vs. 0.30). However, this scaling effect does not transfer to vision: in VIEScore, Qwen3-30B and Qwen3-235B show comparable C_{V} , and GPT-4o-mini occasionally matches or exceeds GPT-4o. This modality-dependent scaling suggests that prompt robustness in vision-language evaluation may rely on capabilities distinct from those improved by scale. Task: Summarization is most stable; vision tasks are not. Within NLP, judgment criteria in SummEval achieves the lowest C_{V} overall, with most models maintaining C_{V}<0.20 . Those in TopicalChat and HelpSteer-2 show moderate variability, though HelpSteer-2 Complexity exhibits notable instability for Gemini-2.5 (1.08). VIEScore consistently yields high C_{V} across all models and subtasks, indicating that prompt sensitivity is a task-level characteristic rather than a model-specific limitation. 5.2 Marginal Reliability ( \rho ) Modality: Vision achieves high reliability. Despite high prompt sensitivity, VIEScore achieves the highest \rho values among all benchmarks (0.80–0.96). This apparent paradox suggests that while vision-language judges are sensitive to exact prompt wording, they produce reliable quality orderings once a specific prompt is fixed. NLP benchmarks show more variance: SummEval achieves consistently high \rho (0.81–0.94), while TopicalChat and HelpSteer-2 exhibit criterion-dependent reliability. Model: Scale law emerges in NLP but not in vision. In NLP benchmarks, larger models consistently achieve higher \rho . Qwen3-235B outperforms Qwen3-30B on SummEval (0.89–0.92 vs. 0.86–0.90) and HelpSteer-2 Helpfulness (0.78 vs. 0.72) and Correctness (0.76 vs. 0.68). Similarly, GPT-4o surpasses GPT-4o-mini on HelpSteer-2 across all criteria (e.g., Helpfulness: 0.84 vs. 0.67, Correctness: 0.76 vs. 0.60). However, this scaling effect does not transfer to vision tasks: in VIEScore, Qwen3-30B and Qwen3-235B achieve comparable \rho (0.90–0.95), and GPT-4o-mini occasionally matches or surpasses GPT-4o. This modality-dependent scaling suggests that vision-language evaluation may rely on capabilities which do not scale with model size. Task: TopicalChat Understandability is least reliable. In NLP, TopicalChat Understandability consistently yields the lowest \rho across all models (0.34–0.53), substantially below the conventional threshold of \rho>0.7 . Other TopicalChat criteria (Naturalness, Coherence, Engagingness) achieve moderate-to-high reliability (0.70–0.87), while Groundedness remains borderline (0.70–0.73). HelpSteer-2 shows criterion-dependent patterns: Helpfulness and Complexity achieve high \rho (0.66–0.89), while Coherence underperforms (0.40–0.68). Criteria in SummEval maintains consistently high reliability across all criteria. Benchmark Gemini-2.5 GPT-4o GPT-4o-mini Qwen3-30b Qwen3-235b Llama-4-m Llama-4-s \theta_{\text{ratio}} D_{W} \theta_{\text{ratio}} D_{W} \theta_{\text{ratio}} D_{W} \theta_{\text{ratio}} D_{W} \theta_{\text{ratio}} D_{W} \theta_{\text{ratio}} D_{W} \theta_{\text{ratio}} D_{W} SummEval Relevance 0.96 0.30 1.55 0.30 1.39 0.34 1.30 0.42 1.47 0.21 1.07 0.25 1.30 0.28 Consistency 1.36 0.26 1.89 0.55 2.20 0.57 1.57 0.34 1.36 0.42 1.29 0.41 1.65 0.55 Fluency 1.71 0.48 1.63 0.42 2.29 0.57 1.97 0.49 1.76 0.50 1.33 0.47 1.57 0.49 Coherence 1.24 0.26 1.50 0.27 1.66 0.31 1.53 0.30 1.55 0.32 0.98 0.24 1.30 0.27 TopicalChat Understandability 2.46 0.33 2.59 0.33 2.59 0.34 2.56 0.36 2.89 0.36 2.29 0.31 2.75 0.33 Naturalness 1.71 0.48 3.19 0.54 2.52 0.55 2.12 0.41 1.96 0.50 2.31 0.50 2.35 0.41 Coherence 1.79 0.45 2.41 0.55 2.01 0.43 1.93 0.43 1.97 0.53 1.88 0.43 1.97 0.43 Engagingness 2.21 0.42 3.07 0.44 2.16 0.43 2.08 0.51 2.21 0.54 2.05 0.44 2.75 0.51 Groundedness 2.51 0.56 2.38 0.56 2.41 0.57 2.46 0.54 2.38 0.56 2.47 0.57 2.46 0.55 HelpSteer-2 Helpfulness 1.80 0.33 1.83 0.33 1.76 0.32 1.61 0.31 1.58 0.33 1.86 0.30 1.65 0.31 Correctness 1.46 0.28 1.55 0.29 1.77 0.32 1.52 0.29 1.60 0.30 1.76 0.34 1.75 0.34 Coherence 1.10 0.25 1.29 0.23 1.34 0.15 1.03 0.16 1.48 0.15 1.24 0.29 1.20 0.21 Complexity 1.26 0.34 1.16 0.30 1.50 0.32 1.10 0.46 1.30 0.36 1.32 0.28 1.40 0.30 Verbosity 1.02 0.37 1.21 0.45 1.46 0.17 0.99 0.53 1.18 0.52 1.40 0.22 1.33 0.23 VIEScore CIG-SC 2.08 0.45 1.92 0.47 1.48 0.45 1.74 0.46 1.37 0.57 1.48 0.43 0.86 0.38 CIG- PQ 3.35 0.51 3.03 0.50 3.11 0.43 3.09 0.46 2.61 0.56 2.76 0.42 2.03 0.33 MIE-SC 1.87 0.55 2.42 0.43 1.96 0.55 2.15 0.49 1.79 0.56 2.00 0.56 1.60 0.52 MIE- PQ 1.56 0.49 1.98 0.39 1.84 0.43 1.75 0.43 1.73 0.43 1.28 0.45 1.12 0.47 TIE-SC 1.99 0.51 2.06 0.51 2.29 0.45 2.20 0.47 2.09 0.50 1.85 0.52 1.71 0.51 TIE- PQ 3.86 0.61 4.40 0.60 4.14 0.52 3.75 0.60 3.92 0.58 3.47 0.53 3.91 0.58 Table 3: Phase 2 human alignment results. \theta_{\text{ratio}} denotes discrimination breadth ratio; D_{W} denotes Wasserstein distance between LLM and human \theta distributions. Gemini-2.5 refers to Gemini 2.5 Flash; Llama-4-m and Llama-4-s refer to Llama-4-Maverick and Llama-4-Scout, respectively. For VIEScore, Qwen3-30B and Qwen3-235B refer to their VL variants. 5.3 Discussion Phase 1 suggests that consistency of current LLM-as-a-judge criteria might depend on how the prompt specifies evaluation criteria. As shown in Table 1 , there is no free lunch; no model showed acceptable consistency across all criteria when we quantified C_{V} and \rho . Thus, one should verify the effect of prompts and models when suggesting a new LLM judgment tasks. It is still questionable why such difference occur. To diagnose the cuase, we conduct two simple ablation studies on evaluation criteria, using TopicalChat 3 3 3 For the result on VIEScore, see Appendix D . First, we test whether detailed evaluation instructions or chain-of-thought prompting improve consistency ( C_{V} ). Second, we examine whether the rating scales used in existing criteria are adequate for reliable judging ( \rho ) by comparing the original 3-point scale with alternative scales: 5-point and 7-point. The result of ablation study suggests proper way to design judgment prompts, as shown in Table 2 . First, instruction specification might affect consistency C_{V} . Providing detailed prompts substantially reduces C_{V} across criteria, and adding chain-of-thought prompting yields further improvement (e.g., Naturalness: GPT-4o achieves C_{V}=0.01 , Qwen3-30b and Llama-4-m achieve C_{V}=0.06 ). However, \rho shows only marginal gains, suggesting that detailed instructions stabilize prompt sensitivity but do not fundamentally improve discriminative capacity. Second, rating scales might affect marginal reliablity \rho . The 5-point scale shows modest improvement in \rho for graded criteria (Naturalness: 0.91–0.95, Coherence: 0.90–0.95, Engagingness: 0.91–0.94), but the 7-point scale does not yield consistent further gains and occasionally decreases reliability. These results emphasizes that our framework is suitable to capture subtle differences due to prompt engineering and scale adjustment. Though prior work reported similar observations that more explicit instructions or chain-of-thought prompting can lead to increased evaluation stability ( 54 ; 52 ; 42 ) , our result further implies guideline for designing a good judgment criteria: More detailed instructions tend to stabilize prompt consistency ( C_{V} ), while scale adjustment affects measurement reliability ( \rho ). 6 Phase 2: Human Alignment Having established intrinsic consistency in Phase 1, we now examine how LLM judges’ quality perception compares to human judgment. Table 3 presents the Phase 2 diagnostic results with two metrics. (1) \theta_{\text{ratio}} (discrimination breadth ratio) measures whether LLMs perceive quality differences with similar breadth to humans: \theta_{\text{ratio}}<1 , \theta_{\text{ratio}}\approx 1 and \theta_{\text{ratio}}>1 indicate hypersensitive, near-human, insensitive calibration. (2) D_{W} (Wasserstein distance) measures distributional divergence from human judgments, where lower values indicate closer alignment. Similar to Phase 1, we analyze both metrics along the same three axes. 6.1 Discrimination Breadth Ratio ( \theta_{\text{ratio}} ) Modality: Vision is more insensitive than NLP. VIEScore exhibits the highest \theta_{\text{ratio}} values, particularly for perceptual quality (PQ) subtasks (2.03–4.40), indicating that VLMs amplify quality differences far beyond human perception. Criteria in NLP benchmarks show more moderate insensitivity (1.0–2.5), with several criteria approaching near-human calibration ( \theta_{\text{ratio}}\approx 1 ). Model: No consistent scaling effect on calibration. Unlike Phase 1, model scale does not show consistent effects on \theta_{\text{ratio}} . Qwen3-235B and Qwen3-30B achieve comparable values across benchmarks, as do GPT-4o and GPT-4o-mini. This suggests that discrimination calibration is depending on task rather than model capacity. Task: HelpSteer-2 shows most appropriate calibration. HelpSteer-2 Coherence and Verbosity most frequently achieve \theta_{\text{ratio}}\approx 1 (e.g., Qwen3-30b: 1.03, 0.99; Gemini-2.5: 1.10, 1.02). SummEval Relevance and Coherence also show near-human calibration for select models. In contrast, TopicalChat and VIEScore PQ consistently exhibit insensitivity across all models, suggesting these tasks inherently elicit amplified quality perception in LLMs. 6.2 Distributional Alignment ( D_{W} ) Modality: No systematic modality effect. D_{W} shows no clear difference between NLP and vision benchmarks. Both modalities exhibit similar ranges (0.15–0.61), with within-modality difference exceeding between-modality difference. Model: Benchmark effects dominate model effects. All models exhibit similar D_{W} patterns within each benchmark, with minimal variation across model families or scales. This suggests that distributional alignment is primarily determined by task characteristics. Task: HelpSteer-2 Coherence achieves closest alignment. HelpSteer-2 Coherence achieves the lowest D_{W} across all benchmarks (0.15–0.29), indicating closest distributional alignment with human judgments. SummEval Coherence also performs well (0.24–0.32). VIEScore TIE-PQ shows the highest D_{W} (0.51–0.61), consistent with its extremely high \theta_{\text{ratio}} values. Figure 1 : Median latent quality ( \theta ) by human score for SummEval-Coherence (up) and VIEScore-MIE PQ (down). Each line represents a different judge model; the black line denotes human. 6.3 Discussion Phase 2 analysis reveals that \theta_{\text{ratio}} and D_{W} capture complementary but incomplete aspects of human alignment. \theta_{\text{ratio}} identifies systematic insensitivity ( \theta_{\text{ratio}}>1 ) across nearly all model-task combinations, indicating that LLM judges consistently perceive quality differences more broadly than humans. D_{W} quantifies overall distributional divergence but shows no clear effect of modality. However, these aggregate metrics mask a critical distinction that emerges only through score-level analysis. Figure 6 visualizes the relationship between human scores and estimated \theta for representative NLP and vision tasks. In NLP tasks (SummEval-Coherence), all models exhibit monotonically increasing \theta trajectories parallel to humans. The slopes differ, but the direction is preserved. This pattern indicates a calibration mismatch: LLMs and humans perceive the same quality construct with different scales. In contrast, vision tasks (VIEScore-MIE PQ) reveal non-monotonic patterns where model \theta values decrease or plateau. This suggests a validity gap between VLMs and humans: VLMs may evaluate different aspects of quality than what humans evaluate. This interpretation is evidence by sample-wise Pearson correlations between human rating scores and machine rating scores ( 39 ; Appendix E ): NLP tasks achieve moderate correlations ( r=0.2 – 0.5 ), while vision tasks approach zero. Together, these findings suggest that \theta_{\text{ratio}}>1 carries different implications depending on the underlying pattern. In NLP tasks, the calibration mismatch indicates that LLMs measure the same construct as humans but with different scales. In vision tasks, the calibration mismatch indicates the validity gap. We provide score-level visualizations for all benchmarks in Appendix F . 7 Conclusion We presented a framework for diagnosing reliability of LLM-as-a-Judge by adapting Item Response Theory to the evaluation setting, where prompt variations serve as measurement items and latent quality \theta captures perceived sample quality. From this formulation, we derive four interpretable metrics organized into two sequential phases. Phase 1 metrics ( C_{V} and \rho ) identify whether inconsistency stems from prompt sensitivity or inadequate use of the rating scale, guiding practitioners toward appropriate adjustments. Phase 2 metrics ( \theta_{\text{ratio}} and D_{W} ) reveal how LLM judges differ from humans, distinguishing calibration mismatch from validity gap. This framework provides practitioners with a principled tool for diagnosing LLM judge reliability. Impact Statement This work aims to promote more responsible use of LLM-as-a-Judge by offering a diagnostic framework that clarifies when automated evaluators behave as reliable measurement instruments and when they do not. By separating intrinsic consistency from human alignment, the proposed approach helps mitigate the risk of over-trusting LLM-based evaluation in benchmarking and model development. At the same time, our study has several limitations that constrain its broader impact. The framework is currently applicable only to point-scale (rating-based) judgments and does not directly extend to pairwise or open-ended evaluation settings. While we analyze consistency and alignment, we do not assess whether the internal reasoning or explanations produced by LLM judges are faithful to their actual decision processes. In addition, our experiments focus on a limited set of languages and modalities; multilingual evaluation and a wider range of modalities remain unexplored and may exhibit different reliability characteristics. Addressing these limitations is an important direction for future work to ensure that reliability diagnostics for LLM-as-a-Judge generalize across settings and do not introduce unintended biases or misuse. The Use of Large Language Models We used AI assistance tools during the writing process of this manuscript. Specifically, we employed Grammarly for grammar checking, and GPT-5 for language polishing and improving clarity of expression. These tools were used for editorial purposes. Appendix A Details of Graded Response Model for LLM Judges A.1 Model Fitting We implement GRM using PyMC 5.25.1 with ArviZ 0.23.0 for diagnostics. For posterior inference, we use the NUTS sampler with 4 chains, 1000 warmup iterations, 1000 sampling iterations, and target acceptance rate of 0.95. We use adapt_diag initialization and fix the random seed to 42 for reproducibility. We apply the following priors: \theta_{j}\sim\mathcal{N}(0,1) for latent quality, \alpha_{p}\sim\text{LogNormal}(0,0.5) for discrimination parameters, and \beta_{pk}\sim\mathcal{N}(0,1) with an ordered constraint ( \beta_{pk}<\beta_{p,k+1} for all k LONLY (attention: 0.0038), JSON -> SJON (attention: 0.0020), task -> tsak (attention: 0.0018), } "[NEWLINE VARIATIONS]": { Extra newlines inserted after line 9, Extra newlines inserted after line 10, Extra newlines inserted after line 14 } "[PARAPHRASE VARIATIONS]": { evaluate -> assess, written -> composed, follow -> adhere, aware -> cognizant, proposed -> suggested } Table 4 : summEval. helpSteer "[Typo VARIATIONS]": { Your -> Youe (attention: 0.0608), Please -> lPease (attention: 0.0039), only -> onyl (attention: 0.0038), } "[NEWLINE VARIATIONS]": { Extra newlines inserted after line 2, Extra newlines inserted after line 6, Extra newlines inserted after line 8 } "[PARAPHRASE VARIATIONS]": { expert -> specialist, assess -> evaluate, following -> subsequent, consistent -> coherent, sophisticated -> complex } Table 5 : helpSteer. Appendix C Prompt Template We adopt prompt templates from the original papers: SummEval and TopicalChat, HelpSteer-2 ( 55 ) , and VIEScore ( 28 ) . For some benchmarks, we modify the output instruction to enforce JSON format. The detailed description and chain-of-thought (CoT) variants in our ablation study are our modifications. C.1 SummEval In this task you will evaluate the quality of summaries written for a news article. To correctly solve this task, follow these steps: 1. Carefully read the news article, be aware of the information it contains. 2. Read the proposed summary. 3. Rate the summary on a scale from 1 (worst) to 3 (best) by its relevance, consistency, fluency, and coherence. Definitions: Relevance: The rating measures how well the summary captures the key points of the article. Consistency: The rating measures whether the facts in the summary are consistent with the facts in the original article. Fluency: This rating measures the quality of individual sentences. Coherence: This rating measures the quality of all sentences collectively. — News Article: {article} — Summary: {summary} — Please provide your ratings in the following JSON format ONLY: "relevance": <1-5>, "consistency": <1-5>, "fluency": <1-5>, "coherence": <1-5> C.2 TopicalChat C.2.1 Original Prompt In this task, you will evaluate the quality of a dialogue response in a knowledge-grounded conversation. Please carefully follow the steps below: 1. Read the dialogue context to understand the conversation history. 2. Read the given response. 3. Rate the summary using integer scores understandability (0–1), naturalness (1–5), coherence (1–5), engagingness (1–5), groundedness (0–1). Definitions: Understandability: judge whether the response is understandable. Naturalness: Judge whether a response is like something a person would naturally say. Coherence: Determine whether this response serves as a valid continuation of the previous conversation. Engagingness: Determine if the response is interesting or dull. Groundedness: Given the fact that this response is conditioned on, determine whether this response uses that fact. — Dialogue Context: {context} — Response: {response} — Please provide your ratings in the following JSON format ONLY: "understandability": <0-1>, "naturalness": <1-3>, "coherence": <1-3>, "engagingness": <1-3>, "groundedness": <0-1> C.2.2 Detailed Prompt In this task, you will evaluate the quality of a dialogue response in a knowledge-grounded conversation. Please carefully follow the steps below: 1. Read the dialogue context to understand the conversation history. 2. Read the given response. 3. Rate the summary using integer scores understandability (0–1), naturalness (1–5), coherence (1–5), engagingness (1–5), groundedness (0–1). Definitions: Understandability: Judge whether the response is understandable in terms of basic meaning and intent. A response is understandable if a reader can clearly grasp what is being said without ambiguity, guesswork, or repeated re-reading. Assign a lower score if the response contains grammatical errors, broken sentence structure, unclear references, or logical gaps that make the meaning difficult or impossible to understand. Naturalness: Judge whether the response sounds like something a real person would naturally say in a conversational setting. Consider fluency, phrasing, tone, and whether the language feels organic rather than robotic, translated, or overly formal. Lower scores indicate stiff, mechanical, repetitive, or unnatural wording, while higher scores indicate smooth, human-like conversational language. Coherence: Determine whether this response serves as a valid and appropriate continuation of the previous conversation. Consider whether the response directly addresses the preceding dialogue, stays on topic, and follows logically from what was said before. Lower scores indicate topic drift, non sequiturs, contradictions, or failure to respond to the prior turn. Engagingness: Determine whether the response is interesting, informative, or engaging for a human reader. Lower scores indicate dull, generic, minimal, or boilerplate responses that add little value to the conversation. Higher scores indicate responses that provide useful information, thoughtful elaboration, or otherwise maintain the reader’s interest. Groundedness: Given the fact or knowledge that this response is conditioned on, determine whether the response correctly uses and reflects that information. Assign a lower score if the response introduces unsupported claims, contradicts the provided context, or ignores relevant grounding facts. Assign a higher score if the response is consistent with, accurately reflects, and appropriately incorporates the given factual information. — Dialogue Context: {context} — Response: {response} — Please provide your ratings in the following JSON format ONLY: "understandability": <0-1>, "naturalness": <1-3>, "coherence": <1-3>, "engagingness": <1-3>, "groundedness": <0-1> C.2.3 Detailed + CoT Prompt In this task, you will evaluate the quality of a dialogue response in a knowledge-grounded conversation. Please carefully follow the steps below: 1. Read the dialogue context to understand the conversation history. 2. Read the given response. 3. Rate the summary using integer scores understandability (0–1), naturalness (1–5), coherence (1–5), engagingness (1–5), groundedness (0–1). Definitions: Understandability: Judge whether the response is understandable in terms of basic meaning and intent. A response is understandable if a reader can clearly grasp what is being said without ambiguity, guesswork, or repeated re-reading. Assign a lower score if the response contains grammatical errors, broken sentence structure, unclear references, or logical gaps that make the meaning difficult or impossible to understand. Naturalness: Judge whether the response sounds like something a real person would naturally say in a conversational setting. Consider fluency, phrasing, tone, and whether the language feels organic rather than robotic, translated, or overly formal. Lower scores indicate stiff, mechanical, repetitive, or unnatural wording, while higher scores indicate smooth, human-like conversational language. Coherence: Determine whether this response serves as a valid and appropriate continuation of the previous conversation. Consider whether the response directly addresses the preceding dialogue, stays on topic, and follows logically from what was said before. Lower scores indicate topic drift, non sequiturs, contradictions, or failure to respond to the prior turn. Engagingness: Determine whether the response is interesting, informative, or engaging for a human reader. Lower scores indicate dull, generic, minimal, or boilerplate responses that add little value to the conversation. Higher scores indicate responses that provide useful information, thoughtful elaboration, or otherwise maintain the reader’s interest. Groundedness: Given the fact or knowledge that this response is conditioned on, determine whether the response correctly uses and reflects that information. Assign a lower score if the response introduces unsupported claims, contradicts the provided context, or ignores relevant grounding facts. Assign a higher score if the response is consistent with, accurately reflects, and appropriately incorporates the given factual information. — Dialogue Context: {context} — Response: {response} — Please evaluate the quality of the judgement based on whether the judgement is grounded on the responses and carefully follows the rubric. Provide some reasoning and analysis to support your evaluation. Next, provide an integer rating in JSON format. — Please provide your ratings in the following JSON format: "understandability": <0-1>, "naturalness": <1-5>, "coherence": <1-5>, "engagingness": <1-5>, "groundedness": <0-1> C.3 HelpSteer-2 You are an expert evaluator for answers to user queries. Your task is to assess responses to user queries based on helpfulness, relevance, accuracy, and clarity. Calculate the following metrics for the response: User Query: {query} Model Response: {response} Metrics: 1. helpfulness (1-5): How well does the response help the user? 2. correctness (1-5): Is the information correct? 3. coherence (1-5): Is the response logically consistent and well-structured? 4. complexity (1-5): How sophisticated is the response? 5. verbosity (1-5): Is the response appropriately detailed? Instructions: Assign a score from 1 (poor) to 5 (excellent) for each metric. — Please provide your ratings in the following JSON format ONLY: "helpfulness": <1-5>, "correctness": <1-5>, "coherence": <1-5>, "complexity": <1-5>, "verbosity": <1-5> C.4 VIEScore-CIG-SC RULES: Two images will be provided: The first being a processed image (e.g. Canny edges, openpose, grayscale, etc.) and the second being an AI-generated image using the first image as guidance. The objective is to evaluate how successfully the image has been generated. On scale 0 to 10: A score from 0 to 10 will be given based on the success in following the prompt. (0 indicates that the second image does not follow the prompt at all. 10 indicates the second image follows the prompt perfectly.) A second score from 0 to 10 will rate how well the generated image is following the guidance image. (0 indicates that the second image does not follow the guidance at all. 10 indicates that the second image is following the guidance image.) Put the score in a list such that output score = [score1, score2], where ’score1’ evaluates the prompt and ’score2’ evaluates the guidance. Text Prompt: text C.5 VIEScore-TIE-SC RULES: Two images will be provided: The first being the original AI-generated image and the second being an edited version of the first. The objective is to evaluate how successfully the editing instruction has been executed in the second image. Note that sometimes the two images might look identical due to the failure of the image edit. On scale of 0 to 10: A score from 0 to 10 will be given based on the success of the editing. (0 indicates that the scene in the edited image does not follow the editing instructions at all. 10 indicates that the scene in the edited image follows the editing instruction text perfectly.) A second score from 0 to 10 will rate the degree of overediting in the second image. (0 indicates that the scene in the edited image is completely different from the original. 10 indicates that the edited image can be recognized as a minimally edited yet effective version of the original.) Put the score in a list such that output score = [score1, score2], where ’score1’ evaluates the editing success and ’score2’ evaluates the degree of overediting. Editing instruction: instruction C.6 VIEScore-PQ RULES: The image is an AI-generated image. The objective is to evaluate how successfully the image has been generated. On a scale 0 to 10: A score from 0 to 10 will be given based on image naturalness. ( 0 indicates that the scene in the image does not look natural at all or gives an unnatural feeling such as a wrong sense of distance, wrong shadow, or wrong lighting. 10 indicates that the image looks natural. ) A second score from 0 to 10 will rate the image artifacts. ( 0 indicates that the image contains a large portion of distortion, watermarks, scratches, blurred faces, unusual body parts, or subjects not harmonized. 10 indicates the image has no artifacts. ) Put the score in a list such that output score = [naturalness, artifacts] Appendix D Details of Intrinsic Reliability Topical Chat Gemini-2.5 GPT-4o GPT-4o-mini Qwen3-30b Qwen3-235b Llama-4-m Llama-4-s C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho C_{V} \rho Detail Prompt + CoT CIG-SC 1.15 0.91 0.63 0.90 0.58 0.79 0.64 0.82 0.67 0.83 0.20 0.58 0.19 0.66 CIG- PQ 1.12 0.93 0.68 0.93 0.51 0.72 0.52 0.86 0.85 0.89 0.21 0.66 0.27 0.75 MIE-SC 0.25 0.93 0.29 0.95 0.68 0.95 0.34 0.86 0.77 0.85 0.33 0.83 0.45 0.82 MIE- PQ 0.22 0.59 0.33 0.86 0.72 0.93 0.36 0.87 0.85 0.79 0.23 0.65 0.26 0.65 TIE-SC 0.99 0.92 0.63 0.94 0.75 0.96 0.79 0.87 0.39 0.87 0.51 0.85 0.26 0.84 TIE- PQ 1.52 0.93 0.73 0.89 0.84 0.94 0.62 0.80 0.70 0.88 0.17 0.64 0.42 0.73 Table 6: VIEScore intrinsic result. Gemini-2.5 refers to Gemini 2.5 Flash; Llama-4-m and Llama-4-s refer to Llama-4-Maverick and Llama-4-Scout, respectively. For VIEScore, Qwen3-30B and Qwen3-235B refer to their VL variants. Appendix E Perason Gemini-2.5 GPT-4o GPT-4o-mini Qwen3-30b Qwen3-235b Llama-4-m Llama-4-s SummEval Relevance 0.49 *** 0.51 *** 0.53 *** 0.54 *** 0.53 *** 0.40 *** 0.51 *** Consistency 0.75 *** 0.61 *** 0.55 *** 0.72 *** 0.73 *** 0.65 *** 0.67 *** Fluency 0.60 *** 0.56 *** 0.45 *** 0.54 *** 0.47 *** 0.49 *** 0.49 *** Coherence 0.54 *** 0.57 *** 0.52 *** 0.51 *** 0.58 *** 0.48 *** 0.52 *** TopicalChat Understandability 0.33 *** 0.38 *** 0.34 *** 0.38 *** 0.30 *** 0.25 *** 0.29 *** Naturalness 0.35 *** 0.49 *** 0.26 *** 0.37 *** 0.46 *** 0.40 *** 0.31 *** Coherence 0.23 *** 0.27 *** 0.30 *** 0.26 *** 0.25 *** 0.21 *** 0.21 *** Engagingness 0.40 *** 0.46 *** 0.39 *** 0.40 *** 0.48 *** 0.31 *** 0.34 *** Groundedness 0.52 *** 0.54 *** 0.53 *** 0.40 *** 0.51 *** 0.42 *** 0.44 *** HelpSteer-2 Helpfulness 0.46 *** 0.49 *** 0.41 *** 0.41 *** 0.47 *** 0.29 *** 0.41 *** Correctness 0.40 *** 0.48 *** 0.47 *** 0.37 *** 0.41 *** 0.34 *** 0.39 *** Coherence 0.35 *** 0.39 *** 0.35 *** 0.30 *** 0.30 *** 0.24 *** 0.27 *** Complexity 0.20 *** 0.40 *** 0.41 *** 0.21 *** 0.26 *** 0.27 *** 0.36 *** Verbosity 0.21 *** 0.31 *** 0.40 *** 0.18 *** -0.03 0.31 *** 0.40 *** VIEScore CIG-SC 0.01 -0.15 -0.15 -0.09 -0.11 -0.25 * -0.17 PQ 0.02 0.03 -0.05 0.12 0.08 0.22 -0.02 MIE-SC 0.04 -0.03 0.05 -0.06 -0.08 -0.12 -0.07 PQ 0.02 0.07 0.16 * 0.14 0.10 0.07 -0.13 TIE-SC 0.08 -0.03 0.12 0.09 0.11 0.03 0.05 PQ 0.06 0.07 -0.13 0.05 0.04 0.03 0.02 Table 7: Pearson correlation between human and LLM judgments. Gemini-2.5 refers to Gemini 2.5 Flash; Llama-4-m and Llama-4-s refer to Llama-4-Maverick and Llama-4-Scout, respectively. For VIEScore, Qwen3-30B and Qwen3-235B refer to their VL variants. * p<0.05 , ** p<0.01 , *** p<0.001 . Refer to Table 7 through Figure 7 . Appendix F Visualization Refer to Figure 2 to Figure 2 : Median \theta by human score for all SummEval metrics. Figure 3 : Median \theta by human score for all TopicalChat metrics. Figure 4 : Median \theta by human score for all HelpSteer-2 metrics. Figure 5 : Median \theta by human score for VIEScore-CIG (SC and PQ). Figure 6 : Median \theta by human score for VIEScore-MIE (SC and PQ). Figure 7 : Median \theta by human score for VIEScore-TIE (SC and PQ).