Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge

Yulong He, Ivan Smirnov, Dmitry Fedrushkov, Sergey Kovalchuk, Ilya Revin · 2026

Read Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge by Yulong He, Ivan Smirnov, Dmitry Fedrushkov, Sergey Kovalchuk, Ilya Revin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03742.txt.

Introduction

Yulong He and colleagues improved how language models judge other models by breaking each verdict into clear criteria, rather than asking for one bare score. They used the Analytic Hierarchy Process to weigh those criteria, then added fuzzy numbers that carry how sure the judge is. Their DualJudge system blends a direct overall score with the structured breakdown, giving more weight to the side that hangs together best. Tested on JudgeBench, the structured methods beat direct scoring, and DualJudge came out on top overall.

Catalog record

CREATORYulong He, Ivan Smirnov, Dmitry Fedrushkov, Sergey Kovalchuk, Ilya Revin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45244
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03742

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge Yulong He, Ivan Smirnov, Dmitry Fedrushkov, Sergey Kovalchuk, Ilya Revin Abstract Effective evaluation of large language models (LLMs) remains a critical bottleneck, as conventional direct scoring often yields inconsistent and opaque judgments. In this work, we adapt the Analytic Hierarchy Process (AHP) to LLM-based evaluation and, more importantly, propose a confidence-aware Fuzzy AHP (FAHP) extension that models epistemic uncertainty via triangular fuzzy numbers modulated by LLM-generated confidence scores. Systematically validated on JudgeBench, our structured approach decomposes assessments into explicit criteria and incorporates uncertainty-aware aggregation, producing more calibrated judgments. Extensive experiments demonstrate that both crisp and fuzzy AHP consistently outperform direct scoring across model scales and dataset splits, with FAHP showing superior stability in uncertain comparison scenarios. Building on these insights, we propose DualJudge , a hybrid framework inspired by Dual-Process Theory that adaptively fuses holistic direct scores with structured AHP outputs via consistency-aware weighting. DualJudge achieves state-of-the-art performance, underscoring the complementary strengths of intuitive and deliberative evaluation paradigms. These results establish uncertainty-aware structured reasoning as a principled pathway toward more reliable LLM assessment. Code is available at https://github.com/hreyulog/AHP_llm_judge . 1 Introduction The rapid progress of large language models (LLMs) has created an urgent demand for reliable and scalable evaluation methodologies. Recent research increasingly adopts LLM-as-a-Judge, where language models automatically assess generated outputs as a substitute for costly human evaluation. Such approaches have demonstrated promising alignment with human judgments across diverse tasks [ 2 , 5 ] . However, despite their growing adoption, existing evaluation paradigms largely rely on a single judging strategy, which introduces systematic biases across domains. Also, this strategy is not robust for different the instructions and prompts. [ 10 ] . The dominant paradigm employs holistic evaluation, where an LLM directly produces an overall score or preference through intuitive judgment. Direct scoring and pairwise comparison are efficient but implicitly compress multiple evaluation dimensions into a single reasoning step. Prior studies show that LLM evaluators are sensitive to prompting formats and reasoning styles, resulting in unstable performance across task types [ 2 ] . Moreover, LLM-based evaluation approach is inconsistent for different scoring scale [ 9 ] . In particular, holistic judgments may fail to capture structured reasoning signals required in logic-intensive domains such as mathematics, coding, and legal analysis. An alternative paradigm introduces structured evaluation, which decomposes judgments into multiple criteria and aggregates them using formal multi-criteria decision-making (MCDM) frameworks such as the Analytic Hierarchy Process (AHP). Recent work has explored integrating AHP with LLMs for specific evaluation and decision-support scenarios [ 11 , 22 , 21 , 8 ] , suggesting that hierarchical reasoning can improve interpretability and alignment with human assessment. Nevertheless, the effectiveness of structured hierarchical evaluation as a general paradigm for LLM-as-a-Judge remains insufficiently understood, and systematic comparisons with holistic judging across heterogeneous domains are largely missing. In this work, we conducted a comprehensive empirical study on JudgeBench [ 17 ] , covering 17 categories spanning mathematical reasoning, programming, scientific knowledge, and soft-knowledge tasks. Our results reveal a clear pattern: structured evaluation based on AHP consistently outperforms direct holistic judging in logic-intensive domains, demonstrating that explicit criterion decomposition improves evaluation reliability. Moreover, we find that modeling judgment uncertainty further enhances performance. By extending AHP with triangular fuzzy numbers to represent preference ambiguity, Fuzzy AHP (FAHP) achieves higher accuracy than traditional crisp AHP, indicating that incorporating uncertainty better reflects the inherently probabilistic nature of LLM-generated judgments. We also treated scoring scale design as a controllable component of the system and showed the impact of the method for different scales. Motivated by the complementary strengths of holistic and structured evaluation, we further introduce DualJudge, a hybrid framework that combines two independent evaluation signals: a direct holistic score and a structured AHP-based score (crisp or fuzzy). Rather than introducing complex interactions, DualJudge aggregates the two scores through simple averaging, yielding consistently improved robustness across domains. Despite its simplicity, this hybrid strategy achieves the best overall performance on JudgeBench. Our contributions are threefold: • We provide a systematic empirical comparison between holistic and structured evaluation paradigms within LLM-as-a-Judge across diverse domains. • We demonstrate that AHP-based structured evaluation significantly outperforms direct judging baselines, and that FAHP further improves performance by modeling judgment uncertainty. • We propose DualJudge, a simple hybrid averaging framework that combines complementary evaluation signals and achieves the strongest overall accuracy. 2 Related Work 2.1 LLM-as-a-Judge The rapid proliferation of LLMs has necessitated scalable evaluation methodologies, leading to the widespread adoption of the LLM-as-a-Judge paradigm [ 5 ] . This approach substitutes costly human annotation with automated assessments, demonstrating promising alignment with human preferences across diverse tasks. The dominant strategy in this domain is holistic evaluation , where an LLM generates a single overall score or preference through intuitive judgment. While computationally efficient, holistic methods implicitly compress multiple evaluation dimensions into a single reasoning step. Prior studies indicate that such evaluators are highly sensitive to prompting formats and reasoning styles, resulting in unstable performance and systematic biases, particularly in logic-intensive domains such as mathematics and coding. To mitigate these limitations, recent research has explored structured evaluation paradigms. Unlike holistic judging, structured approaches decompose complex judgments into multiple criteria and aggregate them using formal frameworks [ 11 , 22 ] . Thus, the widely used prompt-based evaluation solution G-Eval [ 10 ] interacts with predefined evaluation criteria and detailed evaluation steps to obtain the aggregate score. This decomposition forces the model to engage with specific quality dimensions systematically, enhancing interpretability and reducing the opacity inherent in black-box verdicts [ 21 ] . Although initial work suggests that hierarchical reasoning improves alignment with human assessment [ 8 ] , systematic comparisons between holistic and structured paradigms across heterogeneous domains remain insufficiently understood. Our work addresses this gap by empirically validating the superiority of structured reasoning in logic-driven tasks. 2.2 Analytic Hierarchy Process The AHP, originally proposed by Saaty [ 16 , 14 ] , is a classical MCDM framework designed to decompose complex decision problems into hierarchical structures. AHP organizes decision elements into multiple levels, typically consisting of a goal layer, a criteria layer, and an alternative layer, enabling systematic reasoning through pairwise comparisons. Given a set of criteria, decision makers construct a pairwise comparison matrix \mathbf{A}=(a_{ij}),\quad a_{ij}>0 , where a_{ij} denotes the relative importance of criterion i over criterion j . The priority weight vector \mathbf{w} is obtained from the principal eigenvector: \mathbf{A}\mathbf{w}=\lambda_{\max}\mathbf{w}, where \lambda_{\max} represents the maximum eigenvalue of \mathbf{A} . The consistency ratio (CR) is further computed to ensure logical coherence in human judgments. Due to its interpretability and structured reasoning capability, AHP has been widely applied in evaluation, ranking, risk analysis, and resource allocation tasks. In recent years, structured evaluation paradigms inspired by AHP have also been adopted in automated assessment systems and LLM evaluation pipelines, where complex judgments are decomposed into fine-grained criteria [ 11 ] . However, classical AHP assumes precise numerical comparisons between criteria. In practice, human judgments—and particularly LLM-generated evaluations—often exhibit uncertainty, vagueness, and linguistic ambiguity. This limitation motivates extensions that incorporate uncertainty modeling. 2.3 Fuzzy Analytic Hierarchy Process To address uncertainty in pairwise comparisons, researchers introduced fuzzy set theory [ 23 ] into AHP,

Read the complete text