A Survey on Agent-as-a-Judge

Runyang You \equal 1 Hongru Cai \equal 1 Caiqi Zhang 2 Qiancheng Xu 1 Meng Liu 3 Tiezheng Yu 4 Yongqi Li \advisor 1 Wenjie Li 1 1 The Hong Kong Polytechnic University 2 University of Cambridge 3 Shandong Jianzhu University 4 Huawei Technologies \equal Equal contribution \advisor Corresponding author · 2026

Read A Survey on Agent-as-a-Judge by Runyang You \equal 1 Hongru Cai \equal 1 Caiqi Zhang 2 Qiancheng Xu 1 Meng Liu 3 Tiezheng Yu 4 Yongqi Li \advisor 1 Wenjie Li 1 1 The Hong Kong Polytechnic University 2 University of Cambridge 3 Shandong Jianzhu University 4 Huawei Technologies \equal Equal contribution \advisor Corresponding author (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05111.txt.

Introduction

Runyang You and 7 co-authors survey the shift from language-model judges to agent judges in AI evaluation in a 2026 paper. Using a model to grade other systems made evaluation scalable, but complex, specialised, multi-step work strains that approach: single-pass judges show biases, reason shallowly, and cannot verify claims against real-world observations. Agent judges instead plan, use tools, collaborate, and keep persistent memory while evaluating.

The paper maps how this agentic judging works and why it promises more robust, verifiable, and nuanced assessments. As the systems being graded grow more capable, the grader must investigate rather than glance. Better automated evaluation underpins every claim about model progress, so strengthening the judge strengthens the whole field's evidence base.

Catalog record

CREATORRunyang You \equal 1 Hongru Cai \equal 1 Caiqi Zhang 2 Qiancheng Xu 1 Meng Liu 3 Tiezheng Yu 4 Yongqi Li \advisor 1 Wenjie Li 1 1 The Hong Kong Polytechnic University 2 University of Cambridge 3 Shandong Jianzhu University 4 Huawei Technologies \equal Equal contribution \advisor Corresponding author
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05111

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Survey on Agent-as-a-Judge Runyang You \equal 1 Hongru Cai \equal 1 Caiqi Zhang 2 Qiancheng Xu 1 Meng Liu 3 Tiezheng Yu 4 Yongqi Li \advisor 1 Wenjie Li 1 1 The Hong Kong Polytechnic University 2 University of Cambridge 3 Shandong Jianzhu University 4 Huawei Technologies \equal Equal contribution \advisor Corresponding author [email protected], {henry.hongrucai, liyongqi0}@gmail.com, [email protected] Abstract LLM-as-a-Judge has revolutionized AI evaluation by leveraging large language models for scalable assessments. However, as evaluands become increasingly complex, specialized, and multi-step, the reliability of LLM-as-a-Judge has become constrained by inherent biases, shallow single-pass reasoning, and the inability to verify assessments against real-world observations. This has catalyzed the transition to Agent-as-a-Judge , where agentic judges employ planning, tool-augmented verification, multi-agent collaboration, and persistent memory to enable more robust, verifiable, and nuanced evaluations. Despite the rapid proliferation of agentic evaluation systems, the field lacks a unified framework to navigate this shifting landscape. To bridge this gap, we present the first comprehensive survey tracing this evolution. Specifically, we identify key dimensions that characterize this paradigm shift and establish a developmental taxonomy. We organize core methodologies and survey applications across general and professional domains. Furthermore, we analyze frontier challenges and identify promising research directions, ultimately providing a clear roadmap for the next generation of agentic evaluation. \PublicDate 2026-1-8 1 Introduction (a) LLM-as-a-Judge (b) Agent-as-a-Judge Figure 1 : Comparison between LLM-as-a-Judge ( 1(a) ) and Agent-as-a-Judge ( 1(b) ). The former performs direct single-pass evaluation, while the latter leverages planning, memory, and tool-augmented capabilities for enhanced evaluation. The rapid advancement of Large Language Models (LLMs) has revolutionized the field of AI evaluation, giving rise to the LLM-as-a-Judge paradigm [ 63 ] . While traditional metrics fail to capture semantic nuance and human judgment remains unscalable, this new approach leverages LLMs’ advanced understanding and decision-making abilities to deliver near-human quality assessments across diverse domains [ 30 ] . Moreover, serving as a scalable proxy for human preference, LLM judges can provide reward signals for reinforcement learning [ 27 ] and enable the automated curation of massive synthetic datasets [ 38 ] . As such, LLM judgment has established itself as a cornerstone of AI evaluation and optimization pipelines, where the precision of the judge critically determines the success of downstream applications [ 26 ] . However, as generative AI applications evolve from simple text responses to complex, multi-step tasks across specialized domains, the reliability of LLM-as-a-Judge has become inevitably constrained [ 30 , 54 ] . First, single-pass evaluators are prone to inherent parametric biases—such as favoring verbosity or their own output patterns—which compromise their neutrality when assessing high-complexity responses that deviate from their training distribution [ 50 ] . Second, naive LLM judges are passive observers, unable to react to real-world observations; they assess answers based on linguistic patterns without verification, leading to hallucinated evaluations in specialized domains [ 42 ] . Furthermore, in evaluation tasks that require multifaceted assessment rubrics, traditional LLM judges experience cognitive overload when attempting to evaluate all dimensions comprehensively within a single inference step, which results in coarse-grained scores that fail to reflect specific nuances [ 60 ] . These limitations have catalyzed the transition from LLM-as-a-Judge to Agent-as-a-Judge . As shown in Figure 1 , agentic judges proactively engage in evaluation through multiple capabilities: they decompose complex objectives into subtasks, mitigate biases through multi-agent collaboration [ 4 ] , ground assessments via tool-augmented evidence collection and correctness verification [ 42 ] , and enable fine-grained assessment by persisting intermediate states, autonomously planning the evaluation across reasoning steps [ 62 , 11 ] . This paradigm shift enables more robust, verifiable, and nuanced assessments that effectively address the multifaceted nature of sophisticated AI-generated evaluands. Despite the above potentials and rapid proliferation of agentic evaluation systems, the field lacks a survey to summarize and navigate this shifting landscape. To bridge this gap, we present the first comprehensive survey for Agent-as-a-Judge through the following contributions: • We identify and characterize the shift from LLM-as-a-Judge to Agent-as-a-Judge and summarize the agentic judges’ development trend into three progressive stages (Section 2 ). • We organize core methodologies into five key parts according to agent’s abilities (Section 3 ) and survey their applications across general and professional domains (Section 4 ). • We analyze frontier challenges and identify promising research directions, providing a strategic roadmap for the next generation of robust and verifiable AI judgment. 2 Evolution: From LLM-as-a-Judge to Agent-as-a-Judge {forest} Figure 2 : A taxonomy of Agent-as-a-Judge organizing Methodologies (§ 3 ) and Applications (§ 4 ). Background gradients illustrate the coverage of developmental stages, from Procedural to Reactive and then to Self-Evolving . This section traces the evolution of automated evaluation from LLM-as-a-Judge to Agent-as-a-Judge paradigms. We begin by reviewing the foundational LLM-as-a-Judge and its limitations. We then examine the shift toward Agent-as-a-Judge , analyzing key dimensions that characterize the agentic approach. Finally, we summarize Agent-as-a-Judge ’s development trend into three progressive stages with distinct levels of autonomy and adaptability. 2.1 LLM-as-a-Judge LLM-as-a-Judge paradigm emerged to overcome the scalability limits of human evaluation and the semantic insensitivity of traditional metrics. Zheng et al., [ 63 ] formalized this approach by introducing benchmarks like MT-Bench to assess model alignment. Building on this, G-Eval [ 37 ] leveraged chain-of-thought prompting for better alignment in natural language generation (NLG), while Prometheus [ 23 ] induced fine-grained evaluation in open-source models via specialized tuning. To mitigate systematic issues like position and verbosity bias [ 50 ] , JudgeLM [ 65 ] utilized fine-tuning to develop more robust evaluators. 2.2 From LLM-as-a-Judge to Agent-as-a-Judge As evaluands evolve from simple text responses to complex, multi-step tasks across specialized domains, traditional LLM-as-a-Judge has become increasingly inadequate, focusing on final outputs while failing to verify intermediate actions or satisfy the rigorous standards of professional fields [ 30 , 54 ] . To bridge this gap, the paradigm is shifting toward Agent-as-a-Judge that employs decentralized deliberation, executable verification, and fine-grained assessment to mitigate these limitations. Evolving Robustness: From Monolithic to Decentralized. To mitigate the inherent parametric biases of monolithic LLM judges—such as the tendency to favor verbosity or their own output patterns— Agent-as-a-Judge paradigms employ specialized, decentralized agents that collaborate through autonomous decision-making [ 4 , 6 ] . Crucially, this decentralized architecture facilitates the injection of expert prior knowledge: by decomposing complex evaluation goals into sub-tasks or structuring specific interaction workflows, we can enforce domain-specific constraints that a generalist model typically overlooks [ 55 , 61 ] . Furthermore, multi-agent deliberation ensures collective robustness; distinct roles can isolate specific information points to neutralize bias, while debate and self-reflection allow agents to audit their own cognitive shortcuts, ensuring the final judgment transcends the heuristics of any single model [ 39 , 50 ] . Evolving Verification: From Intuition to Execution. Static LLM judges are fundamentally passive observers, unable to react to real-world feedback. They assess answers based on linguistic plausibility – how correct a response looks – without verification or evidence collection, leading to "hallucinated correctness" in complex tasks [ 42 ] . Agent-as-a-Judge bridges this reality gap by replacing intuition with execution. By interacting with external environments, agentic judges can query system states to verify side effects (e.g., file operations)

Read the complete text