SAGE: A Service Agent Graph-guided Evaluation Benchmark DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Multi-agent planning CCS: Computing methodologies Discourse, dialogue and pragmatics
Ling Shi, Yuqin Dai, Ziyin Wang, Ning Gao, Wei Zhang, Chaozheng Wang, Yujie Wang, Wei He, Jinpeng Wang, Deyi Xiong · 2026
Read SAGE: A Service Agent Graph-guided Evaluation Benchmark DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Multi-agent planning CCS: Computing methodologies Discourse, dialogue and pragmatics by Ling Shi, Yuqin Dai, Ziyin Wang, Ning Gao, Wei Zhang, Chaozheng Wang, Yujie Wang, Wei He, Jinpeng Wang, Deyi Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09285.txt.
Introduction
Shi and colleagues propose SAGE, a multi-agent benchmark for service agents that must follow strict operating procedures. Existing benchmarks use static setups and single metrics, missing diverse user behavior and procedure compliance. SAGE turns procedures into dynamic dialogue graphs, enabling precise checks of compliance and path coverage, plus an adversarial intent taxonomy for low-cost use across domains. Tests on 27 language models across 6 scenarios reveal an execution gap: models classify intents well but fail to derive correct next actions.
Catalog record
| CREATOR | Ling Shi, Yuqin Dai, Ziyin Wang, Ning Gao, Wei Zhang, Chaozheng Wang, Yujie Wang, Wei He, Jinpeng Wang, Deyi Xiong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70107 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.09285
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAGE: A Service Agent Graph-guided Evaluation Benchmark DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Multi-agent planning CCS: Computing methodologies Discourse, dialogue and pragmatics Ling Shi, Yuqin Dai, Ziyin Wang, Ning Gao, Wei Zhang, Chaozheng Wang, Yujie Wang, Wei He, Jinpeng Wang, Deyi Xiong Abstract . The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominantly rely on static paradigms and single-dimensional metrics, failing to account for diverse user behaviors or the strict adherence to structured Standard Operating Procedures (SOPs) required in real-world deployments. To bridge this gap, we propose SAGE ( S ervice A gent G raph-guided E valuation), a universal multi-agent benchmark for automated, dual-axis assessment. SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance and comprehensive path coverage. We introduce an Adversarial Intent Taxonomy and a modular Extension Mechanism, enabling low-cost deployment across domains and facilitating automated dialogue data synthesis. Evaluation is conducted via a framework where Judge Agents and a Rule Engine analyze interactions between User and Service Agents to generate deterministic ground truth. Extensive experiments on 27 LLMs across 6 industrial scenarios reveal a significant “Execution Gap” where models accurately classify intents but fail to derive correct subsequent actions. We also observe “Empathy Resilience”, a phenomenon where models maintain polite conversational facades despite underlying logical failures under high adversarial intensity. Code and resources are available at https://anonymous.4open.science/r/SAGE-Bench-4CD3/ . 1. Introduction Figure 1 . Service Agent SOP Example (Telecom Scenario). The rapid advancement of Large Language Models (LLMs) has significantly accelerated the automation process across various industrial sectors ( 2 ; 42 ; 46 ) . Among these, the domain of intelligent customer service has emerged as a pioneer in adopting these technologies to enhance operational efficiency and user experience ( 7 ; 28 ; 43 ) . By leveraging the robust generative capabilities and context understanding of LLMs, enterprises are striving to transition from traditional, rigid rule-based systems to highly adaptive intelligent service agents ( 47 ; 44 ) capable of handling complex interactions ( 53 ; 36 ) , thereby addressing a broader range of customer needs while reducing labor costs. Consequently, benchmarking models on their strict adherence to Standard Operating Procedures (SOPs) for correct workflow transitions, a process requiring the generation of structured responses that align with predefined logic as shown in Figure 1 , has become a priority for enterprises. However, existing service benchmarks face three fundamental limitations. First, insufficient evaluation dimensions : Current benchmarks typically assess either task completion ( 32 ; 20 ; 25 ) or dialogue quality ( 59 ; 41 ) in isolation. Real-world scenarios, however, demand both strict logical compliance with SOPs and appropriate communication skills . This metric insufficiency causes evaluation bias and hinders precise error localization. Second, static interaction paradigms : Relying on fixed datasets like scripts ( 5 ; 34 ; 58 ) , traditional methods fail to test error recovery or cover diverse user behaviors, ranging from cooperative inquiries to adversarial conflicts. Consequently, these static evaluations are incomplete and struggle to reflect real-world performance. Finally, limited scalability : Benchmarks often depend on costly manual annotation ( 33 ; 22 ) and are frequently over-fitted to the SOPs of a single domain, making it prohibitively expensive to adapt to diverse, multi-branch business scenarios. To address these challenges, we propose SAGE ( S ervice A gent G raph-guided E valuation). SAGE integrates three core contributions: (1) Dynamic Multi-Turn Dialogue Graph Modeling , which formalizes SOPs into directed graphs (as shown in Figure 1 ) to enable dynamic verification of logical compliance and ensure comprehensive path coverage; (2) a Multi-Agent multi-dimensional Evaluation utilizes Judge Agents and a Rule Engine to analyze the interaction between User and Service Agents, generating deterministic ground truth for a rigorous, multi-dimensional assessment of logical compliance and chat quality; and (3) a Scenario Extension Mechanism , which enabled the rapid deployment of six industrial scenarios in our study, demonstrating its practical feasibility. To validate SAGE, we evaluated 27 mainstream LLMs across 6 industrial scenarios. Our experiments reveal: (1) The gap between open-source and closed-source models is narrowing, with DeepSeek-V3.2 surpassing several GPT-4 class models; (2) A significant “Execution Gap” in complex scenarios, where high classification accuracy does not guarantee correct action execution, highlighting the challenge of procedural reasoning; (3) Performance degradation in multi-turn dialogues due to context fatigue; and (4) “Empathy Resilience” under high adversarial intensity, where models maintain polite conversational facades despite underlying logical failures. In summary, our contributions are: • We propose SAGE , the first graph-guided multi-agent evaluation benchmark that transforms unstructured SOPs into directed graphs to enable automated, dual-axis assessment of logical compliance and conversational quality. • We introduce dynamic graph modeling and an Adversarial Intent Taxonomy , bridging the gap between static testing and dynamic reality through diverse user behavior simulations. • We uncover critical phenomena such as the “Execution Gap” and “Empathy Resilience” , providing granular diagnostics for agentic capabilities, through extensive experiments on 27 LLMs across 6 distinct scenarios. • We design a modular Extension Mechanism for low-cost adaptation to new scenarios, which also facilitates the automated synthesis of large-scale dialogue datasets for customer service. 2. Related Works 2.1. Evaluation for Large Language Models Large language model evaluation has evolved from basic instruction-following ( 60 ) to complex multi-constraint scenarios. Early benchmarks assessed format compliance ( 48 ; 38 ) , while recent work evaluates real-world complexity through Multi-IF ( 15 ) , FollowBench ( 16 ) , and InfoBench ( 31 ) . Guidebench ( 8 ) introduces domain-specific conditions, and Collie ( 52 ) systematically constructs constrained generation tasks. Domain-specific evaluation spans business process management ( 4 ; 12 ; 18 ; 13 ; 35 ; 17 ; 14 ) , finance ( 49 ; 50 ) , and procedural compliance. SOPBench ( 22 ) and SOP-Bench ( 27 ) evaluate tool-calling sequences but primarily focus on external manipulation rather than deep logical reasoning. Broader agent frameworks include AgentBench ( 25 ) , WebArena ( 61 ) , and WorkArena ( 9 ) . Complementing mathematical tasks, textual logical reasoning, and reading comprehension are assessed through LogiQA ( 24 ) for deductive reasoning, DROP ( 10 ) for discrete reasoning over paragraphs, and HotpotQA ( 51 ) for multi-hop reasoning, which are more closely aligned with the context understanding required in service scenarios. Agent training advances include AGILE ( 30 ) , ReAct ( 53 ) , Reflexion ( 36 ) , Agent-Pro ( 57 ) , AgentTuning ( 54 ) , Self-Refine ( 26 ) , Self-Instruct ( 45 ) , and human feedback training ( 29 ) . Unlike prior benchmarks limited by static datasets and single-dimensional metrics, SAGE introduces a graph-guided multi-agent framework to dynamically evaluate both the logical compliance and conversational quality of service agents. 2.2. Benchmarks for Service Dialogue Systems Service dialogue evaluation addresses customer-facing applications through DialogBench ( 28 ) for human-like conversation, ECom-Bench ( 43 ) for customer support resolution. Multi-turn complexity is captured by MG-ShopDial ( 3 )