A Network Arena for Benchmarking AI Agents on Network Troubleshooting #284

Zhihao Wang, Alessandro Cornacchia, Alessio Sacco, Franco Galante, Marco Canini, Dingde Jiang · 2025

Read A Network Arena for Benchmarking AI Agents on Network Troubleshooting #284 by Zhihao Wang, Alessandro Cornacchia, Alessio Sacco, Franco Galante, Marco Canini, Dingde Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16381.txt.

Introduction

Zhihao Wang and five co-authors build a test arena for network AI in research shared on arXiv in 2025. Language model agents increasingly help engineers configure networks and troubleshoot outages. Progress is slow because there is no standard, low-effort benchmark for judging agents on realistic, changing network failures.

Their benchmark, NIKA, is the largest public one for network incident diagnosis. It replays real-world scenarios with no setup effort and gives agents a clean interface for quick prototyping. The suite holds hundreds of curated incidents across five settings, from data centers to internet service providers, covering 54 representative issues. Trial runs showed bigger models detect problems more often but still struggle to localize faults and name root causes. NIKA is open-source and modular, so new scenarios and failure cases can be added.

Catalog record

CREATORZhihao Wang, Alessandro Cornacchia, Alessio Sacco, Franco Galante, Marco Canini, Dingde Jiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16381

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Network Arena for Benchmarking AI Agents on Network Troubleshooting #284 Zhihao Wang, Alessandro Cornacchia, Alessio Sacco, Franco Galante, Marco Canini, Dingde Jiang Abstract . Agentic systems, powered by Large Language Models (LLMs), assist network engineers with network configuration synthesis and network troubleshooting tasks. For network troubleshooting, progress is hindered by the absence of standardized and accessible benchmarks for evaluating LLM agents in dynamic network settings at low operational effort. We present NIKA , the largest public benchmark to date for LLM-driven network incident diagnosis and troubleshooting. NIKA targets both domain experts and especially AI researchers alike, providing zero-effort replay of real-world network scenarios, and establishing well-defined agent-network interfaces for quick agent prototyping. NIKA comprises hundreds of curated network incidents, spanning five network scenarios, from data centers to ISP networks, and covers 54 representative network issues. Lastly, NIKA is modular and extensible by design, offering APIs to facilitate the integration of new network scenarios and failure cases. We evaluate state-of-the-art LLM agents on NIKA and find that while larger models succeed more often in detecting network issues, they still struggle to localize faults and identify root causes. NIKA is open-source and available to the community: https://github.com/sands-lab/nika . 1. INTRODUCTION Artificial Intelligence (AI) is shifting from human-oriented chat applications to autonomous agentic systems with decision-making capabilities. Google has projected a 1 trillion USD market value for agentic AI by 2040 ( Cloud 2025 ) , with Large Language Models (LLMs) at the core of this transformation. This evolution has accelerated the adoption of LLMs in domains requiring dynamic, context-aware operations, such as networking ( Wu et al. 2024 ; Wang et al. 2025c ; Wang et al. 2025a ; Wang et al. 2024a ; Angi et al. 2025 ; Wang et al. 2024c ; Mekrache et al. 2024 ; Lira et al. 2025 ; Wang et al. 2025b ; Donadel et al. 2024 ) and systems ( Roy et al. 2024 ; Chen et al. 2024b ; Yu et al. 2024 ; Zhang et al. 2024 ; Goel et al. 2024 ; Shetty et al. 2024 ; Cornacchia et al. 2025a ; Wang et al. 2024b ) . Network troubleshooting is a compelling target: given a network incident, engineers undertake mechanical yet cumbersome steps to identify and remediate issues, from probing the network to explain packet drops, to refining detection logic across heterogeneous telemetry and diagnosis tools ( Fayaz et al. 2016 ; Arzani et al. 2018 ) . The process is manual, slow, and prone to errors, requiring expert operators to reason across multiple dimensions. Recent work shows that cloud operators and telco vendors have already deployed AI troubleshooting agents for production networks ( Wang et al. 2025c ; Wang et al. 2025a ; Wang et al. 2024a ; Alan 2025 ) , leveraging LLMs to parse textual data from monitoring tools and assist operators in failure detection and localization. Despite the broad consensus, our community still lacks the benchmarks and platforms needed to explore and compare agent designs in a principled way. For example, existing systems such as NetAssistant ( Wang et al. 2024a ) , BiAn ( Wang et al. 2025c ) , and Confucius ( Wang et al. 2025a ) are evaluated in closed settings with no publicly available datasets and limited disclosure of evaluation scenarios. NetConfEval ( Wang et al. 2024c ) instead provides a public benchmark but focuses on static configuration synthesis, making it impossible for live network troubleshooting or studying how well agents sequence decision-making. Additionally, it cannot capture important dimensions that extend beyond accuracy, such as time to resolution, reasoning quality, or the impact on network performance during diagnosis, which are equally critical in troubleshooting tasks. We can observe two gaps in the existing solutions. First, designers of AI agents need to evaluate a rich design space that includes, among others: agent structure , e.g., monolithic versus specialized sub-agents ( Li et al. 2023 ; Zhuge et al. 2024 ; Pan et al. 2025 ) ; prompting strategies , e.g., how to structure the agent’s reasoning process ( Wei et al. 2022 ; Wang et al. 2023 ; Madaan et al. 2023 ; LangChain 2025a ) ; tool interfaces , e.g., granularity and abstraction of external APIs ( Patil et al. 2024 ; Basu et al. 2025 ) ; state management , e.g., which historical data to keep in the context at each reasoning step ( Xu et al. 2025a ; Lee et al. 2024 ) . Because AI programmers do not necessarily possess domain knowledge, they expect a ready-to-use benchmark to evaluate the impact of their design choices on agent behaviors at low operational effort. Second, experimenting with network troubleshooting scenarios requires a controlled network environment in which to connect AI agents and reproduce failures without disrupting normal user traffic and network operations. Production networks are unsuitable due to safety, privacy, and reproducibility constraints. Existing emulators provide controllable networks, but fall short of offering: (i) streamlined reproducibility of given incident scenarios; (ii) a unified interface to connect AI agents with the network environment; (iii) scoring mechanisms for task objectives ( e.g., localization accuracy) and reasoning-level evaluations. To fill these gaps, we present NIKA , a N etwork I ncidents benchmar K ing FrameworK for A I Agents. NIKA is a unified platform that can offer: (1) a benchmark suite of curated network incidents that covers 54 realistic network issues, ranging from link and host failures to resource contention, and includes five network scenarios, four of which can be instantiated at different topology sizes, spanning campus and data center networks. By combining these dimensions, the benchmark yields 640 distinct troubleshooting incidents for evaluating AI agents. The benchmark can be further extended by randomizing failure locations and composing multiple issues within a single incident. (2) A modular plug-and-play orchestration platform that connects AI agents with the network environment, enabling real-time troubleshooting in realistic conditions, and providing a human-facing interface to monitor agent performance. NIKA orchestrates behind the scenes the components required for traffic generation, failure injection, integration with specialized telemetry instrumentation and diagnosis tools, observability, performance evaluation, and reasoning trajectory analysis, leaving AI programmers to focus only on agent design and evaluation. Through the Model Context Protocol (MCP) ( Anthropic 2025 ) , the framework exposes more than 30 monitoring and troubleshooting tools to third-party agents, ranging from sketches and INT ( P4.org 2020 ) to SDN controller APIs and switch CLIs. Thus, we used NIKA to evaluate three popular LLMs, namely GPT-OSS, GPT-5, and GPT-5-mini, and analyzed how these models behave when troubleshooting various network failures. We have found that, for example, although larger models are more successful in detecting network issues, they still struggle to localize faults and identify root causes. Contributions. In summary, this paper makes the following contributions: • we formalize the problem of benchmarking AI agents for network troubleshooting, identifying key requirements and challenges; • we design a structured methodology to benchmark AI agents in this context and implement a modular platform that orchestrates the end-to-end evaluation workflow and outputs detailed observability and performance reports; • we evaluate three state-of-the-art LLMs to validate the benchmark rigor, summarizing the problems detected with NIKA and share our experience in dealing with

Read the complete text