Benchmarking LLM-Driven Network Configuration Repair DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; ; ISBN: 978-1-4503-XXXX-X/2018/06
Ioannis Protogeros, Rufat Asadli, Benjamin Hoffman, Laurent Vanbever · 2026
Read Benchmarking LLM-Driven Network Configuration Repair DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; ; ISBN: 978-1-4503-XXXX-X/2018/06 by Ioannis Protogeros, Rufat Asadli, Benjamin Hoffman, Laurent Vanbever (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22513.txt.
Introduction
Protogeros and three colleagues present Cornetto, the first benchmark grading language-model repair of network configurations by function and at scale. Fixing large, linked configurations is risky because a repair can break other parts. Cornetto pairs a generator that synthesizes plausible misconfiguration cases with formal verification checking each proposed fix against ground-truth specs. The released set holds 231 problems across networks of 20 to 754 nodes and varied protocols. Testing 9 leading models, the authors find promise but frequent regressions and fading performance as networks grow.
Catalog record
| CREATOR | Ioannis Protogeros, Rufat Asadli, Benjamin Hoffman, Laurent Vanbever |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61134 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22513
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking LLM-Driven Network Configuration Repair DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; ; ISBN: 978-1-4503-XXXX-X/2018/06 Ioannis Protogeros, Rufat Asadli, Benjamin Hoffman, Laurent Vanbever Abstract . There is a rapidly growing interest in using Large Language Models (LLMs) to automate complex network operations, but their reliable adoption requires rigorous assessment of their effectiveness and safety. Existing benchmarks do not address whether LLMs can successfully resolve errors in large-scale, interdependent network configurations without introducing new disruptions. Developing such a benchmark is challenging: scenarios must be diverse and increasingly complex, yet their evaluation must be straightforward and meaningful. In this paper, we present Cornetto , the first benchmark to evaluate LLM-driven network configuration repair functionally and at scale . Cornetto features a generation pipeline that synthesizes representative and plausible misconfiguration scenarios, coupled with an evaluation framework that uses formal verification to assess functional correctness of proposed fixes against ground-truth specifications. Using this pipeline, we synthesize a dataset of 231 problems for fixing configurations across varying network topologies (20–754 nodes) and diverse protocols. We evaluate 9 state-of-the-art LLMs and find that while they show promise, they often introduce regressions and their performance degrades at scale. Our results indicate that reliable LLM-powered network automation requires integrating LLMs into iterative workflows guided by formal verification. 1. Introduction Network correctness, while paramount, remains extremely difficult to achieve and maintain. While network verification and synthesis ( 12 ; 3 ; 23 ; 11 ; 5 ) have made significant strides towards eliminating human-induced misconfigurations, they are not a silver bullet. In particular, their adoption is hindered by limited protocol coverage and inaccuracies in modelling complex network behaviour ( 8 ; 25 ) . More recently, there has been a surge of interest in leveraging Large Language Models (LLMs) as a more flexible approach to automating network operations. Hyperscalers have already begun deploying LLM-based frameworks that assist with such tasks, including ByteDance’s NetAssistant ( 38 ) , Alibaba’s BiAn ( 37 ) , and Meta’s Confucius ( 39 ) . On the one hand, with their impressive capabilities across domains ( 15 ; 32 ) , LLMs appear promising for facilitating complex workflows that are bottlenecked by human reasoning. On the other hand, as probabilistic models, they remain prone to errors and hallucinations ( 7 ; 20 ) , precluding their adoption for managing critical infrastructure. The tension between the tremendous potential for automating network operations and their associated risks makes principled evaluation indispensable. Doing so requires designing a benchmark that challenges LLM reasoning with complex and diverse configuration tasks, alongside with a proper methodology to assess their capabilities on said tasks. Constructing such a diverse benchmark with plausible, well-posed problems is nontrivial. Unlike domains such as mathematics or software engineering, where benchmarks can leverage vast publicly available datasets, network configurations are proprietary and sensitive. Consequently, we must synthesize the dataset. Crucially, this synthesis cannot be random; to ensure relevance, test cases must reflect the complexities that operators face in production networks, including large-scale configurations with feature and protocol dependencies. Lastly, the evaluation must verify the functional correctness of solutions, scalably and automatically . The gap in LLM network configuration benchmarking. LLMs’ progress outpaces our abilities to evaluate them effectively in fields that require deep domain expertise and complex reasoning. Despite growing interest in using LLMs for network operations, we still lack benchmarks to evaluate their performance and explore strategies to improve them. While previous works such as NetConfEval ( 36 ) and NetLLMBench ( 1 ) have established baselines for evaluating LLMs on network configuration tasks, they are severely constrained in scale and complexity relative to the capabilities of current models. Additionally, their proposed evaluation methods rely on proxy metrics (e.g., textual similarity or ping-test validation) that do not guarantee the functional correctness of a configuration. A more recent benchmark, NIKA ( 40 ) , evaluates the diagnostic capabilities of LLM Agents in dynamic, emulated network environments. Yet, it does not support the evaluation of proposed fixes for network faults. Consequently, evaluating the performance of LLMs in repairing realistic, large-scale network configurations correctly and safely remains underexplored. Cornetto : Correct & safe configuration repair. To address this gap, we introduce Cornetto , a benchmark that evaluates end-to-end configuration repair in representative, large-scale networks. 1 1 1 The idea behind the system was presented in a previously-accepted poster ( 33 ) . The task of resolving misconfigurations encapsulates critical challenges in network operations: understanding the interplay among interdependent features and protocols, and bridging the semantic gap between low-level configurations and high-level intent ( 5 ) . To capture this complexity, Cornetto employs a generation pipeline that produces syntactically valid configurations subject to logical and semantic constraints, thereby ensuring structural coherence and consistency. We use this pipeline to synthesize 231 challenging misconfiguration scenarios spanning diverse protocols and scales, where the intended network state is unambiguously defined. To evaluate correctness and safety, Cornetto formally verifies the data plane of the reconfigured network against ground-truth specifications. This ensures that success depends on functional correctness , requiring restoration of intended behaviour. The introduction of new bugs is penalised based on the extent of disruption to previously functional behaviour. Crucially, we also evaluate the diagnostic reasoning (localization and root-cause analysis) that led to the fix. Key findings. Our evaluation of 9 LLMs reveals that while current models can diagnose and fix faults (restoring up to 60% of network state on average), they cannot reliably act as monolithic solvers—the best-performing model successfully resolved only 25% of scenarios. We find that performance degrades in large-scale settings with noisy data, and models often propose partial or unsafe solutions. These insights indicate that reliable automation requires integrating LLMs into systems that filter noisy data, preserve necessary context, and iteratively verify the safety of repairs before application. Key contributions. Our summarized contributions are: • We formulate the problem of automated configuration repair, enabling the quantification of functional disruption caused by misconfigurations and the assessment of fix correctness and safety. • We develop a scenario generation pipeline that synthesizes logically valid network configurations for any topology and systematically injects faults across diverse protocols. • Using this pipeline, we curate a dataset of 231 misconfiguration scenarios of varying scale and complexity by optimizing fault diversity within a minimal number of scenarios. • We design a verification-based evaluation pipeline that utilizes data-plane analysis to automatically infer ground-truth specifications and assess the functional correctness and safety of fixes. • We evaluate 9 state-of-the-art LLMs on Cornetto and comprehensively analyze their performance, examining the effects of scenario complexity on reconfiguration correctness and safety. Outlook. We will open source Cornetto to the community as an extensible and modular framework. 2 2 2 available at https://github.com/nsg-ethz/cornetto Cornetto enables the generation of more challenging tasks, and its architecture supports the evaluation of any LLM-based system, including advanced scaffolds such as Retrieval-Augmented Generation (RAG) systems and agentic setups. By providing a standardized testbed, Cornetto contributes to the continual evaluation of LLMs on configuration repair. 2. Overview Cornetto is a benchmark for assessing LLMs’ capabilities in automated network configuration repair. A Cornetto scenario simulates an end-to-end troubleshooting task: Given a misconfigured network state (including topology and configuration files) and a