Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents
Divyanshu Saxena 1 , Rishikesh Maurya 2 , Xiaoxuan Ou 2 , Gagan Somashekar 2 , Shachee Mishra Gupta 2 Arun Iyer 2 , Yu Kang 2 , Chetan Bansal 2 , Aditya Akella 1 , Saravan Rajmohan 2 1 The University of Texas at Austin, 2 Microsoft · 2025
Read Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents by Divyanshu Saxena 1 , Rishikesh Maurya 2 , Xiaoxuan Ou 2 , Gagan Somashekar 2 , Shachee Mishra Gupta 2 Arun Iyer 2 , Yu Kang 2 , Chetan Bansal 2 , Aditya Akella 1 , Saravan Rajmohan 2 1 The University of Texas at Austin, 2 Microsoft (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10049.txt.
Introduction
Divyanshu Saxena and nine co-authors argue that benchmarks for enterprise AI agents must be generated continuously rather than fixed once. Fixed test sets suit simple coding tasks, but enterprise agents face evolving services, shifting requirements, and sparse ground truth. Their process builds fresh benchmarks as requirements change, starting from semi-structured documents where developers state high-level intent for a migration or service.
State-of-the-art language models turn a small number of such documents into evaluation cases. The authors demonstrate the approach on a real case: migrating a service between deployment platforms at a large public enterprise. The result is a maintainable evaluation framework that gives rapid feedback on agent performance and points improvements at the right targets as the underlying platform keeps moving.
Catalog record
| CREATOR | Divyanshu Saxena 1 , Rishikesh Maurya 2 , Xiaoxuan Ou 2 , Gagan Somashekar 2 , Shachee Mishra Gupta 2 Arun Iyer 2 , Yu Kang 2 , Chetan Bansal 2 , Aditya Akella 1 , Saravan Rajmohan 2 1 The University of Texas at Austin, 2 Microsoft |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10049
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents Divyanshu Saxena 1 , Rishikesh Maurya 2 , Xiaoxuan Ou 2 , Gagan Somashekar 2 , Shachee Mishra Gupta 2 Arun Iyer 2 , Yu Kang 2 , Chetan Bansal 2 , Aditya Akella 1 , Saravan Rajmohan 2 1 The University of Texas at Austin, 2 Microsoft Abstract . The rapid adoption of AI agents across domains has made systematic evaluation crucial for ensuring their usefulness and successful production deployment. Evaluation of AI agents typically involves using a fixed set of benchmarks and computing multiple evaluation metrics for the agent. While sufficient for simple coding tasks, these benchmarks fall short for enterprise-scale agents, where services and requirements evolve continuously and ground-truth examples are sparse. We propose a process of benchmark generation that helps evolve the benchmarks as the requirements change and perform robust evaluation of evolving AI agents. We instantiate this approach for a case study of service migration from one deployment platform to another at a large public enterprise. Our approach relies on semi-structured documents where developers express the high-level intent, and uses state-of-the-art LLMs to generate benchmarks from just a small number of such documents. Overall, this process results in a maintainable evaluation framework, enabling rapid feedback on agent performance and facilitating targeted improvements. 1. Introduction Modern enterprises increasingly rely on large language model (LLM)–based agents to automate software engineering and operational workflows. Examples include build and CI/CD agents ( Xu et al. 2025 ) that adapt build configurations and monitor CI/CD pipelines for issues, incident management and triaging agents ( Wang et al. 2024 ; Yu et al. 2025 ) that route incidents to the appropriate service owners, and troubleshooting agents ( Lou et al. 2025 ) that help reproduce failures and synthesize fixes. These enterprise-scale agents operate across thousands of services and heterogeneous artifacts such as source code, deployment manifests, configuration files, and documentation. Their outputs directly affect production environments—incorrect actions can impact operations, ranging from user dissatisfaction and degraded service quality to complete unavailability of critical services. While several evaluation mechanisms and benchmarks exist for general-purpose coding agents designed for isolated and generic tasks (e.g., function synthesis or bug fixing), enterprise-scale LLM agents pose fundamentally different evaluation requirements. They must reliably produce high-quality outputs, even as the task requirements change. Moreover, these agents themselves evolve – operators continually integrate newer model versions and reasoning capabilities—making evaluation an ongoing necessity rather than a one-time exercise. Existing benchmarks, however, are ill-suited to this setting. Most are static, targeting a fixed and narrow set of tasks. When requirements change, they must be manually revised, often resulting in numerous variants (e.g., SWE-bench ( Jimenez et al. 2024 ) and its extensions ( swe 2024 ; Xia et al. 2025 ; OpenAI 2024 ) ). While useful for reproducibility and enabling transparent evaluation, they fail to capture the continuous evolution typical of enterprise environments. Their rigidity makes longitudinal evaluation difficult and imposes a recurring maintenance burden. We identify two key challenges in evaluating enterprise-scale agents. First, these agents operate under changing dynamics, e.g., deployment platforms, operational policies, or troubleshooting guides may change frequently. An agent that performs well today may degrade as dependencies evolve or organizational requirements shift. Second, the heterogeneity of context and artifacts makes defining ground truth difficult. Conventional benchmarks typically rely on well-defined, static reference outputs, but enterprise-scale tasks span diverse file types, programming languages, and infrastructure components – all of which evolve independently, e.g. for feature additions and bug fixes. Consequently, determining what constitutes the correct or intended outcome for a given task is non-trivial, often requiring significant effort to carefully extract the right ground truths and thus, imposing a recurring maintenance burden. We propose a new approach to evaluating enterprise-scale agents. Instead of relying on fixed, static benchmarks, we introduce a continuous benchmark generation pipeline that evolves alongside changing requirements without imposing significant developer burden on creating these benchmarks. In this paper, we ground the discussion for one such use case from our experience at a major cloud provider Microsoft (anonymized for review), where services were migrated to a new infrastructure platform. File Type Reason Example Script files File path formats differ across operating systems Change directory paths from forward slash (“/”) in Linux to back slash (“\”) in Windows Source and build files Change in supported libraries ‘System.Drawing.Common’ is a .NET library that provides access to Windows Graphics Device Interface. SDC does not exist on Linux systems and must be updated to cross-platform libraries like ImageSharp or SkiaSharp Dependency configuration files Different installation or deployment setups across OSes Jaeger, a popular monitoring library needs to be deployed as a Windows service on Windows, but as a container on Linux/Kubernetes Service orchestration files Changes in requirements from the platform management teams Modify or add new Dockerfile s for successful deployment. Table 1. Examples of file-level changes required during service migration. Motivating Use Case: Service Migration Agents. Enterprise services are typically deployed on shared infrastructure platforms, which themselves evolve over time as underlying components are modified or replaced. We observed one such instance over the past six months in which existing services were transitioned to new infrastructure platform. Such platform-level changes are often intrusive, requiring the dependent services to undergo corresponding migrations. For example, when the underlying operating system changes (e.g., transitioning between Windows and Linux Operating Systems), build and compilation scripts must be updated to reflect the new environment. Similarly, if the service relies on OS-specific libraries, such as the System Drawing package (see example in Table 1 ) that is only functional for Windows, the source code must be updated to use cross-platform libraries (e.g., ImageSharp). As is the case for enterprise-scale agents, this service migration task spanned several heterogeneous artifacts, including build and compilation scripts, deployment files, and even source code. We discuss examples of these changes in more detail in Section 2 . Our Proposal: Continuous Benchmark Generation. Our vision is to develop a methodology that allows benchmarks to be evolved as operational and organizational requirements change, and enables thorough evaluation of agents over a diverse set of tasks and services. We achieve this via two key components. The first of these are developer-authored semi-structured Knowledge Bases (KBs) . The term KB is generally used to denote resources used by LLM agents ( Gupta et al. 2024 ; Wu et al. 2024 ; Wang et al. 2025 ) , but in our context, it describes the specifics of the evaluation task at hand, such as the goal, context, and expected outcomes for the task. We envision that a task (e.g., service migration) can be specified as a suite of KB documents – one document per sub-task (e.g., update the logging library in the service code). Crucially, any updates to the requirements can be captured simply by editing an existing KB document or adding a new one. Moreover, by developing distinct documents for heterogeneous artifacts observed in enterprise-scale services, we can get fine-grained metrics about agent performance. The second component in our proposal are reference implementations derived from a small number of real commits that serve as “gold-standard” examples. By linking these examples to the corresponding KB documents, we can construct the ground truths