CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering
Jinkun Zhao, Yuanshuai Wang, Xingjian Zhang, Ruibo Chen, Xingchuang Liao, Junle Wang, Lei Huang, Kui Zhang, Wenjun Wu · 2025
Read CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering by Jinkun Zhao, Yuanshuai Wang, Xingjian Zhang, Ruibo Chen, Xingchuang Liao, Junle Wang, Lei Huang, Kui Zhang, Wenjun Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22937.txt.
Introduction
Jinkun Zhao and co-authors route IT operations questions to a panel of specialist language models. Their 2025 paper builds CoE-Ops for AIOps, the use of AI to run large software systems. One model cannot master every operations niche, from parsing logs to diagnosing faults to answering build questions.
A general model first classifies each question, then hands it to the expert best suited, with retrieval support for grounding answers. The split covers both high-level tasks like code and testing and low-level ones like anomaly detection. On a DevOps evaluation set, routing accuracy jumped by a large margin over earlier collaboration schemes. Resolution accuracy beat single models and even larger mixture-of-experts systems. The authors argue teamwork among modest specialists beats one giant generalist for operations work. The framework is domain-shaped but the routing lesson travels.
Catalog record
| CREATOR | Jinkun Zhao, Yuanshuai Wang, Xingjian Zhang, Ruibo Chen, Xingchuang Liao, Junle Wang, Lei Huang, Kui Zhang, Wenjun Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22937
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering Jinkun Zhao, Yuanshuai Wang, Xingjian Zhang, Ruibo Chen, Xingchuang Liao, Junle Wang, Lei Huang, Kui Zhang, Wenjun Wu Abstract With the rapid evolution of artificial intelligence, AIOps has emerged as a prominent paradigm in DevOps. Lots of work has been proposed to improve the performance of different AIOps phases. However, constrained by domain-specific knowledge, a single model can only handle the operation requirement of a specific task,such as log parser,root cause analysis. Meanwhile, combining multiple models can achieve more efficient results, which have been proved in both previous ensemble learning and the recent LLM training domain. Inspired by these works,to address the similar challenges in AIOPS, this paper first proposes a collaboration-of-expert framework(CoE-Ops) incorporating a general-purpose large language model task classifier. A retrieval-augmented generation mechanism is introduced to improve the framework’s capability in handling both Question-Answering tasks with high-level(Code,build,Test,etc.) and low-level(fault analysis,anomaly detection,etc.). Finally, the proposed method is implemented in the AIOps domain, and extensive experiments are conducted on the DevOps-EVAL dataset. Experimental results demonstrate that CoE-Ops achieves a 72% improvement in routing accuracy for high-level AIOps tasks compared to existing CoE methods, delivers up to 8% accuracy enhancement over single AIOps models in DevOps problem resolution, and outperforms larger-scale Mixture-of-Experts (MoE) models by up to 14% in accuracy. I Introduction DevOps is a software engineering methodology designed to bridge the gap between software development (Dev) and IT operations (Ops) [ 2 ] . The comprehensive DevOps lifecycle comprises eight iterative phases: Plan, Code, Build, Test, Deploy, Release, Monitor, and Operation. Each phase operates cyclically and encompasses specific subtask categories, as illustrated in Fig. 1 . With advancements in artificial intelligence (AI) and deep learning, emerging paradigms such as MLOps and AIOps have been proposed, representing two distinct approaches to integrating AI with DevOps. Specifically, AIOps employs machine learning models to optimize DevOps workflows [ 11 ] [ 12 ] [ 13 ] . In recent years, the rapid emergence of large language models has spurred research exploring the integration of LLMs with DevOps. These studies primarily focus on leveraging LLMs to optimize DevOps workflows [ 17 ] . Within AIOps implementations that utilize LLMs for DevOps optimization, a critical challenge lies in selecting appropriate AIOps models for different AIOps tasks [ 1 ] [ 13 ] and enabling expert role-switching capabilities across heterogeneous workflows [ 3 ] [ 11 ] [ 19 ] which is not suitable for multi-agent system leveraging multiple AI agents with specialized and fixed roles [ 52 ] . Although domain-specific LLMs tailored for DevOps have emerged, current solutions face limitations due to their reliance on training data from specific domains [ 12 ] [ 15 ] . This results in inadequate coverage of all DevOps phases and their corresponding subtasks [ 17 ] [ 18 ] , leading to deployment failures in unfamiliar scenarios [ 14 ] and representing a persistent bottleneck for AIOps advancements. Fig. 1: Collaboration Scenarios of CoE-Ops Experts on Question-Answering Tasks at Various Levels within DevOps. CoE-Ops is capable of handling DevOps tasks across the entire life-cycle (high-level) and its sub-tasks (low-level), enabling flexible switching between AIOps experts. Based on the current limitations of AIOps, this paper formulates the following three research questions (RQs): • RQ1 (Effectiveness): Can LLM ensembling mitigate the competency gaps between different LLMs? • RQ2 (Scalability): How can LLM ensembling apply for multitask learning in the AIOps domain? • RQ3 (Efficiency): Does the integration of smaller models via LLM ensembling enable performance that surpasses that of larger models? To address the current challenges in AIOps regarding model selection [ 13 ] , role-switching [ 19 ] , and scalability [ 16 ] , we propose the following solutions. First, we enhanced the existing Collaboration-of-Experts framework based on a two-stage expert routing mechanism [ 48 ] [ 49 ] . Subsequently, we integrated the ensemble learning concept and the refined framework into DevOps workflows, leveraging their inherent model-task scalability to enable dynamic selection, composition, and role-switching of AIOps experts (as illustrated in Fig. 1 ). Finally, to address high-level DevOps tasks, we incorporated the task classifier with retrieval-augmented generation. This combination enhances task classification by retrieving relevant contextual knowledge for target problems and integrating it into the prompt, thereby improving the framework’s adaptability to complex operational scenarios. Our key contributions are summarized as follows: • A Collaboration-of-Expert framework CoE-Ops based on two-stage expert routing and a general-purpose large language model as task classifier, enabling dynamic switching across diverse AIOps task domains and LLM ensembles. • An enhanced task classifier empowered by retrieval-augmented generation technology, specifically designed to address high-level task representations inherent in DevOps scenarios. • Comprehensive empirical validation on DevOps-EVAL benchmarks with multiple task-expert configurations and over a dozen AIOps expert models, systematically validating CoE-Ops’s dual scalability in task scalability and model scalability. II Related Work II-A Development and Operations DevOps is a collaborative, cross-domain software development methodology that emphasizes the automation of continuous delivery for software updates [ 3 ] . When integrated with artificial intelligence, its evolutionary trajectory bifurcates into two primary branches: MLOps and AIOps [ 16 ] . MLOps MLOps focuses on applying DevOps practices to machine learning systems, aiming to establish seamless integration between diverse open source tools to enable fully automated execution of ML workflows, spanning dataset construction, model training, and deployment [ 4 ] [ 5 ] . With the recent emergence of large language models, LLMOps [ 7 ] , an extension of MLOps tailored for LLM development and deployment, have gained momentum. LLMOps addresses the unique operational challenges of LLMs [ 8 ] and provides specialized tools for efficient data processing, model training, deployment, and maintenance [ 10 ] . However, both MLOps and LLMOps currently face limitations, including a lack of standardized practices, difficulties in maintaining model consistency and scalability [ 6 ] [ 9 ] , and ambiguous evaluation criteria. AIOps In contrast, AIOps leverages AI and ML technologies to efficiently build and operate large-scale online services and applications in software engineering [ 11 ] . Most existing AIOps implementations rely on data from a limited number of domains [ 12 ] and predominantly employ supervised learning techniques [ 15 ] . Consequently, their proposed models are often confined to specific DevOps subdomains rather than being deployable across the entire ecosystem [ 17 ] . A critical challenge for AIOps lies in selecting and integrating appropriate machine learning models [ 13 ] [ 19 ] to ensure adaptability to diverse use cases while fulfilling heterogeneous [ 18 ] and evolving requirements [ 16 ] . II-B Ensemble Learning with Large Language Models Ensemble learning with large language models involves the systematic utilization of multiple LLMs, each designed to handle user queries during downstream inference to capitalize on their individual strengths [ 20 ] [ 21 ] . Depending on the strategy for model integration, ensemble learning can be categorized into two paradigms: Mixture-of-Experts (MoE) and Collaboration-of-Experts (CoE). Mixture-of-Experts In recent years, MoE models have become a primary choice for foundation models [ 50 ] [ 51 ] due to their computational efficiency and strong generalization capabilities. In MoE systems, different expert modules possess distinct strengths, making efficient