LogPilot: Intent-aware and Scalable Alert Diagnosis for Large-scale Online Service Systems

Zhihan Jiang ¶ , Jinyang Liu ‡ , Yichen Li ‡ , Haiyu Huang ¶ , Xiao He ‡ , Tieying Zhang ‡∗ , Jianjun Chen ‡ , Yi Li ‡ , Rui Shi ‡ , Michael R. Lyu ¶ · 2025

Read LogPilot: Intent-aware and Scalable Alert Diagnosis for Large-scale Online Service Systems by Zhihan Jiang ¶ , Jinyang Liu ‡ , Yichen Li ‡ , Haiyu Huang ¶ , Xiao He ‡ , Tieying Zhang ‡∗ , Jianjun Chen ‡ , Yi Li ‡ , Rui Shi ‡ , Michael R. Lyu ¶ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25874.txt.

Introduction

Jiang and nine co-authors present LogPilot, a 2025 system that uses large language models to diagnose alerts in giant online services. On-call engineers drown in logs when an alarm fires. Existing tools scope logs without regard to the alert's own logic, and reason poorly over messy data. LogPilot reads the alert definition itself to find causally related logs and requests.

It then rebuilds each request as a chain of log events, clusters similar chains, and shows the model a few representative samples. On real alerts from Volcano Engine Cloud, root-cause summaries grew about 50 percent more useful and exact localization about 55 percent more accurate than leading methods. Diagnosis took under a minute and cost about seven cents per alert. The system runs in production.

Catalog record

CREATORZhihan Jiang ¶ , Jinyang Liu ‡ , Yichen Li ‡ , Haiyu Huang ¶ , Xiao He ‡ , Tieying Zhang ‡∗ , Jianjun Chen ‡ , Yi Li ‡ , Rui Shi ‡ , Michael R. Lyu ¶
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LogPilot: Intent-aware and Scalable Alert Diagnosis for Large-scale Online Service Systems Zhihan Jiang ¶ , Jinyang Liu ‡ , Yichen Li ‡ , Haiyu Huang ¶ , Xiao He ‡ , Tieying Zhang ‡∗ , Jianjun Chen ‡ , Yi Li ‡ , Rui Shi ‡ , Michael R. Lyu ¶ Abstract Effective alert diagnosis is essential for ensuring the reliability of large-scale online service systems. However, on-call engineers are often burdened with manually inspecting massive volumes of logs to identify root causes. While various automated tools have been proposed, they struggle in practice due to alert-agnostic log scoping and the inability to organize complex data effectively for reasoning. To overcome these limitations, we introduce LogPilot, an intent-aware and scalable framework powered by Large Language Models (LLMs) for automated log-based alert diagnosis. LogPilot introduces an intent-aware approach, interpreting the logic in alert definitions ( e.g. , PromQL) to precisely identify causally related logs and requests. To achieve scalability, it reconstructs each request’s execution into a spatiotemporal log chain, clusters similar chains to identify recurring execution patterns, and provides representative samples to the LLMs for diagnosis. This clustering-based approach ensures the input is both rich in diagnostic detail and compact enough to fit within the LLM’s context window. Evaluated on real-world alerts from Volcano Engine Cloud, LogPilot improves the usefulness of root cause summarization by 50.34% and exact localization accuracy by 54.79% over state-of-the-art methods. With a diagnosis time under one minute and a cost of only $0.074 per alert, LogPilot has been successfully deployed in production, offering an automated and practical solution for service alert diagnosis. I introduction With the rise of cloud computing, many traditional systems have migrated to cloud platforms as large-scale online services. These services support critical business operations globally, making reliability essential. Despite rigorous testing efforts, unexpected failures still occur in production, often causing significant disruptions and financial losses [ 1 , 2 , 3 , 4 ] . To ensure service reliability, cloud operators deploy monitoring systems that collect low-level metrics such as per-request latency and response codes. These metrics are aggregated into Service Level Indicators (SLIs), such as request success rate or 99th percentile latency [ 5 , 6 ] . SLIs are then continuously evaluated against predefined Service Level Objectives (SLOs) [ 7 , 8 , 9 ] . When violations occur, alerts are triggered and dispatched to on-call engineers (OCEs) for investigation. While alerts indicate that something has gone wrong, they often reveal only the symptom of a deeper issue rather than its root cause. To resolve the underlying problem, engineers must perform root cause analysis (RCA). This process demands engineers to examine vast amounts of fine-grained monitoring data. Among these data, logs, which record detailed runtime events, are the most widely used and crucial diagnostic resource [ 10 , 11 , 4 , 12 ] . For instance, one empirical study found that 93% of failures in distributed systems are manifested in logs, and logs are instrumental in diagnosing nearly 90% of faults [ 13 ] . This finding aligns with our own practical experience, where logs consistently serve as the primary source of evidence during investigations. In current industrial practice, this log-based diagnostic procedure typically involves two phases (§ II-B ): (1) alert-driven log scoping , in which engineers filter massive volumes of logs to extract relevant entries; and (2) investigative log RCA , which infers the underlying cause of the triggered alert. This manual process is time-consuming and labor-intensive, motivating a variety of automated log-based RCA approaches from both academia [ 14 , 15 ] and industry [ 16 , 17 ] . However, we find that existing automated approaches fall short in complex industrial scenarios due to fundamental limitations in both phases. First, for log scoping (§ III-A ), current methods like keyword searches or anomaly detection are alert-agnostic [ 18 , 19 , 20 ] . Lacking the specific context of the alert, they either overwhelm engineers with irrelevant logs or miss critical evidence entirely. Second, for investigative log RCA (§ III-B ), traditional pattern-learning techniques [ 14 , 21 , 12 ] are impractical. They rely on large, manually labeled datasets of past failures, which are costly to create and maintain. While recent LLM-based approaches [ 22 , 23 , 15 ] show promise, their practical application is hindered by ineffective log data organization. This either leads to log data volumes exceeding the LLM’s context window size or presents interleaved and fragmented event information, thereby impairing the model’s reasoning capabilities. To bridge these gaps, we identify two key opportunities for building a practical and automated alert diagnostic framework. First, we recognize that an intent-aware diagnostic process is achievable by leveraging an alert’s definition logic (e.g., Prometheus Query Language (PromQL) expression) to precisely scope the logs relevant to the triggered alert. Second, we observe that the diagnosis can be made scalable even when the volume of scoped logs is large. This is because the underlying execution paths are not random but follow recurring patterns [ 24 , 17 ] . We can distill these patterns into a few representative examples, which are both compact enough for an LLM’s context window and rich enough to support accurate, holistic reasoning about the root causes. Motivated by these insights, we propose LogPilot, to our knowledge, the first intent-aware and scalable framework powered by LLMs for automated log-based alert diagnosis in large-scale online services. LogPilot consists of three key phases. First, in the Intent-Aware Log Scoping phase, an alert-log correlation agent interprets the semantic intent of alerts, typically encoded in PromQL expression, to automatically generate a tailored log filtering tools. These tools extract the logs and corresponding request IDs that are causally linked to the alert, ensuring high precision in log scoping. Second, the Request-Centric Log Chain Processing phase reconstructs a coherent execution for each request by parsing raw logs into structured events and organizing them into spatiotemporal chains. This representation captures the end-to-end execution flow across distributed components, facilitating deeper diagnostic reasoning. Finally, in the Clustering-Based LLM Diagnosis phase, LogPilot groups requests based on the similarity of their log event patterns and selects a representative request from each cluster for detailed analysis by a log-based RCA agent. This approach ensures that the diagnostic input remains within the LLM’s context window while retaining essential diagnostic clues. The identified root causes are then aggregated by a diagnostic summary agent, which synthesizes a final, comprehensive alert diagnosis report. We conducted an extensive evaluation of LogPilot on real-world alert data from large-scale production services at Volcano Engine. The results show that LogPilot significantly outperforms the state-of-the-art baselines [ 22 , 23 ] , improving root cause summarization by 50.34% (human-evaluated Usefulness) and root cause localization by 54.79% (Exact Match). With a response time of under one-minute and an approximate cost of only $0.074 per alert, LogPilot proves both practical and scalable. Furthermore, LogPilot has been successfully deployed in our production services, and we share our experience and practical insights from this deployment. In summary, the main contributions of this paper are: • We investigate the gap between existing log-based diagnosis research and industrial practice, identifying key opportunities for building practical, automated solutions (§ III

Read the complete text