InferLog: Accelerating LLM Inference for Online Log Parsing via ICL-oriented Prefix Caching DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06

Yilun Wang, Pengfei Chen ∗, Haiyu Huang, Zilong He, Gou Tan, Chuanfu Zhang, Jingkai He, Zibin Zheng · 2025

Read InferLog: Accelerating LLM Inference for Online Log Parsing via ICL-oriented Prefix Caching DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 by Yilun Wang, Pengfei Chen ∗, Haiyu Huang, Zilong He, Gou Tan, Chuanfu Zhang, Jingkai He, Zibin Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08523.txt.

Introduction

Yilun Wang and co-authors speed up AI log reading in a 2025 systems paper. Software pours out runtime logs, and parsing them into clean templates feeds failure alarms and root-cause hunts. Language models parse with top accuracy but answer too slowly for live streams, and sending logs to commercial services risks privacy.

InferLog attacks the wait itself rather than the parsing. Its insight is that in-context examples repeat across requests, so caching the model's work on those shared prefixes lets later calls skip ahead. The design speeds the inference engine behind locally hosted parsers, where many requests arrive at once. Faster parsing brings language-model accuracy within reach of real-time operations.

Catalog record

CREATORYilun Wang, Pengfei Chen ∗, Haiyu Huang, Zilong He, Gou Tan, Chuanfu Zhang, Jingkai He, Zibin Zheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08523

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InferLog: Accelerating LLM Inference for Online Log Parsing via ICL-oriented Prefix Caching DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 Yilun Wang, Pengfei Chen ∗, Haiyu Huang, Zilong He, Gou Tan, Chuanfu Zhang, Jingkai He, Zibin Zheng Abstract . Modern software systems generate massive volumes of runtime logs, necessitating efficient and accurate log parsing to enable critical downstream tasks such as anomaly detection and root cause analysis. Recently, large language models (LLMs) have achieved advanced accuracy on log parsing, but their deployment in production environments faces two major limitations: (i) the privacy risks associated with commercial LLMs, driving the adoption of local deployment, and (ii) the stringent latency and throughput requirements imposed by high-volume log streams, which existing LLM-based parsers fail to meet. Although recent efforts have reduced the number of LLM queries, they overlook the high latency of the LLM invocations, where concurrent log parsing requests can cause serve performance degradation of LLM inference system. In this study, we present InferLog , the first LLM inference optimization method for online log parsing. Our key insight is that the inference efficiency emerges as the vital bottleneck in LLM-based online log parsing, rather than parsing accuracy. InferLog accelerates inference by designing (i) A Prefix-aware ICL Refinement policy to refine the examples and permutation of in-context learning to improve the prefix caching efficiency. (ii) A rapid and task-specific configuration tuning pipeline based on meta-learning to find the optimal LLM scheduling-related configuration for dynamic log parsing workloads. The experimental results based on Loghub dataset and vLLM demonstrate that InferLog significantly outperforms existing inference optimization methods and markedly accelerates the state-of-the-art LLM-based log parser without compromising parsing accuracy. 1. Introduction Modern software systems generate massive volumes of runtime logs to record system states, events, and anomalies. Automatically parsing these semi-structured logs into structured templates, a process known as log parsing, is a foundational task in software engineering. Specifically, this task requires separating log messages into two distinct components: static templates - constant patterns explicitly defined in logging statement and dynamic variables - runtime-specific values reflecting system states. It enables critical downstream applications such as anomaly detection, fault localization and root cause analysis, as shown in Fig. 1 . In recent years, large language models (LLMs) have revolutionized log parsing by leveraging their semantic understanding and few-shot learning capabilities ( Xu et al. 2024b ; Jiang et al. 2024a ; Wu et al. 2025 ; Zhong et al. 2024 ; Ma et al. 2024a ; Ma et al. 2024b ) . Current LLM-based parsers achieve state-of-the-art (SOTA) accuracy in template extraction, even for unseen log formats. This advancement stands in contrast to traditional parsing methods, which typically rely on manually designed heuristics and syntactic analysis ( He et al. 2017 ; Du and Li 2016 ; Jiang et al. 2008 ; Messaoudi et al. 2018 ; Yu et al. 2023 ; Le and Zhang 2023 ; Shima 2016 ) . Nevertheless, a critical gap persists between existing methods and modern software system deployment: First, commercial LLMs pose significant privacy risks for log parsing due to potential unauthorized data retention and exposure vulnerabilities ( Ma et al. 2024b ) . Consequently, organizations increasingly adopt locally deployable open-source LLMs to ensure confidential log data remains secure under strict compliance standards. Second, a modern software system can produce log data at rates of several GB per second, imposing stringent requirements on processing latency and throughput ( He et al. 2021 ; Wang et al. 2022 ) . Despite recent works have achieved promising parsing accuracy, the inference performance is far from the Service Level Objectives (SLOs) in production environments. In our experiments using the vLLM system, we observed that the p95 inference latency for concurrent Loghub-2k log parsing requests was in the tens of seconds, it leads to cascading delays in downstream tasks, ultimately compromising the reliability of large-scale software systems. So, it is crucial to develop an efficient local LLM inference system to handle the massive log parsing requests and meet the SLOs. Figure 1. LLM-based online log parsing system. Recent works ( Jin et al. 2024b ; Caching 2024 ; Zheng et al. 2024 ) have highlighted the use of prefix caching (PC) technique to enhance the performance of LLM inference systems. By sharing and reusing intermediate results (i.e., KV Cache) across requests, it effectively accelerates time-to-first-token (TTFT) and reduces computation costs. While PC benefits general LLM workloads, the effectiveness in log parsing tasks remains unexplored. We conduct an empirical study of LLM-based online log parsing workloads under concurrent scenarios on vLLM ( Kwon et al. 2023 ) , three critical performance issues were identified: (i) The prefill phase (processing input sequence before generating outputs) dominates inference latency, causing 85.4% of delays distribution, violating production SLOs. (ii) Traditional prefix caching are ineffective in bypassing computations for tokens in in-context learning (ICL) examples, which make up a significant portion of the context in the prompt. (iii) LLM system parameters, such as maximum token and batch thresholds, significantly impact inference performance, through automated configuration tuning, it can lead to substantial reductions in latency compared with default setting. A detailed analysis can be found in section 2.2 . Generally, There are mainly two challenges in designing the inference optimization techniques for log parsing workloads. C1: Prefix caching imposes strict requirements on same prefix tokens , as the KV cache contains unique positional embeddings within the original statement and the dependencies between the tokens of the sentence. However, log parsing prompts are dynamic across requests, characterized by varying in-context learning (ICL) examples and permutation ( Xu et al. 2024b ) , which often leads to prefix cache misses. Existing methods ( Zheng et al. 2024 ; Liu et al. 2024 ) attempt to mitigate this by reordering requests to group those with identical prefixes for batch processing, however, these approach is not suitable for online requests. C2: Rapid configuration adaptation for dynamic log parsing workloads . Logs generated by different software systems exhibit distinct characteristics and token distributions, necessitating tailored configurations for optimal performance. Existing tuning methods such as Bayesian Optimization ( Cheng et al. 2024 ; Zhang et al. 2021 ; Alipourfard et al. 2017 ) and Reinforcement Learning ( Zhang et al. 2019 ; Li et al. 2019 ; Dou et al. 2022 ) demand extensive online trials. This renders them impractical for online inference tasks that demand swift configuration tuning ( Zhu et al. 2025 ) . InferLog Approach . Our key insight is that the inference efficiency emerges as the vital bottleneck in LLM-based online log parsing, which is ignored in previous works ( Xu et al. 2024b ; Jiang et al. 2024a ; Huang et al. 2025 ; Ma et al. 2024a ; Xiao et al. 2024 ; Wu et al. 2025 ) . To this end, we propose InferLog , a novel framework designed to optimize the LLM Infer ence performance for Log parsing tasks, focusing on reducing latency and improving throughput in online environments. InferLog proposes Prefix-aware ICL Refinement (PAIR) policy to refine the log parsing prompt by adjusting the contents and order of ICL examples.

Read the complete text