LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding

Shiwen Shan ¶ , Yintong Huo ‡ , Hongzhan Zhong ¶ , Zhining Wang ¶ , Yuxin Su ¶∗ , Zibin Zheng ¶ · 2026

Read LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding by Shiwen Shan ¶ , Yintong Huo ‡ , Hongzhan Zhong ¶ , Zhining Wang ¶ , Yuxin Su ¶∗ , Zibin Zheng ¶ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20618.txt.

Introduction

Shan and co-authors present LogFold, a method to compress software logs. Firms keep logs for long spans to trace faults, but log volume grows fast. Past log tools miss patterns inside complex tokens. They also encode all token types in much the same way. LogFold sorts tokens as structured, plain, or static. It mines patterns in structured tokens and encodes each type to fit. On 16 public log sets it beat prior tools. The gain in ratio averaged 11.11 percent.

Catalog record

CREATORShiwen Shan ¶ , Yintong Huo ‡ , Hongzhan Zhong ¶ , Zhining Wang ¶ , Yuxin Su ¶∗ , Zibin Zheng ¶
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62008
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20618

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3787801 ISBN: 979-8-4007-2025-3/2026/04 Thanks: * Yuxin Su is corresponding author. CCS: Software and its engineering Software maintenance tools Shiwen Shan ¶ , Yintong Huo ‡ , Hongzhan Zhong ¶ , Zhining Wang ¶ , Yuxin Su ¶∗ , Zibin Zheng ¶ Abstract . Logs are essential for diagnosing failures and conducting retrospective studies, leading many software organizations to retain log messages for a long time. Nevertheless, the volume of generated log data grows rapidly as software systems grow, necessitating an effective compression method. Apart from general-purpose compressors (e.g., Gzip, Bzip2), many recent studies developed log-specific compression algorithms, but they offer suboptimal performance because of (1) overlooking redundancies within certain complex tokens, and (2) lacking a fine-grained encoding strategy for diverse token types. This work uncovers a new redundancy pattern in structured tokens and proposes a new type-aware encoding strategy to improve log compression. Building on this insight, we introduce LogFold , a novel log compression method consisting of four components: a token analyzer to classifies tokens as structured, unstructured, or static types; a processor that mines recurring patterns within structured tokens based on their delimiter skeletons; a hybrid encoder that tailors data representation according to token types; and a packer that compresses the output into an archive file. Extensive experiments on 16 public log datasets demonstrate that LogFold surpasses state-of-the-art baselines, achieving average compression ratio improvements by 11.11%, with a compression speed of 9.842 MB/s. Ablation studies further indicate the importance of each component. We also conduct sensitivity analyses to verify LogFold ’s robustness and stability across various internal settings. † † cc-license: by 1. Introduction System logs are an indispensable resource in software engineering. They provide valuable run-time information about software behavior for system maintenance ( 17 ; 60 ; 69 ) , ( 49 ; 28 ; 32 ) , performance modeling ( 3 ; 61 ) , and security auditing ( 4 ; 48 ) . Additionally, the detailed information in logs necessitates their storage for extended periods to support post-mortem analysis ( 66 ; 31 ; 27 ; 44 ) . Industry practices have regulations on mandating long-term log storage ( 56 ; 54 ) . For example, some cloud providers must store logs for 180 days ( 56 ) , and others are required to keep them for years ( 47 ) . Legal compliance, such as data privacy laws, also mandates the prolonged storage of user logs ( 62 ) . However, as software systems scale, they generate an enormous volume of logs, imposing huge pressure on storage systems in terms of cost and capacity ( 51 ) . For instance, large-scale services like eBay were reported to produce over a petabyte of log data daily in 2018 ( 47 ; 54 ) . Consequently, to mitigate the challenge, effective log compression has become an indispensable technique. The core idea of log compression is to identify recurring patterns (e.g., repeated templates, common tokens) within the log data and encode these patterns and the remaining data into a compact form to eliminate redundancy. Here, encoding refers to the transformation of original data into a more space-efficient representation. To compress logs, a straightforward solution is to employ general-purpose compressors such as Gzip ( 22 ) and Bzip2 ( 8 ) . However, their compression ratios remain suboptimal because they fail to recognize the various recurring patterns (e.g., highly repeated log templates) of log ( 62 ; 56 ; 47 ) . To further improve log compression, a variety of specialized log compressors have been proposed ( 38 ; 34 ; 64 ; 19 ; 63 ) . The parsing-based log compressors typically target reducing redundancy in log templates (i.e., log events). To extract recurring templates, LogZip ( 38 ) leverages an iterative-clustering method, while LogReducer ( 56 ) and LogShrink ( 34 ) utilize sample-based log parsing techniques. Then, all three methods employ dictionary encoding to replace templates and their parameters with compact indexes. Additionally, LogReducer enhances compression by applying enhanced numeric encoding strategies (e.g., elastic encoding and delta encoding) to number sequences. In contrast, Denum ( 64 ) compresses numeric tokens by grouping them based on length and initial digit, followed by dictionary and optimized numeric encoding. While effective, these approaches face two key limitations: (1) overlooking the recurring patterns within structured tokens . and (2) lacking a comprehensive, type-aware encoding strategy for handling token diversity . The first limitation is the oversight of recurring patterns within structured tokens. Log entries often include tokens composed of delimiters and sub-tokens, such as 2015-07-09 . Current methods typically do not tailor to these more complex structured tokens unless they match exactly. However, these tokens still contain recurring patterns inside, such as the format <>-<>-<> , and the sequence of sub-tokens like ["2015", "07”, “09”] . Ignorance of such intra-token characteristics will hamper the effectiveness of compression. The second limitation is the lack of a comprehensive, type-aware encoding strategy. Log entries contain diverse token types: pure numbers, pure strings, and, most commonly, mixed-type tokens (e.g., a background service process’s name and its ID, such as ftpd[4305] ). However, existing works employ a coarse approach, which applies numeric encoding only to pure numbers while treating pure strings and complex mixed-type tokens alike with plain dictionary encoding. This is sub-optimal, as it ignores the distinct structural characteristics of mixed-type tokens, thereby sacrificing compact representations for encoding diverse tokens. This paper stems from two insights to mitigate the above limitations. First, we recognize and decompose structured tokens based on their delimiter skeletons and sub-token sequences. The deconstruction uncovers novel intra-token patterns, which can be used to enhance compression. Second, we emphasize a fine-grained hybrid encoding strategy that tailors effective encoding techniques to each specific data type. This holistic view ensures that all tokens (i.e., numeric, string, and mixed) are compressed effectively, leading to overall compression improvements. We propose and implement LogFold to capitalize on these insights. It is a delimiter skeleton-aware log compression tool consisting of four components. Specifically, LogFold begins with a Token Analyzer that treats each log entry as a whole. Rather than breaking it apart, the analyzer preserves the original sequence and classifies tokens into three types: structured, unstructured, and static. The identified structured tokens are then passed to the Structured Token Processor designed to exploit the recurring patterns within structured tokens. It begins by deconstructing tokens and grouping them according to their shared delimiter skeleton, reducing redundancy in structural patterns. Next, it applies pattern mining to sub-tokens within those groups, recursively partitioning them into smaller, more homogeneous subgroups. This two-step strategy, progressing from coarse-grained skeletons to fine-grained sub-tokens, enables significantly identifying recurring patterns, and thus enhances compression. Then the resulting refined patterns and the simplified sub-token matrix, along with unstructured tokens and the static token sequences, will be passed to the Hybrid Encoder component. This component utilizes two optimized classic encoding strategies and a novel proposed mixed-type encoding strategy to encode the input data

Read the complete text