LogLLaMA: Transformer-based log anomaly detection with LLaMA

Zhuoyi Yang, Ian G. Harris · 2025

Read LogLLaMA: Transformer-based log anomaly detection with LLaMA by Zhuoyi Yang, Ian G. Harris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14849.txt.

Introduction

Zhuoyi Yang and Ian Harris adapt a large language model to spot bad log lines in a 2025 paper. System logs record events, and odd lines can warn of failures or attacks. Their model first learns the rhythm of normal logs from big data sets. It then learns to predict the next line in a healthy sequence.

A reward based round of training sharpens its eye for odd lines. The model judges new lines by how poorly they fit the learned rhythm. Tests on three classic log sets beat prior best methods. The setup needs no hand labeled attack examples to get started.

Catalog record

CREATORZhuoyi Yang, Ian G. Harris
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14849

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LogLLaMA: Transformer-based log anomaly detection with LLaMA Zhuoyi Yang, Ian G. Harris Abstract Log anomaly detection refers to the task that distinguishes the anomalous log messages from normal log messages. Transformer-based large language models (LLMs) are becoming popular for log anomaly detection because of their superb ability to understand complex and long language patterns. In this paper, we propose LogLLaMA, a novel framework that leverages LLaMA2. LogLLaMA is first finetuned on normal log messages from three large-scale datasets to learn their patterns. After finetuning, the model is capable of generating successive log messages given previous log messages. Our generative model is further trained to identify anomalous log messages using reinforcement learning (RL). The experimental results show that LogLLaMA outperforms the state-of-the-art approaches for anomaly detection on BGL, Thunderbird, and HDFS datasets. I introduction System logs, which record detailed system events, serve as a vital source of information for system monitoring, debugging, and security auditing. They offer insights into system performance and potential issues, with anomalies often indicating system faults, security breaches, or operational failures. As computer systems become increasingly vulnerable to cyberattacks, accurate detection of log anomalies is essential to preventing attacks and malfunctions [ 1 ] . Log anomaly detection frameworks take structured log messages as input. The goal is to classify log messages into normal or abnormal categories with precision. The fundamental approach involves identifying distinct patterns in the logs and differentiating typical behaviors from anomalies. Because system log files are complex, large, and require automation in identifying patterns and making predictions, many machine learning methods are proposed on anomalous log message classifications. Most of such methods are either unsupervised such as Principal Component Analysis (PCA) [ 2 ] , iForest [ 3 ] , and LogCluster [ 4 ] , or semi-supervised such as One-Class Support Vector Machine [ 5 ] . However, traditional machine learning models, which are based on hand-crafted features, are infeasible to capture the temporal information of discrete log messages [ 6 ] . More recently, many deep-learning-based approaches have been proposed to detect anomalous log messages. Most works in this area treat log messages as if they are natural language. For example, DeepLog [ 7 ] uses LSTM models to learn the contextual features in normal log sequences and detect abnormal patterns based on predictions. Experimental results show that these approaches achieve higher accuracy than traditional machine learning approaches at log anomaly detection. However, these models exhibit limitations in effectively capturing long sequential log messages. Identifying an anomalous log message often requires considering a substantial number of preceding messages but LSTM-based models often are constrained in their ability to process sequences of the necessary length. With the development of deep learning, transformers [ 8 ] have been widely adopted, achieving remarkable success in natural language processing (NLP) tasks through large language models (LLMs) [ 9 ] . This success has inspired their applications in log anomaly detection, as log messages are often treated as natural language. Among all the LLMs, BERT [ 10 ] is the most utilized one for log anomaly detection so far. For instance, LogBERT [ 6 ] utilizes a BERT-based model to identify anomalies through masked token prediction. While such models inherit the benefits of BERT, they are also affected by the disadvantages of BERT such as its sensitivity to the number of masked tokens and its maximum input length. Therefore, BERT-based models may not fully capture the sequential and temporal relationships between log messages. We introduce a novel framework LogLLaMA, which is built upon LLaMA2 [ 11 ] . LogLLaMA harnesses the power of the transformer architecture to capture the complex pattern of log messages. Our framework consists of three modules. First, it preprocess the raw log messages by parsing them. Then, the LLaMA2-based foundation model is finetuned with the preprocessed log messages, after which the model is capable of predicting the next log sequences given previous log messages. Finally, we employ reinforcement learning, incorporating the REINFORCE algorithm [ 12 ] as the update policy. The algorithm is augmented with an entropy bonus, reward clipping, and a customized reward system to effectively identify anomalous log messages. Our contributions are as follows: 1) We propose a new framework LogLLaMA, which is capable of capturing the underlying pattern of normal log messages. After finetuning, our model has the ability to generate new log key sequences given previous log messages. Built on top of LLaMA2, LogLLaMA closes the gap between log messages and natural language. 2) We incorporate the REINFORCE algorithm with entropy bonus and reward clipping as well as a customized reward system for anomaly detection. With Top-K selection, LogLLaMA is able to identify anomalous log messages with high accuracy. 3) By conducting experiments on three publicly available datasets, we compare our framework with multiple baselines. The results demonstrate that LogLLaMA outperforms state-of-the-art methods by a significant margin. II related work Log anomaly detection is critical to finding security leaks and maintain system reliability. Various machine learning methods have been proposed: Principal Component Analysis(PCA)-based algorithms [ 2 ] are used to separate out repeating patterns in feature vectors, thereby making abnormal message patterns easier to detect. Isolation forest (iForest) [ 3 ] randomly selects a feature and a split value between its maximum and minimum values. This process creates shorter paths in trees for anomalous data points, distinguishing them from the rest of the data. One-class Support Vector Machines (OCSVM) [ 5 ] operates by defining a boundary around the majority class (normal instances) in the feature space. This boundary is constructed to encapsulate the normal data points, effectively creating a region of normalcy. Although these models are capable of identifying outliers in the log data, a severe issue is that they are trying to extract the semantics of log events through feature engineering, which only focuses on the words in the log event, without considering the interaction between words in the log events, or word order [ 13 ] . However, log messages are often mixed and interdependent, with individual words frequently lacking standalone significance. Another category of approaches for log anomaly detection is deep-learning methods. DeepLog [ 7 ] is the first to use LSTM models to model the sequence of log events. The LSTM learns to predict the next log event given the previous sequences. Its prediction ability is based on the temporal patterns of normal log sequences. LogAnomaly extends DeepLog and proposes the Template2Vec approach to extract both sequential and quantitative features [ 14 ] . Previous studies have shown that deep-learning methods have much higher accuracy than traditional machine learning methods [ 10 , 14 ] . Our experimental results also show that deep-learning methods have a more consistent performance across different datasets than traditional machine learning methods. More recently, LLMs with transformers [ 8 ] have demonstrated their power in different NLP tasks [ 9 , 15 ] . Although LLMs are not specifically trained to understand logs, they can be finetuned on log datasets to learn the patterns and structures of log messages with the right downstream tasks. LogBERT [ 6 ] is the first to use Bidirectional Encoder Representations from Transformers (BERT) as a base model to predict the masked tokens

Read the complete text