Meta Variational Memory Transformer for Anomaly Detection of Multivariate Time Series Authors: Kun Qin, Yuxin Li, Wenchao Chen, Xinyue Hu, Bo Chen, Hongwei Liu Published: 2025 ABSTRACT Detecting anomalies in multivariate time series (MTS) is a crucial task in areas like financial fraud detection and industrial equipment monitoring. Recent research has focused on developing unsupervised probabilistic models to identify anomalous patterns within MTS. However, many of these methods rely on fixed parameter mappings for each MTS, resulting in high computational costs and limited adaptability. To overcome these challenges, we introduce a novel Meta Variational Memory Transformer (MVMT). MVMT captures the diverse patterns across various MTS by encoding them into a set of memory units using a specially developed meta memory attention (MMA) module. Utilizing these learned memory units, we introduce a memory-guided probabilistic generative model that selects relevant memories as priors for latent states, resulting in more expressive MTS representations. A key feature of MVMT is that MMA provides a diversified prior in the latent space, ensuring the generation of various patterns. Finally, we implement a Transformer-based upward–downward variational inference process to estimate the posterior distribution of latent variables. Our extensive experiments on six datasets demonstrate the effectiveness of MVMT in one-for-all anomaly detection tasks. 1. Introduction Anomaly detection in MTS stands as a critical scenario in Information Technology (IT) operations, particularly vital for financial fraud detection and industrial equipment monitoring, such as server machines within cloud infrastructures [1,2,3], content delivery networks [4], and more. Traditional statistical techniques are effective in simpler cases, such as univariate time-series anomaly detection [5,6,7,8,9]. However, their effectiveness may wane in more intricate and massive MTS scenarios. Consequently, attention has shifted towards machine learning-driven anomaly detection techniques, which are generally categorized into supervised learning approaches [5,10,11] and unsupervised methods [1,2,8,12,13,14,15,16]. Since anomalies are often infrequent and hidden within large volumes of normal data, identifying and labeling them is both challenging and costly. Moreover, supervised anomaly detection models often suffer from suboptimal performance due to severe class imbalance [17,18]. Consequently, unsupervised anomaly detection has gained significant traction in recent years. A major obstacle in unsupervised anomaly detection lies in effectively modeling the normal patterns in MTS. Among the various unsupervised methods, probabilistic dynamical models like OmniAnomaly [1] and SDFVAE [4] have excelled by capturing temporal correlations and variations inherent in time series. The emergence of Transformer architectures [19] and their ability to capture long-range dependencies [20,21,22,23] have further motivated their adoption for time-series modeling, and recent Transformer-based anomaly detectors [24] have shown strong representational capability. Since anomalies are often infrequent and hidden within large volumes of normal data, identifying and labeling them is both challenging and costly. Moreover, supervised anomaly detection models often suffer from suboptimal performance due to severe class imbalance [17,18]. Consequently, unsupervised anomaly detection has gained significant traction in recent years. A major obstacle in unsupervised anomaly detection lies in effectively modeling the normal patterns in MTS. Among the various unsupervised methods, probabilistic dynamical models like OmniAnomaly [1] and SDFVAE [4] have excelled by capturing temporal correlations and variations inherent in time series. The emergence of Transformer architectures [19] and their ability to capture long-range dependencies [20,21,22,23] have further motivated their adoption for time series modeling, and recent Transformer-based anomaly detectors [24] have shown strong representational capability. In contrast to physics-embedded MTS models [25], which rely on physical constraints or structure-preserving mappings, our method adopts a purely data-driven probabilistic framework. Rather than enforcing physics-guided latent spaces, we introduce a meta memory prior to capture diverse temporal patterns across heterogeneous MTSs, making the approach suitable when explicit physical laws are unavailable. However, these models typically learn a single shared representation and lack an explicit mechanism to encode and differentiate the diverse normal patterns that arise across heterogeneous MTS sources. As illustrated in Figure 1, different devices or subsystems may exhibit fundamentally different operational modes, making it difficult for a single Transformer encoder to generalize. Without structural priors, attention alone must implicitly capture all such variations, which often leads to underfitting of rare-but-normal behaviors and reduced robustness when intra-class variability is high [26]. These limitations motivate the introduction of a meta memory prior that can store, organize, and retrieve prototypical temporal patterns learned across multiple MTSs. By providing diversified and informative priors for latent states, such a mechanism fills the gap left by existing Transformer-based anomaly detectors and enables more expressive and adaptable representations in a unified one-for-all anomaly detection framework. To overcome the aforementioned limitations, we introduce an innovative memory-guided probabilistic generative model tailored to capture intricate and diverse temporal dependencies across multiple MTSs. Our approach draws inspiration from meta-learning [27], aiming to extract transferable patterns across related tasks. We devised the Meta Memory Attention (MMA) mechanism to identify and retain complex and diverse temporal information shared among multiple MTSs, storing it within the memory units as illustrated in Figure 2. Subsequently, we developed a memory-guided probabilistic generative model within a hierarchical Bayesian framework. This model leverages the diverse temporal dependencies by employing the stored memory from MMA as a prior, thereby facilitating the acquisition of expressive representations crucial for anomaly detection within MTS data. To optimize the inference process for our model, we implement an innovative upward-downward autoencoding variational inference strategy. This approach integrates both bottom-up likelihood assessments and top-down prior knowledge of parameters to achieve a more accurate posterior approximation. Additionally, we enhance the inference network by integrating an anomaly transformer, which refines the latent representations. The key contributions of our work are summarized as follows:To consider diverse nomal patterns within MTSs and achieve anomaly detection, we propose a hierarchical Bayesian network named MVMT. MVMT consists of a developed MMA module that captures and stores various temporal dependencies of MTSs, a memory-guided generative module to capture the shared diverse temporal information, as well as a transformer-powered inference module for accurate posterior approximation.We carefully design an upward–downward variational inference model built upon the transformer blocks for MVMT to approximate the posterior for the latent variables.While most existing methods rely on distinct parameter sets for each MTS, the proposed MVMT utilizes a single group of parameters to model diverse MTSs, allowing it to effectively capture multiple patterns within a unified framework.We perform thorough experiments on six real-world datasets. The quantitative comparison results demonstrate that MVMT surpasses state-of-the-art methods in terms of F1-score with fewer parameters. The qualitative analysis reveals that it effectively captures diverse dynamic memory patterns within different MTSs. To consider diverse nomal patterns within MTSs and achieve anomaly detection, we propose a hierarchical Bayesian network named MVMT. MVMT consists of a developed MMA module that captures and stores various temporal dependencies of MTSs, a memory-guided generative module to capture the shared diverse temporal information, as well as a transformer-powered inference module for accurate posterior approximation. To consider diverse nomal patterns within MTSs and achieve anomaly detection, we propose a hierarchical Bayesian network named MVMT. MVMT consists of a developed MMA module that captures and stores various temporal dependencies of MTSs, a memory-guided generative module to capture the shared diverse temporal information, as well as a transformer-powered inference module for accurate posterior approximation. We carefully design an upward–downward variational inference model built upon the transformer blocks for MVMT to approximate the posterior for the latent variables. We carefully design an upward–downward variational inference model built upon the transformer blocks for MVMT to approximate the posterior for the latent variables. While most existing methods rely on distinct parameter sets for each MTS, the proposed MVMT utilizes a single group of parameters to model diverse MTSs, allowing it to effectively capture multiple patterns within a unified framework. While most existing methods rely on distinct parameter sets for each MTS, the proposed MVMT utilizes a single group of parameters to model diverse MTSs, allowing it to effectively capture multiple patterns within a unified framework. We perform thorough experiments on six real-world datasets. The quantitative comparison results demonstrate that MVMT surpasses state-of-the-art methods in terms of F1-score with fewer parameters. The qualitative analysis reveals that it effectively captures diverse dynamic memory patterns within different MTSs. We perform thorough experiments on six real-world datasets. The quantitative comparison results demonstrate that MVMT surpasses state-of-the-art methods in terms of F1-score with fewer parameters. The qualitative analysis reveals that it effectively captures diverse dynamic memory patterns within different MTSs. Meta Memory Attention (MMA). We constructed this module by incorporating a multi-headed attention mechanism. This approach significantly enhances our model’s memory recall capability. 2. Related Work Recently, unsupervised anomaly detection has become a focal point for operational engineers. Many of these methods adopt an encoding-reconstruction approach, leveraging architectures like AE, VAE, or GAN. To model the characteristics of MTS, these methods are typically based on either RNNs [28] or Transformers [19]. For instance, EncDec-AD [15] uses an LSTM-based AE architecture to capture temporal dependencies in MTS, detecting anomalies through deviations from reconstruction. Similarly, Telemanom [16] leverages LSTM to predict spacecraft telemetry channel values and detects anomalies by evaluating the residual error between predicted and observed values. Additionally, MAD-GAN [13] employs LSTM-RNN within a GAN to model typical temporal and spatial patterns. OmniAnomaly [1] uses a stochastic recurrent neural network (SRNN) to robustly learn representations, while SDFVAE [4] introduces a static and dynamic factorized VAE approach specifically addressing both time-invariant and time-varying elements. VAE-based models, such as OmniAnomaly and SDFVAE, detect anomalies by evaluating likelihood, with lower likelihood values suggesting a higher probability of an anomaly. It is important to highlight that these unsupervised approaches are largely RNN-based, concentrating on capturing temporal dependencies in MTSs. However, there is a recent inclination towards leveraging transformers, known for their efficiency in capturing intricate and long-range temporal dependencies, as well as in MTS anomaly detection [29]. Anomaly transformer [24] has emerged as the top performer in MTS anomaly detection thus far. Although current unsupervised anomaly detection methods have shown effectiveness in practical applications, capturing diverse temporal dependencies across multiple MTSs remains a significant challenge. The common approach of training separate models for each MTS limits comprehensive modeling, leading to high computational costs and reduced adaptability. To address this issue, GmVRNN [30] seeks to handle diverse MTSs with a single set of parameters by using a Gaussian mixture distribution for latent states. Despite its promising results, this approach often struggles to capture all patterns effectively, as a single parameter set may not be sufficient, requiring adjustments for different MTSs to ensure optimal performance. This paper introduces MVMT, a pioneering Transformer-based probabilistic model. MVMT captures the diverse patterns across various MTS by encoding them into a set of memory units using a specially developed meta memory attention (MMA) module. Utilizing these learned memory units, we introduce a memory-guided probabilistic generative model that selects relevant memories as priors for latent states, resulting in more expressive MTS representations. 3. Preliminaries This section presents a brief overview of the problem definition, the MVMT-based anomaly detection approach, and the data processing methodology. Additionally, we clarify the meanings of the symbols used in the paper, as detailed in Table 1. 3.1. Problem Definition Consider the MTS as X={x1,x2,...,xTl}, where Tl represents the total length of X, and xt∈RV is the observation at time t, with V indicating the dimensionality at each timestep. Thus, X∈RTl×V. Anomaly detection in MTS involves determining whether a specific observation at a given time, denoted as xt, is anomalous. In an unsupervised approach, the model is trained exclusively on normal data to capture normal patterns in MTS. During the testing phase, reconstruction errors are used as anomaly scores. The anomaly score function should produce higher values for anomalous MTS compared to normal ones. 3.2. Anomaly Detection Based on MVMT The overarching structure for unsupervised anomaly detection utilizing MVMT is depicted in Figure 3. Our framework consists of three components: MTS preprocessing, representation model, and anomaly detection modules. Specifically, MTS preprocessing is introduced in Section 3.3, and MVMT is employed as the representation model. The reconstruction probability inferred by MVMT is utilized to guide the detection of anomalies. As the model learns the typical patterns of normal MTS, observations that closely match these learned behaviors are reconstructed with high accuracy and confidence. Therefore, we use the reconstruction error as an anomaly score to assess deviations from expected behavior [1,2,8,31]. The anomaly score is computed as follows:(1)St=logp(xt∣zt,ma) where St is the anomaly score, and ma and z are two components in the latent space of the proposed model, which will be explained in detail in Section 4. Because our model is built within a Bayesian probabilistic framework, the reconstruction term is naturally expressed as the log-likelihood. This choice ensures consistency with the hierarchical VAE derivation, whereas MSE residuals correspond to deterministic autoencoders and are not directly aligned with our probabilistic formulation. To classify an observation xt as anomalous, we compare its anomaly score St against a predefined threshold. We use the POT method [32] to set this threshold. In MVMT, smaller anomaly scores are deemed more likely to indicate outliers. Consequently, similar to [1], we select thresholds based on lower-bound criteria to identify extreme values. 3.3. Data Preprocess Similar to the approach in [33], for observed MTSs, we initially normalize them as(2)xt←xt−min(X)max(X)−min(X)+ϵ where max(X) and min(X) denote the maximum and minimum values for each feature across the training time series, respectively, and ε is a small constant (set to 10−6) introduced to avoid division-by-zero instability, and the results are insensitive to its value within a reasonable numerical range. After normalizing the MTSs, we employ a sliding window technique with overlap, following [4], for dataset preprocessing. Here, the window size T=20, and there is an overlap of l=T−1 between two consecutive observed variables. Notably, to capture both time-varying and time-invariant information, we augment each timestep with its w neighbors, set to 5 for anomaly detection tasks. 4. Methodology In this section, we outline the key concepts of the proposed MVMT framework, followed by an in-depth discussion of the MMA model, including its generative and inference components. To facilitate understanding, Table 1 provides a summary of the symbols used throughout the paper. 4.1. Meta Variational Memory Transformer As depicted in Figure 4, MVMT establishes a memory model based on the transformer architecture from a variational Bayesian perspective. Our approach introduces a latent space comprising two components: ma and z. To capture and retain diverse patterns, we devised the MMA module, constructed upon a set of memory units organized using the attention mechanism. ma serves as a memory prior for z, ensuring an expressive latent representation and facilitating the generation of MTS with diverse temporal information. In contrast to the conventional VAE [34], MVMT defines the latent variable z with a memory prior (ma), integrating diverse temporal information from multiple MTSs. This structuring of the latent space is represented as follows in equation form:(3)p(x)=∫ma∫zp(x∣z,ma)p(z∣ma)p(ma)dmadzBased on the latent space defined in Equation (3), we present a refined MMA model alongside a Bayesian generative framework to effectively capture and represent the full latent space. 4.1.1. Meta Memory Attention Module Global diverse temporal information within multiple MTSs is crucial for modeling multiple MTSs with a single model [30]. Inspired by Concept Transformer (CT) [35], we aim to capture and store diverse temporal information within MTS through a meta memory module, employing an attention module to facilitate memory operations. Specifically, considering N memory units M∈RN×dm and MTS features h′∈Rdm, where dm represents the vector dimension, M consists of vectors optimized by a well-designed loss function proposed later. Subsequently, an attention mechanism is utilized to derive the memory embedding Ma. As illustrated in Figure 2, the input feature h′ initially undergoes mapping via a MLP to generate the query matrix Q∈RN×dm for the query-key-value cross-attention module. Following this, the memory units M serve as input to predict the key matrix K∈RN×dm and value matrix V∈RN×dm. The attention weights matrix A=[α]∈RN×N can be obtained from Q and K as(4)α=softmax1dmQK⊤Final result from the MMA is computed by applying the attention map A to the value matrix V, yielding the weighted output.(5)Ma=∑i=1N[AV]iAs shown in Figure 4a, with the memory embedding (Ma) defined in Equation (5), we introduce a memory latent variable ma:(6)p(ma∣Ma)∼Nma∣μma,diagσma2μma=GμMa(Ma)=f(WμMaMa+bμMa)σma2=GσMa(Ma)=f(WσMaMa+bσMa) where ma is treated as a Gaussian random variable, GμMa and GσMa are two functions used to generate ma from Ma. Specifically, we utilize a fully connected network as GMa, and f(·) represents a deterministic non-linear transition function, where WMa∈Rdm×dm denotes the transition matrices, and bMa∈Rdm represents the bias weights. From Equation (6), ma is conditioned on the memory embedding Ma. The advantage of the attention mechanism in MMA allows ma to provide powerful memory information for subsequent models. The memory units are initialized with small random values and are jointly updated with all model parameters through standard back-propagation under the variational objective, which ensures stable training and practical convergence. 4.1.2. Probablistic Memory Generative Model Figure 4a illustrates how we model the diverse temporal dependencies in MTS by utilizing the multiple patterns stored in the MMA module. To achieve this, we design a probabilistic memory generative model based on a hierarchical Bayesian framework [36], where the latent variable is defined as(7)p(z∣ma)∼Nz∣μz,diagσz2μz=Fμma(ma)=f(Wμmama+bμma)σz2=Fσma(ma)=f(Wσmama+bσma) where Fμma and Fσma are two functions used to generate z from ma. Specifically, we employ a fully connected network as Fma, and f(·) represents a deterministic non-linear transition function. Here, Wma∈Rdm×dm denotes the transition matrices, and bma∈Rdm represents the bias weights. Unlike traditional VAEs, where the latent variable z is typically conditioned on a standard Gaussian distribution, in our model, z is conditioned on the latent variable ma. This enables MVMT to consider diverse temporal dependencies based on the meta-information stored in MMA. As depicted in Equation (6), p(ma∣Ma) provides globally diverse temporal information for each latent variable. Similar to [30], for considering the stochasticity within MTS, we assign the generated x to follow a Gaussian distribution, defined as(8)p(x∣z)∼Nx∣μx,diagσx2 where μx and σx2 are mean and variance parameters that are conditioned on the hidden variable z as(9)μx=Fμz(z)=f(Wμzz+bμz)σx=Fσz(z)=f(Wσzz+bσz)Fμz and Fzσ are two functions used to generate x from z. Similar to Equation (7), we utilize a fully connected network as Fma, where model parameters Wz and bz have the same settings. This approach enables the model to transfer the diverse temporal information stored in the MMA from ma to z, ultimately aiding in reconstructing x. The hierarchical information generation allows the model to consider the diversity within multiple MTS and acquire more expressive and robust representations for them. This is essential for unsupervised anomaly detection methods to enhance their performance. 4.1.3. Inference Model As with most Bayesian models, directly computing the exact posterior distribution is intractable, a challenge common in VAEs [34,37]. To address this, we define the variational distribution q(ma,z∣x,Ma), which must be sufficiently flexible to closely approximate the true posterior. To efficiently approximate the posterior for the hierarchically structured latent variables in MVMT, we design an inference network that integrates both bottom-up likelihood information and top-down prior information, allowing for a more accurate approximation of the generative process. The variational distribution is defined as(10)q(ma,z∣Ma,x)=q(ma∣Ma,x)q(z∣ma,x)Specifically, to capture the complex temporal characteristics of MTSs, we initially employ an anomaly-transformer-like block (using the model structure but without the minimax strategy proposed by [24]) to extract the features of MTSs, and the parameters of the block are defined as Wxh0. As depicted in Figure 4c, for the upward flow of information, we compute the variable h0 through the anomaly transformer and then utilize a fully connected network to obtain the variables h1 and h′ as follows:(11)h0=Anomaly Transformer(x)h1=f(Wh0h1h0+bh0h1)h′=f(Wh0h′h0+bh0h′)As depicted in Figure 4b,c, we estimate ma by integrating the extracted latent features with prior knowledge from the MMA module, constructing the variational posterior for ma as follows:(12)q(ma∣Ma,h1)∼Nma∣μ˜ma,diagσ˜ma2μ˜ma=Φ(Ma,h1)=f(W˜maμMa+V˜maμh1+b˜maμ)σ˜ma=Φ(Ma,h1)=f(W˜maσMa+V˜maσh1+b˜maσ) where {W˜maμ,V˜maμ, W˜maσ, V˜maσ }∈Rdm×dm and {b˜maμ, b˜maσ} ∈Rdm are the parameters that the inference model learns during training. As a hierarchical VAE, the inference model requires separate latent flows: h0 estimates the posterior of the lower-level latent variable z, while h1 provides a higher-level approximation for ma, ensuring that deeper latent layers are inferred from correspondingly more abstract features. This approach allows ma to be estimated by integrating both the bottom-up likelihood information present in h1 from the inference network and the contextual memory from the MMA module. By combining these sources, the proposed inference network effectively captures long-range and diverse temporal dependencies, leading to a rich latent representation for MVMT. The latent variable q(ma∣Ma,h1) is then reparameterized as(13)ma=μ˜mak+σ˜makϵ,ϵ∼Uniform(0,1) which allows for back-propagation through z. MVMT combines the obtained latent features by the inference network with the prior from ma to construct the variational posterior as(14)q(z∣ma,h0)∼Nz∣μ˜z,diagσ˜z2μ˜z=Ψ(ma,h0)=f(W˜zμma+V˜zμh0+b˜zμ)σ˜z=Ψ(ma,h0)=f(W˜zσma+V˜zσh0+b˜zσ) where {W˜zμ,V˜zμ, W˜zσ, V˜zσ }∈Rdm×dm and {b˜zμ, b˜zσ} ∈Rdm denote the learnable parameters that map ma and h0 to z. This allows z to be inferred by integrating the likelihood information from h0 with the prior knowledge provided by ma, thus facilitating the generation of rich latent representations for MVMT. Additionally, q(z∣ma,h0) can be reparameterized as(15)z=μ˜z+σ˜zϵ,ϵ∼Uniform(0,1) which facilitates the use of the back-propagation algorithm. We note that MVMT incorporates transformer structure into inference processes, which enables it to model complex temporal dependencies within MTS efficiently in both processes, thus guaranteeing expressive latent representation learning and promising anomaly detection performance. 4.2. Model Properties MVMT leverages the advantages of the hierarchical Bayesian model and the meta memory model enhanced by the attention mechanism, as detailed below. 4.2.1. Learnable Memory Prior The general VAE [34] uses the standard normal distribution as the prior, which does not contain information. We designed the MMA to organize and store the information from the big data and select the corresponding memory for different MTS to generate a dedicated prior. This prior gives our model the ability to adapt quickly to the new MTS. Moreover, we designed an attention-based meta memory module to efficiently manage global information and provide high-quality meta memory priors. 4.2.2. One Model for All MTSs Diverse MTSs often exhibit varying characteristics, making it difficult for anomaly detection models to fully capture this complexity using a single parameter set. MVMT overcomes this by utilizing the MMA and memory generative modules to capture and store a range of dynamics, enabling the model to handle diverse MTSs with a unified set of parameters—a trait we refer to as the one-for-all property. This property can not only ensure MVMT to manage large MTS data but also adapt to new MTS with few observations, which will be proved in the experiment. 4.2.3. Probability Modeling of Hidden Space and Generative Process Previous deterministic transformer-based approaches may face challenges in modeling the variability present in highly structured MTS data, characterized by complex and strong dependencies across different timesteps. To overcome this, we introduce uncertainty into transformers by applying probabilistic modeling in both the hidden space and the generative process. Following similar arguments made by others [38], we propose that this probabilistic framework is particularly useful when the true data distribution is multimodal, which reflects the diverse nature of multivariate time-series data. 4.2.4. Transformer for Time Series Transformers have demonstrated efficiency in capturing complex, long-range temporal dependencies. MVMT applies transformer structure in both memory and inference modules, which enables our model to learn expressive memories and latent representations. Especially for multiple MTSs data, there are diverse patterns within them; traditional recurrent-structured networks may not be powerful enough to capture there dynamics, while our model can handle them efficiently with transformer structure. 4.2.5. Beneficial to Process Large Data MVMT is designed to efficiently handle large-scale MTS data. The learnable prior allows it to capture and store diverse characteristics within the data, while the mixture-distributed latent states help model various structural patterns and non-stationary temporal dependencies across different MTSs. By incorporating probabilistic modeling in the generative process, MVMT accounts for the variability present in complex MTS data. Additionally, the transformer architecture in both the inference and memory modules enhances its ability to capture diverse patterns. The ’one-for-all’ setting of MVMT can avoid the drawback of previous methods that the parameters grow exponentially facing to large data. 5. Model Training To train the model end-to-end, we establish an objective function. This reconstruction-based approach captures the latent representation of the entire MTS. For MVMT, given the model parameters W, the marginal likelihood of the MTS dataset D is formulated asp(D∣W)=∫ma∫zp(x∣z,ma)p(z∣ma)p(ma)dmadz Consistent with VAEs, MVMT aims to optimize by maximizing the evidence lower bound (ELBO) on the log marginal likelihood. The log marginal likelihood for MVMT is then expressed as(16)logp(x)=Ez∼q(z∣ma,x) ma∼q(ma∣Ma,x)[logp(x)] Applying Bayes’ rule [39] for p(x), we incorporate the terms q(z∣ma,x), p(z∣ma), p(ma∣Ma), and q(ma∣Ma,x) to the logp(x) as(17)logp(x)=E[logp(x∣z)p(z)p(z∣x)q(z∣ma,x)q(z∣ma,x)p(z∣ma)p(z∣ma)p(ma∣Ma)p(ma∣Ma)q(ma∣Ma,x)q(ma∣Ma,x)] Organizing Equation (17) can conclude(18)logp(x)=E[logp(x∣z)]−E[logq(z∣ma,x)p(z∣ma)]−E[logq(ma∣Ma,x)p(ma∣Ma)]+E[logq(ma∣Ma,x)p(ma∣Ma)]+Elogq(z∣ma,x)p(z∣ma)+E[logp(z)p(z∣x)] Next, applying the Kullback–Leibler (KL) Divergence, defined as DKL(p∥q)=E[log(p/q)], we can further reformulate Equation (18) as(19)logp(x)=E[logp(x∣z)]−DKL(q(z∣ma,x)∥p(z∣ma))−DKL(q(ma∣Ma,x)∥p(ma∣Ma))+DKL(q(ma∣Ma,x)∥p(ma∣Ma))+DKL(q(z∣ma,x)∥p(z∣ma))+DKL(p(z)∥p(z∣x)) By applying Jensen’s inequality, we can derive the following expression for the ELBO:(20)logp(x)≥E[logp(x∣z)]−DKL(q(z∣ma,x)∥p(z∣ma))−DKL(q(ma∣Ma,x)∥p(ma∣Ma)) Drawing inspiration from beta-VAE [40], we incorporate two hyperparameters, β1>0 and β2>0, which function as temperature coefficients. These coefficients are used in a ‘warm-up’ procedure, where we initially train the model focusing solely on reconstruction error. Gradually, over the first N training epochs, the KL divergence loss is introduced, applying an additional warm-up factor that increases linearly from 0 to 1 during training. This gradual introduction helps address the issue of posterior collapse, resulting in the final optimization target:(21)L=Eq(z),q(ma)[logp(x∣z)]−β1DKL(q(z∣ma,x)∥p(z∣ma))−β2DKL(q(ma∣Ma,x)∥p(ma∣Ma)) Unlike the conventional ELBO, which primarily focuses on the expected log-likelihood for maintaining reconstruction accuracy and the KL divergence for aligning the variational distribution of z with the prior, the ELBO of MVMT learns a set of meta memory prior (M) to guide latent variable ma. The information stored in the memory units passes to z through a KL constraint between ma and z. The prior of z is obtained from the posterior probability of the ma, so an additional KL is needed to give the ma prior. As demonstrated in Algorithm 1, the model parameters are optimized using stochastic gradient descent, allowing for an end-to-end training process. The algorithm implements an autoencoding variational inference strategy, employing an upward–downward scheme, to train and fine-tune the parameters essential for the MVMT model. Leveraging a pre-processed Multivariate Time Series (MTS) training dataset, this process encompasses encoding, memory management, latent variable inference, decoding, and iterative parameter updates, ensuring comprehensive model training and optimization. Algorithm 1 Upward–Downward Autoencoding Variational Inference for MVMTInput:The pre-processed MTS training dataset D(x1:T).Parameter: The encoder parameters Ψ={Wxh0,Wh0h1,bh0h1,Wh0h′,bh0h′,W˜maμ,V˜maμ,b˜maμ,W˜maσ,V˜maσ,b˜ma,W˜zμ,V˜zμ,b˜zμ,W˜zσ,V˜zσ,b˜z}The decoder parameters Φ={WμMa,bμMa,WσMa,bσMa, Wμma,bμma, Wσma,bσma,Wμz,bμz,Wσz,bσz,}The meta memory units parameters M.Output:Parameters after training Ψ, Φ, M. 1:Set mini-banch size as N and hyperparameters β1,β2; 2:Initialize the encoder parameters Ψ, decoder parameters Φ and meta memory units parameters M; 3:while epoch do 4: Randomly select a mini-batch {x1:Ti}i=1N in D(x1:T); 5: Extract features h={h0,h1,h′} using encoder; 6: Inference the meta memory posterior q(ma) through MMA (Equation (12)); 7: Inference the latent variable z through Equation (14); 8: Input the latent variable to get the decoder result; 9: Update the parameters(Ψ, Φ, M) by Equation (21);10:end while11:return Parameters after training Ψ, Φ, M. Input: Input: The pre-processed MTS training dataset D(x1:T). The pre-processed MTS training dataset D(x1:T). Parameter: The encoder parameters Ψ={Wxh0,Wh0h1,bh0h1,Wh0h′,bh0h′,W˜maμ,V˜maμ,b˜maμ,W˜maσ,V˜maσ,b˜ma,W˜zμ,V˜zμ,b˜zμ,W˜zσ,V˜zσ,b˜z} Parameter: The encoder parameters Ψ={Wxh0,Wh0h1,bh0h1,Wh0h′,bh0h′,W˜maμ,V˜maμ,b˜maμ,W˜maσ,V˜maσ,b˜ma,W˜zμ,V˜zμ,b˜zμ,W˜zσ,V˜zσ,b˜z} The decoder parameters Φ={WμMa,bμMa,WσMa,bσMa, Wμma,bμma, Wσma,bσma,Wμz,bμz,Wσz,bσz,} The decoder parameters Φ={WμMa,bμMa,WσMa,bσMa, Wμma,bμma, Wσma,bσma,Wμz,bμz,Wσz,bσz,} The meta memory units parameters M. The meta memory units parameters M. Output: Output: Parameters after training Ψ, Φ, M. Parameters after training Ψ, Φ, M. 1:Set mini-banch size as N and hyperparameters β1,β2; Set mini-banch size as N and hyperparameters β1,β2; 2:Initialize the encoder parameters Ψ, decoder parameters Φ and meta memory units parameters M; Initialize the encoder parameters Ψ, decoder parameters Φ and meta memory units parameters M; 3:while epoch do while epoch do 4: Randomly select a mini-batch {x1:Ti}i=1N in D(x1:T); Randomly select a mini-batch {x1:Ti}i=1N in D(x1:T); 5: Extract features h={h0,h1,h′} using encoder; Extract features h={h0,h1,h′} using encoder; 6: Inference the meta memory posterior q(ma) through MMA (Equation (12)); Inference the meta memory posterior q(ma) through MMA (Equation (12)); 7: Inference the latent variable z through Equation (14); Inference the latent variable z through Equation (14); 8: Input the latent variable to get the decoder result; Input the latent variable to get the decoder result; 9: Update the parameters(Ψ, Φ, M) by Equation (21); Update the parameters(Ψ, Φ, M) by Equation (21); 10:end while end while 11:return Parameters after training Ψ, Φ, M. return Parameters after training Ψ, Φ, M. 6. Experimental Evaluation This section begins by detailing the experimental setup, followed by a comprehensive evaluation of our model through various experiments. 6.1. Experiment Setup 6.1.1. Datasets Our experiments involve five datasets: (1) SMD [1], a 5-week dataset collected from a large internet company, consisting of 38 dimensions. (2) MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite), which are publicly available NASA datasets [16] with 55 dimensions. Both datasets contain telemetry anomaly data sourced from Incident Surprise Anomaly (ISA) reports for spacecraft monitoring. (3) PSM [41], an internally collected dataset from multiple application server nodes at eBay, containing 26 dimensions. (4) SWaT [42], collected from 51 sensors monitoring a critical infrastructure system during continuous operation. (5) DND [43], a real-world multivariate KPI dataset from a major internet company in China, monitoring 12 websites with 36 KPIs per site. These websites offer various services, such as Video on Demand (VoD) and live streaming. The KPIs span roughly 1.5 months, with data points collected every 60 s. For each website, the first half of the KPIs are used for training and the second half for testing. Human operators confirmed the ground-truth anomalies in the DND testing data. Table 2 provides the basic statistical details of the datasets. 6.1.2. Baselines We compare our model with 19 baselines across various categories. For reconstruction-based models, we evaluate against InterFusion [44], BeatGAN [45], OmniAnomaly [1], and LSTM-VAE [46]. For density-estimation methods, we include DAGMM [47], MPPCACD [48], and LOF [49]. Clustering-based approaches like ITAD [50], THOC [51], and Deep-SVDD [52] are also considered. We further compare with autoregression-based models, including CL-MPPCA [53], LSTM [16], and VAR [54], as well as classic methods such as OC-SVM [55] and IsolationForest [56]. State-of-the-art deep learning models such as Anomaly Transformer [24] and InterFusion [44] are part of the evaluation, along with GmVRNN [30], a leading probabilistic model for anomaly detection, and TranAD [33], a recent anomaly detection method. 6.1.3. Implementation Details In our experiments, we implement a three-layer transformer similar to the Anomaly Transformer (without the minimax strategy from [24]), with hidden states of 128 dimensions serving as the encoder. The MLP used to map features h0, h1, and h′ also has a dimension of 128. The MMA module comprises 15 memory units, each with a dimensionality of 128 across all datasets. Both the memory prior ma and latent variable z are set to a dimension of 128. We use hyperparameters β1 and β2 at 0.1 to balance the reconstruction and KL divergence terms across all datasets. The model is optimized using the Adam optimizer with a learning rate of 0.0002 and a batch size of 256. For the threshold in the POT method, we empirically set p to 0.003. Consistent with [51], a sliding window technique is applied to obtain sub-series, with a window size of 20 across all datasets. All experiments were carried out using PyTorch [57] on an NVIDIA RTX 3090 24 GB GPU. 6.1.4. Metric and One-for-All Setting We measure the model’s performance using Precision, Recall, and F1 score, as commonly used in similar studies [1,12,14]. The F1 score, in particular, offers a balanced view by combining both precision and recall. An anomaly is classified as a true positive if any point within the ground-truth anomaly segment is identified correctly [30]. In our experiments, we split each MTS dataset into two halves: the first half for training and the second half for testing. Please note that in the one-for-one experiment, we train individual models specifically for the MTS of each device, effectively preventing any negative impact resulting from mode variations across different devices. Conversely, the one-for-all experiment poses a greater challenge. In this task, we employ a single model for training and testing across all MTS from various devices. 6.2. Main Result 6.2.1. Quantitative Comparison We assessed our model’s performance on five public datasets and one real-world dataset, benchmarking it against a range of competitive methods. The results presented in Table 3 highlight that our model surpasses reconstruction-based methods like OmniAnomaly, density-estimation techniques such as MPPCACD, and even state-of-the-art deep learning models like Anomaly Transformer. Although TranAD [33], the best baseline anomaly detection method, is the one-for-one model, MVMT still exhibits 2.14% higher performance (F1) on average. Table 3 underscores the effectiveness of MVMT in time-series anomaly detection. Our model achieves superior performance across all five datasets, significantly reducing both false alarms and missed detections compared to other methods. This highlights MVMT’s potential for real-world applications where accuracy and reliability are crucial. Although MVMT achieves the best F1 score across all datasets (Table 3), the performance margin is noticeably smaller on datasets such as DND. These datasets exhibit relatively homogeneous normal patterns and limited temporal variability, allowing existing baselines to already model the underlying dynamics effectively. In such cases, the advantage of MVMT’s meta memory mechanism is less pronounced, as the need for capturing diverse temporal concepts is reduced. This observation provides insight into scenarios where MVMT may offer smaller gains despite overall superior performance. 6.2.2. Ablation Study To evaluate the influence of each component within our framework, we carried out ablation studies. The results of these studies are summarized in Figure 5. MVMT-without-VAE refers to using a basic MLP as a decoder, serving as the baseline in our model. Transformer + VAE represents a model where we introduce a single latent variable z without incorporating the ma component. Notably, Transformer + VAE exhibits an average performance 6.61% better than MVMT-without-VAE in Figure 5, demonstrating the effectiveness of employing probabilistic modeling. The complete version of our model, MVMT, integrates the proposed MMA model and the latent variable ma. The MMA model, along with the latent variable ma, achieves the best F1 score, surpassing Transformer + VAE by an average of 8.80% in Figure 5. This result underscores the significance of the MMA module, capable of capturing and storing various temporal dependencies within MTSs. Consequently, MVMT efficiently captures multiple patterns of MTSs with a single set of parameters. Furthermore, introducing the meta memory component in our model results in additional performance enhancements. Our experiments confirm that the diverse temporal information introduced by the MMA module significantly contributes to improving MTS anomaly detection. 6.2.3. Meta Memory Attention In this paper, we aim to improve model performance by leveraging memory units to capture diverse temporal information within MTSs. Efficient retrieval of relevant memory units for various MTSs is crucial. To evaluate this, we analyzed the attention weights of the MMA, as shown in Figure 6. We selected five memory units at random and examined their attention weights. The upper part of the figure depicts MTS data with four distinct patterns, while the lower part displays a bar chart of the mean attention weights, outlined in the red box. The analysis in Figure 6 illustrates that the meta memory attention weights can be tailored to generate specific memories for different patterns in the MTS data. For instance, the yellow bar shows a relatively lower proportion for the first three patterns but becomes more prominent for the fourth pattern. In this section, we provide visual insights into the attention weights of the MMA. Figure 6 showcases the statistical properties across four different patterns of MTSs. Additionally, in Figure 7, we delve into observing subtle changes in a specific MTS pattern. For observation purposes, we randomly selected attention weights from five memory units. The upper section of the figure presents segments of the MTS data with four distinct patterns, while the lower section shows the corresponding attention weights. In contrast to Figure 6, we visualize the attention weights for individual points within each MTS. As depicted in the purple box in Figure 7, the memory attention weights of MMA exhibit four distinct cases across the four patterns of MTS. However, within each pattern, MMA finely tunes the weights corresponding to specific time timesteps. This phenomenon illustrates that MVMT not only provides different memory priors for significant pattern changes but also adjusts the memory priors’ information based on slight variations in each MTS. Regarding abnormal states, in cases where there is no corresponding information in memory, the memory attention weights generate a distinctive response, enhancing the model’s performance in detecting anomalies. 6.2.4. Information Capacity of MVMT A key contribution of this paper is the introduction of a meta variational memory module aimed at learning diverse characteristics across multiple MTSs. The MMA is designed to store and recall historical MTS data, allowing MVMT to adapt quickly to new datasets. To examine how the number of memory units, denoted as N, affects model performance, we tested different N values, as shown in Figure 8. The results indicate that increasing N from 5 to 15 enhances the model’s capacity and performance. However, when N exceeds 15, the model may begin to overfit due to the accumulation of less relevant information. 6.2.5. Adapt to New MTSs In practical settings involving MTS data, such as CDN systems, there is often a need to adapt detection models to new MTSs with limited data availability. Previous methods face challenges in achieving this adaptation, especially in the one-for-one setting, as they often necessitate training individual models for new MTSs. This approach may falter when confronted with limited observations. Our proposed MVMT trains a unified set of parameters for all MTSs and harnesses global information using the MMA module. This capability empowers MVMT to adapt to new MTSs despite limited observations. To demonstrate the efficacy of MVMT in adapting to new MTSs, we designed two new experiments to evaluate the model’s performance in scenarios with minimal data. For the experiments in this section, we allocated 80% of the MTSs as historical data and used the remaining 20% as new data. In practical settings, such as deploying a one-for-one anomaly detection model for new equipment, we might only have a limited number of MTS samples for training. Given that each sample takes about 60 s to acquire and setup time for the equipment is approximately 20 min, we trained the one-for-one models using the initial i samples (i∈1,5,10,20,200) from each new MTS and evaluated the model on the remaining samples. For the one-for-all models (GmVRNN and MVMT), parameters were initially trained on historical MTS data and later fine-tuned using the first i samples (i∈1,5,10,20,200) from each new MTS. Table 4 outlines the detection performance across methods. The results show that one-for-one models, when trained on limited data, fail to accurately capture normal patterns, resulting in reduced performance. In contrast, one-for-all models leverage previously stored information from historical MTSs, leading to a noticeable improvement over the one-for-one approaches. Additionally, to further validate the design of our methods tailored for the one-for-all setting, we conducted an additional experiment by pretraining the one-for-one models on the training data, similar to the pretraining strategy applied in the one-for-all models. As depicted in Table 4, although the one-for-one models perform relatively better due to the involvement of historical MTSs, they fail to capture and retain the diverse characteristics within multiple MTSs. In contrast, our proposed MVMT, equipped with multiple components for preserving and recalling historical information, exhibits significantly superior performance compared to the one-for-one models in this setting. 6.2.6. Qualitative Analysis Beyond quantitative analysis, we also conducted qualitative assessments. We compared the anomaly scores produced by MVMT, Anomaly Transformer, MVMT without ma, and a standard Transformer, as shown in Figure 9. Deterministic models like Transformers tend to generate more fluctuating anomaly scores, as they do not account for the inherent stochasticity in MTS data. In contrast, probabilistic models, such as MVMT without ma, provide smoother anomaly scores due to their more structured probabilistic design. However, dealing with certain situations using a simple Gaussian hidden variable remains challenging. Introducing a memory module into the anomaly detection model plays a vital role in improving performance. By incorporating diverse temporal information, the model can more accurately reconstruct normal MTS patterns, leading to enhanced detection accuracy and a lower false alarm rate. As shown in Figure 9a, the temporal diversity introduced by MMA allows MVMT to generate noticeably lower anomaly scores in normal data scenarios compared to other models. MMA’s retrieval of relevant memory is akin to referencing a vast historical dataset, helping MVMT produce smoother anomaly scores during normal conditions and more pronounced scores when anomalies occur. Figure 9b–d illustrate several cases where MVMT detects anomalies while other methods fail to do so. In (b), Transformer and MVMT without ma fail to generate sufficiently high anomaly scores, leading to detection failure. The more sophisticated model designs of Anomaly Transformer and MVMT yield higher anomaly scores, successfully detecting anomalies. In (c), when two anomalies are proximate, other models may misconstrue them as usual or ignore one, whereas our proposed model can compare memory information to differentiate between these modes using MMV. In (d), the model easily disregards inconspicuous anomalies as noise, while MVMT can sensitively detect them. 6.2.7. Time Efficiency In line with previous studies [30], we assessed the time efficiency of various methods, including our MVMT, transformer-based models like TranAD and Anomaly Transformer, and RNN-based approaches such as GmVRNN and LSTM-VAE. The results, shown in Table 5, indicate that MVMT had notably shorter training and testing times compared to the other methods when using the same GPUs. All models were able to detect anomalies in less than a second per sample, which is beneficial given the 60 s interval between data points. This suggests that these methods are well-suited for real-time anomaly detection. 6.2.8. Number of Parameters We analyzed and compared the number of model parameters across various methods, as shown in Figure 10. For this analysis, we selected LSTM-VAE, OmniAnomaly and Anomaly Transformer as baselines, representing typical RNN-based and Transformer-based methods, respectively. Notably, the parameter count of OmniAnomaly is considerably smaller than that of Anomaly Transformer due to the complexity of the transformer structure. However, with the incorporation of a mixture-distributed latent variable and a meta memory module, our proposed MVMT can efficiently account for the diverse characteristics within various MTSs. This capability allows MVMT to effectively model all MTSs using a single set of parameters, resulting in superior performance compared to previous methods, even surpassing RNN-based methods, with significantly fewer parameters. 7. Conclusions To address anomaly detection in large-scale Multivariate Time Series (MTS), we propose the Meta Variational Memory Transformer (MVMT), which integrates self-attention with a meta memory module and a memory-guided probabilistic generative framework to effectively capture, organize, and utilize temporal concepts across diverse MTSs. The upward–downward autoencoding inference scheme further enhances posterior estimation through the integration of bottom-up likelihood and top-down priors. Across five public datasets and one real-world dataset, MVMT consistently attains the top F1 performance among all compared methods. Beyond accuracy, the ability to apply a single set of parameters to heterogeneous MTS collections highlights MVMT’s practicality and scalability for real-world online anomaly detection. These results collectively demonstrate the effectiveness and applicability of MVMT in large-scale MTS environments.