Machine Learning-Based Anomaly Detection of Correlated Sensor Data: An Integrated Principal Component Analysis-Autoencoder Approach *

Tanish Baranwal, Arnab Das, Srihari Varada, Santanu Das, Mohammad R. Haider · 2025

Read Machine Learning-Based Anomaly Detection of Correlated Sensor Data: An Integrated Principal Component Analysis-Autoencoder Approach * by Tanish Baranwal, Arnab Das, Srihari Varada, Santanu Das, Mohammad R. Haider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24044.txt.

Introduction

Tanish Baranwal and four co-authors built a hybrid system to spot faults in linked sensor streams, in a 2025 paper. Sensors in factories, hospitals, and grids often measure related values, such as heat and pressure in one space. Faults in the stream can undermine whole systems. Real-time spotting is essential, yet linked streams get little study.

Their system pairs a classic summary method with a learning-based autoencoder. The summary method watches the stream and wakes the deeper model when shifts look large. Tests used both real and simulated data. The hybrid matched the deep model alone on a key quality score. It answered much faster and raised fewer false alarms. The design suits settings where computing power is tight, the authors note.

Catalog record

CREATORTanish Baranwal, Arnab Das, Srihari Varada, Santanu Das, Mohammad R. Haider
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24044

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Machine Learning-Based Anomaly Detection of Correlated Sensor Data: An Integrated Principal Component Analysis-Autoencoder Approach * Tanish Baranwal, Arnab Das, Srihari Varada, Santanu Das, Mohammad R. Haider Abstract The growing adoption of IoT systems in industries like transportation, banking, healthcare, and smart energy has increased reliance on sensor networks. However, anomalies in sensor readings can undermine system reliability, making real-time anomaly detection essential. While a large body of research addresses anomaly detection in IoT networks, few studies focus on correlated sensor data streams, such as temperature and pressure within a shared space, especially in resource-constrained environments. To address this, we propose a novel hybrid machine learning approach combining Principal Component Analysis (PCA) and Autoencoders. In this method, PCA continuously monitors sensor data and triggers the Autoencoder when significant variations are detected. This hybrid approach, validated with real-world and simulated data, shows faster response times and fewer false positives. The F1 score of the hybrid method is comparable to Autoencoder, with much faster response time which is driven by PCA. I Introduction Internet of Things (IoT)-based systems are increasingly utilized across applications such as transportation, banking, healthcare, and smart energy [ 1 , 2 ] , relying on sensor networks to collect and transmit data to a central computing resource for seamless operations and data-driven insights. However, anomalies in sensor data—caused by environmental factors, hardware malfunctions, or cyber threats—can undermine the reliability and security of these systems. Detecting and mitigating such anomalies in real-time is critical for ensuring system integrity and operational effectiveness. Existing research on IoT anomaly detection mainly focuses on network-level anomalies, such as abnormal traffic, latency, and cyber threats [ 3 , 4 , 5 ] . However, there is limited research on detecting anomalies in correlated sensor data streams, particularly in resource-constrained environments. Many IoT systems use multiple sensors to measure the same or related parameters, such as temperature and pressure. For example, modern aircraft use multiple altimeters (barometric and radar) to cross-check altitude during critical phases like landing. Current research lacks effective methods for detecting anomalies in such correlated sensor data streams, particularly when sensors are spatially separated or collect data at varying speeds or intervals, as seen in electric vehicles or industrial machines with limited processing power and memory. Traditional statistical methods [ 5 , 6 ] , like time-series analysis, are computationally intensive and require large historical datasets, making them unsuitable for real-time detection in resource-constrained environments. Machine learning (ML) and deep learning (DL) approaches [ 7 , 8 , 9 , 10 , 12 ] have shown promise in anomaly detection but often require substantial computational resources and labeled data, which may not always be available. Additionally, many ML models struggle to differentiate between true anomalies and benign variations, leading to higher false positive rates. This highlights the need for an approach that balances detection speed with computational efficiency, especially in resource-constrained IoT systems. To address these challenges, this paper proposes a novel hybrid machine learning approach combining PCA [ 13 ] and Autoencoders [ 14 ] for anomaly detection in correlated sensor data streams. PCA, a linear dimensionality reduction technique, continuously monitors sensor data for deviations with minimal computational overhead. When significant variations are detected, an Autoencoder—a deep learning model that captures nonlinear dependencies—further analyzes the data to confirm anomalies. This adaptive approach ensures real-time detection, minimizes false positives, and optimizes resource usage. The effectiveness of the proposed method is validated with both real-world and synthesized sensor data. Simulation results show that the hybrid approach reduces computational load, improves response time, and lowers false positive rates compared to standalone PCA or Autoencoder models. The paper is structured as follows: Section II discusses the motivation for our approach, Section III presents the methodology and experimental setup, Section IV provides quantitative analyses and results, and Section V concludes with key findings. II Motivation for Technical Approach The motivation behind this research is to develop a machine learning-based approach for IoT anomaly detection, addressing the challenge of high-dimensional, correlated sensor data. Among dimensionality reduction techniques, PCA, UMAP [ 15 ] , TSNE [ 16 ] , and Autoencoders are very popular and widely used. A comparative overview is provided in Table I , where ‘n’ refers to the training dataset size, ‘input’ is the window size, and ‘output’ denotes the latent space dimension. TABLE I: Comparison of Dimensionality Reduction Techniques Technique Purpose Computational Complexity Retraining Needed t-SNE 2D/3D Visualization O( n^{2} ) Yes UMAP Clustering O( n^{2} ) Yes PCA Linear Compression O( \text{input}\times\text{output} ) No Autoencoder Nonlinear Compression Constant No This study focuses on PCA and Autoencoders, which are particularly useful for feature extraction in high-dimensional spaces [ 17 , 18 ] , while TSNE and UMAP are mainly used for 2D/3D visualization. TSNE and UMAP however require retraining for new data points, making them less efficient for real-time updates. Thus, the comparative analysis in Section IV compares PCA, Autoencoder, and the proposed hybrid model against UMAP. The Autoencoder used in this research is based on a Long Short-Term Memory (LSTM) [ 18 ] encoder-decoder architecture, ideal for processing temporal data and capturing long-term dependencies. To evaluate these techniques, we apply them to a sliding window of 100 historical time samples, reducing the dataset’s dimensionality. We then compute a Euclidean distance matrix, containing the pairwise distances between sensors in matrix form, to quantify pairwise relationships between encoded sensor data. Significant deviations in these distances indicate anomalies. A faulty sensor will show abnormal distances with all other sensors, while functioning sensors exhibit only minor deviations. This approach ensures robust anomaly detection by leveraging both linear and nonlinear dimensionality reduction techniques. III Methods The Methods section is structured into three distinct subsections, each corresponding to the respective sections in the RESULTS. First, we outline the methodology for assessing the effectiveness of PCA and Autoencoders in anomaly detection. Second, we present the approach for the quantitative analysis of PCA and Autoencoder performance. Finally, we introduce the proposed hybrid model and describe the evaluation framework employed to assess its effectiveness. III-A Methods for characterizing the effectiveness of PCA vs. Autoencoder in Anomaly Detection We used sensor data from Intel Berkeley Research Labs [ 19 ] , collected from 54 sensors deployed between February 28th and April 5th, 2004. Mica2Dot sensors [ 20 ] with weatherboards recorded humidity, temperature, light, and voltage every 31 seconds using the TinyDB [ 21 ] in-network query system on the TinyOS platform [ 22 ] . Data from 4 temperature sensors in the same lab area was selected, with readings taken over 2000 time steps. At the 1000th time step, Sensor 4 was deliberately triggered to manifest anomalous behavior. Two anomaly models were studied: (a) an anomaly resulting from a sudden shift in the mean of readings of a sensor in a group of n sensors, where n=4 in this experiment (b) an anomaly resulting from one of the sensors in a group exhibiting ‘erasure error’ (certain readings being set to zero). In anomaly scenario (a), sensor four’s readings were shifted after the 1000th time step by adding a scalar value to its last 1000 readings, altering the mean, while keeping the standard deviation and distribution unchanged. The other three sensors remained normal over

Read the complete text