Evaluating deep learning models for fault diagnosis of a rotating machinery with epistemic and aleatoric uncertainty

Reza Jalayer, Masoud Jalayer, Andrea Mor, Carlotta Orsenigo, Carlo Vercellis · 2024

Read Evaluating deep learning models for fault diagnosis of a rotating machinery with epistemic and aleatoric uncertainty by Reza Jalayer, Masoud Jalayer, Andrea Mor, Carlotta Orsenigo, Carlo Vercellis (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18980.txt.

Introduction

Reza Jalayer and co-authors compare deep learning systems that diagnose faults in rotating machines while reporting their own uncertainty, in a paper from 2024. A diagnosis is safer when the model can say it is unsure. Doubt comes in two kinds: from unseen fault types and from plain noise in the data.

The team benchmarked the leading design families, including dropout sampling, Bayesian networks, and deep ensembles. Two cutoff rules decided when a reading counted as out of place. Against unseen faults, all models caught a large share of the odd data. Ensembles did best under either cutoff rule. Under noise, the picture was more mixed. The findings guide engineers who must field fault systems they can trust.

Catalog record

CREATORReza Jalayer, Masoud Jalayer, Andrea Mor, Carlotta Orsenigo, Carlo Vercellis
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating deep learning models for fault diagnosis of a rotating machinery with epistemic and aleatoric uncertainty Reza Jalayer, Masoud Jalayer, Andrea Mor, Carlotta Orsenigo, Carlo Vercellis Abstract Uncertainty-aware deep learning (DL) models recently gained attention in fault diagnosis as a way to promote the reliable detection of faults when out-of-distribution (OOD) data arise from unseen faults (epistemic uncertainty) or the presence of noise (aleatoric uncertainty). In this paper, we present the first comprehensive comparative study of state-of-the-art uncertainty-aware DL architectures for fault diagnosis in rotating machinery, where different scenarios affected by epistemic uncertainty and different types of aleatoric uncertainty are investigated. The selected architectures include sampling by dropout, Bayesian neural networks, and deep ensembles. Moreover, to distinguish between in-distribution and OOD data in the different scenarios two uncertainty thresholds, one of which is introduced in this paper, are alternatively applied. Our empirical findings offer guidance to practitioners and researchers who have to deploy real-world uncertainty-aware fault diagnosis systems. In particular, they reveal that, in the presence of epistemic uncertainty, all DL models are capable of effectively detecting, on average, a substantial portion of OOD data across all the scenarios. However, deep ensemble models show superior performance, independently of the uncertainty threshold used for discrimination. In the presence of aleatoric uncertainty, the noise level plays an important role. Specifically, low noise levels hinder the models’ ability to effectively detect OOD data. Even in this case, however, deep ensemble models exhibit a milder degradation in performance, dominating the others. These achievements, combined with their shorter inference time, make deep ensemble architectures the preferred choice. [orcid=0000-0003-3440-5658, [email protected], ] [orcid=0000-0001-8013-8613, [email protected], ] [orcid=0000-0001-6131-7229, [email protected], ] [orcid=0000-0001-8688-414X, [email protected], ] [orcid=0000-0002-5020-3688, [email protected], ] keywords Fault Diagnosis ,Uncertainty ,Trustworthy AI ,Industry 4.0 ,Deep Learning ,Out of Distribution Data † † address: Department of Management, Economics and Industrial Engineering, Politecnico di Milano, Via Lambruschini 24/b, 20156, Milan, Italy † † address: Department of Materials and Mechanical Engineering, University of Turku, Vesilinnantie 5, Turku, 20014, Finland † † venue: ArXiv 1 Introduction Rotating machines, such as motors, turbines, and generators, are indispensable components in multiple sectors, including power generation, manufacturing, and transportation. They are crucial for the efficient operation of these industries, providing the required mechanical power and enabling the transformation of energy from one form to another. However, like any complex equipment, rotating machines are prone to faults and failures, which can lead to significant disruptions, financial losses, and safety hazards. For this reason, the development and implementation of effective and intelligent fault diagnosis techniques attracted considerable research interest in recent years Lei et al. 2020 . In the context of rotating machines, deep learning (DL)-based fault diagnosis plays a prominent role, due to the ability of DL models to extract intricate patterns and features from extensive datasets Liu et al. 2018 . In fact, unlike conventional machine learning methods that rely heavily on hand-crafted feature engineering, DL algorithms can automatically learn and uncover concealed fault indicators Zhang et al. 2020 . This characteristic enables the detection and classification of faults with improved accuracy and reliability Jalayer et al. 2021 . By harnessing the power of DL, fault diagnosis becomes more effective, thereby improving the overall performance of the rotating machinery Hoang and Kang 2019 . This improvement contributes to increased productivity, leads to substantial cost savings in industrial operations Shojaeinasab et al. 2022 , and paves the way for transformative advances in the field. With their effectiveness proven, most DL-based methods proposed in the literature are applied under the assumption that the data available for training and the data collected for future predictions share the same distribution. This assumption, however, might be flawed in practice, as future examples could deviate from the distribution of the data used for training. These examples are usually referred to as out-of-distribution (OOD) data Ovadia et al. 2019 . The discrepancy between historic (in-distribution) and future potential OOD data introduces uncertainty in the prediction process led by the DL models, thereby affecting their fault detection accuracy. This uncertainty can be categorized into two main types, i.e., epistemic, often called knowledge uncertainty Abdar et al. 2021 , and aleatoric, also referred to as data uncertainty. Epistemic uncertainty stems from the lack of knowledge and often affects DL models trained with limited data Li et al. 2020 , such as data that do not cover exhaustively all the classes the examples might belong to. This results in a classification model that over-misclassifies data, as it is unaware of all the possible classes. Aleatoric uncertainty, on the other hand, is due to the inherent variability in the data, which is often caused by uncontrollable factors like noise or measurement error. Because of their nature, all the aforementioned cases occur with high frequency. Consequently, it is relevant to develop DL algorithms that are aware of the potential uncertainty in the prediction task, and are therefore capable of discriminating between in-distribution (ID) and OOD data. To deal with epistemic and aleatoric uncertainty, some DL architectures have been proposed. Traditional methods, such as Monte Carlo (MC) dropout, are based on sampling to drop nodes, or weights, in the neural network while predicting the class of future data. By performing multiple predictions for the same example, sampling-based approaches generate a distribution of the output over the known classes, instead of a single outcome typical of deterministic models. This distribution is then exploited to quantify the degree of uncertainty affecting the classification task Gal and Ghahramani 2016 ; McClure and Kriegeskorte 2017 ; Amini et al. 2018 ; Zhou et al. 2023 . More recent approaches are based either on Bayesian neural networks (BNNs) or on ensemble DL techniques Abdar et al. 2021 , called deep ensemble. In particular, BNNs recently received great interest for industrial maintenance applications, where they have been successfully adopted to address uncertainty in the prognosis of the remaining useful life Kim and Liu 2020 ; Li et al. 2020 ; Mazaev et al. 2021 ; Caceres et al. 2021 and in the condition monitoring of fault diagnosis Sun et al. 2020 ; Sajedi and Liang 2021 ; Qi et al. 2022 ; Zhou et al. 2023 ; Moradi et al. 2022 ; Zhou et al. 2022 ; Wu et al. 2020 ; Xiao et al. 2023 ; Feng et al. 2024 . BNNs rely on a distribution of the neural network weights instead of a set of unique values. As a result, repeated predictions for the same example generate different outcomes, giving rise to a distribution over the classes. Similar to the previous approach, this distribution is used to compute an estimate of the underlying uncertainty Jospin et al. 2022 ; Swiatkowski et al. 2020 ; Gal and Ghahramani 2015 ; Blei et al. 2017 ; Salimans et al. 2015 . Finally, deep ensemble methods exploit multiple deep neural networks which are initialized and trained in parallel Lakshminarayanan et al. 2017 ; Hu et al. 2019 ; Jain et al. 2020 . In this case, the distribution of the predicted class values, used to evaluate the level of uncertainty, is obtained by collecting the outcomes generated for a given example by all the models Abdar et al. 2021 ; Zhang et al. 2020 . Deep

Read the complete text