Robust variance estimators in application to segmentation of measurement data distorted by impulsive and non-Gaussian noise

Justyna Witulska, Anna Zaleska, Natalia Kremzer-Osiadacz, Agnieszka Wyłomańska, Ireneusz Jabłoński · 2025

Read Robust variance estimators in application to segmentation of measurement data distorted by impulsive and non-Gaussian noise by Justyna Witulska, Anna Zaleska, Natalia Kremzer-Osiadacz, Agnieszka Wyłomańska, Ireneusz Jabłoński (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10275.txt.

Introduction

Five authors build sound spread gauges for jagged data in 2025, aimed at split points where a system shifts state. Sharp spikes and non-plain spread break old gauges, which need a true spread value to work. Their offline path grows on the classic break test frame but drops that need.

Fake data from two spread forms and real sets from money, machine, and health fields test the path. In the hardest case, state shift error ran 20 times lower than the classic path. Self-drive data tag work is one use in view.

Catalog record

CREATORJustyna Witulska, Anna Zaleska, Natalia Kremzer-Osiadacz, Agnieszka Wyłomańska, Ireneusz Jabłoński
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10275

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust variance estimators in application to segmentation of measurement data distorted by impulsive and non-Gaussian noise Justyna Witulska, Anna Zaleska, Natalia Kremzer-Osiadacz, Agnieszka Wyłomańska, Ireneusz Jabłoński Abstract The paper algorithmizes the problem of regime change point identification for data measured in a system exhibiting impulsive behaviors. This is a fundamental challenge for annotation of measurement data relevant, e.g., for designing data-driven autonomous systems. The contribution consists in the formulation of an offline robust methodology based on the classical approach for structural break detection. The problem of data segmentation is considered in the context of scale change, which physically can be translated into the occurrence of a critical event that reorganizes the system structure. The main advantage of our approach is that it does not require the existence of a variance of the data distribution. The efficiency has been evaluated for simulated data from two distributions and for real-world datasets measured in financial, mechanical, and medical systems. Simulation studies show that in the most challenging case, the error in estimating regime change is 20 times smaller for robust approach compared to the classical one. Journal: Measurement 1 Introduction Measurement science plays a key role in automating the monitoring of a complex system and management of its resources, which is relevant to modern industry (e.g., Industry 4.0/5.0), medicine (including the 4.0/5.0 medicine concept), business operations, environmental monitoring, realization of the smart city conception, etc. [ 1 , 2 , 3 ] . Developments in measurement methodologies and instrumentation have revealed that Gaussian processes are not the only mechanism that governs noisy processes, and even the temporal evolution of a complex system can emerge from physical rules far beyond Gaussianity, such as non-Gaussian quantum sensing, non-Gaussian communication, etc. [ 4 , 5 ] . On the other hand, mathematicians have been exploring the abstract space of non-Gaussianity [ 6 , 7 ] , providing more and more powerful methods and tools for physicists and domain-expert engineers to characterize that class of systems and processes. The signal processing domain has been looking for a robust solution to handle (automatically) a diversity of patterns represented in experimentally measured data at one or multiple sensor nodes, adapting classical transform schemes, including signal filtering, segmentation, decomposition, prediction, etc. [ 8 ] . Starting from offline and univariate analysis, one can even find algorithms for online exploration of complex data patterns, including filtering and prediction of heavy-tailed components [ 9 , 10 ] . Given the characteristics of the process (Gaussian/non-Gaussian/mixed, low/significant dynamics, low/high Signal-to-Noise ratio, etc.) and the resources available for measurement action and signal processing (e.g. one/multiple sensor nodes, limited computing power and memory like in edge sensing [ 11 ] ) serial operating with collected samples - e.g. filtering, segmentation, prediction - can be inefficient, and in dynamic autonomous systems we should be able to reliably segment raw complex data biased with significant impulse and heavy-tailed distortions. In this paper we discuss the problem of offline segmentation (estimation of change point) of non-Gaussian data. From a mathematical point of view, this issue can be considered as the problem of "splitting" of random sample (independent identically distributed observations) into two samples of the same probabilistic properties. The main focus is on the case where the separate segments (samples) differ on scales. However, the proposed methodology can be considered as a general problem, i.e. when some characteristics of the data change over time. Thus, in our research, we assume that a random sample X_{1},...,X_{N} corresponds to the following model: \displaystyle X_{i}\overset{d}{=}\begin{cases}X^{(1)}\sim F(\bar{p},\gamma_{1}),\quad\text{for }i<n^{*},\\ X^{(2)}\sim F(\bar{p},\gamma_{2})\quad\text{for }i\geq n^{*},\end{cases} (1) where n^{*} is a change point, F represents distribution of random variables X_{1},...,X_{N} , \bar{p} is vector of parameters, and \gamma_{1},\gamma_{2} are the scales such that \gamma_{1}\neq\gamma_{2} . As the main focus is on the non-Gaussian data, the problem of segmentation seems to be more challenging in comparison to the classical approaches, where Gaussian (or any finite-variance distribution) is assumed. In the literature, one can find various approaches dedicated to data segmentation. We refer the reader to Section 2 , where the state of the art is presented. The main goal of our research is to provide a relatively easy, effective, and intuitive method of segmenting non-Gaussian data with a possible infinite-variance distribution. Here, we develop two segmentation procedures based on cumulative sums of squares (CSS) that are designated for data with finite variance, namely Iterative Cumulative Sums of Squares [ 12 ] and the quantile method utilizing the ordinary least squares technique [ 13 ] considered in the literature as the classical approaches for segmenting samples with changing variance. The CSS after a simple transformation can be considered as a statistic that estimates the scale of the distribution of the considered random sample. However, when the data exhibit impulsive heavy-tailed behavior, the CSS statistic may be an ineffective estimator of the scale (variance) as it is sensitive for large observations occurring in the time series. As a consequence, the specific behavior of CSS statistics resulting from the scale change may be disturbed. In Fig. 1 we present the schematic diagram that illustrates the considered problem. When the data exhibits impulsive behavior, the classical statistic (here CSS) may not indicate the change point precisely and the algorithms based on it may be inefficient. Thus, we propose robust versions of the classical techniques using robust estimators of scale parameters in the classical algorithms for data segmentation. Despite the proposed methodology is demonstrated for two robust estimators of scale, it can be considered as universal and any other known robust estimator could be applied for the general framework. Figure 1 : Schematic diagram illustrating the discussed problem of change point detection of heavy-tailed impulsive data. One can list three major advantages of the proposed methodology. First, the proposed method is nonparametric which means that no assumptions about data distribution are needed. Moreover, the new approach does not require the existence of variance of the considered distribution. Finally, our method is an effective and simple adaptation of well-known techniques with a clear interpretation, and thus, it can be easily adopted in real environments. The efficiency of the proposed approach is verified for simulated data from two different distributions, namely \alpha- stable distribution [ 14 ] and a mixture Gaussian model [ 15 ] . Both distributions belong to the class of non-Gaussian distributions; however, they exhibit different probabilistic properties. The \alpha -stable distribution can be viewed as an extension of the Gaussian distribution [ 16 ] . The class of \alpha- stable distributions has found widespread applications in various fields, with common use in financial markets [ 17 ] , physics [ 18 ] , and medicine (specifically heartbeats) [ 19 ] . The mixture Gaussian model is defined as the sum of the Gaussian distribution with additive outliers defined as uniformly distributed random variables that appear in the Gaussian noise with a given probability. The corresponding random variables have finite variance; however, the probability of extreme values is much higher than in the classical Gaussian distribution. This distribution also found interesting applications, including medicine [ 20 ] , epidemic spreading [ 21 ] , financial investments, and pricing [ 22 ] . The extensive simulation studies are supported by the analysis

Read the complete text