When lookout sees crackle: Anomaly detection via kernel density estimation

Rob J Hyndman, Sevvandi Kandanaarachchi, Katharine Turner · 2026

Read When lookout sees crackle: Anomaly detection via kernel density estimation by Rob J Hyndman, Sevvandi Kandanaarachchi, Katharine Turner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22636.txt.

Introduction

Hyndman, Kandanaarachchi, and Turner present an updated version of the lookout anomaly detector. The method uses kernel density estimates to flag points that sit in low-density regions. Its bandwidth comes from Rips death diameters, a tool from shape analysis. The authors prove the density estimator is consistent. They also describe a multivariate scaling step that is robust and efficient. Tests on varied examples show the updated method beats the earlier version. The work gives the tool a firmer theory base for practical use.

Catalog record

CREATORRob J Hyndman, Sevvandi Kandanaarachchi, Katharine Turner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67516
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22636

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When lookout sees crackle: Anomaly detection via kernel density estimation Rob J Hyndman, Sevvandi Kandanaarachchi, Katharine Turner Abstract We present an updated version of lookout – an algorithm for detecting anomalies using kernel density estimates with bandwidth based on Rips death diameters – with theoretical guarantees. The kernel density estimator for updated lookout is shown to be consistent, and the proposed multivariate scaling is robust and efficient. We show our updated algorithm performs better than the previous version on diverse examples. keywords Bandwidth selection, Extreme value theory, Generalized Pareto distribution, Outlier detection, Persistent homology, Topological data analysis 1 Introduction Anomaly detection is used in many application domains such as credit card fraud and scam detection ( 14 ) . 12 proposed the lookout algorithm for anomaly detection using a kernel density estimate with bandwidth based on persistence homology. Anomalies are identified by fitting a Generalized Pareto Distribution (GPD) to the negative logarithm of the smallest leave-one-out kernel density estimates, and declaring points with low GPD probabilities as anomalies. In this paper, we explore some of the theoretical properties of the lookout algorithm, particularly the choice of bandwidth based on the Rips death diameters, and the consistency of the kernel density estimates produced by this choice. This leads us to propose a modified lookout algorithm using a different bandwidth, although still based on the Rips death diameters. The modified algorithm improves the performance and robustness of the original algorithm, under a broader range of conditions. First we introduce some background material, before describing both the original lookout algorithm, and our proposed modifications. 1.1 Anomalies and surprisals There is no broadly accepted definition for an anomaly. 8 defines an anomaly or outlier as an observation which deviates so much from other observations as to arouse suspicion it was generated by a different mechanism. 2 define an anomaly or outlier as an observation (or a subset of observations) which appears to be inconsistent with the remainder of that set of data. Both these definitions agree that anomalies are rare and different from other observations. Usually that is applied by fitting some (conditional) probability distribution to the set of observations, and considering whether each observation can be considered a likely random draw from the probability distribution. We formalize this in the following definition. Definition 1 (Anomalies) . Given a set of observations \bm{y}_{i}\in\mathbb{R}^{m} for i\in\{1,\dots,n\} , and a probability density function f , we define observation \bm{y}_{i} as anomalous if \text{Pr}(f(Y)\leq f(\bm{y}_{i}))<\alpha, (1) where Y is a random variable with density f , and \alpha>0 is a chosen threshold. Definition 2 (Surprisals) . Given a probability density function f , the surprisal of an observation \bm{y} drawn from the density f is defined as -\log f(\bm{y}) . These are better known as “log scores” in statistics ( 6 ) , but we prefer the term “surprisals” from information theory ( 18 ) , as it provides a measure of how surprising an observation is, given the probability density function f . Specifically, it takes large values for observations that are unlikely under f . Let S=-\log f(\bm{Y}) be the random variable representing the surprisal of a random vector \bm{Y} drawn from f , and let G(s)=\text{Pr}(S\leq s) be its distribution function. Then the surprisal probability of observation \bm{y}_{i} is given by p_{i}=1-G(s_{i}^{-}), where s_{i}=-\log f(\bm{y}_{i}) , and G(s_{i}^{-})=\lim_{t\uparrow s_{i}}G(t) is the left limit of G at s_{i} . An observation is an anomaly if and only if p_{i}<\alpha . 1.2 Extreme Value Theory Consider n independent and identically distributed random variables X_{1},\dots,X_{n} having a distribution function G , with their maximum M_{n}=\max\{X_{1},\dots,X_{n}\} . Extreme Value Theory (EVT) analyses and characterizes the behavior of M_{n} . One of the fundamental results of EVT is the Fisher-Tippet-Gnedenko Theorem. We restate a version of it from 4 . Theorem 3 (Fisher-Tippett-Gnedenko) . If there exist sequences of constants \{a_{n}>0\} and \{b_{n}\} such that P\left\{(M_{n}-b_{n})/a_{n}\leq z\right\}\rightarrow H(z)\quad\text{as}\quad n\to\infty, for a non-degenerate distribution function H , then H is a member of the Generalized Extreme Value (GEV) family H(z)=\exp\left\{-\left[1+\xi\left(\frac{z-\mu}{\sigma}\right)\right]^{-1/\xi}\right\}\,, (2) where the domain of the function is \{z:1+\xi(z-\mu)/\sigma>0\} and the parameters \mu,\xi\in\mathbb{R} and \sigma>0 . The parameters \mu , \sigma and \xi are called the location, scale and shape parameters respectively. The shape parameter \xi determines the behavior of the tails and differentiates between the Gumbel, Fréchet and Weibull distributions: a Fréchet distribution is obtained when \xi>0 , a Weibull distribution is obtained when \xi<0 , and a Gumbel distribution is obtained in the limit as \xi\rightarrow 0 . Importantly, when F has a finite upper bound, then \xi<0 . A second result from EVT that we will use is Pickands theorem ( 16 ) , here restated from ( 4 , Theorem 4.1) . Theorem 4 (Pickands) . Let X_{1},\dots,X_{n} be a sequence of independent random variables with a common distribution function G , and let M_{n}=\max\{X_{1},\ldots,X_{n}\} . Suppose G satisfies Theorem 3 , so that for large n , P\{M_{n}\leq z\}\approx H(z) , where H(z)=\exp\left\{-\left[1+\xi\left(\frac{z-\mu}{\sigma}\right)\right]^{-1/\xi}\right\}\,, for some \mu,\xi\in\mathbb{R} and \sigma>0 . Then for large enough u , \text{Pr}(X\leq u+x\mid X>u)=1-\left(1+\frac{\xi x}{\sigma_{u}}\right)^{-1/\xi}\,, (3) where x>0 , (1+\xi x)/\sigma_{u}>0 , and \sigma_{u}=\sigma+\xi(u-\mu) . The family of distributions defined by equation ( 3 ) is called the Generalized Pareto Distribution (GPD). The shape parameter \xi is the same in both GPD and GEV distributions. Furthermore, the GEV parameters can be recovered from the GPD formulation by estimating a GPD on all observations above some threshold. 1.3 Kernel Density Estimation Given a set of observations \bm{y}_{i}\in\mathbb{R}^{m} for i\in\{1,\dots,n\} , a kernel density estimate is defined as ( 3 ) \hat{f_{n}}(\bm{y})=\frac{1}{n}\sum_{i=1}^{n}|\bm{H}|^{-1/2}K\big(\bm{H}^{-1/2}(\bm{y}-\bm{y}_{i})\big), (4) where K is a square-integrable spherically-symmetric function, bounded below by 0, with a finite second-order moment and unit integral, and \bm{H} is a symmetric m\times m positive-definite matrix. The expected L_{2} distance between the estimator \hat{f}_{n} and the true density f is given by the mean integrated squared error (MISE) defined as \text{MISE}(\hat{f}_{n})=\text{E}\left[\int\left(\hat{f_{n}}(\bm{u})-f(\bm{u})\right)^{2}d\bm{u}\right]. (5) We say that \hat{f} is a consistent estimator of f if \text{MISE}(\hat{f}_{n})\rightarrow 0 as n\rightarrow\infty . We will need some conditions on the density function, the kernel and the bandwidth matrix to ensure consistency of the kernel density estimator. These are given below. A1 The density function f is square integrable and twice differentiable, with all of its second-order partial derivatives bounded, continuous and square integrable. A2 The kernel K is a square-integrable, spherically symmetric function on \mathbb{R}^{m} that satisfies \int K(\bm{u})d\bm{u}=1 , \int\bm{u}K(\bm{u})d\bm{u}=\bm{0} and \int\bm{u}\bm{u}^{T}K(\bm{u})d\bm{u}=\mu_{2}(K)\bm{I}_{m} for some \mu_{2}(K)>0 . A3 The bandwidth matrix \bm{H} is a sequence of symmetric positive-definite matrices such that \bm{H}\rightarrow\bm{O} and n|\bm{H}|^{1/2}\rightarrow\infty as n\rightarrow\infty . The following result is provided by 3 . Theorem 5 (Consistency) . Let \bm{y}_{1},\dots,\bm{y}_{n} be a sequence of independent random variables from a distribution on \mathbb{R}^{m} that satisfies A1. Then under conditions A2 and A3, \lim_{n\rightarrow\infty}\text{E}\left[\int\left(\hat{f_{n}}(\bm{u})-f(\bm{u})\right)^{2}d\bm{u}\right]=0. We consider matrices of the form \bm{H}=h_{n}\bm{I}_{m} , where h_{n}\in\mathbb{R} is a function of the sample \{\bm{y}_{1},\dots,\bm{y}_{n}\} . We have |\bm{H}|=h_{n}^{m} so n|\bm{H}|^{1/2}\rightarrow\infty precisely when n(h_{n})^{m/2}\to\infty . This leads to our definition of an admissible bandwidth as these are the bandwidths which will determine consistent estimators. Definition 6 (Admissible bandwidth) . We say that h_{n} is an admissible bandwidth for samples in \mathbb{R}^{m} if h_{n}\rightarrow 0 and n(h_{n})^{m/2}\rightarrow\infty as n\rightarrow\infty

Read the complete text