Adaptable High-Dimensional Change Point Detection via Ridge Regularization

Haoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu · 2026

Read Adaptable High-Dimensional Change Point Detection via Ridge Regularization by Haoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24838.txt.

Introduction

Haoran Li leads a method for spotting many shifts in the mean of high-dimensional data streams. The target is dense shifts where dimension rivals sample size. The team builds ridge-regularized CUSUM statistics on an adaptable ridge-regularized Hotelling test. Ridge regularization steadies covariance handling while adapting to the true covariance shape. Scanning multi-scale adjacent segments needs no prior count or place for shifts. Pooling across ridge settings by taking the maximum test value adds adaptability, with limiting laws proved under null and local alternatives.

Catalog record

CREATORHaoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE140639
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptable High-Dimensional Change Point Detection via Ridge Regularization Haoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu, Haoran Li, Haotian Xu Abstract We study the problem of detecting multiple change points in the mean vectors of an independent sequence of high-dimensional observations. We propose a family of ridge-regularized CUSUM statistics built upon the adaptable ridge-regularized Hotelling’s T 2 T^{2} test of Li et al. (Ann. Statist. 48 (2020) 1815–1847). The proposed tests are designed for dense alternatives in the high-dimensional regime where the dimension is comparable to the sample size. By introducing ridge regularization, the procedure achieves a stable form of sample covariance normalization and attains adaptability with respect to the underlying population covariance structure. We derive the limiting distributions of the proposed statistics under mild conditions, both under the null hypothesis and under a class of local alternatives. We further develop a principled framework for selecting the regularization parameter by maximizing asymptotic power. Extensive simulation studies demonstrate that the proposed tests compare favorably with a wide range of existing methods across diverse settings. The performance of the proposed test procedure is illustrated through an application to a panel of daily log-returns from S&P 500 constituents spanning 2007–2025. and ?? Department of Mathematics and Statistics, Auburn University , ??; ?? 1 Introduction Change point detection is a classical problem in statistics with applications across a wide range of disciplines, including econometrics, signal processing, and climate science. In this setting, one observes an ordered sequence of random vectors X_{1},X_{2},\ldots,X_{n}\in\mathbb{R}^{p} that are independent and identically distributed, except for possible structural changes in their mean vectors. In this paper, we consider the model X_{j}=\mu_{j}+\Sigma_{p}^{1/2}Z_{j},\qquad j=1,\ldots,n, (1.1) where \{\mu_{j}\} is a sequence of mean vectors, \{Z_{j}\}\in\mathbb{R}^{p} are i.i.d. random vectors with independent, mean-zero, unit-variance entries, and \Sigma_{p} is a general covariance matrix. Under the null hypothesis of no change point, the mean vectors are constant over time, that is, H_{0}:\ \mu_{j}\equiv\mu,\qquad j=1,\ldots,n. We are interested in testing H_{0} against the alternative that the mean vector undergoes a finite number of abrupt changes: H_{a}:\ \mu_{1}=\cdots=\mu_{k_{1}}\neq\mu_{k_{1}+1}=\cdots=\mu_{k_{s}}\neq\mu_{k_{s}+1}=\cdots=\mu_{n}, (1.2) where s\geq 1 and \{k_{j}\}_{j=1}^{s}\in[1,n) denote the (unknown) number and locations of change points. We henceforth refer to H_{a} as the multiple change point (MC) alternative. In the classical regime where the dimension p is fixed and small, change point detection has been extensively studied; see 9 and 2 for comprehensive reviews of classical methods. To motivate the ideas, consider the case s=1 . That is, we are interested in testing H_{0} against the alternative H_{a}^{\rm SC}:\mu_{1}=\cdots=\mu_{k_{1}}\neq\mu_{k_{1}+1}=\cdots=\mu_{n}, which we henceforth refer to as the single change-point (SC) alternative. For simplicity, assume that the covariance matrix \Sigma_{p} is non-singular and known. The classical CUSUM approach compares the sample means of the two segments \{X_{1},\ldots,X_{k}\} and \{X_{k+1},\ldots,X_{n}\} , normalized by \Sigma_{p}^{-1/2} , over all possible change point locations k . Specifically, the CUSUM test statistic takes the form \max_{1\leq k\leq n-1}C_{0}(k)^{T}\Sigma_{p}^{-1}C_{0}(k), where C_{0}(k)=\frac{1}{n-k}\sum_{j=k+1}^{n}X_{j}-\frac{1}{k}\sum_{j=1}^{k}X_{j} denotes the difference between the sample means of the two segments separated at k . In the low-dimensional regime and under the null hypothesis H_{0} , \Sigma_{p} can be consistently estimated by the pooled sample covariance matrix S_{n}=\frac{1}{n}\sum_{j=1}^{n}(X_{j}-\bar{X})(X_{j}-\bar{X})^{T},\qquad\text{with }~\bar{X}=\frac{1}{n}\sum_{j=1}^{n}X_{j}. (1.3) The CUSUM procedure can therefore be implemented by replacing \Sigma_{p} with S_{n} . Importantly, under H_{0} , the distribution of the resulting test statistic, whether normalized by \Sigma_{p}^{-1} or by S_{n}^{-1} , is invariant with respect to \Sigma_{p} , which highlights the robustness of classical methods to the unknown covariance structure. High-dimensional data are now ubiquitous in many scientific fields. When the dimension p is comparable to or larger than the sample size n , the performance of traditional methods deteriorates substantially. First, accurate estimation of the precision matrix \Sigma_{p}^{-1} becomes challenging in this regime, particularly in the absence of structural assumptions, such as low-rank structure or sparsity. Second, the accumulation of noise across coordinates fundamentally alters the limiting distributions of classical test statistics. There is a substantial literature on change point testing and estimation for the mean of high-dimensional data. Broadly speaking, existing methods can be classified into three main categories. The first class focuses on detecting sparse mean changes, where only a small subset of components of the mean vector undergo change over time. These methods typically rely on the \ell_{\infty} -norm of the mean difference C_{0}(k) and include, among others, 16 and 31 . The second class targets dense alternatives, in which the signal may be distributed across many or all coordinates. For example, 32 and 33 consider tests based on the \ell_{2} -norm of C_{0}(k) without normalization by \Sigma_{p}^{-1} , while 28 proposes a modified procedure motivated by U-statistic techniques. The third class comprises methods that do not presuppose specific alternative structures, such as projection-based approaches 29 and 12 . Other notable works include 26 ; 24 ; 17 ; 8 ; 22 ; 34 . While most existing methods largely overlook the role of covariance scaling and directly utilize the norm of C_{0}(k) , we propose a new class of test statistics that achieve data-driven covariance normalization by scaling the observations with a ridge-regularized sample covariance estimator. Specifically, we transform the observations according to (S_{n}+\lambda I_{p})^{-1/2}X_{j},\qquad j=1,\ldots,n, where S_{n} denotes the pooled sample covariance matrix and \lambda>0 is a tuning parameter. Targeting dense alternatives, we construct test statistics based on the \ell_{2} -norm of the resulting scaled observations; see ( 2.3 ). Our methodology builds on the ridge-regularized Hotelling’s T^{2} (RHT) test for comparing two high-dimensional mean vectors, as proposed in 7 and further developed in 20 . Importantly, the resulting test statistics are rotation-invariant , in the sense that their values are unchanged under arbitrary orthogonal transformations of the data, that is, when X_{j} is replaced by Q^{T}X_{j} for any orthogonal matrix Q^{T} . This invariance is a desirable property when not much prior knowledge about \Sigma_{p} and the alternative is available. We work in a high-dimensional regime where the dimension p is comparable to the sample size n , in the sense that p/n\to\gamma\in(0,\infty) . Detecting dense alternatives in this regime is well motivated by a variety of applications. For example, KEGG pathway–level analyses of DNA copy number alteration (CNA) patterns often involve a few hundred genes, and substantial empirical evidence suggests that such alterations in a cancer cell occur across a large fraction of genes rather than being confined to a few isolated components ( 13 ) . In climate science, spatio-temporal fields are often represented using tens to a few hundred spatial locations or principal components, and regime changes typically induce coherent mean shifts across a large fraction of these components ( 10 ) . In neuroimaging, brain activity is frequently summarized over a few hundred regions of interest, and transitions between cognitive or behavioral states are associated with widespread changes across many regions, resulting in dense mean shifts in moderately high-dimensional signals ( 19 ) . Technically, in the regime p/n\to\gamma , fundamental results from Random Matrix Theory (RMT) indicate that, although the pooled sample covariance matrix S_{n} is a biased estimator of \Sigma_{p} under H_{0} , it exhibits stable limiting spectral behavior under mild conditions, determined solely by the spectral distribution of \Sigma_{p} and the limiting ratio \gamma . The introduction of a ridge

Read the complete text