Joint robust estimation
Xiang Li, Jun S. Liu, Qiang Sun, Lihu Xu · 2025
Read Joint robust estimation by Xiang Li, Jun S. Liu, Qiang Sun, Lihu Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11054.txt.
Introduction
Xiang Li, Jun Liu, and their colleagues build a tuning-free method for estimating a target value and its noise spread together when data have heavy tails. Heavy tails mean rare extreme values can wreck standard estimates, and most robust tools need the spread in advance or a tuning choice. Their Catoni-type frame solves two linked equations at once, with no tuning step.
The frame is worked out for mean estimation and for two forms of regression. The authors prove error bounds that match what an oracle that knew the spread could do, under only a mild moment assumption. A new proof toolkit based on a mirror theorem handles the non-convex equations.
Catalog record
| CREATOR | Xiang Li, Jun S. Liu, Qiang Sun, Lihu Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 141 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11054
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Joint robust estimation Xiang Li, Jun S. Liu, Qiang Sun, Lihu Xu Abstract . We introduce a joint robust estimation method for three parametric statistical models involving heavy-tailed data: mean estimation, linear regression, and ℓ 2 \ell_{2} -penalized linear regression, where both the trend parameters and the error variance are unknown. Our approach is based on solving two coupled Catoni-type equations—one for estimating the target trend parameters and the other for estimating the error variance. Notably, this joint estimation strategy cannot be derived by minimizing a single loss function involving both the trend and the variance parameters. The proposed method offers four key advantages: (i) The length of the resulting ( 1 − ϵ ) (1-\epsilon) -confidence interval scales as log ϵ − 1 \sqrt{\log\epsilon^{-1}} , matching the order achieved by classical estimators for subgaussian data; (ii) It is tuning-free, eliminating the need for separate variance estimation; (iii) It allows flexible selection of the Catoni-type functions tailored to the data characteristics; (iv) It delivers superior performance for high-variance data, thanks to the explicit inclusion of the variance term in the denominators of both equations. We rigorously establish the consistency and asymptotic efficiency of the proposed joint robust estimators using new analytical techniques. Specifically, the coupled equations are inherently complex, rendering the theoretical analysis of their solutions challenging. To address this challenge, we employ the Poincaré–Miranda Theorem to show that the solutions lie within certain geometric regions, such as cylinders or cones, centered around the true parameter values. This methodology is of independent interest and can be extended to solve other statistical problems. 1991 Mathematics Subject Classification Primary 62G35, 62F35; Secondary 62J07, 62H12 1. Introduction A growing body of empirical evidence indicates that many datasets deviate significantly from Gaussian or sub-Gaussian behavior and instead exhibit pronounced heavy tails; for example, see [ 14 ] for financial data, [ 10 ] for complex network data, and [ 42 ] for gene expression data. Unlike sub-Gaussian distributions, heavy-tailed data often contain a substantial number of outliers, causing many classical methods to perform poorly. This has motivated sustained interest in robust methodologies that retain essentially sub-Gaussian-type deviation guarantees even when the data themselves are non-sub-Gaussian (see, e.g., [ 6 , 37 ] ). Robust methods provide a foundational approach for handling outliers, making them versatile across diverse data structures, models, and application domains. Their applicability is not limited to independent observations; they have also been successfully adapted to dependent settings, including various time series contexts ( [ 9 , 7 , 19 , 41 , 44 ] ). Likewise, robustness principles extend beyond simple models and have proven effective in high-dimensional regression ( [ 12 , 27 ] ) as well as in matrix and tensor estimation ( [ 1 , 17 , 25 , 30 ] ). Consequently, robust estimation has been widely adopted in numerous fields such as machine learning ( [ 3 , 18 , 26 , 32 ] ) and finance ( [ 16 , 34 ] ), among others. For a comprehensive treatment of the theoretical foundations and methodological developments in robust statistics, the reader is referred to the survey and monographic expositions in [ 28 , 25 ] . 1.1. Literature review The concept of robust estimation was first introduced by Huber in the 1960s. In [ 20 , 21 ] , Huber proposed a new loss function \ell_{\tau} to replace the classical square loss: \displaystyle\ell_{\tau}(x):=\begin{cases}\frac{1}{2}x^{2},&|x|\leq\tau,\\ \tau|x|-\frac{\tau^{2}}{2},&|x|>\tau,\end{cases} where \tau>0 is the robustification parameter to be tuned. This function is known as Huber’s loss function, and the corresponding estimation procedure is called Huber’s estimation. The main idea of Huber’s estimation is to reduce the influence of outliers whose magnitudes are beyond \tau . The choice of \tau is critical for balancing bias (or and robustness. A large \tau retains information from extreme data points, resulting in smaller bias but weaker robustness, whereas a small \tau suppresses such information, enhancing robustness at the cost of increased bias. Asymptotic properties of the Huber estimator have been well established (see, e.g., [ 21 , 29 , 46 ] and references therein). Regarding the selection of \tau , Huber later suggested setting \tau=1.345\sigma based on the 95\% asymptotic efficiency criterion in [ 22 ] , where \sigma is the standard deviation. However, Sun, Zhou, and Fan [ 37 ] showed through a theoretical analysis that, in linear regression models, the optimal non-asymptotic performance requires choosing \tau adaptively based on sample size, dimension, and moments. Further research on Huber-type robust estimation methods can be found in [ 33 , 43 , 44 , 47 ] and the references therein. In his seminal work [ 6 ] , Catoni introduced a novel non-asymptotic deviation analysis for robust mean estimation under heavy-tailed data, providing key insights into the balance of robustness and bias. More precisely, when the data have finite variance, Catoni proposed a class of non-decreasing influence functions \psi_{1} satisfying (1.1) \displaystyle-\log\left(1-x+\frac{|x|^{2}}{2}\right)\leq\psi_{1}(x)\leq\log\left(1+x+\frac{|x|^{2}}{2}\right). and proposed his robust mean estimator as the following: \widehat{\theta} is defined as the solution of the following equation (1.2) \displaystyle\sum_{i=1}^{n}\psi_{1}\left(\alpha(X_{i}-\theta)\right)=0, where \alpha\propto\sigma\sqrt{\tfrac{\log(\epsilon^{-1})}{n}} is a robustification parameter that balances bias and robustness, analogous to \tau in the Huber loss. According to [ 6 , Proposition 2.4] , the (1-\epsilon) -confidence interval of this estimator has length on the order of O\big(\sigma\sqrt{{\log(\epsilon^{-1})}/{n}}\big) , matching that of the classical sample mean under sub-Gaussian data. In contrast to Huber’s estimator, Catoni’s method ( 1.2 ) allows flexibility in the choice of \psi_{1} . On one hand, the narrowest choice, \displaystyle\psi_{1}(x)=\begin{cases}-\operatorname{sign}(x)\log\left(1-|x|+\frac{x^{2}}{2}\right),&|x|\leq 1,\\ \operatorname{sign}(x)\log(2),&|x|>1,\end{cases} exhibits behavior similar to Huber’s estimator. On the other hand, the widest choice, \displaystyle\psi_{1}(x)=\begin{cases}\log\left(1+x+\frac{x^{2}}{2}\right),&x\geq 0,\\ -\log\left(1-x+\frac{x^{2}}{2}\right),&x<0,\end{cases} preserves information from large-magnitude data at a logarithmic scale. This flexibility enables practitioners to select \psi_{1} between these extremes according to the problem at hand. Catoni’s approach has been widely applied in diverse models and scenarios involving heavy-tailed data (see, e.g., [ 4 , 13 , 30 , 24 , 11 , 37 , 41 , 43 ] ). For distributions with only a finite \beta -th moment, where \beta\in(1,2) , Chen et al. [ 8 ] extended Catoni’s idea by replacing ( 1.1 ) with (1.3) \displaystyle-\log\left(1-x+\frac{|x|^{\beta}}{\beta}\right)\leq\psi_{2}(x)\leq\log\left(1+x+\frac{|x|^{\beta}}{\beta}\right), thereby extending the applicability of Catoni-type estimators to distributions with finite \beta -th moments. In practice, both Huber’s and Catoni’s estimators require the selection of robustification parameters that depend on the variance \sigma^{2} and the sample size. For details on adaptive methods, see [ 37 , 45 ] . Since the variance \sigma^{2} is usually unknown, it must be estimated from the data. Catoni [ 6 ] employed the Lepski method, while Sun et al. [ 37 ] used the sample variance combined with cross-validation. However, for heavy-tailed or high-dimensional problems, neither method is efficient or reliable. In the high-dimensional setting, Sun and Zhang [ 36 ] introduced a novel tuning-free approach by proposing a new loss function for Lasso estimation that simultaneously estimates the trend and the variance parameters. This work inspired the development of the square-root Lasso [ 2 ] . For further research along this line, see [ 5 , 23 , 31 ] and references therein. In the heavy-tailed setting, Sun [ 35 ] proposed a tuning-free method based on a new pseudo-Huber loss, which