On (in)consistency of M-estimators under contamination
Jens Klooster & Bent Nielsen · 2025
Read On (in)consistency of M-estimators under contamination by Jens Klooster & Bent Nielsen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09145.txt.
Introduction
Jens Klooster and Bent Nielsen present a 2025 arXiv paper on robust estimators when data are contaminated unevenly. Estimators like the median and the Huber estimator are trusted to resist bad data. Yet the authors show both turn inconsistent under asymmetric contamination: they settle on the wrong value. The Tukey estimator stays consistent. The catch is scale, since clean inference needs a consistent scale partner. The usual robust scale choices, the interquartile range and the median absolute deviation, also fail under contamination.
Catalog record
| CREATOR | Jens Klooster & Bent Nielsen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09145
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On (in)consistency of M-estimators under contamination Jens Klooster & Bent Nielsen Abstract We consider robust location-scale estimators under contamination. We show that commonly used robust estimators such as the median and the Huber estimator are inconsistent under asymmetric contamination, while the Tukey estimator is consistent. In order to make nuisance parameter free inference based on the Tukey estimator a consistent scale estimator is required. However, standard robust scale estimators such as the interquartile range and the median absolute deviation are inconsistent under contamination. We thank Mikhail Zhelonkin and Chen Zhou for helpful comments. 1 Introduction In the early 1960s, there was an increasing awareness that standard estimators for normal models may not fare well under deviations from normality. Huber, (1964) proposed maximum likelihood-type (M) estimators for location and scale and found that they are more robust to such deviations than traditional estimators. Since then, robustness means that an estimate is only distorted in a bounded way when adding arbitrary observations to the sample ( Hampel,, 1971 ) . While statistical inference theory was developed for data with infinitesimal contamination ( Heritier and Ronchetti,, 1994 ) , little theory is available for cases with more contamination. We analyze popular M-estimators for location and scale when a fixed proportion of observations is contaminated. We find that the Huber estimator for location is not consistent, whereas the Tukey estimator for location has better consistency properties. To conduct inference, the uncontaminated scale has to be estimated. Unfortunately, robust scale estimators such as the interquartile range (IQR) and the median absolute deviation (MAD) are inconsistent under contamination. Thus, valid statistical inference based on M-estimation seems to require detailed modelling of the contamination, which is typically not done in practice. Huber’s article initiated the development of a large body of robust estimators, starting with the location-scale problem and later expanding to regression and many other statistical problems. An attractive feature of Huber’s work was the idea of developing estimators that are not strongly based in statistical models. However, when thinking about inference, statistical models are useful. The lack of models is perhaps the reason that inference theory remains incomplete for the contaminated case. A major problem in the inference theory is that we need consistent estimators for both location and scale. Location and scale are entwined. Without knowing one, it is hard to get to know the other. Huber, (1964) was clearly aware of this. Huber’s \epsilon -contamination model mixes a normal distribution with an \epsilon proportion of a contamination distribution. This model has become the standard contamination model. It is well suited for analyzing the consequences of infinitesimal contamination ( Hampel et al.,, 1986 ; Heritier and Ronchetti,, 1994 ; Huber and Ronchetti,, 2009 ) ; see Avella Medina and Ronchetti, (2015) ; Yu and Yao, (2017) ; Loh, (2025) for recent reviews and Horowitz and Manski, (1995) for a discussion of identification. But, when \epsilon is not vanishing, standard scale estimators are inconsistent and asymptotic inference on the location depends on the contamination, which is usually unknown. Thus, inference for M-estimators is fraught under contamination. Another type of robustness analysis uses the breakdown point ( Hampel,, 1971 ) . The finite sample breakdown point asks how many arbitrary observations can be added to a sample without distorting the estimator unboundedly ( Donoho and Huber,, 1983 ; Huber,, 1984 ) . Location M-estimators typically have a high breakdown point. In practice, it is common to use M-estimators with a high breakdown point, which also maintain high efficiency under the normal model ( Yohai,, 1987 ; Coakley and Hettmansperger,, 1993 ; Yu and Yao,, 2017 ) . Our approach is to consider a model where a proportion of the observations is contaminated. This matches the breakdown point idea. We consider two classes of M-estimators. The first class has an unbounded objective function. It includes the median and the Huber estimator. We find that these estimators are bounded in probability when more than half of the data is uncontaminated, which attests their robustness. Yet, these estimators are typically inconsistent under contamination. The second class has a bounded objective function and includes the Tukey estimator. We derive a lower bound on the proportion of contamination ensuring that these estimators are bounded in probability and consistent under contamination. However, nuisance parameter free inference requires a consistent scale estimator. We show that robust scale estimators such as the IQR and MAD are inconsistent under contamination. These results for M-estimators are disappointing. In the discussion of the results, we point to the least trimmed squares (LTS) estimator ( Rousseeuw,, 1984 ) which fares better. We first revisit M-estimation and classical approaches to the robustness problem. Next, we introduce a new contamination model and analyse boundedness, consistency and asymptotic normality properties. This leads to new insights into the behaviour of M-estimators under contamination. The results are illustrated in a simulation study. 2 Background Suppose we have scalar observations y_{i} for i=1,\dots,n and we want to estimate a location-scale specification y_{i}=\mu+\sigma\varepsilon_{i} where \mu\in\mathbb{R} and \sigma>0 . A model for the errors \varepsilon_{i} will be specified below. 2.1 M-estimators for location with known scale The case of a known scale, \sigma=1 say, is the starting point for the M-estimation theory of Huber, (1964) . The M-estimator \hat{\mu} for location \mu minimizes the objective function \sum_{i=1}^{n}\rho\big(y_{i}-\mu\big). We focus on \rho functions that have a constant hence bounded derivative in the tails, as described in the following assumption. Note, this excludes the sample average with \rho(x)=x^{2} , which is not robust. Assumption 2.1 . The function \rho is defined on \mathbb{R} , positive apart from \rho(0)=0 , continuous, symmetric, and non-decreasing in x>0 , and there are x_{*},\rho_{*}>0 and \psi_{*}\geq 0 such that \rho(x)=\rho_{*}+\psi_{*}(|x|-x_{*}) for |x|\geq x_{*} . We distinguish between two types of estimators. Non-redescending estimators have \psi_{*}>0 and include the median with \rho(x)=|x| and the Huber estimator with \rho(x)=(x^{2}/2)\mathbf{1}_{(|x|\leq c)}+c(|x|-c/2)\mathbf{1}_{(|x|>c)} . Redescending estimators have \psi_{*}=0 as for the Tukey estimator with \rho(x)=(c^{2}/6)\{1-(1-x^{2}/c^{2})^{3}\mathbf{1}_{(|x|\leq c)}\} . The parameter c>0 is a tuning parameter that the researcher needs to set in advance. In Section 2.4 , we discuss standard choices that are used in practice. 2.2 M-estimators with unknown scale In practice the scale \sigma will be unknown. Suppose a preliminary estimator \hat{\sigma} is available. Inserting this estimator in the above objective function for the location case gives a new objective function \displaystyle R_{n}(\mu)=\sum_{i=1}^{n}\rho\left(\frac{y_{i}-\mu}{\hat{\sigma}}\right). (2.1) Note, the median computed from \rho(x)=|x| is scale equivariant and is computable without knowing the scale. It is important that the scale estimator is also robust, because otherwise the location estimator can still easily be affected by ‘outliers’. Standard robust scale estimators include the IQR and the MAD ( Huber and Ronchetti,, 2009 , Chapter 5) . Letting \hat{q}_{p} denote the p quantile of the full sample, we write the estimators as \displaystyle\hat{\sigma}_{\textsc{IQR}}=k(\hat{q}_{3/4}-\hat{q}_{1/4}),\,\hat{\sigma}_{\textsc{MAD}}=k\mathrm{med}\,\left|y_{i}-\hat{q}_{1/2}\right|. (2.2) The tuning parameters, k , must be set in advance by the researcher. Some standard choices are discussed in Section 2.4 . We note that the combination of the median with the MAD is considered a simultaneous M-estimate of location and scale ( Huber and Ronchetti,, 2009 , page 135) . 2.3 The Huber contamination model The Huber \epsilon -contamination model ( Huber,, 1964 ) has become the