Robust Local Polynomial Regression with Similarity Kernels

Yaniv Shulman · 2025

Read Robust Local Polynomial Regression with Similarity Kernels by Yaniv Shulman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10729.txt.

Introduction

Yaniv Shulman proposed in 2025 a more robust way to fit curves to local data. Local polynomial regression fits simple curves to small slices of data, weighted by closeness, which makes it flexible and easy to use. Its weakness is that outliers and extreme points can pull the fit far off course.

His method builds the weights from both the input and the output values, using local density to discount odd points. The result is less pull from outliers without a slow, repeated fitting process. Analysis in the paper quantifies the small bias this weighting adds. Tests showed lower bias than a classic robust method, and results close to the standard method on clean data.

Catalog record

CREATORYaniv Shulman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-10729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Local Polynomial Regression with Similarity Kernels Yaniv Shulman Abstract Local Polynomial Regression (LPR) is a widely used nonparametric method for modeling complex relationships due to its flexibility and simplicity. It estimates a regression function by fitting low-degree polynomials to localized subsets of the data, weighted by proximity. However, traditional LPR is sensitive to outliers and high-leverage points, which can significantly affect estimation accuracy. This paper revisits the kernel function used to compute regression weights and proposes a novel framework that incorporates both predictor and response variables in the weighting mechanism. The focus of this work is a conditional density kernel that robustly estimates weights by mitigating the influence of outliers through localized density estimation. The proposed method is implemented in Python and is publicly available at https://github.com/yaniv-shulman/rsklpr . The population analysis quantifies the bias induced by density-based robust weighting, and the reported experiments show lower empirical bias than iterative robust LOWESS while remaining competitive with standard LOWESS. This advancement provides a promising extension to traditional LPR, opening new possibilities for robust regression applications. 1 Introduction Local polynomial regression (LPR) is a powerful and flexible statistical technique that has gained increasing popularity in recent years due to its ability to model complex relationships between variables. Local polynomial regression generalizes the polynomial regression and moving average methods by fitting a low-degree polynomial to a nearest neighbors subset of the data at the location. The polynomial is fitted using weighted ordinary least-squares, giving more weight to nearby points and less weight to points farther away. The value of the regression function for the point is then obtained by evaluating the fitted local polynomial using the predictor variable value for that data point. Local linear and local polynomial estimators have favorable boundary behavior and strong minimax-efficiency properties under standard smoothness conditions ( Fan, 1993 ; Fan and Gijbels, 1996 ) . The biggest advantage of this class of methods is not requiring a prior specification of a function i.e. a parameterized model. Instead, only a small number of hyperparameters need to be specified such as the type of kernel, a smoothing parameter and the degree of the local polynomial. The method is therefore suitable for modeling complex processes such as non-linear relationships, or complex dependencies for which no theoretical models exist. These two advantages, combined with the simplicity of the method, makes it one of the most attractive of the modern regression methods for applications that fit the general framework of least-squares regression but have a complex deterministic structure. Local polynomial regression incorporates the notion of proximity in two ways. The first is that a smooth function can be reasonably approximated in a local neighborhood by a simple function such as a linear or low order polynomial. The second is the assumption that nearby points carry more importance in the calculation of a simple local approximation or alternatively, that closer points are more likely to interact in simpler ways than far away points. This is achieved by a kernel which produces values that diminish as the distance between the explanatory variables increase to model stronger relationship between closer points. The relevant literature on classical and robust local regression is summarized in Section 2 . LPR is susceptible to outliers, high leverage points and functions with discontinuities in their derivative, which can adversely affect least-squares based local estimates ( Čížek and Sadıkoğlu, 2020 ) . The main contribution of this paper is to revisit the kernel used to produce regression weights. The simple yet effective idea is to generalize the kernel such that both the predictor and the response are used to calculate weights. Within this framework, a non-negative kernel based on conditional density estimation is proposed that assigns robust weights to mitigate the adverse effect of outliers in the local neighborhood. Note the proposed framework does not preclude the use of robust loss functions, robust bandwidth selectors and standardization techniques. In addition the method is implemented in the Python programming language and is made publicly available. In the experiments below, the proposed method substantially improves over iterative robust LOWESS in RMSE and bias diagnostics while remaining competitive with standard LOWESS, however robust LOWESS can still have lower MAE and median absolute error in the public benchmark. The remainder of the paper is organized as follows: Section 2 summarizes related work. In Section 3 , a brief overview of the mathematical formulation of local polynomial regression is provided. In Section 4 , a framework for robust weights and the specific conditional density kernel are proposed. Section 5 provides an analysis of the estimator and a discussion of its properties. In Section 6 , implementation notes and experimental results are provided. Finally, in Section 7 , the paper concludes with directions for future research. 2 Related Work Local polynomial regression and LOWESS/LOESS estimate regression functions by fitting low-degree polynomials in local neighborhoods defined by predictor-space proximity. Classical kernel-regression and LOWESS/LOESS references establish the local-smoothing baseline, while local linear and local polynomial regression show why local polynomial fitting is often preferred over local averaging, especially for design adaptivity and reduced boundary bias ( Nadaraya, 1964 ; Watson, 1964 ; Cleveland, 1979 ; Cleveland and Devlin, 1988 ; Fan, 1993 ) . Robust variants of local regression typically modify the fitting criterion or add residual-based reweighting so that large residuals have limited influence on the local fit. Cleveland’s robust LOWESS is the classical example of iterative residual reweighting, while local M -estimation and adaptive robust local-polynomial methods provide related theoretical frameworks ( Cleveland, 1979 ; Härdle and Gasser, 1984 ; Chichignoud and Lederer, 2014 ) ; broader reviews are given in ( Čížek and Sadıkoğlu, 2020 ; Salibian-Barrera, 2023 ) . The closest adjacent line to the proposed approach is the literature that uses response-space or conditional-density information in local fitting. Local modal regression introduces a response-space kernel and estimates conditional modes rather than conditional means, yielding robustness under skewed, heavy-tailed, or multimodal conditional distributions ( Yao et al., 2012 ) . In parallel, the conditional-density literature studies nonparametric estimators of f_{Y\mid X}(y\mid x) and associated bandwidth-selection methods ( Hyndman et al., 1996 ; Bashtannyk and Hyndman, 2001 ; Hall et al., 2004 ; Cattaneo et al., 2024 ) . RSKLPR sits between these lines: unlike residual-robust LOWESS, it uses estimated local response density rather than fitted residuals to form robustness weights; unlike local modal regression, it does not estimate the conditional mode, but retains a local-polynomial squared-error target whose oracle form is a density-weighted conditional mean. More robust density estimators, including divergence-based or RKHS-based robust KDE variants, are natural extensions left for future work ( Basu et al., 1998 ; Kim and Scott, 2012 ) . 3 Local Polynomial Regression This section provides a brief overview of local polynomial regression and establishes the notation subsequently used. We adopt the following standing assumptions: the training data \mathcal{D}_{T}=\{(X_{i},Y_{i})\}_{i=1}^{T} are an i.i.d. sample from a continuous joint density f_{X,Y} ; the error terms \epsilon_{i} satisfy \mathbb{E}[\epsilon_{i}|X_{i}]=0 and \mathbb{E}[\epsilon_{i}^{2}|X_{i}]=\sigma^{2}(X_{i})<\infty ; the density of the predictors f_{X}(x) is positive in the region of interest; and any kernel function K is a non-negative, symmetric probability density function with finite second moments. Let (X,Y) be a random pair and \mathcal{D}_{T}=\{(X_{i},Y_{i})\}_{i=1}^{T}\subseteq\mathcal{D} be a training

Read the complete text