Calibration Prediction Interval for Non-parametric Regression and Neural Networks
Kejin Wu [email protected], Dimitris N. Politis [email protected] · 2025
Read Calibration Prediction Interval for Non-parametric Regression and Neural Networks by Kejin Wu [email protected], Dimitris N. Politis [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02735.txt.
Introduction
Kejin Wu and Dimitris N. Politis wrote a 2025 statistics paper on prediction intervals. The setting is regression when the usual normal, linear premises fail. A point forecast alone hides its own error. Old-style intervals built on the t-distribution collapse without those premises.
Their calibration interval leans on flexible estimators instead. Deep neural networks or kernel methods supply the fits. The range can fold in fit error too. Many methods ignore that error source. Theory shows the range hits its promised coverage rate as samples grow. That holds under mild rules. Kernel versions carry a high-probability coverage guarantee at large samples. Network versions can work with finite samples too, under stated rules. Tests and two real datasets back the method.
Catalog record
| CREATOR | Kejin Wu [email protected], Dimitris N. Politis [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 123 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02735
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Calibration Prediction Interval for Non-parametric Regression and Neural Networks Kejin Wu [email protected], Dimitris N. Politis [email protected] Abstract Accurate conditional prediction in the regression setting plays an important role in many real-world problems. Typically, a point prediction often falls short since no attempt is made to quantify the prediction accuracy. Classically, under the normality and linearity assumptions, the Prediction Interval (PI) for the response variable can be determined routinely based on the t t distribution. Unfortunately, these two assumptions are rarely met in practice. To fully avoid these two conditions, we develop a so-called calibration PI (cPI) which leverages estimations by Deep Neural Networks (DNN) or kernel methods. Moreover, the cPI can be easily adjusted to capture the estimation variability within the prediction procedure, which is a crucial error source often ignored in practice. Under regular assumptions, we verify that our cPI has an asymptotically valid coverage rate. We also demonstrate that cPI based on the kernel method ensures a coverage rate with a high probability when the sample size is large. Besides, with several conditions, the cPI based on DNN works even with finite samples. A comprehensive simulation study supports the usefulness of cPI, and the convincing performance of cPI with a short sample is confirmed with two empirical datasets. † † shortheadings: Calibration Prediction Interval / Wu and Politis † † firstpage: 1 † † editor: My editor keywords prediction interval, deep neural networks, kernel estimation, uncertainty quantification, coverage guarantee 1 Introduction Accurate prediction conditional on a regressor plays an important role in many real-world applications. Throughout this work, the conditional prediction stands for a complete inference in the regression context, i.e., the point prediction and the corresponding Prediction Interval (PI) for a response variable Y\in\mathbb{R} given some explanatory variable X_{f}\in\mathbb{R}^{d} . Ideally, if the Cumulative Distribution Function (CDF) F_{Y|X_{f}} is known, we have many candidates for the point prediction, e.g., mean and median which are optimal w.r.t. L^{2} and L^{1} loss criteria, respectively. For PI, we also have various choices, e.g., PI with 1-\alpha confidence level symmetric around the mean or one having a minimal length. Unfortunately, F_{Y|X_{f}} is usually unknown in practice. Therefore, we need to estimate the oracle PI and ensure that the resulting PI is at least asymptotically valid. The asymptotic validity means the coverage rate of PI converges to the nominal level as the sample size approaches infinity. Typically, under the normality and linearity assumptions, the PI for Y possesses an analytic form that involves the t distribution and an appropriate standard error term. However, these two assumptions eliminate the possibility of data being non-linear and non-normal. We first propose a non-parametric framework to build PI so that the linearity condition can be eluded. For many real-world problems, it is reasonable to assume that the response variable Y has a normal distribution given corresponding X_{f} human heights, test scores, etc. In this special case, a 1-\alpha oracle symmetric PI can be approximated by \left[\hat{\mu}_{Y|X_{f}}-z_{\alpha/2}\sqrt{\hat{\kappa}_{Y|X_{f}}-\hat{\mu}^{2}_{Y|X_{f}}}\penalty\ ,\penalty\ \hat{\mu}_{Y|X_{f}}+z_{\alpha/2}\sqrt{\hat{\kappa}_{Y|X_{f}}-\hat{\mu}^{2}_{Y|X_{f}}}\right]; (1) z_{\alpha/2} is the right-tail \alpha/2 quantile value of a standard normal random variable; \hat{\kappa}_{Y|X_{f}} is an estimator of \mathbb{E}(Y^{2}|X_{f}) ; \hat{\mu}_{Y|X_{f}} is an estimator of \mathbb{E}(Y|X_{f}) . We should notice that there is no parametric form being assumed about the underlying data-generating model, so PI ( 1 ) is compatible with linear/non-linear data. Moreover, it is asymptotically valid if \hat{\kappa}_{Y|X_{f}} and \hat{\mu}_{Y|X_{f}} are consistent estimators under the normality assumption. Throughout this paper, we take PI ( 1 ) to be the benchmark method. Another non-parametric way to construct PI relies on \widehat{F}_{Y|X_{f}}(\cdot) which is an estimation of the function F_{Y|X_{f}}(\cdot) . Then, a PI can be constructed as follows: \left[\widehat{F}^{-1}_{Y|X_{f}}(\alpha/2);\widehat{F}^{-1}_{Y|X_{f}}(1-\alpha/2)\right]; (2) The PI ( 2 ) can be derived from the oracle equal-tailed PI with the true CDF F_{Y|X_{f}}(\cdot) being replaced by the estimated one \widehat{F}_{Y|X_{f}}(\cdot) . In case that \widehat{F}_{Y|X_{f}}(\cdot) is not continuous and strictly increasing w.r.t. y so that the quantiles are not well-defined, we define \widehat{F}^{-1}_{Y|X_{f}}(\alpha)=\inf\{y:\widehat{F}_{Y|X_{f}}(y)\geq\alpha\} . Under the assumption that \widehat{F}_{Y|X_{f}}(\cdot) converges to F_{Y|X_{f}}(\cdot) consistently at \alpha and 1-\alpha for all X_{f}=x_{f} and y , it is easy to verify PI ( 2 ) is asymptotically valid with a 1-\alpha confidence level when F_{Y|X_{f}} is continuous at F^{-1}_{Y|X_{f}}(\alpha/2) and F^{-1}_{Y|X_{f}}(1-\alpha/2) . A classical candidate for \widehat{F}_{Y|X_{f}}(\cdot) is based on kernel estimation. Then, linearity and normality assumptions are not necessary anymore. In addition, the quantile regression approach can be an alternative that suffers less from the curse of dimensionality compared to kernel methods when d is large. Unfortunately, PI ( 2 ) usually undercovers Y given X_{f} inevitably with finite samples since the estimation variability inherent from \widehat{F}_{Y|X_{f}}(\cdot) is not captured. By our simulation studies, it turns out that the PI ( 1 ) undercovers Y even with the normality assumption and a large sample size. This finding highlights the necessity of capturing estimation variability in the PI; otherwise, undercoverage issues will prevail. To correct the undercoverage of PI due to uncaptured estimation uncertainties, Politis 2015 defined a so-called pertinent PI (PPI) which can capture the estimation variability in finite sample cases, but did not provide the coverage guarantee. Later, Wang and Politis 2021 developed the PPI with standard CDF estimators in the context of regression. It was revealed that the PPI is more accurate and stable in contrast to PI ( 2 ) and conformal prediction of Lei et al. 2018 ; Wu and Politis 2024 further constructed PPI for regression with Deep Neural Networks (DNN) and showed it is superior than other two deep generative methods Zhou et al. 2023 and Liu et al. 2021 . According to folk wisdom and empirical/theoretical evidence, DNN estimators suffer less from the curse of dimensionality than kernel estimators. Thus, it is appealing to utilize the DNN in the prediction procedure. Nevertheless, the above-mentioned DNN-based PI depends on a tuning parameter severely, which renders unavoidable complexity in practice. In this paper, we propose a calibration approach to make prediction intervals with either DNN or kernel-based estimators. We should mention that the word “calibration” in our paper is different from the process of adjusting the predicted probabilities of a model so that they better reflect the true likelihood of an event; see Guo et al. 2017 for a related discussion. More precisely, our calibration method stands for a specifically designed manipulation to extend the PI to capture the estimation variability so that the undercoverage issue can be compensated for. For the DNN-based approach, we consider the standard fully connected forward DNN with ReLU activation functions (abbreviated to DNN hereafter) whose structure is determined by the dimension of input, the depth of the hidden layer and the width for each layer. Our calibration method can be incorporated with DNN without a crucial parameter-tuning step. Although the number of grid points is a required hyperparameter in the setup procedure of calibration PI (cPI), the choice of this number does not affect the performance of PI too much compared to the tuning parameter required in deep generative methods; this fact is revealed by real-data studies later. In addition, this calibration approach is easily adaptive to parallel computation so that the computational time is not necessarily longer than building a single PI ( 1 ) or ( 2 ). In short,