Why Empirical p p -Values Are Not Uniform: Reference Samples, Dependence, and PIT Backtesting
Jakub Lis · 2026
Read Why Empirical p p -Values Are Not Uniform: Reference Samples, Dependence, and PIT Backtesting by Jakub Lis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16221.txt.
Introduction
Lis shows why empirical p-values are not uniform, even though exact probability integral transforms are uniform under correct specification. Finite-sample estimation changes the statistics: shared reference samples push results toward a two-sample Kolmogorov-Smirnov regime, while rolling windows induce autocorrelation and variance suppression. Shared-sample and rolling setups thus break classical one-sample uniformity tests. Backtests based on such transforms need revised calibration that accounts for the two-stage sampling setup.
Catalog record
| CREATOR | Jakub Lis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24677 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Empirical p p -Values Are Not Uniform: Reference Samples, Dependence, and PIT Backtesting Jakub Lis Abstract Probability integral transforms (PITs) and empirical p p -values are widely used to assess the calibration of predictive distributions. While exact PIT values are uniformly distributed under correct model specification, practical implementations rely on empirical estimates constructed from finite samples. We show that this estimation step fundamentally alters the statistical structure of the problem. In particular, common-sample and rolling-window implementations introduce dependence and variance distortions that invalidate classical one-sample uniformity tests. When empirical percentiles are conditioned on a shared reference sample, the resulting statistics converge towards a two-sample Kolmogorov–Smirnov regime, while rolling windows induce autocorrelation and variance suppression. Our findings indicate that treating empirical percentiles as independent uniform draws can distort statistical inference and that backtesting procedures based on PITs require revised calibration methods accounting for the underlying two-stage sampling structure. Keywords: Model validation, backtesting, probability integral transform, Kolmogorov–Smirnov test, empirical processes. JEL Codes: C12, C14, G32. 1 Introduction Assessing the calibration and stability of predictive distributions is a central task in statistical model validation, with applications ranging from probabilistic forecasting to financial risk management. In practice, the probability integral transform (PIT) is widely used as a diagnostic tool for detecting misspecification and distributional change. Procedures based on testing the uniformity of empirical percentiles have therefore become common across a range of applied settings, including ensemble forecasting and financial backtesting. One particularly prominent application arises in financial regulation. In particular, the European Central Bank recommends the use of PIT-based procedures in the validation of internal risk models [ 1 ] . The motivation for this approach is commonly formulated as follows (see [ 1 ] ): If X is a continuous random variable with cumulative distribution function F_{X} , then the transformed variable F_{X}(X) is uniformly distributed on [0,1] . Conversely, if the transformed distribution deviates from uniformity, then F_{X} cannot represent the true distribution of X . This reasoning originates from classical theoretical developments beginning with Rosenblatt’s note [ 2 ] and has been extended in applied statistical literature, including financial econometrics [ 3 , 4 ] and probabilistic forecasting [ 5 ] . Closely related principles also appear in the use of rank histograms for assessing calibration in ensemble forecasting; see, for example, [ 6 ] . The corresponding methodology has become widely adopted in practical model validation and regulatory backtesting frameworks. In this note, we highlight a discrepancy between the theoretical foundation of the test and its practical implementation. While the theory concerns exact probability integral transforms, practical implementations necessarily rely on empirical estimates based on finite reference samples. As a consequence, the quantities subjected to uniformity testing in practice are not uniformly distributed, even under correct model specification. This structural mismatch affects the distribution of standard test statistics and calls into question the direct applicability of classical goodness-of-fit procedures in this setting. The remainder of this paper is organised as follows. Section 2 establishes the theoretical framework of the PIT, defines the interpolation methodology used in regulatory practice, and outlines the standard goodness-of-fit testing suite. Section 3 analyses the distortion introduced by a single fixed reference sample, demonstrating how the test converges to a two-sample problem. In Section 4 , we show that the independent-reference framework typical of rank histograms mitigates the systematic impact of estimation error, as the noise associated with finite reference samples averages out rather than contributing coherently through a common conditioning set. Section 5 explores the more complex case of rolling-window implementations and the resulting autocorrelation and variance suppression effects. We conclude in Section 6 . 2 Tests and Tools 2.1 Test Specification The proposed testing procedure consists of two main steps. First, each new observation is assigned an empirical percentile based on a reference sample. Second, the resulting set of p -values is tested under the null hypothesis that it follows the uniform distribution on [0,1] . The first component estimates the percentile of an observation x relative to a reference sample y_{1},\dots,y_{n} , assuming that x and all y_{i} are independently drawn from the same probability distribution f . The exact percentile is given by p(x)=\int_{-\infty}^{x}f(u)\,\mathrm{d}u. (1) For X drawn from this distribution, the random variable p(X) is uniformly distributed on [0,1] with cumulative distribution function F_{U} , a property that underlies the standard PIT formulation. However, the true distribution is unknown, and this quantity must be estimated empirically. Let y_{(1)}<\cdots<y_{(n)} denote the ordered reference sample. We assign to these observations empirical percentiles \tilde{p}_{i}=i/(n+1) . Intermediate values are then obtained by interpolation. For concreteness and simulations, we consider the interpolation methodology prescribed in ECB internal model guidance [ 1 ] , under which the empirical percentile estimate is defined as follows: \hat{p}(x)=\begin{cases}\displaystyle\frac{y_{(k+1)}-x}{y_{(k+1)}-y_{(k)}}\,\tilde{p}_{k}+\frac{x-y_{(k)}}{y_{(k+1)}-y_{(k)}}\,\tilde{p}_{k+1},&\text{if }y_{(k)}\leq x\leq y_{(k+1)},\quad k=1,\ldots,n-1,\\ \displaystyle\frac{\left(\frac{\tilde{p}_{1}}{1-\tilde{p}_{1}}\right)^{\frac{x}{y_{(1)}}}}{1+\left(\frac{\tilde{p}_{1}}{1-\tilde{p}_{1}}\right)^{\frac{x}{y_{(1)}}}},&\text{if }x<y_{(1)},\\ \displaystyle 1-\frac{\left(\frac{1-\tilde{p}_{n}}{\tilde{p}_{n}}\right)^{\frac{y_{(n)}}{x}}}{1+\left(\frac{1-\tilde{p}_{n}}{\tilde{p}_{n}}\right)^{\frac{y_{(n)}}{x}}},&\text{if }x>y_{(n)}.\end{cases} (2) The second step consists of testing whether the resulting empirical values \{\hat{p}_{i}\} follow a uniform distribution on [0,1] , typically using a Kolmogorov–Smirnov (KS) test. Under the null hypothesis, these values are commonly treated as independent draws from the uniform distribution. Rejection of the null is interpreted as evidence that the evaluated observations and reference sample do not originate from the same distribution. 2.2 Test of Uniformity The empirical cumulative distribution function forms the basis of standard goodness-of-fit procedures. Among these, the KS test is the conceptually simplest, frequently used and, in our context, analytically tractable. However, the concerns we highlight are general and arise in any goodness-of-fit test, including the Anderson–Darling and Cramér–von Mises procedures. The key theoretical result underlying these methods is Donsker’s theorem; see [ 7 ] . Since the probability integral transform p(x)=F(x) is strictly increasing, it is natural to express the problem on the unit interval by writing x=F^{-1}(p) and working with p\in[0,1] . In this representation, the underlying distribution is uniform. Donsker’s theorem then implies that \sqrt{m}\bigl(F_{m}(p)-p\bigr)\Rightarrow B^{0}(p), (3) where B^{0} denotes a Brownian bridge. Figure 1 illustrates the validity of Donsker’s theorem in finite samples and provides a reference for the graphical analyses presented below. It is based on 1000 Monte Carlo samples of size n drawn from the uniform distribution. For each replication, we compute the order statistics of the sample. For each fixed rank i , we then compute the empirical mean and standard deviation of y_{(i)} across replications. As the sample size n increases, the standard deviation of y_{(i)} converges to zero, reflecting the vanishing variability of order statistics in large samples. However, after rescaling by \sqrt{n} , the variance profile across i converges to that of a Brownian bridge, that is, proportional to t(1-t) for t=i/(n+1) . In particular, fluctuations are smallest near the boundaries and largest in the centre of the distribution, reflecting the characteristic shape of the Gaussian bridge. Figure 1: Illustration of Donsker’s theorem in finite samples. As the sample size increases, the dispersion of the order statistics decreases, consistent with the Brownian bridge approximation. After appropriate rescaling, a universal pattern emerges. This result underlies most classical goodness-of-fit procedures, in particular the KS test, which is based on the maximal deviation between the empirical and reference distribution functions. In the present setting, this corresponds to \sup_{p\in[0,1]}\left|F_{n}(p)-p\right|. Under the null hypothesis, Donsker’s theorem implies that \sqrt{n}\,\sup_{p\in[0,1]}\left|F_{n}(p)-p\right|\;\Rightarrow\;\sup_{p\in[0,1]}|B^{0}(p)|. Equivalently, the