Alternative statistical inference for the first normalized incomplete moment

Jiannan Lu, Peng Ding and Anqi Zhao · 2025

Read Alternative statistical inference for the first normalized incomplete moment by Jiannan Lu, Peng Ding and Anqi Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17145.txt.

Introduction

Inequality is often summarized by a curve based measure, the first normalized incomplete moment, but the statistics for trusting that summary lag behind modern data work. Jiannan Lu and two co-authors revisit the measure in 2025 and build a fresh way to draw inferences from it. Their method matches the standard answers where standards exist, yet stays simple to compute.

The gain shows in the awkward cases, where older tools strain. Simulations and real examples carry the demonstration. One finding warns practitioners directly: a common industry habit can quietly wreck the inference, steering decisions wrong. The authors present their approach as the measure's missing statistical engine.

Catalog record

CREATORJiannan Lu, Peng Ding and Anqi Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17145

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Alternative statistical inference for the first normalized incomplete moment Jiannan Lu, Peng Ding and Anqi Zhao Abstract This paper re-examines the first normalized incomplete moment, a well-established measure of inequality with wide applications in economic and social sciences. Despite the popularity of the measure itself, existing statistical inference appears to lag behind the needs of modern-age analytics. To fill this gap, we propose an alternative solution that is intuitive, computationally efficient, mathematically equivalent to the existing solutions for “standard” cases, and easily adaptable to “non-standard” ones. The theoretical and practical advantages of the proposed methodology are demonstrated via both simulated and real-life examples. In particular, we discover that a common practice in industry can lead to highly non-trivial challenges for trustworthy statistical inference, or misleading decision making altogether. Introduction 1.1 Background The past decade witnessed a surge in attention on the equality aspect of artificial intelligence, from recommendation system ( Speicher et al., 2018 ) to machine learning model performance benchmarking ( Lazovich et al., 2022 ) , to A/B testing ( Niculescu et al., 2021 ) . For example, Polonioli et al., 2023 stressed the need to account for all individuals in ML model evaluations, while Lum et al., 2022 pointed out the lack of rigorous measures (and the uncertainty quantifications thereof) for AI systems. From the economic and social science literature on income inequality ( Champernowne and Cowell, 1998 ) , one common measure is the share of income by the p th percentile ( Butler and McDonald, 1989 ) , which finds wide applications in economics and beyond, is free of arbitrary utility functions or “tuning parameters” (e.g., \epsilon in Atkinson’s Index), and is among the most intuitive and actionable in the context of data mining. As an example, Xie and Aurisset, 2016 investigated the correlation between streaming hours and user retention at Netflix, and concluded that “users with low hours are more likely to be on the fence of cancellation, and more sensitive to product changes.” Consequently, a measure of inequality can be share of streaming hours from the bottom 10% active users. Mathematically, let X\in(0,\infty) be a positive-valued random variable with continuous probability density function f(x) and cumulative distribution function F(x). In addition, we let \mu and \sigma^{2} denote the mean and variance of X. Worth noting that f,F,\mu and \sigma^{2} are unknown but may be estimated from the observed data. Given known constant p\in(0,1), the share of X at percentile q=F^{-1}(p) is m(q)=\frac{\mathbb{E}\left(X1_{\{X\leq q\}}\right)}{\mu}=\frac{\int_{0}^{q}sf(s)ds}{\int_{0}^{\infty}sf(s)ds}. (1) Remark 1 . ( 1 ) is the Lorenz curve co-ordinate at q, or as Butler and McDonald, 1989 referred to, the “first normalized incomplete moment.” It is closely related to other measures of inequality, e.g., the Pietra Index ( Eliazar and Sokolov, 2010 ) P=F(\mu)-m(\mu). and the Gini Index ( Lerman and Yitzhaki, 1984 ) G=\int_{0}^{\infty}f(s)\left\{\frac{sF(s)}{\mu}-m(s)\right\}ds.\qed Example 1 . We consider two distributions that are widely leveraged in data mining. First, for the log-normal distribution, let \Phi(x) denote the cumulative distribution function of the standard normal random variable. By definition, the cumulative distribution function of log-normal (\mu,\sigma) is F_{\mathrm{LN}(\mu,\sigma)}(x)=\Phi\left(\frac{\mathrm{ln}x-\mu}{\sigma}\right), resulting q=\mathrm{exp}\left\{\mu+\sigma\Phi^{-1}(p)\right\},\quad\mu=\mathrm{exp}\left(\mu+\sigma^{2}/2\right). For log-normal, ( 1 ) can be solved in closed-form ( Butler and McDonald, 1989 ) , namely, m(q)=F_{\mathrm{LN}(\mu+\sigma^{2},\sigma)}(q),\quad p\in(0,1). Second, consider the exponential distribution. Let f_{\lambda}(x)=\lambda\mathrm{e}^{-\lambda x} denote its probability density function, and F_{\mathrm{exp}(\lambda)}=1-\mathrm{e}^{-\lambda x}, denote its cumulative distribution function, which implies q=-\mathrm{ln}(1-p)/\lambda,\quad\mu=\lambda^{-1}, and m(q)=1-\mathrm{e}^{-\lambda q}(1+\lambda q),\quad q\in(0,\infty). It is worth noting that the scaling factor \lambda is not relevant here. ∎ 1.2 Literature review and contribution Although ( 1 ) is well-established, there are few discussions around its statistical inference. For example, Butler and McDonald, 1989 among many others focused on pre-specified values of q, but in reality we need to estimate q from the observed data – given independently and identically samples X_{1},\ldots,X_{n} from unknown distribution F, how to construct statistically valid point estimate and confidence interval for m(q)? From the literature, analytical ( Beach and Davidson, 1983 ) and bootstrap-based solutions ( Dupuis and Victoria-Feser, 2006 ) had been proposed. Unfortunately, however, they lag behind the needs of modern-age analytics (e.g., on-line A/B experiments). On the one hand, closed-form solutions tend to be unintuitive and less applicable to “non-standard” scenarios in industry. For example, when the sample size n is sufficiently large, should we treat \hat{q} as fixed and the true value for q? On the other hand, re-sampling is computationally expensive or even infeasible for big data ( Kleiner et al., 2014 ) . In light of the opportunities, we make a three-fold contribution to the community. First, we increase awareness to the potential of these measures of inequality in modern-age analytics. Second, we create an alternative solution that is intuitive, closed-form (therefore computationally efficient), mathematically equivalent to Beach and Davidson, 1983 in “standard” settings, and generalizable to others in practice. Third, we reveal a somewhat surprising insight – the often default and seemingly harmless exercise of assuming q\approx\hat{q} leads to highly non-trivial challenges for, or worse, outright misleading, statistical inference. The remainder of the paper is organized as follows. Section 2 employs m -estimation ( Newey and McFadden, 1994 ; Stefanski and Boos, 2002 ) to derive point estimate and confidence interval for m, highlighting why closed-form solutions are preferred, both theoretically and practically. Section 3 illustrates the proposed methodology using simulated and real-life examples. Section 4 concludes and presents future directions. Methodology 2.1 Theory and practice For any fixed p\in(0,1), we estimate q=F^{-1}(p) by its sample analogue \hat{q}=X_{\left(\lfloor np\rfloor\right),} resulting the following plug-in estimator for ( 1 ): \hat{m}(\hat{q})=\frac{\frac{1}{n}\sum_{i=1}^{n}X_{i}1_{\{X_{i}\leq\hat{q}\}}}{\frac{1}{n}\sum_{i=1}^{n}X_{i}}. (2) Its asymptotic consistency is guaranteed by \hat{q}\overset{p}{\rightarrow}q, and by applying the central limit theorem to both the denominator and the numerator of ( 2 ), respectively. To quantify the uncertainty of ( 2 ), although it is possible to leverage bootstrap, closed-form solutions are much preferred. Unfortunately, however, deriving closed-form variance estimate is challenging, because the sample percentile \hat{q} (as a non-linear function of X_{1},\ldots,X_{n} ) renders X_{1}1_{\{X_{1}\leq\hat{q}\}},\ldots,X_{n}1_{\{X_{n}\leq\hat{q}\}} “non-trivially” correlated. To overcome this challenge, we re-write ( 1 ) as \mathbb{E}\left\{X\mathrm{1}_{\{X\leq q\}}-m(q)X\right\}=0. (3) Solving the sample analogue of ( 3 ) yields ( 2 ). To infer uncertainty, we let Y=X\mathrm{1}_{\{X\leq q\}}-m(q)X,\quad Z=\mathrm{1}_{\{X\leq q\}}-p, and augment the estimating equation ( 3 ) as \bm{g}(X,m,q)=\left(Y,Z\right)^{T}, following the strategy of Newey, 1984 . It is trivial that \mathbb{E}\bm{g}=\bm{0}. We now present the main result of the paper. Proposition 1 . Let \bm{B}=\begin{bmatrix}\frac{\mathbb{E}\left\{(Y-qZ)^{2}\right\}}{\mu^{2}}&-\frac{\mathbb{E}\left(YZ\right)+qf(q)\mathbb{E}\left(Z^{2}\right)}{\mu f(q)}\\ -\frac{\mathbb{E}\left(YZ\right)+qf(q)\mathbb{E}\left(Z^{2}\right)}{\mu f(q)}&\frac{p(1-p)}{f^{2}(q)}\end{bmatrix}. The joint asymptotic distribution of \hat{m}(\hat{q}) and \hat{q} is \sqrt{n}\begin{pmatrix}\hat{m}(\hat{q})-m(q)\\ \hat{q}-q\end{pmatrix}\overset{d}{\rightarrow}\bm{N}\left\{\bm{0},\bm{B}\right\}. (4) ∎ Proof. The theory of m- estimation ensures that \sqrt{n}\begin{pmatrix}\hat{m}(\hat{q})-m(q)\\ \hat{q}-q\end{pmatrix}\overset{d}{\rightarrow}\bm{N}\left(\bm{0},\bm{B}\bm{M}\bm{B}^{T}\right), (5) where \bm{B}=\mathbb{E}\left\{\begin{pmatrix}\frac{\partial\bm{g}}{\partial m},&\frac{\partial\bm{g}}{\partial q}\end{pmatrix}^{-1}\right\}=\begin{bmatrix}-\mu&qf(q)\\ 0&f(q)\end{bmatrix}^{-1}=-\frac{1}{\mu f(q)}\begin{bmatrix}f(q)&-qf(q)\\ 0&-\mu\end{bmatrix}, (6) and \bm{M}=\mathbb{E}(\bm{g}\bm{g}^{T})=\begin{bmatrix}\mathbb{E}\left(Y^{2}\right)&\mathbb{E}\left(YZ\right)\\ \mathbb{E}\left(YZ\right)&\mathbb{E}\left(Z^{2}\right)\end{bmatrix}. (7) Combining ( 5 )–( 7 ), \displaystyle\bm{B}\bm{M}\bm{B}^{T} \displaystyle=\frac{1}{\left(f(q)\mu\right)^{2}}\begin{bmatrix}f(q)&-qf(q)\\ 0&-\mu\end{bmatrix}\begin{bmatrix}\mathbb{E}\left(Y^{2}\right)&\mathbb{E}\left(YZ\right)\\ \mathbb{E}\left(YZ\right)&\mathbb{E}\left(Z^{2}\right)\end{bmatrix}\begin{bmatrix}f(q)&0\\ -qf(q)&-\mu\end{bmatrix} \displaystyle=\begin{bmatrix}\frac{f(q)\mathbb{E}\left(Y^{2}\right)-qf(q)\mathbb{E}\left(YZ\right)}{\left(f(q)\mu\right)^{2}}&\frac{f(q)\mathbb{E}\left(YZ\right)-qf(q)\mathbb{E}\left(Z^{2}\right)}{\left(f(q)\mu\right)^{2}}\\ -\frac{\mu\mathbb{E}\left(YZ\right)}{\left(f(q)\mu\right)^{2}}&-\frac{\mu\mathbb{E}\left(Z^{2}\right)}{\left(f(q)\mu\right)^{2}}\end{bmatrix}\begin{bmatrix}f(q)&0\\ -qf(q)&-\mu\end{bmatrix} \displaystyle=\begin{bmatrix}\frac{\mathbb{E}\left(Y^{2}\right)-2q\mathbb{E}\left(YZ\right)+q^{2}\mathbb{E}\left(Z^{2}\right)}{\mu^{2}}&-\frac{\mu\mathbb{E}\left(YZ\right)+qf(q)\mu\mathbb{E}\left(Z^{2}\right)}{f(q)\mu^{2}}\\ -\frac{\mu\mathbb{E}\left(YZ\right)+qf(q)\mu\mathbb{E}\left(Z^{2}\right)}{f(q)\mu^{2}}&\frac{\left(\mu\right)^{2}\mathbb{E}\left(Z^{2}\right)}{f^{2}(q)\mu^{2}}\end{bmatrix} \displaystyle=\begin{bmatrix}\frac{\mathbb{E}\left\{(Y-qZ)^{2}\right\}}{\mu^{2}}&-\frac{\mathbb{E}\left(YZ\right)+qf(q)\mathbb{E}\left(Z^{2}\right)}{\mu f(q)}\\ -\frac{\mathbb{E}\left(YZ\right)+qf(q)\mathbb{E}\left(Z^{2}\right)}{\mu f(q)}&\frac{\mathbb{E}\left(Z^{2}\right)}{f^{2}(q)}\end{bmatrix}. To complete the proof, note that \mathbb{E}\left(Z^{2}\right)=p(1-p) because \mathrm{1}_{\{X\leq q\}}\sim\mathrm{Bern}(p).\qed ∎ Now that the theoretical foundation is established, we can present the variance estimator for ( 2 ). For i=1,\ldots,n, let \hat{Y}_{i}=X_{i}\mathrm{1}_{\{X_{i}\leq\hat{q}\}}-\hat{m}\left(\hat{q}\right)X_{i},\quad\hat{Z}_{i}=\mathrm{1}_{\{X_{i}\leq\hat{q}\}}-p. ( 4 ) implies the following plug-in variance estimator: \displaystyle\hat{\mathbb{V}}_{\mathrm{a}}\left\{\hat{m}(\hat{q})\right\}=\frac{\sum_{i=1}^{n}(\hat{Y}_{i}-\hat{q}\hat{Z}_{i})^{2}}{\left(\sum_{i=1}^{n}X_{i}\right)^{2}}. (8) Remark 2 . We highlight the following key benefits of the proposed alternative defined in ( 4 )–( 8 ): 1. Validity : As a sanity check, ( 4 ) suggests the textbook result ( Van der Vaart,

Read the complete text