On Bessel’s Correction: Unbiased Sample Variance, the Bariance , and a Novel Runtime-Optimized Estimator

Felix Reichel · 2025

Read On Bessel’s Correction: Unbiased Sample Variance, the Bariance , and a Novel Runtime-Optimized Estimator by Felix Reichel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22333.txt.

Introduction

Felix Reichel revisits in 2025 Bessel's correction. It is the switch from dividing by n to dividing by n minus one. That switch makes sample variance unbiased. He gives rigorous algebra for why the switch is needed. He also gives geometric pictures and charts for the point. He then introduces a new spread measure. It is built from pairwise squared differences. It does not rely on the sample mean.

The paper answers an earlier argument that favored n-based estimates on error grounds. That case was made for teaching contexts and for specific applied settings. Simulation studies show the new measure can be found in linear time. That yields faster runtimes while keeping the estimator unbiased. The result serves as both a teaching aid and a practical tool.

Catalog record

CREATORFelix Reichel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Bessel’s Correction: Unbiased Sample Variance, the Bariance , and a Novel Runtime-Optimized Estimator Felix Reichel Abstract Bessel’s correction adjusts the denominator in the sample variance formula from n n to n − 1 n-1 to ensure an unbiased estimator of the population variance. This paper provides rigorous algebraic derivations, geometric interpretations, and visualizations to reinforce the necessity of this correction. It further introduces the concept of Bariance , an alternative dispersion measure based on pairwise squared differences that avoids reliance on the arithmetic mean. Building on this, we address practical concerns raised in Rosenthal’s article [ 8 ] , which advocates for n n -based estimates from a mean squared error (MSE) perspective—particularly in pedagogical contexts and specific applied settings. Finally, the empirical component of this work, based on simulation studies, demonstrates that estimating the population variance via an algebraically optimized Bariance approach can yield a computational advantage. Specifically, the unbiased Bariance estimator can be computed in linear time, resulting in shorter run-times while preserving statistical validity. JEL Codes: C10, C80 Keywords: Unbiased sample variance, Runtime-optimized linear unbiased sample variance estimators “This could be the end of it all, released from the pain. So self-indulgent and so insincere, I’ll never bow to your lies again.” — Megadeth, *Tornado of Souls* The following notation is used throughout the paper: X_{1},X_{2},\dots,X_{n}\in\mathbb{R} . Independent and identically distributed sample from a population. \mu=\E[X_{i}] . Population mean. \sigma^{2}=\Var(X_{i}) . Population variance. \bar{X}=\frac{1}{n}\sum_{i=1}^{n}X_{i} . Sample mean. S^{2}=\frac{1}{n}\sum_{i=1}^{n}(X_{i}-\bar{X})^{2} . Biased sample variance estimator (denominator n ). \hat{S}^{2}=\frac{1}{n-1}\sum_{i=1}^{n}(X_{i}-\bar{X})^{2} . Unbiased sample variance estimator (Bessel-corrected). \text{{Bariance}}=\frac{1}{n(n-1)}\sum_{i\neq j}(X_{i}-X_{j})^{2} . Pairwise variance estimator using all ordered pairs (each unordered pair counted twice). \text{{Bariance}}_{\text{opt}}=\frac{2n}{n(n-1)}\sum_{i=1}^{n}X_{i}^{2}-\frac{2}{n(n-1)}\left(\sum_{i=1}^{n}X_{i}\right)^{2} . Optimized scalar formula for Bariance . \sum_{i<j}(X_{i}-X_{j})^{2} . Sum over all unordered pairwise squared differences (each pair counted once). Note: \sum_{i\neq j}(X_{i}-X_{j})^{2}=2\sum_{i<j}(X_{i}-X_{j})^{2} . \text{Bias}(\hat{\theta})=\E[\hat{\theta}]-\theta . Bias of an estimator \hat{\theta} . \text{Var}(\hat{\theta})=\E[(\hat{\theta}-\E[\hat{\theta}])^{2}] . Variance of an estimator. \text{MSE}(\hat{\theta})=\text{Var}(\hat{\theta})+\text{Bias}^{2}(\hat{\theta}) . Mean squared error of an estimator. \vec{X}\in\mathbb{R}^{n} . Sample vector in Euclidean space. \vec{1}\in\mathbb{R}^{n} . All-ones vector. \vec{r}=\vec{X}-\vec{\mu} . Residual vector after projection onto the mean. \vec{r}\in\vec{1}^{\perp} . Residual lies in the orthogonal complement of the mean vector. \Gamma(k,\theta) . Gamma distribution with shape k and scale \theta . \mathcal{N}(\mu,\sigma^{2}) . Normal distribution with mean \mu and variance \sigma^{2} . \sum X_{i} . Sum of sample values. \sum X_{i}^{2} . Sum of squared sample values. Note: The term Bariance emphasizes variance computed from pairwise squared differences instead of deviations from the mean. 1 Introduction and Motivation Variance estimation is a foundational task in statistics and econometrics, with the sample variance being the default estimator in most applications. The unbiased version, corrected by Bessel’s factor (dividing by n-1 rather than n ), compensates for the loss of one degree of freedom due to pre-estimating the population mean. This correction is not just a simple algebraic trick—it admits deep geometric interpretations via orthogonal projections in \mathbb{R}^{n} and can be derived rigorously from them. Despite its theoretical appeal, the unbiased estimator is not always the most optimal in practice. In small samples especially, its higher variance may lead to suboptimal inference. This has led researchers to consider shrunken estimators that intentionally trade off a small amount of bias for a significant reduction in variance, thereby minimizing mean squared error (MSE). For example, empirical Bayes methods shrink sample variances toward a global prior, stabilizing estimation across thousands of features in genomic studies [ 9 ] . Similar techniques based on James–Stein shrinkage have been explored for variance estimation in high-dimensional settings [ 4 ] . Beyond the univariate case, shrinkage ideas are especially powerful in multivariate settings. In particular, shrinkage estimators for covariance matrices—such as the Ledoit–Wolf estimator [ 6 ] —have gained popularity in fields like econometrics and finance, particularly in the field of asset pricing. These estimators enhance the stability of sample covariance matrices by shrinking them toward structured targets (e.g., the identity matrix), significantly improving conditioning in high-dimensional models, which are known to perform poorly [ 5 ] . This has practical relevance in the construction of variance-covariance matrices for portfolio optimization, factor models, and robust standard error estimation in large-scale regression analysis for econometric applications. In this broader context, this paper revisits classical variance estimation and introduces a novel perspective via an alternative measure of sample dispersion based on the average squared differences between all unordered pairs in a sample. We formally define this estimator as the Bariance , a term that reflects its construction from pairwise distances rather than deviations from a mean. It can be shown that for mean-centered data, the Bariance equals exactly twice the unbiased sample variance . Moreover, a linear-time optimized formulation of the Bariance can be derived using simple algebraic properties that avoids quadratic pairwise computation, making it both theoretically elegant and computationally efficient. Although the pairwise difference approach has roots in classical statistics—such as U-statistics [ 7 ] , dissimilarity-based dispersion measures, and even the Gini coefficient [ 2 ] —the contribution here is a novel, unbiased estimator that is computationally optimized for runtime efficiency. In this respect, Bariance bridges theoretical variance estimation with algorithmic efficiency, a consideration critical in big data contexts, real-time systems, and streaming analytics. While computational efficiency is one of its key advantages, the Bariance measure may also prove valuable in applied scenarios where the concept of central tendency is unstable, ill-defined, or misleading. For example, in domains such as network analysis, genomics, ordinal survey research, or clustering, statistical dispersion is often better captured through relational or pairwise structures rather than deviations from a single global mean. In such contexts, the Bariance shares conceptual kinship with the Gini coefficient, which also operates on pairwise differences but in a distributional inequality framework. Unlike Gini, however, Bariance preserves unbiasedness for variance estimation under i.i.d. sampling and scales naturally in high-dimensional or streaming environments. These features make it particularly attractive for modern applications in unsupervised learning, robust statistics, and high-throughput data pipelines—where traditional variance measures may either fail or become computationally prohibitive. Through an empirical simulation study, I demonstrate that this optimized unbiased sample variance estimator remains unbiased and improves runtime. The simulated empirical runtimes section includes confidence intervals, hardware specifications, seed initialization, and multiple replications, thereby addressing robustness, reproducibility, and statistical reliability. Furthermore, Appendix C replicates the result in a local hardware environment using an alternative high-level programming language. We then revisit the controversial idea—advocated by Rosenthal [ 8 ] —that dividing by n (rather than n-1 ) may yield lower-MSE variance estimators in practice, especially when unbiasedness is not strictly required. To sum up, the Bariance framework bridges computational efficiency with applied relevance, offering a theoretically grounded yet practically flexible alternative to traditional variance estimators. This paper thus aims to bridge classical econometric and statistical theory with modern considerations of efficiency, robustness, and computational scalability, while highlighting the often underestimated choices in estimator design or usage. 2 Definitions and Setup Let X_{1},X_{2},\dots,X_{n}\in\mathbb{R} be i.i.d. random variables with: \E[X_{i}]=\mu,\qquad\Var(X_{i})=\sigma^{2} Define the sample mean and biased/unbiased variance estimates: \bar{X}:=\frac{1}{n}\sum_{i=1}^{n}X_{i},\quad S^{2}:=\frac{1}{n}\sum_{i=1}^{n}(X_{i}-\bar{X})^{2},\quad\hat{S}^{2}:=\frac{1}{n-1}\sum_{i=1}^{n}(X_{i}-\bar{X})^{2} 3 Derivation of Bias and Bessel’s Correction An estimator \hat{\theta} for a parameter \theta is called unbiased if its expected value equals the true value: \E[\hat{\theta}]=\theta The normal n -based sample variance with denominator n is defined as: S^{2}:=\frac{1}{n}\sum_{i=1}^{n}(X_{i}-\bar{X})^{2}

Read the complete text